Deploy a flow to online endpoint for real-time inference with CLI

In this article, you learn how to deploy your flow to a managed online endpoint or a Kubernetes online endpoint for use in real-time inferencing by using Azure Machine Learning v2 CLI.

Before you begin, make sure that you test your flow properly and feel confident that it's ready to be deployed to production. To learn more about testing your flow, see test your flow. After testing your flow, you learn how to create managed online endpoint and deployment, and how to use the endpoint for real-time inferencing.

• This article covers how to use the CLI experience.
• The Python SDK isn't covered in this article. See the GitHub sample notebook instead. To use the Python SDK, you must have The Python SDK v2 for Azure Machine Learning. To learn more, see Install the Python SDK v2 for Azure Machine Learning.

Prerequisites

• The Azure CLI and the Azure Machine Learning extension to the Azure CLI. For more information, see Install, set up, and use the CLI (v2).
• An Azure Machine Learning workspace. If you don't have one, use the steps in the Quickstart: Create workspace resources article to create one.
• Azure role-based access controls (Azure RBAC) are used to grant access to operations in Azure Machine Learning. To perform the steps in this article, your user account must be assigned the owner or contributor role for the Azure Machine Learning workspace, or a custom role allowing "Microsoft.MachineLearningServices/workspaces/onlineEndpoints/". If you use studio to create and manage online endpoints and deployments, you need another permission "Microsoft.Resources/deployments/write" from the resource group owner. For more information, see Manage access to an Azure Machine Learning workspace.

Virtual machine quota allocation for deployment

For managed online endpoints, Azure Machine Learning reserves 20% of your compute resources for performing upgrades. Therefore, if you request a given number of instances in a deployment, you must have a quota for ceil(1.2 * number of instances requested for deployment) * number of cores for the VM SKU available to avoid getting an error. For example, if you request 10 instances of a Standard_DS3_v2 VM (that comes with four cores) in a deployment, you should have a quota for 48 cores (12 instances four cores) available. To view your usage and request quota increases, see View your usage and quotas in the Azure portal.

Get the flow ready for deploy

Each flow has a folder that contains codes, prompts, definition, and other artifacts of the flow. If you develop your flow by using the UI, you can download the flow folder from the flow details page. If you develop your flow by using CLI or SDK, you already have the flow folder.

This article uses the sample flow "basic-chat" as an example to deploy to Azure Machine Learning managed online endpoint.

Set default workspace

Use the following commands to set the default workspace and resource group for the CLI.

az account set --subscription <subscription ID>
az configure --defaults workspace=<Azure Machine Learning workspace name> group=<resource group>

Register the flow as a model (optional)

In the online deployment, you can either refer to a registered model or specify the model path (where to upload the model files from) inline. Register the model and specify the model name and version in the deployment definition. Use the form model:<model_name>:<version>.

The following example shows a model definition for a chat flow.

$schema: https://azuremlschemas.azureedge.net/latest/model.schema.json
name: basic-chat-model
path: ../../../../examples/flows/chat/basic-chat
description: register basic chat flow folder as a custom model
properties:
  # In AuzreML studio UI, endpoint detail UI Test tab needs this property to know it's from prompt flow
  azureml.promptflow.source_flow_id: basic-chat
  
  # Following are properties only for chat flow 
  # endpoint detail UI Test tab needs this property to know it's a chat flow
  azureml.promptflow.mode: chat
  # endpoint detail UI Test tab needs this property to know which is the input column for chat flow
  azureml.promptflow.chat_input: question
  # endpoint detail UI Test tab needs this property to know which is the output column for chat flow
  azureml.promptflow.chat_output: answer

Use az ml model create --file model.yaml to register the model to your workspace.

Define the endpoint

To define an endpoint, specify the following values:

• Endpoint name: The name of the endpoint. It must be unique in the Azure region. For more information on the naming rules, see endpoint limits.
• Authentication mode: The authentication method for the endpoint. Choose between key-based authentication and Azure Machine Learning token-based authentication. A key doesn't expire, but a token does expire. For more information on authenticating, see Authenticate to an online endpoint. Optionally, add a description and tags to your endpoint.
• Optionally, add a description and tags to your endpoint.
• If you want to deploy to a Kubernetes cluster (AKS or Arc enabled cluster) that you attach to your workspace, you can deploy the flow as a Kubernetes online endpoint.

The following example shows an endpoint definition that uses system-assigned identity by default.

$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineEndpoint.schema.json
name: basic-chat-endpoint
auth_mode: key
properties:
# this property only works for system-assigned identity.
# if the deploy user has access to connection secrets, 
# the endpoint system-assigned identity will be auto-assigned connection secrets reader role as well
  enforce_access_to_default_secret_stores: enabled

$schema: https://azuremlschemas.azureedge.net/latest/kubernetesOnlineEndpoint.schema.json
name: basic-chat-endpoint
compute: azureml:<Kubernetes compute name>
auth_mode: key

If you create a Kubernetes online endpoint, you need to specify the following attributes:

For more configurations of endpoint, see managed online endpoint schema.

Use user-assigned identity

By default, when you create an online endpoint, the system automatically generates a system-assigned managed identity for you. You can also specify an existing user-assigned managed identity for the endpoint.

To use a user-assigned identity, specify the following attributes in the endpoint.yaml file:

identity:
  type: user_assigned
  user_assigned_identities:
    - resource_id: user_identity_ARM_id_place_holder

Also, specify the Client ID of the user-assigned identity under environment_variables in the deployment.yaml file as shown in the following example. You can find the Client ID in the Overview of the managed identity in the Azure portal.

environment_variables:
  AZURE_CLIENT_ID: <client_id_of_your_user_assigned_identity>

Define the deployment

A deployment is a set of resources required for hosting the model that does the actual inferencing.

The following example shows a deployment definition. The model section refers to the registered flow model. You can also specify the flow model path in line.

$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
name: blue
endpoint_name: basic-chat-endpoint
model: azureml:basic-chat-model:1
  # You can also specify model files path inline
  # path: examples/flows/chat/basic-chat
environment: 
  image: mcr.microsoft.com/azureml/promptflow/promptflow-runtime:latest
  # inference config is used to build a serving container for online deployments
  inference_config:
    liveness_route:
      path: /health
      port: 8080
    readiness_route:
      path: /health
      port: 8080
    scoring_route:
      path: /score
      port: 8080
instance_type: Standard_E16s_v3
instance_count: 1
environment_variables:
  # for pulling connections from workspace
  PRT_CONFIG_OVERRIDE: deployment.subscription_id=<subscription_id>,deployment.resource_group=<resource_group>,deployment.workspace_name=<workspace_name>,deployment.endpoint_name=<endpoint_name>,deployment.deployment_name=<deployment_name>

  # (Optional) When there are multiple fields in the response, using this env variable will filter the fields to expose in the response.
  # For example, if there are 2 flow outputs: "answer", "context", and I only want to have "answer" in the endpoint response, I can set this env variable to '["answer"]'.
  # If you don't set this environment, by default all flow outputs will be included in the endpoint response.
  # PROMPTFLOW_RESPONSE_INCLUDED_FIELDS: '["category", "evidence"]'

$schema: https://azuremlschemas.azureedge.net/latest/kubernetesOnlineDeployment.schema.json
name: blue
type: kubernetes
endpoint_name: basic-chat-endpoint
model: azureml:basic-chat-model:1
  # You can also specify model files path inline
  # path: examples/flows/chat/basic-chat
environment: 
  image: mcr.microsoft.com/azureml/promptflow/promptflow-runtime:latest
  # inference config is used to build a serving container for online deployments
  inference_config:
    liveness_route:
      path: /health
      port: 8080
    readiness_route:
      path: /health
      port: 8080
    scoring_route:
      path: /score
      port: 8080
instance_type: <kubernetes custom instance type>
instance_count: 1
environment_variables:

  # for pulling connections from workspace
  PRT_CONFIG_OVERRIDE: deployment.subscription_id=<subscription_id>,deployment.resource_group=<resource_group>,deployment.workspace_name=<workspace_name>,deployment.endpoint_name=<endpoint_name>,deployment.deployment_name=<deployment_name>

  # (Optional) When there are multiple fields in the response, using this env variable will filter the fields to expose in the response.
  # For example, if there are 2 flow outputs: "answer", "context", and I only want to have "answer" in the endpoint response, I can set this env variable to '["answer"]'.
  # If you don't set this environment, by default all flow outputs will be included in the endpoint response.
  # PROMPTFLOW_RESPONSE_INCLUDED_FIELDS: '["category", "evidence"]'

If you create a Kubernetes online deployment, specify the following attributes:

Deploy your online endpoint to Azure

To create the endpoint in the cloud, run the following code:

az ml online-endpoint create --file endpoint.yml

To create the deployment named blue under the endpoint, run the following code:

az ml online-deployment create --file blue-deployment.yml --all-traffic

Check status of the endpoint and deployment

To check the status of the endpoint, run the following code:

az ml online-endpoint show -n basic-chat-endpoint

To check the status of the deployment, run the following code:

az ml online-deployment get-logs --name blue --endpoint basic-chat-endpoint

Invoke the endpoint to score data by using your model

Create a sample-request.json file:

{
  "question": "What is Azure Machine Learning?",
  "chat_history":  []
}

az ml online-endpoint invoke --name basic-chat-endpoint --request-file sample-request.json

You can also call the endpoint by using an HTTP client, such as curl:

ENDPOINT_KEY=<your-endpoint-key>
ENDPOINT_URI=<your-endpoint-uri>

curl --request POST "$ENDPOINT_URI" --header "Authorization: Bearer $ENDPOINT_KEY" --header 'Content-Type: application/json' --data '{"question": "What is Azure Machine Learning?", "chat_history":  []}'

Get your endpoint key and your endpoint URI from the Azure Machine Learning workspace in Endpoints > Consume > Basic consumption info.

Advanced configurations

Deploy with different connections from flow development

You might want to override connections of the flow during deployment.

For example, if your flow.dag.yaml file uses a connection named my_connection, you can override it by adding environment variables of the deployment yaml like following:

Option 1: override connection name

environment_variables:
  my_connection: <override_connection_name>

If you want to override a specific field of the connection, you can override by adding environment variables with naming pattern <connection_name>_<field_name>. For example, if your flow uses a connection named my_connection with a configuration key called chat_deployment_name, the serving backend attempts to retrieve chat_deployment_name from the environment variable 'MY_CONNECTION_CHAT_DEPLOYMENT_NAME' by default. If the environment variable isn't set, it uses the original value from the flow definition.

Option 2: override by referring to asset

environment_variables:
  my_connection: ${{azureml://connections/<override_connection_name>}}

Deploy with a custom environment

This section shows you how to use a Docker build context to specify the environment for your deployment, assuming you have knowledge of Docker and Azure Machine Learning environments.

1. In your local environment, create a folder named image_build_with_reqirements that contains the following files:

   |--image_build_with_reqirements
   |  |--requirements.txt
   |  |--Dockerfile
   

   • The requirements.txt file, inherited from the flow folder, tracks the dependencies of the flow.

   • The Dockerfile with content similar to the following example:

     FROM mcr.microsoft.com/azureml/promptflow/promptflow-runtime:latest
     COPY ./requirements.txt .
     RUN pip install -r requirements.txt
     

2. Replace the environment section in the deployment definition YAML file with the following content:

   environment: 
     build:
       path: image_build_with_reqirements
       dockerfile_path: Dockerfile
     # deploy prompt flow is BYOC, so we need to specify the inference config
     inference_config:
       liveness_route:
         path: /health
         port: 8080
       readiness_route:
         path: /health
         port: 8080
       scoring_route:
         path: /score
         port: 8080
   

Use FastAPI serving engine (preview)

By default, prompt flow serving uses the FLASK serving engine. Starting from prompt flow SDK version 1.10.0, FastAPI-based serving engine is supported. You can use the fastapi serving engine by specifying an environment variable PROMPTFLOW_SERVING_ENGINE.

environment_variables:
  PROMPTFLOW_SERVING_ENGINE=fastapi

Configure concurrency for deployment

When you deploy your flow to online deployment, configure two environment variables for concurrency: PROMPTFLOW_WORKER_NUM and PROMPTFLOW_WORKER_THREADS. You also need to set the max_concurrent_requests_per_instance parameter.

The following example shows how to configure these settings in the deployment.yaml file.

request_settings:
  max_concurrent_requests_per_instance: 10
environment_variables:
  PROMPTFLOW_WORKER_NUM: 4
  PROMPTFLOW_WORKER_THREADS: 1

• PROMPTFLOW_WORKER_NUM: This parameter sets the number of workers (processes) that start in one container. The default value equals the number of CPU cores, and the maximum value is twice the number of CPU cores.

• PROMPTFLOW_WORKER_THREADS: This parameter sets the number of threads that start in one worker. The default value is 1.

  Note

  When you set PROMPTFLOW_WORKER_THREADS to a value greater than 1, make sure your flow code is thread-safe.

• max_concurrent_requests_per_instance: The maximum number of concurrent requests per instance allowed for the deployment. The default value is 10.

  The suggested value for max_concurrent_requests_per_instance depends on your request time:

  • If your request time is greater than 200 ms, set max_concurrent_requests_per_instance to PROMPTFLOW_WORKER_NUM * PROMPTFLOW_WORKER_THREADS.
  • If your request time is less than or equal to 200 ms, set max_concurrent_requests_per_instance to (1.5-2) * PROMPTFLOW_WORKER_NUM * PROMPTFLOW_WORKER_THREADS. This setting can improve total throughput by allowing some requests to queue on the server side.
  • If you're sending cross-region requests, you can change the threshold from 200 ms to 1 s.

While tuning these parameters, monitor the following metrics to ensure optimal performance and stability:

• Instance CPU and memory utilization for this deployment
• Non-200 responses (4xx, 5xx)
  • If you receive a 429 response, this status code typically indicates that you need to either retune your concurrency settings following the preceding guide or scale your deployment.
• Azure OpenAI throttle status

Monitor endpoints

Collect general metrics

You can view general metrics of online deployment (request numbers, request latency, network bytes, CPU/GPU/Disk/Memory utilization, and more).

Collect tracing data and system metrics during inference time

You can collect tracing data and prompt flow deployment specific metrics (token consumption, flow latency, and more) during inference time to workspace linked Application Insights by adding a property app_insights_enabled: true in the deployment yaml file. For more information, see trace and metrics of prompt flow deployment.

You can specify prompt flow specific metrics and trace to other Application Insights rather than the workspace linked one. You can specify an environment variable in the deployment yaml file as following. You can find the connection string of your Application Insights in the Overview page in Azure portal.

environment_variables:
  APPLICATIONINSIGHTS_CONNECTION_STRING: <connection_string>

Common errors

Upstream request timeout issue when consuming the endpoint

This error usually happens because of a timeout. By default, the request_timeout_ms value is 5,000 milliseconds. You can set it up to 5 minutes, which is 300,000 milliseconds. The following example shows how to specify the request timeout in the deployment YAML file. For more information about the deployment schema, see Managed online deployment schema.

request_settings:
  request_timeout_ms: 300000

properties:
  # indicate a deployment from prompt flow
  azureml.promptflow.source_flow_id: <value>

Next steps

• Learn more about managed online endpoint schema and managed online deployment schema.
• Learn more about how to test the endpoint in UI and monitor the endpoint.
• Learn more about how to troubleshoot managed online endpoints.
• Troubleshoot prompt flow deployments.
• To deploy an improved version of your flow by using a safe rollout strategy, see Safe rollout for online endpoints.
• Learn more about deploy flows to other platforms, such as a local development service, Docker container, Azure APP service, etc.