Uredi

Monitor generative AI applications (preview)

Warning

Prompt flow in Microsoft Foundry and Azure Machine Learning will be retired on April 20, 2027. Prompt flow is no longer recommended for new development. Migrate existing Prompt flow applications and deployments to Microsoft Agent Framework before April 20, 2027.

Prompt flow container images are no longer receiving updates, including security and package updates. This applies to Prompt flow runtime images, including promptflow-runtime, promptflow-runtime-stable, and promptflow-python.

After April 20, 2027, Prompt flow, including the web authoring experience in Microsoft Foundry and Azure Machine Learning, the VS Code extensions, and related Prompt flow container images, will no longer be supported or available.

If your application depends on Prompt flow deployments or runtime images, plan to move those workloads to supported alternatives such as Microsoft Agent Framework before the retirement date. For migration guidance, see the Prompt flow migration guide and migration code samples.

Use Azure Machine Learning model monitoring to track the quality of an existing generative AI application deployed to a managed online endpoint. The monitor evaluates collected prompts and completions on a recurring schedule and displays metric trends and alerts in Azure Machine Learning studio.

Prompt Flow deployments on Azure Machine Learning managed online endpoints don't have a direct Microsoft Agent Framework equivalent. Review the migration options before you change an existing deployment.

Important

Model monitoring for generative AI applications is currently in public preview. This preview is provided without a service-level agreement and isn't recommended for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

Prerequisites

  • An Azure Machine Learning workspace.
  • An Azure OpenAI resource with quota for an evaluator model deployment. Model availability changes over time. Review Azure OpenAI model retirements and deprecations before you create the deployment.
  • A user-assigned managed identity attached to the workspace. Grant the identity the least-privileged permissions required to access the workspace and the connected Azure OpenAI resource. To create role assignments, your identity needs the Microsoft.Authorization/roleAssignments/write permission at the required scope.
  • An Azure OpenAI workspace connection that uses the managed identity. Keep this connection for as long as the monitor uses it.
  • An existing prompt flow deployment on an Azure Machine Learning managed online endpoint. This procedure is intended for workloads that you maintain until you migrate from prompt flow.

Prepare the deployment

Configure the deployed flow to collect the inputs and outputs required by the monitoring metrics:

  1. Create a compute session, and run your prompt flow.
  2. Deploy the flow to a managed online endpoint.
  3. Under Basic settings, enable Inference data collection.
  4. Under Advanced settings > Outputs & Connections, select completion and any outputs required by the metrics that you plan to use, such as context or ground_truth.
  5. Confirm that the collected input data contains a prompt column and that the collected output data contains a completion column. Record the exact column names for the monitor configuration.

Note

If your compute instance is behind a VNet, see Network isolation in prompt flow.

Understand evaluation metrics

The monitor uses an Azure OpenAI model deployment as an evaluator. It applies evaluation instructions to the collected prompt, completion, and supporting data. For detailed metric definitions and interpretation guidance, see Monitoring evaluation metrics descriptions and use cases.

Metric Prompt Completion Context Ground truth
Coherence Required Required Not required Not required
Fluency Required Required Not required Not required
Groundedness Required Required Required Not required
Relevance Required Required Required Not required
Similarity Required Required Not required Required

Create the monitor

Create a generation safety and quality signal from the data collected by your deployed flow:

  1. In Azure Machine Learning studio, go to the monitoring overview for your workspace, and create a model monitor.
  2. For the model task type, select Prompt & completion.
  3. Select the input and output data assets created by Model Data Collector, and add a generation safety and quality monitoring signal.
  4. Select the workspace connection for the Azure OpenAI evaluator resource, enter the evaluator deployment name, and map the prompt and completion columns. Map the context or ground-truth columns required by the metrics you select.
  5. Keep the automatically generated join configuration unless your data assets use a custom identifier. Model Data Collector generates correlationid to correlate input and output records. Confirm the exact field name in each data asset before you customize the join.
  6. Configure the available metric thresholds, sampling rate, schedule, and notification recipients. Then review the configuration, and create the monitor.

Confirm the monitoring status

On the monitor overview, confirm that the scheduled monitoring pipeline job completes successfully. If the job fails, review its logs and verify the workspace connection, evaluator deployment, data assets, and mapped column names.

Review monitoring results

Use the signal details to review metric trends and respond to alerts:

  1. On the monitor overview, review the status and performance of each monitoring signal.
  2. Open the generation safety and quality signal to view metric trends and score distributions.
  3. Review alerts for metrics that don't meet the configured thresholds.
  4. Adjust the thresholds or notification settings when your monitoring requirements change.

Related content

  • Last updated on