Quickstart: Evaluate your hosted agent

From your azd project directory, verify the agent is deployed and invokable:

azd ai agent show

Send a test prompt:

azd ai agent invoke "Write a haiku about deploying cloud applications."

You should see a response within a few seconds.

1. Open the Foundry portal and go to your project.
2. Select your agent, and then select the Playground tab.
3. Send a test prompt, such as Write a haiku about deploying cloud applications.

You should see a response within a few seconds.

Install the Foundry SDK:

pip install "azure-ai-projects>=2.0.0" azure-identity

Set two environment variables, and then create the project client. Set FOUNDRY_PROJECT_ENDPOINT to your project endpoint and FOUNDRY_MODEL_NAME to a chat-completion deployment to use as the judge model. The following code samples assume you run them in this context:

import os
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"]
model_deployment = os.environ["FOUNDRY_MODEL_NAME"]

credential = DefaultAzureCredential()
project_client = AIProjectClient(endpoint=endpoint, credential=credential)
client = project_client.get_openai_client()

Confirm your deployed agent is registered and available. Replace <your-agent-name> with your hosted agent's name:

agent = project_client.agents.get("<your-agent-name>")
print(f"Found agent: {agent.name}")

The call returns the agent if it exists, or raises an error if the name is wrong or the agent isn't deployed.

Install the Foundry SDK:

npm install @azure/ai-projects @azure/identity dotenv

Set two environment variables, and then create the project client. Set FOUNDRY_PROJECT_ENDPOINT to your project endpoint and FOUNDRY_MODEL_NAME to a chat-completion deployment to use as the judge model. The following code samples assume you run them in this context:

import { DefaultAzureCredential } from "@azure/identity";
import { AIProjectClient } from "@azure/ai-projects";
import "dotenv/config";

const endpoint = process.env["FOUNDRY_PROJECT_ENDPOINT"] || "";
const modelDeployment = process.env["FOUNDRY_MODEL_NAME"] || "";

const projectClient = new AIProjectClient(
  endpoint,
  new DefaultAzureCredential(),
);
const client = projectClient.getOpenAIClient();

Confirm your deployed agent is registered and available. Replace <your-agent-name> with your hosted agent's name:

const agent = await projectClient.agents.get("<your-agent-name>");
console.log(`Found agent: ${agent.name}`);

The call returns the agent if it exists, or raises an error if the name is wrong or the agent isn't deployed.

Reference: AIProjectClient class

First, create a JSONL file of test queries for your agent. Each line is a JSON object with a query field. Save it inside your agent's source folder, as src/<your-agent-name>/tests/queries.jsonl:

{"query": "Write a haiku about deploying cloud applications."}

Then create an eval.yaml file in the same agent source folder, as src/<your-agent-name>/eval.yaml. It points to your dataset and lists the built-in evaluators to apply. The dataset.local_uri path is relative to this folder. Replace <your-agent-name> with your hosted agent's name and <your-chat-completion-deployment> with the judge model deployment:

name: agent-eval
agent:
  name: <your-agent-name>
  kind: hosted
dataset:
  local_uri: tests/queries.jsonl
evaluators:
  - builtin.intent_resolution
  - builtin.task_adherence
options:
  eval_model: <your-chat-completion-deployment>
max_samples: 15

The eval_model value is the judge model that scores responses; you can reuse the deployment your agent already uses.

1. In the Foundry portal, open your agent and select the Evaluation tab, then select Create.
2. For Select evaluation target, select Agent.
3. For Select evaluation scope, select Individual turns.
4. For Select data source, select Existing dataset and choose a CSV or JSONL file of test queries from your project's data assets.
5. If the Configure agents step appears, review the agent and accept the default user prompt, {{item.query}}. Adjust it only if your agent expects a different input format.
6. For Select testing criteria, select one or more agent evaluators, such as Task Adherence and Intent Resolution.

Keep the wizard open. You submit the evaluation in the next step.

First, create a JSONL file of test queries for your agent. Each line is a JSON object with a query field. Save it as queries.jsonl:

{"query": "Write a haiku about deploying cloud applications."}

Upload the file as a dataset in your project:

dataset = project_client.datasets.upload_file(
    name="agent-test-queries",
    version="1",
    file_path="./queries.jsonl",
)

Next, choose built-in evaluators and map their inputs. The data_mapping parameter tells each evaluator where to find the query and the agent response. AI-assisted evaluators need a judge model in initialization_parameters; the value must be a chat-completion deployment in your project.

from azure.ai.projects.models import TestingCriterionAzureAIEvaluator

testing_criteria = [
    TestingCriterionAzureAIEvaluator(
        type="azure_ai_evaluator",
        name="Intent Resolution",
        evaluator_name="builtin.intent_resolution",
        initialization_parameters={"model": model_deployment},
        data_mapping={
            "query": "{{item.query}}",
            "response": "{{sample.output_items}}",
        },
    ),
    TestingCriterionAzureAIEvaluator(
        type="azure_ai_evaluator",
        name="Task Adherence",
        evaluator_name="builtin.task_adherence",
        initialization_parameters={"model": model_deployment},
        data_mapping={
            "query": "{{item.query}}",
            "response": "{{sample.output_items}}",
        },
    ),
]

Create the evaluation. It defines the test data schema and testing criteria, and serves as a container for one or more runs:

from openai.types.eval_create_params import DataSourceConfigCustom

data_source_config = DataSourceConfigCustom(
    type="custom",
    item_schema={
        "type": "object",
        "properties": {"query": {"type": "string"}},
        "required": ["query"],
    },
    include_sample_schema=True,
)

evaluation = client.evals.create(
    name="Agent Quality Evaluation",
    data_source_config=data_source_config,
    testing_criteria=testing_criteria,
)
print(f"Evaluation created: {evaluation.id}")

First, create a JSONL file of test queries for your agent. Each line is a JSON object with a query field. Save it as queries.jsonl:

{"query": "Write a haiku about deploying cloud applications."}

Upload the file as a dataset in your project:

const dataset = await projectClient.datasets.uploadFile(
  "agent-test-queries",
  "1",
  "./queries.jsonl",
);

Next, choose built-in evaluators and map their inputs. The data_mapping parameter tells each evaluator where to find the query and the agent response. AI-assisted evaluators need a judge model in initialization_parameters; the value must be a chat-completion deployment in your project.

const testingCriteria = [
  {
    type: "azure_ai_evaluator",
    name: "Intent Resolution",
    evaluator_name: "builtin.intent_resolution",
    initialization_parameters: { model: modelDeployment },
    data_mapping: {
      query: "{{item.query}}",
      response: "{{sample.output_items}}",
    },
  },
  {
    type: "azure_ai_evaluator",
    name: "Task Adherence",
    evaluator_name: "builtin.task_adherence",
    initialization_parameters: { model: modelDeployment },
    data_mapping: {
      query: "{{item.query}}",
      response: "{{sample.output_items}}",
    },
  },
];

Create the evaluation. It defines the test data schema and testing criteria, and serves as a container for one or more runs:

const dataSourceConfig = {
  type: "custom",
  item_schema: {
    type: "object",
    properties: { query: { type: "string" } },
    required: ["query"],
  },
  include_sample_schema: true,
};

const evaluation = await client.evals.create({
  name: "Agent Quality Evaluation",
  data_source_config: dataSourceConfig,
  testing_criteria: testingCriteria,
});
console.log(`Evaluation created: ${evaluation.id}`);

Run the evaluation from the azd workspace root:

azd ai agent eval run --config eval.yaml

The command reads eval.yaml, sends each query to your agent, scores the responses, and prints a summary when it finishes:

Eval run started
   Eval: eval_b36748dede424e4ba3f8e6c99ca2cf27
   Run:  evalrun_5f72ef189ad24790a32128e6f230b131
   (✓) Done  Eval run

Results:    1 total, 1 passed, 0 failed, 0 errored

Per-criteria results:
  intent_resolution: 1 passed, 0 failed, 0 errored
  task_adherence: 1 passed, 0 failed, 0 errored

1. On the Review and submit step, enter a name for the evaluation.
2. Review the target, scope, data source, and selected evaluators.
3. Select Submit to start the run.

Create a run that sends each test query to your agent and applies the evaluators. Replace <your-agent-name> with your hosted agent's name:

eval_run = client.evals.runs.create(
    eval_id=evaluation.id,
    name="Agent Evaluation Run",
    data_source={
        "type": "azure_ai_target_completions",
        "source": {"type": "file_id", "id": dataset.id},
        "input_messages": {
            "type": "template",
            "template": [
                {
                    "type": "message",
                    "role": "user",
                    "content": {"type": "input_text", "text": "{{item.query}}"},
                }
            ],
        },
        "target": {
            "type": "azure_ai_agent",
            "name": "<your-agent-name>",
            # "version": "1",  # Optional; omit to use the latest version
        },
    },
)

print(f"Evaluation run started: {eval_run.id}")

Create a run that sends each test query to your agent and applies the evaluators. Replace <your-agent-name> with your hosted agent's name:

const evalRun = await client.evals.runs.create(evaluation.id, {
  name: "Agent Evaluation Run",
  data_source: {
    type: "azure_ai_target_completions",
    source: { type: "file_id", id: dataset.id },
    input_messages: {
      type: "template",
      template: [
        {
          type: "message",
          role: "user",
          content: { type: "input_text", text: "{{item.query}}" },
        },
      ],
    },
    target: {
      type: "azure_ai_agent",
      name: "<your-agent-name>",
      // version: "1", // Optional; omit to use the latest version
    },
  },
});

console.log(`Evaluation run started: ${evalRun.id}`);

List recent evaluations:

azd ai agent eval list

    Eval ID                                Name        Status of last run  Runs
    -------                                ----        ------------------  ----
*   eval_b36748dede424e4ba3f8e6c99ca2cf27  agent-eval  Completed           1

* = active eval in current environment

Show the most recent evaluation and its runs:

azd ai agent eval show

Eval:   eval_b36748dede424e4ba3f8e6c99ca2cf27
Name:   agent-eval
Agent:  <your-agent-name>
Runs:   1

Recent runs:
  Run ID                                    Status     Passed  Failed  Created
  ------                                    ------     ------  ------  -------
  evalrun_5f72ef189ad24790a32128e6f230b131  Completed  1/1     0       2026-06-17 14:52 UTC

Use the results to confirm which agent version was evaluated and which evaluator scores were produced. To see per-evaluator details and a link to the report in the Foundry portal, run azd ai agent eval show <eval-id> --eval-run-id <run-id>.

1. The details page shows the target, dataset, status, token usage, and an aggregate score for each evaluator.
2. Select the run name to view row-level results: each query, the agent response, the evaluator score, and the score explanation.

Poll for completion, then print the status and the report URL that opens the results in the Foundry portal:

import time

while True:
    run = client.evals.runs.retrieve(run_id=eval_run.id, eval_id=evaluation.id)
    if run.status in ["completed", "failed"]:
        break
    time.sleep(5)

print(f"Status: {run.status}")
print(f"Report URL: {run.report_url}")

At the run level, you can see aggregated pass and fail counts for each evaluator:

print(run.result_counts)
for criteria in run.per_testing_criteria_results:
    print(criteria.testing_criteria, "passed:", criteria.passed, "failed:", criteria.failed)

ResultCounts(errored=0, failed=0, passed=1, total=1, skipped=0)
Intent Resolution passed: 1 failed: 0
Task Adherence passed: 1 failed: 0

For row-level detail, list the output items. Each result includes the evaluator name, pass or fail, and a score:

for item in client.evals.runs.output_items.list(run_id=eval_run.id, eval_id=evaluation.id):
    for result in item.results:
        print(item.id, result.name, "passed:", result.passed, "score:", result.score)

Poll for completion, and then print the status and the report URL that opens the results in the Foundry portal:

let run = evalRun;
while (!["completed", "failed"].includes(run.status)) {
  run = await client.evals.runs.retrieve(run.id, {
    eval_id: evaluation.id,
  });
  await new Promise((resolve) => setTimeout(resolve, 5000));
}

console.log(`Status: ${run.status}`);
console.log(`Report URL: ${run.report_url}`);

At the run level, you can see aggregated pass and fail counts for each evaluator:

console.log(JSON.stringify(run.result_counts));
for (const criteria of run.per_testing_criteria_results) {
  console.log(
    criteria.testing_criteria,
    "passed:",
    criteria.passed,
    "failed:",
    criteria.failed,
  );
}

{"errored":0,"failed":0,"passed":1,"total":1,"skipped":0}
Intent Resolution passed: 1 failed: 0
Task Adherence passed: 1 failed: 0

For row-level detail, list the output items. Each result includes the evaluator name, pass or fail, and a score:

for await (const item of client.evals.runs.outputItems.list(run.id, {
  eval_id: evaluation.id,
})) {
  for (const result of item.results) {
    console.log(item.id, result.name, "passed:", result.passed, "score:", result.score);
  }
}