Error code: 400 when running General purpose Fluency evaluators

Question

Error code: 400 when running General purpose Fluency evaluators

Justin Obanor 40

Using the sample sample_fluency.py from azure-sdk-for-python at sdk/ai/azure-ai-projects/samples/evaluations/agentic_evaluators/sample_fluency.py, client.evals.create() succeeds but client.evals.runs.create() fails with:

BadRequestError: Error code: 400 - {'error': {'code': 'UserError', 'severity': None, 'message': 'The action cannnot be finished with reason Forbidden', 'messageFormat': 'The action cannnot be finished with reason {error}', 'messageParameters': {'error': 'Forbidden'}, 'referenceCode': None, 'detailsUri': None, 'target': None, 'details': [], 'innerError': {'code': 'UnauthorizedUserAction', 'innerError': None}, 'debugInfo': None, 'additionalInfo': None}, 'correlation': {'operation': 'e9a6cfb5f254b4298cc505acf31ed58c', 'request': '57fc69d1782fbf76'}, 'environment': 'eastus2', 'location': 'eastus2', 'time': '2026-06-22T18:35:59.0328447+00:00', 'componentName': 'raisvc', 'statusCode': 400}

The failure occurs only when creating the evaluation run.

The identity executing the code has the Foundry User role assigned on the Azure AI Foundry resource. I also assigned Foundry User to the project managed identity on the project level.

Inference works successfully using:

client.chat.completions.create(...)

with the same project and model deployment.

Is there additional RBAC required specifically for Evaluation Runs? Or any thing else that might be cauing the RequestError

Anshika Varshney 14,085 Reputation points Microsoft External Staff Moderator

2026-06-24T15:28:18.6433333+00:00

Hello @Justin Obanor

Just checking back to see if you’re still facing the same issue. If the problem persists, please share a few more details and we’ll be happy to help you further.

Thankyou!
Justin Obanor 40 Reputation points

2026-06-24T15:38:07.41+00:00

Hello @Anshika Varshney

Thanks a lot for you help. The issue was indeed with the network restrictions. Setting the network_acls default_action to Allow helped

I was wondering, you mentioned that I can add the required Service Endpoints instead as a permanent fix. Could you please advise where I can get the backend service endpoints so I can add them?

Also you mentioned If using Entra ID authentication for storage. I'm actually using Entra ID just for authentication to the APIs (instead of auth using API keys). I'm curios to know in what scenario I'd use it for storage, as I do not have a storage account hooked up to my foundry account yet. Thanks again
Justin Obanor 40 Reputation points

2026-06-24T16:04:07.6766667+00:00

Hello @Anshika Varshney

Just wanted to clarify some more on my last question, also sorry for getting back late. Looking at the post you shared from learn.microsoft, I can't directly add the service endpoints to the network_acls ip_rules since that field takes in IP addresses. As of now is there an IP address I can use here instead?

Thank you
Anshika Varshney 14,085 Reputation points Microsoft External Staff Moderator

2026-06-25T12:19:48.61+00:00

Hello @Justin Obanor

Sorry for late response. Glad to hear that disabling the network restrictions helped identify the root cause.

Regarding the service endpoints, Azure AI Foundry evaluation services do not currently publish a fixed set of public IP addresses that can be manually added to an allowlist. In most cases, the recommended approach is to use the supported networking configuration (Private Endpoints, Managed Network/VNet integration, or trusted Azure service access where applicable) rather than relying on static IP allowlists. Network-related evaluation failures are commonly caused by storage or dependent resources being inaccessible from the Foundry evaluation service due to firewall restrictions.

For your second question about Entra ID authentication: my earlier comment was referring specifically to scenarios where Azure Storage is used as part of the evaluation workflow. Evaluation runs may read datasets from or write results/artifacts to a storage account. In those cases, the Foundry-managed identity or user identity needs the appropriate RBAC permissions (for example, Storage Blob Data Contributor) on the storage account. This is separate from using Entra ID to authenticate calls to the Foundry or OpenAI APIs.

Since you're not currently using a customer-managed storage account with your Foundry project, that part is likely not relevant to your setup and can be safely ignored.

As for the IP allowlist field, I am not aware of any Microsoft-published IP address range that can be used specifically for Azure AI Foundry evaluation backend services. If your environment requires strict firewall rules, I recommend using the supported private networking options instead of IP-based allowlisting, as the backend service addresses are not documented as fixed customer-consumable endpoints.

Let us know if you run into any additional networking-related issues.

Since I’ve converted comment into an answer, could you please take a moment to mark it as Accepted? This helps others in the community with the same question find the solution more easily.

Thank you!

Answer accepted by question author

Anshika Varshney 14,085 Microsoft External Staff Moderator

Hello @Justin Obanor

Based on the error details and Microsoft's documentation, the issue is likely network restrictions on your Azure AI Foundry account, not missing RBACroles.

Primary Cause: Network Restrictions Blocking raisvc

The error mentions raisvc (the evaluation backend service) and shows "Forbidden" - Microsoft has confirmed this exact issue occurs when the Foundry account uses Selected Networks network restrictions: [learn.microsoft]

Evaluation works when networkAcls.defaultAction = Allow
Evaluation fails when networkAcls.defaultAction = Deny (even with bypass = AzureServices)

The evaluation backend service cannot reach your restricted resources, even though inference works (your direct chat.completions.create call uses a different path).

Solution Steps

1. Check Network Configuration

bash
az cognitive services account show \
  --resource-group <your-resource-group> \
  --name <your-foundry-account-name> \
  --query "networkAcls"

2. Temporary Test: Enable Public Access

Temporarily set defaultAction to Allow to confirm this is the issue:

bash
az cognitive services account update \
  --resource-group <your-resource-group> \
  --name <your-foundry-account-name> \
  --bypass AzureServices \
  --default-action Allow

If evaluations work after this, network restrictions are confirmed as the cause.

3. Permanent Fix: Add Required Service Endpoints

Instead of fully disabling restrictions, add the evaluation service to your permitted networks:

Find the evaluation backend service endpoint
Add it to your network ACLs' allowed IPs/vnets
Or use a private endpoint configuration for evaluations

Secondary Checks (if network isn't the issue)

RBAC Requirements for Evaluation Runs:

While Foundry User should work, verify these minimum roles on the project:

Azure AI Foundry User (you have this)
Cognitive Services OpenAI User (if using Azure OpenAI)
Storage Blob Data Contributor on the storage account (for Entra ID auth)

Storage Account Configuration:

If using Entra ID authentication for storage:

bash
# Verify managed identity has Storage Blob Data Contributor

The storage account must also have publicNetworkAccess = Enabled.

Start with the network restriction check, this matches your exact error pattern (create succeeds, runs.create fails with raisvc/Forbidden) and is the most common cause per Microsoft's recent reports.

I Hope this helps. Do let me know if you have any further queries.
Thankyou!

0 comments

1 additional answer

Your answer

Anshika Varshney 14,085 Reputation points Microsoft External Staff Moderator

2026-06-24T15:28:18.6433333+00:00

Hello @Justin Obanor

Just checking back to see if you’re still facing the same issue. If the problem persists, please share a few more details and we’ll be happy to help you further.

Thankyou!
Justin Obanor 40 Reputation points

2026-06-24T15:38:07.41+00:00

Hello @Anshika Varshney

Thanks a lot for you help. The issue was indeed with the network restrictions. Setting the network_acls default_action to Allow helped

I was wondering, you mentioned that I can add the required Service Endpoints instead as a permanent fix. Could you please advise where I can get the backend service endpoints so I can add them?

Also you mentioned If using Entra ID authentication for storage. I'm actually using Entra ID just for authentication to the APIs (instead of auth using API keys). I'm curios to know in what scenario I'd use it for storage, as I do not have a storage account hooked up to my foundry account yet. Thanks again
Justin Obanor 40 Reputation points

2026-06-24T16:04:07.6766667+00:00

Hello @Anshika Varshney

Just wanted to clarify some more on my last question, also sorry for getting back late. Looking at the post you shared from learn.microsoft, I can't directly add the service endpoints to the network_acls ip_rules since that field takes in IP addresses. As of now is there an IP address I can use here instead?

Thank you

Answer 1

hi Justin Obanor & thx for sharing urs issue here at Q&A portal,

inference is allowed, but the eval runtime path is blocked.

client.evals.create() only creates the eval definition. client.evals.runs.create() actually kicks off the evaluator job, and that can hit extra Foundry/RAI service permissions behind the scenes. The clue is componentName: raisvc + UnauthorizedUserAction.

Foundry User may be enough for normal project usage/inference, but eval runs can need broader project/resource perms, esp if the run has to read data, access model deployment, write eval outputs, or call RAI/evaluation services.

Try to do a quick test give the executing identity Azure AI Developer on the project or Foundry resource, not only Foundry User, then retry. If it works, u have the answer eval runs need more than the current role grants. https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/rbac-azure-ai-foundry Also make sure the project managed identity has access to whatever the eval run uses: model deployment, storage/data asset, connection, and any workspace/project resources. Inference working from the client doesn’t prove the service-side eval job identity can access those deps. Since the error is thrown by raisvc, I’d include the operation/request IDs in a support ticket if RBAC change doesn’t fix it operation: e9a6cfb5f254b4298cc505acf31ed58c & request: 57fc69d1782fbf76

My guess not a bad sample file. More likely eval run authorization is stricter than chat inference, and the backend is returning a bad 400 wrapper around what is really a forbidden action.

rgds,

Alex

&

If my answer was helpful pls mark it and additional thx if u follow me at Q&A portal 
and at my blog https://ctrlaltdel.blog/