Loging ModelNames in API mangement for Background requests

Hello Jeffrey,

Thank you for reaching out on Microsoft Q&A portal.

As you noted, background polling operations via GET requests typically do not return or accept the model name, making it impossible for the metric policy to automatically detect it on the follow-up call. The most reliable workaround is to cache the model name during the initial POST request and retrieve it during the GET request so it can be injected into your metrics.

According to the official Microsoft documentation, the llm-emit-token-metric policy "sends custom metrics to Application Insights about consumption of large language model (LLM) tokens through LLM APIs." You can "configure at most 5 custom dimensions for this policy," and it is strictly evaluated in the <inbound> section. Furthermore, the metrics capture process dictates that "Values in the usage section of the response from the LLM API, when available, are used to determine token metrics."

Here is how you can securely structure your APIM policies using cache-store-value and cache-lookup-value while ensuring your HTTP streams are preserved.

1. Initial POST Request (Store Model)

Extract the requested model from the inbound request and the response_id from the outbound response, using preserveContent: true to prevent the stream from being consumed.

<policies>
    <inbound>
        <base />
        <set-variable name="requestModel" value="@(context.Request.Body.As<JObject>(preserveContent: true)["model"]?.ToString())" />
    </inbound>
    <backend>
        <base />
    </backend>
    <outbound>
        <base />
        <set-variable name="responseId" value="@(context.Response.Body.As<JObject>(preserveContent: true)["response_id"]?.ToString())" />
        
        <choose>
            <when condition="@(!string.IsNullOrEmpty((string)context.Variables["responseId"]) && !string.IsNullOrEmpty((string)context.Variables["requestModel"]))">
                <cache-store-value 
                    key="@("model_" + (string)context.Variables["responseId"])" 
                    value="@((string)context.Variables["requestModel"])" 
                    duration="300" />
            </when>
        </choose>
    </outbound>
</policies>

2. Background GET Request (Retrieve Model & Emit Metric)

Extract the response_id from the URL parameters (assuming it is passed in the path) and emit the custom dimension prior to sending the GET call to the backend.

<policies>
    <inbound>
        <base />
        <set-variable name="pollingResponseId" value="@(context.Request.MatchedParameters["response_id"])" />
        
        <cache-lookup-value 
            key="@("model_" + (string)context.Variables["pollingResponseId"])" 
            variable-name="cachedModel" />
            
        <choose>
            <when condition="@(context.Variables.ContainsKey("cachedModel"))">
                <llm-emit-token-metric namespace="MyLLMMetrics">
                    <dimension name="ModelName" value="@((string)context.Variables["cachedModel"])" />
                </llm-emit-token-metric>
            </when>
            <otherwise>
                <llm-emit-token-metric namespace="MyLLMMetrics" />
            </otherwise>
        </choose>
    </inbound>
    <backend>
        <base />
    </backend>
    <outbound>
        <base />
    </outbound>
</policies>

Please try and let me know if it works.

Note: This response is generated with the help of AI systems.

fantastic looks good im going to try