 I am unable to see the Prod environment Logs in the grafana. Our team has CI/CD cluster in the AZURE cloud for CI/Monitoring. Can you help me out how i can troubleshoot and get Prod environment logs

Vempati Tejaswi, 0 Reputation points
2026-06-17T10:58:21.89+00:00

Can you help me how i can get the prod environment logs from Grafana and from Azure cloud

Azure Monitor
Azure Monitor

An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.

3 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Bharath Y P 10,165 Reputation points Microsoft External Staff Moderator
    2026-06-17T11:46:16.9166667+00:00

    Hello Vempati Tejaswi, It sounds like you’re able to access Grafana, but the Prod environment logs don’t show up there while you also have a CI/CD / monitoring cluster in Azure. Below are some practical troubleshooting directions based on the provided Azure guidance.

    1) First confirm where the “Grafana logs” are supposed to come from

    In Azure Managed Grafana, logs/telemetry shown in Grafana depend on the configured data source (for example Azure Monitor). If Grafana dashboards don’t populate, the most common cause is that the Azure Monitor data source can’t fetch data.

    What to check in Grafana (Azure Managed Grafana)

    1. In your Grafana endpoint, go to: Configurations > Data Sources > Azure Monitor
    2. Verify the top dropdowns are populated with the correct:
      • Subscription
      • Resource group
      • Resource name If “Resource” is set to null or points to the wrong resource group, dashboards won’t show data.
    3. Check authentication method:
      • If using managed identity:
        • Make sure managed identity is enabled in the workspace (Settings > Identity (Preview)).
        • Use Load Subscriptions, then Save & test.
        • If you see “No Log Analytics workspaces found”, the managed identity likely needs Reader role on the Log Analytics workspace.
      • If using App Registration:
        • Validate the Directory (tenant) ID and Application (client) ID are correct.
        • Verify the service principal has the Monitoring Reader role on the Azure Managed Grafana workspace/subscription context (per the guidance), and reapply the client secret if needed.

    2) If you’re using Azure Monitor / Log Analytics for log delivery, validate the “plumbing”

    If you’re expecting diagnostic logs/telemetry in Azure Monitor but they’re missing or delayed, the guidance highlights a few checks:

    Missing/Delayed logs checklist

    1. Check diagnostic settings for the relevant Azure resource:
      • Go to Monitoring > Diagnostic settings
      • Confirm there is a diagnostic setting that exports logs to a destination (for example Azure Storage or Event Hub).
    2. Generate some actual activity
      • Diagnostic logs are created only when the data plane is used (for example making calls), not just by creating/changing the resource.
    3. Allow time for delivery
      • Logs can take 10–20 minutes to appear in the destination.
    4. Verify permissions
      • Ensure the diagnostic setting destination has correct permissions to write.

    3) If your logs are for an AKS/Kubernetes setup (agents must be healthy)

    If the “Prod” logs are coming from containers/nodes, the relevant guidance suggests checking that the log collection agents are actually running:

    • Confirm the Azure Monitor Agent deployments and pods are in expected Running/Ready state (for Linux and Windows nodes if applicable).
    • If agents are running but data is missing, it may still be permissions-related or a collection cap (daily cap is mentioned as a cause of stopping ingestion into Log Analytics once met).

    4) Make sure your destination actually receives application logs

    For App Service on Arc (Preview) specifically: system component logs are forwarded, but application logs are not sent by default. If you’re using that pattern, you may need to collect logs via Kubernetes standard output or configure the extension to send them to a Log Analytics workspace (and be careful with disabling the log processor, because it stops forwarding).

    References documentation

    Hope this helps. If the information was useful, please consider accepting the answer and upvoting. Feel free to reach out if you need any further assistance. Thank you.

    Was this answer helpful?

    0 comments
  2. Vempati Tejaswi, 0 Reputation points
    2026-06-17T11:06:06.8366667+00:00

    Can you send me the query which i can use for producing the prod environment logs

    Was this answer helpful?

    0 comments

  3. AI answer

    2026-06-17T10:58:35.7466667+00:00

    Use a two-step approach: first ensure logs are being collected in Azure, then ensure Grafana is correctly configured to query them.

    1. Verify logs are collected in Azure
      1. In the Azure portal, open the production resource(s) whose logs should appear in Grafana (for example, App Service, AKS, or other workloads).
      2. Confirm that diagnostic settings are configured to send logs to a Log Analytics workspace, storage account, Event Hubs, or partner solution:
        • Go to the resource → Diagnostic settings.
        • Ensure a diagnostic setting exists that sends the relevant log categories (for example, audit, performance, or application logs) to your chosen destination.
        • If using Azure Managed Grafana, follow the same pattern for the Grafana workspace itself: configure diagnostic settings so login and usage events are streamed to Log Analytics or another destination. See “Monitor Azure Managed Grafana using diagnostic settings”.
      3. Validate that logs are actually arriving:
        • For Log Analytics: open the Log Analytics workspace → Logs, run a simple KQL query (for example, on the AzureDiagnostics or relevant table) filtered to the last hour and your production resource.
        • If no rows are returned, adjust diagnostic settings or retention policies until logs appear.
    2. Check Azure Managed Grafana logging and monitoring
      1. In the Azure portal, open the Azure Managed Grafana workspace used by the CI/CD/monitoring cluster.
      2. Confirm diagnostic settings are enabled for this workspace as well, so you can audit access and system events. This helps verify whether queries are being run and if there are errors.
      3. Use the Logs blade on the Managed Grafana resource:
        • Select the Grafana workspace → Logs.
        • Use the suggested queries or write your own to check for authentication events, access patterns, and errors that might explain missing data.
    3. Validate Grafana data source configuration
      1. In Grafana (Azure Managed Grafana):
        • Go to Connections → Data sources → Azure Monitor.
        • Confirm the data source is configured to point to the correct subscription, resource group, and Log Analytics workspace that holds the production logs.
      2. If using Grafana 11 with Azure Managed Grafana and Log Analytics:
        • On the Settings tab of the Azure Monitor data source, optionally enable Basic Logs if using Basic Logs tables in Log Analytics.
        • When building a panel:
          • Set Service to Logs.
          • Set Resource to the correct Log Analytics workspace.
          • Choose the appropriate table plan (Analytics or Basic) matching how your prod logs are stored.
    4. Troubleshoot “no data” in Grafana dashboards
      1. In the Grafana dashboard where prod logs should appear:
        • Check the time range (top-right). Ensure it covers the period when you know logs were generated.
        • Verify the query is targeting the correct workspace, subscription, and resource filters for the production environment.
      2. If dashboards show no data:
        • Confirm the Grafana instance has access to the Azure Monitor workspace that stores your prod logs.
        • Ensure the cluster and applications have been running long enough for logs to be collected and ingested.
    5. Secure logging and retention
      1. Once prod logs are visible, configure log retention policies on the Log Analytics workspace or storage account according to your compliance requirements.
      2. Ensure access to logs and dashboards is controlled via Azure RBAC on both the Log Analytics workspace and the Azure Managed Grafana resource.

    Following these steps ensures:

    • Production resources send logs to a central Azure destination.
    • Azure Managed Grafana is correctly configured to query that destination.
    • Dashboards and queries are scoped and time-ranged correctly so prod logs are visible.

    References:

    AI-generated content may be incorrect. Read our transparency notes for more information.

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.