Azure Files diagnostics are present in Storage Account archive, but only a subset is seen by the Event Hub consumer group

Question

Azure Files diagnostics are present in Storage Account archive, but only a subset is seen by the Event Hub consumer group

Sivabalan J 20

Hi Team,

We are investigating an Azure Files diagnostics case where say 10 file activity events were performed in one short user session.

Azure settings used in this environment:

Azure Storage Account diagnostic settings are configured for File service / File share.
Destination 1: Archive to Storage Account.
Destination 2: Stream to Event Hub.
Enabled categories: StorageRead, StorageWrite, and StorageDelete.

What is happening:

Say All 10 expected events are visible in the Storage Account archive destination.
Only 1 of those 10 events is visible in the Event Hub consumer-group collection logs and in the final product output.
The other 9 events are visible in the archive destination, but they are not seen as exact matching events in the Event Hub consumer-group collection logs.
Some nearby read/property-check events for the same session do still appear in the consumer logs, so the consumer path was not fully down during that time.

This is why the current evidence points to a gap between the archive destination and what is reaching the Event Hub consumer group, rather than only a reporting issue.

The Event Hub consumer implementation in the application is summarized below:

Language: Java.
It uses the Azure Event Hubs Java SDK to connect to the configured Event Hub.
The connection is established using the Event Hub name, connection string, and consumer group.
The transport mode used is AMQP over WebSockets.
When an Event Hub message is received, the application reads the payload, parses the JSON records array, logs receipt of the event, hands the records to the internal processing queue, and then updates the checkpoint.
Checkpointing is custom and database-backed in this application. It is not using Azure Blob checkpoint storage.

If helpful, we can also provide the exact SDK version separately, but the main point here is the delivery behavior rather than the full dependency tree.

Based on the evidence collected so far, we would like to understand the Azure side of this behavior.

Questions:

Is it possible for Azure Files diagnostics to be fully written to the Storage Account archive destination while only a subset of the same events is published to Event Hub?
Are there any known Azure features, platform behaviors, filters, or limitations that can prevent certain Azure Files diagnostic events from a specific file share from being populated into Event Hub while still writing those events to the archive destination?
Is there any Azure-side behavior where specific operations such as CreateDirectory, CreateFile, PutRange, GetFile, DeleteFile, or DeleteDirectory could be selectively missing only from the Event Hub route?
Are there any known cases where diagnostics from a specific file share, path, or object pattern can be excluded or not forwarded to Event Hub even though StorageRead, StorageWrite, and StorageDelete are enabled at the File service / File share diagnostic settings level?
Could Event Hubs factors such as partition ownership, consumer groups, checkpoint state, load balancing, or ordering cause this kind of selective gap without obvious errors, when the archive destination still contains the full event set?
Is there any Microsoft-recommended way to verify whether these missing Azure Files diagnostic records were ever published to the Event Hub namespace and partition before the consumer group tried to read them?
Are there Azure metrics, diagnostic traces, or internal delivery validation methods that can confirm whether the missing records were dropped before Event Hub publish, published but not retained, or published but not delivered to the consumer group as expected?

Any guidance on how to validate this path would be very helpful.

Thank you.

0 comments

Answer accepted by question author

Pilladi Padma Sai Manisha 11,705 Microsoft External Staff Moderator

Hi Sivabalan J,
Thankyou for reaching microsoft Q&A!
You’re right that all 10 Azure Files diagnostics events appearing in the storage archive but only 1 showing up in your Event Hub consumer is possible. Diagnostic settings fan out the same categories (StorageRead/Write/Delete) independently to each destination, and there is no documented Azure‑side feature that selectively filters specific File operations, shares, or paths only on the Event Hubs route once those categories are enabled.

Given that the archive has all 10 events, the most likely causes are on the Event Hubs side: ingestion issues (for example, throttling) or, more commonly, consumer behavior such as batch handling and custom checkpointing. If the Java client processes only the first record in each JSON records array or checkpoints offsets before all records are processed, it will look like “1 of 10” events arrived even though the full payload was published.

To validate this, I’d suggest:

In the Event Hub’s Metrics blade, check IncomingMessages and related metrics for the time window of your test to confirm that multiple messages were accepted.

Enable Event Hubs diagnostic logs (for example, RuntimeAuditLogs) via Diagnostic settings and send them to Log Analytics, then query the documented tables to see if there were errors or throttling during that period.

Optionally, enable Event Hubs Capture temporarily and inspect the captured blobs to verify whether all diagnostics events reached the Event Hub.

If the metrics/logs show normal ingestion but your consumer still sees only one event, the issue is almost certainly in the Java consumer’s parsing or checkpoint logic.

Docs for reference: Diagnostic settings and destinations: https://learn.microsoft.com/azure/azure-monitor/platform/diagnostic-settings Stream monitoring data to Event Hubs: https://learn.microsoft.com/azure/azure-monitor/platform/stream-monitoring-data-event-hubs Monitor Event Hubs (metrics and logs): https://learn.microsoft.com/azure/event-hubs/monitor-event-hubs Monitoring data reference: https://learn.microsoft.com/azure/event-hubs/monitor-event-hubs-reference

Pilladi Padma Sai Manisha 11,705 Reputation points Microsoft External Staff Moderator

2026-05-15T11:18:48.8966667+00:00

Hi Sivabalan J
Following up to see if the below answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

1 additional answer

Your answer

Pilladi Padma Sai Manisha 11,705 Reputation points Microsoft External Staff Moderator

2026-05-15T11:18:48.8966667+00:00

Hi Sivabalan J
Following up to see if the below answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

Event Hubs and Azure Monitor diagnostics are documented as independent destinations that receive the same diagnostic categories (for example, StorageRead/StorageWrite/StorageDelete), but the provided context does not describe any Azure-side behavior where:

A subset of Azure Files diagnostic events is intentionally written only to the archive (Storage) destination and not to Event Hubs, or
Specific operations (CreateDirectory, CreateFile, PutRange, GetFile, DeleteFile, DeleteDirectory) or specific file shares/paths are selectively excluded from the Event Hub stream while still being written to the archive.

The context does describe how Event Hubs consumers work and how gaps can arise on the consumer side due to offsets and checkpointing:

Event Hubs uses a pull model; consumers request events from partitions.
Each consumer group has its own independent view of the stream and tracks its own position via offsets.
Checkpointing is explicitly the responsibility of the consumer. The service exposes offsets, but the consumer must persist them.
If checkpointing is incorrect or inconsistent (for example, custom database-backed checkpoints), a consumer can:
- Skip events (by checkpointing an offset ahead of what was fully processed),
- Re-read events (if checkpointing lags behind), or
- Show apparent gaps if multiple instances compete for partition ownership or if offsets are mismanaged.

From the documented behavior:

A consumer group’s view is entirely determined by its stored offsets and checkpointing logic. If the archive destination shows all 10 events but the consumer group only sees one, the documented mechanism that can cause such a selective gap is consumer-side offset/checkpoint handling, not a documented Azure-side filter on diagnostics.
The guidance for checkpointing emphasizes that when using a checkpoint store (for example, Azure Blob Storage), each consumer group should have its own container and that checkpointing is critical for correct resumption and failover. Custom checkpoint implementations must provide equivalent correctness guarantees.

Regarding verification and validation:

The context states that Event Hubs diagnostic logs (Archive Logs, Operational Logs, Runtime Audit Logs, Application Metric Logs) are written into Azure Monitor Logs tables and can be queried with Log Analytics. These logs capture information about Event Hubs operations and can be used to validate that Event Hubs is emitting diagnostic information, but the context does not provide a specific method to correlate individual Azure Files diagnostic records with their corresponding Event Hub messages.
The Event Hubs monitoring reference notes that all logs are stored as JSON and can be queried, but it does not describe a built-in mechanism to prove that a particular Azure Files diagnostic record was or was not published to a specific Event Hub partition.

Given the available documentation in the context, the only clearly described mechanisms that can explain selective gaps between what exists in the archive and what is visible to a consumer group are related to Event Hubs consumer behavior: offsets, checkpointing, and consumer group state. No Azure-side feature or filter is documented here that would selectively drop or exclude specific Azure Files diagnostic events from the Event Hub destination while still writing them to the archive.

References: