Troubleshoot latency issues on Azure NetApp Files volumes

Elevated latency in Azure NetApp Files can occur because of throughput limits, cool-tier data retrieval, or factors outside the storage service, such as client, network, or application configuration. This article provides a step-by-step workflow to help you isolate the cause of latency and identify appropriate remediation actions.

Symptoms and troubleshooting flow

Elevated latency on an Azure NetApp Files volume typically falls into one of three categories. Work through each branch in order. Start with throughput throttling because it's the most common cause of unexpected latency.

Throughput throttling

Throughput throttling is the most common cause of unexpected latency. A typical symptom is that latency rises when the workload is busy and returns to normal when the load decreases.

Check for throughput throttling

1. Open Azure Monitor for the volume.
2. Review the Throughput limit reached metric.
3. Compare the metric with the time window when latency was high. If the metric has any nonzero values during that time, throughput throttling is the likely cause.

Cause

Every Azure NetApp Files volume has a throughput ceiling. For an auto-QoS volume, the ceiling is determined by the service level multiplied by the volume quota.

When combined read and write throughput reaches the ceiling, the system queues additional I/O operations. Queued I/O increases latency. This behavior indicates that the quality of service (QoS) limit is being enforced; it doesn't necessarily indicate a storage fault.

Resolution

• Auto-QoS: Increase the volume quota. Throughput scales linearly, so doubling the quota doubles the throughput limit. You can set the quota higher than the actual data size. Billing is based on capacity pool allocation, not per-volume used capacity.
• Manual-QoS: Increase the assigned throughput directly on the volume.
• Service-level upgrade: Move the volume to a higher service level to get more throughput per TiB.

Why does increasing application concurrency increase latency after a volume reaches its QoS throughput limit?

If a volume is operating at its QoS throughput limit, latency can continue to increase even though throughput remains relatively unchanged.

When a volume reaches its allocated throughput limit, it can't service additional I/O requests any faster because it has reached its throughput ceiling. As workload demand continues to increase, the system begins queuing incoming I/O requests. This queuing increases latency, even though throughput remains constrained by the QoS allocation.

A common misconception is that increasing application concurrency, such as increasing thread counts, job counts, or outstanding I/O requests, increases throughput after the QoS limit is reached. In practice, once the volume is operating at its maximum throughput allocation, increasing concurrency doesn't increase throughput. Instead, it further increases latency.

The following behavior is expected when a workload reaches a volume's QoS throughput limit:

• Throughput increases as workload demand increases until the QoS limit is reached.
• After the QoS limit is reached, throughput remains relatively constant.
• Additional concurrency primarily increases latency rather than throughput.

If your workload is consistently operating at or near the volume's QoS limit, consider one or more of the following actions:

• Increase the service level or throughput allocation.
• Use Flexible Service Level to increase throughput independently of capacity, where applicable.
• Reduce unnecessary application concurrency if the workload is already saturating the volume.

Increased latency in conjunction with throughput at or near the volume's QoS limit doesn't necessarily indicate a storage service issue. In many cases, this behavior is the expected result of the workload demanding more throughput than has been allocated to the volume.

Cool-tier reads

Use this branch when cool access is enabled. A typical symptom is read-latency spikes, especially when accessing data that isn't recently used. Write latency isn't affected.

Check for cool-tier reads

1. Open Azure Monitor for the volume.
2. Review the Volume cool tier data read size metric.
3. Compare the metric with the time window when latency was high. If the metric is elevated during that time, the system is retrieving data from the cool tier.

Cause

When you enable cool access, the system moves data that you didn't access during the coolness period to a lower-cost Azure storage tier. The default coolness period is 31 days. When you read that data again, the system retrieves it from cool storage.

Cool access trades read performance for cost savings. A latency range of 10-50 ms for cool-tier reads is expected. If latency consistently exceeds 100 ms for cool-tier reads, open a support ticket.

Resolution

• Increase the coolness period: You can configure the coolness period from 2 through 183 days. A longer period keeps recently used data in the hot tier longer.
• Retain read data in the hot tier: Configure the retrieval policy to On-Read so that both sequential and random reads rehydrate data from the cool tier to the hot tier.
• Review the retrieval policy: Review the Cool Access Retrieval Policy setting on the volume.

Client and network factors

Use this branch if the previous checks don't apply. For example, use it if the throughput limit isn't reached, cool access isn't enabled, or cool-tier reads aren't elevated during the latency window.

Typical symptoms include latency that's high regardless of I/O load, affects only specific clients, or affects specific operation types such as small-file operations, metadata-heavy operations, or permission changes.

When to open a support ticket

Open a support ticket if high latency continues after you confirm all of the following conditions:

• The Throughput limit reached metric is zero during the latency window.
• Cool-tier reads aren't elevated during the latency window, or cool access isn't enabled.
• The client and network checklist doesn't identify an issue.

Include the following information in the support ticket:

• Exact UTC timestamps for the latency window.
• The troubleshooting branches that you checked and ruled out.
• Screenshots of throughput, latency, and cool-tier metrics for the same time window.

Providing the time window, branches ruled out, and metric screenshots accelerates the investigation and helps support distinguish storage-layer events from workload, client, and network factors.

Next steps

• Troubleshoot Azure NetApp Files using diagnose and solve problems tool
• General performance considerations for Azure NetApp Files
• Service levels for Azure NetApp Files
• Azure NetApp Files storage with cool access
• Manage Azure NetApp Files storage with cool access