Azure Kubernetes Service (AKS) production upgrade strategies

Use these patterns to safely upgrade production Azure Kubernetes Service (AKS) clusters. The guidance is intended for site reliability engineers and platform teams that need to minimize downtime and control upgrade risk.

AKS production upgrade strategy overview

• Blue-green deployments for minimal-downtime upgrades.
• Staged fleet upgrades across multiple environments.
• Safe Kubernetes version adoption with validation gates.
• Emergency security patching for rapid response to Common Vulnerabilities and Exposures (CVEs).
• Application resilience patterns that minimize upgrade disruption.
• Migration cutover rollback with rapid traffic reversal to the source environment.

Use these patterns when you need minimal downtime and more customization than AKS Automatic's preconfigured defaults provide.

Select the scenario that best matches your business needs:

• Do you need an emergency upgrade?
• Do you have stateful workloads?
• New to production upgrades?
• Are you migrating workloads from another cluster or environment?

Kubernetes prerequisites

This article assumes you understand Kubernetes Deployments and Services, including the following concepts:

• Pod Disruption Budgets (PDBs): Limit how many selected pods can be unavailable during voluntary disruptions like node upgrades. With maxUnavailable: 1, Kubernetes allows at most one selected pod to be unavailable during a drain.
• Readiness probes: Signal when a pod is ready to receive traffic. During upgrades, Kubernetes waits for readiness probes to pass before considering a pod healthy.
• Liveness probes: Restart unhealthy pods automatically. During cluster upgrades, these probes ensure failed pods come back quickly.
• Finalizers: Custom logic that runs before Kubernetes deletes a resource. Long-running finalizers can block pod termination during drains.

Choose your strategy

Role-based starting points

Use this table to find the AKS production upgrade guidance that matches your role.

Scenario 1: Minimal downtime production upgrades

• Challenge: "I need to upgrade my production cluster with less than 2 minutes of downtime during business hours."
• Strategy: Use blue-green deployment with intelligent traffic shifting. Blue-green deployment runs two identical production environments (blue and green). You deploy to the inactive environment, validate it, then switch traffic. If problems occur, traffic switches back to the active environment. Actual convergence time depends on the selected traffic-routing service, health probes, and client or DNS caching.
• With AKS Automatic: Preconfigured automatic upgrade channels and pod readiness SLA handle most of this scenario. However, if you need more control, custom validation gates, or blue-green testing strategies beyond standard automatic upgrades, use the following guidance:

To learn more, see Blue-green deployment patterns and Azure Traffic Manager configuration.

Implementation overview (15 minutes)

# List the Kubernetes versions available in the target region, then select a supported version.
az aks get-versions --location eastus2 --output table

# 1. Create green cluster (parallel to blue)
az aks create --name myaks-green --resource-group myRG \
  --kubernetes-version <supported-kubernetes-version> --enable-cluster-autoscaler \
  --min-count 3 --max-count 10

# 2. Deploy application to green cluster
kubectl config use-context myaks-green
kubectl apply -f ./production-manifests/

# 3. Validate green cluster
# Run your application-specific health checks here
# Examples: API endpoint tests, database connectivity, dependency checks

# 4. Switch traffic (target: less than two minutes, including traffic convergence)
az network traffic-manager endpoint update \
  --resource-group traffic-rg --profile-name prod-tm \
  --name green-endpoint --type azureEndpoints --weight 100
az network traffic-manager endpoint update \
  --resource-group traffic-rg --profile-name prod-tm \
  --name blue-endpoint --type azureEndpoints --weight 0

# Create resource group for green cluster
az group create --name myRG-green --location eastus2

# Create green cluster with same configuration as blue
az aks create \
  --resource-group myRG-green \
  --name myaks-green \
  --kubernetes-version <supported-kubernetes-version> \
  --node-count 3 \
  --enable-cluster-autoscaler \
  --min-count 3 \
  --max-count 10 \
  --enable-addons monitoring \
  --generate-ssh-keys

# Get green cluster credentials
az aks get-credentials --resource-group myRG-green --name myaks-green

# Deploy application stack
# Apply your Kubernetes manifests in order:
kubectl apply -f ./your-manifests/namespace.yaml      # Create namespace
kubectl apply -f ./your-manifests/secrets/           # Deploy secrets
kubectl apply -f ./your-manifests/configmaps/        # Deploy config maps
kubectl apply -f ./your-manifests/deployments/       # Deploy applications
kubectl apply -f ./your-manifests/services/          # Deploy services

# Wait for all pods to be ready
kubectl wait --for=condition=ready pod --all --timeout=300s

# Validate application health
kubectl get pods -A
kubectl logs -l app=my-app --tail=50

# Pre-switch validation
GREEN_ENDPOINT_URL=<green-endpoint-url>
API_URL=<application-url>
curl -f "${GREEN_ENDPOINT_URL}/health"
if [ $? -ne 0 ]; then echo "Green health check failed!"; exit 1; fi

# Execute traffic switch
az network dns record-set cname set-record \
  --resource-group myRG-dns \
  --zone-name mycompany.com \
  --record-set-name api \
  --cname myapp-green.eastus2.cloudapp.azure.com

# Immediate validation
sleep 30
curl -f "${API_URL}/health"

# Monitor traffic and errors for 15 minutes
kubectl top nodes
kubectl top pods
kubectl logs -l app=my-app --since=15m | grep ERROR

# Check application metrics
curl "${API_URL}/metrics" | grep http_requests_total

Troubleshooting

• Domain Name System (DNS) propagation is slow: Use low time-to-live (TTL) values before the upgrade, and validate the DNS cache flush.
• Pods stuck terminating: Check for finalizers, long shutdown hooks, or PDBs with maxUnavailable: 0.
• Traffic not shifting: Validate Azure Load Balancer/Azure Traffic Manager configuration and health probes.
• Rollback fails: Always keep the blue cluster ready until the green cluster is fully validated and the rollback window passes.

Frequently asked questions (FAQs)

Can I use open-source software tools for validation?

Yes. Use kube-no-trouble for API checks and Trivy for image scanning.

What's unique to AKS?

Native integration with Traffic Manager, Azure Kubernetes Fleet Manager, and node image patching helps minimize disruption during upgrades.

Advanced configuration

For applications with a sub-two-minute downtime target, use session affinity to reduce session disruption during traffic convergence:

# Use session affinity during transition
apiVersion: v1
kind: Service
metadata:
  name: my-app
spec:
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 300

Success validation

To validate your progress, use the following checklist:

• Application responds within two seconds.
• No 5xx errors are in logs.
• Database connections are stable.
• User sessions are preserved.

Emergency rollback (if needed)

# Immediate rollback to blue cluster
az network dns record-set cname set-record \
  --resource-group myRG-dns \
  --zone-name mycompany.com \
  --record-set-name api \
  --cname myapp-blue.eastus2.cloudapp.azure.com

Expected outcome: Downtime stays within your validated target and rapid traffic rollback capability. Preventing data loss requires workload-specific replication, write coordination, and tested recovery procedures.

Scenario 2: Stage upgrades across environments

• Challenge: "I need to safely test upgrades through dev, test, and production with proper validation gates."
• Strategy: Use Azure Kubernetes Fleet Manager with staged rollouts.
• With AKS Automatic: AKS Automatic clusters use preconfigured auto-upgrade channels by default. Azure Kubernetes Fleet Manager can orchestrate those upgrades across multiple AKS Automatic clusters in your environment, adding centralized validation gates and staged rollout control.

To learn more, see the Azure Kubernetes Fleet Manager overview and Update orchestration.

Prerequisites

# Install Fleet extension
az extension add --name fleet
az extension update --name fleet

# Create Fleet resource
az fleet create \
  --resource-group fleet-rg \
  --name production-fleet \
  --location eastus

Implementation steps

Step 1: Define stage configuration

Create upgrade-stages.json:

{
  "stages": [
    {
      "name": "development",
      "groups": [{ "name": "dev-clusters" }],
      "afterStageWaitInSeconds": 1800
    },
    {
      "name": "testing",
      "groups": [{ "name": "test-clusters" }],
      "afterStageWaitInSeconds": 3600
    },
    {
      "name": "production",
      "groups": [{ "name": "prod-clusters" }],
      "afterStageWaitInSeconds": 0
    }
  ]
}

Step 2: Add clusters to a fleet

# Add development clusters
az fleet member create \
  --resource-group fleet-rg \
  --fleet-name production-fleet \
  --name dev-east \
  --member-cluster-id "/subscriptions/.../clusters/aks-dev-east" \
  --update-group dev-clusters

# Add test clusters
az fleet member create \
  --resource-group fleet-rg \
  --fleet-name production-fleet \
  --name test-east \
  --member-cluster-id "/subscriptions/.../clusters/aks-test-east" \
  --update-group test-clusters

# Add production clusters
az fleet member create \
  --resource-group fleet-rg \
  --fleet-name production-fleet \
  --name prod-east \
  --member-cluster-id "/subscriptions/.../clusters/aks-prod-east" \
  --update-group prod-clusters

Step 3: Create and run a staged update

Select a target Kubernetes version that every Fleet member supports in the update run.

# Create staged update run
az fleet updaterun create \
  --resource-group fleet-rg \
  --fleet-name production-fleet \
  --name supported-version-upgrade \
  --upgrade-type Full \
  --kubernetes-version <supported-kubernetes-version> \
  --node-image-selection Latest \
  --stages upgrade-stages.json

# Start the staged rollout
az fleet updaterun start \
  --resource-group fleet-rg \
  --fleet-name production-fleet \
  --name supported-version-upgrade

Step 4: Validation gates between stages

After dev stage (30-minute soak):

# Run automated test suite
./scripts/run-e2e-tests.sh dev-cluster
./scripts/performance-baseline.sh dev-cluster

# Check for any regressions
kubectl get events --sort-by='.lastTimestamp' | grep -i warn

After test stage (60-minute soak):

# Extended testing with production-like load
./scripts/load-test.sh test-cluster 1000-users 15-minutes
./scripts/chaos-engineering.sh test-cluster

# Manual approval gate
echo "Approve production deployment? (y/n)"
read approval

Troubleshooting

• Stage fails because of quota: Precheck regional quotas for all clusters in the fleet.
• Validation scripts fail: Verify that test scripts are idempotent and return clear success or failure results.
• Manual approval delays: Use automation for nonproduction. Require manual only for production.

Frequently asked questions (FAQs)

Can I use open-source software tools for validation?

Yes. Integrate Sonobuoy for conformance and kube-bench for security.

What's unique to AKS?

Azure Kubernetes Fleet Manager provides staged rollouts and validation gates.

Scenario 3: Safe Kubernetes version intake

• Challenge: "I need to adopt a newer supported Kubernetes version without breaking existing workloads or APIs."
• Strategy: Use multiphase validation with canary deployment.
• With AKS Automatic: AKS Automatic uses the stable channel by default, which limits adoption to N-1 minor version (where N is the latest supported version). This built-in safety gate handles most version intake safely. If you need to test newer versions or custom validation beyond this default, use the following guidance:

To learn more, see Canary deployments and API deprecation policies.

Implementation steps

Step 1: API deprecation analysis

# Install kubent after reviewing the installation script
sh -c "$(curl -sSL https://git.io/install-kubent)"

# Scan the cluster selected by your current kubeconfig context
kubent -o json > api-deprecation-report.json

# Review and remediate findings
cat api-deprecation-report.json | jq '.[] | select(.deprecated==true)'

To learn more, see the Kubernetes API deprecation guide and kube-no-trouble documentation.

Step 2: Create a canary environment

# Create canary cluster with target version
az aks create \
  --resource-group canary-rg \
  --name aks-canary-target \
  --kubernetes-version <supported-kubernetes-version> \
  --node-count 2 \
  --tier premium \
  --enable-addons monitoring

# Deploy subset of workloads
kubectl apply -f ./canary-manifests/

Step 3: Progressive workload migration

# Phase 1: Stateless services (20% traffic)
kubectl patch service api-service -p '{"spec":{"selector":{"version":"canary"}}}'
./scripts/monitor-error-rate.sh 15-minutes

# Phase 2: Background jobs (50% traffic)
kubectl scale deployment batch-processor --replicas=3
./scripts/validate-job-completion.sh

# Phase 3: Critical services (100% traffic)
kubectl patch deployment critical-api -p '{"spec":{"template":{"metadata":{"labels":{"cluster":"canary"}}}}}'

Step 4: Feature gate validation

# Test features and APIs in the target Kubernetes version
apiVersion: v1
kind: ConfigMap
metadata:
  name: feature-validation
data:
  test-script: |
    # Test new security features
    kubectl auth can-i create pods --as=service-account:default:test-sa

    # Validate performance improvements
    kubectl top nodes --use-protocol-buffers=true

    # Check the API versions enabled on the canary cluster
    kubectl api-versions

Success metrics

• API compatibility: 100% (zero breaking changes)
• Performance: No more than 5% regression in key metrics
• Feature adoption: New features validated in canary

Scenario 4: Fastest security patch deployment

• Challenge: "A critical CVE was announced. I need patches deployed across all clusters within four hours."
• Strategy: Use automated node image patching with minimal disruption.
• With AKS Automatic: AKS Automatic uses the fixed NodeImage channel, which provides weekly node images that include security and bug fixes. You can configure maintenance windows, but you can't change the node OS autoupgrade channel. For AKS Standard clusters that require a different patching cadence, use the following guidance to configure SecurityPatch or trigger an immediate node image upgrade:

To learn more, see Node image upgrade strategies, Auto-upgrade channels, and Security patching best practices.

Implementation steps

Step 1: Emergency response preparation

The first command configures the cluster-level node OS autoupgrade channel for an AKS Standard cluster. This channel is separate from the cluster Kubernetes version autoupgrade channel.

# Configure the cluster-level node OS autoupgrade channel
az aks update \
  --resource-group production-rg \
  --name aks-prod \
  --node-os-upgrade-channel SecurityPatch

# Configure maintenance window for emergency patches
az aks maintenanceconfiguration add \
  --resource-group production-rg \
  --cluster-name aks-prod \
  --name aksManagedNodeOSUpgradeSchedule \
  --schedule-type Weekly \
  --day-of-week Monday \
  --interval-weeks 1 \
  --duration 4 \
  --utc-offset +00:00 \
  --start-time 00:00

To learn more, see Planned maintenance configuration and Autoupgrade channels.

Step 2: Automated security scanning

# security-scan-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: security-scanner
spec:
  schedule: "0 */6 * * *"  # Every 6 hours
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: scanner
            image: aquasec/trivy:latest
            command:
            - trivy
            - k8s
            - --report
            - summary
            - cluster

Step 3: Rapid patch deployment

# Trigger immediate node image upgrade for security patches
az aks nodepool upgrade \
  --resource-group production-rg \
  --cluster-name aks-prod \
  --name nodepool1 \
  --node-image-only \
  --max-surge 50% \
  --drain-timeout 5

# Monitor patch deployment
watch az aks nodepool show \
  --resource-group production-rg \
  --cluster-name aks-prod \
  --name nodepool1 \
  --query "upgradeSettings"

Step 4: Compliance validation

# Verify patch installation
kubectl get nodes -o wide
kubectl describe node | grep "Kernel Version"

# Generate compliance report
./scripts/generate-security-report.sh > security-compliance-$(date +%Y%m%d).json

# Notify security team
curl -X POST "$SLACK_WEBHOOK" -d "{\"text\":\"Security patches deployed to production cluster. Compliance report attached.\"}"

Success metrics

• Deployment time: Less than four hours from CVE announcement
• Coverage: 100% of nodes patched
• Downtime: Less than five minutes per node pool

Scenario 5: Application architecture for resilient upgrades

• Challenge: "I want my applications to handle cluster upgrades gracefully without affecting users."
• Strategy: Use resilient application patterns with graceful degradation.
• With AKS Automatic: These architectural patterns complement AKS Automatic's automatic node repair, pod readiness SLA, and automatic upgrade channels to minimize application impact during upgrades. End-to-end availability still depends on workload design, capacity, probes, PDBs, and dependencies. Implement these patterns on top of AKS Automatic to improve resilience.

To learn more, see Application reliability patterns, PDBs, and Health check best practices.

Implementation steps

Step 1: Implement robust health checks

# robust-health-checks.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: resilient-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: resilient-api
  template:
    metadata:
      labels:
        app: resilient-api
    spec:
      containers:
      - name: api
        image: myapp:latest
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
          timeoutSeconds: 3
          successThreshold: 1
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 15"]

This deployment runs three resilient-api replicas, removes unready pods from service traffic, restarts unhealthy containers, and gives terminating pods 15 seconds to drain.

Step 2: Configure Pod Disruption Budgets (PDBs)

# optimal-pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  selector:
    matchLabels:
      app: resilient-api
  maxUnavailable: 1
  # Ensures at least 2 pods remain available during upgrades
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: database-pdb
spec:
  selector:
    matchLabels:
      app: database
  minAvailable: 2
  # Critical: Always keep majority of database pods running

These budgets allow at most one resilient-api pod to be unavailable and require at least two database pods to remain available during voluntary disruptions.

Step 3: Implement a circuit breaker pattern

// circuit-breaker.js
const CircuitBreaker = require('opossum');

const options = {
  timeout: 3000,
  errorThresholdPercentage: 50,
  resetTimeout: 30000,
  fallback: () => 'Service temporarily unavailable'
};

const breaker = new CircuitBreaker(callExternalService, options);

// Monitor circuit breaker state during upgrades
breaker.on('open', () => console.log('Circuit breaker opened'));
breaker.on('halfOpen', () => console.log('Circuit breaker half-open'));

The circuit breaker opens when at least 50% of requests fail, uses a fallback response, and tests recovery after 30 seconds.

To learn more, see Circuit breaker pattern, Retry pattern, and Application resilience.

Step 4: Database connection resilience

# connection-pool-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: db-config
data:
  database.yml: |
    production:
      adapter: postgresql
      pool: 25
      timeout: 5000
      retry_attempts: 3
      retry_delay: 1000
      connection_validation: true
      validation_query: "SELECT 1"
      test_on_borrow: true

This configuration maintains a pool of 25 database connections, validates connections before use, and retries failed connections up to three times.

Success metrics

• Error rate: Less than 0.01% during upgrades
• Response time: Less than 10% degradation
• Recovery time: Less than 30 seconds after node replacement

Scenario 6: Migration cutover rollback

• Challenge: "I'm migrating workloads from a source cluster, VM estate, or on-premises environment into a new AKS cluster. I need rapid traffic rollback to the source if anything goes wrong after cutover."
• Strategy: Use a blue-green dual-cluster (source cluster = blue, new AKS cluster = green), or use blue/green inside the new AKS cluster by using node pools. Keep the source environment running and reachable by the same external front door (DNS, Traffic Manager, or load balancer). Perform final cutover by switching traffic to the green endpoints. If anything goes wrong, flip traffic back to blue.
• With AKS Automatic: This pattern doesn't apply to AKS Automatic because the source environment is outside AKS. Use AKS Standard for the target cluster so you control the node pool configuration and rollback timing.

Why this works

• Cutover is a single traffic switch operation at the front door or service level.
• The source remains intact as a production fallback. No destructive migration step occurs until you confidently decommission it.
• You can pre-warm caches, validate behavior under real traffic, and cut back immediately when metrics cross abort thresholds.

Implementation overview

# variables (replace)
RG="myResourceGroup"
CLUSTER="myAksCluster"
GREEN_POOL="greenpool"
NODE_COUNT=3
VM_SIZE="Standard_DS2_v2"

# Step 1: Create a green node pool in the target AKS cluster
az aks nodepool add \
  --resource-group $RG \
  --cluster-name $CLUSTER \
  --name $GREEN_POOL \
  --node-count $NODE_COUNT \
  --node-vm-size $VM_SIZE \
  --labels pool=green

# Step 2: Get credentials
az aks get-credentials --resource-group $RG --name $CLUSTER

# Step 3: Cutover - patch the Service selector to green pods
kubectl patch svc myapp -n production -p '{"spec":{"selector":{"app":"myapp","version":"green"}}}'

# Rollback - patch selector back to blue
kubectl patch svc myapp -n production -p '{"spec":{"selector":{"app":"myapp","version":"blue"}}}'

az aks nodepool add \
  --resource-group $RG \
  --cluster-name $CLUSTER \
  --name $GREEN_POOL \
  --node-count $NODE_COUNT \
  --node-vm-size $VM_SIZE \
  --labels pool=green

az aks get-credentials --resource-group $RG --name $CLUSTER

# Confirm nodes are in the new pool
kubectl get nodes --show-labels | grep $GREEN_POOL

# deployment-green.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-green
  labels:
    app: myapp
    version: green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: green
  template:
    metadata:
      labels:
        app: myapp
        version: green
    spec:
      nodeSelector:
        "kubernetes.azure.com/agentpool": "greenpool"
      containers:
      - name: myapp
        image: myregistry.azurecr.io/myapp:green
        ports:
        - containerPort: 80

kubectl apply -f deployment-green.yaml -n production
kubectl rollout status deploy/myapp-green -n production

# Check pod health
kubectl get pods -l app=myapp,version=green -n production
kubectl logs -l app=myapp,version=green -n production --tail=100

# Example: check cluster CPU (adjust resource ID for your cluster)
az monitor metrics list \
  --resource /subscriptions/<sub>/resourceGroups/$RG/providers/Microsoft.ContainerService/managedClusters/$CLUSTER \
  --metric "CpuUsagePercentage" \
  --interval PT1M

kubectl patch svc myapp -n production -p '{"spec":{"selector":{"app":"myapp","version":"green"}}}'

# Verify endpoints updated
kubectl get endpoints myapp -n production -o wide

kubectl patch svc myapp -n production -p '{"spec":{"selector":{"app":"myapp","version":"blue"}}}'

# Verify
kubectl get endpoints myapp -n production -o wide
kubectl rollout status deploy/myapp-blue -n production

Dual-cluster migrations (source is a separate cluster or VMs)

If the source (blue) environment is a separate cluster or VMs, operate the front door externally and switch endpoints there instead of patching a Service inside the new cluster.

• Azure Traffic Manager or Front Door: Flip endpoint priority or weight to route between source and target.
• DNS cutover: Maintain the same DNS name and change the A record with a short TTL to the target cluster IP. Account for client and resolver DNS caches when you plan the cutover.
• External load balancer: Use backend pools for each cluster and toggle backend pool membership.

Rollback triggers (go/no-go metrics)

Define explicit triggers that cause an immediate rollback. Tune thresholds to your service-level objectives (SLOs).

Implement automated alarms in Azure Monitor or Application Insights to notify on-call engineers and optionally trigger runbooks that revert traffic.

Troubleshooting

• DNS propagation is slow: Set short TTLs before cutover and validate DNS cache flush.
• Pods stuck terminating on source: Check for finalizers or long shutdown hooks.
• Traffic not shifting: Validate Traffic Manager health probes and endpoint configuration.
• Rollback fails: Keep the source environment patched and reachable until you formally decommission it.

Success validation checklist

Before cutover:

• [ ] Source (blue) stays running and reachable.
• [ ] Green environment has identical or compatible service endpoints, secrets, and config.
• [ ] Readiness and liveness probes configured and tested.
• [ ] Monitoring and alerts in place for errors, latency, and resource pressure.
• [ ] Scripted rollback action ready and tested.
• [ ] Stakeholders and on-call notified and on standby.

Expected outcome: Rapid traffic rollback and a practiced, metric-driven cutover that keeps the source intact until you choose to decommission it. The achievable recovery time and recovery point depend on traffic convergence, data replication, write coordination, and tested recovery procedures.

Next steps for migration

• AKS migration guidance - start here for migration planning and inventory.
• Blue-green node pool upgrade patterns for AKS - node-pool blue/green and zero-risk upgrades.

Monitor AKS production upgrades

Monitor upgrade progress and application health with Azure Monitor managed service for Prometheus metrics, Container insights logs, and Grafana dashboards. To learn more, see the AKS monitoring overview, Container insights, and Prometheus metrics.

With AKS Automatic

AKS Automatic clusters include monitoring enabled by default:

• Managed Prometheus for metric collection
• Container insights for log collection
• Azure Monitor dashboards with Grafana for visualization

Build additional upgrade-specific monitors on top of these defaults.

Essential metrics to monitor

# upgrade-monitoring.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: upgrade-monitoring
spec:
  groups:
  - name: upgrade.rules
    rules:
    - alert: UpgradeInProgress
      expr: kube_node_spec_unschedulable > 0
      for: 1m
      annotations:
        summary: "Node upgrade in progress"

    - alert: HighErrorRate
      expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.01
      for: 2m
      annotations:
        summary: "High error rate during upgrade"

    - alert: PodEvictionFailed
      expr: increase(kube_pod_container_status_restarts_total[5m]) > 5
      for: 1m
      annotations:
        summary: "Multiple pod restarts detected"

Sample AKS upgrade dashboard signals

The following conceptual JSON groups Prometheus signals for AKS node status and application health. Adapt these signals to Azure Monitor dashboards with Grafana or Azure Managed Grafana; this example isn't an importable dashboard definition.

{
  "dashboard": {
    "title": "AKS Upgrade Dashboard",
    "panels": [
      {
        "title": "Upgrade Progress",
        "targets":
        [
          "kube_node_info",
          "kube_node_status_condition"
        ]
      },
      {
        "title": "Application Health",
        "targets":
        [
          "up{job='kubernetes-pods'}",
          "http_request_duration_seconds"
        ]
      }
    ]
  }
}

Troubleshooting guide

To learn more, see the AKS troubleshooting guide, Node and pod troubleshooting, and AKS error code PodDrainFailure.

Common issues and solutions

Emergency rollback procedures

# Quick rollback script
#!/bin/bash
echo "Initiating emergency rollback..."

# Route traffic back to the previous cluster endpoint
az network traffic-manager endpoint update \
  --resource-group traffic-rg \
  --profile-name production-tm \
  --name current-endpoint \
  --type azureEndpoints \
  --weight 0

az network traffic-manager endpoint update \
  --resource-group traffic-rg \
  --profile-name production-tm \
  --name previous-endpoint \
  --type azureEndpoints \
  --weight 100

# Verify rollback success
API_URL=<application-url>
curl -f "${API_URL}/health"
echo "Rollback completed in $(date)"

Related resources

AKS Automatic (recommended for new production clusters)

• Introduction to AKS Automatic: Features and benefits overview
• Create an AKS Automatic cluster: Get started with managed upgrades

Specialized scenarios

• Stateful workloads: Use PostgreSQL, Redis, and MongoDB upgrade patterns.
• Upgrade scenarios hub: Choose your upgrade path.
• Basic AKS upgrades: Find simple cluster version upgrades.

Supporting tools

• Auto-upgrade configuration: Use automated upgrade channels.
• Maintenance windows: Schedule upgrade windows.
• Upgrade monitoring: Use real-time upgrade alerts.

Best practices

• Cluster reliability: Design for upgrades.
• Security guidelines: Use secure upgrade practices.
• Support policies: Understand upgrade support windows.

Next tasks

For new production clusters

• Start with AKS Automatic: Create an AKS Automatic cluster to benefit from managed upgrades with built-in availability protections.
• Understand capabilities: Review Introduction to AKS Automatic for full feature details.

For existing AKS Standard clusters

• Set up monitoring: Configure upgrade notifications before your first upgrade.
• Practice safely: Test scenarios in staging by using cluster snapshots.
• Automate gradually: Start with auto-upgrade channels for nonproduction.
• Handle stateful data: Review stateful workload patterns if you run databases.