Support policies for Azure Kubernetes Service

This article describes technical support policies and limitations for Azure Kubernetes Service (AKS). It also details agent node management, managed control plane components, non-Microsoft open-source components, and security or patch management.

Service updates and releases

• For release information, see AKS release notes.
• For information on preview features, see the AKS roadmap.

Managed features in AKS

AKS is a mix of Infrastructure as a Service (IaaS) and Platform as a Service (PaaS). Base IaaS cloud components, like compute or networking components, provide access to low-level controls and customization options. By contrast, AKS provides a turnkey Kubernetes deployment that gives you a common set of configurations and capabilities you need for your cluster. As an AKS user, you have limited customization and deployment options and don't manage Kubernetes clusters directly.

With AKS, you get a fully managed control plane. The control plane contains all of the components and services you need to operate and deliver Kubernetes clusters to end users. Microsoft maintains and operates all Kubernetes components.

Microsoft manages and monitors the following components through the control plane:

• The Kubernetes API server and kubelet.
• etcd or a compatible key-value store, providing Quality of Service (QoS), scalability, and runtime.
• DNS services like CoreDNS.
• kube-proxy and cluster networking, except when BYOCNI is used.
• Any other add-ons or system component running in the kube-system namespace.

Some components, like agent nodes, have shared responsibility, where you must help maintain the AKS cluster. User input is required, for example, to apply an agent node operating system (OS) security patch.

The services are managed in the sense that Microsoft and the AKS team deploys, operates, and is responsible for service availability and functionality. Customers can't alter these managed components. Microsoft limits customization to ensure a consistent and scalable user experience.

Shared responsibility

When you create a cluster, you define the Kubernetes agent nodes that AKS creates. Your workloads run on these nodes. Keep the following agent node constraints in mind:

• Microsoft Support has limited access. Because your agent nodes run private code and store sensitive data, Microsoft Support can't sign in to, run commands on, or view logs for these nodes without your express permission or assistance.
• Use Kubernetes-native mechanisms for changes. Any modification made directly to the agent nodes through the IaaS APIs renders the cluster unsupportable. Apply changes by using Kubernetes-native mechanisms like a DaemonSet.
• Don't change system-created metadata. You can add metadata like tags and labels, but changing any system-created metadata renders the cluster unsupported.

Division of responsibility

The following matrix summarizes support responsibility across an AKS cluster, and contrasts AKS with self-managed Kubernetes that you run on-premises. In AKS, Microsoft manages the control plane and core platform. You manage your node configuration, workloads, networking configuration, and data. Shared areas are where Microsoft provides and maintains a capability and you enable, configure, or schedule it. Use the matrix as an at-a-glance guide, then see the detailed sections that follow for support specifics.

Use the following key to read the matrix:

For the shared areas in the following matrix, the table shows what Microsoft provides and what you're responsible for:

AKS support coverage

The following sections describe the supported and unsupported scenarios for AKS technical support.

Supported scenarios

Microsoft provides technical support for the following examples:

Any cluster actions taken by Microsoft or AKS are made with your consent under a built-in Kubernetes role aks-service and built-in role binding aks-service-rolebinding, which binds the role to the aks-support Microsoft Support service identity. This role enables AKS to troubleshoot and diagnose cluster issues, but it can't modify permissions or create roles or role bindings, or do other high privilege actions. Role access is only enabled under active support tickets with just-in-time (JIT) access.

Unsupported scenarios

Microsoft doesn't provide technical support for the following scenarios.

Some of these scenarios have more nuance about what Microsoft can still help with. For more information, see Details about unsupported scenarios.

Details about unsupported scenarios

Several unsupported scenarios have added nuance about what Microsoft can still help with:

• How to use Kubernetes. Microsoft Support doesn't provide advice on how to create custom ingress controllers, use application workloads, or apply non-Microsoft or open-source software packages or tools. Microsoft Support can advise on AKS cluster functionality, customization, and tuning (for example, Kubernetes operations issues and procedures).
• Non-Microsoft open-source projects. These projects aren't provided as part of the Kubernetes control plane or deployed with AKS clusters, and might include Istio, Helm, Envoy, or others. Microsoft can provide best-effort support for projects like Helm. Where the tool integrates with the Kubernetes Azure cloud provider or other AKS-specific bugs, Microsoft supports examples and applications from Microsoft documentation.
• Network customizations. For customizations other than the ones listed in the AKS documentation, Microsoft Support can't configure devices or virtual appliances meant to provide outbound traffic for the cluster, like VPNs or firewalls. On a best-effort basis, Microsoft Support might advise on the configuration needed for Azure Firewall, but not for other non-Microsoft devices.
• Non-Microsoft ingress controllers. For controllers like nginx, kong, or traefik, this includes functionality issues that arise after AKS-specific operations, like an ingress controller ceasing to work following a Kubernetes version upgrade, which might stem from incompatibilities between the ingress controller version and the new Kubernetes version. For a fully supported option, consider a Microsoft-managed ingress controller option.
• Custom DaemonSet scripts. Although using DaemonSet is the recommended approach to tune, modify, or install non-Microsoft software on agent nodes when configuration file parameters are insufficient, Microsoft Support can't troubleshoot issues arising from the custom scripts due to their custom nature.
• Standby and proactive support. Microsoft Support provides reactive support to solve active issues. Standby or proactive support to eliminate operational risks, increase availability, and optimize performance isn't covered. Eligible customers can contact their account team to get nominated for Azure Event Management service, a paid service that includes a proactive solution risk assessment and coverage during the event.
• CVEs less than 30 days old. As long as you run the updated VHD, you shouldn't have any container image CVEs with a vendor fix that's over 30 days old. It's your responsibility to update the VHD, then filter the CVE report and provide Microsoft Support a list of only the CVEs with a vendor fix over 30 days old. Microsoft then works internally to address components with a vendor fix released more than 30 days ago. Microsoft provides CVE-related support only for Microsoft-managed components, like AKS node images and managed container images deployed during cluster creation or through a managed add-on. For more information, see Vulnerability management for Azure Kubernetes Service (AKS).
• Custom code samples. Microsoft Support can provide and review small code samples within a support case to demonstrate how to use features of a Microsoft product, but can't provide custom code samples specific to your environment or application.
• Custom Azure Policy logic. Microsoft Support can provide general guidance on how custom Azure Policy definitions are applied and evaluated in AKS. Detailed troubleshooting of customer-authored policy logic (including Rego-based policies), such as why a specific policy allows or denies a workload, is generally outside the scope of support.

Bring your own CNI support boundaries

When you deploy a cluster with bring your own CNI (BYOCNI) by using --network-plugin none, Microsoft Support can't help with CNI-related issues. You're responsible for the CNI plugin lifecycle, and you should seek support from the CNI plugin vendor. Microsoft still supports issues that aren't related to the CNI.

For more information, see Bring your own CNI (BYOCNI).

AKS support coverage for agent nodes

The following sections describe the Microsoft and customer responsibilities for AKS agent nodes.

The following table summarizes agent node responsibilities at a glance. The sections that follow provide the detail.

Microsoft responsibilities for AKS agent nodes

Microsoft and you share responsibility for Kubernetes agent nodes where:

• The base OS image has required additions like monitoring and networking agents.
• The agent nodes receive OS patches automatically.
• The system automatically remediates issues with the Kubernetes control plane components that run on the agent nodes. These components include the following items:
  • kube-proxy
  • Networking tunnels that provide communication paths to the Kubernetes control plane
  • kubelet
  • containerd

If an agent node isn't operational, AKS might restart individual components or the entire agent node. These restart operations are automated and provide autoremediation for common issues. To learn more about the autoremediation mechanisms, see Node Auto-Repair.

Customer responsibilities for AKS agent nodes

Microsoft provides patches and new images for your image nodes weekly. To keep your agent node OS and runtime components up to date, you should apply these patches and updates regularly either manually or automatically. Microsoft doesn't support node images that are older than 90 days. For more information, see:

• Manually upgrade AKS node images.
• Automatically upgrade AKS node images.

Similarly, AKS regularly releases new Kubernetes patches and minor versions. These updates can contain security or functionality improvements to Kubernetes. You're responsible to keep your clusters' Kubernetes version updated and according to the AKS Kubernetes support version policy.

User customization of agent nodes

AKS manages the lifecycle and operations of agent nodes on your behalf and modifying the IaaS resources associated with the agent nodes is not supported. An example of an unsupported operation is customizing a node pool virtual machine scale set by manually changing configurations in the Azure portal or from the API.

For workload-specific configurations or packages, AKS recommends using a Kubernetes DaemonSet.

Using Kubernetes privileged DaemonSet and init containers enables you to tune/modify or install non-Microsoft software on cluster agent nodes. Examples of such customizations include adding custom security scanning software or updating sysctl settings.

While this path is recommended if the above requirements apply, AKS engineering and support can't help troubleshoot or diagnose modifications that render the node unavailable due to a custom deployed DaemonSet.

Security issues and patching

If a security flaw is found in one or more of the managed components of AKS, the AKS team patches all affected clusters to mitigate the issue. Alternatively, the AKS team provides you with upgrade guidance.

For agent nodes affected by a security flaw, Microsoft notifies you with details on the impact and the steps to fix or mitigate the security issue.

Node maintenance and access

Although you can sign in to agent nodes and change them, don't perform this operation. Changes can make a cluster unsupportable.

Network ports, access, and NSGs

You can customize network security groups (NSGs) only on custom subnets. The following table shows where you can customize NSGs:

AKS has egress requirements to specific endpoints. To control egress and ensure the necessary connectivity, see limit egress traffic. For ingress, the requirements are based on the applications you deploy to the cluster.

Stopped, deallocated, and Not Ready nodes

The following table summarizes what happens to an AKS cluster in each lifecycle state and the associated timeline:

If you don't need your AKS workloads to run continuously, stop the AKS cluster, which stops all node pools and the control plane, and start it again when needed. Manually deallocating nodes by using the IaaS APIs, the Azure CLI, or the Azure portal isn't a supported way to stop a cluster.

AKS reserves the right to archive control planes configured out of support guidelines for periods of 30 days or more. AKS maintains backups of cluster etcd metadata and can reallocate the cluster on any PUT operation that brings it back into support, like an upgrade or scale to active agent nodes.

Unsupported alpha and beta Kubernetes features

AKS supports stable and beta features within the upstream Kubernetes project, but not alpha features unless otherwise documented. The following table summarizes support by feature type:

Preview features or feature flags

For features and functionality that require extended testing and user feedback, Microsoft releases new preview features or features behind a feature flag. Consider these features as prerelease or beta features.

Preview features or feature-flag features aren't meant for production. Ongoing changes in APIs and behavior, bug fixes, and other changes can result in unstable clusters and downtime.

Features in public preview fall under best effort support, as these features are in preview and aren't meant for production. The AKS technical support teams provide support during business hours only. For more information, see Azure Support FAQ.

Upstream bugs and issues

Given the speed of development in the upstream Kubernetes project, bugs invariably arise. Some of these bugs can't be patched or worked around within the AKS system. Instead, bug fixes require larger patches to upstream projects (like Kubernetes, node or agent operating systems, and kernel). For components that Microsoft owns (like the Azure cloud provider), AKS and Azure personnel are committed to fixing issues upstream in the community.

When the root cause of a technical support issue is due to one or more upstream bugs, AKS support and engineering teams will:

• Identify and link the upstream bugs with any supporting details to help explain why this issue affects your cluster or workload. Customers receive links to the required repositories so they can watch the issues and see when a new release provides fixes.
• Provide potential workarounds or mitigation. If the issue can be mitigated, a known issue is filed in the AKS repository. The known-issue filing explains:
  • The issue, including links to upstream bugs.
  • The workaround and details about an upgrade or another persistence of the solution.
  • Rough timelines for the issue's inclusion, based on the upstream release cadence.

Related content

• AKS pricing tiers
• Supported Kubernetes versions in AKS
• Long-term support for AKS
• Automatically upgrade an AKS cluster
• Automatically upgrade AKS node OS images
• Vulnerability management for AKS
• AKS node auto-repair
• Bring your own CNI (BYOCNI)