We have an Azure Linux Virtual Machine that has suddenly become inaccessible.

Hello @Vipul Om

Thanks for the detailed write-up you've given us exactly what we need to diagnose this. Good news: the root cause is clear, and there's a structured recovery path that preserves your data.

Root Cause: Two Related Issues

The Serial Console output tells the whole story:

1.Kernel panic - not syncing: Attempted to kill init! This is the primary cause of the outage. The init process (systemd/PID 1) crashed during boot, which is why SSH, HTTP, and Run Command all fail — the OS never fully starts. This is commonly triggered by a missing or corrupted initramfs, a failed kernel/package update, or corrupted system core libraries.

2.WALinuxAgent `[Errno 2] No such file or directory: 'python3' This is a symptom, not a separate problem. Python3 being missing points to broader system file corruption, consistent with a bad package update or incomplete OS upgrade that left critical binaries in a broken state.

Since Redeploy + Reapply didn't help (expected — those don't fix OS-level corruption), the right path is an offline disk repair.

Recovery Plan — Start Here (Fastest Path)

Step 1: Try ALAR Automated Repair First

Microsoft's Azure Linux Auto Repair (ALAR) tool can automatically regenerate a missing initramfs and GRUB configuration in an unattended mode — no manual disk swap required. Run the following

This creates a temporary repair VM, fixes the initramfs/GRUB config on a copy of your OS disk, swaps the disk back, and cleans up the repair VM automatically. **Your data is not touched.

Recover Azure Linux VM from kernel-related boot issues
https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/kernel-related-boot-issues

Step 2: If Step 1 Doesn't Resolve It — ALAR Kernel Repair

If the issue is specifically with a broken kernel (not just missing initramfs), use the kernel rollback option:

This replaces the broken kernel with the previously installed working version.

Use Azure Linux Auto Repair (ALAR) to fix a Linux VM
https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/repair-linux-vm-using-alar

Step 3: If ALAR Doesn't Fully Resolve It — Manual Repair VM**

If the corruption goes deeper (missing core system libraries, which the python3 error hints at), you'll need to go in manually via a repair VM and chroot:

1. Create a repair VM with the OS disk attached:

az vm repair create -g <YourResourceGroup> -n <YourVMName> \
  --repair-username repairadm --repair-password 'Repair@1234!'

2. SSH into the repair VM, enter the chroot environment, and investigate:

3. Reinstall missing packages if identified (inside chroot), regenerate initramfs, and update GRUB:

# Regenerate initramfs (Ubuntu example)
update-initramfs -u -k all
update-grub
# Regenerate initramfs (RHEL/CentOS example)
dracut -f --regenerate-all
grub2-mkconfig -o /boot/grub2/grub.cfg

4. Restore the disk back to the original VM:

az vm repair restore -g <YourResourceGroup> -n <YourVMName>

Repair a Linux VM by using the Azure Virtual Machine repair commands
https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/repair-linux-vm-using-azure-virtual-machine-repair-commands

Troubleshoot a Linux VM by attaching the OS disk to a recovery VM
https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/troubleshoot-recovery-disks-portal-linux

After the VM Boots Fix WALinuxAgent

Once the OS is back up, fix the python3/WALinuxAgent issue:

Troubleshoot the Azure Linux Agent
https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/linux-azure-guest-agent

Last Resort: Restore from Your Snapshot

You mentioned a snapshot was taken before troubleshooting. If all repair attempts fail, you can restore from that snapshot to get the VM back quickly, then investigate the root cause separately on a copy.

How to restore Azure VM data in Azure portal
https://learn.microsoft.com/en-us/azure/backup/backup-azure-arm-restore-vms

Recommended Order of Actions

Start with Step 1 it handles the majority of kernel panic scenarios automatically and is the least invasive option.

The data on the disk is safe throughout all these steps.

Hope this gets you back online quickly!

Thanks,
Manish.

Hello @Vipul Om,

Thank you for reaching out Q&A forum.

Step 1: Check the VM Power State

In the Azure Portal → Virtual Machines → your VM → Overview, confirm the Status field.

Status Meaning

Running OS-level or network issue

Stopped (deallocated) VM was stopped — just start it

Failed Platform-level provisioning failure

Step 2: Open Boot Diagnostics (Serial Console)

Azure Portal → VM → Support + troubleshooting → Boot diagnostics → Serial log

This is the fastest way to determine the root cause:

What you see in the serial log Likely cause

Kernel panic - not syncing OS/kernel corruption — needs rescue VM

GRUB menu stuck Boot loader issue

Give root password for maintenance fstab mount failure

Started Session / login prompt OS booted fine → network/NSG issue

cloud-init errors VM agent / provisioning failure

Blank / no output Host-level issue or GPU/disk failure

Step 3:

Kernel Panic or filesystem errors → OS-level recovery needed

Detach the OS disk and attach it to a Rescue VM as a data disk

Run fsck, audit /etc/fstab, check /boot for missing initramfs

See the detailed recovery steps in my previous response above

VM appears booted but SSH/HTTP still times out → Network issue

Check these in order:

NSG rules — Does an inbound rule allow port 22 (SSH) or 80/443 from your source IP? (Not just "any")

Azure Firewall / UDR — Is there a route table sending traffic to a firewall that may be blocking it?

sshd service — Use Azure Run Command (if available) to check:

bash

systemctl status sshd

Host-based firewall — Check iptables or firewalld rules inside the VM via Run Command

VM was recently resized, updated, or a disk was attached → Configuration change issue

Review Azure Activity Log (Portal → VM → Activity log) for any recent changes

Check if a failed apt upgrade / yum update broke a kernel or removed Python 3

If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

Hello,

The issue may come from a broken initramfs or python3 not installed (which can break initramfs)

You can try to access the VM via Serial Console (bypasses NSG and SSH timeouts) then apply these steps:

1. When VM is booting up, repeatedly press ESC or Shift to show up GRUB
2. Select Recovery Mode if available, if not, select Previous Kernel Version
3. Fix python3/initramfs installation :
   apt update apt install --reinstall python3 python3-minimal initramfs-tools
4. Update initramfs: update-initramfs -u -f reboot

If kernel panic still appears after trying to enter Recovery Mode/Previous Kernel in GRUB, the initramfs is fully broken. You should create a rescue VM since you made a snapshot of the disk.