Fix 'Kernel panic' in Linux

Understanding the Linux Kernel Panic

A kernel panic is the Linux kernel's equivalent of a critical system crash. When the kernel encounters an unrecoverable error in a context where it cannot safely continue execution, it halts the entire system and displays a diagnostic message. This is not a normal application crash — it means the core of the operating system itself has reached a fatal state.

What Is a Kernel Panic?

At the most fundamental level, a kernel panic occurs when the kernel detects an internal fatal error from which it cannot safely recover. The kernel's primary job is to manage hardware, memory, processes, and system calls. When something goes wrong in kernel space — such as a memory corruption in kernel data structures, an invalid pointer dereference, or a hardware fault that the kernel cannot isolate — the kernel voluntarily stops everything to prevent further data corruption or damage. On modern systems, the kernel will dump its current state to the console (or to a serial console, or to a kdump mechanism) and then halt or reboot depending on configuration.

The panic message typically includes:

Panic type — a string describing what went wrong (e.g., "Kernel panic - not syncing: Attempted to kill init!")
Register dump — CPU registers at the moment of panic
Stack trace — the call trace leading to the panic
Memory info — memory zones, active allocations
Process context — which process was running

Why Fixing Kernel Panics Matters

A kernel panic is not merely an inconvenience. It represents:

Data integrity risk — filesystem operations may be interrupted mid-write, leading to corruption
Service downtime — production servers go offline until someone physically intervenes or a watchdog reboots the machine
Debugging complexity — unlike userspace crashes, kernel panics leave the system in a state where normal debugging tools are unavailable
Hardware insight — recurring panics often reveal failing RAM, overheating CPUs, or buggy firmware
Kernel bug detection — panics can expose bugs in kernel code, especially after updates or when using out-of-tree modules

For developers and system administrators, understanding how to systematically diagnose and resolve kernel panics is a critical skill. It transforms an opaque crash into a structured debugging process.

Common Causes of Kernel Panics

Kernel panics fall into several broad categories. Knowing these helps narrow the investigation quickly:

Hardware faults — bad RAM (bit flips in kernel memory), failing storage controllers, CPU cache errors, overheating causing thermal shutdowns, or faulty PCIe devices
Kernel bugs — null pointer dereferences, use-after-free, race conditions in kernel code (especially in drivers)
Out-of-tree kernel modules — proprietary drivers (NVIDIA, virtualbox, etc.) that do not match the current kernel version
Memory exhaustion — the kernel runs out of memory in a context where the OOM killer cannot intervene
Corrupted initramfs or root filesystem — the kernel cannot mount root or execute init
Firmware / ACPI bugs — BIOS or UEFI firmware provides incorrect tables, causing the kernel to access invalid memory regions
Incorrect kernel parameters — boot arguments that conflict with hardware or kernel configuration

Analyzing the Panic Message

When a kernel panic occurs, the first step is to capture the panic output. On a local console, the text is displayed directly. On headless servers, the output may go to a serial console or be captured via IPMI SOL (Serial Over LAN). In virtualized environments, the hypervisor console shows the panic. The key fields to examine:

# Example panic output (excerpt)
Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
CPU: 0 PID: 1 Comm: init Not tainted 5.15.0-91-generic #101-Ubuntu
Hardware name: Dell Inc. PowerEdge R640/0H28J, BIOS 2.14.1 12/15/2023
Call Trace:
 <TASK>
 dump_stack_lvl+0x64/0x80
 panic+0x11b/0x2f0
 do_exit+0x9a4/0x9b0
 do_group_exit+0x3a/0xa0
 __x64_sys_exit_group+0x14/0x20
 do_syscall_64+0x59/0x90
 ? syscall_exit_to_user_mode+0x17/0x28
 ? do_syscall_64+0x69/0x90
 entry_SYSCALL_64_after_hwframe+0x6e/0xd8
RIP: 0033:0x7f8a2c0a1d10
 </TASK>
Kernel Offset: 0x2c000000 from 0xffffffff81000000
---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100 ]---

Break down the information:

"Attempted to kill init!" — the init process (PID 1) exited. The kernel cannot continue without init. The exit code 0x00000100 often means a segmentation fault or missing shared library.
Call Trace — shows the kernel function chain. Here the panic originated from do_exit(), meaning a process exit triggered the panic (init died).
Not tainted — the kernel has not loaded proprietary modules or encountered hardware errors that would taint it. This increases confidence that the issue is a configuration problem rather than a kernel bug.
RIP — the instruction pointer in userspace when init crashed. Useful for debugging init with gdb if a core dump is available.

Another common panic signature:

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
CPU: 2 PID: 1 Comm: swapper/0 Not tainted 6.5.0-14-generic #14-Ubuntu
Call Trace:
 <TASK>
 panic+0x11b/0x2f0
 mount_block_root+0x1c2/0x1e0
 prepare_namespace+0x136/0x160
 kernel_init_freeable+0x2b4/0x2c0
 kernel_init+0x16/0x130
 ret_from_fork+0x2c/0x40
 </TASK>
---[ end Kernel panic - not syncing: VFS: Unable to mount root fs ]---

This indicates the kernel could not find or mount the root filesystem. Common reasons: missing storage driver in initramfs, incorrect root= parameter, or corrupted initramfs.

Practical Debugging and Recovery Steps

Booting into a Rescue Environment

When a system panics on boot, you need an alternate boot method. The most common approach is using a live USB or netboot image of the same distribution. For GRUB-based systems, you can also edit the boot entry at startup to modify kernel parameters.

To access the GRUB command line during boot:

# Press 'e' at the GRUB menu on the target boot entry
# Then add/modify parameters on the 'linux' line

# Example: boot into single-user mode with verbose logging
linux /boot/vmlinuz-5.15.0-91-generic root=UUID=abc123 ro single ignore_loglevel earlyprintk=ttyS0,115200

# Or boot with a custom init to bypass normal init
linux /boot/vmlinuz-5.15.0-91-generic root=UUID=abc123 ro init=/bin/sh

Using init=/bin/sh boots directly into a root shell. The root filesystem will be mounted read-only. You can remount it read-write with:

mount -o remount,rw /

This gives you a minimal environment to inspect logs, check filesystems, and repair boot-critical files.

Examining Kernel Logs

If the system panics after booting far enough to start the logging subsystem, the panic messages and preceding kernel log entries may survive in the persistent log. After booting a rescue environment, mount the target root filesystem and examine:

# Mount the problematic system's root (assuming it's on /dev/sda2)
mount /dev/sda2 /mnt

# Check the persistent systemd journal
journalctl -D /mnt/var/log/journal --no-pager -b -1 | grep -i -E "panic|oops|bug|error|fail"

# Or check the traditional syslog
less /mnt/var/log/syslog
less /mnt/var/log/messages

# Check dmesg files if they were saved by a crash handler
ls -la /mnt/var/crash/

Look for kernel OOPS messages that precede the panic. An OOPS is a non-fatal kernel error that often leads to a panic when it occurs in an unsafe context. The log entries immediately before a panic often reveal the root cause.

Checking Hardware Issues

Hardware-induced panics are often intermittent and frustrating. Systematic testing is required:

# Memory test (run from rescue environment or GRUB)
# Add memtest to GRUB or boot a memtest86+ ISO
# Let it run for at least 2 complete passes

# Check for MCE (Machine Check Exception) errors in logs
journalctl -D /mnt/var/log/journal | grep -i "mce"
dmesg | grep -i "mce\|machine check"

# Examine EDAC error counts (if ECC memory is present)
grep . /sys/devices/system/edac/mc/mc*/csrow*/ch*_ce_count
grep . /sys/devices/system/edac/mc/mc*/csrow*/ch*_ue_count

# Check disk health
smartctl -a /dev/sda

# Check for thermal issues
sensors  # from lm-sensors package
journalctl -D /mnt/var/log/journal | grep -i "thermal\|critical temp"

If Machine Check Exception (MCE) panics appear, the CPU itself detected a hardware error. These are serious and often indicate failing processors, bad memory, or PCIe bus errors. The MCE log entries contain bank numbers and addresses that can pinpoint the faulty DIMM slot or CPU core.

Fixing Kernel Module Problems

Out-of-tree kernel modules (like proprietary NVIDIA drivers, VirtualBox guest additions, or custom DKMS modules) are a leading cause of post-update panics. When the kernel version changes, these modules must be recompiled against the new kernel headers. If the rebuild fails or the module loads before it's ready, a panic ensues.

To recover from a bad module-induced panic:

# Boot with module blacklisting via kernel parameter
linux /boot/vmlinuz-6.5.0-14-generic root=UUID=abc123 ro module_blacklist=nvidia,nvidia_modeset,nvidia_drm

# Or boot into rescue mode and disable the module permanently
mount /dev/sda2 /mnt
chroot /mnt

# Remove the offending module from initramfs if it's loaded early
lsinitramfs /boot/initrd.img-6.5.0-14-generic | grep nvidia

# Regenerate initramfs without the module
echo "nvidia" >> /etc/initramfs-tools/scripts/init-premount/disable-nvidia
update-initramfs -u -k all

# Or simply remove the module package
apt remove nvidia-driver-550  # Debian/Ubuntu
dnf remove kmod-nvidia        # Fedora/RHEL

# For DKMS modules, check build status
dkms status
dkms remove nvidia/550.107.02 --all

# Update module dependencies and rebuild initramfs
depmod -a
update-initramfs -u -k all

On systems using KMS (Kernel Mode Setting), a bad graphics driver can cause a panic early in the boot process. Adding nomodeset to the kernel command line disables KMS and lets you boot with basic framebuffer support to perform repairs.

Repairing Corrupted Initramfs

A corrupted initramfs (initial ramdisk) prevents the kernel from loading modules needed to mount the root filesystem. The classic symptom is:

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

Recovery steps:

# Boot from live USB, mount the target system
mount /dev/sda2 /mnt
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys
chroot /mnt

# Regenerate initramfs for the installed kernel
update-initramfs -u -k all

# Or for a specific kernel version
update-initramfs -c -k 5.15.0-91-generic

# Verify the new initramfs is not empty
lsinitramfs -l /boot/initrd.img-5.15.0-91-generic | head -20

# Check that storage drivers are included
lsinitramfs /boot/initrd.img-5.15.0-91-generic | grep -E "nvme|ahci|virtio|scsi"

# On Fedora/RHEL, use dracut
dracut --force --kver 5.15.0-91-generic

# Ensure the root filesystem UUID in initramfs matches the actual disk
blkid /dev/sda2
cat /boot/grub/grub.cfg | grep "root=UUID"

If the initramfs build process fails due to missing tools or configuration, check /etc/initramfs-tools/initramfs.conf (Debian/Ubuntu) or /etc/dracut.conf (Fedora/RHEL) for incorrect settings like missing MODULES= directives.

Fixing Bootloader Configuration

Incorrect bootloader configuration is a frequent source of panics. The GRUB configuration may reference a kernel or initramfs that does not exist, or the root= parameter may point to the wrong partition.

# From rescue environment, chroot and fix GRUB
chroot /mnt

# Regenerate GRUB configuration
grub-mkconfig -o /boot/grub/grub.cfg

# Or manually edit /etc/default/grub
# Ensure GRUB_CMDLINE_LINUX contains correct root= parameter
vi /etc/default/grub
# Example line:
GRUB_CMDLINE_LINUX="root=UUID=abc123-def456 ro quiet"

# Update GRUB after editing
update-grub   # Debian/Ubuntu
grub2-mkconfig -o /boot/grub2/grub.cfg  # Fedora/RHEL

# Check that the UUID matches the actual partition
blkid /dev/sda2
ls -la /dev/disk/by-uuid/ | grep abc123

# If using LVM or LUKS, ensure the initramfs includes the necessary hooks
# For LUKS: check /etc/crypttab and initramfs hooks
lsinitramfs /boot/initrd.img-* | grep crypt

For UEFI systems, also verify that the EFI System Partition (ESP) is intact and that the boot entries in NVRAM point to the correct EFI executable:

# Check ESP mount
mount /dev/sda1 /mnt/boot/efi
ls -la /mnt/boot/efi/EFI/

# Use efibootmgr to list and fix boot entries
efibootmgr -v

Recovering from Bad Kernel Updates

A kernel update that introduces a regression or is incompatible with hardware can cause panics on every boot. Recovery strategy:

# Boot from the previous kernel via GRUB menu
# At boot, select "Advanced options" then choose an older kernel

# Once booted, pin the working kernel and remove the bad one
# Debian/Ubuntu:
apt-mark hold linux-image-5.15.0-91-generic  # Pin good kernel
apt remove linux-image-6.5.0-14-generic       # Remove bad kernel
update-grub

# Fedora/RHEL:
dnf remove kernel-6.5.0-14                     # Remove bad kernel
# Keep at least 3 kernels installed via /etc/dnf/dnf.conf:
# max_install_limit=3

# Check which kernels are installed
dpkg -l | grep linux-image   # Debian/Ubuntu
rpm -qa | grep kernel        # Fedora/RHEL

# If the bad kernel is the only one installed, boot live USB and chroot
# Then install an older kernel from repository cache or download
apt install linux-image-5.15.0-91-generic   # specific version

For critical production systems, always test kernel updates on a non-production clone first. Maintain a known-good kernel version as a fallback boot entry in GRUB.

Best Practices to Prevent Kernel Panics

Keep a rescue boot entry — configure GRUB with a fallback entry that boots with nomodeset, init=/bin/sh, and verbose logging.
Use kdump / kexec — configure kdump to capture kernel crash dumps automatically. This preserves the panic state for offline analysis with crash tool.
Enable persistent kernel log — configure CONFIG_PSTORE and use pstore to save kernel logs across reboots, even when the filesystem is not flushed.
Test kernel modules — before updating a kernel, verify that all DKMS modules build successfully with dkms status and dkms autoinstall.
Monitor hardware health — run regular memory tests (memtest86+), SMART disk checks, and EDAC error monitoring. Replace components showing correctable error counts that increase over time.
Avoid tainting the kernel unnecessarily — tainted kernels (marked with flags like 'O' for out-of-tree module) make debugging harder because kernel developers may disregard bug reports.
Document kernel parameters — keep a record of custom kernel command-line parameters and their purpose. Incorrect parameters passed during recovery can introduce new panics.
Use watchdog timers — hardware or software watchdogs can automatically reboot a panicked system, reducing downtime while you investigate the root cause via persistent logs or kdump.
Lock kernel versions in production — use package manager holds or pinning to prevent automatic kernel updates from introducing regressions.
Always verify initramfs after updates — run lsinitramfs or lsinitrd after kernel updates to confirm that storage drivers and filesystem modules are present.

Using kdump for Post-Mortem Analysis

Setting up kdump allows the kernel to save a crash dump to a reserved memory region and then reboot into a capture kernel that writes the dump to disk. This is invaluable for analyzing panics that are hard to reproduce:

# Install kdump tools
apt install kdump-tools   # Debian/Ubuntu
dnf install kexec-tools   # Fedora/RHEL

# Reserve crash kernel memory via kernel parameter
# Add to /etc/default/grub: GRUB_CMDLINE_LINUX="crashkernel=256M"
grub-mkconfig -o /boot/grub/grub.cfg

# Enable kdump service
systemctl enable kdump.service
systemctl start kdump.service

# After a panic, the dump is saved to /var/crash/
# Analyze with the crash tool
crash /usr/lib/debug/boot/vmlinux-5.15.0-91-generic /var/crash/202405011230/vmcore

# Inside crash, examine backtrace, registers, and memory
crash> bt          # backtrace of panicking task
crash> log         # kernel log buffer from the crash
crash> ps          # process list at crash time
crash> files PID   # open files of a process
crash> kmem -i     # memory info
crash> mod         # loaded modules list

This transforms a cryptic panic message into a full interactive debugging session where you can inspect every aspect of kernel state at the moment of failure.

Handling Specific Panic Types

Some panics have well-known signatures and targeted fixes:

# "Kernel panic - not syncing: Attempted to kill init!"
# Cause: init (PID 1) crashed or exited
# Fix: Check what init binary/script does. Boot with init=/bin/sh to bypass.
# Common triggers: missing /lib in chroot, corrupted /sbin/init, missing libc

# "Kernel panic - not syncing: Out of memory and no killable processes"
# Cause: System-wide OOM with no eligible process to kill
# Fix: Increase RAM, reduce kernel memory pressure, check for memory leaks
# Boot parameter to increase reclaim: vm.min_free_kbytes=65536

# "Kernel panic - not syncing: soft lockup - CPU stuck"
# Cause: A CPU has been stuck in kernel mode for too long (usually >20s)
# Fix: Often hardware or driver issue. Boot with nosoftlockup=0 to disable detector
# Investigate which driver/function is spinning in the call trace

# "Kernel panic: Fatal exception in interrupt"
# Cause: An interrupt handler encountered a fatal error
# Fix: Usually bad hardware generating spurious interrupts. Check IRQ stats in /proc/interrupts
# Try booting with irqpoll or pci=nomsi to rule out MSI/MSI-X issues

Conclusion

Fixing a kernel panic in Linux is a systematic process that moves from panic message analysis to hypothesis testing to targeted recovery. The key is understanding that a kernel panic is a symptom with a traceable root cause — whether hardware failure, kernel bug, module incompatibility, or configuration error. By mastering boot recovery techniques, log analysis, initramfs management, and post-mortem tools like kdump, developers and administrators can transform kernel panics from catastrophic black boxes into structured, solvable problems. The best defense remains proactive measures: hardware monitoring, kernel update testing, and maintaining fallback boot configurations that give you a reliable path back into the system when the primary kernel path fails.