Fix 'Kernel panic' in Linux: Complete Troubleshooting Guide

Understanding the Kernel Panic

A kernel panic is the Linux kernel's last-resort safety mechanism when it encounters a fatal error from which it cannot safely recover. Instead of risking data corruption, hardware damage, or cascading failures, the kernel immediately halts all processing, displays a diagnostic message, and either freezes the system or triggers a reboot depending on configuration. The term "panic" originates from the Unix tradition, where the kernel function panic() is called to handle unrecoverable situations.

What Exactly Happens During a Panic?

When the kernel detects a critical inconsistency—such as a memory corruption in kernel space, an invalid pointer dereference, a hardware fault that cannot be masked, or a fatal exception during interrupt handling—it calls panic(). This function:

Disables all interrupt handling to prevent further damage.
Outputs a stack trace and register dump to the console (and optionally to serial/netconsole).
Flashes the keyboard LEDs on some systems (the famous "blinking caps lock" indicator).
If the panic= boot parameter is set, it may reboot after a given timeout; otherwise it halts forever.

Why Troubleshooting Panics Matters

In production servers, embedded devices, or any Linux environment, a kernel panic means total service outage. For developers and system administrators, panics are not just crashes—they are diagnostic signals that reveal deep bugs in kernel code, device drivers, or hardware. Understanding and resolving panics directly improves system reliability, prevents data loss, and helps maintain uptime in mission-critical deployments.

Gathering Information: The First Step

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

Before you can fix a kernel panic, you need to capture the panic message. Since the system typically freezes or reboots, the message vanishes unless you take steps to preserve it. This section covers the essential methods to retrieve the panic log.

Capturing Panic Output via Serial Console or Netconsole

For headless servers or embedded boards, a serial console is the gold standard. By connecting a null-modem cable or a USB-serial adapter, you can log the entire kernel output, including panics, to another machine. Alternatively, netconsole transmits kernel messages over UDP to a remote syslog server, allowing you to capture panics even when local storage is inaccessible.

# Example: enable netconsole on the panicking host (eth0, UDP to 192.168.1.100:6666)
# Add this to kernel boot parameters:
netconsole=@/eth0,@192.168.1.100/

# Or dynamically (requires root):
modprobe netconsole
echo "enabled" > /sys/module/netconsole/parameters/enabled
echo "eth0" > /sys/module/netconsole/parameters/netdev
echo "192.168.1.100" > /sys/module/netconsole/parameters/remote_ip
echo "6666" > /sys/module/netconsole/parameters/remote_port

Using the Kernel Ring Buffer (dmesg) After a Reboot

If the system reboots after a panic (due to panic= timeout or watchdog), the kernel ring buffer contents are lost—unless a mechanism like pstore or persistent RAM (PSTORE/RAMOOPS) is configured. However, on many systems, the firmware or a crash dump mechanism may preserve a fragment of the log. Check immediately after reboot with:

dmesg -T | less

Look for lines containing Kernel panic, BUG, or Oops. Note that if the system did a hard reset, the ring buffer may be empty.

Persistent Logging with systemd-journald

Modern distributions using systemd often keep logs from previous boots. You can retrieve messages from the boot that crashed using:

# List available boots
journalctl --list-boots

# Show logs from the previous boot (-1), filtering for panic
journalctl -b -1 --no-pager | grep -i -E "panic|Oops|BUG|Call trace"

# Or dump the entire previous boot log for offline analysis
journalctl -b -1 > last_boot.log

If journald storage is volatile (default on some embedded systems), consider enabling persistent storage by creating /var/log/journal and ensuring the directory exists.

Decoding the Panic Message

A typical panic output contains a wealth of low-level data. Understanding it allows you to pinpoint the exact function, module, or hardware interaction that caused the crash.

The Anatomy of a Panic Dump

Here is an annotated example of a panic message captured from a real system:

Kernel panic - not syncing: Fatal exception in interrupt
CPU: 2 PID: 0 Comm: swapper/2 Not tainted 5.15.0-91-generic #101-Ubuntu
Hardware name: Dell Inc. PowerEdge R640/0H28TR, BIOS 2.14.2 03/15/2023
Call Trace:
 <IRQ>
 dump_stack+0x6c/0x9e
 panic+0x11b/0x2e3
 ? notifier_call_chain+0x56/0x80
 die+0x1a3/0x1d0
 do_trap+0x68/0xc0
 ? do_IRQ+0x6c/0xe0
 do_handle_exception+0x7b/0x100
 do_general_protection+0x8c/0x140
 ? asm_exc_general_protection+0x10/0x20
 RIP: 0010:mpt3sas_scsih_issue_tm+0x1a5/0x1e0 [mpt3sas]
 Code: 48 8b 43 08 48 8b 50 10 48 85 d2 74 0f 48 89 50 08 48 89 28 48 89 43 08 eb 0c ...
 RSP: 0018:ffffb1e280180bc0 EFLAGS: 00010082
 RAX: dead000000000122 RBX: ffff9a0e8e1c0000 RCX: 0000000000000000
 ...
 Kernel panic - not syncing: Fatal exception in interrupt

Key elements to extract:

Panic reason: The string after not syncing: tells you the immediate cause (e.g., Fatal exception in interrupt).
CPU and Process: CPU: 2 PID: 0 Comm: swapper/2 — PID 0 is the idle thread, indicating the crash happened in interrupt context, not a user process.
Call Trace: The sequence of functions leading to the panic. Read from bottom to top: the last executed function is at the top. Here, mpt3sas_scsih_issue_tm in the mpt3sas driver triggered a general protection fault.
Register dump: Values like RIP (instruction pointer), RAX, RBX can be matched to assembly code. The poisoned value dead000000000122 suggests use-after-free or a stale pointer.

Resolving Addresses to Symbols Using addr2line and gdb

If the panic log only shows hexadecimal addresses (common with stripped kernels), you can translate them to function names and source lines using addr2line or gdb, provided you have the corresponding vmlinux debug image.

# Example: resolve an address from the panic RIP
addr2line -e /usr/lib/debug/boot/vmlinux-5.15.0-91-generic ffffffff9a012345

# Or using gdb to get more context
gdb /usr/lib/debug/boot/vmlinux-5.15.0-91-generic -ex "list *0xffffffff9a012345" -ex "quit"

Make sure the kernel image matches exactly the version and build of the panicking kernel. Distribution debug packages (e.g., linux-image-$(uname -r)-dbg on Debian/Ubuntu) provide these symbols.

Common Causes and Practical Fixes

Most kernel panics fall into predictable categories. Here are the most frequent triggers and step-by-step solutions.

Faulty Kernel Modules (Drivers)

Third-party or buggy drivers are the number one cause. If the panic trace points to a module (like [mpt3sas] above), immediately suspect it. To confirm, blacklist the module and reboot:

# Temporarily prevent loading of the module
sudo modprobe -r mpt3sas   # unload if currently loaded (careful: may crash)
# To persist, create a blacklist file:
echo "blacklist mpt3sas" | sudo tee /etc/modprobe.d/disable-mpt3sas.conf
sudo update-initramfs -u   # update initrd so early boot doesn't load it
sudo reboot

If the system boots stably without the module, you've identified the culprit. Check for known bugs in the module version, apply vendor patches, or use an alternative driver.

Hardware Issues: Memory, CPU, PCI Subsystem

Bad RAM, overheating CPUs, or faulty PCIe devices often cause random panics with no consistent module name. To test hardware:

Memory: Boot memtest86+ from GRUB and run a full pass. Even a single error indicates a DIMM failure. Replace or re-seat modules.
CPU / Machine Check Exceptions (MCE): Panics with Machine check in the log indicate a CPU or platform hardware error. Install mcelog (or use the new rasdaemon) and decode the MCE:
```
mcelog --ascii < /var/log/mcelog
```
PCIe Errors: Use lspci and dmesg to look for AER (Advanced Error Reporting) messages. Try removing suspect PCIe cards or moving them to different slots.

Corrupted Filesystem or Root Device Problems

If the panic occurs early in boot with messages like VFS: Unable to mount root fs or Kernel panic - not syncing: No working init found, the kernel cannot find or mount the root filesystem. Check:

GRUB boot parameters: root=UUID=... or root=/dev/sdaX — ensure the UUID matches the actual root partition (use blkid).
Initrd/initramfs corruption: Regenerate with update-initramfs -u -k all.
Kernel mismatch: A newly installed kernel may lack the driver for your storage controller (e.g., NVMe, RAID). Boot the previous kernel and verify module inclusion.

Incorrect Kernel Boot Parameters (cmdline)

Sometimes the panic is triggered by a kernel feature that interacts badly with your hardware. You can disable or modify behavior via GRUB:

# Edit /etc/default/grub, add to GRUB_CMDLINE_LINUX_DEFAULT:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash panic=30 oops=panic acpi=off"

Useful debugging parameters:

panic=0 — kernel will not automatically reboot; stays frozen for manual capture.
oops=panic — treat a kernel oops (less severe) as a panic, forcing a halt for diagnosis.
acpi=off / noapic / irqpoll — disable power management or interrupt routing if hardware ACPI tables are broken.
nomodeset — prevents graphics driver modesetting, useful when GPU drivers cause panics.
initcall_debug — prints detailed timestamps during driver initialization, helping identify which initcall panics.

Out-of-Tree Drivers (DKMS) and Proprietary Modules

NVIDIA, VirtualBox, and proprietary wireless drivers are notorious for panics on kernel updates. When a kernel ABI changes, DKMS automatically rebuilds these modules, but bugs persist. To isolate:

# List all out-of-tree modules
lsmod | grep -E "nvidia|vboxdrv|wl"

# Remove them temporarily
sudo modprobe -r nvidia_drm nvidia_modeset nvidia
sudo modprobe -r vboxdrv

# Prevent loading on boot
echo "blacklist nvidia" | sudo tee /etc/modprobe.d/nvidia-blacklist.conf
sudo update-initramfs -u

Always test with the open-source nouveau driver (for NVIDIA) or without the proprietary module to see if the panic disappears.

Advanced Analysis with Crash Dumps (kdump)

When a live system freezes, you lose the ability to inspect kernel memory. kdump solves this by booting a second "capture" kernel at the moment of panic, which saves a complete memory dump (vmcore) to disk for offline analysis.

Setting Up kdump

# Install kdump tools (Debian/Ubuntu example)
sudo apt install kdump-tools linux-crashdump

# Reserve memory for the crash kernel by adding to /etc/default/grub:
GRUB_CMDLINE_LINUX_DEFAULT="... crashkernel=256M"
sudo update-grub
sudo reboot

# Verify the crash kernel is loaded
kdump-config show
# Should show "ready to kdump"

The crashkernel= parameter reserves a contiguous block of memory for the second kernel. 256M is typical for x86_64; adjust based on your RAM size and distribution recommendations.

Capturing a Dump and Analyzing with crash

After a panic, the system will reboot into the capture kernel, save /var/crash/vmcore, and then reboot normally. You can now analyze:

# Install crash utility
sudo apt install crash

# Point it to the vmlinux debug image and the vmcore
crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/202501010101/vmcore

# Inside crash, use common commands:
crash> bt          # backtrace of the panicking task
crash> log         # kernel log buffer from the crashed kernel
crash> dis -r RIP  # disassemble around the instruction pointer
crash> struct foo  # examine kernel data structures
crash> mod -S      # list loaded modules with addresses

The bt (backtrace) command gives you the exact call path at the moment of panic, complete with arguments and stack frames. log retrieves the entire kernel ring buffer, including messages just before the panic that might not have been flushed to disk. This is the most powerful method for root-cause analysis.

Best Practices for Preventing Kernel Panics

While you cannot eliminate every possible panic, following these practices dramatically reduces their frequency and ensures you can recover quickly when they occur:

Keep the kernel and modules updated: Use your distribution's stable kernel updates; they include fixes for known panics.
Use well-maintained, in-tree drivers: Avoid out-of-tree modules unless absolutely necessary, and prefer hardware with mainline kernel support.
Enable hardware monitoring: Use lm-sensors, mcelog/rasdaemon, and regular SMART disk checks to detect failing hardware before it causes a panic.
Configure panic= wisely: On servers, set panic=30 to auto-reboot after 30 seconds, minimizing downtime. On development/debugging systems, use panic=0 to freeze for manual capture.
Always set up persistent logging: Enable persistent journald storage and consider a netconsole or serial console for headless systems.
Deploy kdump in production: The vmcore is invaluable for post-mortem analysis; reserve a small amount of memory for the crash kernel.
Test kernel upgrades on a staging system: Before rolling out a new kernel to production, boot it on identical hardware and stress-test with workloads that exercise your drivers.
Monitor for Oopses, not just panics: An oops often precedes a panic; use kerneloops or automated log scanning to catch them early.

Conclusion

A kernel panic is the kernel's ultimate cry for help—it signals that something fundamental has gone wrong. By methodically capturing panic logs, decoding the stack traces, isolating faulty modules or hardware, and leveraging advanced tools like kdump and crash, you can turn a catastrophic crash into a debuggable event. The key is preparation: set up persistent logging, reserve memory for kdump, and understand your hardware and driver stack. With these practices, you transform kernel panics from mysterious system freezes into actionable diagnostics, ensuring your Linux systems remain robust, recoverable, and resilient.