Understanding Kernel Panic in Linux
A kernel panic is the Linux kernel's last-resort safety mechanism when it encounters an unrecoverable fatal error. Think of it as the kernel equivalent of a blue screen of death. When the kernel detects an inconsistency in its internal state, memory corruption, or a hardware fault that makes continued operation dangerous, it halts everything, outputs diagnostic information, and refuses to continue until a reboot occurs.
In production environments, a kernel panic means one thing: unexpected downtime. Unlike a userspace application crash that can be caught and restarted by a process supervisor, a kernel panic takes the entire machine down instantly. All running services, database transactions, in-flight network connections, and unsaved state vanish. For infrastructure running hundreds or thousands of containers, virtual machines, or critical databases, the blast radius is enormous.
What Triggers a Kernel Panic?
Kernel panics stem from several distinct categories of failures:
- Hardware faults: Failing RAM modules (bit flips, ECC errors), overheating CPUs triggering thermal shutdown, power supply voltage drops, or dying storage controllers
- Kernel bugs: NULL pointer dereferences in kernel code, use-after-free in kernel memory, race conditions in filesystem drivers
- Out-of-memory (OOM) in kernel space: When the kernel itself runs out of memory for critical allocations like stack pages or DMA buffers
- Filesystem corruption: Inconsistent on-disk structures that cause the kernel to hit assertion failures in filesystem code
- Faulty kernel modules: Third-party drivers or out-of-tree modules that corrupt kernel memory or violate locking rules
- Hardware-assisted memory errors: MCE (Machine Check Exception) errors from CPUs reporting uncorrectable ECC errors
The Anatomy of a Panic Message
When a panic occurs, the kernel dumps critical information to the console, serial port, or configured panic log destination. Understanding every field in that dump is the first step in root cause analysis. Here is a representative panic message broken down line by line:
[ 187.422451] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
[ 187.423109] IP: [<ffffffffc0452a1a>] ext4_readdir+0x3da/0x650 [ext4]
[ 187.423809] PGD 3c0c7067 PUD 0
[ 187.424301] Thread overran stack, or stack corrupted
[ 187.424998] Oops: 0002 [#1] SMP NOPTI
[ 187.425601] CPU: 3 PID: 28421 Comm: ls Not tainted 5.4.0-109-generic #123-Ubuntu
[ 187.426309] Hardware name: Dell Inc. PowerEdge R640/0H28JN, BIOS 2.11.4 02/15/2021
[ 187.427001] RIP: 0010:ext4_readdir+0x3da/0x650 [ext4]
[ 187.427698] Code: 48 8b 47 28 48 8b 50 28 48 85 d2 0f 84 09 02 00 00 48 8b 47 28 ...
[ 187.428401] RSP: 0018:ffffa1d7c0a03b78 EFLAGS: 00010282
[ 187.429098] RAX: 0000000000000000 RBX: ffff8e5c3b4a1c00 RCX: 0000000000000000
[ 187.429801] RDX: 0000000000000000 RSI: ffff8e5c3b4a1c00 RDI: ffff8e5c3b4a1c00
[ 187.430501] RBP: ffffa1d7c0a03c08 R08: 0000000000000000 R09: 0000000000000001
[ 187.431204] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8e5c4e5a7a00
[ 187.431901] R13: ffffa1d7c0a03c70 R14: 0000000000000000 R15: ffff8e5c3b4a1c00
[ 187.432598] FS: 00007f9a4c3e8d48(0000) GS:ffff8e5c7fb80000(0000) knlGS:0000000000000000
[ 187.433301] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 187.434001] CR2: 0000000000000028 CR3: 000000023d5a0004 CR4: 00000000007706e0
[ 187.434698] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 187.435401] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 187.436098] PKRU: 55555554
[ 187.436798] Call Trace:
[ 187.437501] iterate_dir+0x8e/0x1a0
[ 187.438198] ksys_getdents64+0x78/0x130
[ 187.438901] __x64_sys_getdents64+0x1e/0x30
[ 187.439598] do_syscall_64+0x5b/0x1e0
[ 187.440301] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Key fields to extract immediately:
- BUG/Oops line: "NULL pointer dereference at 0000000000000028" — the offset 0x28 tells you a struct member was accessed on a NULL pointer, giving a strong hint about which data structure is involved
- IP (Instruction Pointer): "ext4_readdir+0x3da/0x650 [ext4]" — pinpoints the exact function and offset in the ext4 module. This is your starting point for source code analysis
- CPU and PID: "CPU: 3 PID: 28421 Comm: ls" — identifies which process context triggered the panic, critical for reproducing the issue
- Kernel version: "5.4.0-109-generic #123-Ubuntu" — lets you check whether a known bug was fixed in a later kernel version
- Hardware name: "Dell Inc. PowerEdge R640" — essential for checking vendor-specific firmware bugs or hardware errata
- Register dump: RAX=0 confirms the NULL pointer; CR2=0x28 confirms the faulting address
- Call trace: The complete stack backtrace shows the syscall path from userspace (getdents64 → iterate_dir → ext4_readdir)
Why Kernel Panic Root Cause Analysis Matters in Production
🚀 Deploy your AI agent in 10 minutes
Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.
Try it free →In production, a kernel panic is not just an incident — it's a systemic failure that demands a thorough RCA. Without it, you risk repeat outages, data corruption, or cascading failures across your infrastructure. Here is why investing time in proper RCA is non-negotiable:
- Prevent recurrence: A panic caused by a specific workload pattern will happen again under the same conditions. RCA identifies the trigger so you can implement a permanent fix or workaround
- Data integrity: Panics during filesystem operations can leave filesystems in inconsistent states. Understanding whether the panic corrupted on-disk structures determines your recovery strategy
- Vendor escalation evidence: When engaging hardware vendors, kernel upstream maintainers, or distribution support, a thorough RCA package dramatically reduces resolution time
- Capacity planning: Memory-exhaustion panics reveal hidden memory pressure that monitoring missed, informing kernel parameter tuning
- Compliance and SLA reporting: Many regulated industries require documented RCA for outages exceeding certain duration thresholds
Prerequisites for Effective RCA
Before you can perform root cause analysis on a kernel panic, your production systems must be configured to capture the necessary forensic data. Without these configurations in place, you will be left with only the console log — often insufficient for deep analysis.
Enabling kdump for Post-Mortem VMcores
The single most important tool for kernel panic RCA is kdump, which boots a secondary crash kernel at panic time to dump the entire kernel memory (a vmcore) to disk. This vmcore can then be analyzed offline using the crash utility. Configure kdump on RHEL/CentOS/Fedora derivatives:
# Install kdump and crash analysis tools
dnf install kexec-tools crash
# Reserve memory for the crash kernel (typically 128-256 MB)
# Add to /etc/default/grub:
GRUB_CMDLINE_LINUX="crashkernel=auto"
# Regenerate grub config
grub2-mkconfig -o /boot/grub2/grub.cfg
# Enable and start kdump
systemctl enable kdump.service
systemctl start kdump.service
# Verify kdump status
kdumpctl status
On Debian/Ubuntu systems:
# Install kdump tools
apt install kdump-tools crash
# Configure crash kernel memory reservation
# Edit /etc/default/kdump-tools and set:
crashkernel="128M"
# Enable kdump
systemctl enable kdump-tools.service
systemctl start kdump-tools.service
# Check configuration
kdump-config show
Configuring netconsole for Remote Logging
When a machine panics, the console output may be the only data you get if kdump fails to trigger. Netconsole sends kernel log messages (including panic output) over UDP to a remote syslog server before the machine dies:
# On the target machine, configure netconsole to send to a remote collector
# Format: netconsole=src_port@src_ip/dev_name,dst_port@dst_ip/dst_mac
# Example: send kernel logs to 10.0.1.50:6666 via eth0
modprobe netconsole netconsole=6666@10.0.0.12/eth0,6666@10.0.1.50/00:1a:2b:3c:4d:5e
# Or make persistent via /etc/modprobe.d/netconsole.conf
echo "options netconsole netconsole=6666@10.0.0.12/eth0,6666@10.0.1.50/00:1a:2b:3c:4d:5e" > /etc/modprobe.d/netconsole.conf
On the receiving side, use a dedicated UDP listener like socat or netcat paired with a log file:
# On the collector machine
socat -u UDP-LISTEN:6666,reuseaddr STDOUT | tee -a /var/log/remote-panics.log
Setting Up Persistent RAM (pstore) for Panic Logs
Modern UEFI firmware exposes a non-volatile storage area called pstore that survives reboots. The kernel can write panic logs there, and you can retrieve them after reboot from /sys/fs/pstore:
# Mount pstore filesystem (usually mounted automatically)
mount -t pstore pstore /sys/fs/pstore
# After a panic and reboot, retrieve the preserved log
ls -la /sys/fs/pstore/
cat /sys/fs/pstore/dmesg-erst-0 # ERST-backed panic log
cat /sys/fs/pstore/dmesg-efi-0 # EFI variable-backed panic log
Step-by-Step Root Cause Analysis Process
A disciplined RCA process turns a cryptic panic message into an actionable fix. Follow this sequence every time, even under time pressure — skipping steps leads to wrong conclusions.
Step 1: Triage — Categorize the Panic Type Immediately
From the panic message header, classify the failure into one of these buckets. Each bucket has a distinct investigation path:
# Common panic headers and their meaning:
"BUG: unable to handle kernel NULL pointer dereference" → Kernel code bug (dereferenced NULL)
"BUG: unable to handle kernel paging request" → Memory corruption or use-after-free
"Kernel panic - not syncing: Out of memory" → Kernel OOM
"Kernel panic - not syncing: Fatal exception" → Hardware interrupt storm or double fault
"Kernel panic - not syncing: Attempted to kill init!" → Critical userspace process (init) died
"Kernel panic - not syncing: No working init found" → Root filesystem or init binary missing
"BUG: soft lockup - CPU# stuck" → Infinite loop in kernel code (usually drivers)
"BUG: hard lockup - CPU# stuck" → Hardware interrupt handler stuck
"Kernel panic - not syncing: Hardware Error" → MCE / uncorrectable ECC error
Step 2: Extract and Preserve All Forensic Artifacts
Collect everything before rebooting — the vmcore, console log, serial console capture, and netconsole remote log. Never reboot without saving these first:
# If the machine is still partially responsive via SSH/serial console:
# 1. Copy the vmcore from /var/crash/
ls -lh /var/crash/
scp /var/crash/127.0.0.1-2025-06-15-14:30:42/vmcore analysis-host:/tmp/
# 2. Capture dmesg if still accessible
dmesg > /tmp/pre-reboot-dmesg.log
dmesg -T > /tmp/pre-reboot-dmesg-human.log
# 3. For machines with BMC/IPMI, retrieve the serial console log
ipmitool sel list # System Event Log
ipmitool sel get # Detailed event records
# 4. Save pstore contents before they are overwritten
tar -czf /tmp/pstore-backup.tar.gz /sys/fs/pstore/
Step 3: Analyze the vmcore with the crash Utility
The crash utility is the primary interactive debugger for vmcore files. It provides a gdb-like interface with kernel-specific commands:
# Start crash analysis on the vmcore
# Syntax: crash /path/to/vmlinux /path/to/vmcore
crash /usr/lib/debug/boot/vmlinux-5.4.0-109-generic /var/crash/127.0.0.1-2025-06-15-14:30:42/vmcore
# Inside crash, examine the system state at panic time:
crash> sys
KERNEL: /usr/lib/debug/boot/vmlinux-5.4.0-109-generic
DUMPFILE: /var/crash/127.0.0.1-2025-06-15-14:30:42/vmcore
CPUS: 16
DATE: Mon Jun 15 14:30:42 2025
UPTIME: 187 days, 04:22:17
MACHINE: Dell Inc. PowerEdge R640
PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000028"
# Examine the backtrace of the panicking task
crash> bt
PID: 28421 TASK: ffff8e5c3b4a1c00 CPU: 3 COMMAND: "ls"
#0 [ffffa1d7c0a03b78] machine_kexec at ffffffff9a07a8b3
#1 [ffffa1d7c0a03bd0] __crash_kexec at ffffffff9a26b892
#2 [ffffa1d7c0a03c98] crash_kexec at ffffffff9a26c7d1
#3 [ffffa1d7c0a03cb0] oops_end at ffffffff9a0398c2
#4 [ffffa1d7c0a03cd0] no_context at ffffffff9a0765fe
#5 [ffffa1d7c0a03d28] __bad_area_nosemaphore at ffffffff9a07688a
#6 [ffffa1d7c0a03d78] bad_area_nosemaphore at ffffffff9a076a28
#7 [ffffa1d7c0a03d88] do_page_fault at ffffffff9a07ac40
#8 [ffffa1d7c0a03db0] page_fault at ffffffff9a076eae
RIP: ffffffffc0452a1a [ext4_readdir+0x3da]
#9 [ffffa1d7c0a03e80] ext4_readdir at ffffffffc0452a1a [ext4]
#10 [ffffa1d7c0a03f18] iterate_dir at ffffffff9a3a1c3e
#11 [ffffa1d7c0a03f40] ksys_getdents64 at ffffffff9a3a2228
#12 [ffffa1d7c0a03f78] __x64_sys_getdents64 at ffffffff9a3a224e
#13 [ffffa1d7c0a03f80] do_syscall_64 at ffffffff9a00545b
#14 [ffffa1d7c0a03fa0] entry_SYSCALL_64_after_hwframe at ffffffff9ac0007c
# Disassemble the faulting instruction area
crash> dis ext4_readdir+0x3da
0xffffffffc0452a1a : mov 0x28(%rax),%rdx
# Examine the register state at fault time
crash> pt -R RIP,RSP,RAX,RBX,RCX,RDX
RIP: ffffffffc0452a1a RSP: ffffa1d7c0a03b78
RAX: 0000000000000000 RBX: ffff8e5c3b4a1c00
RCX: 0000000000000000 RDX: 0000000000000000
# Look at the process's open files (useful for filesystem panics)
crash> files 28421
PID: 28421 TASK: ffff8e5c3b4a1c00 CPU: 3 COMMAND: "ls"
ROOT: / CWD: /home/user/documents
FD FILE FLAGS COUNT
0 ffff8e5c4e5a7a00 r--d--- 1
1 ffff8e5c4e5a7b00 -w---- 1
2 ffff8e5c4e5a7c00 -w---- 1
# Dump kernel log buffer from vmcore
crash> log
[ 187.422451] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
...full panic log...
# Examine memory statistics at panic time
crash> kmem -i
PAGES TOTAL PERCENTAGE
TOTAL MEM 16447676 62.7 GB ----
FREE MEM 1234567 4.7 GB 7.5%
USED MEM 15213109 58.0 GB 92.5%
BUFFERS 456789 1.7 GB 2.8%
CACHED 8901234 33.9 GB 54.1%
# Exit crash
crash> quit
Step 4: Source Code Correlation
With the faulting function and offset identified, correlate with kernel source code. For the ext4_readdir example, look up the exact source line:
# Clone the exact kernel version that panicked
git clone --branch v5.4.109 --depth 1 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
cd linux
# Find the ext4_readdir function
grep -rn "ext4_readdir" fs/ext4/
# Open fs/ext4/dir.c and examine the function
# The offset 0x3da from the function start corresponds to a specific instruction
# In the source, look for pointer dereferences around that area
# For a NULL deref at offset 0x28 into a struct, the code likely does:
# struct something *ptr = NULL;
# ptr->field; // field at offset 0x28
# Use objdump to map instruction offset to source line precisely:
objdump -d -S --start-address=0x$(nm vmlinux | grep ext4_readdir | awk '{print $1}') vmlinux | less
Step 5: Hardware Diagnostics (When Appropriate)
If the panic signature suggests hardware involvement — MCE errors, random memory addresses, unreproducible panics on different CPUs — run hardware diagnostics before concluding it's a software bug:
# Check for MCE (Machine Check Exception) events
mcelog --client --dmi
mcelog --client --cpu
# Or on newer systems using the kernel's mce log:
ras-mc-ctl --summary
ras-mc-ctl --errors
# Run a memory test (requires downtime)
# Boot into memtest86+ and let it run at least 2 complete passes
# Check ECC error counts from EDAC
edac-util -v
edac-util -r
# Check disk health for filesystem-related panics
smartctl -a /dev/sda
smartctl -l error /dev/sda
# Check PCI device errors
lspci -vvv | grep -i error
dmesg | grep -i "PCIe.*error"
Step 6: Determine if the Bug is Known
Before investing in custom debugging, check if your panic is a known issue with an existing fix:
# Search the kernel mailing list archives
# Use the faulting function and panic type as search terms
# Example search on lore.kernel.org:
# "ext4_readdir NULL pointer dereference" site:lore.kernel.org
# Check your distribution's kernel changelog
# On Debian/Ubuntu:
apt changelog linux-image-5.4.0-109-generic | grep -i "ext4\|readdir\|NULL"
# On RHEL/CentOS:
rpm -q --changelog kernel-5.4.0-109 | grep -i "ext4\|readdir\|NULL"
# Check the upstream kernel stable tree for commits touching the function
git log --oneline v5.4.109..v5.4.200 -- fs/ext4/dir.c
Common Panic Patterns and Their Fixes
Certain panic patterns recur across production environments. Here are the most frequent ones, their diagnostic signatures, and proven fixes.
Pattern 1: NULL Pointer Dereference in Filesystem Code
Signature: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000XXX" with IP in ext4, xfs, or btrfs code. CR2 is a small offset like 0x28, 0x30, 0x48.
Root cause: Usually a race condition between directory iteration and concurrent unlink/rename operations, or a corrupted in-memory inode that wasn't properly validated.
Fix:
# 1. Update to the latest kernel in your distribution's stable branch
apt upgrade linux-image-generic
# 2. If the panic is reproducible, apply a kernel boot parameter workaround
# to disable specific filesystem features that may trigger the bug:
# For ext4: disable dir_index and flex_bg
# Add to /etc/default/grub:
GRUB_CMDLINE_LINUX="rootflags=data=ordered,noatime,nodiratime"
# 3. For critical systems, mount the filesystem with conservative options
mount -o remount,data=ordered,noatime,nobarrier /mountpoint
# 4. Schedule an offline fsck to detect and repair corruption
fsck.ext4 -fn /dev/sda2 # Non-destructive check first
fsck.ext4 -fy /dev/sda2 # Repair if corruption found
Pattern 2: Kernel Out-of-Memory Panic
Signature: "Kernel panic - not syncing: Out of memory" or panic triggered from __alloc_pages_slowpath with order value showing large contiguous allocation failure.
Root cause: The kernel failed to allocate memory for critical operations (network buffer rings, filesystem metadata, page tables). Unlike userspace OOM killer, kernel allocations have no fallback and must succeed.
Fix:
# 1. Increase vm.min_free_kbytes to reserve memory for kernel allocations
sysctl -w vm.min_free_kbytes=262144 # 256 MB reserved
echo "vm.min_free_kbytes=262144" >> /etc/sysctl.d/99-kernel-reserve.conf
# 2. Reduce kernel memory pressure from network stack
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
# 3. Disable transparent huge pages if fragmentation is causing large-order allocation failures
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo "kernel.transparent_hugepage=never" >> /etc/sysctl.d/99-thp.conf
# 4. Increase the kernel's overcommit ratio if appropriate
sysctl -w vm.overcommit_ratio=80
# 5. Audit kernel memory consumers using crash's kmem command
crash> kmem -i
crash> kmem -s # Show slab cache usage — identify bloated caches
Pattern 3: Hardware Error (MCE) Panic
Signature: "Kernel panic - not syncing: Hardware Error" with preceding MCE log entries showing bank numbers, address, and error codes like "Uncorrected ECC error".
Root cause: Uncorrectable ECC memory error, CPU cache hierarchy error, or PCIe fatal error. The hardware has detected a fault it cannot correct.
Fix:
# 1. Extract the exact DIMM location from the MCE log
# Look for "Memory Error" and "Physical Address" fields
mcelog --client --dmi
# Example output:
# Memory Error: node 1, channel 2, DIMM 3
# Physical Address: 0x5a3f8000
# 2. Cross-reference with your server's BMC/IPMI to map to physical slot
ipmitool sensor list | grep -i dimm
ipmitool fru print
# 3. For reproducible MCE on specific DIMM, schedule DIMM replacement
# Most server vendors support online DIMM replacement if the fault is isolated
# 4. As a temporary mitigation, offline the affected memory page
# (if the kernel supports memory failure recovery)
echo "offline" > /sys/devices/system/memory/memoryXXX/soft_offline_page
# Or use the hardware memory error subsystem:
echo 1 > /sys/kernel/debug/mce/fake_panic # Test panic handler
# 5. Update server firmware — many MCE bugs are firmware-related
# Dell: check support.dell.com for BIOS and firmware updates
# HP: check support.hpe.com for System ROM and CPLD updates
Pattern 4: Soft Lockup — CPU Stuck
Signature: "BUG: soft lockup - CPU#X stuck for 22s!" followed by the task that's spinning. The kernel's soft lockup detector fires when a task holds the CPU in kernel mode for more than the watchdog threshold (default 20 seconds).
Root cause: Infinite loop in kernel code, usually in a driver's interrupt handler, a tight loop waiting for hardware state that never changes, or a spinlock deadlock where two CPUs wait for each other forever.
Fix:
# 1. From vmcore, identify the looping function
crash> bt -a # Backtrace all CPUs
# Look for the CPU that's stuck — its backtrace will show the function
# that's been running for 22+ seconds without yielding
# 2. Common stuck functions and their causes:
# - nvme_wait_ready: NVMe drive firmware hang → update NVMe firmware
# - i40e_clean_rx_irq: Intel X710 driver bug → update i40e driver
# - __tcp_transmit_skb: Network stack stuck → check netconsole output
# - mutex_lock / spin_lock: Lock contention → check lock owners
# 3. Examine lock state across CPUs
crash> lock
# Shows current lock holders and waiters
# 4. Increase the soft lockup threshold if the workload is legitimate but slow
# (not recommended — fix the root cause instead)
echo 60 > /proc/sys/kernel/watchdog_thresh
# 5. For driver-related lockups, blacklist the offending module
echo "blacklist i40e" >> /etc/modprobe.d/blacklist.conf
# Then reboot and use an alternative driver or hardware
Post-Fix Verification and Monitoring
After applying a fix, verify it holds and set up monitoring to catch future panics early:
# 1. Stress test the fix
# For filesystem fixes: run parallel directory operations
for i in $(seq 1 100); do
(find /mnt/test -type f -exec stat {} \; &)
done
wait
# 2. Enable kernel panic hook for notification before reboot
# Create /etc/sysctl.d/99-panic.conf:
kernel.panic_on_oops = 0 # Don't panic on oops, let kdump handle it
kernel.panic = 10 # Auto-reboot after 10 seconds if kdump fails
kernel.panic_on_io_nmi = 1 # Panic on I/O NMI (hardware fault)
kernel.softlockup_panic = 0 # Don't panic on soft lockup, just log
kernel.hung_task_panic = 0 # Don't panic on hung tasks
# 3. Set up a cron job to monitor pstore for new entries
cat > /etc/cron.d/pstore-monitor << 'EOF'
*/5 * * * * root [ -s /sys/fs/pstore/dmesg-erst-0 ] && \
(cat /sys/fs/pstore/dmesg-erst-0 | mail -s "PANIC: pstore log on $(hostname)" alerts@example.com)
EOF
# 4. Configure systemd to collect crash dumps automatically
systemctl enable systemd-coredump
# Kernel crashes are handled by kdump, userspace crashes by systemd-coredump
# 5. Integrate kdump post-mortem reporting
cat > /etc/kdump-post.d/01-notify.sh << 'EOF'
#!/bin/bash
# This runs after vmcore is saved
HOSTNAME=$(hostname)
DATE=$(date +%Y-%m-%d_%H:%M:%S)
echo "Kernel panic on $HOSTNAME at $DATE. Vmcore saved." | \
mail -s "KERNEL PANIC: $HOSTNAME" alerts@example.com
EOF
chmod +x /etc/kdump-post.d/01-notify.sh
Building a Production-Ready Panic Response Playbook
Document your panic response procedure as a runbook. Here is a template you can adapt:
# Kernel Panic Response Runbook — [Your Organization]
## Phase 1: Detection (0-5 minutes)
1. Monitoring alert fires: host down / service unreachable
2. Verify via out-of-band management (BMC/IPMI) — is machine responsive?
3. If unresponsive, connect via serial console or BMC virtual console
4. Photograph/screenshot the console — capture panic message before reboot
5. If netconsole is configured, check remote log server for panic output
## Phase 2: Preservation (5-15 minutes)
1. DO NOT REBOOT if the machine is partially accessible
2. Copy /var/crash/ contents to a safe analysis host
3. Run `dmesg` and save output if possible
4. Retrieve BMC System Event Log (SEL) via ipmitool
5. Save /sys/fs/pstore/ contents
6. Tag the incident in your ticketing system with panic signature
## Phase 3: Reboot and Restore (15-30 minutes)
1. If kdump completed, reboot the machine
2. Verify filesystems with fsck before full mount
3. Start services in dependency order
4. Run health checks: disk, network, application endpoints
5. Leave the machine in a reduced state (no cron jobs, no batch processing)
## Phase 4: Root Cause Analysis (30 minutes - 48 hours)
1. Transfer vmcore and logs to dedicated analysis workstation
2. Run crash utility as described in this guide
3. Categorize panic type from header
4. Extract backtrace, register state, and memory statistics
5. Correlate with kernel source code at the exact version
6. Check kernel changelogs for known fixes
7. Run hardware diagnostics if hardware involvement suspected
## Phase 5: Remediation and Follow-up
1. Apply kernel update or configuration change
2. Stress-test the fix on a staging clone of the affected machine
3. Gradual rollout: apply to one production node, observe for 48 hours
4. Full rollout to all affected nodes
5. Update runbook with new findings
6. Schedule a post-mortem meeting within 5 business days
Best Practices for Preventing Kernel Panics
-
<