← Back to DevBytes
Linux July 09, 2026 11 min read

Fix 'Read-only file system' Error in Linux in Production: Root Cause Analysis

Understanding the 'Read-only File System' Error

In a production Linux environment, encountering a "Read-only file system" error can bring critical services to an abrupt halt. The error typically surfaces when an application or an administrator tries to write to a disk, and the kernel rejects the operation with a message like: 

touch: cannot touch 'test.txt': Read-only file system
mkdir: cannot create directory 'backup': Read-only file system

At this point, the entire mounted file system has been switched to read-only mode by the kernel. No new data can be written, logs stop accumulating, databases freeze, and user sessions may break. This is not a simple permission issue — it is a kernel-level protective measure triggered by underlying storage problems.

What Actually Happens Under the Hood

The Linux kernel maintains an internal mount state for every file system. When the kernel’s file system driver (ext4, XFS, etc.) or the block layer detects a critical error, it sets the mount point to MS_RDONLY to prevent further data corruption. The error can originate from:

The kernel’s goal is to stop writes immediately so that the file system metadata doesn't become inconsistent. A forced read-only remount is often the only thing preventing a complete filesystem meltdown. That makes this error both alarming and informative — it's the symptom, not the disease.

Why This Matters in Production

🚀 Deploy your AI agent in 10 minutes

Managed Hermes hosting. Zero DevOps. 100M tokens/mo included.

Try it free →

In production, a read-only file system means:

The error often appears suddenly after months of flawless operation, usually linked to hardware wear or an overlooked filesystem consistency check that was never completed after an unclean shutdown. Understanding the root cause is essential not only to fix the immediate issue but to prevent recurrence.

Step-by-Step Root Cause Analysis

The following systematic investigation will help you identify why the file system switched to read-only. All commands assume you have root access or sudo privileges.

1. Identify the Affected Mount Point

First, confirm which file system is in read-only mode. Run mount and look for the ro flag in the mount options: 

mount | grep '(ro,'
# Example output:
# /dev/sda1 on / type ext4 (ro,relatime,errors=remount-ro)
# /dev/sdb1 on /data type xfs (ro,noatime,nodiratime)

The option errors=remount-ro is often present on the root file system; it tells the kernel to remount read-only on error. For data partitions, the ro flag appears without an explicit error option, indicating a forced remount.

2. Inspect Kernel Ring Buffer (dmesg)

The kernel log contains the raw story. Use dmesg to retrieve recent messages, filtering for file system or block device errors: 

dmesg -T | grep -iE 'error|fail|remount|read.only|filesystem|buffer|sda|sdb'

Typical patterns you might see:

3. Check Systemd Journal (If Still Available)

On systems with persistent journal and a writable root (which might be gone after the error), you can query the journal from a previous boot: 

journalctl -b -1 --no-pager | grep -iE 'remount|filesystem|I/O error|sda|sdb'

If the root file system itself is read-only and the journal isn't stored on a separate writable partition, journalctl may fail. In that case, rely on dmesg output captured from the console or a netconsole setup.

4. Check SMART Health of the Disk

Disk health is the most common root cause. Use smartctl to read SMART attributes: 

smartctl -a /dev/sda
# Look for:
# Reallocated_Sector_Ct, Current_Pending_Sector, Offline_Uncorrectable
# SMART overall-health status: PASSED / FAILED

Any non-zero value in Current_Pending_Sector or Offline_Uncorrectable indicates physical bad blocks. A failing disk can intermittently return I/O errors, triggering the kernel’s read-only remount. Even if the SMART health is "PASSED", check the attribute values for recent changes.

5. Check RAID Controller or SAN Logs

If the server uses hardware RAID or a SAN, the kernel may see I/O errors that originate from the storage layer. Check:

6. Look for Recent System Events

Correlate the error with a timeline. Did an unclean shutdown occur? A kernel panic, power failure, or hard reset can leave filesystems in an inconsistent state. Check last reboot or uptime anomalies. An abrupt power loss often leads to orphaned journal entries that require fsck, and if the root file system is mounted with errors=remount-ro, the next boot will hit the error.

Immediate Recovery Actions

Once you have identified the affected file system and the root cause, you need to restore writability while minimizing data loss.

1. Attempt a Manual Remount (Only if Safe)

If the kernel remounted read-only due to a transient, non-disk error (e.g., a SAN path that has been restored, or an NFS server that has recovered), you can remount the file system read-write: 

mount -o remount,rw /mountpoint
# or using the device:
mount -o remount,rw /dev/sda1 /

Warning: Do not attempt this if you suspect actual disk corruption or bad blocks. A forced remount may allow writes onto a damaged filesystem, worsening the corruption.

For the root file system, if the remount fails with "mount: cannot remount read-write, is write-protected", you may need to reset the block device’s write-protection state, but that’s rare.

2. Reboot and Trigger a Forced fsck

For root or critical data partitions, the safest immediate action is a controlled reboot that forces a file system check: 

# Reboot and add fsck flags to kernel command line (for ext4):
# reboot, edit grub, add:  fsck.mode=force fsck.repair=yes
# Or for a data partition, unmount it (if possible) and run:
umount /data
fsck -f /dev/sdb1

For XFS, use xfs_repair instead of fsck. Note that running fsck on a mounted root partition is impossible; a reboot with kernel parameters or booting from a rescue image is required.

3. Boot from Rescue Media

In many production scenarios, the root partition is read-only and you cannot execute fsck online. Boot the server from a Linux Live ISO or a rescue kernel. Then: 

# Identify the root partition
lsblk
# Run fsck on it
fsck -y /dev/sda1
# For XFS root:
xfs_repair /dev/sda1

After repair, reboot normally. The filesystem will mount read-write if the repair succeeded and no new I/O errors occur.

4. Handle NFS-Induced Read-only State

If an NFS client mount turned read-only because the server was unreachable, first ensure the NFS server is stable. Then remount: 

umount -f /mnt/nfs_share
mount -t nfs -o rw,nolock server:/path /mnt/nfs_share

Consider using the hard or soft mount options with appropriate timeouts, but a read-only remount on NFS usually indicates a server-side problem that must be fixed first.

Permanent Fix and Prevention

Addressing the immediate outage is only half the battle. The real value of root cause analysis lies in permanent remediation.

1. Replace Failing Hardware

If SMART attributes or I/O errors confirm a failing disk, schedule an immediate disk replacement. Use RAID rebuild or data migration to avoid downtime. For non-redundant disks, restore from backups after replacing the drive.

2. Adjust Mount Options

The root file system often has errors=remount-ro in /etc/fstab. This is a safety net. Do not simply remove it. Instead, consider: 

# /etc/fstab example for root:
UUID=abc123 / ext4 defaults,errors=remount-ro 0 1

For data partitions without this option, you can add it to catch errors gracefully: 

UUID=def456 /data ext4 defaults,errors=remount-ro 0 2

This ensures the partition becomes read-only on error instead of risking silent corruption. Combine with monitoring so you get alerted immediately.

3. Schedule Regular Filesystem Checks

Even with journaling filesystems, metadata inconsistencies accumulate. For ext4, the tune2fs tool can set a mount-count or interval-based forced fsck: 

tune2fs -c 30 -i 30d /dev/sda1   # Check every 30 mounts or 30 days

For XFS, regular xfs_scrub (online check) and periodic xfs_repair during maintenance windows help detect issues before they become critical.

4. Implement Proper UPS and Power Management

Many read-only remounts follow an unclean shutdown. Ensure servers have redundant power supplies and are connected to a UPS with automatic shutdown scripts. Configure NUT (Network UPS Tools) or vendor-specific agents to gracefully power down when battery is low.

5. Use Multi-Path and Redundancy

For SAN or complex storage, configure multipath with queue_if_no_path to keep I/O queued during path failures, avoiding immediate filesystem errors. Ensure the multipath.conf has appropriate no_path_retry and dev_loss_tmo settings.

6. Monitor and Alert Proactively

Set up monitoring for:

A typical Prometheus alert rule: 

- alert: ReadOnlyFilesystem
  expr: node_filesystem_device{mountpoint="/",fstype!="tmpfs"} 
        and on() node_probe_error{job="node"} > 0
  # Or a custom script checking 'mount' output

Real-World Example: Root Cause Analysis Walkthrough

Let’s simulate a common production scenario. A web server suddenly returns 500 errors. SSH into the server and attempt to create a temporary file: 

$ touch /tmp/test
touch: cannot touch '/tmp/test': Read-only file system

Check mount points: 

$ mount | grep '/ '
/dev/sda1 on / type ext4 (ro,relatime,errors=remount-ro)

The root file system is read-only. Next, check dmesg: 

$ dmesg -T | tail -50
[Tue Jan 14 10:23:45 2025] EXT4-fs error (device sda1): __ext4_get_inode_loc: inode #12345: 
block 67890: comm postgres: unable to read block - I/O error
[Tue Jan 14 10:23:45 2025] EXT4-fs (sda1): Remounting filesystem read-only

The I/O error points to a storage issue. Check SMART: 

$ smartctl -a /dev/sda
...
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always   -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always   -       8
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline  -       3

Here, Current_Pending_Sector = 8 and Offline_Uncorrectable = 3. The disk has physical damage. The root cause: a failing disk that produced an unreadable block, causing the kernel to remount the root as read-only.

Immediate action: schedule downtime, boot from rescue media, run fsck -y /dev/sda1 (which will relocate bad blocks using the bad block inode), replace the disk, and restore from backup. If the server is in a RAID1 array, replace the disk and allow rebuild. The root cause analysis clearly pointed to disk failure, so simply remounting read-write would be dangerous.

Best Practices for Production Resilience

Conclusion

The "Read-only file system" error in Linux production environments is a symptom of deeper storage or filesystem integrity issues. It is a protective mechanism, not a bug. A systematic root cause analysis — checking kernel logs, SMART health, RAID status, and recent system events — reveals whether the trigger is a transient I/O glitch, a failing disk, or a journal corruption from an unclean shutdown. The immediate recovery path depends on the root cause: remounting read-write is only safe for transient errors; for disk failures or metadata corruption, a controlled fsck from rescue media and hardware replacement are mandatory. By implementing robust monitoring, regular filesystem checks, proper mount options, and redundant storage, you can prevent the error from causing extended production outages. Remember, in production, the goal is not just to fix the error, but to understand why the kernel took protective action, ensuring long-term data integrity and service reliability.

🚀 Need a reliable AI agent for your project?

Deploy Hermes Agent in 10 minutes. Managed hosting, zero DevOps.

Get Started — $23.99/mo
← Back to all articles