WenJunjie Blog

sysadmin

Read-only File System

| Comments

原因分析

可能造成文件系统只读的原因

  • 硬盘错误、磁盘坏道 smartctl
  • 系统文件损坏
  • 高磁盘I/O读写导致繁忙I/O重试错误可能使得低水平的磁盘调用失败,导致文件系统保护强制进入ext3只读模式
  • 在SAN上高磁盘I/O或不正确路径的配置
  • RAID进入degraded mode

解决方法

rsync备份
touch file
cat /etc/fstab
df -h
mount
fsck -y
fsck -y -A -f
fsck -y /dev/mapper/VolGroup00-LogVol00
e2fsck -C -p -f -y /dev/sda3
touch file
reboot
umount /app
fuser -mk /app
fsck -t ext3 -y /app
mount /app
mount -o rw,remount /app #读写挂载
mount -o ro,remount /data #只读挂载

注意:只检查系统相关目录,/app数据目录不检查,大磁盘空间不检查检查时间可能较久,半小时以上

若修复不成功,造成无法正常启动,grub无法进入系统 光盘引导 linux rescue

chroot /mnt/sysimage
/sbin/grub-install /dev/sda
cat /boot/grub/grub.conf

default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-274.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-274.el5 ro root=/dev/VolGroup00/LogVol00
        initrd /initrd-2.6.18-274.el5.img

cat /boot/grub/menu.lst

default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-274.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-274.el5 ro root=/dev/VolGroup00/LogVol00
        initrd /initrd-2.6.18-274.el5.img

exit
reboot

umount /app
umount: /app: device is busy.
lsof /app
fuser /app
fuser -km /app
fsck -fy /app

网络pxe引导进入

append ksdevice=eth0 load_ramdisk=1 initrd=initrd-5.7.img rescue

grub-mkrescue –image-type=cdrom /tmp/grub-rescue.iso wodim /tmp/grub-rescue.iso

grub> root (hd0, 0) kernel /vmlinuxz initrd /initrd boot

root (hd0,1) chainloader +1 boot

set root=(hd0,2) chainloader +1 boot

root (hd0,1) kernel /boot/vmlinuz-2.6.18-6-686 root=/dev/sda11 ro initrd /boot/initrd.img-2.6.18-6-686 boot

set root=(hd0,2) linux /boot/vmlinuz-2.6.18-6-686 root=/dev/sda11
initrd /boot/initrd.img-2.6.18-6-686 boot

grub-install /dev/sda16 root (hd0,15) chainloader +1 boot

find /boot/grub/stage1 (optional) root (hdX,Y) setup (hd0) quit

事后预防措施

  • 脚本监控

shell

#!/bin/bash

egrep " ro,|,ro " /proc/mounts > /dev/null 2>&1
status=$?
if [ $status -eq 0 ]
then 
    echo "CRITICAL: `egrep " ro,|,ro " /proc/mounts`"
    exit 2
else
    echo "OK: filesystem is ok"
    exit 0
fi

Comments