Backup of https://hallard.me/raspberry-pi-read-only/

backup.txt

Backup of https://hallard.me/raspberry-pi-read-only/


Protect your Raspberry PI SD card, use Read-Only filesystem
Posted on May 14, 2015 by Charles
Today I decided to protect all my Raspberry Pi SD cards from unproper shutdown and also prevent SD card premature end of life. This means protect them from power outage or just unplugging the power while the Pi is running.

Edit :
April 2016 to works on Jessie and Raspberry PI 3
January 2017 Fix SSH access problem after reboot / moved resolv.conf to tmpfs

Why ?
Most of the time you won’t see or have any incidences when the shutdown is not clean, but depending on what is doing the Pi at this time, you can start to have unpredictable results or data corruption. Also, sometime, applications have really verbose logging and continuously writing log to the SD will for sure bring it to death quicker than expected. SD corruption is not really simple to detect, because sometime it works and sometimes Pi is crashing/blocking and you don’t know why. You have these “hazardous” problems until you decide to change the SD. the worst, is that you were just thinking you done something wrong and searched what you done wrong ? Get time, in doubt, change your SD card first. This is my own experience and as today, I’m on 3 dead SD card.

How ?
Well, the 1st thing I’ve done was to set some folder into temp file system (log file for example) and write the log files to the SD only every hour (a example). It’s working fine but  I wanted to go deeper and have more protection, so searched over Internet people doing with read-only filesystem. I found lot of interesting article with different methods and decided to start with this one from k3a, really thanks to him.

Requirement, do it or not ?
Using a read-only file system is mostly used for Pi working 24H/7 a day and with console mode only, so if you’re using X11 or graphic interface, I won’t recommend this method because it won’t work. May be there are some other but it’s not on this article scope. So when you want you’re Pi in read-only mode it’s acting like a server but some server applications are not compatible with read only file system. Mainly database.

So if you’re running a database on your Pi, don’t use read-only filesystem. May be again there are some specific build or database to minimize write to SD, but once again it’s not in today’s scope. I see more and more people using mySQL database on Pi claiming it works fine. Sure it works, until you’ll start to face on SD corruption, and I hope they all have a good backup, because we know all, admins always have backup ;).

May be I’m the only one claiming that except if you change only few data a day, you shouldn’t use a RPi as a 24/7 database engine. For a pretty rock solid database, use a clouded one or one on a NAS like a Synology with redundant hard disk. It will go so much faster and you’ll get a real reliability. More over you will be able to use your Pi to do action and manage this database, just host your database elsewhere, just my 2 cents.

Let’s do it
Okay, I’ve done nothing yet, I’m writing this article while setting my Ambilight hyperion Rpi in read only mode and following the K3A article. You need to connect to your Pi with ssh to run the following commands, I’m connecting as root so you won’t see any sudo command (I know, I’m a bad boy!)

Get to latest current version
Shell

 apt-get update; apt-get upgrade
1
 apt-get update; apt-get upgrade
Reboot in case of bootloader or kernel version change
Shell

reboot
1
reboot
Wait for reboot and connect back with SSH.

Remove unwanted package and services files
The original article removed cron and fake-hwclock, I decided to let it working assuming my Pi will have the correct date (I never had a problem with date) and also have a date at bootup that is more the last known than 1970 year. fake-hwclock will be written at logout and every day

Shell

apt-get remove --purge wolfram-engine triggerhappy anacron logrotate dphys-swapfile xserver-common lightdm
1
apt-get remove --purge wolfram-engine triggerhappy anacron logrotate dphys-swapfile xserver-common lightdm
Shell

insserv -r x11-common; apt-get autoremove --purge
1
insserv -r x11-common; apt-get autoremove --purge
Replace log management with busybox one
Shell

apt-get install busybox-syslogd; dpkg --purge rsyslog
1
apt-get install busybox-syslogd; dpkg --purge rsyslog
This will put log into circular memory buffer, you will able to see log using
logread
command

Disable swap and filesystem check and set it to read-only.
edit the file
/boot/cmdline.txt
and add the three words
fastboot noswap ro
at the end of the line

mine were (your can be different depending on several options) :


dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait
1
dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait
and now looks like :


dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait fastboot noswap ro
1
dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait fastboot noswap ro
Move some system files to temp filesystem
Shell

rm -rf /var/lib/dhcp/ /var/run /var/spool /var/lock /etc/resolv.conf
ln -s /tmp /var/lib/dhcp
ln -s /tmp /var/run
ln -s /tmp /var/spool
ln -s /tmp /var/lock
touch /tmp/dhcpcd.resolv.conf; ln -s /tmp/dhcpcd.resolv.conf /etc/resolv.conf
1
2
3
4
5
6
rm -rf /var/lib/dhcp/ /var/run /var/spool /var/lock /etc/resolv.conf
ln -s /tmp /var/lib/dhcp
ln -s /tmp /var/run
ln -s /tmp /var/spool
ln -s /tmp /var/lock
touch /tmp/dhcpcd.resolv.conf; ln -s /tmp/dhcpcd.resolv.conf /etc/resolv.conf
On Raspberry PI 3, move some lock file to temp Files System
Thanks to @harlock for the trick

Shell

nano /etc/systemd/system/dhcpcd5
1
nano /etc/systemd/system/dhcpcd5
and be sure to change the line with

Shell

PIDFile=/run/dhcpcd.pid
1
PIDFile=/run/dhcpcd.pid
to

Shell

PIDFile=/var/run/dhcpcd.pid
1
PIDFile=/var/run/dhcpcd.pid
Full file should now looks something like that (on my Jessie Lite on RPI3).


[Unit]
Description=dhcpcd on all interfaces
Wants=network.target
Before=network.target

[Service]
Type=forking
PIDFile=/var/run/dhcpcd.pid
ExecStart=/sbin/dhcpcd -q -b
ExecStop=/sbin/dhcpcd -x

[Install]
WantedBy=multi-user.target
Alias=dhcpcd5
1
2
3
4
5
6
7
8
9
10
11
12
13
14
[Unit]
Description=dhcpcd on all interfaces
Wants=network.target
Before=network.target
 
[Service]
Type=forking
PIDFile=/var/run/dhcpcd.pid
ExecStart=/sbin/dhcpcd -q -b
ExecStop=/sbin/dhcpcd -x
 
[Install]
WantedBy=multi-user.target
Alias=dhcpcd5
On Debian jessie move random-seed  file to writable location

remove existing file

Shell

rm /var/lib/systemd/random-seed
1
rm /var/lib/systemd/random-seed
link the random-seed file to tmpfs location

Shell

ln -s /tmp/random-seed /var/lib/systemd/random-seed
1
ln -s /tmp/random-seed /var/lib/systemd/random-seed
Since file is on tmpfs it will not be created upon, reboot, but we can do it with a kind of magic of systemd system service, this is so powerfull.

To create file on the tmp area at bootup before starting random-seed service, just edit the file service file to add a pre-command to execute :


nano /lib/systemd/system/systemd-random-seed.service
1
nano /lib/systemd/system/systemd-random-seed.service
add the line   ExecStartPre=/bin/echo "" >/tmp/random-seed  under service section, should now looks like this


[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/bin/echo "" >/tmp/random-seed
ExecStart=/lib/systemd/systemd-random-seed load
ExecStop=/lib/systemd/systemd-random-seed save
1
2
3
4
5
6
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/bin/echo "" >/tmp/random-seed
ExecStart=/lib/systemd/systemd-random-seed load
ExecStop=/lib/systemd/systemd-random-seed save
Do not use touch instead of echo, it won’t work because checking RO filesystem

Execute following to tell systemd we made changes


 systemctl daemon-reload
1
 systemctl daemon-reload
Setup the Internet clock sync
I think this one is not needed because on new Raspbian version, the ntp daemon already do the job, but just in case

Shell

apt-get install ntp
1
apt-get install ntp
and be sure to configure your time zone, with raspi-config tool

Shell

raspi-config
1
raspi-config
then go to menu “Internationalisation Options”  then “Change Timezone” and select your time zone.

Edit the hourly cron script that save clock every hour
/etc/cron.hourly/fake-hwclock
and change it to allow saving clock. It sould become

Shell

#!/bin/sh
#
# Simple cron script - save the current clock periodically in case of
# a power failure or other crash

if (command -v fake-hwclock >/dev/null 2>&1) ; then
  mount -o remount,rw /
  fake-hwclock save
  mount -o remount,ro /
fi
1
2
3
4
5
6
7
8
9
10
#!/bin/sh
#
# Simple cron script - save the current clock periodically in case of
# a power failure or other crash
 
if (command -v fake-hwclock >/dev/null 2>&1) ; then
  mount -o remount,rw /
  fake-hwclock save
  mount -o remount,ro /
fi
Edit the the file /etc/ntp.conf  to set redirect driftfile  to writable zone  /var/tmp . My file was like this

Shell

# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help

driftfile /var/lib/ntp/ntp.drift
1
2
3
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help
 
driftfile /var/lib/ntp/ntp.drift
I changed to this, thanks to Gregor for the tip.

Shell

# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help

driftfile /var/tmp/ntp.drift
1
2
3
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help
 
driftfile /var/tmp/ntp.drift
 

Remove some startup scripts

insserv -r bootlogs; insserv -r console-setup
1
insserv -r bootlogs; insserv -r console-setup
Now time to tell the mounted filesystem that we’re in read-only mode

Add “,ro” flag to both block devices in
/etc/fstab
to tell the system to mount them read-only:

edit the file
/etc/fstab
mine was :


proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1
1
2
3
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1
And after modifications it is :


proc            /proc           proc    defaults             0       0
/dev/mmcblk0p1  /boot           vfat    defaults,ro          0       2
/dev/mmcblk0p2  /               ext4    defaults,noatime,ro  0       1

# For Debian Jessie 
tmpfs           /tmp            tmpfs   nosuid,nodev         0       0
tmpfs           /var/log        tmpfs   nosuid,nodev         0       0
tmpfs           /var/tmp        tmpfs   nosuid,nodev         0       0
1
2
3
4
5
6
7
8
proc            /proc           proc    defaults             0       0
/dev/mmcblk0p1  /boot           vfat    defaults,ro          0       2
/dev/mmcblk0p2  /               ext4    defaults,noatime,ro  0       1
 
# For Debian Jessie 
tmpfs           /tmp            tmpfs   nosuid,nodev         0       0
tmpfs           /var/log        tmpfs   nosuid,nodev         0       0
tmpfs           /var/tmp        tmpfs   nosuid,nodev         0       0
We done, reboot

That’s it, you should now be able to reboot your Pi


reboot
1
reboot
If all went fine, it should boot. Well, on mine it rebooted fine and ambilight was still working, then I ssh’ed onto and issued

Shell

root@ambilight:~# mount | grep /dev/root
/dev/root on / type ext4 (ro,noatime,data=ordered)
root@ambilight:~#
1
2
3
root@ambilight:~# mount | grep /dev/root
/dev/root on / type ext4 (ro,noatime,data=ordered)
root@ambilight:~#
you can see root is ro mode


root@ambilight:~# mount | grep /dev/mmc
/dev/mmcblk0p1 on /boot type vfat (ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
root@ambilight:~#
1
2
3
root@ambilight:~# mount | grep /dev/mmc
/dev/mmcblk0p1 on /boot type vfat (ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
root@ambilight:~#
and /boot (fat part of the SD) is also ro mode, great !!!

In case of problem, look at syslog (or logread) and try to find out why. You can try to fix plugging the SD card in a different computer.

Switching from Read-Only mode to Read-Write and vice-versa
Now you’re in read only mode it’s fine and safe, but if you need to install, write or modify files, upgrade, or whatever that need write access, you need to be able to do it, we’ll add this possibility and in visual mode

To set system to Read-Write use
mount -o remount,rw /
and to set it back to Read-Only
mount -o remount,ro /
but I do not remember this tricky syntax and decided to improve things a little. I want to have two simple commands like
ro
for setting mode to read only and
rw
to enable read write mode. I also wanted to know on which mode I am on command prompt.

Add fancy indicating features
Ok for all users just edit the file
/etc/bash.bashrc
and at the end add the following lines

Shell

# set variable identifying the filesystem you work in (used in the prompt below)
set_bash_prompt(){
    fs_mode=$(mount | sed -n -e "s/^\/dev\/.* on \/ .*(\(r[w|o]\).*/\1/p")
    PS1='\[\033[01;32m\]\u@\h${fs_mode:+($fs_mode)}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
}

alias ro='sudo mount -o remount,ro / ; sudo mount -o remount,ro /boot'
alias rw='sudo mount -o remount,rw / ; sudo mount -o remount,rw /boot'

# setup fancy prompt"
PROMPT_COMMAND=set_bash_prompt
1
2
3
4
5
6
7
8
9
10
11
# set variable identifying the filesystem you work in (used in the prompt below)
set_bash_prompt(){
    fs_mode=$(mount | sed -n -e "s/^\/dev\/.* on \/ .*(\(r[w|o]\).*/\1/p")
    PS1='\[\033[01;32m\]\u@\h${fs_mode:+($fs_mode)}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
}
 
alias ro='sudo mount -o remount,ro / ; sudo mount -o remount,ro /boot'
alias rw='sudo mount -o remount,rw / ; sudo mount -o remount,rw /boot'
 
# setup fancy prompt"
PROMPT_COMMAND=set_bash_prompt
Execute this new file and look at the magic, the prompt has also changed and show up file system state


root@ambilight:/var/log# . /etc/bash.bashrc
root@ambilight(rw):/var/log# ro
root@ambilight(ro):/var/log# touch test.txt
touch: cannot touch `test.txt': Read-only file system
root@ambilight(ro):/var/log# rw
root@ambilight(rw):/var/log# touch test.txt
root@ambilight(rw):/var/log# ro
root@ambilight(ro):/var/log#
1
2
3
4
5
6
7
8
root@ambilight:/var/log# . /etc/bash.bashrc
root@ambilight(rw):/var/log# ro
root@ambilight(ro):/var/log# touch test.txt
touch: cannot touch `test.txt': Read-only file system
root@ambilight(ro):/var/log# rw
root@ambilight(rw):/var/log# touch test.txt
root@ambilight(rw):/var/log# ro
root@ambilight(ro):/var/log#
Use logout to save history and force Read-Only mode
To be sure to avoid setting back to Read-Only at logout, add the following line to the file
/etc/bash.bash_logout
(may be you need to create it)


mount -o remount,ro /
mount -o remount,ro /boot
1
2
mount -o remount,ro /
mount -o remount,ro /boot
If you want to have your bash history file saved and last known good date, also, put these lines instead

Shell

mount -o remount,rw /
history -a
fake-hwclock save
mount -o remount,ro /
mount -o remount,ro /boot
1
2
3
4
5
mount -o remount,rw /
history -a
fake-hwclock save
mount -o remount,ro /
mount -o remount,ro /boot
Of course you can also enhance the system like changing colors and/or set file system to read only after a certain amount of time. It’s just the way I’m using it.

PS : depending on your configuration, you may have set prompt elsewhere after /etc/bash.bashrc is executed ( ~/.bashrc for example), that would cause overriding the settings of  /etc/bash.bashrc. So if it does not work, test by putting the lines at the end of the user ~/.bashrc  profile file ( /root/.bashrc for root)

Bonus : health check using Watchdog
If you follow my blog, you probably know that I’m using micro-controller day by day. I love the watchdog feature they have and reading original K3A post, I saw we can do the same on Raspberry, so I decided to give it a try of course.

It’s for advanced users, be sure knowing what you do if you don’t want your Pi going into reset loop, most users don’t need this.

Set system to Read-Write before executing these commands, you sure remember this ?


rw
1
rw
Enable watchdog module :


modprobe bcm2708_wdog; apt-get install watchdog
1
modprobe bcm2708_wdog; apt-get install watchdog
Edit the file and
/etc/watchdog.conf
add the following lines at the end of the file


watchdog-device  = /dev/watchdog
max-load-15      = 25  
watchdog-timeout = 10
1
2
3
watchdog-device  = /dev/watchdog
max-load-15      = 25  
watchdog-timeout = 10
On raspbian before jessie (old system init.d) set the watchdog to start at boot and start it now:


insserv watchdog; /etc/init.d/watchdog start
1
insserv watchdog; /etc/init.d/watchdog start
On Jessie edit the file /lib/systemd/system/watchdog.service  and in section [Install]  add the following


[Install]
WantedBy=multi-user.target
1
2
[Install]
WantedBy=multi-user.target
Always on Jessie, enable it by


systemctl enable watchdog
1
systemctl enable watchdog
In addition to the watchdog, you should set up reboot after a kernel panic. This is done by editing
/etc/sysctl.conf
. Add this line:


kernel.panic = 10
1
kernel.panic = 10
This will cause to wait 10 seconds after a kernel panic, then automatically safely reboot the box.

Last test and validation

As last operation, set back to Read-Only and reboot the Pi


ro
reboot
1
2
ro
reboot
References
Has I already said, I followed the original author (K3A) and just added some features I needed. Now that I have one Pi in this mode, I will update all my Pi and may be I need to add some modifications or enhancement not seen yet. If this the case I will update this post, so stay tuned.

If you want to see more precise explanations, I strongly suggest to read the excellent original K3A article located here

Thank’s to Raspbian France for providing this article image.