IOTstack tutorial - adding a second drive

IOTstack tutorial – adding a second drive.md

IOTstack tutorial – adding a second drive

Problem statement

This issue has come up a few times on Discord. The question can be summarised like this:

My existing IOTstack installation is running out of space on my primary drive. I have another drive. How do I move the "volumes" directory to the new drive?

This example is based on a Raspberry Pi 4 which is booting from a 500GB USB3 SSD. I will show you how to use a 32GB USB3 "thumb drive" as the secondary drive and move ~/IOTstack/volumes onto it.

Don't get hung up on the details of my hardware. The principles are the same, regardless of whether:

• the "primary" is an SD card, SSD or anything else.
• the "secondary" is a thumb drive, SSD or anything else.

Establish a baseline

Before you attach the secondary drive to your Raspberry Pi, list your existing drives:

$ lsblk
sda      8:0    0 465.8G  0 disk 
├─sda1   8:1    0   256M  0 part /boot
└─sda2   8:2    0 465.5G  0 part /

Connect and identify secondary drive

Now, attach the second drive and repeat the command:

$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 465.8G  0 disk 
├─sda1   8:1    0   256M  0 part /boot
└─sda2   8:2    0 465.5G  0 part /
sdb      8:16   1  28.7G  0 disk 
└─sdb1   8:17   1  28.7G  0 part 

Because "sdb" and "sdb1" only appeared after attaching the new drive, it should be self-evident that those elements represent the secondary drive.

Notes:

• There are no guarantees that your own system will use "sdb" and "sdb1" when you attach a new drive. If your system uses different identifiers, you will need to substitute those throughout the remainder of this gist.

• It is really important that you pay attention and make sure that you have identified your secondary drive correctly. Don't guess. If you make a mistake at this point, you could easily destroy all the data on your primary drive.

Each entry in the "NAME" column implies a "/dev/" prefix so, in this example:

• the secondary drive physical disk is: /dev/sdb
• the secondary drive logical volume: /dev/sdb1
• the absence of anything in the "MOUNTPOINT" column to the right of "sdb1" means the secondary drive did not auto-mount.

Note:

• If the "MOUNTPOINT" column to the right of "sdb1" is not empty, you will need to unmount that drive:

   $ sudo umount /dev/sdb1
  

Format secondary drive

You now need to prepare the secondary drive for use by Raspbian. Be very careful with the following commands. Double-check each command before you hit return.

$ sudo parted /dev/sdb mklabel gpt
$ sudo parted -a opt /dev/sdb mkpart primary ext4 0% 100%
$ sudo mkfs -t ext4 /dev/sdb1 
$ sudo e2label /dev/sdb1 IOTstackVolumes
$ sudo partprobe

Stop IOTstack

Take down your running IOTstack:

$ cd ~/IOTstack
$ docker-compose down

Move volumes directory out of the way

Rename the existing ./volumes subdirectory to get it out of the way

$ sudo mv ./volumes ./volumes.old

Create mount-point

Create a new (empty) ./volumes subdirectory. This will become the mount-point for the secondary drive:

$ sudo mkdir ./volumes

Test mount-point

$ sudo mount LABEL=IOTstackVolumes ./volumes

Assuming the volume mounts without error, try creating a file inside it:

$ sudo touch ./volumes/testfile
$ ls -l ./volumes/testfile
-rw-r--r--  1 root     root        0 Mar 23 22:26 ./volumes/testfile

If the file shows up, unmount the volume and try again:

$ sudo umount ./volumes
$ ls -l ./volumes/testfile
ls: cannot access './volumes/testfile': No such file or directory

In this case, the file should not show up because the volumes directory should be empty.

Auto-mount secondary drive on reboot

You need to tell Raspbian to auto-mount the secondary drive on each boot. You do that by editing the file:

/etc/fstab

Start by making a backup of the existing file:

$ sudo cp /etc/fstab /etc/fstab.bak

Then list the file you are going to edit:

$ cat /etc/fstab
proc            /proc           proc    defaults          0       0
PARTUUID=7f8f5227-01  /boot           vfat    defaults          0       2
PARTUUID=7f8f5227-02  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that

The layout of the default file is a little messy and harder to understand than it should be. It's probably a good idea to clean it up by adding spaces (not tabs) to align everything. The line you need to add looks like this:

LABEL=IOTstackVolumes /home/pi/IOTstack/volumes ext4    defaults          0  2

Use sudo and your favourite text editor (eg vi or nano) to edit /etc/fstab so that the final result looks something like this:

proc                  /proc                     proc    defaults          0  0
PARTUUID=1801318c-01  /boot                     vfat    defaults          0  2
PARTUUID=1801318c-02  /                         ext4    defaults,noatime  0  1
LABEL=IOTstackVolumes /home/pi/IOTstack/volumes ext4    defaults          0  2

# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that

From left to right, the fstab fields are:

1. An identifier for the file system to be mounted. The earlier e2label command gave the secondary disk a label of "IOTstackVolumes" so that is what we will use here. We could also use the drive's UUID or PARTUUID but a label is more informative. Using an identifier rather than a device path means that the disk will still mount correctly, even if it appears on a different physical USB port.
2. Where the file system should be mounted. The secondary disk is going to be used for IOTstack volumes data so that is where it should be mounted. Note that fstab is processed by root so an absolute path is required.
3. The type of the file system (ext4 effectively means "Linux Native").
4. The mount options. Defaults are appropriate.
5. The "0" means "don't dump the file system".
6. The priority order for running fsck at boot time.

Save your work and then reboot:

$ sudo reboot

Confirm secondary drive auto-mounts on reboot

Immediately after the reboot, make sure that the volume has mounted correctly:

$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 465.8G  0 disk 
├─sda1   8:1    0   256M  0 part /boot
└─sda2   8:2    0 465.5G  0 part /
sdb      8:16   1  28.7G  0 disk 
└─sdb1   8:17   1  28.7G  0 part /home/pi/IOTstack/volumes

If you see the correct mount-point, it means the fstab has done its work and the secondary disk has mounted as expected. You should also be able to see the test file you created earlier:

$ cd ~/IOTstack/volumes
$ ls -l testfile
-rw-r--r--  1 root     root        0 Mar 23 22:26 testfile

You can remove that file:

$ sudo rm testfile

Migrate your data

You have two choices:

1. EITHER: copy the existing data to the new drive:

   $ cd ~/IOTstack
   $ sudo cp -a ./volumes.old/* ./volumes
   

   This approach has the advantage of being non-destructive. If anything goes wrong with the copy, you still have the existing data and can start over. The disadvantage is that no space is freed-up on your primary drive, and you will have to remember to come back later and delete the old data.

2. OR: move the existing data to the new drive:

   $ cd ~/IOTstack
   $ sudo mv ./volumes.old/* ./volumes
   

   Because the target folder is on a different physical drive, the mv is actually implemented as a copy followed by a remove.

Whichever option you choose, at some future time you should clean up the old folder:

$ cd ~/IOTstack
$ sudo rm -rf ./volumes.old

Reactivate your stack

There is no need to edit your compose file. As far as docker-compose is concerned, the ./volumes folder is in the right place. Just bring up your stack as usual:

$ docker-compose up -d

Caveats

If you have been in the habit of doing any of the following:

1. Renaming or erasing ~/IOTstack
2. Renaming or erasing ~/IOTstack/volumes

you are not going to be able to do that. The absolute path to the /home/pi/IOTstack/volumes mount-point must remain where it is.

One more thing…

Instead of making ~/IOTstack/volumes the mount point for a second drive, you could move the goal-posts slightly and make ~/IOTstack the mount point. That way your your entire IOTstack structure would be on your secondary drive.