Expanding storage on a Raspberry Pi using Amazon S3

gistfile1.md

I've been playing around with a Raspberry Pi hosted with Mythic Beasts and wanted a way to keep a large number of infrequently-accessed files on the server. A cost-effective place to store such data is on Amazon S3. After looking around at ways to mount an S3 bucket on the Raspberry Pi and have it appear as a regular filesystem, I came across s3backer. This provides a single file, divided into blocks, each of which is stored as an S3 object. A filesystem is mounted onto this file using a loopback mount. This setup has many benefits which are explained well in the README. Here's how to set it up on the Raspberry Pi.

I'm going to assume that you have an S3 bucket mybucket created and a user with credentials to access it. Although not explicitly specified, some of the commands below need to be run as root.

First, install prerequisites (the first line is some generic tools that might not be installed on a virgin RPI).

% apt-get install git autoconf automake pkg-config gcc make
% apt-get install libcurl4-openssl-dev libfuse-dev libexpat1-dev libssl-dev fuse

Build and install s3backer. I like to do this in /usr/local/*.

% cd /usr/local/src
% git clone https://github.com/archiecobbs/s3backer.git && cd s3backer
% ./autogen.sh
% ./configure --prefix=/usr/local
% make
% make install

Put the AWS access credentials and a key for encrypting the blocks in /usr/local/etc.

% echo AWSACCESSID:AWSACCESSKEY > /usr/local/etc/s3backer.aws
% echo PASSPHRASE > /usr/local/etc/s3backer.crypt
% chmod 600 /usr/local/etc/s3backer.*

Create the s3backer file. You don't have to specify a prefix, but it lets you store multiple s3backer filesystems in the same S3 bucket. I've specified the London AWS region as that is closest to where my RPI is hosted.

% mkdir /mnt/backup.s3b
% s3backer \
> --accessFile=/usr/local/etc/s3backer.aws \
> --blockSize=256K \
> --listBlocks \
> --encrypt --passwordFile=/usr/local/etc/s3backer.crypt \
> --prefix=backup- \
> --region=eu-west-2 --ssl \
> --size=100G \
> mybucket /mnt/backup.s3b

You can treat /mnt/backup.s3b/file as a block device and create an "upper" filesystem mounted to /backup. I use ext4 but it doesn't have to be.

% mkdir /backup
% mke2fs -t ext4 -E nodiscard -F /mnt/backup.s3b/file
% mount -o loop /mnt/backup.s3b/file /backup

You can now start using /backup. If you want to mount the filesystem automatically on boot, then you'll need firstly to put a couple of entries into /etc/fstab. I've wrapped the lines for readability, but they shouldn't be wrapped in the file.

s3backer#mybucket       /mnt/backup.s3b fuse    noauto,quiet,accessFile=/usr/local/etc/s3backer.aws,blockSize=256K,
                                                encrypt,passwordFile=/usr/local/etc/s3backer.crypt,region=eu-west-2,
                                                size=100G,ssl,prefix=backup-    0   0

/mnt/backup.s3b/file    /backup         ext4     noauto,loop     0   0

Then put the following lines into /etc/rc.local. Without setting the clock before attempting to mount the shares (on my RPI at least), it will fail as it believes the date is 1970-01-01.

/usr/sbin/ntpd -gq
/bin/mount /mnt/backup.s3b
/bin/mount /backup

Reboot to test, and you should be done.

Thanks - Ken

I am getting the following error on running mke2fs command:

pi@raspberry:/usr/local/src/s3backer $ sudo mke2fs -t ext4 -E nodiscard -F /mnt/backup.s3b/file
mke2fs 1.43.4 (31-Jan-2017)
Could not open /mnt/backup.s3b/file: Transport endpoint is not connected

It's possible to use wasabi S3 with this software?
Thank you