The other day, I saw a thread on Reddit where a fellow was looking for a program which could duplicate the functionality of a system called Mailstore Home, a piece of Windows software which can be used to archive copies of your e-mail for later searching. It features the ability to encrypt the emails, separate them by which accounts they were recieved in, and search them quickly. When I saw his question I thought surely some equivalent had to exist already, because all of that is pretty readily done with existing Unix software. In fact, it's pretty easy to think of a way to replicate that functionality using only shell scripts and at least vaguely unix-y programming style. This is because Unix was designed from the ground up to pass around streams of data in plain text for processing by programs that do "one thing" and "one thing well." Whether that one thing and one thing well is uniformly achieved isn't really that relevant because I don't think it can be a goal in-and-of-itself, but what's really important to understand is that Unix accomodates a programming model where separate programs can be connected together by their inputs and outputs, and the core utilities provided by all modern Unix-like operating systems can be used to achieve the goals of this Mailstore program. (After I got through most of this guide it got alot less pipey, but it's still strongly shelly.)
A little bit of shell scripting knowledge will be helpful, but I think most people should be able to follow along with no practical programming knowledge at all and hopefully those people will develop a taste for scripting the shell from this tutorial. That said, I'm not sure that's the whole point. Maybe a zero-configuration equivalent should be available. I'm not an expert, hell I'm not even really a Unix "guy" more just a fan. So here goes.
* Disclaimer - I haven't used Mailstore extensively yet, simply read the publicly available marketing literature on the manufacturers web site and the desired functionality of the redditor seeking a replacement. I won't be fully replacing Mailstore for people who need "advanced" features.
Mailstore's commercial editions consist of a client which exists in several forms, and exists to make decisions about which emails to archive, connect to a server, and provide an interface to the user for configuring and interfacing with the archive, and a server which appers to be some kind of mail server with special archiving features. I'm pretty sure it makes sense to the person who designed it, they made something that enough people who want to use, they clearly had a plan, but it's not the way I would provide the same features for myself.
- Thunderbird: Technically, it would be even easier to do this with Mutt, but I'm pretty sure everyone already configuring Mutt is already doing what they need to do with their .bash_aliases file or something like that anyway.
- encFS: encFS is an encryption system which is capable of creating encrypted folders and mounting them via FUSE(File Systems in User Space).
- An Almquist/Bourne style Shell(Such as bash or dash, they both come with Mint): This is what we'll be using to write short scripts that help us put together our email archiving and searching system.
- A bunch of GNU CoreUtils and pretty standard Unix programs: Such as rsync, mv, find, grep, cat, head, tail, and so on, you should already have most of these.
- A graphical program for prompting the user to input a password. On Ubuntu some of these are ssh-askpass-gnome, razorqt-openssh-askpass, ksshaskpass, and lxqt-openssh-askpass.
- SyncThing: Probably the most controversial part of this plan, certainly the only one you might have to install from a third-party source, SyncThing is a peer-to-peer file-sharing program that people sync files between a few computers using bittorrent to find eachother, which makes it very easy to configure. You could use lots of things for this, but SyncThing is basically an all-in-one solution for this kind of backup.
The first thing that it's important to know is where Thunderbird stores emails on your behalf under Linux Mint and how to reliably make sure you can find the folder from the terminal. This is pretty easy. Ubuntu keeps the user's Thunderbird profile and Thunderbird data under /home/username/.thunderbird. When you access this folder on the terminal you'll want to use the following form.
$HOME/.thunderbird
Because $HOME expands to /home/username.
Next, we'll need to make this a symbolic link to a different location which were going to backup later. I'm going to use this folder: $HOME/.thunderbird-backup/thunderbird. So to create the symbolic link, first create the directory, move the original folder, and create the symbolic link.
mkdir -p $HOME/.thunderbird-backup
mv $HOME/.thunderbird $HOME/.thunderbird-backup/thunderbird
ln -s $HOME/.thunderbird-backup/thunderbird $HOME/.thunderbird
So let's put this together into a script we can use automatically.
#! /usr/bin/env sh
#This is a bash conditional statement, or "if-then" statement.
#A basic if-then statement looks like this
# if [ $CONDITION ]; then
# fi
# -f checks for the existence of a folder, and ! inverts the result, so
# ! -d checks for the non-existence of a folder.
if [ ! -d "$HOME/.thunderbird-backup/.thunderbird" ]; then
#mkdir -p make a directory, and if necessary, all the parent
#directories, in the path to your thunderbird backup folder.
mkdir -p "$HOME/.thunderbird-backup/thunderbird"
#-e checks for the existence of a file or folder, but in the
#case of symbolic links, the file that is linked to. If we know
#that $HOME/.thunderbird is a symlink to something that is
#broken(You'll find out why in the next step) then we'll assume
#that we don't need to copy $HOME/.thunderbird to the backup.
if [ ! -e $HOME/.thunderbird ]; then
echo "Thunderbird symlink found but not available. Aborting normally."
exit 0
elif [ -f $HOME/.thunderbird ]; then
mv $HOME/.thunderbird $HOME/.thunderbird-backup/thunderbird
ln -s $HOME/.thunderbird-backup/thunderbird $HOME/.thunderbird
exit 1
fi
fi
A word on security: In this step we set up encFS, which has a handful of weaknesses that were identified in a recent professional, paid security audit which may be able to weaken it somewhat, and which are being mitigated on the way to the 2.0 release, so please make sure that you make sure you keep your encFS up to date. I think it's good enough to use, but I don't know everything and I don't understand everything. I just read.
Another word on security: In order to do our backups, we'll be relying on two forms of encryption with particular properties to achieve our particular goals. What encFS provides us is file/folder encryption at the endpoint. This is distinct from end-to-end encryption, which we'll use for a different part of the process, and it's important to point this out because except a few special cases involving "Fully Homomorphic Encryption" which isn't available yet, whenever the information is searchable, it is vulnerable to the machine searching it. Not so when it is encrypted. So we use encFS to encrypt and password protect the searchable archive when it's not in use.
First, we'll need to create a folder for use as our encrypted file system. This is easy enough. We just need to create one more folder in our $HOME/.thunderbird-backup directory. Just run:
mkdir -p $HOME/.thunderbird-backup/.thunderbird-encrypted
Next we need to move the contents only the contents to a temporary location for storage.
mkdir -p $HOME/.thunderbird-backup/tmp/
mv $HOME/.thunderbird-backup/thunderbird/* $HOME/.thunderbird-backup/tmp
Mount the $HOME/.thunderbird-backup/thunderbird folder to the $HOME/.thunderbird-backup/.thunderbird-encrypted and move the temporary files back into the $HOME/.thunderbird-backup/thunderbird folder to encrypt them.
encfs --extpass=ssh-askpass $HOME/.thunderbird-backup/.thunderbird-encrypted $HOME/.thunderbird-backup/thunderbird
mv $HOME/.thunderbird-backup/tmp/* mv $HOME/thunderbird-backup/thunderbird/*
With all this in mind, we can set up another script which we can use automatically, but before we do that, think about this: We don't actually need $HOME/thunderbird-backup/thunderbird to be around except when it's mounted to $HOME/thunderbird-backup/.thunderbird-encrypted, and we can use this fact to our advantage. Starting with a cleaned-up version of our script from earlier:
#! /usr/bin/env sh
TIMER=30
if [ ! -f "$HOME/.thunderbird-backup/.thunderbird" ]; then
mkdir -p "$HOME/.thunderbird-backup/thunderbird"
if [ ! -e $HOME/.thunderbird ]; then
echo "Thunderbird symlink found but not available. Prompting $HOME for password."
mkdir -p $HOME/.thunderbird-backup/thunderbird
encfs --extpass=ssh-askpass -i $TIMER $HOME/.thunderbird-backup/.thunderbird-encrypted $HOME/.thunderbird-backup/thunderbird
elif [ -n "$(ls -A $HOME/.thunderbird)" ]; then
mkdir -p $HOME/.thunderbird-backup/thunderbird
encfs --extpass=ssh-askpass -i $TIMER $HOME/.thunderbird-backup/.thunderbird-encrypted $HOME/.thunderbird-backup/thunderbird
elif [ -d $HOME/.thunderbird ]; then
mv $HOME/.thunderbird $HOME/.thunderbird-backup/tmp
ln -s $HOME/.thunderbird-backup/thunderbird $HOME/.thunderbird
mkdir -p $HOME/.thunderbird-backup/thunderbird
encfs --extpass=ssh-askpass -i $TIMER $HOME/.thunderbird-backup/.thunderbird-encrypted $HOME/.thunderbird-backup/thunderbird
mv $HOME/.thunderbird-backup/tmp/* $HOME/.thunderbird-backup/thunderbird/*
fi
fi
We're going to take a break from shell scripting to set up a way to actually make the remote copy of the backup folder. In order to do this, we're going to configure the folder $HOME/.thunderbird-backup/.thunderbird-encrypted folder to sync with a set of other computers via Syncthing, a bittorrent based file-syncing technology that makes it easy for people to host their own file-sharing cloud. Unlike regular bittorrent, though, only you and the people you authorize will be able to access your synced folders. Additionally, only people with your password and your keyfile will be able to see the contents of the $HOME/.thunderbird-backup/.thunderbird-encrypted folder, and the contents of your files will be encrypted in transit.
Please note that Syncthing isn't technically a client-server app, it's a client-client app. The same app runs on both sides, in this case, server just means the computer hosting the backup which doesn't have the key.
First, you'll need to get syncthing. For brevity, and to make sure the instructions stay up-to-date, please take a moment to follow the instructions here. Once you've done that, on both the client and the server, you can launch syncthing by running the app, or from the terminal, or if Syncthing is already running, you can direct your browser to (http://127.0.0.1:8384/)[http://127.0.0.1:8384/] to get to the web interface.
First, go to the "server" and open the web interface. Click on the "Action" button in the top right corner and select the "Show ID" menu item. Then open the web interface on the "client" and select "Add Folder" from the main menu. You'll see some boxes, you'll need to enter a value like "Thunderbird Backup" in the Label field and ~/.thunderbird-backup/.thunderbird-encrypted into the Path field. Once you've done that click save and then click "Add Remote Device" from the main menu. Take the Device ID showing on the server and copy it into the Device ID field on the Add Device menu on the client. Name it something like "Thunderbird Backup Server." Once you've done that, your encrypted folder will be kept in sync between the two machines.
The last feature I'll be attempting to emulate today is the searching of the email archive. Now here's the thing, this approach will place symlink at the folder where thunderbird keeps it's default settings, so if you just want to search with the tools Thunderbird provides, that will work without any further modification. With that in mind, I'll be picking this back up tomorrow because I've been neglecting other things.
More than just that this is possible with these tools, something like this is possible with many tools, depending on your needs. For instance, if you want to keep the files on a remote server and access them from the server without keeping them on the client, it might be more convenient to connect to the remote server by mounting with SSHFS instead of syncing folders between clients.