Script to backup Firefly III database, uploads and config files installed with docker-compose

firefly-iii-backuper.sh

#!/bin/bash

files_to_backup=(.*.env .env docker-compose.yml )

info()  { echo -e "\\033[1;36m[INFO]\\033[0m  \\033[36m$*\\033[0m" >&2; }
warn()  { echo -e "\\033[1;33m[WARNING]\\033[0m  \\033[33m$*\\033[0m" >&2; }
fatal() { echo -e "\\033[1;31m[FATAL]\\033[0m  \\033[31m$*\\033[0m" >&2; exit 1; }

intro () {
    echo " ====================================================="
    echo "     Backup & Restore docker based FireFly III v1.6   "
    echo " ====================================================="
    echo " It automatically detects db & upload volumes based on the name matching the following regex: firefly[_-](iii|)[_-]?"
    echo " Requirements:"
    echo " - Place the script in the same directory where your docker-compose.yml and .env files are saved"
    echo " Warning: The destination directory is created if it does not exist"
}

usage () {
    echo "Usage: $0 backup|restore /tmp/backup/destination/dir [no_files]"
    echo "- backup|restore - Action you want to execute"
    echo "- destination path of your backup file including file name"
    echo "- optionally backup or restore volumns only when no_files parameter is passed"
    echo "Example backup: $0 backup /home/backup/firefly-2022-01-01.tar.gz"
    echo "Example restore: $0 restore /home/backup/firefly-2022-01-01.tar.gz"
    echo "To backup once per day you can add something like this to your cron:"
    echo "1 01 * * * bash /home/myname/backuper.sh backup /home/backup/\$(date '+%F').tar.gz"
    echo "To restore a database of a specific Firefly version follow the steps below"
    echo "- Look for the Firefly version from your backup. It's in <backup>.tar.gz/version.txt"
    echo "- Use the Firefly docker tag (https://hub.docker.com/r/fireflyiii/core/tags) corresponding to your version"
    echo "- Change the firefly image to a tag from your version. Example:"
    echo "  image: fireflyiii/core:version-6.1.10"
    echo "- Run docker compose up"
    echo "- Run this script to restore the backup"
}

backup () {
    script_path="$1"

    if [ ! -d "$(dirname $2)" ]; then
        info "Creating destination directory: $(dirname $2)"
        mkdir -p "$(dirname $2)"
    fi

    full_path=$(realpath $2)
    dest_path="$(dirname $full_path)"
    dest_file="$(basename $full_path)"
    upload_volume="$3"
    no_files=$4

    to_backup=()

    if [ -f "$full_path" ]; then
        warn "Provided file path already exists: $full_path. Overwriting"
    fi

    # Create temporary directory
    if [ ! -d "$dest_path/tmp" ]; then
        mkdir "$dest_path/tmp"
    fi

    # Files backup
    if [ $no_files = "false" ]; then
        not_found=()
        for pattern in "${files_to_backup[@]}"; do
            for file in ${script_path}/${pattern}; do
                if [[ ! -f $file ]]; then
                    not_found+=("$file")
                else
                    cp "$file" "$dest_path/tmp/"
                    to_backup+=($(basename "$file"))
                fi
            done
        done

        if ((${#not_found[@]})); then
            warn "The following files were not found in $script_path: ${not_found[@]}. Skipping."
        fi

        if ((${#to_backup[@]})); then
            info "Backing up the following files in $script_path: ${to_backup[@]}"
        fi
    fi

    # Version
    app_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?(core|app)' | cut -d ' ' -f 1)
    app_version=$(docker exec -it $app_container grep -F "'version'" /var/www/html/config/firefly.php | tr -s ' ' | cut -d "'" -f 4)
    db_version=$(docker exec -it $app_container grep -F "'db_version'" /var/www/html/config/firefly.php | tr -s ' ' | tr -d ',' | cut -d " " -f 4)
    info 'Backing up App & database version numbers.'
    echo -e "Application: $app_version\nDatabase: $db_version" > "$dest_path/tmp/version.txt"
    to_backup+=(version.txt)

    # DB container
    db_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?db' | cut -d ' ' -f 1)
    if [ -z $db_container ]; then
        warn "db container is not running. Not backing up."
    else
        info 'Backing up database'
        docker exec $db_container bash -c '/usr/bin/mariadb-dump -u $MYSQL_USER --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"' > "$dest_path/tmp/firefly_db.sql"
        to_backup+=("firefly_db.sql")
    fi

    # Upload Volume
    if [ -z $upload_volume ]; then
        warn "upload volume does NOT exist. Not backing up."
    else
        info 'Backing up upload volume'
        docker run --rm -v "$upload_volume:/tmp" -v "$dest_path/tmp:/backup" alpine tar -czf "/backup/firefly_upload.tar.gz" -C "/" "tmp"
        to_backup+=("firefly_upload.tar.gz")
    fi

    # Compress
    tar -C "$dest_path/tmp" -czf "$dest_path/$dest_file" --files-from <(printf "%s\n" "${to_backup[@]}")

    # Clean up
    for file in "${to_backup[@]}"; do
        rm -f "$dest_path/tmp/$file"
    done
    rmdir "$dest_path/tmp"
}

restore () {
    script_path="$1"
    full_path=$(realpath $2)
    src_path="$(dirname $full_path)"
    backup_file="$(basename $full_path)"
    upload_volume="$3"
    no_files=$4

    if [ ! -f "$src_path/$backup_file" ]; then
        fatal "Provided backup file does not exist: $path"
    fi

    # Create temporary directory
    if [ ! -d "$src_path/tmp" ]; then
        mkdir "$src_path/tmp"
    fi

    # Files restore
    if [ $no_files = "false" ]; then
        tar -C "$src_path/tmp" -xf "$src_path/$backup_file"
        readarray -t <<<$(tar -tf "$src_path/$backup_file")
        # restored=(${MAPFILE[*]})
        not_found=()
        restored=()
        for f in "${files_to_backup[@]}"; do
            if [ ! -f "$script_path/$f" ]; then
                not_found+=("$f")
            else
                cp "$src_path/tmp/$f" .
                restored+=("$f")
            fi
        done

        if ((${#not_found[@]})); then
            warn "The following files were not found in $script_path: ${not_found[@]}. Skipping."
        fi

        if ((${#restored[@]})); then
            info "Restoring the following files: ${restored[@]}"
        fi
    else
        tar -C "$src_path/tmp" -xf "$src_path/$backup_file" firefly_db.sql firefly_upload.tar.gz
        restored=(firefly_db.sql firefly_upload.tar.gz)
    fi

    if [ ! -z $upload_volume ]; then
        warn "The upload volume exists. Overwriting."
    fi

    docker run --rm -v "$upload_volume:/recover" -v "$src_path/tmp:/backup" alpine tar -xf /backup/firefly_upload.tar.gz -C /recover --strip 1
    restored+=(firefly_upload.tar.gz)

    db_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?db' | cut -d ' ' -f 1)

    if [ -z $db_container ]; then
        warn "The db container is not running. Not restoring."
    else
        info 'Restoring database'
        cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container bash -c '/usr/bin/mariadb -u $MYSQL_USER --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"'
        restored+=(firefly_db.sql)
    fi

    restored+=(version.txt)
    # Clean up
    for file in "${restored[@]}"; do
        rm -f "$src_path/tmp/$file"
    done
    rmdir "$src_path/tmp"
}

main () {
    intro

    if [ $# -lt 2 ]; then
        fatal "Not enough parameters.\n$(usage)"
    fi

    backuper_dir="$(dirname $0)"
    action=$1
    path="$2"
    if [ -z "$3" ]; then
        no_files=false
    else
        no_files=true
    fi

    if [ -d "$path" ]; then
        fatal "Path is an existing directory. It has to be a file path"
    fi

    upload_volume="$(docker volume ls | grep -F "firefly_iii_upload" | tr -s ' ' | cut -d ' ' -f 2)"

    if [ "$action" == 'backup' ]; then
        backup "$backuper_dir" "$path" "$upload_volume" $no_files
    elif [ "$action" == 'restore' ]; then
        restore "$backuper_dir" "$path" "$upload_volume" $no_files
    else
        fatal "Unrecognized action $action\n$(usage)"
    fi
}

main "$@"

Thank you for this script! Really appreciate the work you shared here. I am starting to experiment with firefly-iii, and I ran into a couple issues with this script.

First, I noticed all my containers were being named firefly-iii-*, rather than fireflyiii-*, and the grep statements in the script weren't finding them. I couldn't find a recent commit in the firefly repo that changed the container names, so maybe it's just something I'm not understanding with regex or how docker names things. Changing to grep -E 'firefly-iii[_-]app' seemed to solve the problem.

Second, I see the examples are running the script using a dynamic name for the backup file so that the filename has the day's date in it. My understanding is that you could run that example every day and create a series of backup files, a new one for each day. However, the script uses the realpath command to set the full_path and dest_path variables in the backup function, and realpath fails if the file doesn't currently exist. Am I misunderstanding the example? Should the backup file already exist and we just overwrite it with the latest backup?

First, I noticed all my containers were being named firefly-iii-*, rather than fireflyiii-*, and the grep statements in the script weren't finding them. I couldn't find a recent commit in the firefly repo that changed the container names, so maybe it's just something I'm not understanding with regex or how docker names things. Changing to grep -E 'firefly-iii[_-]app' seemed to solve the problem.

I think, that the base name in front of the container names (app, db) comes from the folder you are in. Is your directory named firefly-iii by chance?
Edit: Normally firefly is used, not fireflyiii or firefly-iii.

Is your directory named firefly-iii by chance? Edit: Normally firefly is used, not fireflyiii or firefly-iii.

It sure is, so that explains that question, appreciate it!

Second, I see the examples are running the script using a dynamic name for the backup file so that the filename has the day's date in it. My understanding is that you could run that example every day and create a series of backup files, a new one for each day.

Right.

However, the script uses the realpath command to set the full_path and dest_path variables in the backup function, and realpath fails if the file doesn't currently exist. Am I misunderstanding the example? Should the backup file already exist and we just overwrite it with the latest backup?

No. The backup file does not need to be existing. If it does, it will be overwritten (and a warning will be shown, but without asking for confirmation).

In the main function line 196, the full path of the backup file is tested. If it is an existing directory, the command would fail. Otherwise (I.e., it does not exist, or it is a file path) it will continue.

The main function later calls the backup function passing the path. At the beginning of the backup function, all the parent directories of the path are created if they do not exist just before calling realpath.

According to realpath's man page:

Print the resolved absolute file name; all but the last component must exist

So, It is not required for the backup file to be already existing.

Edit: Normally firefly is used, not fireflyiii or firefly-iii.

It actually happens that I have firefly-iii also as my directory name. It maybe not be as unusual as you might think. I think the documentation should suggest a name to make the docker container name more consistent. I have filed an issue for that

@dawid-czarnecki
Thank you for the script!
I have some notes (see my fork):

• mysql and mysqldump were deprecated and are now removed from the docker image (reference)
• Compressed tar files (-z flag) usually have .tar.gz file extension
• Creating directories in line 46 is duplicated since it should have been already created in line 34
• I think checking dest_path in line 48 would check the file name against the current working directory, instead of the backup destination directory
• Using fgrep gives a warning of fgrep: warning: fgrep is obsolescent; using grep -F
• Use alpine instead of ubuntu image as it is smaller, but does the job
• Some typos

Edit: Normally firefly is used, not fireflyiii or firefly-iii.

It actually happens that I have firefly-iii also as my directory name. It maybe not be as unusual as you might think. I think the documentation should suggest a name to make the docker container name more consistent. I have filed an issue for that

As per the issue, container names have change to: firefly_iii_core and firefly_iii_db (commit). This will break the current grep in the script. However, the script would be already half-broken for anyone who updates (and pull) his docker file because of "mysql and mysqldump were deprecated and are now removed from the docker image (reference)"

I have updated my fork accordingly.

hi guys, i use Firefly III with Docker on an old Mac Mini that serves as my server. How can I modify the script so that it also works with macOS

What have you identified that doesn't work?

@Pindol83 check my fork. It contains some bug fixes. For more details, see my latest comment. If problems still happens, add more details of the issues you encounter.

@WaleedMortaja Huge thanks for your bug fixes and updates!
I applied all of them and changed the container name regular expressions to catch all that were mentioned here.
It should work both on 5.x and 6.x versions of Firefly.

@jbhammon Thanks for your comments. Should work on your container names now.

Let me know if there are any problems.

Restore

cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container bash -c '/usr/bin/psql --username=hugo fireflyiii'

Hello @hjhp
I have tested your restore and it does not work for me. It's printing many errors, i.e. relations etc. already exist.
Am I missing something here? Have you tested the restore in the meantime?

All I did for the restore was replacing

cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container bash -c '/usr/bin/mariadb -u $MYSQL_USER --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"'

with (my database name is test and my username aswell)

cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container bash -c '/usr/bin/psql --username=test test'

After that I registered a user and made a backup (working with your suggestions).
Then I removed all containers & volumes and started/created it all up. Once running I ran the restore script.

Thank you very much in advance :)

@MuellerNicolas I moved to GnuCash in 2022-11, principally (if memory serves) because I didn't enjoy the way Firefly-III handled debts. I never ended up testing a restore. Some observations though: my restore line is not vastly different from the original by @dawid-czarnecki and @WaleedMortaja's fork, save that they use /usr/bin/mariadb and I use /usr/bin/psql.

I don't have much experience with Postgres; from my understanding the way all these backups work is they generate a file with SQL statements that can act as a restore file by feeding its contents into a SQL prompt, and that this general approach is the same across many databases (Postgres, MySQL, MariaDB…).

Without seeing your error messages exactly, it won't be easy to suggest many ideas:

• If the "relations already exist", then my first thought is that somehow you're not actually starting with a clean slate: could something have been omitted during your deletion process?
• If I read your comment correctly, you followed my script (i.e. Postgres) rather than the others' (using MariaDB). What happens if you try a backup-and-restore on a MariaDB setup?

Restore with postgresql container

Problem was that the container needs to be running for the restore. But when you start the db container it automatically creates a new database.
Solution was to drop the database, create a fresh one and recover the dump. The following is working for me:

info 'Step 1: Drop existing database'
docker exec -i $db_container dropdb -U username databasename
info 'Step 2: Create fresh database'
docker exec -i $db_container createdb -U username databasename
info 'Step 3: Restore dump'
cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container psql -U username -d databasename

It might be easier to comment if you share your Docker command (or docker-compose.yml if that's what you're using).

• At first glance my uncaffeinated brain wants to suggest that whatever image you're using just doesn't have bash ("bash": executable file not found), which is then causing commands like cat not to work, but if you're running fireflyiii/core:latest then this feels improbable (though I haven't tried running Firefly at all in recent times).
• If cat doesn't work, do things like less work? Can you run ls? Do any of the typical commands work?
• Have you verified that the contents of firefly_db.sql makes sense? (This is probably unrelated to any of the errors, just covering all bases.)

I would also note though that you seem to have 2 errors here: "docker: invalid spec" and "OCI runtime exec failed".

Somehow the tar file only gets a date in my case.
2014-01-18.tar
So no version or anything (this part '+%F' doesn't work).
If anyone knows how I can get that fixed..... :)

Other than that the save and restore work fine!

@MuellerNicolas Unfortunately I don't tested it with PostgreSQL so can't help much. With mariadb the restore works just fine.
@PuckStar Not sure why this doesn't work. Do you use bash? Try date '+%Y-%m-%d'

@PuckStar Not sure why this doesn't work. Do you use bash? Try date '+%Y-%m-%d'

The date part already worked. It's the Firefly version that is not included in the filename. But looking at your script again I think I expected something that is not there :). You only include "firefly" in the filename so not also the actual version of Firefly. Right?
So this is in your script as example: firefly-2022-01-01.tar.gz
But mine come out as 2022-01-01.tar
In that case let's ignore it.

Thanks by the way for your script! It's very helpful!

@PuckStar Yes, that's correct. You can still find the firefly and database version inside the archive in version.txt file.
You are welcome! I'm happy it helps :)

Thank you for the script.
Suggestion : Rename ".fidi.env" file to ".importer.env", as its actual name on Firefly-III documetation : https://docs.firefly-iii.org/how-to/data-importer/installation/docker/#:~:text=save%20it%20as-,.importer.env,-next%20to%20the

@Breach7874 Good point. I updated it so it backups all .*.env files. It used to be .fidi.env in previous Data Importer versions so now it supports older and newer versions.

1.5 doesn't backup .env, all the other dot files are.

Thanks @BoomerET for the information. I've just updated it to backup both .*.env and .env files. Additionally, I fixed a bug that caused the script to not find the .*.env files if executed from a different directory.

There's an issue running this in cron that is not the fault of the script but the Firefly III documentation on how to run this with cron. I wanted to post this because I spent an entire day thinking it was the script's fault but it's just a configuration issue.

The Firefly III documentation says to set the cron this way:
1 01 * * * bash /home/myname/backuper.sh backup /home/backup/$(date '+%F').tar

This is incorrect because of the percent sign that is used. Cron uses percents so it needs to be escaped for it to fire off. Correct way:
1 01 * * * bash /home/myname/backuper.sh backup /home/backup/$(date '+\%F').tar

Figured I'd post it here to save everyone time since there aren't any comments on Firefly III side.

@jlindfors Thank you for posting this, it drove me crazy.

@jlindfors @BoomerET Fixed in the documentation.