Script to backup Firefly III database, uploads and config files installed with docker-compose

firefly-iii-backuper.sh

#!/bin/bash

files_to_backup=(.env .db.env .fidi.env docker-compose.yml )

info()  { echo -e "\\033[1;36m[INFO]\\033[0m  \\033[36m$*\\033[0m" >&2; }
warn()  { echo -e "\\033[1;33m[WARNING]\\033[0m  \\033[33m$*\\033[0m" >&2; }
fatal() { echo -e "\\033[1;31m[FATAL]\\033[0m  \\033[31m$*\\033[0m" >&2; exit 1; }

intro () {
    echo " ====================================================="
    echo "     Backup & Restore docker based FireFly III v1.4-1 "
    echo " ====================================================="
    echo " It automatically detects db & upload volumes based on the name matching the following regex: firefly[_-](iii|)[_-]?"
    echo " Requirements:"
    echo " - Place the script in the same directory where your docker-compose.yml and .env files are saved"
    echo " Note: The destination directory is created if it does not exist"
}

usage () {
    echo "Usage: $0 backup|restore destination [no_files]"
    echo "- backup|restore : Action you want to execute"
    echo "- destination : The path for your backup file (including the file name)"
    echo "- (optional) no_files : When passed, exclude the docker and environment files. Only backup or restore volumes"
    echo "Example backup: $0 backup /home/backup/firefly-2022-01-01.tar.gz"
    echo "Example restore: $0 restore /home/backup/firefly-2022-01-01.tar.gz"
    echo "To backup once per day you can add something like this to your cron:"
    echo "1 01 * * * bash \"$(realpath $0)\" backup /home/backup/\$(date '+%F').tar.gz"
}

backup () {
    script_path="$1"
    dest_path="$(dirname $path)"
    dest_file="$(basename $path)"
    upload_volume="$3"
    no_files=$4

    to_backup=()

    if [ ! -d "$dest_path" ]; then
        info "Creating destination directory: $dest_path"
        mkdir -p "$dest_path"
    fi

    if [ -f "$path" ]; then
        warn "Provided file path already exists: $path."
	read -p "Do you want to overwrite? (yes/no) " yn

	case $yn in
		yes) warn overwriting... ;;
		no ) info exiting... ; exit;;
		*  ) fatal invalid response ;;
	esac
    fi

    # Create temporary directory
    if [ ! -d "$dest_path/tmp" ]; then
        mkdir "$dest_path/tmp"
    fi

    # Files backup
    if [ $no_files = "false" ]; then
        not_found=()
        for f in "${files_to_backup[@]}"; do
            if [ ! -f "$script_path/$f" ]; then
                not_found+=("$f")
            else
                cp "$script_path/$f" "$dest_path/tmp/"
                to_backup+=("$f")
            fi
        done

        if ((${#not_found[@]})); then
            warn "The following files were not found in $script_path: ${not_found[@]}. Skipping."
        fi

        if ((${#to_backup[@]})); then
            info "Backing up the following files in $script_path: ${to_backup[@]}"
        fi
    fi

    # Version
    app_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?(core|app)' | cut -d ' ' -f 1)
    app_version=$(docker exec -it $app_container grep -F "'version'" /var/www/html/config/firefly.php | tr -s ' ' | cut -d "'" -f 4)
    db_version=$(docker exec -it $app_container grep -F "'db_version'" /var/www/html/config/firefly.php | tr -s ' ' | tr -d ',' | cut -d " " -f 4)
    info 'Backing up App & database version numbers.'
    echo -e "Application: $app_version\nDatabase: $db_version" > "$dest_path/tmp/version.txt"
    to_backup+=(version.txt)

    # DB container
    db_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?db' | cut -d ' ' -f 1)
    if [ -z $db_container ]; then
        warn "db container is not running. Not backing up."
    else
        info 'Backing up database'
        docker exec $db_container bash -c '/usr/bin/mariadb-dump -u $MYSQL_USER --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"' > "$dest_path/tmp/firefly_db.sql"
        to_backup+=("firefly_db.sql")
    fi

    # Upload Volume
    if [ -z $upload_volume ]; then
        warn "upload volume does NOT exist. Not backing up."
    else
        info 'Backing up upload volume'
        docker run --rm -v "$upload_volume:/tmp" -v "$dest_path/tmp:/backup" alpine tar -czf "/backup/firefly_upload.tar.gz" -C "/" "tmp"
        to_backup+=("firefly_upload.tar.gz")
    fi

    # Compress
    tar -C "$dest_path/tmp" -czf "$dest_path/$dest_file" --files-from <(printf "%s\n" "${to_backup[@]}")

    # Clean up
    for file in "${to_backup[@]}"; do
        rm -f "$dest_path/tmp/$file"
    done
    rmdir "$dest_path/tmp"
}

restore () {
    script_path="$1"
    src_path="$(dirname $path)"
    backup_file="$(basename $path)"
    upload_volume="$3"
    no_files=$4

    if [ ! -f "$path" ]; then
        fatal "Provided backup file does not exist: $path"
    fi

    # Create temporary directory
    if [ ! -d "$src_path/tmp" ]; then
        mkdir "$src_path/tmp"
    fi

    # Files restore
    if [ $no_files = "false" ]; then
        tar -C "$src_path/tmp" -xf "$src_path/$backup_file"
        #readarray -t <<<$(tar -tf "$src_path/$backup_file")
        # restored=(${MAPFILE[*]})
        not_found=()
        restored=()
        for f in "${files_to_backup[@]}"; do
            if [ ! -f "$script_path/$f" ]; then
                not_found+=("$f")
            else
                cp "$src_path/tmp/$f" .
                restored+=("$f")
            fi
        done

        if ((${#not_found[@]})); then
            warn "The following files were not found in $script_path: ${not_found[@]}. Skipping."
        fi

        if ((${#restored[@]})); then
            info "Restoring the following files: ${restored[@]}"
        fi
    else
        tar -C "$src_path/tmp" -xf "$src_path/$backup_file" firefly_db.sql firefly_upload.tar.gz version.txt
        #restored=(firefly_db.sql firefly_upload.tar.gz)
    fi

    if [ ! -z $upload_volume ]; then
        warn "The upload volume exists."
	read -p "Do you want to overwrite? (yes/no) " yn

	case $yn in
		yes)
			warn overwriting...
			docker run --rm -v "$upload_volume:/recover" -v "$src_path/tmp:/backup" alpine tar -xf /backup/firefly_upload.tar.gz -C /recover --strip 1
			restored+=(firefly_upload.tar.gz)
			;;
		no )
			info upload volume is not restored;
			rm -f "$src_path/tmp/firefly_upload.tar.gz"
			;;
		*  ) fatal invalid response ;;
	esac
    fi

    db_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?db' | cut -d ' ' -f 1)

    if [ -z $db_container ]; then
        warn "The db container is not running. Not restoring."
    else
        info 'Restoring database'
        cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container bash -c '/usr/bin/mariadb -u $MYSQL_USER --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"'
        restored+=(firefly_db.sql)
    fi

    cp "$src_path/tmp/version.txt" .
    restored+=(version.txt)

    # Clean up
    for file in "${restored[@]}"; do
        rm -f "$src_path/tmp/$file"
    done
    rmdir "$src_path/tmp"
}

main () {
    intro

    if [ $# -lt 2 ]; then
        fatal "Not enough parameters.\n$(usage)"
    fi

    current_dir="$(dirname $0)"
    action=$1
    path="$2"
    if [ -z "$3" ]; then
        no_files=false
    else
        no_files=true
    fi

    if [ -d "$path" ]; then
        fatal "Path is an existing directory. It has to be a file path"
    fi

    upload_volume="$(docker volume ls | grep -F "firefly_iii_upload" | tr -s ' ' | cut -d ' ' -f 2)"

    if [ "$action" == 'backup' ]; then
        backup "$current_dir" "$path" "$upload_volume" "$no_files"
    elif [ "$action" == 'restore' ]; then
        restore "$current_dir" "$path" "$upload_volume" "$no_files"
    else
        fatal "Unrecognized action $action\n$(usage)"
    fi
}

main "$@"

It seems you were right from the beginning about MacOS being the cause for the problem. It seems it has a different implementation for realpath compared to Linux. Linux: Print the resolved absolute file name; all but the last component must exist MacOS: All components of file_name must exist when realpath() is called.

So, we have to create the file before calling realpath, I will update this gist soon.

I'm not a programmer, so I wouldn't know exactly how to be of help, but by trying with ChatGPT to find a solution, it suggested that it might be possible to replace realpath with dest_file. Could that be a way?

@Pindol83 I found realpath did not seem so required. I removed it and provided some other improvements.
Please check the current version of the script and provide feedback.

Testing the new script gives me new errors, but it does create a backup. However, the backup isn't as large as the one I'm currently doing with a cron job. Could this be due to the fact that my database volume is named "firefly_firefly_iii_db," so it sees "firefly_iii_db" as an unused volume in the container?

[INFO]  Backing up the following files in /Users/macmini2018/docker/firefly: .env .db.env docker-compose.yml
grep: empty (sub)expression
Error response from daemon: No such container: grep
Error response from daemon: No such container: grep
[INFO]  Backing up App & database version numbers.
grep: empty (sub)expression
[WARNING]  db container is not running. Not backing up.
[INFO]  Backing up upload volume```

I found the problem for me, it's in line 90. I had to change this:

db_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?db' | cut -d ' ' -f 1)

to this:

db_container=$(docker ps | grep -E 'firefly_iii_db' | cut -d ' ' -f 1)

It seems that this syntax'firefly[-_](iii|)[_-]?db'doesn't work on MacOS, so I preferred to directly input the container name, even though it's not an elegant solution.

Now the backup runs without any problems.