Script to backup Firefly III database, uploads and config files installed with docker-compose

firefly-iii-backuper.sh

#!/bin/bash

files_to_backup=(.env .db.env .fidi.env docker-compose.yml )

info()  { echo -e "\\033[1;36m[INFO]\\033[0m  \\033[36m$*\\033[0m" >&2; }
warn()  { echo -e "\\033[1;33m[WARNING]\\033[0m  \\033[33m$*\\033[0m" >&2; }
fatal() { echo -e "\\033[1;31m[FATAL]\\033[0m  \\033[31m$*\\033[0m" >&2; exit 1; }

intro () {
    echo " ====================================================="
    echo "     Backup & Restore docker based FireFly III v1.4-1 "
    echo " ====================================================="
    echo " It automatically detects db & upload volumes based on the name matching the following regex: firefly[_-](iii|)[_-]?"
    echo " Requirements:"
    echo " - Place the script in the same directory where your docker-compose.yml and .env files are saved"
    echo " Note: The destination directory is created if it does not exist"
}

usage () {
    echo "Usage: $0 backup|restore destination [no_files]"
    echo "- backup|restore : Action you want to execute"
    echo "- destination : The path for your backup file (including the file name)"
    echo "- (optional) no_files : When passed, exclude the docker and environment files. Only backup or restore volumes"
    echo "Example backup: $0 backup /home/backup/firefly-2022-01-01.tar.gz"
    echo "Example restore: $0 restore /home/backup/firefly-2022-01-01.tar.gz"
    echo "To backup once per day you can add something like this to your cron:"
    echo "1 01 * * * bash \"$(realpath $0)\" backup /home/backup/\$(date '+%F').tar.gz"
}

backup () {
    script_path="$1"
    dest_path="$(dirname $path)"
    dest_file="$(basename $path)"
    upload_volume="$3"
    no_files=$4

    to_backup=()

    if [ ! -d "$dest_path" ]; then
        info "Creating destination directory: $dest_path"
        mkdir -p "$dest_path"
    fi

    if [ -f "$path" ]; then
        warn "Provided file path already exists: $path."
	read -p "Do you want to overwrite? (yes/no) " yn

	case $yn in
		yes) warn overwriting... ;;
		no ) info exiting... ; exit;;
		*  ) fatal invalid response ;;
	esac
    fi

    # Create temporary directory
    if [ ! -d "$dest_path/tmp" ]; then
        mkdir "$dest_path/tmp"
    fi

    # Files backup
    if [ $no_files = "false" ]; then
        not_found=()
        for f in "${files_to_backup[@]}"; do
            if [ ! -f "$script_path/$f" ]; then
                not_found+=("$f")
            else
                cp "$script_path/$f" "$dest_path/tmp/"
                to_backup+=("$f")
            fi
        done

        if ((${#not_found[@]})); then
            warn "The following files were not found in $script_path: ${not_found[@]}. Skipping."
        fi

        if ((${#to_backup[@]})); then
            info "Backing up the following files in $script_path: ${to_backup[@]}"
        fi
    fi

    # Version
    app_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?(core|app)' | cut -d ' ' -f 1)
    app_version=$(docker exec -it $app_container grep -F "'version'" /var/www/html/config/firefly.php | tr -s ' ' | cut -d "'" -f 4)
    db_version=$(docker exec -it $app_container grep -F "'db_version'" /var/www/html/config/firefly.php | tr -s ' ' | tr -d ',' | cut -d " " -f 4)
    info 'Backing up App & database version numbers.'
    echo -e "Application: $app_version\nDatabase: $db_version" > "$dest_path/tmp/version.txt"
    to_backup+=(version.txt)

    # DB container
    db_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?db' | cut -d ' ' -f 1)
    if [ -z $db_container ]; then
        warn "db container is not running. Not backing up."
    else
        info 'Backing up database'
        docker exec $db_container bash -c '/usr/bin/mariadb-dump -u $MYSQL_USER --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"' > "$dest_path/tmp/firefly_db.sql"
        to_backup+=("firefly_db.sql")
    fi

    # Upload Volume
    if [ -z $upload_volume ]; then
        warn "upload volume does NOT exist. Not backing up."
    else
        info 'Backing up upload volume'
        docker run --rm -v "$upload_volume:/tmp" -v "$dest_path/tmp:/backup" alpine tar -czf "/backup/firefly_upload.tar.gz" -C "/" "tmp"
        to_backup+=("firefly_upload.tar.gz")
    fi

    # Compress
    tar -C "$dest_path/tmp" -czf "$dest_path/$dest_file" --files-from <(printf "%s\n" "${to_backup[@]}")

    # Clean up
    for file in "${to_backup[@]}"; do
        rm -f "$dest_path/tmp/$file"
    done
    rmdir "$dest_path/tmp"
}

restore () {
    script_path="$1"
    src_path="$(dirname $path)"
    backup_file="$(basename $path)"
    upload_volume="$3"
    no_files=$4

    if [ ! -f "$path" ]; then
        fatal "Provided backup file does not exist: $path"
    fi

    # Create temporary directory
    if [ ! -d "$src_path/tmp" ]; then
        mkdir "$src_path/tmp"
    fi

    # Files restore
    if [ $no_files = "false" ]; then
        tar -C "$src_path/tmp" -xf "$src_path/$backup_file"
        #readarray -t <<<$(tar -tf "$src_path/$backup_file")
        # restored=(${MAPFILE[*]})
        not_found=()
        restored=()
        for f in "${files_to_backup[@]}"; do
            if [ ! -f "$script_path/$f" ]; then
                not_found+=("$f")
            else
                cp "$src_path/tmp/$f" .
                restored+=("$f")
            fi
        done

        if ((${#not_found[@]})); then
            warn "The following files were not found in $script_path: ${not_found[@]}. Skipping."
        fi

        if ((${#restored[@]})); then
            info "Restoring the following files: ${restored[@]}"
        fi
    else
        tar -C "$src_path/tmp" -xf "$src_path/$backup_file" firefly_db.sql firefly_upload.tar.gz version.txt
        #restored=(firefly_db.sql firefly_upload.tar.gz)
    fi

    if [ ! -z $upload_volume ]; then
        warn "The upload volume exists."
	read -p "Do you want to overwrite? (yes/no) " yn

	case $yn in
		yes)
			warn overwriting...
			docker run --rm -v "$upload_volume:/recover" -v "$src_path/tmp:/backup" alpine tar -xf /backup/firefly_upload.tar.gz -C /recover --strip 1
			restored+=(firefly_upload.tar.gz)
			;;
		no )
			info upload volume is not restored;
			rm -f "$src_path/tmp/firefly_upload.tar.gz"
			;;
		*  ) fatal invalid response ;;
	esac
    fi

    db_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?db' | cut -d ' ' -f 1)

    if [ -z $db_container ]; then
        warn "The db container is not running. Not restoring."
    else
        info 'Restoring database'
        cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container bash -c '/usr/bin/mariadb -u $MYSQL_USER --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"'
        restored+=(firefly_db.sql)
    fi

    cp "$src_path/tmp/version.txt" .
    restored+=(version.txt)

    # Clean up
    for file in "${restored[@]}"; do
        rm -f "$src_path/tmp/$file"
    done
    rmdir "$src_path/tmp"
}

main () {
    intro

    if [ $# -lt 2 ]; then
        fatal "Not enough parameters.\n$(usage)"
    fi

    current_dir="$(dirname $0)"
    action=$1
    path="$2"
    if [ -z "$3" ]; then
        no_files=false
    else
        no_files=true
    fi

    if [ -d "$path" ]; then
        fatal "Path is an existing directory. It has to be a file path"
    fi

    upload_volume="$(docker volume ls | grep -F "firefly_iii_upload" | tr -s ' ' | cut -d ' ' -f 2)"

    if [ "$action" == 'backup' ]; then
        backup "$current_dir" "$path" "$upload_volume" "$no_files"
    elif [ "$action" == 'restore' ]; then
        restore "$current_dir" "$path" "$upload_volume" "$no_files"
    else
        fatal "Unrecognized action $action\n$(usage)"
    fi
}

main "$@"

could it be that the "_" symbol in the folder name "backup_firefly" is the culprit?

I tried redirecting the script to a folder without spaces or strange symbols, but it still gives me the same error.

It seems you were right from the beginning about MacOS being the cause for the problem. It seems it has a different implementation for realpath compared to Linux.
Linux: Print the resolved absolute file name; all but the last component must exist
MacOS: All components of file_name must exist when realpath() is called.

So, we have to create the file before calling realpath, I will update this gist soon.

It seems you were right from the beginning about MacOS being the cause for the problem. It seems it has a different implementation for realpath compared to Linux. Linux: Print the resolved absolute file name; all but the last component must exist MacOS: All components of file_name must exist when realpath() is called.

So, we have to create the file before calling realpath, I will update this gist soon.

I'm not a programmer, so I wouldn't know exactly how to be of help, but by trying with ChatGPT to find a solution, it suggested that it might be possible to replace realpath with dest_file. Could that be a way?

@Pindol83 I found realpath did not seem so required. I removed it and provided some other improvements.
Please check the current version of the script and provide feedback.

Testing the new script gives me new errors, but it does create a backup. However, the backup isn't as large as the one I'm currently doing with a cron job. Could this be due to the fact that my database volume is named "firefly_firefly_iii_db," so it sees "firefly_iii_db" as an unused volume in the container?

[INFO]  Backing up the following files in /Users/macmini2018/docker/firefly: .env .db.env docker-compose.yml
grep: empty (sub)expression
Error response from daemon: No such container: grep
Error response from daemon: No such container: grep
[INFO]  Backing up App & database version numbers.
grep: empty (sub)expression
[WARNING]  db container is not running. Not backing up.
[INFO]  Backing up upload volume```

I found the problem for me, it's in line 90. I had to change this:

db_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?db' | cut -d ' ' -f 1)

to this:

db_container=$(docker ps | grep -E 'firefly_iii_db' | cut -d ' ' -f 1)

It seems that this syntax'firefly[-_](iii|)[_-]?db'doesn't work on MacOS, so I preferred to directly input the container name, even though it's not an elegant solution.

Now the backup runs without any problems.