Script to backup Firefly III database, uploads and config files installed with docker-compose

firefly-iii-backuper.sh

#!/bin/bash

files_to_backup=(.env .db.env .fidi.env docker-compose.yml )

info()  { echo -e "\\033[1;36m[INFO]\\033[0m  \\033[36m$*\\033[0m" >&2; }
warn()  { echo -e "\\033[1;33m[WARNING]\\033[0m  \\033[33m$*\\033[0m" >&2; }
fatal() { echo -e "\\033[1;31m[FATAL]\\033[0m  \\033[31m$*\\033[0m" >&2; exit 1; }

intro () {
    echo " ====================================================="
    echo "     Backup & Restore docker based FireFly III v1.4-1 "
    echo " ====================================================="
    echo " It automatically detects db & upload volumes based on the name matching the following regex: firefly[_-](iii|)[_-]?"
    echo " Requirements:"
    echo " - Place the script in the same directory where your docker-compose.yml and .env files are saved"
    echo " Note: The destination directory is created if it does not exist"
}

usage () {
    echo "Usage: $0 backup|restore destination [no_files]"
    echo "- backup|restore : Action you want to execute"
    echo "- destination : The path for your backup file (including the file name)"
    echo "- (optional) no_files : When passed, exclude the docker and environment files. Only backup or restore volumes"
    echo "Example backup: $0 backup /home/backup/firefly-2022-01-01.tar.gz"
    echo "Example restore: $0 restore /home/backup/firefly-2022-01-01.tar.gz"
    echo "To backup once per day you can add something like this to your cron:"
    echo "1 01 * * * bash \"$(realpath $0)\" backup /home/backup/\$(date '+%F').tar.gz"
}

backup () {
    script_path="$1"
    dest_path="$(dirname $path)"
    dest_file="$(basename $path)"
    upload_volume="$3"
    no_files=$4

    to_backup=()

    if [ ! -d "$dest_path" ]; then
        info "Creating destination directory: $dest_path"
        mkdir -p "$dest_path"
    fi

    if [ -f "$path" ]; then
        warn "Provided file path already exists: $path."
	read -p "Do you want to overwrite? (yes/no) " yn

	case $yn in
		yes) warn overwriting... ;;
		no ) info exiting... ; exit;;
		*  ) fatal invalid response ;;
	esac
    fi

    # Create temporary directory
    if [ ! -d "$dest_path/tmp" ]; then
        mkdir "$dest_path/tmp"
    fi

    # Files backup
    if [ $no_files = "false" ]; then
        not_found=()
        for f in "${files_to_backup[@]}"; do
            if [ ! -f "$script_path/$f" ]; then
                not_found+=("$f")
            else
                cp "$script_path/$f" "$dest_path/tmp/"
                to_backup+=("$f")
            fi
        done

        if ((${#not_found[@]})); then
            warn "The following files were not found in $script_path: ${not_found[@]}. Skipping."
        fi

        if ((${#to_backup[@]})); then
            info "Backing up the following files in $script_path: ${to_backup[@]}"
        fi
    fi

    # Version
    app_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?(core|app)' | cut -d ' ' -f 1)
    app_version=$(docker exec -it $app_container grep -F "'version'" /var/www/html/config/firefly.php | tr -s ' ' | cut -d "'" -f 4)
    db_version=$(docker exec -it $app_container grep -F "'db_version'" /var/www/html/config/firefly.php | tr -s ' ' | tr -d ',' | cut -d " " -f 4)
    info 'Backing up App & database version numbers.'
    echo -e "Application: $app_version\nDatabase: $db_version" > "$dest_path/tmp/version.txt"
    to_backup+=(version.txt)

    # DB container
    db_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?db' | cut -d ' ' -f 1)
    if [ -z $db_container ]; then
        warn "db container is not running. Not backing up."
    else
        info 'Backing up database'
        docker exec $db_container bash -c '/usr/bin/mariadb-dump -u $MYSQL_USER --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"' > "$dest_path/tmp/firefly_db.sql"
        to_backup+=("firefly_db.sql")
    fi

    # Upload Volume
    if [ -z $upload_volume ]; then
        warn "upload volume does NOT exist. Not backing up."
    else
        info 'Backing up upload volume'
        docker run --rm -v "$upload_volume:/tmp" -v "$dest_path/tmp:/backup" alpine tar -czf "/backup/firefly_upload.tar.gz" -C "/" "tmp"
        to_backup+=("firefly_upload.tar.gz")
    fi

    # Compress
    tar -C "$dest_path/tmp" -czf "$dest_path/$dest_file" --files-from <(printf "%s\n" "${to_backup[@]}")

    # Clean up
    for file in "${to_backup[@]}"; do
        rm -f "$dest_path/tmp/$file"
    done
    rmdir "$dest_path/tmp"
}

restore () {
    script_path="$1"
    src_path="$(dirname $path)"
    backup_file="$(basename $path)"
    upload_volume="$3"
    no_files=$4

    if [ ! -f "$path" ]; then
        fatal "Provided backup file does not exist: $path"
    fi

    # Create temporary directory
    if [ ! -d "$src_path/tmp" ]; then
        mkdir "$src_path/tmp"
    fi

    # Files restore
    if [ $no_files = "false" ]; then
        tar -C "$src_path/tmp" -xf "$src_path/$backup_file"
        #readarray -t <<<$(tar -tf "$src_path/$backup_file")
        # restored=(${MAPFILE[*]})
        not_found=()
        restored=()
        for f in "${files_to_backup[@]}"; do
            if [ ! -f "$script_path/$f" ]; then
                not_found+=("$f")
            else
                cp "$src_path/tmp/$f" .
                restored+=("$f")
            fi
        done

        if ((${#not_found[@]})); then
            warn "The following files were not found in $script_path: ${not_found[@]}. Skipping."
        fi

        if ((${#restored[@]})); then
            info "Restoring the following files: ${restored[@]}"
        fi
    else
        tar -C "$src_path/tmp" -xf "$src_path/$backup_file" firefly_db.sql firefly_upload.tar.gz version.txt
        #restored=(firefly_db.sql firefly_upload.tar.gz)
    fi

    if [ ! -z $upload_volume ]; then
        warn "The upload volume exists."
	read -p "Do you want to overwrite? (yes/no) " yn

	case $yn in
		yes)
			warn overwriting...
			docker run --rm -v "$upload_volume:/recover" -v "$src_path/tmp:/backup" alpine tar -xf /backup/firefly_upload.tar.gz -C /recover --strip 1
			restored+=(firefly_upload.tar.gz)
			;;
		no )
			info upload volume is not restored;
			rm -f "$src_path/tmp/firefly_upload.tar.gz"
			;;
		*  ) fatal invalid response ;;
	esac
    fi

    db_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?db' | cut -d ' ' -f 1)

    if [ -z $db_container ]; then
        warn "The db container is not running. Not restoring."
    else
        info 'Restoring database'
        cat "$src_path/tmp/firefly_db.sql" | docker exec -i $db_container bash -c '/usr/bin/mariadb -u $MYSQL_USER --password="$MYSQL_PASSWORD" "$MYSQL_DATABASE"'
        restored+=(firefly_db.sql)
    fi

    cp "$src_path/tmp/version.txt" .
    restored+=(version.txt)

    # Clean up
    for file in "${restored[@]}"; do
        rm -f "$src_path/tmp/$file"
    done
    rmdir "$src_path/tmp"
}

main () {
    intro

    if [ $# -lt 2 ]; then
        fatal "Not enough parameters.\n$(usage)"
    fi

    current_dir="$(dirname $0)"
    action=$1
    path="$2"
    if [ -z "$3" ]; then
        no_files=false
    else
        no_files=true
    fi

    if [ -d "$path" ]; then
        fatal "Path is an existing directory. It has to be a file path"
    fi

    upload_volume="$(docker volume ls | grep -F "firefly_iii_upload" | tr -s ' ' | cut -d ' ' -f 2)"

    if [ "$action" == 'backup' ]; then
        backup "$current_dir" "$path" "$upload_volume" "$no_files"
    elif [ "$action" == 'restore' ]; then
        restore "$current_dir" "$path" "$upload_volume" "$no_files"
    else
        fatal "Unrecognized action $action\n$(usage)"
    fi
}

main "$@"

I want to mention that I'm using MacOs

I do not have experience with that, but I do not think it is the problem.

I've correctly created the directory /mypath/backup_firefly/script

Just to make sure, this is created on the host (MacOs) not in a container.

=====================================================

Backup & Restore docker based FireFly III v1.2-3

It automatically detects db & upload volumes based on the name firefly[_-]iii Requirements:

* Place the script in the same directory where your docker-compose.yml and .env files are saved
  Warning: The destination directory is created if it does not exist
  realpath: /mypath/backup_firefly/script/2023-10-09.tar: No such file or directory

realpath has failed for some reason. the problems indicate there maybe a problem with the path given to the script. Please note that if the path contains white spaces, or special characters, it has to be quoted or properly escaped.
Please provide:

• The full command used (that caused the error)
• proof of path already existing and accessible on the host MacOs (screenshot or paste the output of cd into the directory, or use ls <directory_path_here>)
• realpath is used in lines 38 and 115, try to echo $2 just before each line, and exit after the line, then screenshout or paste the output of the script here

I want to mention that I'm using MacOs

I do not have experience with that, but I do not think it is the problem.

I've correctly created the directory /mypath/backup_firefly/script

Just to make sure, this is created on the host (MacOs) not in a container.

Yes the folder are in the host computer

pwd
/Users/macmini2018/backup_firefly/script

=====================================================

Backup & Restore docker based FireFly III v1.2-3
It automatically detects db & upload volumes based on the name firefly[_-]iii Requirements:

* Place the script in the same directory where your docker-compose.yml and .env files are saved
  Warning: The destination directory is created if it does not exist
  realpath: /mypath/backup_firefly/script/2023-10-09.tar: No such file or directory

realpath has failed for some reason. the problems indicate there maybe a problem with the path given to the script. Please note that if the path contains white spaces, or special characters, it has to be quoted or properly escaped. Please provide:

• The full command used (that caused the error)

This is the command

bash /Users/macmini2018/docker/firefly/firefly-iii-backuper.sh backup /Users/macmini2018/backup_firefly/script/$(date '+%F').tar

• proof of path already existing and accessible on the host MacOs (screenshot or paste the output of cd into the directory, or use ls <directory_path_here>)

pwd
/Users/macmini2018/backup_firefly/script

• realpath is used in lines 38 and 115, try to echo $2 just before each line, and exit after the line, then screenshout or paste the output of the script here

i have change the line like this:

38 - echo $2
full_path=$(realpath $2)
exit
115 - echo $2
full_path=$(realpath $2)
exit

It keeps giving me this error:

 =====================================================
     Backup & Restore docker based FireFly III v1.2-3 
 =====================================================
 It automatically detects db & upload volumes based on the name firefly[_-]iii
 Requirements:
 - Place the script in the same directory where your docker-compose.yml and .env files are saved
 Warning: The destination directory is created if it does not exist
/Users/macmini2018/provascript/2023-10-10.tar
realpath: /Users/macmini2018/provascript/2023-10-10.tar: No such file or directory

could it be that the "_" symbol in the folder name "backup_firefly" is the culprit?

could it be that the "_" symbol in the folder name "backup_firefly" is the culprit?

I tried redirecting the script to a folder without spaces or strange symbols, but it still gives me the same error.

It seems you were right from the beginning about MacOS being the cause for the problem. It seems it has a different implementation for realpath compared to Linux.
Linux: Print the resolved absolute file name; all but the last component must exist
MacOS: All components of file_name must exist when realpath() is called.

So, we have to create the file before calling realpath, I will update this gist soon.

It seems you were right from the beginning about MacOS being the cause for the problem. It seems it has a different implementation for realpath compared to Linux. Linux: Print the resolved absolute file name; all but the last component must exist MacOS: All components of file_name must exist when realpath() is called.

So, we have to create the file before calling realpath, I will update this gist soon.

I'm not a programmer, so I wouldn't know exactly how to be of help, but by trying with ChatGPT to find a solution, it suggested that it might be possible to replace realpath with dest_file. Could that be a way?

@Pindol83 I found realpath did not seem so required. I removed it and provided some other improvements.
Please check the current version of the script and provide feedback.

Testing the new script gives me new errors, but it does create a backup. However, the backup isn't as large as the one I'm currently doing with a cron job. Could this be due to the fact that my database volume is named "firefly_firefly_iii_db," so it sees "firefly_iii_db" as an unused volume in the container?

[INFO]  Backing up the following files in /Users/macmini2018/docker/firefly: .env .db.env docker-compose.yml
grep: empty (sub)expression
Error response from daemon: No such container: grep
Error response from daemon: No such container: grep
[INFO]  Backing up App & database version numbers.
grep: empty (sub)expression
[WARNING]  db container is not running. Not backing up.
[INFO]  Backing up upload volume```

I found the problem for me, it's in line 90. I had to change this:

db_container=$(docker ps | grep -E 'firefly[-_](iii|)[_-]?db' | cut -d ' ' -f 1)

to this:

db_container=$(docker ps | grep -E 'firefly_iii_db' | cut -d ' ' -f 1)

It seems that this syntax'firefly[-_](iii|)[_-]?db'doesn't work on MacOS, so I preferred to directly input the container name, even though it's not an elegant solution.

Now the backup runs without any problems.