Each jail is in its own fileset. Snapshot each fileset, back it up. Destroy the filesets.

1 - The FileSet

# the filesets will be different on each jail, thus, we'll always be doing a full unless 
# we define this for each jail server
#
FileSet {
  Name = "zuul jail snapshots"
  Include {
    Options {
      signature = MD5 
      Exclude = yes
    }

    Exclude Dir Containing = .NOBACKUP

    File = "\\|/usr/home/dan/bin/jail-snapshots-for-backup.sh list"
  }
}

2 - The backup job

Job {
  Name            = "zuul jail snapshots"
  JobDefs         = "DefaultJob"
  Client          = zuul-fd
  FileSet         = "zuul jail snapshots"
  Write Bootstrap = "/usr/local/bacula/bsr/zuul-fd-jail-snapshots.bsr"

  RunScript {
    RunsWhen       = Before
    FailJobOnError = Yes
    Command        = "/usr/home/dan/bin/jail-snapshots-for-backup.sh create"
  }

  RunScript {
    RunsWhen       = After
    FailJobOnError = No
    Command        = "/usr/home/dan/bin/jail-snapshots-for-backup.sh destroy"
  }
}

3 - jail-snapshots-for-backup.sh

#!/bin/sh

ACTION=$1

JLS=/usr/sbin/jls

JAILS=`${JLS} -h path`

SNAPSHOTDIRECTORY='/.zfs/snapshot/'

SNAPNAME="snapshot-for-backup"

for jail in ${JAILS}
do
  # ignore the one value of path, usually the first, because that's the header line.
  if [ ${jail} = 'path' ]
  then
     continue
  fi
  
  DATASETNAME=`/sbin/zfs get -H -o name mountpoint ${jail}`
  
  # the snapshot name is of the form: system/usr/local/jails/fedex@snapshot-for-backup
  SNAPSHOTFORBACKUP="${DATASETNAME}@${SNAPNAME}"
  
  # the backup dir is of the form: /usr/local/jails/fedex/.zfs/snapshot/snapshot-for-backup
  BACKUPDIR="${jail}${SNAPSHOTDIRECTORY}${SNAPNAME}"

  case ${ACTION} in
    "create")
       zfs snapshot ${SNAPSHOTFORBACKUP}
       ;;

    "list")
       # echo back out the directory for backup...
       echo ${BACKUPDIR}
       ;;

    "destroy")
       zfs destroy ${SNAPSHOTFORBACKUP}
       ;;
  esac    
done

4 - jail-snapshots-for-backup.sh

#!/bin/sh

# this script caters for multiple jail locations

ACTION=$1

SNAPSHOTDIRECTORY='/.zfs/snapshot/'

SNAPNAME="snapshot-for-backup"

ZFSJAILROOTS="system/iocage/jails system/jails"

for ZFSJAILROOT in $ZFSJAILROOTS
do

  DATASETS="/sbin/zfs get -r -H -o name -t filesystem name ${ZFSJAILROOT}"

  for dataset in `${DATASETS}`
  do
    # Ignore the jail root.  Nothing in there to backup.
    if [ "${dataset}" = "${ZFSJAILROOT}" ]
    then
       continue
    fi

    MOUNTPOINT=`/sbin/zfs get -H -o value mountpoint ${dataset}`
  
    # the snapshot name is of the form: main_tank/iocage/jails/fedex@snapshot-for-backup
    SNAPSHOTFORBACKUP="${dataset}@${SNAPNAME}"

    # the backup dir is of the form: /usr/local/jails/fedex/.zfs/snapshot/snapshot-for-backup
    BACKUPDIR="${MOUNTPOINT}${SNAPSHOTDIRECTORY}${SNAPNAME}"

    case ${ACTION} in
      "create")
         zfs snapshot ${SNAPSHOTFORBACKUP}
         ;;

      "list")
         # echo back out the directory for backup...
         echo ${BACKUPDIR}
         ;;

      "destroy")
         zfs destroy ${SNAPSHOTFORBACKUP}
         ;;
    esac    
  done

done

A recursive snapshot might be better:
zfs snapshot -r system/usr/local/jails@${SNAPNAME}

as it would be faster than creating the snapshots individually, and avoid flushing all of the extra transaction groups, but then you'd have to hard code the jail root dataset somewhere.

Thanks for sharing Dan

I know we talked about this via IRC, and I think we concluded recursive had a down-side, but I cannot recall what that was. I'll check logs when I'm on my other laptop.

I know now why 'zfs snapshot -r' doesn't help much. Well, it help for creating the snapshots, but you still (for now) have to remove them one at a time. You also have to iterate through the datasets (zfs list -r) to find the mountpoints.

I realized this as I was rewriting the script to work with iocage (https://github.com/pannon/iocage).

It's been a while, zfs destroy -r has worked for a while now :-) I'd like to adapt this for backing up my pools as well, bacula gets upset if things vanish from under it in a multi-day backup

Today I amended the script to cater for multiple jail locations.

Both the original and the new script are listed above.