foremanctl Offline Backup Testing Guide

foremanctl-backup-testing.md

Testing foremanctl Offline Backup

Complete steps to test the offline backup feature for foremanctl in a QE/production environment.

Prerequisites

• Deployed foremanctl installation with Foreman/Katello
• Root or sudo access
• At least 50MB free disk space for backups

Environment Setup

cd /root/foremanctl

# Install dependencies and build Ansible collections
./setup-environment

# Activate Python virtualenv
source .venv/bin/activate

Create QE Inventory (One-time Setup)

The backup playbook expects a host named "quadlet" (from the development environment). For QE/production testing, create a temporary inventory file:

cat > inventories/qe <<'EOF'
[quadlet]
localhost ansible_connection=local
EOF

Note: This file is temporary and should NOT be committed to the repository.

Run Backup

# Use the system-wide foremanctl state directory
export OBSAH_STATE=/var/lib/foremanctl

# Run backup to /var/tmp (or any directory with sufficient space)
./foremanctl backup /var/tmp/foreman-backup-test --wait-for-tasks

Command Options

• --wait-for-tasks - Wait for running Foreman/Pulp tasks to complete (recommended)
• Without --wait-for-tasks - Fail immediately if tasks are running

Expected Behavior

1. Preflight checks - Verify no running tasks, check database health
2. Stop services - foreman.target stopped (includes all Foreman services + PostgreSQL)
3. Restart PostgreSQL - For database dumps
4. Dump databases - All detected databases (foreman, candlepin, pulp, + 5 IOP databases if enabled)
5. Generate metadata - Container image digests, system info
6. Restart services - All services brought back online

Expected downtime: 2-5 minutes depending on database sizes

Verify Results

List Backup Files

# Find the timestamped backup directory
ls -lh /var/tmp/foreman-backup-test/

# Example output:
# foreman-backup-20260511T150849/

Check Backup Contents

# Replace timestamp with your actual backup directory
BACKUP_DIR=/var/tmp/foreman-backup-test/foreman-backup-20260511T150849

# List all files
ls -lh $BACKUP_DIR/

# Expected files:
# - foreman.dump
# - candlepin.dump
# - pulp.dump
# - iop_advisor.dump (if IOP enabled)
# - iop_inventory.dump (if IOP enabled)
# - iop_remediations.dump (if IOP enabled)
# - iop_vmaas.dump (if IOP enabled)
# - iop_vulnerability.dump (if IOP enabled)
# - metadata.yml

Verify Dump File Integrity

# Verify all dumps are valid PostgreSQL backups
file $BACKUP_DIR/*.dump

# Expected output for each:
# PostgreSQL custom database dump - v1.14-0

Check Metadata

cat $BACKUP_DIR/metadata.yml

# Contains:
# - hostname
# - OS version
# - foremanctl version
# - enabled features
# - database names
# - container image digests (for exact restore compatibility)

Verify Database Sizes

# Check total backup size
du -sh $BACKUP_DIR/

# Typical sizes (varies by usage):
# - Small deployment: 2-5 MB
# - With IOP: 15-30 MB
# - Production: 50+ MB

Test Database Dump Validity (Optional)

Verify a dump can be listed (doesn't test full restore, just file validity):

# List objects in a dump file (no restore needed)
pg_restore --list $BACKUP_DIR/foreman.dump | head -20

Cleanup Test Backups

# Remove test backups after verification
rm -rf /var/tmp/foreman-backup-test/

Troubleshooting

"Could not match supplied host pattern: quadlet"

Solution: Create the inventories/qe file as shown above.

"database_mode is undefined"

Solution: Run with OBSAH_STATE=/var/lib/foremanctl to use system deployment parameters.

Services Not Restarting

If services fail to restart after backup:

# Manually restart
systemctl start foreman.target

# Check status
systemctl status foreman.target

IOP Databases Missing from Backup

Expected if IOP feature is not enabled in the deployment. Check:

cat /var/lib/foremanctl/parameters.yaml | grep -i iop

Known Limitations

• No restore functionality - Only backup is implemented; restore must be done manually
• Offline only - Services are stopped during backup (online backup not yet implemented)
• No incremental backups - Full backup only
• No compression of archive - Individual dumps are compressed (PostgreSQL custom format), but not bundled into a tar

Success Criteria

✅ Backup completes without errors
✅ All expected database dumps created
✅ All dump files are valid PostgreSQL custom format
✅ metadata.yml contains system info and container digests
✅ Services automatically restart after backup
✅ Total backup time < 10 minutes

Example Successful Output

TASK [Display backup summary] **************************************************
ok: [localhost] => 
    msg: |-
        Database dumps completed:
        - Total files: 8
        - Total size: 20.46 MB
        - Location: /var/tmp/foreman-backup-test/foreman-backup-20260511T150849

TASK [Display backup completion] ***********************************************
ok: [localhost] => 
    msg: |-
        Backup completed successfully.
        Location: /var/tmp/foreman-backup-test/foreman-backup-20260511T150849
        Databases: foreman, candlepin, pulp, advisor_db, inventory_db, remediations_db, vmaas_db, vulnerability_db

PLAY RECAP *********************************************************************
localhost                  : ok=88   changed=14   unreachable=0    failed=0    skipped=6    rescued=0    ignored=0