foremanctl Restore Implementation Guide

Executive Summary

This guide provides a complete step-by-step process to restore a Foreman/Katello deployment from a foremanctl backup. The backup includes all critical components needed for a full restoration.

Estimated Restore Time: 30-60 minutes (depending on volume data size) Complexity Level: MEDIUM Risk Level: Requires service downtime

Prerequisites

Fresh RHEL 9.7 installation
Podman 4.6+ installed
systemd available
Network connectivity to container registries
Backup directory accessible and complete
Root or sudo access

Backup Components Included

This backup format includes:

8 Database dumps (PostgreSQL custom format)
- foreman.dump
- candlepin.dump
- pulp.dump
- iop_advisor.dump
- iop_inventory.dump
- iop_remediations.dump
- iop_vmaas.dump
- iop_vulnerability.dump
Container Infrastructure
- quadlet-files.tar.gz (47+ container definitions)
- systemd-units.tar.gz (systemd targets and timers)
- podman-networks.json (network configurations)
Configuration & Secrets
- config_files.tar.gz (72 podman secrets)
- foremanctl-state.tar.gz (deployment state and credentials)
Container Images (metadata only, must be pulled)
Volume Data (optional, for IOP services)
- volume-iop-core-kafka-data.tar
- volume-iop-service-vmaas-data.tar

Restore Phases

Phase 1: Infrastructure Preparation

1.1 Install Base Packages

# Update system
dnf update -y

# Install required packages
dnf install -y podman podman-docker git jq postgresql

# Verify podman version
podman --version
# Should be 4.6.0 or higher

# Enable and start podman
systemctl enable --now podman.socket

1.2 Create System Users and Directories

# Create foremanctl directories
mkdir -p /var/lib/foremanctl
mkdir -p /etc/containers/systemd
mkdir -p /usr/share/foremanctl

# Set permissions
chmod 755 /var/lib/foremanctl
chmod 755 /etc/containers/systemd

1.3 Set SELinux Contexts (if SELinux is enforcing)

# Check SELinux status
getenforce

# If enforcing, set proper contexts
semanage fcontext -a -t container_file_t "/etc/containers/systemd(/.*)?"
restorecon -R /etc/containers/systemd

1.4 Configure Firewall

# Enable firewall services for Foreman
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --permanent --add-port=5647/tcp  # Foreman Proxy
firewall-cmd --reload

Phase 2: Restore System Configuration

2.1 Set Backup Directory Variable

# Set this to your backup location
export BACKUP_DIR=/path/to/foreman-backup-TIMESTAMP

# Verify backup exists
ls -lh "$BACKUP_DIR"

2.2 Restore Quadlet Container Definitions

# Extract quadlet files to /etc/containers/systemd
cd /etc/containers
tar -xzf "$BACKUP_DIR/quadlet-files.tar.gz"

# Verify extraction
find /etc/containers/systemd -name "*.container" | wc -l
# Should show 45+ files

# List some containers
ls -la /etc/containers/systemd/*.container | head -10

2.3 Restore Systemd Units

# Extract systemd units to /etc/systemd/system
cd /etc/systemd
tar -xzf "$BACKUP_DIR/systemd-units.tar.gz"

# Move to correct location
mv system/* /etc/systemd/system/ 2>/dev/null || true
rmdir system 2>/dev/null || true

# Reload systemd
systemctl daemon-reload

# Verify units exist
systemctl list-unit-files | grep foreman

2.4 Restore Podman Networks

# Read network configurations from backup
jq -r '.[] | .name' "$BACKUP_DIR/podman-networks.json"

# Create networks (example for iop-core-network)
# Extract network details and recreate
NETWORK_NAME=$(jq -r '.[0].name' "$BACKUP_DIR/podman-networks.json")
NETWORK_SUBNET=$(jq -r '.[0].subnets[0].subnet' "$BACKUP_DIR/podman-networks.json")
NETWORK_GATEWAY=$(jq -r '.[0].subnets[0].gateway' "$BACKUP_DIR/podman-networks.json")

podman network create \
  --subnet="$NETWORK_SUBNET" \
  --gateway="$NETWORK_GATEWAY" \
  "$NETWORK_NAME"

# Verify network creation
podman network ls
podman network inspect "$NETWORK_NAME"

2.5 Restore Foremanctl State

# Extract foremanctl state
cd /var/lib
tar -xzf "$BACKUP_DIR/foremanctl-state.tar.gz"

# Verify restoration
ls -la /var/lib/foremanctl/
cat /var/lib/foremanctl/parameters.yaml

# The state includes:
# - parameters.yaml (deployment parameters)
# - foreman-admin-init-passwd (admin password)
# - foreman-oauth-consumer-key
# - foreman-oauth-consumer-secret
# - .installed (installation marker)

Phase 3: Restore Podman Secrets

3.1 Extract Secrets Archive

# Create temporary directory
mkdir -p /tmp/restore-secrets
cd /tmp/restore-secrets

# Extract config_files.tar.gz
tar -xzf "$BACKUP_DIR/config_files.tar.gz"

# Verify secrets.json exists
ls -lh config_files/secrets.json
jq 'length' config_files/secrets.json
# Should show 72 secrets

3.2 Recreate Podman Secrets

# Script to recreate all secrets from the JSON file
cd /tmp/restore-secrets/config_files

jq -r 'to_entries[] | "\(.key)\t\(.value)"' secrets.json | while IFS=$'\t' read -r name content; do
  echo "Creating secret: $name"
  echo "$content" | podman secret create "$name" -
done

# Verify secret count
podman secret ls | wc -l
# Should show 73 (including header line)

# Verify specific secrets exist
podman secret ls | grep -E '(postgresql-admin-password|foreman-admin-password|candlepin-db-password)'

3.3 Clean Up Temporary Files

rm -rf /tmp/restore-secrets

Phase 4: Restore Container Images

4.1 Review Required Images

# List images from metadata
cat "$BACKUP_DIR/metadata.yml" | grep -A 200 "container_images:" | grep "name:"

# The backup contains image metadata but not the images themselves
# Images must be pulled from the registry

4.2 Pull Container Images

# Pull all required images (this may take 15-30 minutes)
# Images are pulled automatically when containers start, but you can pre-pull:

podman pull quay.io/sclorg/postgresql-13-c9s:latest
podman pull quay.io/sclorg/redis-6-c9s:latest
podman pull quay.io/foreman/candlepin:latest
podman pull quay.io/foreman/foreman:latest
podman pull quay.io/pulp/pulp-minimal:latest
# ... etc

# Or let quadlet handle this automatically during container startup

Phase 5: Restore Volume Data (Optional)

This step is only needed if you backed up volume data and need to restore it.

5.1 Import Volume Archives

# Check if volume backups exist
ls -lh "$BACKUP_DIR"/volume-*.tar

# Import kafka data volume
if [ -f "$BACKUP_DIR/volume-iop-core-kafka-data.tar" ]; then
  podman volume import iop-core-kafka-data "$BACKUP_DIR/volume-iop-core-kafka-data.tar"
fi

# Import vmaas data volume
if [ -f "$BACKUP_DIR/volume-iop-service-vmaas-data.tar" ]; then
  podman volume import iop-service-vmaas-data "$BACKUP_DIR/volume-iop-service-vmaas-data.tar"
fi

# Verify volumes
podman volume ls

Phase 6: Restore Databases

6.1 Start PostgreSQL Container Only

# Start only PostgreSQL service
systemctl start postgresql.service

# Wait for PostgreSQL to be ready
for i in {1..30}; do
  pg_isready -h localhost -p 5432 && break
  echo "Waiting for PostgreSQL..."
  sleep 2
done

6.2 Get PostgreSQL Admin Password

# Extract admin password from secret
POSTGRES_PASSWORD=$(podman secret inspect postgresql-admin-password --format '{{ .SecretData }}' | base64 -d)
echo "PostgreSQL admin password retrieved"

6.3 Create Empty Databases

# Create databases (run as postgres user via container)
for db in foreman candlepin pulp advisor_db inventory_db remediations_db vmaas_db vulnerability_db; do
  echo "Creating database: $db"
  podman exec -e PGPASSWORD="$POSTGRES_PASSWORD" postgresql \
    psql -U postgres -c "CREATE DATABASE $db OWNER postgres;"
done

# Verify databases exist
podman exec -e PGPASSWORD="$POSTGRES_PASSWORD" postgresql \
  psql -U postgres -l

6.4 Restore Database Dumps

# Restore each database using pg_restore
for dump in "$BACKUP_DIR"/*.dump; do
  dbname=$(basename "$dump" .dump)
  echo "Restoring database: $dbname"
  
  podman exec -i -e PGPASSWORD="$POSTGRES_PASSWORD" postgresql \
    pg_restore -U postgres -d "$dbname" -v --no-owner --no-acl < "$dump"
done

# Note: Some errors about existing objects are normal if the database
# was already created. The important thing is that data is restored.

6.5 Verify Database Contents

# Check table counts in foreman database
podman exec -e PGPASSWORD="$POSTGRES_PASSWORD" postgresql \
  psql -U postgres -d foreman -c "\dt" | wc -l

# Check specific tables exist
podman exec -e PGPASSWORD="$POSTGRES_PASSWORD" postgresql \
  psql -U postgres -d foreman -c "SELECT COUNT(*) FROM hosts;"

# Verify all databases
for db in foreman candlepin pulp advisor_db inventory_db remediations_db vmaas_db vulnerability_db; do
  echo "=== Database: $db ==="
  podman exec -e PGPASSWORD="$POSTGRES_PASSWORD" postgresql \
    psql -U postgres -d "$db" -c "\dt" | head -10
done

6.6 Stop PostgreSQL

# Stop PostgreSQL before starting all services
systemctl stop postgresql.service

Phase 7: Start All Services

7.1 Reload Systemd

# Ensure systemd sees all the restored units
systemctl daemon-reload

7.2 Start Foreman Target

# Start all Foreman services
systemctl start foreman.target

# Monitor startup
journalctl -u foreman.target -f

7.3 Wait for Services to Initialize

# Check service status
systemctl status foreman.target

# Check individual containers
podman ps

# Should show containers for:
# - postgresql
# - redis
# - candlepin
# - foreman
# - pulp-api
# - pulp-content
# - pulp-worker instances
# - httpd (if configured)
# - IOP services (if enabled)

7.4 Verify All Containers Running

# Count running containers
podman ps | wc -l

# Check for any failed containers
podman ps -a | grep -v "Up"

# View logs of any problematic containers
podman logs <container-name>

Phase 8: Post-Restore Verification

8.1 Test Foreman UI Access

# Get the Foreman URL from metadata
cat "$BACKUP_DIR/metadata.yml" | grep hostname

# Test HTTP access
curl -k https://$(cat "$BACKUP_DIR/metadata.yml" | grep "hostname:" | awk '{print $2}')

# Access UI in browser
# URL: https://<hostname>
# Username: admin
# Password: <from /var/lib/foremanctl/foreman-admin-init-passwd>

8.2 Verify Database Connectivity

# Test Foreman database connection
podman exec foreman \
  rails runner 'puts "Hosts count: #{Host.count}"'

# Test Pulp
podman exec pulp-api \
  pulpcore-manager shell -c "from pulpcore.app.models import Repository; print(f'Repos: {Repository.objects.count()}')"

8.3 Check SSL Certificates

# Verify certificate files exist
ls -la /var/lib/foremanctl/certs/

# Check certificate validity
openssl x509 -in /var/lib/foremanctl/certs/certs/server-ca.crt -noout -text

8.4 Test Authentication

# Get admin password
ADMIN_PASSWORD=$(cat /var/lib/foremanctl/foreman-admin-init-passwd)

# Test API authentication
curl -k -u admin:$ADMIN_PASSWORD \
  https://localhost/api/status

# Should return JSON with Foreman version and status

8.5 Run Health Checks

# Check Foreman tasks
podman exec foreman \
  rails runner 'puts "Running tasks: #{ForemanTasks::Task.running.count}"'

# Check Pulp tasks
podman exec pulp-api \
  pulpcore-manager shell -c "from pulpcore.app.models import Task; print(f'Running tasks: {Task.objects.filter(state__in=[\"running\", \"waiting\"]).count()}')"

# Verify IOP services (if enabled)
podman exec iop-core-gateway curl -s http://localhost:8000/health

8.6 Verify Enabled Features

# Check what features were enabled in the backup
cat "$BACKUP_DIR/metadata.yml" | grep -A 10 "enabled_features"

# Verify corresponding services are running
cat "$BACKUP_DIR/metadata.yml" | grep "enabled_features" -A 10 | grep -o '- .*' | while read -r feature; do
  feature=$(echo "$feature" | sed 's/^- //')
  echo "Checking feature: $feature"
  # Verify associated containers are running
done

Phase 9: Enable Services at Boot

# Enable foreman.target to start at boot
systemctl enable foreman.target

# Verify enabled status
systemctl is-enabled foreman.target

Known Limitations and Caveats

Hostname Dependency

The backup includes SSL certificates that are tied to the original hostname. If restoring to a different hostname:

Certificates will need to be regenerated
Update Foreman settings with new hostname
Re-register any connected smart proxies

# To regenerate certificates for new hostname:
# (This would require foremanctl to be fully installed)
# foremanctl deploy --regenerate-certificates

Network Configuration

The backup includes network configuration, but network names and IP ranges must not conflict with existing networks on the restore host
If conflicts exist, edit podman-networks.json before restoration

Volume Data Completeness

Volume backups only include critical IOP volumes
Other container volumes (logs, temporary data) are not backed up
Some containers may regenerate data on first startup

Container Images

Container images are NOT included in the backup
Images must be pulled from the registry during restore
Ensure network connectivity to quay.io and other registries
If original images are no longer available, restore may fail

Database Restore Warnings

pg_restore may show warnings about duplicate objects
These are generally safe to ignore if the data is restored
Always verify data after restore by checking table counts

Timing Issues

Services may take 5-10 minutes to fully initialize
Database migrations may run on first startup
Some containers depend on others and may restart if dependencies aren't ready

Troubleshooting

PostgreSQL Won't Start

# Check logs
journalctl -u postgresql.service -n 100

# Verify secret exists
podman secret ls | grep postgresql-admin-password

# Check container definition
cat /etc/containers/systemd/postgresql.container

Container Start Failures

# Check specific container logs
podman logs <container-name>

# Verify network exists
podman network ls

# Check for secret dependencies
podman secret ls

# Inspect container definition
cat /etc/containers/systemd/<container-name>.container

Database Restore Errors

# Check PostgreSQL is accessible
pg_isready -h localhost -p 5432

# Verify database exists
podman exec postgresql psql -U postgres -l

# Try restoring single database with verbose output
podman exec -i postgresql \
  pg_restore -U postgres -d foreman -v --no-owner --no-acl \
  < "$BACKUP_DIR/foreman.dump" 2>&1 | tee restore.log

Service Dependencies Not Met

# Check systemd dependencies
systemctl list-dependencies foreman.target

# Verify all required services are active
systemctl status foreman.target --no-pager

# Check for failed units
systemctl --failed

Permission Errors

# Fix SELinux contexts
restorecon -R /etc/containers
restorecon -R /var/lib/foremanctl

# Check file permissions
ls -laZ /etc/containers/systemd/
ls -laZ /var/lib/foremanctl/

Restore Validation Checklist

After completing the restore, verify:

Time Estimates by Phase

Phase	Task	Estimated Time
1	Infrastructure Preparation	10 minutes
2	Restore System Configuration	5 minutes
3	Restore Secrets	5 minutes
4	Pull Container Images	15-30 minutes
5	Restore Volumes (optional)	5-15 minutes
6	Restore Databases	10-15 minutes
7	Start Services	5-10 minutes
8	Post-Restore Verification	10 minutes
Total		60-90 minutes

Additional Resources

Log Locations

# Systemd service logs
journalctl -u foreman.target -f
journalctl -u postgresql.service -f

# Container logs
podman logs -f foreman
podman logs -f postgresql
podman logs -f pulp-api

# Foreman application logs (inside container)
podman exec foreman tail -f /var/log/foreman/production.log

Configuration File Locations

Quadlet containers: /etc/containers/systemd/*.container
Systemd units: /etc/systemd/system/foreman*
Foremanctl state: /var/lib/foremanctl/
Podman networks: podman network ls
Podman secrets: podman secret ls

Useful Commands

# Check all containers status
podman ps -a --format "table {{.Names}}\t{{.Status}}\t{{.Image}}"

# Restart specific service
systemctl restart <service>.service

# View systemd unit file
systemctl cat <service>.service

# Check container resource usage
podman stats

# Export container logs
podman logs <container> &> /tmp/<container>.log

Recovery from Failed Restore

If the restore fails and you need to start over:

# Stop all services
systemctl stop foreman.target

# Remove all containers
podman rm -af

# Remove all networks (except podman default)
podman network ls --format '{{.Name}}' | grep -v podman | xargs -r podman network rm

# Remove all secrets
podman secret ls --format '{{.Name}}' | xargs -r podman secret rm

# Clear systemd units
rm -f /etc/systemd/system/foreman*
rm -f /etc/systemd/system/pulp*
systemctl daemon-reload

# Clear quadlet files
rm -rf /etc/containers/systemd/*

# Clear foremanctl state
rm -rf /var/lib/foremanctl/*

# Now retry the restore from Phase 2

Support and Debugging

For issues during restore:

Check all logs carefully (systemd + container)
Verify all prerequisites are met
Ensure backup is complete and not corrupted
Check network connectivity to registries
Verify sufficient disk space for volumes
Check for SELinux denials: ausearch -m avc -ts recent

Document Version: 1.0 Last Updated: 2026-05-11 Compatible with: foremanctl backup format v1 (with backed_up_components metadata)

sjha4/RESTORE_IMPLEMENTATION_GUIDE.md