Foremanctl Backup & Restore Testing Guide - Complete Procedure with Restore Script

foremanctl-backup-restore-testing.md

Foremanctl Backup & Restore Testing Guide

Purpose: Complete testing procedure for foremanctl offline backup and restore functionality
Target: Same-box restore testing (hostname must match)
Duration: 30-60 minutes
Date: 2026-05-12

---

Overview

This guide walks through testing the foremanctl backup and restore implementation by:

1. Taking a baseline backup of current state
2. Making deliberate changes (adding repos, changing secrets)
3. Restoring from backup
4. Verifying the system returns to baseline state

Key Test Areas:

• ✅ Database backup and restore (8 databases if IOP enabled)
• ✅ Pulp content and encryption keys
• ✅ Podman secrets (72 secrets)
• ✅ Service lifecycle management
• ✅ Idempotent redeployment

---

Prerequisites

System Requirements

• Running foremanctl deployment on RHEL 9.x
• Foreman/Katello services healthy
• Root or sudo access
• Sufficient disk space (~1.5x database size + Pulp content size)

Verify Current State

# Check services are running
systemctl status foreman.target

# Check containers
podman ps

# Should see: postgresql, redis, foreman, candlepin, pulp-api, pulp-content, pulp-worker@*

Environment Setup

# Navigate to foremanctl directory
cd /root/foremanctl

# Activate Python virtual environment
source .venv/bin/activate

# Set environment variables
export OBSAH_STATE=/var/lib/foremanctl

# Verify foremanctl works
./foremanctl --help

Expected output: List of available commands including backup

---

Testing Procedure

PHASE 1: Initial Backup (Baseline)

Step 1.1: Take Baseline Backup

# Ensure you're in the right directory with venv activated
cd /root/foremanctl
source .venv/bin/activate
export OBSAH_STATE=/var/lib/foremanctl

# Define backup directory
export BACKUP_DIR=/var/tmp/foreman-backup-test

# Run backup (--wait-for-tasks ensures no running tasks block the backup)
./foremanctl backup $BACKUP_DIR --wait-for-tasks

Expected behavior:

• Preflight checks run (tasks, database integrity)
• Services stop (2-5 min downtime)
• Databases dumped (foreman, candlepin, pulp, IOP databases)
• Pulp content archived
• Podman secrets exported
• Container configs archived
• Services restart
• Metadata written

Duration: 5-15 minutes depending on Pulp content size

Step 1.2: Identify and Save Backup Path

# Find the timestamped backup directory
BACKUP_PATH=$(ls -td $BACKUP_DIR/foreman-backup-* | head -1)
echo "Backup saved to: $BACKUP_PATH"

# Save path for later use
echo "$BACKUP_PATH" > /tmp/backup-path.txt

# Verify backup contents
ls -lh "$BACKUP_PATH"

Expected files:

• metadata.yml - Backup metadata
• foreman.dump - Foreman database
• candlepin.dump - Candlepin database
• pulp.dump - Pulp database
• iop_*.dump - IOP databases (if enabled)
• config_files.tar.gz - 72 podman secrets in JSON format
• quadlet-files.tar.gz - Container definitions
• systemd-units.tar.gz - Systemd units
• podman-networks.json - Network configs
• foremanctl-state.tar.gz - Deployment state
• pulp-content.tar.gz - Pulp content + encryption keys ⚠️

Step 1.3: Verify Critical Files

# Check metadata
cat "$BACKUP_PATH/metadata.yml"

# Verify pulp-content.tar.gz contains encryption keys
tar -tzf "$BACKUP_PATH/pulp-content.tar.gz" | grep -E "(database_fields|django_secret)"

Expected: Should see database_fields.symmetric.key and django_secret_key in the tarball

Step 1.4: Document Baseline State

# Get admin password for later verification
ADMIN_PASSWD=$(cat /var/lib/foremanctl/foreman-admin-init-passwd)

# Document current counts
echo "=== BASELINE STATE ===" | tee /tmp/baseline-state.txt
echo "Backup Path: $BACKUP_PATH" | tee -a /tmp/baseline-state.txt
echo "Timestamp: $(date)" | tee -a /tmp/baseline-state.txt
echo "" | tee -a /tmp/baseline-state.txt

# Wait for services to be fully ready
sleep 10

# Host count
podman exec foreman rails runner 'puts "Hosts: #{Host.count}"' 2>/dev/null | tee -a /tmp/baseline-state.txt

# Product count
podman exec foreman rails runner 'puts "Products: #{Katello::Product.count}"' 2>/dev/null | tee -a /tmp/baseline-state.txt

# Repository count
podman exec foreman rails runner 'puts "Repositories: #{Katello::Repository.count}"' 2>/dev/null | tee -a /tmp/baseline-state.txt

# Content view count
podman exec foreman rails runner 'puts "Content Views: #{Katello::ContentView.count}"' 2>/dev/null | tee -a /tmp/baseline-state.txt

# Pulp repository count
podman exec pulp-api pulpcore-manager shell -c "from pulpcore.app.models import Repository; print(f'Pulp Repos: {Repository.objects.count()}')" 2>/dev/null | tee -a /tmp/baseline-state.txt

echo "" | tee -a /tmp/baseline-state.txt
cat /tmp/baseline-state.txt

---

PHASE 2: Make Changes (Test Data)

This phase deliberately modifies the system to verify restore works.

Step 2.1: Get Admin Credentials

ADMIN_PASSWD=$(cat /var/lib/foremanctl/foreman-admin-init-passwd)
HOSTNAME=$(hostname)

echo "================================"
echo "Foreman Web UI Access"
echo "================================"
echo "URL: https://$HOSTNAME"
echo "Username: admin"
echo "Password: $ADMIN_PASSWD"
echo "================================"

Step 2.2: Add Test Product and Repository (Via UI)

Option A: Via Web UI (Recommended)

1. Login to Foreman UI

   • Open browser to https://<your-hostname>
   • Login with credentials from Step 2.1

2. Create Test Product

   • Navigate to: Content > Products
   • Click "Create Product"
   • Name: TEST-PRODUCT-TO-DELETE
   • Label: test_product_to_delete (auto-filled)
   • Click "Save"

3. Create Test Repository

   • Within the product you just created
   • Click "New Repository"
   • Name: TEST-REPO-TO-DELETE
   • Type: yum
   • URL: https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/
   • Click "Save"

4. Optional: Create Test Content View

   • Navigate to: Content > Content Views
   • Click "Create New View"
   • Name: TEST-CV-TO-DELETE
   • Click "Save"

Option B: Via CLI (Alternative)

# Create test product
podman exec foreman rails runner '
  org = Organization.first
  product = Katello::Product.create!(
    name: "TEST-PRODUCT-TO-DELETE",
    organization: org
  )
  puts "✓ Created product: #{product.name} (ID: #{product.id})"
'

# Create test repository
podman exec foreman rails runner '
  product = Katello::Product.find_by(name: "TEST-PRODUCT-TO-DELETE")
  root = Katello::RootRepository.create!(
    name: "TEST-REPO-TO-DELETE",
    product: product,
    content_type: "yum",
    url: "https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/"
  )
  repo = Katello::Repository.create!(
    name: "TEST-REPO-TO-DELETE",
    product: product,
    content_type: "yum",
    url: "https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/",
    root: root
  )
  puts "✓ Created repository: #{repo.name} (ID: #{repo.id})"
'

Step 2.3: Change Podman Secrets (Critical Test!)

This tests that secrets are actually restored correctly:

# Save original secret values
echo "=== ORIGINAL SECRETS ===" > /tmp/secrets-before.txt

# Select 3 critical secrets to test
TEST_SECRETS=(
  "foreman-database-url"
  "pulp-db-password"
  "foreman-seed-admin-password"
)

echo "Backing up and changing test secrets..."
for secret in "${TEST_SECRETS[@]}"; do
  # Get and save original value
  original=$(podman secret inspect "$secret" --format '{{ .SecretData }}' 2>/dev/null || echo "NOT FOUND")
  echo "$secret: $original" >> /tmp/secrets-before.txt
  
  if [ "$original" != "NOT FOUND" ]; then
    echo "  Changing secret: $secret"
    
    # Remove old secret
    podman secret rm "$secret" 2>/dev/null
    
    # Create new secret with changed value
    echo "CHANGED-TEST-VALUE-$(date +%s)" | podman secret create "$secret" -
    
    # Verify it changed
    new_value=$(podman secret inspect "$secret" --format '{{ .SecretData }}')
    echo "$secret: $new_value" >> /tmp/secrets-changed.txt
  fi
done

echo "✓ Original secrets saved to: /tmp/secrets-before.txt"
echo "✓ Changed secrets saved to: /tmp/secrets-changed.txt"
echo ""
echo "⚠️  WARNING: Services may fail with changed secrets - this is expected!"
echo "⚠️  Restore will fix this."

Step 2.4: Document Changed State

# Wait a moment for changes to propagate
sleep 5

echo "=== CHANGED STATE ===" | tee /tmp/changed-state.txt
echo "Timestamp: $(date)" | tee -a /tmp/changed-state.txt
echo "" | tee -a /tmp/changed-state.txt

# Get new counts (should be HIGHER than baseline)
podman exec foreman rails runner 'puts "Hosts: #{Host.count}"' 2>/dev/null | tee -a /tmp/changed-state.txt
podman exec foreman rails runner 'puts "Products: #{Katello::Product.count}"' 2>/dev/null | tee -a /tmp/changed-state.txt
podman exec foreman rails runner 'puts "Repositories: #{Katello::Repository.count}"' 2>/dev/null | tee -a /tmp/changed-state.txt
podman exec foreman rails runner 'puts "Content Views: #{Katello::ContentView.count}"' 2>/dev/null | tee -a /tmp/changed-state.txt
podman exec pulp-api pulpcore-manager shell -c "from pulpcore.app.models import Repository; print(f'Pulp Repos: {Repository.objects.count()}')" 2>/dev/null | tee -a /tmp/changed-state.txt

echo "" | tee -a /tmp/changed-state.txt
echo "✓ Changed state documented"
cat /tmp/changed-state.txt

---

PHASE 3: Run Restore

Step 3.1: Create Restore Script

Save the following script as /root/test-restore.sh:

cat > /root/test-restore.sh << 'RESTORE_SCRIPT_EOF'
#!/bin/bash
#
# test-restore.sh - Same-box restore testing for foremanctl backup
#
# Usage: ./test-restore.sh /path/to/backup/foreman-backup-TIMESTAMP
#
# This script performs a DESTRUCTIVE restore on the same box:
# - Restores podman secrets (MUST be done before starting PostgreSQL)
# - Drops and recreates all databases
# - Restores database dumps
# - Fixes database ownership and permissions
# - Restores Pulp content and encryption keys
#
# WARNING: This will OVERWRITE current data!
#
# FIXES APPLIED:
# 1. Secrets restored BEFORE PostgreSQL starts (was causing PostgreSQL to fail)
# 2. Database ownership and permissions fixed after pg_restore (was causing permission denied errors)
#

set -e  # Exit on error

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

# Helper functions
log_info() {
    echo -e "${GREEN}[INFO]${NC} $1"
}

log_warn() {
    echo -e "${YELLOW}[WARN]${NC} $1"
}

log_error() {
    echo -e "${RED}[ERROR]${NC} $1"
}

# Check arguments
if [ $# -ne 1 ]; then
    echo "Usage: $0 /path/to/backup/foreman-backup-TIMESTAMP"
    exit 1
fi

BACKUP_PATH="$1"

if [ ! -d "$BACKUP_PATH" ]; then
    log_error "Backup directory does not exist: $BACKUP_PATH"
    exit 1
fi

log_info "Starting restore from: $BACKUP_PATH"

# Verify backup contains required files
log_info "Verifying backup contents..."
REQUIRED_FILES=(
    "metadata.yml"
    "foreman.dump"
    "candlepin.dump"
    "pulp.dump"
    "config_files.tar.gz"
)

for file in "${REQUIRED_FILES[@]}"; do
    if [ ! -f "$BACKUP_PATH/$file" ]; then
        log_error "Required file missing: $file"
        exit 1
    fi
done
log_info "✓ Backup verification passed"

# Confirmation prompt
log_warn "════════════════════════════════════════════════════════════════"
log_warn "WARNING: This will DESTROY current data and restore from backup!"
log_warn "Backup path: $BACKUP_PATH"
log_warn "════════════════════════════════════════════════════════════════"
read -p "Are you sure you want to continue? (type 'yes' to confirm): " confirm
if [ "$confirm" != "yes" ]; then
    log_info "Restore cancelled"
    exit 0
fi

# ============================================================================
# PHASE 1: Stop Services
# ============================================================================
log_info "Phase 1: Stopping Foreman services..."
systemctl stop foreman.target
sleep 5
log_info "✓ Services stopped"

# ============================================================================
# PHASE 2: Restore Podman Secrets (BEFORE starting PostgreSQL!)
# ============================================================================
log_info "Phase 2: Restoring podman secrets..."
log_warn "CRITICAL: Secrets MUST be restored before PostgreSQL starts!"

# Extract secrets from backup
TEMP_DIR=$(mktemp -d)
cd "$TEMP_DIR"
tar -xzf "$BACKUP_PATH/config_files.tar.gz" || {
    log_error "Failed to extract config_files.tar.gz"
    exit 1
}

if [ ! -f "secrets.json" ]; then
    log_error "secrets.json not found in config_files.tar.gz"
    exit 1
fi

# Count secrets in backup
backup_secret_count=$(jq 'length' secrets.json)
log_info "Found $backup_secret_count secrets in backup"

# List current secrets
current_secret_count=$(podman secret ls --format '{{.Name}}' | wc -l)
log_info "Currently have $current_secret_count secrets"

# Remove all existing secrets (WARNING: DESTRUCTIVE!)
log_warn "Removing all existing podman secrets..."
podman secret ls --format '{{.Name}}' | while read secret_name; do
    podman secret rm "$secret_name" 2>/dev/null || log_warn "Failed to remove secret: $secret_name"
done

# Recreate secrets from backup
log_info "Recreating secrets from backup..."
secret_count=0
jq -r '.[] | .[0] | @json' secrets.json | while read -r secret_json; do
    name=$(echo "$secret_json" | jq -r '.Spec.Name')
    data=$(echo "$secret_json" | jq -r '.SecretData')

    echo "$data" | podman secret create "$name" - > /dev/null 2>&1 || {
        log_warn "Failed to create secret: $name"
        continue
    }
    secret_count=$((secret_count + 1))
done

# Verify secret count
restored_secret_count=$(podman secret ls --format '{{.Name}}' | wc -l)
log_info "  ✓ Restored $restored_secret_count podman secrets"

if [ "$restored_secret_count" -ne "$backup_secret_count" ]; then
    log_warn "Secret count mismatch! Expected: $backup_secret_count, Got: $restored_secret_count"
fi

# Cleanup temp directory
rm -rf "$TEMP_DIR"

# ============================================================================
# PHASE 3: Start PostgreSQL (now that secrets exist)
# ============================================================================
log_info "Phase 3: Starting PostgreSQL (secrets are now available)..."
systemctl start postgresql.service

log_info "Waiting for PostgreSQL to be ready..."
for i in {1..30}; do
    if pg_isready -h localhost -p 5432 > /dev/null 2>&1; then
        log_info "✓ PostgreSQL is ready"
        break
    fi
    echo -n "."
    sleep 2
done
echo ""

# Get PostgreSQL password (now that secrets are restored)
POSTGRES_PASSWORD=$(podman secret inspect postgresql-admin-password --format '{{ .SecretData }}' | base64 -d)

# ============================================================================
# PHASE 4: Restore Databases
# ============================================================================
log_info "Phase 4: Restoring databases..."

# Detect which databases to restore
DATABASES=()
for dump in "$BACKUP_PATH"/*.dump; do
    dbname=$(basename "$dump" .dump)
    DATABASES+=("$dbname")
done

log_info "Found ${#DATABASES[@]} databases to restore: ${DATABASES[*]}"

# Drop and recreate each database
for dbname in "${DATABASES[@]}"; do
    log_info "Dropping and recreating database: $dbname"
    podman exec -e PGPASSWORD="$POSTGRES_PASSWORD" postgresql \
        psql -U postgres -c "DROP DATABASE IF EXISTS $dbname;" 2>&1 | grep -v "NOTICE" || true

    podman exec -e PGPASSWORD="$POSTGRES_PASSWORD" postgresql \
        psql -U postgres -c "CREATE DATABASE $dbname OWNER postgres;" || {
        log_error "Failed to create database: $dbname"
        exit 1
    }
done

# Restore each database dump
for dbname in "${DATABASES[@]}"; do
    dump_file="$BACKUP_PATH/${dbname}.dump"
    log_info "Restoring database: $dbname from $(basename $dump_file)"

    podman exec -i -e PGPASSWORD="$POSTGRES_PASSWORD" postgresql \
        pg_restore -U postgres -d "$dbname" -v --no-owner --no-acl < "$dump_file" \
        > "/tmp/restore-${dbname}.log" 2>&1 || {
        log_warn "pg_restore reported errors (this may be normal - check /tmp/restore-${dbname}.log)"
    }

    # Verify restore
    table_count=$(podman exec -e PGPASSWORD="$POSTGRES_PASSWORD" postgresql \
        psql -U postgres -d "$dbname" -t -c "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = 'public';" | tr -d ' ')
    log_info "  ✓ Database $dbname restored ($table_count tables)"
done

log_info "✓ All databases restored"

# ============================================================================
# PHASE 5: Fix Database Ownership and Permissions
# ============================================================================
log_info "Phase 5: Fixing database ownership and permissions..."
log_warn "This is required because pg_restore used --no-owner --no-acl flags"

# Define database ownership mappings
declare -A DB_OWNERS=(
    ["foreman"]="foreman"
    ["candlepin"]="candlepin"
    ["pulp"]="pulp"
    ["advisor_db"]="advisor_user"
    ["inventory_db"]="inventory_admin"
    ["remediations_db"]="remediations_user"
    ["vmaas_db"]="vmaas_admin"
    ["vulnerability_db"]="vulnerability_admin"
    ["iop_advisor"]="postgres"
    ["iop_inventory"]="postgres"
    ["iop_remediations"]="postgres"
    ["iop_vmaas"]="postgres"
    ["iop_vulnerability"]="postgres"
)

# Fix ownership for each database
for dbname in "${!DB_OWNERS[@]}"; do
    owner="${DB_OWNERS[$dbname]}"

    # Check if database exists in this backup
    if [[ ! " ${DATABASES[@]} " =~ " ${dbname} " ]] && [[ ! " ${DATABASES[@]} " =~ " iop_${dbname#iop_} " ]]; then
        continue
    fi

    # Handle IOP database naming (dump files are iop_advisor.dump but db is advisor_db)
    actual_dbname="$dbname"
    if [[ "$dbname" == iop_* ]]; then
        actual_dbname="$dbname"
    elif [[ "$dbname" == *_db ]]; then
        # Check if we have iop_ version in DATABASES
        iop_version="iop_${dbname%_db}"
        if [[ " ${DATABASES[@]} " =~ " ${iop_version} " ]]; then
            actual_dbname="$iop_version"
        fi
    fi

    log_info "Setting ownership for $actual_dbname to $owner..."

    # Change database owner
    podman exec -e PGPASSWORD="$POSTGRES_PASSWORD" postgresql \
        psql -U postgres -c "ALTER DATABASE \"$actual_dbname\" OWNER TO \"$owner\";" 2>&1 | grep -v "NOTICE" || true

    # Grant privileges on all tables and sequences
    if [ "$owner" != "postgres" ]; then
        podman exec -e PGPASSWORD="$POSTGRES_PASSWORD" postgresql \
            psql -U postgres -d "$actual_dbname" -c "GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO \"$owner\";" 2>&1 | grep -v "NOTICE" || true

        podman exec -e PGPASSWORD="$POSTGRES_PASSWORD" postgresql \
            psql -U postgres -d "$actual_dbname" -c "GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO \"$owner\";" 2>&1 | grep -v "NOTICE" || true
    fi
done

log_info "✓ Database ownership and permissions fixed"

# ============================================================================
# PHASE 6: Restore Pulp Content and Encryption Keys
# ============================================================================
log_info "Phase 6: Restoring Pulp content and encryption keys..."

if [ -f "$BACKUP_PATH/pulp-content.tar.gz" ]; then
    log_info "Backing up current Pulp media directory..."
    if [ -d /var/lib/pulp/media ]; then
        mv /var/lib/pulp/media /var/lib/pulp/media.backup-$(date +%s)
    fi

    log_info "Extracting Pulp content to /var/lib/pulp..."
    cd /var/lib/pulp
    tar -xzf "$BACKUP_PATH/pulp-content.tar.gz" || {
        log_error "Failed to extract Pulp content"
        exit 1
    }

    # Verify encryption keys were restored
    if [ -f /var/lib/pulp/database_fields.symmetric.key ]; then
        log_info "  ✓ Pulp encryption key restored"
    else
        log_error "Pulp encryption key NOT found after restore!"
        exit 1
    fi

    if [ -f /var/lib/pulp/django_secret_key ]; then
        log_info "  ✓ Django secret key restored"
    else
        log_error "Django secret key NOT found after restore!"
        exit 1
    fi

    # Count restored files
    file_count=$(find /var/lib/pulp/media -type f 2>/dev/null | wc -l)
    log_info "  ✓ Pulp content restored ($file_count files in media directory)"
else
    log_warn "No pulp-content.tar.gz found in backup (may have been skipped)"
fi

# ============================================================================
# PHASE 7: Restore Foremanctl State
# ============================================================================
log_info "Phase 7: Restoring foremanctl state..."

if [ -f "$BACKUP_PATH/foremanctl-state.tar.gz" ]; then
    log_info "Backing up current foremanctl state..."
    if [ -d /var/lib/foremanctl ]; then
        # Backup current state (excluding certs)
        tar -czf /tmp/foremanctl-state-backup-$(date +%s).tar.gz \
            --exclude=/var/lib/foremanctl/certs \
            /var/lib/foremanctl 2>/dev/null || true
    fi

    log_info "Extracting foremanctl state to /var/lib/foremanctl..."
    cd /var/lib
    tar -xzf "$BACKUP_PATH/foremanctl-state.tar.gz" || {
        log_error "Failed to extract foremanctl state"
        exit 1
    }

    log_info "  ✓ Foremanctl state restored"
else
    log_warn "No foremanctl-state.tar.gz found in backup"
fi

# ============================================================================
# PHASE 8: Stop PostgreSQL and Start All Services
# ============================================================================
log_info "Phase 8: Restarting services..."

log_info "Stopping PostgreSQL..."
systemctl stop postgresql.service
sleep 3

log_info "Starting Foreman services..."
systemctl start foreman.target

log_info "Waiting for services to start (60 seconds)..."
sleep 60

# ============================================================================
# PHASE 9: Verification
# ============================================================================
log_info "Phase 9: Verifying restore..."

# Check service status
log_info "Checking service status..."
if systemctl is-active --quiet foreman.target; then
    log_info "  ✓ foreman.target is active"
else
    log_error "  ✗ foreman.target is NOT active"
fi

# Check containers
log_info "Checking running containers..."
container_count=$(podman ps --format '{{.Names}}' | wc -l)
log_info "  Running containers: $container_count"

# Check PostgreSQL
if podman ps --format '{{.Names}}' | grep -q "^postgresql$"; then
    log_info "  ✓ PostgreSQL container running"
else
    log_error "  ✗ PostgreSQL container NOT running"
fi

# Check Foreman
if podman ps --format '{{.Names}}' | grep -q "^foreman$"; then
    log_info "  ✓ Foreman container running"
else
    log_error "  ✗ Foreman container NOT running"
fi

# Check Pulp
if podman ps --format '{{.Names}}' | grep -q "^pulp-api$"; then
    log_info "  ✓ Pulp API container running"
else
    log_error "  ✗ Pulp API container NOT running"
fi

# Test database connectivity
log_info "Testing database connectivity..."
sleep 10  # Give services more time to fully start

if podman exec foreman rails runner 'puts "Hosts: #{Host::Managed.count}"' 2>/dev/null; then
    log_info "  ✓ Foreman database connection working"
else
    log_warn "  ✗ Foreman database connection failed (may need more time to start)"
fi

# Test Pulp database connectivity
if podman exec pulp-api pulpcore-manager shell -c "from pulpcore.app.models import Repository; print(f'Repos: {Repository.objects.count()}')" 2>/dev/null; then
    log_info "  ✓ Pulp database connection working"
else
    log_warn "  ✗ Pulp database connection failed (may need more time to start)"
fi

# Check for recent errors
log_info "Checking for recent errors in logs..."
error_count=$(journalctl -u foreman.target --since "5 minutes ago" 2>/dev/null | grep -i error | wc -l)
if [ "$error_count" -gt 0 ]; then
    log_warn "  Found $error_count error messages in logs (check: journalctl -u foreman.target --since '5 minutes ago')"
else
    log_info "  ✓ No errors in recent logs"
fi

# ============================================================================
# COMPLETION
# ============================================================================
echo ""
log_info "════════════════════════════════════════════════════════════════"
log_info "Restore completed!"
log_info "════════════════════════════════════════════════════════════════"
log_info "Next steps:"
log_info "1. Access Foreman UI: https://$(hostname)"
log_info "2. Login with credentials from: /var/lib/foremanctl/foreman-admin-init-passwd"
log_info "3. Verify your changes are GONE (system restored to backup state)"
log_info "4. Check for any errors: journalctl -u foreman.target --since '5 minutes ago'"
log_info ""
log_info "Database restore logs: /tmp/restore-*.log"
log_info "════════════════════════════════════════════════════════════════"
log_info ""
log_info "FIXES APPLIED IN THIS VERSION:"
log_info "1. ✓ Secrets restored BEFORE PostgreSQL starts (Phase 2)"
log_info "2. ✓ Database ownership and permissions fixed after restore (Phase 5)"
log_info "════════════════════════════════════════════════════════════════"
RESTORE_SCRIPT_EOF

chmod +x /root/test-restore.sh
echo "✓ Restore script created at /root/test-restore.sh"

Step 3.2: Run Restore

# Retrieve backup path
BACKUP_PATH=$(cat /tmp/backup-path.txt)
echo "Restoring from: $BACKUP_PATH"

# Execute restore
cd /root
./test-restore.sh "$BACKUP_PATH"

What happens:

1. Verification of backup files
2. Confirmation prompt (type yes)
3. Services stop
4. PostgreSQL starts in isolation
5. Databases dropped and recreated
6. Database dumps restored
7. Pulp content + encryption keys restored
8. Podman secrets restored (all 72)
9. Foremanctl state restored
10. Services restart
11. Verification checks

Duration: 5-15 minutes

---

PHASE 4: Verify Restore

Step 4.1: Wait for Services

# Services may need extra time to fully initialize
echo "Waiting for services to fully start..."
sleep 30

# Check service status
systemctl status foreman.target

# Check running containers
podman ps --format 'table {{.Names}}\t{{.Status}}'

Expected containers:

• postgresql
• redis
• candlepin
• foreman
• foreman-proxy
• pulp-api
• pulp-content
• pulp-worker@1 through pulp-worker@N
• IOP containers (if enabled)

Step 4.2: Verify Data Restored to Baseline

echo "=== RESTORED STATE (should match BASELINE) ===" | tee /tmp/restored-state.txt
echo "Timestamp: $(date)" | tee -a /tmp/restored-state.txt
echo "" | tee -a /tmp/restored-state.txt

# Get current counts
podman exec foreman rails runner 'puts "Hosts: #{Host.count}"' 2>/dev/null | tee -a /tmp/restored-state.txt
podman exec foreman rails runner 'puts "Products: #{Katello::Product.count}"' 2>/dev/null | tee -a /tmp/restored-state.txt
podman exec foreman rails runner 'puts "Repositories: #{Katello::Repository.count}"' 2>/dev/null | tee -a /tmp/restored-state.txt
podman exec foreman rails runner 'puts "Content Views: #{Katello::ContentView.count}"' 2>/dev/null | tee -a /tmp/restored-state.txt
podman exec pulp-api pulpcore-manager shell -c "from pulpcore.app.models import Repository; print(f'Pulp Repos: {Repository.objects.count()}')" 2>/dev/null | tee -a /tmp/restored-state.txt

# Compare states
echo ""
echo "═══════════════════════════════════════"
echo "STATE COMPARISON"
echo "═══════════════════════════════════════"
echo ""
echo "BASELINE (original state):"
cat /tmp/baseline-state.txt | grep -E "(Hosts|Products|Repositories|Views|Pulp)"
echo ""
echo "CHANGED (after modifications):"
cat /tmp/changed-state.txt | grep -E "(Hosts|Products|Repositories|Views|Pulp)"
echo ""
echo "RESTORED (after restore - should match BASELINE):"
cat /tmp/restored-state.txt | grep -E "(Hosts|Products|Repositories|Views|Pulp)"
echo "═══════════════════════════════════════"

Success Criteria: Restored counts MUST match baseline, NOT changed state

Step 4.3: Verify Test Changes Are Gone

echo ""
echo "Verifying test changes were removed..."

# Check test product does NOT exist
podman exec foreman rails runner '
  product = Katello::Product.find_by(name: "TEST-PRODUCT-TO-DELETE")
  if product.nil?
    puts "✓ TEST PRODUCT NOT FOUND (correct - removed by restore)"
  else
    puts "✗ TEST PRODUCT STILL EXISTS (restore FAILED!)"
    exit 1
  end
'

# Check test repository does NOT exist
podman exec foreman rails runner '
  repo = Katello::Repository.find_by(name: "TEST-REPO-TO-DELETE")
  if repo.nil?
    puts "✓ TEST REPOSITORY NOT FOUND (correct - removed by restore)"
  else
    puts "✗ TEST REPOSITORY STILL EXISTS (restore FAILED!)"
    exit 1
  end
'

# Check test content view does NOT exist (if you created one)
podman exec foreman rails runner '
  cv = Katello::ContentView.find_by(name: "TEST-CV-TO-DELETE")
  if cv.nil?
    puts "✓ TEST CONTENT VIEW NOT FOUND (correct - removed by restore)"
  else
    puts "✗ TEST CONTENT VIEW STILL EXISTS (restore FAILED!)"
    exit 1
  end
'

Step 4.4: Verify Secrets Were Restored

echo ""
echo "Verifying podman secrets restored correctly..."

# Get restored secret values
echo "=== RESTORED SECRETS ===" > /tmp/secrets-after.txt

TEST_SECRETS=(
  "foreman-database-url"
  "pulp-db-password"
  "foreman-seed-admin-password"
)

for secret in "${TEST_SECRETS[@]}"; do
  restored=$(podman secret inspect "$secret" --format '{{ .SecretData }}' 2>/dev/null || echo "NOT FOUND")
  echo "$secret: $restored" >> /tmp/secrets-after.txt
done

# Compare secrets
echo "═══════════════════════════════════════"
echo "SECRET COMPARISON"
echo "═══════════════════════════════════════"
echo ""
echo "BEFORE CHANGES (original):"
cat /tmp/secrets-before.txt
echo ""
echo "AFTER CHANGES (modified):"
cat /tmp/secrets-changed.txt
echo ""
echo "AFTER RESTORE (should match original):"
cat /tmp/secrets-after.txt
echo "═══════════════════════════════════════"

# Verify match
if diff /tmp/secrets-before.txt /tmp/secrets-after.txt > /dev/null 2>&1; then
    echo "✓ SECRETS RESTORED CORRECTLY!"
else
    echo "✗ SECRETS DO NOT MATCH!"
    echo "Differences:"
    diff /tmp/secrets-before.txt /tmp/secrets-after.txt
fi

Step 4.5: Verify Pulp Encryption Keys

echo ""
echo "Verifying Pulp encryption keys..."

# Check keys exist
if [ -f /var/lib/pulp/database_fields.symmetric.key ]; then
    echo "✓ Pulp encryption key exists"
    ls -lh /var/lib/pulp/database_fields.symmetric.key
else
    echo "✗ Pulp encryption key MISSING!"
    exit 1
fi

if [ -f /var/lib/pulp/django_secret_key ]; then
    echo "✓ Django secret key exists"
    ls -lh /var/lib/pulp/django_secret_key
else
    echo "✗ Django secret key MISSING!"
    exit 1
fi

# Test encrypted fields are readable
echo "Testing encrypted database fields..."
podman exec pulp-api pulpcore-manager shell -c "
from pulpcore.app.models import Repository
repos = Repository.objects.all()[:5]
for repo in repos:
    print(f'Repo: {repo.name}')
print('✓ Encrypted fields are readable')
" 2>&1 | grep -E "(Repo:|readable)" || echo "✗ Error reading encrypted fields!"

Step 4.6: Verify Pulp Content

echo ""
echo "Verifying Pulp content..."

# Count files
pulp_files=$(find /var/lib/pulp/media -type f 2>/dev/null | wc -l)
echo "Pulp content files: $pulp_files"

# Check artifacts
if [ -d /var/lib/pulp/media/artifact ]; then
    artifact_count=$(find /var/lib/pulp/media/artifact -type f 2>/dev/null | wc -l)
    echo "✓ Artifacts directory exists ($artifact_count artifacts)"
else
    echo "✗ Artifacts directory missing!"
fi

Step 4.7: Check for Errors

echo ""
echo "Checking for errors in logs..."

# Check systemd logs
error_count=$(journalctl -u foreman.target --since "10 minutes ago" 2>/dev/null | grep -i error | wc -l)
if [ "$error_count" -eq 0 ]; then
    echo "✓ No errors in systemd logs"
else
    echo "⚠️  Found $error_count error messages"
    echo "Review with: journalctl -u foreman.target --since '10 minutes ago' | grep -i error"
fi

# Check container logs
echo ""
echo "Checking container logs..."
for container in foreman pulp-api candlepin; do
    container_errors=$(podman logs $container --since 10m 2>&1 | grep -i error | wc -l)
    if [ "$container_errors" -eq 0 ]; then
        echo "✓ $container: no errors"
    else
        echo "⚠️  $container: $container_errors errors (check: podman logs $container)"
    fi
done

Step 4.8: Verify UI Access

ADMIN_PASSWD=$(cat /var/lib/foremanctl/foreman-admin-init-passwd)
HOSTNAME=$(hostname)

echo ""
echo "═══════════════════════════════════════"
echo "MANUAL UI VERIFICATION REQUIRED"
echo "═══════════════════════════════════════"
echo ""
echo "1. Open browser to: https://$HOSTNAME"
echo "2. Login:"
echo "   Username: admin"
echo "   Password: $ADMIN_PASSWD"
echo ""
echo "3. Verify in UI:"
echo "   ✓ Navigate to Content > Products"
echo "   ✓ Confirm 'TEST-PRODUCT-TO-DELETE' is NOT in list"
echo "   ✓ Confirm original products ARE present"
echo ""
echo "   ✓ Navigate to Content > Repositories"
echo "   ✓ Confirm 'TEST-REPO-TO-DELETE' is NOT in list"
echo ""
echo "   ✓ Check Content > Content Views"
echo "   ✓ Confirm 'TEST-CV-TO-DELETE' is NOT in list"
echo "═══════════════════════════════════════"

---

PHASE 5: Test Idempotent Redeployment

This verifies that encryption keys survive a redeploy (critical!)

Step 5.1: Run foremanctl deploy

# Activate venv if not already
cd /root/foremanctl
source .venv/bin/activate
export OBSAH_STATE=/var/lib/foremanctl

# Redeploy (should be idempotent - no changes)
./foremanctl deploy --foreman-initial-admin-password=changeme

Expected behavior:

• Deployment runs
• Encryption key generation tasks are SKIPPED (files already exist)
• Services may restart
• No errors

Step 5.2: Verify Keys NOT Regenerated

echo ""
echo "Verifying encryption keys were preserved during redeploy..."

# Check keys still exist
if [ -f /var/lib/pulp/database_fields.symmetric.key ]; then
    echo "✓ Pulp encryption key still exists after redeploy"
else
    echo "✗ Pulp encryption key DELETED by redeploy!"
    exit 1
fi

if [ -f /var/lib/pulp/django_secret_key ]; then
    echo "✓ Django secret key still exists after redeploy"
else
    echo "✗ Django secret key DELETED by redeploy!"
    exit 1
fi

# Test encrypted fields still work
echo "Testing encrypted fields after redeploy..."
podman exec pulp-api pulpcore-manager shell -c "
from pulpcore.app.models import Repository
repos = Repository.objects.all()[:5]
for repo in repos:
    print(f'Repo: {repo.name}')
print('✓ Encrypted fields still readable after redeploy')
" 2>&1 | grep -E "(Repo:|readable)" || echo "✗ Encrypted fields unreadable!"

Success Criteria: Keys exist AND encrypted fields are still readable

---

Success Criteria Summary

✅ Test PASSES if ALL are true:

1. Backup Phase

• Backup completes without critical errors
• All expected files created
• Metadata contains correct information
• pulp-content.tar.gz contains encryption keys

2. Restore Phase

• Restore completes without critical errors
• All databases restored
• Pulp content restored
• All secrets restored (72 total)
• Services restart successfully

3. Data Verification

• Restored counts MATCH baseline (not changed state)
• Test product/repo/CV do NOT exist
• Original data IS present

4. Secrets Verification

• Changed secrets restored to original values
• Secret count matches backup (72)

5. Encryption Keys

• database_fields.symmetric.key exists
• django_secret_key exists
• Encrypted DB fields are readable
• No decryption errors in logs

6. Pulp Content

• Media directory exists
• Artifacts directory populated
• File count reasonable

7. Services Health

• All containers running
• No critical errors in logs
• UI accessible
• Database connections working

8. Idempotent Redeploy

• Keys NOT regenerated
• Encrypted fields still readable
• No errors during redeploy

---

Troubleshooting

Services Won't Start

# Check which services failed
systemctl status foreman.target
podman ps -a

# Check specific container logs
podman logs foreman
podman logs pulp-api
podman logs postgresql

# Restart services
systemctl restart foreman.target

Database Errors

# Check restore logs
cat /tmp/restore-foreman.log
cat /tmp/restore-pulp.log

# Test database connectivity
podman exec postgresql psql -U postgres -l

# Check PostgreSQL logs
journalctl -u postgresql.service --since "15 minutes ago"

Secret Errors

# List all secrets
podman secret ls

# Count secrets
podman secret ls | wc -l
# Should be 72 (or 73 with header)

# Inspect specific secret
podman secret inspect foreman-database-url --format '{{ .SecretData }}'

# Re-run restore if secrets are wrong
./test-restore.sh "$BACKUP_PATH"

Pulp Encryption Errors

# Check if keys exist
ls -la /var/lib/pulp/*.key

# Check pulp-content.tar.gz contents
tar -tzf "$BACKUP_PATH/pulp-content.tar.gz" | grep -E "(database_fields|django_secret)"

# If keys missing, manually extract
cd /var/lib/pulp
tar -xzf "$BACKUP_PATH/pulp-content.tar.gz" database_fields.symmetric.key django_secret_key

# Restart Pulp services
podman restart pulp-api pulp-content
for i in {1..8}; do podman restart pulp-worker@$i; done

UI Not Accessible

# Check httpd is running (if containerized)
podman ps | grep httpd

# Or check host httpd service
systemctl status httpd

# Check Foreman container
podman logs foreman | tail -50

# Try accessing API
ADMIN_PASSWD=$(cat /var/lib/foremanctl/foreman-admin-init-passwd)
curl -k -u admin:$ADMIN_PASSWD https://localhost/api/status

---

Cleanup After Testing

# Remove test state files
rm -f /tmp/baseline-state.txt
rm -f /tmp/changed-state.txt
rm -f /tmp/restored-state.txt
rm -f /tmp/secrets-*.txt
rm -f /tmp/restore-*.log
rm -f /tmp/backup-path.txt

# Optional: Remove backup directory
# WARNING: Only if completely done with testing!
# rm -rf /var/tmp/foreman-backup-test

echo "✓ Cleanup complete"

---

Test Results Template

Document your results:

FOREMANCTL BACKUP/RESTORE TEST RESULTS
======================================
Date: [YYYY-MM-DD]
Tester: [Your name]
Hostname: [system hostname]
Foremanctl Version: [git describe --tags]

ENVIRONMENT:
- OS: [cat /etc/redhat-release]
- Podman: [podman --version]
- IOP Enabled: [yes/no]

BACKUP:
- Backup Path: [path]
- Backup Size: [du -sh]
- Databases: [count]
- Duration: [X minutes]

RESTORE:
- Restore Duration: [X minutes]
- Databases Restored: [count]
- Secrets Restored: [count]
- Pulp Files: [count]

VERIFICATION RESULTS:
✓/✗ Data Restored to Baseline
✓/✗ Test Changes Removed
✓/✗ Secrets Restored Correctly
✓/✗ Pulp Keys Working
✓/✗ Pulp Content Restored
✓/✗ Services Healthy
✓/✗ Redeploy Idempotent

OVERALL: PASS / FAIL

ISSUES FOUND:
[List any problems, warnings, or unexpected behavior]

NOTES:
[Additional observations]

---

Quick Reference Commands

# Activate environment
source .venv/bin/activate && export OBSAH_STATE=/var/lib/foremanctl

# Take backup
./foremanctl backup /var/tmp/foreman-backup-test --wait-for-tasks

# Get backup path
BACKUP_PATH=$(ls -td /var/tmp/foreman-backup-test/foreman-backup-* | head -1)

# Run restore
./test-restore.sh "$BACKUP_PATH"

# Check services
systemctl status foreman.target
podman ps

# Check logs
journalctl -u foreman.target --since "10 minutes ago"

# Check DB connectivity
podman exec foreman rails runner 'puts Host.count'
podman exec pulp-api pulpcore-manager shell -c "from pulpcore.app.models import Repository; print(Repository.objects.count())"

# Get admin password
cat /var/lib/foremanctl/foreman-admin-init-passwd

---

End of Testing Guide

For questions or issues, refer to:

• Restore Guide: https://gist.github.com/sjha4/68673cb98b0b531aa607e06a8ade4ccc
• Backup Summary: https://gist.github.com/sjha4/eb71a9393ede8600fa34b6b857112666

Why You Had to Run foremanctl deploy Manually
Why You Had to Run foremanctl deploy Manually

The foremanctl backup/restore is a lower-level tool that:

• ✅ Backs up databases, secrets, files
• ✅ Restores databases, secrets, files
• ❌ Does NOT automatically run migrations

Whereas foreman-maintain is a higher-level orchestration tool that:

• Calls the installer
• Runs migrations
• Handles upgrades
• Complete end-to-end restore

Recommendation for Foremanctl

The foremanctl restore documentation/script should include:

After manual restore:

echo "Running migrations..."
cd /root/foremanctl
source .venv/bin/activate
export OBSAH_STATE=/var/lib/foremanctl
./foremanctl deploy

Or better yet, the restore script could automatically run foremanctl deploy at the end, similar to how foreman-maintain does it!