Foremanctl Restore Implementation Evaluation

Date: 2026-05-11
Context: Evaluating restore functionality for foremanctl based on backup implementation and foreman-maintain restore behavior
Related: Backup Evaluation

Executive Summary

This document evaluates how to implement foremanctl restore for the containerized/quadlet-based Foreman deployment. The restore functionality must handle backups created by both foremanctl backup and potentially foreman-maintain backup offline, adapting traditional RPM-based restore logic to the containerized world.

Key Challenge: Foreman-maintain assumes RPM packages and traditional file layouts; foremanctl uses containers and quadlet definitions. The restore process must bridge this gap.

1. Foreman-Maintain Restore Analysis

1.1 Restore Workflow (foreman-maintain)

Command:

foreman-maintain restore /path/to/backup-dir
foreman-maintain restore /path/to/backup-dir --incremental
foreman-maintain restore /path/to/backup-dir --dry-run

Restore Steps (from definitions/scenarios/restore.rb):

Validation Phase (Pre-flight)
- ✅ Check running as root
- ✅ Validate backup directory contains required files
- ✅ Validate hostname matches backup metadata
- ✅ Validate network interfaces match (DNS/DHCP features)
- ✅ Validate PostgreSQL dump file permissions
Confirmation Phase (if not --dry-run)
- ⚠️ Show user what will be restored
- ⚠️ Confirm destructive operation (drops databases)
- ⚠️ Confirm required packages will be installed
Preparation Phase
- 📦 Install required packages from backup metadata (RPMs)
- 📁 Restore config files from config_files.tar.gz
- ⏸️ Stop cron/timers
- ⏸️ Stop all services
Installer Reset (if not incremental)
- 🔧 Reset foreman-installer to clean state
- 🔧 This allows installer to reconfigure from restored configs
Database Restore Phase
- 🗑️ Drop existing databases (foreman, candlepin, pulp, 5 IOP DBs)
- ▶️ Start PostgreSQL service
- 📥 Restore database dumps using pg_restore -C -d postgres
  - foreman.dump
  - candlepin.dump
  - pulpcore.dump (renamed from pulp.dump)
  - iop_advisor.dump
  - iop_inventory.dump
  - iop_remediations.dump
  - iop_vmaas.dump
  - iop_vulnerability.dump
- ⏸️ Stop PostgreSQL service
File Extraction (if pulp_data.tar exists)
- 📁 Extract Pulp content to /var/lib/pulp/
Reconfiguration Phase
- 🔧 Run foreman-installer (re-applies configuration)
- 🔧 Run upgrade rake tasks (db:migrate, etc.)
Restart Phase
- ▶️ Start cron/timers
- ▶️ Services started by installer

Rescue Scenario (on failure):

Stop cron/timers
Leave services stopped for manual intervention

1.2 Database Restore Logic

From base_database.rb:

def restore_dump(file, localdb)
  if localdb
    # Local database: restore with CREATE DATABASE
    dump_cmd = "runuser - postgres -c 'pg_restore -C -d postgres #{file}'"
    execute!(dump_cmd)
  else
    # Remote database: restore into existing database
    dump_cmd = 'pg_restore --no-privileges --clean --disable-triggers -n public ' \
               "-d #{configuration['database']} #{file}"
    execute!(dump_cmd, :env => base_env, :valid_exit_statuses => [0, 1])
  end
end

Key differences:

Local DB: Uses pg_restore -C -d postgres (creates database from dump)
Remote DB: Uses pg_restore --clean --disable-triggers (restores into existing DB)

1.3 File Mapping (Utils::Backup)

Expected files in backup directory:

file_map = {
  :pulp_data => 'pulp_data.tar',              # Optional: Pulp content
  :foreman_dump => 'foreman.dump',            # Required for Server
  :candlepin_dump => 'candlepin.dump',        # Required for Server
  :pulpcore_dump => 'pulpcore.dump',          # Required for Server/Capsule
  :iop_advisor_dump => 'iop_advisor.dump',           # Optional: IOP
  :iop_inventory_dump => 'iop_inventory.dump',       # Optional: IOP
  :iop_remediations_dump => 'iop_remediations.dump', # Optional: IOP
  :iop_vmaas_dump => 'iop_vmaas.dump',               # Optional: IOP
  :iop_vulnerability_dump => 'iop_vulnerability.dump', # Optional: IOP
  :config_files => 'config_files.tar.gz',     # Required: Config files
  :metadata => 'metadata.yml',                # Required: Metadata
}

Note: Our foremanctl backup creates different filenames for IOP databases:

✅ We use: advisor.dump, inventory.dump, remediations.dump, vmaas.dump, vulnerability.dump
❌ foreman-maintain expects: iop_advisor.dump, iop_inventory.dump, etc.

ACTION REQUIRED: Update backup implementation to use iop_* naming for compatibility.

2. Foremanctl Environment Differences

2.1 Container vs RPM Deployment

Aspect	foreman-maintain (RPM)	foremanctl (Containers)
Package management	`yum install` RPMs	Container images (already present)
Config files	`/etc/foreman/`, `/etc/foreman-proxy/`	Podman secrets + mounted configs
Service management	`systemctl` individual services	`systemctl foreman.target` (quadlets)
Installer	`foreman-installer` script	No installer (config via Ansible playbooks)
Database access	Direct `psql`, `pg_restore`	`podman exec postgresql psql/pg_restore`
File locations	Traditional Linux paths	Container volumes + host paths

2.2 Configuration Restoration Challenges

foreman-maintain approach: Extract config_files.tar.gz to /

foremanctl challenges:

Quadlet definitions live in /etc/containers/systemd/*.container
- Overwriting these could break containerized services
- May need selective extraction or skip entirely
Secrets stored in Podman secrets (not files)
- foreman-database-url, foreman-seed-admin-password, etc.
- Cannot be extracted from tar.gz
- May need to reconstruct from backup metadata
Container-specific configs mounted as secrets
- /etc/foreman/settings.yaml → Podman secret foreman-settings-yaml
- /etc/foreman/plugins/katello.yaml → Podman secret foreman-katello-yaml
- Restoration requires updating secrets, not files
Parameters persistence in /var/lib/foremanctl/parameters.yaml
- This is foremanctl-specific
- Not present in foreman-maintain backups

2.3 No Installer in Foremanctl

foreman-maintain: Runs foreman-installer after restore to:

Regenerate configs
Re-apply settings
Run db:migrate

foremanctl alternative:

No foreman-installer command exists
Must use foremanctl deploy or equivalent to reconfigure
But deploy expects fresh system, not restored system
Challenge: How to run db:migrate and post-restore tasks?

Possible solutions:

Run db:migrate manually: podman exec foreman foreman-rake db:migrate
Create a new playbook: foremanctl post-restore (runs rake tasks only)
Use existing deploy with --skip-* flags (if they exist)

3. Foremanctl Restore Design

3.1 Proposed Architecture

Structure:

src/playbooks/restore/
├── metadata.obsah.yaml          # CLI parameter definitions
├── restore.yaml                 # Main orchestration playbook
└── tasks/
    ├── validate_backup.yaml     # Validate backup directory
    ├── validate_hostname.yaml   # Check hostname match
    ├── confirmation.yaml        # User confirmation prompt
    ├── drop_databases.yaml      # Drop existing databases
    ├── restore_databases.yaml   # Restore database dumps
    └── post_restore.yaml        # Run db:migrate, etc.

3.2 Restore Workflow (foremanctl)

Command:

foremanctl restore /path/to/backup-dir
foremanctl restore /path/to/backup-dir --dry-run
foremanctl restore /path/to/backup-dir --force-hostname-mismatch

Proposed Steps:

Phase 1: Validation

- Validate backup directory exists
- Validate metadata.yml present
- Validate required dump files present (based on metadata.databases)
- Validate hostname matches (warn if mismatch, fail without --force)
- Validate OS version compatibility (warn if different)
- Validate PostgreSQL version compatibility
- Display backup summary (databases, timestamp, source hostname)

Phase 2: Confirmation (Interactive)

- Display what will be restored:
  - Databases: foreman, candlepin, pulp, advisor, inventory, etc.
  - Source: backup timestamp, source hostname
  - Warning: THIS WILL DROP EXISTING DATABASES
- Prompt for confirmation (skip if --assumeyes or non-interactive)

Phase 3: Service Shutdown

- Stop foreman.target (all services)
- Verify services stopped

Phase 4: Database Restoration

- Start PostgreSQL service (if internal)
- Drop existing databases:
  - For each database in backup metadata.databases:
    - DROP DATABASE IF EXISTS <dbname> (via podman exec postgresql)
- Restore database dumps:
  - For each .dump file in backup directory:
    - podman exec postgresql pg_restore -C -d postgres /path/to/dump
    - Or mount backup dir and restore from container
- Verify restoration (check database exists, check table count)
- Stop PostgreSQL service (if internal)

Phase 5: Post-Restore Tasks

- Start PostgreSQL service (if internal)
- Run database migrations:
  - podman exec foreman foreman-rake db:migrate
- Run seed tasks (if needed):
  - podman exec foreman foreman-rake db:seed
- Stop PostgreSQL service (if internal)

Phase 6: Service Restart

- Start foreman.target (all services)
- Wait for services to be ready
- Verify Foreman API responds
- Display restore summary

Rescue Scenario:

rescue:
  - Start foreman.target (ensure services running)
  - Report error with backup directory path
  - Advise manual recovery steps

3.3 Database Restore Implementation

Challenge: Access PostgreSQL inside container

Option 1: Mount backup directory into container

- name: Restore Foreman database
  ansible.builtin.command:
    cmd: >
      podman exec -v {{ backup_dir }}:/backup:ro postgresql
      pg_restore -C -d postgres /backup/foreman.dump
  environment:
    PGUSER: postgres
    PGPASSWORD: "{{ postgresql_admin_password }}"

Option 2: Copy dump into container, then restore

- name: Copy dump to container
  ansible.builtin.command:
    cmd: podman cp {{ backup_dir }}/foreman.dump postgresql:/tmp/

- name: Restore dump
  ansible.builtin.command:
    cmd: >
      podman exec postgresql
      pg_restore -U postgres -C -d postgres /tmp/foreman.dump

Option 3: Stream dump into container (preferred for large dumps)

- name: Restore Foreman database
  ansible.builtin.shell:
    cmd: >
      cat {{ backup_dir }}/foreman.dump |
      podman exec -i postgresql pg_restore -U postgres -C -d postgres
  environment:
    PGPASSWORD: "{{ postgresql_admin_password }}"

Recommendation: Use Option 3 (streaming) - no disk space overhead, works with large dumps.

3.4 Handling Configuration Files

Decision: DO NOT restore config files in MVP restore.

Rationale:

Quadlet configs in /etc/containers/systemd/ are deployment-specific
Secrets are in Podman secrets, not filesystem
Restoring traditional /etc/foreman/ would conflict with containerized approach
Parameters are in /var/lib/foremanctl/parameters.yaml (foremanctl-specific)

Future enhancement:

Restore only specific configs if needed
Update Podman secrets from backup metadata
Sync parameters.yaml from backup metadata

MVP scope: Database-only restore (matches offline backup scope)

4. Metadata Compatibility

4.1 Backup Metadata Comparison

foreman-maintain metadata.yml:

hostname: server.example.com
os_version: Red Hat Enterprise Linux 8.6
rpms:
  - foreman-3.10.0
  - katello-4.13.0
  - ...
online: false
incremental: false
proxy_config:
  dns: false
  dhcp: false

foremanctl metadata.yml (our implementation):

hostname: server.example.com
os_version: Red Hat Enterprise Linux 9.2
foremanctl_version: 0.1.0-dev
online: false
incremental: false
timestamp: 20260511T140630
databases:
  - foreman
  - candlepin
  - pulp
  - advisor_db
  - inventory_db
  - remediations_db
  - vmaas_db
  - vulnerability_db
iop_enabled: true
enabled_features:
  - katello
  - iop
  - foreman-proxy
database_mode: internal
container_images:
  - quay.io/foreman/foreman:nightly
  - quay.io/foreman/pulp:foreman-nightly
  - ...

4.2 Cross-Compatibility

Can foremanctl restore from foreman-maintain backup?

✅ Partially YES - Database dumps are compatible:

foreman.dump → ✅ Can restore
candlepin.dump → ✅ Can restore
pulpcore.dump → ✅ Can restore (note: filename changed from pulp.dump)
iop_*.dump → ✅ Can restore (if present)

❌ NO - Config files are incompatible:

config_files.tar.gz contains RPM-based paths
Would conflict with containerized deployment
Solution: Skip config restoration, only restore databases

Can foreman-maintain restore from foremanctl backup?

⚠️ Partially - Database dumps compatible, but:

❌ No config_files.tar.gz (foreman-maintain requires it)
❌ Different metadata structure (may fail validation)
✅ Database dumps would work if validation bypassed

Recommendation: Support restoring foremanctl backups only (for MVP). Document incompatibility with foreman-maintain backups due to missing config files.

5. Implementation Details

5.1 CLI Parameters (metadata.obsah.yaml)

---
help: |
  Restore Foreman databases from backup directory

variables:
  backup_dir:
    parameter: backup_dir
    help: Directory containing backup files to restore
    type: AbsolutePath
    persist: false

  dry_run:
    help: Validate backup without performing restore
    action: store_true
    persist: false

  force_hostname_mismatch:
    help: Proceed even if hostname does not match backup
    action: store_true
    persist: false

  assumeyes:
    help: Automatically answer yes to confirmation prompts
    action: store_true
    persist: false

5.2 Validation Tasks (tasks/validate_backup.yaml)

---
- name: Check backup directory exists
  ansible.builtin.stat:
    path: "{{ backup_dir }}"
  register: backup_dir_stat
  failed_when: not backup_dir_stat.stat.exists or not backup_dir_stat.stat.isdir

- name: Check metadata file exists
  ansible.builtin.stat:
    path: "{{ backup_dir }}/metadata.yml"
  register: metadata_stat
  failed_when: not metadata_stat.stat.exists

- name: Read backup metadata
  ansible.builtin.slurp:
    src: "{{ backup_dir }}/metadata.yml"
  register: metadata_slurp

- name: Parse backup metadata
  ansible.builtin.set_fact:
    backup_metadata: "{{ metadata_slurp.content | b64decode | from_yaml }}"

- name: Validate required fields in metadata
  ansible.builtin.assert:
    that:
      - backup_metadata.hostname is defined
      - backup_metadata.databases is defined
      - backup_metadata.databases | length > 0
    fail_msg: "Backup metadata is missing required fields"

- name: Check database dump files exist
  ansible.builtin.stat:
    path: "{{ backup_dir }}/{{ item }}.dump"
  register: dump_files
  loop: "{{ backup_metadata.databases }}"
  failed_when: not dump_files.stat.exists

5.3 Hostname Validation (tasks/validate_hostname.yaml)

---
- name: Get current hostname
  ansible.builtin.command:
    cmd: hostname -f
  register: current_hostname
  changed_when: false

- name: Compare hostnames
  ansible.builtin.set_fact:
    hostname_matches: "{{ current_hostname.stdout == backup_metadata.hostname }}"

- name: Warn on hostname mismatch
  ansible.builtin.debug:
    msg: |
      WARNING: Hostname mismatch!
      Current: {{ current_hostname.stdout }}
      Backup:  {{ backup_metadata.hostname }}
      Restoring to a different hostname may cause issues.
  when: not hostname_matches

- name: Fail on hostname mismatch (without --force)
  ansible.builtin.fail:
    msg: |
      Hostname mismatch detected. Use --force-hostname-mismatch to proceed anyway.
      Current: {{ current_hostname.stdout }}
      Backup:  {{ backup_metadata.hostname }}
  when:
    - not hostname_matches
    - not force_hostname_mismatch | default(false)

5.4 Database Drop (tasks/drop_databases.yaml)

---
- name: Start PostgreSQL service
  ansible.builtin.systemd:
    name: postgresql.service
    state: started
  when: database_mode == 'internal'

- name: Wait for PostgreSQL readiness
  ansible.builtin.command:
    cmd: pg_isready -h {{ database_host }} -p {{ database_port }}
  register: pg_ready
  retries: 10
  delay: 2
  until: pg_ready.rc == 0
  changed_when: false

- name: Drop databases from backup
  ansible.builtin.command:
    cmd: >
      podman exec postgresql psql -U postgres -c
      'DROP DATABASE IF EXISTS {{ item }}'
  loop: "{{ backup_metadata.databases }}"
  changed_when: true
  register: drop_result
  failed_when: false  # Don't fail if database doesn't exist

- name: Display dropped databases
  ansible.builtin.debug:
    msg: "Dropped {{ backup_metadata.databases | length }} database(s)"

5.5 Database Restore (tasks/restore_databases.yaml)

---
# Map database names to dump filenames
# backup_metadata.databases contains: [foreman, candlepin, pulp, advisor_db, ...]
# dump files are: foreman.dump, candlepin.dump, pulp.dump, advisor.dump, ...

- name: Build database to dump file mapping
  ansible.builtin.set_fact:
    db_dump_map:
      foreman: foreman.dump
      candlepin: candlepin.dump
      pulp: pulp.dump
      advisor_db: advisor.dump
      inventory_db: inventory.dump
      remediations_db: remediations.dump
      vmaas_db: vmaas.dump
      vulnerability_db: vulnerability.dump

- name: Restore database dumps
  ansible.builtin.shell:
    cmd: >
      cat {{ backup_dir }}/{{ db_dump_map[item] }} |
      podman exec -i postgresql
      pg_restore -U postgres -C -d postgres
  loop: "{{ backup_metadata.databases }}"
  when: item in db_dump_map
  environment:
    PGPASSWORD: "{{ postgresql_admin_password }}"
  register: restore_result
  changed_when: true

- name: Verify databases restored
  ansible.builtin.command:
    cmd: >
      podman exec postgresql psql -U postgres -c
      '\l {{ item }}'
  loop: "{{ backup_metadata.databases }}"
  register: verify_result
  changed_when: false
  failed_when: "'does not exist' in verify_result.stderr"

- name: Display restore summary
  ansible.builtin.debug:
    msg: "Restored {{ backup_metadata.databases | length }} database(s) successfully"

5.6 Post-Restore Tasks (tasks/post_restore.yaml)

---
- name: Start PostgreSQL service (for migrations)
  ansible.builtin.systemd:
    name: postgresql.service
    state: started
  when: database_mode == 'internal'

- name: Run database migrations
  ansible.builtin.command:
    cmd: podman exec foreman foreman-rake db:migrate
  register: migrate_result
  changed_when: "'Migrating' in migrate_result.stdout"

- name: Run db:seed if needed (for fresh restore)
  ansible.builtin.command:
    cmd: podman exec foreman foreman-rake db:seed
  when: migrate_result is changed
  register: seed_result
  changed_when: true
  failed_when: false  # Seed may fail if already seeded

- name: Stop PostgreSQL service
  ansible.builtin.systemd:
    name: postgresql.service
    state: stopped
  when: database_mode == 'internal'

5.7 Main Restore Playbook (restore.yaml)

---
- name: Restore Foreman databases from backup
  hosts: quadlet
  become: true
  gather_facts: true
  vars_files:
    - "../../vars/database.yml"
    - "../../vars/database_iop.yml"
    - "../../vars/foreman.yml"

  vars:
    service_stopped: false

  tasks:
    - name: Validation phase
      block:
        - name: Validate backup directory
          ansible.builtin.include_tasks:
            file: tasks/validate_backup.yaml

        - name: Validate hostname
          ansible.builtin.include_tasks:
            file: tasks/validate_hostname.yaml

        - name: Display backup summary
          ansible.builtin.debug:
            msg: |
              Backup Summary:
              - Source: {{ backup_metadata.hostname }}
              - Date: {{ backup_metadata.timestamp }}
              - Databases: {{ backup_metadata.databases | join(', ') }}
              - IOP: {{ 'Yes' if backup_metadata.iop_enabled else 'No' }}

    - name: Dry run exit
      when: dry_run | default(false)
      block:
        - name: Dry run successful
          ansible.builtin.debug:
            msg: "Dry run completed. Backup is valid and can be restored."
        - name: End playbook
          ansible.builtin.meta: end_play

    - name: Confirmation
      when: not assumeyes | default(false)
      ansible.builtin.pause:
        prompt: |
          WARNING: This will DROP and REPLACE the following databases:
          {{ backup_metadata.databases | join(', ') }}

          All existing data in these databases will be LOST.

          Do you want to continue? (yes/no)
      register: confirmation
      failed_when: confirmation.user_input | lower != 'yes'

    - name: Restore operations
      block:
        - name: Stop Foreman services
          ansible.builtin.systemd:
            name: foreman.target
            state: stopped

        - name: Mark services as stopped
          ansible.builtin.set_fact:
            service_stopped: true

        - name: Drop existing databases
          ansible.builtin.include_tasks:
            file: tasks/drop_databases.yaml

        - name: Restore database dumps
          ansible.builtin.include_tasks:
            file: tasks/restore_databases.yaml

        - name: Run post-restore tasks
          ansible.builtin.include_tasks:
            file: tasks/post_restore.yaml

        - name: Start Foreman services
          ansible.builtin.systemd:
            name: foreman.target
            state: started

        - name: Mark services as started
          ansible.builtin.set_fact:
            service_stopped: false

        - name: Wait for services to be ready
          ansible.builtin.uri:
            url: "https://{{ ansible_fqdn }}/api/status"
            validate_certs: false
            status_code: 200
          register: api_status
          retries: 30
          delay: 10
          until: api_status.status == 200

        - name: Display restore completion
          ansible.builtin.debug:
            msg: |
              Restore completed successfully!
              - Restored {{ backup_metadata.databases | length }} database(s)
              - Services are running
              - Foreman API is responding

      rescue:
        - name: Restore Foreman services on failure
          ansible.builtin.systemd:
            name: foreman.target
            state: started
          when: service_stopped | default(false)
          ignore_errors: true

        - name: Report failure
          ansible.builtin.fail:
            msg: |
              Restore failed: {{ ansible_failed_result.msg | default('Unknown error') }}
              Services have been restarted.
              Manual intervention may be required.
              Check logs: journalctl -u foreman.service

6. Critical Challenges and Solutions

Challenge 1: Database Name vs Dump Filename Mismatch

Problem:

Database name in PostgreSQL: advisor_db
Dump filename in backup: advisor.dump (not advisor_db.dump)
Metadata lists database names, not filenames

Solution: Create a mapping dictionary in restore playbook:

db_dump_map:
  foreman: foreman.dump
  candlepin: candlepin.dump
  pulp: pulp.dump
  advisor_db: advisor.dump
  inventory_db: inventory.dump
  remediations_db: remediations.dump
  vmaas_db: vmaas.dump
  vulnerability_db: vulnerability.dump

Better solution: Update backup implementation to use consistent naming:

Either use DB names: advisor_db.dump, inventory_db.dump
Or update metadata to include both DB name and filename

Challenge 2: Podman Exec vs Direct psql/pg_restore

Problem:

foreman-maintain runs: pg_restore directly on host
foremanctl must run: podman exec postgresql pg_restore
Passing large dumps via stdin to podman exec -i

Solution: Use shell piping with stdin redirect:

cat /path/to/backup/foreman.dump | podman exec -i postgresql pg_restore -U postgres -C -d postgres

This streams the dump into the container without copying the entire file.

Alternative: Mount backup directory as volume:

podman exec -v /path/to/backup:/backup:ro postgresql pg_restore -U postgres -C -d postgres /backup/foreman.dump

Challenge 3: No foreman-installer Equivalent

Problem:

foreman-maintain runs foreman-installer after restore
foremanctl has no installer command
Need to run db:migrate and setup tasks

Solution: Run rake tasks directly inside container:

- name: Run migrations
  ansible.builtin.command:
    cmd: podman exec foreman foreman-rake db:migrate

- name: Seed database (if needed)
  ansible.builtin.command:
    cmd: podman exec foreman foreman-rake db:seed

Alternative: Create a foremanctl post-restore command that runs minimal setup:

db:migrate
db:seed (if needed)
Clear caches
Restart services

Challenge 4: Secrets Restoration

Problem:

Podman secrets cannot be read or extracted
Backup doesn't contain secrets (only database dumps)
After restore, secrets may be out of sync with database

Solution (MVP):

Don't restore secrets - assume existing secrets are compatible
Document requirement: restore only works on same system (or with same secrets)
User must manually update secrets if restoring to different system

Future enhancement: Include encrypted secrets in backup metadata:

# In metadata.yml (encrypted section)
secrets:
  foreman_admin_password: <encrypted>
  database_password: <encrypted>

Then restore secrets with:

echo "$password" | podman secret create foreman-seed-admin-password -

Challenge 5: Hostname Change Handling

Problem:

Foreman stores its hostname in database (settings table)
Restoring to different hostname breaks:
- SSL certificates
- Pulp content URLs
- Smart proxy registration

Solution:

Validate hostname matches (default behavior)
Allow mismatch with --force-hostname-mismatch flag
Update hostname in database after restore:

UPDATE settings SET value = 'new.hostname.com' WHERE name = 'foreman_url';

Recommended approach:

Fail by default on hostname mismatch
Provide clear error message with remediation steps
Document hostname change procedure separately

Challenge 6: IOP Database Filename Compatibility

Problem: Our backup creates:

advisor.dump, inventory.dump, remediations.dump, vmaas.dump, vulnerability.dump

foreman-maintain expects:

iop_advisor.dump, iop_inventory.dump, etc.

Solution: Update backup implementation to match foreman-maintain naming:

In tasks/database_dumps.yaml:

- name: Dump IOP Advisor database
  ansible.builtin.command:
    cmd: pg_dump ... -f {{ backup_dir_full }}/iop_advisor.dump  # Changed from advisor.dump

This ensures cross-compatibility with foreman-maintain.

7. Testing Strategy

7.1 Test Scenarios

Test 1: Basic Database-Only Restore

# Create backup
foremanctl backup /tmp/test-backup

# Simulate restore on same system
foremanctl restore /tmp/test-backup/foreman-backup-*/ --assumeyes

# Verify
- All services running
- Databases contain data
- Foreman API responds
- Can log into web UI

Test 2: Dry Run Validation

# Create backup
foremanctl backup /tmp/test-backup

# Test dry run
foremanctl restore /tmp/test-backup/foreman-backup-*/ --dry-run

# Expected: Validation passes, no changes made

Test 3: Hostname Mismatch

# Create backup on system A (hostname: server-a.example.com)
foremanctl backup /tmp/test-backup

# Copy to system B (hostname: server-b.example.com)
foremanctl restore /tmp/test-backup/foreman-backup-*/

# Expected: Fails with hostname mismatch error

# Force restore
foremanctl restore /tmp/test-backup/foreman-backup-*/ --force-hostname-mismatch --assumeyes

# Expected: Succeeds with warnings

Test 4: Partial Backup (No IOP)

# System without IOP
foremanctl backup /tmp/test-backup

# Restore on IOP-enabled system
foremanctl restore /tmp/test-backup/foreman-backup-*/ --assumeyes

# Expected: Restores only core databases, skips IOP

Test 5: Full Backup (With IOP)

# System with IOP
foremanctl backup /tmp/test-backup

# Restore
foremanctl restore /tmp/test-backup/foreman-backup-*/ --assumeyes

# Expected: Restores all 8 databases

Test 6: Interrupted Restore (Rescue Scenario)

# Simulate failure during restore (e.g., kill process)
foremanctl restore /tmp/test-backup/foreman-backup-*/ &
sleep 30 && kill %1

# Expected: Services restarted, clear error message
systemctl is-active foreman.target
# Should be: active

7.2 Verification Commands

# After restore, verify:

# 1. Services running
systemctl status foreman.target

# 2. Databases exist
podman exec postgresql psql -U postgres -c '\l' | grep -E "(foreman|candlepin|pulp|advisor|inventory)"

# 3. Database size (should match backup)
podman exec postgresql psql -U postgres -c "SELECT datname, pg_size_pretty(pg_database_size(datname)) FROM pg_database WHERE datname IN ('foreman', 'candlepin', 'pulp');"

# 4. Foreman API responds
curl -k https://localhost/api/status

# 5. Can log in
curl -k -u admin:changeme https://localhost/api/v2/hosts

# 6. Check migrations applied
podman exec foreman foreman-rake db:version

# 7. Check for errors
journalctl -u foreman.service --since "5 minutes ago" | grep -i error

8. Implementation Roadmap

Phase 1: MVP - Database-Only Restore

Create restore playbook structure
Implement validation tasks (backup, hostname)
Implement database drop
Implement database restore (streaming method)
Implement post-restore tasks (db:migrate)
Update backup to use iop_* naming for IOP dumps
Test on same-system restore
Document limitations (no config restore)

Phase 2: Enhanced Validation

Add PostgreSQL version check
Add OS version compatibility check
Add disk space validation
Add network interface validation (if proxy features)
Improve error messages with remediation steps

Phase 3: Cross-System Restore

Implement hostname change handling
Add certificate regeneration support
Document manual steps for different hostname
Test restore to different system

Phase 4: Config Restoration (Future)

Design selective config restoration
Implement Podman secret updates from backup
Restore parameters.yaml
Restore quadlet definitions (carefully)

Phase 5: Advanced Features (Future)

Incremental restore support
Restore from foreman-maintain backups (config skipping)
Encrypted backup support
Remote backup source (S3, NFS)
Point-in-time recovery (if incremental backups)

9. Acceptance Criteria

9.1 MVP Success Criteria

Restore command works:

foremanctl restore /path/to/backup succeeds
All databases from backup are restored
Services restart automatically after restore
Foreman API is accessible after restore
Can log into web UI after restore
db:migrate runs successfully

Validation works:

--dry-run validates without making changes
Hostname mismatch detected and fails (without --force)
Missing backup files detected with clear error
Invalid metadata detected with clear error

Error handling works:

Services restart even if restore fails
Clear error messages on failure
Rescue scenario leaves system in safe state

Documentation complete:

Usage examples in help text
Limitations documented (database-only)
Same-system restore tested and documented
Cross-system restore limitations documented

9.2 Future Enhancement Criteria

Phase 2+:

Config files can be selectively restored
Secrets can be updated from backup
Hostname changes handled automatically
Can restore foreman-maintain backups (database portion)
Incremental restore supported

10. Known Limitations (MVP)

Database-only restore - Does not restore:
- Configuration files
- Pulp content (/var/lib/pulp/)
- Podman secrets
- Quadlet definitions
Same-system restore only - Hostname must match (or use --force)
- Different hostname requires manual certificate regeneration
- Smart proxy registration may break
No incremental restore - Only full restores supported
No config restoration - User must manually:
- Update secrets if needed
- Regenerate certificates if hostname changed
- Update parameters.yaml if needed
PostgreSQL version compatibility - Restore may fail if:
- Source and target PostgreSQL versions differ significantly
- Custom extensions in source not available in target
No rollback - Once databases are dropped, cannot undo
- Recommend testing with --dry-run first
- Keep original backup safe

11. Comparison Matrix

Feature	foreman-maintain restore	foremanctl restore (MVP)	foremanctl restore (Future)
Database restore	✅ All DBs	✅ All DBs	✅ All DBs
Config files	✅ Full restore	❌ Not restored	⚠️ Selective restore
Pulp content	✅ Optional restore	❌ Not restored	✅ Optional restore
Package install	✅ RPM install	➖ N/A (containers)	➖ N/A
Installer run	✅ foreman-installer	➖ db:migrate only	⚠️ Post-restore tasks
Dry run	✅ Supported	✅ Supported	✅ Supported
Incremental	✅ Supported	❌ Not supported	✅ Planned
Hostname change	⚠️ Manual steps	⚠️ Manual steps	✅ Automated
Secrets restore	➖ N/A (file-based)	❌ Not restored	✅ Planned
Cross-compat	➖ RPM→RPM only	⚠️ foremanctl→foremanctl	⚠️ Limited foreman-maintain compat

12. Risk Assessment

High Risk

Data loss during restore - Dropping databases is irreversible
- Mitigation: Require explicit confirmation, --dry-run testing, backup verification
Service downtime longer than expected - Restore takes time
- Mitigation: Display estimated time, progress indicators, timeout handling
Incompatible PostgreSQL dumps - Version mismatches cause failures
- Mitigation: Validate PostgreSQL version, clear error messages

Medium Risk

Hostname mismatch issues - Restoring to different hostname breaks features
- Mitigation: Validate hostname, document manual fix steps, --force flag
Secret/config mismatch - Secrets don't match restored database
- Mitigation: Document same-system requirement, validate before restore
Disk space exhaustion - Large database restores fill disk
- Mitigation: Check available space before restore, stream dumps

Low Risk

Metadata parsing errors - Malformed metadata.yml
- Mitigation: Robust YAML parsing, validation with clear errors
Container not running - PostgreSQL container stopped/missing
- Mitigation: Check container status before operations, auto-start if needed

13. Summary and Recommendations

Recommended Approach for MVP

Start with database-only restore - Core functionality, manageable scope
Ensure compatibility with foremanctl backups - Same tool, same format
Skip config restoration - Too complex for MVP, quadlet-specific challenges
Focus on same-system restore - Avoid hostname/cert complexity initially
Strong validation - Prevent data loss through thorough pre-flight checks
Clear documentation - Set user expectations about limitations

Critical Action Items

Update backup implementation:
- Change IOP dump filenames: advisor.dump → iop_advisor.dump
- Ensures compatibility with foreman-maintain metadata expectations
Create db_dump_map:
- Map database names to dump filenames
- Handle name mismatches gracefully
Implement streaming restore:
- Use cat | podman exec -i postgresql pg_restore
- Avoid disk space issues with large dumps
Add robust validation:
- Hostname check with --force override
- Backup completeness check
- PostgreSQL version compatibility check
Test thoroughly:
- Same-system restore (primary use case)
- Dry run validation
- Error recovery scenarios

Success Criteria

✅ User can restore a foremanctl backup to the same system
✅ All databases are restored correctly
✅ Services restart automatically
✅ Foreman is functional after restore
✅ Clear error messages on failures
✅ Documented limitations and workarounds

Future Work (Post-MVP)

Selective config restoration
Hostname change automation
Secret updates from backup
Incremental restore support
Cross-compatibility with foreman-maintain backups

End of Evaluation

Appendix A: Example Usage

Basic Restore

# Restore from backup directory
foremanctl restore /tmp/foreman-backup-20260511T140630/

# Interactive confirmation prompt appears
# Services stop → Databases drop → Restore → Migrate → Services start

Dry Run

# Validate backup without restoring
foremanctl restore /tmp/foreman-backup-20260511T140630/ --dry-run

# Output:
# ✓ Backup directory valid
# ✓ Metadata present
# ✓ All required dump files found
# ✓ Hostname matches
# Dry run successful. Backup can be restored.

Force Hostname Mismatch

# Restore to different hostname
foremanctl restore /tmp/foreman-backup-20260511T140630/ --force-hostname-mismatch --assumeyes

# Warning displayed about hostname mismatch
# Proceeds with restore anyway

Non-Interactive

# Automated restore (no prompts)
foremanctl restore /tmp/foreman-backup-20260511T140630/ --assumeyes

Appendix B: Error Scenarios and Messages

Scenario 1: Backup Directory Not Found

Error: Backup directory does not exist: /tmp/nonexistent/
Check the path and try again.

Scenario 2: Missing Metadata

Error: Backup metadata file not found: /tmp/backup/metadata.yml
This does not appear to be a valid foremanctl backup directory.

Scenario 3: Hostname Mismatch

Error: Hostname mismatch detected.
Current hostname: server-b.example.com
Backup hostname:  server-a.example.com

Restoring to a different hostname may cause issues with:
- SSL certificates
- Smart proxy registration
- Pulp content URLs

To proceed anyway, use: --force-hostname-mismatch

Scenario 4: Missing Database Dump

Error: Required database dump file missing: /tmp/backup/foreman.dump
Backup appears incomplete. Cannot proceed with restore.

Scenario 5: Restore Failed

Error: Database restore failed: pg_restore exited with code 1

Services have been restarted to prevent prolonged downtime.
Check PostgreSQL logs for details: journalctl -u postgresql.service

The system may be in an inconsistent state.
Recommendation: Restore from a known-good backup.

Appendix C: File Structure Comparison

foreman-maintain backup directory:

foreman-backup-2026-05-11-14-06-30/
├── config_files.tar.gz       # Required: /etc configs
├── foreman.dump              # Required: Foreman DB
├── candlepin.dump            # Required: Candlepin DB
├── pulpcore.dump             # Required: Pulp DB
├── iop_advisor.dump          # Optional: IOP
├── iop_inventory.dump        # Optional: IOP
├── iop_remediations.dump     # Optional: IOP
├── iop_vmaas.dump            # Optional: IOP
├── iop_vulnerability.dump    # Optional: IOP
├── pulp_data.tar             # Optional: Pulp content
└── metadata.yml              # Required: Metadata

foremanctl backup directory (current):

foreman-backup-20260511T140630/
├── foreman.dump              # ✅ Core DB
├── candlepin.dump            # ✅ Core DB
├── pulp.dump                 # ✅ Core DB
├── advisor.dump              # ❌ Should be iop_advisor.dump
├── inventory.dump            # ❌ Should be iop_inventory.dump
├── remediations.dump         # ❌ Should be iop_remediations.dump
├── vmaas.dump                # ❌ Should be iop_vmaas.dump
├── vulnerability.dump        # ❌ Should be iop_vulnerability.dump
└── metadata.yml              # ✅ Metadata

foremanctl backup directory (corrected):

foreman-backup-20260511T140630/
├── foreman.dump              # ✅ Core DB
├── candlepin.dump            # ✅ Core DB
├── pulp.dump                 # ✅ Core DB (or pulpcore.dump for compat?)
├── iop_advisor.dump          # ✅ IOP DB (renamed)
├── iop_inventory.dump        # ✅ IOP DB (renamed)
├── iop_remediations.dump     # ✅ IOP DB (renamed)
├── iop_vmaas.dump            # ✅ IOP DB (renamed)
├── iop_vulnerability.dump    # ✅ IOP DB (renamed)
└── metadata.yml              # ✅ Metadata

Document Version: 1.0
Status: Draft for Review
Next Steps: Review with team, prioritize MVP features, begin implementation

sjha4/foremanctl-restore-evaluation.md

Foremanctl Restore Implementation Evaluation

Executive Summary

1. Foreman-Maintain Restore Analysis

1.1 Restore Workflow (foreman-maintain)

1.2 Database Restore Logic

1.3 File Mapping (Utils::Backup)

2. Foremanctl Environment Differences

2.1 Container vs RPM Deployment

2.2 Configuration Restoration Challenges

2.3 No Installer in Foremanctl

3. Foremanctl Restore Design

3.1 Proposed Architecture

3.2 Restore Workflow (foremanctl)

Phase 1: Validation

Phase 2: Confirmation (Interactive)

Phase 3: Service Shutdown

Phase 4: Database Restoration

Phase 5: Post-Restore Tasks

Phase 6: Service Restart

3.3 Database Restore Implementation

3.4 Handling Configuration Files

4. Metadata Compatibility

4.1 Backup Metadata Comparison

4.2 Cross-Compatibility

5. Implementation Details

5.1 CLI Parameters (metadata.obsah.yaml)

5.2 Validation Tasks (tasks/validate_backup.yaml)

5.3 Hostname Validation (tasks/validate_hostname.yaml)

5.4 Database Drop (tasks/drop_databases.yaml)

5.5 Database Restore (tasks/restore_databases.yaml)

5.6 Post-Restore Tasks (tasks/post_restore.yaml)

5.7 Main Restore Playbook (restore.yaml)

6. Critical Challenges and Solutions

Challenge 1: Database Name vs Dump Filename Mismatch

Challenge 2: Podman Exec vs Direct psql/pg_restore

Challenge 3: No foreman-installer Equivalent

Challenge 4: Secrets Restoration

Challenge 5: Hostname Change Handling

Challenge 6: IOP Database Filename Compatibility

7. Testing Strategy

7.1 Test Scenarios

7.2 Verification Commands

8. Implementation Roadmap

Phase 1: MVP - Database-Only Restore

Phase 2: Enhanced Validation

Phase 3: Cross-System Restore

Phase 4: Config Restoration (Future)

Phase 5: Advanced Features (Future)

9. Acceptance Criteria

9.1 MVP Success Criteria

9.2 Future Enhancement Criteria

10. Known Limitations (MVP)

11. Comparison Matrix

12. Risk Assessment

High Risk

Medium Risk

Low Risk

13. Summary and Recommendations

Recommended Approach for MVP

Critical Action Items

Success Criteria

Future Work (Post-MVP)

Appendix A: Example Usage

Basic Restore

Dry Run

Force Hostname Mismatch

Non-Interactive

Appendix B: Error Scenarios and Messages

Scenario 1: Backup Directory Not Found

Scenario 2: Missing Metadata

Scenario 3: Hostname Mismatch

Scenario 4: Missing Database Dump

Scenario 5: Restore Failed

Appendix C: File Structure Comparison