Date: 2026-05-11
Focus: Offline backup implementation for foremanctl (porting from foreman-maintain)
The current plan in foremanctl-backup-implementation-plan.md provides a solid foundation but has critical gaps around IOP (Insights Operator Platform) database handling. This evaluation identifies these gaps and proposes a corrected design aligned with the actual running services on this system.
# Core Foreman Services (managed by foreman.target)
- foreman.service # Main Foreman Rails app
- postgresql.service # PostgreSQL 13 database
- redis.service # Redis cache
- candlepin.service # Subscription management
- foreman-proxy.service # Smart proxy
- pulp-api.service # Pulp API
- pulp-content.service # Pulp content delivery
- pulp-worker@{1-6}.service # 6 Pulp workers
- dynflow-sidekiq@*.service # Background job workers
# IOP Services (19 total, also in foreman.target)
- iop-core-engine.service
- iop-core-gateway.service
- iop-core-host-inventory*.service
- iop-core-ingress.service
- iop-core-kafka.service
- iop-core-puptoo.service
- iop-core-yuptoo.service
- iop-service-advisor-backend*.service
- iop-service-remediations-api.service
- iop-service-vmaas*.service
- iop-service-vuln-*.serviceKey Finding: foreman.target includes ALL services (core + IOP). Stopping foreman.target stops everything.
postgres=# \l
Name | Owner |
------------------+---------------------+
foreman | foreman | # Core: Foreman main DB
candlepin | candlepin | # Core: Subscription mgmt
pulp | pulp | # Core: Content mgmt
advisor_db | advisor_user | # IOP: Advisor service
inventory_db | inventory_admin | # IOP: Host inventory
remediations_db | remediations_user | # IOP: Remediations
vmaas_db | vmaas_admin | # IOP: VMaaS
vulnerability_db | vulnerability_admin | # IOP: VulnerabilityTotal databases to backup: 8 (3 core + 5 IOP)
Core databases:
- Config:
/usr/share/foremanctl/src/vars/database.yml - Host:
localhost:5432 - Databases: foreman, candlepin, pulp
- Users/passwords defined per database
IOP databases:
- Config:
/usr/share/foremanctl/src/vars/database_iop.yml - Host:
host.containers.internal:5432(resolves to localhost) - Databases: advisor_db, inventory_db, remediations_db, vmaas_db, vulnerability_db
- Users/passwords defined per database
Conditional deployment:
- IOP databases only exist when
'iop' in enabled_features - Need runtime detection to handle systems with/without IOP
- Service orchestration: Correctly uses
foreman.targetfor service management - Metadata before shutdown: Generates metadata before stopping services (matches foreman-maintain)
- PostgreSQL restart: Correctly restarts postgresql.service for offline dumps
- Database format: Uses
pg_dump -Fc(PostgreSQL custom format, compressed) - Error handling: Block/rescue pattern ensures services always restart
- Architecture: Follows foremanctl playbook patterns (obsah, tasks/, metadata.obsah.yaml)
- Task checking: Includes foreman and pulp task preflight checks
- Directory structure: Timestamped backup directories with proper permissions
Issue: Plan only mentions 3 databases (foreman, candlepin, pulpcore) but this system has 8 databases (3 core + 5 IOP).
Impact: Backup would be incomplete, losing IOP data:
- Advisor recommendations
- Host inventory
- Remediation plans
- VMaaS vulnerability data
- Vulnerability assessments
Evidence from foreman-maintain:
# definitions/scenarios/backup.rb lines 106-116
def add_database_backup_steps
add_steps_with_context(
Procedures::Backup::Online::CandlepinDB,
Procedures::Backup::Online::ForemanDB,
Procedures::Backup::Online::IopAdvisorDB, # ← IOP databases
Procedures::Backup::Online::IopInventoryDB, # ← IOP databases
Procedures::Backup::Online::IopRemediationsDB, # ← IOP databases
Procedures::Backup::Online::IopVmaasDB, # ← IOP databases
Procedures::Backup::Online::IopVulnerabilityDB, # ← IOP databases
Procedures::Backup::Online::PulpcoreDB
)
endIssue: Plan's vars_files only loads database.yml and foreman.yml, missing database_iop.yml.
Fix needed:
vars_files:
- "../../vars/database.yml"
- "../../vars/database_iop.yml" # ← ADD THIS
- "../../vars/foreman.yml"Issue: tasks/database_detection.yaml in the plan only checks for foreman, candlepin, pulpcore, container_gateway.
Missing checks: advisor_db, inventory_db, remediations_db, vmaas_db, vulnerability_db
Required pattern:
# Check each of the 8 databases
- Check foreman exists
- Check candlepin exists
- Check pulp exists
- Check advisor_db exists # ← ADD
- Check inventory_db exists # ← ADD
- Check remediations_db exists # ← ADD
- Check vmaas_db exists # ← ADD
- Check vulnerability_db exists # ← ADD
# Build list of all existing databases
- databases_to_backup: [list of all found DBs]Issue: tasks/database_dumps.yaml only dumps foreman, candlepin, pulp.
Missing dumps: 5 IOP databases
Required additions:
# Dump advisor_db if exists
- name: Dump IOP Advisor database
when: advisor_db_exists
ansible.builtin.command:
cmd: pg_dump -h {{ iop_advisor_database_host }} -p {{ iop_advisor_database_port }}
-U {{ iop_advisor_database_user }} -Fc
-f {{ backup_dir_full }}/advisor.dump {{ iop_advisor_database_name }}
environment:
PGPASSWORD: "{{ iop_advisor_database_password }}"
# Repeat for inventory_db, remediations_db, vmaas_db, vulnerability_dbIssue: Metadata doesn't capture IOP database presence.
Fix needed:
backup_metadata:
# ... existing fields ...
databases: "{{ databases_to_backup }}" # Will now include IOP DBs
iop_enabled: "{{ 'iop' in enabled_features }}" # Track IOP feature stateApproach: Extend the existing plan to handle IOP databases conditionally.
Key principles:
- Runtime detection: Check which databases actually exist (not all systems have IOP)
- Backward compatibility: Works on systems with or without IOP
- Follows foreman-maintain: Matches foreman-maintain's backup behavior exactly
- Offline backup only: Focus on offline backup (as requested)
src/playbooks/backup/
├── metadata.obsah.yaml # CLI parameters (unchanged from plan)
├── backup.yaml # Main playbook (UPDATED: load database_iop.yml)
└── tasks/
├── preflight.yaml # Task checking (unchanged from plan)
├── database_detection.yaml # Database detection (UPDATED: add 5 IOP DBs)
├── database_dumps.yaml # pg_dump operations (UPDATED: add 5 IOP DBs)
└── metadata.yaml # Metadata generation (UPDATED: add IOP flag)
---
- name: Backup Foreman databases and configuration
hosts: quadlet
become: true
gather_facts: true
vars_files:
- "../../vars/database.yml"
- "../../vars/database_iop.yml" # ← ADD THIS
- "../../vars/foreman.yml"Rationale: IOP database connection info lives in database_iop.yml. Even on non-IOP systems, loading this file is safe (vars just won't be used).
Current plan covers: foreman, candlepin, pulpcore, container_gateway (4 DBs)
Add detection for:
- advisor_db (IOP Advisor)
- inventory_db (IOP Inventory)
- remediations_db (IOP Remediations)
- vmaas_db (IOP VMaaS)
- vulnerability_db (IOP Vulnerability)
Implementation pattern (repeat for each DB):
# Example for advisor_db
- name: Check if IOP Advisor database exists
community.postgresql.postgresql_query:
db: postgres
login_host: "{{ iop_advisor_database_host }}"
login_port: "{{ iop_advisor_database_port }}"
login_user: postgres
login_password: "{{ postgresql_admin_password }}"
query: SELECT 1 FROM pg_database WHERE datname = '{{ iop_advisor_database_name }}'
register: advisor_db_check
failed_when: false # Don't fail if DB doesn't exist
- name: Set advisor_db existence fact
ansible.builtin.set_fact:
advisor_db_exists: "{{ advisor_db_check.query_result | default([]) | length > 0 }}"Build final list:
- name: Build complete database backup list
ansible.builtin.set_fact:
databases_to_backup: >-
{{
(foreman_database_exists | ternary([foreman_database_name], [])) +
(candlepin_database_exists | ternary([candlepin_database_name], [])) +
(pulpcore_database_exists | ternary([pulp_database_name], [])) +
(advisor_db_exists | ternary([iop_advisor_database_name], [])) +
(inventory_db_exists | ternary([iop_inventory_database_name], [])) +
(remediations_db_exists | ternary([iop_remediation_database_name], [])) +
(vmaas_db_exists | ternary([iop_vmaas_database_name], [])) +
(vulnerability_db_exists | ternary([iop_vulnerability_database_name], []))
}}Add 5 IOP database dumps:
# 1. IOP Advisor Database
- name: Dump IOP Advisor database
when: advisor_db_exists | default(false)
ansible.builtin.command:
cmd: >
pg_dump
-h {{ iop_advisor_database_host }}
-p {{ iop_advisor_database_port }}
-U {{ iop_advisor_database_user }}
-Fc
-f {{ backup_dir_full }}/advisor.dump
{{ iop_advisor_database_name }}
environment:
PGPASSWORD: "{{ iop_advisor_database_password }}"
- name: Verify advisor dump file
when: advisor_db_exists | default(false)
ansible.builtin.stat:
path: "{{ backup_dir_full }}/advisor.dump"
register: advisor_dump_stat
- name: Assert advisor dump succeeded
when: advisor_db_exists | default(false)
ansible.builtin.assert:
that:
- advisor_dump_stat.stat.exists
- advisor_dump_stat.stat.size > 0
fail_msg: "IOP Advisor database dump failed or produced empty file"
# 2. IOP Inventory Database
- name: Dump IOP Inventory database
when: inventory_db_exists | default(false)
ansible.builtin.command:
cmd: >
pg_dump
-h {{ iop_inventory_database_host }}
-p {{ iop_inventory_database_port }}
-U {{ iop_inventory_database_user }}
-Fc
-f {{ backup_dir_full }}/inventory.dump
{{ iop_inventory_database_name }}
environment:
PGPASSWORD: "{{ iop_inventory_database_password }}"
- name: Verify inventory dump file
when: inventory_db_exists | default(false)
ansible.builtin.stat:
path: "{{ backup_dir_full }}/inventory.dump"
register: inventory_dump_stat
- name: Assert inventory dump succeeded
when: inventory_db_exists | default(false)
ansible.builtin.assert:
that:
- inventory_dump_stat.stat.exists
- inventory_dump_stat.stat.size > 0
fail_msg: "IOP Inventory database dump failed or produced empty file"
# 3. IOP Remediations Database
- name: Dump IOP Remediations database
when: remediations_db_exists | default(false)
ansible.builtin.command:
cmd: >
pg_dump
-h {{ iop_remediation_database_host }}
-p {{ iop_remediation_database_port }}
-U {{ iop_remediation_database_user }}
-Fc
-f {{ backup_dir_full }}/remediations.dump
{{ iop_remediation_database_name }}
environment:
PGPASSWORD: "{{ iop_remediation_database_password }}"
- name: Verify remediations dump file
when: remediations_db_exists | default(false)
ansible.builtin.stat:
path: "{{ backup_dir_full }}/remediations.dump"
register: remediations_dump_stat
- name: Assert remediations dump succeeded
when: remediations_db_exists | default(false)
ansible.builtin.assert:
that:
- remediations_dump_stat.stat.exists
- remediations_dump_stat.stat.size > 0
fail_msg: "IOP Remediations database dump failed or produced empty file"
# 4. IOP VMaaS Database
- name: Dump IOP VMaaS database
when: vmaas_db_exists | default(false)
ansible.builtin.command:
cmd: >
pg_dump
-h {{ iop_vmaas_database_host }}
-p {{ iop_vmaas_database_port }}
-U {{ iop_vmaas_database_user }}
-Fc
-f {{ backup_dir_full }}/vmaas.dump
{{ iop_vmaas_database_name }}
environment:
PGPASSWORD: "{{ iop_vmaas_database_password }}"
- name: Verify vmaas dump file
when: vmaas_db_exists | default(false)
ansible.builtin.stat:
path: "{{ backup_dir_full }}/vmaas.dump"
register: vmaas_dump_stat
- name: Assert vmaas dump succeeded
when: vmaas_db_exists | default(false)
ansible.builtin.assert:
that:
- vmaas_dump_stat.stat.exists
- vmaas_dump_stat.stat.size > 0
fail_msg: "IOP VMaaS database dump failed or produced empty file"
# 5. IOP Vulnerability Database
- name: Dump IOP Vulnerability database
when: vulnerability_db_exists | default(false)
ansible.builtin.command:
cmd: >
pg_dump
-h {{ iop_vulnerability_database_host }}
-p {{ iop_vulnerability_database_port }}
-U {{ iop_vulnerability_database_user }}
-Fc
-f {{ backup_dir_full }}/vulnerability.dump
{{ iop_vulnerability_database_name }}
environment:
PGPASSWORD: "{{ iop_vulnerability_database_password }}"
- name: Verify vulnerability dump file
when: vulnerability_db_exists | default(false)
ansible.builtin.stat:
path: "{{ backup_dir_full }}/vulnerability.dump"
register: vulnerability_dump_stat
- name: Assert vulnerability dump succeeded
when: vulnerability_db_exists | default(false)
ansible.builtin.assert:
that:
- vulnerability_dump_stat.stat.exists
- vulnerability_dump_stat.stat.size > 0
fail_msg: "IOP Vulnerability database dump failed or produced empty file"Note: Each database follows the same three-step pattern:
- Dump with
pg_dump -Fc - Verify file exists with
stat - Assert file is non-empty
backup_metadata:
hostname: "{{ hostname_result.stdout }}"
os_version: "{{ os_version_result.stdout }}"
foremanctl_version: "{{ foremanctl_version_result.stdout | default('unknown') }}"
online: false
incremental: false
timestamp: "{{ backup_timestamp }}"
databases: "{{ databases_to_backup }}" # Now includes IOP DBs
iop_enabled: "{{ 'iop' in enabled_features }}" # ← ADD THIS
enabled_features: "{{ enabled_features }}"
database_mode: "{{ database_mode }}"
container_images: "{{ container_images | map(attribute='RepoTags') | flatten | list }}"
parameters_hash: "{{ lookup('file', parameters_yaml_path) | from_yaml | hash('sha256') }}"Rationale: iop_enabled flag helps future restore operations know if IOP was part of the backup.
Core databases (database.yml):
Host: localhost:5432
Databases:
- foreman (user: foreman, password: CHANGEME)
- candlepin (user: candlepin, password: CHANGEME)
- pulp (user: pulp, password: CHANGEME)IOP databases (database_iop.yml):
Host: host.containers.internal:5432 # Resolves to localhost from containers
Databases:
- advisor_db (user: advisor_user, password: CHANGEME)
- inventory_db (user: inventory_admin, password: CHANGEME)
- remediations_db (user: remediations_user, password: CHANGEME)
- vmaas_db (user: vmaas_admin, password: CHANGEME)
- vulnerability_db (user: vulnerability_admin, password: CHANGEME)PostgreSQL admin access:
- User:
postgres(superuser) - Password:
{{ postgresql_admin_password }}from vars - Used for: Database existence checks (
SELECT 1 FROM pg_database WHERE datname = '...')
The plan's service management is CORRECT:
# Stop all services (including IOP)
- name: Stop Foreman services
ansible.builtin.systemd:
name: foreman.target
state: stopped
# Start PostgreSQL for dumps (if internal DB)
- name: Start PostgreSQL for dumps
ansible.builtin.systemd:
name: postgresql.service
state: started
when: database_mode == 'internal'
# ... dumps happen ...
# Stop PostgreSQL
- name: Stop PostgreSQL
ansible.builtin.systemd:
name: postgresql.service
state: stopped
when: database_mode == 'internal'
# Start all services (including IOP)
- name: Start Foreman services
ansible.builtin.systemd:
name: foreman.target
state: startedVerification:
$ systemctl list-dependencies foreman.target | head -30
foreman.target
● ├─candlepin.service
● ├─dynflow-sidekiq@orchestrator.service
● ├─foreman-proxy.service
● ├─foreman.service
● ├─postgresql.service
● ├─pulp-api.service
● ├─pulp-content.service
● ├─pulp-worker@{1-6}.service
● ├─redis.service
# (IOP services are also part of foreman.target but not shown in list-dependencies)Confirmed: foreman.target manages all services. No changes needed.
Server instance (with IOP):
- Creates
foreman.dump - Creates
candlepin.dump - Creates
pulp.dump - Creates
advisor.dump(if IOP enabled) - Creates
inventory.dump(if IOP enabled) - Creates
remediations.dump(if IOP enabled) - Creates
vmaas.dump(if IOP enabled) - Creates
vulnerability.dump(if IOP enabled)
Server instance (without IOP):
- Creates only
foreman.dump,candlepin.dump,pulp.dump - Skips IOP databases gracefully
Capsule instance:
- Creates only
pulp.dump(andcontainer_gateway.dumpif exists)
-
metadata.ymlincludes all backed-up databases indatabases:list -
metadata.ymlincludesiop_enabled: true/false -
metadata.ymlincludesenabled_featureslist
- Each dump file is non-empty (size > 0)
- Each dump file is PostgreSQL custom format:
file *.dumpshows "PostgreSQL custom database dump" - Each dump can be listed:
pg_restore --list <file>.dumpsucceeds
Test 1: Full IOP backup
# System: Server with IOP enabled
foremanctl backup /tmp/test-backup
# Verify 8 dump files exist
ls /tmp/test-backup/foreman-backup-*/
# Expected: foreman.dump, candlepin.dump, pulp.dump,
# advisor.dump, inventory.dump, remediations.dump,
# vmaas.dump, vulnerability.dump, metadata.yml
# Verify metadata
cat /tmp/test-backup/foreman-backup-*/metadata.yml | grep iop_enabled
# Expected: iop_enabled: true
cat /tmp/test-backup/foreman-backup-*/metadata.yml | grep -A 8 databases
# Expected: all 8 databases listedTest 2: Non-IOP backup
# System: Server without IOP
foremanctl backup /tmp/test-backup
# Verify 3 dump files exist
ls /tmp/test-backup/foreman-backup-*/
# Expected: foreman.dump, candlepin.dump, pulp.dump, metadata.yml
# Verify metadata
cat /tmp/test-backup/foreman-backup-*/metadata.yml | grep iop_enabled
# Expected: iop_enabled: falseTest 3: Dump format verification
# Verify PostgreSQL custom format
file /tmp/test-backup/foreman-backup-*/foreman.dump
# Expected: PostgreSQL custom database dump
# Verify can be restored
pg_restore --list /tmp/test-backup/foreman-backup-*/foreman.dump
# Expected: table of contents listingTest 4: Service restart on failure
# Simulate failure (e.g., full disk)
dd if=/dev/zero of=/tmp/fillup bs=1M # Fill disk
foremanctl backup /tmp/test-backup
# Verify services restarted despite failure
systemctl is-active foreman.target
# Expected: active
systemctl is-active postgresql.service
# Expected: active# 1. Check all databases exist
podman exec postgresql psql -U postgres -c '\l' | grep -E "foreman|candlepin|pulp|advisor|inventory|remediation|vmaas|vulnerability"
# 2. Run backup
foremanctl backup /tmp/test-backup
# 3. Verify dump files
ls -lh /tmp/test-backup/foreman-backup-*/
# 4. Check dump contents
for dump in /tmp/test-backup/foreman-backup-*/*.dump; do
echo "=== $dump ==="
pg_restore --list "$dump" | head -20
done
# 5. Verify metadata
cat /tmp/test-backup/foreman-backup-*/metadata.yml
# 6. Verify services running
systemctl status foreman.target- Update
tasks/database_detection.yaml - Add advisor_db detection query
- Add inventory_db detection query
- Add remediations_db detection query
- Add vmaas_db detection query
- Add vulnerability_db detection query
- Update
databases_to_backupfact to include all 8 DBs
- Update
tasks/database_dumps.yaml - Add advisor.dump creation
- Add inventory.dump creation
- Add remediations.dump creation
- Add vmaas.dump creation
- Add vulnerability.dump creation
- Add verification for each IOP dump
- Update
tasks/metadata.yaml - Add
iop_enabledfield to metadata dict - Ensure
databasesfield includes all dumped DBs
- Update
backup.yamlvars_files - Add
database_iop.ymlto vars_files list
- Test on system with IOP (this system)
- Test on system without IOP (if available)
- Test service recovery on failure
- Test dump file integrity
- Verify metadata completeness
- Database list: Both backup 8 databases (3 core + 5 IOP)
- Dump format: Both use
pg_dump -Fc(custom format) - Metadata before shutdown: Both generate metadata before stopping services
- Service orchestration: Both stop all services, start PostgreSQL, dump, restart all
- Task checking: Both check Foreman and Pulp tasks before backup
- Error handling: Both ensure services restart on failure
- No config file backup: foremanctl doesn't backup quadlet definitions (future enhancement)
- No Pulp content backup: /var/lib/pulp/ not backed up (future enhancement)
- No compression: foremanctl relies on -Fc compression, no additional tar.gz
- Container images vs RPMs: Metadata includes container images instead of RPM list
- Parameters.yaml vs installer answers: Metadata references parameters.yaml not foreman-installer
| Component | Current Plan | Required Update |
|---|---|---|
backup.yaml |
Loads database.yml only |
Add database_iop.yml to vars_files |
database_detection.yaml |
Checks 3-4 DBs | Add checks for 5 IOP databases |
database_dumps.yaml |
Dumps 3 DBs | Add dumps for 5 IOP databases |
metadata.yaml |
No IOP tracking | Add iop_enabled field |
| Acceptance criteria | Lists 3 DBs | Update to list up to 8 DBs |
Likelihood: Low
Impact: High (backup fails)
Mitigation: Test database connectivity in preflight checks
Likelihood: Medium
Impact: Medium (backup incomplete)
Mitigation: Runtime detection with graceful skipping (already in design)
Likelihood: Low
Impact: High (can't connect to IOP DBs)
Mitigation: IOP DBs use host.containers.internal which should resolve to localhost; verify in tests
Likelihood: Low
Impact: Medium (dump writes fail)
Mitigation: Plan already sets mode 0770 and group postgres for internal DB mode
- Review this evaluation with stakeholders
- Update the original plan with IOP database handling
- Implement the changes following the checklist (Phase 1-4)
- Test thoroughly on this IOP-enabled system
- Document limitations (no Pulp content, no config files) for users
- Plan future enhancements (restore command, config backup, Pulp content)
foreman:
host: localhost
port: 5432
user: foreman
password: CHANGEME
database: foreman
candlepin:
host: localhost
port: 5432
user: candlepin
password: CHANGEME
database: candlepin
pulp:
host: localhost
port: 5432
user: pulp
password: CHANGEME
database: pulpadvisor:
host: host.containers.internal
port: 5432
user: advisor_user
password: CHANGEME
database: advisor_db
inventory:
host: host.containers.internal
port: 5432
user: inventory_admin
password: CHANGEME
database: inventory_db
remediations:
host: host.containers.internal
port: 5432
user: remediations_user
password: CHANGEME
database: remediations_db
vmaas:
host: host.containers.internal
port: 5432
user: vmaas_admin
password: CHANGEME
database: vmaas_db
vulnerability:
host: host.containers.internal
port: 5432
user: vulnerability_admin
password: CHANGEME
database: vulnerability_dbpostgres:
user: postgres
password: "{{ postgresql_admin_password }}"
# Used for: SELECT 1 FROM pg_database WHERE datname = '...'/tmp/foreman-backup-20260511T143000/
├── foreman.dump # 1.2 GB - Foreman main DB
├── candlepin.dump # 45 MB - Subscription data
├── pulp.dump # 3.5 GB - Content management
├── advisor.dump # 15 MB - IOP Advisor
├── inventory.dump # 120 MB - IOP Host inventory
├── remediations.dump # 8 MB - IOP Remediations
├── vmaas.dump # 250 MB - IOP VMaaS
├── vulnerability.dump # 180 MB - IOP Vulnerability
└── metadata.yml # 2 KB - Backup metadata
Total backup size (estimate): ~5.3 GB (varies by data volume)
End of Evaluation and Design Document