Pulp Filesystem Data Backup Analysis

What foreman-maintain Does

Location: /root/foreman_maintain/definitions/procedures/backup/pulp.rb

Backs Up: `/var/lib/pulp`

Includes:

/var/lib/pulp/media - Repository content (RPMs, errata, container images, etc.)
- This is where all synced repository artifacts are stored
- Can be hundreds of GB to TBs in production

Excludes (regenerated on installer run):

/var/lib/pulp/assets - Static files (CSS, JS)
/var/lib/pulp/exports - Temporary export data
/var/lib/pulp/imports - Temporary import data
/var/lib/pulp/sync_imports - Temporary sync data
/var/lib/pulp/tmp - Temporary worker files

Method:

tar --create \
    --gzip \
    --listed-incremental .pulp.snar \
    --transform 's,^,var/lib/pulp/,S' \
    --file pulp_data.tar \
    --exclude assets --exclude exports --exclude imports --exclude sync_imports --exclude tmp \
    *

Features:

Incremental backup support (.pulp.snar file tracks changes)
File change detection (ensures data doesn't change during backup)
Optional tar volume splitting for large content (--tar-volume-size)
Skip option (--skip-pulp-content)

Typical Size:

Fresh install: ~10 KB (just symmetric keys)
Small deployment: 1-10 GB (RHEL repos)
Medium deployment: 50-200 GB (RHEL + satellite capsule)
Large deployment: 500 GB - 2 TB+ (multiple products, versions, architectures)

Containerized foremanctl Setup

Current State on This System

Directory Structure:

/var/lib/pulp/
├── assets/              (0 bytes - empty)
├── database_fields.symmetric.key  (45 bytes)
├── django_secret_key   (68 bytes)
├── media/              (0 bytes - empty, no synced content)
└── tmp/                (4 KB - worker temp dirs)

Pulp Database:

0 artifacts (no synced repositories)

Key Files Needed for Restore:

database_fields.symmetric.key - Encrypts sensitive database fields
django_secret_key - Django application secret

Where Data Lives

Secrets:

✅ /var/lib/pulp/database_fields.symmetric.key → pulp-symmetric-key (podman secret)
✅ /var/lib/pulp/django_secret_key → pulp-django-secret-key (podman secret)
✅ Already backed up in config_files.tar.gz (72 podman secrets)

Repository Content:

Would be in /var/lib/pulp/media/ when repositories are synced
Bind-mounted into pulp-api, pulp-content, pulp-worker containers
Database tracks artifact locations via core_artifact table

What We Currently Backup

✅ Already Covered

Pulp Database (pulp.dump)
- All artifact metadata (paths, checksums, sizes)
- Repository structure
- Content associations
- Publications and distributions
Pulp Secrets (in config_files.tar.gz)
- pulp-symmetric-key (matches database_fields.symmetric.key)
- pulp-django-secret-key (matches django_secret_key)
- pulp-db-password
- pulp-db-ca
Pulp Systemd Units (in systemd-units.tar.gz)
- pulp-worker.target
- pulp-worker@1-6.service instances

❌ NOT Currently Backed Up

Repository Content Files:

/var/lib/pulp/media/** - Actual RPMs, errata, container layers
Currently 0 bytes on this system (no synced repos)
Would be critical in production with synced content

Recommendation for Containerized Setup

Option 1: Conditional Pulp Content Backup (RECOMMENDED)

Only backup /var/lib/pulp/media if it contains data.

Implementation:

# File: /root/foremanctl/src/playbooks/backup/tasks/pulp_content.yaml
---
- name: Check if Pulp media directory has content
  ansible.builtin.find:
    paths: /var/lib/pulp/media
    recurse: yes
    file_type: file
  register: pulp_media_files
  when: database_mode == 'internal'

- name: Check Pulp media directory size
  ansible.builtin.command:
    cmd: du -sb /var/lib/pulp/media
  register: pulp_media_size
  changed_when: false
  when:
    - database_mode == 'internal'
    - pulp_media_files.files | length > 0

- name: Backup Pulp content directory
  ansible.builtin.archive:
    path: /var/lib/pulp/media
    dest: "{{ backup_dir_full }}/pulp_content.tar.gz"
    format: gz
    exclude_path:
      - /var/lib/pulp/assets
      - /var/lib/pulp/exports
      - /var/lib/pulp/imports
      - /var/lib/pulp/sync_imports
      - /var/lib/pulp/tmp
  when:
    - database_mode == 'internal'
    - pulp_media_files.files | length > 0
  register: pulp_content_backup

- name: Display Pulp content backup info
  ansible.builtin.debug:
    msg: "Backed up {{ pulp_media_files.files | length }} Pulp artifacts ({{ pulp_media_size.stdout | int | human_readable }})"
  when:
    - database_mode == 'internal'
    - pulp_media_files.files | length > 0

Advantages:

Only creates tar if content exists
Follows foreman-maintain exclusion pattern
Shows size in backup summary
Works for both fresh and populated systems

Size Impact:

Fresh system (this one): +0 bytes (skipped)
With synced repos: +size of /var/lib/pulp/media

Option 2: Skip Pulp Content (Not Recommended for Production)

Accept that repository content must be re-synced after restore.

Pros:

Smaller backups
Faster backup/restore

Cons:

Data loss - all synced repositories gone
Must re-sync all content (hours to days depending on bandwidth)
Offline during re-sync
May not be acceptable for air-gapped environments

Option 3: External Storage Backend

If Pulp is configured with S3/Azure/etc storage backend:

Content is NOT in /var/lib/pulp/media
Backup database only (already doing this)
Restore relies on external storage availability

Production Considerations

Size Planning

Typical Pulp Content Sizes:

RHEL 9 AppStream: ~7 GB
RHEL 9 BaseOS: ~2 GB
Satellite Tools: ~500 MB
Container images: varies widely (100 MB - 50 GB)

Example Production Backup:

Database dumps:        25 MB
Podman secrets:        45 KB
Quadlet files:         7 KB
Systemd units:         1 KB
Networks:              5 KB
State:                 24 KB
Volumes (kafka+vmaas): 1.5 GB
Pulp content:          150 GB  ← NEW
────────────────────────────────
Total:                 ~152 GB

Incremental Backup

foreman-maintain uses incremental backups for Pulp content:

First backup: Full (~150 GB)
Subsequent: Only changes (~5-10 GB/week)

We could implement:

- name: Incremental Pulp content backup
  ansible.builtin.command:
    cmd: >
      tar --create --gzip
      --listed-incremental {{ backup_dir_full }}/.pulp.snar
      --file {{ backup_dir_full }}/pulp_content.tar.gz
      -C /var/lib/pulp/media .

Skip Option

Add user control via parameter:

# metadata.obsah.yaml
skip_pulp_content:
  help: Skip Pulp content directory backup
  action: store_true
  persist: false

Current Status: What's Missing?

Critical for Systems with Synced Content

NOT BACKED UP: /var/lib/pulp/media/**

Impact:

If this system had synced repositories, they would NOT be backed up
Restore would succeed but all repository content would be gone
Must re-sync all content (potentially hundreds of GB, hours/days)

OK for This System

Current state: 0 artifacts, /var/lib/pulp/media is empty

No data loss risk right now
Backup is complete for current deployment state

Recommended Action

Immediate (Next Commit)

✅ Add Pulp content backup with conditional logic:

Create pulp_content.yaml task (see Option 1 above)
Add to backup.yaml after database dumps
Update metadata to include pulp_content in backed_up_components
Add --skip-pulp-content flag for large deployments

Future Enhancements

Incremental backup support - Use .pulp.snar for delta backups
Parallel compression - Use pigz instead of gzip for large content
Deduplication - Check for hardlinks (Pulp uses them)
Progress reporting - Show backup progress for large content
Validation - Verify artifact checksums against database

Testing

Test on system with synced content:

Sync a repository:

hammer repository synchronize --id <repo-id>

Verify content exists:

du -sh /var/lib/pulp/media
find /var/lib/pulp/media -type f | wc -l

Run backup:

./foremanctl backup /var/tmp/test-with-content

Verify pulp_content.tar.gz created and contains files

Summary

Component	foreman-maintain	foremanctl (current)	Recommended
Pulp database	✅ Backed up	✅ Backed up	✅ Keep
Pulp secrets	✅ In config tar	✅ In config tar (podman secrets)	✅ Keep
Pulp systemd	✅ Backed up	✅ Backed up	✅ Keep
`/var/lib/pulp/media`	✅ Backed up (excluding tmp/assets/etc)	❌ NOT backed up	⚠️ ADD
Incremental support	✅ Yes	❌ No	🔮 Future
Skip option	✅ Yes	❌ No	➕ Add flag

Verdict: We're missing Pulp content backup. This is OK for fresh systems but CRITICAL for production systems with synced repositories.

sjha4/PULP_FILESYSTEM_BACKUP_ANALYSIS.md

Select an option

No results found

Select an option

No results found

Pulp Filesystem Data Backup Analysis

What foreman-maintain Does

Backs Up: `/var/lib/pulp`

Containerized foremanctl Setup

Current State on This System

Where Data Lives

What We Currently Backup

✅ Already Covered

❌ NOT Currently Backed Up

Recommendation for Containerized Setup

Option 1: Conditional Pulp Content Backup (RECOMMENDED)

Option 2: Skip Pulp Content (Not Recommended for Production)

Option 3: External Storage Backend

Production Considerations

Size Planning

Incremental Backup

Skip Option

Current Status: What's Missing?

Critical for Systems with Synced Content

OK for This System

Recommended Action

Immediate (Next Commit)

Future Enhancements

Testing

Summary

sjha4/PULP_FILESYSTEM_BACKUP_ANALYSIS.md

Pulp Filesystem Data Backup Analysis

What foreman-maintain Does

Backs Up: /var/lib/pulp

Containerized foremanctl Setup

Current State on This System

Where Data Lives

What We Currently Backup

✅ Already Covered

❌ NOT Currently Backed Up

Recommendation for Containerized Setup

Option 1: Conditional Pulp Content Backup (RECOMMENDED)

Option 2: Skip Pulp Content (Not Recommended for Production)

Option 3: External Storage Backend

Production Considerations

Size Planning

Incremental Backup

Skip Option

Current Status: What's Missing?

Critical for Systems with Synced Content

OK for This System

Recommended Action

Immediate (Next Commit)

Future Enhancements

Testing

Summary

Backs Up: `/var/lib/pulp`