Idiosyncracies of BackBlaze S3 on Proxmox Backup Server

gistfile1.txt

____             _    _     _                   ____  ____  
| __ )  __ _  ___| | _| |__ | | __ _ _______   _|  _ \|___ \ 
|  _ \ / _` |/ __| |/ / '_ \| |/ _` |_  / _ \ (_) |_) | __) |
| |_) | (_| | (__|   <| |_) | | (_| |/ /  __/  _|  __/ / __/ 
|____/ \__,_|\___|_|\_\_.__/|_|\__,_/___\___| (_)_|   |_____|
                                                              
 ____                                      ____  ____  ____  
|  _ \ _ __ _____  ___ __ ___   _____  __ | __ )/ ___|| __ ) 
| |_) | '__/ _ \ \/ / '_ ` _ \ / _ \ \/ / |  _ \\___ \|  _ \ 
|  __/| | | (_) >  <| | | | | | (_) >  <  | |_) |___) | |_) |
|_|   |_|  \___/_/\_\_| |_| |_|\___/_/\_\ |____/|____/|____/ 
                                                              
          🦾 ARM64 Edition (CM3588) 🦾

📖 Introduction
This guide documents the complete setup and troubleshooting process for integrating Proxmox Backup Server with Backblaze B2 S3-compatible storage on ARM64 architecture (specifically CM3588).
This records some unusualess and the "work arounds" required to get BackBlaze S3 working on Proxmox Backup Server v4.2.1 
Backblaze B2 offers cost-effective cloud storage at $6/TB/month, making it an excellent choice for offsite backups. However, the ARM64 platform and Backblaze's S3 implementation come with unique challenges that this guide addresses.

🎯 What You'll Learn
✅ Configuring Backblaze B2 S3 endpoint in Proxmox Backup Server
✅ Solving overlay filesystem issues with O_TMPFILE operations
✅ Understanding Backblaze bucket naming requirements
✅ Listing buckets via CLI
✅ Quick reference commands and troubleshooting
🔧 Prerequisites
Proxmox Backup Server compiled for ARM64 (requires swap space during compilation)  - this guide is based on the Wofferl unofficial build at https://github.com/wofferl/proxmox-backup-arm64/releases 
Backblaze B2 account with S3 credentials
Backblaze bucket created (must follow naming requirements - see below)
Physical storage mount for symlink workaround
📋 Listing Backblaze Buckets via CLI
Once you've configured your S3 endpoint in Proxmox Backup Server, you can list available buckets using the following command:

Command
bash
Copy
proxmox-backup-manager s3 endpoint list-buckets eu-central-003
Note: Replace eu-central-003 with your endpoint name as configured in Proxmox Backup Server.

Example Output
┌────────────────────────┬─────────────────────────┐
│ name                   │ creation-date           │
╞════════════════════════╪═════════════════════════╡
│ pxbs-backups      │ 2025-01-15T10:23:45.000Z│
│ my-other-bucket        │ 2025-01-10T08:15:30.000Z│
└────────────────────────┴─────────────────────────┘
This confirms connectivity and proper S3 endpoint configuration.

🐛 The Overlay Filesystem Issue (CRITICAL FIX)
The Problem
When running Proxmox Backup Server on systems with overlay filesystems (common in container environments or certain storage configurations), you may encounter this error:

Error: EOPNOTSUPP: Operation not supported
Failed to create statistics file in /var/lib/proxmox-backup/s3-statistics
mmap() or O_TMPFILE operations not supported on overlay filesystem
Why This Happens
Overlay filesystems (OverlayFS) do not support certain operations:

O_TMPFILE flag (atomic temporary file creation)
mmap() on certain file types
Direct file operations that require underlying filesystem features
Proxmox Backup Server uses these operations for S3 statistics tracking, causing failures on overlay mounts.

✅ The Solution: Symlink to Physical Storage
Move the s3-statistics directory to a physical filesystem and create a symlink:

Step-by-Step Commands
bash
Copy
# 1. Stop Proxmox Backup Server (if running)
systemctl stop proxmox-backup

# 2. Create directory on physical storage
mkdir -p /mnt/storage/s3-statistics

# 3. Remove existing directory (if it exists)
rm -rf /var/lib/proxmox-backup/s3-statistics

# 4. Create symlink from expected location to physical storage
ln -s /mnt/storage/s3-statistics /var/lib/proxmox-backup/s3-statistics

# 5. Set proper permissions
chown backup:backup /mnt/storage/s3-statistics
chmod 750 /mnt/storage/s3-statistics

# 6. Start Proxmox Backup Server
systemctl start proxmox-backup

# 7. Verify the symlink
ls -la /var/lib/proxmox-backup/ | grep s3-statistics
Expected Output
lrwxrwxrwx 1 root root 30 Jan 15 10:30 s3-statistics -> /mnt/storage/s3-statistics
⚠️ Important: Replace /mnt/storage with your actual physical mount point (ext4, xfs, btrfs, etc.).

🏷️ Bucket Naming Requirements
Backblaze B2 buckets accessed via S3 API have strict naming requirements that differ from standard Backblaze naming.

❌ Invalid Bucket Names
The following will NOT work with Proxmox Backup Server S3 integration:

PXBS-Backups     ❌ Contains uppercase letters
12BB_PXBS_Backups     ❌ Contains uppercase + underscores
Backup-Server         ❌ Starts with uppercase
my_bucket_name        ❌ Underscores not allowed in S3 mode
✅ Valid Bucket Names
Buckets must match this regex pattern:

regex
Copy
^[a-z0-9][a-z0-9-]*[a-z0-9]$
Rules:

✅ Lowercase letters (a-z)
✅ Numbers (0-9)
✅ Hyphens (-) for separators
✅ Must start and end with alphanumeric character
❌ NO uppercase letters
❌ NO underscores
❌ NO special characters
Working Example
pxbs-backups     ✅ Perfect!
proxmox-backup-2025   ✅ Valid
my-backup-bucket      ✅ Valid
backup123             ✅ Valid
💡 Tip: If you already created a bucket with uppercase/underscores, you must create a new bucket with a valid name. Renaming is not supported.

⚡ Quick Reference Commands
Configure Backblaze B2 S3 Endpoint
bash
Copy
# Add S3 endpoint
proxmox-backup-manager s3 endpoint create \
  --endpoint s3.eu-central-003.backblazeb2.com \
  --region eu-central-003 \
  --access-key YOUR_ACCESS_KEY_ID \
  --secret-key YOUR_SECRET_KEY \
  --path-style true \
  --skip-if-none-match true
List Configured Endpoints
bash
Copy
proxmox-backup-manager s3 endpoint list
List Buckets
bash
Copy
proxmox-backup-manager s3 endpoint list-buckets ENDPOINT_NAME
Test S3 Connectivity
bash
Copy
proxmox-backup-manager s3 endpoint test ENDPOINT_NAME --bucket YOUR_BUCKET_NAME
Check Service Status
bash
Copy
systemctl status proxmox-backup
journalctl -u proxmox-backup -f
🔍 Troubleshooting Tips
Issue: "EOPNOTSUPP" Errors
Solution: Implement the overlay filesystem symlink fix (see above)

Issue: Bucket Not Found
Symptoms:

Error: Bucket not found or permission denied
Checklist:

✅ Verify bucket name is lowercase only with hyphens
✅ Check S3 credentials have bucket access
✅ Confirm endpoint region matches bucket region
✅ Ensure path-style: true is set
Issue: Authentication Failures
Symptoms:

Error: 401 Unauthorized
Error: Invalid access key
Solutions:

Generate new S3-compatible credentials in Backblaze (not master credentials)
Verify credentials have listBuckets and read/write permissions
Check for whitespace in access-key or secret-key
Issue: Compilation Fails on ARM64
Symptoms:

Out of memory during compilation
Killed
Solution:

bash
Copy
# Add swap space (minimum 4GB recommended)
fallocate -l 4G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile

# Make permanent
echo '/swapfile none swap sw 0 0' >> /etc/fstab
Issue: Path-Style vs Virtual-Hosted Style
Backblaze B2 S3 requires path-style addressing:

✅ path-style: https://s3.eu-central-003.backblazeb2.com/bucket-name/object
❌ virtual-hosted: https://bucket-name.s3.eu-central-003.backblazeb2.com/object
Ensure --path-style true in endpoint configuration.

📚 Additional Resources
Backblaze B2 S3 Compatible API
Proxmox Backup Server Documentation
Backblaze Bucket Naming Rules
🙏 Acknowledgments
This guide was created after successfully deploying Proxmox Backup Server on ARM64 (CM3588) with Backblaze B2 storage. Special thanks to the Proxmox and Backblaze communities for their documentation.

📝 Configuration Summary
For reference, here's a working configuration:

Parameter	Value
Platform	ARM64 (CM3588)
Endpoint	s3.eu-central-003.backblazeb2.com
Region	eu-central-003
Path-Style	Enabled
Skip If-None-Match	Enabled
Bucket Name	pxbs-backups
Statistics Path	/mnt/storage/s3-statistics (symlinked)
Last Updated: June 2026
Platform: ARM64/CM3588
PBS Version: proxmox-backup-server (ARM64 build)

⭐ If this guide helped you, consider sharing it with others facing similar challenges!