lab_example_workflow.md

title
About This Documentation

This documentation site is fully automated - generated from live infrastructure using Ansible, Git workflows, and static site generators.

Last Updated: Auto-updated on every deployment
Build System: Astro + Starlight
Source Repository: obsidian-notebook

Overview

The documentation is a living document that reflects the current state of the infrastructure. When you view these pages, you're seeing data pulled directly from:

• Proxmox VE API (cluster state, VMs, LXCs, firewall)
• UniFi Network API (devices, networks, firewall policies)
• Cloudflare API (DNS, Zero Trust, Magic Transit)
• Docker hosts (container inventories)
• Database servers (PostgreSQL, MariaDB, MongoDB, Redis)
• Configuration files (Caddy, Prometheus, Grafana, etc.)

How It Works

1. Infrastructure → Artifacts

Ansible playbooks connect to live infrastructure and save JSON snapshots:

ansible-playbook playbooks/proxmox/inventory.yml
ansible-playbook playbooks/unifi/inventory.yml
ansible-playbook playbooks/cloudflare/inventory.yml

Output: ansible/artifacts/ directory with timestamped JSON files

2. Artifacts → Markdown

Ansible roles read the JSON artifacts and generate markdown documentation:

ansible-playbook playbooks/proxmox/update_docs.yml
ansible-playbook playbooks/unifi/update_docs.yml
ansible-playbook playbooks/cloudflare/update_docs.yml

Output: docs/ directory with auto-generated markdown files

3. Markdown → Static Site

Astro builds the markdown into a static website:

cd astro-docs
npm run build

Output: astro-docs/dist/ directory with static HTML/CSS/JS

4. Deploy → Caddy

CI/CD Actions deploys the built site to Caddy reverse proxy:

# .ci/workflows/deploy-astro-docs.yml
- Build Astro site
- Deploy to Caddy at 10.1.1.1:/var/www/lab
- Purge Redis cache

Result: Documentation live at https://docs.example.com/lab

Auto-Generated Documentation

The following files are 100% auto-generated from live infrastructure:

Infrastructure Documentation

File	Source	Update Frequency
Proxmox Cluster	Proxmox API	On demand
Proxmox LXC Containers	Proxmox API	On demand
UniFi Network	UniFi API	On demand
Cloudflare (Prod)	Cloudflare API	On demand
Cloudflare (Testing)	Cloudflare API	On demand
Caddy Reverse Proxy	Caddy API	On deploy
Database Server	Live server	On demand
Prometheus	Prometheus server	On demand
Grafana	Grafana server	On demand
Loki	Loki server	On demand
Docker Hosts	Docker API	On demand
Runner Server	Live server	On demand
UPS Monitor	Peanut API	On demand

Ansible Documentation

File	Source	Update Frequency
Ansible Index	File system	On demand
Ansible Roles	File system (60 roles)	On demand
Playbook Categories	playbook_metadata.json	On demand

Note: Individual playbook category READMEs in ansible/playbooks/*/README.md are also auto-generated.

Configuration as Code

All infrastructure configuration lives in Git:

config/
├── caddy/Caddyfile              # Reverse proxy routes
├── cloudflare/                  # Zero Trust, DNS, Magic Transit
│   ├── gateway-lists/           # Gateway Lists (YAML)
│   ├── gateway-rules/           # Gateway Rules (YAML)
│   ├── access-apps/             # Access Applications (YAML)
│   ├── dns-records/             # DNS Records (YAML)
│   └── magic-firewall/          # Magic Firewall (YAML)
├── database/                    # PostgreSQL, MariaDB, MongoDB, Redis
├── prometheus/prometheus.yml    # Monitoring config
├── docker/docker_hosts.json     # Docker host manifest
└── ufw/                         # Firewall rules (YAML)

Deploy Workflow

All config changes follow the same pattern:

1. Edit locally - Modify config files in config/
2. Validate - Ansible validates syntax
3. Backup - Current config backed up with timestamp
4. Deploy - Ansible pushes to server
5. Verify - Service restarted and tested
6. Commit - Changes committed to Git

Example:

# Edit config
vim config/caddy/Caddyfile

# Deploy (validates, backs up, deploys, reloads)
ansible-playbook playbooks/caddy/deploy.yml

# Pull back live config if edited on server
ansible-playbook playbooks/caddy/pull.yml

Full Sync

Run a complete sync (all infrastructure → artifacts → docs):

cd ansible

# Testing account only (default)
ansible-playbook playbooks/site.yml

# Include production Cloudflare
ansible-playbook playbooks/site.yml -e cloudflare_account=all

# Only pull configs (no doc generation)
ansible-playbook playbooks/site.yml --tags pull

# Only generate docs (no API calls)
ansible-playbook playbooks/site.yml --tags docs

Document Structure

Every auto-generated page follows this template:

---
title: "Page Title"
---

Brief description of the service/infrastructure.

> **Last Updated:** 2026-02-15T07:20:01Z  
> Auto-generated by Ansible

## Quick Reference

| Parameter | Value |
|-----------|-------|
| Key info  | Value |

## Summary

High-level statistics and counts

## Detailed Sections

Main content...

## See Also

- Cross-references to related docs

---

*Regenerate: `ansible-playbook playbooks/*/update_docs.yml`*

This ensures consistency across all documentation.

Technology Stack

Documentation Generation

• Ansible - Infrastructure automation and doc generation
• Python - Custom scripts for complex transformations
• Jinja2 - Template engine for markdown generation
• JSON/YAML - Data storage and configuration

Static Site

• Astro - Static site generator
• Starlight - Documentation theme
• Pagefind - Search functionality
• TypeScript - Type-safe development

Deployment

• CI/CD Actions - CI/CD pipeline
• Caddy - Reverse proxy and web server
• Redis - Cache layer (24h TTL)
• SSH - Secure deployment (ED25519 keys)

Infrastructure

• Proxmox VE - Virtualization cluster (8 nodes)
• UniFi Network - Gateway and network management
• Cloudflare - Zero Trust, DNS, Magic Transit
• Docker - Container orchestration
• Git - Version control

Source Repository Structure

obsidian-notebook/
├── docs/                         # Markdown documentation (auto-generated)
├── config/                       # Source of truth configs
├── ansible/
│   ├── playbooks/                # Automation playbooks
│   ├── roles/                    # Reusable roles
│   ├── artifacts/                # JSON snapshots from APIs
│   └── playbook_metadata.json    # Playbook descriptions
├── astro-docs/                   # Astro site source
│   ├── src/content/docs/         # Symlink to docs/
│   └── dist/                     # Built static site
└── .ci/workflows/                # CI/CD pipelines

Common Workflows

Here are real-world examples of how the automation works in practice:

Example 1: Add a New Reverse Proxy Route

Scenario: You want to add a new service behind Caddy.

# 1. Edit the Caddyfile locally
vim config/caddy/Caddyfile

# Add your new route:
# newservice.example.com {
#     reverse_proxy http://10.1.1.50:8080
# }

# 2. Commit and push to Git
git add config/caddy/Caddyfile
git commit -m "feat: add newservice reverse proxy route"
git push

What happens automatically:

1. ✅ CI/CD Actions workflow triggers (.ci/workflows/deploy-caddy.yml)
2. ✅ Caddyfile syntax validated
3. ✅ Current config backed up to config/caddy/backups/
4. ✅ New config deployed to Caddy server via Admin API
5. ✅ Caddy automatically reloads (zero downtime)
6. ✅ Route is immediately live at https://newservice.example.com
7. ✅ Documentation at docs/caddy/index.md auto-updates with new route
8. ✅ Redis cache purged for instant doc updates

Total time: ~30 seconds from commit to live!

Example 2: Add a New LXC Container

Scenario: You create a new LXC container in Proxmox for a new service.

# 1. Create LXC in Proxmox UI (or via API)
# VMID: 10005, Name: "monitoring", IP: 10.1.1.35

# 2. Update the documentation
cd ansible
ansible-playbook playbooks/proxmox/inventory.yml
ansible-playbook playbooks/proxmox/update_docs.yml

What happens automatically:

1. ✅ Proxmox API queried for all VMs/LXCs
2. ✅ New container discovered: monitoring (10005)
3. ✅ Container specs collected (vCPUs, RAM, disk, status)
4. ✅ docs/proxmox/index.md updated with new container
5. ✅ docs/proxmox/lxc/monitoring.md created with full details
6. ✅ docs/proxmox/lxc/index.md updated with container count
7. ✅ Artifacts saved to ansible/artifacts/proxmox/vms.json

Next deployment: Astro site rebuilds and new container appears in documentation automatically.

Example 3: Update Cloudflare Gateway Rules

Scenario: You need to block a new malicious domain in Cloudflare Gateway.

# 1. Edit the Gateway Lists config
vim config/cloudflare/gateway-lists/malicious-domains.yml

# Add new domain to the list:
# items:
#   - badsite.com
#   - evilsite.net
#   - newbadsite.com  # NEW

# 2. Deploy to Cloudflare
cd ansible
ansible-playbook playbooks/cloudflare/gateway_lists.yml

What happens automatically:

1. ✅ YAML config validated
2. ✅ Cloudflare API called for both accounts (prod + testing)
3. ✅ Gateway List updated with new domain
4. ✅ All Gateway Rules using this list automatically pick up the change
5. ✅ Domain is immediately blocked across all WARP clients
6. ✅ Documentation updated showing new list count

Bonus: Run the pull playbook later to sync back to Git:

ansible-playbook playbooks/cloudflare/gateway_lists_pull.yml

This ensures your Git repo stays in sync with live Cloudflare config.

Example 4: Full Infrastructure Sync

Scenario: It's Monday morning and you want to ensure all documentation is current.

cd ansible

# Single command syncs EVERYTHING
ansible-playbook playbooks/site.yml -e cloudflare_account=all

What happens automatically:

1. ✅ Proxmox: All 8 nodes, 16 LXCs, 3 VMs inventoried
2. ✅ UniFi: 16 devices, 9 networks, 120 firewall policies collected
3. ✅ Cloudflare (prod + testing): DNS zones, Access apps, Gateway rules, Tunnels
4. ✅ Caddy: Current routes and certificates exported
5. ✅ Database: PostgreSQL/MariaDB/MongoDB/Redis inventories
6. ✅ Docker: All containers across all hosts inventoried
7. ✅ Prometheus/Grafana/Loki: Configs and status checked
8. ✅ All documentation regenerated from fresh data
9. ✅ Ansible playbook/role READMEs updated
10. ✅ Navigation breadcrumbs refreshed

Total artifacts generated: 90+ JSON files
Total documentation updated: 100+ markdown files
Total time: ~5 minutes for complete infrastructure snapshot

Example 5: Add a New DNS Record

Scenario: You need to create a new DNS record in Cloudflare.

# 1. Edit DNS records config
vim config/cloudflare/dns-records/prod/example.com.yml

# Add new record:
# - name: api
#   type: A
#   content: 10.1.1.60
#   proxied: true

# 2. Deploy to Cloudflare
cd ansible
ansible-playbook playbooks/cloudflare/dns_records.yml -e cloudflare_account=prod

What happens automatically:

1. ✅ DNS record created via Cloudflare API
2. ✅ Record immediately resolvable worldwide
3. ✅ Documentation updated: docs/cloudflare/prod/dns/example.com.md
4. ✅ Record count incremented in zone summary
5. ✅ Change committed to Git for audit trail

Alternative: Use the pull playbook to capture manual changes:

# If you added record via Cloudflare Dashboard
ansible-playbook playbooks/cloudflare/dns_records_pull.yml -e cloudflare_account=prod

# Now it's in Git!
git diff config/cloudflare/dns-records/prod/
git add -A && git commit -m "chore: sync DNS records from Cloudflare"

Example 6: Update Database Configuration

Scenario: You need to change PostgreSQL connection limits.

# 1. Edit PostgreSQL config
vim config/database/postgresql/postgresql.conf

# Change: max_connections = 200  # was 100

# 2. Deploy to database server
cd ansible
ansible-playbook playbooks/database/deploy.yml -e db_engine=postgresql

What happens automatically:

1. ✅ Config validated for syntax errors
2. ✅ Current config backed up to config/database/postgresql/backups/
3. ✅ New config deployed to database server (10.1.1.5)
4. ✅ PostgreSQL service restarted
5. ✅ Connection test performed to verify service is up
6. ✅ Documentation updated with new connection limit
7. ✅ Change logged with timestamp

Rollback is easy:

# Just restore the backup and redeploy
cp config/database/postgresql/backups/postgresql.conf.2026-02-15_07-30-00 \
   config/database/postgresql/postgresql.conf
ansible-playbook playbooks/database/deploy.yml -e db_engine=postgresql

The Pattern

Notice the pattern across all workflows:

1. Edit - Make changes in Git repository
2. Commit - Version control the change
3. Deploy - Ansible/CI/CD Actions applies it
4. Verify - Automatic health checks
5. Document - Auto-generated docs update
6. Done - Change is live and documented

No manual steps. No documentation drift. Complete audit trail.

Key Features

1. Self-Documenting Infrastructure

When you add a new service, VM, or network:

• Run the inventory playbook
• Documentation automatically updates
• No manual editing required

2. Accurate and Current

Documentation is generated from live API data, not manually maintained text files that go stale.

3. Version Controlled

Every change is tracked in Git with full history.

4. Searchable

Pagefind indexes all content for instant search.

5. Fast Updates

Caching ensures docs load quickly. Deployments purge cache automatically.

Adding New Documentation

To add a new auto-generated page:

1. Create Ansible role (e.g., service_docs)
2. Add to playbook (e.g., playbooks/service/update_docs.yml)
3. Generate artifacts (e.g., from service API)
4. Create Jinja2 template (e.g., templates/service.md.j2)
5. Run playbook to generate markdown
6. Deploy via CI/CD Actions or manual push

Example role structure:

ansible/roles/service_docs/
├── tasks/main.yml              # Data collection and doc generation
├── templates/
│   └── service_index.md.j2     # Markdown template
└── defaults/main.yml            # Default variables (optional)

Maintenance

Regular Tasks

# Weekly: Full sync and doc update
ansible-playbook playbooks/site.yml -e cloudflare_account=all

# After infrastructure changes
ansible-playbook playbooks/proxmox/inventory.yml
ansible-playbook playbooks/proxmox/update_docs.yml

# After config changes
ansible-playbook playbooks/caddy/deploy.yml

Troubleshooting

Docs out of date?

ansible-playbook playbooks/update_docs_all.yml

Missing data in artifacts?

# Re-run inventory collection
ansible-playbook playbooks/*/inventory.yml

Build failing?

cd astro-docs
npm run build  # Check for errors

Benefits

• ✅ Always accurate - Docs match reality
• ✅ Low maintenance - Automated updates
• ✅ Version controlled - Full Git history
• ✅ Searchable - Instant search with Pagefind
• ✅ Fast - Redis caching, static site
• ✅ Consistent - Standardized templates
• ✅ Scalable - Easy to add new services
• ✅ Auditable - All changes tracked in Git

Future Improvements

Potential enhancements:

• Real-time updates via webhooks
• Diff viewer for config changes
• Integration with monitoring alerts
• Automated screenshot capture
• Network topology diagrams
• Change log generation
• Backup/restore procedures
• Runbook integration

Contributing

This documentation system is maintained by the infrastructure team.

To suggest improvements:

1. Open an issue in the repository
2. Submit a pull request
3. Contact the infrastructure team

---

This page itself is manually written to explain the automation. Meta, right? 😄