Created
October 3, 2025 17:33
-
-
Save jwiegley/355ad81c3f02918bd348cf83a84e364a to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| commit f299585a768d894385a47e8ee613ba64a924056b | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Fri Oct 3 10:29:35 2025 -0700 | |
| opnsense-exporter: Add API transformer proxy workaround | |
| - **OPNSENSE-EXPORTER-WORKAROUND.md**: Add comprehensive documentation for | |
| Python-based HTTP proxy workaround that fixes type mismatch bug in | |
| opnsense-exporter v0.0.11 gateway collector, including problem description, | |
| implementation details, testing steps, revert instructions when upstream fix | |
| is released, and alternative solutions | |
| - **opnsense-api-transformer.nix**: Add Python HTTP proxy service that | |
| intercepts OPNsense API requests on localhost:8444, transforms | |
| /api/routing/settings/searchGateway responses to convert monitor_disable | |
| boolean to string and priority number to string before forwarding to | |
| exporter, fixing "json: cannot unmarshal bool" error in gateway collector | |
| - **opnsense-exporter-quadlet.nix**: Reconfigure container to route API | |
| requests through transformer proxy at 10.88.0.1:8444 instead of direct | |
| OPNsense API connection, change protocol to http, keep insecure mode | |
| enabled, add opnsense-api-transformer.service to systemd dependencies, and | |
| expand inline documentation with detailed workaround removal steps | |
| - **quadlet.nix**: Import opnsense-api-transformer module with inline comment | |
| explaining its purpose as gateway collector fix | |
| commit 3943ca5a98abb080f7e407908010fef997656399 | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Fri Oct 3 10:06:17 2025 -0700 | |
| OPNSENSE-EXPORTER-SETUP: Add OPNsense exporter docs, config | |
| - **OPNSENSE-EXPORTER-SETUP.md**: Add comprehensive documentation for OPNsense | |
| Prometheus exporter setup, including API credential configuration, required | |
| permissions, SOPS secret format, deployment steps, verification procedures, | |
| troubleshooting guide, and available metrics reference | |
| - **opnsense-exporter-quadlet.nix**: Add Podman quadlet configuration for | |
| OPNsense exporter container using | |
| ghcr.io/athennamind/opnsense-exporter:latest, exposing metrics on | |
| localhost:9273, with SOPS-encrypted API credentials and temporary insecure | |
| mode workaround for gateway collector issue | |
| - **quadlet.nix**: Import opnsense-exporter-quadlet module to enable container | |
| service | |
| - **prometheus-monitoring.nix**: Add "opnsense" scrape job targeting | |
| localhost:9273 with 30s interval and labels (alias="opnsense-router", | |
| role="gateway", device_type="router") | |
| - **secrets.yaml**: Add encrypted opnsense-exporter-secrets entry containing | |
| OPNSENSE_EXPORTER_OPS_API_KEY and OPNSENSE_EXPORTER_OPS_API_SECRET environment | |
| variables | |
| commit 50b02937fbf35bbda193c12453875f3aee7a8378 | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Thu Oct 2 17:35:03 2025 -0700 | |
| scripts/imapdedup.py: Remove trailing whitespace | |
| commit 1a1bb256c0489c92491519ebfd65dedb4673d872 | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Thu Oct 2 17:34:40 2025 -0700 | |
| prometheus-monitoring: Add OPNsense router monitoring | |
| - **prometheus-monitoring.nix**: Add new scrape job "node_opnsense" to monitor | |
| OPNsense router at 192.168.1.1:9100 with 30s interval and labels for alias, | |
| role, and device type | |
| commit a008f21955ff6cf2119482141edc18608bf029b5 | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Thu Oct 2 17:31:30 2025 -0700 | |
| imapdedup: Add weekly dedup service for Dovecot | |
| - **vulcan/default.nix**: Import imapdedup service module | |
| - **imapdedup.nix**: Add systemd service and timer to run weekly IMAP | |
| deduplication for user mailboxes with proper security hardening and Dovecot | |
| integration | |
| - **imapdedup.py**: Add Python script to identify and mark/delete duplicate | |
| messages in IMAP mailboxes based on Message-ID or content checksum | |
| commit 4e947069a878af8c6e418895340b6638d4be1186 | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Thu Oct 2 17:04:20 2025 -0700 | |
| homepage: Add Nextcloud to Productivity section | |
| - modules/services/homepage.nix: | |
| - Add new "Productivity" section configuration with row style and 3 columns | |
| - Add Nextcloud service entry under Productivity section with icon, URL | |
| (https://nextcloud.vulcan.lan), description and ping configuration | |
| - modules/services/nextcloud.nix: | |
| - Fix nextcloud-monitoring-password secret ownership from "nextcloud" | |
| user/group to "nextcloud-exporter" user/group for proper access | |
| permissions | |
| commit a7386ef6b85d772021d5ca3902a990582ac2c9e3 | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Thu Oct 2 17:03:54 2025 -0700 | |
| mkMbsyncModule: Switch mail storage to /var/mail | |
| - **mkMbsyncModule.nix**: Change ReadWritePaths from `/tank/Maildir/${name}` | |
| to `/var/mail/${name}` to use standard mail directory location | |
| - **dovecot.nix**: Update mailLocation from `maildir:/home/%u/Maildir` to | |
| `maildir:/var/mail/%u` and revise comment to reflect ZFS storage usage | |
| commit d04d7c07220ae1e3bfdf32eac2bbd96e0fbd6c48 | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Thu Oct 2 10:53:13 2025 -0700 | |
| nextcloud: Upgrade to Nextcloud 31 | |
| - **modules/services/nextcloud.nix**: Update package version from nextcloud30 | |
| to nextcloud31 | |
| commit 736cafb85094d293be0dd3759237d63eceedac04 | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Thu Oct 2 10:44:45 2025 -0700 | |
| nextcloud: Add Nextcloud server with monitoring | |
| - **.claude/settings.local.json** | |
| - Add sequential-thinking MCP tool to allowed list | |
| - Add `zfs list` bash command to allowed tools | |
| - **certs/renew-nginx-certs.sh** | |
| - Add nextcloud.vulcan.lan to certificate renewal domains list | |
| - **hosts/vulcan/default.nix** | |
| - Import nextcloud service module | |
| - **modules/monitoring/alerts/nextcloud.yaml** | |
| - Add comprehensive Nextcloud monitoring alerts | |
| - Monitor service availability with 5m downtime threshold | |
| - Add storage space warnings (<20GB) and critical alerts (<10GB) | |
| - Track active users, database health, and app updates | |
| - Monitor PHP-FPM pool utilization (warn at >80%) | |
| - Alert on stalled background jobs (>10 minutes) | |
| - **modules/services/databases.nix** | |
| - Add nextcloud database and user to PostgreSQL | |
| - Ensure nextcloud user has database ownership | |
| - Add systemd service to set PostgreSQL password for nextcloud from SOPS | |
| secret | |
| - Configure password setup to run before nextcloud-setup.service | |
| - **modules/services/nextcloud.nix** | |
| - Configure Nextcloud 30 with PostgreSQL backend | |
| - Set up SOPS secrets for admin, database, and monitoring passwords | |
| - Enable Redis caching (distributed and locking) with dedicated instance | |
| - Configure PHP with 512M memory, 10G uploads, 3600s timeouts | |
| - Tune PHP-FPM with dynamic pool (5-50 workers) | |
| - Set up Nginx vhost with TLS and security headers | |
| - Enable Prometheus exporter on port 9205 | |
| - Bind mount /tank/Nextcloud to /var/lib/nextcloud/data | |
| - Configure email via local Postfix | |
| - Apply systemd hardening to PHP-FPM service | |
| - **modules/services/prometheus-monitoring.nix** | |
| - Add nextcloud.yaml to alert rules list | |
| - Configure Prometheus scrape job for Nextcloud exporter | |
| - **modules/storage/backups.nix** | |
| - Add Nextcloud to backup schedule | |
| - Exclude cache, previews, tmp, and updater files from backups | |
| - **secrets.yaml** | |
| - Add SOPS-encrypted passwords for Nextcloud admin, database, and monitoring | |
| users | |
| commit f883736f1db43853234987b57bf2b1bf58e69470 | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Thu Oct 2 08:33:03 2025 -0700 | |
| mbsync: Exclude Import folder from sync patterns | |
| - modules/services/mbsync.nix: | |
| * Update sync patterns to exclude "Import" folder and its subfolders | |
| * Add negative pattern "!Import !Import/*" to prevent syncing Import directory | |
| * Maintain existing wildcard pattern for all other folders | |
| commit f8442ad13ea630cd530a55f6c507347eb06560f1 | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Thu Oct 2 08:20:00 2025 -0700 | |
| renew-nginx-certs, validate-certificates: Remove smokeping | |
| - **certs/renew-nginx-certs.sh**: Remove smokeping.vulcan.lan from domain list | |
| - **certs/validate-certificates.sh**: Remove smokeping.vulcan.lan from TLS | |
| endpoint checks | |
| - **docs/prd.md**: Remove SmokePing from infrastructure services list | |
| - **modules/core/boot.nix**: Remove smokeping.service restart from boot script | |
| - **modules/services/blackbox-monitoring.nix**: Update documentation to remove | |
| references to SmokePing integration and complementary monitoring approach | |
| - **modules/services/homepage.nix**: Remove SmokePing dashboard entry from | |
| homepage configuration | |
| - **modules/services/monitoring.nix**: Remove entire smokeping service | |
| configuration including alert, database, probe, and target configs; remove | |
| nginx virtual host configuration for smokeping and smokeping.vulcan.lan | |
| - **modules/services/prometheus-monitoring.nix**: Remove | |
| https://smokeping.vulcan.lan from blackbox exporter target list | |
| - **modules/services/service-reliability.nix**: Remove smokeping systemd | |
| service restart configuration | |
| commit 4ee5a2404f1e4991b8f5888abb805ddbaf28a22a | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Thu Oct 2 08:19:22 2025 -0700 | |
| prometheus-monitoring: Remove August locks and Ring doorbell | |
| - **modules/services/prometheus-monitoring.nix**: Comment out monitoring | |
| targets for three August smart locks (front door, garage door, side door) | |
| and the Ring doorbell, removing them from active Prometheus monitoring while | |
| preserving their configuration for potential future use. | |
| commit 8ebb49a70b1199b571a842e8983624ba6b9236d6 | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Wed Oct 1 23:17:23 2025 -0700 | |
| Restore mbsync metrics | |
| commit 74705d0c9658c357229dd21015416a0b60d3e99d | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Wed Oct 1 22:36:08 2025 -0700 | |
| prometheus-monitoring: Add postfix, ZFS, restic exporters | |
| - **modules/services/prometheus-monitoring.nix**: Enable textfile collector on | |
| node exporter with dedicated directory for custom metrics. Add postfix | |
| exporter for mail server monitoring (port 9154) and ZFS exporter for pool | |
| monitoring (port 9134). Implement restic metrics collection via textfile | |
| collector with comprehensive bash script that gathers stats (snapshot count, | |
| repo size, restore size, unique files, deduplication info, timestamps) for | |
| nine S3-backed repositories (Audio, Backups, Databases, Home, Nasim, Photos, | |
| Video, doc, src). Add systemd service and 6-hour timer for automated restic | |
| metrics collection. Configure firewall rules for new exporter ports. Add | |
| scrape configs for postfix and ZFS exporters. Update documentation to | |
| describe all monitoring endpoints and restic metrics approach. | |
| - **modules/storage/backups.nix**: Add comment noting that restic metrics | |
| collection uses existing aws-keys and restic-password secrets via the | |
| textfile collector approach. | |
| commit d94350c904f647b38e5654f919c8aee81367ba47 | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Wed Oct 1 20:24:06 2025 -0700 | |
| dovecot: Update mailLocation to use /home/%u/Maildir | |
| - **modules/services/dovecot.nix**: Change mail storage location from | |
| `/tank/Maildir/%u` to `/home/%u/Maildir` to use standard home | |
| directory-based mailbox structure instead of tank-based storage | |
| commit db3cdb985d75ea26355ec32c444ea08ed049a3b6 | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Wed Oct 1 18:05:40 2025 -0700 | |
| monitoring: Fix certificate validation script | |
| - **modules/services/monitoring.nix**: Fix the `certificateValidationScript` | |
| to properly execute the validation script by removing unnecessary quotes | |
| around the command path and ensuring required runtime dependencies (`gawk`, | |
| `gnugrep`) are included in the script's environment. | |
| commit 7dc6170f4f0e02f3ba7081f1c930c87086893da0 | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Wed Oct 1 18:02:24 2025 -0700 | |
| prometheus-monitoring: Remove traeger-grill from targets | |
| - modules/services/prometheus-monitoring.nix: Comment out "traeger-grill.lan" | |
| from the blackbox exporter targets list, effectively removing it from | |
| monitoring | |
| commit fb6ebc56adda9b0a56e1ec12fa1f17e2fb0a0db4 | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Wed Oct 1 17:54:55 2025 -0700 | |
| system: Move systemd alerts to dedicated file | |
| - **modules/monitoring/alerts/system.yaml**: Remove basic SystemdServiceFailed | |
| alert that was duplicated and will be replaced by more comprehensive checks | |
| in the new dedicated file. | |
| - **modules/monitoring/alerts/systemd.yaml**: Add comprehensive systemd | |
| monitoring with specific alerts for: | |
| - Failed services detection across all systemd units | |
| - Critical service monitoring (sshd, postgresql, nginx, prometheus, tailscaled, step-ca, docker) | |
| - Frequent restart detection using rate metrics | |
| - Services stuck in activating or deactivating states | |
| - Monitoring service availability (prometheus services) | |
| - Backup service failure detection (restic-backups) | |
| - Timer unit status monitoring for scheduled tasks | |
| - Socket unit listening status checks | |
| - Network service health (NetworkManager, tailscaled, nebula) | |
| - Database service availability (postgresql) | |
| - Certificate authority service status (step-ca) with TLS-specific context | |
| - **modules/services/prometheus-monitoring.nix**: Add systemd.yaml to alert | |
| rules list and comment out non-critical IoT devices (ADT security panel, | |
| Nest thermostats) from blackbox monitoring targets. Update documentation to | |
| reference new systemd alerts file. | |
| commit b574b0d7c5da466515e70fbb59bfdd59aca5ca9d | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Wed Oct 1 17:27:00 2025 -0700 | |
| network: Adjust jitter alert threshold to 500ms | |
| - modules/monitoring/alerts/network.yaml: Increase the HighNetworkJitter alert | |
| threshold from 0.2s (200ms) to 0.5s (500ms) standard deviation over 10 | |
| minutes. Update the alert description to reflect the new 500ms threshold | |
| instead of the previous 50ms value (which was incorrectly documented - the | |
| actual threshold was 200ms). | |
| commit 87599f7bce4caa5e0d14a081cafa4f0477c0fc7a | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Wed Oct 1 17:13:34 2025 -0700 | |
| chainweb: Fix alert description thresholds | |
| - modules/monitoring/alerts/chainweb.yaml | |
| * Update ChainwebBlockProgressionSlow description to reflect correct | |
| threshold of 100 blocks per 15 minutes (was incorrectly showing 5 blocks) | |
| - modules/monitoring/alerts/network.yaml | |
| * Fix HighLatency alert description to show correct threshold of >5s (was | |
| showing >100ms) | |
| * Fix CriticalLatency alert description to show correct threshold of >10s | |
| (was showing >500ms) | |
| commit 602a94c289bc86ab4448bcefefa1b641d1ca18fd | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Wed Oct 1 16:48:19 2025 -0700 | |
| Add Prometheus UI access and adjust alert thresholds | |
| - **certs/renew-nginx-certs.sh** | |
| - Add "prometheus.vulcan.lan" domain to certificate renewal list | |
| - **modules/monitoring/alerts/network.yaml** | |
| - Remove all runbook_url references from alert annotations | |
| - Increase HighLatency alert threshold from 0.1s to 5s | |
| - Increase CriticalLatency alert threshold from 0.5s to 10s | |
| - Increase HighNetworkJitter threshold from 0.05s to 0.2s standard deviation | |
| - Relax alert sensitivity to reduce false positives for local network | |
| - **modules/services/blackbox-monitoring.nix** | |
| - Add new "https_2xx_local" probe module with step-ca root certificate verification | |
| - Create setup-blackbox-ca systemd service to copy step-ca root cert to accessible location | |
| - Add service dependency on setup-blackbox-ca before blackbox exporter starts | |
| - Configure BindReadOnlyPaths to allow reading step-ca certificate from /etc/ssl/certs | |
| - **modules/services/homepage.nix** | |
| - Move Grafana from Infrastructure to new Monitoring section | |
| - Add Prometheus entry to Monitoring section with icon, URL, and description | |
| - Reorder Monitoring services: Grafana, Prometheus, Alertmanager, Smokeping | |
| - **modules/services/prometheus-monitoring.nix** | |
| - Enable Prometheus admin API with --web.enable-admin-api flag | |
| - Clean up and alphabetically sort smart home device monitoring targets | |
| - Remove inactive devices from ICMP monitoring (watches, phones, laptops) | |
| - Remove "www.indiana.edu" from university backbone monitoring | |
| - Split HTTPS monitoring into public and local jobs | |
| - Add blackbox_https_local job using https_2xx_local module for step-ca secured services | |
| - Add "prometheus.vulcan.lan" to local HTTPS monitoring targets | |
| - Add Nginx reverse proxy configuration for Prometheus UI at prometheus.vulcan.lan | |
| commit 3df8600cd240b9d05f40d6f0801e5ad0245aa9ae | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Wed Oct 1 15:54:04 2025 -0700 | |
| renew-nginx-certs: Remove uptime domain, update monitoring | |
| • **certs/renew-nginx-certs.sh**: Remove "uptime.vulcan.lan" from the domains | |
| array | |
| • **modules/services/prometheus-monitoring.nix**: Replace dynamic blackbox | |
| monitoring target with comprehensive static host list including local network | |
| devices, IoT devices, external DNS servers, and web services; add internal | |
| HTTPS service monitoring endpoints; expand DNS monitoring targets with | |
| additional public DNS servers and local gateway | |
| commit 93dea3bbe3391bfabc092ee7d76572450231adce | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Wed Oct 1 15:31:40 2025 -0700 | |
| chainweb: Increase low block progression threshold | |
| - **chainweb.yaml**: Increase alert threshold from 5 to 100 blocks per 15 | |
| minutes for ChainwebLowBlockProgression alert | |
| commit 56ad98e0de5145128deeb6860f41730f8b056b50 | |
| Author: John Wiegley <johnw@newartisans.com> | |
| Date: Wed Oct 1 15:28:03 2025 -0700 | |
| backups: Exclude Kadena from Audio backup |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment