You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Analysis of the Datum Cloud DNS service (dns-operator) for production readiness, combining an infra-deployment survey, peer service comparison within the platform, and Google SRE production-readiness standards from the SRE books and CRE launch checklist.
Current State
The DNS operator has:
Service catalog registration: Service resource published, ServiceConfiguration planned (Phase 2, issue #44)
IAM: ProtectedResources for all 4 CRDs, admin/viewer RBAC roles, GCS bucket IAM for LightningStream
Kustomize overlays: production/staging for control-plane, downstream/agent, storage; edge variant for the DaemonSet
Metrics scraping: PodMonitors for pdns-auth and the managed DaemonSet; Vector metrics gateway for edge
Resource metrics policy: dns-metrics.yaml generating Prometheus gauges for zones and record sets
E2E dashboard: Grafana dashboard tracking Chainsaw test results for DNS setup
CRDs: 4 resources under dns.networking.miloapis.com/v1alpha1
CI/CD: GitHub Actions for test, lint, publish (Docker + Kustomize bundle), and e2e (3-cluster kind)
No README in apps/dns-operator/ explaining deployment layout, components, or how to operate it
9 of 46 apps have READMEs
Crossplane resources
❌ Missing
GCS bucket for LightningStream uses raw k8s resources instead of Crossplane Composition
ipam-system, backstage, datum-iam-system, etc.
Runbooks (alerts pages)
❌ Missing
No docs/runbooks/dns/ directory. Every other subsystem (karmada, kubelet, flux, bgp, envoy, cnpg, billing, victoriametrics, workload-memory) has one.
Entire docs/runbooks/ README documents the convention
Rate limiting / WAF for DNS
❌ Missing
Platform has Envoy rate-limiting and Coraza WAF for HTTP, but UDP/TCP 53 is uncovered
tests/construct/networking/rate-limiting/ + WAF tests exist for HTTP
DNSSEC
❌ Missing
Neither PowerDNS nor Knot DNS has DNSSEC configured
Not configured anywhere in infra
Incident response
❌ Missing
No on-call rotation, escalation path, or incident response procedure defined for DNS
Billing, Karmada, and others have documented runbooks
Part 2: Gaps vs. Google SRE Production Readiness Standards
Based on the Google SRE book, SRE workbook, and CRE (Customer Reliability Engineering) launch checklists.
2.1 Monitoring & Alerting
Requirement
Status
Gap
Symptom-based (not cause-based) alerts
❌
No alerts at all
Only 3 output types: pages, tickets, logs
❌
Not categorized
Alert thresholds aligned with SLOs
❌
No SLOs defined
Every paging alert has a runbook
❌
No runbooks
Black-box monitoring (end-to-end)
❌
No synthetic DNS query monitoring from outside
White-box monitoring (internal state)
⚠️ Partial
Resource metrics exist but no controller health alerts
Pages immediately actionable
❌
Nothing pages
Maximum 2 incidents per on-call shift
❌
No on-call defined
Monitoring the monitoring
❌
Not addressed
Dashboards link to playbooks; playbooks link to consoles
❌
No playbooks exist
2.2 SLIs, SLOs, Error Budgets
Requirement
Status
Gap
SLIs defined in user-centric terms (measure at client, not server)
❌
Not defined
SLOs published (specific availability/latency targets)
❌
Not published
Error budget policy defined and enforced
❌
Not defined
SLO document signed off by dev + SRE
❌
Not written
SLO-burn-rate alerting
❌
Not implemented
Suggested initial SLOs (common for authoritative DNS services):
Availability: 99.99% of DNS queries to the serving layer succeed (non-SERVFAIL, non-NXDOMAIN for existing records)
Latency P99 (global): <50ms for recursive resolution
Propagation freshness: zone changes replicated to all edges within 120s (P99)
Zone programming: DNSZone/DNSRecordSet reaches Programmed=True within 60s of creation
2.3 Capacity Planning
Requirement
Status
Gap
Load test completed at max acceptable latency
❌
No load testing framework exists
Capacity forecast for 6 months out
❌
Not documented
End-to-end perf test under projected peak (15-25x normal)
❌
Not done
Storage capacity verified (LMDB, logs, backups)
❌
Not documented
N+2 redundancy
❌
No DR plan for total DNS-serving failure
Spare capacity for 10x growth
❌
No growth model
Documented max zones/nodes/records/QPS
❌
Not published
Autoscaling policy for PowerDNS/recursor edge nodes
❌
Not defined
2.4 Failure Modes & Resilience
Requirement
Status
Gap
Behavior documented per failure mode: machine dies, rack fails, DC offline, network split
❌
Not documented
Graceful degradation defined per dependency
❌
Not defined
Exponential backoff WITH jitter on all retries
⚠️ Partial
Controller-runtime has rate limiting but not all retries have jitter
Load shedding implemented (reject early in overload, serve degraded)
❌
PowerDNS has no documented load shedding for overload
Request deadlines/timeouts to prevent resource exhaustion
❌
Not validated
Data backup/restore tested
⚠️ Partial
LightningStream syncs LMDB to GCS, but restore procedure undocumented
Disaster recovery plan documented and drilled
❌
Not documented
"Fail sanely" on bad input (continue operating on previous state + alert)
❌
Not validated
Known failure mode: Orphaned downstream records (documented in docs/troubleshooting/dnsrecordset-downstream-orphan.md). The replicator can miss upstream deletions during restarts or timing gaps, leaving stale records in PowerDNS indefinitely. This caused a real incident (www.ab.dk stuck for ~24 hours). The replicator has no post-discovery mechanism for orphan cleanup.
2.5 Change Management & Release Process
Requirement
Status
Gap
Non-emergency rollouts proceed in stages
❌
No canary/release-channel mechanism for DNS
Canary first: deploy to small % then observe
❌
No canary deployment config
Staged rollouts across geographies
❌
No staged rollout
Rollback capability (roll back first, diagnose second)
⚠️ Partial
Flux can revert Kustomization, but LRRT is unknown
Feature flags for toggling features independently of binaries
❌
No feature flags
Rollouts supervised (monitoring watching for regressions)
⚠️ Partial
CI passes but no post-deploy monitoring
Releases cut on separate release branches
❌
Not observed
All code AND config in version control
✅
GitOps via Flux
2.6 Incident Response
Requirement
Status
Gap
Incident Command System with defined roles
❌
Not defined for DNS
Clear escalation paths documented
❌
Not documented
Declare incidents early (not "debug via bug tracker")
❌
No incident criteria defined
Playbooks/runbooks for every alert type
❌
No alerts → no runbooks
Postmortems required: blameless, with tracked action items
❌
No postmortem process for DNS
Practice incident response with drills
❌
Not done
Generic mitigations available: rollback, traffic drain, feature flag disable
Measure LRRT (how long from decision to revert to traffic restored)
Release branch strategy for hotfixes vs. normal releases
Post-deploy monitoring window (watch dashboards/alerts for 30min after every release)
3.7 Disaster Recovery
Document RPO and RTO targets for DNS service
Document and test LMDB restore procedure from GCS (LightningStream recovery)
Test failover from one edge cluster to another
Document procedure for total DNS rebuild (create PowerDNS, sync from GCS, update BGP)
Practice DR drill quarterly
Document procedure for handling TLD/registrar issues (domain expires, registrar unresponsive)
3.8 Customer-Facing (Go-to-Market)
Publish pricing page (free tier: X zones, Y records, Z queries/month)
Publish SLA commitment (availability %, credits for breaches)
Publish API deprecation policy (how long v1alpha1 is supported, migration path)
Create self-service signup flow (not just kubectl)
Write getting-started guide (5-minute quickstart)
Write migration guides from Route53, CloudDNS, Azure DNS, etc.
Publish customer-facing changelog and release notes
Create status page (or integrate with existing) for DNS-specific outages
Define data portability (export all zones/records as zone files or Terraform HCL)
3.9 Developer & Agent Experience
Build Terraform/Crossplane provider for DNS resource management
Build CLI plugin for common operations
Create UI/portal integration for DNS management (create zone, add records, view status)
Expose DNS Zone Discovery results in the portal (live DNS queries visible to users)
Create SDK or API client library if direct API access is supported
Document the customer's SLA/SLO expectations and how to monitor their own DNS
3.10 Known Technical Debt
Fix orphaned downstream record gap (replicator should reconcile shadow existence upstream periodically, not just on watch events)
Define programmatic interface for zone ownership accounting (ConfigMap is fragile — consider a CRD or a lock per zone)
Add end-to-end circuit breakers for PowerDNS API calls (avoid cascading failures)
Add cross-cluster health checking (replicator should detect downstream cluster unavailability and surface Programmed=False with appropriate reason)
Document the replication freshness budget (how stale can a shadow be before it's worth alerting)
Key Principles From Google SRE
Measure at the user, not the server — Gmail's perceived availability jumped 0.9% just by switching measurement to client-side. DNS should measure from recursive resolvers, not just from PowerDNS internal stats.
"Fail sanely" — On bad input, continue operating on previous valid state AND alert. Never accept empty/partial config that would cause a full outage. (Example: if LightningStream returns an empty snapshot, don't replace the LMDB with an empty database.)
"Roll back first, diagnose second" — Minimize MTTR. Document the exact rollback command and measure how long it takes.
Error budgets resolve dev vs. SRE tension — Define one. When the budget is spent, releases freeze until reliability is restored.
No email-only alerts — They are "the moral equivalent of piping to /dev/null." Every alert must page or be a dashboard panel.
Maximum 2 incidents per on-call shift — If DNS pages the on-call more than twice per 12-hour shift, prioritize fixing the pager load before adding features.
50% project time cap — If >50% of operational work is toil (manual intervention, firefighting), the service isn't production-ready. Automate or eliminate the toil.
The CRE launch checklist — Before a service can serve production traffic: (a) signed SLO document, (b) monitoring & alerting linked to SLOs, (c) incident response plan, (d) documented rollback procedure, (e) capacity tested, (f) security reviewed.
Summary
Your 5 original categories (Billing, Observability, Developer/Agent Experience, Documentation, Performance) cover ~60% of what's needed. The biggest gaps in your list:
Missing Category
Urgency
Why
Alerting + runbooks
Critical
You cannot operate without on-call pages. Every other subsystem has them.
SLOs + error budgets
Critical
You cannot measure or defend reliability without targets.
Security (DNSSEC, rate limiting, DDoS)
High
DNS is trivially amplifiable; customers can't trust unsigned responses.
Incident response + on-call
High
Who gets paged at 3am and what do they do?
Capacity planning + load testing
High
You have no idea how much traffic the system can handle.
Disaster recovery
Medium
LightningStream gives you GCS backups but no tested restore procedure.
Network policy
Medium
DNS control plane and agents have no network restrictions.
