Datum Cloud Infrastructure Service Catalog & API Server Inventory (2026-04-10)

gistfile0.txt

# Datum Cloud Infrastructure Service Catalog

> Generated 2026-04-10 from live Kubernetes clusters (prod + staging).  
> Covers all Deployments, StatefulSets, and DaemonSets across both environments.

---

## API Server Inventory

### Aggregated API Servers in the Host GKE Cluster

Only **3** services are registered as true Kubernetes aggregated API servers (`APIService` objects backed by a real service endpoint):

| APIService Group | Backing Service | Namespace | Purpose |
|---|---|---|---|
| `v1beta1.metrics.k8s.io` | `metrics-server` | `kube-system` | Standard GKE pod/node metrics |
| `v1alpha1.identity.miloapis.com` | `auth-provider-zitadel-apiserver` | `datum-iam-system` | Zitadel identity aggregated API |
| `v1alpha1.acme.datumapis.com` | `externaldns-webhook` | `cert-manager` | DNS-01 challenge webhook shim for cert-manager |

Everything else in the host cluster API surface (`infrastructure.miloapis.com`, `dns.networking.miloapis.com`, Flux, Kyverno, Crossplane, etc.) is served **locally via CRDs** — no separate process, all stored in the host etcd.

### The 4 Other `*-apiserver` Deployments

These do not appear in the host cluster's `APIService` list because they are registered into a second Kubernetes control plane (Karmada), not the host GKE cluster.

| Deployment | Namespace | Image | What It Actually Is |
|---|---|---|---|
| `karmada-apiserver` | `karmada-system` | `kube-apiserver:v1.33.2` | A **real kube-apiserver binary** — Karmada's own full Kubernetes control plane, backed by its own etcd. Not kubebuilder, not an app server. |
| `karmada-aggregated-apiserver` | `karmada-system` | `karmada/karmada-aggregated-apiserver:v1.15.0` | Kubebuilder-pattern aggregated API server registered **inside** the Karmada cluster for Karmada-specific APIs (e.g. `cluster.karmada.io`). |
| `milo-apiserver` | `datum-system` | `datum-cloud/milo:v0.24.3` | Kubebuilder/apiserver-builder pattern — Milo platform API server. Registered inside the Karmada cluster, not the host GKE cluster (no host-side `*.miloapis.com` aggregation except identity). |
| `activity-apiserver` | `activity-system` | `datum-cloud/activity:v0.3.3` | Same pattern — activity timeline aggregated API server, registered inside the Karmada cluster. |

### Classification: "Inside Kubernetes" vs "Inside Application"

**Aggregated API server pattern (kubebuilder/apiserver-builder — "inside Kubernetes"):**
- `auth-provider-zitadel-apiserver` → registered in host GKE cluster
- `karmada-aggregated-apiserver` → registered in Karmada cluster
- `milo-apiserver` → registered in Karmada cluster
- `activity-apiserver` → registered in Karmada cluster

**Full Kubernetes control plane (is a kube-apiserver binary):**
- `karmada-apiserver` → Karmada's own peer Kubernetes cluster API server

**Webhook registered via aggregation (not a true domain API server):**
- `externaldns-webhook` → uses aggregation mechanism as a cert-manager DNS-01 hook

**Total: 7 API server processes across 2 Kubernetes control planes** (host GKE + Karmada virtual cluster). The per-application pattern (one aggregated API server per domain: identity, activity, milo-platform) is consistent with the "Custom Kubernetes API servers for multi-tenancy" architecture.

---

## Service Catalog

> **Uptime requirement methodology:** If a dependent service has a 99.9% SLO, anything in its synchronous call path must be at least as reliable. Services shared across many dependents need more nines because their downtime simultaneously caps every downstream SLO.

| Tier | Required | Max Downtime/Year | Who |
|---|---|---|---|
| **4 nines** | 99.99% | ~52 min | etcd, NATS, Cilium/CNI, cert-manager, Kyverno admission, Envoy proxies, Zitadel, OpenFGA, PowerDNS, ClickHouse Keeper, KMS plugin, GKE PD CSI, GKE Metadata Server, Karmada APIServer |
| **3 nines** | 99.9% | ~8.7 hr | Most platform services, app services, Redis, VictoriaMetrics, Loki, cloud-portal, DNS operator, Meilisearch |
| **2.5 nines** | 99.5% | ~43 hr | Tempo, Pyroscope (non-critical observability, single-node SPOFs) |
| **2 nines** | 99% | ~87 hr | Sentry, NetBox, MCP servers, Storybook (dev tooling / internal) |

### Platform Infrastructure

| Service | Namespace | Kind | Prod Replicas | Data Persistence | Required Uptime | Notes |
|---|---|---|---|---|---|---|
| GKE Cluster (Kubernetes API) | kube-system | Managed | HA | Google Cloud (provider-managed) | 99.95% | Provider SLO |
| etcd (platform) | etcd-system | StatefulSet | 3 | PersistentVolume (GCE PD) — 3-node Raft | **99.99%** | Every k8s resource and controller depends on it |
| etcd (Karmada) | karmada-etcd-system | StatefulSet | 3 | PersistentVolume (GCE PD) — 3-node Raft | **99.99%** | Backs Karmada API; multi-cluster dispatch depends on it |
| Cilium CNI (anetd) | kube-system | DaemonSet | per-node | Stateless (node kernel state) | **99.99%** | Node networking; loss = pod connectivity loss cluster-wide |
| kube-dns | kube-system | Deployment | 5 | Stateless (backed by etcd) | **99.99%** | Service discovery for every in-cluster workload |
| cert-manager | cert-manager | Deployment | 1 | k8s etcd | **99.99%** | Issues TLS certs for all services; outage blocks new pod starts and cert renewals |
| cert-manager-cainjector | cert-manager | Deployment | 1 | k8s etcd | **99.99%** | Injects CA bundles into webhooks; outage breaks admission |
| cert-manager-webhook | cert-manager | Deployment | 1 | Stateless | **99.99%** | Validates Certificate resources; outage blocks cert creation |
| cert-manager CSI Driver | cert-manager | DaemonSet | per-node | Stateless | **99.99%** | Mounts certs into pods; loss prevents pod startup |
| External Secrets Operator | external-secrets | Deployment | 1 | k8s etcd | **99.99%** | Pulls all secrets from GCP Secret Manager |
| external-secrets-webhook | external-secrets | Deployment | 1 | Stateless | **99.99%** | Validates ExternalSecret resources |
| Crossplane | crossplane-system | Deployment | 1 | k8s etcd | 99.9% | Provisions GCP Cloud SQL / GCS / networking |
| Crossplane GCP Providers | crossplane-system | Deployment | 1 each | k8s etcd + GCP APIs | 99.9% | Manages cloud resources; runtime impact only if resources need reprovisioning |
| ClickHouse Operator | clickhouse-system | Deployment | 1 | k8s etcd | 99.9% | Manages all ClickHouse installs; outage freezes CH config changes |
| k8s CloudKMS Plugin | kube-system | DaemonSet | per-node | Stateless (wraps GCP KMS) | **99.99%** | Encrypts etcd at rest |
| GKE Persistent Disk CSI | kube-system | DaemonSet | per-node | Stateless | **99.99%** | Provides PersistentVolume mounts; loss = stateful pods cannot start |
| GKE Metadata Server | kube-system | DaemonSet | per-node | Stateless | **99.99%** | Provides Workload Identity tokens; loss = GCP API calls fail cluster-wide |

### GitOps

| Service | Namespace | Kind | Prod Replicas | Data Persistence | Required Uptime | Notes |
|---|---|---|---|---|---|---|
| Flux (source + kustomize + helm) | flux-system | Deployment | 1 each | k8s etcd + Git OCI | 99.9% | Drives GitOps reconciliation; outage freezes config changes but running workloads continue |
| Flux Image Automation | flux-system | Deployment | 1 | k8s etcd | 99.9% | Auto-updates image tags in Git; non-critical path |

### Networking

| Service | Namespace | Kind | Prod Replicas | Data Persistence | Required Uptime | Notes |
|---|---|---|---|---|---|---|
| Envoy Gateway (control plane) | envoy-gateway-system | Deployment | 1 | k8s etcd (Gateway API CRDs) | **99.99%** | Programs all ingress Envoy proxies |
| Envoy Proxies (per-gateway) | envoy-gateway-system | Deployment | 3 each | Stateless | **99.99%** | Handles all external traffic; direct user-facing critical path |
| AI Gateway Controller | envoy-gateway-system | Deployment | 1 | Stateless | 99.9% | AI traffic routing; subset of traffic |
| External DNS | external-dns | Deployment | 1 | Stateless | 99.9% | Syncs ingress hostnames to DNS |
| external-dns-webhook (GKE→datum) | external-dns-gke-to-datum-dns | Deployment | 1 | Stateless | 99.9% | Routes GKE LB IPs to Datum DNS; prod-only |
| Datum-Net | datum-net | Deployment | 2 | Stateless | 99.9% | Network resource management service |
| Galactic Router | galactic-router | Deployment | 1 | Stateless | 99.9% | Traffic routing service; prod-only |
| Iroh Relay (downstream gateway) | datum-downstream-gateway | Deployment | 1 | Stateless | 99.9% | Relay for Iroh P2P protocol; staging-only currently |
| KnotDNS (auth DNS agent) | datum-auth-dns | DaemonSet | per-node | Local LMDB | **99.99%** | Edge authoritative DNS; staging-only |

### Networking / DNS

| Service | Namespace | Kind | Prod Replicas | Data Persistence | Required Uptime | Notes |
|---|---|---|---|---|---|---|
| DNS Operator Controller | datum-dns-system | Deployment | 1 | k8s etcd | **99.99%** | Reconciles DNS zones and records |
| PowerDNS Auth + Recursor | datum-dns-system | StatefulSet | 3 | LMDB + LightningStream replication | **99.99%** | Authoritative DNS for datum.net; outage = DNS resolution failure |

### Multi-Cluster (Karmada)

| Service | Namespace | Kind | Prod Replicas | Data Persistence | Required Uptime | Notes |
|---|---|---|---|---|---|---|
| Karmada API Server | karmada-system | Deployment | 3 | karmada-etcd | **99.99%** | Multi-cluster API; workload dispatch depends on it |
| Karmada Aggregated APIServer | karmada-system | Deployment | 2 | karmada-etcd | 99.9% | Karmada-specific APIs (cluster.karmada.io) |
| Karmada Controller Manager | karmada-system | Deployment | 2 | karmada-etcd | 99.9% | Reconciles multi-cluster resources |
| Karmada Scheduler | karmada-system | Deployment | 2 | karmada-etcd | 99.9% | Schedules workloads to member clusters |
| Karmada Webhook | karmada-system | Deployment | 2 | Stateless | **99.99%** | Admission for multi-cluster resources |

### Security / IAM

| Service | Namespace | Kind | Prod Replicas | Data Persistence | Required Uptime | Notes |
|---|---|---|---|---|---|---|
| Zitadel | datum-iam-system | Deployment | 3 | External PostgreSQL (Cloud SQL) | **99.99%** | Issues tokens for every user/service auth; outage = login failure everywhere |
| Zitadel Login UI | datum-iam-system | Deployment | 1 | Stateless | **99.99%** | User-facing login flow |
| Auth Provider (Zitadel) APIServer | datum-iam-system | Deployment | 1 | k8s etcd | **99.99%** | Aggregated API server for `identity.miloapis.com` |
| Auth Provider (Zitadel) Webhook | datum-iam-system | Deployment | 1 | Stateless | **99.99%** | AuthN webhook; outage blocks pod auth |
| OpenFGA | datum-iam-system | Deployment | 2 | External PostgreSQL (Cloud SQL) | **99.99%** | Fine-grained authorization; every API call goes through it |
| Auth Provider (OpenFGA) Webhook | datum-iam-system | Deployment | 3 | Stateless | **99.99%** | AuthZ webhook; outage = all requests denied or allowed |
| Auth UI | datum-iam-system | Deployment | 2 | Stateless | 99.9% | User-facing authentication pages |
| Kyverno (admission) | kyverno-system | Deployment | 1 | k8s etcd | **99.99%** | Admission webhook; failClosed = no new pods if down |
| Fraud Detection | fraud-system | Deployment | 1 | k8s etcd | 99.9% | Fraud detection controller |

### Messaging

| Service | Namespace | Kind | Prod Replicas | Data Persistence | Required Uptime | Notes |
|---|---|---|---|---|---|---|
| NATS | nats-system | StatefulSet | 3 | PersistentVolume (JetStream — 3-node cluster) | **99.99%** | Core event bus for activity and control plane messaging |
| NACK (JetStream controller) | nats-system | Deployment | 1 | k8s etcd | 99.9% | Manages JetStream streams/consumers declaratively |

### Platform Control Plane

| Service | Namespace | Kind | Prod Replicas | Data Persistence | Required Uptime | Notes |
|---|---|---|---|---|---|---|
| Milo API Server | datum-system | Deployment | 3 | k8s etcd (Milo CRDs) | **99.99%** | Core platform API; all resource management goes through it |
| Milo Controller Manager | datum-system | Deployment | 1 | k8s etcd | 99.9% | Reconciles Milo resources |
| Datum Controller Manager | datum-system | Deployment | 1 | k8s etcd | 99.9% | Core resource lifecycle management |
| Network Services Operator | datum-system | Deployment | 3 | Redis + k8s etcd | 99.9% | Manages network service resources |
| Infra Provider GCP | datum-system | Deployment | 1 | k8s etcd | 99.9% | Provisions GCP infrastructure resources |
| Workload Operator | datum-system | Deployment | 1 | k8s etcd | 99.9% | Manages compute workload lifecycle |
| Telemetry Services Operator | datum-system | Deployment | 1 | k8s etcd | 99.9% | Manages telemetry pipeline resources |
| Activity API Server | activity-system | Deployment | 3 | ClickHouse + k8s etcd | 99.9% | Serves activity timeline APIs |
| Activity Controller | activity-system | Deployment | 1 | ClickHouse + k8s etcd | 99.9% | Processes events into activity records |
| Activity Processor | activity-system | Deployment | 2 | ClickHouse | 99.9% | Processes activity stream events |

### Applications

| Service | Namespace | Kind | Prod Replicas | Data Persistence | Required Uptime | Notes |
|---|---|---|---|---|---|---|
| Cloud Portal | cloud-portal | Deployment | 2 | Stateless (backed by Redis + Milo + Zitadel) | 99.9% | Primary user-facing web portal |
| GraphQL Gateway | graphql-gateway | Deployment | 2 | Stateless | 99.9% | API gateway for GraphQL consumers |
| CloudValid Backend | cloudvalid-system | Deployment | 2 | External (Cloud SQL) | 99.9% | Backend for CloudValid service |
| CloudValid Frontend | cloudvalid-system | Deployment | 2 | Stateless | 99.9% | User-facing CloudValid UI |
| CloudValid Docs | cloudvalid-system | Deployment | 1 | Stateless | 99% | Static documentation site |
| Milo OS (marketing site) | milo-os-com | Deployment | 2 | Stateless (static/SSR) | 99.9% | Public-facing marketing/product site |
| Email Provider (Loops) | datum-notification-system | Deployment | 1 | k8s etcd | 99.9% | Sends emails via Loops |
| Email Provider (Resend) | datum-notification-system | Deployment | 1 | k8s etcd | 99.9% | Sends emails via Resend |

### Data Stores

| Service | Namespace | Kind | Prod Replicas | Data Persistence | Required Uptime | Notes |
|---|---|---|---|---|---|---|
| ClickHouse cluster (activity) | activity-system | StatefulSet | 3 (1 pod/shard) | PersistentVolume — 3-shard | 99.9% | Stores all activity timeline data |
| ClickHouse Keeper (activity) | activity-system | StatefulSet | 3 | PersistentVolume — 3-node quorum | **99.99%** | Coordinates ClickHouse cluster; loss = writes stop |
| ClickHouse (edge-logs) | edge-logs-system | StatefulSet | 1 | PersistentVolume (GCE PD) — single node | 99.9% | Stores edge log data; single-node SPOF |
| Redis (network-services) | datum-system | StatefulSet | 1 | PersistentVolume — standalone | 99.9% | Session/cache for network-services-operator |
| Redis (cloud-portal) | cloud-portal | StatefulSet | 1 | PersistentVolume — standalone | 99.9% | Session store for cloud-portal |
| Meilisearch | search-system | StatefulSet | 1 | PersistentVolume — single node | 99.9% | Full-text search; single-node SPOF |
| Valkey (NetBox primary) | netbox-system | StatefulSet | 1 | PersistentVolume | 99% | Cache for NetBox only |
| Valkey (NetBox replicas) | netbox-system | StatefulSet | 3 | PersistentVolume | 99% | Read replicas for NetBox Valkey |
| Sentry ClickHouse | sentry-system | StatefulSet | 1 | PersistentVolume — single node | 99% | Stores Sentry event data; single-node SPOF |
| Sentry Kafka | sentry-system | StatefulSet | 3 | PersistentVolume — 3-node KRaft | 99% | Event queue for Sentry ingest only |
| Sentry Redis | sentry-system | StatefulSet | 1+1 | PersistentVolume | 99% | Cache/queue for Sentry only |

### Observability

| Service | Namespace | Kind | Prod Replicas | Data Persistence | Required Uptime | Notes |
|---|---|---|---|---|---|---|
| VictoriaMetrics vmselect | telemetry-system | StatefulSet | 4 | Stateless (queries vmstorage) | 99.9% | Query layer; 4 replicas provide HA |
| VictoriaMetrics vmstorage | telemetry-system | StatefulSet | 3 | PersistentVolume — 3 shards | 99.9% | Stores all platform metrics |
| VMAgent (resource metrics) | datum-system | Deployment | 1 | Stateless (scraper) | 99.9% | Scrapes and ships metrics to VM |
| VMAlert | datum-system | Deployment | 1 | Stateless | 99.9% | Evaluates alerting rules |
| VMSingle (resource metrics) | datum-system | Deployment | 1 | PersistentVolume | 99.9% | Single-node VM for datum resource metrics |
| VMAlertmanager | telemetry-system | StatefulSet | 1 | PersistentVolume (silences) | 99.9% | Routes alerts; brief outage = missed alerts |
| Loki (single-binary) | telemetry-system | StatefulSet | 1 | GCS Object Storage | 99.9% | Log aggregation; **single-node SPOF** |
| Tempo | telemetry-system | StatefulSet | 1 | PersistentVolume — single node | 99.5% | Distributed tracing; **single-node SPOF** |
| Pyroscope | telemetry-system | StatefulSet | 1 | PersistentVolume | 99.5% | Continuous profiling; non-critical |
| Pyroscope Alloy | telemetry-system | StatefulSet | 1 | Stateless | 99.5% | Profiling data collector |
| OpenTelemetry Operator | open-telemetry-system | Deployment | 1 | k8s etcd | 99.9% | Manages OTel collector CRDs |
| OpenTelemetry Collector (platform) | open-telemetry-system | Deployment | 1 | Stateless (pipeline) | 99.9% | Central telemetry aggregation pipeline |
| OTel Collector (logs) | edge-logs-system | Deployment | 2 | Stateless (pipeline) | 99.9% | Collects and routes logs to edge ClickHouse |
| Vector Aggregator (activity) | activity-system | Deployment | 2 | Stateless (pipeline) | 99.9% | Routes log/event data to ClickHouse |
| Custom Resource State Metrics | datum-system | Deployment | 1 | Stateless | 99.9% | Exposes CRD-level metrics to VictoriaMetrics |

### Internal Tooling / Dev Tools

| Service | Namespace | Kind | Prod Replicas | Data Persistence | Required Uptime | Notes |
|---|---|---|---|---|---|---|
| Sentry | sentry-system | Deployment | varies | ClickHouse + Kafka + Redis + PostgreSQL | 99% | Error tracking; not in customer-facing runtime path |
| Sentry Taskbrokers (×4) | sentry-system | StatefulSet | 1 each | Stateless (backed by Kafka) | 99% | Async task processing for Sentry |
| NetBox | netbox-system | Deployment | 1 | PostgreSQL (Cloud SQL) + Valkey | 99% | IPAM/DCIM; internal tooling; prod-only |
| flux-mcp-server | mcp-system | Deployment | 1 | Stateless | 99% | MCP server for AI tooling access to Flux |
| victoria-metrics-mcp-server | mcp-system | Deployment | 1 | Stateless | 99% | MCP server for AI tooling access to metrics |
| Datum Storybook | datum-storybook | Deployment | 1 | Stateless | 99% | Component library documentation |

---

## Data Persistence Summary

| Storage Type | Services Using It |
|---|---|
| **k8s etcd (host GKE)** | All CRD-backed operators: Flux, Crossplane, cert-manager, External Secrets, Kyverno, Network Services Operator, Milo controllers, DNS Operator, OTel Operator, etc. |
| **Separate etcd (GCE PD, 3-node Raft)** | Platform etcd (`etcd-system`), Karmada etcd (`karmada-etcd-system`) |
| **GCE Persistent Disk (PV)** | ClickHouse (activity, edge-logs, sentry), Redis (cloud-portal, network-services, sentry), Valkey (netbox), Meilisearch, VMStorage, VMAlertmanager, Loki (chunks cache), Tempo, Pyroscope |
| **GCS Object Storage** | Loki (primary chunks + index store via Crossplane-managed bucket) |
| **External PostgreSQL (Cloud SQL)** | Zitadel, OpenFGA, Sentry, NetBox, CloudValid |
| **LMDB + LightningStream replication** | PowerDNS (datum-dns-system) — local fast LMDB replicated across nodes via LightningStream |
| **NATS JetStream (PV-backed)** | NATS cluster itself (3 PVs, 3-node quorum) |
| **Stateless** | Most controllers, proxies, and API gateways |