I want to build a model and its APIs using the kubebuilder pattern. Don't use code examples or any particular programming language. Walk me through at the REST API layer or other interfaces how I would need to build in my language of choice
I'll walk you through the conceptual architecture and interfaces you need to build, independent of language.
You're essentially building two parallel systems that share data sources:
- The Reconciliation System (Kubebuilder's domain)
- The REST API System (Your webapp domain)
Both need access to the Kubernetes API and your database, but they serve different purposes.
You need an abstraction that can:
- GET a resource by namespace + name
- LIST resources (with optional label selectors, field selectors)
- CREATE a new resource
- UPDATE an existing resource (including status subresource)
- DELETE a resource
- WATCH for changes (streaming interface for reconciliation loop)
This client needs to understand:
- Your CRD's type system (marshaling/unmarshaling)
- Kubernetes authentication (usually in-cluster service account or kubeconfig)
- Retries and backoff for transient failures
Key point: Your REST handlers and your reconciler both use this same interface. The reconciler uses WATCH + GET/UPDATE. Your REST API primarily uses GET/LIST/UPDATE/DELETE.
A standard repository/DAO pattern:
- Find by resource UID (joining app data to K8s resources)
- Find by time range (metrics, history)
- Find by user (annotations, audit logs)
- Create/Update/Delete records
- Transaction support (for multi-table operations)
Design this with resource UID as the foreign key to link database records to Kubernetes resources. The UID is immutable, so it's perfect for joins.
This is where Kubebuilder patterns live. You need:
Input: A work queue that receives notifications when:
- A resource is created/updated/deleted
- A related resource changes (e.g., the Deployment your CRD manages)
- Time passes (requeue after delay)
Processing:
receive notification(namespace, name)
↓
fetch current resource state from K8s
↓
compute desired state
↓
compare current vs desired
↓
take actions (create/update/delete K8s objects)
↓
update resource status
↓
return (success | requeue after delay | error)
Side effects to database: Your reconciler can optionally write to the database:
- Log deployment events when spec changes
- Record reconciliation attempts
- Store computed metrics
But the reconciler never reads from the database to make decisions. It's purely K8s state → K8s state.
Here's where you diverge from pure Kubebuilder patterns. Your handlers need to:
receive HTTP request
↓
extract namespace, name from path
↓
[PARALLEL FETCH]
├─→ K8s: fetch CRD resource
└─→ DB: fetch related records by UID
↓
join the data structures
↓
transform to response DTO
↓
serialize to JSON
↓
return HTTP 200
Error handling:
- K8s returns 404 → HTTP 404
- DB query fails → HTTP 500
- Resource exists but no DB data → return K8s data with nulls for DB fields
receive HTTP request
↓
parse and validate JSON body
↓
construct CRD resource object
↓
K8s: create the resource
↓
[OPTIONAL] DB: create initial record with new UID
↓
return HTTP 201 with resource
Important: The reconciler will kick in automatically after K8s create completes. You don't need to trigger it manually.
receive HTTP request
↓
fetch current resource from K8s
↓
apply changes to spec
↓
K8s: update the resource
↓
[OPTIONAL] DB: log the change event
↓
return HTTP 200
The reconciler sees the spec change and acts on it.
This is pure database - doesn't touch K8s at all:
receive HTTP request
↓
verify resource exists in K8s (optional but recommended)
↓
parse annotation data
↓
DB: insert new annotation record with resource UID
↓
return HTTP 201
This is your "Rails model" layer. Build functions that understand how to combine data:
Function: GetResourceDetail(namespace, name)
k8s_data = kubernetes_client.Get(namespace, name)
resource_uid = k8s_data.metadata.uid
db_history = database.FindDeploymentHistory(resource_uid, limit=10)
db_metrics = database.FindRecentMetrics(resource_uid, since=now-5min)
db_annotations = database.FindAnnotations(resource_uid)
return DetailView {
name: k8s_data.metadata.name,
namespace: k8s_data.metadata.namespace,
spec: k8s_data.spec,
status: k8s_data.status,
deployment_history: db_history,
metrics: aggregate(db_metrics),
annotations: db_annotations,
health_score: calculate_health(k8s_data, db_metrics)
}
Function: ListResourcesWithMetrics(namespace, label_selector)
k8s_list = kubernetes_client.List(namespace, label_selector)
for each resource in k8s_list:
uid = resource.metadata.uid
metrics = database.GetLatestMetric(uid)
resource.enriched_data = metrics
return k8s_list
You need a single HTTP server process that:
Routes:
/healthz→ liveness probe (checks if process is alive)/readyz→ readiness probe (checks K8s client + DB are reachable)/metrics→ Prometheus metrics (optional but recommended)/api/*→ your REST API handlers
Middleware chain:
- Request logging
- Authentication (JWT, API keys, whatever you need)
- Authorization (RBAC based on K8s user + namespace)
- Error handling (catch panics, return proper status codes)
- CORS (if frontend needs it)
When your operator process starts:
1. Initialize logging
2. Load configuration (DB connection, K8s config)
3. Connect to database
- Run migrations if needed
- Verify connectivity
4. Connect to Kubernetes
- Load in-cluster config or kubeconfig
- Verify can list your CRDs
5. Start reconciliation manager
- Register your CRD reconciler
- Start work queues
- Begin watching resources
6. Start HTTP server
- Bind to port (e.g., 8080 for API, 8081 for metrics)
- Register all routes
7. Wait for shutdown signal
8. Gracefully shutdown
- Stop accepting new HTTP requests
- Drain in-flight reconciliations
- Close DB connections
1. Should REST API mutations go through K8s or directly to DB?
- Anything that affects desired state → goes through K8s (creates/updates to CRD)
- Anything that's supplementary metadata → can go directly to DB (annotations, comments)
2. Should the reconciler write to the database?
Optional, but useful for:
- Audit trail (what actions did the reconciler take?)
- Deployment history (log when spec.image changes)
- Error tracking (log reconciliation failures)
3. Should REST handlers trigger reconciliation?
No. Kubernetes watches handle this. When you update a CRD via the REST API, the watch stream notifies the reconciler automatically.
4. How do you handle eventual consistency?
The REST API shows what's in K8s + DB right now. The reconciler is working toward eventual consistency in the background. Your API responses should reflect current state, not promised future state.
Status fields tell you what the reconciler has accomplished:
spec.replicas = 5(what user wants)status.readyReplicas = 3(what exists now)
You need two separate auth contexts:
Your operator (reconciler) authenticates to K8s using:
- Service account token mounted at
/var/run/secrets/kubernetes.io/serviceaccount/token - Needs RBAC permissions to watch/get/update your CRDs and manage child resources
Your REST API authenticates users using:
- Whatever you choose (JWT, OAuth, API keys)
- Then maps user identity to K8s permissions (can they access this namespace?)
- Possibly uses K8s SubjectAccessReview API to check permissions
Think of your system as:
Kubebuilder gives you: A robust framework for "when K8s state changes, do this"
You're adding: A traditional webapp layer that says "let users view/edit K8s state + app data together"
The reconciler ensures K8s resources match their desired state. The REST API lets humans interact with that state and see it enriched with application data.
Does this give you the conceptual roadmap you need to implement in your language of choice?