AI Grid Progress Summary — July 14

Summary

AI Grid has moved from architecture sketch to repeatable Kind validation for the core routing story.

The routing foundation is now real enough to review: multi-cluster routing, API fallback, full-grid mixed backend routing, metrics-driven routing, GridSite discovery, and lost-peer route-away all have repeatable validation commands.
The next work is not adding more demo surfaces. The highest-value work is hardening the implementation behind the demos.
The main architecture boundary is now clear: Praxis core should stay generic, while AI/Grid routing semantics move to the AI repo.
The remaining design decisions are around production credential lifecycle, stale-state policy, GridSite lifecycle automation, metrics normalization, and deferred product surfaces like KV cache affinity, MCP federation, and budgets.

Architecture Shape

The system is settling into three layers:

Praxis core provides generic proxy infrastructure: filter runtime, routing hooks, TLS identity, config loading, and generic security primitives.
The AI repo owns AI-specific filters and protocol behavior, including Grid route inference/MCP routing after the closed Praxis PRs are moved.
The Grid operator owns control-plane state: provider inventory, scoring inputs, routing overlays, SWIM/CRDT membership, and Kubernetes-facing CRDs.

Layer	Responsibility	Current status
Praxis core	Generic proxy runtime, filter registry, HTTP/TLS plumbing, peer identity, generic trust/security hooks	Mostly established; should not grow Grid-specific semantics
AI repo	AI filters, API adapters, llm-d/EPP integration, Grid route inference/MCP filters	Existing AI filter registry is present; Grid route migration is active next work
Grid operator	CRDs, provider inventory, scoring, overlays, SWIM/CRDT state, Kind validation	Core routing path is Kind-proven
Grid/xtask generated config	Demo consumer/provider gateway config, route candidates, header injection	Demo-proven, but some behavior still harness-owned
SWIM/CRDT control plane	Membership, distributed provider propagation, stale detection	Proven for core cases; recovery/TTL/seed management remain

Claimable Demo Summary

These are the current engineering-review claims. The table intentionally separates “demo-proven” from production completeness.

Demo	Current status	What is proven	Main caveat
Demo 1: Basic Multi-Cluster Model Routing	Enabled / Kind-proven	Consumer routes `model-east` and `model-west` through provider gateways, `ext_proc`, mock EPP, endpoint selector, and inference-sim	Real production llm-d topology is not the claim
Demo 2: API Provider Fallback	Enabled / Kind-proven, caveated	Mock API fallback and gateway credential injection are proven; Secret-backed CRD-to-Kubernetes-Secret-to-Praxis config path is validated in xtask	Operator controller does not yet own credential lifecycle
Demo 3: Full Grid — Clusters + Cloud + APIs	Enabled / Kind-proven	One consumer gateway routes across local, remote, cloud-managed mock, and API-provider mock backends	Real cloud protocols/auth such as SigV4/OAuth2 are not exercised
Demo 4: Metrics-Driven Load Balancing	Enabled / smaller version Kind-proven	Prometheus-format queue-depth metrics change overlay order and request routing to the lower-queue backend	API-provider spillover under saturation still needs policy semantics
Demo 7: Site Join and Discovery	Enabled / discovery v1 + lifecycle v1 Kind-proven	Operator creates `GridSite` from SWIM Alive membership; lifecycle preservation, routing metadata, and isolation are proven	Cert exchange and automatic transition to `Active` remain
Demo 9: Failover Under Partition	Advanced validation / route-away Kind-proven	Lost peer becomes `fresh=false`; shared-model traffic routes through healthy east fallback	Real network partition, hard stale exclusion, recovery, and TTL remain
Demo 5: KV Cache Affinity	Deferred	Not claimable yet	Needs real prefix-cache signal source and provider-vs-endpoint routing design
Demo 8: MCP Tool Federation	Deferred	Not claimable yet	Tool catalog gossip, policy filtering, session semantics, and failover are separate product work
Demo 6: Budget Enforcement	Deferred	Not claimable yet	Needs budget/accounting/enforcement CRD/API design first

Praxis / AI Filter Boundary

The filter-location decision is no longer open: Grid route inference and MCP routing are moving to the AI repo.

Praxis should keep generic mechanics: filter registration, context propagation, mTLS identity, header/routing primitives, and generic trust enforcement.
AI should own model/provider/tool semantics: grid_route, inference candidates, freshness/locality policy, MCP tool candidates, and AI protocol behavior.
Closed Praxis PRs #802 and #804 should be ported to AI rather than reopened in Praxis.
Praxis PR #803 is different: peer identity enforcement is a generic security primitive and can remain in Praxis if named/scoped generically.

Item	Direction	Reason
`grid_route` inference routing	Move to AI repo	Model names, providers, locality, freshness, and overlays are AI/Grid semantics
`grid_route` MCP tool routing	Move to AI repo	MCP tool routing is an AI/MCP product concern
mTLS peer identity enforcement	Keep generic primitive in Praxis	Allow/deny by peer identity is generally useful proxy security behavior
Temporary Grid proxy server	Avoid unless blocked	It creates a second data-plane path to explain, test, and later remove

Needed AI repo work:

Recover the closed #802/#804 code and port equivalent filters into the AI filter crate.
Register grid_route through the AI registry so Praxis core does not need to know Grid semantics.
Update AI’s pinned Praxis SHA to current upstream with the needed generic primitives.
Move docs/examples into AI and verify Grid-generated config still matches the AI filter shape.

Grid Operator Capabilities

The operator now handles more than static overlay rendering. It has a working control-plane loop for local providers, remote CRDT providers, metrics, SWIM membership, stale state, and GridSite discovery.

Overlay generation is the central integration point. The operator ranks candidates and writes config that the data plane consumes.
SWIM and CRDT now provide real distributed state propagation for remote provider inventory.
Metrics are wired into scoring through configured endpoints, but the metric contract is still narrow.
GridSite discovery v1 is now operator-owned when explicitly enabled.

Capability	Current implementation	Remaining gap
Routing overlay generation	Local and CRDT-remote providers become ordered route candidates	Final data-plane ownership depends on AI-side `grid_route` migration
Candidate ordering	Score ordering plus `fresh=true` before `fresh=false` tiebreaker	Need recovery and explicit stale policy
Provider health/freshness	HTTP/HTTPS probes and phase-to-fresh mapping	mTLS health probing not wired
Metrics scoring	`metricsConfig` scrapes Prometheus-format queue-depth during reconcile	Raw vLLM/llm-d metric normalization undecided
SWIM membership	foca-backed runtime forms membership and reports connected sites	CRD-driven runtime seed management remains
CRDT propagation	Provider snapshots gossip over SWIM and feed overlay input	TTL/stale-age behavior remains
GridSite discovery	Opt-in GridNetwork label creates GridSite for Alive SWIM peers	Cert exchange and automatic phase transitions remain

Credential and API Provider Handling

The API-provider demos are credible for behavior, but the credential lifecycle is not production-complete.

Current validation proves a real Secret reference path exists: InferenceProvider.spec.auth.secretRef points to a Kubernetes Secret, xtask resolves it, and the generated Praxis config injects the bearer token.
This is still harness-driven. The operator controller does not yet read spec.auth and own production config generation.
Kubernetes Secrets are acceptable as a v1 backend, but the design should leave room for Vault, External Secrets, OAuth2, SigV4, and workload identity.
Embedding a token in a ConfigMap is acceptable for the validation harness, not a production target.

Area	Current state	Next step
Credential source	`spec.auth.secretRef` validated through xtask	Move resolution/projection into controller-owned path
Secret backend	Kubernetes Secret v1	Keep resolver boundary open for external backends
Config delivery	Token appears in generated ConfigMap	Prefer Secret volume or native Praxis secret reference
Injection scope	Filter-chain header injection	Add per-cluster/per-upstream injection semantics
Rotation	Not implemented	Add watch/reconcile/reload behavior

Site Join and Discovery

GridSite discovery has crossed the important threshold from harness-created to operator-created.

When the GridNetwork has the opt-in label, the operator observes remote Alive SWIM members and creates a GridSite.
The created GridSite includes network reference, egress address, and starts at phase=Discovered.
The lifecycle state model exists and phase preservation is proven, but automatic progression beyond Discovered is still future work.
The next major feature is trust bootstrap: certificate exchange, identity verification, and readiness-driven transition to Active.

Phase / component	Current behavior	Outstanding

| Discovery trigger | Remote Alive SWIM member | Opt-in UX may become CRD field/default later | | GridSite creation | Operator applies {network}-{site_id} GridSite | Need stale cleanup/TTL policy | | Lifecycle | Harness can patch phases; operator preserves them | Operator does not drive transitions beyond Discovered | | Trust bootstrap | Not implemented | Cert exchange and identity verification | | Data-plane readiness | Metadata exists | Need readiness checks before Active |

Failover and Partition Behavior

Failover is now more than an overlay-only signal, but it is still not the full partition story.

The operator observes lost SWIM peers and downgrades their CRDT providers to Degraded.
The overlay keeps stale candidates for observability but marks them fresh=false.
Shared-model route-away is proven: after west is killed, the healthy east candidate wins and the consumer request returns 200.
The policy question remains: should stale candidates be deprioritized, used only as last resort, or excluded entirely?

Capability	Current proof	Remaining work
Lost peer detection	Process kill drives SWIM Dead/Suspect behavior	Add real network partition simulation
Stale signaling	Remote candidates become `fresh=false`	Add stale-age/TTL policy
Route-away	Shared model routes to healthy fallback	Decide hard exclusion vs last-resort behavior
Recovery	Not proven	Prove rejoin restores `fresh=true`
Observability	Stale candidate remains in overlay	Decide if retained stale candidates are acceptable in production config

Metrics and Scoring

Metrics-driven routing is proven for a bounded case, not yet a complete capacity policy.

The current validation uses a Prometheus-format metrics endpoint that exposes normalized queue depth.
The operator scrapes that endpoint during reconcile and changes route order based on queue depth.
This should not be described as “Prometheus server integration.” It is endpoint scraping of Prometheus-format metrics.
Before KV cache affinity, we need to decide whether Grid consumes normalized metrics or owns normalization from raw vLLM/llm-d metrics.

Topic	Current state	Decision needed
Queue depth	Prometheus-format endpoint scraped during reconcile	Keep wording precise
Normalization	Demo uses pre-normalized 0.0-1.0 values	Decide Grid-owned vs exporter-owned normalization
Invalid values	Happy path works	Handle NaN/out-of-range/stale values
Saturation spillover	Not claimed	Define when API provider beats self-hosted capacity
KV cache affinity	Deferred	Define signal ownership and routing layer split

Deferred Demo Surfaces

The deferred demos are valuable, but starting them now would pull effort away from hardening the validated routing/SWIM foundation.

KV cache affinity depends on signal ownership: Grid-level provider routing and llm-d/EPP endpoint routing must not fight each other.
MCP federation is a separate product surface. Moving grid_route MCP routing to AI is only the first step, not full federation.
Budget enforcement is too large to treat as a routing demo. It needs identity, accounting, policy, enforcement, and API design first.

Deferred area	Why deferred	Revisit trigger
KV Cache Affinity	Needs prefix-cache signal source and provider-vs-endpoint routing design	After metrics normalization is settled
MCP Tool Federation	Needs distributed catalog, policy filtering, session semantics, failover	After AI-side MCP routing and Grid capability records are stable
Budget Enforcement	Needs budget/accounting/enforcement CRD/API design	After credential and request-accounting boundaries are clear

Validation and CI Posture

The project is using validation as the forcing function for architecture claims.

Static gates are consistently part of the workflow: fmt, check, test, lint, doc, and diff-check.
Production coverage is above the 80% gate, with xtask excluded because it is a Kind harness.
Kind validations are repeatable but should run sequentially, especially SWIM-heavy tests.
Generated artifacts such as coverage.json are ignored.

Area	Current state	Notes
Static gates	fmt/check/test/lint/doc/diff-check used on implementation chunks	Continue before commits
Coverage	Production coverage gate passes above 80%	xtask excluded from threshold
Kind validations	Core demo commands are repeatable	Run SWIM-heavy validations sequentially
CI hardening	Coverage, MSRV, unicode safety workflows added	Keep aligned with Praxis/AI practices

Test-Driven Development Strategy

The best pattern so far is: state the product claim, encode it as a repeatable validation command, then add unit tests around the logic seams that make the command reliable.

Start with a claim: what should the system prove after this change?
Add pure tests for policy logic before wiring Kubernetes or networking.
Add negative tests at production boundaries: missing Secret, invalid metric, stale peer, bad seed, empty candidates.
Keep xtask as the validation harness. If xtask owns product behavior, the claim is weaker.
Require a Kind proof for any externally stated demo claim.
Keep docs honest by saying what the validation does not prove.

Test layer	Purpose	Current examples
Pure unit tests	Pin deterministic logic without Kubernetes/networking	scoring, freshness ordering, CRDT merge, GridSite naming
Controller unit tests	Validate controller helper behavior	discovered-site extraction, grid ID resolution, staleness override
Schema/config tests	Prove generated config matches runtime expectations	Grid overlay-to-filter shape, AI/Praxis examples
Focused integration tests	Validate one runtime boundary	AI registry tests, mock API credential proof
Kind E2E	Prove user-visible product claims	demo routing, metrics, discovery, failover commands
Regression suite	Prevent prior demos from breaking	sequential core Kind validations
Coverage gate	Catch production logic with no tests	production crates over 80%; xtask excluded

Current Decision List

These are the real remaining architecture decisions. Filter location is not listed because that decision is made: Grid route filters move to AI.

Should we ever use a temporary Grid proxy server? Recommendation: avoid unless blocked, because it creates a second data-plane story.
How should API credentials be projected? Recommendation: Kubernetes Secret as v1 backend, with resolver boundary for Vault, External Secrets, OAuth2, SigV4, and workload identity.
What does fresh=false mean in production? Current behavior is deprioritize and retain; we need to decide last-resort vs hard exclusion.
How should GridSite auto-discovery be enabled long-term? Current opt-in label is safe; a CRD field/default may be better product UX.
Who normalizes model-serving metrics? Decide before KV cache affinity.

Next Steps

The next sequence should harden existing claims before opening more surfaces.

Move closed Praxis #802/#804 route filters into AI and bump AI Praxis pins.
Make API credential projection controller-owned instead of xtask-owned.
Implement GridSite join lifecycle phase 2: cert exchange, trust bootstrap, and transition toward Active.
Add failover recovery validation proving rejoin restores freshness and routing eligibility.
Wire CRD-driven SWIM seeds so operator deployment is less env-var dependent.
Define the metrics contract and normalization model, then revisit KV cache affinity (this is a big lift and tightly coupled to llm-d which is still meh on integration so tbd 😬).

nerdalert/grid-status-jul14.md

Select an option

No results found