issue-44-grid.txt

Demo:         grid-glb-demo
Subcommand:   run-grid-glb-demo
Forge config: /home/fedora/praxxis/ai-grid/prs/44-grid-demo-image-source/demos/demos/grid-glb-demo/forge.yaml
Demo root:    /home/fedora/praxxis/ai-grid/prs/44-grid-demo-image-source/demos/demos/grid-glb-demo
Grid repo:    /home/fedora/praxxis/ai-grid/prs/44-grid-demo-image-source/grid

   Compiling xtask v0.1.3 (/home/fedora/praxxis/ai-grid/prs/44-grid-demo-image-source/grid/xtask)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 7.95s
   Compiling libc v0.2.186
   Compiling serde_core v1.0.228
   Compiling serde v1.0.228
   Compiling serde_json v1.0.150
   Compiling clap_builder v4.6.0
   Compiling getrandom v0.2.17
   Compiling getrandom v0.3.4
   Compiling getrandom v0.4.3
   Compiling ring v0.17.14
   Compiling tempfile v3.27.0
   Compiling rand_core v0.9.5
   Compiling rand_chacha v0.9.0
   Compiling rand v0.9.4
   Compiling clap v4.6.1
   Compiling rcgen v0.14.8
   Compiling toml_datetime v0.6.11
   Compiling serde_spanned v0.6.9
   Compiling serde_yaml v0.9.34+deprecated
   Compiling toml_edit v0.22.27
   Compiling certs v0.1.3 (/home/fedora/praxxis/ai-grid/prs/44-grid-demo-image-source/grid/certs)
   Compiling toml v0.8.23
   Compiling xtask v0.1.3 (/home/fedora/praxxis/ai-grid/prs/44-grid-demo-image-source/grid/xtask)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 16.46s
     Running `target/debug/xtask env run-grid-glb-demo --forge-config /home/fedora/praxxis/ai-grid/prs/44-grid-demo-image-source/demos/demos/grid-glb-demo/forge.yaml --quick --teardown --keep-on-failure`
Forge config: /home/fedora/praxxis/ai-grid/prs/44-grid-demo-image-source/demos/demos/grid-glb-demo/forge.yaml
Demo root:    /home/fedora/praxxis/ai-grid/prs/44-grid-demo-image-source/demos/demos/grid-glb-demo

===============================================================================
ENVIRONMENT SETUP
===============================================================================

[SETUP 1/9] Staging demo certificates and provider identities
  generated CA certificate
  generated cert for east-edge (SAN: east-edge.grid.internal)
  generated cert for west-edge (SAN: west-edge.grid.internal)
  generated cert for edge-untrusted (SAN: edge-untrusted.grid.internal)
  generated cert for east-provider (SAN: east-provider.grid.internal)
  generated cert for west-provider (SAN: west-provider.grid.internal)
  generated wrong-org cert (org=not-ai-grid, expected=ai-grid)
  reusing existing CA certificate

[SETUP 2/9] Creating five Kind clusters on one shared cross-cluster network
            Forge will report again after cluster creation completes.

[SETUP 3/9] Resolving and loading runtime images
  gateway:       ghcr.io/praxis-proxy/grid-ai-rollup:v0.1.3
  operator:      ghcr.io/praxis-proxy/grid-operator:v0.1.3
  vcr:           ghcr.io/neuralmagic/vllm-vcr:vllm0.23
  pull policy:   IfNotPresent

[SETUP 4/9] Installing MetalLB, SWIM services, and Grid operators

[SETUP 5/9] Installing provider trust, credentials, and policy
namespace/grid-system created
secret/edge-gateway-tls created
secret/edge-gateway-tls created
secret/gtm-emulator-tls created
configmap/provider-gateway-config created
secret/provider-gateway-tls created
secret/vcr-inference-credential created
secret/vcr-inference-secondary-credential created
configmap/provider-gateway-config created
secret/provider-gateway-tls created
secret/vcr-inference-credential created
grid-routing: edge identities, GTM certificate, provider configs, mTLS material, and credentials installed

[SETUP 6/9] Deploying two provider gateways and three private inference providers

[SETUP 7/9] Configuring edge trust and deploying Praxis edge gateways
  [OK] east-edge: local edge site configured
  [OK] west-edge: local edge site configured

  east-edge trust view: authorizing providers discovered through Grid
  [OK] GridSite "glb-demo-east-provider" auto-created by operator (gridNetworkRef="glb-demo")
  [OK] GridSite "glb-demo-east-provider": advertised certificate matches the staged identity
  [OK] GridSite "glb-demo-east-provider": canonicalFingerprints + serverName="east-provider.grid.internal" patched
  [OK] GridSite "glb-demo-east-provider": phase="Active" (confirmed)
  [OK] GridSite "glb-demo-west-provider" auto-created by operator (gridNetworkRef="glb-demo")
  [OK] GridSite "glb-demo-west-provider": advertised certificate matches the staged identity
  [OK] GridSite "glb-demo-west-provider": canonicalFingerprints + serverName="west-provider.grid.internal" patched
  [OK] GridSite "glb-demo-west-provider": phase="Active" (confirmed)

  west-edge trust view: authorizing providers discovered through Grid
  [OK] GridSite "glb-demo-east-provider" auto-created by operator (gridNetworkRef="glb-demo")
  [OK] GridSite "glb-demo-east-provider": advertised certificate matches the staged identity
  [OK] GridSite "glb-demo-east-provider": canonicalFingerprints + serverName="east-provider.grid.internal" patched
  [OK] GridSite "glb-demo-east-provider": phase="Active" (confirmed)
  [OK] GridSite "glb-demo-west-provider" auto-created by operator (gridNetworkRef="glb-demo")
  [OK] GridSite "glb-demo-west-provider": advertised certificate matches the staged identity
  [OK] GridSite "glb-demo-west-provider": canonicalFingerprints + serverName="west-provider.grid.internal" patched
  [OK] GridSite "glb-demo-west-provider": phase="Active" (confirmed)
  [OK] east-edge: Praxis edge gateway deployed
  [OK] west-edge: Praxis edge gateway deployed

[SETUP 8/9] Deploying the GTM emulator in front of both edges

[SETUP 9/9] Waiting for both edge-local routing overlays to converge
            Each edge must receive one complete, versioned provider view.
            This proves distribution; Praxis acceptance is verified next.
  distributed → east-edge: revision=eb0212bde1647334…
  distributed → west-edge: revision=0c95ac5f3f9871dd…

[READY] Environment deployed from /home/fedora/praxxis/ai-grid/prs/44-grid-demo-image-source/demos/demos/grid-glb-demo/.forge.resolved.yaml
        east-edge=3 candidates, projected `ConfigMap`, west-edge=3 candidates, projected `ConfigMap`

===============================================================================
PRAXIS GRID GLOBAL INGRESS DEMO
===============================================================================

Mode: QUICK
Proof policy: Every PASS comes from a runtime assertion; manifest intent is not
              counted as proof.

EXPECTED PATH
  client -> stable Praxis HTTPS -> selected Praxis edge
         -> Grid-selected provider gateway -> private backend

Live edge/provider paths appear under OBSERVED ROUTES after Demo 1
runtime validation completes.

===============================================================================
DEMO 1 | ACTIVE/ACTIVE GLOBAL AND PROVIDER ROUTING
===============================================================================

USER STORY
  As an application owner, I need one stable HTTPS endpoint backed by active
  edges while Grid independently selects an admitted provider.

DETAILED RUNTIME PROOF
-------------------------------------------------------------------------------
  [CHECK] Prerequisites
  [CHECK] validating forge config
  [CHECK] checking environment status
  [CHECK] checking clusters live
  [CHECK] checking provider gateway IPs
  [CHECK] checking SWIM LB services
  [CHECK] checking operator SWIM advertise address
  [CHECK] checking GridNetwork seeds
  [CHECK] checking overlay candidate metadata
  rendered → revision=eb0212bde1647334…
  distributed → revision=eb0212bde1647334…, resourceVersion=5688
  [CHECK] checking multiple providers in one site overlay
  [CHECK] checking provider gateway self-discovery
  [CHECK] checking remote GridSite egress addresses
  [CHECK] checking provider service selector
  [CHECK] checking provider gateway pods
  [CHECK] checking backend private service
  [CHECK] checking backend network policy
  [CHECK] checking provider mTLS trust matrix
  [CHECK] checking provider peer policy
  [CHECK] checking provider request boundary
  [CHECK] checking edge config applied
  [CHECK] checking edge overlay projections
  distributed → east-edge: revision=eb0212bde1647334…
  distributed → west-edge: revision=0c95ac5f3f9871dd…
  [CHECK] capturing edge gateway pod identity
  accepted → east-edge: revision=eb0212bde1647334…
  accepted → west-edge: revision=0c95ac5f3f9871dd…
  [CHECK] sending inference request
  serving → revision=eb0212bde1647334…
  [CHECK] routing to both providers hosted by the east site

RUNTIME PROOF RESULTS
-------------------------------------------------------------------------------
Grid must update live edge routing from discovered provider state while authenticated provider gateways protect private backends.


| Step | Result | Evidence |
|---|---|---|
| prerequisites | **PASS** | config validation passed |
| forge status | **PASS** | forge status returned OK |
| clusters live | **PASS** | all 5 clusters live |
| provider gateway IPs | **PASS** | east-provider=172.18.255.232, west-provider=172.18.255.212 |
| swim lb services | **PASS** | east-edge=172.18.255.191, east-provider=172.18.255.231, west-edge=172.18.255.171, west-provider=172.18.255.211 |
| swim advertise addr | **PASS** | east-edge=172.18.255.191:7946, east-provider=172.18.255.231:7946, west-edge=172.18.255.171:7946, west-provider=172.18.255.211:7946 |
| gridnetwork seeds | **PASS** | seeds: east-edge=3, east-provider=3, west-edge=3, west-provider=3 |
| overlay metadata | **PASS** | dual-key: 3 candidate(s), validated: stable_id, admission_state, selection_tier, rank, generated_at; envelope: schema=1.0.0, revision=eb0212bde1647334…, annotations and Grid status verified |
| same-site provider candidates | **PASS** | east-edge: east-provider clusters=[vcr-east-provider,vcr-east-provider-secondary], stable_ids=[869189d6,46f71925]; west-edge: east-provider clusters=[vcr-east-provider,vcr-east-provider-secondary], stable_ids=[869189d6,46f71925] |
| provider gateway addr | **PASS** | east-provider=172.18.255.232:8443 (self-discovered, broadcast via SWIM), west-provider=172.18.255.212:8443 (self-discovered, broadcast via SWIM) |
| remote gridsite egress | **PASS** | east-provider=172.18.255.232:8443, west-provider=172.18.255.212:8443 |
| provider service selector | **PASS** | east-provider: port=8443, selector={"app.kubernetes.io/instance":"provider-gateway","app.kubernetes.io/name":"provider-gateway"}; west-provider: port=8443, selector={"app.kubernetes.io/instance":"provider-gateway","app.kubernetes.io/name":"provider-gateway"} |
| provider gateway pods | **PASS** | east-provider: Running, image=ghcr.io/praxis-proxy/grid-ai-rollup:v0.1.3; west-provider: Running, image=ghcr.io/praxis-proxy/grid-ai-rollup:v0.1.3 |
| backend private service | **PASS** | east-provider/vcr-inference: type=ClusterIP; east-provider/vcr-inference-secondary: type=ClusterIP; west-provider/vcr-inference: type=ClusterIP |
| backend network policy | **PASS** | east-provider/primary: allowed=connected, unlabeled=denied, no_auth=HTTP_401, client_auth=HTTP_403; east-provider/secondary: allowed=connected, unlabeled=denied, no_auth=HTTP_401, client_auth=HTTP_403; west-provider/primary: allowed=connected, unlabeled=denied, no_auth=HTTP_401, client_auth=HTTP_403 |
| provider mTLS trust matrix | **PASS** | east-provider: valid_edges=2, no_cert/wrong_sni/wrong_ca=TLS_REFUSED; west-provider: valid_edges=2, no_cert/wrong_sni/wrong_ca=TLS_REFUSED |
| provider peer policy | **PASS** | east-provider: wrong_org=HTTP_403, unknown_digest=HTTP_403; west-provider: wrong_org=HTTP_403, unknown_digest=HTTP_403 |
| provider request boundary | **PASS** | east-provider: unknown_candidate=HTTP_403, wrong_path=HTTP_404; west-provider: unknown_candidate=HTTP_403, wrong_path=HTTP_404 |
| edge config applied | **PASS** | GridNetwork 'glb-demo' applied on both edge clusters |
| edge overlays mounted | **PASS** | east-edge=3 candidates, projected `ConfigMap`, west-edge=3 candidates, projected `ConfigMap` |
| edge gateways running | **PASS** | east-edge=uid:8b32da16-8fa…, restarts:0, accepted_revision:eb0212bde1647334…, west-edge=uid:f0bb2f33-125…, restarts:0, accepted_revision:0c95ac5f3f9871dd… |
| inference routed | **PASS** | HTTP 200, model=Qwen/Qwen3-0.6B, provider=east-provider, gateway=east-provider, overlay_revision=eb0212bde1647334…, provider credential replaced client-supplied credential |
| same-site providers routed | **PASS** | east-provider gateway served two independent requests for Qwen/Qwen3-0.6B; distinct backend stable IDs were validated in the distributed overlay |

[PASS] All routing and provider-boundary proof points passed.

OBSERVED ROUTES
  [PASS] east-edge -> east-provider  (fixtures: narrated-edge-1 / narrated-provider-1)
  [PASS] west-edge -> west-provider  (fixtures: narrated-edge-0 / narrated-provider-0)
         client -> east-edge -> east-provider gateway -> east-provider backend
         client -> west-edge -> west-provider gateway -> west-provider backend

===============================================================================
DEMO 2 | SECURE PROVIDER BOUNDARY
===============================================================================

USER STORY
  As a provider and security owner, I need authenticated Grid traffic, exact
  local policy, private backend isolation, and final-hop credential
  replacement.

[PASS] Both provider gateways required mTLS, accepted both pinned edge
       identities, rejected missing or invalid TLS identities, and enforced
       exact candidate/model/path policy for three provider candidates.

[PASS] All three private provider paths are isolated by NetworkPolicy and
       provider-local credentials; the two east providers use distinct backends
       and credentials behind one site gateway.

[SKIP] Demos 3-5 run only in full mode.

===============================================================================
DEMONSTRATED BOUNDARY
===============================================================================

[PROVEN] Two Praxis edges served one verified HTTPS name, and Grid routed
         across three provider candidates.
[PROVEN] Versioned per-edge overlays with three candidates, including two
         independently routed providers in one cluster, plus exact
         rendered/distributed/accepted/serving revision evidence.
[PROVEN] Health-driven provider withdrawal, hot reload, provider mTLS, and peer
         authorization.
[PROVEN] Provider-local credential replacement and NetworkPolicy-enforced
         private backend access.
[OUT OF SCOPE] Managed DNS/Anycast, internet DDoS/WAF, geo-latency GTM
               steering, shared affinity storage, or in-flight stream
               migration.

[CLEANUP] Tearing down demo clusters...
[CLEANUP] Teardown complete.

===============================================================================
FINAL RESULT
===============================================================================
[PASS   ] Active/active routing
          2 edges observed; 3 provider candidates include 2 independently
          routed providers in one cluster
[PASS   ] Observable overlay contract
          one revision matched rendered/distributed/accepted/serving evidence
[PASS   ] Secure provider boundary
          mTLS, peer auth, NetworkPolicy, credential replacement verified
[SKIPPED] Session affinity and drain
          quick mode
[SKIPPED] Edge withdrawal and recovery
          quick mode
[SKIPPED] Grid restart recovery and soak
          quick mode
[SKIPPED] Live external provider
          not enabled; OpenAI Secret, InferenceProvider, provider route, and
          edge clusters absent

OVERALL   PASS
MODE      QUICK
ELAPSED   380.4s
EVIDENCE  /home/fedora/praxxis/ai-grid/prs/44-grid-demo-image-source/demos/demos/grid-glb-demo/.forge/evidence/glb-demo-20260812T230547Z
[EVIDENCE] Human narration and results.json written to /home/fedora/praxxis/ai-grid/prs/44-grid-demo-image-source/demos/demos/grid-glb-demo/.forge/evidence/glb-demo-20260812T230547Z