Skip to content

Instantly share code, notes, and snippets.

@edeediong

edeediong/standup-summary-2026-04-17.md

Created April 17, 2026 00:38
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save edeediong/7d267b50c8b0b6b34d850ea578c0f1bb to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save edeediong/7d267b50c8b0b6b34d850ea578c0f1bb to your computer and use it in GitHub Desktop.
Download ZIP
Standup Summary — April 17, 2026 — Canton Validator Infrastructure
Raw
standup-summary-2026-04-17.md

Standup Summary — April 17, 2026

1. Istio VirtualService Routing Fix (PR #301)

Rewrote helm/eu-central-9/base/istio/templates/canton-virtualservice.yaml to match the proven eu-central-5 Traefik routing pattern:

  • Added CNS host routing (cns.<name>.<cantonDomain>)
  • Added /api/json-api/ route with URI rewrite to participant JSON-API (:7575)
  • Added /api/validator/ and /api/validator (exact) routes to validator-app (:5003)
  • Added default catch-all routes for wallet-web-ui and ans-web-ui
  • Updated values.yaml with services.jsonApi.host: canton-validator-1-devnet-participant override since the participant service name doesn't match the client key
  • PR was merged but ArgoCD didn't pick it up — fixed with kubectl annotate application dev-istio argocd.argoproj.io/refresh=hard --overwrite

2. Auth0 Authority URL Fix — Double .well-known Path (PR #302)

Root cause: The Auth0 Terraform module stored the URL as https://stakeway-dev.eu.auth0.com/.well-known/openid-configuration in AWS Secrets Manager. Splice wallet's oidc-client-ts library appends /.well-known/openid-configuration to the authority value automatically — resulting in a double path that returned 404, causing "TypeError: Failed to fetch."

Fix: Changed terraform/modules/auth0/canton-validator/main.tf to store just "https://${var.auth0_domain}" (without the .well-known suffix) in the aws_secretsmanager_secret_version.validator_ui_auth0 resource.

Post-merge steps:

  • Triggered Auth0 GitHub Actions workflow (run ID: 24536900184)
  • Force-synced the ExternalSecret: kubectl annotate externalsecret splice-app-wallet-ui-auth force-sync=$(date +%s) --overwrite
  • Restarted wallet-web-ui deployment to pick up the new secret
  • Verified config.js now shows authority: "https://stakeway-dev.eu.auth0.com" (correct)

3. Auth0 Callback URL Mismatch Fix (PR #303)

Root cause: After fixing the authority URL, Auth0 rejected the login redirect with "Callback URL mismatch." Investigated by reading the minified Splice wallet-web-ui bundle (3.6 MB) and found the oidcAuthToProviderProps function sets redirect_uri: window.location.origin (no /callback path). Auth0 requires exact match on redirect URIs.

Fix (two files):

  • terraform/modules/auth0/canton-validator/main.tf — Added origin URLs (without /callback) to the callbacks list alongside the existing /callback variants
  • terragrunt/auth0/canton-devnet/terragrunt.hcl — Added CNS canton domain URLs and origin URLs to all three lists: allowed_callbacks, allowed_origins, allowed_logout

Result: Wallet OAuth2 login at https://wallet.stakeway.canton.dev.eu-central-9.gateway.fm works end-to-end.

4. Full Ingress Audit

After wallet login was working, audited all Canton ingress routes:

  • wallet.stakeway.canton... — 200 OK (working)
  • cns.stakeway.canton... — 200 OK (serving ans-web-ui)
  • /api/validator/v0/wallet/user-status via both hosts — 401 (correct, auth required)
  • /api/json-api/ via wallet host — routed to participant on :7575
  • validator.stakeway.canton... — has DNS but no VirtualService route (orphaned DNS record, optional cleanup)

5. CNS UI "TypeError: Failed to fetch" — Pod Restart

Symptom: User tested CNS UI and got "TypeError: Failed to fetch" (same error wallet had earlier).

Investigation: Checked the splice-app-cns-ui-auth K8s secret — the URL value was already correct (https://stakeway-dev.eu.auth0.com). But curl-ing the CNS config.js showed the old .well-known authority URL still being served.

Root cause: The ans-web-ui pod was 46 hours old — it predated the secret fix. Even though the K8s secret was updated, the pod had the old value cached in its environment variables. The reloader.stakater.com/auto: "true" annotation didn't trigger because the ExternalSecret sync timing didn't change the secret checksum while the pod was watching.

Fix: kubectl rollout restart deployment/ans-web-ui -n canton-val-validator-1-devnet

Verified: config.js now shows authority: "https://stakeway-dev.eu.auth0.com" (correct).

6. CNS UI Infinite Loading — Deep Investigation

Symptom: After auth fix, CNS UI gets past Auth0 login but shows infinite loading spinner.

Investigation:

  • Checked validator-app logs — requests from the CNS browser IP (10.51.145.48) initially returned 200 OK, then started returning 401 (token lost/expired)
  • Found the CNS UI calls scan-proxy/* endpoints (e.g., scan-proxy/ans-entries, scan-proxy/dso, scan-proxy/open-and-issuing-mining-rounds)
  • Initially thought scan-proxy was returning 404 (broken), but deeper investigation showed:
    • scan-proxy/automation-service/status → 404 (that specific path doesn't exist — red herring)
    • scan-proxy/dso → 401 (scan-proxy works, just needs auth)
    • scan-proxy/ans-entries → 401 (works, needs auth)
  • Extracted the CNS UI JS bundle to confirm all API paths it calls — all are scan-proxy/* variants
  • Checked the splice-validator Helm chart (splice-validator-0.5.18.tgz) — no separate scan-proxy feature flag; it's built into the validator-app when scan client is configured (which it is, BFT mode with seed URLs)

Conclusion: Not an infrastructure/routing issue. The CNS browser is losing its auth token after initial login — likely stale OIDC session state from the pre-fix era cached in the browser. Advised user to clear browser local storage/cookies for the CNS domain and re-login.

Status: Awaiting user re-test after clearing browser state.

7. Client-Facing Canton Validator Onboarding Guide

Created ~/gatewayfm/client-onboarding-guide.md — a polished, client-facing document for the Sales team covering the full lifecycle:

  • Phase 1: DevNet (Sandbox) — zero client effort, Gateway provisions everything
  • Phase 2: TestNet (Staging) — client AWS KMS + IAM setup required
    • Option A: Terraform Module — references the existing canton-client-kms module at github.com/gateway-fm/stakeway-terraform//terraform/modules/aws/canton-client-kms
    • Option B: AWS Console — full 5-step manual walkthrough (Create KMS key → Create IAM policy → Create IAM role with trust policy → Tighten KMS key policy → Share details)
  • Phase 3: MainNet (Production) — marked as "Coming Soon"
  • FAQ — token expiry, KMS region support, key rotation, revocation, multi-validator
  • Endpoints reference table per environment

Gist: https://gist.github.com/edeediong/eafa8499f89706355d17eb1621963656

8. MainNet Infrastructure Requirements & Cost Analysis

Audited the entire infrastructure stack across both repos (stakeway-terraform and stakeway-deployments) to build an honest cost estimate:

What MainNet needs beyond current infra:

  • Dedicated NAT Gateway + Elastic IP (Canton requires unique egress IPs per network)
  • Auth0 production tenant (not yet created)
  • Route53 hosted zone for MainNet domain
  • TLS trust-manager with strict CA validation
  • Cross-region backup replication (currently disabled on dev)
  • Formal SV sponsor agreement

Cost summary:

  • Shared infrastructure: ~$207/month (EKS control plane, system nodes, NAT, NLB, Route53)
  • Per validator: ~$91-104/month (compute, storage, backups, secrets, KMS)
  • 1 MainNet validator: ~$300-310/month
  • 3 validators (DevNet + TestNet + MainNet): ~$480-520/month
  • 5 validators (multi-client): ~$680-730/month
  • Current dev cluster spend (2 validators): ~$400-450/month

Gist: https://gist.github.com/edeediong/6455c1fa7e9b3dbd1702dc98e0ba93a5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment