newton verifiable credential - identity data ref fetch from DB

gistfile1.txt

So the answer is: yes, operators DO have KMS access. The DataProviderConfig (loaded by the operator's DataProvider) reads
  DATA_PROVIDER_KMS_RSA_KEY_ID from the environment, creates an AWS KMS client, and uses it for decrypt_data_kms. This is the pre-HPKE
  path where operators decrypt identity data locally.

  But the critical problem remains: operators don't have API keys, so the gateway RPC approach I implemented won't work. The gateway
  authenticates every RPC call via API key.

  ★ Insight ─────────────────────────────────────
  This means the gateway-side resolution is the correct approach:

  1. Gateway already has DB access to identity_data_refs
  2. Gateway already has the task context (policy client, intent, signatures)
  3. Gateway can do the on-chain lookup (it has chain RPC)
  4. Gateway passes the encrypted blob to operators (like it does for privacy_data)
  5. Operators decrypt with KMS locally (they already have KMS access)

  The flow should be:
  Gateway: on-chain lookup → data_ref_id → DB fetch → encrypted_data
  Gateway → operator: pass encrypted_data via ConsensusCommitRequest or similar
  Operator: KMS decrypt → plaintext → Rego injection
  ─────────────────────────────────────────────────

  I need to revert the operator-calls-gateway-RPC approach and implement the gateway-side resolution instead. Want me to proceed with
  that refactor? The gateway RPC endpoint (newt_getIdentityEncrypted) can still be useful for external tooling, but the operator flow
  should go through the gateway's task broadcast path.

✻ Cooked for 44s

❯ also, note the newton vc underlying encryption and decryption will be migrated over from aws kms to Newton Privacy Layer (HPKE), so
considering this, evaluate the different approaches above to design our architecture

⏺ Great question. Let me evaluate the approaches with the HPKE migration in mind.

  Architecture Evaluation

  Current state: KMS RSA-OAEP encryption, operators decrypt locally
  Future state: HPKE encryption, gateway decrypts (centralized) or threshold decrypt via Prepare phase (Phase 2)

  The privacy flow already went through this exact transition. Here's how each approach maps to the HPKE future:

  Option A: Gateway resolves + passes encrypted blob to operators (operators decrypt)

  Now (KMS):   Gateway: DB fetch → encrypted blob → operator → KMS decrypt
  After HPKE:  Gateway: DB fetch → HPKE envelope → ???

  Problem: After HPKE migration, the gateway holds the HPKE private key (derived from ECDSA), not the operator. In threshold mode,
  decryption requires the Prepare phase partial decryptions. Either way, the operator can't decrypt HPKE. This approach dead-ends at
  HPKE migration — you'd have to move decryption to the gateway anyway.

  Option B: Gateway resolves + decrypts + passes plaintext (gateway decrypts)

  Now (KMS):   Gateway: DB fetch → KMS decrypt → plaintext → operator via ConsensusCommitRequest.identity_data
  After HPKE:  Gateway: DB fetch → HPKE decrypt → plaintext → operator via ConsensusCommitRequest.identity_data

  This is the privacy_data pattern exactly. When HPKE migration happens, you only swap decrypt_data_kms() → decrypt_envelopes() in the
  gateway. Operators never change. The ConsensusCommitRequest.identity_data field carries plaintext in both eras.

  ┌───────────────────────────┬───────────────────────────────────────────────────────┬────────────────────────────────────────┐
  │          Factor           │             Option A (operator decrypts)              │      Option B (gateway decrypts)       │
  ├───────────────────────────┼───────────────────────────────────────────────────────┼────────────────────────────────────────┤
  │ HPKE migration impact     │ Must redesign (move decrypt to gateway)               │ Swap one function call in gateway      │
  ├───────────────────────────┼───────────────────────────────────────────────────────┼────────────────────────────────────────┤
  │ Threshold DKG (Phase 2)   │ Doesn't work (operator can't threshold-decrypt alone) │ Works — same as privacy threshold path │
  ├───────────────────────────┼───────────────────────────────────────────────────────┼────────────────────────────────────────┤
  │ Operator complexity       │ Needs KMS credentials + gateway URL                   │ Just reads identity_data from request  │
  ├───────────────────────────┼───────────────────────────────────────────────────────┼────────────────────────────────────────┤
  │ Mirrors privacy pattern   │ No                                                    │ Yes                                    │
  ├───────────────────────────┼───────────────────────────────────────────────────────┼────────────────────────────────────────┤
  │ Gateway KMS access needed │ No                                                    │ Yes — needs kms_rsa_key_id             │
  └───────────────────────────┴───────────────────────────────────────────────────────┴────────────────────────────────────────┘

  Recommendation: Option B

  The only question is: does the gateway currently have KMS access? The privacy layer uses HPKE (not KMS), so the gateway may not have
  DATA_PROVIDER_KMS_RSA_KEY_ID configured today.

  If the gateway already has KMS access (or can easily get it): implement Option B now, and the HPKE migration for identity is a
  one-line swap.

  If not: we could still do Option B but the gateway needs KMS credentials added to its config. Alternatively, ship the 0x fix + Option
  A as a stopgap, then migrate to Option B as part of the HPKE migration (since that's when the gateway takes over decryption anyway).

  What's the gateway's current KMS situation?