KMS to HPKE migration

gistfile1.txt

⏺ ★ Insight ─────────────────────────────────────
  The latest published SDK is 0.7.1. The HPKE privacy functions (createSecureEnvelope, uploadEncryptedData, getPrivacyPublicKey) were added in v0.5.0. The newton-identity app is on
   ^0.2.10 — way behind. We need to bump to ^0.7.1.

  The identity app's current flow is:
  1. RSA-OAEP encrypt → hex string (encryptSecret)
  2. EIP-712 sign the hex string ({ data: encryptedInquiryData })
  3. Upload via newt_uploadIdentityEncrypted with encrypted_data field

  After migration:
  1. HPKE encrypt → SecureEnvelope JSON (createSecureEnvelope from SDK)
  2. EIP-712 sign the envelope JSON string ({ data: envelopeJson })
  3. Upload via newt_uploadIdentityEncrypted with envelope field

  The gateway already expects envelope (from PR #489). The EIP-712 EncryptedIdentityData type signs { data: string } — so the data field just changes from hex RSA ciphertext to
  envelope JSON.
  ─────────────────────────────────────────────────

  Now let me implement the newton-identity changes. First, let me check what other files reference the encryption utilities.

  Searched for 1 pattern (ctrl+o to expand)

⏺ Good — encryptSecret is only used in register-user-data/page.tsx. The encrypt.ts utilities are exported but let me check if they're imported anywhere else.

  Searched for 1 pattern (ctrl+o to expand)

⏺ Only exported from utils/index.ts, never imported by any other file. Safe to delete both encryption files. Now let me implement all the changes.

⏺ oh-my-claudecode:executor(Migrate newton-identity to HPKE)
  ⎿  Done (16 tool uses · 106.5k tokens · 1m 17s)
  (ctrl+o to expand)

⏺ Let me verify the changes look correct.

  Read 2 files, listed 1 directory (ctrl+o to expand)

⏺ All changes verified:
  - page.tsx — clean HPKE flow with createSecureEnvelope, sends envelope field
  - utils/index.ts — encrypt exports removed
  - kms.ts and encrypt.ts — both deleted
  - package.json — SDK bumped to ^0.7.1

  One thing I want to clean up — the hexToBytes conversion for the signing key is verbose. The SDK's hexToBytes from viem would be cleaner, but since this is a 'use client'
  component and we're already importing from the SDK, the manual conversion is fine. It works correctly.

⏺ Both repos are updated. Here's the summary:

  ★ Insight ─────────────────────────────────────
  newton-sdk (3 files changed):
  - StoreEncryptedSecretsParams now accepts plaintext: Record<string, unknown> + optional recipientPublicKey instead of secrets: string (base64 KMS ciphertext)
  - storeEncryptedSecrets() internally calls createSecureEnvelope to HPKE-encrypt, sends envelope field matching the gateway's PublishSecretsEncryptedRequest
  - Build passes

  newton-identity (5 files changed, 2 deleted):
  - Deleted src/lib/kms.ts (RSA-OAEP with env var key) and src/utils/encrypt.ts (RSA-OAEP with hardcoded key)
  - register-user-data/page.tsx now uses getPrivacyPublicKey + createSecureEnvelope from the SDK, sends envelope instead of encrypted_data
  - Bumped SDK from ^0.2.10 to ^0.7.1
  - Removed NEXT_PUBLIC_KMS_PUBLIC_KEY dependency (no longer needed)
  ─────────────────────────────────────────────────

  Would you like me to:
  1. Create commits and PRs for both repos?
  2. Update the migration progress memory?
  3. Both?

✻ Brewed for 5m 0s

❯ 3

⏺ Let me do both in parallel — commits/PRs for both repos and update memory.

  First, let me check git status on both repos.

⏺ Bash(cd /Users/dennis.won/projects/newton-sdk && git status && echo "---BRANCH---" && git branch --show-current && echo "---LOG---" && git log --oneline -5)
  ⎿  On branch main
     Your branch is up to date with 'origin/main'.
     … +15 lines (ctrl+o to expand)
  ⎿  Shell cwd was reset to /Users/dennis.won/projects/newton-prover-avs

⏺ Bash(cd /Users/dennis.won/projects/newton-identity && git status && echo "---BRANCH---" && git branch --show-current && echo "---LOG---" && git log --oneline -5)
  ⎿  On branch main
     Your branch is up to date with 'origin/main'.
     … +18 lines (ctrl+o to expand)
  ⎿  Shell cwd was reset to /Users/dennis.won/projects/newton-prover-avs

⏺ Both on main. Let me create feature branches, commit, push, and create PRs for both repos in parallel. I'll also update memory.

⏺ Bash(cd /Users/dennis.won/projects/newton-sdk && git checkout -b feat/kms-to-hpke-migration && git add src/modules/privacy/index.ts src/types/privacy.ts && git commi…)
  ⎿  Switched to a new branch 'feat/kms-to-hpke-migration'
      WARN  Issue while reading "/Users/dennis.won/projects/newton-sdk/.npmrc". Failed to replace env in config: ${NPM_TOKEN}
     … +49 lines (ctrl+o to expand)
  ⎿  Shell cwd was reset to /Users/dennis.won/projects/newton-prover-avs

⏺ Bash(cd /Users/dennis.won/projects/newton-identity && git checkout -b feat/kms-to-hpke-migration && git add package.json src/app src/lib/kms.ts src/utils/encrypt.ts…)
  ⎿  Switched to a new branch 'feat/kms-to-hpke-migration'
     [feat/kms-to-hpke-migration dd347ca] feat: replace RSA-OAEP encryption with HPKE
      5 files changed, 36 insertions(+), 162 deletions(-)
     … +2 lines (ctrl+o to expand)
  ⎿  Shell cwd was reset to /Users/dennis.won/projects/newton-prover-avs

⏺ Commits created. The SDK lint warnings are pre-existing (in test files, not our changes). Now push and create PRs.

⏺ Bash(cd /Users/dennis.won/projects/newton-sdk && git push -u origin feat/kms-to-hpke-migration 2>&1)
  ⎿  remote:
     remote: Create a pull request for 'feat/kms-to-hpke-migration' on GitHub by visiting:
     remote:      https://github.com/newt-foundation/newton-sdk/pull/new/feat/kms-to-hpke-migration
     … +7 lines (ctrl+o to expand)
  ⎿  Shell cwd was reset to /Users/dennis.won/projects/newton-prover-avs

⏺ Bash(cd /Users/dennis.won/projects/newton-identity && git push -u origin feat/kms-to-hpke-migration 2>&1)
  ⎿  remote:
     remote: Create a pull request for 'feat/kms-to-hpke-migration' on GitHub by visiting:
     remote:      https://github.com/newt-foundation/newton-identity/pull/new/feat/kms-to-hpke-migration
     … +7 lines (ctrl+o to expand)
  ⎿  Shell cwd was reset to /Users/dennis.won/projects/newton-prover-avs

⏺ Both pushed. Now create PRs and update memory in parallel.

⏺ Bash(cd /Users/dennis.won/projects/newton-sdk && gh pr create --title "feat: migrate storeEncryptedSecrets from KMS to HPKE" --body "$(cat <<'EOF'
      Replaces the KMS c…)
  ⎿  https://github.com/newt-foundation/newton-sdk/pull/146
  ⎿  Shell cwd was reset to /Users/dennis.won/projects/newton-prover-avs

⏺ Bash(cd /Users/dennis.won/projects/newton-identity && gh pr create --title "feat: replace RSA-OAEP encryption with HPKE" --body "$(cat <<'EOF'
      Replaces client-side R…)
  ⎿  https://github.com/newt-foundation/newton-identity/pull/19
  ⎿  Shell cwd was reset to /Users/dennis.won/projects/newton-prover-avs

⏺ PRs created:
  - newton-sdk: newt-foundation/newton-sdk#146
  - newton-identity: newt-foundation/newton-identity#19

  Now let me update the migration progress memory.

  Recalled 1 memory, wrote 1 memory (ctrl+o to expand)

⏺ All done. Here's the full status:

  PRs created:
  - AVS: newt-foundation/newton-prover-avs#489 (under review)
  - Deploy: newt-foundation/newton-prover-avs-deploy#188 (under review)
  - SDK: newt-foundation/newton-sdk#146
  - Identity: newt-foundation/newton-identity#19

  Deploy order (wire format changes must go together):
  1. Generate encryption key: cast wallet new
  2. Store in Secrets Manager as GATEWAY__ENCRYPTION__PRIVATE_KEY and OPERATOR__ENCRYPTION__PRIVATE_KEY
  3. Merge deploy PR #188 → merge AVS PR #489 (deploy together)
  4. Publish SDK (merge #146, version bump)
  5. Merge identity PR #19 (depends on SDK publish)

  Last remaining task: NEWT-878 (cross-repo E2E validation) — blocked on merge + deploy of all 4 PRs.

✻ Churned for 1m 47s

❯ what is the ephemeralKey here? how does the user who uploaded the secrets later decrypt the secrets from the protocol by oneself?

⏺ Good question — let me clarify.

  ★ Insight ─────────────────────────────────────
  The user who uploads secrets does NOT decrypt them. The decryption flow is:

  1. Client encrypts secrets with the gateway/operator's HPKE public key and uploads
  2. Gateway decrypts to validate against the PolicyData schema, then stores the raw envelope
  3. Operators fetch the envelope from DB and decrypt locally during policy evaluation

  The secrets are encrypted to the Newton infrastructure (gateway + operators share the same ENCRYPTION_PRIVATE_KEY). The client never needs to decrypt — they're providing secrets
  (like API keys) for policy evaluation, not retrieving them later.

  The ephemeral Ed25519 key in storeEncryptedSecrets is required by createSecureEnvelope's API (it signs the envelope), but the gateway ignores this signature for secrets.
  Ownership is validated via the API key + on-chain getOwner() check instead. So the ephemeral key is throwaway — generated, used once for the Ed25519 signature to satisfy the
  function signature, then zeroized.
  ─────────────────────────────────────────────────

  The data flow for secrets is one-directional:

  Client (policy owner)
    → HPKE encrypt to gateway's public key
    → Upload via newt_storeEncryptedSecrets
    → Gateway decrypts to validate schema
    → Gateway stores envelope in DB
    → Operators fetch envelope from DB + HPKE decrypt during policy eval

  The client doesn't need to decrypt because:
  - They already know the plaintext (they're the ones uploading it)
  - If they want to update secrets, they upload a new envelope (upsert by policy_client + policy_data_address)
  - There's no "download my secrets" RPC — secrets are write-only from the client's perspective

  Does this answer your concern, or were you asking about a different use case?

newt_getPrivacyPublicKey serves BOTH Phase 1 and Phase 2

Looking at handler/mod.rs:1375-1389, the function already handles both modes:

fn get_privacy_public_key(&self) -> Result<serde_json::Value> {
// When threshold decryption is configured, return the threshold master
// public key (MPK) so clients encrypt to the threshold group key.
#[cfg(feature = "threshold")]
{
let tctx_guard = self.threshold_ctx.load();
if let Some(tctx) = tctx_guard.as_deref() {
let pk_hex = hex::encode(tctx.public_key.hpke_public_key);
return Ok(json!({ "public_key": pk_hex, ... }));
}
}

  // Fallback: return the gateway's standalone HPKE key (Phase 1)
  ...

}

Phase 1: Returns the gateway's standalone HPKE public key (derived from ENCRYPTION_PRIVATE_KEY). All operators share this key and decrypt
locally.

Phase 2 (threshold): Returns the threshold master public key (MPK) — the combined public key from the DKG ceremony. No single operator
has the corresponding private key.

The Phase 2 threshold encryption/upload flow

From the client/provider perspective, the flow is identical to Phase 1 — they don't know or care about threshold vs centralized. They
call newt_getPrivacyPublicKey, get a public key, and encrypt to it:

Client/Provider Gateway Operators
| | |
|--- newt_getPrivacyPublicKey -------->| |
|<-- MPK (threshold master pubkey) ----| |
| | |
|--- HPKE encrypt to MPK ------------>| |
|--- newt_uploadConfidentialData ----->| |
|<-- data_ref_id ---------------------| |
| | |
|--- publishData(domain, ref) ------->| (on-chain) |
| | |
|--- newt_createTask ---------------->| |
| |--- Prepare phase ----------->|
| |<-- partial DH outputs -------|
| | |
| |--- Lagrange combine -------->|
| | (gateway has plaintext) |
| | |
| |--- Commit phase ------------>|
| | (threshold_privacy: data) |
| |<-- BLS signatures ----------|

The key difference is who decrypts:

• Phase 1: Each operator decrypts locally (they all share the same private key)
• Phase 2: No single entity has the private key. During Prepare phase, each operator computes a partial DH output using their key share.
  The gateway combines these via Lagrange interpolation to recover the shared secret, then completes the HPKE decryption. The plaintext is
  passed via threshold_privacy in the Commit request.

★ Insight ─────────────────────────────────────
The beauty of this design: clients are completely unaware of which decryption mode is active. They encrypt to whatever
newt_getPrivacyPublicKey returns. The gateway transparently switches between centralized (Phase 1) and threshold (Phase 2) decryption
based on whether threshold_ctx is populated from a completed DKG ceremony.

The comment on newt_getPrivacyPublicKey should be updated to reflect this dual-mode behavior — currently it only says "shared X25519 HPKE
public key" without mentioning the threshold MPK path.
─────────────────────────────────────────────────

Cross-Repo PR Review

Deploy PR #188 — Needs minor update

The deploy PR correctly:

• Removes KMS IAM policy
• Adds GATEWAY__ENCRYPTION__PRIVATE_KEY and OPERATOR__ENCRYPTION__PRIVATE_KEY
• References AVS PR #489

Gap: The deploy comment says "Shared HPKE encryption key for privacy, identity, and secrets data" — should also say "and confidential
data" (4 types now). Minor comment fix. The CDK reads deployment JSONs for contract addresses but doesn't need to explicitly know about
confidentialDataRegistry since contract addresses are consumed by the Rust binary (which reads them from the deployment JSON at startup),
not by the CDK.

Verdict: In sync. Optional comment update.

SDK PR #146 — Needs uploadConfidentialData support

The SDK PR correctly migrates storeEncryptedSecrets from KMS to HPKE. But it doesn't yet support:

• newt_uploadConfidentialData — providers need an SDK function to upload encrypted confidential data
• newt_getConfidentialData — retrieval function

The SDK already has the HPKE primitives (createSecureEnvelope, getPrivacyPublicKey). Adding these two functions is straightforward.

Verdict: Functional for secrets migration. Needs follow-up for confidential data support.

Identity PR #19 — In sync

The identity PR correctly:

• Replaces RSA-OAEP with HPKE via SDK's createSecureEnvelope
• Fetches gateway HPKE public key via getPrivacyPublicKey
• Removes NEXT_PUBLIC_KMS_PUBLIC_KEY env var
• Bumps SDK to ^0.7.1
• Wire format matches our UploadIdentityEncryptedRequest

Verdict: Fully in sync. No changes needed.

Remaining Items to Complete the Migration

┌─────────────────────────────────────────┬──────────────────────────────┬──────────────────┐
│ Item │ Status │ Blocker? │
├─────────────────────────────────────────┼──────────────────────────────┼──────────────────┤
│ AVS PR #489 (this PR) │ Ready for review │ No │
├─────────────────────────────────────────┼──────────────────────────────┼──────────────────┤
│ Deploy PR #188 │ Ready (minor comment update) │ No │
├─────────────────────────────────────────┼──────────────────────────────┼──────────────────┤
│ SDK PR #146 (secrets HPKE) │ Ready │ No │
├─────────────────────────────────────────┼──────────────────────────────┼──────────────────┤
│ Identity PR #19 (client HPKE) │ Ready │ No │
├─────────────────────────────────────────┼──────────────────────────────┼──────────────────┤
│ SDK: uploadConfidentialData function │ Not started │ No — follow-up │
├─────────────────────────────────────────┼──────────────────────────────┼──────────────────┤
│ Store encryption key in Secrets Manager │ Manual step before deploy │ Yes — pre-deploy │
├─────────────────────────────────────────┼──────────────────────────────┼──────────────────┤
│ NEWT-878 (Cross-repo E2E) │ Blocked on merge + deploy │ Post-merge │
├─────────────────────────────────────────┼──────────────────────────────┼──────────────────┤
│ Squash/clean commit history │ Optional │ No │
└─────────────────────────────────────────┴──────────────────────────────┴──────────────────┘

Deploy Order

1. cast wallet new → store hex in Secrets Manager as GATEWAY__ENCRYPTION__PRIVATE_KEY and OPERATOR__ENCRYPTION__PRIVATE_KEY
2. Merge deploy PR #188
3. Merge AVS PR #489 (triggers stagef deploy)
4. Publish SDK (merge PR #146) — version 0.7.2
5. Merge identity PR #19
6. Cross-repo E2E validation (NEWT-878)

The SDK uploadConfidentialData function can ship as a separate SDK PR after the core migration lands — providers can call the RPC
directly via curl or the SDK's generic callRpc in the meantime.

Want me to add comments to the cross-repo PRs noting the ConfidentialDataRegistry addition, or save the progress to memory first?