-
-
Save denniswon/4fc35b235596272c1692b0ea940de951 to your computer and use it in GitHub Desktop.
| --- | |
| Newton Privacy Layer — Full Recall | |
| ★ Insight ───────────────────────────────────── | |
| The Privacy Layer follows a progressive decentralization strategy: centralized gateway decryption (Phase 1) → threshold DKG with | |
| Feldman VSS/FROST (Phase 2) → MPC/ZK evaluation on encrypted data (Phase 3). The cryptographic stack bridges three incompatible | |
| curves: BN254 (BLS attestation), X25519 (HPKE encryption), and Ristretto255 (FROST DKG) — connected by a single to_montgomery() | |
| call at the combined DH output. | |
| ───────────────────────────────────────────────── | |
| Phase 1 — COMPLETE (shipped) | |
| - All AVS-side code merged (PR #368) | |
| - SDK released as v0.5.0 (2026-03-11), latest v0.5.4 | |
| - Covers: HPKE module (RFC 9180), Ed25519 key derivation, SecureEnvelope with AAD binding, encrypted_data_refs storage, | |
| dual-signature privacy auth, newt_getPrivacyPublicKey RPC, gateway centralized decryption, E2E tests, Prometheus metrics | |
| - Linear issues: NEWT-627, NEWT-182, NEWT-179, NEWT-437, NEWT-436, NEWT-180, NEWT-181 — all Done | |
| Phase 2 — MOSTLY COMPLETE (2A + 2B merged, 2C/2D not started) | |
| What's merged to main (verified just now): | |
| The entire PR stack has landed: | |
| - PR #426 → Feldman VSS, DLEQ proofs, threshold HPKE decrypt, consensus integration | |
| - PR #428 → DKG config loading (TOML + keystore priority), E2E test | |
| - PR #429 → FROST DKG ceremony wrapper (round1/round2/round3, Ristretto255→Edwards) | |
| - PR #443 → Gateway integration fixes (threshold MPK switch, single-phase rejection, fallback removal, Prometheus counter) | |
| - PR #476 → Batch transaction submission (BatchTaskManager) | |
| - PR #478 → FROST DKG integration (the most recent merge — DKG coordinator, encrypted keystore, ArcSwap hot-reload, operator RPC | |
| handlers, ceremony E2E) | |
| DKG module on main (crates/core/src/dkg/): | |
| ┌─────────────┬────────────────────────────────────────────────────────────────────────┐ | |
| │ File │ Purpose │ | |
| ├─────────────┼────────────────────────────────────────────────────────────────────────┤ | |
| │ types.rs │ ThresholdConfig, KeyShare, VssCommitment, PartialDecryption, DleqProof │ | |
| ├─────────────┼────────────────────────────────────────────────────────────────────────┤ | |
| │ dealer.rs │ Trusted dealer Feldman VSS share generation │ | |
| ├─────────────┼────────────────────────────────────────────────────────────────────────┤ | |
| │ dleq.rs │ Non-interactive DLEQ proofs (Fiat-Shamir) │ | |
| ├─────────────┼────────────────────────────────────────────────────────────────────────┤ | |
| │ combine.rs │ Lagrange interpolation, threshold_decrypt, compute_partial_decryption │ | |
| ├─────────────┼────────────────────────────────────────────────────────────────────────┤ | |
| │ config.rs │ TOML-friendly config with keystore-priority loading │ | |
| ├─────────────┼────────────────────────────────────────────────────────────────────────┤ | |
| │ frost.rs │ FROST DKG ceremony (RFC 9591): round1(), round2(), round3() │ | |
| ├─────────────┼────────────────────────────────────────────────────────────────────────┤ | |
| │ keystore.rs │ Encrypted keystore (scrypt + AES-256-GCM-SIV) │ | |
| └─────────────┴────────────────────────────────────────────────────────────────────────┘ | |
| Gateway DKG coordinator (crates/gateway/src/dkg/): | |
| ┌────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────┐ | |
| │ File │ Purpose │ | |
| ├────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤ | |
| │ coordinator.rs │ DkgCoordinator state machine (Idle→Round1→Round2→Round3→Complete/Failed), sequential polling │ | |
| ├────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────┤ | |
| │ RPC endpoints │ newt_dkg_initiate, newt_dkg_getStatus, newt_dkg_cancel │ | |
| └────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────┘ | |
| Key architecture decisions baked in: | |
| - ArcSwap for threshold_ctx (gateway) and threshold_key_share (operator) — wait-free reads on hot path, writes only after DKG | |
| ceremonies | |
| - Sequential polling per round (not parallel) — avoids race conditions where operators receive Round 2 packages before Round 1 | |
| completes | |
| - NATS decoupled — DKG uses gateway-as-relay over HTTP/JSON-RPC, NATS tracked separately | |
| - Single global MPK — all chains share one threshold key set (per-chain threshold is a future consideration) | |
| - Keystore persistence — operator key shares encrypted to disk (scrypt + AES-256-GCM-SIV) | |
| - FROST min_signers >= 2 — RFC 9591 constraint; E2E tests use 2-of-2 default | |
| Sub-phase Status (updated) | |
| ┌────────────────────────────────────────────────────────────────────┬─────────────┬──────────────────────────────┐ | |
| │ Sub-Phase │ Status │ PRs │ | |
| ├────────────────────────────────────────────────────────────────────┼─────────────┼──────────────────────────────┤ | |
| │ 2A — Feldman VSS, threshold decrypt, DLEQ, consensus integration │ MERGED │ #426, #428, #443 │ | |
| ├────────────────────────────────────────────────────────────────────┼─────────────┼──────────────────────────────┤ | |
| │ 2B — FROST DKG coordinator, encrypted keystore, ArcSwap hot-reload │ MERGED │ #429, #478 │ | |
| ├────────────────────────────────────────────────────────────────────┼─────────────┼──────────────────────────────┤ | |
| │ 2C — PSS epoch rotation, EpochRegistry contract │ Not started │ NEWT-628, NEWT-174 │ | |
| ├────────────────────────────────────────────────────────────────────┼─────────────┼──────────────────────────────┤ | |
| │ 2D — PrivacySlasher, on-chain DLEQ verification │ Not started │ NEWT-629, NEWT-640, NEWT-641 │ | |
| └────────────────────────────────────────────────────────────────────┴─────────────┴──────────────────────────────┘ | |
| Phase 2 Known Limitations | |
| 1. Single envelope only — threshold decrypt handles refs[0], multi-envelope needs wire type change | |
| 2. Gateway still sees plaintext — threshold removes single-key custody but gateway reconstructs after combining partials (Phase | |
| 3 MPC solves this) | |
| 3. On-chain DLEQ gas — EVM has no Curve25519 precompiles; pure-Solidity ~500k+ gas; may need BN254-based DLEQ alternative | |
| 4. No epoch rotation — keys are static until Phase 2C (PSS + EpochRegistry) | |
| Phases 3-4 — NOT STARTED (backlog) | |
| ┌───────────────────┬────────────────────────────────────────────────────────────┬─────────────────────────────────────────┐ | |
| │ Phase │ Key Items │ Linear Issues │ | |
| ├───────────────────┼────────────────────────────────────────────────────────────┼─────────────────────────────────────────┤ | |
| │ 3 — Advanced │ Benchmark HPKE vs IBE, conditional decryption, MPC │ NEWT-631, NEWT-632, │ | |
| │ Privacy │ evaluation │ NEWT-167/168/171/172/173 │ | |
| ├───────────────────┼────────────────────────────────────────────────────────────┼─────────────────────────────────────────┤ | |
| │ 4 — Ecosystem │ EVM precompiles for L2s, confidential intent │ NEWT-633, NEWT-634, NEWT-159-166/170 │ | |
| │ │ (MaybeEncrypted<T>), FHE │ │ | |
| └───────────────────┴────────────────────────────────────────────────────────────┴─────────────────────────────────────────┘ | |
| Related Work (also on main) | |
| - Private Storage Layer — redb + JMT architecture, Phase 1 (Postgres stopgap) merged via PR #459. Phase 2 (HPKE migration) | |
| blocked on Privacy Layer Phase 2 (now unblocked) | |
| - zkTLS / TLSNotary — PRs #3 (regorus) and #380 approved, uses same additional_data merge pattern as privacy | |
| - Batch Transaction Submission — PR #476 merged, BatchTaskManager for high-throughput on-chain submission | |
| Key References | |
| - Linear project: https://linear.app/magiclabs/project/newton-privacy-layer-e360cf0b6436 | |
| - Competitive analysis gist: https://gist.github.com/denniswon/32ba5dad6d44ee72de34710735a8d03e | |
| - Architecture doc: docs/THRESHOLD_DKG.md, docs/PRIVACY.md | |
| - E2E: make threshold-e2e (Feldman VSS), make threshold-e2e frost=true (FROST DKG) | |
| - Devnet: make gateway frost=true (gateway with FROST DKG + privacy + two-phase) | |
| --- | |
| ★ Insight ───────────────────────────────────── | |
| Where we are now: Phase 2A+2B are fully merged — the entire threshold decryption pipeline and FROST DKG ceremony coordination | |
| are on main. The next meaningful privacy work is Phase 2C (epoch rotation with PSS — Proactive Secret Sharing) which gives | |
| forward secrecy by rotating the master key periodically without re-encrypting existing data, and Phase 2D (on-chain slashing for | |
| privacy violations). The Private Storage Layer's Phase 2 (HPKE migration from KMS) is now unblocked since the threshold DKG | |
| stack landed. | |
| ───────────────────────────────────────────────── |
Good — memory is saved and indexed. Here's the consolidated requirements summary from our 4 batches:
Consolidated Requirements: KMS → HPKE Migration
★ Insight ─────────────────────────────────────
This migration touches the deepest trust boundary in Newton — who can decrypt user data. Moving from AWS KMS (centralized cloud
HSM) to operator-side HPKE decryption fundamentally shifts the trust model: instead of "trust AWS + the gateway," it becomes
"trust the operators who are economically staked via EigenLayer." This aligns the encryption trust model with the consensus
trust model for the first time.
─────────────────────────────────────────────────
Decisions Locked
┌─────┬───────────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────┐
│ # │ Decision │ Detail │
├─────┼───────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
│ 1 │ Clean-cut, no backward │ Drop existing KMS-encrypted data. Policy clients re-upload, identity owners re-register │
│ │ compat │ │
├─────┼───────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
│ │ New │ Separate key (not derived from task_generator_signer). Shared across gateway + all │
│ 2 │ ENCRYPTION_PRIVATE_KEY │ operators in Phase 1. Injected via Secrets Manager. TODO: replaced by per-operator │
│ │ │ FROST key shares in Phase 2 │
├─────┼───────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
│ 3 │ Operator-side decryption │ Each operator decrypts HPKE locally from DB. No plaintext traverses the network. │
│ │ │ Applies to ALL data types (privacy, identity, secrets) │
├─────┼───────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
│ 4 │ Table unification │ Merge identity_data_refs + wasm_secrets into encrypted_data_refs with data_type enum │
│ │ │ column (privacy, identity, secrets) │
├─────┼───────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
│ 5 │ Secrets validation via │ Gateway asks ONE operator to decrypt + validate against on-chain secrets_schema_cid. No │
│ │ operator │ quorum — first response suffices │
├─────┼───────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
│ 6 │ SDK: unified │ One method with data_type param for privacy, identity, and secrets. Replaces │
│ │ uploadEncryptedData │ storeEncryptedSecrets │
├─────┼───────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
│ 7 │ newton-identity: direct │ Calls newt_getPrivacyPublicKey directly, no SDK dependency. Delete kms.ts + encrypt.ts │
│ │ RPC │ │
├─────┼───────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
│ 8 │ Deployment order │ AVS → deploy repo → SDK → identity app │
├─────┼───────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────┤
│ 9 │ Single key for all data │ newt_getPrivacyPublicKey returns one X25519 public key used for privacy, identity, and │
│ │ types │ secrets │
└─────┴───────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────┘
What Changes Per Repo
newton-prover-avs (biggest change):
- DB migration: merge tables, add data_type column, drop old tables
- Remove all aws-sdk-kms / aws-config dependencies and KMS code
- Add ENCRYPTION_PRIVATE_KEY config field to gateway + operator
- Derive HPKE keypair from new encryption key (not task_generator_signer)
- Operator: HPKE decrypt from DB for all data types (privacy, identity, secrets)
- Remove ConsensusCommitRequest.privacy_data — operators decrypt locally
- New RPC: newt_validateSecrets (operator decrypts + schema validation)
- Update newt_storeEncryptedSecrets → accept SecureEnvelope, delegate validation to operator
- Update all docs, tests, Makefile targets
newton-prover-avs-deploy:
- Remove DATA_PROVIDER__KMS_RSA_KEY_ID from all services
- Remove KMS IAM policy from ecs_service.py
- Add ENCRYPTION_PRIVATE_KEY to gateway + operator Secrets Manager injection
newton-sdk:
- Unify into uploadEncryptedData(data_type, ...) using createSecureEnvelope internally
- Remove storeEncryptedSecrets
- Update docs, site pages, deployment checklist
newton-identity:
- Delete src/lib/kms.ts, src/utils/encrypt.ts
- Direct newt_getPrivacyPublicKey RPC → HPKE encrypt with hpke-js
- Remove NEXT_PUBLIC_KMS_PUBLIC_KEY env var
- Update register-user-data popup flow
Shall I proceed to write a detailed implementation plan with file-level changes, or do you want to adjust any of these decisions
first?
⏺ Excellent — now I have a complete picture of the KMS footprint across all four repos. Here are my first batch of questions:
★ Insight ─────────────────────────────────────
The KMS usage splits into two distinct encryption domains with different migration paths:
identity_data_refs, decrypted by operators via AWS KMS
RSA-OAEP, stored in wasm_secrets, decrypted by gateway inline
Both need to migrate to HPKE SecureEnvelope, but the data flow and who-decrypts-what differs significantly.
─────────────────────────────────────────────────
Batch 1: Scope & Data Migration
There are existing rows in identity_data_refs and wasm_secrets tables on stagef (and possibly prod) that contain KMS RSA-OAEP
ciphertext. Since you said "clean-cut, no backward compatibility," does that mean:
greenfield)
Currently, the gateway decrypts policy client secrets (KMS inline) before injecting them into WASM execution. In the HPKE
migration, the gateway would also decrypt (using its HPKE private key in centralized mode, or threshold in Phase 2). Is this the
intended model — gateway remains the decryptor for secrets, same as it already is for privacy data? Or should operators decrypt
secrets too (like they currently do for identity data via direct DB access)?
Right now operators have a stopgap: direct DB access (identity_database_url) to fetch KMS ciphertext, then KMS-decrypt locally.
The Privacy Layer Phase 1 model for encrypted_data_refs is: gateway decrypts HPKE → passes plaintext via
ConsensusCommitRequest.privacy_data. Should identity data follow this same gateway-mediated path (eliminating operator DB
access), or do you want to keep operator direct DB access but swap KMS for HPKE decryption?
✻ Cogitated for 3m 13s
❯ 1. (a) just drop and have them redeploy/register/upload for either policy client secret or newton vc identity data. 2. yes that
is the intended model, gateway descrypts using hpke in centralized mode, or each operator decrypts using the same hpke key in
two phase consensus mode. Then, with newton privacy layer phase 2 in the future, threshold decryption. 3. We need to keep
operator direct DB access but swap KMS for HPKE decryption, and we should NOT have the privacy data be transferred over network
between operators or gateway for security reasons. So this needs an architecture fix for the privacy layer phase 1 where privacy
data is included in ConsensusCommitRequest.privacy_data, isntead each operator should decrypt.
⏺ That's a significant architectural shift on point 3. Let me make sure I understand the implications before the next batch.
★ Insight ─────────────────────────────────────
Point 3 fundamentally changes the Phase 1 privacy data flow. Currently:
plaintext
The new model: each operator has HPKE decryption capability (either the full key in centralized mode, or their own key share in
threshold mode) and decrypts locally from DB. No plaintext ever traverses the network. This is strictly better for security but
means operators need: (a) DB access to encrypted blobs, and (b) the HPKE private key or key share.
─────────────────────────────────────────────────
Batch 2: Operator Decryption Architecture & Key Distribution
In the current Phase 1 design, only the gateway holds the HPKE private key (derived from task_generator_signer ECDSA key). If
each operator now needs to decrypt HPKE envelopes locally, they need the same private key material. Three options I see:
operators) — simple but means every operator holds the master decryption key
complexity
single-key custody but changes the client encryption model
Which model do you have in mind for centralized mode? Or is the plan to skip straight to threshold (Phase 2 is already merged)
and only support two-phase consensus for privacy tasks?
Both tables will store HPKE SecureEnvelope blobs after migration. The newton-sdk docs/identity/hpke-migration.md already notes
this possibility. Should we:
column
keccak256(envelope))
Currently the gateway KMS-decrypts the uploaded secrets blob to validate it matches the PolicyData JSON schema (from the
secrets_schema_cid). If the gateway can no longer decrypt (because we're moving to operator-side decryption for identity data),
should the gateway still decrypt-and-validate for policy client secrets? Or should validation move to the operator at evaluation
time? This affects whether policy clients get immediate feedback on malformed secrets.
❯ 1. (a) for now, but we need to leave a clear TODO note saying that with phase 2 migration, that shared task private key would
be removed. Also, instead of re-using task generator signer private key to derive hpke key, let's have a separate private key to
use as a encryption private key for gateway/operators (key shared in phase 1 whether centralized vs two phase consensus. Unique
per operator in threshold phase 2) 2. identity_data_refs and wasm_secrets should be merged into encrypted_data_refs. Is having a
data_type column necessary? Explain why. 3. let's have gateway to ask operators for validation of policy data secrets schema
before storing the encrypted data. This kind of flow will be needed for phase 2 anyways in the future for threshold decryption.
But we don't need quorum check or anything as this is not a task on the protocol. Just as soon as one of the operators give them
response, gateway can proceed.
⏺ Good answers. Let me address your question on #2 and then ask the final batch.
★ Insight ─────────────────────────────────────
On the data_type column: Yes, it's needed. encrypted_data_refs will store three semantically different blob types — privacy data
(intent fields), identity data (KYC/VC), and policy client secrets (API keys/credentials). Without a discriminator:
(JSON schema for secrets vs. identity domain dispatch vs. raw privacy bytes)
has different authorization (linked identity owners). A data_type lets the gateway enforce different access policies per type
owner unlinks), privacy data is ephemeral (per-task). Different retention policies need the discriminator
So yes — a data_type enum column (privacy, identity, secrets) is necessary.
─────────────────────────────────────────────────
Batch 3: SDK, Identity App & Deployment Boundaries
Currently storeEncryptedSecrets accepts a pre-encrypted KMS blob. In the new model, the SDK would use createSecureEnvelope
(already implemented) to HPKE-encrypt the secrets client-side, then upload. Should we:
uploadSecureEnvelope) — keeping the developer-facing API name but changing internals
data, and secrets
Currently src/lib/kms.ts reads NEXT_PUBLIC_KMS_PUBLIC_KEY from env. In the HPKE model, the popup needs the gateway's X25519
public key. The SDK already has getPrivacyPublicKey() which calls newt_getPrivacyPublicKey RPC. Should the newton-identity
popup:
Since this is a clean-cut with no backward compatibility, all four repos need to ship together. But they have deployment
dependencies. My assumption is:
Is this the right order, or do you want all PRs staged and merged simultaneously?
❯ 1. Unify all encrypted uploads into uploadEncryptedData with a data_type parameter — one method for privacy data, identity
data, and secrets 2. Make a direct newt_getPrivacyPublicKey RPC call without the SDK (lighter, no new dependency) 3. that is
the right order.
⏺ All clear. Let me ask one final batch to nail down the remaining technical details before I write the plan.
Batch 4: Key Management & Operator Config Details
You said a separate private key (not derived from task_generator_signer). For Phase 1 (shared key), operators need this key to
decrypt HPKE envelopes locally. How should operators receive it?
repo, same as BLS/ECDSA keys
operators derive different keys (breaks the "shared key" model)
I'm assuming (a) — a single ENCRYPTION_PRIVATE_KEY env var shared across all operators and the gateway, stored in Secrets
Manager. Confirm?
Currently this RPC returns a single X25519 public key. With the unified encrypted_data_refs table, all three data types
(privacy, identity, secrets) encrypt to the same key. Is that correct — one key for all data types? Or do you foresee different
keys for different data types (e.g., secrets encrypted to gateway-only key, identity encrypted to a key operators also hold)?
You said the gateway asks one operator to validate the secrets schema before storing. The operator needs to:
Does the operator need to fetch the schema CID from the policy contract on-chain, or does the gateway pass both the envelope and
the expected schema in the validation request?
❯ 1. New config field (e.g., encryption_private_key in operator config / env var) — injected via Secrets Manager in deploy repo,
same as BLS/ECDSA keys. a single ENCRYPTION_PRIVATE_KEY env var shared across all operators and the gateway, stored in Secrets
Manager. 2. yes that is correct. 3. just operator fetch the schema cid from onchain
⏺ All questions answered. Let me save this decision context to memory, then present the consolidated requirements summary.