-
-
Save denniswon/06d1c34ba5ca47a4fcfefc4de1ef49bf to your computer and use it in GitHub Desktop.
| --- | |
| Meeting: TEE Integration for Newton Protocol | |
| --- | |
| Problem Statement | |
| Newton Protocol accumulates sensitive data (identity PII, confidential data, policy client secrets) that operators must decrypt and | |
| process during policy evaluation. Two core concerns: | |
| 1. Operator data leakage — operators currently decrypt private data locally, meaning a malicious or compromised operator could exfiltrate | |
| PII | |
| 2. Regulatory compliance (GDPR) — the safest posture for regulators is that private data is never exposed in plaintext outside a | |
| hardware-attested environment | |
| ★ Insight ───────────────────────────────────── | |
| This maps directly to Newton's existing three-path privacy model: identity data (data.identity.*), confidential data | |
| (data.confidential.*), and inline ephemeral (data.privacy.inline[0].*). Today, operators decrypt all three paths locally — either with | |
| their own HPKE key (centralized mode) or via DKG key shares (threshold mode). TEE would move that decryption boundary inside a hardware | |
| enclave, so operators never see plaintext. | |
| ───────────────────────────────────────────────── | |
| --- | |
| Approaches Evaluated | |
| ┌─────────────┬─────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┐ | |
| │ Approach │ Verdict │ Reasoning │ | |
| ├─────────────┼─────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ | |
| │ TEE (AWS │ Chosen │ Practical, well-documented, prior team experience with Intel TDX, meets GDPR requirements, │ | |
| │ Nitro) │ │ hardware attestation valued by institutional partners │ | |
| ├─────────────┼─────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ | |
| │ │ Deferred │ Jacob called it "Pandora's box" — higher risk, longer timeline, no guaranteed results in 2-4 │ | |
| │ MPC │ (research │ weeks. Dennis noted MPC isn't scalable as TEE. Existing DKG/PSS work mitigates some │ | |
| │ │ phase) │ operator-leaving risks already │ | |
| ├─────────────┼─────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ | |
| │ FHE │ Rejected │ Too expensive, circuit arithmetic limited to basic operations ("very old days of ZK"), not │ | |
| │ │ │ practical for Rego evaluation │ | |
| ├─────────────┼─────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ | |
| │ ZK proofs │ Doesn't solve │ ZK proves computation correctness but doesn't prevent data decryption — you still need to decrypt │ | |
| │ │ the problem │ before running the circuit │ | |
| └─────────────┴─────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘ | |
| --- | |
| Agreed Architecture | |
| What goes inside the TEE enclave: | |
| - Rego policy evaluation only (Phase 1) — where private data is actually loaded and used | |
| - WASM execution is a secondary step (Phase 2), lower priority since WASM only handles policy client secrets, not PII | |
| What stays outside the enclave: | |
| - Application code (operator binary) | |
| - WASM data provider execution (less sensitive) | |
| - All non-privacy policy evaluations | |
| Communication: Enclave ↔ Application via VSOCK (attached secure encrypted channel). No external network connections needed from the | |
| enclave — it only talks to the local operator application. | |
| Fallback: If TEE communication fails, fall back to raw (non-enclave) policy evaluation. Only privacy-involved evaluations run inside the | |
| TEE. | |
| ┌─────────────────────────────────────────────┐ | |
| │ Operator Process (outside enclave) │ | |
| │ ┌────────────┐ ┌──────────────────────┐ │ | |
| │ │ WASM Exec │ │ Operator Service │ │ | |
| │ │ (data │ │ (RPC, BLS signing, │ │ | |
| │ │ provider) │ │ chain interaction) │ │ | |
| │ └────────────┘ └──────┬───────────────┘ │ | |
| │ │ VSOCK │ | |
| │ ┌──────────────────────▼───────────────┐ │ | |
| │ │ AWS Nitro Enclave │ │ | |
| │ │ ┌──────────────────────────────┐ │ │ | |
| │ │ │ Rego Policy Evaluation │ │ │ | |
| │ │ │ + Private data decryption │ │ │ | |
| │ │ │ + HPKE decrypt → eval → │ │ │ | |
| │ │ │ zeroize → return result │ │ │ | |
| │ │ └──────────────────────────────┘ │ │ | |
| │ │ Remote Attestation (built-in) │ │ | |
| │ └──────────────────────────────────────┘ │ | |
| └─────────────────────────────────────────────┘ | |
| ★ Insight ───────────────────────────────────── | |
| 1. Memory is not a concern — Jacob noted TEE memory limits of 256-512 MB per CPU. Dennis confirmed Rego evaluation is lightweight and | |
| fits comfortably. This is a much smaller footprint than running the full WASM data provider. | |
| 2. The zeroization pattern already exists — Newton already zeroizes decrypted private data outside WASM (lessons.md documents this). TEE | |
| takes this further by ensuring the plaintext never exists outside the enclave at all. | |
| 3. Secret Network is the closest analog — Wesley identified Secret Network's model as nearly identical: encrypted transactions enter the | |
| TEE, are decrypted inside a Cosmos SDK runtime, executed, and only the output exits. Newton's model substitutes Rego evaluation for | |
| Cosmos SDK execution. | |
| ───────────────────────────────────────────────── | |
| --- | |
| Key Design Decisions | |
| 1. AWS Nitro as preferred platform — best performance, mature remote attestation, existing AWS infra alignment. Intel TDX / AMD SEV as | |
| future portability options (not high priority now). | |
| 2. Every operator runs their own enclave — not a single centralized enclave. Rationale: preserves the operator model where each operator | |
| independently evaluates. Wesley proposed a single-enclave model but Dennis noted it "kills the purpose of having multiple operators." | |
| 3. Institutional operators, not general validators — Newton targets financial institutions and strategic partners, not Block Daemon / | |
| Luganodes-style validator shops. These operators are motivated by "I don't want to touch PII data" — TEE gives them hardware-level | |
| guarantees. | |
| 4. Tiered operator model (Alec's suggestion, Dennis acknowledged) — institutional operators run enclaves for privacy-sensitive work; | |
| other operators could handle non-sensitive tasks. Operator running is permissioned, staking is permissionless. | |
| 5. No vendor lock-in panic — cloud-native Nitro enclaves (VM-based, not bare metal). Dennis noted no additional pricing overhead beyond | |
| Nitro-enabled EC2 instances. | |
| --- | |
| Concerns Raised | |
| ┌────────────────────────────────────────┬─────────────┬─────────────────────────────────────────────────────────────────────────────┐ | |
| │ Concern │ Raised By │ Resolution │ | |
| ├────────────────────────────────────────┼─────────────┼─────────────────────────────────────────────────────────────────────────────┤ | |
| │ SGX/TDX security gaps (physical access │ Jacob │ Dennis: these are research-level, not practical threats. Nitro has stronger │ | |
| │ attacks) │ │ attestation than SGX │ | |
| ├────────────────────────────────────────┼─────────────┼─────────────────────────────────────────────────────────────────────────────┤ | |
| │ Vendor lock-in to AWS │ Jacob, │ Accepted as trade-off for speed. Future TDX/AMD support possible │ | |
| │ │ Wesley │ │ | |
| ├────────────────────────────────────────┼─────────────┼─────────────────────────────────────────────────────────────────────────────┤ | |
| │ Operator onboarding complexity │ Wesley │ Mitigated by targeting institutional operators who can handle it │ | |
| ├────────────────────────────────────────┼─────────────┼─────────────────────────────────────────────────────────────────────────────┤ | |
| │ Centralization risk │ Wesley │ Each operator runs own enclave. Duplication + fallback for availability │ | |
| ├────────────────────────────────────────┼─────────────┼─────────────────────────────────────────────────────────────────────────────┤ | |
| │ Operator key compromise on departure │ Jacob │ Already mitigated by DKG epoch rotation + upcoming PSS (NEWT-628). TEE adds │ | |
| │ │ │ defense-in-depth │ | |
| ├────────────────────────────────────────┼─────────────┼─────────────────────────────────────────────────────────────────────────────┤ | |
| │ MPC as alternative │ Jacob │ Dennis: not ruled out long-term, but can't be sole bet without a Plan B │ | |
| └────────────────────────────────────────┴─────────────┴─────────────────────────────────────────────────────────────────────────────┘ | |
| --- | |
| Action Items | |
| 1. Wesley Coakley — Lead the TEE integration initiative. Draft a detailed technical document covering: | |
| - Enclave architecture | |
| - Communication protocols (VSOCK) | |
| - How Rego policy evaluation runs inside the enclave | |
| - Staged WASM integration plan | |
| 2. Team — Include TEE-based policy evaluation in upcoming sprint. Start technical exploration immediately (today/tomorrow). | |
| 3. Dennis + team — Monitor MPC as a future alternative while prioritizing TEE now. | |
| --- | |
| Relationship to Existing Newton Architecture | |
| This maps onto the current codebase as follows: | |
| ┌───────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────────┐ | |
| │ Current Component │ TEE Impact │ | |
| ├───────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────┤ | |
| │ crates/core/src/crypto/envelope.rs (HPKE decrypt) │ Decryption moves inside enclave │ | |
| ├───────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────┤ | |
| │ crates/operator/src/core.rs (Rego evaluation with privacy │ Evaluation moves inside enclave │ | |
| │ extensions) │ │ | |
| ├───────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────┤ | |
| │ crates/core/src/dkg/ (threshold key management) │ DKG key shares would be loaded inside enclave only │ | |
| ├───────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────┤ | |
| │ crates/data-provider/ (WASM execution) │ Stays outside enclave (Phase 1). Policy client secrets injection may │ | |
| │ │ move inside later (Phase 2) │ | |
| ├───────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────┤ | |
| │ Privacy data zeroization (already in place) │ Becomes hardware-enforced rather than software-enforced │ | |
| └───────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────────┘ | |
| ★ Insight ───────────────────────────────────── | |
| 1. TEE is complementary to the existing DKG/PSS work, not a replacement. DKG handles key rotation so departed operators can't decrypt | |
| historical data. TEE handles runtime protection so active operators can't exfiltrate plaintext during evaluation. The Phase 2C PSS work | |
| (PRs #523/#525/#526) remains relevant. | |
| 2. The "only privacy evaluations in TEE" decision is architecturally clean — it means the enclave binary is small (just Rego engine + | |
| HPKE decrypt), the VSOCK interface is narrow (encrypted input in, boolean/evaluation result out), and non-privacy policies see zero | |
| performance impact. | |
| 3. Fallback to non-TEE evaluation is a pragmatic choice that matches the existing pattern where the gateway can degrade gracefully (e.g., | |
| circuit breaker patterns in crates/gateway/src/task/circuit_breaker.rs). | |
| ───────────────────────────────────────────────── |
⏺ Question 8: Does WASM-in-enclave apply to ALL tasks or only privacy tasks?
Today, WASM execution runs for every task that has a data provider plugin — privacy or not. The question is whether Phase 2 moves ALL WASM
execution into the enclave, or only WASM execution for tasks that have secrets.
- (A) Only tasks with secrets: If the policy client has stored encrypted secrets via newt_storeEncryptedSecrets, WASM runs inside the
enclave. Otherwise WASM stays on the host. This minimizes enclave load — most tasks don't have secrets. - (B) All tasks with WASM: Every WASM data provider runs inside the enclave regardless of whether secrets are configured. Simpler routing
logic (no conditional), and it means all policyTaskData is enclave-attested. But it adds Egress Service overhead to every task. - (C) All privacy tasks: WASM runs in enclave if the task involves any privacy data (identity, confidential, ephemeral). Non-privacy tasks
with WASM but no secrets stay on host.
My recommendation: Option A (only tasks with secrets). The motivation for Phase 2 is protecting secrets (API keys, endpoint configs).
Non-secret WASM execution (e.g., a price feed plugin with no API key) doesn't benefit from enclave isolation. Option A keeps the hot path
fast — the majority of tasks with public data providers run on the host with zero Egress overhead. The operator builder already knows
whether secrets are configured (it fetches them from DB during Prepare).
The routing decision: if secrets_envelope.is_some() { enclave_wasm } else { host_wasm }.
What's your view?
Non-secret WASM execution (e.g., a price feed plugin with no API key) doesn't benefit from enclave isolation. Option A keeps the hot path
fast — the majority of tasks with public data providers run on the host with zero Egress overhead. The operator builder already knows
whether secrets are configured (it fetches them from DB during Prepare).
The routing decision: if secrets_envelope.is_some() { enclave_wasm } else { host_wasm }.
What's your view?
✻ Churned for 35s
❯ A
⏺ Question 9: How does the Egress Service on the host handle concurrent HTTP requests?
The WASM fetch() host function can make multiple sequential HTTP calls during a single WASM execution (e.g., fetch price from Pyth, then
fetch from Chainlink as fallback). The Egress Service on VSOCK_PORT_EGRESS needs to handle these.
- (A) Per-connection model (same as compute port): One VSOCK connection per HTTP request. The enclave connects to port 5006, sends an
HttpRequest, gets an HttpResponse, closes. Simple, matches existing pattern. Downside: connection setup overhead per HTTP call (~50-200μs). - (B) Multiplexed connection: The enclave opens one VSOCK connection to port 5006 at WASM execution start, sends multiple
HttpRequest/HttpResponse pairs over it, closes when WASM execution completes. Lower overhead for multiple sequential requests. - (C) Shared long-lived connection pool: Persistent VSOCK connections between enclave and Egress Service, shared across WASM executions.
Most efficient but adds connection management complexity.
My recommendation: Option A. WASM plugins typically make 1-2 HTTP calls per execution. The 50-200μs overhead per call is negligible vs the
200-1000ms external API latency. The per-connection model provides clean failure isolation (one stuck request can't block another) and
matches the compute port pattern. It also means the Egress Service is stateless — no session tracking, no connection lifecycle management.
If profiling later shows connection overhead is a bottleneck (unlikely), upgrade to B.
What's your view?
✻ Cogitated for 39s
※ recap: We're designing Phase 2 TEE (WASM execution inside enclave). We've resolved 9 design questions through a grill session — WASM +
secrets move into enclave, Egress Service proxies HTTP via framed VSOCK. Next: answer the remaining design questions and document decisions.
❯ for performance and reliability, which option is the optimal one?
⏺ Option B is optimal for both performance and reliability. Here's why:
Performance:
- WASM plugins that make 2-3 sequential API calls (common pattern: price feed + fallback + validation) pay 50-200μs × 3 = 150-600μs overhead
with Option A. With Option B, it's 50-200μs × 1 = one connection for the whole WASM execution. - More importantly, VSOCK connection setup involves kernel context switches between the enclave VM and parent VM. Under load (10+ concurrent
tasks), this adds up.
Reliability:
- Option A: if the Egress listener is temporarily overloaded, each HTTP request independently retries the VSOCK connect. 3 requests = 3
chances to hit a connect timeout. - Option B: one connection at WASM start. If it connects, all HTTP requests flow through. If it fails, you get one clean error before WASM
starts — no partial execution where the first fetch succeeds but the second fails to connect.
Why not Option C:
- Long-lived connections across WASM executions mean connection state outlives request scope. A stuck connection from a previous task
poisons the next task. The compute port already proved that per-request connections (Option A pattern) are reliable — Option B is the
natural middle ground: per-WASM-execution, not per-HTTP-request.
Revised recommendation: Option B. One VSOCK connection opened when WASM execution starts, multiple HttpRequest/HttpResponse frames flow
through it, closed when WASM completes. Clean failure isolation per WASM execution, minimal overhead.
What's your view?
✻ Churned for 21s
❯ agreed. option B
⏺ Question 10: What security controls should the Egress Service enforce?
The Egress Service on the host runs reqwest on behalf of the enclave. A malicious or compromised WASM plugin could try to abuse this to
exfiltrate data or attack internal services. What restrictions should the Egress proxy enforce?
- (A) No restrictions: The enclave can make any HTTP request to any URL. Simple but dangerous — a malicious WASM plugin could call
http://169.254.169.254/latest/meta-data/ (AWS instance metadata) or scan internal networks. - (B) Domain allowlist: Only requests to pre-configured domains are proxied. The policy client specifies allowed domains at registration
time (e.g., api.pyth.network, hermes.pyth.network). Any request to a non-whitelisted domain is rejected by the Egress Service. - (C) URL pattern allowlist + rate limiting: Domain allowlist (from B) plus rate limiting per WASM execution (max N requests, max M bytes
response) to prevent resource exhaustion. Block private IP ranges (10.0.0.0/8, 172.16.0.0/12, 169.254.0.0/16, localhost) unconditionally.
My recommendation: Option C. The private IP blocking is essential (prevents SSRF against AWS metadata and internal services). Rate limiting
prevents a buggy or malicious WASM plugin from DDoSing external APIs through the operator. The domain allowlist can start as a deny-list
(block private ranges) and evolve into a per-policy-client allowlist in a future iteration.
MVP implementation: block private IP ranges + block AWS metadata endpoint + configurable max requests per WASM execution (default 10) +
configurable max response body size (default 1MB).
What's your view?
✻ Cogitated for 21s
❯ C
⏺ I've covered the major decision branches. Let me summarize the design and check if there are remaining questions.
Phase 2 Design Summary
┌───────────────────────┬────────────────────────────────────────────────┬─────────────────────────────────────────────────────────────┐
│ Decision │ Choice │ Rationale │
├───────────────────────┼────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ What moves inside │ WASM execution + secrets (Option B) │ Protects secrets at rest; IPFS/policy fetch stays on host │
├───────────────────────┼────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ When does WASM │ Prepare phase (Option A) │ Preserves Two-Phase Consensus model │
│ execute │ │ │
├───────────────────────┼────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ Egress Service │ Framed VSOCK request/response (Option A) │ Matches existing compute port pattern; Option C (end-to-end │
│ transport │ │ TLS) deferred to Phase 2.5 │
├───────────────────────┼────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ Secrets delivery │ Encrypted via VSOCK, decrypt inside enclave │ Secrets never in host memory │
│ │ (Option A) │ │
├───────────────────────┼────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ WASM binary handling │ Host sends raw bytes, enclave compiles + │ Enclave independently verifies CID/policyCodeHash integrity │
│ │ caches (Option B) │ │
├───────────────────────┼────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────┤
├───────────────────────┼────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ VSOCK message flow │ Single PrepareEval message (Option A) │ One round-trip for WASM + partials │
├───────────────────────┼────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ Egress connection │ Per-WASM-execution (Option B) │ Clean failure isolation, minimal overhead │
│ model │ │ │
├───────────────────────┼────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────┤
│ Egress security │ Private IP blocking + rate limiting + domain │ Prevents SSRF, resource exhaustion │
│ │ deny-list (Option C) │ │
└───────────────────────┴────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────┘
Two remaining questions:
Question 11: Does the LoopbackEnclave need to simulate the Egress Service for testing?
In the LoopbackEnclave (in-process, no VSOCK), the WASM fetch() host function currently calls reqwest directly. For Phase 2, the production
path would be VSOCK→Egress→reqwest. The LoopbackEnclave needs to either:
- (A) Keep direct reqwest in LoopbackEnclave: WASM fetch() in loopback mode calls reqwest directly (no Egress simulation). Tests exercise
the WASM compilation + secrets decryption + Rego eval path but skip the Egress proxy path. - (B) Simulate Egress in LoopbackEnclave: LoopbackEnclave spawns a local Egress Service thread that the in-process WASM connects to via
localhost TCP. Full path testing but more complex test setup.
My recommendation: Option A. The Egress Service is a simple proxy — its correctness is testable independently (unit test for framing +
security controls). The LoopbackEnclave's value is testing the crypto + WASM + Rego pipeline, not network proxying. Separate integration
test for the Egress Service.
What's your view?
✻ Crunched for 28s
※ recap: We're grilling Phase 2 TEE design (WASM execution inside enclave). All major decisions are resolved. Last question: should
LoopbackEnclave simulate the Egress Service for testing, or keep direct reqwest. I recommend direct reqwest (option A) for simplicity.
❯ what are pros vs cons?
⏺ Option A: Direct reqwest in LoopbackEnclave
┌───────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────┐
│ Pros │ Cons │
├───────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Simple — no test infra changes, existing │ Egress security controls (IP blocking, rate limiting) never tested in E2E │
│ LoopbackEnclave pattern unchanged │ │
├───────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ │ Code path divergence: loopback uses reqwest directly, production uses │
│ Fast — no proxy overhead in tests │ VSOCK→Egress→reqwest. A bug in the Egress framing/serialization could pass loopback but │
│ │ fail production │
├───────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────┤
│ Egress Service testable independently via │ If WASM fetch() has different behavior when going through the proxy (e.g., timeout │
│ unit tests │ handling, response size limits), E2E tests won't catch it │
└───────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────┘
Option B: Simulate Egress in LoopbackEnclave
┌───────────────────────────────────────────────────────────────────────────┬───────────────────────────────────────────────────────────┐
│ Pros │ Cons │
├───────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
│ Full pipeline tested end-to-end — same code path as production │ More complex test setup — need a local TCP listener │
│ │ simulating the Egress Service │
├───────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
│ Egress security controls (IP blocking, rate limiting) exercised in every │ Slower test setup (~50ms for spawning listener) │
│ E2E run │ │
├───────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
│ Catches serialization bugs in HttpRequest/HttpResponse framing │ LoopbackEnclave becomes less "loop back" and more "local │
│ │ integration" — blurs the line │
├───────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────┤
│ Production and test code paths are identical — no conditional if loopback │ Requires the WASM executor to be parameterized by │
│ { reqwest } else { vsock_egress } │ transport (reqwest vs proxy client) │
└───────────────────────────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────┘
The real risk with Option A: the WASM fetch() implementation inside the enclave would need an if loopback { reqwest::Client } else {
EgressClient } branch. Every conditional path that only runs in production is a bug waiting to happen. The Phase 1 LoopbackEnclave works
because EnclaveState::evaluate() is identical in both modes — no conditional branching.
Revised recommendation: Option B. The LoopbackEnclave should simulate the Egress path with a local TCP proxy. This keeps the WASM fetch()
implementation unconditional — it always goes through the EgressClient. In loopback mode, the EgressClient connects to a localhost TCP
listener instead of VSOCK. In production, it connects to VSOCK port 5006. The transport is parameterized, not the fetch logic.
This mirrors how EnclaveTransport is already parameterized: LoopbackEnclave (in-process) vs VsockEnclave (VSOCK). The Egress Service gets
the same treatment: LoopbackEgress (localhost TCP) vs VsockEgress (VSOCK port 5006).
What's your view?
✻ Cogitated for 24s
❯ Option B yes
⏺ Question 12 (final): What's the incremental delivery plan?
Phase 2 is bigger than Phase 1 — it touches the enclave binary (wasmtime), the WASM executor, the Egress Service (new component), the
operator Prepare handler, and the VSOCK protocol. Shipping it all at once is risky. How should we slice it?
- (A) One big PR: Everything in one branch. Fast to develop, hard to review, high merge risk.
- (B) Bottom-up vertical slices:
a. Egress Service — new VSOCK_PORT_EGRESS listener on the operator host, framed HttpRequest/HttpResponse protocol, security controls (IP
blocking, rate limiting). Standalone, testable independently. No enclave changes.
b. Enclave WASM runtime — add wasmtime to enclave binary, EgressClient transport (parameterized: loopback TCP vs VSOCK), WASM compilation
- caching inside enclave. LoopbackEgress for testing. No secrets yet.
c. Secrets in enclave — encrypted secrets delivered via VSOCK, decrypted inside enclave, injected into WASM args. Operator routing: if
secrets { enclave_wasm } else { host_wasm }.
d. PrepareEval message — unified single-message flow (WASM + partials in one request). Wire protocol version bump.
e. E2E test — just e2e-tee --wasm true with a test WASM plugin that uses encrypted secrets.
- (C) Feature-flag gated: Everything behind #[cfg(feature = "enclave-wasm")], ship incrementally but merge to main behind the flag.
My recommendation: Option B. Each slice is independently reviewable, testable, and mergeable. Slice 1 (Egress Service) can ship before any
enclave changes. Slice 2 proves WASM compiles and runs inside the enclave. Slice 3 adds the security-critical secrets path. Slice 4
optimizes the wire protocol. Slice 5 validates end-to-end.
This matches how Phase 1 was delivered: core crypto → operator plumbing → gateway wiring → E2E test.
What's your view?
✻ Worked for 24s
❯ B
⏺ All decision branches resolved. Here's the final summary.
Phase 2 TEE: WASM in Enclave — Agreed Design
┌─────┬────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ # │ Decision │ Choice │
├─────┼────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 1 │ What moves inside │ WASM execution + secrets (not IPFS/policy fetch) │
├─────┼────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 2 │ When WASM executes │ Prepare phase (preserves Two-Phase Consensus) │
├─────┼────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 3 │ Egress transport │ Framed VSOCK HttpRequest/HttpResponse on port 5006 (Option C end-to-end TLS deferred to Phase 2.5) │
├─────┼────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 4 │ Secrets delivery │ Encrypted SecureEnvelope via VSOCK, decrypt inside enclave │
├─────┼────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 5 │ WASM binary │ Host sends raw bytes, enclave compiles + caches + verifies CID/policyCodeHash │
├─────┼────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 6 │ Which tasks │ Only tasks with encrypted secrets (secrets_envelope.is_some()) │
├─────┼────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 7 │ VSOCK message │ Single PrepareEval message (WASM + secrets + enc_points + peer_pubkeys → policyTaskData + │
│ │ │ encrypted_partials) │
├─────┼────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 8 │ Egress connection │ Per-WASM-execution (one connection for all HTTP calls within one WASM run) │
│ │ model │ │
├─────┼────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 9 │ Egress security │ Block private IPs + AWS metadata, rate limit per execution (10 requests, 1MB response), domain │
│ │ │ deny-list │
├─────┼────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 10 │ LoopbackEnclave │ Simulate Egress via LoopbackEgress (localhost TCP). Parameterized transport like EnclaveTransport. │
│ │ testing │ │
├─────┼────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 11 │ Delivery plan │ 5 vertical slices: Egress Service → Enclave WASM runtime → Secrets in enclave → PrepareEval message → │
│ │ │ E2E test │
└─────┴────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Future improvement (Phase 2.5): End-to-end TLS from enclave to external APIs via vsock-to-TCP bridge. Protects secrets in transit through
the parent — currently only protected at rest.
Now let me document this in docs/ENCLAVE.md, add the Phase 2.5 improvement note, and create the Linear tickets. Let me do that now.
⏺ Skill(/save-progress)
⎿ Successfully loaded skill
⏺ Here's the progress preview:
Project: TEE Integration (Phase 2: WASM in Enclave)
Linear: NEWT-952
Summary: Completed design grill for Phase 2 TEE — WASM execution inside enclave with Egress Service. Resolved 12 architectural decisions
through structured questioning. Also completed Phase 1b E2E testing (all tests passing) and fixed several production-path issues (Prepare
quorum elevation, operator_id, on-chain key registration, test DB URLs).
Key Decisions (Phase 2 design):
- WASM + secrets move inside enclave; IPFS/policy fetch stays on host
- WASM runs in Prepare phase (preserves Two-Phase Consensus)
- Egress: framed VSOCK HttpRequest/HttpResponse on port 5006 (end-to-end TLS deferred to Phase 2.5)
- Secrets: encrypted SecureEnvelope via VSOCK, decrypted inside enclave
- WASM binary: host sends raw bytes, enclave compiles/caches/verifies CID+policyCodeHash
- Only tasks with encrypted secrets route WASM to enclave
- Single PrepareEval VSOCK message (WASM + secrets + partials in one round-trip)
- Per-WASM-execution Egress connection (one connection for all HTTP calls within one run)
- Egress security: block private IPs, rate limit (10 req, 1MB), domain deny-list
- LoopbackEgress simulates Egress via localhost TCP (parameterized transport, no conditional branching)
- Delivery: 5 vertical slices (Egress → WASM runtime → Secrets → PrepareEval → E2E test)
Phase 1b fixes this session:
- Prepare quorum: max(bls_quorum%, threshold_min%) when threshold configured
- EnclaveState.operator_id set from task_signer.operator_id() at build time
- On-chain enclave key registration in E2E test (mirrors production)
- test_database_url() shared helper across all 7 E2E operator configs
- get_enclave_version_registry_address address helper
- GetPublicKeyRpcResponse parsing fix (result.public_key, not bare result)
- NewtonPolicy/NewtonPolicyData SemVerMixin fix (cherry-picked to main)
- All E2E tests passing: e2e-tee, e2e-tee --threshold true, e2e --scenario privacy, e2e-gateway
Remaining TODOs:
- Document Phase 2 design in ENCLAVE.md
- Create Linear tickets for the 5 delivery slices
- Create Linear ticket for Phase 2.5 (end-to-end TLS)
- Get PR #594 reviewed and merged
Looks good?
My revised recommendation: Start with Option A for Phase 2 MVP (ships fast, matches existing VSOCK pattern, performance-optimal). Add Option
C as Phase 2.5 when the threat model requires end-to-end TLS from enclave to external APIs (e.g., when handling highly sensitive financial
API keys where even the parent shouldn't see them).
The key insight: Option A protects secrets at rest (encrypted in DB, decrypted only inside enclave). Option C additionally protects secrets
in transit through the parent. Whether transit protection is worth the complexity depends on how much you trust the operator's own host.
Which approach do you want?
✻ Brewed for 28s
❯ yes option A, but let's document about option C improvements in our docs @docs/ENCLAVE.md and also make sure we track in Linear as well
⎿ Read docs/ENCLAVE.md (669 lines)
⏺ Agreed. Let me continue the grill first, then we'll document and create the Linear ticket at the end.
❯ yes option A, but let's document about option C improvements in our docs @docs/ENCLAVE.md and also make sure we track in Linear as well
⎿ Read docs/ENCLAVE.md (669 lines)
⏺ Agreed. Let me continue the grill first, then we'll document and create the Linear ticket at the end.
Question 4: How do secrets get into the enclave for WASM execution?
Today, WASM secrets (API keys, endpoint URLs) are encrypted per-operator via newt_storeEncryptedSecrets and stored in the DB. During
Prepare, the operator host fetches from DB, HPKE-decrypts with its individual key, and injects into the WASM sandbox args.
With WASM inside the enclave, the enclave needs the secrets. Three approaches:
decrypt — it only has the HPKE key inside the enclave). It passes the raw encrypted blob via the VSOCK EnclaveEvalRequest. The enclave
decrypts with its HPKE key and injects into WASM args. Secrets never touch host memory in plaintext.
host memory briefly. Defeats the purpose of Phase 2.
to enclave binary — increases TCB significantly.
My recommendation: Option A. The operator host already fetches encrypted blobs from DB (it has DB access). It passes them as opaque bytes
over VSOCK. The enclave decrypts inside its isolation boundary. This is the natural extension of the existing SecureEnvelope pattern — same
as how identity/confidential envelopes are already passed. The EnclaveEvalRequest already carries encrypted envelopes; adding encrypted
secrets is one more field.
The key insight: secrets are encrypted to the individual operator HPKE key (via newt_getSecretsPublicKey), not the threshold MPK. So each
enclave can decrypt its own secrets independently — no threshold partials needed.
What's your view?
✻ Cooked for 24s
❯ agreed option A
⏺ Question 5: What about the WASM binary itself — does it need to be inside the enclave?
Today the WASM component is fetched from IPFS, compiled via wasmtime, cached in-memory (wasm_component_cache), and executed on the host. The
WASM binary is policy logic (not secret) — it's content-addressed via CID and integrity-verified.
Three options for where WASM compilation and caching live:
module, sends over VSOCK. Enclave deserializes and runs. Problem: serialized wasmtime modules are platform-specific and large (~MBs). Also,
a malicious host could send a modified WASM binary.
sends bytes over VSOCK. Enclave compiles with wasmtime and caches in-memory. Enclave can independently verify keccak256(wasm_bytes) ==
policyCodeHash from on-chain. Problem: wasmtime compiler adds ~20MB to enclave binary and compilation takes 400-500ms (from logs: wasm
component compiled and cached compile_ms=507).
(500ms penalty per task).
My recommendation: Option B. The enclave must verify WASM integrity independently (defense against malicious host injecting modified WASM).
CID + policyCodeHash checks are already in the codebase. The 500ms compile cost is a one-time hit per unique WASM binary — the in-memory
cache amortizes it across tasks. wasmtime adds binary size but is already validated for the RISC-V SP1 target, so it should compile for the
enclave's x86 target.
The alternative (option A) trusts the host to provide correct compiled code — which violates the TEE security model where the host is
semi-trusted.
What's your view?
✻ Baked for 28s
※ recap: We're designing Phase 2 TEE (WASM execution inside enclave). I was grilling you on design decisions — answered Q1-Q4 (all option A/B
agreed). Q5 is pending: should the enclave compile WASM from raw bytes (my recommendation) or trust pre-compiled modules from the host?
❯ B
⏺ Question 6: How does the enclave binary size and dependency footprint change?
Today the enclave binary is minimal: Regorus + HPKE + domain types. No Tokio server, no network libs, no DB drivers. This keeps the Trusted
Computing Base (TCB) small and auditable.
Phase 2 adds:
DON'T need reqwest inside the enclave. The enclave sends HttpRequest structs over VSOCK and the host runs reqwest.
So the enclave gains:
It does NOT gain:
My recommendation: Accept the wasmtime dependency increase. It's the core of the feature. The PCR0 changes (new binary measurement), so the
EnclaveVersionRegistry needs a version activation for the Phase 2 enclave image. This is expected — same as any enclave binary update.
One concern: memory pressure. Nitro Enclaves have limited memory (configurable, default ~512MB). wasmtime compilation + WASM execution +
Rego evaluation + HPKE decryption all compete for memory. The enclave config already has memory_mib (default from config). Do we need to
increase the default for Phase 2?
Searched for 1 pattern, read 1 file (ctrl+o to expand)
⎿ PreToolUse:Read hook error ⎿ Failed with non-blocking status code: [TokenEfficiency] Hard rule
violation: Re-reading config.rs (2x) — file unchanged since last
read. Consider using cached knowledge.
⏺ 4096 MiB (4 GB) default. That's generous — wasmtime compiles a 144KB WASM binary in ~500ms using maybe 50-100MB peak. With Rego eval + HPKE
No config change needed. Moving on.
Question 7: What's the new VSOCK message flow for Prepare phase with WASM?
Today's Prepare flow:
Host: IPFS fetch WASM → compile → execute (reqwest HTTP) → policyTaskData
Host: compute enc_points from envelopes
Host → Enclave: PartialDh(enc_points, peer_pubkeys)
Enclave → Host: EncryptedPartialDH blobs
Host → Gateway: policyTaskData + encrypted_partials
Phase 2 Prepare flow — two options:
(A) Single new message type PrepareEval: The host sends WASM bytes + encrypted secrets + WASM args + enc_points + peer_pubkeys in one
request. The enclave compiles WASM, decrypts secrets, executes WASM (proxying HTTP via Egress), produces policyTaskData, then computes
partial DHs. Returns policyTaskData + encrypted_partials in one response. One VSOCK round-trip.
(B) Two messages: First WasmExec (WASM bytes + secrets + args → policyTaskData via Egress), then PartialDh (enc_points + peer_pubkeys →
encrypted_partials). Two VSOCK round-trips but simpler — each message does one thing.
My recommendation: Option A. The host doesn't need intermediate policyTaskData access (it flows straight back to the gateway). Combining
into one round-trip saves ~100-200μs connect overhead and simplifies the operator Prepare handler. The enclave already handles multi-step
logic in evaluate() (decrypt → domain → Rego). Adding WASM exec at the front is the same pattern.
What's your view?
✻ Baked for 55s
※ recap: We're designing Phase 2 TEE (WASM execution inside enclave). Grilling through design decisions — decided on framed VSOCK egress,
encrypted secrets via VSOCK, WASM compiled inside enclave. Currently on question 7: whether Prepare phase uses one combined VSOCK message or
two separate ones. Awaiting your answer on that.
❯ A