Skip to content

Instantly share code, notes, and snippets.

@denniswon

denniswon/gist:2625d7112c3e9eea95c46c5a2be95a10

Last active May 15, 2026 01:32
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save denniswon/2625d7112c3e9eea95c46c5a2be95a10 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save denniswon/2625d7112c3e9eea95c46c5a2be95a10 to your computer and use it in GitHub Desktop.
Download ZIP
Protocol state architecture migation
Raw
gistfile1.txt
★ Insight ─────────────────────────────────────
- EigenDA was the trustless intermediary for upload broadcast — gateway posts blob, operators pull, operators verify the KZG cert
against an L1 anchor before applying. Direct gateway broadcast trades that property for latency.
- The 120s commit cadence (Q3) and the daCertHash field on StateCommit (IStateRootCommittable.sol, locked in §6, error selectors
0x822ef683/0xdc4e1d57) were both calibrated to EigenDA economics (~$/blob, 14-day retention cliff). Removing EigenDA from the hot path
unlocks both knobs.
- The compliance constraint in §S.2 ("gateway MUST NOT hold authoritative private data") was load-bearing for the original topology.
Dennis's meeting line — "if bootnode is also we are operating it, then it's not really a difference" — collapses gateway and bootnode
into one Newton-trust footprint, which means the §S.2 invariant needs to be re-derived from a new principle, not just relaxed.
─────────────────────────────────────────────────
Meeting outcome — my reading
Three locked shifts (please confirm before I go deeper):
1. EigenDA removed from upload hot path. Gateway broadcasts uploads directly to operators AND a single-instance bootnode. Operators
write to local redb on receipt; no EigenDA pull step.
2. Bootnode = single-instance Newton-operated HTTP server with redb (not Postgres, not load-balanced). Two roles: (a) replica receiver
for every gateway broadcast, (b) snapshot+delta index for new-operator catch-up.
3. Compliance constraint relaxed. Gateway holding encrypted-PII is now equivalent to bootnode holding it — both are Newton-operated.
Tentative (unresolved in the call): commit cadence may drop below 120s; EigenDA's role outside the hot path is unsettled — Alec is
holding his EigenDA PR pending your architectural decision.
Cascade in docs/PRIVATE_DATA_STORAGE.md
§3 Topology · §6 Commit Protocol (daCertHash, cadence, §6.6 CommitProposalV1) · §8 entire EigenDA section · §9 Data Flow upload path ·
§S.2 gateway data-plane invariant · §S.4 EigenDA cert rot · §S.18 retrieval ladder · §S.19 bootnode SPOF (amplified) · §F.6
drop-bootnode-for-P2P (now reframed) · Q2/Q12/Q13/Q14/Q18/Q21/Q29 in design-decisions.
First lock to pin down
The pivotal axis is EigenDA's role going forward — every other decision (cadence, daCertHash field, tiered recovery, catch-up SLA,
snapshot storage) cascades from it.
⏺ User answered Claude's questions:
⎿  · In the new architecture, what should EigenDA's role be going forward? → Defer to Phase 1.5+
⏺ ★ Insight ─────────────────────────────────────
- EigenDA's deferral cleanly removes a 14-day retention cliff that was forcing the §6/§S.4/§S.18 tiered-recovery scaffolding. But it
transfers the "trustless data availability" property to the bootnode's redb — which is operationally cheap but cryptographically opaque.
- The Q12 tiered recovery (0-7d / 7-14d / 14d+) collapses to a single tier. §F.6 ("drop bootnode, operator P2P") gets repositioned: P2P
becomes the trustless-recovery story, not a Phase 2 nice-to-have.
- A deferred-but-not-deleted EigenDA needs a clean re-entry seam in the spec — a Phase 1.5 marker noting which fields/flows would
re-activate. Otherwise re-introduction in 6 months means re-deriving the design from scratch.
─────────────────────────────────────────────────
Confirmed: EigenDA out of Phase 1
That locks the upload-path simplification and means I can now resolve the cascading questions in dependency order: contract surface
first (hardest to unwind), then config knobs, then operations.
Next dependency: StateCommit field semantics
The StateCommit struct in IStateRootCommittable.sol carries daCertHash: bytes32 — a field whose original purpose was binding the
BLS-signed commit to a specific EigenDA blob (the trustless integrity anchor). With EigenDA gone, this field is either obsolete,
repurposable, or a vestigial slot.
This matters because the field is in the BLS message preimage (keccak256(abi.encode(commit)) per CertificateMessageHashMismatch selector
0x822ef683). Whatever we do here affects every operator's signing path AND the on-chain verifier.
The principal cryptographic question: how do we prevent a Byzantine gateway from fork-broadcasting different delta bytes to different
operators? EigenDA's KZG cert was the public binding. In the new flow, we need a substitute.
My recommended option is (1) Repurpose daCertHash → deltaContentHash. Keep the field, change the semantic to
keccak256(canonical_delta_bytes). Operators compute on receipt and include in their BLS partial; mismatch on aggregation surfaces a
Byzantine gateway. This preserves the Q14 leader-ECDSA invariant and keeps the existing 7-selector taxonomy in error-selectors.md
intact.
⎿  Tool use rejected
⏺ The user wants to clarify the question before answering. Let me ask what needs explanation.
What would you like to clarify about the StateCommit.daCertHash question? Some dimensions you might want to dig into:
- What "delta canonical bytes" means — the exact serialization scope (just the writes? writes + ordering invariant? full Delta blob
frame including DELTA_BLOB_FORMAT_V1 header from Q29?)
- Why this field needs to change at all — i.e., what specifically breaks if we leave daCertHash semantics alone but stop populating
EigenDA
- What a Byzantine-gateway-fork attack looks like concretely — the scenario this field defends against, and whether existing primitives
(Q14 leader ECDSA + Q17 CommitProposalV1.proposer_id) already cover it
- Migration cost of bumping V1 → V2 — given that UnsupportedStateCommitVersion (0xb681668e) is already the typed selector, what
coordination is needed between gateway/operator/challenger versions
- The struct-V2 contract migration mechanics — given PR #589's StateCommitRegistry is still in review, whether Option 2/3 is "free"
right now versus a real upgrade later
- Whether this field is needed at all — broader architectural question: do we need any delta-content binding on-chain, or is off-chain
CommitProposalV1 + bootnode-served bytes sufficient
Or something else entirely about the framing of the question. Let me know what direction you want to take it.
✻ Baked for 5m 43s
▐▛███▜▌ Claude Code v2.1.132
▝▜█████▛▘ Opus 4.7 (1M context) with max effort · Claude API
▘▘ ▝▝ ~/projects/newton-prover-avs
✻ Conversation compacted (ctrl+o for history)
⎿  Read docs/CLAUDE.md (7 lines)
⎿  Read ../../.claude/RTK.md (30 lines)
⎿  Read ../../CLAUDE.md (39 lines)
⎿  Referenced file .claude/CLAUDE.md
⎿  Referenced file .claude/rules/architecture.md
⎿  Skills restored (superpowers:brainstorming)
⏺ ★ Insight ─────────────────────────────────────
EigenDA's removal collapses Phase 1 into a single trust path: gateway → operators → on-chain root. This is structurally simpler than the
prior tiered (S3 → EigenDA → blob recovery) ladder, but concentrates SPOF risk on the bootnode for catch-up. JMT determinism already
guarantees state correctness from (prevRoot, newRoot) alone, so the on-chain deltaContentHash is a fast-verification anchor for catch-up
consumers — not a state-integrity primitive. That separation cleans up the §S.18 retrieval ladder considerably.
─────────────────────────────────────────────────
Two decisions are locked (EigenDA → Phase 1.5+, on-chain deltaContentHash). Here's the hardened design for the remaining cascading
questions — push back on any section.
1. Commit cadence — keep 120s default, per-chain config seam
EigenDA economics no longer gate it, but BLS aggregation budget, gas, and operator processing windows are all calibrated to 120s. Add
gateway.state_commit.interval_seconds per-chain config so we can tune under real mainnet load without a contract change. Cadence is
decoupled from StateCommit semantics — timestamp is monotonic regardless.
2. Bootnode posture — single instance Phase 1, hot standby Phase 1.5
Phase 1 ships one Newton-operated bootnode with redb-backed delta archive (acknowledged SPOF, documented in §S.19). Phase 1.5 adds
hot-standby behind a load balancer with replicated redb. Avoids over-engineering before real load patterns; matches staged-rollout
philosophy.
3. §S.2 compliance principle — recast as "gateway authority + operator data-plane"
Original §S.2 said gateway never holds raw plaintext. EigenDA removal doesn't change this — gateway still terminates auth, dispatches
HPKE-wrapped blobs to operators, never persists. New framing: gateway is the authority/admission boundary; operators are the sole
data-plane. Closes one trust hop, doesn't add any.
4. Canonical delta bytes — RLP-encoded Vec<(LeafKey, LeafHeaderV1, ciphertext)> sorted by leaf_key
deltaContentHash = keccak256(rlp_encode(sorted_delta)). Sort BEFORE encoding so any two operators with the same logical delta compute
identical hashes. RLP avoids the JSON-ordering hazards (per lessons.md "Cross-service trust"). Reserve a version byte for migration.
5. Phase 2 P2P transition — §F.6 reframed
Original §F.6 was "drop bootnode for P2P". Reframe: operator-to-operator gossip for delta replication only (catch-up, not consensus).
Triggers: bootnode bandwidth saturation, geographic latency, trust diversification. StateCommitRegistry root remains the on-chain anchor
— P2P is data-plane only.
6. Cascade updates required
- docs/PRIVATE_DATA_STORAGE.md: §3 topology, §6 commit, §8 delete, §9 flow, §S.2 reframe, §S.4 delete, §S.18 rewrite, §S.19, §F.6
reframe
- docs/superpowers/specs/2026-04-20-private-data-migration-design-decisions.md: Q2, Q12, Q13, Q14, Q18, Q21, Q29 amendments
- docs/STATE_TREE_ARCHITECTURE.md: no change (interface preserved)
- .claude/rules/architecture.md: StateCommitRegistry row note (daCertHash → deltaContentHash)
Approve sections 1–5 to lock? On approval I'll write the spec to docs/superpowers/specs/2026-05-07-pds-eigenda-removal-design.md, run
spec self-review, then transition to writing-plans for the cascade.
✻ Cogitated for 4m 34s
❯ Approve except 1: commit cadence 120s is too long. Why is it set to 120 seconds? where does the requirement / bottleneck come from? 2:
operator p2p is for censorship resistance as well, deprecating gateway (replaced by p2p) and bootnode just become an archive node (it
already kind of is)
⏺ ★ Insight ─────────────────────────────────────
The 120s cadence is a composite number — three contributors stacked: operator response budget (~20s realistic), tx confirmation + reorg
margin (10-20s, chain-specific), and EigenDA blob cert confirmation (30-60s, NOW REMOVED). With EigenDA gone, the floor drops
dramatically. The remaining bottleneck is per-chain (gas, block time, finality), so cadence becomes a chain-specific knob, not a global
default.
─────────────────────────────────────────────────
1 (revised). Commit cadence — per-chain config, aggressive defaults
Where 120s came from:
- ~20s operator response budget (RPC fan-out → BLS partial sign → aggregator collect)
- ~30-60s EigenDA blob cert confirmation (REMOVED)
- ~10-20s tx confirmation + reorg margin (chain-specific)
- Historical reuse from prior policy-task consensus tick — no real consensus driver
Real floor now: operator budget (~20s) + tx confirmation. ~30s on fast L2s, ~45s on L1.
Per-chain interval_seconds config with proposed defaults:
- L1 mainnet: 60s — gas dominates; reorg margin meaningful
- L2 (Base, Arb): 30s — cheap gas, fast finality, write→read latency wins
- Local/dev: 5s — throughput testing
Halves staleness on L1, quarters it on L2. Operator budget stays comfortable. Going below ~25s risks stragglers on slow regional links
and butts up against tx confirmation.
★ Insight ─────────────────────────────────────
The P2P reframe is bigger than catch-up. Phase 1's hub-and-spoke (gateway = single ingress) becomes Phase 2's mesh (federated gateways +
operator gossip), and Phase 3's gateway deprecation. Bootnode contracts to archive — already its actual role; the redb delta history is
exactly an archive. Censorship resistance comes from redundancy at every layer: multiple gateways, operator-direct submission, multiple
archives.
─────────────────────────────────────────────────
5 (revised). P2P reframe — censorship resistance + gateway deprecation endgame
Phase 1: gateway → operators direct, bootnode broadcasts catch-up.
Phase 2 (P2P transition):
- (a) Operator-to-operator libp2p gossip for delta replication — catch-up without bootnode dependency
- (b) Federated gateways: anyone can run one; censorship resistance via gateway diversity
- (c) Optional direct user → operator P2P submission for advanced clients
- (d) Bootnode demoted to pure archive: passive gossip subscriber, redb persistence for cold queries. This just promotes its actual
existing role.
Phase 3 endgame: gateway deprecated. Operators are first-class P2P endpoints. Gateway optional as a paid fast-lane for high-throughput
users. Permissionless gossip with peer scoring, DoS protection, per-message sig verification at the operator boundary.
StateCommitRegistry on-chain root remains the anchor across all phases — P2P is data-plane only, never consensus.
§F.6 in PRIVATE_DATA_STORAGE.md becomes a 3-phase trajectory: catch-up → censorship-resistance → gateway-deprecation. Each compounds on
the prior. §S.19 bootnode-SPOF section evolves alongside: SPOF risk diminishes as Phase 2 lands, vanishes by Phase 3.
Sections 2, 3, 4, 6 locked. Cadence (1) and P2P (5) revised above — approve to lock the design and proceed to spec write?
✻ Crunched for 1m 53s
❯ 1. - ~20s operator response budget (RPC fan-out → BLS partial sign → aggregator collect) --> where did you get this number 20s from?
our goal is to be 1 second, and definitely it HAS to be (requirement) to be under 4 seconds max.
⏺ ★ Insight ─────────────────────────────────────
20s was wrong — I anchored on policy-task heuristics where stragglers and complex Rego evaluation extend the window. State commit is
purely root-read + BLS-sign, dominated by network RTT not computation. Real steady-state response is sub-second; 4s is generous
tail-budget; 1s target requires regional clustering or quorum-deadline tuning. Owning the bad number — that was a heuristic transplant
from the wrong domain.
─────────────────────────────────────────────────
You're right — 20s was wrong. Recomputing with accurate per-step costs:
Operator response breakdown (target 1s, max 4s)
┌──────────────────────────────────────────────────────────────────┬──────────────────┬───────────────────────┐
│ Step │ Steady state │ Worst case │
├──────────────────────────────────────────────────────────────────┼──────────────────┼───────────────────────┤
│ Aggregator → operator request (JSON-RPC) │ 50ms same-region │ 200ms cross-continent │
├──────────────────────────────────────────────────────────────────┼──────────────────┼───────────────────────┤
│ Operator: read JMT cached root from redb │ <5ms │ <50ms cold cache │
├──────────────────────────────────────────────────────────────────┼──────────────────┼───────────────────────┤
│ Operator: build StateCommit, BLS partial sign (BN254) │ 30-50ms │ 100ms │
├──────────────────────────────────────────────────────────────────┼──────────────────┼───────────────────────┤
│ Operator → aggregator return │ 50ms │ 200ms │
├──────────────────────────────────────────────────────────────────┼──────────────────┼───────────────────────┤
│ Aggregator: collect 2/3 quorum (waits for slowest qualifying op) │ 200ms │ 1.5s regional tail │
├──────────────────────────────────────────────────────────────────┼──────────────────┼───────────────────────┤
│ BLS aggregation + verification │ 50-100ms │ 200ms │
├──────────────────────────────────────────────────────────────────┼──────────────────┼───────────────────────┤
│ Total │ ~400ms │ ~2.3s │
└──────────────────────────────────────────────────────────────────┴──────────────────┴───────────────────────┘
The 4s envelope holds with healthy margin. 1s target needs regionally-clustered operators or a quorum-collection deadline shorter than
the slowest-op RTT (excludes the tail from this cycle, includes it next).
Hard design requirements this exposes:
- Operators MUST maintain JMT root incrementally — no on-demand re-walk. Builder updates root on every leaf insert/update with sub-ms
cost. Already the STATE_TREE_ARCHITECTURE.md design assumption; making it an explicit invariant for state-commit RPC.
- BLS signing is host-only — no VSOCK round-trip into the enclave (already the case).
- Aggregator quorum-collection deadline ≤ 2s; stragglers dropped from current cycle.
★ Insight ─────────────────────────────────────
With response budget under 4s, the cadence floor shifts entirely to tx confirmation + reorg policy. StateCommitRegistry already has
SequenceGap + StateRootMismatch errors that cleanly reject stale views — so an aggressive-cadence + idempotent-retry strategy lets
reorged commits recover for free. The error semantics that were designed for "stale operator" failures double as the reorg-tolerance
primitive. That's an unexpected payoff from the existing design.
─────────────────────────────────────────────────
Revised cadence math
┌───────────────┬────────────┬──────────────┬─────────────────────────────────────────────────────────────────┐
│ Chain │ Aggressive │ Conservative │ Reorg posture │
├───────────────┼────────────┼──────────────┼─────────────────────────────────────────────────────────────────┤
│ L1 mainnet │ 12s │ 30-40s │ aggressive: accept reorg + retry; conservative: 3 confirmations │
├───────────────┼────────────┼──────────────┼─────────────────────────────────────────────────────────────────┤
│ L2 (Base/Arb) │ 5-10s │ 15s │ aggressive safe — low reorg + fast finality │
├───────────────┼────────────┼──────────────┼─────────────────────────────────────────────────────────────────┤
│ Local/dev │ 1-2s │ 5s │ n/a │
└───────────────┴────────────┴──────────────┴─────────────────────────────────────────────────────────────────┘
Recommendation: per-chain config with L1: 30s default (12s opt-in flag), L2: 10s default, Local: 2s. Drops L1 staleness 4× from 120s;
drops L2 staleness 12×. Operator response budget stays well under 4s; 1s aspirational target documented as "achievable with regional
clustering or sub-RTT quorum deadline."
Reorg-recovery as first-class flow — want me to fold a §6.x subsection into the spec covering: aggressive-cadence operator behavior on
SequenceGap/StateRootMismatch revert (re-read latest committed root, rebuild against fresh prev, re-aggregate)? That makes "aggressive
on L2" the safe default rather than a calculated risk.
Sections 2, 3, 4, 6 locked. P2P (5) locked from prior turn. Cadence (1) revised above — approve, or push the L2 default lower?
✻ Churned for 1m 22s
❯ approve
⏺ Writing the spec to docs/superpowers/specs/2026-05-07-pds-architecture-pivot-design.md, then I'll run spec self-review.
⏺ Write(docs/superpowers/specs/2026-05-07-pds-architecture-pivot-design.md)
⎿  Wrote 329 lines to docs/superpowers/specs/2026-05-07-pds-architecture-pivot-design.md
1 ---
2 title: Private Data Storage — Architecture Pivot Design (EigenDA removal, cadence, P2P trajectory)
3 date: 2026-05-07
4 status: approved
5 applies_to: Phase 1 MVP and forward
6 supersedes_amendments:
7 - docs/superpowers/specs/2026-04-20-private-data-migration-design-decisions.md (Q2, Q12, Q13, Q14, Q18, Q21, Q29)
8 meeting_source: Newton PDE stand-up 2026-05-07 (Fireflies 01KQDY7XR840ANYYYCDT5GVVS9)
9 ---
… +392 lines (ctrl+o to expand)
⏺ Running spec self-review now per brainstorming workflow (placeholder scan, internal consistency, scope check, ambiguity check).
Findings from self-review:
1. Placeholder scan: No TBDs, TODOs, or vague requirements. "Open questions for implementation plan" section is intentional — frames
plan-stage decisions, not unresolved design.
2. Internal consistency: D4 bootnode 3-phase aligns with D7 P2P 3-phase. D8 reorg-recovery references the existing
SequenceGap/StateRootMismatch selectors correctly. Hard invariants don't contradict any decision.
3. Scope check: Single implementation plan size — sizable but cohesive (9 doc sections, 1 field rename, 1 config schema change, 1 helper
crate, 1 RPC handler enhancement).
4. Ambiguity check: Two minor fixes needed.
★ Insight ─────────────────────────────────────
Self-review caught two phrasings that read fine to the writer but mislead a fresh reader: (1) "active broadcast endpoint" for the
bootnode could be parsed as bootnode relaying to operators (it doesn't — gateway broadcasts to it); (2) hardcoded "2/3 quorum" in the
latency budget could be read as a locked threshold (it's illustrative; the real value is quorum_threshold_bps config). Both are
clarification edits, not design changes.
─────────────────────────────────────────────────
⏺ Update(docs/superpowers/specs/2026-05-07-pds-architecture-pivot-design.md)
⎿  Added 1 line, removed 1 line
140
141 **Decision.** Bootnode role evolves alongside the P2P trajectory.
142
143 -**Phase 1 (this MVP).** Single Newton-operated bootnode. redb-backed delta archive. Active broadcast endpoint: gateway p
-ushes encrypted deltas to bootnode AND directly to operators. Bootnode serves catch-up to new operators on demand.
143 +**Phase 1 (this MVP).** Single Newton-operated bootnode. redb-backed delta archive. Receives gateway broadcasts and pers
+ists encrypted deltas to redb (passive in steady state). Serves catch-up to new operators on demand (active only when qu
+eried). Gateway dispatches encrypted deltas in parallel to operators (data plane) and to bootnode (archive).
144
145 **Phase 1.5.** Hot-standby bootnode behind a load balancer with replicated redb. SPOF mitigation. Triggered by either (a
) operational maturity milestone, (b) observed bootnode bandwidth pressure, or (c) regulatory requirement for redundancy
.
146
⏺ Update(docs/superpowers/specs/2026-05-07-pds-architecture-pivot-design.md)
⎿  Added 1 line, removed 1 line
124 | Operator: read JMT cached root from redb | <5ms | <50ms cold cache |
125 | Operator: build `StateCommit`, BLS partial sign (BN254) | 30-50ms | 100ms |
126 | Operator → aggregator return | 50ms | 200ms |
127 -| Aggregator: collect 2/3 quorum (slowest qualifying op) | 200ms | 1.5s regional tail |
127 +| Aggregator: collect quorum per `quorum_threshold_bps` (slowest qualifying op) | 200ms | 1.5s regional tail |
128 | BLS aggregation + verification | 50-100ms | 200ms |
129 | **Total** | **~400ms** | **~2.3s** |
130
⏺ ★ Insight ─────────────────────────────────────
Self-review pass complete. Both ambiguity fixes were micro-edits (no design changes). The spec is the authoritative reference for the
implementation plan — it locks 8 decisions, lists 8 system-wide hard invariants the plan must respect, and explicitly fences off Phase
2/3 work to avoid scope creep. Note docs/superpowers/** is gitignored on this repo per project memory, so the spec lives locally —
committing it would be a no-op.
─────────────────────────────────────────────────
Spec written to docs/superpowers/specs/2026-05-07-pds-architecture-pivot-design.md
Self-review pass: clean.
- Placeholder scan: no TBDs/TODOs (open questions section is plan-stage only, not design-stage).
- Internal consistency: D4 bootnode 3-phase ↔ D7 P2P 3-phase fully aligned; D8 reorg-recovery references correct selectors.
- Scope check: cohesive single-plan size.
- Ambiguity check: 2 wording fixes applied (bootnode role wording, quorum threshold reference).
Per the brainstorming workflow, this is the user review gate — please read the spec end-to-end and flag anything you want changed.
Particular things worth a fresh-eyes pass:
1. D3 cadence defaults — L1: 30s default / 12s aggressive opt-in, L2: 10s default. Are those numbers what you want shipped, or do you
want L1 lower?
2. D4 Phase 1.5 trigger — left intentionally soft (operational maturity OR bandwidth pressure OR regulatory). Want a concrete trigger
written into the spec?
3. D6 RLP vs alternatives — committed to RLP. If you'd rather use bincode or SSZ, now's the time.
4. D7 Phase 3 endgame — gateway-as-paid-fast-lane is one possible endpoint. If you want gateway gone entirely (not even as a paid tier),
the spec should say so.
5. D8 reorg recovery — adds operator behavior on SequenceGap/StateRootMismatch. Plan-level work; want it kept in this spec or split out?
6. Out-of-scope list — anything you want pulled into scope?
After your approval, I'll invoke the writing-plans skill to generate the implementation plan covering: field rename (daCertHash →
deltaContentHash), per-chain cadence config schema, RLP delta encoding helper crate, reorg-recovery RPC handler logic, and the 9-section
cascade through PRIVATE_DATA_STORAGE.md.
✻ Worked for 5m 13s
❯ 1. approve 2. soft 3. help me compare rlp vs alternatives 4. sounds good for now, with "ultimate" goal to be deprecated/dissolve
(potentially) 5. keep in this spec and plan 6. no. But before we move onto implementation plan, let's do a thorough, detailed,
/grill-me session that goes through every single component of the protocol under this direction
⏺ ★ Insight ─────────────────────────────────────
Encoding comparison reveals a non-obvious split: "deterministic by design" candidates are Borsh (literally designed for hashing), RLP,
and SSZ; bincode is the fastest in pure Rust but has determinism caveats fatal for chain anchoring. The choice between RLP and Borsh
hinges on whether a future zkVM verifier needs to consume delta bytes in-circuit — Borsh is leaner there. For Phase 1 with no zkVM
consumer of deltaContentHash, RLP's Ethereum-stack idiom wins.
─────────────────────────────────────────────────
Item 3: Encoding comparison
┌─────────────┬───────────────────┬──────────────────┬──────────┬─────────────────────┬─────────────┬────────────────────────────┐
│ Format │ Deterministic │ Ethereum fit │ Rust │ Cross-lang │ Versioning │ Verdict │
│ │ │ │ perf │ │ │ │
├─────────────┼───────────────────┼──────────────────┼──────────┼─────────────────────┼─────────────┼────────────────────────────┤
│ RLP │ Structural │ Native │ Standard │ Yes │ Manual │ Current pick │
│ │ │ (alloy::rlp) │ │ │ │ │
├─────────────┼───────────────────┼──────────────────┼──────────┼─────────────────────┼─────────────┼────────────────────────────┤
│ Borsh │ Structural │ None │ Fast │ Decent │ Manual │ Strong alt │
│ │ │ │ │ (Rust/JS/Go/Py) │ │ │
├─────────────┼───────────────────┼──────────────────┼──────────┼─────────────────────┼─────────────┼────────────────────────────┤
│ bincode │ Configurable, │ None │ Fastest │ Rust-only really │ Built-in │ Determinism caveats — │
│ │ v1≠v2 │ │ │ │ │ fatal here │
├─────────────┼───────────────────┼──────────────────┼──────────┼─────────────────────┼─────────────┼────────────────────────────┤
│ SSZ │ Structural │ Eth2 CL │ Fast │ Yes │ Type-pinned │ Heavy; merkleization │
│ │ │ │ │ │ │ unused │
├─────────────┼───────────────────┼──────────────────┼──────────┼─────────────────────┼─────────────┼────────────────────────────┤
│ CBOR │ Canonical mode │ None │ Standard │ Excellent │ Manual │ Multi-encoding hazard │
│ │ only │ │ │ │ │ │
├─────────────┼───────────────────┼──────────────────┼──────────┼─────────────────────┼─────────────┼────────────────────────────┤
│ Custom │ Yours to make │ None │ Best │ Yours │ Yours │ Reinvention cost │
│ packed │ │ │ │ │ │ │
└─────────────┴───────────────────┴──────────────────┴──────────┴─────────────────────┴─────────────┴────────────────────────────┘
Why RLP still wins for Phase 1: catch-up consumers verifying deltaContentHash are already pulling alloy. No new dep. Ethereum-stack
idiom — anyone debugging keccak256(bytes) expects RLP first. Determinism is structural, no encoder-config flag to forget.
When to switch: Borsh if SP1/RISC0 ever verifies delta bytes in-circuit (smaller in-circuit decoder). Never bincode for hash anchoring.
Recommendation: keep RLP for Phase 1. Re-evaluate when/if a zkVM consumer enters the picture.
Item 4: Phase 3 spec edit (ultimate dissolution)
⏺ Update(docs/superpowers/specs/2026-05-07-pds-architecture-pivot-design.md)
⎿  Added 2 lines, removed 1 line
215
216 - Gateway optional; operators are first-class P2P endpoints.
217 - Permissionless gossip with peer scoring, DoS protection, per-message signature verification at the operator boundary.
218 -- Gateway lives on as a paid fast-lane for high-throughput users (think CDN tier on top of permissionless ingress).
218 +- Gateway initially persists as a paid fast-lane for high-throughput users (CDN tier on top of permissionless ingress).
219 +- **Ultimate trajectory: gateway potentially dissolves entirely.** The fast-lane role is itself transitional. If/when pe
+rmissionless ingress reaches throughput parity (peer scoring matures, DoS hardening stabilizes), the gateway has no rema
+ining function and the protocol becomes fully decentralized at the ingress layer. Whether to dissolve is a future decisi
+on contingent on ecosystem maturity, not committed here.
220
221 **Anchor invariant across all phases.** `StateCommitRegistry` on-chain root remains the consensus anchor. P2P is data-pl
ane only — never used for consensus or finality. This holds from Phase 1 through Phase 3 unchanged.
222
⏺ ★ Insight ─────────────────────────────────────
The grill-me sequence isn't arbitrary — it stacks bottom-up: state-tree invariants are foundation for consensus invariants, consensus
invariants are foundation for catch-up flow. If we grill out of order, later clusters keep referencing decisions from earlier clusters
that aren't locked yet, and the questions get hand-wavy. The 7-cluster sequence preserves that dependency order.
─────────────────────────────────────────────────
Phase 3 dissolution edit applied to spec.
Item 6: Grill-me agenda
Phase 3 dissolution edit applied to spec.
Item 6: Grill-me agenda
7 component clusters, ~10-15 probes each. Each cluster spans 1-3 turns. Locked answers numbered G-1, G-2, ..., folded back into spec
under a new "Grill amendments" section.
┌─────┬───────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ # │ Cluster │ Sample probes │
├─────┼───────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 1 │ Authority + │ Gateway auth termination on Phase 3 dissolution; per-chain dispatch fanout failure modes; │
│ │ admission │ plaintext-blind invariant under partial gateway failure; DoS surface │
├─────┼───────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 2 │ Operator │ Incremental JMT root invariant edge cases; redb write-tx contention with state-commit RPC; snapshot │
│ │ state-tree │ isolation for concurrent reads; leaf overwrite ordering; orphan ciphertext on aborted writes │
├─────┼───────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 3 │ State-commit │ Quorum-collection deadline straggler policy; BLS partial-sig retention/replay; reorg-recovery race │
│ │ consensus │ conditions; multi-aggregator scenarios; sequence-number gap policy │
├─────┼───────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 4 │ Catch-up + │ Delta retrieval API shape; consumer-side deltaContentHash verification; bootnode eviction policy; │
│ │ bootnode │ cold-start operator joining at sequence N≫0; hot-standby cutover semantics │
├─────┼───────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 5 │ P2P trajectory │ Phase 2 gossip topic structure; libp2p peer discovery via on-chain registry; federation cross-gateway │
│ │ │ state sync; Phase 3 fast-lane economic model; gossip-vs-consensus boundary enforcement │
├─────┼───────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 6 │ Cross-cutting │ New metrics surface (cadence, reorg counters, quorum-collection p99); error selector additions; │
│ │ │ multichain isolation; ECDSA key rotation impact on signed-read; EigenLayer slashing intersect │
├─────┼───────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 7 │ Test surface │ Fork-test reorg-recovery scenarios; P2P partition simulation; bootnode failover harness; multichain │
│ │ │ cadence matrix; Phase 1.5 hot-standby integration test │
└─────┴───────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Format per turn:
- One focused question (sometimes paired if naturally coupled).
- You answer; I push back if the answer leaves a gap or contradicts a locked decision.
- Lock with G-N numbering; brief rationale captured.
- Move to next cluster only when current cluster has no remaining gaps.
Estimated turn count: 60-90 total across all clusters. Front-loaded — clusters 1-3 are dense (foundational invariants), 4-7 lighter
(downstream consequences).
Approve agenda + sequence?
If yes, I'll start with Cluster 1, G-1: gateway authority termination semantics under Phase 3 dissolution — specifically what replaces
the gateway's API-key-to-PolicyClientRegistry-owner mapping when there's no gateway. That binding is load-bearing for newt_signedRead
admission and currently the strongest argument against full dissolution; resolving it first sets the ceiling on Phase 3 ambition.
If you want to reorder (e.g., grill state-tree first since it's the deepest invariant) or skip clusters, redirect now.
✻ Baked for 2m 25s
❯ grill state-tree first. We currently doont have a concrete plan or timeline to deprecate gateway, and will operator with that trust
assumption, but minimize the trust/authority that gateway has.
⏺ ★ Insight ─────────────────────────────────────
"Operate with gateway trust but minimize what we trust it for" is a directional principle, not a decision — it shapes every subsequent
grill answer without prescribing them. Folding it into the spec as a stated design philosophy makes future answers verifiable against
it. Without that explicit anchor, "trust minimization" gets re-litigated in every cluster instead of just being applied.
─────────────────────────────────────────────────
Adding the principle to spec, then starting state-tree grill.
⏺ Update(docs/superpowers/specs/2026-05-07-pds-architecture-pivot-design.md)
⎿  Added 14 lines
47
48 Newton has not deployed to mainnet. There are no live records, no state migration concerns, no operator-fleet upgrade ord
ering. The locked design ships as-is on first mainnet deploy. This removes a large class of compatibility constraints fro
m every downstream decision.
49
50 +# Design philosophy
51 +
52 +Directional principles informing every decision below. Recorded explicitly so future grill answers and implementation cho
+ices remain verifiable against them.
53 +
54 +## P1: Minimize gateway trust
55 +
56 +Gateway is operationally trusted in Phase 1 — there is no concrete plan or timeline to deprecate it. However, the protoco
+l design treats every gateway-asserted fact as something the operator should independently verify when verification is fe
+asible, and explicitly trust when it is not.
57 +
58 +- **When verification is feasible** (operator can derive the same fact from on-chain state, cryptographic primitives, or
+its own local data): operator verifies. Gateway's assertion is taken as a hint, not a source of truth.
59 +- **When verification is infeasible** (the fact is gateway-internal, e.g., API-key authentication state): operator trusts
+. Trust scope is enumerated explicitly in the implementation plan.
60 +- **Conflict resolution**: when this principle conflicts with operational simplicity (e.g., adding an on-chain read per w
+rite doubles latency), the principle wins unless an explicit decision waives it for that path. Waivers are documented; de
+faults are not.
61 +
62 +This principle takes effect immediately in Phase 1 design. Phase 2/3 evolution (federation, eventual potential dissolutio
+n per D7) builds on this foundation but is not what this principle is about — it's about shrinking the gateway's trusted
+surface NOW, regardless of when or whether it dissolves.
63 +
64 # Decisions
65
66 ## D1: EigenDA out of Phase 1, deferred to Phase 1.5+
⏺ Spec amendment applied. Design philosophy P1 (minimize gateway trust) anchored in the spec.
Reordered agenda: state-tree → state-commit consensus → catch-up + bootnode → P2P → authority/admission → cross-cutting → test surface.
★ Insight ─────────────────────────────────────
The state-tree's write path is the most important place to apply P1 — it's where untrusted data becomes structured, authenticated state.
Every read downstream (Rego eval, signed-read, state-commit root) trusts leaves to be well-formed. If garbage gets past the write
boundary, it propagates everywhere; catching at write is far cheaper than retrofitting validation at every read site. Trust minimization
at write = trust budget for everything downstream.
─────────────────────────────────────────────────
Cluster 1 (State-tree), G-1: Trust boundary on gateway-pushed writes
For each fact the gateway asserts on a write request, the operator either VERIFIES (independently derives the truth) or TRUSTS (accepts
gateway's word). The disposition sets the operator's validation surface.
┌─────────────────────────────────────────────────┬──────────────────────────────────────┬──────────────────────────┬──────────────┐
│ Asserted fact │ Verifiable by operator? │ Cost to verify │ Current │
│ │ │ │ default │
├─────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────┼──────────────┤
│ Authenticated client (API key valid, signed │ No — gateway holds auth state │ n/a │ Trusted │
│ envelope) │ │ │ │
├─────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────┼──────────────┤
│ policy_client_owner = client_id │ Yes — read PolicyClientRegistry │ 1 RPC (cacheable) │ Trusts │
│ │ on-chain │ │ │
├─────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────┼──────────────┤
│ Leaf key/namespace inside policy client's │ Yes — derive from policy contract + │ Local computation │ Trusts │
│ authorized scope │ namespace prefix │ │ │
├─────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────┼──────────────┤
│ Ciphertext well-formed HPKE/AEAD to operator's │ Yes — recipient-side decap probe │ ~1ms HPKE decap, no │ Undefined │
│ pubkey (per-op namespaces) │ │ plaintext stored │ │
├─────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────┼──────────────┤
│ epoch_id in LeafHeaderV1 matches current │ Yes — operator already reads │ Local cache │ Should │
│ EpochRegistry epoch │ EpochRegistry │ │ verify │
├─────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────┼──────────────┤
│ created_at monotonic w.r.t. operator's │ Partial — clamp to local clock + │ Local │ Should clamp │
│ last-write │ last-leaf-timestamp │ │ │
├─────────────────────────────────────────────────┼──────────────────────────────────────┼──────────────────────────┼──────────────┤
│ Leaf destined for THIS operator (not misrouted) │ Yes — HPKE recipient pubkey │ Local crypto │ Could verify │
└─────────────────────────────────────────────────┴──────────────────────────────────────┴──────────────────────────┴──────────────┘
Three sub-questions:
G-1a: For each row, does "Current default" align with P1? Specifically — should policy_client_owner and namespace_scope move from TRUSTS
→ VERIFIES in Phase 1?
G-1b: Is the recipient-side decap probe a Phase 1 requirement? Cost is ~1ms HPKE decap per leaf, no plaintext stored, catches
gateway-injected garbage and misrouted leaves in one check. Caveat: only applies to per-operator HPKE namespaces (identity, possibly
confidential). Threshold-encrypted secrets can't be probed by an individual operator.
G-1c: Missing rows you want added? Plausible candidates:
- Ciphertext size bounds (DoS protection)
- Per-policy-client leaf rate cap
- Delete-then-write atomicity (if deletes return in Phase 2)
- Tombstone validity (Phase 2)
- Replay protection — same (leaf_key, epoch_id) written twice with different ciphertext
G-1 cascades into:
- G-2 — write-handler RPC contract shape
- G-3 — validation order (fail fast on cheap checks per lessons.md "Admission gates ordering")
- G-4 — trust delta evolution as Phase 2 federation lands (multiple gateways means each operator-side check applies independently per
source gateway)
Answer G-1a, G-1b, G-1c — reorder if you want a different anchoring question.
✻ Sautéed for 2m 30s
※ recap: Goal: harden PDS architecture pivot (EigenDA out, on-chain deltaContentHash, per-chain cadence, P2P trajectory) before writing
the implementation plan. Current task: grill-me session through every protocol component. Next: you answer G-1a/b/c on state-tree
write-path trust boundary. (disable recaps in /config)
❯ G1a: Operator "Verifies" to minimize trust on gateway as much as possible, but if that "hurts" latency/performance, then we should make
some sacrifices there. Help me evaluate. But operators are trusting that "gateway is not malicious", but gateway stays minimal as an
orchestrator for bls aggregation/consensus as much as possible. (tldr: gateway -> until we have decentralized consensus, centralized
consensus orechestrator that is trusted to be honest but can be verified that gateway is honest, operated Newton DAO/foundation) G1b,c
--> while we do want to operator to verify as much as possible, again, "performance" is one of the most important so in the order
importance: security > reliabilty > performance > verifiability > decentralization (trustlessness)
⏺ Update(docs/superpowers/specs/2026-05-07-pds-architecture-pivot-design.md)
⎿  Added 25 lines
61
62 This principle takes effect immediately in Phase 1 design. Phase 2/3 evolution (federation, eventual potential dissolutio
n per D7) builds on this foundation but is not what this principle is about — it's about shrinking the gateway's trusted
surface NOW, regardless of when or whether it dissolves.
63
64 +## P2: Priority hierarchy
65 +
66 +When trade-offs arise across protocol design choices, prioritize in this order:
67 +
68 +1. **Security** — protocol correctness, cryptographic soundness, defense against malicious actors
69 +2. **Reliability** — uptime, fault tolerance, predictable behavior under load
70 +3. **Performance** — latency, throughput, resource efficiency
71 +4. **Verifiability** — operator-side independent verification of gateway-asserted facts (P1's operational expression)
72 +5. **Decentralization** — trustlessness, permissionless ingress
73 +
74 +Higher-ranked concerns trump lower-ranked when they conflict. Security is never sacrificed for performance. Performance c
+an be sacrificed for reliability. Verifiability (P1) is sacrificed for performance only when the cost is materially signi
+ficant; otherwise verify by default.
75 +
76 +This ordering matches Phase 1's mainnet-readiness posture. As Phase 2/3 land, decentralization rises in priority but neve
+r above security.
77 +
78 +## P3: Gateway trust model
79 +
80 +Gateway is a **centralized consensus orchestrator**, trusted to be honest but verifiable to be honest. Specifically:
81 +
82 +- **Trust assumption**: gateway is non-malicious. It does not forge requests, manipulate state assertions, or selectively
+ censor beyond DoS protection. This trust is operational, backed by Newton DAO/Foundation operation.
83 +- **Verifiable honesty**: every gateway-asserted fact that operators CAN verify (per P1 + P2 cost analysis), they MUST ve
+rify. The trust model is not "trust without checks" — it's "trust + auditability."
84 +- **Minimal scope**: gateway's role is BLS aggregation orchestration, request fanout, and admission control. It is NOT th
+e data plane (D5), NOT the consensus authority (`StateCommitRegistry` is), NOT the source of truth for any state operator
+ can derive elsewhere.
85 +- **Future**: this centralized trust model persists until decentralized consensus orchestration replaces it. Phase 2/3 P2
+P trajectory is the long-term evolution; no concrete timeline.
86 +
87 +Operator-side validation logic targets "honest-but-misconfigured gateway" failure modes (transient incorrectness) over "m
+alicious gateway" attacks (active subversion). The latter is out of scope for Phase 1 protocol design but IS in scope for
+ Phase 1 audit surface — every TRUST decision must be reviewable.
88 +
89 # Decisions
90
91 ## D1: EigenDA out of Phase 1, deferred to Phase 1.5+
⏺ Spec amendments P2 (priority hierarchy) + P3 (gateway trust model) anchored. Now applying them to G-1.
★ Insight ─────────────────────────────────────
The priority hierarchy makes G-1 a cost analysis problem, not a values problem. Security > reliability > performance forces VERIFY by
default. The interesting question is whether any single check crosses a performance threshold worth waiving for. With aggressive
caching, even the most expensive (HPKE decap probe at ~1ms) doesn't impact the state-commit RPC hot path — state-commit reads the cached
JMT root, not raw writes. The two paths are decoupled, so write-side cost stays in the write budget.
─────────────────────────────────────────────────
Cost evaluation per verification (hot path = write-handler):
┌────────────────────────────────────────────────┬──────────────────────┬───────────────────┬──────────────────┐
│ Verification │ Cost │ Write-path impact │ Verdict under P2 │
├────────────────────────────────────────────────┼──────────────────────┼───────────────────┼──────────────────┤
│ policy_client_owner (cached, TTL=10min) │ 1µs warm / 50ms cold │ <0.01% warm │ VERIFY │
├────────────────────────────────────────────────┼──────────────────────┼───────────────────┼──────────────────┤
│ Namespace scope │ 10µs local │ <0.01% │ VERIFY │
├────────────────────────────────────────────────┼──────────────────────┼───────────────────┼──────────────────┤
│ HPKE decap probe (per-op HPKE) │ ~1ms │ dominant cost │ VERIFY │
├────────────────────────────────────────────────┼──────────────────────┼───────────────────┼──────────────────┤
│ Structural validation (threshold ns) │ 10µs │ <0.01% │ VERIFY │
├────────────────────────────────────────────────┼──────────────────────┼───────────────────┼──────────────────┤
│ epoch_id (cached EpochRegistry view) │ 1µs │ <0.01% │ VERIFY │
├────────────────────────────────────────────────┼──────────────────────┼───────────────────┼──────────────────┤
│ created_at clamp │ 1µs │ <0.01% │ CLAMP │
├────────────────────────────────────────────────┼──────────────────────┼───────────────────┼──────────────────┤
│ Size bound check │ 1µs │ <0.01% │ VERIFY │
├────────────────────────────────────────────────┼──────────────────────┼───────────────────┼──────────────────┤
│ Per-policy-client rate counter │ 1µs │ <0.01% │ VERIFY │
├────────────────────────────────────────────────┼──────────────────────┼───────────────────┼──────────────────┤
│ Replay protection ((leaf_key, epoch_id) dedup) │ 5µs │ <0.01% │ VERIFY │
└────────────────────────────────────────────────┴──────────────────────┴───────────────────┴──────────────────┘
Aggregate verified write hot path: ~1.1ms (per-op HPKE namespaces) / ~150µs (threshold namespaces). Single-threaded write throughput
floor: ~900 writes/sec. Acceptable for Phase 1.
State-commit RPC reads the in-memory cached JMT root, so write-side verification cost does NOT cascade into the 1s/4s state-commit
response budget. The two paths are independent.
G-1 lock proposal:
- G-1a: All verifiable rows → VERIFIES. Only "client authenticated" stays TRUSTED (gateway-internal, infeasible to verify per P3).
- G-1b: HPKE decap probe is MANDATORY for per-operator HPKE namespaces (identity at minimum, confidential pending G-3
namespace-by-namespace review). Threshold-encrypted namespaces (privacy/secrets in threshold mode) get structural validation only —
individual operator lacks full share for decap.
- G-1c: Three rows added to validation surface:
- Ciphertext size bound (DoS protection — fail leaves over max_leaf_bytes config, default 1 MiB)
- Per-policy-client leaf rate cap (DoS defense-in-depth — operator-side counter, even though gateway should rate-limit upstream per
P3)
- Replay protection — (leaf_key, epoch_id) dedup catches double-write with different ciphertext, which is a clear
malicious-or-misconfigured gateway signal
Net write-handler validation surface: 9 checks, ~1.1ms warm-cache hot path, throughput floor 900 writes/sec, no cascade into
state-commit RPC budget.
Confirm G-1 lock? On approval I move to G-2: write-handler RPC contract shape — input type schema, validation order (cheapest-first per
lessons.md "Admission gates ordering"), atomicity boundary (per-leaf vs per-batch redb tx), error response variants (one selector per
failure class so off-chain diagnostics are unambiguous).
✻ Baked for 2m 27s
※ recap: Goal: harden the Private Data Storage architecture pivot (EigenDA out, P2P trajectory, minimize gateway trust) before writing the
implementation plan. Current task: grilling Cluster 1 state-tree, awaiting your G-1 lock confirmation so I can move to G-2
write-handler RPC contract. (disable recaps in /config)
❯ sounds good.but operators' rpc endpoints are authenticated by "gateway" right? how should we do this?
⏺ ★ Insight ─────────────────────────────────────
The two-layer auth model is exactly how P1 + P3 work together: P3 says "gateway is honest operationally" which gives the OUTER trust
hop; P1 says "verify what you can" which gives the INNER per-leaf verification. They're not in tension — they cover orthogonal concerns.
Outer layer = cross-service identity (gateway proving it's the gateway); inner layer = content truth (request contents matching
on-chain state). This matches lessons.md "Cross-service trust: single trust boundary per identity check" — operator verifies the
envelope, not per-client identity (that's the gateway's job, and reproducing it at every hop costs complexity for no gain).
─────────────────────────────────────────────────
Good catch — outer envelope auth was implicit in G-1 but never made explicit. Lock as G-1.5: Envelope authentication before G-2 builds
on it.
Existing production pattern. All operator non-DKG RPC endpoints (signed-read, get-state-commit-proposal, sign-state-commit,
simulate-policy variants) are already guarded by Authenticated<T> envelopes via authenticate_gateway_call. Verifies four things:
1. ECDSA signature recovers to an address registered as task-generator on OperatorRegistryEpochGovernance
2. chain_id matches operator's chain context
▐▛███▜▌ Claude Code v2.1.132
▝▜█████▛▘ Opus 4.7 (1M context) with max effort · Claude API
▘▘ ▝▝ ~/projects/newton-prover-avs
✻ Conversation compacted (ctrl+o for history)
⎿  Referenced file docs/superpowers/specs/2026-05-07-pds-architecture-pivot-design.md
⎿  Read ../../.claude/RTK.md (30 lines)
⎿  Read ../../CLAUDE.md (39 lines)
⎿  Read docs/CLAUDE.md (7 lines)
⎿  Skills restored (superpowers:brainstorming)
Searched for 1 pattern, read 1 file (ctrl+o to expand)
⏺ ★ Insight ─────────────────────────────────────
- The Authenticated<T> pattern already exists in crates/chainio/src/operator_rpc_auth.rs for the newt_signedRead endpoint — its design
rationale (chain_id binding, expires_at, ParamsHashMismatch defense) is the canonical reference. Reusing the EXACT same envelope avoids
two implementations of "trust gateway as task generator."
- The "cheapest gates first" admission pattern in lessons.md (admission gates must run cheapest-first) is the lens for understanding
expires_at: it's a sub-microsecond timestamp comparison that filters out stale envelopes BEFORE the ~50µs ECDSA recover and BEFORE the
~5ms state lookup. Without it, an attacker can force expensive crypto on every replayed packet.
- The task-generator set is mutated through the epoch-governance queue (NEWT-1175), so it rotates on epoch boundaries — typically
minutes to hours. expires_at with 120s window guarantees a captured envelope cannot survive across an epoch rotation, defending against
the "former-but-now-revoked signer" attack class.
─────────────────────────────────────────────────
You're right, I had P3 framed as if multi-gateway were aspirational. Let me amend the spec to lock the production reality.
⏺ Update(docs/superpowers/specs/2026-05-07-pds-architecture-pivot-design.md)
⎿  Added 1 line
82 - **Trust assumption**: gateway is non-malicious. It does not forge requests, manipulate state assertions, or selectively
censor beyond DoS protection. This trust is operational, backed by Newton DAO/Foundation operation.
83 - **Verifiable honesty**: every gateway-asserted fact that operators CAN verify (per P1 + P2 cost analysis), they MUST ve
rify. The trust model is not "trust without checks" — it's "trust + auditability."
84 - **Minimal scope**: gateway's role is BLS aggregation orchestration, request fanout, and admission control. It is NOT th
e data plane (D5), NOT the consensus authority (`StateCommitRegistry` is), NOT the source of truth for any state operator
can derive elsewhere.
85 +- **Multiple gateway instances are production reality, not Phase 2.** Stagef runs 1 gateway server. Prod runs 10 gateway
+servers for horizontal scaling, each with a distinct ECDSA signing key but all enrolled into the **single, shared task-ge
+nerator set** on `OperatorRegistryEpochGovernance`. From the operator's verification perspective there is no difference b
+etween "1 gateway" and "N gateways" — admission requires `recovered_signer ∈ task_generator_set` regardless of N. Adding/
+removing gateway instances is therefore an operational concern (provision a new key, queue `addTaskGenerator`, apply at e
+poch boundary, deprovision the old key, queue `removeTaskGenerator`), not a protocol change. Phase 1 ships this directly.
86 - **Future**: this centralized trust model persists until decentralized consensus orchestration replaces it. Phase 2/3 P2
P trajectory is the long-term evolution; no concrete timeline.
87
88 Operator-side validation logic targets "honest-but-misconfigured gateway" failure modes (transient incorrectness) over "m
alicious gateway" attacks (active subversion). The latter is out of scope for Phase 1 protocol design but IS in scope for
Phase 1 audit surface — every TRUST decision must be reviewable.
@denniswon

denniswon commented May 10, 2026

Copy link
Copy Markdown
Author 
⏺ Re-sending the full throughput / cadence answer:

Cycle Duration

There are two distinct cycles running concurrently — this distinction is what motivated G-4-Q1:

┌────────────────────────┬──────────────────┬───────────────────────────────────────┬──────────────────────────────────────────────┐
│ Cycle │ Duration │ Purpose │ What signs │
├────────────────────────┼──────────────────┼───────────────────────────────────────┼──────────────────────────────────────────────┤
│ Layer 1 — Per-task │ ≤1s typical, 2s │ User-facing SLA: returns BLS cert so │ keccak256(canonical_bincode(TxResult)) │
│ attestation │ hard cap │ user can execute intent │ │
├────────────────────────┼──────────────────┼───────────────────────────────────────┼──────────────────────────────────────────────┤
│ Layer 2 — Per-cycle │ ~120s │ On-chain slashing anchor; advances │ keccak256(abi.encode(StateCommit{prev, new, │
│ state-commit │ │ JMT version once │ ...})) │
└────────────────────────┴──────────────────┴───────────────────────────────────────┴──────────────────────────────────────────────┘

The "cycle" in G-3-Q8 lock = Layer 2 (120s). JMT advances version once per 120s, batching all tx WriteBatches between commits. Per-task
attestation does NOT wait for the on-chain anchor.

★ Insight ─────────────────────────────────────
The two-layer model exists because the on-chain commit cost (gas + L1/L2 confirmation) cannot fit a sub-second SLA, but cross-operator
determinism (Invariants 11+14) means every operator computes the same TxResult for the same tx — so per-task BLS aggregation is just
signing already-deterministic bytes. The 120s state-commit anchors the JMT root retrospectively for slashing; the per-task cert is what
the user uses to execute the intent.
─────────────────────────────────────────────────

Transactions Per Cycle (120s)

Per-operator throughput × 120s = transactions per cycle. The actual ceiling depends on dominant eval cost:

┌────────────────────────────────────────────────┬────────────────────────┬────────────────┐
│ Eval cost (per tx) │ Per-operator sustained │ Per 120s cycle │
├────────────────────────────────────────────────┼────────────────────────┼────────────────┤
│ 10ms (simple Rego policies) │ 500–1,000 tx/sec │ 60,000–120,000 │
├────────────────────────────────────────────────┼────────────────────────┼────────────────┤
│ 50ms (complex policies, multiple data.* reads) │ 200–400 tx/sec │ 24,000–48,000 │
├────────────────────────────────────────────────┼────────────────────────┼────────────────┤
│ 200ms (heavy crypto / fanout) │ 50–100 tx/sec │ 6,000–12,000 │
└────────────────────────────────────────────────┴────────────────────────┴────────────────┘

Bottlenecks beyond raw eval:

  • Aggregator BLS aggregation rate: ~low-thousands partial sigs/sec per aggregator → not the bottleneck for ≤1k tx/sec
  • redb I/O: one batch_write per 120s cycle (G-3-Q8 lock) amortizes to negligible per-tx
  • HPKE encryption: ~50-100µs per leaf payload (libsodium tuned), parallelizable per-tx
  • Network round-trip operator↔aggregator: ~10-50ms typical, dominates BLS sign time

Maximum Throughput

┌────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Layer │ Bound │
├────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Per-operator │ 500-1,000 tx/sec (10ms eval, 16-core node, parallelizable Phase 2) │
├────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Cluster │ min(per-operator-rates) — every operator signs every task, so throughput is the slowest │
│ │ operator, not the sum │
├────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ BLS aggregation │ Aggregator handles thousands of partials/sec → not the cluster bottleneck │
├────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Per-cycle (120s) absolute │ ~120,000 tx at 1k tx/sec sustained │
│ ceiling │ │
└────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘

Sub-Second SLA Path (Happy Path)

End-to-end latency budget for a single task evaluation request:

Gateway receives request t = 0
↓ Forward to operator(s)
Operator runs Phase 1 (PutLeafs into overlay) t ≈ 1-5ms
↓
Operator runs Phase 2 (Rego eval, EmitLeafs) t ≈ 10-50ms (eval cost)
↓
Operator computes TxResult, BLS partial-sign t ≈ +1ms
↓ Send to aggregator
Aggregator collects quorum partial sigs t ≈ +50-200ms (network)
↓
Aggregator BLS-aggregates t ≈ +5ms
↓ Return to gateway
Gateway returns cert to user t ≈ +10-50ms
────────────
≈ 80-310ms typical

2s hard cap accounts for: slow operators (tail latency), network jitter, RPC retries, aggregator quorum-wait timeout. If quorum doesn't
reach in 2s, gateway returns timeout error — task can be retried or deadlined out.

Why This Lock Made G-4-Q1 Necessary

The spec already had per-cycle BLS state-commit nailed down (Hard Invariants 19, 20). It did NOT explicitly nail per-task attestation
under the new D9 transaction model. Sub-second SLA forces:

  1. Per-task BLS aggregation must NOT wait for cycle-end (would push latency to 120s)
  2. Digest must be deterministic across operators (Invariants 11, 14) so partial sigs aggregate
  3. TxResult shape must be locked so canonical encoding is unambiguous

That's the three sub-decisions G-4-Q1 wraps. Now — back to your clarification: which axis of G-4-Q1 do you want to probe?

✻ Baked for 1m 20s

❯ help me understand this: while all other operations other policy evaluation task is "fast", policy evaluation is intrinsically slow:
it runs rego policy (inside TEE wasm if private data is involved + two-phase consensus for threshold decryption) / potentially also
wasm for policy data generation (inside TEE enclave if secrets are involved). This seems to be the fundamental bottleneck of low
latency, high throughput policy evaluation. The problems I see: 1) putting putleafs and evalauteandemit in the same path/transaction or
cycle seems we should more carefully evaluate this design. maybe we should separate policy data generation (wasm execution) and rego
evaluation as two separate operations that are batched/linked? unless is there a way to remove "two-phase" consensus, which was
introduced because policy data could produce numeric data and there could be small discrepancies across operations' wasm execution
results (rounding errors, etc.), by making all wasm executions to be deterministic. or maybe we can remove wasm entirely if there is a
way for rego policies to get supplied with outside external data for policy evaluation (which is the wasm's purpose in the first place,
to run some arbitrary process/code to generate some data with flexibility for policy evaluation) 2. what if the wasm execution for
policy data generation itself takes a long time and that slows down the entire transaction increasing the max(tx latency) and slows
down the process? /clarify-plan

⏺ ★ Insight ─────────────────────────────────────
You've identified the cleanest structural tension in the current design: the two-phase per-tx model (D9) treats state mutation and
policy evaluation as a single atomic unit, but they have fundamentally different latency characteristics. State mutation is fast (memory
write + redb batch), but policy evaluation can be 10ms–10s+ depending on WASM cost, threshold decryption rounds, and Rego complexity.
Coupling them forces the slow path to dominate cycle scheduling and per-tx attestation budgets.
─────────────────────────────────────────────────

Let me walk through your three concerns in order, because they're each correct in different ways.

Where the Real Coupling Lives

The current D9 model couples PutLeaf + EvaluateAndEmit at the per-tx level, but parallel-snapshot Phase 2 means slow tx don't block fast
tx. The real coupling is more subtle:

┌─────────────────────────────────┬─────────────────────────┬──────────────────────────────────────────────────────────────────────┐
│ Concern │ Coupled today? │ Why │
├─────────────────────────────────┼─────────────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Slow tx blocks fast tx │ No │ Phase 2 parallel-snapshot — each tx evaluates independently against │
│ │ │ (JMT @ root_k, own-tx overlay) │
├─────────────────────────────────┼─────────────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Slow tx blocks cycle-end commit │ Maybe — depends on │ If cycle-end waits for all in-flight tx, yes; if cycle-end is fixed │
│ │ cycle-end semantics │ wall-clock and defers in-flight to next cycle, no │
├─────────────────────────────────┼─────────────────────────┼──────────────────────────────────────────────────────────────────────┤
│ Slow tx blocks own attestation │ Yes │ Per-task TxResult digest only signs after Phase 2 completes — so │
│ │ │ slow Phase 2 = slow attestation │
├─────────────────────────────────┼─────────────────────────┼──────────────────────────────────────────────────────────────────────┤
│ WASM non-determinism forces │ Yes (today) │ Median/modal aggregation in Prepare/Commit handles operator output │
│ two-phase consensus │ │ drift │
└─────────────────────────────────┴─────────────────────────┴──────────────────────────────────────────────────────────────────────┘

Your point #1 is asking whether the per-tx coupling is the right model when most txs aren't mixed (PutLeaf vs Eval are usually
disjoint). Your point #2 is asking whether tail latency on a single slow WASM gates the whole thing.

Concern 1.a — Separating Policy Data Generation from Rego Evaluation

The proposal: split policy_data_gen(WASM) → rego_eval(Rego) into two separate ops that are batched/linked, rather than fused into one
tx's Phase 2.

This already half-exists in Newton via the policy_data registry pattern:

[once per policy_data refresh] [per task evaluation]
WASM(raw_inputs) → policy_data Rego(intent, policy_data, identity, ...)
Sign + register policy_data at version V Read policy_data @ version V
Anchor to JMT (immutable until next refresh) Run eval
Emit attestation
↑ Slow path, runs offline ↑ Fast path, runs per-task

Why this works: WASM execution is moved out of the hot eval path. All operators read the same anchored policy_data deterministically.
Rego itself is fast (10-50ms typical) because it's pure logic over already-fetched data.

The gap right now: the spec doesn't explicitly forbid inline WASM in Phase 2. If a policy's WASM runs at task-eval time (not
pre-cached), you get exactly the latency problem you're describing.

★ Insight ─────────────────────────────────────
The cleanest version of "separate WASM from Rego" is to disallow inline WASM in Phase 2 entirely for the task namespace (0x04). WASM
only runs in policy_data registration paths, which are NOT in the per-task hot path. Per-task eval becomes pure Rego over pre-anchored
state — sub-second by construction. This is the architectural decision that needs explicit locking.
─────────────────────────────────────────────────

Concern 1.b — Making WASM Deterministic to Drop Two-Phase Consensus

You're right that two-phase consensus exists primarily because WASM outputs can drift across operators. The drift sources:

┌───────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Source │ Deterministic if... │
├───────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Floating-point │ WASM spec mandates IEEE 754 — already deterministic if all operators use same wasmtime version │
├───────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Time / Random │ Disallow clock_time_get / random_get from imports — pure compute │
├───────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Network I/O │ Disallow fd_read / sockets — no ambient I/O │
├───────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ HTTP fetch │ Replicate-fetch (each operator fetches independently and votes) OR pre-attest (one fetch, signed feed, all │
│ results │ operators read) │
└───────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

If we constrain WASM to strict deterministic mode (no syscalls, no I/O, no clock, no random, pinned wasmtime version), then for the same
input bytes every operator produces the same output bytes. Two-phase median aggregation becomes redundant.

Proposed lock: All WASM in PDS Phase 1 must be deterministic. Outputs must be bytewise-identical across operators. One-phase BLS
aggregation suffices. Two-phase consensus retired for the new PDS path (orthogonal to existing AVS task lifecycle which can keep its
semantics).

Concern 1.c — Removing WASM Entirely?

WASM serves two distinct purposes — keep one, retire one:

┌─────────────────────────────────────┬────────────────┬───────────────────────────────────────────────────────────────────────────┐
│ Purpose │ Keep / retire? │ Why │
├─────────────────────────────────────┼────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ Pre-compute policy_data from raw │ Keep │ Rego can't do HTTP fetches, signed feed parsing, or cryptographic │
│ external inputs │ │ processing efficiently │
├─────────────────────────────────────┼────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ Inline computation during task │ Retire for hot │ Forces operators to run identical untrusted code at request time; any │
│ evaluation │ path │ drift = slashing risk; high latency │
└─────────────────────────────────────┴────────────────┴───────────────────────────────────────────────────────────────────────────┘

Rego is intentionally restricted (no recursion, no I/O) — that's its safety property. Retrying to push everything into Rego loses you
that. So WASM stays, but its execution boundary moves out of the per-task eval path.

Concern 2 — Tail Latency on Slow WASM

Even with all the above, a single policy's WASM might still take a long time when it runs (just not on the hot path). What about that?

Slow path (offline, batched per policy_data refresh):
WASM execution can be 1s, 10s, even minutes
Doesn't block any task eval
Anchored result becomes available at version V
All subsequent task evals at version ≥ V use it deterministically

Fast path (online, per-task):
Pure Rego eval against pre-anchored data
Bounded by Rego policy complexity (typically 10-50ms)
EvalTimeout (TxError variant) caps at ~500ms operator-side
Per-task attestation budget: 2s end-to-end (G-4-Q1 proposed lock)

The 2s hard cap belongs to the fast path only. Slow WASM lives in the slow path and never gates user-facing latency.

If a policy's WASM is so slow that policy_data refresh can't keep up with the desired refresh frequency — that's a policy-author
problem, not a protocol problem. The protocol guarantees: whatever policy_data is anchored at version V, all operators agree on it
deterministically.

Recommended Refined Model

Consolidating: I'd propose a clean three-axis decoupling, possibly worth opening G-5 grill series:

┌─────────────────┬──────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────┐
│ Axis │ Current (D9) │ Proposed │
├─────────────────┼──────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┤
│ State mutation │ Per-tx Phase 1, batched per │ Unchanged — per-tx Phase 1, cycle-batched │
│ cadence │ cycle (120s) │ │
├─────────────────┼──────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┤
│ Policy data │ Implicitly inline in Phase 2 │ Decoupled — runs offline in policy_data registration paths; pre-anchored to JMT │
│ generation │ WASM │ │
├─────────────────┼──────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┤
│ Per-task │ Phase 2 = WASM + threshold │ Pure Rego — reads pre-anchored policy_data + identity/confidential/privacy │
│ evaluation │ decrypt + Rego + emit │ state; emits attestation │
├─────────────────┼──────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┤
│ Consensus model │ Two-phase Prepare/Commit │ One-phase — deterministic WASM + deterministic Rego = bytewise-identical │
│ │ with median │ TxResult across operators │
├─────────────────┼──────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┤
│ Threshold │ Two-phase consensus during │ Inline one-pass — partial DH gather happens during Phase 2 but doesn't need │
│ decryption │ eval │ median aggregation since each operator's partial is provably correct via DLEQ │
└─────────────────┴──────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────┘

★ Insight ─────────────────────────────────────
The key reframing: policy evaluation is read-mostly compute, not state mutation. The D9 two-phase per-tx model was designed for atomic
state transitions, but task evaluation isn't really that — it reads state, evaluates, emits an attestation that stores in JMT only as
audit trail. Treating eval as a separate operation class (not "another tx with Phase 1 = empty") simplifies the latency story
dramatically.
─────────────────────────────────────────────────

What I Need You to Clarify Next

Before I propose this as cascading edits to the spec (Hard Invariants, Open Questions G-5-Q*, Decision History), I need to know which of
these resonate vs which you want to push back on:

  1. Drop inline WASM in per-task Phase 2 — only pre-anchored policy_data is readable in eval. Agree?
  2. Mandate deterministic WASM for all PDS paths (no clock, no random, no network I/O, pinned wasmtime version). Agree?
  3. Drop two-phase consensus for PDS task evaluation in favor of one-phase BLS aggregation over deterministic TxResult. Agree?
  4. Keep threshold decryption as a Phase 2 sub-step but treat partial DH outputs as deterministic (DLEQ-verified) so no median
    aggregation. Agree?
  5. Treat task evaluation as a distinct operation class from JMT mutation (different latency budgets, different consensus shapes). Agree?

Which of these (or which combination) do you want to dig into, push back on, or refine?

✻ Churned for 2m 35s

❯ no one of the core value props of newton protocol is the flexibilty that wasm provides where developers can use arbitrary code logic
and data, including making network I/O so that those offchain process/compute/data could be used for policy evaluation for onchain
intents. But essentially, what I do like is this potential approach of separating data generation with policy evaluation. The intuition
comes from what Accountable is doing https://docs.accountable.capital/accountable-documentation/vault-as-a-service
https://docs.accountable.capital/accountable-documentation/data-verification-network-dvn/nodes . this connector approach sounds
interesting https://docs.accountable.capital/accountable-documentation/data-verification-network-dvn/connectors. Is there anything from
Accountable design/approach that we can get insights from? (we already have TEE integration where wasm and rego are executed for
private data involved or secrets involved cases)

⏺ ★ Insight ─────────────────────────────────────
You're right — I overcorrected. Network I/O flexibility through WASM is exactly what makes Newton policies more powerful than pure
on-chain rule engines. Killing inline WASM would kill that. But Accountable's design points to a way to keep full WASM flexibility AND
get sub-second eval: separate the data layer (connectors with network I/O, possibly slow, possibly multi-phase consensus) from the
evaluation layer (pure read + Rego, sub-second, one-phase). The flexibility lives in connectors, not in per-task hot paths.
─────────────────────────────────────────────────

Let me walk through what's transferable from Accountable's design.

Accountable's Relevant Primitives

┌────────────────────┬────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────┐
│ Accountable │ What it does │ Insight for Newton │
│ concept │ │ │
├────────────────────┼────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────┤
│ Vault-as-a-Service │ Programmable vault that executes strategies under │ Newton's policy contracts already serve this role. Less │
│ │ verifiable constraints │ novel transfer here. │
├────────────────────┼────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────┤
│ DVN Nodes │ Network of operators that fetch + verify off-chain │ Map to Newton operators, but with a separate consensus │
│ │ data, reach consensus │ lane for connector outputs (not policy eval) │
├────────────────────┼────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────┤
│ Connectors │ Pluggable adapters with their own input/output │ The core insight — promote WASM data fetchers to │
│ │ schemas, refresh cadence, attestation lifecycle │ first-class network primitives, not per-policy concerns │
├────────────────────┼────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────┤
│ Verified data │ Connector outputs attested + made available to │ Map to JMT-anchored connector output leaves, read at │
│ feeds │ consumers │ policy eval time │
└────────────────────┴────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────┘

The conceptual shift: in Accountable, data fetching is a network service with its own consensus and refresh schedule. Vaults consume
already-verified data. Newton currently has WASM as a per-policy implementation detail — same code might run inside a TEE during a
privacy task, or on the host for a public task, with no formal abstraction binding the data lifecycle.

What This Lets Us Do Without Sacrificing Flexibility

The Accountable-inspired refinement keeps WASM's network I/O fully intact, but moves where it runs:

┌───────────────┬──────────────────────────────────────┬────────────┬──────────────────────────┬──────────────────────────────────┐
│ │ │ Network │ │ │
│ Layer │ What runs │ I/O │ Cadence │ Consensus shape │
│ │ │ allowed? │ │ │
├───────────────┼──────────────────────────────────────┼────────────┼──────────────────────────┼──────────────────────────────────┤
│ Connector │ WASM connectors — full flexibility │ │ Per-connector schedule │ Two-phase with median/modal — │
│ layer │ (HTTP, oracle feeds, signed APIs, │ Yes — full │ (e.g., 100ms, 1s, 10s, │ handles natural drift in fetched │
│ │ complex preprocessing) │ │ 60s, on-demand) │ data │
├───────────────┼──────────────────────────────────────┼────────────┼──────────────────────────┼──────────────────────────────────┤
│ Privacy │ WASM connectors handling │ Yes — but │ │ Two-phase + TEE attestation │
│ connector │ secrets/confidential data │ inside TEE │ Same as above │ (existing pattern) │
│ layer │ │ │ │ │
├───────────────┼──────────────────────────────────────┼────────────┼──────────────────────────┼──────────────────────────────────┤
│ Evaluation │ Pure Rego over connector outputs + │ No — pure │ │ One-phase BLS — │
│ layer │ identity + confidential + privacy │ compute │ Per-task (sub-second) │ bytewise-deterministic Rego │
│ │ state │ │ │ │
└───────────────┴──────────────────────────────────────┴────────────┴──────────────────────────┴──────────────────────────────────┘

★ Insight ─────────────────────────────────────
The key architectural shift is connectors are anchored, policies are reactive. A policy doesn't fetch data anymore — it consumes the
latest anchored connector outputs at eval time. The connector network refreshes data continuously at whatever rate the connector spec
demands; policies read whatever's current. This is the same model Chainlink uses (price oracles refresh on their own cadence; consumers
read the latest), but generalized to arbitrary WASM-based data sources.
─────────────────────────────────────────────────

Concrete Mapping to Newton's Architecture

Here's how Accountable's primitives translate into Newton's existing PDS components:

                  Slow path (continuous, scheduled)
                  ─────────────────────────────────
  [External Source]         (HTTP API, oracle, RPC, signed feed)
         │
         ▼
  [WASM Connector]          (registered in ConnectorRegistry, versioned)
         │
         ▼  Two-phase consensus across operators
  [Median/Modal Output]     (handles natural network drift)
         │
         ▼
  [JMT-anchored Leaf]       (namespace 0x06: ConnectorOutput, leaf_key = connector_id)
         │
         │  Available to all policy evaluations
         ▼
                  Fast path (per-task, sub-second)
                  ────────────────────────────────
  [User Intent Request]
         │
         ▼
  [Operator reads]          (identity + confidential + privacy + connector outputs from JMT)
         │
         ▼
  [Pure Rego eval]          (deterministic, ~10-50ms)
         │
         ▼  One-phase BLS aggregation (deterministic TxResult bytes)
  [Per-task attestation]    (returned to user, sub-second)

This preserves everything you care about:

  • Full WASM flexibility — any code, any I/O — lives in connectors
  • TEE-isolated WASM for secrets — connectors that fetch secrets run inside the TEE (existing infrastructure reused)
  • Threshold decryption for privacy — happens during Phase 2 eval but is deterministic (DLEQ-verified partials) so doesn't need median
    aggregation
  • Newton's value prop — "policies can consume arbitrary off-chain data" — fully intact
  • Sub-second SLA — eval layer is pure Rego, deterministic, one-phase

Specific Insights from Accountable's Connector Model

Three patterns from Accountable that directly translate:

  1. Connectors are versioned + registered

Every connector has a unique ID, a code hash (WASM bytes), an input/output schema, a refresh cadence. The ConnectorRegistry (analogous
to existing PolicyClientRegistry) tracks them. Operators sync connector code on registration. New connectors are added via governance /
policy_client registration flow.

This eliminates the current ambiguity where each policy might have its own bespoke WASM. Connectors become reusable across policies.

  1. Connector outputs are anchored separately from policy state

Newton already has multiple namespace prefixes. Adding 0x06: ConnectorOutput (or extending 0x03 policy_data semantics) gives connector
outputs their own JMT subtree. Connector layer writes here on its schedule; eval layer reads here at task time. This isolates connector
consensus drama from policy eval.

  1. Connectors can have configurable consensus thresholds

Some connectors fetch from a single canonical source (e.g., a signed Chainlink feed) — for these, deterministic re-fetch + bytewise
comparison works.
Others aggregate from multiple sources (e.g., median of 5 price APIs) — for these, two-phase with median is appropriate.

The connector spec declares its consensus model. The DVN-equivalent layer (Newton operators) runs the appropriate aggregation. This is
more flexible than the current implicit "everything goes through two-phase consensus" model.

What Doesn't Transfer

A few things from Accountable that we should NOT copy:

┌──────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────┐
│ Their pattern │ Why we shouldn't copy │
├──────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
│ Vaults as protocol-level abstraction │ Newton's policy contracts already serve this; vaults are application-layer in Newton │
├──────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
│ Their specific staking economics │ EigenLayer restaking is our staking primitive; their model is different │
├──────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────┤
│ DVN as a separate network │ Newton's same operator set should run both layers (avoids splitting security budget) │
└──────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────┘

Open Questions This Reframing Surfaces

If we adopt the connector + evaluation two-layer model, several decisions need locking:

  1. Connector cadence model — fixed schedule (every N seconds), pull-based (refresh when stale), or hybrid?
  2. Connector output staleness — how do policies declare "I need data fresher than X"? Reject eval if connector output is too old?
  3. Connector failure handling — if a connector can't reach consensus (e.g., upstream API down), do dependent policies fail gracefully or
    block?
  4. Connector composition — can a connector consume another connector's output? Or only external data?
  5. TEE-vs-host connector dispatch — same code, different execution strategy based on connector spec? (Newton already has
    EnclaveWasmStrategy — this maps.)
  6. Connector slashing — if a connector operator emits a wrong output, what's the slashing condition? Same as policy slashing or
    different?
  7. Migration path — existing per-policy WASM patterns need to convert to connectors. What's the boundary?

★ Insight ─────────────────────────────────────
Q4 (connector composition) is interesting because it could enable layered abstractions: a "ETH-USD price" connector can consume a
"BTC-USD price" connector + a "ETH-BTC price" connector. This composition lets developers build sophisticated data pipelines without
writing one giant connector. It's also where you can get into trouble — composition cycles, refresh-cascade timing, version-skew issues.
Likely needs to be locked carefully or deferred to Phase 2.
─────────────────────────────────────────────────

My Recommendation

This is significant enough to warrant a new grill series — call it G-5: Connector / Evaluation Separation — alongside (not replacing)
G-4. The cascading edits would touch:

  • New Hard Invariants (likely 21-24 range)
  • Possibly a new namespace prefix in NamespacePrefix enum (e.g., 0x06 ConnectorOutput)
  • New ConnectorRegistry contract spec (parallel to PolicyClientRegistry)
  • Refined evaluation flow under D9 — Phase 2 reads connector outputs but doesn't run WASM
  • Decision History rows
  • Possibly a new architecture section in the spec doc

Before I draft G-5 questions formally, three things I want to confirm:

  1. Direction confirmed? — Is the "connector layer + evaluation layer" two-tier model the direction you want to lock, with WASM
    flexibility moved to the connector layer?
  2. Naming — "Connector" (Accountable's term) vs "DataFeed" vs "Oracle" vs "PolicyDataProvider" (extending existing Newton vocab)? I'd
    lean toward "Connector" for clarity, but it's bikesheddable.
  3. Scope for Phase 1 MVP — should connector model be in scope for the first PDS migration, or deferred to a Phase 2 (and Phase 1 keeps
    the current ad-hoc per-policy WASM but with the eval-vs-mutation separation we're proposing)?

Which of those should we settle first?

✻ Brewed for 1m 52s

@denniswon

denniswon commented May 14, 2026

Copy link
Copy Markdown
Author

◼ Cascade G-6 locks into spec doc
for operators to process every block (around 1 second block time). No atomocity at the block level. Currently we dont have fee
mechanism so it is first come first served in linear order for operations, but ONLY for put leaf operations. TLDR: put leaf operations
(deploy policy, register policy client to a policy, user upload identity, grant authorization to a poicy client for identity usage,
etc.) should ALL be linearly ordered like blockchain transactions from the gateway sequencer mempool (should we have ordering per
user/client/sender nonce, like ethereum EOA nonce? For atomicity and guaranteed ordering, the responsibility can be moved to the
operation/task request sender to specifically "batch" operations. Every "block" with 1 second block time, operations are sequentially
ordered for each policy client/operation sender account nonce for putleaf operations only. ex: if the client wants to ensure that they
want to have their put leaf operation of register policy client to a policy before the task evaluation request, they need to set the
nonce correctly to ensure that order. If they want to batch it for atomocity for all or nothing experience, they need to specifically
batch them into a batched operation.) Then, from the mempool, the gateway continuously "sequences" operations (in phase 1, just first
come first served, sequential sequencing) and constructs a block. And each block, at the beginning of the block, each operator take a
snapshot of the state, and execute in order one by one to the snapshot, 1 batch state write at the end of the block. Some operations
can succeed, some operations fail. No atomocity in the block. Another important thing is that in phase 1 before the fee market, ONLY
Put leaf "operations" are sequentially ordered like this in blocks. All policy evaluation "Tasks" or all policy data ingestion "Tasks"
are executed in "parallel" (important) separately. we should have 3 separate block pipeline that runs in parallel where 1: first
pipeline for all put leaf operations, in operation linear sequential processing (again should we have a concept of account nonce for
put leaf operations only? not sure if we need this), 2: policy evaluation task block pipeline, no account nonce involved (account nonce
only relevant for putleaf operations if we need), all tasks executed in parallel, takes a state snapshot at the beginning of the
block, 1 batch write at the end. Could involved threshold decryption with 2 phase consensus for certain operations in the block. No
atomocity in the block. 3: Similar to 2, a separate policy data ingestion task block pipeline where all tasks are executed in parallel.
No atomocity in the block. Does encryption for policy data feed if the policy data template is a private data. Having such 3 separate
pipelines of blocks, this means sender can ONLY batch and guarantee specific order in a block for put leaf operations, not tasks.
(Conceptually, operations --> customer writes, Tasks --> operators executes per request/cadence). This structure is for maximal,
optimal latency for policy evaluation and policy data ingestion tasks, and separation of failure points in the protocol. #Point 2#
operators should not operate under the assumption that other operators are honest --> meaning BLS aggregation/offchain finality after
consensus is required for operators to finalize the state transition, just like ethereum block. Before we have bft consensus with
operator p2p, operators need someone/some source to tell them what the consensus-achieved/bls aggregate signed state is. Because state
commit happens ONLY every cycle not every block, after each block, Gateway can "broadcast" the finalized state root is for each block.
If operators see discrepancy, they need to "FIRST" reconcile the discrepancy by getting the finalized block delta state from the
bootnode (in later phase, from other operators once we have p2p layer). TLDR: before bfs consensus with p2p layer, operators do need to
trust gateway and bootnode operated by newton. If no discrepancy, they just finalize to actually update the state db (before this
block confirmation from gateway, not actually written to db yet). #Point 3# Let's actually remove active_binding mapping and do not
strictly enforce validateAttestationDirect to check the policy client policy binding (only optionally). Optimizing for user experience
and performance/speed. #Point 4# I have some thoughts on attestation being spent flow. Let's do a grill session on this. High level, I
want to introduce the concept of "Relayers" and improve the developer/user experience to the contract "callback" pattern while the
current only supported pattern is users/clients getting back the attestation and directly spending it. (They still can but this
callback pattern support is an improvement) The concept is similar to how Chainlink oracle works. INewtonPolicyClients contracts of
developers can implement the callback function where the "relayers" will relay the attestation on the behalf of the users (the
attestation already contains the exact transaction intent (from, to, data, value)). This is an improvement in developer experience
because for example: let's say the developer want to guard its mint() function on the contract. Currently, they have to change their
existing mint(some params...) function scheme to mint(NewtonAttestation) function scheme. But with the callback approach, they can
leave mint() function scheme as is, and just call requestPolicyEvaluation() inside the mint() function, then just implement an
additional separate callback function of INewtonPolicyClient for "relayers" to call (function evaluationCallback(NewtonAttestation))
and that callback just then executes the intent contained inside the attestation.

⏺ Four major design changes — three architectural restructures (Points 1, 2, 3) and one new feature with grill request (Point 4). Let me
work through cascade impact for each, then kick off the Point 4 grill.

★ Insight ─────────────────────────────────────
Points 1+2 together resolve a tension I'd been quietly carrying: the current spec has the gateway as "admission boundary, not
correctness path" (D5/D14), but the unified Transaction model (D9) implicitly required gateway-blind sequencing — which is incoherent
before P2P+BFT. Reframing as gateway-as-trusted-sequencer with operator-side verification + bootnode reconciliation is honest about
Phase 1's actual trust posture (Newton operates both gateway and bootnode → effective 1-of-1 trust until Phase 2 federation lands).
─────────────────────────────────────────────────

Point 1: Three-pipeline architecture (block-level, not tx-level)

What this invalidates in the current spec:

Spec section: D9 unified Transaction model
Impact: Removed. Replaced by 3 separate pipelines
────────────────────────────────────────
Spec section: G-3-Q3 (two-phase per-tx execution: Phase 1 PutLeafs → Phase 2 Evals)
Impact: Removed. Cross-op-class composition within one unit no longer exists
────────────────────────────────────────
Spec section: G-3-Q8 (per-cycle JMT versioning + cycle-end commit pipeline)
Impact: Restructured. Each pipeline has its own block-end snapshot+writeback
────────────────────────────────────────
Spec section: Invariant 14 (two-phase per-tx execution)
Impact: Removed
────────────────────────────────────────
Spec section: Invariant 9 ("Transactions are atomic; cycles are not")
Impact: Restructured. Atomicity moves to "batched operations within Pipeline 1 only"
────────────────────────────────────────
Spec section: G-3-Q1 content-hash tx_id
Impact: Repurposes as "operation_id" or "batch_id" depending on whether ordering or atomicity is the primary use
────────────────────────────────────────
Spec section: G-2-Q5 op-exclusivity invariant
Impact: Strengthened. Op-class doesn't just bind namespace — it binds pipeline

New conceptual split — operations vs tasks:

Pipeline: 1: PutLeaf
Unit: Operation (or Batch[Operation] for atomicity)
Trigger: Client-submitted via gateway mempool
Ordering: Linear (sender-nonce-ordered?)
Atomicity: Batch-scoped if explicitly batched
Cadence: ~1s block
────────────────────────────────────────
Pipeline: 2: Policy evaluation
Unit: Task
Trigger: Per-client-request
Ordering: Parallel within block
Atomicity: None
Cadence: ~1s block, possibly 2-phase for private inputs
────────────────────────────────────────
Pipeline: 3: PolicyData ingestion
Unit: Task
Trigger: Per-cadence IngestionTick
Ordering: Parallel within block
Atomicity: None
Cadence: ~1s block, encrypts for private feeds

The nonce question — my recommendation:

★ Insight ─────────────────────────────────────
The right answer depends on whether the SDK or the protocol owns ordering responsibility. Without nonces, "first-come-first-served at
gateway" is non-deterministic from the sender's perspective — network jitter between two PutLeaf submissions can flip their order. That
breaks the canonical use case "register policy client to a policy BEFORE uploading identity that references it."
─────────────────────────────────────────────────

Yes to nonces for Pipeline 1, modeled on Ethereum EOA nonces:

  • Tuple: (sender_owner, chain_id, nonce: u64). Per-(sender, chain) nonce space.
  • Gateway tracks last_committed_nonce per (sender_owner, chain_id) in the existing Postgres journal (Invariant 36 — same table, new
    column).
  • Submission rule: nonce > last_committed_nonce. Gap-fills queue up to a bound (say 32 pending); past that, gateway rejects with
    NonceGapTooWide.
  • Mempool ordering: ascending nonce per sender, FCFS across senders.
  • Replay protection comes for free (same (sender, nonce) cannot land twice).
  • SDK auto-increments; pre-fetches last_committed_nonce on session start.
  • Batching: Batch[Operation] consumes ONE nonce slot and is atomic (all-or-nothing within Pipeline 1's block).

Why not "first-come-first-served + opt-in batching only":

  • Bare FCFS gives no ordering guarantee for a single sender across blocks. SDK would need a separate request-ordering protocol layer on
    top — re-inventing nonces.
  • The mental model "Pipeline 1 is Ethereum-tx-like" is load-bearing for developer onboarding. Nonces are the part of EOA semantics that
    developers already know.
  • Gateway-side nonce tracking is one column in the existing replication journal — zero new infrastructure surface.

Trade-off accepted: Phase 1 single-gateway means nonces are gateway-scoped, not chain-scoped. If gateway state is lost without a
graceful failover, in-flight nonces stall. The Phase 1.5 hot-standby (D4) inherits the nonce ledger from replicated Postgres —
recoverable. Phase 2 federated gateways need a cross-gateway nonce reservation protocol (likely OPTIMISTIC reservation with
conflict-resolution at submission). Defer Phase 2 mechanism design; Phase 1 is bottleneck-free.

Cascade — what needs to be written/rewritten:

  • New D-decision block: "D17: Three-pipeline architecture"
  • Rewrite D9 entirely
  • New invariant block replacing Invariants 9, 14
  • Drop G-3-Q3, G-3-Q8 lock entries; mark superseded
  • New G-7 round to lock pipeline boundary semantics (open questions: cross-pipeline read snapshots, cross-pipeline failure isolation,
    cross-pipeline observability)

Point 2: Block vs cycle, gateway-broadcast finalization, bootnode reconciliation

Terminology lock proposal:

  • Block (~1s): unit of operation/task processing per pipeline. Each pipeline produces one block per ~1s tick.
  • Cycle (≥1 block, network-dependent): unit of on-chain state commit. Cycles aggregate K blocks. K is network-class config: mainnet K=60
    (60s cadence per D3), testnet K=30, local K=2.

New finalization flow:

t=0 Block N starts. Each pipeline snapshots local JMT @ root_k.
t=0+ Pipeline 1: operations sequential into block N's writeset.
Pipeline 2: tasks parallel against snapshot.
Pipeline 3: tasks parallel against snapshot.
t=1s Block N ends. Each pipeline computes its block-end root.
t=1s+ Gateway aggregates per-pipeline roots → broadcasts canonical (root_1, root_2, root_3)_block_N.
t=1s+ Operators independently compute their own per-pipeline roots.
MATCH: operator writes block N to local redb (finalized).
MISMATCH: operator stops, fetches block N delta from bootnode,
reconciles, then finalizes.
t=K×1s Cycle end. BLS quorum signs StateCommit over the cycle's aggregated root.
Aggregator submits commitStateRoot on-chain.

Trust topology:

┌────────────┬───────────────────────────────────┬──────────────────────────────────────────────────────────┐
│ Actor │ Role │ Trust model │
├────────────┼───────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ Gateway │ Sequencer + finalizer broadcaster │ Trusted to sequence and broadcast canonical block deltas │
├────────────┼───────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ Bootnode │ Tiebreaker delta source │ Trusted as canonical reconciliation source │
├────────────┼───────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ Operators │ Verifiers + BLS signers │ Not trusted; must verify gateway broadcasts │
├────────────┼───────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ Aggregator │ Cycle-end BLS collector │ Trusted to assemble quorum │
└────────────┴───────────────────────────────────┴──────────────────────────────────────────────────────────┘

This is Phase 1 reality — Newton operates both gateway and bootnode, so the trust set is effectively {Newton}. Phase 2 federated
gateways spread that trust horizontally; Phase 3 P2P+BFT removes the trust assumption entirely (operators reach consensus among
themselves).

What this clarifies:

  • D5 reframe stays — gateway is still plaintext-blind on private data; "trusted sequencer" doesn't mean "trusted with content."
  • D7's 3-phase P2P trajectory stays unchanged.
  • The Layer 1 (≤1s) vs Layer 2 (60s mainnet) distinction from D3 holds — block finalization is Layer 1, cycle commit is Layer 2.
  • Invariant 11 (pre-cycle JMT state + canonical operation ordering is sufficient for cross-operator determinism) holds — gateway
    broadcast IS the canonical ordering signal.
  • Invariant 38 (90s replication SLO) still holds for cross-chain replication of cycle commits.

Cascade:

  • New D18: "Gateway-as-trusted-sequencer model + bootnode reconciliation"
  • Rewrite D3 with block vs cycle distinction made explicit
  • New invariant: "Pre-finalization state is not durable; durability requires gateway-broadcast match OR bootnode reconciliation"
  • New gateway component: BlockFinalizer (broadcasts (block_id, per_pipeline_roots, block_delta_hash) over the existing dispatch channel)
  • New operator component: FinalizationVerifier (compares local roots against gateway broadcast, triggers bootnode catch-up on mismatch)
  • Bootnode HTTP API extension: GET /block-delta/{block_id} (already implied by D4/Invariant 29 — make it explicit)

Point 3: Remove active_binding map; validateAttestationDirect binding check optional

Reverts D13 entirely. The commitStateRoot(StateCommit, NewBinding[]) calldata simplifies to commitStateRoot(StateCommit). No more
active_binding mapping, no more BindingActivated event.

Resolves Q5 (the race I flagged). With no on-chain active_binding map, there's no stale-read window to engineer around. The two consumer
paths simplify:

Path: Optimistic (off-chain or on-chain via attestation arg)
Verification: BLS aggregate sig inline
Cost: BN254 pairing, ~2-3M gas
────────────────────────────────────────
Path: Canonical post-commit (JMT inclusion proof)
Verification: Merkle proof against StateCommitRegistry.lastCommittedStateRoot
Cost: ~50-100K gas

The on-chain cheap-SLOAD path is gone. ~50K vs ~2K is acceptable — the SLOAD shortcut was a perf optimization, not a correctness
primitive.

validateAttestationDirect becomes structurally lenient. Policy-client-policy binding check moves from "MUST" to "MAY" — DApps that need
binding verification do it via JMT inclusion proof against lastCommittedStateRoot; DApps that trust the BLS attestation's policy field
alone skip the binding check entirely.

Cascade:

  • Strike D13 from the spec entirely
  • Strike Invariant 33 (read-path cost asymmetry table) — replace with simpler "post-commit reads use JMT inclusion proof"
  • Strike NewBinding[] from commitStateRoot signature in §6 cascade items
  • Strike BindingActivated event
  • Strike active_binding[bindingKey] mapping from docs/PRIVATE_DATA_STORAGE.md §6
  • Update validateAttestationDirect ABI to reflect optional binding check
  • Q5 grill round prediction → closed by design change, no grill round needed

Point 4: Relayer/callback pattern — kicking off the grill

★ Insight ─────────────────────────────────────
The pattern you're describing is the Chainlink VRF / CCIP request-callback model: developer contract makes a request, gets back a
callback when the off-chain work completes. Newton's twist: the "off-chain work" is policy evaluation producing a BLS-signed
attestation, and the callback delivers (NewtonAttestation, intent). This is genuinely better DX for the "guard an existing function"
case — mint() stays as mint(), with an internal requestPolicyEvaluation() call and a separate evaluationCallback(NewtonAttestation)
handler.

The interesting design surface is who runs the relayer + how is it economically sustainable. Chainlink relays are economically gated by
LINK payment from the requesting contract. Newton has no analogous fee token yet (Phase 1 is gateway-gas-subsidized per D12), so the
relayer economics inherit the same Phase 1-vs-Phase 2 split.
─────────────────────────────────────────────────

To structure this grill, here's the question set I'd like to walk through one at a time. Let me start with the most foundational one —
once this locks, the others narrow:

G-7-Q1: Relayer trust + operation model in Phase 1

Who runs the relayer that calls INewtonPolicyClient.evaluationCallback(NewtonAttestation)?

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
☐ Relayer operator

G-7-Q1: Who runs the relayer that submits attestations to dApp callbacks in Phase 1?

❯ 1. Gateway-operated (Newton-only)
The Phase 1 gateway runs the relayer as a built-in service. Gas paid from the same per-chain hot-wallet that subsidizes
commitStateRoot per D12. Simplest Phase 1 ops; one trust point; tight integration with attestation production. Phase 2 federation
gives multi-gateway relay redundancy.
2. Permissionless Phase 1, gas-paid by requester
Anyone can be a relayer. Requesting contract escrows native gas at requestPolicyEvaluation() time; relayer collects it on successful
callback. Open marketplace from day one; no Newton ops cost; but harder to bootstrap (relayer infra must exist before adoption) and
adds escrow contract complexity.
3. Gateway-operated Phase 1, permissionless from Phase 2
Newton's gateway is the sole relayer in Phase 1 (matches D7 hub-and-spoke posture and D12 gas-subsidy economics). When federation
lands in Phase 2, third-party relayers can opt in alongside Newton's gateway. Mirrors the Phase 1 → Phase 2 trust expansion across
gateway, bootnode, and now relayers.
4. Optional relayer; client-direct stays first-class
Both patterns are supported equally. Client gets the attestation back from gateway as today (direct spend pattern). Separately, dApps
that want callback ergonomics implement INewtonPolicyClient and a permissionless / Newton-operated relayer service picks up
requests. No primary pattern; developer chooses per-contract.
5. Type something.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
6. Chat about this

Enter to select · ↑/↓ to navigate · Esc to cancel

@denniswon

denniswon commented May 15, 2026

Copy link
Copy Markdown
Author

❯ consolidate the G-7 round into a structured lock summary before we move on

⏺ G-7 Lock Summary — Relayer-Callback Pattern + Attestation Lifecycle

Round scope

G-7 covers the post-pivot attestation consumption model: how attestations are delivered (direct vs callback), who can consume them, how
single-consumption is enforced, what the attestation contains, and how dApps integrate. It sits on top of three foundational changes
from earlier in this round (Points 1-3) which restructured the protocol layers below it.

Foundation locks (Points 1-3 of this round; outside G-7 numbering but G-7 depends on them)

Lock: Three-pipeline architecture (replaces D9 unified Transaction model)
Decision: Pipeline 1 = PutLeaf operations, gateway-sequenced, sender-nonce-ordered. Pipeline 2 = Policy evaluation tasks, parallel.
Pipeline 3 = PolicyData ingestion tasks, parallel. ~1s blocks per pipeline. No cross-pipeline atomicity.
Cascade: Strike D9, G-3-Q3, G-3-Q8, Invariants 9 + 14. New D17 + replacement invariants. New per-(sender, chain_id) nonce ledger in
gateway redb.
────────────────────────────────────────
Lock: Operator pending vs confirmed state
Decision: Per-block computation produces in-memory pending root; gateway broadcast of canonical finalization root is the durability
signal. Match → flush pending to redb. Mismatch → fetch canonical block delta from bootnode → apply → flush.
Cascade: Rewrite Invariant 11 + 19. New D18: "Gateway-as-trusted-sequencer + bootnode reconciliation." New gateway BlockFinalizer
component, operator FinalizationVerifier.
────────────────────────────────────────
Lock: Single gateway + sidecar bootnode
Decision: Single active gateway + hot-standby (10-instance load-balanced topology dropped — original scaling rationale no longer
applies). Bootnode runs as co-located sidecar with own redb.
Cascade: Drop Postgres for protocol coordination (keep for API keys + indexer, UX only). Drop Redis entirely. Strike Invariants 36-39
(G-6 round mostly obsolete).
────────────────────────────────────────
Lock: Remove active_binding map
Decision: commitStateRoot(StateCommit) — no NewBinding[] parameter. validateAttestationDirect policy-binding check optional, not
enforced. Post-commit reads use JMT inclusion proof (~50-100K gas) instead of SLOAD (~2K gas).
Cascade: Strike D13. Strike Invariant 33. Strike BindingActivated event. Update validateAttestation* ABI.

★ Insight ─────────────────────────────────────
The foundation changes drop Postgres + Redis dependencies, multi-instance coordination, and the on-chain binding cache all in one move.
Phase 1 protocol storage collapses to a single primitive: redb. The dev/ops surface shrinks dramatically — operators, gateway, and
bootnode all run the same storage backend; no replication-journal complexity; no multi-instance lease primitives; no per-instance state
coordination. This is the architectural simplification that makes the rest of G-7 cleanly designable.
─────────────────────────────────────────────────

G-7 locks (in question order)

G-7-Q1: Relayer pool + registration

Lock: Registered pool of relayers; gateway broadcasts to all registered relayers via long-poll claim queue (see Q1.B below). Stagef pool
size = 1; prod pool size = 10 candidate (pending scalability evaluation — see open questions). On-chain RelayerRegistry contract owns
the allowlist; DAO-gated in Phase 1, permissionless in Phase 2.

Cascade:

  • New contract RelayerRegistry.sol — DAO-gated allowlist; emits RelayerRegistered/RelayerRevoked
  • New contract NewtonMulticallRelayer.sol — dispatcher; onlyRegisteredRelayer modifier reads RelayerRegistry
  • New gateway module crates/gateway/src/relayer/ — claim queue + broadcast logic, redb-backed
  • New ref-impl binary bin/newton-relayer/

G-7-Q1.B (sub-lock): Claim queue transport — HTTP long-poll (30s timeout). Gateway holds connection open until claim available or
timeout. Defers WebSocket to Phase 2 contingent on relayer_claim_latency_p99_ms measurement.

G-7-Q2: Delivery lifecycle + path opt-in

Lock: Both client-direct and callback paths coexist; neither is first-class. Mechanism:

  • Client SDK passes wantsCallback: bool (or DeliveryFlags) at requestPolicyEvaluation() time.
  • dApp's INewtonPolicyClient.acceptedPaths() returns a bitmap declaring which paths it accepts.
  • Gateway always returns attestation to client (zero gateway cost — connection is open).
  • Gateway additionally broadcasts to relayer pool iff client.wantsCallback AND dApp.acceptedPaths() & RELAYER_CALLBACK_BIT != 0.
  • AttestationValidator.spent[attestationHash] enforces single consumption across all delivery paths.

Cascade:

  • New acceptedPaths() method in INewtonPolicyClient interface
  • RELAYER_CALLBACK_BIT = 0x02, CLIENT_DIRECT_BIT = 0x01 constants
  • Gateway routing logic at requestPolicyEvaluation handler

G-7-Q3: Relayer economics

Lock: Phase 1 Newton-subsidized (mirrors D12 gateway gas-subsidy posture). Phase 2 fee mechanism designed as a single coherent question
covering replication (D12) + relay (G-7) + future Newton-on-behalf-of-user chain calls.

Cascade:

  • Per-chain operational treasury for relayer pool; separate hot-wallet from commitStateRoot wallet for clean accounting
  • New metric relayer_subsidy_remaining_native_units{chain_id} mirroring gateway_subsidy_remaining_native_units{chain_id}
  • Out-of-scope (Phase 2): RelayerEscrow, permit-style allowances, on-chain payment rails

G-7-Q4: Consume permissioning

Lock: Hybrid of consume-folded-into-validate (Option 2) + permissioned caller (Option 4). validateAttestation* is the atomic
verify+consume primitive, permissioned to msg.sender ∈ {policyClient, MULTICALL_RELAYER, approved_delegate}. Public preview() view
function exists for off-chain inspection. Relayer multicall pre-consumes before invoking dApp's evaluationCallback (centralized
enforcement; dApp can't forget).

Cascade:

  • AttestationValidator._checkAuthorized enforces three authorized callers (policyClient / multicall / delegate)
  • AttestationValidator.preview(att) view function — verify without consume
  • Multicall dispatcher pattern: validate → invoke dApp callback → emit event per item; per-item revert is non-fatal to multicall

G-7-Q5a: Intent signature handling — revised per last turn

Lock: intentSignature: bytes stays in schema (variable-length, can be empty). AttestationValidator does NOT verify intent sig — defers
to dApp via separation of digest helper from verification scheme. AttestationValidator.intentDigest(intent) is the canonical helper.

Cascade:

  • New abstract NewtonPolicyClientBase with _verifyIntentSignatureEcdsa, _verifyIntentSignatureErc1271, _verifyIntentSignatureAuto
    helpers
  • Per-dApp guidance docs: three integration patterns (EOA-signed, smart-account ERC-1271, trust-protocol-only)
  • Smart-account integration uses _verifyIntentSignatureErc1271; intent.from = smart account address
  • Gas saving: dApps that opt out save ~3-5K gas per validate

G-7-Q5b: intent.to vs policyClient relationship

Lock: Allow intent.to != ctx.policyClient via delegate pattern. INewtonPolicyClient.isApprovedDelegate(address) view returns (bool).
AttestationValidator._checkAuthorized adds delegate clause: msg.sender == intent.to && intent.to != policyClient &&
policyClient.isApprovedDelegate(intent.to).

Cascade:

  • New isApprovedDelegate method on INewtonPolicyClient
  • Three-clause authorization check in AttestationValidator._checkAuthorized
  • Documented use case: router patterns (account abstraction, multicall)

G-7-Q5c: Expiration semantics

Recommended (pending explicit confirmation): cycleId-based expiration. Intent.expirationCycle: uint64 — absolute cycleId after which
attestation is invalid. Validate reads StateCommitRegistry.currentSequenceNo() (~2100 gas) and reverts on expiry. Rationale:
cadence-uniform across chains, reuses existing ctx.cycleId temporal axis, off-chain relayer filtering simpler, gas overhead negligible
vs validate's other costs. Eliminates expireAfter=0 misconfiguration trap by moving expiration to intent-level (SDK-set), not
policy-level.

Cascade (pending confirm):

  • StateCommitRegistry.currentSequenceNo() view exists per existing design — no contract change
  • SDK pattern: intent.expirationCycle = ctx.cycleId + sensibleDefault (e.g., 600 cycles)
  • Strike per-policy expireAfter config

G-7-Q4 (sub-lock): validateAttestationDirect smart-fallback

Lock: Unified entry point (Option A) — single schema carries both proofs optionally populated; validator picks cheapest available.

  • AttestationProof = { BlsProof, JmtProof } — JmtProof empty if cycle not committed
  • Validator checks att.ctx.cycleId <= currentSequenceNo() && jmtProof.siblingPath.length > 0 → use cheap JMT path; otherwise BLS-inline
    path
  • validateAttestationDirect kept as legacy-compat alias delegating to validateAttestation

Cascade: Single validateAttestation entry handles both pre-commit and post-commit. SDK constructs both proofs when cycle is committed;
just BLS otherwise.

Final locked schemas

contract NewtonMessage {
struct Intent {
address from;
address to;
uint256 value;
bytes data;
uint256 chainId;
uint64 expirationCycle; // Q5c recommended; awaiting explicit confirm
}

  struct AttestationContext {
      bytes32 policyId;
      address policyClient;
      bytes32 taskId;
      uint64 cycleId;
  }

  struct BlsProof {
      BN254G1Point signature;
      bytes32 operatorSetHash;
      uint64 referenceBlock;
  }

  struct JmtProof {
      bytes32 stateRoot;
      uint64 jmtVersion;
      bytes32[] siblingPath;
      uint64 anchorSequenceNo;
  }

  struct AttestationProof {
      BlsProof blsProof;
      JmtProof jmtProof;  // empty if cycle not yet committed
  }

  struct NewtonAttestation {
      AttestationContext ctx;
      Intent intent;
      bytes intentSignature;  // opaque to validator; dApp interprets
      AttestationProof proof;
  }

}

interface INewtonPolicyClient {
function isApprovedDelegate(address delegate) external view returns (bool);
function acceptedPaths() external view returns (bytes1);
function evaluationCallback(NewtonAttestation calldata att) external;
}

// Nullifier key — path-agnostic
function attestationHash(NewtonAttestation memory att) internal pure returns (bytes32) {
return keccak256(abi.encode(att.ctx, att.intent));
}

New contracts (cascade summary)

Contract: RelayerRegistry
Role: DAO-gated allowlist of registered relayers
Phase 1 trust gate: Newton DAO multisig
────────────────────────────────────────
Contract: NewtonMulticallRelayer
Role: Batch dispatcher to dApp callbacks; pre-consumes via validateAttestation
Phase 1 trust gate: onlyRegisteredRelayer via RelayerRegistry
────────────────────────────────────────
Contract: AttestationValidator
Role: Unified validate+consume primitive; smart-fallback proof routing; nullifier map
Phase 1 trust gate: Permissioned to {policyClient, MULTICALL_RELAYER, approved_delegate}
────────────────────────────────────────
Contract: NewtonPolicyClientBase (abstract)
Role: dApp inheritance helpers (ECDSA / ERC-1271 / Auto sig verification)
Phase 1 trust gate: dApp inherits
────────────────────────────────────────
Contract: NewtonAddressesProvider
Role: (existing) Adds multicall_relayer slot
Phase 1 trust gate: Newton DAO

Cascade — spec doc updates required

File / section: pds-architecture-pivot-design.md D9
Edit: Replace with D17 three-pipeline architecture
────────────────────────────────────────
File / section: Same doc — D13
Edit: Strike; replaced by Point 3 active_binding removal
────────────────────────────────────────
File / section: Same doc — Invariants 9, 11, 14, 19
Edit: Rewrite for block-vs-cycle distinction + pending/confirmed state
────────────────────────────────────────
File / section: Same doc — Invariants 33, 36-39
Edit: Strike (G-6 multi-instance round obsolete)
────────────────────────────────────────
File / section: Same doc — new section
Edit: "G-7 attestation consumption model" with lock summary above
────────────────────────────────────────
File / section: Same doc — new D17, D18
Edit: "Three-pipeline architecture" + "Gateway-as-trusted-sequencer"
────────────────────────────────────────
File / section: architecture.md Key Contracts
Edit: Add RelayerRegistry, NewtonMulticallRelayer, updated AttestationValidator; strike ConfidentialDataRegistry row
────────────────────────────────────────
File / section: error-selectors.md
Edit: Add: Unauthorized (consume), AlreadySpent, InvalidIntentSignature, DelegateNotApproved, AttestationExpired (cycleId-based),
NotRegisteredRelayer, others as they're assigned selectors
────────────────────────────────────────
File / section: RPC_API.md
Edit: Document gateway relayer claim endpoint (long-poll); deliveryFlags param on requestPolicyEvaluation
────────────────────────────────────────
File / section: OPERATOR.md
Edit: No changes (operators uninvolved in callback path); document attestation construction includes cycleId
────────────────────────────────────────
File / section: DEVELOPER_GUIDE.md (or equivalent)
Edit: New patterns section: "Integration with NewtonPolicyClientBase" with three example dApps

Open / pending

Item: G-7-Q5c expiration semantics
Status: Recommended cycleId-based; awaiting explicit confirmation
────────────────────────────────────────
Item: 10 vs N relayers in prod
Status: Locked direction (multi-relayer better for parallelism); exact N TBD on capacity benchmarks during implementation
────────────────────────────────────────
Item: Multicall failure modes
Status: Need explicit design: per-callback gas cap, retry semantics, on-chain accounting of failed deliveries. Likely G-8 grill.
────────────────────────────────────────
Item: Permit-style allowance for Phase 2 economics
Status: Deferred to Phase 2 grill round (combined with D12 federation econ)
────────────────────────────────────────
Item: Cross-pipeline read snapshot semantics
Status: Locked at block boundary (Pipeline 2/3 reads against confirmed root); concrete operator-side implementation detail open
────────────────────────────────────────
Item: EIP-712 domain separator scope
Status: Single per-chain DOMAIN_SEPARATOR in AttestationValidator? Or per-policyClient? Likely the former for simplicity; minor open.

What's intentionally deferred to other grill rounds

  • Phase 2 federation economics (gateway + relayer + future infra) — single Phase 2 round combining D12 + G-7-Q3
  • Phase 2 P2P design (libp2p protocol, peer scoring) — separate Phase 2/3 spec
  • N-shot attestation semantics — explicitly out of scope; the 1-shot nullifier in AttestationValidator.spent is the locked Phase 1 model
  • Permissionless relayer registration economics — Phase 2 expansion of RelayerRegistry model

This consolidates everything G-7 produced. Ready to write up the spec edits, or move to the next grill round (probably G-8 around
multicall failure modes + cross-pipeline read semantics) — your call.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment