Gateway: optimistic_insert_task only writes successful tasks

gistfile1.txt

Now I see the full picture. There are two gaps:

  Gap 1: optimistic_insert_task only writes successful tasks

  Lines 42-49:
  let task = match response.task.as_ref() {
      Some(t) => t,
      None => return Ok(()),
  };
  let task_response = match response.task_response.as_ref() {
      Some(tr) => tr,
      None => return Ok(()),
  };

  When a task fails (quorum not reached, policy eval error), task and task_response are None — so the function returns early and nothing
   is written to the indexer DB. Failed tasks are invisible in the explorer unless the on-chain indexer picks up the NewTaskCreated
  event separately.

  Gap 2: No error or operator_errors columns in the insert

  Even if we fixed gap 1, the SQL INSERT (lines 80-113) has no columns for error or operator_errors. The indexer DB task table doesn't
  store failure information at all.

  Gap 3: On-chain indexer also can't see the error

  The on-chain indexer (separate service) sees NewTaskCreated events but has no access to the gateway's operator_errors — those exist
  only in the RPC response. So neither path (optimistic write OR on-chain indexer) gets the detailed error to the explorer.

  ★ Insight ─────────────────────────────────────

  The data pipeline for task errors is:

  Gateway RPC response → has operator_errors ✅
    ↓ (direct caller via SDK sees this)

  Optimistic indexer write → skips failed tasks entirely ❌
  On-chain indexer → only sees events, not gateway errors ❌
  Dashboard API → reads from DB, no error data ❌
  Explorer UI → shows generic "failed" status ❌

  The fix needs two things:
  1. Gateway: Write failed tasks to indexer DB with error and operator_errors fields
  2. Indexer DB schema: Add error TEXT and operator_errors JSONB columns to the task table

  ─────────────────────────────────────────────────

  This is a separate issue from PR #479. Want me to file a Linear ticket and/or start implementing the fix? The gateway-side change is
  straightforward — add a second code path in optimistic_insert_task for failed tasks that writes the available fields (task_id,
  chain_id, error, operator_errors, timestamp) even when task/task_response are None.

Here's the status of NEWT-864 against what we've done:

newton-indexer (this repo) — All Done

┌────────────────────────────────────────────────────────────┬───────────────────────────────────────────────────────┐
│ Task │ Status │
├────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────┤
│ Alembic migration: add error TEXT NULL and operator_errors │ Done │
│ JSONB NULL to task table │ │
├────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────┤
│ shared/db/models/task.py: add error and operator_errors │ Done │
│ mapped columns │ │
├────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────┤
│ explorer_api/api/v1/schema/task.py: add fields to │ Done │
│ TaskDetail and TaskItem │ │
├────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────┤
│ explorer_api/services/task.py: include error fields in │ N/A — from_instance() already maps all model fields, │
│ query results │ no separate service query change needed │
└────────────────────────────────────────────────────────────┴───────────────────────────────────────────────────────┘

Plus we did additional work not in the ticket:

• Fixed created_block to Optional[int] (would crash on gateway-written failure records)
• Added per-environment CORS (was wildcard everywhere)
• Full docs-sync across the entire codebase

newton-prover-avs (separate repo) — Not Started

┌─────────────────────────────────────────────────────────────────────────────────────────────────┬──────────────────┐
│ Task │ Status │
├─────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────┤
│ crates/gateway/src/indexer_db.rs: add write path for failed tasks that persists task_id, │ Not done — │
│ chain_id, error, operator_errors, and timestamp when task/task_response are None │ different repo │
├─────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────┤
│ error from CreateTaskResponse.error │ Not done │
├─────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────┤
│ operator_errors from CreateTaskResponse.operator_errors (serialized as JSONB) │ Not done │
└─────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────┘

The indexer side is complete. The gateway side (newton-prover-avs) is the remaining work — that's where the actual failed
task writes will originate. Without that change, the new columns exist but will remain empty.