Skip to content

Instantly share code, notes, and snippets.

@rjpower

rjpower/iris_job_failure_report.md

Created April 16, 2026 18:59
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save rjpower/aa8afbc8f9b46a7fa0360c83bbffa18a to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save rjpower/aa8afbc8f9b46a7fa0360c83bbffa18a to your computer and use it in GitHub Desktop.
Download ZIP
Iris job failure: controller RPC timeout
Raw
iris_job_failure_report.md

Iris Job Failure Report

/larry/iris-run-job-20260414-211246

Task: grug-train-moe-v16-compute-opt-d1280-2.83e-19 (Attempt 48) Date: 2026-04-16 09:08:33 — 09:21:17 UTC Duration: ~13 minutes Exit Code: 1

Root Cause

Iris Controller RPC Timeout — The job dispatcher repeatedly failed to submit a Grug training task to the Iris controller due to RPC deadline exceeded errors.

Failure Timeline

  • 09:08:33 — Grug training step starts dispatching to Iris backend
  • 09:08:33–09:21:17 — 19 retry attempts, all timing out
    • Attempt 1 @ +30s elapsed
    • Attempt 2 @ +60s elapsed
    • ...
    • Attempt 19 @ +722s elapsed (deadline exhausted, max retries exceeded)
  • 09:21:17 — Step failed after final retry exhaustion
  • 09:21:22 — Task container stopped

Error Stack Trace

Primary Error:

connectrpc.errors.ConnectError: Request timed out

Lower-level Error:

TimeoutError: Request failed: error sending request for url 
(http://10.128.0.3:10000/iris.cluster.ControllerService/LaunchJob): 
operation timed out

Call Stack:

grug/moe/launch.py:111  → run_grug_moe_trial()
grug/moe/train.py:553   → dispatch_grug_training_run()
grug/dispatch.py:53     → current_client().submit(request)
  ↓
fray/v2/iris_backend.py:563 → self._iris.submit()
iris/client/client.py:661    → self._cluster_client.submit_job()
iris/cluster/client/remote_client.py:139 → call_with_retry()
  ↓ [Exponential backoff, 20 max retries]
iris/rpc/controller_connect.py:1644 → launch_job(request)
connectrpc/_client_sync.py:336 → _send_request_unary()
  ↓
HTTP POST to 10.128.0.3:10000/iris.cluster.ControllerService/LaunchJob
  ↗ TIMEOUT

Retry Behavior

The client used exponential backoff with max 20 retries:

  • Attempt 1: 30.0s (first timeout immediately)
  • Attempt 2–19: 30–60s intervals with jitter
  • Attempt 20: Not reached; exhausted after attempt 19

Each attempt individually timed out before completing.

Failure Indicators

From Controller Logs:

[2026-04-16 09:33:14] iris.rpc.interceptors RPC LaunchJob failed after 0ms: 
  Cannot submit job: parent job /larry/iris-run-job-20260414-211246 has 
  terminated (state=JOB_STATE_FAILED)

The parent job marked as failed after this task's container exited.

Likely Causes

  1. Controller Under Load — The controller (10.128.0.3) may have been experiencing high latency or resource contention, causing all LaunchJob RPCs to exceed the timeout.
  2. Network Connectivity — Possible network partition or high latency between the worker and controller.
  3. Controller Crash/Restart — Controller may have crashed or been restarting during this window.
  4. GCS Operations — Prior metadata writes took ~4 seconds (09:08:23–09:08:27), which might indicate GCS latency affecting controller state.

Evidence

Pre-failure Activity (Successful):

  • ✅ Step inspection completed
  • ✅ Statuses read (40 steps)
  • ✅ Metadata written to GCS
  • ✅ 39 of 40 training steps skipped (already succeeded)
  • ✅ Grug training step ready to dispatch

Failure Point:

  • ❌ LaunchJob RPC to controller times out repeatedly
  • ❌ No recovery after 19 retries over 12+ minutes

Recommendations

  1. Check Iris Controller Health — Run iris process status and iris cluster status to verify controller is responsive.
  2. Review Controller Logs — Check /system/controller logs for performance degradation or crashes around 2026-04-16 09:08:00.
  3. Check Network — Verify connectivity from worker marin-tpu-v5p-preemptible-8-us-east5-a-20260416-0006-db4d1806-worker-0 to controller 10.128.0.3:10000.
  4. Increase RPC Timeout — If transient, consider increasing the LaunchJob RPC timeout in iris/rpc/controller_connect.py.
  5. Retry Externally — The step itself is idempotent; re-run the executor should attempt dispatch again.

Files Involved

  • /app/experiments/grug/moe/launch.py — Entry point
  • /app/experiments/grug/dispatch.py — Dispatch logic (fray client submit)
  • /app/lib/fray/src/fray/v2/iris_backend.py — Iris backend for Fray
  • /app/lib/iris/src/iris/cluster/client/remote_client.py — RPC retry logic
  • /app/lib/iris/src/iris/rpc/controller_connect.py — LaunchJob RPC
Raw
iris_job_raw_logs.txt
syncing deps
installing pip deps
activating venv
python=/app/.venv/bin/python
sys.path: ['', '/app', '/uv/cache/python/cpython-3.11.14-linux-x86_64-gnu/lib/python311.zip', '/uv/cache/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11', '/uv/cache/python/cpython-3.11-linux-x86_64-gnu/lib/python3.11/lib-dynload', '/app/.venv/lib/python3.11/site-packages', '/app/lib/marin/src', '/app/lib/fray/src', '/app/lib/haliax/src', '/app/lib/iris/src', '/app/lib/rigging/src', '/app/lib/zephyr/src', '/app/lib/levanter/src']
running user command
I20260416 09:08:22 140430218774336 marin.execution.executor ### Inspecting the 4 provided steps ###
W20260416 09:08:22 140430218774336 marin.execution.executor Output path gs://marin-us-east5/tokenized/nemotron_cc/hq_actual-a09c23 doesn't match given override tokenized/nemotron_cc/hq_actual-5af4cc, using the latter.
W20260416 09:08:22 140430218774336 marin.execution.executor Output path gs://marin-us-east5/tokenized/nemotron_cc/hq_synth-9cac1c doesn't match given override tokenized/nemotron_cc/hq_synth-3525e2, using the latter.
W20260416 09:08:22 140430218774336 marin.execution.executor Output path gs://marin-us-east5/tokenized/nemotron_cc/medium_high-746531 doesn't match given override tokenized/nemotron_cc/medium_high-d21701, using the latter.
W20260416 09:08:22 140430218774336 marin.execution.executor Output path gs://marin-us-east5/tokenized/nemotron_cc/medium-60cd43 doesn't match given override tokenized/nemotron_cc/medium-d86506, using the latter.
W20260416 09:08:22 140430218774336 marin.execution.executor Output path gs://marin-us-east5/tokenized/nemotron_cc/medium_low-826d01 doesn't match given override tokenized/nemotron_cc/medium_low-0fdb07, using the latter.
W20260416 09:08:22 140430218774336 marin.execution.executor Output path gs://marin-us-east5/tokenized/nemotron_cc/low_actual-753f37 doesn't match given override tokenized/nemotron_cc/low_actual-cb3f2c, using the latter.
W20260416 09:08:22 140430218774336 marin.execution.executor Output path gs://marin-us-east5/tokenized/nemotron_cc/low_synth-c73afe doesn't match given override tokenized/nemotron_cc/low_synth-3c57b3, using the latter.
W20260416 09:08:22 140430218774336 marin.execution.executor Output path gs://marin-us-east5/raw/starcoderdata-49ea09 doesn't match given override gs://marin-us-east5/raw/starcoderdata-720c8c, using the latter.
W20260416 09:08:22 140430218774336 marin.execution.executor Output path gs://marin-us-east5/tokenized/starcoderdata-ae0339 doesn't match given override tokenized/starcoderdata-12f018/, using the latter.
W20260416 09:08:22 140430218774336 marin.execution.executor Output path gs://marin-us-east5/raw/proof-pile-2-48f0af doesn't match given override gs://marin-us-east5/raw/proof-pile-2-f1b1d8, using the latter.
W20260416 09:08:22 140430218774336 marin.execution.executor Output path gs://marin-us-east5/tokenized/proofpile_2-0751d5 doesn't match given override tokenized/proofpile_2-4a35c7/, using the latter.
W20260416 09:08:22 140430218774336 marin.execution.executor Output path gs://marin-us-east5/raw/uncheatable-eval/latest-d529bd doesn't match given override gs://marin-us-east5/raw/uncheatable-eval/latest_43d8c7f4, using the latter.
W20260416 09:08:22 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:22 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:22 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:22 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:22 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:22 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:22 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:22 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
I20260416 09:08:23 140430218774336 marin.execution.executor ### Reading 40 statuses ###
I20260416 09:08:23 140430218774336 marin.execution.executor ### Writing metadata ###
W20260416 09:08:23 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:23 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:23 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:23 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:23 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:23 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:23 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:23 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:23 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:23 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:23 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:23 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:23 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:23 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:23 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:23 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
I20260416 09:08:23 140430218774336 marin.execution.executor Writing executor info to gs://marin-us-east5/experiments/launch-fc9ad9.json
I20260416 09:08:23 140430218774336 marin.execution.executor To view the experiment page, go to:
I20260416 09:08:23 140430218774336 marin.execution.executor
I20260416 09:08:23 140430218774336 marin.execution.executor https://marin.community/data-browser/experiment?path=gs%3A//marin-us-east5/experiments/launch-fc9ad9.json
I20260416 09:08:23 140430218774336 marin.execution.executor
W20260416 09:08:27 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:27 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:27 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:27 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:27 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:27 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:27 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:27 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:27 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:27 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:27 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:27 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:27 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:27 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:27 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:27 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:28 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:28 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:28 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:28 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:28 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:28 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:28 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:28 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:28 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:28 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:28 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:28 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:28 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:28 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:28 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
W20260416 09:08:28 140430218774336 marin.utilities.json_encoder Could not serialize object of type <class 'jax.sharding.PartitionSpec'>: PartitionSpec(('data', 'expert'),)
I20260416 09:08:28 140430218774336 marin.execution.executor ### Launching 40 steps ###
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 fray.v2.client current_client: using Iris backend (auto-detected)
I20260416 09:08:30 140430218774336 marin.execution.step_runner Step = tokenized/nemotron_cc/hq_actual_33f82033 Params = {} Output_path = gs://marin-us-east5/tokenized/nemotron_cc/hq_actual-5af4cc
I20260416 09:08:30 140430218774336 marin.execution.step_runner Skip tokenized/nemotron_cc/hq_actual_33f82033: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = tokenized/nemotron_cc/hq_synth_414e3cac Params = {} Output_path = gs://marin-us-east5/tokenized/nemotron_cc/hq_synth-3525e2
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip tokenized/nemotron_cc/hq_synth_414e3cac: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = tokenized/nemotron_cc/medium_high_f64db2dc Params = {} Output_path = gs://marin-us-east5/tokenized/nemotron_cc/medium_high-d21701
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip tokenized/nemotron_cc/medium_high_f64db2dc: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = tokenized/nemotron_cc/medium_aba322a7 Params = {} Output_path = gs://marin-us-east5/tokenized/nemotron_cc/medium-d86506
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip tokenized/nemotron_cc/medium_aba322a7: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = tokenized/nemotron_cc/medium_low_665d11e5 Params = {} Output_path = gs://marin-us-east5/tokenized/nemotron_cc/medium_low-0fdb07
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip tokenized/nemotron_cc/medium_low_665d11e5: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = tokenized/nemotron_cc/low_actual_40c91646 Params = {} Output_path = gs://marin-us-east5/tokenized/nemotron_cc/low_actual-cb3f2c
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip tokenized/nemotron_cc/low_actual_40c91646: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = tokenized/nemotron_cc/low_synth_8fac7840 Params = {} Output_path = gs://marin-us-east5/tokenized/nemotron_cc/low_synth-3c57b3
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip tokenized/nemotron_cc/low_synth_8fac7840: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = raw/starcoderdata_3857c719 Params = {'hf_dataset_id': 'bigcode/starcoderdata', 'revision': '9fc30b5', 'hf_urls_glob': [], 'append_sha_to_path': False} Output_path = gs://marin-us-east5/raw/starcoderdata-720c8c
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip raw/starcoderdata_3857c719: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = tokenized/starcoderdata_eacabc46 Params = {} Output_path = gs://marin-us-east5/tokenized/starcoderdata-12f018/
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip tokenized/starcoderdata_eacabc46: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = raw/proof-pile-2_3f4c728c Params = {'hf_dataset_id': 'EleutherAI/proof-pile-2', 'revision': '901a927', 'hf_urls_glob': [], 'append_sha_to_path': False} Output_path = gs://marin-us-east5/raw/proof-pile-2-f1b1d8
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip raw/proof-pile-2_3f4c728c: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = tokenized/proofpile_2_91e7dd8b Params = {} Output_path = gs://marin-us-east5/tokenized/proofpile_2-4a35c7/
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip tokenized/proofpile_2_91e7dd8b: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = raw/paloma_cd2bb710 Params = {} Output_path = gs://marin-us-east5/raw/paloma-fc6827
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip raw/paloma_cd2bb710: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = tokenized/paloma/4chan_8ac7eed7 Params = {} Output_path = gs://marin-us-east5/tokenized/paloma/4chan-496ad5
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip tokenized/paloma/4chan_8ac7eed7: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = tokenized/paloma/c4_100_domains_ec3ef039 Params = {} Output_path = gs://marin-us-east5/tokenized/paloma/c4_100_domains-2b6db7
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip tokenized/paloma/c4_100_domains_ec3ef039: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = tokenized/paloma/c4_en_1b54a048 Params = {} Output_path = gs://marin-us-east5/tokenized/paloma/c4_en-cf1f79
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip tokenized/paloma/c4_en_1b54a048: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = tokenized/paloma/dolma-v1_5_2400d613 Params = {} Output_path = gs://marin-us-east5/tokenized/paloma/dolma-v1_5-d3bed7
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip tokenized/paloma/dolma-v1_5_2400d613: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = tokenized/paloma/dolma_100_programing_languages_589dc97a Params = {} Output_path = gs://marin-us-east5/tokenized/paloma/dolma_100_programing_languages-369132
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip tokenized/paloma/dolma_100_programing_languages_589dc97a: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = tokenized/paloma/dolma_100_subreddits_71c47035 Params = {} Output_path = gs://marin-us-east5/tokenized/paloma/dolma_100_subreddits-f25f70
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip tokenized/paloma/dolma_100_subreddits_71c47035: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = tokenized/paloma/falcon-refinedweb_f47ed79d Params = {} Output_path = gs://marin-us-east5/tokenized/paloma/falcon-refinedweb-75d43b
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip tokenized/paloma/falcon-refinedweb_f47ed79d: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = tokenized/paloma/gab_ef9b35f3 Params = {} Output_path = gs://marin-us-east5/tokenized/paloma/gab-ccaced
I20260416 09:08:31 140430218774336 marin.execution.step_runner Skip tokenized/paloma/gab_ef9b35f3: already succeeded
I20260416 09:08:31 140430218774336 marin.execution.step_runner Step = tokenized/paloma/m2d2_s2orc_unsplit_7e73503d Params = {} Output_path = gs://marin-us-east5/tokenized/paloma/m2d2_s2orc_unsplit-7dbcc1
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip tokenized/paloma/m2d2_s2orc_unsplit_7e73503d: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = tokenized/paloma/m2d2_wikipedia_unsplit_9549a2ed Params = {} Output_path = gs://marin-us-east5/tokenized/paloma/m2d2_wikipedia_unsplit-b33d23
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip tokenized/paloma/m2d2_wikipedia_unsplit_9549a2ed: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = tokenized/paloma/manosphere_meta_sep_8c003ee8 Params = {} Output_path = gs://marin-us-east5/tokenized/paloma/manosphere_meta_sep-a07891
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip tokenized/paloma/manosphere_meta_sep_8c003ee8: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = tokenized/paloma/mc4_39284cb6 Params = {} Output_path = gs://marin-us-east5/tokenized/paloma/mc4-ea36a2
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip tokenized/paloma/mc4_39284cb6: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = tokenized/paloma/ptb_d87d8587 Params = {} Output_path = gs://marin-us-east5/tokenized/paloma/ptb-628036
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip tokenized/paloma/ptb_d87d8587: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = tokenized/paloma/redpajama_351a036b Params = {} Output_path = gs://marin-us-east5/tokenized/paloma/redpajama-9d4ddd
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip tokenized/paloma/redpajama_351a036b: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = tokenized/paloma/twitterAAE_HELM_fixed_03e0986c Params = {} Output_path = gs://marin-us-east5/tokenized/paloma/twitterAAE_HELM_fixed-2e17c1
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip tokenized/paloma/twitterAAE_HELM_fixed_03e0986c: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = tokenized/paloma/wikitext_103_bef9584a Params = {} Output_path = gs://marin-us-east5/tokenized/paloma/wikitext_103-1f5636
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip tokenized/paloma/wikitext_103_bef9584a: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = raw/uncheatable-eval/latest_43d8c7f4 Params = {'repo_owner': 'ziqing-huang', 'repo_name': 'uncheatable_eval', 'data_path': 'data', 'branch': 'master'} Output_path = gs://marin-us-east5/raw/uncheatable-eval/latest_43d8c7f4
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip raw/uncheatable-eval/latest_43d8c7f4: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = tokenized/uncheatable_eval/wikipedia_english_7d7e291a Params = {} Output_path = gs://marin-us-east5/tokenized/uncheatable_eval/wikipedia_english-6330df
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip tokenized/uncheatable_eval/wikipedia_english_7d7e291a: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = tokenized/uncheatable_eval/github_python_d90f66a0 Params = {} Output_path = gs://marin-us-east5/tokenized/uncheatable_eval/github_python-baab41
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip tokenized/uncheatable_eval/github_python_d90f66a0: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = tokenized/uncheatable_eval/github_cpp_4811d99a Params = {} Output_path = gs://marin-us-east5/tokenized/uncheatable_eval/github_cpp-a9de07
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip tokenized/uncheatable_eval/github_cpp_4811d99a: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = tokenized/uncheatable_eval/bbc_news_cb3bbc95 Params = {} Output_path = gs://marin-us-east5/tokenized/uncheatable_eval/bbc_news-4df59f
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip tokenized/uncheatable_eval/bbc_news_cb3bbc95: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = tokenized/uncheatable_eval/arxiv_physics_a0672af6 Params = {} Output_path = gs://marin-us-east5/tokenized/uncheatable_eval/arxiv_physics-f4ad8c
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip tokenized/uncheatable_eval/arxiv_physics_a0672af6: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = tokenized/uncheatable_eval/arxiv_computer_science_9084ff07 Params = {} Output_path = gs://marin-us-east5/tokenized/uncheatable_eval/arxiv_computer_science-2b4f07
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip tokenized/uncheatable_eval/arxiv_computer_science_9084ff07: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = tokenized/uncheatable_eval/ao3_english_e017c8cb Params = {} Output_path = gs://marin-us-east5/tokenized/uncheatable_eval/ao3_english-bb5666
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip tokenized/uncheatable_eval/ao3_english_e017c8cb: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = grug/moe-v16-compute-opt-d512-2.19e+17_de6b4f18 Params = {} Output_path = gs://marin-us-east5/grug/moe-v16-compute-opt-d512-2.19e+17-d3b963
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip grug/moe-v16-compute-opt-d512-2.19e+17_de6b4f18: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = grug/moe-v16-compute-opt-d768-1.70e+18_df6e896d Params = {} Output_path = gs://marin-us-east5/grug/moe-v16-compute-opt-d768-1.70e+18-69d861
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip grug/moe-v16-compute-opt-d768-1.70e+18_df6e896d: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = grug/moe-v16-compute-opt-d1024-9.00e+18_2247830d Params = {} Output_path = gs://marin-us-east5/grug/moe-v16-compute-opt-d1024-9.00e+18-d3bc52
I20260416 09:08:32 140430218774336 marin.execution.step_runner Skip grug/moe-v16-compute-opt-d1024-9.00e+18_2247830d: already succeeded
I20260416 09:08:32 140430218774336 marin.execution.step_runner Step = grug/moe-v16-compute-opt-d1280-2.83e+19_cc655c3a Params = {} Output_path = gs://marin-us-east5/grug/moe-v16-compute-opt-d1280-2.83e+19-264579
I20260416 09:08:33 140425605318336 marin.execution.executor_step_status [t1v-n-dd338f52-w-0-140425605318336] Status grug/moe-v16-compute-opt-d1280-2.83e+19_cc655c3a: RUNNING
I20260416 09:08:33 140425605318336 marin.execution.executor_step_status [t1v-n-dd338f52-w-0-140425605318336] Step grug/moe-v16-compute-opt-d1280-2.83e+19_cc655c3a has no active lock, taking over.
I20260416 09:08:33 140425605318336 marin.execution.executor_step_status [t1v-n-dd338f52-w-0-140425605318336] Attempting to acquire lock for grug/moe-v16-compute-opt-d1280-2.83e+19_cc655c3a
I20260416 09:08:33 140425605318336 marin.execution.executor_step_status [t1v-n-dd338f52-w-0-140425605318336] Acquired lock for grug/moe-v16-compute-opt-d1280-2.83e+19_cc655c3a
I20260416 09:08:33 140425605318336 experiments.grug.dispatch Dispatching grug training via Fray: grug-train-moe-v16-compute-opt-d1280-2.83e-19
I20260416 09:08:33 140425605318336 fray.v2.client current_client: using Iris backend (auto-detected)
W20260416 09:09:03 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 1/20, 30.0s elapsed), retrying in 0.55s: Request timed out
W20260416 09:09:34 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 2/20, 60.6s elapsed), retrying in 1.09s: Request timed out
W20260416 09:10:05 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 3/20, 91.6s elapsed), retrying in 1.91s: Request timed out
W20260416 09:10:37 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 4/20, 123.6s elapsed), retrying in 4.34s: Request timed out
W20260416 09:11:11 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 5/20, 157.9s elapsed), retrying in 7.31s: Request timed out
W20260416 09:11:48 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 6/20, 195.2s elapsed), retrying in 9.50s: Request timed out
W20260416 09:12:28 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 7/20, 234.7s elapsed), retrying in 9.26s: Request timed out
W20260416 09:13:07 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 8/20, 274.0s elapsed), retrying in 9.10s: Request timed out
W20260416 09:13:46 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 9/20, 313.1s elapsed), retrying in 11.00s: Request timed out
W20260416 09:14:27 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 10/20, 354.1s elapsed), retrying in 9.53s: Request timed out
W20260416 09:15:07 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 11/20, 393.6s elapsed), retrying in 10.78s: Request timed out
W20260416 09:15:47 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 12/20, 434.4s elapsed), retrying in 9.16s: Request timed out
W20260416 09:16:27 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 13/20, 473.6s elapsed), retrying in 10.14s: Request timed out
W20260416 09:17:07 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 14/20, 513.7s elapsed), retrying in 9.21s: Request timed out
memray: Deactivating tracking: 10 seconds have elapsed
Failed to compress input file
W20260416 09:17:55 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 15/20, 561.7s elapsed), retrying in 10.92s: Request timed out
W20260416 09:18:36 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 16/20, 602.7s elapsed), retrying in 9.33s: Request timed out
W20260416 09:19:15 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 17/20, 642.0s elapsed), retrying in 9.17s: Request timed out
W20260416 09:19:54 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 18/20, 681.2s elapsed), retrying in 10.87s: Request timed out
W20260416 09:20:35 140425605318336 rigging.timing Operation launch_job(/larry/iris-run-job-20260414-211246/grug-train-moe-v16-compute-opt-d1280-2.83e-19) failed (attempt 19/20, 722.0s elapsed), retrying in 10.32s: Request timed out
E20260416 09:21:17 140430218774336 marin.execution.step_runner Step failed: grug/moe-v16-compute-opt-d1280-2.83e+19_cc655c3a
Traceback (most recent call last):
File "/app/.venv/lib/python3.11/site-packages/connectrpc/_client_sync.py", line 304, in _send_request_unary
resp = self._http_client.post(
^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: Request failed: error sending request for url (http://10.128.0.3:10000/iris.cluster.ControllerService/LaunchJob): operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/lib/marin/src/marin/execution/step_runner.py", line 153, in _harvest
status = handle.wait(raise_on_failure=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/lib/fray/src/fray/v2/local_backend.py", line 65, in wait
self._future.result(timeout=timeout)
File "/uv/cache/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/uv/cache/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/uv/cache/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/lib/marin/src/marin/execution/step_runner.py", line 281, in worker_fn
run_step(step)
File "/app/lib/marin/src/marin/execution/step_runner.py", line 328, in run_step
result = step.fn(output_path) # pyrefly: ignore[not-callable]
^^^^^^^^^^^^^^^^^^^^
File "/app/lib/marin/src/marin/execution/executor.py", line 717, in resolved_fn
return captured_fn(captured_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/experiments/grug/moe/launch.py", line 111, in run_grug_moe_trial
run_grug(run_config)
File "/app/experiments/grug/moe/train.py", line 553, in run_grug
dispatch_grug_training_run(
File "/app/experiments/grug/dispatch.py", line 53, in dispatch_grug_training_run
job = current_client().submit(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/lib/fray/src/fray/v2/iris_backend.py", line 563, in submit
job = self._iris.submit(
^^^^^^^^^^^^^^^^^^
File "/app/lib/iris/src/iris/client/client.py", line 661, in submit
canonical_id = self._cluster_client.submit_job(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/lib/iris/src/iris/cluster/client/remote_client.py", line 139, in submit_job
response = call_with_retry(f"launch_job({job_id})", _call)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/lib/iris/src/iris/rpc/errors.py", line 181, in call_with_retry
return retry_with_backoff(
^^^^^^^^^^^^^^^^^^^
File "/app/lib/rigging/src/rigging/timing.py", line 596, in retry_with_backoff
return call_fn()
^^^^^^^^^
File "/app/lib/iris/src/iris/cluster/client/remote_client.py", line 137, in _call
return self._client.launch_job(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/lib/iris/src/iris/rpc/controller_connect.py", line 1644, in launch_job
return self.execute_unary(
^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.11/site-packages/connectrpc/_client_sync.py", line 210, in execute_unary
return self._execute_unary(request, ctx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.11/site-packages/connectrpc/_client_sync.py", line 336, in _send_request_unary
raise ConnectError(Code.DEADLINE_EXCEEDED, "Request timed out") from e
connectrpc.errors.ConnectError: Request timed out
Traceback (most recent call last):
File "/app/.venv/lib/python3.11/site-packages/connectrpc/_client_sync.py", line 304, in _send_request_unary
resp = self._http_client.post(
^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: Request failed: error sending request for url (http://10.128.0.3:10000/iris.cluster.ControllerService/LaunchJob): operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/lib/marin/src/marin/execution/step_runner.py", line 153, in _harvest
status = handle.wait(raise_on_failure=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/lib/fray/src/fray/v2/local_backend.py", line 65, in wait
self._future.result(timeout=timeout)
File "/uv/cache/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/uv/cache/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/uv/cache/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/lib/marin/src/marin/execution/step_runner.py", line 281, in worker_fn
run_step(step)
File "/app/lib/marin/src/marin/execution/step_runner.py", line 328, in run_step
result = step.fn(output_path) # pyrefly: ignore[not-callable]
^^^^^^^^^^^^^^^^^^^^
File "/app/lib/marin/src/marin/execution/executor.py", line 717, in resolved_fn
return captured_fn(captured_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/experiments/grug/moe/launch.py", line 111, in run_grug_moe_trial
run_grug(run_config)
File "/app/experiments/grug/moe/train.py", line 553, in run_grug
dispatch_grug_training_run(
File "/app/experiments/grug/dispatch.py", line 53, in dispatch_grug_training_run
job = current_client().submit(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/lib/fray/src/fray/v2/iris_backend.py", line 563, in submit
job = self._iris.submit(
^^^^^^^^^^^^^^^^^^
File "/app/lib/iris/src/iris/client/client.py", line 661, in submit
canonical_id = self._cluster_client.submit_job(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/lib/iris/src/iris/cluster/client/remote_client.py", line 139, in submit_job
response = call_with_retry(f"launch_job({job_id})", _call)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/lib/iris/src/iris/rpc/errors.py", line 181, in call_with_retry
return retry_with_backoff(
^^^^^^^^^^^^^^^^^^^
File "/app/lib/rigging/src/rigging/timing.py", line 596, in retry_with_backoff
return call_fn()
^^^^^^^^^
File "/app/lib/iris/src/iris/cluster/client/remote_client.py", line 137, in _call
return self._client.launch_job(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/lib/iris/src/iris/rpc/controller_connect.py", line 1644, in launch_job
return self.execute_unary(
^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.11/site-packages/connectrpc/_client_sync.py", line 210, in execute_unary
return self._execute_unary(request, ctx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/.venv/lib/python3.11/site-packages/connectrpc/_client_sync.py", line 336, in _send_request_unary
raise ConnectError(Code.DEADLINE_EXCEEDED, "Request timed out") from e
connectrpc.errors.ConnectError: Request timed out
The above exception was the direct cause of the following exception:
RuntimeError: Step failed: grug/moe-v16-compute-opt-d1280-2.83e+19_cc655c3a
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/app/experiments/grug/moe/launch.py", line 174, in <module>
executor_main(
File "/app/.venv/lib/python3.11/site-packages/draccus/argparsing.py", line 228, in wrapper_inner
response = fn(cfg, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/lib/marin/src/marin/execution/executor.py", line 1638, in executor_main
executor.run(
File "/app/lib/marin/src/marin/execution/executor.py", line 1273, in run
StepRunner().run(
File "/app/lib/marin/src/marin/execution/step_runner.py", line 234, in run
raise RuntimeError(f"{len(failures)} step(s) failed") from failures[0]
RuntimeError: 1 step(s) failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment