dynamic_slurm_pilot_issues.md

Dynamic Slurm Pilot Issues

Found 2026-08-21 while validating the SWF-22 Rapthor Marimo demo (swf22_rapthor_broker_demo.py) against pilot_resource=slurm on a long-lived dev cluster (deploy-ubuntu, ~9 days uptime at the time). None of these are specific to that notebook — they block any job requesting pilot_resource=slurm on an affected cluster. SWF-22's default was changed to kubernetes to work around this; see its lib module for the pointer back here.

1. Stale/zombie dynamic Slurm node blocks all pilot dispatch for a site

Symptom: every pilot_resource=slurm job sits at WAITING_REDISPATCH / SUBMITTED indefinitely. squeue shows a job in CG (COMPLETING) state that never clears, with a RunTime of days, on a node whose EndTime (per scontrol show job <id>) already passed — sometimes by several days.

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
 4272   compute pilot-dy     root CG 6-07:38:33      1 compute-1

scontrol show node <node> reports:

State=IDLE+COMPLETING+DYNAMIC_NORM+NOT_RESPONDING ... SlurmdStartTime=None

NOT_RESPONDING + SlurmdStartTime=None indicates the dynamic node's slurmd registration is dead (most likely because the underlying slurm-worker-compute-N pod restarted at some point and slurmctld never cleared the old registration for it), and the completing pilot job is stuck because it can't get an epilog confirmation from a node that no longer exists.

Impact: since the dynamic pilot dispatcher appears to treat "a non-terminal pilot job already exists for this site" as "don't submit another," a single stuck zombie job can block all new Slurm pilot dispatch for that site indefinitely, even when other nodes in the same partition are healthy and idle.

Workaround (clears in seconds, no cluster restart needed):

kubectl exec -n slurm slurm-controller-0 -c slurmctld -- \
  scontrol update NodeName=<node> State=DOWN Reason="stale-dynamic-node-cleanup"
kubectl exec -n slurm slurm-controller-0 -c slurmctld -- \
  scontrol delete NodeName=<node>

scancel <jobid> alone is not sufficient — the job stays wedged in CG because slurmctld is waiting on the (dead) node's epilog, not on the job itself. Marking the node DOWN releases that wait; scontrol delete then removes the stale dynamic registration entirely so the pod can re-register cleanly the next time it's needed.

Real fix (not done here): figure out why slurm-worker-compute-N pod restarts don't clean up / re-register their own dynamic node entry, and whether the dynamic pilot dispatcher should treat a NOT_RESPONDING node's job as already-dead rather than blocking on it.

2. Slurm pilot IDTOKEN path mismatch (job-gateway vs. Slurm worker view of /srv/storage)

Symptom: every dynamically-dispatched Slurm pilot fails within ~15s of being submitted (see job-gateway-1's logs: Reaped slurm pilot ... job <id> (failed), retried every ~15s in a loop). The pilot's own stderr (/srv/storage/slurm-pilot/<SITE>/pilot-dynamic-slurm-<SITE>-<hash>.err) shows:

WARNING: skipping mount of /srv/storage/slurm-pilot/<SITE>/idtoken: stat ...: no such file or directory
FATAL:   container creation failed: mount hook function failure: mount /srv/storage/slurm-pilot/<SITE>/idtoken->/root/secrets/idtoken error: ... doesn't exist

Root cause: job-gateway-1's pod mounts a namespaced site-scratch PVC at /srv/storage (see its values.yaml's sharedStoragePvcs) — an isolated, per-gateway view. When the gateway writes the pilot's HTCondor IDTOKEN to what it thinks is /srv/storage/slurm-pilot/<SITE>/idtoken, that write actually lands on the PVC's backing hostPath (on this dev cluster: out/<cluster>/volumes/site-scratch-1/slurm-pilot/<SITE>/idtoken).

Slurm worker nodes are plain OS processes, not Kubernetes pods — they see /srv/storage as the raw k3d-wide bind mount (bin/cluster.sh's -v .../volumes:/srv/storage@all), i.e. out/<cluster>/volumes/slurm-pilot/<SITE>/idtoken (no site-scratch-1/ prefix). Same nominal path, two different physical locations — the token the gateway wrote is invisible to the Slurm pilot's apptainer bind-mount.

Workaround (until the write path is fixed): copy the token to where Slurm pilots actually look:

cp out/<cluster>/volumes/site-scratch-1/slurm-pilot/<SITE>/idtoken \
   out/<cluster>/volumes/slurm-pilot/<SITE>/idtoken

Real fix (not done here): either have the gateway write the Slurm pilot IDTOKEN to the raw /srv/storage hostPath (bypassing the namespaced site-scratch PVC for this one file), or have Slurm's pilot launch mount the token from wherever the gateway's PVC actually resolves to. Whoever owns ska_src_ef_job_gateway.clients.slurm_provisioner should confirm which is intended.

3. pilot-entrypoint.sh invokes kubectl inside a Slurm/apptainer pilot

Symptom: once issues #1 and #2 above are worked around, the Slurm pilot's apptainer container actually starts (SIF cached, scratch dir created), but then immediately fails:

[entrypoint] Created PVC/scratch directory at /srv/storage/pilot-scratch-1/pilot-dynamic-slurm-<SITE>-<hash> (0711)
/usr/local/bin/pilot-entrypoint.sh: line 140: environment: command not found
/usr/local/bin/pilot-entrypoint.sh: line 140: kubectl: command not found
/usr/local/bin/pilot-entrypoint.sh: line 140: pilot.sh: command not found

This is the actual blocker for Slurm pilots right now, after #1 and #2 are cleared. kubectl has no reason to run inside a bare Slurm/apptainer pilot at all — this smells like a branch-selection bug in the shared pilot-entrypoint.sh (used by both kubernetes and Slurm pilots) picking the kubernetes-style bootstrap path regardless of how the pilot was actually dispatched, or a wrapper-command variable that's unset/empty in the Slurm path, causing the following literal words (environment, kubectl, pilot.sh) to each be attempted as bare commands instead of args to one.

Not investigated further — this is inside the shared htcondor-pilot image's entrypoint script, used by every pilot regardless of backend, and is a different repo/component than whatever surfaced #1/#2. Kubernetes pilots do not hit this (confirmed working in the same cluster, same session).

Workaround: use pilot_resource=kubernetes until this is fixed.

Summary

#	Symptom	Workaround	Real fix owner
1	All Slurm dispatch stuck at WAITING_REDISPATCH/SUBMITTED	scontrol update NodeName=... State=DOWN then scontrol delete NodeName=...	Slurm dynamic node lifecycle / dynamic pilot dispatcher
2	Pilot FATALs on missing idtoken mount, reaped every ~15s	Copy idtoken from the gateway's PVC-backed path to the raw hostPath path	ska_src_ef_job_gateway.clients.slurm_provisioner
3	Pilot starts, then kubectl: command not found	Use pilot_resource=kubernetes	Shared htcondor-pilot image's pilot-entrypoint.sh