d3v-null

Dynamic Slurm Pilot Issues

Found 2026-08-21 while validating the SWF-22 Rapthor Marimo demo (swf22_rapthor_broker_demo.py) against pilot_resource=slurm on a long-lived dev cluster (deploy-ubuntu, ~9 days uptime at the time). None of these are specific to that notebook — they block any job requesting pilot_resource=slurm on an affected cluster. SWF-22's default was changed to kubernetes to work around this; see its lib module for the pointer back here.

d3v-null

Switching the SRCNet demos over to Oracle OKE

About this gist. Gists are flat, so the companion files below are prefixed (script-, manifest-, benchmon-) copies of the repository tree at docs/oke-demo/{scripts,manifests,benchmon}/ on the oke-demo-2026-08-27 branch of ska-src-api-deployment-stack. A path written as docs/oke-demo/scripts/remove-site1-ms.sh below is script-remove-site1-ms.sh here.

d3v-null

OKE PoC — payload delivery across a network boundary

Federating a second HTCondor pilot pool — Kueue + job-gateway on Oracle's managed OKE — with a schedd/collector that lives somewhere else entirely.

Status (2026-08-14): working end to end. A job submitted on the VM is matched to a pilot running in OKE, its sandbox is transferred across the boundary, the payload executes on an Oracle Linux node, and stdout comes back:

d3v-null

Federated Execution (Cool) Stack — Privileged-Access Security Audit (Oracle K8s + future Pawsey Slurm)

Architecture in one paragraph

A client POSTs a job to the Computing Broker, which authorizes via PAPI, exchanges tokens via AAPI, and intersects SCAPI (site capability) with DMAPI (data location) into a candidate set of sites — it never picks the final site itself. The broker submits exactly one run leader (a condor-native local-universe job — this replaced the old Toil/WES stack entirely) to the central HTCondor pool. The run leader fans the job out as a vanilla-universe payload cluster. Independently, each site's Job Gateway is a pure pilot factory: it spawns pilot pods (Kubernetes, Kueue-gated) or Slurm jobs (via slurmrestd + Apptainer) sized to idle demand. When a pilot joins the pool, HTCondor's negotiator matches it to an idle task — this match, not the broker or the gateway, is what decides where a job actually runs ("late binding"). The pilot then runs the payload u

d3v-null

Rapthor + HTCondor/GlideinWMS: Architecture, Late-Binding, and Integration Options

Technical analysis for radio astronomers and SKA/SRCNet engineers Date: May 2026

---

1. Background: The Two Systems

1.1 Rapthor

d3v-null

This didn't work but I'm documenting it here in case I want to try it again.

take calibration solutions, fit linear model, update cable lengths in metafits before a second round of preprocessing.

singularity shell --cleanenv --bind /data/curtin_mwaeor/src/mwa-demo/demo:/demo --bind $PWD --workdir $PWD /data/curtin_mwaeor/singularity/mwa-demo_autos_cuda12.5.1.sif
/demo/82_calfit.py \
    --metafits /data/curtin_mwaeor/dev/1274010302/raw/1274010302.metafits \
    --solns /data/curtin_mwaeor/dev/1274010302/hyp_soln_1274010302_kr_75-1667l_src1k_500it_sdc50_el36.692.fits \
 --name hyp_soln --out-dir /data/curtin_mwaeor/dev/1274010302/ \

d3v-null

Rapthor + PanDA WMS: Architecture, Late-Binding, and Integration Options

Technical analysis for radio astronomers and SKA/SRCNet engineers Date: May 2026

---

1. Background: The Two Systems

1.1 Rapthor

d3v-null

d3v-null

from astropy.io import fits
from pprint import pprint
import numpy as np
from sys import argv
from os.path import exists

path = argv[-1]
if not exists(path):
    print(f"""Usage: {argv[0]} <metafits>
          File {path} does not exist.""")

d3v-null