oke-poc.md

OKE PoC — payload delivery across a network boundary

Federating a second HTCondor pilot pool — Kueue + job-gateway on Oracle's managed OKE — with a schedd/collector that lives somewhere else entirely.

Status (2026-08-14): working end to end. A job submitted on the VM is matched to a pilot running in OKE, its sandbox is transferred across the boundary, the payload executes on an Oracle Linux node, and stdout comes back:

=== WHERE AM I ===
hostname : pilot-dynamic-96c3a43f-n9jll
ip       : 10.244.1.34                        <- OKE pod network
kernel   : Linux 5.15.0-322.203.3.4.el8uek.x86_64   <- Oracle Linux UEK node
os       : "Ubuntu 22.04.5 LTS"                     <- pilot image
=== WORK ===
sum(1..1e6) = 500000500000
=== DONE ===

Update (2026-08-14, later): the production path — submit through the broker, not condor_submit — was verified separately. Broker.run(..., site="OKE_SITE1") now exists (a new --site preselection hint, checked alongside every other eligibility filter, never bypassing them) and OKE_SITE1 is registered with SCAPI, so it is a real, capability-checked candidate rather than a hand-picked PILOT_SITE requirement. Dispatch, matching, and placement all work identically to the direct-condor test above — preselection_trace shows OKE_SITE1 selected and the sibling sites rejected with not_requested_site when pinned. The payload itself then fails for a different, new reason: see Run-directory staging below. Also now live: GET /v1/sites/capacity on the broker, which shows OKE_SITE1 — budget, used, live pilot count, SCAPI ceiling — merged from the collector's own startd ads, with no new network path and no per-caller HTCondor credential. The netstudy notebook's live-capacity widget uses it.

Why this document exists

In the real deployment the services being simulated on this VM — collector, schedd, broker, IAM, storage — sit in another country and not on Oracle. Only the compute is on OKE. So the question this PoC answers for the Oracle operators is narrow and concrete:

What has to be open, and in which direction, for compute in OKE to run payloads dispatched from a service plane outside Oracle?

The short answer is at the top of the next section, and it is better news than expected.

---

The headline finding: nothing inbound to OKE

Every TCP connection in the working system is opened from OKE towards the service plane. Not one is opened towards OKE.

That means:

• OKE needs no inbound security-list rules, no public IPs on nodes, no load balancer, no NAT hairpin, and no VPN inbound path for the compute plane.
• The pilots can sit behind NAT on a private pod network (10.244.0.0/16 here) that is not routable from the outside at all — and it is not; we verified the service plane cannot reach it.
• All the firewall work lands on the service-plane side: three TCP ports reachable from OKE's egress addresses.

For the HTCondor traffic this is a direct consequence of how CCB (Condor Connection Broker) works, described in Why the direction inverts. For the global SRCNet services it simply falls out of everything being an outbound HTTPS client call.

flowchart LR
    subgraph OKE["OKE — Oracle · compute + site-local services"]
        pilot["pilot pod<br/>10.244.x.x<br/><b>no inbound needed</b>"]
        site["site-local services<br/><i>not yet deployed</i><br/>battle API · StoRM storage · registry"]
        pilot -. "③ storage, images,<br/>identity / stage-in<br/>stay inside the site" .-> site
    end
    subgraph SP["Service plane — elsewhere, not Oracle"]
        coll["collector :9618"]
        sched["schedd :9622"]
        glob["global SRCNet services :443<br/>IAM · SCAPI · DMAPI<br/>Rucio · broker"]
    end
    pilot -- "① register + advertise" --> coll
    pilot -- "② CCB reverse connect<br/>claim, payload, file transfer" --> sched
    pilot -- "④ token, metadata, data" --> glob

    classDef ok stroke:#2e9e4f,stroke-width:2px
    classDef future stroke:#888,stroke-width:2px,stroke-dasharray:5 5
    class pilot,coll,sched,glob ok
    class site future

Loading

Three ports cross the boundary — 9618, 9622, 443 — all outbound from OKE.

What does not cross it is anything site-local: the battle API (identity and stage-in), StoRM storage, and the container registry all run in OKE beside the compute. Storage especially — the point of an SRC is that the data is already at the site, so bulk I/O never touches the boundary. Only the global control-plane services (IAM, SCAPI, DMAPI, Rucio, broker) are remote by design, and they carry tokens and metadata rather than bytes.

---

Firewall rules required

Source is OKE's node subnet, not the pod CIDR: pods SNAT to their node's IP on the way out. In this PoC the three worker nodes are 10.20.12.196, 10.20.15.151, 10.20.15.75, covered by 10.20.12.0/22.

Inbound to the service plane (required)

Port	Proto	Purpose	Opened by
9618	TCP	Collector. Pilot registration, ClassAd advertising, CCB registration + request channel.	pilot → collector
9622	TCP	Schedd shared port. Every CCB reverse connection: REQUEST_CLAIM, ACTIVATE_CLAIM, shadow↔starter, sandbox file transfer.	pilot → schedd
443	TCP	Global SRCNet services — see the table below.	payload/pilot → services

Ports 9618/9622 are the HTCondor defaults for collector and (here) a dedicated schedd shared port. They are not magic — what matters is that the collector and the schedd each have one address reachable from OKE, and that the schedd advertises exactly that address. (In this simulation the 443 endpoints are served on 8443, because the VM's ingress is bound there.)

Which global services, and why

Every one of these was exercised by an authenticated payload running in OKE, and the list below is taken from the TLS SNI in the packet capture — not from guessing what the architecture ought to need.

Service	SNI seen	What the payload does with it	Target placement
IAM	iam.test	Obtain / exchange the OAuth token that authorises everything else	global
SCAPI	scapi.test	Site capabilities lookup	global
DMAPI	dmapi.test	Data management — resolve a DID to replicas	global
Rucio	rucio.test	Replica catalogue	global
Broker	broker.test	Job/task coordination	global
StoRM WebDAV	storm1.test, storm2.test	The actual bytes — GET/PUT	site-local
Registry	harbor.test	Container image pulls	site-local

The five marked global are the cross-boundary allow-list if operators prefer names to CIDRs. All ordinary outbound HTTPS, all carrying tokens and metadata rather than bulk data.

StoRM and the registry appear in the capture only because this site has neither yet, so the payload reached back for the VM's. In the target architecture both are site-local and their traffic never crosses the boundary — which matters most for StoRM, since that is the one flow that would otherwise carry the data volume.

Staging and identity are also site-local. The battle API — identity mapping and stage-in for the PREPARE_JOB hook — will run in OKE alongside the compute. It does not exist at this site yet.

Egress from OKE

The three ports above, outbound. OCI allows all egress by default; if that has been tightened, they need explicit egress rules and a working NAT gateway.

Inbound to OKE

None.

Observed traffic

Counters on the service-plane host after a full working run, confirming the only three ports OKE ever touched:

pkts bytes target  prot  source           destination
2585  155K ACCEPT  tcp   10.20.12.0/22    tcp dpt:9618   # collector
 247 14820 ACCEPT  tcp   10.20.12.0/22    tcp dpt:9622   # schedd (CCB reverse)
  43  2580 ACCEPT  tcp   10.20.12.0/22    tcp dpt:8443   # global services (443 here)
 786  162K REJECT  all   0.0.0.0/0                       # default deny

A separate packet capture of an authenticated payload running in OKE — full chain of IAM token → SCAPI → DMAPI → StoRM WebDAV PUT/GET round-trip — counted every connection it opened:

  32  ->  10.0.0.90:8443    global services (TLS)
  11  ->  10.0.0.90:9618    collector
   0  ->  anything else

Nothing outside the service plane, and nothing inbound. (That capture predates the schedd path, which is why 9622 does not appear in it.)

Two firewalls must agree. On Oracle, a host iptables rule alone is not enough — the OCI security list for the service-plane subnet must carry the matching ingress rule or packets are silently dropped before they reach the host. Every port above appears in both. Debugging one while forgetting the other burns a lot of time, because the failure mode is identical to an application hang.

---

Why the direction inverts

CCB is usually described as "relaying" connections. It does not relay. It reverses them.

When daemon A wants to talk to daemon B, and B is behind NAT, A does not connect to B. A asks the collector's CCB server to pass a message down B's existing outbound registration channel saying "connect back to me at this address". B then dials A.

sequenceDiagram
    autonumber
    participant P as pilot startd<br/>(OKE, NAT'd)
    participant C as collector CCB<br/>(:9618)
    participant S as schedd / shadow<br/>(:9622)

    Note over P,C: at pilot start-up
    P->>C: register, keep channel open (ccbid 7)
    P->>C: advertise ClassAd (slot, PILOT_SITE)

    Note over S,C: job matched to this pilot
    S->>C: CCB_REQUEST → ccbid 7,<br/>"connect back to <10.0.0.90:9622?sock=schedd_…>"
    C-->>P: push request down the open channel
    P->>S: dial 10.0.0.90:9622  ← the reversal
    S->>P: REQUEST_CLAIM over that connection

    Note over S,C: shadow starts, needs the startd
    S->>C: CCB_REQUEST → ccbid 7,<br/>"connect back to <10.0.0.90:9622?sock=shadow_…>"
    C-->>P: push request
    P->>S: dial 10.0.0.90:9622
    S->>P: ACTIVATE_CLAIM, sandbox transfer, payload runs

Loading

Two consequences that drive the whole design:

1. The NAT'd side never needs to be reachable. Hence no inbound to OKE.
2. The reachable side must be genuinely reachable at the address it advertises, from OKE and from inside its own cluster — because the same ClassAd is handed to both. This is why 10.0.0.90:9622 has to work as a hairpin from local pods too.

---

Configuration that this depends on

Four settings, each of which produced a distinct and misleading failure when wrong.

1. The schedd must NOT set CCB_ADDRESS

This is the one that cost the most. The schedd is directly reachable, so it has no reason to register with CCB — and registering actively breaks the boundary crossing, because every address the submit side then hands out carries a CCBID whose value is the collector's in-cluster address:

<10.0.0.90:9622?CCBID=10.42.0.240:9618...&sock=shadow_28_7a7a_7>
                       ^^^^^^^^^^^^^^^^ unroutable from OKE

The pilot has to dial that address. Seeing a CCBID on it, it tries to reach the CCB server at 10.42.0.240:9618 and never falls back to the perfectly reachable 10.0.0.90:9622:

Failed to send CCB_REQUEST to collector 10.42.0.240:9618...:
  SECMAN:2003:TCP connection to collector ... failed.
CCBClient: no more CCB servers to try ...; giving up.
CCBListener: failed to create reversed connection for request id 370

The symptom is deceptive. The schedd's own REQUEST_CLAIM succeeds (that reversal goes the other way, and its CCBID is never dialled by anything off-cluster), so jobs match, claim, and transfer files — then die at ACTIVATE_CLAIM, which reads like a startd fault:

CCBClient: Timed out waiting for response after requesting reversed
  connection from startd slot1@oke-pilot-… ccbid 7
DCStartd::activateClaim: Failed to send command ACTIVATE_CLAIM to the startd

It also breaks local submission, because condor_submit/condor_release resolve the schedd through the collector, see the CCBID, and prefer a CCB reversal that times out — ERROR: Failed to connect to local queue manager.

The evidence is only visible in the pilot's SharedPortLog, not the startd's log; the shared-port daemon owns the CCB listener.

2. PRIVATE_NETWORK_NAME — opposite settings on each side

Side	Value	Why
schedd	blank (PRIVATE_NETWORK_NAME =)	Omitting it is not enough — it defaults to $(FULL_HOSTNAME), so the schedd advertises PrivNet=<pod-name>, CCB sees two mismatched private networks and refuses to relay. Assigning empty removes it from the ad and the schedd is treated as public.
pilot	set (PRIVATE_NETWORK_NAME = oke-pilot)	The pilot's own master and startd live in the same pod and must talk directly. Blanking it makes them attempt CCB — which needs the collector's in-cluster address — and the startd exits.

A mismatch between the two logs a warning that is not fatal and can be ignored:

CCBClient: WARNING: ... this appears to be a connection from one private
network to another, which is not supported by CCB. ... Assuming the latter.

3. TCP_FORWARDING_PORT is read but not applied under SHARED_PORT

With shared port in use, the advertised port comes from the shared-port daemon, so TCP_FORWARDING_PORT is silently ignored — the schedd kept advertising :9618, which is the collector's forward, sending pilots to the wrong daemon.

The fix is to make the real port match rather than translate it: set SHARED_PORT_PORT = 9622. That drags in two more changes that will otherwise stall the rollout — the Service targetPort, and the container port, whose readiness probe references it by name.

submit:
  extraConfig: |
    PRIVATE_NETWORK_NAME =
    SHARED_PORT_PORT     = 9622
    TCP_FORWARDING_HOST  = 10.0.0.90     # the address pilots must reach

4. Pilot config must ride in the file the provisioner actually stages

The chart's gateway.pilotExtraConfig rendered into a separate 98-extra.conf ConfigMap key. Both provisioners stage exactly one file by name — the Kubernetes one subPath-mounts 99-pilot.conf, the Slurm one binds a single staged 99-pilot.conf — so the extra key was rendered correctly and then never delivered. Off-cluster pilots came up with no CCB_ADDRESS, advertised their unroutable pod IP, matched jobs, and failed every shadow connection.

Fixed by appending pilotExtraConfig into 99-pilot.conf (charts/job-gateway/templates/pilot-configmap.yaml), which fixes both provisioners at once.

---

Full topology

flowchart TB
    subgraph OKE["OKE — Oracle Cloud · compute only · NO INBOUND"]
        direction TB
        gwoke["job-gateway-oke<br/>SITE_ID=OKE_SITE1<br/>CONDOR_HOST=10.0.0.90"]
        pjob["pilot Job (batch/v1)<br/>label kueue.x-k8s.io/queue-name=pilots<br/>suspend=true"]
        kueue["Kueue controller<br/>LocalQueue pilots<br/>→ ClusterQueue pilot-pool-oke"]
        pilotoke["pilot-dynamic-* pods<br/>10.244.x.x (private, NAT'd)<br/>PRIVATE_NETWORK_NAME=oke-pilot<br/>CCB_ADDRESS=10.0.0.90"]
        sitesvc["site-local services<br/><i>not yet deployed</i><br/>battle API (stage-in/out, identity)<br/>StoRM storage · registry mirror"]

        gwoke -- "① create suspended Job" --> pjob
        pjob -. "② Workload admitted<br/>against quota" .-> kueue
        kueue -- "③ unsuspend" --> pjob
        pjob -- "④ Job controller<br/>creates pod" --> pilotoke
        gwoke -. "reads quota<br/>(capacity reporting only)" .-> kueue
        pilotoke -. "payload I/O + PREPARE_JOB hook<br/>never leaves the site" .-> sitesvc
    end

    subgraph EDGE["OKE egress · SNAT to node IPs 10.20.12.0/22"]
        nat["NAT gateway"]
    end

    subgraph FW["Service-plane firewall — BOTH layers must agree"]
        direction TB
        oci["OCI security list<br/>ingress <b>9618, 9622, 443</b><br/>(443 served on 8443 here)<br/>from 10.20.12.0/22"]
        ipt["host iptables INPUT<br/>same three rules<br/>(default REJECT)"]
        oci --> ipt
    end

    subgraph VM["Service plane — simulated here on the VM"]
        direction TB
        s618["socat 10.0.0.90:9618<br/>→ NodePort 30618"]
        s622["socat 10.0.0.90:9622<br/>→ NodePort 30622"]
        cmext["Service htcondor-cm-external"]
        smext["Service htcondor-submit-external"]
        cm["htcondor-cm · collector + CCB server"]
        submit["htcondor-submit · schedd<br/>SHARED_PORT_PORT=9622<br/>TCP_FORWARDING_HOST=10.0.0.90<br/>PRIVATE_NETWORK_NAME= (blank)<br/><b>no CCB_ADDRESS</b>"]
        exec1["htcondor-execute x2"]
        broker["broker"]
        ingress["ingress :8443 (stands in for :443)<br/>IAM · SCAPI · DMAPI · Rucio · broker<br/><i>+ StoRM &amp; registry until the site has its own</i>"]

        s618 --> cmext ---> cm
        s622 --> smext --> submit
        submit <--> cm
        exec1 <--> cm
        broker -- "submits run leaders" --> submit
    end

    pilotoke --> nat
    gwoke --> nat
    nat --> oci
    ipt --> s618
    ipt --> s622
    ipt --> ingress

    classDef ok stroke:#2e9e4f,stroke-width:2px
    classDef future stroke:#888,stroke-width:2px,stroke-dasharray:5 5
    class pilotoke,submit,cm,ingress ok
    class sitesvc future

Loading

How Kueue actually gates a pilot

Kueue is not an API the gateway submits to, and it does not create pods. The gateway creates an ordinary batch/v1 Job, carrying the label kueue.x-k8s.io/queue-name: pilots and spec.suspend: true. Kueue watches for that label, builds a Workload from the Job, admits it against the ClusterQueue quota, and flips suspend off — at which point the Job controller, not Kueue, creates the pilot pod.

Consequences worth knowing:

• The gateway's only direct call to Kueue is read-only, to report free capacity. Everything else is mediated by a label on a Job.
• suspend is set only when gated (suspend: gated in the provisioner). A suspended Job with no queue label would hang forever, since Kueue runs with manageJobsWithoutQueueName=false. With no LocalQueue configured, pilots are ungated and start immediately with Kueue not involved at all.
• The pool's total size is the ClusterQueue quota. A pilot beyond the free quota sits suspended with zero pods until quota frees.
• A pilot sized larger than the whole ClusterQueue can never be admitted and stays suspended forever — a slot leak, which is why the provisioner reaps Jobs that have been suspended without ever starting.

Flow table

Flow	Path	Direction
pilot → collector	10.0.0.90:9618 → socat → NodePort 30618 → htcondor-cm	out of OKE
pilot → schedd (CCB reverse)	10.0.0.90:9622 → socat → NodePort 30622 → htcondor-submit	out of OKE
gateway → schedd/collector	same two addresses	out of OKE
payload → global services	:443 — IAM, SCAPI, DMAPI, Rucio, broker	out of OKE
payload → storage / registry	site-local StoRM + registry mirror (not yet deployed)	stays in OKE
pilot → staging/identity	site-local service in OKE (not yet deployed)	stays in OKE
in-cluster clients → schedd	the same 10.0.0.90:9622, hairpinned	local
anything → pilot	never happens	—

The hairpin

Local pods reach the schedd at the same external 10.0.0.90:9622 the OKE pilots use, because the collector hands everyone one ClassAd. This works — pods route to 10.0.0.90 fine. What blocks it is the host firewall's default REJECT catching traffic from the k3d bridge subnet, so that needs its own rule:

sudo iptables -I INPUT -p tcp -s 172.31.0.0/24 --dport 9622 -m state --state NEW -j ACCEPT
sudo iptables -I INPUT -p tcp -s 172.31.0.0/24 --dport 9618 -m state --state NEW -j ACCEPT

Confirmed by the REJECT counter incrementing on a pod connect, and by tcpdump showing the SYNs arriving from 172.31.0.2. One advertised address therefore serves in-cluster and off-cluster clients alike.

If the schedd advertises a port with no listener behind it, local submission breaks too — everything resolves the schedd through the collector, so the advertised address must be live for in-cluster clients as well.

---

Reproducing the end-to-end run

# 1. Confirm the pilot is in the pool
condor_status -af Name State Activity PILOT_SITE | grep OKE_SITE1
# slot1@pilot-dynamic-96c3a43f-n9jll Unclaimed Idle OKE_SITE1

# 2. Submit a job that can only run there
cat > oke-e2e.sub <<'EOS'
executable              = payload.sh
output                  = out.$(ClusterId).$(ProcId)
error                   = err.$(ClusterId).$(ProcId)
log                     = e2e.log
requirements            = (PILOT_SITE == "OKE_SITE1")
should_transfer_files   = YES
when_to_transfer_output = ON_EXIT
queue 1
EOS
condor_submit oke-e2e.sub

payload.sh just prints hostname, hostname -i, uname -sr and does a trivial sum — enough to prove where it ran.

Submit as a real pool user, not root: the schedd refuses condor@password with ERROR: Failed to create new User record ... The given user is not allowed to own jobs.

Where to look when it stalls

Symptom	Look at
Job Idle, negotiator says "1 jobs matched" every cycle	Schedd can't reach the startd — check the pilot's SharedPortLog for CCBListener: failed to create reversed connection
Job Idle, -better-analyze says 0 slots match	Matchmaking, not networking — check PILOT_SITE on the slot
Claim succeeds then ACTIVATE_CLAIM fails	The CCB_ADDRESS-on-the-schedd trap above
Failed to connect to local queue manager	Same trap, seen from the local side
Failed communication with collector	Stale pool password / IDTOKEN, or the socat forward is dead

condor_ping is the fastest way to separate network from HTCondor problems — run it from the pilot, against the schedd's advertised address:

condor_ping -addr '<10.0.0.90:9622?addrs=10.0.0.90-9622&noUDP&sock=schedd_25_b9f5>' READ
# READ command ... succeeded as unauthenticated@unmapped to daemon at <10.0.0.90:9622>.

A bare TCP check is not sufficient here: socat accepts the client connection before dialling its downstream, so a successful connect proves nothing about the rest of the chain.

---

Two compute pools in SCAPI

OKE_SITE1 originally advertised one compute block — OKE_SITE1_COMPUTE1, ceiling 8 vCPU / 16384 MiB per pilot, matching pool1's node size. A second block was registered for a bigger pilot shape:

OKE_SITE1_COMPUTE1   kubernetes   8 vCPU  / 16384  MiB
OKE_SITE1_COMPUTE2   kubernetes   36 vCPU / 163840 MiB   <- new

via the same ska_src_site_capabilities_api.client.integration CLI used to register the site itself (register_compute + register_backend, no register_service needed a second time — one job_gateway service per site is enough; _site_has_job_gateway only checks that some compute on the site has one).

This is a capability record, not a placement rule. The broker doesn't route a job to a specific compute block — _aggregate_pilot_cap takes the max across all of a site's compute blocks for a given backend. So this registration raises OKE_SITE1's advertised kubernetes ceiling from 8/16GB to 36/160GB site-wide, rather than adding a separately-selectable pool. Verified functionally: a job requesting 20 vCPU / 40GB pinned to OKE_SITE1 — which the old 8/16GB ceiling rejected outright with exceeds_site_capacity — now clears preselection (candidate_sites: ["OKE_SITE1"]).

At the time this was registered, no real hardware backed the bigger number — see Second compute pool below for the node pool that now does.

---

Second compute pool: large pilots with Cluster Autoscaler

The original pool1 (3 × VM.Standard.E5.Flex, 4 OCPU/32GB fixed size) only ever fits the small pilot shape this PoC started with. A second, larger pool was added for bigger jobs — scaling from zero rather than sitting there paid for idle:

	pool1 (existing)	pool2-large (new)
Shape	VM.Standard.E5.Flex	VM.Standard.E5.Flex
Size per node	4 OCPU (8 vCPU) / 32 GB	18 OCPU (36 vCPU) / 160 GB
Node count	fixed 3	autoscaled 0 → 10
OCID	...ctkla4a	...xjdroeha

"36 core" here means vCPU — the unit Kubernetes/HTCondor actually see, matching the SCAPI OKE_SITE1_COMPUTE2 registration in Two compute pools in SCAPI above — which is 18 OCPU on the OCI side, the same 1 OCPU : 2 vCPU ratio pool1 already uses.

Scale-to-zero via the OCI Cluster Autoscaler

pool2-large was created at size: 0 and is managed by the Kubernetes Cluster Autoscaler's OCI cloud provider, deployed into kube-system with instance-principal auth (OCI_USE_INSTANCE_PRINCIPAL=true — no credential file, no secret):

containers:
  - image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.36.1  # matches this cluster's k8s version
    command:
      - ./cluster-autoscaler
      - --cloud-provider=oci
      - --nodes=0:10:ocid1.nodepool.oc1.ap-sydney-1.aaaaaaaagpdxzu2sk647crfvohfibmme5ouzckcpvnzn3epfonv4xjdroeha
      - --scale-down-delay-after-add=10m
      - --scale-down-unneeded-time=10m
    env:
      - name: OCI_USE_INSTANCE_PRINCIPAL
        value: "true"

No new IAM policy was needed. The upstream docs call for a dynamic group plus a policy granting manage cluster-node-pools (+ instance/network/volume permissions) to the nodes that will run the autoscaler pod. This cluster already had exactly that — rookce-oke-cluster-nodes-acoq (matching every instance in this compartment) already carries manage cluster-node-pools, manage instance-family, use subnets, use vnics, use private-ips, and inspect compartments, because the existing CSI/CCM setup needed the same shape of access. Confirmed from a cold start — no auth errors, node pool found immediately:

oci_manager.go:105  using instance principal provider
oci_manager.go:286  static node spec constructed: id:...xjdroeha minSize:0 maxSize:10
oci_manager.go:452  Refreshed NodePool list, next refresh after ...

One RBAC gap in the upstream example manifest, not an OCI/IAM issue: the ClusterRole it ships doesn't grant watch/list/get on volumeattachments.storage.k8s.io (a newer resource type the example predates). Harmless — the autoscaler keeps running — but noisy (reflector.go:227 "Failed to watch" ... forbidden) until patched:

kubectl patch clusterrole cluster-autoscaler --type=json -p='[{"op":"add",
  "path":"/rules/-","value":{"apiGroups":["storage.k8s.io"],
  "resources":["volumeattachments"],"verbs":["watch","list","get"]}}]'

What "10" doesn't mean

The 10 here is a real ceiling: the autoscaler will actually launch up to 10 real VM.Standard.E5.Flex (18 OCPU/160GB) nodes if enough oversized pending work demands it, at real OCI cost. It is a separate number from the OKE_SITE1_COMPUTE2 SCAPI registration (Two compute pools in SCAPI), which only tells the broker "a pilot up to 36 vCPU/160GB can fit here" — SCAPI has no field for pool depth. The two have to be kept in sync by hand: SCAPI's ceiling says what size job the broker will offer this site; the node pool's max says how many of that size OKE can actually deliver before pilots start Kueue-suspending.

---

Production gaps

These are not networking problems, but they stand between this PoC and a real off-site pilot.

Identity and stage-in — site-local services that do not exist yet. The pilot's PREPARE_JOB hook (start_payload.sh) is fail-closed on BATTLE_API_BASE and performs identity mapping and stage-in against the battle API. In the target architecture that API is site-local: it runs in OKE beside the pilots, so the hook never crosses the boundary and adds no firewall rule. It has not been deployed at this site yet, so the PoC pilots have no battle API to talk to and the hook is disabled (STARTER_JOB_HOOK_KEYWORD =), letting payloads run directly.

This is only about the battle API. It does not reduce the 443 requirement — the payload still reaches IAM, SCAPI, DMAPI, Rucio, StoRM and the registry across the boundary, and that traffic is already proven (see Which global services). Deploying the site-local staging and identity services in OKE is the remaining work; it keeps the hook's traffic inside the site rather than adding a rule.

The PoC did confirm a pilot can reach the VM's battle API across the boundary (HTTP/2 200 via the ingress), which is a useful fallback to know about but is not the design.

The socat forwards are not persistent. Both 9618 and 9622 are plain socat processes that die on reboot, and 9622 is now load-bearing for local submission too. They need systemd units:

socat TCP-LISTEN:9618,bind=10.0.0.90,fork,reuseaddr TCP:172.31.0.2:30618
socat TCP-LISTEN:9622,bind=10.0.0.90,fork,reuseaddr TCP:172.31.0.2:30622

In a real deployment these disappear entirely — the collector and schedd are reachable services in their own right, and the forwards exist only because the service plane is being simulated inside a k3d cluster on one VM.

Secrets are a point-in-time copy. htcondor-pool-password and htcondor-gateway-idtoken in job-gateway-oke were copied by hand from the local Vault-synced values. They are not synced. Any pool-password rotation stales them, and the failure looks exactly like a network fault (Failed communication with collector). Re-copy after any remint:

kubectl create secret generic htcondor-pool-password -n job-gateway-oke \
  --from-file=pool_password=<current-value> --dry-run=client -o yaml | kubectl apply -f -
kubectl create secret generic htcondor-gateway-idtoken -n job-gateway-oke \
  --from-file=gateway.token=<current-READ_IDTOKEN> --dry-run=client -o yaml | kubectl apply -f -

Order matters when rotating: write the new password, force the ExternalSecret to resync, restart collector → schedd → execute in that order, then re-mint tokens, then restart consumers. Minting tokens before the daemons have the new password orphans them.

Credential staleness generally. Any long-running process that reads an IAM-issued or pool credential once at start-up needs a rollout restart after that credential rotates. PAPI, JupyterHub's hub, ska-src-auth-api, the broker's OAuth exchange and the HTCondor pool password have all hit this after IAM/Vault resets in dev.

Still open

• Gateway placement. job-gateway-oke runs on OKE and reaches the schedd fine now. The alternative — run the gateway beside the schedd and have it provision pilots into OKE — needs a patch to kubernetes_provisioner.py, which hard-codes load_incluster_config(), plus a way to give pilots a different CONDOR_HOST from the gateway's own. A local pod can already reach the OKE API server, so it is feasible; it is no longer necessary.
• Site-local services. None of the battle API, StoRM storage, or a registry mirror is deployed in OKE yet. Until the battle API is, the PREPARE_JOB hook stays disabled on these pilots and payloads run without identity mapping or stage-in; until StoRM and the registry are, payload I/O and image pulls cross the boundary that they are not supposed to cross.
• Multi-site parallel execution. One job on OKE works. Fanning a single logical job across OKE and local pilots simultaneously is the next test.
• Run-directory staging assumes shared storage that doesn't exist across the boundary. Submitting through the broker (not condor_submit directly) and pinning to OKE_SITE1 with the new --site hint works end to end — the job dispatches, matches, and the run leader gets as far as invoking the payload — but the payload script itself fails (sh exit 2). The leader stages task-N.sh/task-N.json under /srv/storage/.ska-leader/runs/... and the script's first line redirects its own output back to that same path, assuming the pilot mounts the identical filesystem the leader does. True for every local pilot (same k3d hostPath); false for OKE, whose /srv/storage is its own node-local XFS volume with no such run directory. This is a sibling gap to the site-local services above — the fix is the same shape (a site-local staging path, or object storage, rather than a POSIX path assumed shared with the VM) — but it blocks payload output specifically, not just identity/data.
• The STARTER_JOB_HOOK_KEYWORD blanking is currently applied to a live pilot by hand — done: it now lives in the job-gateway-oke Helm values (pilotExtraConfig) and survives pilot turnover. Verified: an old pilot with the hook still active reproduced the original BATTLE_API_BASE failure; a fresh pilot spawned after the Helm upgrade had STARTER_JOB_HOOK_KEYWORD correctly unset. Note for next time: the pilot ConfigMap is subPath-mounted, which Kubernetes never live-updates — an already-running pilot needs to be replaced (kubectl delete pod), not just condor_reconfig'd, to pick up a ConfigMap change.