gistfile1.txt

  Root Cause Analysis: OIDC Listener TCP Accept Delays

  The Smoking Gun: Processor Thread Starvation via Synchronous HTTP

  There is a clear, plausible code-level explanation for why only the OIDC listener experiences TCP accept delays while the Kerberos listener on the same broker is
  unaffected. The issue is a cascading backpressure chain caused by synchronous HTTP calls during SASL/OAUTHBEARER authentication.

  The Chain of Events

  Here's the exact code path that causes the problem:

  1. Authentication happens on the Processor (network) thread

  In Selector.java:546-547, during every poll() cycle, for every channel that isn't yet authenticated:
  if (channel.isConnected() && !channel.ready()) {
      channel.prepare();  // ← this calls authenticator.authenticate() SYNCHRONOUSLY

  This runs on the Processor thread — there is no offloading to a separate thread pool.

  2. For OAUTHBEARER, evaluateResponse() calls the Strimzi callback handler

  In OAuthBearerSaslServer.java:155-156:
  callbackHandler.handle(new Callback[] {callback});

  This invokes JaasServerOauthValidatorCallbackHandler.handle() → validateToken() → validator.validate(token).

  3. If using introspection validation, this makes a synchronous HTTP POST

  In OAuthIntrospectionValidator.java:303-314:
  response = HttpUtil.doWithRetries(retries, retryPauseMillis, ..., () ->
      post(introspectionURI, ...)  // ← SYNCHRONOUS HTTP call on the Processor thread
  );

  Even with JWKS-based validation (local JWT verification), initial JWKS fetches or refreshes could stall. But the introspection path is the worst case — every single
   token validation blocks the Processor thread waiting for an HTTP response from the OIDC provider.

  4. If retries are configured, blocking is multiplied with Thread.sleep()

  In HttpUtil.java:74-77:
  if (i > 1 && retryPauseMillis > 0) {
      Thread.sleep(retryPauseMillis);  // ← sleeps the Processor thread!
  }

  5. The HTTP timeouts are 60 seconds by default

  From HttpUtil.java:39: connect timeout and read timeout default to 60 seconds each.

  6. JDK's HttpURLConnection pool is limited to ~5 connections per destination

  From the comment in HttpUtil.java:33-37: The internal JDK connection pool per destination defaults to 5 (http.maxConnections). If you have num.network.threads=8 (8
  Processor threads) all trying to call the same OIDC introspection endpoint simultaneously, 3 of them will block waiting for an HTTP connection from the pool.

  7. When Processor threads are blocked, their newConnections queues fill up

  Each Processor has an ArrayBlockingQueue with capacity 20 (SocketServer.scala:890). The Processor can only drain this queue when it completes a poll cycle
  (configureNewConnections() at line 1012). If the Processor thread is blocked on HTTP I/O, the queue fills.

  8. When ALL Processor queues are full, the Acceptor thread blocks

  In SocketServer.scala:760-771, the Acceptor tries each Processor in round-robin. On the last attempt (retriesLeft == 0), it calls newConnections.put() at line 1267
  — a blocking call. The Acceptor thread stops accepting TCP connections entirely.

  9. Once the Acceptor blocks, TCP connections pile up in the kernel backlog

  The default socket.listen.backlog.size is 50 (SocketServerConfigs.java:104). Once this is exhausted, the OS stops ACKing SYN packets. From the client's perspective,
   connect() hangs for seconds until either a slot opens up or TCP retransmit timers kick in.

  Why Kerberos Is Not Affected

  For GSSAPI/Kerberos, saslServer.evaluateResponse() is a purely in-memory cryptographic operation. It uses the local Kerberos keytab and the JVM's GSS-API library —
  no network I/O whatsoever on the Processor thread. The Processor threads process authentication in microseconds and are always available to drain their
  newConnections queues, so the Acceptor never blocks.

  Why Only the Busiest Clusters Are Affected

  The backpressure only triggers when enough concurrent OIDC authentications are in-flight simultaneously to saturate all Processor threads on a listener. On quieter
  clusters, the authentication rate is low enough that Processors complete their HTTP calls and drain their queues before new connections arrive.

  Why Some Brokers Are Hit More Than Others

  This is likely due to:
  - Uneven partition leadership / client affinity causing uneven auth load
  - The JDK HTTP connection pool (http.maxConnections default = 5) creating a bottleneck on brokers with more concurrent auths — once 5 Processors are using HTTP
  connections, the rest queue behind them
  - Possible GC pauses or OIDC provider latency variance amplifying the effect

  Recommendations

  1. Check if you're using introspection (oauth.validation.url) vs JWKS (oauth.jwks.endpoint.uri). If introspection, every auth makes a synchronous HTTP call. Switch
  to JWKS-based validation if possible — it validates tokens locally after the initial key fetch.
  2. Increase num.network.threads for the OIDC listener specifically:
  listener.name.<OIDC_LISTENER>.num.network.threads=32
  2. More Processor threads = more capacity to absorb HTTP blocking.
  3. Set http.maxConnections system property to match or exceed num.network.threads:
  -Dhttp.maxConnections=32
  3. This prevents Processor threads from blocking on the JDK HTTP connection pool.
  4. Lower the HTTP timeouts from the 60-second defaults:
  -Doauth.connect.timeout.seconds=5
  -Doauth.read.timeout.seconds=5
  5. Monitor the AcceptorBlockedPercent metric for the OIDC listener — this directly measures how long the Acceptor is blocked waiting for Processor queue space.
  6. Increase socket.listen.backlog.size as a band-aid to buffer more pending connections in the kernel while the Acceptor is blocked.