Performance analysis of Authorino's wristband token issuance and JWKS serving paths, with concrete fixes and estimated improvement factors.
Based on code analysis of the Authorino codebase in the context of the multi-cluster auth design for odh-model-controller.
For a wristband token request (GET /api/v1/auth/token), the auth pipeline executes:
- Identity:
kubernetesTokenReview(K8s API call) + condition checks onwristband-userand any external JWT evaluators - Authorization: skipped for token path (conditions exclude it), but conditions still evaluated
- Response: 5 response headers each with
whenconditions + wristbandCall()with custom claims
Each condition check and each custom claim resolution calls GetAuthorizationJSON(), which
triggers a full serialize/deserialize cycle.
Files: pkg/service/auth_pipeline.go:592-669, pkg/expressions/cel/expressions.go:39-88
GetAuthorizationJSON()
├─ RLock 5 pipeline maps (Identity, Metadata, Authorization, Response, Callbacks)
├─ Copy all map entries
├─ json.Marshal entire pipeline state → string [alloc + serialize]
└─ return string
Then the caller (CEL predicate or expression) does:
AuthJsonToCel(json)
├─ protojson.Unmarshal(json) → structpb.Struct [alloc + deserialize]
└─ Extract 5 top-level fields
Then for expressions (not predicates):
ValueToJSON(result)
└─ protojson.Marshal(result) → string [alloc + serialize]
That's 2-3 serialization cycles per call. With ~11 calls per wristband token request:
| Call site | Count | Trigger |
|---|---|---|
| AuthConfig-level conditions | 1 | pipeline.Evaluate() entry |
wristband-user identity condition |
1 | request.path != '/api/v1/auth/token' |
inference-access authz condition |
1 | path exclusion check |
inference-access-delegate authz condition |
1 | path exclusion + header check |
x-gateway-inference-fairness-id response condition |
1 | (no condition, but wired) |
x-gateway-inference-objective response condition |
1 | (no condition) |
x-maas-user response condition |
1 | path check |
x-maas-groups response condition |
1 | path check |
x-wristband-token response condition |
1 | request.path == '/api/v1/auth/token' |
| Wristband cache key resolution | 1 | response.go:85 |
| Wristband custom claims | 1 | wristband.go:120 (single call, resolves all claims) |
~11 calls x 2-3 serialization cycles = ~25-33 serialize/deserialize operations per request.
type AuthPipeline struct {
// ...existing fields...
authJSONCache string
authJSONCacheDirty bool
}
func (pipeline *AuthPipeline) GetAuthorizationJSON() string {
pipeline.mu.RLock()
if !pipeline.authJSONCacheDirty && pipeline.authJSONCache != "" {
defer pipeline.mu.RUnlock()
return pipeline.authJSONCache
}
pipeline.mu.RUnlock()
pipeline.mu.Lock()
defer pipeline.mu.Unlock()
// double-check after acquiring write lock
if !pipeline.authJSONCacheDirty && pipeline.authJSONCache != "" {
return pipeline.authJSONCache
}
pipeline.authJSONCache = pipeline.buildAuthorizationJSON()
pipeline.authJSONCacheDirty = false
return pipeline.authJSONCache
}
// Invalidate cache when phase results change
func (pipeline *AuthPipeline) setIdentityObj(conf *evaluators.IdentityConfig, obj interface{}) {
pipeline.mu.Lock()
defer pipeline.mu.Unlock()
pipeline.Identity[conf] = obj
pipeline.authJSONCacheDirty = true
}
// ...same for setMetadataObj, setAuthorizationObj, setResponseObj, setCallbackObjWithin a single phase (e.g., all response evaluators), the pipeline state doesn't change - all response evaluators run against the same snapshot. The cache is only invalidated when a new phase writes results.
Estimated improvement: 11 calls → 3-4 actual serializations (one per phase transition). ~3x reduction in JSON serialization per request. At 1000 req/s, this eliminates ~7000-8000 unnecessary serialize/deserialize cycles per second.
The deeper issue is that GetAuthorizationJSON() serializes Go structs to JSON, then
AuthJsonToCel() deserializes JSON back to protobuf structs, just to pass data to CEL.
This is a Go map → JSON string → protobuf struct round-trip on every evaluation.
A more invasive but higher-impact fix: build the CEL input (map[string]interface{}) directly
from the pipeline maps without the JSON intermediate representation.
func (pipeline *AuthPipeline) GetCELInput() (map[string]interface{}, error) {
// Build protobuf structs directly from pipeline maps
// Skip json.Marshal + protojson.Unmarshal entirely
}This would require changing the Matches(json string) and ResolveFor(json string)
signatures to accept structured input, which is a larger refactor but eliminates the
serialization entirely.
Estimated improvement: 5-10x for CEL-heavy AuthConfigs. The JSON round-trip
dominates the cost of simple predicate evaluations like request.path == '/api/v1/auth/token'.
Files: pkg/evaluators/response/wristband.go:170-186, pkg/service/oidc.go:54-65
GET /.well-known/openid-connect/certs
└─ OidcService.ServeHTTP()
└─ wristband.JWKS()
├─ for each signingKey:
│ └─ signingKey.Public() [crypto: extract public key from private]
├─ allocate []jose.JSONWebKey
└─ json.Marshal(jwks) [serialize]
No caching. No ETag. No Cache-Control headers. Every request recomputes from scratch.
With N peer clusters polling every 30s, that's N requests/30s, each doing crypto + JSON serialization for an output that only changes on AuthConfig reconciliation.
type Wristband struct {
Issuer string
CustomClaims []json.JSONProperty
TokenDuration int64
SigningKeys []jose.JSONWebKey
// Cached outputs - computed once, invalidated on key change
jwksCache string
jwksCacheOnce sync.Once
oidcConfigCache string
}
func (w *Wristband) JWKS() (string, error) {
var err error
w.jwksCacheOnce.Do(func() {
publicKeys := make([]jose.JSONWebKey, 0, len(w.SigningKeys))
for _, signingKey := range w.SigningKeys {
publicKeys = append(publicKeys, signingKey.Public())
}
jwks := jose.JSONWebKeySet{Keys: publicKeys}
var encoded []byte
encoded, err = gojson.Marshal(jwks)
if err == nil {
w.jwksCache = string(encoded)
}
})
return w.jwksCache, err
}Since Wristband instances are created during AuthConfig reconciliation and replaced
atomically (new reconcile = new instance), sync.Once is safe - the cached value is
valid for the entire lifetime of the instance.
Add HTTP cache headers in OidcService.ServeHTTP():
case "/.well-known/openid-connect/certs":
responseBody, err = wristband.JWKS()
if err == nil {
// JWKS only changes on AuthConfig reconciliation
writer.Header().Set("Cache-Control", "public, max-age=300")
writer.Header().Set("ETag", wristband.JWKSETag())
}Estimated improvement: For JWKS serving, O(1) instead of O(keys) per request.
At 10 clusters polling every 30s with 5 signing keys each, this eliminates 50 crypto
operations and 10 JSON serializations per 30-second window. The Cache-Control header
lets clients skip requests entirely.
File: pkg/evaluators/response/wristband.go:102-105
idStr, _ := gojson.Marshal(resolvedIdentityObject) // serialize identity to JSON
hash := sha256.New()
hash.Write(idStr) // hash the JSON
sub := fmt.Sprintf("%x", hash.Sum(nil)) // hex-encodeThe identity object is JSON-serialized here, then GetAuthorizationJSON() serializes the
entire pipeline (including the identity) again for custom claim resolution at line 120.
Two marshals of the same identity object within 18 lines.
Since GetAuthorizationJSON() will be called anyway for custom claims, extract the
identity portion from it instead of marshaling separately:
func (w *Wristband) Call(pipeline auth.AuthPipeline, ctx context.Context) (interface{}, error) {
resolvedIdentity, _ := pipeline.GetResolvedIdentity()
// ...issuer check...
authJSON := pipeline.GetAuthorizationJSON() // will be cached per Issue 1 fix
// Derive sub from the identity portion of authJSON
identityJSON := gjson.Get(authJSON, "auth.identity").Raw
hash := sha256.Sum256([]byte(identityJSON))
sub := fmt.Sprintf("%x", hash[:])
// ...rest of claims using authJSON...
}This eliminates one json.Marshal call per wristband issuance.
Estimated improvement: ~10-15% reduction in wristband issuance latency by eliminating one redundant JSON serialization of the identity object.
Files: pkg/service/auth.go:238-309, main.go:72
The gRPC server allows MaxConcurrentStreams: 10000. Each wristband token request triggers
a kubernetesTokenReview (HTTP call to K8s API server). With no admission control, 10,000
concurrent token requests means 10,000 concurrent TokenReview API calls.
The K8s API server has its own rate limiting, but Authorino doesn't respect it - failed TokenReviews bubble up as auth failures rather than backpressure signals.
// In main.go or a shared config
var k8sAPISemaphore = make(chan struct{}, 100) // limit concurrent K8s API calls
// In identity/kubernetes.go, wrap the TokenReview call
func (k *KubernetesAuth) Call(pipeline auth.AuthPipeline, ctx context.Context) (interface{}, error) {
select {
case k8sAPISemaphore <- struct{}{}:
defer func() { <-k8sAPISemaphore }()
case <-ctx.Done():
return nil, ctx.Err()
}
// ...existing TokenReview logic...
}Same pattern for SubjectAccessReview in authorization evaluators.
Estimated improvement: Prevents K8s API server saturation under load spikes. No throughput improvement under normal load, but prevents cascading failures at >100 concurrent requests (configurable). The difference between "auth latency increases under load" and "K8s API server becomes unresponsive for the entire cluster."
File: pkg/index/index.go:50-66
Every auth request does index.Get(host) (read lock). Every OIDC request does
index.FindKeys(realm) + index.Get(host) (two read locks). AuthConfig reconciliation
does index.Set() or index.Delete() (write lock).
During reconciliation (write lock held), all concurrent auth and OIDC requests block. With 28 InferenceServices triggering AuthConfig reconciliation in rapid succession (as seen in RHOAISUP-25), that's sustained write lock contention.
type authConfigTree struct {
current atomic.Pointer[treeSnapshot] // lock-free reads
mu sync.Mutex // writes only
}
type treeSnapshot struct {
root *treeNode
keys map[string][]string
}
func (c *authConfigTree) Get(key string) *evaluators.AuthConfig {
snapshot := c.current.Load() // atomic, no lock
if entry := snapshot.root.get(revertKey(key)); entry != nil {
return &entry.AuthConfig
}
return nil
}
func (c *authConfigTree) Set(id, key string, config evaluators.AuthConfig, override bool) error {
c.mu.Lock()
defer c.mu.Unlock()
// clone current snapshot, modify clone, atomic swap
newSnapshot := c.current.Load().clone()
// ...modify newSnapshot...
c.current.Store(newSnapshot)
return nil
}Reads become lock-free. Writes clone-and-swap, so concurrent reads see a consistent snapshot without blocking.
Estimated improvement: Eliminates read-side contention entirely. Under sustained reconciliation (e.g., 28 AuthConfig updates), auth request latency no longer spikes. P99 latency improvement of 10-100x during reconciliation storms, no change during steady state.
File: pkg/evaluators/response/wristband.go:138
ES256 (ECDSA P-256) signing takes ~0.1ms per operation. At 10,000 tokens/second, that's 1 full CPU core just for signing. This is inherent to the crypto and can't be optimized away, but it can be parallelized.
ES256 signing is already fast. At realistic scales (100-1000 tokens/second), signing consumes 1-10% of a CPU core. The serialization overhead (Issues 1-3) dominates.
If signing becomes a bottleneck at extreme scales, consider:
- Horizontal scaling (multiple Authorino replicas behind the same Gateway)
- Pre-signing token batches during idle periods (not practical for per-user tokens)
No fix needed unless token issuance exceeds ~5000/second sustained.
| # | Fix | Effort | Improvement | Metric |
|---|---|---|---|---|
| 1a | Cache GetAuthorizationJSON() per phase |
Small | ~3x fewer serializations | CPU, alloc/s |
| 1b | Pass CEL inputs directly (no JSON round-trip) | Large | 5-10x for CEL evaluations | CPU, latency |
| 2 | Memoize JWKS + HTTP cache headers | Small | O(1) JWKS serving | CPU, network |
| 3 | Reuse serialized identity for sub hash |
Small | ~10-15% per issuance | CPU, alloc |
| 4 | K8s API call semaphore | Small | Prevents cascading failure | Availability |
| 5 | Copy-on-write index | Medium | 10-100x P99 during reconciliation | Tail latency |
Fix 1a + Fix 2 + Fix 3 together would reduce per-request CPU cost by roughly 2-3x for the wristband token path. These are localized changes with no API surface impact.
Fix 1b (eliminate JSON round-trip for CEL) would be the single largest performance improvement but requires changing internal interfaces. This benefits all AuthConfig evaluations, not just wristbands.
Fix 4 + Fix 5 don't improve steady-state throughput but prevent the system from falling over under load spikes and reconciliation storms - exactly the scenario seen in RHOAISUP-25.
| Operation | Count | Cost each | Total |
|---|---|---|---|
json.Marshal (pipeline state) |
~11 | ~50us | ~550us |
protojson.Unmarshal (CEL input) |
~11 | ~30us | ~330us |
json.Marshal (identity for sub) |
1 | ~10us | ~10us |
| K8s TokenReview API call | 1 | ~5ms | ~5ms |
| ES256 signing | 1 | ~100us | ~100us |
| Map copies + RLocks | ~11 | ~1us | ~11us |
| Total (excl. network) | ~6ms | ||
| Serialization overhead | ~890us (~15%) |
| Operation | Count | Cost each | Total |
|---|---|---|---|
json.Marshal (pipeline state) |
~3 | ~50us | ~150us |
protojson.Unmarshal (CEL input) |
~3 | ~30us | ~90us |
| K8s TokenReview API call | 1 | ~5ms | ~5ms |
| ES256 signing | 1 | ~100us | ~100us |
| Cache lookups | ~8 | ~0.1us | ~1us |
| Total (excl. network) | ~5.3ms | ||
| Serialization overhead | ~240us (~4.5%) |
The K8s API call dominates regardless. The serialization fixes matter most when the K8s API is fast (cached TokenReview) or when running at high concurrency where CPU becomes the bottleneck before network does.