tls-is-always-painful.md

If the maas-api log shows a TLS handshake error like remote error: tls: bad certificate, this usually means the client side (Authorino) rejected the certificate rather than the other way around.

This looks like the Authorino-to-maas-api TLS path. maas-api serves HTTPS with an OpenShift service-serving cert, so Authorino needs the OpenShift service CA bundle when it calls the tier lookup endpoint. The Authorino CR does not expose generic env vars, so the current workaround is to mount the CA via the Authorino CR and set SSL_CERT_FILE / REQUESTS_CA_BUNDLE on the generated deployment.

There are two separate TLS connections in the MaaS auth flow:

1. Gateway -> Authorino (listener TLS) -- requires spec.listener.tls.enabled: true on the Authorino CR and the security.opendatahub.io/authorino-tls-bootstrap: "true" annotation on the gateway. On RHOAI/ODH this is usually already configured as a platform prerequisite.

2. Authorino -> maas-api (outbound TLS) -- Authorino makes HTTPS calls to maas-api for tier lookups (e.g. https://maas-api.<namespace>.svc.cluster.local:8443/v1/tiers/lookup). This is the leg that's likely failing.

Fix

First, confirm the service CA ConfigMap exists in the Authorino namespace:

oc -n "$AUTHORINO_NS" get configmap openshift-service-ca.crt

Then patch the Authorino CR to mount it. Note: this replaces spec.volumes.items -- if you already have custom volumes on your Authorino CR, merge this entry into the existing list rather than applying it as-is.

oc -n "$AUTHORINO_NS" patch authorino authorino --type=merge --patch '
{
  "spec": {
    "volumes": {
      "items": [
        {
          "name": "openshift-service-ca",
          "mountPath": "/etc/ssl/certs/openshift-service-ca",
          "configMaps": ["openshift-service-ca.crt"],
          "items": [
            {
              "key": "service-ca.crt",
              "path": "service-ca-bundle.crt"
            }
          ]
        }
      ]
    }
  }
}'

Set the env vars on the deployment so Authorino uses the bundle (this should trigger a rollout on its own):

oc -n "$AUTHORINO_NS" set env deployment/"$AUTHORINO_DEPLOY" \
  SSL_CERT_FILE=/etc/ssl/certs/openshift-service-ca/service-ca-bundle.crt \
  REQUESTS_CA_BUNDLE=/etc/ssl/certs/openshift-service-ca/service-ca-bundle.crt

As a belt-and-suspenders step, restart the deployment to make sure it picked everything up:

oc -n "$AUTHORINO_NS" rollout restart deployment/"$AUTHORINO_DEPLOY"

Also verify that the MaaS gateway has both required annotations:

oc -n "$GATEWAY_NS" get gateway maas-default-gateway -o yaml \
  | grep -E 'opendatahub.io/managed|authorino-tls-bootstrap'

Expected output:

opendatahub.io/managed: "false"
security.opendatahub.io/authorino-tls-bootstrap: "true"

Additional Diagnostics if the Fix Doesn't Work :_(

Set these variables to match the cluster. Defaults shown are for a typical RHOAI deployment:

AUTHORINO_NS=kuadrant-system
MAAS_NS=redhat-ods-applications
GATEWAY_NS=openshift-ingress
AUTHORINO_DEPLOY=authorino

Logs

# Authorino logs
oc -n "$AUTHORINO_NS" logs deployment/"$AUTHORINO_DEPLOY" --tail=200 --since=30m | \
  grep -iE 'tls|certificate|x509|handshake|dial|maas|tier' || true

# maas-api logs
oc -n "$MAAS_NS" logs deployment/maas-api --tail=200 --since=30m | \
  grep -iE 'tls|certificate|x509|handshake|bad certificate|remote error' || true

Authorino Deployment State

# Are the CA env vars set?
oc -n "$AUTHORINO_NS" get deploy "$AUTHORINO_DEPLOY" \
  -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}={.value}{"\n"}{end}'

# Is the CA volume present?
oc -n "$AUTHORINO_NS" get deploy "$AUTHORINO_DEPLOY" \
  -o jsonpath='{range .spec.template.spec.volumes[*]}{.name}{"\n"}{end}'

# Is it mounted into the container?
oc -n "$AUTHORINO_NS" get deploy "$AUTHORINO_DEPLOY" \
  -o jsonpath='{range .spec.template.spec.containers[0].volumeMounts[*]}{.name} -> {.mountPath}{"\n"}{end}'

# Does the CA file exist in the running Authorino pod?
oc -n "$AUTHORINO_NS" exec deployment/"$AUTHORINO_DEPLOY" -- \
  ls -la /etc/ssl/certs/openshift-service-ca/

Certificate Checks

# Does the OpenShift service CA ConfigMap have a cert?
oc -n "$AUTHORINO_NS" get configmap openshift-service-ca.crt \
  -o jsonpath='{.data.service-ca\.crt}' | openssl x509 -subject -issuer -enddate -noout

# Does the maas-api serving cert exist?
oc -n "$MAAS_NS" get secret maas-api-serving-cert

# Is it still valid?
oc -n "$MAAS_NS" get secret maas-api-serving-cert \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -enddate -noout

# Who issued it?
oc -n "$MAAS_NS" get secret maas-api-serving-cert \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -issuer -noout

Infrastructure Checks

# Confirm maas-api service is asking OpenShift to generate the serving cert
oc -n "$MAAS_NS" get svc maas-api \
  -o jsonpath='{.metadata.annotations.service\.beta\.openshift\.io/serving-cert-secret-name}{"\n"}'


# Confirm Authorino listener TLS is enabled
oc -n "$AUTHORINO_NS" get authorino authorino \
  -o jsonpath='{.spec.listener.tls.enabled}{" "}{.spec.listener.tls.certSecretRef.name}{"\n"}'

# Confirm the gateway bootstrap annotations
oc -n "$GATEWAY_NS" get gateway maas-default-gateway -o yaml | \
  grep -E 'opendatahub.io/managed|authorino-tls-bootstrap'

# Confirm the tier lookup URL Authorino is using
oc -n "$GATEWAY_NS" get authpolicy gateway-auth-policy \
  -o jsonpath='{.spec.rules.metadata.matchedTier.http.url}{"\n"}'

# Check pod events for mount/restart issues
oc -n "$AUTHORINO_NS" get events --sort-by='.lastTimestamp' \
  --field-selector involvedObject.kind=Pod | grep authorino | tail -20

Connection Test

This runs a temporary curl pod with the authorino-resource=authorino label so it matches the same NetworkPolicy rules as Authorino. It tests both bypassed and CA-verified connections to maas-api:

oc -n "$AUTHORINO_NS" run maas-ca-test --rm -i --restart=Never \
  --image=curlimages/curl \
  --labels=authorino-resource=authorino \
  --command -- sh -c "
    echo 'Bypass cert verification:'
    curl -skv https://maas-api.${MAAS_NS}.svc.cluster.local:8443/health >/dev/null &&
    echo OK

    echo 'Verify with OpenShift service CA:'
    curl -v --cacert /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt \
      https://maas-api.${MAAS_NS}.svc.cluster.local:8443/health
  "

Interpreting results:

• Bypass succeeds, verified call fails -- confirms the CA trust problem. Apply the fix below.
• Both fail -- look at NetworkPolicy, service routing, or whether maas-api is running.
• Both succeed from the curl pod but Authorino still fails -- focus on whether the env vars and mounted CA file are present in the actual Authorino deployment.

Caveat: Authorino is operator-managed, so the oc set env step patches the generated deployment directly. If the Authorino operator reconciles or gets upgraded, it may regenerate the deployment without those env vars, bringing the error back. If that happens, reapply the set env command. The volumes patch on the Authorino CR itself should survive reconciliation since it's part of the CR spec.