pr662-logs.md

MaaS PR662 Install Log

$ ./scripts/deploy.sh --operator-type odh
[INFO] ===================================================
[INFO]   Models-as-a-Service Deployment
[INFO] ===================================================
[INFO] Validating configuration...
[INFO] Configuration validated successfully
[INFO] Deployment configuration:
[INFO]   Mode: operator
[INFO]   Operator: odh
[INFO]   Policy Engine: kuadrant
[INFO]   Namespace: opendatahub
[INFO]   TLS Backend: true
[INFO]   External OIDC: false
[INFO] Starting operator-based deployment...
[INFO] Checking if there are any conflicting operators...
[INFO] No conflicting operators found. Proceeding to installing the primary operator.
[INFO] Installing optional operators in parallel...
[INFO] Applying cert-manager and LeaderWorkerSet subscriptions...
namespace/openshift-lws-operator created
namespace/cert-manager-operator created
operatorgroup.operators.coreos.com/cert-manager-operator created
operatorgroup.operators.coreos.com/leader-worker-set created
subscription.operators.coreos.com/leader-worker-set created
subscription.operators.coreos.com/openshift-cert-manager-operator created
[INFO] Waiting for operators to be installed...
  * Waiting for Subscription cert-manager-operator/openshift-cert-manager-operator to start setup...
  * Waiting for Subscription openshift-lws-operator/leader-worker-set to start setup...
subscription.operators.coreos.com/leader-worker-set condition met
  * Waiting for Subscription setup to finish setup. CSV = leader-worker-set.v1.0.0 ...
clusterserviceversion.operators.coreos.com/leader-worker-set.v1.0.0 condition met
subscription.operators.coreos.com/openshift-cert-manager-operator condition met
  * Waiting for Subscription setup to finish setup. CSV = cert-manager-operator.v1.18.1 ...
clusterserviceversion.operators.coreos.com/cert-manager-operator.v1.18.1 condition met
[INFO] Activating LeaderWorkerSet API...
leaderworkersetoperator.operator.openshift.io/cluster created
[INFO] Optional operators installed
[INFO] Installing policy engine: kuadrant
[INFO] Installing Kuadrant v1.4.2 (upstream community)
[INFO] Creating Kuadrant v1.4.2 catalog source...
namespace/kuadrant-system created
catalogsource.operators.coreos.com/kuadrant-operator-catalog created
[INFO] Waiting for Kuadrant catalog to be ready...
operatorgroup.operators.coreos.com/kuadrant-operator-group created
[INFO] Installing operator: kuadrant-operator in namespace: kuadrant-system
namespace/kuadrant-system condition met
[INFO] Creating Subscription for kuadrant-operator from kuadrant-operator-catalog (channel: stable)
subscription.operators.coreos.com/kuadrant-operator created
[INFO] Waiting for subscription to install...
  * Waiting for Subscription kuadrant-system/kuadrant-operator to start setup...
subscription.operators.coreos.com/kuadrant-operator condition met
  * Waiting for Subscription setup to finish setup. CSV = kuadrant-operator.v1.4.2 ...
clusterserviceversion.operators.coreos.com/kuadrant-operator.v1.4.2 condition met
[INFO] Operator kuadrant-operator installed successfully
[INFO] Patching kuadrant-operator CSV for OpenShift Gateway controller...
clusterserviceversion.operators.coreos.com/kuadrant-operator.v1.4.2 patched
[INFO] CSV patched for OpenShift Gateway controller
[INFO] Forcing operator restart to apply new Gateway controller configuration...
pod "kuadrant-operator-controller-manager-5f895dd7d5-x9ptd" force deleted from kuadrant-system namespace
pod "limitador-operator-controller-manager-85c4996f8c-tcfm7" force deleted from kuadrant-system namespace
[INFO] Waiting for operator pod to restart...
Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "kuadrant-operator-controller-manager" rollout to finish: 1 old replicas are pending termination...
deployment "kuadrant-operator-controller-manager" successfully rolled out
[WARN] Operator pod may not have correct env yet:
[INFO] Waiting 15s for operator to fully initialize with Gateway controller configuration...
[INFO] Initializing Gateway API and ModelsAsService gateway...
[INFO] Setting up Gateway API infrastructure...
[INFO] Creating GatewayClass openshift-default...
gatewayclass.gateway.networking.k8s.io/openshift-default created
[INFO] Setting up ModelsAsService gateway...
[INFO]   Cluster domain: apps.ci-ln-4dxqsg2-76ef8.aws-4.ci.openshift.org
[INFO]   Detecting TLS certificate secret...
[INFO]   * Found certificate from router deployment: router-certs-default
[INFO]   TLS certificate secret: router-certs-default
[INFO] Creating maas-default-gateway resource (allowing routes from all namespaces)...
gateway.gateway.networking.k8s.io/maas-default-gateway serverside-applied
[INFO] Waiting for Gateway to be Programmed (Service Mesh initialization)...
gateway.gateway.networking.k8s.io/maas-default-gateway condition met
[INFO] Applying Kuadrant custom resource in kuadrant-system...
kuadrant.kuadrant.io/kuadrant created
[INFO] Waiting for Kuadrant to become ready (initial check)...
[INFO] Waiting for: Kuadrant ready in kuadrant-system (timeout: 60s)
[ERROR] Kuadrant ready in kuadrant-system - Timeout after 60s
[INFO] Kuadrant shows MissingDependency - restarting operator to re-register Gateway controller...
pod "authorino-f99f4b5cd-d6962" force deleted from kuadrant-system namespace
pod "kuadrant-operator-controller-manager-6bc9f4c76f-69clt" force deleted from kuadrant-system namespace
pod "limitador-operator-controller-manager-85c4996f8c-9wml5" force deleted from kuadrant-system namespace
[INFO] Retrying Kuadrant readiness check after operator restart...
[INFO] Waiting for: Kuadrant ready in kuadrant-system (timeout: 120s)
[INFO] Kuadrant ready in kuadrant-system - Ready
[INFO] Kuadrant setup complete
[INFO] Installing primary operator: odh
[INFO] Installing ODH operator...
[INFO] Installing operator: opendatahub-operator in namespace: opendatahub
[INFO] Creating namespace: opendatahub
namespace/opendatahub created
namespace/opendatahub condition met
[INFO] Creating OperatorGroup in opendatahub for AllNamespaces mode
operatorgroup.operators.coreos.com/opendatahub-operatorgroup created
[INFO] Creating Subscription for opendatahub-operator from community-operators (channel: fast-3, installPlanApproval: Manual, startingCSV: opendatahub-operator.v3.4.0-ea.1)
subscription.operators.coreos.com/opendatahub-operator created
[INFO] Manual Subscription: approving initial InstallPlan so first install can proceed...
[INFO] Approving initial InstallPlan install-hvcph (Manual subscription)
installplan.operators.coreos.com/install-hvcph patched
[INFO] Waiting for subscription to install...
  * Waiting for Subscription opendatahub/opendatahub-operator to start setup...
subscription.operators.coreos.com/opendatahub-operator condition met
  * Waiting for Subscription setup to finish setup. CSV = opendatahub-operator.v3.4.0-ea.1 ...
clusterserviceversion.operators.coreos.com/opendatahub-operator.v3.4.0-ea.1 condition met
[INFO] Operator opendatahub-operator installed successfully
[INFO] Applying custom resources...
[INFO] Waiting for operator CRDs to be established...
⏳ Waiting for CRD datascienceclusters.datasciencecluster.opendatahub.io to appear (timeout: 180s)…
✅ CRD datascienceclusters.datasciencecluster.opendatahub.io detected, waiting for it to become Established...
customresourcedefinition.apiextensions.k8s.io/datascienceclusters.datasciencecluster.opendatahub.io condition met
[INFO] Waiting for operator webhook to be ready...
* Waiting for deployment/opendatahub-operator-controller-manager in opendatahub (timeout: 120s)...
  * Found deployment/opendatahub-operator-controller-manager
deployment.apps/opendatahub-operator-controller-manager condition met
[INFO] Applying DSCInitialization...
dscinitialization.dscinitialization.opendatahub.io/default-dsci created
[INFO] Applying DataScienceCluster with ModelsAsService...
datasciencecluster.datasciencecluster.opendatahub.io/default-dsc serverside-applied
[INFO] Waiting for DataScienceCluster to be ready...
* Waiting for DataScienceCluster 'default-dsc' KServe and ModelsAsService components to be ready...
  - KServe state: , KserveReady: , ModelsAsServiceReady: , ModelControllerReady:
  - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: False, ModelControllerReady: False
  - KServe state: Managed, KserveReady: False, ModelsAsServiceReady: False, ModelControllerReady: True
  * KServe and ModelsAsService are ready in DataScienceCluster 'default-dsc'
🔧 Deploying PostgreSQL for API key storage in namespace 'opendatahub'...
  Generated random PostgreSQL password (stored in secret postgres-creds)
  Creating PostgreSQL deployment...
  ⚠️  Using POC configuration (ephemeral storage)

secret/postgres-creds created
deployment.apps/postgres created
service/postgres created
secret/maas-db-config created
  Waiting for PostgreSQL to be ready...
deployment.apps/postgres condition met

✅ PostgreSQL deployed successfully
  Database: maas
  User: maas
  Secret: maas-db-config (contains DB_CONNECTION_URL)

  ⚠️  For production, use AWS RDS, Crunchy Operator, or Azure Database
  Note: Schema migrations run automatically when maas-api starts
  * MAAS_API_IMAGE not set, using operator default
[INFO] Configuring TLS backend for Authorino and MaaS API...
* Waiting for deployment/authorino in kuadrant-system (timeout: 300s)...
  * Found deployment/authorino
[INFO] Running TLS configuration script...
[INFO] TLS configuration script completed successfully
[INFO] Restarting deployments to pick up TLS configuration...
deployment.apps/maas-api restarted
deployment.apps/authorino restarted
[INFO] Waiting for Authorino deployment to be ready...
Waiting for deployment "authorino" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment spec update to be observed...
Waiting for deployment spec update to be observed...
Waiting for deployment "authorino" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "authorino" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "authorino" rollout to finish: 1 old replicas are pending termination...
deployment "authorino" successfully rolled out
[INFO] TLS backend configuration complete
[INFO] Checking cluster OIDC audience...
[INFO] Standard Kubernetes audience detected, no patching needed
[INFO] Operator deployment completed
[INFO]
[INFO] MaaS Subscription Controller...
[INFO]   Installing controller (CRDs, RBAC, deployment, default-deny policy)...
customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io created
customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io created
customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io created
customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io created
serviceaccount/maas-controller created
role.rbac.authorization.k8s.io/maas-controller-leader-election-role created
clusterrole.rbac.authorization.k8s.io/maas-controller-role created
rolebinding.rbac.authorization.k8s.io/maas-controller-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/maas-controller-rolebinding created
deployment.apps/maas-controller created
[INFO]   Waiting for maas-controller to be ready...
Waiting for deployment "maas-controller" rollout to finish: 0 of 1 updated replicas are available...
deployment "maas-controller" successfully rolled out
[INFO]   Subscription controller ready.
[INFO]   Create MaaSModelRef, MaaSAuthPolicy, and MaaSSubscription to enable per-model auth and rate limiting.
[INFO] ===================================================
[INFO]   Deployment completed successfully!
[INFO] ===================================================
$ kubectl apply -k deployment/overlays/odh
customresourcedefinition.apiextensions.k8s.io/externalmodels.maas.opendatahub.io unchanged
customresourcedefinition.apiextensions.k8s.io/maasauthpolicies.maas.opendatahub.io unchanged
customresourcedefinition.apiextensions.k8s.io/maasmodelrefs.maas.opendatahub.io unchanged
customresourcedefinition.apiextensions.k8s.io/maassubscriptions.maas.opendatahub.io unchanged
Warning: resource serviceaccounts/maas-api is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
serviceaccount/maas-api configured
serviceaccount/maas-controller unchanged
serviceaccount/payload-processing created
role.rbac.authorization.k8s.io/maas-controller-leader-election-role unchanged
Warning: resource clusterroles/maas-api is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
clusterrole.rbac.authorization.k8s.io/maas-api configured
clusterrole.rbac.authorization.k8s.io/maas-controller-role unchanged
clusterrole.rbac.authorization.k8s.io/payload-processing-reader created
rolebinding.rbac.authorization.k8s.io/maas-controller-leader-election-rolebinding unchanged
Warning: resource clusterrolebindings/maas-api is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
clusterrolebinding.rbac.authorization.k8s.io/maas-api configured
clusterrolebinding.rbac.authorization.k8s.io/maas-controller-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/payload-processing-reader created
Warning: resource configmaps/maas-parameters is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
configmap/maas-parameters configured
Warning: resource services/maas-api is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
service/maas-api configured
service/payload-processing created
Warning: resource deployments/maas-api is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
deployment.apps/maas-api configured
deployment.apps/maas-controller configured
deployment.apps/payload-processing created
cronjob.batch/maas-api-key-cleanup created
Warning: resource httproutes/maas-api-route is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
httproute.gateway.networking.k8s.io/maas-api-route configured
Warning: resource authpolicies/maas-api-auth-policy is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
authpolicy.kuadrant.io/maas-api-auth-policy configured
authpolicy.kuadrant.io/gateway-default-auth created
tokenratelimitpolicy.kuadrant.io/gateway-default-deny created
Warning: resource destinationrules/maas-api-backend-tls is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
destinationrule.networking.istio.io/maas-api-backend-tls configured
destinationrule.networking.istio.io/payload-processing created
envoyfilter.networking.istio.io/payload-processing created
networkpolicy.networking.k8s.io/maas-api-cleanup-restrict created
Warning: resource networkpolicies/maas-authorino-allow is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
networkpolicy.networking.k8s.io/maas-authorino-allow configured

$ kubectl create ns llm --dry-run=client -o yaml | kubectl apply -f -
  kubectl create ns models-as-a-service --dry-run=client -o yaml | kubectl apply -f -
  kustomize build docs/samples/maas-system | kubectl apply -f -
  kubectl wait --for=condition=Ready llminferenceservice/facebook-opt-125m-simulated -n llm --timeout=300s
namespace/llm created
Warning: resource namespaces/models-as-a-service is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
namespace/models-as-a-service configured
maasauthpolicy.maas.opendatahub.io/premium-simulator-access created
maasauthpolicy.maas.opendatahub.io/simulator-access created
maasmodelref.maas.opendatahub.io/facebook-opt-125m-simulated created
maasmodelref.maas.opendatahub.io/premium-simulated-simulated-premium created
maassubscription.maas.opendatahub.io/premium-simulator-subscription created
maassubscription.maas.opendatahub.io/simulator-subscription created
llminferenceservice.serving.kserve.io/facebook-opt-125m-simulated created
llminferenceservice.serving.kserve.io/premium-simulated-simulated-premium created
llminferenceservice.serving.kserve.io/facebook-opt-125m-simulated condition met