Collection of logs/snippets to show how to debug a failed APB provision in OCP 3.6.0 with Ansible Service Broker

Debugging a Failed APB Provision - 3.6.0 MediaWiki & PostgreSQL

# oc get projects
NAME                               DISPLAY NAME   STATUS
default                                           Active
demo                                              Active
kube-public                                       Active
kube-service-catalog                              Active
kube-system                                       Active
logging                                           Active
management-infra                                  Active
openshift                                         Active
openshift-ansible-service-broker                  Active
openshift-infra                                   Active



2 APBs were provisioned into the 'demo' project
Both APBs errored out

# oc get pods --namespace=demo
NAME                                       READY     STATUS      RESTARTS   AGE
apb-0183fc4d-ce19-4a01-8e37-50de36b1c631   0/1       Error       0          6h
apb-7eab770f-4848-46c8-9a10-9b0992b563d1   0/1       Completed   0          6h
mediawiki123-1-deploy                      0/1       Error       0          6h
postgresql-1-deploy                        0/1       Error       0          6h


# oc logs apb-0183fc4d-ce19-4a01-8e37-50de36b1c631 --namespace=demo
Openshift cluster credentials not provided. Assuming the broker is running inside an Openshift cluster
Attempting to login with a service account...
Logged into "https://kubernetes.default:443" as "system:serviceaccount:demo:apb-0183fc4d-ce19-4a01-8e37-50de36b1c631" using the token provided.
You have one project on this server: "demo"
Using project "demo".
Welcome! See 'oc help' to get started.
PLAY [Deploy rhscl-postgresql-apb to openshift] ********************************
TASK [ansible.kubernetes-modules : Intall latest openshift client] *************
skipping: [localhost]
TASK [rhscl-postgresql-apb-openshift : create namespace] ***********************
ok: [localhost]
TASK [rhscl-postgresql-apb-openshift : set volume state to present] ************
changed: [localhost]
TASK [rhscl-postgresql-apb-openshift : set service state to present] ***********
changed: [localhost]
TASK [rhscl-postgresql-apb-openshift : scale deployment down] ******************
skipping: [localhost]
TASK [rhscl-postgresql-apb-openshift : set deployment config state to present] ***
changed: [localhost]
TASK [rhscl-postgresql-apb-openshift : delete replication controller] **********
skipping: [localhost]
TASK [rhscl-postgresql-apb-openshift : Wait for postgres to come up] ***********
fatal: [localhost]: FAILED! => {"changed": false, "elapsed": 300, "failed": true, "msg": "Timeout when waiting for 172.30.120.119:5432"}
PLAY RECAP *********************************************************************
localhost                  : ok=4    changed=3    unreachable=0    failed=1   
 [WARNING]: Could not create retry file '/opt/apb/actions/provision.retry'.
[Errno 13] Permission denied: u'/opt/apb/actions/provision.retry'


#### That didn't show us much beyond the APB ran and it waited 10 minutes for postgres to be deployed.
#### postgres failed to come up, so after 10 minutes the APB timed out.  
#### We don't know the cause of the failure just yet.


# oc logs postgresql-1-deploy --namespace=demo
--> Scaling postgresql-1 to 1
--> Waiting up to 10m0s for pods in rc postgresql-1 to become ready
error: update acceptor rejected postgresql-1: pods for rc "postgresql-1" took longer than 600 seconds to become ready


# oc logs mediawiki123-1-deploy --namespace=demo
--> Scaling mediawiki123-1 to 1
--> Waiting up to 10m0s for pods in rc mediawiki123-1 to become ready
error: update acceptor rejected mediawiki123-1: pods for rc "mediawiki123-1" took longer than 600 seconds to become ready


### Still not much to go on.
### No obvious errors beyond a timeout


# oc get pvc --namespace=demo
NAME               STATUS    VOLUME    CAPACITY   ACCESSMODES   STORAGECLASS   AGE
mediawiki123-pvc   Pending                                                     6h
postgresql         Pending                                                     6h

 oc get pv
NAME          CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                                   STORAGECLASS   REASON    AGE
etcd-volume   10G        RWO           Retain          Bound     openshift-ansible-service-broker/etcd                            6h

### Now we realize that the 2 APBs we were deploying required PV's and our setup lacked available PV's to satisfy the PVCs the APBs required.