OCP Telco Notes.md

Troubleshooting CPU isolation

• Identifying cores assigned to a particular container

ID=$(crictl ps --label io.kubernetes.container.name=<name-of-container> -o json | jq -r '.containers[0].id')
PID=$(crictl inspect $ID | jq -r '.info.pid')
taskset -pc $PID 

Troubleshooting

• Troubleshooting etcd performance on a lab environment NOTE: DO NOT USE THESE COMMANDS ON A PRODUCTION CLUSTER

id=$(sudo crictl ps —name etcd-member | awk ‘FNR==2{ print $1}’) && sudo crictl exec -it $id /bin/sh
export ETCDCTL_API=3 ETCDCTL_CACERT=/etc/ssl/etcd/ca.crt ETCDCTL_CERT=$(find /etc/ssl/ -name *peer*crt) ETCDCTL_KEY=$(find /etc/ssl/ -name *peer*key)

etcdctl check perf

etcdctl endpoint health
etcdctl endpoint status

etcdctl member list

• Find Pods status and QoS

# List QoS of all pods not in Running or Completed/Succeeded
oc get pods -A -o custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace,QOS-CLASS:.status.qosClass --field-selector=status.phase!=Running,status.phase!=Succeeded

# List QoS and Status of all pods not in Running or Completed/Succeeded
oc get pods -A -o custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace,QOS-CLASS:.status.qosClass,STATUS:.status.phase --field-selector=status.phase!=Running,status.phase!=Succeeded

# List all Pods in Pending state on particular Node 
oc get pods -A -o custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace,STATUS:.status.phase,NODE:.spec.nodeName --field-selector=spec.nodeName=m0,status.phase=Pending

• Cincinnati protocol output of upgrade graph for a channel

export CHANNEL=stable-4.8

curl -sH 'Accept:application/json' "https://api.openshift.com/api/upgrades_info/v1/graph?channel=${CHANNEL}" | jq .