Skip to content

Instantly share code, notes, and snippets.

@tuminoid

tuminoid/k8s_scheduler_pending_issue.txt

Created January 23, 2019 13:25
Show Gist options
  • Save tuminoid/5e6367070eb95d45c0d59550b888cd0e to your computer and use it in GitHub Desktop.
Save tuminoid/5e6367070eb95d45c0d59550b888cd0e to your computer and use it in GitHub Desktop.
Download ZIP
Scheduler fails to schedule pods that share hostpath backed pvc in a single node cluster
Raw
k8s_scheduler_pending_issue.txt
$ kubectl get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
audit-699596cd5f-qc8ps 0/1 Pending 0 6m39s <none> <none> <none>
consul-7bfbdfd5-qjmhr 0/1 Pending 0 6m39s <none> <none> <none>
logger-7d44d67647-v7dwg 0/1 Pending 0 6m39s <none> <none> <none>
postgres-647d8f555c-gqg6d 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-42t5k 0/1 Pending 0 6m37s <none> <none> <none>
static-web-9dcb56dbb-45bnb 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-45hn8 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-47qbd 0/1 Pending 0 6m37s <none> <none> <none>
static-web-9dcb56dbb-4gq6b 0/1 Pending 0 6m37s <none> <none> <none>
static-web-9dcb56dbb-4k74w 0/1 Pending 0 6m37s <none> <none> <none>
static-web-9dcb56dbb-4tg9s 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-55gkz 0/1 Pending 0 6m37s <none> <none> <none>
static-web-9dcb56dbb-5wdft 0/1 Pending 0 6m37s <none> <none> <none>
static-web-9dcb56dbb-69vrg 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-6k8fk 0/1 Pending 0 6m37s <none> <none> <none>
static-web-9dcb56dbb-78p4t 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-7x5sm 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-85hwt 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-8nnfc 1/1 Running 0 6m38s 172.30.215.68 192.168.10.30 <none>
static-web-9dcb56dbb-8nx8h 0/1 Pending 0 6m37s <none> <none> <none>
static-web-9dcb56dbb-9j55q 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-9ms5n 0/1 Pending 0 6m37s <none> <none> <none>
static-web-9dcb56dbb-c2bgj 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-cgfbx 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-cvc9c 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-cz5m8 0/1 Pending 0 6m37s <none> <none> <none>
static-web-9dcb56dbb-fbtk4 0/1 Pending 0 6m37s <none> <none> <none>
static-web-9dcb56dbb-fd25h 0/1 Pending 0 6m37s <none> <none> <none>
static-web-9dcb56dbb-gmhhw 0/1 Pending 0 6m37s <none> <none> <none>
static-web-9dcb56dbb-hng87 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-hqxgh 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-j9xdl 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-k6kcn 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-kjtzj 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-kxksk 0/1 Pending 0 6m37s <none> <none> <none>
static-web-9dcb56dbb-mjqkj 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-n295z 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-n898d 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-n8rwl 0/1 Pending 0 6m37s <none> <none> <none>
static-web-9dcb56dbb-ndh8r 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-ndnlh 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-nhhnw 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-nvgh2 0/1 Pending 0 6m37s <none> <none> <none>
static-web-9dcb56dbb-p98r4 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-pdmrt 0/1 Pending 0 6m37s <none> <none> <none>
static-web-9dcb56dbb-phgqq 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-qjxrw 0/1 Pending 0 6m37s <none> <none> <none>
static-web-9dcb56dbb-rqd4z 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-t2wnj 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-trlsf 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-vgf2m 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-vmvsq 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-vz5x5 0/1 Pending 0 6m38s <none> <none> <none>
static-web-9dcb56dbb-wvcvh 0/1 Pending 0 6m37s <none> <none> <none>
vault-7f957948bf-cplsf 0/1 Pending 0 6m39s <none> <none> <none>
$ kubectl describe pod static-web-9dcb56dbb-zf6qc
Name: static-web-9dcb56dbb-qjxrw
Namespace: foo
Node: <none>
Labels: access-logger=true
pod-template-hash=9dcb56dbb
service=static-web
Annotations: <none>
Status: Pending
IP:
Controlled By: ReplicaSet/static-web-9dcb56dbb
Containers:
static-web:
Image: registry:5000/static-web:latest
Port: 8000/TCP
Host Port: 0/TCP
Limits:
memory: 32Mi
Requests:
memory: 16Mi
Liveness: http-get http://:static-web/healthz delay=10s timeout=2s period=10s #success=1 #failure=3
Readiness: http-get http://:static-web/healthz delay=5s timeout=2s period=5s #success=1 #failure=3
Mounts:
/keys from pv-keys-foo (rw)
/tokens from pv-vault-proxy (rw)
Conditions:
Type Status
PodScheduled False
Volumes:
pv-vault-proxy:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: pvc-vault-ro
ReadOnly: false
pv-keys-foo:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 7m22s (x4 over 7m32s) default-scheduler pod has unbound immediate PersistentVolumeClaims
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-audit Bound pv-audit 1Gi RWO sc-audit 83s
pvc-consul Bound pv-consul 2G RWO sc-consul 83s
pvc-consul-tokens-ro Bound pv-consul-tokens-ro 10Mi ROX sc-consul-tokens-ro 83s
pvc-consul-tokens-rw Bound pv-consul-tokens-rw 10Mi RWO sc-consul-tokens-rw 83s
pvc-logger Bound pv-logger 10Gi RWO sc-logger 83s
pvc-postgres-data Bound pv-postgres-data 10G RWO sc-postgres 83s
pvc-vault-ro Bound pv-vault-ro 10Mi ROX sc-vault-ro 83s
pvc-vault-rw Bound pv-vault-rw 10Mi RWO sc-vault-rw 83s
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv-audit 1Gi RWO Retain Bound foo/pvc-audit sc-audit 112s
pv-consul 2G RWO Retain Bound foo/pvc-consul sc-consul 112s
pv-consul-tokens-ro 10Mi ROX Retain Bound foo/pvc-consul-tokens-ro sc-consul-tokens-ro 112s
pv-consul-tokens-rw 10Mi RWO Retain Bound foo/pvc-consul-tokens-rw sc-consul-tokens-rw 112s
pv-logger 10Gi RWO Retain Bound foo/pvc-logger sc-logger 112s
pv-postgres-data 10G RWO Retain Bound foo/pvc-postgres-data sc-postgres 111s
pv-vault-ro 10Mi ROX Retain Bound foo/pvc-vault-ro sc-vault-ro 112s
pv-vault-rw 10Mi RWO Retain Bound foo/pvc-vault-rw sc-vault-rw 112s
$ kubectl describe pvc pvc-vault-ro
Name: pvc-vault-ro
Namespace: foo
StorageClass: sc-vault-ro
Status: Bound
Volume: pv-vault-ro
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"pvc-vault-ro","namespace":"foo"},"spec":{"accessMod...
pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 10Mi
Access Modes: ROX
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 2m48s persistentvolume-controller storageclass.storage.k8s.io "sc-vault-ro" not found
Mounted By: audit-699596cd5f-qc8ps
consul-7bfbdfd5-qjmhr
logger-7d44d67647-v7dwg
postgres-647d8f555c-gqg6d
static-web-9dcb56dbb-42t5k
static-web-9dcb56dbb-45bnb
static-web-9dcb56dbb-45hn8
static-web-9dcb56dbb-47qbd
static-web-9dcb56dbb-4gq6b
static-web-9dcb56dbb-4k74w
static-web-9dcb56dbb-4tg9s
static-web-9dcb56dbb-55gkz
static-web-9dcb56dbb-5wdft
static-web-9dcb56dbb-69vrg
static-web-9dcb56dbb-6k8fk
static-web-9dcb56dbb-78p4t
static-web-9dcb56dbb-7x5sm
static-web-9dcb56dbb-85hwt
static-web-9dcb56dbb-8nnfc
static-web-9dcb56dbb-8nx8h
static-web-9dcb56dbb-9j55q
static-web-9dcb56dbb-9ms5n
static-web-9dcb56dbb-c2bgj
static-web-9dcb56dbb-cgfbx
static-web-9dcb56dbb-cvc9c
static-web-9dcb56dbb-cz5m8
static-web-9dcb56dbb-fbtk4
static-web-9dcb56dbb-fd25h
static-web-9dcb56dbb-gmhhw
static-web-9dcb56dbb-hng87
static-web-9dcb56dbb-hqxgh
static-web-9dcb56dbb-j9xdl
static-web-9dcb56dbb-k6kcn
static-web-9dcb56dbb-kjtzj
static-web-9dcb56dbb-kxksk
static-web-9dcb56dbb-mjqkj
static-web-9dcb56dbb-n295z
static-web-9dcb56dbb-n898d
static-web-9dcb56dbb-n8rwl
static-web-9dcb56dbb-ndh8r
static-web-9dcb56dbb-ndnlh
static-web-9dcb56dbb-nhhnw
static-web-9dcb56dbb-nvgh2
static-web-9dcb56dbb-p98r4
static-web-9dcb56dbb-pdmrt
static-web-9dcb56dbb-phgqq
static-web-9dcb56dbb-qjxrw
static-web-9dcb56dbb-rqd4z
static-web-9dcb56dbb-t2wnj
static-web-9dcb56dbb-trlsf
static-web-9dcb56dbb-vgf2m
static-web-9dcb56dbb-vmvsq
static-web-9dcb56dbb-vz5x5
static-web-9dcb56dbb-wvcvh
$ cat /var/log/kubernetes/scheduler/kube-scheduler.ERROR
...<endless amount of same stuff>...
E0123 13:15:55.959312 1 factory.go:1483] Error scheduling esm/static-web-9dcb56dbb-cz5m8: pod has unbound immediate PersistentVolumeClaims; retrying
E0123 13:15:55.970124 1 factory.go:1483] Error scheduling esm/consul-7bfbdfd5-qjmhr: pod has unbound immediate PersistentVolumeClaims; retrying
E0123 13:15:55.991050 1 factory.go:1483] Error scheduling esm/static-web-9dcb56dbb-c2bgj: pod has unbound immediate PersistentVolumeClaims; retrying
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment