HostPathType Debuging Log

gistfile1.txt

node-01 system # docker logs kubelet
2017-09-13 02:13:40.186868 I | proto: duplicate proto type registered: google.protobuf.Any
2017-09-13 02:13:40.187008 I | proto: duplicate proto type registered: google.protobuf.Duration
2017-09-13 02:13:40.187026 I | proto: duplicate proto type registered: google.protobuf.Timestamp
Flag --network-plugin-dir has been deprecated, Use --cni-bin-dir instead. This flag will be removed in a future version.
I0913 02:13:40.222607   11616 feature_gate.go:156] feature gates: map[]
I0913 02:13:40.228852   11616 client.go:75] Connecting to docker on unix:///var/run/docker.sock
I0913 02:13:40.229874   11616 client.go:95] Start docker client with request timeout=2m0s
W0913 02:13:40.232306   11616 cni.go:196] Unable to update cni config: No networks found in /etc/kubernetes/cni/net.d
W0913 02:13:40.257783   11616 server.go:288] --cloud-provider=auto-detect is deprecated. The desired cloud provider should be set explicitly
I0913 02:13:40.261151   11616 manager.go:149] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct"
W0913 02:13:40.271023   11616 manager.go:157] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
W0913 02:13:40.271093   11616 manager.go:166] unable to connect to CRI-O api service: Get http://%2Fvar%2Frun%2Fcrio.sock/info: dial unix /var/run/crio.sock: connect: no such file or directory
I0913 02:13:40.285134   11616 fs.go:139] Filesystem UUIDs: map[]
I0913 02:13:40.285191   11616 fs.go:140] Filesystem partitions: map[tmpfs:{mountpoint:/dev major:0 minor:55 fsType:tmpfs blockSize:0} /dev/sda9:{mountpoint:/var/lib/docker/overlay2 major:8 minor:9 fsType:ext4 blockSize:0} /dev/mapper/usr:{mountpoint:/rootfs/usr major:254 minor:0 fsType:ext4 blockSize:0} /dev/sda6:{mountpoint:/rootfs/usr/share/oem major:8 minor:6 fsType:ext4 blockSize:0} shm:{mountpoint:/rootfs/var/lib/docker/containers/b00db8424d71c02839a032c4cd6d400febd7e69378808d630f92c2442f1d1f20/shm major:0 minor:53 fsType:tmpfs blockSize:0}]
I0913 02:13:40.286542   11616 manager.go:216] Machine: {NumCores:1 CpuFrequency:3503751 MemoryCapacity:1045037056 HugePages:[{PageSize:2048 NumPages:0}] MachineID:0f59a2f640ba4efa8f021839be1c5376 SystemUUID:E1904120-004C-4DF0-816C-0FBB7DC711B1 BootID:84628a57-e2b9-43e0-8782-0f2428737a9f Filesystems:[{Device:shm DeviceMajor:0 DeviceMinor:53 Capacity:67108864 Type:vfs Inodes:127568 HasInodes:true} {Device:overlay DeviceMajor:0 DeviceMinor:40 Capacity:16718393344 Type:vfs Inodes:4234880 HasInodes:true} {Device:tmpfs DeviceMajor:0 DeviceMinor:55 Capacity:522518528 Type:vfs Inodes:127568 HasInodes:true} {Device:/dev/sda9 DeviceMajor:8 DeviceMinor:9 Capacity:16718393344 Type:vfs Inodes:4234880 HasInodes:true} {Device:/dev/mapper/usr DeviceMajor:254 DeviceMinor:0 Capacity:1031946240 Type:vfs Inodes:260096 HasInodes:true} {Device:/dev/sda6 DeviceMajor:8 DeviceMinor:6 Capacity:113229824 Type:vfs Inodes:32768 HasInodes:true}] DiskMap:map[254:0:{Name:dm-0 Major:254 Minor:0 Size:1065345024 Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size:19818086400 Scheduler:cfq}] NetworkDevices:[{Name:eth0 MacAddress:08:00:27:b3:b3:af Speed:-1 Mtu:1500} {Name:eth1 MacAddress:08:00:27:16:b5:5d Speed:-1 Mtu:1500}] Topology:[{Id:0 Memory:1045037056 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:4194304 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
I0913 02:13:40.287265   11616 manager.go:222] Version: {KernelVersion:4.11.0-coreos ContainerOsVersion:Debian GNU/Linux 8 (jessie) DockerVersion:17.05.0-ce DockerAPIVersion:1.29 CadvisorVersion: CadvisorRevision:}
I0913 02:13:40.287581   11616 server.go:408] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
I0913 02:13:40.290612   11616 container_manager_linux.go:252] container manager verified user specified cgroup-root exists: /
I0913 02:13:40.290678   11616 container_manager_linux.go:257] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} ExperimentalQOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s}
I0913 02:13:40.290774   11616 container_manager_linux.go:288] Creating device plugin handler: false
I0913 02:13:40.290864   11616 kubelet.go:271] Adding manifest file: /etc/kubernetes/manifests
I0913 02:13:40.290885   11616 kubelet.go:281] Watching apiserver
W0913 02:13:40.342114   11616 kubelet_network.go:69] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
I0913 02:13:40.342195   11616 kubelet.go:514] Hairpin mode set to "hairpin-veth"
W0913 02:13:40.346982   11616 cni.go:196] Unable to update cni config: No networks found in /etc/cni/net.d
I0913 02:13:40.368037   11616 docker_service.go:207] Docker cri networking managed by kubernetes.io/no-op
I0913 02:13:40.375698   11616 docker_service.go:224] Setting cgroupDriver to cgroupfs
I0913 02:13:40.395897   11616 remote_runtime.go:43] Connecting to runtime service unix:///var/run/dockershim.sock
I0913 02:13:40.397577   11616 kuberuntime_manager.go:177] Container runtime docker initialized, version: 17.05.0-ce, apiVersion: 1.29.0
W0913 02:13:40.398500   11616 probe.go:215] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
I0913 02:13:40.399493   11616 server.go:704] Started kubelet v1.8.0-alpha.3.1179+6a0ebd96c73962-dirty
E0913 02:13:40.399583   11616 kubelet.go:1212] Image garbage collection failed once. Stats initialization may not have completed yet: unable to find data for container /
I0913 02:13:40.402969   11616 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
I0913 02:13:40.403032   11616 status_manager.go:140] Starting to sync pod status with apiserver
I0913 02:13:40.403052   11616 kubelet.go:1746] Starting kubelet main sync loop.
I0913 02:13:40.403082   11616 kubelet.go:1757] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s]
I0913 02:13:40.403177   11616 server.go:128] Starting to listen on 172.17.8.102:10250
I0913 02:13:40.403976   11616 server.go:296] Adding debug handlers to kubelet server.
E0913 02:13:40.411947   11616 container_manager_linux.go:603] [ContainerManager]: Fail to get rootfs information unable to find data for container /
I0913 02:13:40.412013   11616 volume_manager.go:246] Starting Kubelet Volume Manager
I0913 02:13:40.440002   11616 factory.go:355] Registering Docker factory
W0913 02:13:40.440078   11616 manager.go:265] Registration of the rkt container factory failed: unable to communicate with Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
W0913 02:13:40.440175   11616 manager.go:276] Registration of the crio container factory failed: Get http://%2Fvar%2Frun%2Fcrio.sock/info: dial unix /var/run/crio.sock: connect: no such file or directory
I0913 02:13:40.440197   11616 factory.go:54] Registering systemd factory
I0913 02:13:40.441605   11616 factory.go:86] Registering Raw factory
I0913 02:13:40.447607   11616 manager.go:1140] Started watching for new ooms in manager
I0913 02:13:40.448336   11616 manager.go:311] Starting recovery of all containers
I0913 02:13:40.514682   11616 kubelet_node_status.go:276] Setting node annotation to enable volume controller attach/detach
I0913 02:13:40.527049   11616 kubelet_node_status.go:83] Attempting to register node 172.17.8.102
I0913 02:13:40.546173   11616 kubelet_node_status.go:134] Node 172.17.8.102 was previously registered
I0913 02:13:40.546237   11616 kubelet_node_status.go:86] Successfully registered node 172.17.8.102
I0913 02:13:40.576981   11616 manager.go:316] Recovery completed
I0913 02:13:40.627426   11616 kubelet_node_status.go:786] Node became not ready: {Type:Ready Status:False LastHeartbeatTime:2017-09-13 02:13:40.62741201 +0000 UTC LastTransitionTime:2017-09-13 02:13:40.62741201 +0000 UTC Reason:KubeletNotReady Message:container runtime is down. WARNING: CPU hardcapping unsupported}
E0913 02:13:40.821635   11616 helpers.go:826] Could not find capacity information for resource ephemeral-storage
W0913 02:13:40.821719   11616 helpers.go:837] eviction manager: no observation found for eviction signal allocatableNodeFs.available
E0913 02:13:45.414235   11616 kubelet.go:1590] Failed creating a mirror pod for "kube-proxy-172.17.8.102_kube-system(0e20c9966e505a9cf3d881345764dc95)": pods "kube-proxy-172.17.8.102" already exists
I0913 02:13:45.512394   11616 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "ssl-certs" (UniqueName: "kubernetes.io/host-path/0e20c9966e505a9cf3d881345764dc95-ssl-certs") pod "kube-proxy-172.17.8.102" (UID: "0e20c9966e505a9cf3d881345764dc95")
I0913 02:13:45.512544   11616 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/0e20c9966e505a9cf3d881345764dc95-kubeconfig") pod "kube-proxy-172.17.8.102" (UID: "0e20c9966e505a9cf3d881345764dc95")
I0913 02:13:45.512597   11616 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "etc-kube-ssl" (UniqueName: "kubernetes.io/host-path/0e20c9966e505a9cf3d881345764dc95-etc-kube-ssl") pod "kube-proxy-172.17.8.102" (UID: "0e20c9966e505a9cf3d881345764dc95")
I0913 02:13:45.512639   11616 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-dns-config" (UniqueName: "kubernetes.io/configmap/bbcdc458-97a2-11e7-8c7a-080027b3b3af-kube-dns-config") pod "kube-dns-4223714313-n8xmf" (UID: "bbcdc458-97a2-11e7-8c7a-080027b3b3af")
I0913 02:13:45.512680   11616 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-gzx3w" (UniqueName: "kubernetes.io/secret/bbcdc458-97a2-11e7-8c7a-080027b3b3af-default-token-gzx3w") pod "kube-dns-4223714313-n8xmf" (UID: "bbcdc458-97a2-11e7-8c7a-080027b3b3af")
W0913 02:13:46.499175   11616 kuberuntime_container.go:191] Non-root verification doesn't support non-numeric user (nobody)
E0913 02:13:46.845008   11616 helpers.go:468] PercpuUsage had 0 cpus, but the actual number is 1; ignoring extra CPUs
W0913 02:14:00.841144   11616 conversion.go:110] Could not get instant cpu stats: different number of cpus
E0913 02:14:41.192607   11616 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: du command failed on /rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d with output stdout: 727252  /rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d
, stderr: du: cannot access '/rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d/merged/proc/12003/task/12003/fd/3': No such file or directory
du: cannot access '/rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d/merged/proc/12003/task/12003/fdinfo/3': No such file or directory
du: cannot access '/rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d/merged/proc/12003/fd/4': No such file or directory
du: cannot access '/rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d/merged/proc/12003/fdinfo/4': No such file or directory
 - exit status 1, rootInodeErr: <nil>, extraDiskErr: <nil>
I0913 02:15:41.538246   11616 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "test-volume" (UniqueName: "kubernetes.io/host-path/70f55342-9829-11e7-9887-080027b3b3af-test-volume") pod "hostpath" (UID: "70f55342-9829-11e7-9887-080027b3b3af")
I0913 02:15:41.538378   11616 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-8td07" (UniqueName: "kubernetes.io/secret/70f55342-9829-11e7-9887-080027b3b3af-default-token-8td07") pod "hostpath" (UID: "70f55342-9829-11e7-9887-080027b3b3af")
E0913 02:15:41.641797   11616 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/host-path/70f55342-9829-11e7-9887-080027b3b3af-test-volume\" (\"70f55342-9829-11e7-9887-080027b3b3af\")" failed. No retries permitted until 2017-09-13 02:15:42.141797238 +0000 UTC (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "test-volume" (UniqueName: "kubernetes.io/host-path/70f55342-9829-11e7-9887-080027b3b3af-test-volume") pod "hostpath" (UID: "70f55342-9829-11e7-9887-080027b3b3af") : hostPath type check failed: /data is not a socket file
E0913 02:15:42.145170   11616 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/host-path/70f55342-9829-11e7-9887-080027b3b3af-test-volume\" (\"70f55342-9829-11e7-9887-080027b3b3af\")" failed. No retries permitted until 2017-09-13 02:15:43.145148435 +0000 UTC (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "test-volume" (UniqueName: "kubernetes.io/host-path/70f55342-9829-11e7-9887-080027b3b3af-test-volume") pod "hostpath" (UID: "70f55342-9829-11e7-9887-080027b3b3af") : hostPath type check failed: /data is not a socket file
E0913 02:15:43.152506   11616 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/host-path/70f55342-9829-11e7-9887-080027b3b3af-test-volume\" (\"70f55342-9829-11e7-9887-080027b3b3af\")" failed. No retries permitted until 2017-09-13 02:15:45.152474594 +0000 UTC (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "test-volume" (UniqueName: "kubernetes.io/host-path/70f55342-9829-11e7-9887-080027b3b3af-test-volume") pod "hostpath" (UID: "70f55342-9829-11e7-9887-080027b3b3af") : hostPath type check failed: /data is not a socket file
E0913 02:15:45.166836   11616 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/host-path/70f55342-9829-11e7-9887-080027b3b3af-test-volume\" (\"70f55342-9829-11e7-9887-080027b3b3af\")" failed. No retries permitted until 2017-09-13 02:15:49.166814184 +0000 UTC (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "test-volume" (UniqueName: "kubernetes.io/host-path/70f55342-9829-11e7-9887-080027b3b3af-test-volume") pod "hostpath" (UID: "70f55342-9829-11e7-9887-080027b3b3af") : hostPath type check failed: /data is not a socket file
E0913 02:15:49.205310   11616 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/host-path/70f55342-9829-11e7-9887-080027b3b3af-test-volume\" (\"70f55342-9829-11e7-9887-080027b3b3af\")" failed. No retries permitted until 2017-09-13 02:15:57.2052852 +0000 UTC (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "test-volume" (UniqueName: "kubernetes.io/host-path/70f55342-9829-11e7-9887-080027b3b3af-test-volume") pod "hostpath" (UID: "70f55342-9829-11e7-9887-080027b3b3af") : hostPath type check failed: /data is not a socket file
E0913 02:15:57.277220   11616 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/host-path/70f55342-9829-11e7-9887-080027b3b3af-test-volume\" (\"70f55342-9829-11e7-9887-080027b3b3af\")" failed. No retries permitted until 2017-09-13 02:16:13.277194203 +0000 UTC (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "test-volume" (UniqueName: "kubernetes.io/host-path/70f55342-9829-11e7-9887-080027b3b3af-test-volume") pod "hostpath" (UID: "70f55342-9829-11e7-9887-080027b3b3af") : hostPath type check failed: /data is not a socket file
E0913 02:16:13.351910   11616 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/host-path/70f55342-9829-11e7-9887-080027b3b3af-test-volume\" (\"70f55342-9829-11e7-9887-080027b3b3af\")" failed. No retries permitted until 2017-09-13 02:16:45.351875572 +0000 UTC (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "test-volume" (UniqueName: "kubernetes.io/host-path/70f55342-9829-11e7-9887-080027b3b3af-test-volume") pod "hostpath" (UID: "70f55342-9829-11e7-9887-080027b3b3af") : hostPath type check failed: /data is not a socket file
E0913 02:16:41.415632   11616 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: du command failed on /rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d with output stdout: 727252  /rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d
, stderr: du: cannot access '/rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d/merged/proc/12119/task/12119/fd/3': No such file or directory
du: cannot access '/rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d/merged/proc/12119/task/12119/fdinfo/3': No such file or directory
du: cannot access '/rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d/merged/proc/12119/fd/4': No such file or directory
du: cannot access '/rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d/merged/proc/12119/fdinfo/4': No such file or directory
 - exit status 1, rootInodeErr: <nil>, extraDiskErr: <nil>
E0913 02:16:45.402644   11616 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/host-path/70f55342-9829-11e7-9887-080027b3b3af-test-volume\" (\"70f55342-9829-11e7-9887-080027b3b3af\")" failed. No retries permitted until 2017-09-13 02:17:49.4026201 +0000 UTC (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "test-volume" (UniqueName: "kubernetes.io/host-path/70f55342-9829-11e7-9887-080027b3b3af-test-volume") pod "hostpath" (UID: "70f55342-9829-11e7-9887-080027b3b3af") : hostPath type check failed: /data is not a socket file
I0913 02:16:59.332058   11616 reconciler.go:186] operationExecutor.UnmountVolume started for volume "default-token-8td07" (UniqueName: "kubernetes.io/secret/70f55342-9829-11e7-9887-080027b3b3af-default-token-8td07") pod "70f55342-9829-11e7-9887-080027b3b3af" (UID: "70f55342-9829-11e7-9887-080027b3b3af")
I0913 02:16:59.332933   11616 reconciler.go:290] Volume detached for volume "test-volume" (UniqueName: "kubernetes.io/host-path/70f55342-9829-11e7-9887-080027b3b3af-test-volume") on node "172.17.8.102" DevicePath ""
I0913 02:16:59.354473   11616 operation_generator.go:535] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f55342-9829-11e7-9887-080027b3b3af-default-token-8td07" (OuterVolumeSpecName: "default-token-8td07") pod "70f55342-9829-11e7-9887-080027b3b3af" (UID: "70f55342-9829-11e7-9887-080027b3b3af"). InnerVolumeSpecName "default-token-8td07". PluginName "kubernetes.io/secret", VolumeGidValue ""
I0913 02:16:59.433138   11616 reconciler.go:290] Volume detached for volume "default-token-8td07" (UniqueName: "kubernetes.io/secret/70f55342-9829-11e7-9887-080027b3b3af-default-token-8td07") on node "172.17.8.102" DevicePath ""
E0913 02:17:44.432035   11616 kubelet.go:1606] Unable to mount volumes for pod "hostpath_default(70f55342-9829-11e7-9887-080027b3b3af)": timeout expired waiting for volumes to attach/mount for pod "default"/"hostpath". list of unattached/unmounted volumes=[test-volume default-token-8td07]; skipping pod
E0913 02:17:44.432165   11616 pod_workers.go:182] Error syncing pod 70f55342-9829-11e7-9887-080027b3b3af ("hostpath_default(70f55342-9829-11e7-9887-080027b3b3af)"), skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"hostpath". list of unattached/unmounted volumes=[test-volume default-token-8td07]
I0913 02:17:47.477456   11616 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "test-volume" (UniqueName: "kubernetes.io/host-path/bc110a19-9829-11e7-9887-080027b3b3af-test-volume") pod "hostpath" (UID: "bc110a19-9829-11e7-9887-080027b3b3af")
I0913 02:17:47.477574   11616 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-8td07" (UniqueName: "kubernetes.io/secret/bc110a19-9829-11e7-9887-080027b3b3af-default-token-8td07") pod "hostpath" (UID: "bc110a19-9829-11e7-9887-080027b3b3af")
E0913 02:17:47.587997   11616 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/host-path/bc110a19-9829-11e7-9887-080027b3b3af-test-volume\" (\"bc110a19-9829-11e7-9887-080027b3b3af\")" failed. No retries permitted until 2017-09-13 02:17:48.087981425 +0000 UTC (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "test-volume" (UniqueName: "kubernetes.io/host-path/bc110a19-9829-11e7-9887-080027b3b3af-test-volume") pod "hostpath" (UID: "bc110a19-9829-11e7-9887-080027b3b3af") : hostPath type check failed: /data is not a directory
E0913 02:17:48.183263   11616 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/host-path/bc110a19-9829-11e7-9887-080027b3b3af-test-volume\" (\"bc110a19-9829-11e7-9887-080027b3b3af\")" failed. No retries permitted until 2017-09-13 02:17:49.183226654 +0000 UTC (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "test-volume" (UniqueName: "kubernetes.io/host-path/bc110a19-9829-11e7-9887-080027b3b3af-test-volume") pod "hostpath" (UID: "bc110a19-9829-11e7-9887-080027b3b3af") : hostPath type check failed: /data is not a directory
E0913 02:17:49.193803   11616 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/host-path/bc110a19-9829-11e7-9887-080027b3b3af-test-volume\" (\"bc110a19-9829-11e7-9887-080027b3b3af\")" failed. No retries permitted until 2017-09-13 02:17:51.193767553 +0000 UTC (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "test-volume" (UniqueName: "kubernetes.io/host-path/bc110a19-9829-11e7-9887-080027b3b3af-test-volume") pod "hostpath" (UID: "bc110a19-9829-11e7-9887-080027b3b3af") : hostPath type check failed: /data is not a directory
E0913 02:17:51.215954   11616 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/host-path/bc110a19-9829-11e7-9887-080027b3b3af-test-volume\" (\"bc110a19-9829-11e7-9887-080027b3b3af\")" failed. No retries permitted until 2017-09-13 02:17:55.215934778 +0000 UTC (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "test-volume" (UniqueName: "kubernetes.io/host-path/bc110a19-9829-11e7-9887-080027b3b3af-test-volume") pod "hostpath" (UID: "bc110a19-9829-11e7-9887-080027b3b3af") : hostPath type check failed: /data is not a directory
E0913 02:17:55.257548   11616 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/host-path/bc110a19-9829-11e7-9887-080027b3b3af-test-volume\" (\"bc110a19-9829-11e7-9887-080027b3b3af\")" failed. No retries permitted until 2017-09-13 02:18:03.257524247 +0000 UTC (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "test-volume" (UniqueName: "kubernetes.io/host-path/bc110a19-9829-11e7-9887-080027b3b3af-test-volume") pod "hostpath" (UID: "bc110a19-9829-11e7-9887-080027b3b3af") : hostPath type check failed: /data is not a directory
E0913 02:18:03.331307   11616 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/host-path/bc110a19-9829-11e7-9887-080027b3b3af-test-volume\" (\"bc110a19-9829-11e7-9887-080027b3b3af\")" failed. No retries permitted until 2017-09-13 02:18:19.331280852 +0000 UTC (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "test-volume" (UniqueName: "kubernetes.io/host-path/bc110a19-9829-11e7-9887-080027b3b3af-test-volume") pod "hostpath" (UID: "bc110a19-9829-11e7-9887-080027b3b3af") : hostPath type check failed: /data is not a directory
I0913 02:18:18.049302   11616 reconciler.go:186] operationExecutor.UnmountVolume started for volume "default-token-8td07" (UniqueName: "kubernetes.io/secret/bc110a19-9829-11e7-9887-080027b3b3af-default-token-8td07") pod "bc110a19-9829-11e7-9887-080027b3b3af" (UID: "bc110a19-9829-11e7-9887-080027b3b3af")
I0913 02:18:18.049928   11616 reconciler.go:290] Volume detached for volume "test-volume" (UniqueName: "kubernetes.io/host-path/bc110a19-9829-11e7-9887-080027b3b3af-test-volume") on node "172.17.8.102" DevicePath ""
I0913 02:18:18.063959   11616 operation_generator.go:535] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc110a19-9829-11e7-9887-080027b3b3af-default-token-8td07" (OuterVolumeSpecName: "default-token-8td07") pod "bc110a19-9829-11e7-9887-080027b3b3af" (UID: "bc110a19-9829-11e7-9887-080027b3b3af"). InnerVolumeSpecName "default-token-8td07". PluginName "kubernetes.io/secret", VolumeGidValue ""
I0913 02:18:18.151494   11616 reconciler.go:290] Volume detached for volume "default-token-8td07" (UniqueName: "kubernetes.io/secret/bc110a19-9829-11e7-9887-080027b3b3af-default-token-8td07") on node "172.17.8.102" DevicePath ""
I0913 02:18:59.717385   11616 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "test-volume" (UniqueName: "kubernetes.io/host-path/e71da115-9829-11e7-9887-080027b3b3af-test-volume") pod "hostpath" (UID: "e71da115-9829-11e7-9887-080027b3b3af")
I0913 02:18:59.717460   11616 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-8td07" (UniqueName: "kubernetes.io/secret/e71da115-9829-11e7-9887-080027b3b3af-default-token-8td07") pod "hostpath" (UID: "e71da115-9829-11e7-9887-080027b3b3af")
E0913 02:18:59.860185   11616 helpers.go:468] PercpuUsage had 0 cpus, but the actual number is 1; ignoring extra CPUs
E0913 02:19:50.445275   11616 kubelet.go:1606] Unable to mount volumes for pod "hostpath_default(bc110a19-9829-11e7-9887-080027b3b3af)": timeout expired waiting for volumes to attach/mount for pod "default"/"hostpath". list of unattached/unmounted volumes=[test-volume default-token-8td07]; skipping pod
E0913 02:19:50.445659   11616 pod_workers.go:182] Error syncing pod bc110a19-9829-11e7-9887-080027b3b3af ("hostpath_default(bc110a19-9829-11e7-9887-080027b3b3af)"), skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"hostpath". list of unattached/unmounted volumes=[test-volume default-token-8td07]
E0913 02:20:41.657402   11616 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: du command failed on /rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d with output stdout: 727332  /rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d
, stderr: du: cannot access '/rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d/merged/proc/12450/task/12450/fd/3': No such file or directory
du: cannot access '/rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d/merged/proc/12450/task/12450/fdinfo/3': No such file or directory
du: cannot access '/rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d/merged/proc/12450/fd/4': No such file or directory
du: cannot access '/rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d/merged/proc/12450/fdinfo/4': No such file or directory
 - exit status 1, rootInodeErr: <nil>, extraDiskErr: <nil>
E0913 02:28:41.890020   11616 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: du command failed on /rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d with output stdout: 727332  /rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d
, stderr: du: cannot access '/rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d/merged/proc/12931/task/12931/fd/3': No such file or directory
du: cannot access '/rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d/merged/proc/12931/task/12931/fdinfo/3': No such file or directory
du: cannot access '/rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d/merged/proc/12931/fd/4': No such file or directory
du: cannot access '/rootfs/var/lib/docker/overlay2/db26f414b3746aa5b60fb62a2c07604959f747536aa0db3a0aa356ce044f910d/merged/proc/12931/fdinfo/4': No such file or directory
 - exit status 1, rootInodeErr: <nil>, extraDiskErr: <nil>
E0913 02:32:49.729982   11616 helpers.go:468] PercpuUsage had 0 cpus, but the actual number is 1; ignoring extra CPUs
I0913 02:32:49.730199   11616 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-8td07" (UniqueName: "kubernetes.io/secret/d5dcfe89-982b-11e7-9887-080027b3b3af-default-token-8td07") pod "web2-59dd8c8747-5mvvx" (UID: "d5dcfe89-982b-11e7-9887-080027b3b3af")
E0913 02:32:49.767443   11616 helpers.go:468] PercpuUsage had 0 cpus, but the actual number is 1; ignoring extra CPUs
W0913 02:32:49.889734   11616 container.go:367] Failed to get RecentStats("/system.slice/run-r31aa95315b59438bab89fb3283c148e9.scope") while determining the next housekeeping: unable to find data for container /system.slice/run-r31aa95315b59438bab89fb3283c148e9.scope
I0913 02:32:49.930920   11616 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-8td07" (UniqueName: "kubernetes.io/secret/d5de1a77-982b-11e7-9887-080027b3b3af-default-token-8td07") pod "web2-59dd8c8747-kzjhs" (UID: "d5de1a77-982b-11e7-9887-080027b3b3af")
I0913 02:32:49.931042   11616 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-8td07" (UniqueName: "kubernetes.io/secret/d5e16de9-982b-11e7-9887-080027b3b3af-default-token-8td07") pod "web2-59dd8c8747-6hj6z" (UID: "d5e16de9-982b-11e7-9887-080027b3b3af")
I0913 02:32:49.931090   11616 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-8td07" (UniqueName: "kubernetes.io/secret/d5de660f-982b-11e7-9887-080027b3b3af-default-token-8td07") pod "web2-59dd8c8747-6k8nh" (UID: "d5de660f-982b-11e7-9887-080027b3b3af")
I0913 02:32:49.931122   11616 reconciler.go:212] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-8td07" (UniqueName: "kubernetes.io/secret/d5e184c5-982b-11e7-9887-080027b3b3af-default-token-8td07") pod "web2-59dd8c8747-9b4h4" (UID: "d5e184c5-982b-11e7-9887-080027b3b3af")
W0913 02:33:02.384196   11616 conversion.go:110] Could not get instant cpu stats: different number of cpus
W0913 02:33:12.401948   11616 conversion.go:110] Could not get instant cpu stats: different number of cpus
W0913 02:33:12.403661   11616 conversion.go:110] Could not get instant cpu stats: different number of cpus