Mount/Unmount Error

gistfile1.txt

Nov 09 05:22:33 ip-172-20-61-52 kubelet[1579]: I1109 05:22:33.565089    1579 kubelet.go:1883] SyncLoop (ADD, "api"): "app_default(763983fe-e3df-11e8-baec-0a3a1cea024a)"
Nov 09 05:22:33 ip-172-20-61-52 kubelet[1579]: I1109 05:22:33.729498    1579 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "test-pv" (Unique
Name: "kubernetes.io/aws-ebs/vol-0eedee73dff51ca75") pod "app" (UID: "763983fe-e3df-11e8-baec-0a3a1cea024a")
Nov 09 05:22:33 ip-172-20-61-52 kubelet[1579]: I1109 05:22:33.729552    1579 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-7c
x7h" (UniqueName: "kubernetes.io/secret/763983fe-e3df-11e8-baec-0a3a1cea024a-default-token-7cx7h") pod "app" (UID: "763983fe-e3df-11e8-baec-0a3a1cea024a")
Nov 09 05:22:33 ip-172-20-61-52 kubelet[1579]: E1109 05:22:33.729742    1579 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/aws-ebs/vol-0eedee73dff51ca75\"" failed.
No retries permitted until 2018-11-09 05:22:34.229616326 +0000 UTC m=+27420.808969191 (durationBeforeRetry 500ms). Error: "Volume has not been added to the list of VolumesInUse in t
he node's volume status for volume \"test-pv\" (UniqueName: \"kubernetes.io/aws-ebs/vol-0eedee73dff51ca75\") pod \"app\" (UID: \"763983fe-e3df-11e8-baec-0a3a1cea024a\") "
Nov 09 05:22:33 ip-172-20-61-52 kubelet[1579]: I1109 05:22:33.829878    1579 reconciler.go:252] operationExecutor.MountVolume started for volume "default-token-7cx7h" (UniqueName: "
kubernetes.io/secret/763983fe-e3df-11e8-baec-0a3a1cea024a-default-token-7cx7h") pod "app" (UID: "763983fe-e3df-11e8-baec-0a3a1cea024a")
Nov 09 05:22:33 ip-172-20-61-52 kubelet[1579]: I1109 05:22:33.837079    1579 operation_generator.go:635] MountVolume.SetUp succeeded for volume "default-token-7cx7h" (UniqueName: "k
ubernetes.io/secret/763983fe-e3df-11e8-baec-0a3a1cea024a-default-token-7cx7h") pod "app" (UID: "763983fe-e3df-11e8-baec-0a3a1cea024a")
Nov 09 05:22:34 ip-172-20-61-52 kubelet[1579]: I1109 05:22:34.231500    1579 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "test-pv" (Unique
Name: "kubernetes.io/aws-ebs/vol-0eedee73dff51ca75") pod "app" (UID: "763983fe-e3df-11e8-baec-0a3a1cea024a")
Nov 09 05:22:34 ip-172-20-61-52 kubelet[1579]: E1109 05:22:34.231603    1579 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/aws-ebs/vol-0eedee73dff51ca75\"" failed.
No retries permitted until 2018-11-09 05:22:35.231575853 +0000 UTC m=+27421.810928785 (durationBeforeRetry 1s). Error: "Volume has not been added to the list of VolumesInUse in the
node's volume status for volume \"test-pv\" (UniqueName: \"kubernetes.io/aws-ebs/vol-0eedee73dff51ca75\") pod \"app\" (UID: \"763983fe-e3df-11e8-baec-0a3a1cea024a\") "
Nov 09 05:22:35 ip-172-20-61-52 kubelet[1579]: I1109 05:22:35.234103    1579 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "test-pv" (Unique
Name: "kubernetes.io/aws-ebs/vol-0eedee73dff51ca75") pod "app" (UID: "763983fe-e3df-11e8-baec-0a3a1cea024a")
Nov 09 05:22:35 ip-172-20-61-52 kubelet[1579]: E1109 05:22:35.234198    1579 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/aws-ebs/vol-0eedee73dff51ca75\"" failed.
No retries permitted until 2018-11-09 05:22:37.234169454 +0000 UTC m=+27423.813522381 (durationBeforeRetry 2s). Error: "Volume has not been added to the list of VolumesInUse in the
node's volume status for volume \"test-pv\" (UniqueName: \"kubernetes.io/aws-ebs/vol-0eedee73dff51ca75\") pod \"app\" (UID: \"763983fe-e3df-11e8-baec-0a3a1cea024a\") "
Nov 09 05:22:37 ip-172-20-61-52 kubelet[1579]: I1109 05:22:37.238832    1579 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "test-pv" (Unique
Name: "kubernetes.io/aws-ebs/vol-0eedee73dff51ca75") pod "app" (UID: "763983fe-e3df-11e8-baec-0a3a1cea024a")
Nov 09 05:22:37 ip-172-20-61-52 kubelet[1579]: E1109 05:22:37.239466    1579 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/aws-ebs/vol-0eedee73dff51ca75\"" failed.
No retries permitted until 2018-11-09 05:22:41.239437145 +0000 UTC m=+27427.818790026 (durationBeforeRetry 4s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"test-pv\" (UniqueName: \"kubernetes.io/aws-ebs/vol-0eedee73dff51ca75\") pod \"app\" (UID: \"763983fe-e3df-11e8-baec-0a3a1cea024a\") "
Nov 09 05:22:41 ip-172-20-61-52 kubelet[1579]: I1109 05:22:41.248987    1579 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "test-pv" (UniqueName: "kubernetes.io/aws-ebs/vol-0eedee73dff51ca75") pod "app" (UID: "763983fe-e3df-11e8-baec-0a3a1cea024a")
Nov 09 05:22:41 ip-172-20-61-52 kubelet[1579]: E1109 05:22:41.249099    1579 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/aws-ebs/vol-0eedee73dff51ca75\"" failed. No retries permitted until 2018-11-09 05:22:49.249074233 +0000 UTC m=+27435.828427098 (durationBeforeRetry 8s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"test-pv\" (UniqueName: \"kubernetes.io/aws-ebs/vol-0eedee73dff51ca75\") pod \"app\" (UID: \"763983fe-e3df-11e8-baec-0a3a1cea024a\") "
Nov 09 05:22:43 ip-172-20-61-52 kubelet[1579]: W1109 05:22:43.022890    1579 setters.go:144] replacing cloudprovider-reported hostname of ip-172-20-61-52.ec2.internal with overridden hostname of ip-172-20-61-52.ec2.internal
Nov 09 05:22:49 ip-172-20-61-52 kubelet[1579]: I1109 05:22:49.269760    1579 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "test-pv" (UniqueName: "kubernetes.io/aws-ebs/vol-0eedee73dff51ca75") pod "app" (UID: "763983fe-e3df-11e8-baec-0a3a1cea024a")
Nov 09 05:22:49 ip-172-20-61-52 kubelet[1579]: W1109 05:22:49.273062    1579 operation_generator.go:1298] volumeToMount.VolumeName: kubernetes.io/aws-ebs/vol-0eedee73dff51ca75
Nov 09 05:22:49 ip-172-20-61-52 kubelet[1579]: W1109 05:22:49.273086    1579 operation_generator.go:1299] node.Status.VolumesAttached: [{kubernetes.io/aws-ebs/vol-0eedee73dff51ca75 csi-b7a72fc7d19e2b8f44332833e726788c8afe1be897e6faaabd92cedc47a84cff}]
Nov 09 05:22:49 ip-172-20-61-52 kubelet[1579]: W1109 05:22:49.273098    1579 operation_generator.go:1301] attachedVolume.Name: kubernetes.io/aws-ebs/vol-0eedee73dff51ca75
Nov 09 05:22:49 ip-172-20-61-52 kubelet[1579]: I1109 05:22:49.273122    1579 operation_generator.go:1305] Controller attach succeeded for volume "test-pv" (UniqueName: "kubernetes.io/aws-ebs/vol-0eedee73dff51ca75") pod "app" (UID: "763983fe-e3df-11e8-baec-0a3a1cea024a") device path: "csi-b7a72fc7d19e2b8f44332833e726788c8afe1be897e6faaabd92cedc47a84cff"
Nov 09 05:22:49 ip-172-20-61-52 kubelet[1579]: I1109 05:22:49.370306    1579 reconciler.go:252] operationExecutor.MountVolume started for volume "test-pv" (UniqueName: "kubernetes.io/aws-ebs/vol-0eedee73dff51ca75") pod "app" (UID: "763983fe-e3df-11e8-baec-0a3a1cea024a")
Nov 09 05:22:49 ip-172-20-61-52 kubelet[1579]: I1109 05:22:49.370360    1579 operation_generator.go:565] MountVolume.WaitForAttach entering for volume "test-pv" (UniqueName: "kubernetes.io/aws-ebs/vol-0eedee73dff51ca75") pod "app" (UID: "763983fe-e3df-11e8-baec-0a3a1cea024a") DevicePath "csi-b7a72fc7d19e2b8f44332833e726788c8afe1be897e6faaabd92cedc47a84cff"
Nov 09 05:22:49 ip-172-20-61-52 kubelet[1579]: I1109 05:22:49.372720    1579 operation_generator.go:574] MountVolume.WaitForAttach succeeded for volume "test-pv" (UniqueName: "kubernetes.io/aws-ebs/vol-0eedee73dff51ca75") pod "app" (UID: "763983fe-e3df-11e8-baec-0a3a1cea024a") DevicePath "csi-b7a72fc7d19e2b8f44332833e726788c8afe1be897e6faaabd92cedc47a84cff"
Nov 09 05:22:49 ip-172-20-61-52 kubelet[1579]: I1109 05:22:49.406791    1579 operation_generator.go:595] MountVolume.MountDevice succeeded for volume "test-pv" (UniqueName: "kubernetes.io/aws-ebs/vol-0eedee73dff51ca75") pod "app" (UID: "763983fe-e3df-11e8-baec-0a3a1cea024a") device mount path "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/test-pv/globalmount"
Nov 09 05:22:49 ip-172-20-61-52 kubelet[1579]: I1109 05:22:49.415952    1579 operation_generator.go:635] MountVolume.SetUp succeeded for volume "test-pv" (UniqueName: "kubernetes.io/aws-ebs/vol-0eedee73dff51ca75") pod "app" (UID: "763983fe-e3df-11e8-baec-0a3a1cea024a")
Nov 09 05:22:49 ip-172-20-61-52 kubelet[1579]: I1109 05:22:49.474297    1579 kuberuntime_manager.go:397] No sandbox for pod "app_default(763983fe-e3df-11e8-baec-0a3a1cea024a)" can be found. Need to start a new one