Skip to content

Instantly share code, notes, and snippets.

@jmcarp

jmcarp/controller-manager

Last active December 14, 2016 19:53
Show Gist options
  • Save jmcarp/284562d081d5102b073d33434607717c to your computer and use it in GitHub Desktop.
Save jmcarp/284562d081d5102b073d33434607717c to your computer and use it in GitHub Desktop.
Download ZIP
k8s-ebs-debug
Raw
controller-manager
------------ STARTING controller-manager_ctl at Wed Dec 14 18:38:00 UTC 2016 --------------
I1214 18:38:00.697691 5417 leaderelection.go:247] lock is held by 306e0ee4-b6a8-454b-b199-6c8547deb849 and has not yet expired
I1214 18:38:04.160237 5417 leaderelection.go:188] sucessfully acquired lease kube-system/kube-controller-manager
I1214 18:38:04.160557 5417 event.go:217] Event(api.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-controller-manager", UID:"72c92afa-c22c-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5820764", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' 7ec92627-d913-44ad-b3f7-3ea5c1fb9b65 became leader
I1214 18:38:04.169568 5417 aws.go:746] Building AWS cloudprovider
I1214 18:38:04.169636 5417 aws.go:700] Zone not specified in configuration file; querying AWS metadata service
I1214 18:38:04.171050 5417 replication_controller.go:219] Starting RC Manager
I1214 18:38:04.284034 5417 replication_controller.go:653] Waiting for pods controller to sync, requeuing rc kube-system/kube-dns-v17.1
I1214 18:38:04.359264 5417 aws.go:832] AWS cloud filtering on tags: map[KubernetesCluster:kubernetes-staging]
W1214 18:38:04.359520 5417 controllermanager.go:285] Unsuccessful parsing of cluster CIDR : invalid CIDR address:
W1214 18:38:04.359651 5417 controllermanager.go:289] Unsuccessful parsing of service CIDR : invalid CIDR address:
I1214 18:38:04.360061 5417 nodecontroller.go:189] Sending events to api server.
I1214 18:38:04.360732 5417 controllermanager.go:322] Will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true.
E1214 18:38:04.361448 5417 util.go:45] Metric for replenishment_controller already registered
E1214 18:38:04.361500 5417 util.go:45] Metric for replenishment_controller already registered
E1214 18:38:04.361600 5417 util.go:45] Metric for replenishment_controller already registered
E1214 18:38:04.361681 5417 util.go:45] Metric for replenishment_controller already registered
E1214 18:38:04.361778 5417 util.go:45] Metric for replenishment_controller already registered
I1214 18:38:04.396843 5417 replication_controller.go:653] Waiting for pods controller to sync, requeuing rc kube-system/kube-dns-v17.1
I1214 18:38:04.398255 5417 controllermanager.go:403] Starting extensions/v1beta1 apis
I1214 18:38:04.398329 5417 controllermanager.go:406] Starting daemon set controller
I1214 18:38:04.409540 5417 controllermanager.go:413] Starting job controller
I1214 18:38:04.410394 5417 controllermanager.go:420] Starting deployment controller
I1214 18:38:04.411289 5417 controllermanager.go:427] Starting ReplicaSet controller
I1214 18:38:04.411953 5417 daemoncontroller.go:192] Starting Daemon Sets controller manager
I1214 18:38:04.412003 5417 deployment_controller.go:132] Starting deployment controller
I1214 18:38:04.415326 5417 controllermanager.go:436] Attempting to start horizontal pod autoscaler controller, full resource map map[batch/v1:&APIResourceList{GroupVersion:batch/v1,APIResources:[{jobs true Job} {jobs/status true Job}],} extensions/v1beta1:&APIResourceList{GroupVersion:extensions/v1beta1,APIResources:[{daemonsets true DaemonSet} {daemonsets/status true DaemonSet} {deployments true Deployment} {deployments/rollback true DeploymentRollback} {deployments/scale true Scale} {deployments/status true Deployment} {horizontalpodautoscalers true HorizontalPodAutoscaler} {horizontalpodautoscalers/status true HorizontalPodAutoscaler} {ingresses true Ingress} {ingresses/status true Ingress} {jobs true Job} {jobs/status true Job} {networkpolicies true NetworkPolicy} {replicasets true ReplicaSet} {replicasets/scale true Scale} {replicasets/status true ReplicaSet} {replicationcontrollers true ReplicationControllerDummy} {replicationcontrollers/scale true Scale} {thirdpartyresources false ThirdPartyResource}],} policy/v1beta1:&APIResourceList{GroupVersion:policy/v1beta1,APIResources:[{poddisruptionbudgets true PodDisruptionBudget} {poddisruptionbudgets/status true PodDisruptionBudget}],} rbac.authorization.k8s.io/v1alpha1:&APIResourceList{GroupVersion:rbac.authorization.k8s.io/v1alpha1,APIResources:[{clusterrolebindings false ClusterRoleBinding} {clusterroles false ClusterRole} {rolebindings true RoleBinding} {roles true Role}],} apps/v1beta1:&APIResourceList{GroupVersion:apps/v1beta1,APIResources:[{statefulsets true StatefulSet} {statefulsets/status true StatefulSet}],} authentication.k8s.io/v1beta1:&APIResourceList{GroupVersion:authentication.k8s.io/v1beta1,APIResources:[{tokenreviews false TokenReview}],} authorization.k8s.io/v1beta1:&APIResourceList{GroupVersion:authorization.k8s.io/v1beta1,APIResources:[{localsubjectaccessreviews true LocalSubjectAccessReview} {selfsubjectaccessreviews false SelfSubjectAccessReview} {subjectaccessreviews false SubjectAccessReview}],} v1:&APIResourceList{GroupVersion:v1,APIResources:[{bindings true Binding} {componentstatuses false ComponentStatus} {configmaps true ConfigMap} {endpoints true Endpoints} {events true Event} {limitranges true LimitRange} {namespaces false Namespace} {namespaces/finalize false Namespace} {namespaces/status false Namespace} {nodes false Node} {nodes/proxy false Node} {nodes/status false Node} {persistentvolumeclaims true PersistentVolumeClaim} {persistentvolumeclaims/status true PersistentVolumeClaim} {persistentvolumes false PersistentVolume} {persistentvolumes/status false PersistentVolume} {pods true Pod} {pods/attach true Pod} {pods/binding true Binding} {pods/eviction true Eviction} {pods/exec true Pod} {pods/log true Pod} {pods/portforward true Pod} {pods/proxy true Pod} {pods/status true Pod} {podtemplates true PodTemplate} {replicationcontrollers true ReplicationController} {replicationcontrollers/scale true Scale} {replicationcontrollers/status true ReplicationController} {resourcequotas true ResourceQuota} {resourcequotas/status true ResourceQuota} {secrets true Secret} {serviceaccounts true ServiceAccount} {services true Service} {services/proxy true Service} {services/status true Service}],} autoscaling/v1:&APIResourceList{GroupVersion:autoscaling/v1,APIResources:[{horizontalpodautoscalers true HorizontalPodAutoscaler} {horizontalpodautoscalers/status true HorizontalPodAutoscaler}],} certificates.k8s.io/v1alpha1:&APIResourceList{GroupVersion:certificates.k8s.io/v1alpha1,APIResources:[{certificatesigningrequests false CertificateSigningRequest} {certificatesigningrequests/approval false CertificateSigningRequest} {certificatesigningrequests/status false CertificateSigningRequest}],} storage.k8s.io/v1beta1:&APIResourceList{GroupVersion:storage.k8s.io/v1beta1,APIResources:[{storageclasses false StorageClass}],}]
I1214 18:38:04.415669 5417 controllermanager.go:438] Starting autoscaling/v1 apis
I1214 18:38:04.415712 5417 controllermanager.go:440] Starting horizontal pod controller.
I1214 18:38:04.416077 5417 controllermanager.go:458] Attempting to start disruption controller, full resource map map[autoscaling/v1:&APIResourceList{GroupVersion:autoscaling/v1,APIResources:[{horizontalpodautoscalers true HorizontalPodAutoscaler} {horizontalpodautoscalers/status true HorizontalPodAutoscaler}],} certificates.k8s.io/v1alpha1:&APIResourceList{GroupVersion:certificates.k8s.io/v1alpha1,APIResources:[{certificatesigningrequests false CertificateSigningRequest} {certificatesigningrequests/approval false CertificateSigningRequest} {certificatesigningrequests/status false CertificateSigningRequest}],} storage.k8s.io/v1beta1:&APIResourceList{GroupVersion:storage.k8s.io/v1beta1,APIResources:[{storageclasses false StorageClass}],} v1:&APIResourceList{GroupVersion:v1,APIResources:[{bindings true Binding} {componentstatuses false ComponentStatus} {configmaps true ConfigMap} {endpoints true Endpoints} {events true Event} {limitranges true LimitRange} {namespaces false Namespace} {namespaces/finalize false Namespace} {namespaces/status false Namespace} {nodes false Node} {nodes/proxy false Node} {nodes/status false Node} {persistentvolumeclaims true PersistentVolumeClaim} {persistentvolumeclaims/status true PersistentVolumeClaim} {persistentvolumes false PersistentVolume} {persistentvolumes/status false PersistentVolume} {pods true Pod} {pods/attach true Pod} {pods/binding true Binding} {pods/eviction true Eviction} {pods/exec true Pod} {pods/log true Pod} {pods/portforward true Pod} {pods/proxy true Pod} {pods/status true Pod} {podtemplates true PodTemplate} {replicationcontrollers true ReplicationController} {replicationcontrollers/scale true Scale} {replicationcontrollers/status true ReplicationController} {resourcequotas true ResourceQuota} {resourcequotas/status true ResourceQuota} {secrets true Secret} {serviceaccounts true ServiceAccount} {services true Service} {services/proxy true Service} {services/status true Service}],} rbac.authorization.k8s.io/v1alpha1:&APIResourceList{GroupVersion:rbac.authorization.k8s.io/v1alpha1,APIResources:[{clusterrolebindings false ClusterRoleBinding} {clusterroles false ClusterRole} {rolebindings true RoleBinding} {roles true Role}],} apps/v1beta1:&APIResourceList{GroupVersion:apps/v1beta1,APIResources:[{statefulsets true StatefulSet} {statefulsets/status true StatefulSet}],} authentication.k8s.io/v1beta1:&APIResourceList{GroupVersion:authentication.k8s.io/v1beta1,APIResources:[{tokenreviews false TokenReview}],} authorization.k8s.io/v1beta1:&APIResourceList{GroupVersion:authorization.k8s.io/v1beta1,APIResources:[{localsubjectaccessreviews true LocalSubjectAccessReview} {selfsubjectaccessreviews false SelfSubjectAccessReview} {subjectaccessreviews false SubjectAccessReview}],} batch/v1:&APIResourceList{GroupVersion:batch/v1,APIResources:[{jobs true Job} {jobs/status true Job}],} extensions/v1beta1:&APIResourceList{GroupVersion:extensions/v1beta1,APIResources:[{daemonsets true DaemonSet} {daemonsets/status true DaemonSet} {deployments true Deployment} {deployments/rollback true DeploymentRollback} {deployments/scale true Scale} {deployments/status true Deployment} {horizontalpodautoscalers true HorizontalPodAutoscaler} {horizontalpodautoscalers/status true HorizontalPodAutoscaler} {ingresses true Ingress} {ingresses/status true Ingress} {jobs true Job} {jobs/status true Job} {networkpolicies true NetworkPolicy} {replicasets true ReplicaSet} {replicasets/scale true Scale} {replicasets/status true ReplicaSet} {replicationcontrollers true ReplicationControllerDummy} {replicationcontrollers/scale true Scale} {thirdpartyresources false ThirdPartyResource}],} policy/v1beta1:&APIResourceList{GroupVersion:policy/v1beta1,APIResources:[{poddisruptionbudgets true PodDisruptionBudget} {poddisruptionbudgets/status true PodDisruptionBudget}],}]
I1214 18:38:04.416383 5417 controllermanager.go:460] Starting policy/v1beta1 apis
I1214 18:38:04.416449 5417 controllermanager.go:462] Starting disruption controller
I1214 18:38:04.417308 5417 controllermanager.go:470] Attempting to start statefulset, full resource map map[autoscaling/v1:&APIResourceList{GroupVersion:autoscaling/v1,APIResources:[{horizontalpodautoscalers true HorizontalPodAutoscaler} {horizontalpodautoscalers/status true HorizontalPodAutoscaler}],} certificates.k8s.io/v1alpha1:&APIResourceList{GroupVersion:certificates.k8s.io/v1alpha1,APIResources:[{certificatesigningrequests false CertificateSigningRequest} {certificatesigningrequests/approval false CertificateSigningRequest} {certificatesigningrequests/status false CertificateSigningRequest}],} storage.k8s.io/v1beta1:&APIResourceList{GroupVersion:storage.k8s.io/v1beta1,APIResources:[{storageclasses false StorageClass}],} v1:&APIResourceList{GroupVersion:v1,APIResources:[{bindings true Binding} {componentstatuses false ComponentStatus} {configmaps true ConfigMap} {endpoints true Endpoints} {events true Event} {limitranges true LimitRange} {namespaces false Namespace} {namespaces/finalize false Namespace} {namespaces/status false Namespace} {nodes false Node} {nodes/proxy false Node} {nodes/status false Node} {persistentvolumeclaims true PersistentVolumeClaim} {persistentvolumeclaims/status true PersistentVolumeClaim} {persistentvolumes false PersistentVolume} {persistentvolumes/status false PersistentVolume} {pods true Pod} {pods/attach true Pod} {pods/binding true Binding} {pods/eviction true Eviction} {pods/exec true Pod} {pods/log true Pod} {pods/portforward true Pod} {pods/proxy true Pod} {pods/status true Pod} {podtemplates true PodTemplate} {replicationcontrollers true ReplicationController} {replicationcontrollers/scale true Scale} {replicationcontrollers/status true ReplicationController} {resourcequotas true ResourceQuota} {resourcequotas/status true ResourceQuota} {secrets true Secret} {serviceaccounts true ServiceAccount} {services true Service} {services/proxy true Service} {services/status true Service}],} rbac.authorization.k8s.io/v1alpha1:&APIResourceList{GroupVersion:rbac.authorization.k8s.io/v1alpha1,APIResources:[{clusterrolebindings false ClusterRoleBinding} {clusterroles false ClusterRole} {rolebindings true RoleBinding} {roles true Role}],} apps/v1beta1:&APIResourceList{GroupVersion:apps/v1beta1,APIResources:[{statefulsets true StatefulSet} {statefulsets/status true StatefulSet}],} authentication.k8s.io/v1beta1:&APIResourceList{GroupVersion:authentication.k8s.io/v1beta1,APIResources:[{tokenreviews false TokenReview}],} authorization.k8s.io/v1beta1:&APIResourceList{GroupVersion:authorization.k8s.io/v1beta1,APIResources:[{localsubjectaccessreviews true LocalSubjectAccessReview} {selfsubjectaccessreviews false SelfSubjectAccessReview} {subjectaccessreviews false SubjectAccessReview}],} batch/v1:&APIResourceList{GroupVersion:batch/v1,APIResources:[{jobs true Job} {jobs/status true Job}],} extensions/v1beta1:&APIResourceList{GroupVersion:extensions/v1beta1,APIResources:[{daemonsets true DaemonSet} {daemonsets/status true DaemonSet} {deployments true Deployment} {deployments/rollback true DeploymentRollback} {deployments/scale true Scale} {deployments/status true Deployment} {horizontalpodautoscalers true HorizontalPodAutoscaler} {horizontalpodautoscalers/status true HorizontalPodAutoscaler} {ingresses true Ingress} {ingresses/status true Ingress} {jobs true Job} {jobs/status true Job} {networkpolicies true NetworkPolicy} {replicasets true ReplicaSet} {replicasets/scale true Scale} {replicasets/status true ReplicaSet} {replicationcontrollers true ReplicationControllerDummy} {replicationcontrollers/scale true Scale} {thirdpartyresources false ThirdPartyResource}],} policy/v1beta1:&APIResourceList{GroupVersion:policy/v1beta1,APIResources:[{poddisruptionbudgets true PodDisruptionBudget} {poddisruptionbudgets/status true PodDisruptionBudget}],}]
I1214 18:38:04.417574 5417 controllermanager.go:472] Starting apps/v1beta1 apis
I1214 18:38:04.417613 5417 controllermanager.go:474] Starting StatefulSet controller
I1214 18:38:04.418268 5417 controllermanager.go:499] Not starting batch/v2alpha1 apis
I1214 18:38:04.424693 5417 replica_set.go:162] Starting ReplicaSet controller
I1214 18:38:04.424762 5417 horizontal.go:132] Starting HPA Controller
I1214 18:38:04.425631 5417 disruption.go:274] Starting disruption controller
I1214 18:38:04.425669 5417 disruption.go:276] Sending events to api server.
I1214 18:38:04.425853 5417 pet_set.go:146] Starting statefulset controller
I1214 18:38:04.450377 5417 controllermanager.go:537] Attempting to start certificates, full resource map map[rbac.authorization.k8s.io/v1alpha1:&APIResourceList{GroupVersion:rbac.authorization.k8s.io/v1alpha1,APIResources:[{clusterrolebindings false ClusterRoleBinding} {clusterroles false ClusterRole} {rolebindings true RoleBinding} {roles true Role}],} apps/v1beta1:&APIResourceList{GroupVersion:apps/v1beta1,APIResources:[{statefulsets true StatefulSet} {statefulsets/status true StatefulSet}],} authentication.k8s.io/v1beta1:&APIResourceList{GroupVersion:authentication.k8s.io/v1beta1,APIResources:[{tokenreviews false TokenReview}],} authorization.k8s.io/v1beta1:&APIResourceList{GroupVersion:authorization.k8s.io/v1beta1,APIResources:[{localsubjectaccessreviews true LocalSubjectAccessReview} {selfsubjectaccessreviews false SelfSubjectAccessReview} {subjectaccessreviews false SubjectAccessReview}],} batch/v1:&APIResourceList{GroupVersion:batch/v1,APIResources:[{jobs true Job} {jobs/status true Job}],} extensions/v1beta1:&APIResourceList{GroupVersion:extensions/v1beta1,APIResources:[{daemonsets true DaemonSet} {daemonsets/status true DaemonSet} {deployments true Deployment} {deployments/rollback true DeploymentRollback} {deployments/scale true Scale} {deployments/status true Deployment} {horizontalpodautoscalers true HorizontalPodAutoscaler} {horizontalpodautoscalers/status true HorizontalPodAutoscaler} {ingresses true Ingress} {ingresses/status true Ingress} {jobs true Job} {jobs/status true Job} {networkpolicies true NetworkPolicy} {replicasets true ReplicaSet} {replicasets/scale true Scale} {replicasets/status true ReplicaSet} {replicationcontrollers true ReplicationControllerDummy} {replicationcontrollers/scale true Scale} {thirdpartyresources false ThirdPartyResource}],} policy/v1beta1:&APIResourceList{GroupVersion:policy/v1beta1,APIResources:[{poddisruptionbudgets true PodDisruptionBudget} {poddisruptionbudgets/status true PodDisruptionBudget}],} autoscaling/v1:&APIResourceList{GroupVersion:autoscaling/v1,APIResources:[{horizontalpodautoscalers true HorizontalPodAutoscaler} {horizontalpodautoscalers/status true HorizontalPodAutoscaler}],} certificates.k8s.io/v1alpha1:&APIResourceList{GroupVersion:certificates.k8s.io/v1alpha1,APIResources:[{certificatesigningrequests false CertificateSigningRequest} {certificatesigningrequests/approval false CertificateSigningRequest} {certificatesigningrequests/status false CertificateSigningRequest}],} storage.k8s.io/v1beta1:&APIResourceList{GroupVersion:storage.k8s.io/v1beta1,APIResources:[{storageclasses false StorageClass}],} v1:&APIResourceList{GroupVersion:v1,APIResources:[{bindings true Binding} {componentstatuses false ComponentStatus} {configmaps true ConfigMap} {endpoints true Endpoints} {events true Event} {limitranges true LimitRange} {namespaces false Namespace} {namespaces/finalize false Namespace} {namespaces/status false Namespace} {nodes false Node} {nodes/proxy false Node} {nodes/status false Node} {persistentvolumeclaims true PersistentVolumeClaim} {persistentvolumeclaims/status true PersistentVolumeClaim} {persistentvolumes false PersistentVolume} {persistentvolumes/status false PersistentVolume} {pods true Pod} {pods/attach true Pod} {pods/binding true Binding} {pods/eviction true Eviction} {pods/exec true Pod} {pods/log true Pod} {pods/portforward true Pod} {pods/proxy true Pod} {pods/status true Pod} {podtemplates true PodTemplate} {replicationcontrollers true ReplicationController} {replicationcontrollers/scale true Scale} {replicationcontrollers/status true ReplicationController} {resourcequotas true ResourceQuota} {resourcequotas/status true ResourceQuota} {secrets true Secret} {serviceaccounts true ServiceAccount} {services true Service} {services/proxy true Service} {services/status true Service}],}]
I1214 18:38:04.450715 5417 controllermanager.go:539] Starting certificates.k8s.io/v1alpha1 apis
I1214 18:38:04.450757 5417 controllermanager.go:541] Starting certificate request controller
I1214 18:38:04.451345 5417 attach_detach_controller.go:203] Starting Attach Detach Controller
E1214 18:38:04.453425 5417 controllermanager.go:551] Failed to start certificate controller: open /etc/kubernetes/ca/ca.pem: no such file or directory
E1214 18:38:04.453921 5417 util.go:45] Metric for serviceaccount_controller already registered
I1214 18:38:04.454367 5417 serviceaccounts_controller.go:120] Starting ServiceAccount controller
I1214 18:38:04.474951 5417 garbagecollector.go:758] Garbage Collector: Initializing
I1214 18:38:04.517770 5417 replication_controller.go:653] Waiting for pods controller to sync, requeuing rc kube-system/kube-dns-v17.1
E1214 18:38:04.612603 5417 actual_state_of_world.go:462] Failed to set statusUpdateNeeded to needed true because nodeName="ip-10-9-30-229.us-gov-west-1.compute.internal" does not exist
E1214 18:38:04.612648 5417 actual_state_of_world.go:462] Failed to set statusUpdateNeeded to needed true because nodeName="ip-10-9-30-231.us-gov-west-1.compute.internal" does not exist
E1214 18:38:04.612669 5417 actual_state_of_world.go:462] Failed to set statusUpdateNeeded to needed true because nodeName="ip-10-9-30-232.us-gov-west-1.compute.internal" does not exist
E1214 18:38:04.612726 5417 actual_state_of_world.go:462] Failed to set statusUpdateNeeded to needed true because nodeName="ip-10-9-30-233.us-gov-west-1.compute.internal" does not exist
I1214 18:38:04.661615 5417 replication_controller.go:653] Waiting for pods controller to sync, requeuing rc kube-system/kube-dns-v17.1
I1214 18:38:04.706881 5417 nodecontroller.go:429] Initializing eviction metric for zone: us-gov-west-1::us-gov-west-1a
W1214 18:38:04.706960 5417 nodecontroller.go:678] Missing timestamp for Node ip-10-9-30-229.us-gov-west-1.compute.internal. Assuming now as a timestamp.
W1214 18:38:04.707025 5417 nodecontroller.go:678] Missing timestamp for Node ip-10-9-30-231.us-gov-west-1.compute.internal. Assuming now as a timestamp.
W1214 18:38:04.707074 5417 nodecontroller.go:678] Missing timestamp for Node ip-10-9-30-232.us-gov-west-1.compute.internal. Assuming now as a timestamp.
W1214 18:38:04.707120 5417 nodecontroller.go:678] Missing timestamp for Node ip-10-9-30-233.us-gov-west-1.compute.internal. Assuming now as a timestamp.
I1214 18:38:04.707172 5417 nodecontroller.go:608] NodeController detected that zone us-gov-west-1::us-gov-west-1a is now in state Normal.
I1214 18:38:04.707409 5417 event.go:217] Event(api.ObjectReference{Kind:"Node", Namespace:"", Name:"ip-10-9-30-233.us-gov-west-1.compute.internal", UID:"6e4186ee-c1b7-11e6-921a-02201d3e0adb", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node ip-10-9-30-233.us-gov-west-1.compute.internal event: Registered Node ip-10-9-30-233.us-gov-west-1.compute.internal in NodeController
I1214 18:38:04.707446 5417 event.go:217] Event(api.ObjectReference{Kind:"Node", Namespace:"", Name:"ip-10-9-30-229.us-gov-west-1.compute.internal", UID:"86873eb0-c22b-11e6-b611-0219c1053c6d", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node ip-10-9-30-229.us-gov-west-1.compute.internal event: Registered Node ip-10-9-30-229.us-gov-west-1.compute.internal in NodeController
I1214 18:38:04.707463 5417 event.go:217] Event(api.ObjectReference{Kind:"Node", Namespace:"", Name:"ip-10-9-30-231.us-gov-west-1.compute.internal", UID:"fe9f85a8-c1b4-11e6-921a-02201d3e0adb", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node ip-10-9-30-231.us-gov-west-1.compute.internal event: Registered Node ip-10-9-30-231.us-gov-west-1.compute.internal in NodeController
I1214 18:38:04.707478 5417 event.go:217] Event(api.ObjectReference{Kind:"Node", Namespace:"", Name:"ip-10-9-30-232.us-gov-west-1.compute.internal", UID:"f49178f0-c1b5-11e6-921a-02201d3e0adb", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node ip-10-9-30-232.us-gov-west-1.compute.internal event: Registered Node ip-10-9-30-232.us-gov-west-1.compute.internal in NodeController
I1214 18:38:14.475207 5417 garbagecollector.go:772] Garbage Collector: All monitored resources synced. Proceeding to collect garbage
I1214 18:39:44.364274 5417 servicecontroller.go:642] Detected change in list of current cluster nodes. New node set: [ip-10-9-30-229.us-gov-west-1.compute.internal ip-10-9-30-231.us-gov-west-1.compute.internal ip-10-9-30-232.us-gov-west-1.compute.internal ip-10-9-30-233.us-gov-west-1.compute.internal]
I1214 18:39:44.364328 5417 servicecontroller.go:650] Successfully updated 22 out of 22 load balancers to direct traffic to the updated set of nodes
E1214 18:42:29.475908 5417 actual_state_of_world.go:462] Failed to set statusUpdateNeeded to needed true because nodeName="ip-10-9-30-230.us-gov-west-1.compute.internal" does not exist
I1214 18:42:29.495643 5417 event.go:217] Event(api.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"fluentd-cloudwatch-v1", UID:"98b1cc3b-c227-11e6-921a-02201d3e0adb", APIVersion:"extensions", ResourceVersion:"5820351", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: fluentd-cloudwatch-v1-jd8l4
I1214 18:42:29.508856 5417 event.go:217] Event(api.ObjectReference{Kind:"DaemonSet", Namespace:"default", Name:"kube2iam", UID:"97592ba6-c227-11e6-921a-02201d3e0adb", APIVersion:"extensions", ResourceVersion:"5820356", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube2iam-xbpjt
E1214 18:42:29.541483 5417 daemoncontroller.go:225] default/kube2iam failed with : error storing status for daemon set &extensions.DaemonSet{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"kube2iam", GenerateName:"", Namespace:"default", SelfLink:"/apis/extensions/v1beta1/namespaces/default/daemonsets/kube2iam", UID:"97592ba6-c227-11e6-921a-02201d3e0adb", ResourceVersion:"5820356", Generation:1, CreationTimestamp:unversioned.Time{Time:time.Time{sec:63617335398, nsec:0, loc:(*time.Location)(0x3923440)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"kube2iam"}, Annotations:map[string]string{"kubectl.kubernetes.io/last-applied-configuration":"{\"kind\":\"DaemonSet\",\"apiVersion\":\"extensions/v1beta1\",\"metadata\":{\"name\":\"kube2iam\",\"creationTimestamp\":null,\"labels\":{\"app\":\"kube2iam\"}},\"spec\":{\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"name\":\"kube2iam\"}},\"spec\":{\"containers\":[{\"name\":\"kube2iam\",\"image\":\"jtblin/kube2iam:latest\",\"args\":[\"--base-role-arn=$(BASE_ROLE_ARN)\",\"--default-role=$(DEFAULT_ROLE)\",\"--host-ip=$(HOST_IP)\",\"--iptables=true\"],\"ports\":[{\"name\":\"http\",\"hostPort\":8181,\"containerPort\":8181}],\"env\":[{\"name\":\"AWS_REGION\",\"value\":\"us-gov-west-1\"},{\"name\":\"BASE_ROLE_ARN\",\"value\":\"arn:aws-us-gov:iam::135676904304:role/\"},{\"name\":\"DEFAULT_ROLE\",\"value\":\"k8s-node\"},{\"name\":\"HOST_IP\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}}],\"resources\":{},\"securityContext\":{\"privileged\":true}}],\"hostNetwork\":true}}},\"status\":{\"currentNumberScheduled\":0,\"numberMisscheduled\":0,\"desiredNumberScheduled\":0,\"numberReady\":0}}"}, OwnerReferences:[]api.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:extensions.DaemonSetSpec{Selector:(*unversioned.LabelSelector)(0xc420d921e0), Template:api.PodTemplateSpec{ObjectMeta:api.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"name":"kube2iam"}, Annotations:map[string]string(nil), OwnerReferences:[]api.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:api.PodSpec{Volumes:[]api.Volume(nil), InitContainers:[]api.Container(nil), Containers:[]api.Container{api.Container{Name:"kube2iam", Image:"jtblin/kube2iam:latest", Command:[]string(nil), Args:[]string{"--base-role-arn=$(BASE_ROLE_ARN)", "--default-role=$(DEFAULT_ROLE)", "--host-ip=$(HOST_IP)", "--iptables=true"}, WorkingDir:"", Ports:[]api.ContainerPort{api.ContainerPort{Name:"http", HostPort:8181, ContainerPort:8181, Protocol:"TCP", HostIP:""}}, Env:[]api.EnvVar{api.EnvVar{Name:"AWS_REGION", Value:"us-gov-west-1", ValueFrom:(*api.EnvVarSource)(nil)}, api.EnvVar{Name:"BASE_ROLE_ARN", Value:"arn:aws-us-gov:iam::135676904304:role/", ValueFrom:(*api.EnvVarSource)(nil)}, api.EnvVar{Name:"DEFAULT_ROLE", Value:"k8s-node", ValueFrom:(*api.EnvVarSource)(nil)}, api.EnvVar{Name:"HOST_IP", Value:"", ValueFrom:(*api.EnvVarSource)(0xc420d922a0)}}, Resources:api.ResourceRequirements{Limits:api.ResourceList(nil), Requests:api.ResourceList(nil)}, VolumeMounts:[]api.VolumeMount(nil), LivenessProbe:(*api.Probe)(nil), ReadinessProbe:(*api.Probe)(nil), Lifecycle:(*api.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", ImagePullPolicy:"Always", SecurityContext:(*api.SecurityContext)(0xc421168840), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc420d90290), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"", NodeName:"", SecurityContext:(*api.PodSecurityContext)(0xc420d94380), ImagePullSecrets:[]api.LocalObjectReference(nil), Hostname:"", Subdomain:""}}}, Status:extensions.DaemonSetStatus{CurrentNumberScheduled:5, NumberMisscheduled:0, DesiredNumberScheduled:5, NumberReady:4}}: Operation cannot be fulfilled on daemonsets.extensions "kube2iam": the object has been modified; please apply your changes to the latest version and try again
W1214 18:42:29.851360 5417 nodecontroller.go:678] Missing timestamp for Node ip-10-9-30-230.us-gov-west-1.compute.internal. Assuming now as a timestamp.
I1214 18:42:29.851640 5417 event.go:217] Event(api.ObjectReference{Kind:"Node", Namespace:"", Name:"ip-10-9-30-230.us-gov-west-1.compute.internal", UID:"10ec70ef-c22d-11e6-b611-0219c1053c6d", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node ip-10-9-30-230.us-gov-west-1.compute.internal event: Registered Node ip-10-9-30-230.us-gov-west-1.compute.internal in NodeController
I1214 18:42:51.008730 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" to node "ip-10-9-30-231.us-gov-west-1.compute.internal"
I1214 18:42:51.008881 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" to node "ip-10-9-30-231.us-gov-west-1.compute.internal"
I1214 18:42:51.008945 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" to node "ip-10-9-30-231.us-gov-west-1.compute.internal"
I1214 18:42:51.009021 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" to node "ip-10-9-30-231.us-gov-west-1.compute.internal"
I1214 18:42:51.009098 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" to node "ip-10-9-30-231.us-gov-west-1.compute.internal"
I1214 18:42:51.012318 5417 event.go:217] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"x864fe17555694-189251105", UID:"8f2d4cf4-b38a-11e6-b781-02ba9586f3cd", APIVersion:"extensions", ResourceVersion:"5819369", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: x864fe17555694-189251105-lxw4k
I1214 18:42:51.012434 5417 event.go:217] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"xf2ee8dcfaed64-master-4003242301", UID:"752c69c0-ac43-11e6-b31f-02ac00f5d061", APIVersion:"extensions", ResourceVersion:"5819399", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: xf2ee8dcfaed64-master-4003242301-2t794
I1214 18:42:51.063136 5417 event.go:217] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"x5fc37d87a5e74-4085760040", UID:"509616e4-b777-11e6-a25b-02ab705d877f", APIVersion:"extensions", ResourceVersion:"5819347", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: x5fc37d87a5e74-4085760040-095xm
I1214 18:42:51.067823 5417 event.go:217] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"x1a7c7862478a4-3477520091", UID:"83c4126c-b70e-11e6-a25b-02ab705d877f", APIVersion:"extensions", ResourceVersion:"5819361", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: x1a7c7862478a4-3477520091-gp7hg
I1214 18:42:51.160436 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-0db40747620fe777c" to node "ip-10-9-30-231.us-gov-west-1.compute.internal"
I1214 18:42:51.164392 5417 event.go:217] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"xc5b7e5d80dd34-1944633621", UID:"177175a3-b71a-11e6-a25b-02ab705d877f", APIVersion:"extensions", ResourceVersion:"5819351", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: xc5b7e5d80dd34-1944633621-qw3l7
I1214 18:42:51.239809 5417 event.go:217] Event(api.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"xd5b5b021000b4-master-649573568", UID:"cbf641ab-c19b-11e6-8fed-0231a80237ed", APIVersion:"extensions", ResourceVersion:"5819358", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: xd5b5b021000b4-master-649573568-2fj4g
W1214 18:42:51.572991 5417 aws.go:1209] Got assignment call for already-assigned volume: bc@vol-0db40747620fe777c
W1214 18:42:51.577505 5417 aws.go:1209] Got assignment call for already-assigned volume: be@vol-18c1b983
W1214 18:42:51.602079 5417 aws.go:1209] Got assignment call for already-assigned volume: ba@vol-781962e3
W1214 18:42:51.604729 5417 aws.go:1209] Got assignment call for already-assigned volume: bf@vol-36ccb4ad
W1214 18:42:51.612597 5417 aws.go:1209] Got assignment call for already-assigned volume: bd@vol-15562f8e
W1214 18:42:51.614048 5417 aws.go:1209] Got assignment call for already-assigned volume: bb@vol-97aad60c
E1214 18:42:51.686666 5417 aws.go:1462] endAttaching called for disk "vol-0db40747620fe777c" when attach not in progress
I1214 18:42:51.686700 5417 operation_executor.go:620] AttachVolume.Attach succeeded for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-0db40747620fe777c" (spec.Name: "pvc-cc008d44-c19b-11e6-8f74-02015b8bdf19") from node "ip-10-9-30-231.us-gov-west-1.compute.internal".
E1214 18:42:51.691902 5417 aws.go:1462] endAttaching called for disk "vol-18c1b983" when attach not in progress
I1214 18:42:51.691931 5417 operation_executor.go:620] AttachVolume.Attach succeeded for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") from node "ip-10-9-30-231.us-gov-west-1.compute.internal".
I1214 18:42:51.709846 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-0db40747620fe777c" to node "ip-10-9-30-229.us-gov-west-1.compute.internal"
I1214 18:42:51.710023 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
E1214 18:42:51.732941 5417 aws.go:1462] endAttaching called for disk "vol-781962e3" when attach not in progress
I1214 18:42:51.733004 5417 operation_executor.go:620] AttachVolume.Attach succeeded for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") from node "ip-10-9-30-231.us-gov-west-1.compute.internal".
E1214 18:42:51.739932 5417 aws.go:1462] endAttaching called for disk "vol-36ccb4ad" when attach not in progress
I1214 18:42:51.739959 5417 operation_executor.go:620] AttachVolume.Attach succeeded for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") from node "ip-10-9-30-231.us-gov-west-1.compute.internal".
E1214 18:42:51.748250 5417 aws.go:1462] endAttaching called for disk "vol-97aad60c" when attach not in progress
I1214 18:42:51.748277 5417 operation_executor.go:620] AttachVolume.Attach succeeded for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") from node "ip-10-9-30-231.us-gov-west-1.compute.internal".
E1214 18:42:51.748998 5417 aws.go:1462] endAttaching called for disk "vol-15562f8e" when attach not in progress
I1214 18:42:51.749022 5417 operation_executor.go:620] AttachVolume.Attach succeeded for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") from node "ip-10-9-30-231.us-gov-west-1.compute.internal".
I1214 18:42:51.828908 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
I1214 18:42:51.828961 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
I1214 18:42:51.829050 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
I1214 18:42:51.829072 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
E1214 18:42:52.130740 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-18c1b983": Error attaching EBS volume "vol-18c1b983" to instance "i-0158364a043743c5f": VolumeInUse: vol-18c1b983 is already attached to an instance
status code: 400, request id:
E1214 18:42:52.130849 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983\"" failed. No retries permitted until 2016-12-14 18:42:52.630809493 +0000 UTC (durationBeforeRetry 500ms). Error: Failed to attach volume "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-18c1b983" to instance "i-0158364a043743c5f": VolumeInUse: vol-18c1b983 is already attached to an instance
status code: 400, request id:
I1214 18:42:52.131049 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"xc5b7e5d80dd34-1944633621-qw3l7", UID:"1dc5ab51-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821297", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-18c1b983" to instance "i-0158364a043743c5f": VolumeInUse: vol-18c1b983 is already attached to an instance
status code: 400, request id:
E1214 18:42:52.184837 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-0db40747620fe777c": Error attaching EBS volume "vol-0db40747620fe777c" to instance "i-094d19e4da8daf58e": VolumeInUse: vol-0db40747620fe777c is already attached to an instance
status code: 400, request id:
E1214 18:42:52.184995 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-0db40747620fe777c\"" failed. No retries permitted until 2016-12-14 18:42:52.684965205 +0000 UTC (durationBeforeRetry 500ms). Error: Failed to attach volume "pvc-cc008d44-c19b-11e6-8f74-02015b8bdf19" on node "ip-10-9-30-229.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-0db40747620fe777c" to instance "i-094d19e4da8daf58e": VolumeInUse: vol-0db40747620fe777c is already attached to an instance
status code: 400, request id:
I1214 18:42:52.185163 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"xd5b5b021000b4-master-649573568-2fj4g", UID:"1dde3cac-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821312", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-cc008d44-c19b-11e6-8f74-02015b8bdf19" on node "ip-10-9-30-229.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-0db40747620fe777c" to instance "i-094d19e4da8daf58e": VolumeInUse: vol-0db40747620fe777c is already attached to an instance
status code: 400, request id:
E1214 18:42:52.298899 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-781962e3": Error attaching EBS volume "vol-781962e3" to instance "i-0158364a043743c5f": VolumeInUse: vol-781962e3 is already attached to an instance
status code: 400, request id:
E1214 18:42:52.298986 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3\"" failed. No retries permitted until 2016-12-14 18:42:52.798959992 +0000 UTC (durationBeforeRetry 500ms). Error: Failed to attach volume "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-781962e3" to instance "i-0158364a043743c5f": VolumeInUse: vol-781962e3 is already attached to an instance
status code: 400, request id:
I1214 18:42:52.299183 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"x5fc37d87a5e74-4085760040-095xm", UID:"1dc5073b-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821294", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-781962e3" to instance "i-0158364a043743c5f": VolumeInUse: vol-781962e3 is already attached to an instance
status code: 400, request id:
E1214 18:42:52.333460 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-15562f8e": Error attaching EBS volume "vol-15562f8e" to instance "i-0158364a043743c5f": VolumeInUse: vol-15562f8e is already attached to an instance
status code: 400, request id:
E1214 18:42:52.333597 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e\"" failed. No retries permitted until 2016-12-14 18:42:52.833568191 +0000 UTC (durationBeforeRetry 500ms). Error: Failed to attach volume "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-15562f8e" to instance "i-0158364a043743c5f": VolumeInUse: vol-15562f8e is already attached to an instance
status code: 400, request id:
I1214 18:42:52.333744 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"x864fe17555694-189251105-lxw4k", UID:"1dc2012b-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821279", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-15562f8e" to instance "i-0158364a043743c5f": VolumeInUse: vol-15562f8e is already attached to an instance
status code: 400, request id:
E1214 18:42:52.337836 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-97aad60c": Error attaching EBS volume "vol-97aad60c" to instance "i-0158364a043743c5f": VolumeInUse: vol-97aad60c is already attached to an instance
status code: 400, request id:
E1214 18:42:52.337928 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c\"" failed. No retries permitted until 2016-12-14 18:42:52.837906815 +0000 UTC (durationBeforeRetry 500ms). Error: Failed to attach volume "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-97aad60c" to instance "i-0158364a043743c5f": VolumeInUse: vol-97aad60c is already attached to an instance
status code: 400, request id:
I1214 18:42:52.337991 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"xf2ee8dcfaed64-master-4003242301-2t794", UID:"1dc22f6d-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821278", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-97aad60c" to instance "i-0158364a043743c5f": VolumeInUse: vol-97aad60c is already attached to an instance
status code: 400, request id:
E1214 18:42:52.359098 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-36ccb4ad": Error attaching EBS volume "vol-36ccb4ad" to instance "i-0158364a043743c5f": VolumeInUse: vol-36ccb4ad is already attached to an instance
status code: 400, request id:
E1214 18:42:52.359185 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad\"" failed. No retries permitted until 2016-12-14 18:42:52.859158247 +0000 UTC (durationBeforeRetry 500ms). Error: Failed to attach volume "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-36ccb4ad" to instance "i-0158364a043743c5f": VolumeInUse: vol-36ccb4ad is already attached to an instance
status code: 400, request id:
I1214 18:42:52.359398 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"x1a7c7862478a4-3477520091-gp7hg", UID:"1dc4c710-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821290", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-36ccb4ad" to instance "i-0158364a043743c5f": VolumeInUse: vol-36ccb4ad is already attached to an instance
status code: 400, request id:
I1214 18:42:52.652190 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
I1214 18:42:52.752576 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-0db40747620fe777c" to node "ip-10-9-30-229.us-gov-west-1.compute.internal"
I1214 18:42:52.852968 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
I1214 18:42:52.853018 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
I1214 18:42:52.853051 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
I1214 18:42:52.954339 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
E1214 18:42:53.227474 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-18c1b983": Error attaching EBS volume "vol-18c1b983" to instance "i-0158364a043743c5f": VolumeInUse: vol-18c1b983 is already attached to an instance
status code: 400, request id:
E1214 18:42:53.227610 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983\"" failed. No retries permitted until 2016-12-14 18:42:54.227534007 +0000 UTC (durationBeforeRetry 1s). Error: Failed to attach volume "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-18c1b983" to instance "i-0158364a043743c5f": VolumeInUse: vol-18c1b983 is already attached to an instance
status code: 400, request id:
I1214 18:42:53.227642 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"xc5b7e5d80dd34-1944633621-qw3l7", UID:"1dc5ab51-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821297", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-18c1b983" to instance "i-0158364a043743c5f": VolumeInUse: vol-18c1b983 is already attached to an instance
status code: 400, request id:
E1214 18:42:53.233032 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-0db40747620fe777c": Error attaching EBS volume "vol-0db40747620fe777c" to instance "i-094d19e4da8daf58e": VolumeInUse: vol-0db40747620fe777c is already attached to an instance
status code: 400, request id:
E1214 18:42:53.233188 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-0db40747620fe777c\"" failed. No retries permitted until 2016-12-14 18:42:54.233113846 +0000 UTC (durationBeforeRetry 1s). Error: Failed to attach volume "pvc-cc008d44-c19b-11e6-8f74-02015b8bdf19" on node "ip-10-9-30-229.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-0db40747620fe777c" to instance "i-094d19e4da8daf58e": VolumeInUse: vol-0db40747620fe777c is already attached to an instance
status code: 400, request id:
I1214 18:42:53.233217 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"xd5b5b021000b4-master-649573568-2fj4g", UID:"1dde3cac-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821312", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-cc008d44-c19b-11e6-8f74-02015b8bdf19" on node "ip-10-9-30-229.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-0db40747620fe777c" to instance "i-094d19e4da8daf58e": VolumeInUse: vol-0db40747620fe777c is already attached to an instance
status code: 400, request id:
E1214 18:42:53.257867 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-781962e3": Error attaching EBS volume "vol-781962e3" to instance "i-0158364a043743c5f": VolumeInUse: vol-781962e3 is already attached to an instance
status code: 400, request id:
E1214 18:42:53.257943 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3\"" failed. No retries permitted until 2016-12-14 18:42:54.257919864 +0000 UTC (durationBeforeRetry 1s). Error: Failed to attach volume "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-781962e3" to instance "i-0158364a043743c5f": VolumeInUse: vol-781962e3 is already attached to an instance
status code: 400, request id:
I1214 18:42:53.257965 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"x5fc37d87a5e74-4085760040-095xm", UID:"1dc5073b-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821294", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-781962e3" to instance "i-0158364a043743c5f": VolumeInUse: vol-781962e3 is already attached to an instance
status code: 400, request id:
E1214 18:42:53.303133 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-15562f8e": Error attaching EBS volume "vol-15562f8e" to instance "i-0158364a043743c5f": VolumeInUse: vol-15562f8e is already attached to an instance
status code: 400, request id:
E1214 18:42:53.303221 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e\"" failed. No retries permitted until 2016-12-14 18:42:54.303193454 +0000 UTC (durationBeforeRetry 1s). Error: Failed to attach volume "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-15562f8e" to instance "i-0158364a043743c5f": VolumeInUse: vol-15562f8e is already attached to an instance
status code: 400, request id:
I1214 18:42:53.303384 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"x864fe17555694-189251105-lxw4k", UID:"1dc2012b-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821279", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-15562f8e" to instance "i-0158364a043743c5f": VolumeInUse: vol-15562f8e is already attached to an instance
status code: 400, request id:
E1214 18:42:53.334342 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-97aad60c": Error attaching EBS volume "vol-97aad60c" to instance "i-0158364a043743c5f": VolumeInUse: vol-97aad60c is already attached to an instance
status code: 400, request id:
E1214 18:42:53.334496 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c\"" failed. No retries permitted until 2016-12-14 18:42:54.334471303 +0000 UTC (durationBeforeRetry 1s). Error: Failed to attach volume "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-97aad60c" to instance "i-0158364a043743c5f": VolumeInUse: vol-97aad60c is already attached to an instance
status code: 400, request id:
I1214 18:42:53.334576 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"xf2ee8dcfaed64-master-4003242301-2t794", UID:"1dc22f6d-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821278", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-97aad60c" to instance "i-0158364a043743c5f": VolumeInUse: vol-97aad60c is already attached to an instance
status code: 400, request id:
E1214 18:42:53.410215 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-36ccb4ad": Error attaching EBS volume "vol-36ccb4ad" to instance "i-0158364a043743c5f": VolumeInUse: vol-36ccb4ad is already attached to an instance
status code: 400, request id:
E1214 18:42:53.410331 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad\"" failed. No retries permitted until 2016-12-14 18:42:54.410301625 +0000 UTC (durationBeforeRetry 1s). Error: Failed to attach volume "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-36ccb4ad" to instance "i-0158364a043743c5f": VolumeInUse: vol-36ccb4ad is already attached to an instance
status code: 400, request id:
I1214 18:42:53.410466 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"x1a7c7862478a4-3477520091-gp7hg", UID:"1dc4c710-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821290", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-36ccb4ad" to instance "i-0158364a043743c5f": VolumeInUse: vol-36ccb4ad is already attached to an instance
status code: 400, request id:
I1214 18:42:54.259501 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
I1214 18:42:54.259602 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
I1214 18:42:54.259634 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-0db40747620fe777c" to node "ip-10-9-30-229.us-gov-west-1.compute.internal"
I1214 18:42:54.359913 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
I1214 18:42:54.359955 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
I1214 18:42:54.460292 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
E1214 18:42:54.729635 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-781962e3": Error attaching EBS volume "vol-781962e3" to instance "i-0158364a043743c5f": VolumeInUse: vol-781962e3 is already attached to an instance
status code: 400, request id:
E1214 18:42:54.729803 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3\"" failed. No retries permitted until 2016-12-14 18:42:56.729703664 +0000 UTC (durationBeforeRetry 2s). Error: Failed to attach volume "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-781962e3" to instance "i-0158364a043743c5f": VolumeInUse: vol-781962e3 is already attached to an instance
status code: 400, request id:
E1214 18:42:54.729635 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-0db40747620fe777c": Error attaching EBS volume "vol-0db40747620fe777c" to instance "i-094d19e4da8daf58e": VolumeInUse: vol-0db40747620fe777c is already attached to an instance
status code: 400, request id:
E1214 18:42:54.730151 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-0db40747620fe777c\"" failed. No retries permitted until 2016-12-14 18:42:56.73009575 +0000 UTC (durationBeforeRetry 2s). Error: Failed to attach volume "pvc-cc008d44-c19b-11e6-8f74-02015b8bdf19" on node "ip-10-9-30-229.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-0db40747620fe777c" to instance "i-094d19e4da8daf58e": VolumeInUse: vol-0db40747620fe777c is already attached to an instance
status code: 400, request id:
I1214 18:42:54.730396 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"x5fc37d87a5e74-4085760040-095xm", UID:"1dc5073b-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821294", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-781962e3" to instance "i-0158364a043743c5f": VolumeInUse: vol-781962e3 is already attached to an instance
status code: 400, request id:
I1214 18:42:54.730475 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"xd5b5b021000b4-master-649573568-2fj4g", UID:"1dde3cac-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821312", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-cc008d44-c19b-11e6-8f74-02015b8bdf19" on node "ip-10-9-30-229.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-0db40747620fe777c" to instance "i-094d19e4da8daf58e": VolumeInUse: vol-0db40747620fe777c is already attached to an instance
status code: 400, request id:
E1214 18:42:54.750000 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-18c1b983": Error attaching EBS volume "vol-18c1b983" to instance "i-0158364a043743c5f": VolumeInUse: vol-18c1b983 is already attached to an instance
status code: 400, request id:
E1214 18:42:54.750120 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983\"" failed. No retries permitted until 2016-12-14 18:42:56.750094803 +0000 UTC (durationBeforeRetry 2s). Error: Failed to attach volume "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-18c1b983" to instance "i-0158364a043743c5f": VolumeInUse: vol-18c1b983 is already attached to an instance
status code: 400, request id:
I1214 18:42:54.750147 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"xc5b7e5d80dd34-1944633621-qw3l7", UID:"1dc5ab51-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821297", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-18c1b983" to instance "i-0158364a043743c5f": VolumeInUse: vol-18c1b983 is already attached to an instance
status code: 400, request id:
E1214 18:42:54.783088 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-15562f8e": Error attaching EBS volume "vol-15562f8e" to instance "i-0158364a043743c5f": VolumeInUse: vol-15562f8e is already attached to an instance
status code: 400, request id:
E1214 18:42:54.783398 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e\"" failed. No retries permitted until 2016-12-14 18:42:56.783374303 +0000 UTC (durationBeforeRetry 2s). Error: Failed to attach volume "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-15562f8e" to instance "i-0158364a043743c5f": VolumeInUse: vol-15562f8e is already attached to an instance
status code: 400, request id:
I1214 18:42:54.783642 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"x864fe17555694-189251105-lxw4k", UID:"1dc2012b-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821279", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-15562f8e" to instance "i-0158364a043743c5f": VolumeInUse: vol-15562f8e is already attached to an instance
status code: 400, request id:
E1214 18:42:54.812932 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-97aad60c": Error attaching EBS volume "vol-97aad60c" to instance "i-0158364a043743c5f": VolumeInUse: vol-97aad60c is already attached to an instance
status code: 400, request id:
E1214 18:42:54.813109 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c\"" failed. No retries permitted until 2016-12-14 18:42:56.813016266 +0000 UTC (durationBeforeRetry 2s). Error: Failed to attach volume "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-97aad60c" to instance "i-0158364a043743c5f": VolumeInUse: vol-97aad60c is already attached to an instance
status code: 400, request id:
I1214 18:42:54.813371 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"xf2ee8dcfaed64-master-4003242301-2t794", UID:"1dc22f6d-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821278", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-97aad60c" to instance "i-0158364a043743c5f": VolumeInUse: vol-97aad60c is already attached to an instance
status code: 400, request id:
E1214 18:42:54.933205 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-36ccb4ad": Error attaching EBS volume "vol-36ccb4ad" to instance "i-0158364a043743c5f": VolumeInUse: vol-36ccb4ad is already attached to an instance
status code: 400, request id:
E1214 18:42:54.933490 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad\"" failed. No retries permitted until 2016-12-14 18:42:56.933461803 +0000 UTC (durationBeforeRetry 2s). Error: Failed to attach volume "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-36ccb4ad" to instance "i-0158364a043743c5f": VolumeInUse: vol-36ccb4ad is already attached to an instance
status code: 400, request id:
I1214 18:42:54.933636 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"x1a7c7862478a4-3477520091-gp7hg", UID:"1dc4c710-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821290", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-36ccb4ad" to instance "i-0158364a043743c5f": VolumeInUse: vol-36ccb4ad is already attached to an instance
status code: 400, request id:
I1214 18:42:56.766258 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
I1214 18:42:56.766300 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
I1214 18:42:56.766396 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-0db40747620fe777c" to node "ip-10-9-30-229.us-gov-west-1.compute.internal"
I1214 18:42:56.866695 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
I1214 18:42:56.866769 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
I1214 18:42:56.967005 5417 reconciler.go:202] Started AttachVolume for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" to node "ip-10-9-30-230.us-gov-west-1.compute.internal"
E1214 18:42:57.245915 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-18c1b983": Error attaching EBS volume "vol-18c1b983" to instance "i-0158364a043743c5f": VolumeInUse: vol-18c1b983 is already attached to an instance
status code: 400, request id:
E1214 18:42:57.246069 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983\"" failed. No retries permitted until 2016-12-14 18:43:01.246036141 +0000 UTC (durationBeforeRetry 4s). Error: Failed to attach volume "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-18c1b983" to instance "i-0158364a043743c5f": VolumeInUse: vol-18c1b983 is already attached to an instance
status code: 400, request id:
I1214 18:42:57.246122 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"xc5b7e5d80dd34-1944633621-qw3l7", UID:"1dc5ab51-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821297", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f" on node "ip-10-9-30-230.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-18c1b983" to instance "i-0158364a043743c5f": VolumeInUse: vol-18c1b983 is already attached to an instance
status code: 400, request id:
E1214 18:42:57.250298 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-0db40747620fe777c": Error attaching EBS volume "vol-0db40747620fe777c" to instance "i-094d19e4da8daf58e": VolumeInUse: vol-0db40747620fe777c is already attached to an instance
status code: 400, request id:
E1214 18:42:57.250417 5417 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-0db40747620fe777c\"" failed. No retries permitted until 2016-12-14 18:43:01.250393509 +0000 UTC (durationBeforeRetry 4s). Error: Failed to attach volume "pvc-cc008d44-c19b-11e6-8f74-02015b8bdf19" on node "ip-10-9-30-229.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-0db40747620fe777c" to instance "i-094d19e4da8daf58e": VolumeInUse: vol-0db40747620fe777c is already attached to an instance
status code: 400, request id:
I1214 18:42:57.250448 5417 event.go:217] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"xd5b5b021000b4-master-649573568-2fj4g", UID:"1dde3cac-c22d-11e6-ac5f-028961b0177d", APIVersion:"v1", ResourceVersion:"5821312", FieldPath:""}): type: 'Warning' reason: 'FailedMount' Failed to attach volume "pvc-cc008d44-c19b-11e6-8f74-02015b8bdf19" on node "ip-10-9-30-229.us-gov-west-1.compute.internal" with: Error attaching EBS volume "vol-0db40747620fe777c" to instance "i-094d19e4da8daf58e": VolumeInUse: vol-0db40747620fe777c is already attached to an instance
status code: 400, request id:
E1214 18:42:57.266960 5417 attacher.go:73] Error attaching volume "aws://us-gov-west-1a/vol-781962e3": Error attaching EBS volume "vol-781962e3" to instance "i-0158364a043743c5f": VolumeInUse: vol-781962e3 is already attached to an instance
status code: 400, request id:
Raw
info
setup: five-node k8s cluster (3 masters, 2 workers, all nodes schedulable) managed by bosh using https://github.com/18F/kubernetes-release
steps: restarts masters one by one
results: several pods with pvcs are stuck in `ContainerCreating` forever, with ebs volumes attached to the wrong host
Raw
kubectl describe pod
master/86d23dea-1b83-4a3a-b335-862cb498dd2e:~# kubectl describe pod x5fc37d87a5e74-915003763-89gbb
Name: x5fc37d87a5e74-915003763-89gbb
Namespace: default
Node: ip-10-9-30-233.us-gov-west-1.compute.internal/10.9.30.233
Start Time: Wed, 14 Dec 2016 18:26:58 +0000
Labels: idx_and_short_serviceid=x5fc37d87a5e74
managed_by=TAP
pod-template-hash=915003763
service_id=5fc37d87-a5e7-42fd-840f-f249cf806973
Status: Pending
IP:
Controllers: ReplicaSet/x5fc37d87a5e74-915003763
Containers:
k-redis28:
Container ID:
Image: frodenas/redis
Image ID:
Port: 6379/TCP
Limits:
memory: 512M
Requests:
memory: 100M
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Volume Mounts:
/data from redis-persistent-storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-d1n4j (ro)
Environment Variables:
MANAGED_BY: TAP
REDIS_PASSWORD: <set to the key 'redis-password' in secret 'x5fc37d87a5e74-redis-credentials'>
TAP_K8S: true
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
redis-persistent-storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: x5fc37d87a5e74
ReadOnly: false
default-token-d1n4j:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-d1n4j
QoS Class: Burstable
Tolerations: <none>
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1h 1m 31 {kubelet ip-10-9-30-233.us-gov-west-1.compute.internal}Warning FailedMount Unable to mount volumes for pod "x5fc37d87a5e74-915003763-89gbb_default(3e9eb27d-c22a-11e6-8fb5-0293618a388d)": timeout expired waiting for volumes to attach/mount for pod "x5fc37d87a5e74-915003763-89gbb"/"default". list of unattached/unmounted volumes=[redis-persistent-storage]
1h 1m 31 {kubelet ip-10-9-30-233.us-gov-west-1.compute.internal}Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "x5fc37d87a5e74-915003763-89gbb"/"default". list of unattached/unmounted volumes=[redis-persistent-storage]
Raw
kubectl get pods
master/86d23dea-1b83-4a3a-b335-862cb498dd2e:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
kube2iam-7lbl1 1/1 Running 0 46m
kube2iam-9fkw1 1/1 Running 0 1h
kube2iam-vflwd 1/1 Running 0 1h
kube2iam-xbpjt 1/1 Running 0 53m
kube2iam-z8hj9 1/1 Running 0 1h
x1267f4b2ef244-1252513940-qwmjf 1/1 Running 0 1h
x1a0b21da21594-2244090092-fxfx4 1/1 Running 0 15h
x1a0b21da21594-3934597687-6zm1t 0/1 ContainerCreating 0 1h
x1a7c7862478a4-3477520091-gp7hg 1/1 Running 0 53m
x1a7c7862478a4-3601586214-c4jjb 0/1 ContainerCreating 0 1h
x3d5cef62b25f4-3573399908-g58bx 1/1 Running 0 1h
x511b293cb9f74-881603435-b00gr 1/1 Running 0 15h
x5fc37d87a5e74-4085760040-095xm 1/1 Running 0 53m
x5fc37d87a5e74-915003763-89gbb 0/1 ContainerCreating 0 1h
x78e0caa05f934-3261668110-vw1h6 1/1 Running 0 15h
x7c06d2d7cab54-729095091-3hk79 0/1 ContainerCreating 0 1h
x7e07fde5f4e04-1435179930-wdbxr 1/1 Running 0 15h
x864fe17555694-189251105-lxw4k 1/1 Running 0 53m
x864fe17555694-3958495084-jhd37 0/1 ContainerCreating 0 1h
x9dbf08f3084a4-3781244526-clwz8 1/1 Running 0 15h
x9fd6ed6913fb4-master-3410075224-f4r89 0/1 ContainerCreating 0 1h
xa23f1b6a96af4-3177585766-ghnn3 1/1 Running 0 1h
xc5b7e5d80dd34-1944633621-qw3l7 1/1 Running 0 53m
xcfa4a916fa564-master-2182526851-7h63q 1/1 Running 0 1h
xd250f51bbd354-320353218-mlvk3 1/1 Running 0 15h
xd5b5b021000b4-master-649573568-2fj4g 1/1 Running 0 53m
xec9bed0feca04-master-1122869484-mm96x 1/1 Running 0 1h
xf2ee8dcfaed64-master-4003242301-2t794 1/1 Running 0 53m
Raw
kubelet
------------ STARTING kubelet_ctl at Wed Dec 14 18:42:27 UTC 2016 --------------
Flag --api-servers has been deprecated, Use --kubeconfig instead. Will be removed in a future version.
Flag --config has been deprecated, Use --pod-manifest-path instead. Will be removed in a future version.
I1214 18:42:27.911320 11428 feature_gate.go:181] feature gates: map[]
I1214 18:42:27.911470 11428 aws.go:746] Building AWS cloudprovider
I1214 18:42:27.911506 11428 aws.go:700] Zone not specified in configuration file; querying AWS metadata service
I1214 18:42:28.128138 11428 aws.go:832] AWS cloud filtering on tags: map[KubernetesCluster:kubernetes-staging]
I1214 18:42:28.128304 11428 server.go:595] Multiple api servers specified. Picking first one
W1214 18:42:28.128423 11428 server.go:605] Could not load kubeconfig file /var/lib/kubelet/kubeconfig: stat /var/lib/kubelet/kubeconfig: no such file or directory. Using default client config instead.
I1214 18:42:28.275579 11428 docker.go:356] Connecting to docker on unix:///var/vcap/sys/run/docker/docker.sock
I1214 18:42:28.275647 11428 docker.go:376] Start docker client with request timeout=2m0s
E1214 18:42:29.249644 11428 cni.go:163] error updating cni config: No networks found in /etc/cni/net.d
I1214 18:42:29.253367 11428 manager.go:143] cAdvisor running in container: "/"
W1214 18:42:29.253629 11428 manager.go:147] Unable to connect to Docker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
W1214 18:42:29.254039 11428 manager.go:151] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp 127.0.0.1:15441: getsockopt: connection refused
I1214 18:42:29.255060 11428 fs.go:117] Filesystem partitions: map[/dev/xvdf1:{mountpoint:/var/vcap/store major:202 minor:81 fsType:ext4 blockSize:0} /dev/xvda1:{mountpoint:/ major:202 minor:1 fsType:ext4 blockSize:0} /dev/xvdb2:{mountpoint:/var/vcap/data major:202 minor:18 fsType:ext4 blockSize:0}]
I1214 18:42:29.259064 11428 info.go:47] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"
I1214 18:42:29.259153 11428 manager.go:198] Machine: {NumCores:2 CpuFrequency:2500038 MemoryCapacity:7841607680 MachineID: SystemUUID:EC289008-DB82-AC80-E533-1C72AE835E68 BootID:14c37675-ab9f-409b-b7db-691b7781d7e3 Filesystems:[{Device:/dev/xvda1 Capacity:3103703040 Type:vfs Inodes:195840 HasInodes:true} {Device:/dev/xvdb2 Capacity:25959673856 Type:vfs Inodes:1618848 HasInodes:true} {Device:/dev/xvdf1 Capacity:67501568000 Type:vfs Inodes:4194304 HasInodes:true}] DiskMap:map[202:0:{Name:xvda Major:202 Minor:0 Size:3221225472 Scheduler:none} 202:16:{Name:xvdb Major:202 Minor:16 Size:34359738368 Scheduler:none} 202:80:{Name:xvdf Major:202 Minor:80 Size:68719476736 Scheduler:none}] NetworkDevices:[{Name:eth0 MacAddress:02:89:61:b0:17:7d Speed:0 Mtu:9001} {Name:flannel0 MacAddress: Speed:10 Mtu:8973}] Topology:[{Id:0 Memory:7841607680 Cores:[{Id:0 Threads:[0 1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:26214400 Type:Unified Level:3}]}] CloudProvider:AWS InstanceType:m3.large InstanceID:i-0158364a043743c5f}
I1214 18:42:29.260216 11428 manager.go:204] Version: {KernelVersion:4.4.0-53-generic ContainerOsVersion:Ubuntu 14.04.5 LTS DockerVersion:Unknown CadvisorVersion: CadvisorRevision:}
W1214 18:42:29.263376 11428 container_manager_linux.go:205] Running with swap on is not supported, please disable swap! This will be a fatal error by default starting in K8s v1.6! In the meantime, you can opt-in to making this a fatal error by enabling --experimental-fail-swap-on.
I1214 18:42:29.263603 11428 kubelet.go:242] Adding manifest file: /var/vcap/sys/run/kubelet/manifests
I1214 18:42:29.263641 11428 kubelet.go:252] Watching apiserver
W1214 18:42:29.268979 11428 kubelet_network.go:69] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
I1214 18:42:29.269053 11428 kubelet.go:477] Hairpin mode set to "hairpin-veth"
I1214 18:42:29.311989 11428 docker_manager.go:257] Setting dockerRoot to /var/vcap/store/docker/docker
I1214 18:42:29.312018 11428 docker_manager.go:260] Setting cgroupDriver to cgroupfs
I1214 18:42:29.313394 11428 server.go:770] Started kubelet v1.5.0
E1214 18:42:29.313743 11428 kubelet.go:1145] Image garbage collection failed: unable to find data for container /
I1214 18:42:29.313935 11428 server.go:123] Starting to listen on 10.9.30.230:10250
I1214 18:42:29.314336 11428 kubelet_node_status.go:204] Setting node annotation to enable volume controller attach/detach
I1214 18:42:29.314358 11428 kubelet_node_status.go:246] Adding node label from cloud provider: beta.kubernetes.io/instance-type=m3.large
I1214 18:42:29.314371 11428 kubelet_node_status.go:257] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=us-gov-west-1a
I1214 18:42:29.314381 11428 kubelet_node_status.go:261] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=us-gov-west-1
E1214 18:42:29.343511 11428 kubelet.go:1634] Failed to check if disk space is available for the runtime: failed to get fs info for "runtime": unable to find data for container /
E1214 18:42:29.343548 11428 kubelet.go:1642] Failed to check if disk space is available on the root partition: failed to get fs info for "root": unable to find data for container /
E1214 18:42:29.344154 11428 container_manager_linux.go:809] Error parsing docker version "Unknown": No Major.Minor.Patch elements found
I1214 18:42:29.344192 11428 fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
I1214 18:42:29.344230 11428 status_manager.go:129] Starting to sync pod status with apiserver
I1214 18:42:29.344250 11428 kubelet.go:1714] Starting kubelet main sync loop.
I1214 18:42:29.344286 11428 kubelet.go:1725] skipping pod synchronization - [container runtime is down]
I1214 18:42:29.344897 11428 volume_manager.go:242] Starting Kubelet Volume Manager
W1214 18:42:29.351286 11428 manager.go:242] Docker container factory registration failed: failed to validate Docker info: failed to detect Docker info: Cannot connect to the Docker daemon. Is the docker daemon running on this host?.
W1214 18:42:29.351329 11428 manager.go:247] Registration of the rkt container factory failed: unable to communicate with Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp 127.0.0.1:15441: getsockopt: connection refused
I1214 18:42:29.351342 11428 factory.go:54] Registering systemd factory
I1214 18:42:29.351674 11428 factory.go:86] Registering Raw factory
I1214 18:42:29.352008 11428 manager.go:1106] Started watching for new ooms in manager
I1214 18:42:29.352114 11428 oomparser.go:200] OOM parser using kernel log file: "/var/log/kern.log"
I1214 18:42:29.353398 11428 manager.go:288] Starting recovery of all containers
I1214 18:42:29.354357 11428 manager.go:293] Recovery completed
I1214 18:42:29.455191 11428 kubelet_node_status.go:204] Setting node annotation to enable volume controller attach/detach
I1214 18:42:29.455236 11428 kubelet_node_status.go:246] Adding node label from cloud provider: beta.kubernetes.io/instance-type=m3.large
I1214 18:42:29.455251 11428 kubelet_node_status.go:257] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=us-gov-west-1a
I1214 18:42:29.455262 11428 kubelet_node_status.go:261] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=us-gov-west-1
I1214 18:42:29.465956 11428 kubelet_node_status.go:74] Attempting to register node ip-10-9-30-230.us-gov-west-1.compute.internal
I1214 18:42:29.483212 11428 kubelet_node_status.go:77] Successfully registered node ip-10-9-30-230.us-gov-west-1.compute.internal
W1214 18:42:34.344847 11428 pod_container_deletor.go:77] Container "07dedadd201fe17cf07f78477fe1dd6e7224509e1dfb0e505a17e06b99c6707b" not found in pod's containers
W1214 18:42:34.344902 11428 pod_container_deletor.go:77] Container "57931d06aa6497117dc6972da42f3516dad979c52fe0145d93fbba6e13127843" not found in pod's containers
W1214 18:42:34.344930 11428 pod_container_deletor.go:77] Container "60510e641807ee5218c0bbdbcbbce12ada80842d81fbf12751171da3cd438537" not found in pod's containers
I1214 18:42:34.365495 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/host-path/10efd90d-c22d-11e6-ac5f-028961b0177d-varlogdockercontainers" (spec.Name: "varlogdockercontainers") pod "10efd90d-c22d-11e6-ac5f-028961b0177d" (UID: "10efd90d-c22d-11e6-ac5f-028961b0177d")
I1214 18:42:34.365550 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/host-path/10efd90d-c22d-11e6-ac5f-028961b0177d-varlibdockercontainers" (spec.Name: "varlibdockercontainers") pod "10efd90d-c22d-11e6-ac5f-028961b0177d" (UID: "10efd90d-c22d-11e6-ac5f-028961b0177d")
I1214 18:42:34.365585 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/10efd90d-c22d-11e6-ac5f-028961b0177d-default-token-piiyk" (spec.Name: "default-token-piiyk") pod "10efd90d-c22d-11e6-ac5f-028961b0177d" (UID: "10efd90d-c22d-11e6-ac5f-028961b0177d")
I1214 18:42:34.365629 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/10f03cce-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "10f03cce-c22d-11e6-ac5f-028961b0177d" (UID: "10f03cce-c22d-11e6-ac5f-028961b0177d")
I1214 18:42:34.365667 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/host-path/c1c18ee8fe5b5187d85718c9917e243d-secrets" (spec.Name: "secrets") pod "c1c18ee8fe5b5187d85718c9917e243d" (UID: "c1c18ee8fe5b5187d85718c9917e243d")
I1214 18:42:34.365705 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/host-path/10efd90d-c22d-11e6-ac5f-028961b0177d-varlog" (spec.Name: "varlog") pod "10efd90d-c22d-11e6-ac5f-028961b0177d" (UID: "10efd90d-c22d-11e6-ac5f-028961b0177d")
I1214 18:42:34.466313 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/host-path/c1c18ee8fe5b5187d85718c9917e243d-secrets" (spec.Name: "secrets") pod "c1c18ee8fe5b5187d85718c9917e243d" (UID: "c1c18ee8fe5b5187d85718c9917e243d").
I1214 18:42:34.466313 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/host-path/10efd90d-c22d-11e6-ac5f-028961b0177d-varlibdockercontainers" (spec.Name: "varlibdockercontainers") pod "10efd90d-c22d-11e6-ac5f-028961b0177d" (UID: "10efd90d-c22d-11e6-ac5f-028961b0177d").
I1214 18:42:34.466530 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/host-path/10efd90d-c22d-11e6-ac5f-028961b0177d-varlog" (spec.Name: "varlog") pod "10efd90d-c22d-11e6-ac5f-028961b0177d" (UID: "10efd90d-c22d-11e6-ac5f-028961b0177d").
I1214 18:42:34.466682 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/host-path/10efd90d-c22d-11e6-ac5f-028961b0177d-varlogdockercontainers" (spec.Name: "varlogdockercontainers") pod "10efd90d-c22d-11e6-ac5f-028961b0177d" (UID: "10efd90d-c22d-11e6-ac5f-028961b0177d").
I1214 18:42:34.476510 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/10f03cce-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "10f03cce-c22d-11e6-ac5f-028961b0177d" (UID: "10f03cce-c22d-11e6-ac5f-028961b0177d").
I1214 18:42:34.481732 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/10efd90d-c22d-11e6-ac5f-028961b0177d-default-token-piiyk" (spec.Name: "default-token-piiyk") pod "10efd90d-c22d-11e6-ac5f-028961b0177d" (UID: "10efd90d-c22d-11e6-ac5f-028961b0177d").
I1214 18:42:34.968334 11428 hairpin.go:61] Unable to find pair interface, setting up all interfaces: exec: "nsenter": executable file not found in $PATH
I1214 18:42:34.968636 11428 docker_manager.go:2236] Determined pod ip after infra change: "fluentd-cloudwatch-v1-jd8l4_kube-system(10efd90d-c22d-11e6-ac5f-028961b0177d)": "10.100.82.2"
I1214 18:42:35.675415 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/10efd90d-c22d-11e6-ac5f-028961b0177d-default-token-piiyk" (spec.Name: "default-token-piiyk") pod "10efd90d-c22d-11e6-ac5f-028961b0177d" (UID: "10efd90d-c22d-11e6-ac5f-028961b0177d").
I1214 18:42:36.678963 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/10f03cce-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "10f03cce-c22d-11e6-ac5f-028961b0177d" (UID: "10f03cce-c22d-11e6-ac5f-028961b0177d").
I1214 18:42:36.679070 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/10efd90d-c22d-11e6-ac5f-028961b0177d-default-token-piiyk" (spec.Name: "default-token-piiyk") pod "10efd90d-c22d-11e6-ac5f-028961b0177d" (UID: "10efd90d-c22d-11e6-ac5f-028961b0177d").
I1214 18:42:37.682809 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/10f03cce-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "10f03cce-c22d-11e6-ac5f-028961b0177d" (UID: "10f03cce-c22d-11e6-ac5f-028961b0177d").
W1214 18:42:49.413111 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
W1214 18:42:49.414705 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
I1214 18:42:51.210886 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/1dc22f6d-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d")
I1214 18:42:51.211050 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:51.211272 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c\"" failed. No retries permitted until 2016-12-14 18:42:51.711228949 +0000 UTC (durationBeforeRetry 500ms). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
I1214 18:42:51.614192 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:51.614375 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e\"" failed. No retries permitted until 2016-12-14 18:42:52.114340928 +0000 UTC (durationBeforeRetry 500ms). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
I1214 18:42:51.614433 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/1dc2012b-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d")
I1214 18:42:51.714808 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:51.716884 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c\"" failed. No retries permitted until 2016-12-14 18:42:52.716798198 +0000 UTC (durationBeforeRetry 1s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
I1214 18:42:51.732326 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/1dc22f6d-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d").
I1214 18:42:52.116438 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:52.116717 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e\"" failed. No retries permitted until 2016-12-14 18:42:53.116683983 +0000 UTC (durationBeforeRetry 1s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
I1214 18:42:52.330929 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/1dc2012b-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d").
I1214 18:42:52.618281 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d")
I1214 18:42:52.618329 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/1dc4c710-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:52.618539 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad\"" failed. No retries permitted until 2016-12-14 18:42:53.1185096 +0000 UTC (durationBeforeRetry 500ms). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
I1214 18:42:52.718935 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:52.719160 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c\"" failed. No retries permitted until 2016-12-14 18:42:54.719111094 +0000 UTC (durationBeforeRetry 2s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
I1214 18:42:52.931202 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/1dc4c710-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d").
I1214 18:42:53.120774 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:53.120926 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad\"" failed. No retries permitted until 2016-12-14 18:42:54.120891464 +0000 UTC (durationBeforeRetry 1s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
E1214 18:42:53.121010 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e\"" failed. No retries permitted until 2016-12-14 18:42:55.120985999 +0000 UTC (durationBeforeRetry 2s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
I1214 18:42:53.120833 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d")
I1214 18:42:53.221275 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d")
I1214 18:42:53.221326 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/1dc5073b-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:53.221506 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3\"" failed. No retries permitted until 2016-12-14 18:42:53.7214633 +0000 UTC (durationBeforeRetry 500ms). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
I1214 18:42:53.531465 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/1dc5073b-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d").
I1214 18:42:53.723035 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:53.723200 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3\"" failed. No retries permitted until 2016-12-14 18:42:54.723168772 +0000 UTC (durationBeforeRetry 1s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
I1214 18:42:53.823352 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d")
I1214 18:42:53.823405 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/secret/1dc5ab51-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:53.823542 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983\"" failed. No retries permitted until 2016-12-14 18:42:54.32352059 +0000 UTC (durationBeforeRetry 500ms). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
I1214 18:42:53.931553 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/1dc5ab51-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d").
I1214 18:42:54.124663 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:54.124855 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad\"" failed. No retries permitted until 2016-12-14 18:42:56.124820587 +0000 UTC (durationBeforeRetry 2s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
I1214 18:42:54.325161 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:54.325314 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983\"" failed. No retries permitted until 2016-12-14 18:42:55.325259207 +0000 UTC (durationBeforeRetry 1s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
E1214 18:42:54.726664 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c\"" failed. No retries permitted until 2016-12-14 18:42:58.726628296 +0000 UTC (durationBeforeRetry 4s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
I1214 18:42:54.726702 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d")
I1214 18:42:54.726747 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:54.726857 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3\"" failed. No retries permitted until 2016-12-14 18:42:56.726837173 +0000 UTC (durationBeforeRetry 2s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
I1214 18:42:55.128018 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:55.128197 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e\"" failed. No retries permitted until 2016-12-14 18:42:59.128176785 +0000 UTC (durationBeforeRetry 4s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
I1214 18:42:55.328702 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:55.328928 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983\"" failed. No retries permitted until 2016-12-14 18:42:57.328899269 +0000 UTC (durationBeforeRetry 2s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
I1214 18:42:56.131200 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:56.131451 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad\"" failed. No retries permitted until 2016-12-14 18:43:00.13133298 +0000 UTC (durationBeforeRetry 4s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
I1214 18:42:56.732964 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:56.733119 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3\"" failed. No retries permitted until 2016-12-14 18:43:00.733092858 +0000 UTC (durationBeforeRetry 4s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
I1214 18:42:57.334999 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:57.335168 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983\"" failed. No retries permitted until 2016-12-14 18:43:01.335136193 +0000 UTC (durationBeforeRetry 4s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
E1214 18:42:58.739265 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c\"" failed. No retries permitted until 2016-12-14 18:43:06.739235516 +0000 UTC (durationBeforeRetry 8s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
I1214 18:42:58.739184 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d")
I1214 18:42:59.140522 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d")
E1214 18:42:59.140666 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e\"" failed. No retries permitted until 2016-12-14 18:43:07.140638331 +0000 UTC (durationBeforeRetry 8s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d") has not yet been added to the list of VolumesInUse in the node's volume status.
W1214 18:42:59.446291 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
W1214 18:42:59.447110 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
W1214 18:42:59.447749 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
W1214 18:42:59.448519 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
W1214 18:42:59.449205 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
W1214 18:42:59.449879 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
W1214 18:42:59.450544 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
W1214 18:42:59.451896 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
I1214 18:43:00.143416 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d")
E1214 18:43:00.146182 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad\"" failed. No retries permitted until 2016-12-14 18:43:08.146162236 +0000 UTC (durationBeforeRetry 8s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:43:00.745401 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d")
E1214 18:43:00.748278 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3\"" failed. No retries permitted until 2016-12-14 18:43:08.748247784 +0000 UTC (durationBeforeRetry 8s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:43:01.347097 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d")
E1214 18:43:01.350784 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983\"" failed. No retries permitted until 2016-12-14 18:43:09.350763882 +0000 UTC (durationBeforeRetry 8s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
W1214 18:43:05.075450 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
W1214 18:43:05.077023 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
W1214 18:43:05.078320 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
W1214 18:43:05.079789 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
W1214 18:43:05.080523 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
W1214 18:43:05.081219 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
I1214 18:43:06.762678 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d")
E1214 18:43:06.764956 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c\"" failed. No retries permitted until 2016-12-14 18:43:22.764935134 +0000 UTC (durationBeforeRetry 16s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:43:07.163795 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d")
E1214 18:43:07.166151 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e\"" failed. No retries permitted until 2016-12-14 18:43:23.16613075 +0000 UTC (durationBeforeRetry 16s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:43:08.166545 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d")
E1214 18:43:08.168856 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad\"" failed. No retries permitted until 2016-12-14 18:43:24.168836263 +0000 UTC (durationBeforeRetry 16s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:43:08.768023 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d")
E1214 18:43:08.770440 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3\"" failed. No retries permitted until 2016-12-14 18:43:24.770418205 +0000 UTC (durationBeforeRetry 16s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:43:09.369553 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d")
E1214 18:43:09.371808 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983\"" failed. No retries permitted until 2016-12-14 18:43:25.37178467 +0000 UTC (durationBeforeRetry 16s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
W1214 18:43:09.471725 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
W1214 18:43:09.471766 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
W1214 18:43:09.471822 11428 conversion.go:110] Could not get instant cpu stats: different number of cpus
I1214 18:43:22.807185 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d")
E1214 18:43:22.809721 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c\"" failed. No retries permitted until 2016-12-14 18:43:54.809700398 +0000 UTC (durationBeforeRetry 32s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:43:23.208388 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d")
E1214 18:43:23.211260 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e\"" failed. No retries permitted until 2016-12-14 18:43:55.211240474 +0000 UTC (durationBeforeRetry 32s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:43:24.211426 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d")
E1214 18:43:24.214217 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad\"" failed. No retries permitted until 2016-12-14 18:43:56.214196596 +0000 UTC (durationBeforeRetry 32s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:43:24.813210 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d")
E1214 18:43:24.815882 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3\"" failed. No retries permitted until 2016-12-14 18:43:56.815851439 +0000 UTC (durationBeforeRetry 32s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:43:25.414883 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d")
E1214 18:43:25.417573 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983\"" failed. No retries permitted until 2016-12-14 18:43:57.417552523 +0000 UTC (durationBeforeRetry 32s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:43:50.395030 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/10f03cce-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "10f03cce-c22d-11e6-ac5f-028961b0177d" (UID: "10f03cce-c22d-11e6-ac5f-028961b0177d").
I1214 18:43:54.903743 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d")
E1214 18:43:54.906556 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c\"" failed. No retries permitted until 2016-12-14 18:44:58.906535857 +0000 UTC (durationBeforeRetry 1m4s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:43:55.305232 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d")
E1214 18:43:55.307991 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e\"" failed. No retries permitted until 2016-12-14 18:44:59.307961722 +0000 UTC (durationBeforeRetry 1m4s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:43:56.308616 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d")
E1214 18:43:56.311514 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad\"" failed. No retries permitted until 2016-12-14 18:45:00.311481727 +0000 UTC (durationBeforeRetry 1m4s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:43:56.910562 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d")
E1214 18:43:56.913658 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3\"" failed. No retries permitted until 2016-12-14 18:45:00.913630783 +0000 UTC (durationBeforeRetry 1m4s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:43:57.512524 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d")
E1214 18:43:57.515620 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983\"" failed. No retries permitted until 2016-12-14 18:45:01.515592618 +0000 UTC (durationBeforeRetry 1m4s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:44:03.437132 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/10efd90d-c22d-11e6-ac5f-028961b0177d-default-token-piiyk" (spec.Name: "default-token-piiyk") pod "10efd90d-c22d-11e6-ac5f-028961b0177d" (UID: "10efd90d-c22d-11e6-ac5f-028961b0177d").
E1214 18:44:51.023522 11428 kubelet.go:1522] Unable to mount volumes for pod "xf2ee8dcfaed64-master-4003242301-2t794_default(1dc22f6d-c22d-11e6-ac5f-028961b0177d)": timeout expired waiting for volumes to attach/mount for pod "xf2ee8dcfaed64-master-4003242301-2t794"/"default". list of unattached/unmounted volumes=[mongo-persistent-storage]; skipping pod
E1214 18:44:51.023741 11428 pod_workers.go:184] Error syncing pod 1dc22f6d-c22d-11e6-ac5f-028961b0177d, skipping: timeout expired waiting for volumes to attach/mount for pod "xf2ee8dcfaed64-master-4003242301-2t794"/"default". list of unattached/unmounted volumes=[mongo-persistent-storage]
E1214 18:44:51.029496 11428 kubelet.go:1522] Unable to mount volumes for pod "x864fe17555694-189251105-lxw4k_default(1dc2012b-c22d-11e6-ac5f-028961b0177d)": timeout expired waiting for volumes to attach/mount for pod "x864fe17555694-189251105-lxw4k"/"default". list of unattached/unmounted volumes=[redis-persistent-storage]; skipping pod
E1214 18:44:51.029525 11428 pod_workers.go:184] Error syncing pod 1dc2012b-c22d-11e6-ac5f-028961b0177d, skipping: timeout expired waiting for volumes to attach/mount for pod "x864fe17555694-189251105-lxw4k"/"default". list of unattached/unmounted volumes=[redis-persistent-storage]
E1214 18:44:51.058130 11428 kubelet.go:1522] Unable to mount volumes for pod "x1a7c7862478a4-3477520091-gp7hg_default(1dc4c710-c22d-11e6-ac5f-028961b0177d)": timeout expired waiting for volumes to attach/mount for pod "x1a7c7862478a4-3477520091-gp7hg"/"default". list of unattached/unmounted volumes=[redis-persistent-storage]; skipping pod
E1214 18:44:51.058165 11428 pod_workers.go:184] Error syncing pod 1dc4c710-c22d-11e6-ac5f-028961b0177d, skipping: timeout expired waiting for volumes to attach/mount for pod "x1a7c7862478a4-3477520091-gp7hg"/"default". list of unattached/unmounted volumes=[redis-persistent-storage]
E1214 18:44:51.058470 11428 kubelet.go:1522] Unable to mount volumes for pod "x5fc37d87a5e74-4085760040-095xm_default(1dc5073b-c22d-11e6-ac5f-028961b0177d)": timeout expired waiting for volumes to attach/mount for pod "x5fc37d87a5e74-4085760040-095xm"/"default". list of unattached/unmounted volumes=[redis-persistent-storage]; skipping pod
E1214 18:44:51.058503 11428 pod_workers.go:184] Error syncing pod 1dc5073b-c22d-11e6-ac5f-028961b0177d, skipping: timeout expired waiting for volumes to attach/mount for pod "x5fc37d87a5e74-4085760040-095xm"/"default". list of unattached/unmounted volumes=[redis-persistent-storage]
E1214 18:44:51.075030 11428 kubelet.go:1522] Unable to mount volumes for pod "xc5b7e5d80dd34-1944633621-qw3l7_default(1dc5ab51-c22d-11e6-ac5f-028961b0177d)": timeout expired waiting for volumes to attach/mount for pod "xc5b7e5d80dd34-1944633621-qw3l7"/"default". list of unattached/unmounted volumes=[redis-persistent-storage]; skipping pod
E1214 18:44:51.075058 11428 pod_workers.go:184] Error syncing pod 1dc5ab51-c22d-11e6-ac5f-028961b0177d, skipping: timeout expired waiting for volumes to attach/mount for pod "xc5b7e5d80dd34-1944633621-qw3l7"/"default". list of unattached/unmounted volumes=[redis-persistent-storage]
I1214 18:44:59.003856 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d")
E1214 18:44:59.006157 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c\"" failed. No retries permitted until 2016-12-14 18:46:59.006136713 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:44:59.405021 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d")
E1214 18:44:59.407782 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e\"" failed. No retries permitted until 2016-12-14 18:46:59.407762115 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:45:00.407860 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d")
E1214 18:45:00.410666 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad\"" failed. No retries permitted until 2016-12-14 18:47:00.410644725 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:45:01.009950 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d")
E1214 18:45:01.012404 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3\"" failed. No retries permitted until 2016-12-14 18:47:01.012378563 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:45:01.611587 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d")
E1214 18:45:01.614828 11428 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983\"" failed. No retries permitted until 2016-12-14 18:47:01.614806535 +0000 UTC (durationBeforeRetry 2m0s). Error: Volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d") is not yet attached according to node status.
I1214 18:45:02.417507 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/1dc4c710-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d").
I1214 18:45:02.417606 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/1dc5073b-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d").
I1214 18:45:03.420077 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/1dc5ab51-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d").
I1214 18:45:05.426634 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/1dc2012b-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d").
I1214 18:45:05.427105 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/1dc22f6d-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d").
I1214 18:45:14.352826 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/10efd90d-c22d-11e6-ac5f-028961b0177d-default-token-piiyk" (spec.Name: "default-token-piiyk") pod "10efd90d-c22d-11e6-ac5f-028961b0177d" (UID: "10efd90d-c22d-11e6-ac5f-028961b0177d").
I1214 18:45:18.374678 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/10f03cce-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "10f03cce-c22d-11e6-ac5f-028961b0177d" (UID: "10f03cce-c22d-11e6-ac5f-028961b0177d").
I1214 18:46:25.350954 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/10efd90d-c22d-11e6-ac5f-028961b0177d-default-token-piiyk" (spec.Name: "default-token-piiyk") pod "10efd90d-c22d-11e6-ac5f-028961b0177d" (UID: "10efd90d-c22d-11e6-ac5f-028961b0177d").
I1214 18:46:46.404444 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/10f03cce-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "10f03cce-c22d-11e6-ac5f-028961b0177d" (UID: "10f03cce-c22d-11e6-ac5f-028961b0177d").
I1214 18:46:59.032573 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d")
I1214 18:46:59.035512 11428 operation_executor.go:1219] Controller successfully attached volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d") devicePath: "/dev/xvdbd"
I1214 18:46:59.132977 11428 operation_executor.go:812] Entering MountVolume.WaitForAttach for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d") DevicePath: "/dev/xvdbd"
I1214 18:46:59.433968 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d")
I1214 18:46:59.436842 11428 operation_executor.go:1219] Controller successfully attached volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d") devicePath: "/dev/xvdbb"
I1214 18:46:59.534506 11428 operation_executor.go:812] Entering MountVolume.WaitForAttach for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d") DevicePath: "/dev/xvdbb"
I1214 18:47:00.133268 11428 attacher.go:148] Successfully found attached AWS Volume "aws://us-gov-west-1a/vol-97aad60c".
I1214 18:47:00.133314 11428 operation_executor.go:832] MountVolume.WaitForAttach succeeded for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d").
I1214 18:47:00.182259 11428 operation_executor.go:871] MountVolume.MountDevice succeeded for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d") device mount path "/var/vcap/data/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-gov-west-1a/vol-97aad60c"
I1214 18:47:00.187311 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-97aad60c" (spec.Name: "pvc-752687cb-ac43-11e6-bfb3-0268f725c9ef") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d").
I1214 18:47:00.431005 11428 hairpin.go:61] Unable to find pair interface, setting up all interfaces: exec: "nsenter": executable file not found in $PATH
I1214 18:47:00.431348 11428 docker_manager.go:2236] Determined pod ip after infra change: "xf2ee8dcfaed64-master-4003242301-2t794_default(1dc22f6d-c22d-11e6-ac5f-028961b0177d)": "10.100.82.3"
I1214 18:47:00.437216 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d")
I1214 18:47:00.440475 11428 operation_executor.go:1219] Controller successfully attached volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d") devicePath: "/dev/xvdbe"
I1214 18:47:00.534762 11428 attacher.go:148] Successfully found attached AWS Volume "aws://us-gov-west-1a/vol-15562f8e".
I1214 18:47:00.534894 11428 operation_executor.go:832] MountVolume.WaitForAttach succeeded for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d").
I1214 18:47:00.537752 11428 operation_executor.go:812] Entering MountVolume.WaitForAttach for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d") DevicePath: "/dev/xvdbe"
I1214 18:47:00.591926 11428 operation_executor.go:871] MountVolume.MountDevice succeeded for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d") device mount path "/var/vcap/data/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-gov-west-1a/vol-15562f8e"
I1214 18:47:00.596861 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-15562f8e" (spec.Name: "pvc-8f232c59-b38a-11e6-b781-02ba9586f3cd") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d").
I1214 18:47:01.040485 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d")
I1214 18:47:01.043713 11428 operation_executor.go:1219] Controller successfully attached volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d") devicePath: "/dev/xvdbc"
I1214 18:47:01.111162 11428 hairpin.go:61] Unable to find pair interface, setting up all interfaces: exec: "nsenter": executable file not found in $PATH
I1214 18:47:01.111525 11428 docker_manager.go:2236] Determined pod ip after infra change: "x864fe17555694-189251105-lxw4k_default(1dc2012b-c22d-11e6-ac5f-028961b0177d)": "10.100.82.4"
I1214 18:47:01.141171 11428 operation_executor.go:812] Entering MountVolume.WaitForAttach for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-781962e3" (spec.Name: "pvc-508cd719-b777-11e6-aba4-021e4e50ed5f") pod "1dc5073b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5073b-c22d-11e6-ac5f-028961b0177d") DevicePath: "/dev/xvdbc"
I1214 18:47:01.449887 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/1dc2012b-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "1dc2012b-c22d-11e6-ac5f-028961b0177d" (UID: "1dc2012b-c22d-11e6-ac5f-028961b0177d").
I1214 18:47:01.450316 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/1dc22f6d-c22d-11e6-ac5f-028961b0177d-default-token-d1n4j" (spec.Name: "default-token-d1n4j") pod "1dc22f6d-c22d-11e6-ac5f-028961b0177d" (UID: "1dc22f6d-c22d-11e6-ac5f-028961b0177d").
I1214 18:47:01.538206 11428 attacher.go:148] Successfully found attached AWS Volume "aws://us-gov-west-1a/vol-36ccb4ad".
I1214 18:47:01.538245 11428 operation_executor.go:832] MountVolume.WaitForAttach succeeded for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d").
I1214 18:47:01.592142 11428 operation_executor.go:871] MountVolume.MountDevice succeeded for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d") device mount path "/var/vcap/data/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-gov-west-1a/vol-36ccb4ad"
I1214 18:47:01.598000 11428 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-36ccb4ad" (spec.Name: "pvc-83b70a4d-b70e-11e6-aba4-021e4e50ed5f") pod "1dc4c710-c22d-11e6-ac5f-028961b0177d" (UID: "1dc4c710-c22d-11e6-ac5f-028961b0177d").
I1214 18:47:01.647771 11428 reconciler.go:230] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d")
I1214 18:47:01.650516 11428 operation_executor.go:1219] Controller successfully attached volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d") devicePath: "/dev/xvdba"
I1214 18:47:01.748283 11428 operation_executor.go:812] Entering MountVolume.WaitForAttach for volume "kubernetes.io/aws-ebs/aws://us-gov-west-1a/vol-18c1b983" (spec.Name: "pvc-17665461-b71a-11e6-aba4-021e4e50ed5f") pod "1dc5ab51-c22d-11e6-ac5f-028961b0177d" (UID: "1dc5ab51-c22d-11e6-ac5f-028961b0177d") DevicePath: "/dev/xvdba"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment