gitschaub · March 22, 2016 21:46
diff --git a/kubelet_kube-proxy.log b/kubelet_kube-proxy.log
 core@k8s-master /var/log $ journalctl -u kube-proxy
 Mar 21 17:38:39 k8s-master systemd[1]: Started Kubernetes Kube-Proxy Server.
 Mar 21 17:38:39 k8s-master kube-proxy[713]: E0321 17:38:39.544714     713 server.go:357] Not trying iptables proxy: can't get Node "k8s-master": Get https://147.75.197.101:443/api/v1/nodes/k8s-master: d
 Mar 21 17:38:39 k8s-master kube-proxy[713]: F0321 17:38:39.544968     713 server.go:272] Unable to create proxier: failed to select a host interface: Unable to select an IP.
 Mar 21 17:38:39 k8s-master systemd[1]: kube-proxy.service: Main process exited, code=exited, status=255/n/a
 Mar 21 17:38:39 k8s-master systemd[1]: kube-proxy.service: Unit entered failed state.
 Mar 21 17:38:39 k8s-master systemd[1]: kube-proxy.service: Failed with result 'exit-code'.
 Mar 21 17:38:39 k8s-master systemd[1]: kube-proxy.service: Service hold-off time over, scheduling restart.
 Mar 21 17:38:39 k8s-master systemd[1]: Stopped Kubernetes Kube-Proxy Server.
 Mar 21 17:38:39 k8s-master systemd[1]: Started Kubernetes Kube-Proxy Server.
 Mar 21 17:38:39 k8s-master kube-proxy[826]: E0321 17:38:39.821154     826 server.go:357] Not trying iptables proxy: can't get Node "k8s-master": Get https://147.75.197.101:443/api/v1/nodes/k8s-master: d
 Mar 21 17:38:39 k8s-master kube-proxy[826]: E0321 17:38:39.892598     826 proxier.go:193] Error removing pure-iptables proxy rule: error checking rule: exit status 2: iptables v1.4.21: Couldn't load tar
 Mar 21 17:38:39 k8s-master kube-proxy[826]: Try `iptables -h' or 'iptables --help' for more information.
 Mar 21 17:38:39 k8s-master kube-proxy[826]: E0321 17:38:39.894663     826 proxier.go:197] Error removing pure-iptables proxy rule: error checking rule: exit status 2: iptables v1.4.21: Couldn't load tar
 Mar 21 17:38:39 k8s-master kube-proxy[826]: Try `iptables -h' or 'iptables --help' for more information.
 Mar 21 17:38:39 k8s-master kube-proxy[826]: E0321 17:38:39.897963     826 event.go:197] Unable to write event: 'Post https://147.75.197.101:443/api/v1/namespaces/default/events: dial tcp 147.75.197.101:
 Mar 21 18:18:43 k8s-master kube-proxy[826]: E0321 18:18:43.039623     826 proxysocket.go:216] ReadFrom failed, exiting ProxyLoop: read udp [::]:53506: use of closed network connection
 Mar 21 18:28:42 k8s-master kube-proxy[826]: E0321 18:28:42.184967     826 proxysocket.go:216] ReadFrom failed, exiting ProxyLoop: read udp [::]:56024: use of closed network connection
 Mar 21 19:07:52 k8s-master kube-proxy[826]: I0321 19:07:52.936293     826 proxier.go:614] Opened iptables from-containers public port for service "default/frontend:" on TCP port 30303
 Mar 21 19:07:52 k8s-master kube-proxy[826]: I0321 19:07:52.945833     826 proxier.go:625] Opened iptables from-host public port for service "default/frontend:" on TCP port 30303
 Mar 21 19:08:56 k8s-master kube-proxy[826]: E0321 19:08:56.695503     826 proxysocket.go:93] Couldn't find an endpoint for default/frontend:: missing service entry
 Mar 21 19:08:56 k8s-master kube-proxy[826]: E0321 19:08:56.695601     826 proxysocket.go:138] Failed to connect to balancer: missing service entry
 Mar 21 19:09:20 k8s-master kube-proxy[826]: E0321 19:09:20.080846     826 proxysocket.go:93] Couldn't find an endpoint for default/frontend:: missing service entry
 Mar 21 19:09:20 k8s-master kube-proxy[826]: E0321 19:09:20.080920     826 proxysocket.go:138] Failed to connect to balancer: missing service entry
 Mar 21 19:09:20 k8s-master kube-proxy[826]: E0321 19:09:20.081004     826 proxysocket.go:93] Couldn't find an endpoint for default/frontend:: missing service entry
 Mar 21 19:09:20 k8s-master kube-proxy[826]: E0321 19:09:20.081029     826 proxysocket.go:138] Failed to connect to balancer: missing service entry
 Mar 21 19:18:42 k8s-master kube-proxy[826]: E0321 19:18:42.308173     826 proxysocket.go:216] ReadFrom failed, exiting ProxyLoop: read udp [::]:37830: use of closed network connection
 Mar 21 19:28:18 k8s-master kube-proxy[826]: I0321 19:28:18.389125     826 proxier.go:614] Opened iptables from-containers public port for service "kube-system/kubernetes-dashboard:" on TCP port 30303
 Mar 21 19:28:18 k8s-master kube-proxy[826]: I0321 19:28:18.395413     826 proxier.go:625] Opened iptables from-host public port for service "kube-system/kubernetes-dashboard:" on TCP port 30303
 Mar 21 19:58:46 k8s-master kube-proxy[826]: E0321 19:58:46.189404     826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry
 ...
 Mar 21 21:14:43 k8s-master kube-proxy[826]: I0321 21:14:43.452205     826 proxier.go:614] Opened iptables from-containers public port for service "kube-system/kubernetes-dashboard:" on TCP port 30303
 Mar 21 21:14:43 k8s-master kube-proxy[826]: I0321 21:14:43.461299     826 proxier.go:625] Opened iptables from-host public port for service "kube-system/kubernetes-dashboard:" on TCP port 30303
 Mar 21 21:18:42 k8s-master kube-proxy[826]: E0321 21:18:42.980344     826 proxysocket.go:216] ReadFrom failed, exiting ProxyLoop: read udp [::]:45959: use of closed network connection
 Mar 21 21:37:47 k8s-master kube-proxy[826]: E0321 21:37:47.070528     826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing endpoints
 Mar 21 21:37:52 k8s-master kube-proxy[826]: E0321 21:37:52.079845     826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing endpoints
 Mar 21 21:37:57 k8s-master kube-proxy[826]: E0321 21:37:57.090008     826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry
 Mar 21 21:38:02 k8s-master kube-proxy[826]: E0321 21:38:02.099143     826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry
 Mar 21 21:38:07 k8s-master kube-proxy[826]: E0321 21:38:07.124469     826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry
 Mar 21 21:38:12 k8s-master kube-proxy[826]: E0321 21:38:12.133999     826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry
 Mar 21 21:38:17 k8s-master kube-proxy[826]: E0321 21:38:17.143064     826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry
 Mar 21 21:38:22 k8s-master kube-proxy[826]: E0321 21:38:22.152267     826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry
 Mar 21 21:38:27 k8s-master kube-proxy[826]: E0321 21:38:27.162103     826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry
 Mar 21 21:38:42 k8s-master kube-proxy[826]: E0321 21:38:42.164397     826 proxysocket.go:216] ReadFrom failed, exiting ProxyLoop: read udp [::]:34804: use of closed network connection
 Mar 21 21:38:43 k8s-master kube-proxy[826]: E0321 21:38:43.262957     826 proxysocket.go:165] I/O error: write tcp 172.16.10.5:56414: broken pipe
 Mar 21 21:38:43 k8s-master kube-proxy[826]: E0321 21:38:43.271413     826 proxysocket.go:165] I/O error: write tcp 172.16.10.5:56426: broken pipe
 ...

 core@k8s-master /var/log $ docker ps -a | grep "About an"
 a2c388b6c92f        gcr.io/google_containers/kube2sky:1.14                "/kube2sky --domain=c"   About an hour ago   Exited (255) About an hour ago                       k8s_kube2sky.74cfb978_kube-dns-v11-0zjme_kube-system_01d1da63-f06a-11e5-822b-0cc47a205a18_382d1c18
 c9bbc237d7ce        gcr.io/google_containers/etcd-amd64:2.2.1             "/usr/local/bin/etcd "   About an hour ago   Exited (0) About an hour ago                         k8s_etcd.822e3531_kube-dns-v11-0zjme_kube-system_01d1da63-f06a-11e5-822b-0cc47a205a18_4fef3a06
 6294677152dc        gcr.io/google_containers/exechealthz:1.0              "/exechealthz '-cmd=n"   About an hour ago   Exited (2) About an hour ago                         k8s_healthz.40c4147f_kube-dns-v11-0zjme_kube-system_01d1da63-f06a-11e5-822b-0cc47a205a18_95480b8d
 b913400bc940        gcr.io/google_containers/skydns:2015-10-13-8c72f8c    "/skydns -machines=ht"   About an hour ago   Exited (2) About an hour ago                         k8s_skydns.7b5d3ad2_kube-dns-v11-0zjme_kube-system_01d1da63-f06a-11e5-822b-0cc47a205a18_0044eab2
 4418c7c1bbfe        gcr.io/google_containers/pause:0.8.0                  "/pause"                 About an hour ago   Exited (0) About an hour ago                         k8s_POD.63f848fb_kube-dns-v11-0zjme_kube-system_01d1da63-f06a-11e5-822b-0cc47a205a18_cd2fa60a
 core@k8s-master /var/log $ docker logs a2c388b6c92f (kube2sky)
 I0322 20:09:40.985805       1 kube2sky.go:462] Etcd server found: http://127.0.0.1:4001
 I0322 20:09:41.994632       1 kube2sky.go:529] Using https://10.254.0.1:443 for kubernetes master
 I0322 20:09:41.994664       1 kube2sky.go:530] Using kubernetes API <nil>
 I0322 20:09:41.995502       1 kube2sky.go:598] Waiting for service: default/kubernetes
 I0322 20:09:42.039937       1 kube2sky.go:660] Successfully added DNS record for Kubernetes service.
 F0322 20:18:43.511325       1 kube2sky.go:625] Received signal terminated
 core@k8s-master /var/log $ docker logs c9bbc237d7ce (etcd)
 2016-03-22 20:09:40.806024 I | etcdmain: etcd Version: 2.2.1
 2016-03-22 20:09:40.806105 I | etcdmain: Git SHA: 75f8282
 2016-03-22 20:09:40.806123 I | etcdmain: Go Version: go1.5.1
 2016-03-22 20:09:40.806140 I | etcdmain: Go OS/Arch: linux/amd64
 2016-03-22 20:09:40.806166 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
 2016-03-22 20:09:40.806679 I | etcdmain: listening for peers on http://localhost:2380
 2016-03-22 20:09:40.806862 I | etcdmain: listening for peers on http://localhost:7001
 2016-03-22 20:09:40.806952 I | etcdmain: listening for client requests on http://127.0.0.1:2379
 2016-03-22 20:09:40.807025 I | etcdmain: listening for client requests on http://127.0.0.1:4001
 2016-03-22 20:09:40.807440 I | etcdserver: name = default
 2016-03-22 20:09:40.807470 I | etcdserver: data dir = /var/etcd/data
 2016-03-22 20:09:40.807489 I | etcdserver: member dir = /var/etcd/data/member
 2016-03-22 20:09:40.807506 I | etcdserver: heartbeat = 100ms
 2016-03-22 20:09:40.807522 I | etcdserver: election = 1000ms
 2016-03-22 20:09:40.807539 I | etcdserver: snapshot count = 10000
 2016-03-22 20:09:40.807565 I | etcdserver: advertise client URLs = http://127.0.0.1:2379,http://127.0.0.1:4001
 2016-03-22 20:09:40.807589 I | etcdserver: initial advertise peer URLs = http://localhost:2380,http://localhost:7001
 2016-03-22 20:09:40.807623 I | etcdserver: initial cluster = default=http://localhost:2380,default=http://localhost:7001
 2016-03-22 20:09:40.809518 I | etcdserver: starting member 6a5871dbdd12c17c in cluster f68652439e3f8f2a
 2016-03-22 20:09:40.809623 I | raft: 6a5871dbdd12c17c became follower at term 0
 2016-03-22 20:09:40.809658 I | raft: newRaft 6a5871dbdd12c17c [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
 2016-03-22 20:09:40.809678 I | raft: 6a5871dbdd12c17c became follower at term 1
 2016-03-22 20:09:40.809974 I | etcdserver: starting server... [version: 2.2.1, cluster version: to_be_decided]
 2016-03-22 20:09:40.812454 N | etcdserver: added local member 6a5871dbdd12c17c [http://localhost:2380 http://localhost:7001] to cluster f68652439e3f8f2a
 2016-03-22 20:09:41.610073 I | raft: 6a5871dbdd12c17c is starting a new election at term 1
 2016-03-22 20:09:41.610224 I | raft: 6a5871dbdd12c17c became candidate at term 2
 2016-03-22 20:09:41.610245 I | raft: 6a5871dbdd12c17c received vote from 6a5871dbdd12c17c at term 2
 2016-03-22 20:09:41.610279 I | raft: 6a5871dbdd12c17c became leader at term 2
 2016-03-22 20:09:41.610305 I | raft: raft.node: 6a5871dbdd12c17c elected leader 6a5871dbdd12c17c at term 2
 2016-03-22 20:09:41.610976 I | etcdserver: setting up the initial cluster version to 2.2
 2016-03-22 20:09:41.611212 I | etcdserver: published {Name:default ClientURLs:[http://127.0.0.1:2379 http://127.0.0.1:4001]} to cluster f68652439e3f8f2a
 2016-03-22 20:09:41.611356 N | etcdserver: set the initial cluster version to 2.2
 2016-03-22 20:18:43.509341 N | osutil: received terminated signal, shutting down...
 core@k8s-master /var/log $ docker logs 6294677152dc (healthz)
 ...
 2016/03/22 20:11:52 Worker running nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
 2016/03/22 20:11:54 Worker running nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
 2016/03/22 20:11:56 Worker running nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
 2016/03/22 20:11:58 Worker running nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
 2016/03/22 20:12:00 Worker running nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
 2016/03/22 20:12:01 Client ip 172.16.10.1:36484 requesting /healthz probe servicing cmd nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
 ...

 core@k8s-master /var/log $ docker logs b913400bc940 (skydns)
 2016/03/22 20:09:40 skydns: falling back to default configuration, could not read from etcd: 501: All the given peers are not reachable (failed to propose on members [http://127.0.0.1:4001] twice [last error: Get http://127.0.0.1:4001/v2/keys/skydns/config?quorum=false&recursive=false&sorted=false: dial tcp 127.0.0.1:4001: connection refused]) [0]
 2016/03/22 20:09:40 skydns: ready for queries on cluster.local. for tcp://0.0.0.0:53 [rcache 0]
 2016/03/22 20:09:40 skydns: ready for queries on cluster.local. for udp://0.0.0.0:53 [rcache 0]
	core@k8s-master /var/log $ journalctl -u kube-proxy
	Mar 21 17:38:39 k8s-master systemd[1]: Started Kubernetes Kube-Proxy Server.
	Mar 21 17:38:39 k8s-master kube-proxy[713]: E0321 17:38:39.544714 713 server.go:357] Not trying iptables proxy: can't get Node "k8s-master": Get https://147.75.197.101:443/api/v1/nodes/k8s-master: d
	Mar 21 17:38:39 k8s-master kube-proxy[713]: F0321 17:38:39.544968 713 server.go:272] Unable to create proxier: failed to select a host interface: Unable to select an IP.
	Mar 21 17:38:39 k8s-master systemd[1]: kube-proxy.service: Main process exited, code=exited, status=255/n/a
	Mar 21 17:38:39 k8s-master systemd[1]: kube-proxy.service: Unit entered failed state.
	Mar 21 17:38:39 k8s-master systemd[1]: kube-proxy.service: Failed with result 'exit-code'.
	Mar 21 17:38:39 k8s-master systemd[1]: kube-proxy.service: Service hold-off time over, scheduling restart.
	Mar 21 17:38:39 k8s-master systemd[1]: Stopped Kubernetes Kube-Proxy Server.
	Mar 21 17:38:39 k8s-master systemd[1]: Started Kubernetes Kube-Proxy Server.
	Mar 21 17:38:39 k8s-master kube-proxy[826]: E0321 17:38:39.821154 826 server.go:357] Not trying iptables proxy: can't get Node "k8s-master": Get https://147.75.197.101:443/api/v1/nodes/k8s-master: d
	Mar 21 17:38:39 k8s-master kube-proxy[826]: E0321 17:38:39.892598 826 proxier.go:193] Error removing pure-iptables proxy rule: error checking rule: exit status 2: iptables v1.4.21: Couldn't load tar
	Mar 21 17:38:39 k8s-master kube-proxy[826]: Try `iptables -h' or 'iptables --help' for more information.
	Mar 21 17:38:39 k8s-master kube-proxy[826]: E0321 17:38:39.894663 826 proxier.go:197] Error removing pure-iptables proxy rule: error checking rule: exit status 2: iptables v1.4.21: Couldn't load tar
	Mar 21 17:38:39 k8s-master kube-proxy[826]: Try `iptables -h' or 'iptables --help' for more information.
	Mar 21 17:38:39 k8s-master kube-proxy[826]: E0321 17:38:39.897963 826 event.go:197] Unable to write event: 'Post https://147.75.197.101:443/api/v1/namespaces/default/events: dial tcp 147.75.197.101:
	Mar 21 18:18:43 k8s-master kube-proxy[826]: E0321 18:18:43.039623 826 proxysocket.go:216] ReadFrom failed, exiting ProxyLoop: read udp [::]:53506: use of closed network connection
	Mar 21 18:28:42 k8s-master kube-proxy[826]: E0321 18:28:42.184967 826 proxysocket.go:216] ReadFrom failed, exiting ProxyLoop: read udp [::]:56024: use of closed network connection
	Mar 21 19:07:52 k8s-master kube-proxy[826]: I0321 19:07:52.936293 826 proxier.go:614] Opened iptables from-containers public port for service "default/frontend:" on TCP port 30303
	Mar 21 19:07:52 k8s-master kube-proxy[826]: I0321 19:07:52.945833 826 proxier.go:625] Opened iptables from-host public port for service "default/frontend:" on TCP port 30303
	Mar 21 19:08:56 k8s-master kube-proxy[826]: E0321 19:08:56.695503 826 proxysocket.go:93] Couldn't find an endpoint for default/frontend:: missing service entry
	Mar 21 19:08:56 k8s-master kube-proxy[826]: E0321 19:08:56.695601 826 proxysocket.go:138] Failed to connect to balancer: missing service entry
	Mar 21 19:09:20 k8s-master kube-proxy[826]: E0321 19:09:20.080846 826 proxysocket.go:93] Couldn't find an endpoint for default/frontend:: missing service entry
	Mar 21 19:09:20 k8s-master kube-proxy[826]: E0321 19:09:20.080920 826 proxysocket.go:138] Failed to connect to balancer: missing service entry
	Mar 21 19:09:20 k8s-master kube-proxy[826]: E0321 19:09:20.081004 826 proxysocket.go:93] Couldn't find an endpoint for default/frontend:: missing service entry
	Mar 21 19:09:20 k8s-master kube-proxy[826]: E0321 19:09:20.081029 826 proxysocket.go:138] Failed to connect to balancer: missing service entry
	Mar 21 19:18:42 k8s-master kube-proxy[826]: E0321 19:18:42.308173 826 proxysocket.go:216] ReadFrom failed, exiting ProxyLoop: read udp [::]:37830: use of closed network connection
	Mar 21 19:28:18 k8s-master kube-proxy[826]: I0321 19:28:18.389125 826 proxier.go:614] Opened iptables from-containers public port for service "kube-system/kubernetes-dashboard:" on TCP port 30303
	Mar 21 19:28:18 k8s-master kube-proxy[826]: I0321 19:28:18.395413 826 proxier.go:625] Opened iptables from-host public port for service "kube-system/kubernetes-dashboard:" on TCP port 30303
	Mar 21 19:58:46 k8s-master kube-proxy[826]: E0321 19:58:46.189404 826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry
	...
	Mar 21 21:14:43 k8s-master kube-proxy[826]: I0321 21:14:43.452205 826 proxier.go:614] Opened iptables from-containers public port for service "kube-system/kubernetes-dashboard:" on TCP port 30303
	Mar 21 21:14:43 k8s-master kube-proxy[826]: I0321 21:14:43.461299 826 proxier.go:625] Opened iptables from-host public port for service "kube-system/kubernetes-dashboard:" on TCP port 30303
	Mar 21 21:18:42 k8s-master kube-proxy[826]: E0321 21:18:42.980344 826 proxysocket.go:216] ReadFrom failed, exiting ProxyLoop: read udp [::]:45959: use of closed network connection
	Mar 21 21:37:47 k8s-master kube-proxy[826]: E0321 21:37:47.070528 826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing endpoints
	Mar 21 21:37:52 k8s-master kube-proxy[826]: E0321 21:37:52.079845 826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing endpoints
	Mar 21 21:37:57 k8s-master kube-proxy[826]: E0321 21:37:57.090008 826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry
	Mar 21 21:38:02 k8s-master kube-proxy[826]: E0321 21:38:02.099143 826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry
	Mar 21 21:38:07 k8s-master kube-proxy[826]: E0321 21:38:07.124469 826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry
	Mar 21 21:38:12 k8s-master kube-proxy[826]: E0321 21:38:12.133999 826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry
	Mar 21 21:38:17 k8s-master kube-proxy[826]: E0321 21:38:17.143064 826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry
	Mar 21 21:38:22 k8s-master kube-proxy[826]: E0321 21:38:22.152267 826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry
	Mar 21 21:38:27 k8s-master kube-proxy[826]: E0321 21:38:27.162103 826 proxysocket.go:93] Couldn't find an endpoint for kube-system/kube-dns:dns: missing service entry
	Mar 21 21:38:42 k8s-master kube-proxy[826]: E0321 21:38:42.164397 826 proxysocket.go:216] ReadFrom failed, exiting ProxyLoop: read udp [::]:34804: use of closed network connection
	Mar 21 21:38:43 k8s-master kube-proxy[826]: E0321 21:38:43.262957 826 proxysocket.go:165] I/O error: write tcp 172.16.10.5:56414: broken pipe
	Mar 21 21:38:43 k8s-master kube-proxy[826]: E0321 21:38:43.271413 826 proxysocket.go:165] I/O error: write tcp 172.16.10.5:56426: broken pipe
	...

	core@k8s-master /var/log $ docker ps -a \| grep "About an"
	a2c388b6c92f gcr.io/google_containers/kube2sky:1.14 "/kube2sky --domain=c" About an hour ago Exited (255) About an hour ago k8s_kube2sky.74cfb978_kube-dns-v11-0zjme_kube-system_01d1da63-f06a-11e5-822b-0cc47a205a18_382d1c18
	c9bbc237d7ce gcr.io/google_containers/etcd-amd64:2.2.1 "/usr/local/bin/etcd " About an hour ago Exited (0) About an hour ago k8s_etcd.822e3531_kube-dns-v11-0zjme_kube-system_01d1da63-f06a-11e5-822b-0cc47a205a18_4fef3a06
	6294677152dc gcr.io/google_containers/exechealthz:1.0 "/exechealthz '-cmd=n" About an hour ago Exited (2) About an hour ago k8s_healthz.40c4147f_kube-dns-v11-0zjme_kube-system_01d1da63-f06a-11e5-822b-0cc47a205a18_95480b8d
	b913400bc940 gcr.io/google_containers/skydns:2015-10-13-8c72f8c "/skydns -machines=ht" About an hour ago Exited (2) About an hour ago k8s_skydns.7b5d3ad2_kube-dns-v11-0zjme_kube-system_01d1da63-f06a-11e5-822b-0cc47a205a18_0044eab2
	4418c7c1bbfe gcr.io/google_containers/pause:0.8.0 "/pause" About an hour ago Exited (0) About an hour ago k8s_POD.63f848fb_kube-dns-v11-0zjme_kube-system_01d1da63-f06a-11e5-822b-0cc47a205a18_cd2fa60a
	core@k8s-master /var/log $ docker logs a2c388b6c92f (kube2sky)
	I0322 20:09:40.985805 1 kube2sky.go:462] Etcd server found: http://127.0.0.1:4001
	I0322 20:09:41.994632 1 kube2sky.go:529] Using https://10.254.0.1:443 for kubernetes master
	I0322 20:09:41.994664 1 kube2sky.go:530] Using kubernetes API <nil>
	I0322 20:09:41.995502 1 kube2sky.go:598] Waiting for service: default/kubernetes
	I0322 20:09:42.039937 1 kube2sky.go:660] Successfully added DNS record for Kubernetes service.
	F0322 20:18:43.511325 1 kube2sky.go:625] Received signal terminated
	core@k8s-master /var/log $ docker logs c9bbc237d7ce (etcd)
	2016-03-22 20:09:40.806024 I \| etcdmain: etcd Version: 2.2.1
	2016-03-22 20:09:40.806105 I \| etcdmain: Git SHA: 75f8282
	2016-03-22 20:09:40.806123 I \| etcdmain: Go Version: go1.5.1
	2016-03-22 20:09:40.806140 I \| etcdmain: Go OS/Arch: linux/amd64
	2016-03-22 20:09:40.806166 I \| etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
	2016-03-22 20:09:40.806679 I \| etcdmain: listening for peers on http://localhost:2380
	2016-03-22 20:09:40.806862 I \| etcdmain: listening for peers on http://localhost:7001
	2016-03-22 20:09:40.806952 I \| etcdmain: listening for client requests on http://127.0.0.1:2379
	2016-03-22 20:09:40.807025 I \| etcdmain: listening for client requests on http://127.0.0.1:4001
	2016-03-22 20:09:40.807440 I \| etcdserver: name = default
	2016-03-22 20:09:40.807470 I \| etcdserver: data dir = /var/etcd/data
	2016-03-22 20:09:40.807489 I \| etcdserver: member dir = /var/etcd/data/member
	2016-03-22 20:09:40.807506 I \| etcdserver: heartbeat = 100ms
	2016-03-22 20:09:40.807522 I \| etcdserver: election = 1000ms
	2016-03-22 20:09:40.807539 I \| etcdserver: snapshot count = 10000
	2016-03-22 20:09:40.807565 I \| etcdserver: advertise client URLs = http://127.0.0.1:2379,http://127.0.0.1:4001
	2016-03-22 20:09:40.807589 I \| etcdserver: initial advertise peer URLs = http://localhost:2380,http://localhost:7001
	2016-03-22 20:09:40.807623 I \| etcdserver: initial cluster = default=http://localhost:2380,default=http://localhost:7001
	2016-03-22 20:09:40.809518 I \| etcdserver: starting member 6a5871dbdd12c17c in cluster f68652439e3f8f2a
	2016-03-22 20:09:40.809623 I \| raft: 6a5871dbdd12c17c became follower at term 0
	2016-03-22 20:09:40.809658 I \| raft: newRaft 6a5871dbdd12c17c [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
	2016-03-22 20:09:40.809678 I \| raft: 6a5871dbdd12c17c became follower at term 1
	2016-03-22 20:09:40.809974 I \| etcdserver: starting server... [version: 2.2.1, cluster version: to_be_decided]
	2016-03-22 20:09:40.812454 N \| etcdserver: added local member 6a5871dbdd12c17c [http://localhost:2380 http://localhost:7001] to cluster f68652439e3f8f2a
	2016-03-22 20:09:41.610073 I \| raft: 6a5871dbdd12c17c is starting a new election at term 1
	2016-03-22 20:09:41.610224 I \| raft: 6a5871dbdd12c17c became candidate at term 2
	2016-03-22 20:09:41.610245 I \| raft: 6a5871dbdd12c17c received vote from 6a5871dbdd12c17c at term 2
	2016-03-22 20:09:41.610279 I \| raft: 6a5871dbdd12c17c became leader at term 2
	2016-03-22 20:09:41.610305 I \| raft: raft.node: 6a5871dbdd12c17c elected leader 6a5871dbdd12c17c at term 2
	2016-03-22 20:09:41.610976 I \| etcdserver: setting up the initial cluster version to 2.2
	2016-03-22 20:09:41.611212 I \| etcdserver: published {Name:default ClientURLs:[http://127.0.0.1:2379 http://127.0.0.1:4001]} to cluster f68652439e3f8f2a
	2016-03-22 20:09:41.611356 N \| etcdserver: set the initial cluster version to 2.2
	2016-03-22 20:18:43.509341 N \| osutil: received terminated signal, shutting down...
	core@k8s-master /var/log $ docker logs 6294677152dc (healthz)
	...
	2016/03/22 20:11:52 Worker running nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
	2016/03/22 20:11:54 Worker running nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
	2016/03/22 20:11:56 Worker running nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
	2016/03/22 20:11:58 Worker running nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
	2016/03/22 20:12:00 Worker running nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
	2016/03/22 20:12:01 Client ip 172.16.10.1:36484 requesting /healthz probe servicing cmd nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
	...

	core@k8s-master /var/log $ docker logs b913400bc940 (skydns)
	2016/03/22 20:09:40 skydns: falling back to default configuration, could not read from etcd: 501: All the given peers are not reachable (failed to propose on members [http://127.0.0.1:4001] twice [last error: Get http://127.0.0.1:4001/v2/keys/skydns/config?quorum=false&recursive=false&sorted=false: dial tcp 127.0.0.1:4001: connection refused]) [0]
	2016/03/22 20:09:40 skydns: ready for queries on cluster.local. for tcp://0.0.0.0:53 [rcache 0]
	2016/03/22 20:09:40 skydns: ready for queries on cluster.local. for udp://0.0.0.0:53 [rcache 0]