Skip to content

Instantly share code, notes, and snippets.

@etiennetremel

etiennetremel/Previous kube-dns logs

Last active April 20, 2017 08:56
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save etiennetremel/622e14bc77375c8f3d256901896439cb to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save etiennetremel/622e14bc77375c8f3d256901896439cb to your computer and use it in GitHub Desktop.
Download ZIP
kube-dns bug report
Raw
kube-dns describe
❯ kubectl describe po kube-dns-3263495268-9ltl2 -n kube-system
Name: kube-dns-3263495268-9ltl2
Namespace: kube-system
Node: gke-development-euro-auto-scaling-pre-d396a110-6fgw/172.20.93.12
Start Time: Thu, 20 Apr 2017 08:52:26 +0200
Labels: k8s-app=kube-dns
pod-template-hash=3263495268
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"kube-dns-3263495268","uid":"05ebeee4-1dee-11e7-93d2-42010a84...
scheduler.alpha.kubernetes.io/critical-pod=
Status: Pending
IP: 10.248.21.76
Controllers: ReplicaSet/kube-dns-3263495268
Containers:
kubedns:
Container ID: docker://f964eac2a9e4d3076ef255996044f974c36d81f6fe777743d7b59423372e3120
Image: eu.gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.1
Image ID: docker://sha256:f8363dbf447b571de528c7c03b8cb71725d627e30f2d0a0d403df40c2afe32e8
Ports: 10053/UDP, 10053/TCP, 10055/TCP
Args:
--domain=cluster.local.
--dns-port=10053
--config-dir=/kube-dns-config
--v=2
State: Running
Started: Thu, 20 Apr 2017 09:17:51 +0200
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Mon, 01 Jan 0001 00:00:00 +0000
Finished: Thu, 20 Apr 2017 09:17:28 +0200
Ready: True
Restart Count: 2
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Liveness: http-get http://:10054/healthcheck/kubedns delay=60s timeout=5s period=10s #success=1 #failure=5
Readiness: http-get http://:8081/readiness delay=3s timeout=5s period=10s #success=1 #failure=3
Environment:
PROMETHEUS_PORT: 10055
Mounts:
/kube-dns-config from kube-dns-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-896k2 (ro)
dnsmasq:
Container ID:
Image: eu.gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1
Image ID:
Ports: 53/UDP, 53/TCP
Args:
-v=2
-logtostderr
-configDir=/etc/k8s/dns/dnsmasq-nanny
-restartDnsmasq=true
--
-k
--cache-size=1000
--log-facility=-
--server=/cluster.local/127.0.0.1#10053
--server=/in-addr.arpa/127.0.0.1#10053
--server=/ip6.arpa/127.0.0.1#10053
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Requests:
cpu: 150m
memory: 20Mi
Liveness: http-get http://:10054/healthcheck/dnsmasq delay=60s timeout=5s period=10s #success=1 #failure=5
Environment: <none>
Mounts:
/etc/k8s/dns/dnsmasq-nanny from kube-dns-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-896k2 (ro)
sidecar:
Container ID: docker://c454d983ea45b47f38323d376065812e9672fd0be4c6967c66a1f1002d8445ad
Image: eu.gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.1
Image ID: docker://sha256:fc5e302d8309a8deb3315452b428969aed4aa931189b10a3231ce48fc0687224
Port: 10054/TCP
Args:
--v=2
--logtostderr
--probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A
--probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A
State: Running
Started: Thu, 20 Apr 2017 08:52:27 +0200
Ready: True
Restart Count: 0
Requests:
cpu: 10m
memory: 20Mi
Liveness: http-get http://:10054/metrics delay=60s timeout=5s period=10s #success=1 #failure=5
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-896k2 (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
kube-dns-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kube-dns
Optional: true
kube-dns-token-896k2:
Type: Secret (a volume populated by a Secret)
SecretName: kube-dns-token-896k2
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: CriticalAddonsOnly=:Exists
node.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s
node.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1h 2m 27 kubelet, gke-development-euro-auto-scaling-pre-d396a110-6fgw spec.containers{dnsmasq} Normal Pulling pulling image "eu.gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1"
1h 2m 22 kubelet, gke-development-euro-auto-scaling-pre-d396a110-6fgw Warning FailedSync (events with common reason combined)
1h 2m 18 kubelet, gke-development-euro-auto-scaling-pre-d396a110-6fgw spec.containers{dnsmasq} Warning Failed (events with common reason combined)
1h <invalid> 506 kubelet, gke-development-euro-auto-scaling-pre-d396a110-6fgw spec.containers{dnsmasq} Normal BackOff Back-off pulling image "eu.gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1"
1h <invalid> 503 kubelet, gke-development-euro-auto-scaling-pre-d396a110-6fgw Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "dnsmasq" with ImagePullBackOff: "Back-off pulling image \"eu.gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.1\""
Raw
Previous kube-dns logs
❯ kubectl logs kube-dns-3263495268-9ltl2 -n kube-system -c kubedns --previous:
[...]
fatal error: I0420 07:17:28.519915 1 dns.go:264] New service: feedmaterializationcoordinator
concurrent map writes
goroutine 65 [running]:
runtime.throw(0x162294a, 0x15)
/usr/local/go/src/runtime/panic.go:566 +0x95 fp=0xc4208d55c8 sp=0xc4208d55a8
runtime.mapassign1(0x1465c20, 0xc420303980, 0xc4208d5820, 0xc4208d57d8)
/usr/local/go/src/runtime/hashmap.go:458 +0x8ef fp=0xc4208d56b0 sp=0xc4208d55c8
k8s.io/dns/pkg/dns.(*KubeDNS).generateRecordsForHeadlessService(0xc42038c000, 0xc4207bba78, 0xc4208b5140, 0x0, 0x0)
/go/src/k8s.io/dns/pkg/dns/dns.go:504 +0x860 fp=0xc4208d5850 sp=0xc4208d56b0
k8s.io/dns/pkg/dns.(*KubeDNS).addDNSUsingEndpoints(0xc42038c000, 0xc4207bba78, 0xc4208d5a40, 0xc4208d5a38)
/go/src/k8s.io/dns/pkg/dns/dns.go:420 +0xc2 fp=0xc4208d5890 sp=0xc4208d5850
k8s.io/dns/pkg/dns.(*KubeDNS).handleEndpointAdd(0xc42038c000, 0x15e9060, 0xc4207bba78)
/go/src/k8s.io/dns/pkg/dns/dns.go:319 +0x52 fp=0xc4208d58c0 sp=0xc4208d5890
k8s.io/dns/pkg/dns.(*KubeDNS).handleEndpointUpdate(0xc42038c000, 0x15e9060, 0xc4207bba78, 0x15e9060, 0xc4207bba78)
/go/src/k8s.io/dns/pkg/dns/dns.go:382 +0x4f8 fp=0xc4208d5b28 sp=0xc4208d58c0
k8s.io/dns/pkg/dns.(*KubeDNS).(k8s.io/dns/pkg/dns.handleEndpointUpdate)-fm(0x15e9060, 0xc4207bba78, 0x15e9060, 0xc4207bba78)
/go/src/k8s.io/dns/pkg/dns/dns.go:246 +0x52 fp=0xc4208d5b60 sp=0xc4208d5b28
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(0xc4204bc5d0, 0xc4204bc5e0, 0xc4204bc5f0, 0x15e9060, 0xc4207bba78, 0x15e9060, 0xc4207bba78)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/controller.go:180 +0x5d fp=0xc4208d5b90 sp=0xc4208d5b60
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnUpdate(0xc420489320, 0x15e9060, 0xc4207bba78, 0x15e9060, 0xc4207bba78)
<autogenerated>:51 +0x8c fp=0xc4208d5bd8 sp=0xc4208d5b90
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.NewInformer.func1(0x148db60, 0xc42090b1c0, 0xc42090b1c0, 0x148db60)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/controller.go:246 +0x335 fp=0xc4208d5ca8 sp=0xc4208d5bd8
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc42012a420, 0xc420303ad0, 0x0, 0x0, 0x0, 0x0)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:420 +0x22a fp=0xc4208d5d80 sp=0xc4208d5ca8
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*Controller).processLoop(0xc420209110)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0x3c fp=0xc4208d5dc0 sp=0xc4208d5d80
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*Controller).(k8s.io/dns/vendor/k8s.io/client-go/tools/cache.processLoop)-fm()
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/controller.go:102 +0x2a fp=0xc4208d5dd8 sp=0xc4208d5dc0
k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait.JitterUntil.func1(0xc4208d5f70)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait/wait.go:96 +0x5e fp=0xc4208d5e10 sp=0xc4208d5dd8
k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait.JitterUntil(0xc4208d5f70, 0x3b9aca00, 0x0, 0x13e8501, 0xc42003be60)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait/wait.go:97 +0xad fp=0xc4208d5ed8 sp=0xc4208d5e10
k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait.Until(0xc4208d5f70, 0x3b9aca00, 0xc42003be60)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait/wait.go:52 +0x4d fp=0xc4208d5f10 sp=0xc4208d5ed8
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*Controller).Run(0xc420209110, 0xc42003be60)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/controller.go:102 +0x1af fp=0xc4208d5f90 sp=0xc4208d5f10
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4208d5f98 sp=0xc4208d5f90
created by k8s.io/dns/pkg/dns.(*KubeDNS).Start
/go/src/k8s.io/dns/pkg/dns/dns.go:148 +0x9f
goroutine 1 [IO wait]:
net.runtime_pollWait(0x7f576b801550, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc420952fb0, 0x72, 0xc4208d7b00, 0xc420014140)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc420952fb0, 0x1e771c0, 0xc420014140)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).accept(0xc420952f50, 0x0, 0x1e74a80, 0xc42095f160)
/usr/local/go/src/net/fd_unix.go:419 +0x238
net.(*TCPListener).accept(0xc4200ea000, 0x29e8d60800, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:132 +0x2e
net.(*TCPListener).AcceptTCP(0xc4200ea000, 0xc4208d7c28, 0xc4208d7c30, 0xc4208d7c20)
/usr/local/go/src/net/tcpsock.go:209 +0x49
net/http.tcpKeepAliveListener.Accept(0xc4200ea000, 0x16e1ff0, 0xc4204b5280, 0x1e83e80, 0xc42062cba0)
/usr/local/go/src/net/http/server.go:2608 +0x2f
net/http.(*Server).Serve(0xc4203a1900, 0x1e83680, 0xc4200ea000, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2273 +0x1ce
net/http.(*Server).ListenAndServe(0xc4203a1900, 0xc4203a1900, 0xc420993380)
/usr/local/go/src/net/http/server.go:2219 +0xb4
net/http.ListenAndServe(0xc420993380, 0x5, 0x0, 0x0, 0x1, 0xc420993380)
/usr/local/go/src/net/http/server.go:2351 +0xa0
k8s.io/dns/cmd/kube-dns/app.(*KubeDNSServer).Run(0xc4200efc20)
/go/src/k8s.io/dns/cmd/kube-dns/app/server.go:123 +0x1ee
main.main()
/go/src/k8s.io/dns/cmd/kube-dns/dns.go:52 +0x2e5
goroutine 5 [chan receive]:
k8s.io/dns/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x1ec7580)
/go/src/k8s.io/dns/vendor/github.com/golang/glog/glog.go:882 +0x7a
created by k8s.io/dns/vendor/github.com/golang/glog.init.1
/go/src/k8s.io/dns/vendor/github.com/golang/glog/glog.go:410 +0x21d
goroutine 6 [syscall, 5 minutes]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
/usr/local/go/src/os/signal/signal_unix.go:28 +0x41
goroutine 37 [IO wait]:
net.runtime_pollWait(0x7f576b801790, 0x72, 0xc4205098d8)
/usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4202f2060, 0x72, 0xc420509a18, 0xc420014140)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4202f2060, 0x1e771c0, 0xc420014140)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).readMsg(0xc4202f2000, 0xc4201e9a00, 0x200, 0x200, 0xc4207e6180, 0x28, 0x28, 0xffffffffffffffff, 0x0, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:297 +0x243
net.(*UDPConn).readMsg(0xc4200ea050, 0xc4201e9a00, 0x200, 0x200, 0xc4207e6180, 0x28, 0x28, 0x0, 0xc420509b98, 0x51111d, ...)
/usr/local/go/src/net/udpsock_posix.go:55 +0x91
net.(*UDPConn).ReadMsgUDP(0xc4200ea050, 0xc4201e9a00, 0x200, 0x200, 0xc4207e6180, 0x28, 0x28, 0x0, 0x0, 0xc420509c98, ...)
/usr/local/go/src/net/udpsock.go:116 +0xaa
k8s.io/dns/vendor/github.com/miekg/dns.ReadFromSessionUDP(0xc4200ea050, 0xc4201e9a00, 0x200, 0x200, 0x200, 0x200, 0xc4203400a0, 0x4fd904)
/go/src/k8s.io/dns/vendor/github.com/miekg/dns/udp.go:47 +0xb4
k8s.io/dns/vendor/github.com/miekg/dns.(*Server).readUDP(0xc420340000, 0xc4200ea050, 0x77359400, 0x8389f7, 0x454f10, 0xc420509d68, 0xc420509d70, 0xc420509d60, 0x8389f7)
/go/src/k8s.io/dns/vendor/github.com/miekg/dns/server.go:647 +0xf8
k8s.io/dns/vendor/github.com/miekg/dns.(*defaultReader).ReadUDP(0xc4200ea118, 0xc4200ea050, 0x77359400, 0x1e79f00, 0xc4207e6090, 0x1e6e740, 0xc42040e000, 0xc4201e9800, 0x36)
/go/src/k8s.io/dns/vendor/github.com/miekg/dns/server.go:254 +0x46
k8s.io/dns/vendor/github.com/miekg/dns.(*Server).serveUDP(0xc420340000, 0xc4200ea050, 0x0, 0x0)
/go/src/k8s.io/dns/vendor/github.com/miekg/dns/server.go:520 +0x13d
k8s.io/dns/vendor/github.com/miekg/dns.(*Server).ListenAndServe(0xc420340000, 0x0, 0x0)
/go/src/k8s.io/dns/vendor/github.com/miekg/dns/server.go:371 +0x2e8
k8s.io/dns/vendor/github.com/miekg/dns.ListenAndServe(0xc4204bc980, 0xd, 0x160c19d, 0x3, 0x1e6e740, 0xc42040e000, 0xc420467fa8, 0x1196752)
/go/src/k8s.io/dns/vendor/github.com/miekg/dns/server.go:101 +0xc0
k8s.io/dns/vendor/github.com/skynetservices/skydns/server.(*server).Run.func5(0xc420286c80, 0xc42040e000)
/go/src/k8s.io/dns/vendor/github.com/skynetservices/skydns/server/server.go:113 +0xa0
created by k8s.io/dns/vendor/github.com/skynetservices/skydns/server.(*server).Run
/go/src/k8s.io/dns/vendor/github.com/skynetservices/skydns/server/server.go:116 +0x5d3
goroutine 36 [IO wait, 5 minutes]:
net.runtime_pollWait(0x7f576b8016d0, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4202f20d0, 0x72, 0xc42050ac58, 0xc420014140)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4202f20d0, 0x1e771c0, 0xc420014140)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).accept(0xc4202f2070, 0x0, 0x1e74a80, 0xc420400160)
/usr/local/go/src/net/fd_unix.go:419 +0x238
net.(*TCPListener).accept(0xc4200ea120, 0xc42050ad40, 0xc42050ad50, 0xc42050ad30)
/usr/local/go/src/net/tcpsock_posix.go:132 +0x2e
net.(*TCPListener).Accept(0xc4200ea120, 0xc4200ea128, 0xc4200ea120, 0x6, 0xc4202f2070)
/usr/local/go/src/net/tcpsock.go:222 +0x49
k8s.io/dns/vendor/github.com/miekg/dns.(*Server).serveTCP(0xc4203400d0, 0x1e82640, 0xc4200ea120, 0x0, 0x0)
/go/src/k8s.io/dns/vendor/github.com/miekg/dns/server.go:477 +0x140
k8s.io/dns/vendor/github.com/miekg/dns.(*Server).ListenAndServe(0xc4203400d0, 0x0, 0x0)
/go/src/k8s.io/dns/vendor/github.com/miekg/dns/server.go:335 +0x4e2
k8s.io/dns/vendor/github.com/miekg/dns.ListenAndServe(0xc4204bc980, 0xd, 0x160c191, 0x3, 0x1e6e740, 0xc42040e000, 0x7804b9, 0x4326a8)
/go/src/k8s.io/dns/vendor/github.com/miekg/dns/server.go:101 +0xc0
k8s.io/dns/vendor/github.com/skynetservices/skydns/server.(*server).Run.func4(0xc420286c80, 0xc42040e000)
/go/src/k8s.io/dns/vendor/github.com/skynetservices/skydns/server/server.go:105 +0xa0
created by k8s.io/dns/vendor/github.com/skynetservices/skydns/server.(*server).Run
/go/src/k8s.io/dns/vendor/github.com/skynetservices/skydns/server/server.go:108 +0x543
goroutine 19 [select]:
k8s.io/dns/vendor/k8s.io/kubernetes/pkg/util/wait.JitterUntil(0x16dbd38, 0x12a05f200, 0x0, 0x432601, 0xc42003be60)
/go/src/k8s.io/dns/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:108 +0x16e
k8s.io/dns/vendor/k8s.io/kubernetes/pkg/util/wait.Until(0x16dbd38, 0x12a05f200, 0xc42003be60)
/go/src/k8s.io/dns/vendor/k8s.io/kubernetes/pkg/util/wait/wait.go:52 +0x4d
created by k8s.io/dns/vendor/k8s.io/kubernetes/pkg/util/logs.InitLogs
/go/src/k8s.io/dns/vendor/k8s.io/kubernetes/pkg/util/logs/logs.go:50 +0xa8
goroutine 20 [select, 5 minutes, locked to thread]:
runtime.gopark(0x16e2698, 0x0, 0x160eff8, 0x6, 0x18, 0x2)
/usr/local/go/src/runtime/proc.go:259 +0x13a
runtime.selectgoImpl(0xc42001ef30, 0x0, 0x18)
/usr/local/go/src/runtime/select.go:423 +0x11d9
runtime.selectgo(0xc42001ef30)
/usr/local/go/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
/usr/local/go/src/runtime/signal1_unix.go:304 +0x2f3
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1
goroutine 21 [chan receive, 5 minutes]:
k8s.io/dns/cmd/kube-dns/app.setupSignalHandlers.func1(0xc4200e4c60)
/go/src/k8s.io/dns/cmd/kube-dns/app/server.go:153 +0x74
created by k8s.io/dns/cmd/kube-dns/app.setupSignalHandlers
/go/src/k8s.io/dns/cmd/kube-dns/app/server.go:156 +0x148
goroutine 31 [IO wait, 5 minutes]:
net.runtime_pollWait(0x7f576b801850, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4202c2140, 0x72, 0xc4204aad00, 0xc420014140)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4202c2140, 0x1e771c0, 0xc420014140)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).accept(0xc4202c20e0, 0x0, 0x1e74a80, 0xc420430060)
/usr/local/go/src/net/fd_unix.go:419 +0x238
net.(*TCPListener).accept(0xc420026060, 0xc4203804e0, 0xc4204aae10, 0x786770)
/usr/local/go/src/net/tcpsock_posix.go:132 +0x2e
net.(*TCPListener).AcceptTCP(0xc420026060, 0xc42045e25c, 0xc4204aae00, 0x59d9e0)
/usr/local/go/src/net/tcpsock.go:209 +0x49
net/http.tcpKeepAliveListener.Accept(0xc420026060, 0xc4203803c0, 0x14475e0, 0x1eb4ea0, 0x14e6a60)
/usr/local/go/src/net/http/server.go:2608 +0x2f
net/http.(*Server).Serve(0xc42045e200, 0x1e83680, 0xc420026060, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2273 +0x1ce
net/http.(*Server).ListenAndServe(0xc42045e200, 0xc42045e200, 0xc42042c030)
/usr/local/go/src/net/http/server.go:2219 +0xb4
net/http.ListenAndServe(0xc42042c030, 0x6, 0x0, 0x0, 0x5, 0xc42042c030)
/usr/local/go/src/net/http/server.go:2351 +0xa0
k8s.io/dns/vendor/github.com/skynetservices/skydns/metrics.Metrics.func1()
/go/src/k8s.io/dns/vendor/github.com/skynetservices/skydns/metrics/metrics.go:121 +0x85
created by k8s.io/dns/vendor/github.com/skynetservices/skydns/metrics.Metrics
/go/src/k8s.io/dns/vendor/github.com/skynetservices/skydns/metrics/metrics.go:122 +0x265
goroutine 32 [semacquire, 5 minutes]:
sync.runtime_Semacquire(0xc4204bc9cc)
/usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc4204bc9c0)
/usr/local/go/src/sync/waitgroup.go:131 +0x97
k8s.io/dns/vendor/github.com/skynetservices/skydns/server.(*server).Run(0xc420286c80, 0xc4205cfca8, 0x2)
/go/src/k8s.io/dns/vendor/github.com/skynetservices/skydns/server/server.go:120 +0x276
created by k8s.io/dns/cmd/kube-dns/app.(*KubeDNSServer).startSkyDNSServer
/go/src/k8s.io/dns/cmd/kube-dns/app/server.go:202 +0x933
goroutine 66 [runnable]:
path.Clean(0xc420926a20, 0x56, 0x7, 0x160b8ae)
/usr/local/go/src/path/path.go:124 +0x65a
path.Join(0xc42053b2d0, 0x7, 0x7, 0x6, 0xc42053c480)
/usr/local/go/src/path/path.go:153 +0xbf
k8s.io/dns/vendor/github.com/skynetservices/skydns/msg.Path(0xc420940000, 0x4e, 0x0, 0x0)
/go/src/k8s.io/dns/vendor/github.com/skynetservices/skydns/msg/service.go:122 +0x265
k8s.io/dns/pkg/dns/treecache.(*treeCache).SetEntry(0xc42064d270, 0xc42064d3a0, 0x10, 0xc42053b180, 0xc420940000, 0x4e, 0x0, 0x0, 0x0)
/go/src/k8s.io/dns/pkg/dns/treecache/treecache.go:91 +0x7e
k8s.io/dns/pkg/dns.(*KubeDNS).newPortalService(0xc42038c000, 0xc4208ad260)
/go/src/k8s.io/dns/pkg/dns/dns.go:453 +0x230
k8s.io/dns/pkg/dns.(*KubeDNS).newService(0xc42038c000, 0x15eba60, 0xc4208ad260)
/go/src/k8s.io/dns/pkg/dns/dns.go:281 +0x1f1
k8s.io/dns/pkg/dns.(*KubeDNS).updateService(0xc42038c000, 0x15eba60, 0xc4208ad260, 0x15eba60, 0xc4208ad260)
/go/src/k8s.io/dns/pkg/dns/dns.go:312 +0xe5
k8s.io/dns/pkg/dns.(*KubeDNS).(k8s.io/dns/pkg/dns.updateService)-fm(0x15eba60, 0xc4208ad260, 0x15eba60, 0xc4208ad260)
/go/src/k8s.io/dns/pkg/dns/dns.go:226 +0x52
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(0xc4204bc640, 0xc4204bc660, 0xc4204bc650, 0x15eba60, 0xc4208ad260, 0x15eba60, 0xc4208ad260)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/controller.go:180 +0x5d
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnUpdate(0xc420489360, 0x15eba60, 0xc4208ad260, 0x15eba60, 0xc4208ad260)
<autogenerated>:51 +0x8c
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.NewInformer.func1(0x148db60, 0xc420ae17a0, 0xc420ae17a0, 0x148db60)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/controller.go:246 +0x335
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc42012a4d0, 0xc42031c2a0, 0x0, 0x0, 0x0, 0x0)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:420 +0x22a
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*Controller).processLoop(0xc420209180)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0x3c
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*Controller).(k8s.io/dns/vendor/k8s.io/client-go/tools/cache.processLoop)-fm()
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/controller.go:102 +0x2a
k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait.JitterUntil.func1(0xc4205bbf70)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait/wait.go:96 +0x5e
k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait.JitterUntil(0xc4205bbf70, 0x3b9aca00, 0x0, 0x13e8501, 0xc42003be60)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait/wait.go:97 +0xad
k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait.Until(0xc4205bbf70, 0x3b9aca00, 0xc42003be60)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait/wait.go:52 +0x4d
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*Controller).Run(0xc420209180, 0xc42003be60)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/controller.go:102 +0x1af
created by k8s.io/dns/pkg/dns.(*KubeDNS).Start
/go/src/k8s.io/dns/pkg/dns/dns.go:151 +0x11d
goroutine 67 [chan receive]:
k8s.io/dns/pkg/dns/config.(*kubeSync).Periodic.func1(0xc420303830)
/go/src/k8s.io/dns/pkg/dns/config/sync.go:85 +0x7e
created by k8s.io/dns/pkg/dns/config.(*kubeSync).Periodic
/go/src/k8s.io/dns/pkg/dns/config/sync.go:95 +0x3f
goroutine 68 [chan receive, 5 minutes]:
k8s.io/dns/pkg/dns.(*KubeDNS).syncConfigMap(0xc42038c000, 0xc4200e4c00)
/go/src/k8s.io/dns/pkg/dns/dns.go:194 +0x56
created by k8s.io/dns/pkg/dns.(*KubeDNS).startConfigMapSync
/go/src/k8s.io/dns/pkg/dns/dns.go:189 +0xbf
goroutine 69 [select]:
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc42008cc80, 0x1e79780, 0xc420652030, 0xc420065c40, 0xc420734240, 0xc42003be60, 0x0, 0x0)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:346 +0x11cb
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc42008cc80, 0xc42003be60, 0x0, 0x0)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:317 +0xacf
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*Reflector).RunUntil.func1()
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:198 +0x33
k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait.JitterUntil.func1(0xc420348e40)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait/wait.go:96 +0x5e
k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait.JitterUntil(0xc420348e40, 0x3b9aca00, 0x0, 0x1, 0xc42003be60)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait/wait.go:97 +0xad
k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait.Until(0xc420348e40, 0x3b9aca00, 0xc42003be60)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait/wait.go:52 +0x4d
created by k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*Reflector).RunUntil
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:201 +0x1c4
goroutine 52 [select, 4 minutes]:
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*Reflector).watchHandler(0xc42045e380, 0x1e79780, 0xc42073c0f0, 0xc42047fc40, 0xc4200e4de0, 0xc42003be60, 0x0, 0x0)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:346 +0x11cb
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc42045e380, 0xc42003be60, 0x0, 0x0)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:317 +0xacf
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*Reflector).RunUntil.func1()
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:198 +0x33
k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait.JitterUntil.func1(0xc420430080)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait/wait.go:96 +0x5e
k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait.JitterUntil(0xc420430080, 0x3b9aca00, 0x0, 0x1, 0xc42003be60)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait/wait.go:97 +0xad
k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait.Until(0xc420430080, 0x3b9aca00, 0xc42003be60)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/util/wait/wait.go:52 +0x4d
created by k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*Reflector).RunUntil
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:201 +0x1c4
goroutine 82 [select]:
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc42048c1f8, 0xc42003be60, 0xc4207342a0, 0xc42008cc80, 0xc420734240, 0xc42048c200)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:266 +0x2eb
created by k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:281 +0x520
goroutine 43 [semacquire, 4 minutes]:
sync.runtime_notifyListWait(0xc4200936c0, 0x1)
/usr/local/go/src/runtime/sema.go:267 +0x122
sync.(*Cond).Wait(0xc4200936b0)
/usr/local/go/src/sync/cond.go:57 +0x80
k8s.io/dns/vendor/golang.org/x/net/http2.(*pipe).Read(0xc4200936a8, 0xc4209472e0, 0x4, 0x4, 0x0, 0x0, 0x0)
/go/src/k8s.io/dns/vendor/golang.org/x/net/http2/pipe.go:60 +0xd9
k8s.io/dns/vendor/golang.org/x/net/http2.transportResponseBody.Read(0xc420093680, 0xc4209472e0, 0x4, 0x4, 0x0, 0x0, 0x0)
/go/src/k8s.io/dns/vendor/golang.org/x/net/http2/transport.go:1589 +0xd9
io.ReadAtLeast(0x7f576b855b38, 0xc420093680, 0xc4209472e0, 0x4, 0x4, 0x4, 0x7, 0xc420abd560, 0x0)
/usr/local/go/src/io/io.go:307 +0xa4
k8s.io/dns/vendor/k8s.io/client-go/pkg/util/framer.(*lengthDelimitedFrameReader).Read(0xc4207f2000, 0xc4202a0a00, 0x800, 0xa00, 0x40, 0x38, 0x14cb6c0)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/util/framer/framer.go:76 +0x2ce
k8s.io/dns/vendor/k8s.io/client-go/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc4207da050, 0x0, 0x1e728c0, 0xc420679980, 0x452ab0, 0xc4207f6000, 0xc42028ab40, 0x0, 0xc4208dbed8)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/runtime/serializer/streaming/streaming.go:77 +0x85
k8s.io/dns/vendor/k8s.io/client-go/pkg/watch/versioned.(*Decoder).Decode(0xc4207f2020, 0xc4207f6000, 0xc4208dbf68, 0x1e70700, 0xc42028ab40, 0x0, 0x0)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/watch/versioned/decoder.go:48 +0xa1
k8s.io/dns/vendor/k8s.io/client-go/pkg/watch.(*StreamWatcher).receive(0xc42073c0f0)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/watch/streamwatcher.go:93 +0xf4
created by k8s.io/dns/vendor/k8s.io/client-go/pkg/watch.NewStreamWatcher
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/watch/streamwatcher.go:60 +0xd5
goroutine 38 [chan receive]:
k8s.io/dns/pkg/dns/config.(*kubeFileSyncSource).Periodic.func1(0xc420303800)
/go/src/k8s.io/dns/pkg/dns/config/sync_dir.go:72 +0x188
created by k8s.io/dns/pkg/dns/config.(*kubeFileSyncSource).Periodic
/go/src/k8s.io/dns/pkg/dns/config/sync_dir.go:74 +0x3f
goroutine 41 [IO wait]:
net.runtime_pollWait(0x7f576b801610, 0x72, 0xb)
/usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4202c2370, 0x72, 0xc4205fa770, 0xc420014140)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4202c2370, 0x1e771c0, 0xc420014140)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc4202c2310, 0xc4201ac000, 0x8000, 0x8000, 0x0, 0x1e771c0, 0xc420014140)
/usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc4200ea138, 0xc4201ac000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:173 +0x70
crypto/tls.(*block).readFromUntil(0xc420428480, 0x7f576b774db8, 0xc4200ea138, 0x5, 0xc4200ea138, 0x0)
/usr/local/go/src/crypto/tls/conn.go:476 +0x91
crypto/tls.(*Conn).readRecord(0xc4201f8000, 0x16e2717, 0xc4201f8108, 0x0)
/usr/local/go/src/crypto/tls/conn.go:578 +0xc4
crypto/tls.(*Conn).Read(0xc4201f8000, 0xc4205fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1113 +0x116
bufio.(*Reader).fill(0xc4202a89c0)
/usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Read(0xc4202a89c0, 0xc420340378, 0x9, 0x9, 0x0, 0xc420274000, 0x0)
/usr/local/go/src/bufio/bufio.go:209 +0x1bc
io.ReadAtLeast(0x1e6dbc0, 0xc4202a89c0, 0xc420340378, 0x9, 0x9, 0x9, 0x4, 0xc4205fac90, 0x43d9b0)
/usr/local/go/src/io/io.go:307 +0xa4
io.ReadFull(0x1e6dbc0, 0xc4202a89c0, 0xc420340378, 0x9, 0x9, 0xc42048e440, 0xc4205facb8, 0x4546a0)
/usr/local/go/src/io/io.go:325 +0x58
k8s.io/dns/vendor/golang.org/x/net/http2.readFrameHeader(0xc420340378, 0x9, 0x9, 0x1e6dbc0, 0xc4202a89c0, 0x0, 0xc400000000, 0x117d209, 0xc42015a028)
/go/src/k8s.io/dns/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
k8s.io/dns/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420340340, 0xc4206109f0, 0x0, 0x0, 0x0)
/go/src/k8s.io/dns/vendor/golang.org/x/net/http2/frame.go:469 +0x8e
k8s.io/dns/vendor/golang.org/x/net/http2.(*clientConnReadLoop).run(0xc4205faf80, 0x16ddc20, 0xc420463790)
/go/src/k8s.io/dns/vendor/golang.org/x/net/http2/transport.go:1343 +0xbb
k8s.io/dns/vendor/golang.org/x/net/http2.(*ClientConn).readLoop(0xc420058820)
/go/src/k8s.io/dns/vendor/golang.org/x/net/http2/transport.go:1269 +0xa6
created by k8s.io/dns/vendor/golang.org/x/net/http2.(*Transport).newClientConn
/go/src/k8s.io/dns/vendor/golang.org/x/net/http2/transport.go:534 +0x791
goroutine 55 [select]:
k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch.func1(0xc420026098, 0xc42003be60, 0xc4200e4f00, 0xc42045e380, 0xc4200e4de0, 0xc420026108)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:266 +0x2eb
created by k8s.io/dns/vendor/k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch
/go/src/k8s.io/dns/vendor/k8s.io/client-go/tools/cache/reflector.go:281 +0x520
goroutine 57 [semacquire]:
sync.runtime_notifyListWait(0xc42015a040, 0x238)
/usr/local/go/src/runtime/sema.go:267 +0x122
sync.(*Cond).Wait(0xc42015a030)
/usr/local/go/src/sync/cond.go:57 +0x80
k8s.io/dns/vendor/golang.org/x/net/http2.(*pipe).Read(0xc42015a028, 0xc420a8cf28, 0x4, 0x4, 0x0, 0x0, 0x0)
/go/src/k8s.io/dns/vendor/golang.org/x/net/http2/pipe.go:60 +0xd9
k8s.io/dns/vendor/golang.org/x/net/http2.transportResponseBody.Read(0xc42015a000, 0xc420a8cf28, 0x4, 0x4, 0x0, 0x0, 0x0)
/go/src/k8s.io/dns/vendor/golang.org/x/net/http2/transport.go:1589 +0xd9
io.ReadAtLeast(0x7f576b855b38, 0xc42015a000, 0xc420a8cf28, 0x4, 0x4, 0x4, 0x7, 0xc420610a50, 0x0)
/usr/local/go/src/io/io.go:307 +0xa4
k8s.io/dns/vendor/k8s.io/client-go/pkg/util/framer.(*lengthDelimitedFrameReader).Read(0xc42072ff60, 0xc420336a00, 0x800, 0xa00, 0x40, 0x38, 0x14cb6c0)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/util/framer/framer.go:76 +0x2ce
k8s.io/dns/vendor/k8s.io/client-go/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc420648ff0, 0x0, 0x1e728c0, 0xc420a45340, 0x452ab0, 0xc420631500, 0xc4205b10e0, 0x0, 0xc420481ed8)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/runtime/serializer/streaming/streaming.go:77 +0x85
k8s.io/dns/vendor/k8s.io/client-go/pkg/watch/versioned.(*Decoder).Decode(0xc42072ff80, 0xc420631500, 0xc420481f68, 0x1e6fe80, 0xc4205b10e0, 0x0, 0x0)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/watch/versioned/decoder.go:48 +0xa1
k8s.io/dns/vendor/k8s.io/client-go/pkg/watch.(*StreamWatcher).receive(0xc420652030)
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/watch/streamwatcher.go:93 +0xf4
created by k8s.io/dns/vendor/k8s.io/client-go/pkg/watch.NewStreamWatcher
/go/src/k8s.io/dns/vendor/k8s.io/client-go/pkg/watch/streamwatcher.go:60 +0xd5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment