Skip to content

Instantly share code, notes, and snippets.

@krsna1729
Last active March 14, 2019 17:12
Show Gist options
  • Save krsna1729/fb1ef568051e578cd00e4a6dd2581855 to your computer and use it in GitHub Desktop.
Save krsna1729/fb1ef568051e578cd00e4a6dd2581855 to your computer and use it in GitHub Desktop.
Debugging crio

bring up v1.2 of cloud-native-setup. setup_system is done as part of vagrant all 3 machines have crio running at this point

  • one error in journalctl
dmeventd libdevmapper-event-lvm2thin.so dlopen failed: libdevmapper-event-lvm2thin.so: cannot open shared object file: No such file or directory.

kubeadm init caused this on master

[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: time="2019-03-14T16:27:07Z" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = pinging docker registry returned: Get https://k8s.gcr.io/v2/: net/http: TLS handshake timeout", error: exit status 1

rerunning it went ahead

On the workers, the moment we do kubeadm join crio panics with below snippet

```
Mar 14 16:30:36 clr-02 crio[19232]: panic: runtime error: invalid memory address or nil pointer dereference
Mar 14 16:30:36 clr-02 crio[19232]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xe58e1e]
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 957 [running]:
Mar 14 16:30:36 clr-02 crio[19232]: panic(0x1774b00, 0x2b97230)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/panic.go:565 +0x2c5 fp=0xc000cc7858 sp=0xc000cc77c8 pc=0x435fe5
Mar 14 16:30:36 clr-02 crio[19232]: runtime.panicmem(...)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/panic.go:82
Mar 14 16:30:36 clr-02 crio[19232]: runtime.sigpanic()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/signal_unix.go:390 +0x411 fp=0xc000cc7888 sp=0xc000cc7858 pc=0x44bbf1
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/utils.GetDiskUsageStats.func1(0xc000d6fc00, 0x72, 0x0, 0x0, 0x1c1b680, 0xc000a37a70, 0x20, 0x18056a0)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/utils/filesystem.go:16 +0x6e fp=0xc000cc78e8 sp=0xc000cc7888 pc=0xe5
8e1e
Mar 14 16:30:36 clr-02 crio[19232]: path/filepath.Walk(0xc000d6fc00, 0x72, 0xc0009abe40, 0x72, 0x1)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/path/filepath/path.go:407 +0x6a fp=0xc000cc7948 sp=0xc000cc78e8 pc=0x518eaa
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/utils.GetDiskUsageStats(0xc000d6fc00, 0x72, 0x2, 0xc000d6fc00, 0x72, 0x1e)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/utils/filesystem.go:13 +0xb0 fp=0xc000cc7990 sp=0xc000cc7948 pc=0xe5
71b0
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/server.buildContainerStats(0xc000566cb0, 0xc00154c000, 0xc0007d8b50)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/server/container_stats.go:19 +0xd8 fp=0xc000cc7a88 sp=0xc000cc7990 p
c=0x15587d8
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/server.(*Server).ListContainerStats(0xc0000cd680, 0x1c60c20, 0xc000a36cf0, 0xc0000be678, 0x0, 0x0, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/server/container_stats_list.go:42 +0x288 fp=0xc000cc7bc8 sp=0xc000cc
7a88 pc=0x15591e8
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2._RuntimeService_ListContainerStats_Handler(0x197b740, 0xc000
0cd680, 0x1c60c20, 0xc000a36cf0, 0xc00002acd0, 0x0, 0x0, 0x0, 0x0, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2/api.p
b.go:4488 +0x23e fp=0xc000cc7c38 sp=0xc000cc7bc8 pc=0xa08dde
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc0003d8780, 0x1c793a0, 0xc0005a2840, 0xc0005b5600, 0xc00023fb30,
0x2baaba8, 0x0, 0x0, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc/server.go:843 +0x94c fp=0xc000cc7e48 s
p=0xc000cc7c38 pc=0x9f32fc
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc.(*Server).handleStream(0xc0003d8780, 0x1c793a0, 0xc0005a2840, 0xc0005b5600, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc/server.go:1040 +0x12b5 fp=0xc000cc7f80
sp=0xc000cc7e48 pc=0x9f6325
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00037c700, 0xc0003d8780, 0x1c793a0, 0xc0005a2840, 0xc0005b5
600)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc/server.go:589 +0x9f fp=0xc000cc7fb8 sp
=0xc000cc7f80 pc=0x9f7c0f
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000cc7fc0 sp=0xc000cc7fb8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc/server.go:587 +0xa1
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 1 [select, 14 minutes]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41c88, 0x0, 0x1809, 0x1)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc000194768 sp=0xc000194748 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.selectgo(0xc000194b68, 0xc000194954, 0x3, 0x1c673a0, 0xc0000e5860)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/select.go:313 +0xcb8 fp=0xc000194890 sp=0xc000194768 pc=0x447d48
Mar 14 16:30:36 clr-02 crio[19232]: main.main.func2(0xc00056c420, 0xc00056c420, 0xc00055ae3f)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/cmd/crio/main.go:740 +0xc8a fp=0xc000194e40 sp=0xc000194890 pc=0x15b
194a
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/github.com/urfave/cli.HandleAction(0x1703440, 0xc000576650, 0xc00056c420, 0xc00057ad20, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/github.com/urfave/cli/app.go:490 +0xc8 fp=0xc000194e68 sp=0xc
000194e40 pc=0x1583768
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/github.com/urfave/cli.(*App).Run(0xc0005b61a0, 0xc0000b0170, 0x1, 0x1, 0x0, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/github.com/urfave/cli/app.go:264 +0x57c fp=0xc000195158 sp=0x
c000194e68 pc=0x158198c
Mar 14 16:30:36 clr-02 crio[19232]: main.main()
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/cmd/crio/main.go:765 +0x224e fp=0xc000195f98 sp=0xc000195158 pc=0x15
af7ae
Mar 14 16:30:36 clr-02 crio[19232]: runtime.main()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:200 +0x20c fp=0xc000195fe0 sp=0xc000195f98 pc=0x437cec
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000195fe8 sp=0xc000195fe0 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 2 [force gc (idle), 3 minutes]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41c48, 0x2d4c770, 0x1410, 0x1)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc000068fb0 sp=0xc000068f90 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goparkunlock(...)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:307
Mar 14 16:30:36 clr-02 crio[19232]: runtime.forcegchelper()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:250 +0xb7 fp=0xc000068fe0 sp=0xc000068fb0 pc=0x437f87
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000068fe8 sp=0xc000068fe0 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by runtime.init.5
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:239 +0x35
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 3 [GC sweep wait]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41c48, 0x2d4c9e0, 0x140c, 0x1)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc0000697a8 sp=0xc000069788 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goparkunlock(...)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:307
Mar 14 16:30:36 clr-02 crio[19232]: runtime.bgsweep(0xc00008e000)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgcsweep.go:89 +0x134 fp=0xc0000697d8 sp=0xc0000697a8 pc=0x42a324
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000697e0 sp=0xc0000697d8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by runtime.gcenable
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:208 +0x58
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 18 [finalizer wait]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41c48, 0x2d6b3e8, 0xc0001d140f, 0x1)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc000068758 sp=0xc000068738 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goparkunlock(...)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:307
Mar 14 16:30:36 clr-02 crio[19232]: runtime.runfinq()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mfinal.go:175 +0xa6 fp=0xc0000687e0 sp=0xc000068758 pc=0x421046
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000687e8 sp=0xc0000687e0 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by runtime.createfing
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mfinal.go:156 +0x61
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 19 [syscall]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.notetsleepg(0x2d52720, 0x1bf08cc629, 0x1414)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/lock_futex.go:227 +0x34 fp=0xc000064760 sp=0xc000064730 pc=0x413924
Mar 14 16:30:36 clr-02 crio[19232]: runtime.timerproc(0x2d52700)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/time.go:311 +0x2ea fp=0xc0000647d8 sp=0xc000064760 pc=0x45509a
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000647e0 sp=0xc0000647d8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by runtime.(*timersBucket).addtimerLocked
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/time.go:169 +0x10e
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 20 [syscall, 14 minutes]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.notetsleepg(0x2d6bbe0, 0xffffffffffffffff, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/lock_futex.go:227 +0x34 fp=0xc000064f98 sp=0xc000064f68 pc=0x413924
Mar 14 16:30:36 clr-02 crio[19232]: os/signal.signal_recv(0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/sigqueue.go:139 +0x9c fp=0xc000064fc0 sp=0xc000064f98 pc=0x44c9dc
Mar 14 16:30:36 clr-02 crio[19232]: os/signal.loop()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x22 fp=0xc000064fe0 sp=0xc000064fc0 pc=0x8404c2
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000064fe8 sp=0xc000064fe0 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by os/signal.init.0
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/os/signal/signal_unix.go:29 +0x41
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 21 [chan receive]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41c48, 0xc00003e118, 0x170d, 0x3)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc0000656d0 sp=0xc0000656b0 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goparkunlock(...)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:307
Mar 14 16:30:36 clr-02 crio[19232]: runtime.chanrecv(0xc00003e0c0, 0xc0000657b0, 0x2d4db01, 0x2d4db98)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/chan.go:524 +0x2d0 fp=0xc000065760 sp=0xc0000656d0 pc=0x40e9f0
Mar 14 16:30:36 clr-02 crio[19232]: runtime.chanrecv2(0xc00003e0c0, 0xc0000657b0, 0x1)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/chan.go:411 +0x2b fp=0xc000065790 sp=0xc000065760 pc=0x40e70b
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x2d4db80)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/k8s.io/klog/klog.go:941 +0x8b fp=0xc0000657d8 sp=0xc000065790
pc=0xac704b
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000657e0 sp=0xc0000657d8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by github.com/kubernetes-sigs/cri-o/vendor/k8s.io/klog.init.0
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/k8s.io/klog/klog.go:403 +0x6c
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 4 [syscall]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.notetsleepg(0x2d526a0, 0x24f561a59, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/lock_futex.go:227 +0x34 fp=0xc000069f60 sp=0xc000069f30 pc=0x413924
Mar 14 16:30:36 clr-02 crio[19232]: runtime.timerproc(0x2d52680)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/time.go:311 +0x2ea fp=0xc000069fd8 sp=0xc000069f60 pc=0x45509a
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000069fe0 sp=0xc000069fd8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by runtime.(*timersBucket).addtimerLocked
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/time.go:169 +0x10e
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 22 [GC worker (idle)]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41ad8, 0xc0004be000, 0x1417, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc000065f60 sp=0xc000065f40 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gcBgMarkWorker(0xc000048000)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:1836 +0xff fp=0xc000065fd8 sp=0xc000065f60 pc=0x424a8f
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000065fe0 sp=0xc000065fd8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by runtime.gcBgMarkStartWorkers
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:1784 +0x77
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 34 [GC worker (idle)]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41ad8, 0xc0004be010, 0x1417, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc0004c4760 sp=0xc0004c4740 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gcBgMarkWorker(0xc00004a500)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:1836 +0xff fp=0xc0004c47d8 sp=0xc0004c4760 pc=0x424a8f
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0004c47e0 sp=0xc0004c47d8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by runtime.gcBgMarkStartWorkers
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:1784 +0x77
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 35 [GC worker (idle)]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41ad8, 0xc0004be020, 0x1417, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc0004c4f60 sp=0xc0004c4f40 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gcBgMarkWorker(0xc00004ca00)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:1836 +0xff fp=0xc0004c4fd8 sp=0xc0004c4f60 pc=0x424a8f
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0004c4fe0 sp=0xc0004c4fd8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by runtime.gcBgMarkStartWorkers
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:1784 +0x77
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 36 [GC worker (idle)]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41ad8, 0xc0004be030, 0x1417, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc0004c5760 sp=0xc0004c5740 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gcBgMarkWorker(0xc00004ef00)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:1836 +0xff fp=0xc0004c57d8 sp=0xc0004c5760 pc=0x424a8f
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0004c57e0 sp=0xc0004c57d8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by runtime.gcBgMarkStartWorkers
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:1784 +0x77
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 23 [GC worker (idle)]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41ad8, 0xc0004be030, 0x1417, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc0004c5760 sp=0xc0004c5740 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gcBgMarkWorker(0xc00004ef00)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:1836 +0xff fp=0xc0004c57d8 sp=0xc0004c5760 pc=0x424a8f
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0004c57e0 sp=0xc0004c57d8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by runtime.gcBgMarkStartWorkers
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:1784 +0x77
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 23 [GC worker (idle)]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41ad8, 0xc0004be040, 0x1417, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc000066760 sp=0xc000066740 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gcBgMarkWorker(0xc000051400)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:1836 +0xff fp=0xc0000667d8 sp=0xc000066760 pc=0x424a8f
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000667e0 sp=0xc0000667d8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by runtime.gcBgMarkStartWorkers
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:1784 +0x77
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 37 [GC worker (idle)]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41ad8, 0xc0004be050, 0x1417, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc0004c5f60 sp=0xc0004c5f40 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gcBgMarkWorker(0xc000053900)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:1836 +0xff fp=0xc0004c5fd8 sp=0xc0004c5f60 pc=0x424a8f
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0004c5fe0 sp=0xc0004c5fd8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by runtime.gcBgMarkStartWorkers
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:1784 +0x77
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 38 [GC worker (idle)]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41ad8, 0xc0004be060, 0x1417, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc0004c6760 sp=0xc0004c6740 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gcBgMarkWorker(0xc000056000)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:1836 +0xff fp=0xc0004c67d8 sp=0xc0004c6760 pc=0x424a8f
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0004c67e0 sp=0xc0004c67d8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by runtime.gcBgMarkStartWorkers
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:1784 +0x77
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 39 [GC worker (idle)]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41ad8, 0xc00037c100, 0x1417, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc0004c6f60 sp=0xc0004c6f40 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gcBgMarkWorker(0xc000058500)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:1836 +0xff fp=0xc0004c6fd8 sp=0xc0004c6f60 pc=0x424a8f
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0004c6fe0 sp=0xc0004c6fd8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by runtime.gcBgMarkStartWorkers
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/mgc.go:1784 +0x77
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 25 [select, 14 minutes]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41c88, 0x0, 0x1809, 0x1)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc0004c7d90 sp=0xc0004c7d70 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.selectgo(0xc0004c7f68, 0xc0004c7ef0, 0x2, 0xc00003e1e0, 0xc0004d4300)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/select.go:313 +0xcb8 fp=0xc0004c7eb8 sp=0xc0004c7d90 pc=0x447d48
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/github.com/cri-o/ocicni/pkg/ocicni.(*cniNetworkPlugin).monitorNetDir.func1(0xc0002e0190, 0xc00008e880, 0xc0004fa000)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go:128 +0xe9 fp=0xc
0004c7fc8 sp=0xc0004c7eb8 pc=0x14f8879
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit() [9769/10197]Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0004c7fd0 sp=0xc0004c7fc8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by github.com/kubernetes-sigs/cri-o/vendor/github.com/cri-o/ocicni/pkg/ocicni.(*cniNetworkPlugin).monitorNetDir
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go:126 +0x2e8
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 14 [chan receive]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41c48, 0xc00057a058, 0xc00006170d, 0x3)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc00006aeb8 sp=0xc00006ae98 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goparkunlock(...)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:307
Mar 14 16:30:36 clr-02 crio[19232]: runtime.chanrecv(0xc00057a000, 0xc00006afb0, 0xc00006af01, 0x400101)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/chan.go:524 +0x2d0 fp=0xc00006af48 sp=0xc00006aeb8 pc=0x40e9f0
Mar 14 16:30:36 clr-02 crio[19232]: runtime.chanrecv2(0xc00057a000, 0xc00006afb0, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/chan.go:411 +0x2b fp=0xc00006af78 sp=0xc00006af48 pc=0x40e70b
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/github.com/containers/storage/drivers/devmapper.(*DeviceSet).startDeviceDeletionWorker(0xc0003803c0)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/github.com/containers/storage/drivers/devmapper/deviceset.go:
711 +0xe7 fp=0xc00006afd8 sp=0xc00006af78 pc=0x7b7237
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00006afe0 sp=0xc00006afd8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by github.com/kubernetes-sigs/cri-o/vendor/github.com/containers/storage/drivers/devmapper.(*DeviceSet).initMetaData
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/github.com/containers/storage/drivers/devmapper/deviceset.go:
743 +0x233
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 82 [chan receive, 14 minutes]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41c48, 0xc0000a6dd8, 0xc00007170d, 0x3)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc000078e80 sp=0xc000078e60 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goparkunlock(...)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:307
Mar 14 16:30:36 clr-02 crio[19232]: runtime.chanrecv(0xc0000a6d80, 0x0, 0xc0004d4401, 0x14f5448)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/chan.go:524 +0x2d0 fp=0xc000078f10 sp=0xc000078e80 pc=0x40e9f0
Mar 14 16:30:36 clr-02 crio[19232]: runtime.chanrecv1(0xc0000a6d80, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/chan.go:406 +0x2b fp=0xc000078f40 sp=0xc000078f10 pc=0x40e6cb
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/github.com/cri-o/ocicni/pkg/ocicni.(*cniNetworkPlugin).monitorNetDir(0xc00008e880)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go:154 +0x306 fp=0x
c000078fd8 sp=0xc000078f40 pc=0x14f5466
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000078fe0 sp=0xc000078fd8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by github.com/kubernetes-sigs/cri-o/vendor/github.com/cri-o/ocicni/pkg/ocicni.InitCNI
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go:184 +0x1d9
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 24 [syscall, 14 minutes]:
Mar 14 16:30:36 clr-02 crio[19232]: syscall.Syscall6(0xe8, 0x11, 0xc00054bd14, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0xc00054bcb8, 0x41c635, 0xc000235568)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/syscall/asm_linux_amd64.s:44 +0x5 fp=0xc00054bc48 sp=0xc00054bc40 pc=0x47ea65
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/golang.org/x/sys/unix.EpollWait(0x11, 0xc00054bd14, 0x7, 0x7, 0xffffffffffffffff, 0xc00054bd10, 0x47, 0xc0004d4600)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:1518 +0x72 fp=0
xc00054bcb8 sp=0xc00054bc48 pc=0x5ab4e2
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/github.com/fsnotify/fsnotify.(*fdPoller).wait(0xc00012a280, 0x1a3f600, 0xc00012a280, 0x99)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/github.com/fsnotify/fsnotify/inotify_poller.go:86 +0x91 fp=0x
c00054bd80 sp=0xc00054bcb8 pc=0xf798c1
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc0002e0190)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/github.com/fsnotify/fsnotify/inotify.go:192 +0x18a fp=0xc0005
5bfd8 sp=0xc00054bd80 pc=0xf78a6a
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00055bfe0 sp=0xc00055bfd8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by github.com/kubernetes-sigs/cri-o/vendor/github.com/fsnotify/fsnotify.NewWatcher
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/github.com/fsnotify/fsnotify/inotify.go:61 +0x1e2
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 32 [runnable]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41c88, 0x0, 0x1809, 0x1)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc0007d2d20 sp=0xc0007d2d00 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.selectgo(0xc0007d2ec0, 0xc0007d2e84, 0x2, 0x0, 0xc000376700)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/select.go:313 +0xcb8 fp=0xc0007d2e48 sp=0xc0007d2d20 pc=0x447d48
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc/transport.loopyWriter(0x1c60b60, 0xc0000c0e00, 0xc000150600, 0xc0007d2fb8)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc/transport/transport.go:750 +0x27a fp=0
xc0007d2f98 sp=0xc0007d2e48 pc=0x9e688a
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc/transport.newHTTP2Server.func1(0xc0005a2840)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc/transport/http2_server.go:252 +0x66 fp
=0xc0007d2fd8 sp=0xc0007d2f98 pc=0x9e8106
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0007d2fe0 sp=0xc0007d2fd8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc/transport.newHTTP2Server
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc/transport/http2_server.go:251 +0xdfd
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 115 [IO wait]:
Mar 14 16:30:36 clr-02 crio[19232]: runtime.gopark(0x1a41c18, 0x7f08526a53c8, 0xc0004d1b02, 0x5)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/proc.go:301 +0xef fp=0xc0007d3a20 sp=0xc0007d3a00 pc=0x4380df
Mar 14 16:30:36 clr-02 crio[19232]: runtime.netpollblock(0x7f08526a53a0, 0x72, 0xc000258000)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/netpoll.go:389 +0x9a fp=0xc0007d3a58 sp=0xc0007d3a20 pc=0x43381a
Mar 14 16:30:36 clr-02 crio[19232]: internal/poll.runtime_pollWait(0x7f08526a53a0, 0x72, 0xffffffffffffffff)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/netpoll.go:182 +0x56 fp=0xc0007d3a80 sp=0xc0007d3a58 pc=0x432e16
Mar 14 16:30:36 clr-02 crio[19232]: internal/poll.(*pollDesc).wait(0xc000376618, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x9b fp=0xc0007d3ab0 sp=0xc0007d3a80 pc=0x4932fb
Mar 14 16:30:36 clr-02 crio[19232]: internal/poll.(*pollDesc).waitRead(...)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92
Mar 14 16:30:36 clr-02 crio[19232]: internal/poll.(*FD).Read(0xc000376600, 0xc000258000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/internal/poll/fd_unix.go:169 +0x19b fp=0xc0007d3b08 sp=0xc0007d3ab0 pc=0x49410b
Mar 14 16:30:36 clr-02 crio[19232]: net.(*netFD).Read(0xc000376600, 0xc000258000, 0x8000, 0x8000, 0x1e, 0x0, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/net/fd_unix.go:202 +0x4f fp=0xc0007d3b68 sp=0xc0007d3b08 pc=0x6658ef
Mar 14 16:30:36 clr-02 crio[19232]: net.(*conn).Read(0xc00057e0b8, 0xc000258000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/net/net.go:177 +0x69 fp=0xc0007d3bc8 sp=0xc0007d3b68 pc=0x67a409
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/github.com/soheilhy/cmux.(*bufferedReader).Read(0xc000202320, 0xc000258000, 0x8000, 0x8000, 0x43819c, 0xc0007d3c60,
0x462310)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/github.com/soheilhy/cmux/buffer.go:53 +0x134 fp=0xc0007d3c30
sp=0xc0007d3bc8 pc=0x15a6714
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/github.com/soheilhy/cmux.(*MuxConn).Read(0xc000202310, 0xc000258000, 0x8000, 0x8000, 0xc0007d3db8, 0xc00002c480, 0xc
0007d3d28)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/github.com/soheilhy/cmux/cmux.go:260 +0x4f fp=0xc0007d3c78 sp
=0xc0007d3c30 pc=0x15a7a1f
Mar 14 16:30:36 clr-02 crio[19232]: bufio.(*Reader).Read(0xc0002b0840, 0xc00015a2d8, 0x9, 0x9, 0xc0004d2700, 0x7f085690a460, 0x414a79)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/bufio/bufio.go:223 +0x22f fp=0xc0007d3cd0 sp=0xc0007d3c78 pc=0x52febf
Mar 14 16:30:36 clr-02 crio[19232]: io.ReadAtLeast(0x1c18d20, 0xc0002b0840, 0xc00015a2d8, 0x9, 0x9, 0x9, 0xc00004ef00, 0x9e4efe, 0xc000150ea8)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/io/io.go:310 +0x88 fp=0xc0007d3d30 sp=0xc0007d3cd0 pc=0x46fed8
Mar 14 16:30:36 clr-02 crio[19232]: io.ReadFull(...)
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/io/io.go:329
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/golang.org/x/net/http2.readFrameHeader(0xc00015a2d8, 0x9, 0x9, 0x1c18d20, 0xc0002b0840, 0x0, 0xc000000000, 0x9dd49a,
0xbf1abe4b30c4f501)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/golang.org/x/net/http2/frame.go:237 +0x88 fp=0xc0007d3d98 sp=
0xc0007d3d30 pc=0x9a45d8
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00015a2a0, 0xc000a95780, 0xc000a95780, 0x0, 0x0)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 fp=0xc0007d3e58 sp=
0xc0007d3d98 pc=0x9a4e31
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc0005a2f20, 0xc000150f30, 0x1a409f8)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc/transport/http2_server.go:383 +0x37 fp
=0xc0007d3ec8 sp=0xc0007d3e58 pc=0x9db867
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc0003d8780, 0x1c793a0, 0xc0005a2f20)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc/server.go:585 +0x12b fp=0xc0007d3f08 s
p=0xc0007d3ec8 pc=0x9f1d0b
Mar 14 16:30:36 clr-02 crio[19232]: github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc.(*Server).handleRawConn(0xc0003d8780, 0x1c7fca0, 0xc000202310)
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc/server.go:546 +0x583 fp=0xc0007d3fc8 s
p=0xc0007d3f08 pc=0x9f1683
Mar 14 16:30:36 clr-02 crio[19232]: runtime.goexit()
Mar 14 16:30:36 clr-02 crio[19232]: /usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0007d3fd0 sp=0xc0007d3fc8 pc=0x4658c1
Mar 14 16:30:36 clr-02 crio[19232]: created by github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc.(*Server).Serve
Mar 14 16:30:36 clr-02 crio[19232]: /builddir/build/BUILD/cri-o-1.13.1/_output/src/github.com/kubernetes-sigs/cri-o/vendor/google.golang.org/grpc/server.go:507 +0x541
Mar 14 16:30:36 clr-02 crio[19232]: goroutine 42 [syscall, 14 minutes]:
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment