Skip to content

Instantly share code, notes, and snippets.

@tbg
Created May 22, 2019 00:45
Show Gist options
  • Save tbg/9e79b193c88b28fc78038416e6b141cf to your computer and use it in GitHub Desktop.
Save tbg/9e79b193c88b28fc78038416e6b141cf to your computer and use it in GitHub Desktop.
I190522 00:41:14.287997 1 rand.go:87 Random seed: 2857119843189535667
=== RUN TestChangefeedDataTTL
=== RUN TestChangefeedDataTTL/sinkless
W190522 00:41:14.295254 15 server/status/runtime.go:322 [n?] Could not parse build timestamp: parsing time "" as "2006/01/02 15:04:05": cannot parse "" as "2006"
I190522 00:41:14.304266 15 server/server.go:888 [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
I190522 00:41:14.304507 15 base/addr_validation.go:279 [n?] server certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node
I190522 00:41:14.304543 15 base/addr_validation.go:319 [n?] web UI certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node
I190522 00:41:14.307296 15 server/config.go:506 [n?] 1 storage engine initialized
I190522 00:41:14.307323 15 server/config.go:509 [n?] RocksDB cache size: 128 MiB
I190522 00:41:14.307334 15 server/config.go:509 [n?] store 0: in-memory, size 0 B
I190522 00:41:14.307442 15 server/server.go:1462 [n?] no stores bootstrapped and --join flag specified, awaiting init command or join with an already initialized node.
I190522 00:41:14.307504 15 server/server.go:1472 [n?] **** add additional nodes by specifying --join=127.0.0.1:32803
I190522 00:41:14.310486 15 server/node.go:320 [n?] **** cluster eca62c41-243c-4b6b-a8a8-e2bee862612e has been created
I190522 00:41:14.310692 15 gossip/gossip.go:398 [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:32803" > attrs:<> locality:<> ServerVersion:<major_val:19 minor_val:1 patch:0 unstable:3 > build_tag:"v19.2.0-alpha.00000000-579-gea1f028aba" started_at:1558485674310584463
I190522 00:41:14.313822 15 server/node.go:422 [n1] initialized store [n1,s1]: disk (capacity=512 MiB, available=512 MiB, used=0 B, logicalBytes=8.5 KiB), ranges=20, leases=0, queries=0.00, writes=0.00, bytesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=887.00 pMax=7614.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00 pMax=0.00}
I190522 00:41:14.313916 15 storage/stores.go:244 [n1] read 0 node addresses from persistent storage
W190522 00:41:14.314269 361 storage/store.go:1534 [n1,s1,r6/1:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r6: replica (n1,s1):1 not lease holder; lease holder unknown
I190522 00:41:14.314368 15 server/node.go:636 [n1] connecting to gossip network to verify cluster ID...
I190522 00:41:14.314399 15 server/node.go:656 [n1] node connected via gossip and verified as part of cluster "eca62c41-243c-4b6b-a8a8-e2bee862612e"
I190522 00:41:14.314516 15 server/node.go:503 [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I190522 00:41:14.314600 15 server/server.go:1914 [n1] Could not start goroutine dumper worker due to: directory to store dumps could not be determined
I190522 00:41:14.316574 15 server/status/recorder.go:612 [n1] available memory from cgroups (8.0 EiB) exceeds system memory 31 GiB, using system memory
I190522 00:41:14.316618 15 server/server.go:1925 [n1] Could not start heap profiler worker due to: directory to store profiles could not be determined
I190522 00:41:14.316744 15 server/server.go:1592 [n1] starting https server at 127.0.0.1:35383 (use: 127.0.0.1:35383)
I190522 00:41:14.316773 15 server/server.go:1594 [n1] starting grpc/postgres server at 127.0.0.1:32803
I190522 00:41:14.316787 15 server/server.go:1595 [n1] advertising CockroachDB node at 127.0.0.1:32803
I190522 00:41:14.334581 60 sql/event_log.go:135 [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root}
I190522 00:41:14.344270 149 sql/event_log.go:135 [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:19.1-3 User:root}
I190522 00:41:14.350105 471 sql/event_log.go:135 [n1,intExec=disableNetTrace] Event: "set_cluster_setting", target: 0, info: {SettingName:trace.debug.enable Value:false User:root}
I190522 00:41:14.363094 748 sql/event_log.go:135 [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:f98f30e4-0415-474d-9834-9ddcbf76b4d2 User:root}
I190522 00:41:14.370670 826 sql/event_log.go:135 [n1,intExec=create-default-db] Event: "create_database", target: 50, info: {DatabaseName:defaultdb Statement:CREATE DATABASE IF NOT EXISTS defaultdb User:root}
I190522 00:41:14.373631 777 sql/event_log.go:135 [n1,intExec=create-default-db] Event: "create_database", target: 51, info: {DatabaseName:postgres Statement:CREATE DATABASE IF NOT EXISTS postgres User:root}
I190522 00:41:14.378914 15 server/server.go:1663 [n1] done ensuring all necessary migrations have run
I190522 00:41:14.378953 15 server/server.go:1666 [n1] serving sql connections
I190522 00:41:14.381409 759 server/server_update.go:72 [n1] no need to upgrade, cluster already at the newest version
I190522 00:41:14.383038 762 sql/event_log.go:135 [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:127.0.0.1:32803 Attrs: Locality: ServerVersion:19.1-3 BuildTag:v19.2.0-alpha.00000000-579-gea1f028aba StartedAt:1558485674310584463 LocalityAddress:[] XXX_NoUnkeyedLiteral:{} XXX_sizecache:0} ClusterID:eca62c41-243c-4b6b-a8a8-e2bee862612e StartedAt:1558485674310584463 LastUp:1558485674310584463}
I190522 00:41:14.394905 601 sql/event_log.go:135 [n1,client=127.0.0.1:53778,user=root] Event: "set_cluster_setting", target: 0, info: {SettingName:kv.rangefeed.enabled Value:true User:root}
I190522 00:41:14.399504 601 sql/event_log.go:135 [n1,client=127.0.0.1:53778,user=root] Event: "set_cluster_setting", target: 0, info: {SettingName:kv.closed_timestamp.target_duration Value:00:00:01 User:root}
I190522 00:41:14.403003 601 sql/event_log.go:135 [n1,client=127.0.0.1:53778,user=root] Event: "set_cluster_setting", target: 0, info: {SettingName:changefeed.experimental_poll_interval Value:00:00:00.01 User:root}
I190522 00:41:14.405985 601 sql/event_log.go:135 [n1,client=127.0.0.1:53778,user=root] Event: "create_database", target: 52, info: {DatabaseName:d Statement:CREATE DATABASE d User:root}
I190522 00:41:14.409661 601 sql/event_log.go:135 [n1,client=127.0.0.1:53778,user=root] Event: "create_table", target: 53, info: {TableName:d.public.foo Statement:CREATE TABLE foo (a INT8 PRIMARY KEY, b STRING) User:root}
I190522 00:41:14.411285 1160 storage/replica_command.go:295 [n1,split,s1,r20/1:/{Table/24-Max}] initiating a split of this range at key /Table/53 [r21] (zone config)
I190522 00:41:14.442757 1289 storage/replica_rangefeed.go:509 [n1,client=127.0.0.1:53788,user=root,s1,r21/1:/{Table/53-Max}] RangeFeed closed timestamp is empty
I190522 00:41:14.456961 15 ccl/changefeedccl/changefeed_test.go:1190 ignoring message foo: [1]->{"after": {"a": 1, "b": "version 1"}}
I190522 00:41:14.457129 15 ccl/changefeedccl/changefeed_test.go:1190 ignoring message foo: [1]->{"after": {"a": 1, "b": "version 2"}}
I190522 00:41:14.462538 15 ccl/changefeedccl/changefeed_test.go:1190 ignoring message foo: [1]->{"after": {"a": 1, "b": "version 3"}}
I190522 00:41:14.462674 15 ccl/changefeedccl/changefeed_test.go:1190 ignoring message foo: [1]->{"after": {"a": 1, "b": "version 4"}}
I190522 00:41:15.869988 108 gossip/gossip.go:1516 [n1] node has connected to cluster via gossip
I190522 00:41:15.870089 108 storage/stores.go:263 [n1] wrote 0 node addresses to persistent storage
I190522 00:41:17.319408 1309 storage/replica_consistency.go:198 [n1,consistencyChecker,s1,r4/1:/System{/tsd-tse}] triggering stats recomputation to resolve delta of {ContainsEstimates:true LastUpdateNanos:1558485674324116066 IntentAge:0 GCBytesAge:0 LiveBytes:-30261 LiveCount:-603 KeyBytes:-29118 KeyCount:-603 ValBytes:-1143 ValCount:-603 IntentBytes:0 IntentCount:0 SysBytes:0 SysCount:0 XXX_NoUnkeyedLiteral:{} XXX_sizecache:0}
I190522 00:41:24.322879 406 server/status/runtime.go:500 [n1] runtime stats: 101 MiB RSS, 327 goroutines, 21 MiB/36 MiB/71 MiB GO alloc/idle/total, 21 MiB/30 MiB CGO alloc/total, 0.0 CGO/sec, 0.0/0.0 %(u/s)time, 0.0 %gc (9x), 10 MiB/10 MiB (r/w)net
I190522 00:41:34.324017 406 server/status/runtime.go:500 [n1] runtime stats: 105 MiB RSS, 326 goroutines, 14 MiB/40 MiB/71 MiB GO alloc/idle/total, 21 MiB/33 MiB CGO alloc/total, 2609.4 CGO/sec, 4.8/1.5 %(u/s)time, 0.0 %gc (2x), 10 MiB/10 MiB (r/w)net
I190522 00:41:44.325007 406 server/status/runtime.go:500 [n1] runtime stats: 107 MiB RSS, 326 goroutines, 14 MiB/41 MiB/71 MiB GO alloc/idle/total, 21 MiB/34 MiB CGO alloc/total, 2527.2 CGO/sec, 4.9/1.0 %(u/s)time, 0.0 %gc (1x), 10 MiB/10 MiB (r/w)net
I190522 00:41:54.325969 406 server/status/runtime.go:500 [n1] runtime stats: 108 MiB RSS, 327 goroutines, 14 MiB/41 MiB/71 MiB GO alloc/idle/total, 21 MiB/35 MiB CGO alloc/total, 2535.3 CGO/sec, 4.9/0.9 %(u/s)time, 0.0 %gc (1x), 10 MiB/10 MiB (r/w)net
I190522 00:42:04.327270 406 server/status/runtime.go:500 [n1] runtime stats: 111 MiB RSS, 327 goroutines, 15 MiB/41 MiB/71 MiB GO alloc/idle/total, 21 MiB/37 MiB CGO alloc/total, 2527.9 CGO/sec, 4.6/1.1 %(u/s)time, 0.0 %gc (1x), 10 MiB/10 MiB (r/w)net
I190522 00:42:14.316604 402 gossip/gossip.go:563 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
gossip connectivity
n1 [sentinel];
I190522 00:42:14.328494 406 server/status/runtime.go:500 [n1] runtime stats: 112 MiB RSS, 330 goroutines, 16 MiB/40 MiB/71 MiB GO alloc/idle/total, 21 MiB/37 MiB CGO alloc/total, 2528.9 CGO/sec, 4.7/1.2 %(u/s)time, 0.0 %gc (1x), 10 MiB/10 MiB (r/w)net
I190522 00:42:14.329930 249 storage/queue.go:471 [n1,s1,r11/1:/Table/1{5-6}] rate limited in MaybeAdd (merge): context canceled
I190522 00:42:24.329600 406 server/status/runtime.go:500 [n1] runtime stats: 113 MiB RSS, 326 goroutines, 20 MiB/37 MiB/71 MiB GO alloc/idle/total, 21 MiB/37 MiB CGO alloc/total, 2589.9 CGO/sec, 5.0/1.0 %(u/s)time, 0.0 %gc (1x), 10 MiB/10 MiB (r/w)net
I190522 00:42:34.330901 406 server/status/runtime.go:500 [n1] runtime stats: 114 MiB RSS, 329 goroutines, 18 MiB/39 MiB/71 MiB GO alloc/idle/total, 20 MiB/37 MiB CGO alloc/total, 2516.5 CGO/sec, 4.6/1.2 %(u/s)time, 0.0 %gc (1x), 10 MiB/10 MiB (r/w)net
I190522 00:42:44.332097 406 server/status/runtime.go:500 [n1] runtime stats: 114 MiB RSS, 327 goroutines, 17 MiB/39 MiB/71 MiB GO alloc/idle/total, 20 MiB/38 MiB CGO alloc/total, 2535.2 CGO/sec, 4.9/0.9 %(u/s)time, 0.0 %gc (1x), 10 MiB/10 MiB (r/w)net
I190522 00:42:54.333290 406 server/status/runtime.go:500 [n1] runtime stats: 115 MiB RSS, 327 goroutines, 16 MiB/40 MiB/71 MiB GO alloc/idle/total, 20 MiB/38 MiB CGO alloc/total, 2521.5 CGO/sec, 4.9/1.0 %(u/s)time, 0.0 %gc (1x), 10 MiB/10 MiB (r/w)net
I190522 00:43:04.334545 406 server/status/runtime.go:500 [n1] runtime stats: 118 MiB RSS, 327 goroutines, 15 MiB/40 MiB/71 MiB GO alloc/idle/total, 21 MiB/41 MiB CGO alloc/total, 2526.4 CGO/sec, 4.4/1.3 %(u/s)time, 0.0 %gc (1x), 10 MiB/10 MiB (r/w)net
panic: test timed out after 2m0s
goroutine 2662 [running]:
testing.(*M).startAlarm.func1()
/usr/local/go/src/testing/testing.go:1334 +0xdf
created by time.goFunc
/usr/local/go/src/time/sleep.go:169 +0x44
goroutine 1 [chan receive, 2 minutes]:
testing.(*T).Run(0xc0009d6200, 0x328822a, 0x15, 0x33c4910, 0x74fb01)
/usr/local/go/src/testing/testing.go:917 +0x37e
testing.runTests.func1(0xc0009d6100)
/usr/local/go/src/testing/testing.go:1157 +0x78
testing.tRunner(0xc0009d6100, 0xc000981e10)
/usr/local/go/src/testing/testing.go:865 +0xc0
testing.runTests(0xc0003ac1e0, 0x57932a0, 0x30, 0x30, 0x0)
/usr/local/go/src/testing/testing.go:1155 +0x2a9
testing.(*M).Run(0xc000a04000, 0x0)
/usr/local/go/src/testing/testing.go:1072 +0x162
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.TestMain(0xc000a04000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/main_test.go:31 +0xda
main.main()
_testmain.go:136 +0x13e
goroutine 6 [syscall, 2 minutes]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:29 +0x41
goroutine 9 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1226 +0xf3
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:629 +0x121
goroutine 10 [chan receive, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:636 +0x126
created by github.com/cockroachdb/cockroach/pkg/util/log.init.0
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:630 +0x139
goroutine 611 [IO wait]:
internal/poll.runtime_pollWait(0x7f4ad7716af8, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc001014098, 0x72, 0x700, 0x719, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc001014080, 0xc0013e9000, 0x719, 0x719, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc001014080, 0xc0013e9000, 0x719, 0x719, 0xc0017bd500, 0x39aaf01, 0x5a65bc0)
/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0009ac010, 0xc0013e9000, 0x719, 0x719, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:177 +0x69
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc00142e000, 0xc0013e9000, 0x719, 0x719, 0x0, 0x39ab260, 0x10)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111
crypto/tls.(*atLeastReader).Read(0xc000434480, 0xc0013e9000, 0x719, 0x719, 0x0, 0x39ab260, 0xc0017bd618)
/usr/local/go/src/crypto/tls/conn.go:761 +0x60
bytes.(*Buffer).ReadFrom(0xc001410258, 0x39a5f60, 0xc000434480, 0x6a05b5, 0x2ebbd80, 0x30901e0)
/usr/local/go/src/bytes/buffer.go:207 +0xbd
crypto/tls.(*Conn).readFromUntil(0xc001410000, 0x7f4ad1a0b048, 0xc00142e000, 0x5, 0xc00142e000, 0xc0017bd6e0)
/usr/local/go/src/crypto/tls/conn.go:783 +0xf8
crypto/tls.(*Conn).readRecordOrCCS(0xc001410000, 0x33caa00, 0xc001410138, 0xed4769222)
/usr/local/go/src/crypto/tls/conn.go:590 +0x125
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:558
crypto/tls.(*Conn).Read(0xc001410000, 0xc00159b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1236 +0x137
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*readTimeoutConn).Read(0xc0003c2f60, 0xc00159b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:1939 +0x110
bufio.(*Reader).fill(0xc0004db048)
/usr/local/go/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).ReadByte(0xc0004db048, 0xc0004db618, 0xc0016fdb00, 0x1)
/usr/local/go/src/bufio/bufio.go:249 +0x39
github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase.(*ReadBuffer).ReadTypedMsg(0xc0004db6f0, 0x39e9020, 0xc0004db048, 0x39dec20, 0x5a66340, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase/encoding.go:121 +0x35
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl(0xc0004db000, 0x3a13f20, 0xc000a8dbc0, 0xc000463c80, 0xc000c44000, 0x5400, 0x15000, 0xc00014e3d0, 0x0, 0x0, ...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:315 +0x474
github.com/cockroachdb/cockroach/pkg/sql/pgwire.serveConn(0x3a13f20, 0xc000a8dbc0, 0x3a47fa0, 0xc001410000, 0xc0011cc020, 0x4, 0xc0011e8540, 0x39dd7e0, 0xc0007ba4b0, 0x4000, ...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:171 +0x18d
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc00014e240, 0x3a13f20, 0xc000a8dbc0, 0x3a47fa0, 0xc001410000, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:528 +0x87e
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func20.1(0x3a48840, 0xc00142e000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1721 +0x163
github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith.func1(0xc0009e6360, 0x3a13fe0, 0xc0011be000, 0xc0004aa000, 0x3a48840, 0xc00142e000, 0xc0011be030)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:139 +0xa9
created by github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:136 +0x222
goroutine 12 [select]:
github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view.(*worker).start(0xc000505900)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:152 +0xdd
created by github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view.init.0
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:29 +0x57
goroutine 14 [chan receive, 2 minutes]:
testing.(*T).Run(0xc0009d6300, 0x326be9e, 0x8, 0xc0003de540, 0xa477838561a)
/usr/local/go/src/testing/testing.go:917 +0x37e
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.TestChangefeedDataTTL(0xc0009d6200)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:1195 +0x96
testing.tRunner(0xc0009d6200, 0x33c4910)
/usr/local/go/src/testing/testing.go:865 +0xc0
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:916 +0x357
goroutine 15 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f4ad7716bc8, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc001749d18, 0x72, 0x700, 0x791, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc001749d00, 0xc000a75800, 0x791, 0x791, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc001749d00, 0xc000a75800, 0x791, 0x791, 0x203000, 0xc001937130, 0x7613e6)
/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000105d8, 0xc000a75800, 0x791, 0x791, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:177 +0x69
crypto/tls.(*atLeastReader).Read(0xc0015b4940, 0xc000a75800, 0x791, 0x791, 0x0, 0x0, 0xc0019371a8)
/usr/local/go/src/crypto/tls/conn.go:761 +0x60
bytes.(*Buffer).ReadFrom(0xc001451e58, 0x39a5f60, 0xc0015b4940, 0x6a05b5, 0x2ebbd80, 0x31c9cc0)
/usr/local/go/src/bytes/buffer.go:207 +0xbd
crypto/tls.(*Conn).readFromUntil(0xc001451c00, 0x39ab2a0, 0xc0000105d8, 0x5, 0xc0000105d8, 0x40)
/usr/local/go/src/crypto/tls/conn.go:783 +0xf8
crypto/tls.(*Conn).readRecordOrCCS(0xc001451c00, 0x33caa00, 0xc001451d38, 0xc001937488)
/usr/local/go/src/crypto/tls/conn.go:590 +0x125
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:558
crypto/tls.(*Conn).Read(0xc001451c00, 0xc001172af3, 0x50d, 0x50d, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1236 +0x137
io.ReadAtLeast(0x7f4ad771b698, 0xc001451c00, 0xc001172af3, 0x50d, 0x50d, 0x5, 0xc00035ef00, 0x3, 0xc000066070)
/usr/local/go/src/io/io.go:310 +0x88
github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/chunkreader.(*ChunkReader).appendAtLeast(...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/chunkreader/chunkreader.go:73
github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/chunkreader.(*ChunkReader).Next(0xc001668240, 0x5, 0x33caa48, 0x6aa700, 0xc001937540, 0xc001937560, 0x21856af)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/chunkreader/chunkreader.go:63 +0x143
github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/pgproto3.(*Frontend).Receive(0xc001174000, 0x1, 0x0, 0xc001937628, 0x6a24a9)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/pgproto3/frontend.go:56 +0x3e2
github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx.(*Conn).rxMsg(0xc0015a9a40, 0xc0007dbd50, 0x5430915, 0x24, 0xc0019a6000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/conn.go:1419 +0x4f
github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx.(*Rows).Next(0xc00056d940, 0xc0007dbd50)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/query.go:126 +0x2c4
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/cdctest.(*sinklessFeed).Next(0xc000f60f00, 0xc0000520d8, 0x1, 0x1)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/cdctest/testfeed.go:142 +0x87
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.TestChangefeedDataTTL.func1(0xc0009d6300, 0xc0006a4300, 0x39d21e0, 0xc001400480)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:1185 +0x42a
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.sinklessTest.func1(0xc0009d6300)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:231 +0x566
testing.tRunner(0xc0009d6300, 0xc0003de540)
/usr/local/go/src/testing/testing.go:865 +0xc0
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:916 +0x357
goroutine 16 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc0009e6360, 0xc0009f4840)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:179 +0x14a
created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:176 +0x49
goroutine 66 [chan receive, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.func1(0x3a13fe0, 0xc0009d2090)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:448 +0x52
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0004291a0, 0xc0009e6360, 0xc00033f500)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 67 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.newInfoStore.func1(0x3a13fe0, 0xc00089e270)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/gossip/infostore.go:194 +0x15b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429260, 0xc0009e6360, 0xc000429250)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 26 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc000315da8, 0x0)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc000315d98)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc000315c80)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:679 +0x70
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:666 +0x255
goroutine 27 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a13fe0, 0xc00018f230)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:96 +0x140
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003df250, 0xc0009e6360, 0xc0003df240)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 28 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a13fe0, 0xc00018f290)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:96 +0x140
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003df270, 0xc0009e6360, 0xc0003df260)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 29 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a13fe0, 0xc00018ec30)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:96 +0x140
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003df290, 0xc0009e6360, 0xc0003df280)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 30 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a13fe0, 0xc0008a2390)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:96 +0x140
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003df2b0, 0xc0009e6360, 0xc0003df2a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 31 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a13fe0, 0xc0008a2420)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:96 +0x140
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003df2d0, 0xc0009e6360, 0xc0003df2c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 32 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a13fe0, 0xc0008a24b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:96 +0x140
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003df2f0, 0xc0009e6360, 0xc0003df2e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 33 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a13fe0, 0xc0008a2300)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:96 +0x140
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003df330, 0xc0009e6360, 0xc0003df310)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 82 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a13fe0, 0xc0008a2540)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:96 +0x140
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003df360, 0xc0009e6360, 0xc0003df350)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 83 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a13fe0, 0xc0008a2570)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:96 +0x140
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003df3c0, 0xc0009e6360, 0xc0003df3b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 84 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a13fe0, 0xc0008a2600)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:96 +0x140
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003df420, 0xc0009e6360, 0xc0003df410)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 85 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a13fe0, 0xc0008a2630)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:96 +0x140
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003df460, 0xc0009e6360, 0xc0003df430)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 86 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a13fe0, 0xc0008a24e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:96 +0x140
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003df4f0, 0xc0009e6360, 0xc0003df480)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 87 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a13fe0, 0xc00018f6e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:96 +0x140
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003df590, 0xc0009e6360, 0xc0003df570)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 88 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a13fe0, 0xc0008a2690)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:96 +0x140
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003df5b0, 0xc0009e6360, 0xc0003df5a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 89 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a13fe0, 0xc000210000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:96 +0x140
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003df5d0, 0xc0009e6360, 0xc0003df5c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 90 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x3a13fe0, 0xc0004b45d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:96 +0x140
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003df5f0, 0xc0009e6360, 0xc0003df5e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 91 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc000a005a0, 0x3a13f20, 0xc0000cb180)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1290 +0xdf
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:83 +0x293
goroutine 92 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc000a006c0, 0x3a13f20, 0xc0000cb180)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1290 +0xdf
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:83 +0x293
goroutine 68 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x3a13fe0, 0xc0009ef320)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1657 +0x385
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0000b9d30, 0xc0009e6360, 0xc0009ef2f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 69 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).PeriodicallyRefreshSomeLeases.func1(0x3a13fe0, 0xc0004b4690)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1732 +0x257
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0000b9d50, 0xc0009e6360, 0xc0000b9d40)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 70 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps.func1(0x33cac98, 0xc000558230, 0xc0003be460, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:185 +0x11b
created by github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:178 +0x105
goroutine 93 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x3a13fe0, 0xc0009c01e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:99 +0x65
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002e41b0, 0xc0009e6360, 0xc000408080)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 94 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func2(0x3a13fe0, 0xc0009c0390)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1175 +0x54
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002e45d0, 0xc0009e6360, 0xc000408200)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 95 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f4ad7716d68, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc000a04098, 0x72, 0x0, 0x0, 0x3268a5d)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000a04080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba
net.(*netFD).accept(0xc000a04080, 0xc000538000, 0x6ef6f0, 0x10)
/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0004aa038, 0x6c2bdf, 0xc0000cab70, 0xc000e20ef0)
/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0004aa038, 0x33c8040, 0xc00014c540, 0xc00041c120, 0x0)
/usr/local/go/src/net/tcpsock.go:260 +0x48
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc00014c540, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x88
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func3(0x3a13fe0, 0xc0004b46f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1187 +0x2f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002e45f0, 0xc0009e6360, 0xc000408260)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 96 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
net/http.(*Server).Serve(0xc0000e7450, 0x39ff3a0, 0xc000408220, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2859 +0x22d
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func4(0x3a13fe0, 0xc0009c03c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1199 +0x127
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002e4610, 0xc0009e6360, 0xc0009ef5f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 97 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
crypto/tls.(*listener).Accept(0xc000408280, 0x3096120, 0xc0008a28d0, 0x2e4c1e0, 0x576ba80)
/usr/local/go/src/crypto/tls/tls.go:62 +0x37
net/http.(*Server).Serve(0xc0009c4000, 0x39e91e0, 0xc000408280, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2859 +0x22d
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func5(0x3a13fe0, 0xc0008a2750)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1206 +0x42
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002e4640, 0xc0009e6360, 0xc0004082c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 98 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func6(0x3a13fe0, 0xc000210570)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1210 +0x7d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002e4670, 0xc0009e6360, 0xc00014c580)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 99 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc000251dc0, 0x39ff3a0, 0xc0004080e0, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:544 +0x1f2
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func7(0x3a13fe0, 0xc0009d2060)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1222 +0x46
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002e46a0, 0xc0009e6360, 0xc0004082e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 71 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x3a13fe0, 0xc0009c0570)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:99 +0x65
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0000b9e70, 0xc0009e6360, 0xc0003fa0c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 100 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func9(0x3a13fe0, 0xc0004b4720)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1300 +0x7d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002e49f0, 0xc0009e6360, 0xc0009ef8c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 102 [select, 1 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc0009ef980)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:109 +0x13f
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).start
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:95 +0x3f
goroutine 103 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func12(0x3a13fe0, 0xc0008a2900)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1330 +0x54
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000428010, 0xc0009e6360, 0xc0004085a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 72 [select, 1 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0008a4340)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x110
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x13e
goroutine 73 [select, 1 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc00021c000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1384 +0x1c4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc00021c000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:960 +0x1d9
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:951 +0xe1
goroutine 75 [select]:
net.(*pipe).read(0xc000a04200, 0xc00042c000, 0x791, 0x791, 0x203000, 0x0, 0x784)
/usr/local/go/src/net/pipe.go:164 +0x19b
net.(*pipe).Read(0xc000a04200, 0xc00042c000, 0x791, 0x791, 0xc00042c00d, 0x11, 0x10)
/usr/local/go/src/net/pipe.go:147 +0x4d
crypto/tls.(*atLeastReader).Read(0xc001752f60, 0xc00042c000, 0x791, 0x791, 0x2e, 0x39a6220, 0xc000b46ac8)
/usr/local/go/src/crypto/tls/conn.go:761 +0x60
bytes.(*Buffer).ReadFrom(0xc000539058, 0x39a5f60, 0xc001752f60, 0x6a05b5, 0x2ebbd80, 0x30d2360)
/usr/local/go/src/bytes/buffer.go:207 +0xbd
crypto/tls.(*Conn).readFromUntil(0xc000538e00, 0x7f4ad1a4b028, 0xc000a04200, 0x5, 0xc000a04200, 0x11)
/usr/local/go/src/crypto/tls/conn.go:783 +0xf8
crypto/tls.(*Conn).readRecordOrCCS(0xc000538e00, 0x33caa00, 0xc000538f38, 0x0)
/usr/local/go/src/crypto/tls/conn.go:590 +0x125
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:558
crypto/tls.(*Conn).Read(0xc000538e00, 0xc000a26000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1236 +0x137
bufio.(*Reader).Read(0xc0005114a0, 0xc0003282d8, 0x9, 0x9, 0x6a24a9, 0xc00062e180, 0xc000b46e48)
/usr/local/go/src/bufio/bufio.go:223 +0x22f
io.ReadAtLeast(0x39a5d60, 0xc0005114a0, 0xc0003282d8, 0x9, 0x9, 0x9, 0x0, 0xc0008ce6b8, 0x6edb80)
/usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(...)
/usr/local/go/src/io/io.go:329
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0003282d8, 0x9, 0x9, 0x39a5d60, 0xc0005114a0, 0x0, 0xc000000000, 0xc000b46eaf, 0x1)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0003282a0, 0xc000b71d40, 0xc000b71d40, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc000aac000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1135 +0x123
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:270 +0xba9
goroutine 104 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00013f028, 0xc000000135)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00013f018)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc00013ef00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:679 +0x70
created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:666 +0x255
goroutine 105 [select]:
github.com/cockroachdb/cockroach/pkg/server.startAssertEngineHealth.func1(0x3a13fe0, 0xc000211a10)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server_engine_health.go:48 +0x16b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0004281d0, 0xc0009e6360, 0xc0009efa70)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 106 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x3a13fe0, 0xc0009d20f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:387 +0x5b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000428290, 0xc0009e6360, 0xc000408de0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 107 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x3a13fe0, 0xc0009d2120)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1350 +0x3b4
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0004282b0, 0xc0009e6360, 0xc0004282a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 108 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x3a13fe0, 0xc0008a2990)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1385 +0x33c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0004282d0, 0xc0009e6360, 0xc0004282c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 109 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7f4ad7716e38, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0003afd18, 0x72, 0x0, 0x0, 0x3268a5d)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0003afd00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba
net.(*netFD).accept(0xc0003afd00, 0xc0006e2101, 0x0, 0x0)
/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0004aa020, 0xc000a16e80, 0xc000a16e88, 0x28)
/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0004aa020, 0x33c8048, 0xc00014c4c0, 0x3a48a80, 0xc0006e2160)
/usr/local/go/src/net/tcpsock.go:260 +0x48
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc00014c4c0, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x88
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func13.1()
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1434 +0x2f
sync.(*Once).Do(0xc000252140, 0xc000c41750)
/usr/local/go/src/sync/once.go:44 +0xb3
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func13(0x3a13fe0, 0xc00089e2d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1433 +0x63
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0004282e0, 0xc0009e6360, 0xc000408e00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 110 [semacquire, 1 minutes]:
sync.runtime_SemacquireMutex(0xc000252144, 0xc000c41f00)
/usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc000252140)
/usr/local/go/src/sync/mutex.go:134 +0xff
sync.(*Once).Do(0xc000252140, 0xc000c41f50)
/usr/local/go/src/sync/once.go:40 +0x3b
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func15(0x3a13fe0, 0xc00089e330)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1491 +0x63
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000428390, 0xc0009e6360, 0xc000409900)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 111 [select]:
github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).runCloser(0xc0008a8e00, 0x3a13fe0, 0xc00089e360)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:138 +0x333
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000428470, 0xc0009e6360, 0xc000428460)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 146 [chan receive]:
github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x3a13fe0, 0xc0008a29c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:221 +0x147
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000462000, 0xc0009e6360, 0xc0003e4000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 113 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc000412460, 0x3a13f60, 0xc0000520d8)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:410 +0x319
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0009e6360, 0x3a13f60, 0xc0000520d8, 0xc000056db0, 0x22, 0x0, 0x0, 0xc000428660)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:325 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320 +0x131
goroutine 162 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc000412620, 0x3a13f60, 0xc0000520d8)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:410 +0x319
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0009e6360, 0x3a13f60, 0xc0000520d8, 0xc000056e10, 0x22, 0x0, 0x0, 0xc000428670)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:325 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320 +0x131
goroutine 131 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*controlBuffer).get(0xc0008a4980, 0x1, 0x0, 0x0, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:300 +0x104
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*loopyWriter).run(0xc00089d0e0, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:387 +0x1b6
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc000aac000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:303 +0x7b
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:301 +0xd09
goroutine 132 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*controlBuffer).get(0xc000505c00, 0x1, 0x0, 0x0, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:300 +0x104
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*loopyWriter).run(0xc0005c26c0, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:387 +0x1b6
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server.func2(0xc000af6000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:277 +0xcb
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:274 +0xe64
goroutine 133 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc000af6000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:875 +0x1ed
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:283 +0xe89
goroutine 134 [select]:
net.(*pipe).read(0xc000a04180, 0xc000a74000, 0x75b, 0x75b, 0x203000, 0x0, 0x74e)
/usr/local/go/src/net/pipe.go:164 +0x19b
net.(*pipe).Read(0xc000a04180, 0xc000a74000, 0x75b, 0x75b, 0xc000a7400d, 0x11, 0x10)
/usr/local/go/src/net/pipe.go:147 +0x4d
crypto/tls.(*atLeastReader).Read(0xc001752f80, 0xc000a74000, 0x75b, 0x75b, 0x2e, 0x39a6220, 0xc000be6a30)
/usr/local/go/src/crypto/tls/conn.go:761 +0x60
bytes.(*Buffer).ReadFrom(0xc000544958, 0x39a5f60, 0xc001752f80, 0x6a05b5, 0x2ebbd80, 0x30d2360)
/usr/local/go/src/bytes/buffer.go:207 +0xbd
crypto/tls.(*Conn).readFromUntil(0xc000544700, 0x7f4ad1a4b028, 0xc000a04180, 0x5, 0xc000a04180, 0x11)
/usr/local/go/src/crypto/tls/conn.go:783 +0xf8
crypto/tls.(*Conn).readRecordOrCCS(0xc000544700, 0x33caa00, 0xc000544838, 0x0)
/usr/local/go/src/crypto/tls/conn.go:590 +0x125
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:558
crypto/tls.(*Conn).Read(0xc000544700, 0xc000a94000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1236 +0x137
bufio.(*Reader).Read(0xc00089d020, 0xc00051a038, 0x9, 0x9, 0xc00020f180, 0x7f4ad776c8b8, 0x0)
/usr/local/go/src/bufio/bufio.go:223 +0x22f
io.ReadAtLeast(0x39a5d60, 0xc00089d020, 0xc00051a038, 0x9, 0x9, 0x9, 0xbd65c5, 0xc000b71d6c, 0xc0002f47b0)
/usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(...)
/usr/local/go/src/io/io.go:329
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00051a038, 0x9, 0x9, 0x39a5d60, 0xc00089d020, 0x0, 0x0, 0xc000b71d60, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00051a000, 0xc000b71d60, 0xc000b71d60, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc000af6000, 0xc0009d22d0, 0x33c9c38)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:434 +0x7c
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc000251dc0, 0x3a4c9e0, 0xc000af6000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:676 +0xdd
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:631 +0x3c
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3(0xc00033f0a0, 0xc000251dc0, 0x7f4ad1a0b028, 0xc000af6000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:639 +0x27
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:638 +0x5e4
goroutine 163 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x3a13fe0, 0xc000903230)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:155 +0x4a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000428ea0, 0xc0009e6360, 0xc0003d0a60)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 164 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x241)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903260)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903260)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000428ed0, 0xc0009e6360, 0xc000428eb0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 165 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x243)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903290)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903290)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000428ef0, 0xc0009e6360, 0xc000428ee0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 166 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x242)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009032c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009032c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000428f20, 0xc0009e6360, 0xc000428f10)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 167 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x184)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009032f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009032f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000428f50, 0xc0009e6360, 0xc000428f40)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 168 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x185)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903320)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903320)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000428f80, 0xc0009e6360, 0xc000428f70)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 169 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x186)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903350)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903350)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429010, 0xc0009e6360, 0xc000429000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 170 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x187)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903380)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903380)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429030, 0xc0009e6360, 0xc000429020)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 171 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x240)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903200)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903200)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429060, 0xc0009e6360, 0xc000429050)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 172 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x189)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009033e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009033e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429090, 0xc0009e6360, 0xc000429070)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 173 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x18b)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903410)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903410)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0004290c0, 0xc0009e6360, 0xc0004290a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 174 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x18e)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903440)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903440)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0004290e0, 0xc0009e6360, 0xc0004290d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 175 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x190)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903470)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903470)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429110, 0xc0009e6360, 0xc000429100)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 176 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x192)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009034a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009034a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429160, 0xc0009e6360, 0xc000429150)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 177 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x194)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009034d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009034d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429180, 0xc0009e6360, 0xc000429170)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 178 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x197)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903500)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903500)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0004291e0, 0xc0009e6360, 0xc0004291a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 179 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x199)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903530)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903530)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429220, 0xc0009e6360, 0xc000429210)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 180 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x188)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009033b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009033b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429240, 0xc0009e6360, 0xc000429230)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 181 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x18c)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e3c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e3c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429260, 0xc0009e6360, 0xc000429250)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 182 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x18d)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e3f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e3f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0004292c0, 0xc0009e6360, 0xc000429290)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 183 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x18f)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e420)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e420)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0004292f0, 0xc0009e6360, 0xc0004292d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 184 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x191)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e450)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e450)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429320, 0xc0009e6360, 0xc000429310)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 185 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x193)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e480)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e480)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429350, 0xc0009e6360, 0xc000429340)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 186 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x196)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e4b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e4b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0004297e0, 0xc0009e6360, 0xc0004297c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 187 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x195)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e4e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e4e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429820, 0xc0009e6360, 0xc000429810)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 188 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x198)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e510)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e510)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429880, 0xc0009e6360, 0xc000429870)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 189 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x19a)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e540)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e540)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0004298b0, 0xc0009e6360, 0xc0004298a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 190 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x18a)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e390)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e390)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0004298d0, 0xc0009e6360, 0xc0004298c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 191 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x19c)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903590)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903590)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429900, 0xc0009e6360, 0xc0004298f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 192 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1a0)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e5a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e5a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429930, 0xc0009e6360, 0xc000429910)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 193 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1a1)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e5d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e5d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429960, 0xc0009e6360, 0xc000429940)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 194 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1a3)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e600)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e600)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429990, 0xc0009e6360, 0xc000429970)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 195 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1a5)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e630)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e630)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0004299c0, 0xc0009e6360, 0xc0004299a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 196 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1a7)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e660)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e660)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0004299e0, 0xc0009e6360, 0xc0004299d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 197 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1aa)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e690)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e690)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429a10, 0xc0009e6360, 0xc000429a00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 198 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1ab)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e6c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e6c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429a40, 0xc0009e6360, 0xc000429a30)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 199 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1ac)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009036e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009036e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429a70, 0xc0009e6360, 0xc000429a60)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 200 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x19d)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e570)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e570)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429b00, 0xc0009e6360, 0xc000429af0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 201 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1a8)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009036b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009036b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429b20, 0xc0009e6360, 0xc000429b10)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 202 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1a6)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903680)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903680)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429b40, 0xc0009e6360, 0xc000429b30)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 203 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1ae)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0008a2cc0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0008a2cc0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429b60, 0xc0009e6360, 0xc000429b50)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 204 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1b0)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0008a2d20)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0008a2d20)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429b80, 0xc0009e6360, 0xc000429b70)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 205 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1a9)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0008a2c60)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0008a2c60)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429ba0, 0xc0009e6360, 0xc000429b90)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 206 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x19e)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009035c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009035c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429bc0, 0xc0009e6360, 0xc000429bb0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 207 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x19f)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009035f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009035f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429be0, 0xc0009e6360, 0xc000429bd0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 208 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1a2)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903620)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903620)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429c00, 0xc0009e6360, 0xc000429bf0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 209 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1a4)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903650)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903650)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429c20, 0xc0009e6360, 0xc000429c10)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 210 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x19b)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903560)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903560)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429c40, 0xc0009e6360, 0xc000429c30)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 211 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1b2)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903740)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903740)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429c60, 0xc0009e6360, 0xc000429c50)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 212 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1b3)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903770)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903770)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429c80, 0xc0009e6360, 0xc000429c70)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 213 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1b6)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009037a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009037a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429ca0, 0xc0009e6360, 0xc000429c90)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 214 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1ba)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009037d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009037d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429cc0, 0xc0009e6360, 0xc000429cb0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 215 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1bc)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903800)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903800)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429ce0, 0xc0009e6360, 0xc000429cd0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 216 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1be)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903830)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903830)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429d00, 0xc0009e6360, 0xc000429cf0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 217 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1bf)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e7e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e7e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429d20, 0xc0009e6360, 0xc000429d10)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 218 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1c0)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903860)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903860)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429d40, 0xc0009e6360, 0xc000429d30)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 219 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1ad)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903710)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903710)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429d60, 0xc0009e6360, 0xc000429d50)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 220 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1b1)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e720)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e720)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429d90, 0xc0009e6360, 0xc000429d80)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 221 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1b7)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0008a2de0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0008a2de0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429dc0, 0xc0009e6360, 0xc000429da0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 222 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1b9)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0008a2e40)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0008a2e40)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429df0, 0xc0009e6360, 0xc000429dd0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 223 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1bd)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0008a2ea0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0008a2ea0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429e10, 0xc0009e6360, 0xc000429e00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 224 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1b4)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0008a2d80)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0008a2d80)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429e40, 0xc0009e6360, 0xc000429e30)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 225 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1b5)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e750)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e750)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429e90, 0xc0009e6360, 0xc000429e80)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 226 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1b8)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e780)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e780)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429eb0, 0xc0009e6360, 0xc000429ea0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 227 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1bb)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e7b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e7b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429ed0, 0xc0009e6360, 0xc000429ec0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 228 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1c3)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009d23c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009d23c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429ef0, 0xc0009e6360, 0xc000429ee0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 229 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1af)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e6f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e6f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429f10, 0xc0009e6360, 0xc000429f00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 230 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1c6)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e840)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e840)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429f30, 0xc0009e6360, 0xc000429f20)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 231 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1ce)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e8d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e8d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429f50, 0xc0009e6360, 0xc000429f40)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 232 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1d1)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000211ad0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000211ad0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429f70, 0xc0009e6360, 0xc000429f60)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 233 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1cc)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000211aa0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000211aa0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429f90, 0xc0009e6360, 0xc000429f80)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 234 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1c9)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e870)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e870)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429fb0, 0xc0009e6360, 0xc000429fa0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 235 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1ca)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009d24b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009d24b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429fd0, 0xc0009e6360, 0xc000429fc0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 236 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1cb)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e8a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e8a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000429ff0, 0xc0009e6360, 0xc000429fe0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 237 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1c2)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e810)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e810)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2030, 0xc0009e6360, 0xc0002b2010)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 238 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1c5)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009d2420)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009d2420)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2050, 0xc0009e6360, 0xc0002b2040)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 239 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1c7)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009d2450)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009d2450)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2080, 0xc0009e6360, 0xc0002b2070)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 240 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1c8)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009d2480)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009d2480)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b20b0, 0xc0009e6360, 0xc0002b20a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 241 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1c4)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009d23f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009d23f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2170, 0xc0009e6360, 0xc0002b20d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 242 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x208)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0008a2ff0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0008a2ff0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2190, 0xc0009e6360, 0xc0002b2180)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 243 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1fe)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009d2690)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009d2690)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b21d0, 0xc0009e6360, 0xc0002b21b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 244 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x202)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0008a2f30)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0008a2f30)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2200, 0xc0009e6360, 0xc0002b21f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 245 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x203)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00086c000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00086c000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2230, 0xc0009e6360, 0xc0002b2220)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 246 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1c1)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0008a2f00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0008a2f00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2260, 0xc0009e6360, 0xc0002b2250)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 247 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1d0)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009038c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009038c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b23b0, 0xc0009e6360, 0xc0002b2330)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 248 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1d4)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009038f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009038f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b23d0, 0xc0009e6360, 0xc0002b23c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 249 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1d7)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903920)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903920)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2470, 0xc0009e6360, 0xc0002b23e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 250 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1da)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903950)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903950)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b24d0, 0xc0009e6360, 0xc0002b24b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 251 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1dd)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903980)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903980)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2510, 0xc0009e6360, 0xc0002b2500)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 252 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1e1)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009039b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009039b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2570, 0xc0009e6360, 0xc0002b2520)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 253 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1e3)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009039e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009039e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2600, 0xc0009e6360, 0xc0002b2580)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 254 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1ec)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903a10)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903a10)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2650, 0xc0009e6360, 0xc0002b2610)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 255 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1cf)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903890)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903890)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2690, 0xc0009e6360, 0xc0002b2660)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 256 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1d6)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009d2510)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009d2510)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b26d0, 0xc0009e6360, 0xc0002b26a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 257 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1d8)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009d2540)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009d2540)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2710, 0xc0009e6360, 0xc0002b26e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 258 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1dc)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009d2570)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009d2570)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2760, 0xc0009e6360, 0xc0002b2720)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 259 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1e0)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009d25a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009d25a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b27c0, 0xc0009e6360, 0xc0002b2780)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 260 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1e5)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009d25d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009d25d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2840, 0xc0009e6360, 0xc0002b27e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 261 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1ee)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009d2600)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009d2600)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2880, 0xc0009e6360, 0xc0002b2870)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 262 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1e9)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009d2630)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009d2630)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2910, 0xc0009e6360, 0xc0002b2900)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 263 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1f0)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009d2660)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009d2660)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b29e0, 0xc0009e6360, 0xc0002b29d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 264 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x201)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0004b5230)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0004b5230)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2a10, 0xc0009e6360, 0xc0002b2a00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 265 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1cd)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0009d24e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0009d24e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2a80, 0xc0009e6360, 0xc0002b2a50)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 266 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1d9)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000211bf0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000211bf0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2aa0, 0xc0009e6360, 0xc0002b2a90)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 267 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1df)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0007ba000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0007ba000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2ac0, 0xc0009e6360, 0xc0002b2ab0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 268 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1e4)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0007ba030)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0007ba030)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2ae0, 0xc0009e6360, 0xc0002b2ad0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 269 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1eb)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0007ba060)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0007ba060)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2b00, 0xc0009e6360, 0xc0002b2af0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 270 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1d3)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000211b30)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000211b30)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2b20, 0xc0009e6360, 0xc0002b2b10)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 271 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1d5)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e930)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e930)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2b40, 0xc0009e6360, 0xc0002b2b30)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 272 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1db)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e960)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e960)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2b60, 0xc0009e6360, 0xc0002b2b50)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 273 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1de)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e990)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e990)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2b80, 0xc0009e6360, 0xc0002b2b70)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 274 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1e2)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e9c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e9c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2ba0, 0xc0009e6360, 0xc0002b2b90)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 275 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1d2)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e900)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e900)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2bc0, 0xc0009e6360, 0xc0002b2bb0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 276 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1f1)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0007ba0c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0007ba0c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2be0, 0xc0009e6360, 0xc0002b2bd0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 277 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1e8)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0007ba0f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0007ba0f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2c00, 0xc0009e6360, 0xc0002b2bf0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 278 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1f2)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0007ba120)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0007ba120)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2c20, 0xc0009e6360, 0xc0002b2c10)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 279 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1f4)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0007ba150)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0007ba150)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2c40, 0xc0009e6360, 0xc0002b2c30)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 280 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1f7)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0007ba180)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0007ba180)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2c60, 0xc0009e6360, 0xc0002b2c50)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 281 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1ef)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0007ba090)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0007ba090)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2c80, 0xc0009e6360, 0xc0002b2c70)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 282 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1ea)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089ea20)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089ea20)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2ca0, 0xc0009e6360, 0xc0002b2c90)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 283 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1e7)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089ea50)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089ea50)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2cc0, 0xc0009e6360, 0xc0002b2cb0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 284 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1e6)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089ea80)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089ea80)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2cf0, 0xc0009e6360, 0xc0002b2cd0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 285 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1f3)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089eab0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089eab0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2d30, 0xc0009e6360, 0xc0002b2d00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 286 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1f5)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089eae0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089eae0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2d90, 0xc0009e6360, 0xc0002b2d40)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 287 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1ed)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089e9f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089e9f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2dd0, 0xc0009e6360, 0xc0002b2da0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 288 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1f8)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089eb40)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089eb40)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2e10, 0xc0009e6360, 0xc0002b2de0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 289 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1fa)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089eb70)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089eb70)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2e60, 0xc0009e6360, 0xc0002b2e30)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 290 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1fd)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089eba0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089eba0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2ea0, 0xc0009e6360, 0xc0002b2e70)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 291 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1ff)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089ebd0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089ebd0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2f00, 0xc0009e6360, 0xc0002b2ee0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 292 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x20a)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903a70)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903a70)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2f40, 0xc0009e6360, 0xc0002b2f10)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 293 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x20b)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903aa0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903aa0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b2fd0, 0xc0009e6360, 0xc0002b2fc0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 294 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x204)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903a40)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903a40)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3000, 0xc0009e6360, 0xc0002b2ff0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 295 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x209)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089ec00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089ec00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3030, 0xc0009e6360, 0xc0002b3010)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 296 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x211)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00086c0c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00086c0c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3050, 0xc0009e6360, 0xc0002b3040)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 297 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1f9)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089eb10)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089eb10)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3080, 0xc0009e6360, 0xc0002b3070)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 298 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1fb)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0007ba1e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0007ba1e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b30b0, 0xc0009e6360, 0xc0002b3090)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 299 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1fc)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0007ba210)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0007ba210)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b30d0, 0xc0009e6360, 0xc0002b30c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 300 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x200)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0007ba240)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0007ba240)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3100, 0xc0009e6360, 0xc0002b30f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 301 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x206)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0008a2f90)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0008a2f90)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3130, 0xc0009e6360, 0xc0002b3110)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 302 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x1f6)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0007ba1b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0007ba1b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3170, 0xc0009e6360, 0xc0002b3140)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 303 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x207)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00086c060)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00086c060)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3190, 0xc0009e6360, 0xc0002b3180)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 304 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x20c)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00086c090)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00086c090)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b31e0, 0xc0009e6360, 0xc0002b31c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 305 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x20d)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0007ba270)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0007ba270)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3220, 0xc0009e6360, 0xc0002b31f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 306 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x205)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00086c030)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00086c030)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3250, 0xc0009e6360, 0xc0002b3240)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 307 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x21a)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0008a3800)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0008a3800)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3290, 0xc0009e6360, 0xc0002b3280)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 308 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x215)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089ec60)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089ec60)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b32f0, 0xc0009e6360, 0xc0002b32b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 309 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x212)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089ec30)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089ec30)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3390, 0xc0009e6360, 0xc0002b3370)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 310 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x214)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00086c0f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00086c0f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b33d0, 0xc0009e6360, 0xc0002b33c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 311 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x21b)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089ec90)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089ec90)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3410, 0xc0009e6360, 0xc0002b33f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 312 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x20e)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0008a3770)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0008a3770)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3440, 0xc0009e6360, 0xc0002b3430)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 313 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x21e)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b34d0, 0xc0009e6360, 0xc0002b3470)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 314 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x213)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903b00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903b00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3500, 0xc0009e6360, 0xc0002b34f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 315 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x20f)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903ad0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903ad0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3530, 0xc0009e6360, 0xc0002b3510)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 316 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x216)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0007ba2d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0007ba2d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3550, 0xc0009e6360, 0xc0002b3540)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 317 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x217)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903b30)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903b30)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3580, 0xc0009e6360, 0xc0002b3570)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 318 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x210)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0007ba2a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0007ba2a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b35b0, 0xc0009e6360, 0xc0002b3590)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 319 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x21c)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0007ba300)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0007ba300)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b35e0, 0xc0009e6360, 0xc0002b35c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 320 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x21d)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00086c150)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00086c150)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3600, 0xc0009e6360, 0xc0002b35f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 321 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x218)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00086c120)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00086c120)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3640, 0xc0009e6360, 0xc0002b3620)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 322 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x219)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000903b60)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000903b60)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3660, 0xc0009e6360, 0xc0002b3650)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 323 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x220)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc00089ecc0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc00089ecc0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3690, 0xc0009e6360, 0xc0002b3680)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 324 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x21f)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc0007ba330)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc0007ba330)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b36c0, 0xc0009e6360, 0xc0002b36b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 325 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x223)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e060)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e060)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3710, 0xc0009e6360, 0xc0002b36e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 326 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x222)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e090)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e090)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3730, 0xc0009e6360, 0xc0002b3720)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 327 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x224)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e0c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e0c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3760, 0xc0009e6360, 0xc0002b3750)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 328 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x225)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e0f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e0f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3790, 0xc0009e6360, 0xc0002b3780)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 329 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x226)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e120)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e120)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b37c0, 0xc0009e6360, 0xc0002b37b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 330 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x228)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e150)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e150)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3870, 0xc0009e6360, 0xc0002b3860)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 331 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x227)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e180)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e180)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b38b0, 0xc0009e6360, 0xc0002b3880)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 332 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x229)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e1b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e1b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b38e0, 0xc0009e6360, 0xc0002b38d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 333 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x22b)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e1e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e1e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3910, 0xc0009e6360, 0xc0002b3900)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 334 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x22a)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e210)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e210)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3940, 0xc0009e6360, 0xc0002b3930)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 335 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x22c)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e240)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e240)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3970, 0xc0009e6360, 0xc0002b3950)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 336 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x22f)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e2d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e2d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b39b0, 0xc0009e6360, 0xc0002b39a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 337 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x23c)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000afc000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000afc000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3a60, 0xc0009e6360, 0xc0002b39d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 338 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x22d)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e270)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e270)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3a80, 0xc0009e6360, 0xc0002b3a70)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 339 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x22e)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e2a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e2a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3ab0, 0xc0009e6360, 0xc0002b3aa0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 340 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x221)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e030)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e030)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3ae0, 0xc0009e6360, 0xc0002b3ac0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 341 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x231)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e330)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e330)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3b00, 0xc0009e6360, 0xc0002b3af0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 342 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x232)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e360)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e360)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3b20, 0xc0009e6360, 0xc0002b3b10)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 343 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x234)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e390)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e390)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3b40, 0xc0009e6360, 0xc0002b3b30)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 344 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x233)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e3c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e3c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3b60, 0xc0009e6360, 0xc0002b3b50)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 345 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x235)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e3f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e3f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3b90, 0xc0009e6360, 0xc0002b3b80)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 346 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x236)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e420)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e420)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3bc0, 0xc0009e6360, 0xc0002b3ba0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 347 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x237)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e450)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e450)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3c00, 0xc0009e6360, 0xc0002b3bd0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 348 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x238)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e480)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e480)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3c30, 0xc0009e6360, 0xc0002b3c20)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 349 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x23a)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e4b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e4b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3c80, 0xc0009e6360, 0xc0002b3c60)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 350 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x239)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e4e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e4e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3cc0, 0xc0009e6360, 0xc0002b3c90)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 351 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x23b)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e510)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e510)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3cf0, 0xc0009e6360, 0xc0002b3cd0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 352 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x230)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e300)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e300)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3d30, 0xc0009e6360, 0xc0002b3d20)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 353 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x23d)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e570)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e570)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3da0, 0xc0009e6360, 0xc0002b3d50)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 354 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x23e)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e5a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e5a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3df0, 0xc0009e6360, 0xc0002b3dd0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 355 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d450, 0x23f)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d440)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc000a04700, 0x3a13fe0, 0xc000a9e5d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0x7c
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x3a13fe0, 0xc000a9e5d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3e30, 0xc0009e6360, 0xc0002b3e10)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 356 [semacquire, 2 minutes]:
sync.runtime_Semacquire(0xc000a04778)
/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc000a04770)
/usr/local/go/src/sync/waitgroup.go:130 +0x65
github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:171
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3e70, 0xc0009e6360, 0xc0002b3e40)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 357 [runnable]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop(0xc00015aa00, 0x3a13fe0, 0xc000a9e630)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3824 +0x264
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3ea0, 0xc0009e6360, 0xc0002b3e90)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 358 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).coalescedHeartbeatsLoop(0xc00015aa00, 0x3a13fe0, 0xc00086c1b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3880 +0x129
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3ee0, 0xc0009e6360, 0xc0002b3ed0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 359 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x3a13fe0, 0xc000a9e540)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1405 +0xe6
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002b3ff0, 0xc0009e6360, 0xc0003d0a80)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 360 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x3a13fe0, 0xc000afc060)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1546 +0x350
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036b680, 0xc0009e6360, 0xc0009d1bd0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 361 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x3a13fe0, 0xc00086c1e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1546 +0x350
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036b6a0, 0xc0009e6360, 0xc0009d1c20)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 362 [runnable]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x3a13fe0, 0xc0007ba390)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1546 +0x350
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036b6b0, 0xc0009e6360, 0xc0009d1c70)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 364 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startLeaseRenewer.func1(0x3a13fe0, 0xc00086c180)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1595 +0x2ea
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036b710, 0xc0009e6360, 0xc00036b6f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 365 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00014d390, 0x256)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00014d380)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc0008a8e00, 0x3a13fe0, 0xc000993740, 0xc000a520c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:292 +0x3d8
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startClosedTimestampRangefeedSubscriber.func1(0x3a13fe0, 0xc000993740)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1616 +0x57
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0009e6360, 0x3a13fe0, 0xc000993740, 0xc0002a6e10, 0x25, 0x0, 0x0, 0xc0003d0aa0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:325 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320 +0x131
goroutine 366 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*Store).startClosedTimestampRangefeedSubscriber.func2(0x3a13fe0, 0xc0007ba360)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1624 +0x2db
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036b720, 0xc0009e6360, 0xc0003d0ac0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 367 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*StoreRebalancer).Start.func1(0x3a13fe0, 0xc00089ecf0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/store_rebalancer.go:186 +0x181
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036b740, 0xc0009e6360, 0xc0003d0ae0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 368 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/storage/compactor.(*Compactor).Start.func1.1(0x3a13fe0, 0xc00089ed20)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/compactor/compactor.go:127 +0x148
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036b750, 0xc0009e6360, 0xc0003d0b20)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 403 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/server.periodicallyPersistHLCUpperBound(0xc000558230, 0xc000255650, 0xc00036bd70, 0x33cac98, 0xc00091c420, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:991 +0x18b
github.com/cockroachdb/cockroach/pkg/server.(*Server).startPersistingHLCUpperBound.func2(0x3a13fe0, 0xc0009d26f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1065 +0x6b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036bd80, 0xc0009e6360, 0xc000b0cb70)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 387 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x3a13fe0, 0xc000b0ca50)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:698 +0x15e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00033c010, 0xc0009e6360, 0xc0003c2020)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 388 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x3a13fe0, 0xc000b0ca20)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:698 +0x15e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00033c020, 0xc0009e6360, 0xc0003c2040)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 389 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x3a13fe0, 0xc000b0ca80)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:698 +0x15e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00033c030, 0xc0009e6360, 0xc0003c2060)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 390 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x3a13fe0, 0xc0007ba3c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:698 +0x15e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00033c040, 0xc0009e6360, 0xc0003c2080)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 391 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x3a13fe0, 0xc000b0c9c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:698 +0x15e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00033c050, 0xc0009e6360, 0xc0003c20a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 392 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x3a13fe0, 0xc000b0c990)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:698 +0x15e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00033c060, 0xc0009e6360, 0xc0003c20c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 393 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x3a13fe0, 0xc000b0c9f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:698 +0x15e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00033c070, 0xc0009e6360, 0xc0003c20e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 394 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x3a13fe0, 0xc00086c360)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:698 +0x15e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00033c080, 0xc0009e6360, 0xc0003c2100)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 402 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1(0x3a13fe0, 0xc00089ed50)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:684 +0x23e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036bd10, 0xc0009e6360, 0xc0003d0de0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 422 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func1(0x3a13fe0, 0xc000902840)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:781 +0x101
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003f0990, 0xc0009e6360, 0xc0003faee0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 369 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1(0x3a13fe0, 0xc0009d26c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:719 +0x11d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036bcb0, 0xc0009e6360, 0xc0003d0da0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 395 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x3a13fe0, 0xc00086c390)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:698 +0x15e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00033c090, 0xc0009e6360, 0xc0003c2120)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 396 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).waitAndProcess(0xc0001a61a0, 0x3a13fe0, 0xc000a9e660, 0xc0009e6360, 0x14487ed0, 0xed4769212, 0x0, 0xc0008dd400, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:221 +0x1d1
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1.1(0xc0008dd400, 0xc000993201)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:283 +0x81
github.com/cockroachdb/cockroach/pkg/storage.(*storeReplicaVisitor).Visit(0xc0009932f0, 0xc001a1ef00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:358 +0x1ab
github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1(0x3a13fe0, 0xc000a9e660)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:281 +0x19d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00033c0b0, 0xc0009e6360, 0xc0003c2140)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 404 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).refreshSettings.func2(0x3a13fe0, 0xc000a9e690)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:114 +0x107
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036bd90, 0xc0009e6360, 0xc0003d0e80)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 405 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startSampleEnvironment.func1(0x3a13fe0, 0xc000a9e6c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1940 +0x182
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036bee0, 0xc0009e6360, 0xc0003d1c20)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 406 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startSampleEnvironment.func2(0x3a13fe0, 0xc0007ba3f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1956 +0x15f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036bef0, 0xc0009e6360, 0xc000b0ccf0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 407 [select]:
github.com/cockroachdb/cockroach/pkg/ts.(*poller).start.func1(0x3a13fe0, 0xc000a9e6f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ts/db.go:183 +0x112
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036bf10, 0xc0009e6360, 0xc00036bf00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 408 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x3a13fe0, 0xc0007ba420)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1763 +0x835
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036bf20, 0xc0009e6360, 0xc0003d1c60)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 409 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*flowScheduler).Start.func1(0x3a13fe0, 0xc0007ba510)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow_scheduler.go:151 +0x14f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036bf60, 0xc0009e6360, 0xc00036bf40)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 410 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*Server).Start.func1(0x3a13fe0, 0xc0007ba540)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:317 +0xfa
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036bf70, 0xc0009e6360, 0xc0003d1c80)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 411 [select, 2 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*Server).PeriodicallyClearStmtStats.func1(0x3a13fe0, 0xc0009d2720)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:674 +0x234
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036bf80, 0xc0009e6360, 0xc0003d1ca0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 412 [select]:
github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1(0x3a13fe0, 0xc0007ba570)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:492 +0x484
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0004c2090, 0xc0009e6360, 0xc000efc0f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 423 [select]:
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func1(0x3a13fe0, 0xc000902030)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:299 +0x12b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003f09a0, 0xc0009e6360, 0xc000afd6e0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 424 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func2(0x3a13fe0, 0xc000594060)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:310 +0xda
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003f09b0, 0xc0009e6360, 0xc0003faf00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 425 [select]:
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func3(0x3a13fe0, 0xc000902870)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:324 +0xfb
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003f09c0, 0xc0009e6360, 0xc000afd710)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 426 [select]:
github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1(0x3a13fe0, 0xc0009028d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:249 +0x193
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003f09d0, 0xc0009e6360, 0xc0003faf20)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 119 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).addToPurgatoryLocked.func2(0x3a13fe0, 0xc000a9fa40)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:1033 +0x325
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0005a45b0, 0xc0009e6360, 0xc0003e4c00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 1299 [select, 1 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0016abc40)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x110
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x13e
goroutine 1257 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).Next(0xc000b231a0, 0x3a13fe0, 0xc001292600, 0xc00014e480, 0xc00156fe50, 0x3, 0x3, 0x3a66d00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:151 +0x116
github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Next(0xc001620a00, 0xc0017422d0, 0x3, 0x3, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:175 +0x43c
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.Run(0x3a13fe0, 0xc001292600, 0x3a1e820, 0xc001620a00, 0x39f48a0, 0xc0015af880)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/base.go:175 +0x35
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*ProcessorBase).Run(0xc001620a00, 0x3a13fe0, 0xc001292600)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/processors.go:808 +0x92
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*Flow).Run(0xc001a14f00, 0x3a13fe0, 0xc001292600, 0x33c55d8, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:626 +0x1ec
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc000343180, 0xc000b23200, 0xc0017421b0, 0xc00107a868, 0xc0019fa2c0, 0xc00156ff28, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:253 +0x8fe
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0xc000343180, 0x3a13fe0, 0xc001292360, 0xc00156ff28, 0xc000b23200, 0xc0017421b0, 0x3a15820, 0xc000b231a0, 0xc0019fa2c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:841 +0x1fb
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc00156fb00, 0x3a13fe0, 0xc001292360, 0xc00156fe50, 0x3, 0x7f4ad7638a80, 0xc0019dcf20, 0xc00107aa00, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:898 +0x284
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc00156fb00, 0x3a13fe0, 0xc001292360, 0xc00156fe50, 0x7f4ad7638a80, 0xc0019dcf20, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:733 +0x6ce
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc00156fb00, 0x3a13fe0, 0xc001292360, 0x3a184a0, 0xc0019daa00, 0xc0006332de, 0x25, 0x0, 0x1, 0xc000b23020, ...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:414 +0xb05
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc00156fb00, 0x3a13fe0, 0xc001292360, 0x3a184a0, 0xc0019daa00, 0xc0006332de, 0x25, 0x0, 0x1, 0xc000b23020, ...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:100 +0x5f2
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc00156fb00, 0x3a13f20, 0xc00181f880, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1260 +0xf17
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc00156fb00, 0x3a13f20, 0xc0015dfa40, 0xc00014e338, 0x5400, 0x15000, 0xc00014e3d0, 0xc00164e390, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1137 +0x1a3
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc000c44000, 0x3a13f20, 0xc0015dfa40, 0xc00156fb00, 0x5400, 0x15000, 0xc00014e3d0, 0xc00164e390, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:437 +0xce
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc0014a19e1, 0xc0017eab40, 0x3a13f20, 0xc0015dfa40, 0xc00164e390, 0xc000c44000, 0xc0017fc000, 0x3a177a0, 0xc0017eab20, 0xc0016172c0, ...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:580 +0x21c
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:515 +0x17b
goroutine 831 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changeFrontier).Start.func1(0x3a13fe0, 0xc001293590, 0xc0019ee800)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:475 +0x48
created by github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changeFrontier).Start
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:474 +0x384
goroutine 832 [runnable]:
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*poller).pollTableHistory(0xc000909520, 0x3a13f20, 0xc0016aa240, 0xc000993710, 0x39dee60)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/poller.go:549 +0x171
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1(0x8, 0x33caa68)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:170 +0x3a
github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc001293800, 0xc001293830)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x57
created by github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66
goroutine 1315 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*controlBuffer).get(0xc00108e640, 0x1, 0x0, 0x0, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:300 +0x104
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*loopyWriter).run(0xc001656cc0, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:387 +0x1b6
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc0014d0000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:303 +0x7b
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:301 +0xd09
goroutine 1284 [select]:
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*poller).rangefeedImpl.func2(0x3a13f20, 0xc0016ab0c0, 0xc000993710, 0xc000118f90)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/poller.go:292 +0x11c
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1(0x8, 0x33caa68)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:170 +0x3a
github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc00172cd80, 0xc00172ce40)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x57
created by github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66
goroutine 1298 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc000592420, 0xc00166bb00, 0xc0002c0010, 0xf, 0xc000f162a0, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:863 +0x276
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x3a13fe0, 0xc00172def0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:785 +0x79
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0011deb20, 0xc0009e6360, 0xc0002f60f0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 764 [select, 1 minutes]:
database/sql.(*DB).connectionOpener(0xc0006a4300, 0x3a13f20, 0xc00108f1c0)
/usr/local/go/src/database/sql/sql.go:1000 +0xe8
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:670 +0x15e
goroutine 765 [select, 1 minutes]:
database/sql.(*DB).connectionResetter(0xc0006a4300, 0x3a13f20, 0xc00108f1c0)
/usr/local/go/src/database/sql/sql.go:1013 +0xfb
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:671 +0x194
goroutine 1249 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).keepalive(0xc0014d0000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1184 +0x113
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:265 +0x1455
goroutine 830 [semacquire, 1 minutes]:
sync.runtime_Semacquire(0xc001293810)
/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc001293808)
/usr/local/go/src/sync/waitgroup.go:130 +0x65
github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Wait(0xc001293800, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:40 +0x31
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait(0xc001293800, 0x3a13f20, 0xc0016aa240, 0xc0011de380, 0xc0016aa240)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:146 +0x57
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*poller).RunUsingRangefeeds(0xc000909520, 0x3a13f20, 0xc0016aa180, 0x6c2bdf, 0xc000b201b0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/poller.go:203 +0x12d
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changeAggregator).Start.func1(0x3a13f20, 0xc0016aa180)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:195 +0xbc
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0009e6360, 0x3a13f20, 0xc0016aa180, 0xc0004a6a00, 0x19, 0x3a60180, 0xc0003ac320, 0xc0011de330)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:325 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320 +0x131
goroutine 1300 [select, 1 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc0019fadc0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1384 +0x1c4
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc0019fadc0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:960 +0x1d9
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:951 +0xe1
goroutine 1283 [semacquire, 1 minutes]:
sync.runtime_Semacquire(0xc00172cee0)
/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc00172ced8)
/usr/local/go/src/sync/waitgroup.go:130 +0x65
github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Wait(0xc00172ced0, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:40 +0x31
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait(0xc00172ced0, 0x3a13f20, 0xc0016ab1c0, 0xc00166a480, 0xc0016ab1c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:146 +0x57
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).RangeFeed(0xc0004fd200, 0x3a13f20, 0xc0016ab0c0, 0xc0003bc798, 0x2, 0x8, 0xc0003bc7a0, 0x2, 0x2, 0x0, ...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender_rangefeed.go:85 +0x3e4
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*poller).rangefeedImpl.func1(0x3a13f20, 0xc0016ab0c0, 0x0, 0xc000117f90)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/poller.go:287 +0xd4
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1(0x8, 0x33caa68)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:170 +0x3a
github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc00172cd80, 0xc00172cde0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x57
created by github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66
goroutine 1256 [IO wait]:
internal/poll.runtime_pollWait(0x7f4ad7716a28, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0009b9618, 0x72, 0x700, 0x719, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0009b9600, 0xc0017f2000, 0x719, 0x719, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc0009b9600, 0xc0017f2000, 0x719, 0x719, 0xc001793500, 0x39aaf01, 0x5a65bc0)
/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0013c0470, 0xc0017f2000, 0x719, 0x719, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:177 +0x69
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc0016169c0, 0xc0017f2000, 0x719, 0x719, 0x0, 0x39ab260, 0x10)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111
crypto/tls.(*atLeastReader).Read(0xc0004342a0, 0xc0017f2000, 0x719, 0x719, 0x0, 0x39ab260, 0xc001793618)
/usr/local/go/src/crypto/tls/conn.go:761 +0x60
bytes.(*Buffer).ReadFrom(0xc00116e258, 0x39a5f60, 0xc0004342a0, 0x6a05b5, 0x2ebbd80, 0x30901e0)
/usr/local/go/src/bytes/buffer.go:207 +0xbd
crypto/tls.(*Conn).readFromUntil(0xc00116e000, 0x7f4ad1a0b048, 0xc0016169c0, 0x5, 0xc0016169c0, 0xc0017936e0)
/usr/local/go/src/crypto/tls/conn.go:783 +0xf8
crypto/tls.(*Conn).readRecordOrCCS(0xc00116e000, 0x33caa00, 0xc00116e138, 0xed4769222)
/usr/local/go/src/crypto/tls/conn.go:590 +0x125
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:558
crypto/tls.(*Conn).Read(0xc00116e000, 0xc0017fb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1236 +0x137
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*readTimeoutConn).Read(0xc0017eaae0, 0xc0017fb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:1939 +0x110
bufio.(*Reader).fill(0xc0017fc048)
/usr/local/go/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).ReadByte(0xc0017fc048, 0xc0017fc618, 0xc001805b00, 0x1)
/usr/local/go/src/bufio/bufio.go:249 +0x39
github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase.(*ReadBuffer).ReadTypedMsg(0xc0017fc6f0, 0x39e9020, 0xc0017fc048, 0x39dec20, 0x5a66340, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase/encoding.go:121 +0x35
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl(0xc0017fc000, 0x3a13f20, 0xc0015df700, 0xc00164e370, 0xc000c44000, 0x5400, 0x15000, 0xc00014e3d0, 0x0, 0x0, ...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:315 +0x474
github.com/cockroachdb/cockroach/pkg/sql/pgwire.serveConn(0x3a13f20, 0xc0015df700, 0x3a47fa0, 0xc00116e000, 0xc00116d00d, 0x4, 0xc00146ba40, 0x39dd7e0, 0xc00146af90, 0x4000, ...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:171 +0x18d
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc00014e240, 0x3a13f20, 0xc0015df700, 0x3a47fa0, 0xc00116e000, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:528 +0x87e
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func20.1(0x3a48840, 0xc0016169c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1721 +0x163
github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith.func1(0xc0009e6360, 0x3a13fe0, 0xc0011be000, 0xc0004aa000, 0x3a48840, 0xc0016169c0, 0xc0011be030)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:139 +0xa9
created by github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:136 +0x222
goroutine 1165 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/storage/idalloc.(*Allocator).start.func1(0x3a13fe0, 0xc00172c660)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/idalloc/id_alloc.go:127 +0x436
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00036be50, 0xc0009e6360, 0xc00036be40)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 1289 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).RangeFeed(0xc001621400, 0xc001710d20, 0x3a40bc0, 0xc0016b9380, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_rangefeed.go:233 +0x4c3
github.com/cockroachdb/cockroach/pkg/storage.(*Store).RangeFeed(0xc00015aa00, 0xc001710d20, 0x3a40bc0, 0xc0016b9380, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3131 +0xf9
github.com/cockroachdb/cockroach/pkg/storage.(*Stores).RangeFeed(0xc00032eaa0, 0xc001710d20, 0x3a40bc0, 0xc0016b9380, 0xc000119770)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:210 +0xe0
github.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0xc000503200, 0xc001710d20, 0x3a40bc0, 0xc0016b9380, 0xc000119778, 0x16fdc7a)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:1026 +0x54
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1(0xc0011de7d0, 0x39d2ba0, 0xc000503200, 0xc001710d20, 0x3a13f20, 0xc0016ab2c0, 0xc00166a540, 0xc00166a5a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:567 +0xbe
created by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:565 +0x11d
goroutine 1314 [IO wait]:
internal/poll.runtime_pollWait(0x7f4ad7716958, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0008a6a98, 0x72, 0x700, 0x791, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0008a6a80, 0xc000a75000, 0x791, 0x791, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc0008a6a80, 0xc000a75000, 0x791, 0x791, 0x203000, 0x0, 0x784)
/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0019e0488, 0xc000a75000, 0x791, 0x791, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:177 +0x69
crypto/tls.(*atLeastReader).Read(0xc00045c880, 0xc000a75000, 0x791, 0x791, 0xc001945af0, 0x6da682, 0xc001945ac8)
/usr/local/go/src/crypto/tls/conn.go:761 +0x60
bytes.(*Buffer).ReadFrom(0xc000a92958, 0x39a5f60, 0xc00045c880, 0x6a05b5, 0x2ebbd80, 0x31c9cc0)
/usr/local/go/src/bytes/buffer.go:207 +0xbd
crypto/tls.(*Conn).readFromUntil(0xc000a92700, 0x39ab2a0, 0xc0019e0488, 0x5, 0xc0019e0488, 0x7f4ad776a6d0)
/usr/local/go/src/crypto/tls/conn.go:783 +0xf8
crypto/tls.(*Conn).readRecordOrCCS(0xc000a92700, 0x33caa00, 0xc000a92838, 0x69bd2f)
/usr/local/go/src/crypto/tls/conn.go:590 +0x125
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:558
crypto/tls.(*Conn).Read(0xc000a92700, 0xc0010a4000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1236 +0x137
bufio.(*Reader).Read(0xc0009df6e0, 0xc001756498, 0x9, 0x9, 0xc00035c000, 0x7f4ad776a6d0, 0x0)
/usr/local/go/src/bufio/bufio.go:223 +0x22f
io.ReadAtLeast(0x39a5d60, 0xc0009df6e0, 0xc001756498, 0x9, 0x9, 0x9, 0xc27da4, 0xc0019e0460, 0x3a13fe0)
/usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(...)
/usr/local/go/src/io/io.go:329
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc001756498, 0x9, 0x9, 0x39a5d60, 0xc0009df6e0, 0x0, 0x0, 0xc0014d0000, 0xc001945edd)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc001756460, 0xc001891530, 0xc001891530, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc0014d0000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:1135 +0x123
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Client
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_client.go:270 +0xba9
goroutine 1305 [IO wait]:
internal/poll.runtime_pollWait(0x7f4ad7716888, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc001141098, 0x72, 0x700, 0x75b, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc001141080, 0xc00038c800, 0x75b, 0x75b, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc001141080, 0xc00038c800, 0x75b, 0x75b, 0xc000c1d918, 0x2a00000701, 0x2000)
/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0006e2160, 0xc00038c800, 0x75b, 0x75b, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:177 +0x69
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc001656c00, 0xc00038c800, 0x75b, 0x75b, 0x0, 0x0, 0x10)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111
crypto/tls.(*atLeastReader).Read(0xc001712860, 0xc00038c800, 0x75b, 0x75b, 0x40, 0x8, 0xc000c1da30)
/usr/local/go/src/crypto/tls/conn.go:761 +0x60
bytes.(*Buffer).ReadFrom(0xc000a76cd8, 0x39a5f60, 0xc001712860, 0x6a05b5, 0x2ebbd80, 0x30901e0)
/usr/local/go/src/bytes/buffer.go:207 +0xbd
crypto/tls.(*Conn).readFromUntil(0xc000a76a80, 0x7f4ad1a0b048, 0xc001656c00, 0x5, 0xc001656c00, 0x1)
/usr/local/go/src/crypto/tls/conn.go:783 +0xf8
crypto/tls.(*Conn).readRecordOrCCS(0xc000a76a80, 0x33caa00, 0xc000a76bb8, 0xc001c01bf0)
/usr/local/go/src/crypto/tls/conn.go:590 +0x125
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:558
crypto/tls.(*Conn).Read(0xc000a76a80, 0xc00124c000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1236 +0x137
bufio.(*Reader).Read(0xc0009de120, 0xc0017563b8, 0x9, 0x9, 0xc001bbd408, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:223 +0x22f
io.ReadAtLeast(0x39a5d60, 0xc0009de120, 0xc0017563b8, 0x9, 0x9, 0x9, 0x63943516c8c65901, 0x6300000000000000, 0x0)
/usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(...)
/usr/local/go/src/io/io.go:329
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0017563b8, 0x9, 0x9, 0x39a5d60, 0xc0009de120, 0x0, 0x7900000000, 0xc00102cb00, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc001756380, 0xc000992570, 0xc000992570, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc000c36200, 0xc0013149c0, 0x33c9c38)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:434 +0x7c
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc000251dc0, 0x3a4c9e0, 0xc000c36200)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:676 +0xdd
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func2()
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:631 +0x3c
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func3(0xc0002c24c0, 0xc000251dc0, 0x7f4ad1a0b028, 0xc000c36200)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:639 +0x27
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:638 +0x5e4
goroutine 761 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startSystemLogsGC.func1(0x3a13fe0, 0xc0009c0600)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server_systemlog_gc.go:175 +0x539
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013ccd50, 0xc0009e6360, 0xc000465020)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 1304 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*http2Server).keepalive(0xc000c36200)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:875 +0x1ed
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:283 +0xe89
goroutine 1303 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*controlBuffer).get(0xc00108e600, 0x1, 0x0, 0x0, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:300 +0x104
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.(*loopyWriter).run(0xc001656d20, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/controlbuf.go:387 +0x1b6
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server.func2(0xc000c36200)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:277 +0xcb
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport.newHTTP2Server
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/transport/http2_server.go:274 +0xe64
goroutine 601 [sync.Cond.Wait, 1 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc00103c250, 0x22f942ad0000000e)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc00103c240)
/usr/local/go/src/sync/cond.go:56 +0x91
github.com/cockroachdb/cockroach/pkg/sql.(*StmtBuf).curCmd(0xc0004db618, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_io.go:441 +0x9d
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc0014ee380, 0x3a13f20, 0xc00103c280, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1161 +0x52
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc0014ee380, 0x3a13f20, 0xc00103c280, 0xc00014e338, 0x5400, 0x15000, 0xc00014e3d0, 0xc000463ca0, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1137 +0x1a3
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc000c44000, 0x3a13f20, 0xc00103c280, 0xc0014ee380, 0x5400, 0x15000, 0xc00014e3d0, 0xc000463ca0, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:437 +0xce
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc000253651, 0xc0003c2fc0, 0x3a13f20, 0xc00103c280, 0xc000463ca0, 0xc000c44000, 0xc0004db000, 0x3a177a0, 0xc0003c2fa0, 0xc0009abb60, ...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:580 +0x21c
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:515 +0x17b
goroutine 760 [chan receive, 1 minutes]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith(0xc0004aa000, 0x3a13fe0, 0xc0011be000, 0xc0009e6360, 0x39ff3a0, 0xc0004080c0, 0xc0011be030, 0x7f4ad776c1f0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:118 +0x50
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func20(0x3a13fe0, 0xc0011be000)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1717 +0x195
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0013ccd40, 0xc0009e6360, 0xc00108f080)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 1293 [select]:
github.com/cockroachdb/cockroach/pkg/storage/rangefeed.(*registration).outputLoop(0xc001284700, 0x3a13f20, 0xc0016ab3c0, 0xc0016ab3c0, 0xc0011de870)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/rangefeed/registry.go:184 +0x1f8
github.com/cockroachdb/cockroach/pkg/storage/rangefeed.(*registration).runOutputLoop(0xc001284700, 0x3a13f20, 0xc0016ab3c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/rangefeed/registry.go:201 +0xc2
github.com/cockroachdb/cockroach/pkg/storage/rangefeed.(*Processor).Start.func1.1(0x3a13f20, 0xc0016ab300)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/rangefeed/processor.go:220 +0x64
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0009e6360, 0x3a13f20, 0xc0016ab300, 0xc0004a7140, 0x1e, 0x0, 0x0, 0xc0016b94c0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:325 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320 +0x131
goroutine 1290 [select]:
github.com/cockroachdb/cockroach/pkg/storage/rangefeed.(*Processor).Start.func1(0x3a13fe0, 0xc00172d230)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/storage/rangefeed/processor.go:202 +0x3e1
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0011de830, 0xc0009e6360, 0xc00172d200)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:200 +0xfb
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:193 +0xa8
goroutine 1297 [select, 1 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc00172dec0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:109 +0x13f
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccResolverWrapper).start
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:95 +0x3f
goroutine 833 [semacquire, 1 minutes]:
sync.runtime_Semacquire(0xc00172cd90)
/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc00172cd88)
/usr/local/go/src/sync/waitgroup.go:130 +0x65
github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Wait(0xc00172cd80, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:40 +0x31
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait(0xc00172cd80, 0x3a13f20, 0xc0016ab0c0, 0xc0016b92c0, 0xc0003bc7a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:146 +0x57
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*poller).rangefeedImpl(0xc000909520, 0x3a13f20, 0xc0016aa240, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/poller.go:369 +0x361
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1(0x8, 0x33caa68)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:170 +0x3a
github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc001293800, 0xc001293860)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x57
created by github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66
goroutine 1285 [select]:
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*memBuffer).getRow(0xc00025f680, 0x3a13f20, 0xc0016ab0c0, 0x6f06a0, 0xc001946af0, 0x2, 0x2, 0xc001946aec)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/buffer.go:236 +0x1d0
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*memBuffer).Get(0xc00025f680, 0x3a13f20, 0xc0016ab0c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/buffer.go:176 +0x7b
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*poller).rangefeedImpl.func3(0x3a13f20, 0xc0016ab0c0, 0x1ca552e9f7024f47, 0xc000119790)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/poller.go:320 +0x83
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1(0xb8b2333300000008, 0x33caa68)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:170 +0x3a
github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc00172cd80, 0xc00172ce70)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x57
created by github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66
goroutine 1286 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).RangeFeed.func1(0x3a13f20, 0xc0016ab1c0, 0x0, 0xc000117f90)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender_rangefeed.go:68 +0x13d
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1(0x8, 0x33caa68)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:170 +0x3a
github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc00172ced0, 0xc00172cf00)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x57
created by github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66
goroutine 1288 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(0x3a13f20, 0xc0016ab2c0, 0xc00166a540, 0xc00166a5a0, 0xc00151ca1c, 0x2, 0x2)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:513 +0xd1
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).singleRangeFeed(0xc0004fd200, 0x3a13f20, 0xc0016ab1c0, 0xc0003bc798, 0x2, 0x8, 0xc0003bc7a0, 0x2, 0x2, 0x0, ...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender_rangefeed.go:257 +0x4d4
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).partialRangeFeed(0xc0004fd200, 0x3a13f20, 0xc0016ab1c0, 0xc0016bcfa0, 0xc00166a420, 0xc00166a0c0, 0x2, 0x2)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender_rangefeed.go:149 +0x2c8
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).RangeFeed.func1.1(0x3a13f20, 0xc0016ab1c0, 0x1ca552e9f7024f47, 0xc000119790)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender_rangefeed.go:72 +0x59
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1(0xb8b2333300000008, 0x33caa68)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:170 +0x3a
github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc00172ced0, 0xc00172cfc0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x57
created by github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66
goroutine 829 [select]:
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*buffer).Get(0xc0019e02e0, 0x3a13f20, 0xc0016aa180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/buffer.go:73 +0x103
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.kvsToRows.func2(0x3a13f20, 0xc0016aa180, 0xed4769222, 0x0, 0x1, 0x478cc93b, 0xc00114a370)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed.go:131 +0x89
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.emitEntries.func2(0x3a13f20, 0xc0016aa180, 0x0, 0x0, 0x1, 0xc00033f4a0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed.go:215 +0xc0
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changeAggregator).tick(0xc000a23b80, 0x1, 0x8)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:271 +0x5a
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changeAggregator).Next(0xc000a23b80, 0x0, 0x0, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:253 +0x77
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changeFrontier).Next(0xc0019ee800, 0xc001237680, 0x4, 0x4, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:517 +0x99
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.Run(0x3a13fe0, 0xc001293590, 0x3a1e4a0, 0xc0019ee800, 0x39f48a0, 0xc000b53180)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/base.go:175 +0x35
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*ProcessorBase).Run(0xc0019ee800, 0x3a13fe0, 0xc001293590)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/processors.go:808 +0x92
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*Flow).Run(0xc001a154a0, 0x3a13fe0, 0xc001293590, 0x33c55d8, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:626 +0x1ec
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc000343180, 0xc000b23500, 0x0, 0xc00114b2f8, 0xc0019fa840, 0xc00014eb40, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:253 +0x8fe
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.distChangefeedFlow(0x3a13fe0, 0xc001292600, 0x3a7fee0, 0xc00156fe50, 0x0, 0xc0012930b0, 0x0, 0x0, 0xc001292690, 0x15a0da76b4899ca3, ...)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_dist.go:198 +0xd15
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.changefeedPlanHook.func2(0x3a13fe0, 0xc001292600, 0x0, 0x0, 0x0, 0xc00091d440, 0x0, 0x0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_stmt.go:280 +0xbe3
github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec.func1(0xc000b231a0, 0x3a13fe0, 0xc001292600, 0xc00014e480, 0xc00156fe50)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:139 +0x7c
created by github.com/cockroachdb/cockroach/pkg/sql.(*hookFnNode).startExec
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/planhook.go:138 +0xcc
goroutine 2457 [select]:
github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1.1(0x3a13fe0, 0xc0009028d0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:261 +0x11f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0009e6360, 0x3a13fe0, 0xc0009028d0, 0xc0013d31a0, 0x2a, 0x0, 0x0, 0xc001fab1a0)
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:325 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
/home/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:320 +0x131
ERROR: exit status 2
6421 runs completed, 1 failures, over 7m17s
context canceled
FAIL
FAIL github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl 436.592s
Makefile:907: recipe for target 'stress' failed
make: *** [stress] Error 1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment