Skip to content

Instantly share code, notes, and snippets.

@tbg

tbg/gist:798d2b207b67925350bdecc628ee81f1

Created March 6, 2020 11:10
Show Gist options
  • Save tbg/798d2b207b67925350bdecc628ee81f1 to your computer and use it in GitHub Desktop.
Save tbg/798d2b207b67925350bdecc628ee81f1 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
------- Stdout: -------
=== RUN TestDropDatabaseDeleteData
I200306 10:45:57.367324 195580 base/addr_validation.go:296 [n?] server certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node
I200306 10:45:57.367466 195580 base/addr_validation.go:342 [n?] web UI certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node
W200306 10:45:57.417251 195580 server/status/runtime.go:308 [n?] Could not parse build timestamp: parsing time "" as "2006/01/02 15:04:05": cannot parse "" as "2006"
I200306 10:45:57.427202 195580 server/server.go:1034 [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled
I200306 10:45:57.452255 195580 server/config.go:572 [n?] 1 storage engine initialized
I200306 10:45:57.452281 195580 server/config.go:575 [n?] RocksDB cache size: 128 MiB
I200306 10:45:57.452290 195580 server/config.go:575 [n?] store 0: in-memory, size 0 B
I200306 10:45:57.452386 195580 server/server.go:1486 [n?] no stores bootstrapped and --join flag specified, awaiting init command or join with an already initialized node.
I200306 10:45:57.452415 195580 server/server.go:1496 [n?] **** add additional nodes by specifying --join=127.0.0.1:35915
I200306 10:45:57.456622 195580 server/node.go:324 [n?] **** cluster 4464e02d-0567-4b3c-9803-931ed07cd57b has been created
I200306 10:45:57.456800 195580 gossip/gossip.go:395 [n1] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:35915" > attrs:<> locality:<> ServerVersion:<major_val:19 minor_val:2 patch:0 unstable:14 > build_tag:"v20.1.0-beta.2-469-g450e122" started_at:1583491557456718037 cluster_name:"" sql_address:<network_field:"tcp" address_field:"127.0.0.1:45333" >
I200306 10:45:57.464559 195580 server/node.go:428 [n1] initialized store [n1,s1]: disk (capacity=512 MiB, available=512 MiB, used=0 B, logicalBytes=16 KiB), ranges=32, leases=0, queries=0.00, writes=0.00, bytesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=166.00 pMax=13265.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00 pMax=0.00}
I200306 10:45:57.464641 195580 kv/kvserver/stores.go:234 [n1] read 0 node addresses from persistent storage
I200306 10:45:57.464705 195580 server/node.go:645 [n1] connecting to gossip network to verify cluster ID...
W200306 10:45:57.465330 195758 kv/kvserver/store.go:1541 [n1,s1,r6/1:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r6: replica (n1,s1):1 not lease holder; lease holder unknown
I200306 10:45:57.465391 195580 server/node.go:665 [n1] node connected via gossip and verified as part of cluster "4464e02d-0567-4b3c-9803-931ed07cd57b"
I200306 10:45:57.465480 195580 server/node.go:512 [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I200306 10:45:57.465533 195580 server/server.go:2180 [n1] Could not start goroutine dumper worker due to: directory to store dumps could not be determined
I200306 10:45:57.465598 195580 server/server.go:1616 [n1] starting https server at 127.0.0.1:35079 (use: 127.0.0.1:35079)
I200306 10:45:57.465622 195580 server/server.go:1621 [n1] starting postgres server at 127.0.0.1:45333 (use: 127.0.0.1:45333)
I200306 10:45:57.465635 195580 server/server.go:1623 [n1] starting grpc server at 127.0.0.1:35915
I200306 10:45:57.465647 195580 server/server.go:1624 [n1] advertising CockroachDB node at 127.0.0.1:35915
I200306 10:45:57.511266 195775 sql/event_log.go:132 [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root}
I200306 10:45:57.527739 195878 sql/event_log.go:132 [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:19.2-14 User:root}
I200306 10:45:57.539032 195719 kv/kvserver/queue.go:578 [n1,s1,r7/1:/Table/1{1-2}] rate limited in MaybeAdd (merge): throttled on async limiting semaphore
I200306 10:45:57.545609 196270 sql/event_log.go:132 [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:f26fa3ed-82ff-4a01-96a7-b16662a43bd5 User:root}
I200306 10:45:57.551650 196230 sql/event_log.go:132 [n1,intExec=create-default-db] Event: "create_database", target: 50, info: {DatabaseName:defaultdb Statement:CREATE DATABASE IF NOT EXISTS defaultdb User:root}
I200306 10:45:57.572921 196210 sql/event_log.go:132 [n1,intExec=create-default-db] Event: "create_database", target: 51, info: {DatabaseName:postgres Statement:CREATE DATABASE IF NOT EXISTS postgres User:root}
I200306 10:45:57.597570 196565 sql/sqlbase/structured.go:1554 [n1,intExec=depublicize-system-comments] publish: descID=24 (comments) version=2 mtime=1970-01-01 00:00:00 +0000 UTC
I200306 10:45:57.602137 196565 sql/schema_changer.go:973 [n1,intExec=depublicize-system-comments,scExec] schema change on comments (24 v2) mutation 0 starting execution...
I200306 10:45:57.602156 196565 sql/schema_changer.go:1263 [n1,intExec=depublicize-system-comments,scExec] waiting for a single version of table 24...
I200306 10:45:57.605297 196565 sql/schema_changer.go:1265 [n1,intExec=depublicize-system-comments,scExec] waiting for a single version of table 24... done (at v 2)
I200306 10:45:57.625148 196570 sql/sqlbase/structured.go:1554 [n1,intExec=depublicize-system-comments] publish: descID=24 (comments) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200306 10:45:57.639088 196570 sql/schema_changer.go:973 [n1,intExec=depublicize-system-comments,scExec] schema change on comments (24 v3) mutation 0 starting execution...
I200306 10:45:57.639120 196570 sql/schema_changer.go:1263 [n1,intExec=depublicize-system-comments,scExec] waiting for a single version of table 24...
I200306 10:45:57.640743 196570 sql/schema_changer.go:1265 [n1,intExec=depublicize-system-comments,scExec] waiting for a single version of table 24... done (at v 3)
I200306 10:45:57.643495 196707 sql/sqlbase/structured.go:1554 [n1,intExec=depublicize-system-comments] publish: descID=24 (comments) version=4 mtime=1970-01-01 00:00:00 +0000 UTC
I200306 10:45:57.650598 196707 sql/schema_changer.go:973 [n1,intExec=depublicize-system-comments,scExec] schema change on comments (24 v4) mutation 0 starting execution...
I200306 10:45:57.650799 196707 sql/schema_changer.go:1263 [n1,intExec=depublicize-system-comments,scExec] waiting for a single version of table 24...
I200306 10:45:57.659216 196707 sql/schema_changer.go:1265 [n1,intExec=depublicize-system-comments,scExec] waiting for a single version of table 24... done (at v 4)
I200306 10:45:57.660982 196712 sql/sqlbase/structured.go:1554 [n1,intExec=depublicize-system-comments] publish: descID=24 (comments) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200306 10:45:57.680628 196712 sql/schema_changer.go:973 [n1,intExec=depublicize-system-comments,scExec] schema change on comments (24 v5) mutation 0 starting execution...
I200306 10:45:57.680715 196712 sql/schema_changer.go:1263 [n1,intExec=depublicize-system-comments,scExec] waiting for a single version of table 24...
I200306 10:45:57.690177 196712 sql/schema_changer.go:1265 [n1,intExec=depublicize-system-comments,scExec] waiting for a single version of table 24... done (at v 5)
I200306 10:45:57.694730 195580 server/server.go:1702 [n1] done ensuring all necessary migrations have run
I200306 10:45:57.694858 195580 server/server.go:1985 [n1] serving sql connections
I200306 10:45:57.697068 196854 server/server_update.go:55 [n1] no need to upgrade, cluster already at the newest version
I200306 10:45:57.701568 196856 sql/event_log.go:132 [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:127.0.0.1:35915 Attrs: Locality: ServerVersion:19.2-14 BuildTag:v20.1.0-beta.2-469-g450e122 StartedAt:1583491557456718037 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:45333} ClusterID:4464e02d-0567-4b3c-9803-931ed07cd57b StartedAt:1583491557456718037 LastUp:1583491557456718037}
I200306 10:45:57.729563 196719 sql/event_log.go:132 [n1,client=127.0.0.1:44448,user=root] Event: "create_database", target: 52, info: {DatabaseName:t Statement:CREATE DATABASE t User:root}
I200306 10:45:57.743231 196719 sql/event_log.go:132 [n1,client=127.0.0.1:44448,user=root] Event: "create_table", target: 53, info: {TableName:t.public.kv Statement:CREATE TABLE t.public.kv (k CHAR PRIMARY KEY, v CHAR, FAMILY (k), FAMILY (v)) User:root}
I200306 10:45:57.745917 196680 kv/kvserver/replica_command.go:397 [n1,split,s1,r32/1:/{Table/36-Max}] initiating a split of this range at key /Table/53 [r33] (zone config)
I200306 10:45:57.759870 196719 sql/event_log.go:132 [n1,client=127.0.0.1:44448,user=root] Event: "create_table", target: 54, info: {TableName:t.public.kv2 Statement:CREATE TABLE t.public.kv2 (k CHAR PRIMARY KEY, v CHAR, FAMILY (k), FAMILY (v)) User:root}
I200306 10:45:57.764334 196951 kv/kvserver/replica_command.go:397 [n1,split,s1,r33/1:/{Table/53-Max}] initiating a split of this range at key /Table/54 [r34] (zone config)
I200306 10:45:57.796087 196719 sql/sqlbase/structured.go:1554 [n1,client=127.0.0.1:44448,user=root] publish: descID=53 (kv) version=2 mtime=1970-01-01 00:00:00 +0000 UTC
I200306 10:45:57.806554 196719 sql/sqlbase/structured.go:1554 [n1,client=127.0.0.1:44448,user=root] publish: descID=54 (kv2) version=2 mtime=1970-01-01 00:00:00 +0000 UTC
I200306 10:45:59.065288 195671 gossip/gossip.go:1532 [n1] node has connected to cluster via gossip
I200306 10:45:59.065449 195671 kv/kvserver/stores.go:253 [n1] wrote 0 node addresses to persistent storage
I200306 10:46:07.469098 195690 server/status/runtime.go:498 [n1] runtime stats: 315 MiB RSS, 188 goroutines, 63 MiB/178 MiB/118 MiB GO alloc/idle/total, 18 MiB/50 MiB CGO alloc/total, 0.0 CGO/sec, 0.0/0.0 %(u/s)time, 0.0 %gc (249x), 7.0 MiB/7.0 MiB (r/w)net
I200306 10:46:16.498445 197601 kv/kvserver/replica_consistency.go:255 [n1,consistencyChecker,s1,r4/1:/System{/tsd-tse}] triggering stats recomputation to resolve delta of {ContainsEstimates:2740 LastUpdateNanos:1583491567544019260 IntentAge:0 GCBytesAge:0 LiveBytes:-7715 LiveCount:-658 KeyBytes:-31967 KeyCount:-658 ValBytes:24252 ValCount:-658 IntentBytes:0 IntentCount:0 SysBytes:0 SysCount:0}
I200306 10:46:17.466206 195685 kv/kvserver/store.go:2383 [n1,s1] sstables (read amplification = 0):
I200306 10:46:17.466366 195685 kv/kvserver/store.go:2384 [n1,s1]
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
** Compaction Stats [default] **
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Uptime(secs): 20.0 total, 20.0 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
estimated_pending_compaction_bytes: 0 B
I200306 10:46:17.468602 195690 server/status/runtime.go:498 [n1] runtime stats: 316 MiB RSS, 188 goroutines, 71 MiB/171 MiB/118 MiB GO alloc/idle/total, 18 MiB/51 MiB CGO alloc/total, 239.1 CGO/sec, 0.9/0.3 %(u/s)time, 0.0 %gc (0x), 10 MiB/10 MiB (r/w)net
I200306 10:46:27.468835 195690 server/status/runtime.go:498 [n1] runtime stats: 315 MiB RSS, 188 goroutines, 78 MiB/165 MiB/122 MiB GO alloc/idle/total, 18 MiB/50 MiB CGO alloc/total, 95.1 CGO/sec, 0.7/0.3 %(u/s)time, 0.0 %gc (0x), 17 MiB/17 MiB (r/w)net
I200306 10:46:37.469265 195690 server/status/runtime.go:498 [n1] runtime stats: 316 MiB RSS, 190 goroutines, 48 MiB/182 MiB/125 MiB GO alloc/idle/total, 19 MiB/51 MiB CGO alloc/total, 55.5 CGO/sec, 1.4/0.1 %(u/s)time, 0.0 %gc (1x), 2.7 MiB/2.7 MiB (r/w)net
I200306 10:46:47.470698 195690 server/status/runtime.go:498 [n1] runtime stats: 317 MiB RSS, 190 goroutines, 55 MiB/176 MiB/125 MiB GO alloc/idle/total, 19 MiB/51 MiB CGO alloc/total, 45.8 CGO/sec, 0.7/0.2 %(u/s)time, 0.0 %gc (0x), 5.3 MiB/5.3 MiB (r/w)net
I200306 10:46:57.465755 195686 gossip/gossip.go:567 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
gossip connectivity
n1 [sentinel];
I200306 10:46:57.470909 195690 server/status/runtime.go:498 [n1] runtime stats: 317 MiB RSS, 193 goroutines, 61 MiB/171 MiB/125 MiB GO alloc/idle/total, 19 MiB/51 MiB CGO alloc/total, 44.7 CGO/sec, 0.8/0.1 %(u/s)time, 0.0 %gc (0x), 8.0 MiB/8.0 MiB (r/w)net
W200306 10:46:57.827002 196719 kv/kvserver/txnwait/queue.go:517 [n1,s1,r6/1:/Table/{SystemCon…-11}] pusher f713d97f: have been waiting 60.00s for pushee 2be15fb8
I200306 10:47:07.470918 195690 server/status/runtime.go:498 [n1] runtime stats: 317 MiB RSS, 193 goroutines, 70 MiB/165 MiB/125 MiB GO alloc/idle/total, 19 MiB/51 MiB CGO alloc/total, 138.4 CGO/sec, 1.4/0.2 %(u/s)time, 0.0 %gc (0x), 8.6 MiB/8.6 MiB (r/w)net
I200306 10:47:17.474632 195690 server/status/runtime.go:498 [n1] runtime stats: 318 MiB RSS, 193 goroutines, 77 MiB/159 MiB/126 MiB GO alloc/idle/total, 19 MiB/51 MiB CGO alloc/total, 92.9 CGO/sec, 1.2/0.4 %(u/s)time, 0.0 %gc (0x), 10 MiB/10 MiB (r/w)net
I200306 10:47:27.474312 195690 server/status/runtime.go:498 [n1] runtime stats: 319 MiB RSS, 193 goroutines, 84 MiB/153 MiB/131 MiB GO alloc/idle/total, 20 MiB/52 MiB CGO alloc/total, 101.9 CGO/sec, 1.3/0.2 %(u/s)time, 0.0 %gc (0x), 10 MiB/10 MiB (r/w)net
W200306 10:47:27.713046 197786 kv/kvserver/txnwait/queue.go:517 [n1,s1,r6/1:/Table/{SystemCon…-11}] pusher 1d889b04: have been waiting 60.00s for pushee 2be15fb8
I200306 10:47:37.475467 195690 server/status/runtime.go:498 [n1] runtime stats: 319 MiB RSS, 193 goroutines, 39 MiB/190 MiB/113 MiB GO alloc/idle/total, 20 MiB/52 MiB CGO alloc/total, 93.4 CGO/sec, 2.4/1.2 %(u/s)time, 0.0 %gc (1x), 1.4 MiB/1.4 MiB (r/w)net
I200306 10:47:47.475558 195690 server/status/runtime.go:498 [n1] runtime stats: 319 MiB RSS, 193 goroutines, 46 MiB/185 MiB/113 MiB GO alloc/idle/total, 20 MiB/52 MiB CGO alloc/total, 96.9 CGO/sec, 1.2/0.2 %(u/s)time, 0.0 %gc (0x), 3.0 MiB/3.0 MiB (r/w)net
I200306 10:47:57.465732 195686 gossip/gossip.go:567 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I200306 10:47:57.475805 195690 server/status/runtime.go:498 [n1] runtime stats: 320 MiB RSS, 193 goroutines, 53 MiB/179 MiB/113 MiB GO alloc/idle/total, 20 MiB/52 MiB CGO alloc/total, 97.4 CGO/sec, 1.1/0.3 %(u/s)time, 0.0 %gc (0x), 1.3 MiB/1.3 MiB (r/w)net
W200306 10:47:57.510102 197661 kv/kvserver/txnwait/queue.go:517 [n1,s1,r6/1:/Table/{SystemCon…-11}] pusher d39d3a50: have been waiting 60.00s for pushee 2be15fb8
I200306 10:48:07.481066 195690 server/status/runtime.go:498 [n1] runtime stats: 320 MiB RSS, 193 goroutines, 62 MiB/173 MiB/113 MiB GO alloc/idle/total, 20 MiB/52 MiB CGO alloc/total, 121.0 CGO/sec, 1.4/0.2 %(u/s)time, 0.0 %gc (0x), 311 KiB/311 KiB (r/w)net
I200306 10:48:17.481073 195690 server/status/runtime.go:498 [n1] runtime stats: 321 MiB RSS, 193 goroutines, 69 MiB/167 MiB/114 MiB GO alloc/idle/total, 23 MiB/55 MiB CGO alloc/total, 104.3 CGO/sec, 1.5/0.2 %(u/s)time, 0.0 %gc (0x), 1.1 MiB/1.1 MiB (r/w)net
I200306 10:48:27.481388 195690 server/status/runtime.go:498 [n1] runtime stats: 321 MiB RSS, 193 goroutines, 33 MiB/196 MiB/101 MiB GO alloc/idle/total, 23 MiB/55 MiB CGO alloc/total, 95.2 CGO/sec, 1.9/2.1 %(u/s)time, 0.0 %gc (1x), 1.8 MiB/1.8 MiB (r/w)net
I200306 10:48:37.481357 195690 server/status/runtime.go:498 [n1] runtime stats: 321 MiB RSS, 197 goroutines, 42 MiB/190 MiB/101 MiB GO alloc/idle/total, 23 MiB/55 MiB CGO alloc/total, 106.6 CGO/sec, 1.4/0.2 %(u/s)time, 0.0 %gc (0x), 1.6 MiB/1.6 MiB (r/w)net
I200306 10:48:47.481522 195690 server/status/runtime.go:498 [n1] runtime stats: 322 MiB RSS, 197 goroutines, 49 MiB/184 MiB/101 MiB GO alloc/idle/total, 23 MiB/55 MiB CGO alloc/total, 98.5 CGO/sec, 1.3/0.2 %(u/s)time, 0.0 %gc (0x), 1.4 MiB/1.4 MiB (r/w)net
I200306 10:48:57.465747 195686 gossip/gossip.go:567 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I200306 10:48:57.481626 195690 server/status/runtime.go:498 [n1] runtime stats: 321 MiB RSS, 197 goroutines, 56 MiB/178 MiB/102 MiB GO alloc/idle/total, 23 MiB/53 MiB CGO alloc/total, 97.6 CGO/sec, 1.4/0.2 %(u/s)time, 0.0 %gc (0x), 1.6 MiB/1.6 MiB (r/w)net
I200306 10:49:07.481828 195690 server/status/runtime.go:498 [n1] runtime stats: 321 MiB RSS, 197 goroutines, 35 MiB/194 MiB/106 MiB GO alloc/idle/total, 23 MiB/53 MiB CGO alloc/total, 122.4 CGO/sec, 1.8/0.2 %(u/s)time, 0.0 %gc (1x), 1.2 MiB/1.2 MiB (r/w)net
I200306 10:49:17.482089 195690 server/status/runtime.go:498 [n1] runtime stats: 322 MiB RSS, 197 goroutines, 42 MiB/189 MiB/106 MiB GO alloc/idle/total, 23 MiB/53 MiB CGO alloc/total, 95.7 CGO/sec, 1.2/0.4 %(u/s)time, 0.0 %gc (0x), 662 KiB/662 KiB (r/w)net
I200306 10:49:27.482149 195690 server/status/runtime.go:498 [n1] runtime stats: 322 MiB RSS, 197 goroutines, 50 MiB/183 MiB/106 MiB GO alloc/idle/total, 23 MiB/53 MiB CGO alloc/total, 98.8 CGO/sec, 1.4/0.1 %(u/s)time, 0.0 %gc (0x), 153 KiB/153 KiB (r/w)net
W200306 10:49:27.910709 199088 kv/kvserver/txnwait/queue.go:517 [n1,s1,r6/1:/Table/{SystemCon…-11}] pusher ddf1c78b: have been waiting 60.00s for pushee 2be15fb8
W200306 10:49:27.911353 199089 kv/kvserver/txnwait/queue.go:517 [n1,s1,r6/1:/Table/{SystemCon…-11}] pusher cea7f210: have been waiting 60.00s for pushee 2be15fb8
I200306 10:49:37.482187 195690 server/status/runtime.go:498 [n1] runtime stats: 323 MiB RSS, 197 goroutines, 57 MiB/177 MiB/106 MiB GO alloc/idle/total, 23 MiB/53 MiB CGO alloc/total, 96.7 CGO/sec, 1.0/0.6 %(u/s)time, 0.0 %gc (0x), 76 KiB/76 KiB (r/w)net
I200306 10:49:47.482367 195690 server/status/runtime.go:498 [n1] runtime stats: 323 MiB RSS, 197 goroutines, 33 MiB/196 MiB/104 MiB GO alloc/idle/total, 28 MiB/57 MiB CGO alloc/total, 101.1 CGO/sec, 2.0/0.4 %(u/s)time, 0.0 %gc (1x), 2.5 MiB/2.5 MiB (r/w)net
I200306 10:49:57.465761 195686 gossip/gossip.go:567 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I200306 10:49:57.482510 195690 server/status/runtime.go:498 [n1] runtime stats: 323 MiB RSS, 197 goroutines, 40 MiB/191 MiB/104 MiB GO alloc/idle/total, 28 MiB/57 MiB CGO alloc/total, 98.4 CGO/sec, 1.3/0.4 %(u/s)time, 0.0 %gc (0x), 17 MiB/17 MiB (r/w)net
I200306 10:50:07.486036 195690 server/status/runtime.go:498 [n1] runtime stats: 324 MiB RSS, 197 goroutines, 48 MiB/185 MiB/104 MiB GO alloc/idle/total, 28 MiB/57 MiB CGO alloc/total, 125.9 CGO/sec, 1.4/0.1 %(u/s)time, 0.0 %gc (0x), 34 MiB/34 MiB (r/w)net
I200306 10:50:17.486126 195690 server/status/runtime.go:498 [n1] runtime stats: 324 MiB RSS, 197 goroutines, 56 MiB/179 MiB/104 MiB GO alloc/idle/total, 28 MiB/57 MiB CGO alloc/total, 98.2 CGO/sec, 1.3/0.3 %(u/s)time, 0.0 %gc (0x), 22 MiB/22 MiB (r/w)net
I200306 10:50:27.486743 195690 server/status/runtime.go:498 [n1] runtime stats: 324 MiB RSS, 197 goroutines, 32 MiB/197 MiB/105 MiB GO alloc/idle/total, 28 MiB/57 MiB CGO alloc/total, 97.7 CGO/sec, 1.7/0.4 %(u/s)time, 0.0 %gc (1x), 22 MiB/22 MiB (r/w)net
I200306 10:50:37.486453 195690 server/status/runtime.go:498 [n1] runtime stats: 325 MiB RSS, 197 goroutines, 40 MiB/191 MiB/105 MiB GO alloc/idle/total, 28 MiB/57 MiB CGO alloc/total, 98.1 CGO/sec, 1.3/0.3 %(u/s)time, 0.0 %gc (0x), 22 MiB/22 MiB (r/w)net
I200306 10:50:47.487453 195690 server/status/runtime.go:498 [n1] runtime stats: 326 MiB RSS, 197 goroutines, 47 MiB/186 MiB/105 MiB GO alloc/idle/total, 27 MiB/57 MiB CGO alloc/total, 95.3 CGO/sec, 1.4/0.1 %(u/s)time, 0.0 %gc (0x), 29 MiB/29 MiB (r/w)net
I200306 10:50:57.465767 195686 gossip/gossip.go:567 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I200306 10:50:57.487380 195690 server/status/runtime.go:498 [n1] runtime stats: 326 MiB RSS, 197 goroutines, 54 MiB/179 MiB/105 MiB GO alloc/idle/total, 27 MiB/57 MiB CGO alloc/total, 100.7 CGO/sec, 1.5/0.2 %(u/s)time, 0.0 %gc (0x), 26 MiB/26 MiB (r/w)net
I200306 10:51:07.488534 195690 server/status/runtime.go:498 [n1] runtime stats: 326 MiB RSS, 197 goroutines, 34 MiB/196 MiB/105 MiB GO alloc/idle/total, 27 MiB/57 MiB CGO alloc/total, 119.6 CGO/sec, 1.6/0.4 %(u/s)time, 0.0 %gc (1x), 20 MiB/20 MiB (r/w)net
I200306 10:51:17.487638 195690 server/status/runtime.go:498 [n1] runtime stats: 327 MiB RSS, 197 goroutines, 41 MiB/190 MiB/105 MiB GO alloc/idle/total, 27 MiB/57 MiB CGO alloc/total, 101.8 CGO/sec, 1.2/0.5 %(u/s)time, 0.0 %gc (0x), 18 MiB/18 MiB (r/w)net
I200306 10:51:27.487863 195690 server/status/runtime.go:498 [n1] runtime stats: 328 MiB RSS, 197 goroutines, 49 MiB/184 MiB/105 MiB GO alloc/idle/total, 27 MiB/57 MiB CGO alloc/total, 96.4 CGO/sec, 1.4/0.2 %(u/s)time, 0.0 %gc (0x), 27 MiB/27 MiB (r/w)net
I200306 10:51:37.487947 195690 server/status/runtime.go:498 [n1] runtime stats: 328 MiB RSS, 197 goroutines, 56 MiB/178 MiB/105 MiB GO alloc/idle/total, 35 MiB/65 MiB CGO alloc/total, 102.7 CGO/sec, 1.3/0.3 %(u/s)time, 0.0 %gc (0x), 39 MiB/39 MiB (r/w)net
I200306 10:51:47.488025 195690 server/status/runtime.go:498 [n1] runtime stats: 328 MiB RSS, 197 goroutines, 34 MiB/195 MiB/106 MiB GO alloc/idle/total, 35 MiB/65 MiB CGO alloc/total, 98.2 CGO/sec, 1.7/0.3 %(u/s)time, 0.0 %gc (1x), 15 MiB/15 MiB (r/w)net
I200306 10:51:57.465671 195686 gossip/gossip.go:567 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I200306 10:51:57.489327 195690 server/status/runtime.go:498 [n1] runtime stats: 326 MiB RSS, 197 goroutines, 42 MiB/189 MiB/106 MiB GO alloc/idle/total, 35 MiB/63 MiB CGO alloc/total, 100.2 CGO/sec, 1.3/0.3 %(u/s)time, 0.0 %gc (0x), 17 MiB/17 MiB (r/w)net
I200306 10:52:07.488575 195690 server/status/runtime.go:498 [n1] runtime stats: 328 MiB RSS, 197 goroutines, 50 MiB/183 MiB/106 MiB GO alloc/idle/total, 35 MiB/63 MiB CGO alloc/total, 123.1 CGO/sec, 1.4/0.1 %(u/s)time, 0.0 %gc (0x), 16 MiB/16 MiB (r/w)net
I200306 10:52:17.488746 195690 server/status/runtime.go:498 [n1] runtime stats: 328 MiB RSS, 197 goroutines, 58 MiB/177 MiB/106 MiB GO alloc/idle/total, 35 MiB/63 MiB CGO alloc/total, 97.1 CGO/sec, 1.3/0.3 %(u/s)time, 0.0 %gc (0x), 17 MiB/17 MiB (r/w)net
I200306 10:52:27.488892 195690 server/status/runtime.go:498 [n1] runtime stats: 328 MiB RSS, 198 goroutines, 35 MiB/194 MiB/106 MiB GO alloc/idle/total, 35 MiB/63 MiB CGO alloc/total, 98.4 CGO/sec, 1.8/0.3 %(u/s)time, 0.1 %gc (1x), 16 MiB/16 MiB (r/w)net
I200306 10:52:37.488892 195690 server/status/runtime.go:498 [n1] runtime stats: 328 MiB RSS, 197 goroutines, 42 MiB/189 MiB/106 MiB GO alloc/idle/total, 35 MiB/63 MiB CGO alloc/total, 97.5 CGO/sec, 1.3/0.2 %(u/s)time, 0.0 %gc (0x), 19 MiB/19 MiB (r/w)net
I200306 10:52:47.489020 195690 server/status/runtime.go:498 [n1] runtime stats: 329 MiB RSS, 197 goroutines, 49 MiB/184 MiB/106 MiB GO alloc/idle/total, 35 MiB/63 MiB CGO alloc/total, 101.7 CGO/sec, 1.4/0.1 %(u/s)time, 0.0 %gc (0x), 31 MiB/31 MiB (r/w)net
I200306 10:52:57.469353 195686 gossip/gossip.go:567 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I200306 10:52:57.489086 195690 server/status/runtime.go:498 [n1] runtime stats: 330 MiB RSS, 197 goroutines, 56 MiB/178 MiB/106 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 98.1 CGO/sec, 1.5/0.2 %(u/s)time, 0.0 %gc (0x), 41 MiB/41 MiB (r/w)net
I200306 10:53:07.491838 195690 server/status/runtime.go:498 [n1] runtime stats: 330 MiB RSS, 197 goroutines, 31 MiB/198 MiB/99 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 121.9 CGO/sec, 2.4/0.5 %(u/s)time, 0.0 %gc (1x), 27 MiB/27 MiB (r/w)net
I200306 10:53:17.493997 195690 server/status/runtime.go:498 [n1] runtime stats: 330 MiB RSS, 197 goroutines, 38 MiB/193 MiB/99 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 100.0 CGO/sec, 1.5/0.2 %(u/s)time, 0.0 %gc (0x), 25 MiB/25 MiB (r/w)net
I200306 10:53:27.489641 195690 server/status/runtime.go:498 [n1] runtime stats: 330 MiB RSS, 197 goroutines, 45 MiB/187 MiB/99 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 100.2 CGO/sec, 1.4/0.1 %(u/s)time, 0.0 %gc (0x), 33 MiB/33 MiB (r/w)net
I200306 10:53:37.489810 195690 server/status/runtime.go:498 [n1] runtime stats: 331 MiB RSS, 197 goroutines, 52 MiB/181 MiB/99 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 98.2 CGO/sec, 1.0/0.6 %(u/s)time, 0.0 %gc (0x), 42 MiB/42 MiB (r/w)net
I200306 10:53:47.489942 195690 server/status/runtime.go:498 [n1] runtime stats: 332 MiB RSS, 197 goroutines, 33 MiB/196 MiB/102 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 96.1 CGO/sec, 1.5/0.6 %(u/s)time, 0.0 %gc (1x), 57 MiB/57 MiB (r/w)net
I200306 10:53:57.465700 195686 gossip/gossip.go:567 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I200306 10:53:57.490029 195690 server/status/runtime.go:498 [n1] runtime stats: 332 MiB RSS, 197 goroutines, 41 MiB/191 MiB/102 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 98.7 CGO/sec, 1.2/0.5 %(u/s)time, 0.0 %gc (0x), 23 MiB/23 MiB (r/w)net
I200306 10:54:07.490420 195690 server/status/runtime.go:498 [n1] runtime stats: 332 MiB RSS, 197 goroutines, 49 MiB/184 MiB/102 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 123.1 CGO/sec, 1.2/0.4 %(u/s)time, 0.0 %gc (0x), 32 MiB/32 MiB (r/w)net
I200306 10:54:17.490322 195690 server/status/runtime.go:498 [n1] runtime stats: 333 MiB RSS, 197 goroutines, 56 MiB/178 MiB/102 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 101.6 CGO/sec, 1.3/0.3 %(u/s)time, 0.0 %gc (0x), 37 MiB/37 MiB (r/w)net
I200306 10:54:27.490446 195690 server/status/runtime.go:498 [n1] runtime stats: 334 MiB RSS, 197 goroutines, 34 MiB/195 MiB/105 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 95.7 CGO/sec, 1.7/0.3 %(u/s)time, 0.0 %gc (1x), 32 MiB/32 MiB (r/w)net
I200306 10:54:37.490689 195690 server/status/runtime.go:498 [n1] runtime stats: 334 MiB RSS, 197 goroutines, 41 MiB/190 MiB/105 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 97.0 CGO/sec, 1.5/0.2 %(u/s)time, 0.0 %gc (0x), 54 MiB/54 MiB (r/w)net
I200306 10:54:47.490595 195690 server/status/runtime.go:498 [n1] runtime stats: 335 MiB RSS, 197 goroutines, 48 MiB/184 MiB/105 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 101.8 CGO/sec, 1.4/0.3 %(u/s)time, 0.0 %gc (0x), 55 MiB/55 MiB (r/w)net
I200306 10:54:57.465712 195686 gossip/gossip.go:567 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I200306 10:54:57.490997 195690 server/status/runtime.go:498 [n1] runtime stats: 335 MiB RSS, 197 goroutines, 56 MiB/178 MiB/105 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 101.6 CGO/sec, 1.2/0.5 %(u/s)time, 0.0 %gc (0x), 30 MiB/30 MiB (r/w)net
I200306 10:55:07.491127 195690 server/status/runtime.go:498 [n1] runtime stats: 336 MiB RSS, 197 goroutines, 31 MiB/197 MiB/101 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 122.2 CGO/sec, 2.4/0.4 %(u/s)time, 0.0 %gc (1x), 35 MiB/35 MiB (r/w)net
I200306 10:55:17.491289 195690 server/status/runtime.go:498 [n1] runtime stats: 337 MiB RSS, 197 goroutines, 38 MiB/192 MiB/101 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 97.7 CGO/sec, 1.5/0.1 %(u/s)time, 0.0 %gc (0x), 45 MiB/45 MiB (r/w)net
I200306 10:55:27.491315 195690 server/status/runtime.go:498 [n1] runtime stats: 337 MiB RSS, 197 goroutines, 46 MiB/186 MiB/101 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 98.4 CGO/sec, 1.5/0.3 %(u/s)time, 0.0 %gc (0x), 69 MiB/69 MiB (r/w)net
I200306 10:55:37.491656 195690 server/status/runtime.go:498 [n1] runtime stats: 337 MiB RSS, 197 goroutines, 54 MiB/180 MiB/101 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 100.6 CGO/sec, 1.1/0.4 %(u/s)time, 0.0 %gc (0x), 28 MiB/28 MiB (r/w)net
I200306 10:55:47.494167 195690 server/status/runtime.go:498 [n1] runtime stats: 337 MiB RSS, 197 goroutines, 34 MiB/195 MiB/103 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 103.4 CGO/sec, 1.6/0.4 %(u/s)time, 0.0 %gc (1x), 30 MiB/30 MiB (r/w)net
I200306 10:55:57.465708 195686 gossip/gossip.go:567 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I200306 10:55:57.491824 195690 server/status/runtime.go:498 [n1] runtime stats: 338 MiB RSS, 197 goroutines, 42 MiB/189 MiB/103 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 98.7 CGO/sec, 1.1/0.6 %(u/s)time, 0.0 %gc (0x), 24 MiB/24 MiB (r/w)net
E200306 10:55:57.510798 195897 kv/kvserver/queue.go:1204 [n1,replicate] 33 replicas failing with "0 of 1 live stores are able to take a new replica for the range (1 already has a replica); likely not enough nodes in cluster"
I200306 10:56:07.492038 195690 server/status/runtime.go:498 [n1] runtime stats: 339 MiB RSS, 197 goroutines, 51 MiB/183 MiB/103 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 127.3 CGO/sec, 1.3/0.3 %(u/s)time, 0.0 %gc (0x), 6.9 MiB/6.9 MiB (r/w)net
I200306 10:56:17.466090 195685 kv/kvserver/store.go:2383 [n1,s1] sstables (read amplification = 0):
I200306 10:56:17.466202 195685 kv/kvserver/store.go:2384 [n1,s1]
** Compaction Stats [default] **
Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0
** Compaction Stats [default] **
Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Uptime(secs): 620.0 total, 600.0 interval
Flush(GB): cumulative 0.000, interval 0.000
AddFile(GB): cumulative 0.000, interval 0.000
AddFile(Total Files): cumulative 0, interval 0
AddFile(L0 Files): cumulative 0, interval 0
AddFile(Keys): cumulative 0, interval 0
Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
estimated_pending_compaction_bytes: 0 B
I200306 10:56:17.492334 195690 server/status/runtime.go:498 [n1] runtime stats: 340 MiB RSS, 197 goroutines, 58 MiB/177 MiB/103 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 98.7 CGO/sec, 0.9/0.6 %(u/s)time, 0.0 %gc (0x), 1.8 KiB/1.8 KiB (r/w)net
I200306 10:56:27.492505 195690 server/status/runtime.go:498 [n1] runtime stats: 340 MiB RSS, 197 goroutines, 35 MiB/194 MiB/106 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 100.2 CGO/sec, 1.3/0.5 %(u/s)time, 0.0 %gc (1x), 1.2 KiB/1.2 KiB (r/w)net
I200306 10:56:37.493067 195690 server/status/runtime.go:498 [n1] runtime stats: 340 MiB RSS, 197 goroutines, 43 MiB/189 MiB/106 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 100.4 CGO/sec, 0.7/0.7 %(u/s)time, 0.0 %gc (0x), 1.4 KiB/1.4 KiB (r/w)net
I200306 10:56:47.493143 195690 server/status/runtime.go:498 [n1] runtime stats: 341 MiB RSS, 197 goroutines, 50 MiB/183 MiB/106 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 100.8 CGO/sec, 0.5/0.8 %(u/s)time, 0.0 %gc (0x), 1.8 KiB/1.8 KiB (r/w)net
I200306 10:56:57.465716 195686 gossip/gossip.go:567 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I200306 10:56:57.493517 195690 server/status/runtime.go:498 [n1] runtime stats: 341 MiB RSS, 197 goroutines, 57 MiB/177 MiB/106 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 98.5 CGO/sec, 0.7/0.6 %(u/s)time, 0.0 %gc (0x), 1.2 KiB/1.2 KiB (r/w)net
I200306 10:57:07.493526 195690 server/status/runtime.go:498 [n1] runtime stats: 342 MiB RSS, 197 goroutines, 32 MiB/197 MiB/102 MiB GO alloc/idle/total, 45 MiB/73 MiB CGO alloc/total, 120.5 CGO/sec, 1.9/0.8 %(u/s)time, 0.0 %gc (1x), 1.4 KiB/1.4 KiB (r/w)net
I200306 10:57:17.493892 195690 server/status/runtime.go:498 [n1] runtime stats: 342 MiB RSS, 197 goroutines, 39 MiB/192 MiB/102 MiB GO alloc/idle/total, 53 MiB/81 MiB CGO alloc/total, 101.6 CGO/sec, 0.7/0.6 %(u/s)time, 0.0 %gc (0x), 1.8 KiB/1.8 KiB (r/w)net
I200306 10:57:27.493839 195690 server/status/runtime.go:498 [n1] runtime stats: 342 MiB RSS, 197 goroutines, 46 MiB/186 MiB/102 MiB GO alloc/idle/total, 53 MiB/81 MiB CGO alloc/total, 98.3 CGO/sec, 0.5/0.8 %(u/s)time, 0.0 %gc (0x), 1.2 KiB/1.2 KiB (r/w)net
I200306 10:57:37.493976 195690 server/status/runtime.go:498 [n1] runtime stats: 343 MiB RSS, 197 goroutines, 54 MiB/180 MiB/102 MiB GO alloc/idle/total, 53 MiB/81 MiB CGO alloc/total, 99.1 CGO/sec, 0.7/0.7 %(u/s)time, 0.0 %gc (0x), 1.4 KiB/1.4 KiB (r/w)net
I200306 10:57:47.494439 195690 server/status/runtime.go:498 [n1] runtime stats: 343 MiB RSS, 197 goroutines, 32 MiB/197 MiB/103 MiB GO alloc/idle/total, 53 MiB/81 MiB CGO alloc/total, 96.3 CGO/sec, 1.1/0.6 %(u/s)time, 0.0 %gc (1x), 1.6 KiB/1.6 KiB (r/w)net
I200306 10:57:57.465771 195686 gossip/gossip.go:567 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I200306 10:57:57.494366 195690 server/status/runtime.go:498 [n1] runtime stats: 344 MiB RSS, 197 goroutines, 40 MiB/192 MiB/103 MiB GO alloc/idle/total, 53 MiB/81 MiB CGO alloc/total, 102.1 CGO/sec, 0.8/0.5 %(u/s)time, 0.0 %gc (0x), 1.4 KiB/1.4 KiB (r/w)net
I200306 10:58:07.494621 195690 server/status/runtime.go:498 [n1] runtime stats: 344 MiB RSS, 197 goroutines, 48 MiB/185 MiB/103 MiB GO alloc/idle/total, 53 MiB/81 MiB CGO alloc/total, 121.3 CGO/sec, 1.0/0.5 %(u/s)time, 0.0 %gc (0x), 1.4 KiB/1.4 KiB (r/w)net
I200306 10:58:17.495034 195690 server/status/runtime.go:498 [n1] runtime stats: 344 MiB RSS, 197 goroutines, 55 MiB/179 MiB/103 MiB GO alloc/idle/total, 53 MiB/81 MiB CGO alloc/total, 101.6 CGO/sec, 0.4/0.9 %(u/s)time, 0.0 %gc (0x), 1.6 KiB/1.6 KiB (r/w)net
I200306 10:58:27.494866 195690 server/status/runtime.go:498 [n1] runtime stats: 345 MiB RSS, 197 goroutines, 35 MiB/194 MiB/104 MiB GO alloc/idle/total, 53 MiB/81 MiB CGO alloc/total, 98.8 CGO/sec, 1.4/0.4 %(u/s)time, 0.0 %gc (1x), 1.4 KiB/1.4 KiB (r/w)net
I200306 10:58:37.495179 195690 server/status/runtime.go:498 [n1] runtime stats: 346 MiB RSS, 197 goroutines, 42 MiB/189 MiB/104 MiB GO alloc/idle/total, 53 MiB/81 MiB CGO alloc/total, 94.9 CGO/sec, 0.6/0.8 %(u/s)time, 0.0 %gc (0x), 1.4 KiB/1.4 KiB (r/w)net
I200306 10:58:47.495205 195690 server/status/runtime.go:498 [n1] runtime stats: 346 MiB RSS, 197 goroutines, 50 MiB/183 MiB/104 MiB GO alloc/idle/total, 53 MiB/81 MiB CGO alloc/total, 102.0 CGO/sec, 0.8/0.6 %(u/s)time, 0.0 %gc (0x), 1.7 KiB/1.7 KiB (r/w)net
I200306 10:58:57.465814 195686 gossip/gossip.go:567 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I200306 10:58:57.495452 195690 server/status/runtime.go:498 [n1] runtime stats: 346 MiB RSS, 197 goroutines, 57 MiB/177 MiB/104 MiB GO alloc/idle/total, 53 MiB/81 MiB CGO alloc/total, 96.6 CGO/sec, 0.6/0.7 %(u/s)time, 0.0 %gc (0x), 1.4 KiB/1.4 KiB (r/w)net
I200306 10:59:07.495614 195690 server/status/runtime.go:498 [n1] runtime stats: 347 MiB RSS, 197 goroutines, 33 MiB/196 MiB/104 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 124.5 CGO/sec, 1.6/0.7 %(u/s)time, 0.0 %gc (1x), 1.4 KiB/1.4 KiB (r/w)net
I200306 10:59:17.495930 195690 server/status/runtime.go:498 [n1] runtime stats: 347 MiB RSS, 198 goroutines, 40 MiB/191 MiB/104 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 97.9 CGO/sec, 0.5/0.9 %(u/s)time, 0.0 %gc (0x), 1.6 KiB/1.6 KiB (r/w)net
I200306 10:59:27.496164 195690 server/status/runtime.go:498 [n1] runtime stats: 347 MiB RSS, 197 goroutines, 48 MiB/185 MiB/104 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 102.1 CGO/sec, 0.8/0.5 %(u/s)time, 0.0 %gc (0x), 1.4 KiB/1.4 KiB (r/w)net
I200306 10:59:37.496250 195690 server/status/runtime.go:498 [n1] runtime stats: 348 MiB RSS, 197 goroutines, 55 MiB/179 MiB/104 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 94.8 CGO/sec, 0.7/0.6 %(u/s)time, 0.0 %gc (0x), 1.4 KiB/1.4 KiB (r/w)net
I200306 10:59:47.496419 195690 server/status/runtime.go:498 [n1] runtime stats: 348 MiB RSS, 197 goroutines, 35 MiB/194 MiB/104 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 98.5 CGO/sec, 1.1/0.6 %(u/s)time, 0.0 %gc (1x), 1.6 KiB/1.6 KiB (r/w)net
I200306 10:59:57.465710 195686 gossip/gossip.go:567 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I200306 10:59:57.496659 195690 server/status/runtime.go:498 [n1] runtime stats: 349 MiB RSS, 197 goroutines, 43 MiB/188 MiB/104 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 98.7 CGO/sec, 0.8/0.5 %(u/s)time, 0.0 %gc (0x), 1.5 KiB/1.5 KiB (r/w)net
I200306 11:00:07.496910 195690 server/status/runtime.go:498 [n1] runtime stats: 349 MiB RSS, 197 goroutines, 51 MiB/182 MiB/104 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 125.0 CGO/sec, 0.8/0.6 %(u/s)time, 0.0 %gc (0x), 1.5 KiB/1.5 KiB (r/w)net
I200306 11:00:17.497148 195690 server/status/runtime.go:498 [n1] runtime stats: 349 MiB RSS, 197 goroutines, 59 MiB/176 MiB/104 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 105.3 CGO/sec, 1.2/0.2 %(u/s)time, 0.0 %gc (0x), 1.7 KiB/1.7 KiB (r/w)net
I200306 11:00:27.497309 195690 server/status/runtime.go:498 [n1] runtime stats: 349 MiB RSS, 197 goroutines, 35 MiB/194 MiB/106 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 96.9 CGO/sec, 1.2/0.7 %(u/s)time, 0.0 %gc (1x), 1.4 KiB/1.4 KiB (r/w)net
I200306 11:00:37.497456 195690 server/status/runtime.go:498 [n1] runtime stats: 350 MiB RSS, 197 goroutines, 42 MiB/189 MiB/106 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 100.4 CGO/sec, 0.5/0.9 %(u/s)time, 0.0 %gc (0x), 1.4 KiB/1.4 KiB (r/w)net
I200306 11:00:47.497438 195690 server/status/runtime.go:498 [n1] runtime stats: 351 MiB RSS, 197 goroutines, 50 MiB/183 MiB/106 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 98.4 CGO/sec, 0.6/0.8 %(u/s)time, 0.0 %gc (0x), 1.6 KiB/1.6 KiB (r/w)net
I200306 11:00:57.465734 195686 gossip/gossip.go:567 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I200306 11:00:57.497617 195690 server/status/runtime.go:498 [n1] runtime stats: 351 MiB RSS, 197 goroutines, 58 MiB/176 MiB/106 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 100.5 CGO/sec, 0.2/1.1 %(u/s)time, 0.0 %gc (0x), 1.5 KiB/1.5 KiB (r/w)net
I200306 11:01:07.497670 195690 server/status/runtime.go:498 [n1] runtime stats: 351 MiB RSS, 197 goroutines, 32 MiB/197 MiB/102 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 122.5 CGO/sec, 2.0/0.7 %(u/s)time, 0.0 %gc (1x), 1.4 KiB/1.4 KiB (r/w)net
I200306 11:01:17.497950 195690 server/status/runtime.go:498 [n1] runtime stats: 352 MiB RSS, 197 goroutines, 40 MiB/192 MiB/102 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 100.6 CGO/sec, 0.8/0.5 %(u/s)time, 0.0 %gc (0x), 1.6 KiB/1.6 KiB (r/w)net
I200306 11:01:27.498029 195690 server/status/runtime.go:498 [n1] runtime stats: 353 MiB RSS, 197 goroutines, 47 MiB/186 MiB/102 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 100.7 CGO/sec, 1.1/0.3 %(u/s)time, 0.0 %gc (0x), 1.4 KiB/1.4 KiB (r/w)net
I200306 11:01:37.498068 195690 server/status/runtime.go:498 [n1] runtime stats: 353 MiB RSS, 197 goroutines, 54 MiB/180 MiB/102 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 100.7 CGO/sec, 0.6/0.7 %(u/s)time, 0.0 %gc (0x), 1.5 KiB/1.5 KiB (r/w)net
I200306 11:01:47.498166 195690 server/status/runtime.go:498 [n1] runtime stats: 354 MiB RSS, 197 goroutines, 33 MiB/197 MiB/104 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 100.4 CGO/sec, 1.3/0.5 %(u/s)time, 0.0 %gc (1x), 1.6 KiB/1.6 KiB (r/w)net
I200306 11:01:57.465708 195686 gossip/gossip.go:567 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I200306 11:01:57.498311 195690 server/status/runtime.go:498 [n1] runtime stats: 354 MiB RSS, 197 goroutines, 42 MiB/190 MiB/104 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 101.0 CGO/sec, 0.9/0.5 %(u/s)time, 0.0 %gc (0x), 1.5 KiB/1.5 KiB (r/w)net
I200306 11:02:07.498437 195690 server/status/runtime.go:498 [n1] runtime stats: 354 MiB RSS, 197 goroutines, 48 MiB/185 MiB/104 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 124.5 CGO/sec, 0.5/0.8 %(u/s)time, 0.0 %gc (0x), 1.4 KiB/1.4 KiB (r/w)net
I200306 11:02:17.498535 195690 server/status/runtime.go:498 [n1] runtime stats: 355 MiB RSS, 197 goroutines, 55 MiB/179 MiB/104 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 96.5 CGO/sec, 0.8/0.6 %(u/s)time, 0.0 %gc (0x), 1.6 KiB/1.6 KiB (r/w)net
I200306 11:02:27.498909 195690 server/status/runtime.go:498 [n1] runtime stats: 356 MiB RSS, 197 goroutines, 34 MiB/195 MiB/104 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 100.7 CGO/sec, 1.4/0.3 %(u/s)time, 0.0 %gc (1x), 1.4 KiB/1.4 KiB (r/w)net
I200306 11:02:37.498845 195690 server/status/runtime.go:498 [n1] runtime stats: 356 MiB RSS, 197 goroutines, 42 MiB/190 MiB/104 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 98.5 CGO/sec, 0.4/0.9 %(u/s)time, 0.0 %gc (0x), 1.4 KiB/1.4 KiB (r/w)net
I200306 11:02:47.499060 195690 server/status/runtime.go:498 [n1] runtime stats: 356 MiB RSS, 197 goroutines, 49 MiB/184 MiB/104 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 98.4 CGO/sec, 0.7/0.6 %(u/s)time, 0.0 %gc (0x), 1.7 KiB/1.7 KiB (r/w)net
I200306 11:02:57.465761 195686 gossip/gossip.go:567 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I200306 11:02:57.499266 195690 server/status/runtime.go:498 [n1] runtime stats: 357 MiB RSS, 197 goroutines, 57 MiB/176 MiB/104 MiB GO alloc/idle/total, 73 MiB/101 MiB CGO alloc/total, 99.4 CGO/sec, 0.7/0.7 %(u/s)time, 0.0 %gc (0x), 1.4 KiB/1.4 KiB (r/w)net
I200306 11:03:07.499535 195690 server/status/runtime.go:498 [n1] runtime stats: 358 MiB RSS, 197 goroutines, 32 MiB/197 MiB/101 MiB GO alloc/idle/total, 81 MiB/109 MiB CGO alloc/total, 123.6 CGO/sec, 1.6/0.7 %(u/s)time, 0.0 %gc (1x), 1.4 KiB/1.4 KiB (r/w)net
I200306 11:03:17.499468 195690 server/status/runtime.go:498 [n1] runtime stats: 358 MiB RSS, 197 goroutines, 39 MiB/192 MiB/101 MiB GO alloc/idle/total, 81 MiB/109 MiB CGO alloc/total, 102.1 CGO/sec, 0.8/0.6 %(u/s)time, 0.0 %gc (0x), 1.6 KiB/1.6 KiB (r/w)net
I200306 11:03:27.499793 195690 server/status/runtime.go:498 [n1] runtime stats: 358 MiB RSS, 197 goroutines, 47 MiB/186 MiB/101 MiB GO alloc/idle/total, 81 MiB/109 MiB CGO alloc/total, 98.6 CGO/sec, 0.5/0.7 %(u/s)time, 0.0 %gc (0x), 1.4 KiB/1.4 KiB (r/w)net
I200306 11:03:37.499695 195690 server/status/runtime.go:498 [n1] runtime stats: 359 MiB RSS, 197 goroutines, 54 MiB/180 MiB/102 MiB GO alloc/idle/total, 81 MiB/109 MiB CGO alloc/total, 98.0 CGO/sec, 1.0/0.3 %(u/s)time, 0.0 %gc (0x), 1.4 KiB/1.4 KiB (r/w)net
I200306 11:03:47.499854 195690 server/status/runtime.go:498 [n1] runtime stats: 359 MiB RSS, 197 goroutines, 32 MiB/197 MiB/103 MiB GO alloc/idle/total, 81 MiB/109 MiB CGO alloc/total, 97.3 CGO/sec, 0.9/0.7 %(u/s)time, 0.0 %gc (1x), 1.6 KiB/1.6 KiB (r/w)net
I200306 11:03:57.465677 195686 gossip/gossip.go:567 [n1] gossip status (ok, 1 node)
gossip client (0/3 cur/max conns)
gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received)
I200306 11:03:57.499946 195690 server/status/runtime.go:498 [n1] runtime stats: 360 MiB RSS, 197 goroutines, 43 MiB/189 MiB/103 MiB GO alloc/idle/total, 81 MiB/109 MiB CGO alloc/total, 101.9 CGO/sec, 0.8/0.6 %(u/s)time, 0.0 %gc (0x), 1.4 KiB/1.4 KiB (r/w)net
I200306 11:04:07.499986 195690 server/status/runtime.go:498 [n1] runtime stats: 360 MiB RSS, 197 goroutines, 49 MiB/185 MiB/103 MiB GO alloc/idle/total, 81 MiB/109 MiB CGO alloc/total, 127.1 CGO/sec, 1.0/0.3 %(u/s)time, 0.0 %gc (0x), 1.4 KiB/1.4 KiB (r/w)net
I200306 11:04:17.500346 195690 server/status/runtime.go:498 [n1] runtime stats: 360 MiB RSS, 197 goroutines, 56 MiB/178 MiB/103 MiB GO alloc/idle/total, 81 MiB/109 MiB CGO alloc/total, 96.9 CGO/sec, 0.4/1.0 %(u/s)time, 0.0 %gc (0x), 1.6 KiB/1.6 KiB (r/w)net
I200306 11:04:27.500303 195690 server/status/runtime.go:498 [n1] runtime stats: 360 MiB RSS, 197 goroutines, 36 MiB/194 MiB/103 MiB GO alloc/idle/total, 81 MiB/109 MiB CGO alloc/total, 100.1 CGO/sec, 1.0/0.8 %(u/s)time, 0.0 %gc (1x), 1.4 KiB/1.4 KiB (r/w)net
I200306 11:04:37.500386 195690 server/status/runtime.go:498 [n1] runtime stats: 361 MiB RSS, 197 goroutines, 43 MiB/188 MiB/103 MiB GO alloc/idle/total, 81 MiB/109 MiB CGO alloc/total, 98.5 CGO/sec, 0.6/0.7 %(u/s)time, 0.0 %gc (0x), 1.4 KiB/1.4 KiB (r/w)net
panic: test timed out after 20m0s
goroutine 209831 [running]:
testing.(*M).startAlarm.func1()
/usr/local/go/src/testing/testing.go:1377 +0xdf
created by time.goFunc
/usr/local/go/src/time/sleep.go:168 +0x44
goroutine 1 [chan receive, 18 minutes]:
testing.(*T).Run(0xc004ea2500, 0x3b9fd1c, 0x1a, 0x3cf60d0, 0x726701)
/usr/local/go/src/testing/testing.go:961 +0x377
testing.runTests.func1(0xc000682200)
/usr/local/go/src/testing/testing.go:1202 +0x78
testing.tRunner(0xc000682200, 0xc0007d7da0)
/usr/local/go/src/testing/testing.go:909 +0xc9
testing.runTests(0xc00020a360, 0x631cc40, 0x123, 0x123, 0x0)
/usr/local/go/src/testing/testing.go:1200 +0x2a7
testing.(*M).Run(0xc0004f0800, 0x0)
/usr/local/go/src/testing/testing.go:1117 +0x176
github.com/cockroachdb/cockroach/pkg/sql_test.TestMain(0xc0004f0800)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/main_test.go:32 +0xb6
main.main()
_testmain.go:634 +0x135
goroutine 7 [syscall, 20 minutes]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:29 +0x41
goroutine 8 [chan receive]:
github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:73 +0x73
created by github.com/cockroachdb/cockroach/pkg/util/log.init.7
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:45 +0x35
goroutine 9 [chan receive, 20 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:100 +0x124
created by github.com/cockroachdb/cockroach/pkg/util/log.init.7
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:46 +0x4d
goroutine 42 [select]:
github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view.(*worker).start(0xc0005cafc0)
/go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:152 +0xdd
created by github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view.init.0
/go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:29 +0x57
goroutine 195585 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x4388300, 0xc0039b9aa0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:99 +0x13f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ecba0, 0xc0028cb5f0, 0xc0059ecb90)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 196858 [select]:
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func1(0x4388300, 0xc004371290)
/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:437 +0x142
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059aebe0, 0xc0028cb5f0, 0xc004270e10)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195496 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001764000, 0x1, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x11a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc003259440, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1e3
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00325c1c0)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xee5
goroutine 195749 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18be)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432ed0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432ed0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20730, 0xc0028cb5f0, 0xc005f20700)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195638 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x4388300, 0xc0039b9c80)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:99 +0x13f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ecc70, 0xc0028cb5f0, 0xc0059ecc40)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 197660 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*createStatsNode).Next(0xc000270380, 0x4388240, 0xc002f9c280, 0xc001744000, 0xc002d683e8, 0x0, 0x4388240, 0xc002f9c280)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/create_stats.go:91 +0xf5
github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Next(0xc000820000, 0xc002d683e8, 0x43899c0, 0xc000270380, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:171 +0x439
github.com/cockroachdb/cockroach/pkg/sql/execinfra.Run(0x4388240, 0xc002f9c280, 0x43a73c0, 0xc000820000, 0x436bcc0, 0xc0039ca380)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/base.go:170 +0x35
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBase).Run(0xc000820000, 0x4388240, 0xc002f9c280)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:749 +0x92
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run(0xc000384fc0, 0x4388240, 0xc002f9c280, 0x3cf3740, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:370 +0x247
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc0004403c0, 0xc007d0a8a0, 0xc0059e6120, 0xc001b186b8, 0xc002c3e840, 0xc002d684c8, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:378 +0x426
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0xc0004403c0, 0x4388300, 0xc001160510, 0xc002d684c8, 0xc007d0a8a0, 0xc0059e6120, 0x43899c0, 0xc000270380, 0xc002c3e840, 0xc002d681c8)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:988 +0x225
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc002d68000, 0x4388300, 0xc001160510, 0xc002d683e8, 0x1, 0x43e4420, 0xc007d0a840, 0xc001b18a00, 0xc00643c188, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:843 +0x3cf
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc002d68000, 0x4388300, 0xc001160510, 0xc002d683e8, 0x43e4420, 0xc007d0a840, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:750 +0x6cd
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc002d68000, 0x4388300, 0xc001160510, 0x43a0700, 0xc00643c050, 0xc000cee000, 0x59, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:455 +0xac8
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc002d68000, 0x4388300, 0xc001160510, 0x43a0700, 0xc00643c050, 0xc000cee000, 0x59, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:117 +0x4f6
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc002d68000, 0x4388240, 0xc002f9c200, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1396 +0x1aa8
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc002d68000, 0x4388300, 0xc001160060, 0xc004343ae0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1325 +0x1f2
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx.func1(0xc002d68000, 0x4388300, 0xc001160060, 0xc007722180, 0xc000232160, 0x0, 0xc0005ec694)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:158 +0x64
created by github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx
/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:157 +0x33a
goroutine 195779 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func1(0x4388300, 0xc005a96030)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:808 +0x176
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0076d65c0, 0xc0028cb5f0, 0xc00799dec0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195813 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1(0x4388300, 0xc001d46c60)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:254 +0x1ae
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22930, 0xc0028cb5f0, 0xc00799db80)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195729 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18ae)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432a80)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432a80)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f203a0, 0xc0028cb5f0, 0xc005f20390)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195795 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*NodeLiveness).StartHeartbeat.func1(0x4388300, 0xc006243830)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/node_liveness.go:475 +0x4ce
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d230b0, 0xc0028cb5f0, 0xc004d7ad20)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195712 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18d0)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc0014326c0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc0014326c0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20040, 0xc0028cb5f0, 0xc005f20030)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195719 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x189d)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432870)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432870)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20140, 0xc0028cb5f0, 0xc005f20130)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195690 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startSampleEnvironment.func1(0x4388300, 0xc004e7d3e0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2215 +0x216
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22eb0, 0xc0028cb5f0, 0xc005ed2d50)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 197255 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc004b14000)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x218
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfc9
goroutine 195720 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18b1)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc0014328a0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc0014328a0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20160, 0xc0028cb5f0, 0xc005f20150)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195653 [semacquire, 16 minutes]:
sync.runtime_Semacquire(0xc003e2a098)
/usr/local/go/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc003e2a090)
/usr/local/go/src/sync/waitgroup.go:130 +0x64
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).refreshSomeLeases(0xc003372360, 0x4388300, 0xc008a97410)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1883 +0x61d
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).PeriodicallyRefreshSomeLeases.func1(0x4388300, 0xc008a97410)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1839 +0x1e1
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ed330, 0xc0028cb5f0, 0xc0059ed320)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195671 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x4388300, 0xc004d35fb0)
/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1401 +0x3a2
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ed8d0, 0xc0028cb5f0, 0xc0059ed8b0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195688 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports.(*Reporter).Start.func2(0x4388300, 0xc004b62870)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports/reporter.go:150 +0x41d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22dc0, 0xc0028cb5f0, 0xc002178dc0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195706 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18d1)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432570)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432570)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400df70, 0xc0028cb5f0, 0xc00400df50)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195686 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1(0x4388300, 0xc005a96060)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:693 +0x275
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22d30, 0xc0028cb5f0, 0xc002178d20)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195752 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18b5)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432f60)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432f60)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20840, 0xc0028cb5f0, 0xc005f20830)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195759 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startGossip.func4(0x4388300, 0xc004bcc390)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:1553 +0x364
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f209b0, 0xc0028cb5f0, 0xc00027a0f0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195734 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18a2)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432b70)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432b70)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20490, 0xc0028cb5f0, 0xc005f20480)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195115 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18c6)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc0014321b0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc0014321b0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400dd00, 0xc0028cb5f0, 0xc00400dcd0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195733 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18ab)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432b40)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432b40)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20470, 0xc0028cb5f0, 0xc005f20460)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195705 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18cf)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc0014324e0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc0014324e0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400df40, 0xc0028cb5f0, 0xc00400df30)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195581 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc0028cb5f0, 0xc0045d31e0)
/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:175 +0x148
created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler
/go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:172 +0x49
goroutine 195726 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18a5)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc0014329f0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc0014329f0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20330, 0xc0028cb5f0, 0xc005f202f0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 197129 [select]:
github.com/cockroachdb/cockroach/pkg/kv.(*txnHeartbeater).heartbeatLoop(0xc004053cc8, 0x4388240, 0xc001f42180)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_heartbeater.go:257 +0x19c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0028cb5f0, 0x4388240, 0xc001f42180, 0xc001c79a40, 0x29, 0x43f24a0, 0xc00881d560, 0xc00162d740)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:323 +0x152
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:318 +0x131
goroutine 195647 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x4388300, 0xc0039b9e60)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:99 +0x13f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ecda0, 0xc0028cb5f0, 0xc0059ecd90)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 197786 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait.(*Queue).MaybeWaitForPush(0xc00602f9a0, 0x4388300, 0xc004bcc6f0, 0xc005e64b40, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait/queue.go:513 +0xf95
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).maybeInterceptReq(0xc003e24d40, 0x4388300, 0xc004bcc6f0, 0x0, 0x15f9b135967024dd, 0x0, 0x0, 0x0, 0xc002ac9600, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:189 +0xa9
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc003e24d40, 0x4388300, 0xc004bcc6f0, 0xc004bb07e0, 0x0, 0x15f9b135967024dd, 0x0, 0x0, 0x0, 0xc002ac9600, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:142 +0xb8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc003e24d40, 0x4388300, 0xc004bcc6f0, 0x0, 0x0, 0x15f9b135967024dd, 0x0, 0x0, 0x0, 0xc002ac9600, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:121 +0xfb
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc00267ea00, 0x4388300, 0xc004bcc6f0, 0xc002ac9700, 0x3cf2c18, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:192 +0x270
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc00267ea00, 0x4388300, 0xc004bcc6c0, 0x6, 0xc002ac9700, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:94 +0x68a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc00267ea00, 0x4388300, 0xc004bcc6c0, 0x15f9b135967024dd, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36 +0x91
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0029ee700, 0x4388300, 0xc004bcc660, 0x15f9b135967024dd, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:204 +0x6c0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc004183a40, 0x4388300, 0xc004bcc660, 0x15f9b135967024dd, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:175 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x4388300, 0xc004bcc660, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:925 +0x201
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0028cb5f0, 0x4388300, 0xc004bcc660, 0x3b83687, 0x10, 0xc00094a7b0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:302 +0x140
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0024a4a00, 0x4388300, 0xc004bcc660, 0xc002ac9680, 0xc004bcc660, 0xc004bcc420, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:913 +0x194
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0024a4a00, 0x4388300, 0xc004bcc630, 0xc002ac9680, 0xc004bcc540, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:951 +0x9f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:536
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc003869b80, 0x4388300, 0xc004bcc630, 0x1, 0x434d440, 0xc0059ed430, 0x15f9b135967024dd, 0x0, 0x100000001, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x127
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc003869b80, 0x4388300, 0xc004bcc540, 0x15f9b135967024dd, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x160
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc003372120, 0x4388300, 0xc004bcc540, 0x15f9b135967024dd, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1623 +0x2a2
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc003372120, 0x4388300, 0xc004bcc540, 0x15f9b135967024dd, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:460 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc003372120, 0x4388300, 0xc004bcc540, 0x15f9b135967024dd, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:542 +0x2ee
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc003372120, 0x4388300, 0xc004bcc540, 0x15f9b135967024dd, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1394 +0x3cf
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc003372120, 0x4388300, 0xc004bcc540, 0x15f9b135967024dd, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1079 +0x1580
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc003372120, 0x4388300, 0xc004bcc540, 0x15f9b135967024dd, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:729 +0x7c1
github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc0004ca860, 0x4388300, 0xc004bcc540, 0x15f9b135967024dd, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:223 +0x95
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc0004ca800, 0x4388300, 0xc004bcc540, 0x15f9b135967024dd, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:754 +0x122
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:736
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x4388300, 0xc004bcc540, 0xc00094c548, 0xc00458e100, 0xc00094cb68, 0xc00094c7b0)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:672 +0xf7
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc0004ca800, 0x4388300, 0xc004bcc540, 0xc00458e100, 0x19, 0xc00094c780)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:695 +0x9c
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).MaybePushTransactions(0xc003945790, 0x4388300, 0xc004bcc540, 0xc00094cb68, 0x15f9b13595c996f1, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:357 +0x6e9
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).PushTransaction(0xc003945790, 0x4388300, 0xc004bcc540, 0xc004bb0718, 0x15f9b13595c996f1, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:267 +0x170
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).pushLockTxn(0xc003733170, 0x4388300, 0xc004bcc540, 0xc0005f0b40, 0x15f9b13595c996f1, 0x0, 0x0, 0x0, 0xc00027b8b0, 0x5, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:331 +0x132
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).WaitOn(0xc003733170, 0x4388300, 0xc004bcc540, 0xc0005f0b40, 0x15f9b13595c996f1, 0x0, 0x0, 0x0, 0xc00027b8b0, 0x5, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:270 +0x24e
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc003e24fc0, 0x4388300, 0xc004bcc540, 0xc00065a0e0, 0xc0005f0c60, 0x15f9b13595c996f1, 0x0, 0x0, 0x0, 0xc00027b8b0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:170 +0x43e
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc003e24fc0, 0x4388300, 0xc004bcc540, 0xc00065a0e0, 0xc0005f0c60, 0x15f9b13595c996f1, 0x0, 0x0, 0x0, 0xc00027b8b0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:121 +0xfb
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc00404bc00, 0x4388300, 0xc004bcc540, 0xc002ac9400, 0x3cf2c00, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:192 +0x270
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc00404bc00, 0x4388300, 0xc004bcc510, 0xb, 0xc002ac9400, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:98 +0x2f3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc00404bc00, 0x4388300, 0xc004bcc510, 0x15f9b13595c996f1, 0x0, 0x100000001, 0x1, 0x0, 0xb, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36 +0x91
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0029ee700, 0x4388300, 0xc004bcc4b0, 0x15f9b13595c996f1, 0x0, 0x100000001, 0x1, 0x0, 0xb, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:204 +0x6c0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc004183a40, 0x4388300, 0xc004bcc4b0, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0xb, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:175 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x4388300, 0xc004bcc4b0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:925 +0x201
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0028cb5f0, 0x4388300, 0xc004bcc4b0, 0x3b83687, 0x10, 0xc00094e858, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:302 +0x140
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0024a4a00, 0x4388300, 0xc004bcc4b0, 0xc002ac9280, 0xc004bcc4b0, 0x80, 0x80)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:913 +0x194
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0024a4a00, 0x4388300, 0xc004bcc480, 0xc002ac9280, 0xc004bcc420, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:951 +0x9f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:536
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc003869980, 0x4388300, 0xc004bcc480, 0x1, 0x434d440, 0xc0059ed430, 0x0, 0x0, 0x100000001, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x127
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc003869980, 0x4388300, 0xc004bcc420, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0xb, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x160
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc003372120, 0x4388300, 0xc004bcc420, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1623 +0x2a2
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc003372120, 0x4388300, 0xc004bcc420, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:460 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc003372120, 0x4388300, 0xc004bcc420, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:542 +0x2ee
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc003372120, 0x4388300, 0xc004bcc420, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1394 +0x3cf
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc003372120, 0x4388300, 0xc004bcc420, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1079 +0x1580
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc003372120, 0x4388300, 0xc004bcc420, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:729 +0x7c1
github.com/cockroachdb/cockroach/pkg/kv.(*txnLockGatekeeper).SendLocked(0xc00311da88, 0x4388300, 0xc004bcc420, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_lock_gatekeeper.go:86 +0x10b
github.com/cockroachdb/cockroach/pkg/kv.(*txnMetricRecorder).SendLocked(0xc00311da50, 0x4388300, 0xc004bcc420, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_metric_recorder.go:46 +0x8d
github.com/cockroachdb/cockroach/pkg/kv.(*txnCommitter).SendLocked(0xc00311da20, 0x4388300, 0xc004bcc420, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_committer.go:126 +0x6f9
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).sendLockedWithRefreshAttempts(0xc00311d9b8, 0x4388300, 0xc004bcc420, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:234 +0x9b
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).SendLocked(0xc00311d9b8, 0x4388300, 0xc004bcc420, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:179 +0x158
github.com/cockroachdb/cockroach/pkg/kv.(*txnPipeliner).SendLocked(0xc00311d908, 0x4388300, 0xc004bcc420, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_pipeliner.go:209 +0x159
github.com/cockroachdb/cockroach/pkg/kv.(*txnSeqNumAllocator).SendLocked(0xc00311d8e8, 0x4388300, 0xc004bcc420, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_seq_num_allocator.go:105 +0x20c
github.com/cockroachdb/cockroach/pkg/kv.(*txnHeartbeater).SendLocked(0xc00311d848, 0x4388300, 0xc004bcc420, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_heartbeater.go:172 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc00311d680, 0x4388300, 0xc004bcc420, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:503 +0x406
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc0004ca800, 0x4388240, 0xc003869740, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:754 +0x122
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send(0xc004cdebd0, 0x4388240, 0xc003869740, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:870 +0x11e
github.com/cockroachdb/cockroach/pkg/sql/row.makeKVBatchFetcher.func1(0x4388240, 0xc003869740, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:154 +0x76
github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).fetch(0xc002d67380, 0x4388240, 0xc003869740, 0x203000, 0x203001)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:284 +0x671
github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).nextBatch(0xc002d67380, 0x4388240, 0xc003869740, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:368 +0x1af
github.com/cockroachdb/cockroach/pkg/sql/row.(*KVFetcher).NextKV(0xc002ac9200, 0x4388240, 0xc003869740, 0xc000951c10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/kv_fetcher.go:87 +0x114
github.com/cockroachdb/cockroach/pkg/sql/row.(*Fetcher).NextKey(0xc0005f07e0, 0x4388240, 0xc003869740, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/fetcher.go:598 +0x5e
github.com/cockroachdb/cockroach/pkg/sql/row.(*Fetcher).StartScanFrom(0xc0005f07e0, 0x4388240, 0xc003869740, 0x43494c0, 0xc002d67380, 0x0, 0x100000000)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/fetcher.go:588 +0x97
github.com/cockroachdb/cockroach/pkg/sql/row.(*Fetcher).StartScan(0xc0005f07e0, 0x4388240, 0xc003869740, 0xc004cdebd0, 0xc003ceab40, 0x5, 0x5, 0xc000951f01, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/row/fetcher.go:484 +0x1da
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*tableReader).Start(0xc00404b500, 0x4388240, 0xc003869740, 0x1, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/tablereader.go:162 +0x23c
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexJoiner).Start(0xc000154000, 0x4388240, 0xc003869740, 0xc0055b8000, 0x14)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/indexjoiner.go:138 +0x52
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*sortAllProcessor).Start(0xc0024a4000, 0x4388240, 0xc003869740, 0x6610400, 0x35ad6c0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/rowexec/sorter.go:275 +0x52
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBase).Run(0xc0024a4000, 0x4388240, 0xc003869740)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:748 +0x52
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run(0xc0005f06c0, 0x4388240, 0xc003869740, 0x3cf3740, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:370 +0x247
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc0004403c0, 0xc003259c20, 0xc004cdebd0, 0xc0009526b8, 0xc000367600, 0xc003c234c8, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:378 +0x426
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0xc0004403c0, 0x4388300, 0xc0021d7290, 0xc003c234c8, 0xc003259c20, 0xc004cdebd0, 0x438a480, 0xc00441bae0, 0xc000367600, 0xc003c231c8)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:988 +0x225
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc003c23000, 0x4388300, 0xc0021d7290, 0xc003c233e8, 0x3, 0x43e4420, 0xc003258a80, 0xc000952a00, 0xc00027adb8, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:843 +0x3cf
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc003c23000, 0x4388300, 0xc0021d7290, 0xc003c233e8, 0x43e4420, 0xc003258a80, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:750 +0x6cd
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc003c23000, 0x4388300, 0xc0021d7290, 0x43a1900, 0xc00027a640, 0x3c50ae4, 0x78, 0x5, 0x1, 0xc002ac8700, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:455 +0xac8
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc003c23000, 0x4388300, 0xc0021d7290, 0x43a1900, 0xc00027a640, 0x3c50ae4, 0x78, 0x5, 0x1, 0xc002ac8700, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:117 +0x4f6
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc003c23000, 0x4388240, 0xc003868f00, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1458 +0xed8
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc003c23000, 0x4388300, 0xc0021d6300, 0xc004343ae0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1325 +0x1f2
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx.func1(0xc003c23000, 0x4388300, 0xc0021d6300, 0xc007722180, 0xc00505e320, 0x0, 0xc004dd2030)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:158 +0x64
created by github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx
/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:157 +0x33a
goroutine 195645 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x4388300, 0xc0039b9e30)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:99 +0x13f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ecd50, 0xc0028cb5f0, 0xc0059ecd40)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195747 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18bd)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432e70)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432e70)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f206d0, 0xc0028cb5f0, 0xc005f206c0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195696 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*Server).PeriodicallyClearSQLStats.func1(0x4388300, 0xc00318cc00)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:765 +0x248
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22f70, 0xc0028cb5f0, 0xc005ed2ea0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195497 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0010e0940, 0x1, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x11a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc003259500, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1e3
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc001811980)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfa4
goroutine 199085 [select, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.acquireNodeLease(0x4388300, 0xc008a97410, 0xc003372360, 0xc000000036, 0xc002e88e40, 0x0, 0x1)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1020 +0x23b
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).refreshSomeLeases.func1(0x4388300, 0xc008a97410)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1875 +0xab
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunLimitedAsyncTask.func2(0xc0028cb5f0, 0x4388300, 0xc008a97410, 0xc0016f6020, 0x16, 0xc00750c920, 0x0, 0x0, 0xc0002cc000)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:382 +0x19b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunLimitedAsyncTask
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:376 +0x274
goroutine 195639 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x4388300, 0xc0039b9cb0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:99 +0x13f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ecc90, 0xc0028cb5f0, 0xc0059ecc80)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195723 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18a7)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432960)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432960)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f201c0, 0xc0028cb5f0, 0xc005f201b0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195586 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x4388300, 0xc0039b9b30)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:99 +0x13f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ecbc0, 0xc0028cb5f0, 0xc0059ecbb0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 197283 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc002480b00)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x656
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0x120
goroutine 195684 [chan receive]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/provider.(*Provider).Notify.func1(0x4388300, 0xc005ed2810)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/provider/provider.go:220 +0x147
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22c20, 0xc0028cb5f0, 0xc0021789a0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195661 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func3(0x4388300, 0xc004d34630)
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1891 +0x7c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ed590, 0xc0028cb5f0, 0xc003e24640)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195637 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x4388300, 0xc0039b9c50)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:99 +0x13f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ecc20, 0xc0028cb5f0, 0xc0059ecc10)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195762 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bef10, 0x750)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bef00)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/provider.(*Provider).Subscribe(0xc001caddc0, 0x4388300, 0xc004bd7320, 0xc00659bec0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/provider/provider.go:293 +0x455
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startClosedTimestampRangefeedSubscriber.func1(0x4388300, 0xc004bd7320)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:1623 +0x57
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0028cb5f0, 0x4388300, 0xc004bd7320, 0xc000186990, 0x25, 0x0, 0x0, 0xc00334d200)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:323 +0x152
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:318 +0x131
goroutine 195649 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).gcDaemon(0xc003a491f0, 0x4388240, 0xc003e24180)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_gc.go:44 +0xcc
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:87 +0x2b6
goroutine 195736 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18ad)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432bd0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432bd0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20500, 0xc0028cb5f0, 0xc005f204e0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195682 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func1(0x4388300, 0xc005ed2c90)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:808 +0x176
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0076d65b0, 0xc0028cb5f0, 0xc00799de80)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 197256 [IO wait]:
internal/poll.runtime_pollWait(0x7f22e76e1590, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc000bb4998, 0x72, 0x700, 0x775, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000bb4980, 0xc000a74800, 0x775, 0x775, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc000bb4980, 0xc000a74800, 0x775, 0x775, 0x22800000088, 0x294a5294a5, 0x37)
/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc005cd3928, 0xc000a74800, 0x775, 0x775, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:184 +0x68
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc0060549c0, 0xc000a74800, 0x775, 0x775, 0xc005e6b401, 0x10, 0x10)
/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x10f
crypto/tls.(*atLeastReader).Read(0xc00286d360, 0xc000a74800, 0x775, 0x775, 0x0, 0x0, 0xc0010429a8)
/usr/local/go/src/crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc000e6ccd8, 0x430bb00, 0xc00286d360, 0x66a285, 0x36fce40, 0x394f4e0)
/usr/local/go/src/bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc000e6ca80, 0x7f22e7719d40, 0xc0060549c0, 0x5, 0xc0060549c0, 0x48)
/usr/local/go/src/crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc000e6ca80, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc000e6ca80, 0xc001440000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1255 +0x161
bufio.(*Reader).Read(0xc005d8c3c0, 0xc001a241f8, 0x9, 0x9, 0xc0042d9e88, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x430b940, 0xc005d8c3c0, 0xc001a241f8, 0x9, 0x9, 0x9, 0x2e4d473d40b9a701, 0x2e00000000000000, 0x0)
/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
/usr/local/go/src/io/io.go:329
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc001a241f8, 0x9, 0x9, 0x430b940, 0xc005d8c3c0, 0x0, 0x33100000000, 0xc003f88700, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x87
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc001a241c0, 0xc005b20b70, 0xc005b20b70, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc004b14000, 0xc005b20cc0, 0x3cf8f48)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x86
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc007026f00, 0x43dea00, 0xc004b14000)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xf3
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc007026f00, 0x43dea00, 0xc004b14000)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x5e0
goroutine 195113 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18cc)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432150)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432150)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400dca0, 0xc0028cb5f0, 0xc00400dc90)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195758 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startGossip.func4(0x4388300, 0xc005ed28d0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:1553 +0x364
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20920, 0xc0028cb5f0, 0xc000e9bf90)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195701 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18c8)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432390)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432390)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400deb0, 0xc0028cb5f0, 0xc00400dea0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195651 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).gcDaemon(0xc003a492d0, 0x4388240, 0xc003e24180)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_gc.go:44 +0xcc
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:87 +0x2b6
goroutine 195678 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func1(0x4388300, 0xc004bcc360)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:808 +0x176
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0076d6550, 0xc0028cb5f0, 0xc00799de00)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195582 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/rpc.NewContextWithTestingKnobs.func1(0x4388300, 0xc008a97350)
/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:474 +0x51
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0004357e0, 0xc0028cb5f0, 0xc00528c940)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195687 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/server.periodicallyPersistHLCUpperBound(0xc004e59630, 0xc001f3da40, 0xc004d22d90, 0x3cf9fb8, 0xc002e88ea0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1132 +0x18b
github.com/cockroachdb/cockroach/pkg/server.(*Server).startPersistingHLCUpperBound.func2(0x4388300, 0xc0062437d0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1206 +0x6a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22da0, 0xc0028cb5f0, 0xc005ed2cf0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195642 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x4388300, 0xc0039b9d40)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:99 +0x13f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059eccf0, 0xc0028cb5f0, 0xc0059ecce0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195699 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18c5)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432330)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432330)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400de70, 0xc0028cb5f0, 0xc00400de60)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195676 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func1(0x4388300, 0xc005ed2ab0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:808 +0x176
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0076d6530, 0xc0028cb5f0, 0xc00799ddc0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 198201 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait.(*Queue).MaybeWaitForQuery(0xc00602f9a0, 0x4388300, 0xc0048a6810, 0xc0021a8210, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait/queue.go:747 +0x504
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).maybeInterceptReq(0xc003e24d40, 0x4388300, 0xc0048a6810, 0x0, 0x15f9b23440e967bc, 0x0, 0x0, 0x0, 0xc00438c980, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:199 +0x22f
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc003e24d40, 0x4388300, 0xc0048a6810, 0xc004a76000, 0x0, 0x15f9b23440e967bc, 0x0, 0x0, 0x0, 0xc00438c980, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:142 +0xb8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc003e24d40, 0x4388300, 0xc0048a6810, 0x0, 0x0, 0x15f9b23440e967bc, 0x0, 0x0, 0x0, 0xc00438c980, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:121 +0xfb
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc00267ea00, 0x4388300, 0xc0048a6810, 0xc00438ca80, 0x3cf2c00, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:192 +0x270
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc00267ea00, 0x4388300, 0xc0048a67e0, 0x6, 0xc00438ca80, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:98 +0x2f3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc00267ea00, 0x4388300, 0xc0048a67e0, 0x15f9b23440e967bc, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36 +0x91
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0029ee700, 0x4388300, 0xc0048a6780, 0x15f9b23440e967bc, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:204 +0x6c0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc004183a40, 0x4388300, 0xc0048a6780, 0x15f9b23440e967bc, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:175 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x4388300, 0xc0048a6780, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:925 +0x201
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0028cb5f0, 0x4388300, 0xc0048a6780, 0x3b83687, 0x10, 0xc001f07db8, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:302 +0x140
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0024a4a00, 0x4388300, 0xc0048a6780, 0xc00438ca00, 0xc0048a6780, 0x80, 0x80)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:913 +0x194
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0024a4a00, 0x4388300, 0xc0048a6750, 0xc00438ca00, 0xc002f9cf80, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:951 +0x9f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:536
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc00103e8c0, 0x4388300, 0xc0048a6750, 0x1, 0x434d440, 0xc0059ed430, 0x15f9b23440e967bc, 0x0, 0x100000001, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x127
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc00103e8c0, 0x4388240, 0xc002f9cf80, 0x15f9b23440e967bc, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x160
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc003372120, 0x4388240, 0xc002f9cf80, 0x15f9b23440e967bc, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1623 +0x2a2
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc003372120, 0x4388240, 0xc002f9cf80, 0x15f9b23440e967bc, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:460 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc003372120, 0x4388240, 0xc002f9cf80, 0x15f9b23440e967bc, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:542 +0x2ee
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc003372120, 0x4388240, 0xc002f9cf80, 0x15f9b23440e967bc, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1394 +0x3cf
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc003372120, 0x4388240, 0xc002f9cf80, 0x15f9b23440e967bc, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1079 +0x1580
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc003372120, 0x4388240, 0xc002f9cf80, 0x15f9b23440e967bc, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:729 +0x7c1
github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc0004ca860, 0x4388240, 0xc002f9cf80, 0x15f9b23440e967bc, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:223 +0x95
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc0004ca800, 0x4388240, 0xc002f9cf80, 0x15f9b23440e967bc, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:754 +0x122
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:736
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x4388240, 0xc002f9cf80, 0xc001f09b50, 0xc0010fdb80, 0x0, 0xc001f09bc8)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:672 +0xf7
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc0004ca800, 0x4388240, 0xc002f9cf80, 0xc0010fdb80, 0x15f9b23440e96273, 0x8)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:695 +0x9c
github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait.(*Queue).queryTxnStatus(0xc00602f9a0, 0x4388240, 0xc002f9cf80, 0x944d01a3503a9dd3, 0x5d111ceaae658690, 0xc000052748, 0x1, 0x8, 0x0, 0x15f9b13589b562ad, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait/queue.go:874 +0x199
github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait.(*Queue).startQueryPusherTxn.func1(0x4388240, 0xc002f9cf80)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait/queue.go:796 +0x436
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0028cb5f0, 0x4388240, 0xc002f9cf80, 0xc0014f4580, 0x1d, 0x43f24a0, 0xc00881d560, 0xc002f9cfc0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:323 +0x152
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:318 +0x131
goroutine 199087 [select, 16 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.acquireNodeLease(0x4388300, 0xc008a97410, 0xc003372360, 0x35, 0xc000052130, 0x43e9020, 0xc003c27c00)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1020 +0x23b
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).refreshSomeLeases.func1(0x4388300, 0xc008a97410)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1875 +0xab
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunLimitedAsyncTask.func2(0xc0028cb5f0, 0x4388300, 0xc008a97410, 0xc0016f6080, 0x16, 0xc00750c650, 0x0, 0x0, 0xc0002cc040)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:382 +0x19b
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunLimitedAsyncTask
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:376 +0x274
goroutine 195636 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x4388300, 0xc0039b9c20)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:99 +0x13f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ecc00, 0xc0028cb5f0, 0xc0059ecbf0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195753 [semacquire, 18 minutes]:
sync.runtime_Semacquire(0xc0034b99f8)
/usr/local/go/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc0034b99f0)
/usr/local/go/src/sync/waitgroup.go:130 +0x64
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Wait(...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:172
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20860, 0xc0028cb5f0, 0xc005f20850)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 197661 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait.(*Queue).MaybeWaitForPush(0xc00602f9a0, 0x4388300, 0xc000f6e270, 0xc002448000, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait/queue.go:513 +0xf95
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).maybeInterceptReq(0xc003e24d40, 0x4388300, 0xc000f6e270, 0x0, 0x15f9b13c867ac76a, 0x0, 0x0, 0x0, 0xc0037e8780, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:189 +0xa9
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc003e24d40, 0x4388300, 0xc000f6e270, 0xc0086d2ee0, 0x0, 0x15f9b13c867ac76a, 0x0, 0x0, 0x0, 0xc0037e8780, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:142 +0xb8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc003e24d40, 0x4388300, 0xc000f6e270, 0x0, 0x0, 0x15f9b13c867ac76a, 0x0, 0x0, 0x0, 0xc0037e8780, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:121 +0xfb
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc00267ea00, 0x4388300, 0xc000f6e270, 0xc0037e8900, 0x3cf2c18, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:192 +0x270
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc00267ea00, 0x4388300, 0xc000f6e240, 0x6, 0xc0037e8900, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:94 +0x68a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc00267ea00, 0x4388300, 0xc000f6e240, 0x15f9b13c867ac76a, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36 +0x91
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0029ee700, 0x4388300, 0xc000f6e150, 0x15f9b13c867ac76a, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:204 +0x6c0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc004183a40, 0x4388300, 0xc000f6e150, 0x15f9b13c867ac76a, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:175 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x4388300, 0xc000f6e150, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:925 +0x201
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0028cb5f0, 0x4388300, 0xc000f6e150, 0x3b83687, 0x10, 0xc001c94718, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:302 +0x140
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0024a4a00, 0x4388300, 0xc000f6e150, 0xc0037e8880, 0xc000f6e150, 0xc001160b10, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:913 +0x194
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0024a4a00, 0x4388300, 0xc000f6e120, 0xc0037e8880, 0xc001160c90, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:951 +0x9f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:536
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc002f9ce40, 0x4388300, 0xc000f6e120, 0x1, 0x434d440, 0xc0059ed430, 0x15f9b13c867ac76a, 0x0, 0x100000001, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x127
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc002f9ce40, 0x4388300, 0xc001160c90, 0x15f9b13c867ac76a, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x160
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc003372120, 0x4388300, 0xc001160c90, 0x15f9b13c867ac76a, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1623 +0x2a2
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc003372120, 0x4388300, 0xc001160c90, 0x15f9b13c867ac76a, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:460 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc003372120, 0x4388300, 0xc001160c90, 0x15f9b13c867ac76a, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:542 +0x2ee
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc003372120, 0x4388300, 0xc001160c90, 0x15f9b13c867ac76a, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1394 +0x3cf
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc003372120, 0x4388300, 0xc001160c90, 0x15f9b13c867ac76a, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1079 +0x1580
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc003372120, 0x4388300, 0xc001160c90, 0x15f9b13c867ac76a, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:729 +0x7c1
github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc0004ca860, 0x4388300, 0xc001160c90, 0x15f9b13c867ac76a, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:223 +0x95
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc0004ca800, 0x4388300, 0xc001160c90, 0x15f9b13c867ac76a, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:754 +0x122
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:736
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x4388300, 0xc001160c90, 0xc001c964b0, 0xc002e6d080, 0xc001c96ad0, 0xc001c96718)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:672 +0xf7
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc0004ca800, 0x4388300, 0xc001160c90, 0xc002e6d080, 0x19, 0xc001c966e8)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:695 +0x9c
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).MaybePushTransactions(0xc003945790, 0x4388300, 0xc001160c90, 0xc001c96ad0, 0x15f9b13c85d87c72, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:357 +0x6e9
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).PushTransaction(0xc003945790, 0x4388300, 0xc001160c90, 0xc000270868, 0x15f9b13c85d87c72, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:267 +0x170
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).pushLockTxn(0xc007c9f530, 0x4388300, 0xc001160c90, 0xc000385200, 0x15f9b13589b562ad, 0x0, 0x0, 0x0, 0xc0037e8300, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:331 +0x132
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).WaitOn(0xc007c9f530, 0x4388300, 0xc001160c90, 0xc000385200, 0x15f9b13589b562ad, 0x0, 0x0, 0x0, 0xc0037e8300, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:270 +0x24e
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc003e24d40, 0x4388300, 0xc001160c90, 0xc004520070, 0xc000385200, 0x15f9b13589b562ad, 0x0, 0x0, 0x0, 0xc0037e8300, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:170 +0x43e
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc003e24d40, 0x4388300, 0xc001160c90, 0xc004520070, 0xc000385200, 0x15f9b13589b562ad, 0x0, 0x0, 0x0, 0xc0037e8300, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:121 +0xfb
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc00267ea00, 0x4388300, 0xc001160c90, 0xc0037e8400, 0x3cf2c00, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:192 +0x270
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc00267ea00, 0x4388300, 0xc001160c60, 0x6, 0xc0037e8400, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:98 +0x2f3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc00267ea00, 0x4388300, 0xc001160c60, 0x15f9b13589b562ad, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36 +0x91
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0029ee700, 0x4388300, 0xc001160ba0, 0x15f9b13589b562ad, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:204 +0x6c0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc004183a40, 0x4388300, 0xc001160ba0, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:175 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x4388300, 0xc001160ba0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:925 +0x201
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0028cb5f0, 0x4388300, 0xc001160ba0, 0x3b83687, 0x10, 0xc001b167c0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:302 +0x140
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0024a4a00, 0x4388300, 0xc001160ba0, 0xc0037e8380, 0xc001160ba0, 0x80, 0x80)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:913 +0x194
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0024a4a00, 0x4388300, 0xc001160b70, 0xc0037e8380, 0xc001160b10, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:951 +0x9f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:536
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc002f9c3c0, 0x4388300, 0xc001160b70, 0x1, 0x434d440, 0xc0059ed430, 0x0, 0x0, 0x100000001, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x127
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc002f9c3c0, 0x4388300, 0xc001160b10, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x160
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc003372120, 0x4388300, 0xc001160b10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1623 +0x2a2
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc003372120, 0x4388300, 0xc001160b10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:460 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc003372120, 0x4388300, 0xc001160b10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:542 +0x2ee
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc003372120, 0x4388300, 0xc001160b10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1394 +0x3cf
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc003372120, 0x4388300, 0xc001160b10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1079 +0x1580
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc003372120, 0x4388300, 0xc001160b10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:729 +0x7c1
github.com/cockroachdb/cockroach/pkg/kv.(*txnLockGatekeeper).SendLocked(0xc0049e4408, 0x4388300, 0xc001160b10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_lock_gatekeeper.go:86 +0x10b
github.com/cockroachdb/cockroach/pkg/kv.(*txnMetricRecorder).SendLocked(0xc0049e43d0, 0x4388300, 0xc001160b10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_metric_recorder.go:46 +0x8d
github.com/cockroachdb/cockroach/pkg/kv.(*txnCommitter).SendLocked(0xc0049e43a0, 0x4388300, 0xc001160b10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_committer.go:126 +0x6f9
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).sendLockedWithRefreshAttempts(0xc0049e4338, 0x4388300, 0xc001160b10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:234 +0x9b
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).SendLocked(0xc0049e4338, 0x4388300, 0xc001160b10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:179 +0x158
github.com/cockroachdb/cockroach/pkg/kv.(*txnPipeliner).SendLocked(0xc0049e4288, 0x4388300, 0xc001160b10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_pipeliner.go:209 +0x159
github.com/cockroachdb/cockroach/pkg/kv.(*txnSeqNumAllocator).SendLocked(0xc0049e4268, 0x4388300, 0xc001160b10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_seq_num_allocator.go:105 +0x20c
github.com/cockroachdb/cockroach/pkg/kv.(*txnHeartbeater).SendLocked(0xc0049e41c8, 0x4388300, 0xc001160b10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_heartbeater.go:172 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc0049e4000, 0x4388300, 0xc001160b10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:503 +0x406
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc0004ca800, 0x4388240, 0xc002f9c280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:754 +0x122
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send(0xc0059e6120, 0x4388240, 0xc002f9c280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:870 +0x11e
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x4388240, 0xc002f9c280, 0xc002afb488, 0xc002e6c580, 0x1, 0x1)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:672 +0xf7
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Run(0xc0059e6120, 0x4388240, 0xc002f9c280, 0xc002e6c580, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:538 +0xfa
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Get(0xc0059e6120, 0x4388240, 0xc002f9c280, 0x391f980, 0xc0002325c0, 0x8, 0xc002afb6b8, 0x11b64ba, 0xc0005ecb90, 0xc002afb6b8, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:348 +0xbc
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).GetProtoTs(0xc0059e6120, 0x4388240, 0xc002f9c280, 0x391f980, 0xc0002325c0, 0x43c2020, 0xc004adc270, 0xc002d68000, 0x4312060, 0xc004adc090, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:370 +0x66
github.com/cockroachdb/cockroach/pkg/sql/sqlbase.GetDatabaseDescFromID(0x4388240, 0xc002f9c280, 0x430bf20, 0xc0059e6120, 0x34, 0x0, 0x0, 0x15f9b13c85d842c6)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/structured.go:342 +0xc7
github.com/cockroachdb/cockroach/pkg/sql.(*planner).getQualifiedTableName(0xc002d683e8, 0x4388240, 0xc002f9c280, 0xc003775200, 0x35, 0x7f22e779db00, 0xc003775200, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/resolver.go:373 +0x78
github.com/cockroachdb/cockroach/pkg/sql.(*createStatsNode).makeJobRecord(0xc000270380, 0x4388240, 0xc002f9c280, 0xc002ff4e78, 0x8b2387ef, 0x10)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/create_stats.go:164 +0x993
github.com/cockroachdb/cockroach/pkg/sql.(*createStatsNode).startJob(0xc000270380, 0x4388240, 0xc002f9c280, 0xc000cee540, 0x0, 0xc002ff4f98)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/create_stats.go:106 +0x68
github.com/cockroachdb/cockroach/pkg/sql.(*createStatsNode).startExec.func1(0xc000270380, 0x4388240, 0xc002f9c280, 0xc001744000, 0xc002d683e8)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/create_stats.go:79 +0x63
created by github.com/cockroachdb/cockroach/pkg/sql.(*createStatsNode).startExec
/go/src/github.com/cockroachdb/cockroach/pkg/sql/create_stats.go:78 +0x129
goroutine 195640 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x4388300, 0xc0039b9a70)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:99 +0x13f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059eccb0, 0xc0028cb5f0, 0xc0059ecca0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195710 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x189a)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432660)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432660)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400dff0, 0xc0028cb5f0, 0xc00400dfe0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195680 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func1(0x4388300, 0xc005ed2c60)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:808 +0x176
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0076d6570, 0xc0028cb5f0, 0xc00799de40)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195641 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x4388300, 0xc0039b9d10)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:99 +0x13f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059eccd0, 0xc0028cb5f0, 0xc0059eccc0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 197278 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00a96d840, 0x1, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x11a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00601be00, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1e3
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc001aea1c0)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xee5
goroutine 197254 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0010e0400, 0x1, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x11a
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc005d8c420, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1e3
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc004b14000)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfa4
goroutine 195702 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18c9)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc0014323c0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc0014323c0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400ded0, 0xc0028cb5f0, 0xc00400dec0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 196855 [IO wait, 18 minutes]:
internal/poll.runtime_pollWait(0x7f22e76e1730, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc003da7798, 0x72, 0x0, 0x0, 0x3b6d4d2)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc003da7780, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1f8
net.(*netFD).accept(0xc003da7780, 0x37e11d600, 0x0, 0x0)
/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0052e0b00, 0xc000f4ade8, 0xc000f4adf0, 0x38)
/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0052e0b00, 0x3cf6b18, 0xc0028cb5f0, 0x4388300, 0xc004615290)
/usr/local/go/src/net/tcpsock.go:261 +0x47
github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith(0xc000df08e0, 0x4388300, 0xc004615290, 0xc0028cb5f0, 0x4374600, 0xc0052e0b00, 0xc0046152c0, 0xc004615290, 0xc001b5cee8)
/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:126 +0x50
github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeSQL.func1(0x4388300, 0xc004615290)
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1994 +0xdd
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059aeb90, 0xc0028cb5f0, 0xc004270db0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195703 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x1899)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc0014323f0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc0014323f0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400def0, 0xc0028cb5f0, 0xc00400dee0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195670 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x4388300, 0xc004d86450)
/go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1366 +0x3f7
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ed8a0, 0xc0028cb5f0, 0xc0059ed890)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195655 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x4388300, 0xc0039b9ec0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:107 +0x64
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ed490, 0xc0028cb5f0, 0xc0052e09e0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195689 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).refreshSettings.func2(0x4388300, 0xc004e7d410)
/go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:110 +0x106
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22dd0, 0xc0028cb5f0, 0xc002178de0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195725 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x1898)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc0014329c0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc0014329c0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20200, 0xc0028cb5f0, 0xc005f201f0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195731 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18a0)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432ae0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432ae0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20410, 0xc0028cb5f0, 0xc005f203d0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195643 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x4388300, 0xc0039b9d70)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:99 +0x13f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ecd10, 0xc0028cb5f0, 0xc0059ecd00)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195119 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18c2)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432270)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432270)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400dde0, 0xc0028cb5f0, 0xc00400ddd0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195742 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18b3)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432d50)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432d50)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20620, 0xc0028cb5f0, 0xc005f20610)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195764 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*StoreRebalancer).Start.func1(0x4388300, 0xc00318cb40)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_rebalancer.go:186 +0x19a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20a40, 0xc0028cb5f0, 0xc00334d240)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195738 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18b9)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432c60)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432c60)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20550, 0xc0028cb5f0, 0xc005f20540)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195120 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18c7)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc0014322a0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc0014322a0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400de00, 0xc0028cb5f0, 0xc00400ddf0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195739 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18b4)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432c90)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432c90)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f205b0, 0xc0028cb5f0, 0xc005f205a0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195121 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18d2)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc0014322d0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc0014322d0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400de30, 0xc0028cb5f0, 0xc00400de10)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195114 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18cd)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432180)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432180)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400dcc0, 0xc0028cb5f0, 0xc00400dcb0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195693 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowScheduler).Start.func1(0x4388300, 0xc004e7d560)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow_scheduler.go:149 +0x15d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22f40, 0xc0028cb5f0, 0xc004d22f00)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 197277 [IO wait]:
internal/poll.runtime_pollWait(0x7f22e77482d8, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc000970298, 0x72, 0x800, 0x888, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000970280, 0xc0000e3200, 0x888, 0x888, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc000970280, 0xc0000e3200, 0x888, 0x888, 0x203001, 0x0, 0xc001041978)
/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0004c6678, 0xc0000e3200, 0x888, 0x888, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:184 +0x68
crypto/tls.(*atLeastReader).Read(0xc0066d6ba0, 0xc0000e3200, 0x888, 0x888, 0x7f22e0794d38, 0xc0000b2700, 0xc001041a08)
/usr/local/go/src/crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc000e70258, 0x430bb00, 0xc0066d6ba0, 0x66a285, 0x36fce40, 0x3ac61c0)
/usr/local/go/src/bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc000e70000, 0x4311220, 0xc0004c6678, 0x5, 0xc0004c6678, 0x8)
/usr/local/go/src/crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc000e70000, 0x0, 0x0, 0xc0044873d8)
/usr/local/go/src/crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc000e70000, 0xc001480000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1255 +0x161
bufio.(*Reader).Read(0xc008a6a720, 0xc0017103b8, 0x9, 0x9, 0x20, 0xc001041d18, 0xc8f31e)
/usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x430b940, 0xc008a6a720, 0xc0017103b8, 0x9, 0x9, 0x9, 0xc2979c8ba7391001, 0x0, 0x0)
/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
/usr/local/go/src/io/io.go:329
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0017103b8, 0x9, 0x9, 0x430b940, 0xc008a6a720, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x87
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc001710380, 0xc004e77500, 0xc004e77500, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc001aea1c0)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x17e
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd48
goroutine 196862 [select, 18 minutes]:
database/sql.(*DB).connectionResetter(0xc00295acc0, 0x4388240, 0xc0049506c0)
/usr/local/go/src/database/sql/sql.go:1065 +0xfb
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:723 +0x193
goroutine 195737 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18b0)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432c00)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432c00)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20530, 0xc0028cb5f0, 0xc005f20520)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195751 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18b7)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432f30)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432f30)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20820, 0xc0028cb5f0, 0xc005f207a0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195650 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).gcDaemon(0xc003a49260, 0x4388240, 0xc003e24180)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_gc.go:44 +0xcc
created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:87 +0x2b6
goroutine 195660 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func1(0x4388300, 0xc004d34210)
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1834 +0x53
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ed570, 0xc0028cb5f0, 0xc0052e0b20)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195750 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18c1)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432f00)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432f00)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20790, 0xc0028cb5f0, 0xc005f20760)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195814 [select, 1 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptcache.(*Cache).periodicallyRefreshProtectedtsCache(0xc0045d38c0, 0x4388300, 0xc005a31020)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptcache/cache.go:160 +0x278
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0028cb5f0, 0x4388300, 0xc005a31020, 0xc001d08150, 0x2e, 0x0, 0x0, 0xc004d22940)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:323 +0x152
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:318 +0x131
goroutine 195730 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18b6)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432ab0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432ab0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f203c0, 0xc0028cb5f0, 0xc005f203b0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195707 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18ca)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc0014325a0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc0014325a0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400df90, 0xc0028cb5f0, 0xc00400df80)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195746 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18ba)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432e40)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432e40)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f206b0, 0xc0028cb5f0, 0xc005f206a0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195658 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
net/http.(*Server).Serve(0xc0002f5500, 0x4375580, 0xc0052e0a40, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2896 +0x280
github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func3(0x4388300, 0xc0039b9f20)
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1965 +0x127
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ed510, 0xc0028cb5f0, 0xc004739b30)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195695 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*Server).PeriodicallyClearSQLStats.func1(0x4388300, 0xc004e7d590)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:765 +0x248
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22f60, 0xc0028cb5f0, 0xc005ed2e70)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195691 [select]:
github.com/cockroachdb/cockroach/pkg/ts.(*poller).start.func1(0x4388300, 0xc004b62b40)
/go/src/github.com/cockroachdb/cockroach/pkg/ts/db.go:166 +0x124
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22ed0, 0xc0028cb5f0, 0xc004d22ec0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195722 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18a1)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432930)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432930)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f201a0, 0xc0028cb5f0, 0xc005f20190)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195714 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x189b)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432720)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432720)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20080, 0xc0028cb5f0, 0xc005f20070)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195761 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startLeaseRenewer.func1(0x4388300, 0xc005ed2a80)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:1602 +0x2fe
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20a10, 0xc0028cb5f0, 0xc005f20a00)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195812 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func1(0x4388300, 0xc001d46c00)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:790 +0x125
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22920, 0xc0028cb5f0, 0xc00799db40)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195654 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps.func1(0x3cf9fb8, 0xc004e59630, 0xc001e16a10, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:181 +0x11a
created by github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps
/go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:174 +0x104
goroutine 195741 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18bc)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432d20)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432d20)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20600, 0xc0028cb5f0, 0xc005f205f0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195685 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1(0x4388300, 0xc004b62840)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:728 +0x13b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22d00, 0xc0028cb5f0, 0xc002178ce0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195675 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func1(0x4388300, 0xc005ed2b10)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:808 +0x176
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0076d6520, 0xc0028cb5f0, 0xc00799dda0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195757 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startGossip.func4(0x4388300, 0xc001433050)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:1553 +0x364
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20910, 0xc0028cb5f0, 0xc000e9bef0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 196853 [select, 9 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startSystemLogsGC.func1(0x4388300, 0xc004cb5260)
/go/src/github.com/cockroachdb/cockroach/pkg/server/server_systemlog_gc.go:177 +0x553
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059aea60, 0xc0028cb5f0, 0xc006cb3c80)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195494 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc004d2e580)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x656
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0x120
goroutine 195602 [semacquire, 18 minutes]:
sync.runtime_SemacquireMutex(0xc002056118, 0x0, 0x1)
/usr/local/go/src/runtime/sema.go:71 +0x47
sync.(*Mutex).lockSlow(0xc002056114)
/usr/local/go/src/sync/mutex.go:138 +0xfc
sync.(*Mutex).Lock(...)
/usr/local/go/src/sync/mutex.go:81
sync.(*Once).doSlow(0xc002056110, 0xc001cf5ed0)
/usr/local/go/src/sync/once.go:62 +0x122
sync.(*Once).Do(...)
/usr/local/go/src/sync/once.go:57
github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1(0x4388300, 0xc005ed2870)
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1915 +0x73
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22af0, 0xc0028cb5f0, 0xc002178760)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195583 [select]:
github.com/cockroachdb/cockroach/pkg/gossip.newInfoStore.func1(0x4388300, 0xc0039b9a40)
/go/src/github.com/cockroachdb/cockroach/pkg/gossip/infostore.go:190 +0x169
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0004358c0, 0xc0028cb5f0, 0xc0004358a0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195665 [select, 18 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc003e24880)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x10f
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x169
goroutine 196860 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal(0xc007722180, 0x4388300, 0xc0021d6300, 0x3b72096, 0x9, 0x0, 0x3b68183, 0x4, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:475 +0xa3e
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternal(0xc007722180, 0x4388300, 0xc004615320, 0x3b72096, 0x9, 0x0, 0x3b68183, 0x4, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:224 +0xef
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryEx(0xc007722180, 0x4388300, 0xc004615320, 0x3b72096, 0x9, 0x0, 0x3b68183, 0x4, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:199 +0xec
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).Query(0xc007722180, 0x4388300, 0xc004615320, 0x3b72096, 0x9, 0x0, 0x3c50ae3, 0x79, 0xc00027a2d0, 0x5, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:183 +0x195
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).maybeAdoptJob(0xc003385b00, 0x4388300, 0xc004615320, 0x4348000, 0xc004ea2a00, 0xc004ae6438, 0xc001e89670)
/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:922 +0x116
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func3(0x4388300, 0xc004615320)
/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:465 +0xa6
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func4(0x4388300, 0xc004615320)
/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:478 +0x166
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059aec40, 0xc0028cb5f0, 0xc004270e40)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195648 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x4388300, 0xc0039b9e90)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:99 +0x13f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ecdc0, 0xc0028cb5f0, 0xc0059ecdb0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195584 [sync.Cond.Wait, 18 minutes]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc007027328, 0x0)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc007027318)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/storage.(*RocksDB).syncLoop(0xc007027200)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/rocksdb.go:678 +0x80
created by github.com/cockroachdb/cockroach/pkg/storage.(*RocksDB).open
/go/src/github.com/cockroachdb/cockroach/pkg/storage/rocksdb.go:665 +0x316
goroutine 195668 [select]:
github.com/cockroachdb/cockroach/pkg/server.(*Node).startAssertEngineHealth.func1(0x4388300, 0xc004d35f50)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node_engine_health.go:43 +0x17a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ed820, 0xc0028cb5f0, 0xc004bd6120)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195763 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startClosedTimestampRangefeedSubscriber.func2(0x4388300, 0xc00318cb70)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:1631 +0x300
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20a20, 0xc0028cb5f0, 0xc00334d220)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195815 [select, 3 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptreconcile.(*Reconciler).run(0xc00174a000, 0x4388300, 0xc005a31020, 0xc0028cb5f0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptreconcile/reconciler.go:120 +0x2ad
github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptreconcile.(*Reconciler).Start.func1(0x4388300, 0xc005a31020)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptreconcile/reconciler.go:99 +0x47
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0028cb5f0, 0x4388300, 0xc005a31020, 0xc001d08180, 0x22, 0x0, 0x0, 0xc00799dba0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:323 +0x152
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:318 +0x131
goroutine 195681 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func1(0x4388300, 0xc005ed2ba0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:808 +0x176
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0076d6580, 0xc0028cb5f0, 0xc00799de60)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195118 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18c0)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432240)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432240)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400ddc0, 0xc0028cb5f0, 0xc00400ddb0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195498 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc001811980)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x218
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfc9
goroutine 199089 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait.(*Queue).MaybeWaitForPush(0xc00602f9a0, 0x4388300, 0xc004a5f050, 0xc0004ea000, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait/queue.go:513 +0xf95
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).maybeInterceptReq(0xc003e24d40, 0x4388300, 0xc004a5f050, 0x0, 0x15f9b15192ccceb4, 0x0, 0x0, 0x0, 0xc0025c5e00, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:189 +0xa9
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc003e24d40, 0x4388300, 0xc004a5f050, 0xc00504d500, 0x0, 0x15f9b15192ccceb4, 0x0, 0x0, 0x0, 0xc0025c5e00, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:142 +0xb8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc003e24d40, 0x4388300, 0xc004a5f050, 0x0, 0x0, 0x15f9b15192ccceb4, 0x0, 0x0, 0x0, 0xc0025c5e00, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:121 +0xfb
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc00267ea00, 0x4388300, 0xc004a5f050, 0xc0025c5f00, 0x3cf2c18, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:192 +0x270
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc00267ea00, 0x4388300, 0xc004a5f020, 0x6, 0xc0025c5f00, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:94 +0x68a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc00267ea00, 0x4388300, 0xc004a5f020, 0x15f9b15192ccceb4, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36 +0x91
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0029ee700, 0x4388300, 0xc004a5efc0, 0x15f9b15192ccceb4, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:204 +0x6c0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc004183a40, 0x4388300, 0xc004a5efc0, 0x15f9b15192ccceb4, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:175 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x4388300, 0xc004a5efc0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:925 +0x201
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0028cb5f0, 0x4388300, 0xc004a5efc0, 0x3b83687, 0x10, 0xc001965798, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:302 +0x140
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0024a4a00, 0x4388300, 0xc004a5efc0, 0xc0025c5e80, 0xc004a5efc0, 0xc0060fdce0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:913 +0x194
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0024a4a00, 0x4388300, 0xc004a5ef90, 0xc0025c5e80, 0xc0060fde00, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:951 +0x9f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:536
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc0005cb700, 0x4388300, 0xc004a5ef90, 0x1, 0x434d440, 0xc0059ed430, 0x15f9b15192ccceb4, 0x0, 0x100000001, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x127
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc0005cb700, 0x4388300, 0xc0060fde00, 0x15f9b15192ccceb4, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x160
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc003372120, 0x4388300, 0xc0060fde00, 0x15f9b15192ccceb4, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1623 +0x2a2
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc003372120, 0x4388300, 0xc0060fde00, 0x15f9b15192ccceb4, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:460 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc003372120, 0x4388300, 0xc0060fde00, 0x15f9b15192ccceb4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:542 +0x2ee
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc003372120, 0x4388300, 0xc0060fde00, 0x15f9b15192ccceb4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1394 +0x3cf
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc003372120, 0x4388300, 0xc0060fde00, 0x15f9b15192ccceb4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1079 +0x1580
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc003372120, 0x4388300, 0xc0060fde00, 0x15f9b15192ccceb4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:729 +0x7c1
github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc0004ca860, 0x4388300, 0xc0060fde00, 0x15f9b15192ccceb4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:223 +0x95
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc0004ca800, 0x4388300, 0xc0060fde00, 0x15f9b15192ccceb4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:754 +0x122
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:736
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x4388300, 0xc0060fde00, 0xc001967530, 0xc000fc2000, 0xc001967b50, 0xc001967798)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:672 +0xf7
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc0004ca800, 0x4388300, 0xc0060fde00, 0xc000fc2000, 0x19, 0xc001967768)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:695 +0x9c
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).MaybePushTransactions(0xc003945790, 0x4388300, 0xc0060fde00, 0xc001967b50, 0x15f9b151922d5ea4, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:357 +0x6e9
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).PushTransaction(0xc003945790, 0x4388300, 0xc0060fde00, 0xc004a762b8, 0x15f9b151922d5ea4, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:267 +0x170
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).pushLockTxn(0xc007c9f530, 0x4388300, 0xc0060fde00, 0xc0017c86c0, 0x15f9b151922d5ea4, 0x0, 0x0, 0x0, 0xc003cd5500, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:331 +0x132
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).WaitOn(0xc007c9f530, 0x4388300, 0xc0060fde00, 0xc0017c86c0, 0x15f9b151922d5ea4, 0x0, 0x0, 0x0, 0xc003cd5500, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:270 +0x24e
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc003e24d40, 0x4388300, 0xc0060fde00, 0xc000270a10, 0xc0017c8ea0, 0x15f9b151922d5ea4, 0x0, 0x0, 0x0, 0xc003cd5500, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:170 +0x43e
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc003e24d40, 0x4388300, 0xc0060fde00, 0xc000270a10, 0xc0017c8ea0, 0x15f9b151922d5ea4, 0x0, 0x0, 0x0, 0xc003cd5500, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:121 +0xfb
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc00267ea00, 0x4388300, 0xc0060fde00, 0xc003cd5600, 0x3cf2c00, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:192 +0x270
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc00267ea00, 0x4388300, 0xc0060fddd0, 0x6, 0xc003cd5600, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:98 +0x2f3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc00267ea00, 0x4388300, 0xc0060fddd0, 0x15f9b151922d5ea4, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36 +0x91
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0029ee700, 0x4388300, 0xc0060fdd70, 0x15f9b151922d5ea4, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:204 +0x6c0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc004183a40, 0x4388300, 0xc0060fdd70, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:175 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x4388300, 0xc0060fdd70, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:925 +0x201
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0028cb5f0, 0x4388300, 0xc0060fdd70, 0x3b83687, 0x10, 0xc000a5f840, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:302 +0x140
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0024a4a00, 0x4388300, 0xc0060fdd70, 0xc003cd5580, 0xc0060fdd70, 0x80, 0x80)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:913 +0x194
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0024a4a00, 0x4388300, 0xc0060fdd40, 0xc003cd5580, 0xc0060fdce0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:951 +0x9f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:536
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc006199300, 0x4388300, 0xc0060fdd40, 0x1, 0x434d440, 0xc0059ed430, 0x0, 0x0, 0x100000001, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x127
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc006199300, 0x4388300, 0xc0060fdce0, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x160
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc003372120, 0x4388300, 0xc0060fdce0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1623 +0x2a2
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc003372120, 0x4388300, 0xc0060fdce0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:460 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc003372120, 0x4388300, 0xc0060fdce0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:542 +0x2ee
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc003372120, 0x4388300, 0xc0060fdce0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1394 +0x3cf
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc003372120, 0x4388300, 0xc0060fdce0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1079 +0x1580
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc003372120, 0x4388300, 0xc0060fdce0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:729 +0x7c1
github.com/cockroachdb/cockroach/pkg/kv.(*txnLockGatekeeper).SendLocked(0xc004368888, 0x4388300, 0xc0060fdce0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_lock_gatekeeper.go:86 +0x10b
github.com/cockroachdb/cockroach/pkg/kv.(*txnMetricRecorder).SendLocked(0xc004368850, 0x4388300, 0xc0060fdce0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_metric_recorder.go:46 +0x8d
github.com/cockroachdb/cockroach/pkg/kv.(*txnCommitter).SendLocked(0xc004368820, 0x4388300, 0xc0060fdce0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_committer.go:126 +0x6f9
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).sendLockedWithRefreshAttempts(0xc0043687b8, 0x4388300, 0xc0060fdce0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:234 +0x9b
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).SendLocked(0xc0043687b8, 0x4388300, 0xc0060fdce0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:179 +0x158
github.com/cockroachdb/cockroach/pkg/kv.(*txnPipeliner).SendLocked(0xc004368708, 0x4388300, 0xc0060fdce0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_pipeliner.go:209 +0x159
github.com/cockroachdb/cockroach/pkg/kv.(*txnSeqNumAllocator).SendLocked(0xc0043686e8, 0x4388300, 0xc0060fdce0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_seq_num_allocator.go:105 +0x20c
github.com/cockroachdb/cockroach/pkg/kv.(*txnHeartbeater).SendLocked(0xc004368648, 0x4388300, 0xc0060fdce0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_heartbeater.go:172 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc004368480, 0x4388300, 0xc0060fdce0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:503 +0x406
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc0004ca800, 0x4388240, 0xc0061991c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:754 +0x122
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send(0xc004c48480, 0x4388240, 0xc0061991c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:870 +0x11e
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x4388240, 0xc0061991c0, 0xc000b58508, 0xc0005a0100, 0x1, 0x1)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:672 +0xf7
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Run(0xc004c48480, 0x4388240, 0xc0061991c0, 0xc0005a0100, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:538 +0xfa
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Get(0xc004c48480, 0x4388240, 0xc0061991c0, 0x391f980, 0xc0002cc180, 0x8, 0xc0018e4738, 0x11b64ba, 0xc003e2a168, 0xc0018e4738, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:348 +0xbc
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).GetProtoTs(0xc004c48480, 0x4388240, 0xc0061991c0, 0x391f980, 0xc0002cc180, 0x43c2020, 0xc004d22050, 0x3cf9d58, 0xc0018e47c0, 0xc0018e47a8, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:370 +0x66
github.com/cockroachdb/cockroach/pkg/sql/sqlbase.getTableDescFromIDRaw(0x4388240, 0xc0061991c0, 0x430bf20, 0xc004c48480, 0xc000000036, 0x7f1261, 0xc00000e060, 0x3b837effa1a1008)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/structured.go:404 +0xc7
github.com/cockroachdb/cockroach/pkg/sql/sqlbase.GetTableDescFromID(0x4388240, 0xc0061991c0, 0x430bf20, 0xc004c48480, 0x36, 0x4388300, 0x10000c005cc3d70, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/structured.go:361 +0x55
github.com/cockroachdb/cockroach/pkg/sql.LeaseStore.acquire.func1(0x4388240, 0xc0061991c0, 0xc004c48480, 0x440cc20, 0xc004368480)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:193 +0x197
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn.func1(0x4388240, 0xc0061991c0, 0xc004c48480, 0x440cc20, 0xc004368480)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:717 +0x43
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).exec(0xc004c48480, 0x4388240, 0xc0061991c0, 0xc0024f34b8, 0xc004c48480, 0xc0069be180)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:759 +0xd9
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn(0xc0004ca800, 0x4388240, 0xc0061991c0, 0xc0018e58d0, 0x65c4360, 0x20b23277)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:716 +0xd1
github.com/cockroachdb/cockroach/pkg/sql.LeaseStore.acquire(0xc001102000, 0xc0004ca800, 0xc004e59630, 0x43c7e40, 0xc007722180, 0xc000351500, 0xc000435f70, 0x45d964b800, 0x3fa999999999999a, 0xdf8475800, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:183 +0x112
github.com/cockroachdb/cockroach/pkg/sql.acquireNodeLease.func1(0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1003 +0x29b
github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall(0xc000435f70, 0xc004d7b770, 0xc003e2a170, 0x9, 0xc0060fdbc0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:128 +0x2e
created by github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).DoChan
/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:121 +0x2bc
goroutine 195716 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18a4)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc0014327e0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc0014327e0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f200d0, 0xc0028cb5f0, 0xc005f200c0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195656 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func1(0x4388300, 0xc008a97440)
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1936 +0x53
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ed4d0, 0xc0028cb5f0, 0xc0052e0a20)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195698 [select]:
github.com/cockroachdb/cockroach/pkg/sql.(*Server).PeriodicallyPollForStatementInfoRequests.func1(0x4388300, 0xc005a96150)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:790 +0x1fe
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22f90, 0xc0028cb5f0, 0xc002178f00)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195679 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func1(0x4388300, 0xc005ed2bd0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:808 +0x176
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0076d6560, 0xc0028cb5f0, 0xc00799de20)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195744 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18a8)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432db0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432db0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20670, 0xc0028cb5f0, 0xc005f20660)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195117 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18bb)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432210)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432210)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400dd90, 0xc0028cb5f0, 0xc00400dd70)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 197249 [select, 18 minutes]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0025d7980)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x10f
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x169
goroutine 195657 [IO wait, 18 minutes]:
internal/poll.runtime_pollWait(0x7f22e76e13f0, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc003da7698, 0x72, 0x0, 0x0, 0x3b6d4d2)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc003da7680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1f8
net.(*netFD).accept(0xc003da7680, 0xc0000b3500, 0x7f22e779c008, 0x0)
/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0052e0a00, 0x66cb18, 0x10, 0x68e44a)
/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0052e0a00, 0xc00617c0f0, 0x30, 0xc0039b9ef0, 0xc001dd7e70)
/usr/local/go/src/net/tcpsock.go:261 +0x47
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc003e24540, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0xaf
github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func2(0x4388300, 0xc0039b9ef0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1949 +0x2f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ed4f0, 0xc0028cb5f0, 0xc0052e0a80)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195743 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18af)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432d80)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432d80)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20640, 0xc0028cb5f0, 0xc005f20630)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195672 [IO wait, 18 minutes]:
internal/poll.runtime_pollWait(0x7f22e627c6c8, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc003da7718, 0x72, 0x0, 0x0, 0x3b6d4d2)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc003da7700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1f8
net.(*netFD).accept(0xc003da7700, 0x37e11d600, 0x0, 0x0)
/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0052e0ae0, 0xc00008dd50, 0xc00008dd58, 0x28)
/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0052e0ae0, 0x3cf70a8, 0xc003e24600, 0x43d8e80, 0xc005cd3928)
/usr/local/go/src/net/tcpsock.go:261 +0x47
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc003e24600, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0xaf
github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1.1()
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1916 +0x2f
sync.(*Once).doSlow(0xc002056110, 0xc001cf46d0)
/usr/local/go/src/sync/once.go:66 +0xe3
sync.(*Once).Do(...)
/usr/local/go/src/sync/once.go:57
github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1(0x4388300, 0xc004d864b0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1915 +0x73
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ed8e0, 0xc0028cb5f0, 0xc0052e1920)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195700 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18ce)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432360)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432360)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400de90, 0xc0028cb5f0, 0xc00400de80)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195499 [select]:
net.(*pipe).read(0xc003da7800, 0xc000a74000, 0x775, 0x775, 0x203001, 0x0, 0x770)
/usr/local/go/src/net/pipe.go:164 +0x19b
net.(*pipe).Read(0xc003da7800, 0xc000a74000, 0x775, 0x775, 0xc000a74005, 0x12, 0x10)
/usr/local/go/src/net/pipe.go:147 +0x4d
crypto/tls.(*atLeastReader).Read(0xc005edc820, 0xc000a74000, 0x775, 0x775, 0x27, 0x430bdc0, 0xc0000929a8)
/usr/local/go/src/crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc00274a5d8, 0x430bb00, 0xc005edc820, 0x66a285, 0x36fce40, 0x39a06a0)
/usr/local/go/src/bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc00274a380, 0x7f22e7670aa8, 0xc003da7800, 0x5, 0xc003da7800, 0x11)
/usr/local/go/src/crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc00274a380, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc00274a380, 0xc003c0e000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1255 +0x161
bufio.(*Reader).Read(0xc003259320, 0xc0002f5998, 0x9, 0x9, 0xc0000b2700, 0x7f22e779cd98, 0x0)
/usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x430b940, 0xc003259320, 0xc0002f5998, 0x9, 0x9, 0x9, 0xc4d275, 0xc004c83d2c, 0xc000092e28)
/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
/usr/local/go/src/io/io.go:329
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0002f5998, 0x9, 0x9, 0x430b940, 0xc003259320, 0x0, 0x0, 0xc004c83d20, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x87
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0002f5960, 0xc004c83d20, 0xc004c83d20, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc001811980, 0xc004d35ce0, 0x3cf8f48)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x86
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc007026f00, 0x43dea00, 0xc001811980)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xf3
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc007026f00, 0x43dea00, 0xc001811980)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x5e0
goroutine 195635 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x4388300, 0xc0039b9bf0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:99 +0x13f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ecbe0, 0xc0028cb5f0, 0xc0059ecbd0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195756 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Start.func2(0x4388300, 0xc001433020)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:1412 +0xe5
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f208f0, 0xc0028cb5f0, 0xc00334d1e0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195748 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18bf)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432ea0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432ea0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f206f0, 0xc0028cb5f0, 0xc005f206e0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195694 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*Server).Start.func1(0x4388300, 0xc004e7d440)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:325 +0xf9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22f50, 0xc0028cb5f0, 0xc002178ee0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195724 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18aa)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432990)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432990)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f201e0, 0xc0028cb5f0, 0xc005f201d0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195709 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x1895)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432630)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432630)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400dfd0, 0xc0028cb5f0, 0xc00400dfc0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195897 [select, 9 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).addToPurgatoryLocked.func2(0x4388300, 0xc00658ede0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:1148 +0x33e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005295010, 0xc0028cb5f0, 0xc002d15860)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195713 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x1896)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc0014326f0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc0014326f0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20060, 0xc0028cb5f0, 0xc005f20050)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195740 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18b8)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432cf0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432cf0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f205e0, 0xc0028cb5f0, 0xc005f205d0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 199088 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait.(*Queue).MaybeWaitForPush(0xc00602f9a0, 0x4388300, 0xc004b80180, 0xc002448780, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait/queue.go:513 +0xf95
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).maybeInterceptReq(0xc003e24d40, 0x4388300, 0xc004b80180, 0x0, 0x15f9b15192c6e840, 0x0, 0x0, 0x0, 0xc00386c280, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:189 +0xa9
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc003e24d40, 0x4388300, 0xc004b80180, 0xc004ca9110, 0x0, 0x15f9b15192c6e840, 0x0, 0x0, 0x0, 0xc00386c280, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:142 +0xb8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc003e24d40, 0x4388300, 0xc004b80180, 0x0, 0x0, 0x15f9b15192c6e840, 0x0, 0x0, 0x0, 0xc00386c280, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:121 +0xfb
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc00267ea00, 0x4388300, 0xc004b80180, 0xc00386c380, 0x3cf2c18, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:192 +0x270
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc00267ea00, 0x4388300, 0xc004b80150, 0x6, 0xc00386c380, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:94 +0x68a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc00267ea00, 0x4388300, 0xc004b80150, 0x15f9b15192c6e840, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36 +0x91
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0029ee700, 0x4388300, 0xc004b800f0, 0x15f9b15192c6e840, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:204 +0x6c0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc004183a40, 0x4388300, 0xc004b800f0, 0x15f9b15192c6e840, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:175 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x4388300, 0xc004b800f0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:925 +0x201
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0028cb5f0, 0x4388300, 0xc004b800f0, 0x3b83687, 0x10, 0xc0017e1798, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:302 +0x140
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0024a4a00, 0x4388300, 0xc004b800f0, 0xc00386c300, 0xc004b800f0, 0xc0060fda10, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:913 +0x194
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0024a4a00, 0x4388300, 0xc004b800c0, 0xc00386c300, 0xc0060fdb30, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:951 +0x9f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:536
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc0027b3a80, 0x4388300, 0xc004b800c0, 0x1, 0x434d440, 0xc0059ed430, 0x15f9b15192c6e840, 0x0, 0x100000001, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x127
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc0027b3a80, 0x4388300, 0xc0060fdb30, 0x15f9b15192c6e840, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x160
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc003372120, 0x4388300, 0xc0060fdb30, 0x15f9b15192c6e840, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1623 +0x2a2
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc003372120, 0x4388300, 0xc0060fdb30, 0x15f9b15192c6e840, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:460 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc003372120, 0x4388300, 0xc0060fdb30, 0x15f9b15192c6e840, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:542 +0x2ee
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc003372120, 0x4388300, 0xc0060fdb30, 0x15f9b15192c6e840, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1394 +0x3cf
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc003372120, 0x4388300, 0xc0060fdb30, 0x15f9b15192c6e840, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1079 +0x1580
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc003372120, 0x4388300, 0xc0060fdb30, 0x15f9b15192c6e840, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:729 +0x7c1
github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc0004ca860, 0x4388300, 0xc0060fdb30, 0x15f9b15192c6e840, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:223 +0x95
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc0004ca800, 0x4388300, 0xc0060fdb30, 0x15f9b15192c6e840, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:754 +0x122
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:736
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x4388300, 0xc0060fdb30, 0xc0017e3530, 0xc000987600, 0xc0017e3b50, 0xc0017e3798)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:672 +0xf7
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc0004ca800, 0x4388300, 0xc0060fdb30, 0xc000987600, 0x19, 0xc0017e3768)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:695 +0x9c
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).MaybePushTransactions(0xc003945790, 0x4388300, 0xc0060fdb30, 0xc0017e3b50, 0x15f9b15192258504, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:357 +0x6e9
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).PushTransaction(0xc003945790, 0x4388300, 0xc0060fdb30, 0xc004a76168, 0x15f9b15192258504, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:267 +0x170
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).pushLockTxn(0xc007c9f530, 0x4388300, 0xc0060fdb30, 0xc004b45b00, 0x15f9b15192258504, 0x0, 0x0, 0x0, 0xc003cd5380, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:331 +0x132
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).WaitOn(0xc007c9f530, 0x4388300, 0xc0060fdb30, 0xc004b45b00, 0x15f9b15192258504, 0x0, 0x0, 0x0, 0xc003cd5380, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:270 +0x24e
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc003e24d40, 0x4388300, 0xc0060fdb30, 0xc004bb0850, 0xc004b45e60, 0x15f9b15192258504, 0x0, 0x0, 0x0, 0xc003cd5380, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:170 +0x43e
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc003e24d40, 0x4388300, 0xc0060fdb30, 0xc004bb0850, 0xc004b45e60, 0x15f9b15192258504, 0x0, 0x0, 0x0, 0xc003cd5380, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:121 +0xfb
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc00267ea00, 0x4388300, 0xc0060fdb30, 0xc003cd5480, 0x3cf2c00, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:192 +0x270
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc00267ea00, 0x4388300, 0xc0060fdb00, 0x6, 0xc003cd5480, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:98 +0x2f3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc00267ea00, 0x4388300, 0xc0060fdb00, 0x15f9b15192258504, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36 +0x91
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0029ee700, 0x4388300, 0xc0060fdaa0, 0x15f9b15192258504, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:204 +0x6c0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc004183a40, 0x4388300, 0xc0060fdaa0, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:175 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x4388300, 0xc0060fdaa0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:925 +0x201
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0028cb5f0, 0x4388300, 0xc0060fdaa0, 0x3b83687, 0x10, 0xc000a5f840, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:302 +0x140
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0024a4a00, 0x4388300, 0xc0060fdaa0, 0xc003cd5400, 0xc0060fdaa0, 0x80, 0x80)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:913 +0x194
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0024a4a00, 0x4388300, 0xc0060fda70, 0xc003cd5400, 0xc0060fda10, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:951 +0x9f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:536
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc0061990c0, 0x4388300, 0xc0060fda70, 0x1, 0x434d440, 0xc0059ed430, 0x0, 0x0, 0x100000001, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x127
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc0061990c0, 0x4388300, 0xc0060fda10, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x160
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc003372120, 0x4388300, 0xc0060fda10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1623 +0x2a2
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc003372120, 0x4388300, 0xc0060fda10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:460 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc003372120, 0x4388300, 0xc0060fda10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:542 +0x2ee
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc003372120, 0x4388300, 0xc0060fda10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1394 +0x3cf
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc003372120, 0x4388300, 0xc0060fda10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1079 +0x1580
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc003372120, 0x4388300, 0xc0060fda10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:729 +0x7c1
github.com/cockroachdb/cockroach/pkg/kv.(*txnLockGatekeeper).SendLocked(0xc004368408, 0x4388300, 0xc0060fda10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_lock_gatekeeper.go:86 +0x10b
github.com/cockroachdb/cockroach/pkg/kv.(*txnMetricRecorder).SendLocked(0xc0043683d0, 0x4388300, 0xc0060fda10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_metric_recorder.go:46 +0x8d
github.com/cockroachdb/cockroach/pkg/kv.(*txnCommitter).SendLocked(0xc0043683a0, 0x4388300, 0xc0060fda10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_committer.go:126 +0x6f9
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).sendLockedWithRefreshAttempts(0xc004368338, 0x4388300, 0xc0060fda10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:234 +0x9b
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).SendLocked(0xc004368338, 0x4388300, 0xc0060fda10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:179 +0x158
github.com/cockroachdb/cockroach/pkg/kv.(*txnPipeliner).SendLocked(0xc004368288, 0x4388300, 0xc0060fda10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_pipeliner.go:209 +0x159
github.com/cockroachdb/cockroach/pkg/kv.(*txnSeqNumAllocator).SendLocked(0xc004368268, 0x4388300, 0xc0060fda10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_seq_num_allocator.go:105 +0x20c
github.com/cockroachdb/cockroach/pkg/kv.(*txnHeartbeater).SendLocked(0xc0043681c8, 0x4388300, 0xc0060fda10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_heartbeater.go:172 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc004368000, 0x4388300, 0xc0060fda10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:503 +0x406
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc0004ca800, 0x4388240, 0xc006198f80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:754 +0x122
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send(0xc004c483f0, 0x4388240, 0xc006198f80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:870 +0x11e
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x4388240, 0xc006198f80, 0xc000b58508, 0xc00059fb80, 0x1, 0x1)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:672 +0xf7
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Run(0xc004c483f0, 0x4388240, 0xc006198f80, 0xc00059fb80, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:538 +0xfa
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Get(0xc004c483f0, 0x4388240, 0xc006198f80, 0x391f980, 0xc0002cc060, 0x8, 0xc0018e4738, 0x11b64ba, 0xc003e2a0e0, 0xc0018e4738, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:348 +0xbc
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).GetProtoTs(0xc004c483f0, 0x4388240, 0xc006198f80, 0x391f980, 0xc0002cc060, 0x43c2020, 0xc004d22010, 0x3cf9d58, 0xc0018e47c0, 0xc0018e47a8, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:370 +0x66
github.com/cockroachdb/cockroach/pkg/sql/sqlbase.getTableDescFromIDRaw(0x4388240, 0xc006198f80, 0x430bf20, 0xc004c483f0, 0xc000000035, 0x7f1261, 0xc00000e060, 0x405bff1af3dcb7b5)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/structured.go:404 +0xc7
github.com/cockroachdb/cockroach/pkg/sql/sqlbase.GetTableDescFromID(0x4388240, 0xc006198f80, 0x430bf20, 0xc004c483f0, 0x35, 0x4388300, 0x10000c005cc3d70, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/structured.go:361 +0x55
github.com/cockroachdb/cockroach/pkg/sql.LeaseStore.acquire.func1(0x4388240, 0xc006198f80, 0xc004c483f0, 0x440cc20, 0xc004368000)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:193 +0x197
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn.func1(0x4388240, 0xc006198f80, 0xc004c483f0, 0x440cc20, 0xc004368000)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:717 +0x43
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).exec(0xc004c483f0, 0x4388240, 0xc006198f80, 0xc0023a34b8, 0xc004c483f0, 0xc001f30508)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:759 +0xd9
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn(0xc0004ca800, 0x4388240, 0xc006198f80, 0xc0018e58d0, 0xc001f30648, 0x280a014)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:716 +0xd1
github.com/cockroachdb/cockroach/pkg/sql.LeaseStore.acquire(0xc001102000, 0xc0004ca800, 0xc004e59630, 0x43c7e40, 0xc007722180, 0xc000351500, 0xc000435f70, 0x45d964b800, 0x3fa999999999999a, 0xdf8475800, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:183 +0x112
github.com/cockroachdb/cockroach/pkg/sql.acquireNodeLease.func1(0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1003 +0x29b
github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).doCall(0xc000435f70, 0xc004d7b680, 0xc003e2a0d0, 0x9, 0xc0060fd8c0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:128 +0x2e
created by github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight.(*Group).DoChan
/go/src/github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight/singleflight.go:121 +0x2bc
goroutine 195721 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18a3)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc0014328d0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc0014328d0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20180, 0xc0028cb5f0, 0xc005f20170)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195704 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x1894)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432450)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432450)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400df20, 0xc0028cb5f0, 0xc00400df10)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195683 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/provider.(*Provider).runCloser(0xc001caddc0, 0x4388300, 0xc005ed27e0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/provider/provider.go:136 +0x375
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22bb0, 0xc0028cb5f0, 0xc004d22ba0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195728 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18a9)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432a50)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432a50)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20380, 0xc0028cb5f0, 0xc005f20370)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195669 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x4388300, 0xc004d35f80)
/go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:383 +0x5a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ed880, 0xc0028cb5f0, 0xc0052e1900)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195715 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x189e)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432750)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432750)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f200a0, 0xc0028cb5f0, 0xc005f20090)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195697 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*Server).PeriodicallyClearSQLStats.func1(0x4388300, 0xc006243800)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:765 +0x248
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22f80, 0xc0028cb5f0, 0xc005ed2ed0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195677 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func1(0x4388300, 0xc005ed2b70)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/queue.go:808 +0x176
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0076d6540, 0xc0028cb5f0, 0xc00799dde0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195659 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
crypto/tls.(*listener).Accept(0xc0052e0aa0, 0x3954c60, 0xc004d34600, 0x3693340, 0x62d6610)
/usr/local/go/src/crypto/tls/tls.go:59 +0x37
net/http.(*Server).Serve(0xc0048a5420, 0x435e240, 0xc0052e0aa0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2896 +0x280
github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func4(0x4388300, 0xc004d342a0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1976 +0x3f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ed520, 0xc0028cb5f0, 0xc0052e0ac0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 197250 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc003b82000, 0xc004afd130, 0xc002aade00, 0xf, 0xc001503b60, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:1053 +0x30b
github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x4388300, 0xc004c1aed0)
/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:988 +0x79
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001734c90, 0xc0028cb5f0, 0xc004afd180)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 196685 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/idalloc.(*Allocator).start.func1(0x4388300, 0xc001849a10)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/idalloc/id_alloc.go:123 +0x449
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002c3b30, 0xc0028cb5f0, 0xc0002c3b20)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195674 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc0010b5680, 0x4388280, 0xc000052130)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:411 +0x390
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0028cb5f0, 0x4388280, 0xc000052130, 0xc007868ed0, 0x22, 0x0, 0x0, 0xc0059edfa0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:323 +0x152
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:318 +0x131
goroutine 195780 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).waitAndProcess(0xc003944f70, 0x4388300, 0xc00318cbd0, 0xc0028cb5f0, 0x362c6a1c, 0xed5f42545, 0x0, 0xc0043e3100, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scanner.go:217 +0x1dc
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).scanLoop.func1.1(0xc0043e3100, 0xc004bd6d01)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scanner.go:279 +0x81
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*storeReplicaVisitor).Visit(0xc004bd6d80, 0xc004951880)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:345 +0x154
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).scanLoop.func1(0x4388300, 0xc00318cbd0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scanner.go:277 +0x1da
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0076d65d0, 0xc0028cb5f0, 0xc00799dee0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195666 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func4(0x4388300, 0xc008a97470)
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1353 +0x53
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ed6f0, 0xc0028cb5f0, 0xc0052e0de0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195644 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x4388300, 0xc0039b9ce0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:99 +0x13f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ecd30, 0xc0028cb5f0, 0xc0059ecd20)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195580 [IO wait, 18 minutes]:
internal/poll.runtime_pollWait(0x7f22e76e19a0, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc001732298, 0x72, 0x800, 0x888, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc001732280, 0xc000877200, 0x888, 0x888, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc001732280, 0xc000877200, 0x888, 0x888, 0x203001, 0x0, 0x7ff)
/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000df0908, 0xc000877200, 0x888, 0x888, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:184 +0x68
crypto/tls.(*atLeastReader).Read(0xc007dfbb00, 0xc000877200, 0x888, 0x888, 0x2e, 0x37b, 0xc0056b2820)
/usr/local/go/src/crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc003c56258, 0x430bb00, 0xc007dfbb00, 0x66a285, 0x36fce40, 0x3ac61c0)
/usr/local/go/src/bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc003c56000, 0x4311220, 0xc000df0908, 0x5, 0xc000df0908, 0x3cf92f8)
/usr/local/go/src/crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc003c56000, 0x0, 0x0, 0x380)
/usr/local/go/src/crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc003c56000, 0xc002354000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1255 +0x161
bufio.(*Reader).Read(0xc004ab3680, 0xc0004798e0, 0x5, 0x200, 0xc0056b2b60, 0xc0056b2bf0, 0x8f7e28)
/usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x430b940, 0xc004ab3680, 0xc0004798e0, 0x5, 0x200, 0x5, 0x8f7d3a, 0x0, 0x0)
/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
/usr/local/go/src/io/io.go:329
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).recvMessage(0xc0004798c0, 0xc0010a2cd0, 0x200, 0x1d, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:939 +0x103
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).recv1Buf(0xc0004798c0, 0xc0010a2cd0, 0x17)
/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:990 +0x39
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).recv1(...)
/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:1015
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).simpleExec(0xc0004798c0, 0x3b958b3, 0x17, 0x37, 0x8, 0x0, 0x0, 0x66913b, 0xc0006f0000)
/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:613 +0x1bc
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).Exec(0xc0004798c0, 0x3b958b3, 0x17, 0x660e0d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:867 +0x2a1
github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).ExecContext(0xc0004798c0, 0x4388280, 0xc000052130, 0x3b958b3, 0x17, 0x660e0d0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn_go18.go:42 +0x1aa
database/sql.ctxDriverExec(0x4388280, 0xc000052130, 0x7f22e7682c78, 0xc0004798c0, 0x0, 0x0, 0x3b958b3, 0x17, 0x660e0d0, 0x0, ...)
/usr/local/go/src/database/sql/ctxutil.go:31 +0x227
database/sql.(*DB).execDC.func2()
/usr/local/go/src/database/sql/sql.go:1519 +0x1db
database/sql.withLock(0x4347b00, 0xc001732500, 0xc0056b3230)
/usr/local/go/src/database/sql/sql.go:3184 +0x6d
database/sql.(*DB).execDC(0xc00295acc0, 0x4388280, 0xc000052130, 0xc001732500, 0xc0056b3330, 0x3b958b3, 0x17, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/database/sql/sql.go:1514 +0x4b2
database/sql.(*DB).exec(0xc00295acc0, 0x4388280, 0xc000052130, 0x3b958b3, 0x17, 0x0, 0x0, 0x0, 0x1, 0xc0056b33e0, ...)
/usr/local/go/src/database/sql/sql.go:1499 +0x139
database/sql.(*DB).ExecContext(0xc00295acc0, 0x4388280, 0xc000052130, 0x3b958b3, 0x17, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/database/sql/sql.go:1477 +0xde
database/sql.(*DB).Exec(...)
/usr/local/go/src/database/sql/sql.go:1491
github.com/cockroachdb/cockroach/pkg/sql_test.TestDropDatabaseDeleteData(0xc004ea2500)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/drop_test.go:369 +0xcc7
testing.tRunner(0xc004ea2500, 0x3cf60d0)
/usr/local/go/src/testing/testing.go:909 +0xc9
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:960 +0x350
goroutine 195663 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func1(0x4388300, 0xc004d34720)
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1323 +0x7c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ed600, 0xc0028cb5f0, 0xc004739f20)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195122 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18c3)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432300)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432300)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400de50, 0xc0028cb5f0, 0xc00400de40)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 197487 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal(0xc007722180, 0x4388300, 0xc001160060, 0x3b7941b, 0xc, 0x0, 0x3b68183, 0x4, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:475 +0xa3e
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).ExecEx(0xc007722180, 0x4388300, 0xc001d46c60, 0x3b7941b, 0xc, 0x0, 0x3b68183, 0x4, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:295 +0xef
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).Exec(0xc007722180, 0x4388300, 0xc001d46c60, 0x3b7941b, 0xc, 0x0, 0xc000cee000, 0x59, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:279 +0x195
github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).refreshStats(0xc00602e960, 0x4388300, 0xc001d46c60, 0x35, 0x6fc23ac00, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:467 +0x225
github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).maybeRefreshStats(0xc00602e960, 0x4388300, 0xc001d46c60, 0xc0028cb5f0, 0x35, 0x80000002, 0x6fc23ac00)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:433 +0x21d
github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1.1(0x4388300, 0xc001d46c60)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:280 +0x28d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0028cb5f0, 0x4388300, 0xc001d46c60, 0xc0021083f0, 0x2a, 0x0, 0x0, 0xc003405a70)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:323 +0x152
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:318 +0x131
goroutine 196861 [select, 18 minutes]:
database/sql.(*DB).connectionOpener(0xc00295acc0, 0x4388240, 0xc0049506c0)
/usr/local/go/src/database/sql/sql.go:1052 +0xe8
created by database/sql.OpenDB
/usr/local/go/src/database/sql/sql.go:722 +0x15d
goroutine 195673 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc0010b5560, 0x4388280, 0xc000052130)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:411 +0x390
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0028cb5f0, 0x4388280, 0xc000052130, 0xc007868e70, 0x22, 0x0, 0x0, 0xc0059edf60)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:323 +0x152
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:318 +0x131
goroutine 193147 [select]:
net.(*pipe).read(0xc003da7880, 0xc003aac000, 0x888, 0x888, 0x203001, 0x7f22e073e800, 0x40)
/usr/local/go/src/net/pipe.go:164 +0x19b
net.(*pipe).Read(0xc003da7880, 0xc003aac000, 0x888, 0x888, 0xc003aac000, 0x12, 0x10)
/usr/local/go/src/net/pipe.go:147 +0x4d
crypto/tls.(*atLeastReader).Read(0xc005edc800, 0xc003aac000, 0x888, 0x888, 0x27, 0x430bdc0, 0xc001f22a50)
/usr/local/go/src/crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc00274a958, 0x430bb00, 0xc005edc800, 0x66a285, 0x36fce40, 0x39a06a0)
/usr/local/go/src/bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc00274a700, 0x7f22e7670aa8, 0xc003da7880, 0x5, 0xc003da7880, 0x11)
/usr/local/go/src/crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc00274a700, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc00274a700, 0xc003bae000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1255 +0x161
bufio.(*Reader).Read(0xc004930360, 0xc000cd6038, 0x9, 0x9, 0x66c2a6, 0xc0003c7980, 0xc001f22e08)
/usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x430b940, 0xc004930360, 0xc000cd6038, 0x9, 0x9, 0x9, 0x0, 0xc00130a8f8, 0x6b9440)
/usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
/usr/local/go/src/io/io.go:329
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc000cd6038, 0x9, 0x9, 0x430b940, 0xc004930360, 0x0, 0xc000000000, 0xc001f22e6f, 0x1)
/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x87
github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000cd6000, 0xc004c83d00, 0xc004c83d00, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00325c1c0)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x17e
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd48
goroutine 195116 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18c4)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc0014321e0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc0014321e0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400dd30, 0xc0028cb5f0, 0xc00400dd10)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195718 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x189f)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432840)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432840)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20120, 0xc0028cb5f0, 0xc005f20110)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195646 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x4388300, 0xc0039b9da0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:99 +0x13f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ecd80, 0xc0028cb5f0, 0xc0059ecd70)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195727 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18ac)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432a20)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432a20)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20360, 0xc0028cb5f0, 0xc005f20340)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 196719 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait.(*Queue).MaybeWaitForPush(0xc00602f9a0, 0x4388300, 0xc005ca1e30, 0xc00447e960, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait/queue.go:513 +0xf95
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).maybeInterceptReq(0xc003e24d40, 0x4388300, 0xc005ca1e30, 0x0, 0x15f9b12ea10357c4, 0x0, 0x0, 0x0, 0xc003a2e680, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:189 +0xa9
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc003e24d40, 0x4388300, 0xc005ca1e30, 0xc004da6a80, 0x0, 0x15f9b12ea10357c4, 0x0, 0x0, 0x0, 0xc003a2e680, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:142 +0xb8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc003e24d40, 0x4388300, 0xc005ca1e30, 0xc004da6a80, 0x0, 0x15f9b12ea10357c4, 0x0, 0x0, 0x0, 0xc003a2e680, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:121 +0xfb
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc00267ea00, 0x4388300, 0xc005ca1e30, 0xc003a2e780, 0x3cf2c18, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:192 +0x270
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc00267ea00, 0x4388300, 0xc005ca1e00, 0x6, 0xc003a2e780, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:94 +0x68a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc00267ea00, 0x4388300, 0xc005ca1e00, 0x15f9b12ea10357c4, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36 +0x91
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0029ee700, 0x4388300, 0xc005ca1ce0, 0x15f9b12ea10357c4, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:204 +0x6c0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc004183a40, 0x4388300, 0xc005ca1ce0, 0x15f9b12ea10357c4, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:175 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x4388300, 0xc005ca1ce0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:925 +0x201
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0028cb5f0, 0x4388300, 0xc005ca1ce0, 0x3b83687, 0x10, 0xc00145b860, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:302 +0x140
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0024a4a00, 0x4388300, 0xc005ca1ce0, 0xc003a2e700, 0xc005ca1ce0, 0xc0051b1b60, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:913 +0x194
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0024a4a00, 0x4388300, 0xc005ca1cb0, 0xc003a2e700, 0xc0051b1e60, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:951 +0x9f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:536
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc001629d00, 0x4388300, 0xc005ca1cb0, 0x1, 0x434d440, 0xc0059ed430, 0x15f9b12ea10357c4, 0x0, 0x100000001, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x127
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc001629d00, 0x4388300, 0xc0051b1e60, 0x15f9b12ea10357c4, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x160
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc003372120, 0x4388300, 0xc0051b1e60, 0x15f9b12ea10357c4, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1623 +0x2a2
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc003372120, 0x4388300, 0xc0051b1e60, 0x15f9b12ea10357c4, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:460 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc003372120, 0x4388300, 0xc0051b1e60, 0x15f9b12ea10357c4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:542 +0x2ee
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc003372120, 0x4388300, 0xc0051b1e60, 0x15f9b12ea10357c4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1394 +0x3cf
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc003372120, 0x4388300, 0xc0051b1e60, 0x15f9b12ea10357c4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1079 +0x1580
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc003372120, 0x4388300, 0xc0051b1e60, 0x15f9b12ea10357c4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:729 +0x7c1
github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc0004ca860, 0x4388300, 0xc0051b1e60, 0x15f9b12ea10357c4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:223 +0x95
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc0004ca800, 0x4388300, 0xc0051b1e60, 0x15f9b12ea10357c4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:754 +0x122
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:736
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x4388300, 0xc0051b1e60, 0xc00145d5f8, 0xc000f5bb80, 0xc00145dc18, 0xc00145d860)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:672 +0xf7
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc0004ca800, 0x4388300, 0xc0051b1e60, 0xc000f5bb80, 0x19, 0xc00145d830)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:695 +0x9c
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).MaybePushTransactions(0xc003945790, 0x4388300, 0xc0051b1e60, 0xc00145dc18, 0x15f9b12ea067963f, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:357 +0x6e9
github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).PushTransaction(0xc003945790, 0x4388300, 0xc0051b1e60, 0xc004a77278, 0x15f9b12ea067963f, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:267 +0x170
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).pushLockTxn(0xc0019b43c0, 0x4388300, 0xc0051b1e60, 0xc0017c99e0, 0x15f9b12ea067963f, 0x0, 0x0, 0x0, 0xc000bb4380, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:331 +0x132
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).WaitOn(0xc0019b43c0, 0x4388300, 0xc0051b1e60, 0xc0017c99e0, 0x15f9b12ea067963f, 0x0, 0x0, 0x0, 0xc000bb4380, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:270 +0x24e
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc003bf9800, 0x4388300, 0xc0051b1e60, 0xc00081a230, 0xc0017c9b00, 0x15f9b12ea067963f, 0x0, 0x0, 0x0, 0xc000bb4380, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:170 +0x43e
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc003bf9800, 0x4388300, 0xc0051b1e60, 0xc00081a230, 0xc0017c9b00, 0x15f9b12ea067963f, 0x0, 0x0, 0x0, 0xc000bb4380, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:121 +0xfb
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc003c28300, 0x4388300, 0xc0051b1e60, 0xc000bb4500, 0x3cf2c00, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:192 +0x270
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID(0xc003c28300, 0x4388300, 0xc0051b1e30, 0x1a, 0xc000bb4500, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:98 +0x2f3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send(0xc003c28300, 0x4388300, 0xc0051b1e30, 0x15f9b12ea067963f, 0x0, 0x100000001, 0x1, 0x0, 0x1a, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36 +0x91
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send(0xc0029ee700, 0x4388300, 0xc0051b1d70, 0x15f9b12ea067963f, 0x0, 0x100000001, 0x1, 0x0, 0x1a, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:204 +0x6c0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send(0xc004183a40, 0x4388300, 0xc0051b1d70, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x1a, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:175 +0xed
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x4388300, 0xc0051b1d70, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:925 +0x201
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0028cb5f0, 0x4388300, 0xc0051b1d70, 0x3b83687, 0x10, 0xc00145f908, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:302 +0x140
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0024a4a00, 0x4388300, 0xc0051b1d70, 0xc000bb4400, 0xc0051b1d70, 0x80, 0x80)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:913 +0x194
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0024a4a00, 0x4388300, 0xc0051b1bf0, 0xc000bb4400, 0xc0051b1b60, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:951 +0x9f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:536
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc00a96d300, 0x4388300, 0xc0051b1bf0, 0x1, 0x434d440, 0xc0059ed430, 0x0, 0x0, 0x100000001, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x127
github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc00a96d300, 0x4388300, 0xc0051b1b60, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x1a, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x160
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc003372120, 0x4388300, 0xc0051b1b60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1a, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1623 +0x2a2
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc003372120, 0x4388300, 0xc0051b1b60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1a, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:460 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc003372120, 0x4388300, 0xc0051b1b60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:542 +0x2ee
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc003372120, 0x4388300, 0xc0051b1b60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1394 +0x3cf
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc003372120, 0x4388300, 0xc0051b1b60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1079 +0x1580
github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc003372120, 0x4388300, 0xc0051b1b60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:729 +0x7c1
github.com/cockroachdb/cockroach/pkg/kv.(*txnLockGatekeeper).SendLocked(0xc0032e0d08, 0x4388300, 0xc0051b1b60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_lock_gatekeeper.go:86 +0x10b
github.com/cockroachdb/cockroach/pkg/kv.(*txnMetricRecorder).SendLocked(0xc0032e0cd0, 0x4388300, 0xc0051b1b60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_metric_recorder.go:46 +0x8d
github.com/cockroachdb/cockroach/pkg/kv.(*txnCommitter).SendLocked(0xc0032e0ca0, 0x4388300, 0xc0051b1b60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_committer.go:126 +0x6f9
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).sendLockedWithRefreshAttempts(0xc0032e0c38, 0x4388300, 0xc0051b1b60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:234 +0x9b
github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).SendLocked(0xc0032e0c38, 0x4388300, 0xc0051b1b60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:179 +0x158
github.com/cockroachdb/cockroach/pkg/kv.(*txnPipeliner).SendLocked(0xc0032e0b88, 0x4388300, 0xc0051b1b60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_pipeliner.go:209 +0x159
github.com/cockroachdb/cockroach/pkg/kv.(*txnSeqNumAllocator).SendLocked(0xc0032e0b68, 0x4388300, 0xc0051b1b60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_seq_num_allocator.go:105 +0x20c
github.com/cockroachdb/cockroach/pkg/kv.(*txnHeartbeater).SendLocked(0xc0032e0ac8, 0x4388300, 0xc0051b1b60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_heartbeater.go:172 +0x1a9
github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc0032e0900, 0x4388300, 0xc0051b1b60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:503 +0x406
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc0004ca800, 0x4388300, 0xc0051b0300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:754 +0x122
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send(0xc0078b6bd0, 0x4388300, 0xc0051b0300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:870 +0x11e
github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x4388300, 0xc0051b0300, 0xc0014625d0, 0xc000fa7180, 0x1, 0x1)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:672 +0xf7
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Run(0xc0078b6bd0, 0x4388300, 0xc0051b0300, 0xc000fa7180, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:538 +0xfa
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Get(0xc0078b6bd0, 0x4388300, 0xc0051b0300, 0x391f980, 0xc000232e00, 0xc001462810, 0x1, 0x1, 0xc0014627f0, 0xc000435bb0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:348 +0xbc
github.com/cockroachdb/cockroach/pkg/sql/sqlbase.LookupObjectID(0x4388300, 0xc0051b0300, 0xc0078b6bd0, 0x0, 0x633c274, 0x1, 0xc0032e0c38, 0x43dcc60, 0xc0032e0ca0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/namespace.go:165 +0x1cb
github.com/cockroachdb/cockroach/pkg/sql/sqlbase.LookupDatabaseID(...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/namespace.go:215
github.com/cockroachdb/cockroach/pkg/sql.getDatabaseID(0x4388300, 0xc0051b0300, 0xc0078b6bd0, 0x633c274, 0x1, 0x0, 0x0, 0xc001462968, 0x76a18c)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/database.go:82 +0x7e
github.com/cockroachdb/cockroach/pkg/sql.(*databaseCache).getDatabaseID.func1(0x4388300, 0xc0051b0300, 0xc0078b6bd0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/database.go:226 +0x62
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn.func1(0x4388300, 0xc0051b0300, 0xc0078b6bd0, 0x440cc20, 0xc0032e0900)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:717 +0x43
github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).exec(0xc0078b6bd0, 0x4388300, 0xc0051b0300, 0xc001beacf0, 0xc0078b6bd0, 0x28)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:759 +0xd9
github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn(0xc0004ca800, 0x4388300, 0xc0051b0300, 0xc0051b1ad0, 0x1, 0xc0051b1ad0)
/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:716 +0xd1
github.com/cockroachdb/cockroach/pkg/sql.(*databaseCache).getDatabaseID(0xc005211110, 0x4388300, 0xc0051b0300, 0xc001462e28, 0x633c274, 0x1, 0x0, 0xc004468d80, 0x0, 0xc001462e90)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/database.go:224 +0x19d
github.com/cockroachdb/cockroach/pkg/sql.(*TableCollection).getMutableTableDescriptor(0xc0007ae228, 0x4388300, 0xc0051b0300, 0xc003bedc20, 0xc0007ae8c0, 0x10000, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/table.go:204 +0x49b
github.com/cockroachdb/cockroach/pkg/sql.(*CachedPhysicalAccessor).GetObjectDesc(0xc008713f20, 0x4388300, 0xc0051b0300, 0xc003bedc20, 0xc000351500, 0xc0007ae8c0, 0x10000, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/physical_schema_accessors.go:302 +0xa2
github.com/cockroachdb/cockroach/pkg/sql.(*LogicalSchemaAccessor).GetObjectDesc(0xc008713f60, 0x4388300, 0xc0051b0300, 0xc003bedc20, 0xc000351500, 0xc0007ae8c0, 0x10000, 0x6, 0x101, 0xc001463bb8, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/logical_schema_accessors.go:100 +0x619
github.com/cockroachdb/cockroach/pkg/sql.(*planner).LookupObject(0xc0007ae3e8, 0x4388300, 0xc0051b0300, 0x10000, 0x633c274, 0x1, 0x3b6bc37, 0x6, 0xc0055445fd, 0x2, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/resolver.go:300 +0x19d
github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*TableName).ResolveExisting(0xc004a77110, 0x4388300, 0xc0051b0300, 0x7f22e774dbc8, 0xc0007ae3e8, 0x10000, 0x3b72528, 0x9, 0xc001deacf0, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/name_resolution.go:287 +0x7c9
github.com/cockroachdb/cockroach/pkg/sql.resolveExistingObjectImpl(0x4388300, 0xc0051b0300, 0x43ddf20, 0xc0007ae3e8, 0xc004a77110, 0x7f0000010000, 0x0, 0x20, 0xc000232dc0, 0xc008713f60, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/resolver.go:136 +0x176
github.com/cockroachdb/cockroach/pkg/sql.ResolveMutableExistingObject(0x4388300, 0xc0051b0300, 0x43ddf20, 0xc0007ae3e8, 0xc004a77110, 0xc001463e00, 0x0, 0x20, 0x37bcc80, 0x62f9d01)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/resolver.go:122 +0x7e
github.com/cockroachdb/cockroach/pkg/sql.(*planner).ResolveMutableTableDescriptor(...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/resolver.go:205
github.com/cockroachdb/cockroach/pkg/sql.(*planner).prepareDrop(0xc0007ae3e8, 0x4388300, 0xc0051b0300, 0xc004a77110, 0xc004a77000, 0x0, 0x2, 0x2, 0x1)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/drop_table.go:174 +0x75
github.com/cockroachdb/cockroach/pkg/sql.(*planner).DropDatabase(0xc0007ae3e8, 0x4388300, 0xc0051b0300, 0xc007dfbb20, 0xc001590e10, 0xc0043f1540, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/drop_database.go:102 +0x4ee
github.com/cockroachdb/cockroach/pkg/sql.buildOpaque(0x4388300, 0xc0051b0300, 0xc0007ae3f8, 0xc0007ae4c8, 0x43a0e00, 0xc007dfbb20, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/opaque.go:81 +0x1a84
github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).tryBuildOpaque(0xc00439a180, 0x43a0e00, 0xc007dfbb20, 0xc00326fc00, 0xc0007ae4c8)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/opaque.go:70 +0x131
github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmt(0xc00439a180, 0x43a0e00, 0xc007dfbb20, 0x0, 0x0, 0x0, 0xc00326fc00, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:322 +0x2f7
github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).buildStmtAtRoot(0xc00439a180, 0x43a0e00, 0xc007dfbb20, 0x0, 0x0, 0x0, 0xc00326fc00, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:217 +0xe3
github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder.(*Builder).Build(0xc00439a180, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder/builder.go:188 +0x172
github.com/cockroachdb/cockroach/pkg/sql.(*optPlanningCtx).buildExecMemo(0xc0007aead0, 0x4388300, 0xc0051b0300, 0x3a04280, 0x43f24a0, 0x660e0d0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:421 +0x14f
github.com/cockroachdb/cockroach/pkg/sql.(*planner).makeOptimizerPlan(0xc0007ae3e8, 0x4388300, 0xc0051b0300, 0xc0014646c8, 0x726776)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:159 +0x7d
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).makeExecPlan(0xc0007ae000, 0x4388300, 0xc0051b0300, 0xc0007ae3e8, 0x1, 0xc001464790)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:772 +0xc2
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc0007ae000, 0x4388300, 0xc0051b0300, 0xc0007ae3e8, 0x7f22e7682cf0, 0xc002fe1c80, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:665 +0xf9
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc0007ae000, 0x4388300, 0xc0051b0300, 0x43a0e00, 0xc007dfbb20, 0xc0036721ad, 0x17, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:455 +0xac8
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc0007ae000, 0x4388300, 0xc0051b0300, 0x43a0e00, 0xc007dfbb20, 0xc0036721ad, 0x17, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:117 +0x4f6
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc0007ae000, 0x4388240, 0xc006199440, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1396 +0x1aa8
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc0007ae000, 0x4388240, 0xc002ec4b40, 0xc004874e80, 0x5400, 0x15000, 0xc004874f18, 0xc001deac50, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1325 +0x1f2
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc0038a6680, 0x4388240, 0xc002ec4b40, 0xc0007ae000, 0x5400, 0x15000, 0xc004874f18, 0xc001deac50, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:479 +0x104
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc0035dc589, 0xc008713e60, 0x4388240, 0xc002ec4b40, 0xc001deac50, 0xc0038a6680, 0xc002fe1600, 0x439f7c0, 0xc008713e40, 0xc001d52420, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:566 +0x2e8
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync
/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:498 +0x17e
goroutine 196560 [IO wait]:
internal/poll.runtime_pollWait(0x7f22e6279c20, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc0052d2398, 0x72, 0x700, 0x733, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0052d2380, 0xc000bc0000, 0x733, 0x733, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc0052d2380, 0xc000bc0000, 0x733, 0x733, 0x203000, 0x4310fa0, 0x660dac0)
/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000105b8, 0xc000bc0000, 0x733, 0x733, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:184 +0x68
crypto/tls.(*atLeastReader).Read(0xc00286d660, 0xc000bc0000, 0x733, 0x733, 0x0, 0x43111e0, 0xc0011174c8)
/usr/local/go/src/crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc0000b3e58, 0x430bb00, 0xc00286d660, 0x66a285, 0x36fce40, 0x3ac61c0)
/usr/local/go/src/bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc0000b3c00, 0x4311220, 0xc0000105b8, 0x5, 0xc0000105b8, 0x80)
/usr/local/go/src/crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc0000b3c00, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc0000b3c00, 0xc000502000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1255 +0x161
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*readTimeoutConn).Read(0xc008713e00, 0xc000502000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:1539 +0x13d
bufio.(*Reader).fill(0xc002fe1648)
/usr/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).ReadByte(0xc002fe1648, 0xc0012f3840, 0x2a245b1, 0x0)
/usr/local/go/src/bufio/bufio.go:252 +0x39
github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase.(*ReadBuffer).ReadTypedMsg(0xc002fe1dd0, 0x435e080, 0xc002fe1648, 0x434d740, 0x660e0d0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase/encoding.go:118 +0x35
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl(0xc002fe1600, 0x4388240, 0xc00168dc80, 0xc001deac20, 0xc0038a6680, 0x5400, 0x15000, 0xc004874f18, 0x0, 0x4, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:298 +0x49b
github.com/cockroachdb/cockroach/pkg/sql/pgwire.serveConn(0x4388240, 0xc00168dc80, 0x43d8160, 0xc0000b3c00, 0xc0027fd046, 0x4, 0xc0041498f0, 0x434c280, 0xc004615380, 0x4000, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:153 +0x196
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc004874d80, 0x4388240, 0xc00168dc80, 0x43d8160, 0xc0000b3c00, 0xc005295901, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:512 +0x746
github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeSQL.func1.1(0x43d8e80, 0xc0000105b8)
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1998 +0x16a
github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith.func1(0xc0028cb5f0, 0x4388300, 0xc004615290, 0xc000df08e0, 0x43d8e80, 0xc0000105b8, 0xc0046152c0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:147 +0xc8
created by github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith
/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:144 +0x22b
goroutine 195745 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18a6)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432de0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432de0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20690, 0xc0028cb5f0, 0xc005f20680)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195667 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0052491a8, 0x1199)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc005249198)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/storage.(*RocksDB).syncLoop(0xc005249080)
/go/src/github.com/cockroachdb/cockroach/pkg/storage/rocksdb.go:678 +0x80
created by github.com/cockroachdb/cockroach/pkg/storage.(*RocksDB).open
/go/src/github.com/cockroachdb/cockroach/pkg/storage/rocksdb.go:665 +0x316
goroutine 195765 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver/compactor.(*Compactor).Start.func1.1(0x4388300, 0xc0014320f0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/compactor/compactor.go:123 +0x166
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20a50, 0xc0028cb5f0, 0xc00334d260)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195652 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x4388300, 0xc008a97380)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1750 +0x369
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ed310, 0xc0028cb5f0, 0xc004739740)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195717 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x1897)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432810)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432810)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20100, 0xc0028cb5f0, 0xc005f200f0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195692 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x4388300, 0xc004e7d470)
/go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:2153 +0x802
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004d22ee0, 0xc0028cb5f0, 0xc002178ea0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 197276 [select]:
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc001aea1c0)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113
created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1695
goroutine 195112 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func1(0x4388300, 0xc001432120)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:156 +0x4d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400dc80, 0xc0028cb5f0, 0xc00334d1c0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195708 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18cb)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432600)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432600)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00400dfb0, 0xc0028cb5f0, 0xc00400dfa0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 196859 [select, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func2(0x4388300, 0xc0046152f0)
/go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:448 +0xe9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059aec10, 0xc0028cb5f0, 0xc006cb3d40)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195755 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).coalescedHeartbeatsLoop(0xc0029ee700, 0x4388300, 0xc001432ff0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:635 +0x13b
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f208a0, 0xc0028cb5f0, 0xc005f20890)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195662 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
/go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc007026f00, 0x4375580, 0xc0052e0b40, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:587 +0x22e
github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func4(0x4388300, 0xc004d346c0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1906 +0x46
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0059ed5a0, 0xc0028cb5f0, 0xc0052e0b60)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195711 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x189c)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432690)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432690)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20020, 0xc0028cb5f0, 0xc005f20000)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195735 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x1893)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432ba0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432ba0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f204d0, 0xc0028cb5f0, 0xc005f204b0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195732 [sync.Cond.Wait]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0069bf110, 0x18b2)
/usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0069bf100)
/usr/local/go/src/sync/cond.go:56 +0x9d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0034b9980, 0x4388300, 0xc001432b10)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:197 +0xaa
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2(0x4388300, 0xc001432b10)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/scheduler.go:166 +0x3e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20450, 0xc0028cb5f0, 0xc005f20430)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195754 [select]:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).raftTickLoop(0xc0029ee700, 0x4388300, 0xc001432fc0)
/go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:579 +0x26e
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc005f20880, 0xc0028cb5f0, 0xc005f20870)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
goroutine 195492 [chan receive, 18 minutes]:
github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x4388300, 0xc004d341e0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:107 +0x64
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004b2b4d0, 0xc0028cb5f0, 0xc005e99340)
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:198 +0x13e
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:191 +0xa8
FAIL github.com/cockroachdb/cockroach/pkg/sql 1200.233s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment