Skip to content

Instantly share code, notes, and snippets.

@tamird

tamird/testrace.log

Created September 2, 2016 19:23
Show Gist options
  • Save tamird/8871a296306df3d1aa303ce1105b75a6 to your computer and use it in GitHub Desktop.
Save tamird/8871a296306df3d1aa303ce1105b75a6 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
testrace.log
=== RUN TestRaceWithBackfill
I160831 21:46:17.852726 18171 storage/engine/rocksdb.go:354 opening in memory rocksdb instance
I160831 21:46:17.900512 18171 gossip/gossip.go:230 [n?] initial resolvers: []
W160831 21:46:17.900731 18171 gossip/gossip.go:1041 [n?] no resolvers found; use --join to specify a connected node
W160831 21:46:17.907434 18171 server/status/runtime.go:109 Could not parse build timestamp: parsing time "" as "2006/01/02 15:04:05": cannot parse "" as "2006"
I160831 21:46:17.915087 18171 server/node.go:413 [n?] store store=0:0 not bootstrapped
I160831 21:46:17.945704 18154 storage/replica.go:2237 [s1,r1{/Min-/Max}] applying command with forced error: storage/replica.go:2162: no-op on empty Raft entry
E160831 21:46:17.946976 18154 storage/replica.go:2272 [s1,r1{/Min-/Max}] error executing raft command: storage/replica.go:2162: no-op on empty Raft entry
I160831 21:46:17.948354 18154 storage/replica_trigger.go:308 [s1,r1{/Min-/Max}] new range lease replica {1 1 1} 1970-01-01 00:00:00 +0000 UTC 409077h46m26.943491973s following replica {0 0 0} 1970-01-01 00:00:00 +0000 UTC 0s [physicalTime=2016-08-31 21:46:17.947962917 +0000 UTC]
I160831 21:46:17.949723 18154 storage/replica.go:1950 [s1,r1{/Min-/Max}] pending commands: refurbished 0, reproposing 1 (at 12.0); reasonNewLeader
I160831 21:46:17.951283 18151 storage/replica.go:2237 [s1,r1{/Min-/Max}] applying command with forced error: storage/replica.go:2193: command observed at lease index 1, but required < 1
E160831 21:46:17.952354 18151 storage/replica.go:2272 [s1,r1{/Min-/Max}] error executing raft command: storage/replica.go:2193: command observed at lease index 1, but required < 1
I160831 21:46:17.954551 18171 util/stop/stopper.go:342 stop has been called, stopping or quiescing all running tasks
I160831 21:46:17.954891 18171 server/node.go:345 [n?] **** cluster {c478fa64-ffca-403b-a4bb-77c542e52d23} has been created
I160831 21:46:17.954974 18171 server/node.go:346 [n?] **** add additional nodes by specifying --join=127.0.0.1:43238
I160831 21:46:17.956860 18171 gossip/gossip.go:264 [n?] NodeID set to 1
I160831 21:46:17.965302 18171 server/node.go:426 [n?] initialized store store=1:1: {Capacity:536870912 Available:536870912 RangeCount:0}
I160831 21:46:17.965512 18171 server/node.go:318 [n?] node ID 1 initialized
I160831 21:46:17.965749 18171 gossip/gossip.go:278 [n?] NodeDescriptor set to node_id:1 address:<network_field:"tcp" address_field:"127.0.0.1:43238" > attrs:<>
I160831 21:46:17.966212 18171 storage/stores.go:313 read 0 node addresses from persistent storage
I160831 21:46:17.966461 18171 server/node.go:555 [n?] connecting to gossip network to verify cluster ID...
I160831 21:46:17.967359 18198 storage/replica.go:2845 [n?,s1,r1{/Min-/Max}] gossiping cluster id {"c478fa64-ffca-403b-a4bb-77c542e52d23"} from store 1, range 1
I160831 21:46:17.967849 18198 storage/replica.go:2867 [n?,s1,r1{/Min-/Max}] gossiping sentinel from store 1, range 1
I160831 21:46:17.968566 18198 storage/replica.go:2874 [n?,s1,r1{/Min-/Max}] gossiping first range from store 1, range 1: [{1 1 1}]
I160831 21:46:17.970767 18203 kv/dist_sender.go:223 [n?] gossipped first range descriptor: [{NodeID:1 StoreID:1 ReplicaID:1}]
I160831 21:46:17.971106 18171 server/node.go:576 [n?] node connected via gossip and verified as part of cluster {"c478fa64-ffca-403b-a4bb-77c542e52d23"}
I160831 21:46:17.971771 18171 server/node.go:364 [n?] node=1: started with [[]=] engine(s) and attributes []
I160831 21:46:17.972764 18171 server/server.go:490 [n1] starting https server at 127.0.0.1:50337
I160831 21:46:17.972869 18171 server/server.go:491 [n1] starting grpc/postgres server at 127.0.0.1:43238
I160831 21:46:17.973237 18171 rpc/context.go:203 dialing 127.0.0.1:43238
I160831 21:46:17.975551 18209 storage/split_queue.go:99 splitting store=1:1 1{/Min-/Max} at keys [/Table/11/0 /Table/12/0 /Table/13/0 /Table/14/0]
I160831 21:46:17.977086 18233 sql/event_log.go:109 [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:{NetworkField:tcp AddressField:127.0.0.1:43238} Attrs:} ClusterID:{UUID:c478fa64-ffca-403b-a4bb-77c542e52d23} StartedAt:1472679977971180966}
I160831 21:46:18.049835 18161 storage/replica.go:2237 [n1,s1,r1{/Min-/Max}] applying command with forced error: storage/replica.go:2162: no-op on empty Raft entry
E160831 21:46:18.051427 18161 storage/replica.go:2272 [n1,s1,r1{/Min-/Max}] error executing raft command: storage/replica.go:2162: no-op on empty Raft entry
I160831 21:46:18.055552 18209 storage/replica_command.go:2190 [n1,s1,r1{/Min-/Max}] initiating a split of this range at key /Table/11
I160831 21:46:18.072608 18233 kv/dist_sender.go:1036 [n1] application error: retry txn "server/node.go:740 (*Node).recordJoinEvent.func1" id=34fc5aa9 key=/Table/12/1/2016-08-31T21:46:17.977178Z/"\x14p\x02\xa30E\xcd\xfb\x00"/0 rw=false pri=0.06244406 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.007732973,1 orig=1472679977.978000578,0 max=1472679977.978000578,0 wto=false rop=false
I160831 21:46:18.073612 18233 kv/dist_sender.go:771 [n1] reply error BeginTransaction [/Table/12/1/2016-08-31T21:46:17.977178Z/"\x14p\x02\xa30E\xcd\xfb\x00"/0,/Min), ConditionalPut [/Table/12/1/2016-08-31T21:46:17.977178Z/"\x14p\x02\xa30E\xcd\xfb\x00"/0,/Min), ConditionalPut [/Table/12/1/2016-08-31T21:46:17.977178Z/"\x14p\x02\xa30E\xcd\xfb\x00"/2/1,/Min), ConditionalPut [/Table/12/1/2016-08-31T21:46:17.977178Z/"\x14p\x02\xa30E\xcd\xfb\x00"/3/1,/Min), ConditionalPut [/Table/12/1/2016-08-31T21:46:17.977178Z/"\x14p\x02\xa30E\xcd\xfb\x00"/4/1,/Min), ConditionalPut [/Table/12/1/2016-08-31T21:46:17.977178Z/"\x14p\x02\xa30E\xcd\xfb\x00"/5/1,/Min): retry txn "server/node.go:740 (*Node).recordJoinEvent.func1" id=34fc5aa9 key=/Table/12/1/2016-08-31T21:46:17.977178Z/"\x14p\x02\xa30E\xcd\xfb\x00"/0 rw=false pri=0.06244406 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.007732973,1 orig=1472679977.978000578,0 max=1472679977.978000578,0 wto=false rop=false
I160831 21:46:18.074179 18233 internal/client/db.go:498 failed batch: retry txn "server/node.go:740 (*Node).recordJoinEvent.func1" id=34fc5aa9 key=/Table/12/1/2016-08-31T21:46:17.977178Z/"\x14p\x02\xa30E\xcd\xfb\x00"/0 rw=false pri=0.06244406 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679978.007732973,1 orig=1472679978.007732973,1 max=1472679978.028000578,0 wto=false rop=false
I160831 21:46:18.074427 18233 sql/event_log.go:109 [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:{NetworkField:tcp AddressField:127.0.0.1:43238} Attrs:} ClusterID:{UUID:c478fa64-ffca-403b-a4bb-77c542e52d23} StartedAt:1472679977971180966}
I160831 21:46:18.106802 18160 storage/replica_command.go:1357 [n1,s1,r1{/Min-/Max}] <nil> failed to push 4802be53: priority
I160831 21:46:18.107350 18160 storage/replica_command.go:1363 [n1,s1,r1{/Min-/Max}] failed to push "storage/replica_command.go:2236 (*Replica).AdminSplit" id=4802be53 key=/Local/Range/""/RangeDescriptor rw=true pri=0.01528696 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.056986437,0 orig=1472679978.056986437,0 max=1472679978.056986437,0 wto=false rop=false
I160831 21:46:18.149077 18209 storage/log.go:85 Range Event: "split", range: 1, info: {"UpdatedDesc":{"range_id":1,"end_key":"kw==","replicas":[{"node_id":1,"store_id":1,"replica_id":1}],"next_replica_id":2},"NewDesc":{"range_id":2,"start_key":"kw==","end_key":"//8=","replicas":[{"node_id":1,"store_id":1,"replica_id":1}],"next_replica_id":2}}
I160831 21:46:18.154078 18171 kv/dist_sender.go:1036 application error: failed to push "storage/replica_command.go:2236 (*Replica).AdminSplit" id=4802be53 key=/Local/Range/""/RangeDescriptor rw=true pri=0.01528696 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.056986437,0 orig=1472679978.056986437,0 max=1472679978.056986437,0 wto=false rop=false
I160831 21:46:18.154394 18171 kv/dist_sender.go:771 reply error PushTxn [/Local/Range/""/RangeDescriptor,/Min), PushTxn [/Local/Range/""/RangeDescriptor,/Min): failed to push "storage/replica_command.go:2236 (*Replica).AdminSplit" id=4802be53 key=/Local/Range/""/RangeDescriptor rw=true pri=0.01528696 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.056986437,0 orig=1472679978.056986437,0 max=1472679978.056986437,0 wto=false rop=false
I160831 21:46:18.154628 18171 internal/client/db.go:498 failed batch: failed to push "storage/replica_command.go:2236 (*Replica).AdminSplit" id=4802be53 key=/Local/Range/""/RangeDescriptor rw=true pri=0.01528696 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.056986437,0 orig=1472679978.056986437,0 max=1472679978.056986437,0 wto=false rop=false
I160831 21:46:18.154838 18171 storage/intent_resolver.go:102 [n1,s1] on Scan: failed to push "storage/replica_command.go:2236 (*Replica).AdminSplit" id=4802be53 key=/Local/Range/""/RangeDescriptor rw=true pri=0.01528696 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.056986437,0 orig=1472679978.056986437,0 max=1472679978.056986437,0 wto=false rop=false
W160831 21:46:18.155043 18171 storage/store.go:2096 [n1,s1] conflicting intents on [/Meta2/Table/11 /Meta2/Max]: resolved? false
I160831 21:46:18.225493 18160 storage/queue.go:293 [replicate] store=1:1 2{/Table/11-/Max}: split needed; not adding
I160831 21:46:18.245288 18209 kv/dist_sender.go:1036 application error: key range /Table/12/0-/Min outside of bounds of range /Min-/Table/11
I160831 21:46:18.245618 18209 kv/dist_sender.go:771 reply error AdminSplit [/Table/12/0,/Min): key range /Table/12/0-/Min outside of bounds of range /Min-/Table/11
W160831 21:46:18.246234 18209 kv/dist_sender.go:816 key range /Table/12/0-/Min outside of bounds of range /Min-/Table/11
I160831 21:46:18.248014 18209 storage/replica_command.go:2190 [n1,s1,r2{/Table/11-/Max}] initiating a split of this range at key /Table/12
I160831 21:46:18.252941 18210 storage/replica.go:2237 [n1,s1,r2{/Table/11-/Max}] applying command with forced error: storage/replica.go:2162: no-op on empty Raft entry
E160831 21:46:18.254504 18210 storage/replica.go:2272 [n1,s1,r2{/Table/11-/Max}] error executing raft command: storage/replica.go:2162: no-op on empty Raft entry
I160831 21:46:18.260768 18209 storage/log.go:85 Range Event: "split", range: 2, info: {"UpdatedDesc":{"range_id":2,"start_key":"kw==","end_key":"lA==","replicas":[{"node_id":1,"store_id":1,"replica_id":1}],"next_replica_id":2},"NewDesc":{"range_id":3,"start_key":"lA==","end_key":"//8=","replicas":[{"node_id":1,"store_id":1,"replica_id":1}],"next_replica_id":2}}
I160831 21:46:18.282793 18210 storage/queue.go:293 [replicate] store=1:1 3{/Table/12-/Max}: split needed; not adding
I160831 21:46:18.286957 18161 storage/replica.go:2237 [n1,s1,r3{/Table/12-/Max}] applying command with forced error: storage/replica.go:2162: no-op on empty Raft entry
I160831 21:46:18.333133 18244 rpc/clock_offset.go:147 0 of 0 nodes are within the maximum offset of 50ms
E160831 21:46:18.349531 18161 storage/replica.go:2272 [n1,s1,r3{/Table/12-/Max}] error executing raft command: storage/replica.go:2162: no-op on empty Raft entry
I160831 21:46:18.377215 18209 kv/dist_sender.go:1036 application error: key range /Table/13/0-/Min outside of bounds of range /Table/11-/Table/12
I160831 21:46:18.386673 18209 kv/dist_sender.go:771 reply error AdminSplit [/Table/13/0,/Min): key range /Table/13/0-/Min outside of bounds of range /Table/11-/Table/12
W160831 21:46:18.387170 18209 kv/dist_sender.go:816 key range /Table/13/0-/Min outside of bounds of range /Table/11-/Table/12
I160831 21:46:18.388506 18209 storage/replica_command.go:2190 [n1,s1,r3{/Table/12-/Max}] initiating a split of this range at key /Table/13
I160831 21:46:18.422717 18250 kv/dist_sender.go:1036 application error: end key /Local/Range/"\x93"/"rdsc\x00" must be greater than start /Local/Range/"\x93"/RangeDescriptor
I160831 21:46:18.423059 18250 kv/dist_sender.go:771 reply error GC [/Local/Range/"\x93"/RangeDescriptor,/Local/Range/"\x93"/"rdsc\x00"): end key /Local/Range/"\x93"/"rdsc\x00" must be greater than start /Local/Range/"\x93"/RangeDescriptor
I160831 21:46:18.423252 18250 internal/client/db.go:498 failed batch: end key /Local/Range/"\x93"/"rdsc\x00" must be greater than start /Local/Range/"\x93"/RangeDescriptor
W160831 21:46:18.423353 18250 storage/intent_resolver.go:344 could not GC completed transaction: end key /Local/Range/"\x93"/"rdsc\x00" must be greater than start /Local/Range/"\x93"/RangeDescriptor
W160831 21:46:18.450546 18171 storage/store.go:2091 [n1,s1] conflicting intents on [/Meta2/Table/12 /Meta2/Max]: resolved? false
I160831 21:46:18.494299 18209 storage/log.go:85 Range Event: "split", range: 3, info: {"UpdatedDesc":{"range_id":3,"start_key":"lA==","end_key":"lQ==","replicas":[{"node_id":1,"store_id":1,"replica_id":1}],"next_replica_id":2},"NewDesc":{"range_id":4,"start_key":"lQ==","end_key":"//8=","replicas":[{"node_id":1,"store_id":1,"replica_id":1}],"next_replica_id":2}}
I160831 21:46:18.514650 18210 storage/replica_command.go:1357 [n1,s1,r3{/Table/12-/Max}] <nil> pushed 935373f0: priority
W160831 21:46:18.519247 18171 storage/store.go:2091 [n1,s1] conflicting intents on [/Meta2/Table/13 /Meta2/Max]: resolved? false
I160831 21:46:18.541138 18160 storage/replica_command.go:1357 [n1,s1,r3{/Table/12-/Max}] <nil> failed to push 935373f0: priority
I160831 21:46:18.543420 18160 storage/replica_command.go:1363 [n1,s1,r3{/Table/12-/Max}] failed to push "storage/replica_command.go:2236 (*Replica).AdminSplit" id=935373f0 key=/Local/Range/"\x94"/RangeDescriptor rw=true pri=0.07068231 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.510985367,1 orig=1472679978.388986362,0 max=1472679978.388986362,0 wto=false rop=false
I160831 21:46:18.560129 18171 kv/dist_sender.go:1036 application error: failed to push "storage/replica_command.go:2236 (*Replica).AdminSplit" id=935373f0 key=/Local/Range/"\x94"/RangeDescriptor rw=true pri=0.07068231 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.510985367,1 orig=1472679978.388986362,0 max=1472679978.388986362,0 wto=false rop=false
I160831 21:46:18.560641 18171 kv/dist_sender.go:771 reply error PushTxn [/Local/Range/"\x94"/RangeDescriptor,/Min), PushTxn [/Local/Range/"\x94"/RangeDescriptor,/Min): failed to push "storage/replica_command.go:2236 (*Replica).AdminSplit" id=935373f0 key=/Local/Range/"\x94"/RangeDescriptor rw=true pri=0.07068231 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.510985367,1 orig=1472679978.388986362,0 max=1472679978.388986362,0 wto=false rop=false
I160831 21:46:18.560885 18171 internal/client/db.go:498 failed batch: failed to push "storage/replica_command.go:2236 (*Replica).AdminSplit" id=935373f0 key=/Local/Range/"\x94"/RangeDescriptor rw=true pri=0.07068231 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.510985367,1 orig=1472679978.388986362,0 max=1472679978.388986362,0 wto=false rop=false
I160831 21:46:18.561156 18171 storage/intent_resolver.go:102 [n1,s1] on Scan: failed to push "storage/replica_command.go:2236 (*Replica).AdminSplit" id=935373f0 key=/Local/Range/"\x94"/RangeDescriptor rw=true pri=0.07068231 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.510985367,1 orig=1472679978.388986362,0 max=1472679978.388986362,0 wto=false rop=false
W160831 21:46:18.561410 18171 storage/store.go:2096 [n1,s1] conflicting intents on [/Meta2/Table/13 /Meta2/Max]: resolved? false
I160831 21:46:18.568577 18209 kv/dist_sender.go:1036 application error: retry txn "storage/replica_command.go:2236 (*Replica).AdminSplit" id=935373f0 key=/Local/Range/"\x94"/RangeDescriptor rw=true pri=0.07068231 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.510985367,1 orig=1472679978.388986362,0 max=1472679978.388986362,0 wto=false rop=false
I160831 21:46:18.569349 18209 kv/dist_sender.go:771 reply error EndTransaction [/Local/Range/"\x94"/RangeDescriptor,/Min): retry txn "storage/replica_command.go:2236 (*Replica).AdminSplit" id=935373f0 key=/Local/Range/"\x94"/RangeDescriptor rw=true pri=0.07068231 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.510985367,1 orig=1472679978.388986362,0 max=1472679978.388986362,0 wto=false rop=false
I160831 21:46:18.570241 18209 internal/client/db.go:498 failed batch: retry txn "storage/replica_command.go:2236 (*Replica).AdminSplit" id=935373f0 key=/Local/Range/"\x94"/RangeDescriptor rw=true pri=0.07068231 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679978.510985367,1 orig=1472679978.510985367,1 max=1472679978.438986362,0 wto=false rop=false
I160831 21:46:18.588214 18209 storage/log.go:85 Range Event: "split", range: 3, info: {"UpdatedDesc":{"range_id":3,"start_key":"lA==","end_key":"lQ==","replicas":[{"node_id":1,"store_id":1,"replica_id":1}],"next_replica_id":2},"NewDesc":{"range_id":4,"start_key":"lQ==","end_key":"//8=","replicas":[{"node_id":1,"store_id":1,"replica_id":1}],"next_replica_id":2}}
I160831 21:46:18.640456 18210 storage/queue.go:293 [replicate] store=1:1 4{/Table/13-/Max}: split needed; not adding
I160831 21:46:18.645137 18209 kv/dist_sender.go:1036 application error: key range /Table/14/0-/Min outside of bounds of range /Table/12-/Table/13
I160831 21:46:18.645429 18209 kv/dist_sender.go:771 reply error AdminSplit [/Table/14/0,/Min): key range /Table/14/0-/Min outside of bounds of range /Table/12-/Table/13
W160831 21:46:18.646017 18209 kv/dist_sender.go:816 key range /Table/14/0-/Min outside of bounds of range /Table/12-/Table/13
I160831 21:46:18.647522 18209 storage/replica_command.go:2190 [n1,s1,r4{/Table/13-/Max}] initiating a split of this range at key /Table/14
I160831 21:46:18.655094 18161 storage/replica.go:2237 [n1,s1,r4{/Table/13-/Max}] applying command with forced error: storage/replica.go:2162: no-op on empty Raft entry
E160831 21:46:18.656267 18161 storage/replica.go:2272 [n1,s1,r4{/Table/13-/Max}] error executing raft command: storage/replica.go:2162: no-op on empty Raft entry
I160831 21:46:18.660659 18217 kv/dist_sender.go:1036 application error: end key /Local/Range/"\x94"/"rdsc\x00" must be greater than start /Local/Range/"\x94"/RangeDescriptor
I160831 21:46:18.660997 18217 kv/dist_sender.go:771 reply error GC [/Local/Range/"\x94"/RangeDescriptor,/Local/Range/"\x94"/"rdsc\x00"): end key /Local/Range/"\x94"/"rdsc\x00" must be greater than start /Local/Range/"\x94"/RangeDescriptor
I160831 21:46:18.661170 18217 internal/client/db.go:498 failed batch: end key /Local/Range/"\x94"/"rdsc\x00" must be greater than start /Local/Range/"\x94"/RangeDescriptor
W160831 21:46:18.661260 18217 storage/intent_resolver.go:344 could not GC completed transaction: end key /Local/Range/"\x94"/"rdsc\x00" must be greater than start /Local/Range/"\x94"/RangeDescriptor
W160831 21:46:18.663365 18171 storage/store.go:2091 [n1,s1] conflicting intents on [/Meta2/Table/13 /Meta2/Max]: resolved? false
I160831 21:46:18.675634 18209 storage/log.go:85 Range Event: "split", range: 4, info: {"UpdatedDesc":{"range_id":4,"start_key":"lQ==","end_key":"lg==","replicas":[{"node_id":1,"store_id":1,"replica_id":1}],"next_replica_id":2},"NewDesc":{"range_id":5,"start_key":"lg==","end_key":"//8=","replicas":[{"node_id":1,"store_id":1,"replica_id":1}],"next_replica_id":2}}
I160831 21:46:18.687661 18160 storage/replica_command.go:1357 [n1,s1,r4{/Table/13-/Max}] <nil> pushed 9052e97d: priority
W160831 21:46:18.695626 18171 storage/store.go:2091 [n1,s1] conflicting intents on [/Meta2/Table/14 /Meta2/Max]: resolved? false
I160831 21:46:18.713587 18209 kv/dist_sender.go:1036 application error: retry txn "storage/replica_command.go:2236 (*Replica).AdminSplit" id=9052e97d key=/Local/Range/"\x95"/RangeDescriptor rw=true pri=0.04569166 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.667585506,1 orig=1472679978.648149755,0 max=1472679978.648149755,0 wto=false rop=false
I160831 21:46:18.713848 18209 kv/dist_sender.go:771 reply error EndTransaction [/Local/Range/"\x95"/RangeDescriptor,/Min): retry txn "storage/replica_command.go:2236 (*Replica).AdminSplit" id=9052e97d key=/Local/Range/"\x95"/RangeDescriptor rw=true pri=0.04569166 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.667585506,1 orig=1472679978.648149755,0 max=1472679978.648149755,0 wto=false rop=false
I160831 21:46:18.715570 18160 storage/replica_command.go:1357 [n1,s1,r4{/Table/13-/Max}] <nil> failed to push 9052e97d: priority
I160831 21:46:18.718594 18160 storage/replica_command.go:1363 [n1,s1,r4{/Table/13-/Max}] failed to push "storage/replica_command.go:2236 (*Replica).AdminSplit" id=9052e97d key=/Local/Range/"\x95"/RangeDescriptor rw=true pri=0.04569166 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.667585506,1 orig=1472679978.648149755,0 max=1472679978.648149755,0 wto=false rop=false
I160831 21:46:18.721944 18171 kv/dist_sender.go:1036 application error: failed to push "storage/replica_command.go:2236 (*Replica).AdminSplit" id=9052e97d key=/Local/Range/"\x95"/RangeDescriptor rw=true pri=0.04569166 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.667585506,1 orig=1472679978.648149755,0 max=1472679978.648149755,0 wto=false rop=false
I160831 21:46:18.722267 18171 kv/dist_sender.go:771 reply error PushTxn [/Local/Range/"\x95"/RangeDescriptor,/Min), PushTxn [/Local/Range/"\x95"/RangeDescriptor,/Min): failed to push "storage/replica_command.go:2236 (*Replica).AdminSplit" id=9052e97d key=/Local/Range/"\x95"/RangeDescriptor rw=true pri=0.04569166 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.667585506,1 orig=1472679978.648149755,0 max=1472679978.648149755,0 wto=false rop=false
I160831 21:46:18.722605 18209 internal/client/db.go:498 failed batch: retry txn "storage/replica_command.go:2236 (*Replica).AdminSplit" id=9052e97d key=/Local/Range/"\x95"/RangeDescriptor rw=true pri=0.04569166 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679978.667585506,1 orig=1472679978.667585506,1 max=1472679978.698149755,0 wto=false rop=false
I160831 21:46:18.748297 18171 internal/client/db.go:498 failed batch: failed to push "storage/replica_command.go:2236 (*Replica).AdminSplit" id=9052e97d key=/Local/Range/"\x95"/RangeDescriptor rw=true pri=0.04569166 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.667585506,1 orig=1472679978.648149755,0 max=1472679978.648149755,0 wto=false rop=false
I160831 21:46:18.748601 18171 storage/intent_resolver.go:102 [n1,s1] on Scan: failed to push "storage/replica_command.go:2236 (*Replica).AdminSplit" id=9052e97d key=/Local/Range/"\x95"/RangeDescriptor rw=true pri=0.04569166 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679978.667585506,1 orig=1472679978.648149755,0 max=1472679978.648149755,0 wto=false rop=false
W160831 21:46:18.748847 18171 storage/store.go:2096 [n1,s1] conflicting intents on [/Meta2/Table/14 /Meta2/Max]: resolved? false
I160831 21:46:18.756933 18209 storage/log.go:85 Range Event: "split", range: 4, info: {"UpdatedDesc":{"range_id":4,"start_key":"lQ==","end_key":"lg==","replicas":[{"node_id":1,"store_id":1,"replica_id":1}],"next_replica_id":2},"NewDesc":{"range_id":5,"start_key":"lg==","end_key":"//8=","replicas":[{"node_id":1,"store_id":1,"replica_id":1}],"next_replica_id":2}}
I160831 21:46:18.808483 18161 storage/replica.go:2237 [n1,s1,r5{/Table/14-/Max}] applying command with forced error: storage/replica.go:2162: no-op on empty Raft entry
E160831 21:46:18.810477 18161 storage/replica.go:2272 [n1,s1,r5{/Table/14-/Max}] error executing raft command: storage/replica.go:2162: no-op on empty Raft entry
I160831 21:46:18.847618 18219 kv/dist_sender.go:1036 application error: end key /Local/Range/"\x95"/"rdsc\x00" must be greater than start /Local/Range/"\x95"/RangeDescriptor
I160831 21:46:18.848108 18219 kv/dist_sender.go:771 reply error GC [/Local/Range/"\x95"/RangeDescriptor,/Local/Range/"\x95"/"rdsc\x00"): end key /Local/Range/"\x95"/"rdsc\x00" must be greater than start /Local/Range/"\x95"/RangeDescriptor
I160831 21:46:18.848354 18219 internal/client/db.go:498 failed batch: end key /Local/Range/"\x95"/"rdsc\x00" must be greater than start /Local/Range/"\x95"/RangeDescriptor
W160831 21:46:18.848476 18219 storage/intent_resolver.go:344 could not GC completed transaction: end key /Local/Range/"\x95"/"rdsc\x00" must be greater than start /Local/Range/"\x95"/RangeDescriptor
W160831 21:46:19.058623 18222 sql/pgwire/v3.go:176 unrecognized configuration parameter "client_encoding"
W160831 21:46:19.083519 18222 sql/pgwire/v3.go:176 unrecognized configuration parameter "datestyle"
W160831 21:46:19.083695 18222 sql/pgwire/v3.go:176 unrecognized configuration parameter "extra_float_digits"
I160831 21:46:19.106336 18222 sql/event_log.go:109 [n1] Event: "create_database", target: 50, info: {DatabaseName:t Statement:CREATE DATABASE t User:root}
I160831 21:46:19.186503 18209 storage/split_queue.go:99 splitting store=1:1 5{/Table/14-/Max} at keys [/Table/50/0]
I160831 21:46:19.188525 18209 kv/dist_sender.go:1036 application error: key range /Table/50/0-/Min outside of bounds of range /Table/13-/Table/14
I160831 21:46:19.188766 18209 kv/dist_sender.go:771 reply error AdminSplit [/Table/50/0,/Min): key range /Table/50/0-/Min outside of bounds of range /Table/13-/Table/14
W160831 21:46:19.205279 18209 kv/dist_sender.go:816 key range /Table/50/0-/Min outside of bounds of range /Table/13-/Table/14
I160831 21:46:19.207075 18209 storage/replica_command.go:2190 [n1,s1,r5{/Table/14-/Max}] initiating a split of this range at key /Table/50
I160831 21:46:19.220761 18222 sql/event_log.go:109 [n1] Event: "create_table", target: 51, info: {TableName:t.test Statement:CREATE TABLE t.test (k INT PRIMARY KEY, v INT, pi DECIMAL DEFAULT (CAST('3.14' AS DECIMAL))) User:root}
I160831 21:46:19.246362 18209 storage/log.go:85 Range Event: "split", range: 5, info: {"UpdatedDesc":{"range_id":5,"start_key":"lg==","end_key":"ug==","replicas":[{"node_id":1,"store_id":1,"replica_id":1}],"next_replica_id":2},"NewDesc":{"range_id":6,"start_key":"ug==","end_key":"//8=","replicas":[{"node_id":1,"store_id":1,"replica_id":1}],"next_replica_id":2}}
I160831 21:46:19.320866 18210 storage/queue.go:293 [replicate] store=1:1 6{/Table/50-/Max}: split needed; not adding
I160831 21:46:19.338995 18161 storage/replica.go:2237 [n1,s1,r6{/Table/50-/Max}] applying command with forced error: storage/replica.go:2162: no-op on empty Raft entry
E160831 21:46:19.341499 18161 storage/replica.go:2272 [n1,s1,r6{/Table/50-/Max}] error executing raft command: storage/replica.go:2162: no-op on empty Raft entry
I160831 21:46:19.344325 18222 sql/event_log.go:109 [n1] Event: "create_index", target: 51, info: {TableName:test IndexName:vidx Statement:CREATE UNIQUE INDEX vidx ON t.test (v) User:root MutationID:1}
I160831 21:46:19.359846 18262 kv/dist_sender.go:1036 application error: end key /Local/Range/"\x96"/"rdsc\x00" must be greater than start /Local/Range/"\x96"/RangeDescriptor
I160831 21:46:19.360324 18262 kv/dist_sender.go:771 reply error GC [/Local/Range/"\x96"/RangeDescriptor,/Local/Range/"\x96"/"rdsc\x00"): end key /Local/Range/"\x96"/"rdsc\x00" must be greater than start /Local/Range/"\x96"/RangeDescriptor
I160831 21:46:19.360541 18262 internal/client/db.go:498 failed batch: end key /Local/Range/"\x96"/"rdsc\x00" must be greater than start /Local/Range/"\x96"/RangeDescriptor
W160831 21:46:19.360667 18262 storage/intent_resolver.go:344 could not GC completed transaction: end key /Local/Range/"\x96"/"rdsc\x00" must be greater than start /Local/Range/"\x96"/RangeDescriptor
I160831 21:46:19.403250 18209 storage/split_queue.go:99 splitting store=1:1 6{/Table/50-/Max} at keys [/Table/51/0]
I160831 21:46:19.404994 18209 kv/dist_sender.go:1036 application error: key range /Table/51/0-/Min outside of bounds of range /Table/14-/Table/50
I160831 21:46:19.405189 18209 kv/dist_sender.go:771 reply error AdminSplit [/Table/51/0,/Min): key range /Table/51/0-/Min outside of bounds of range /Table/14-/Table/50
W160831 21:46:19.405661 18209 kv/dist_sender.go:816 key range /Table/51/0-/Min outside of bounds of range /Table/14-/Table/50
I160831 21:46:19.406810 18209 storage/replica_command.go:2190 [n1,s1,r6{/Table/50-/Max}] initiating a split of this range at key /Table/51
I160831 21:46:19.431940 18209 storage/log.go:85 Range Event: "split", range: 6, info: {"UpdatedDesc":{"range_id":6,"start_key":"ug==","end_key":"uw==","replicas":[{"node_id":1,"store_id":1,"replica_id":1}],"next_replica_id":2},"NewDesc":{"range_id":7,"start_key":"uw==","end_key":"//8=","replicas":[{"node_id":1,"store_id":1,"replica_id":1}],"next_replica_id":2}}
I160831 21:46:19.495850 18161 storage/replica.go:2237 [n1,s1,r7{/Table/51-/Max}] applying command with forced error: storage/replica.go:2162: no-op on empty Raft entry
E160831 21:46:19.497246 18161 storage/replica.go:2272 [n1,s1,r7{/Table/51-/Max}] error executing raft command: storage/replica.go:2162: no-op on empty Raft entry
I160831 21:46:19.556324 18270 kv/dist_sender.go:1036 application error: end key /Local/Range/"\xba"/"rdsc\x00" must be greater than start /Local/Range/"\xba"/RangeDescriptor
I160831 21:46:19.567741 18270 kv/dist_sender.go:771 reply error GC [/Local/Range/"\xba"/RangeDescriptor,/Local/Range/"\xba"/"rdsc\x00"): end key /Local/Range/"\xba"/"rdsc\x00" must be greater than start /Local/Range/"\xba"/RangeDescriptor
I160831 21:46:19.568584 18270 internal/client/db.go:498 failed batch: end key /Local/Range/"\xba"/"rdsc\x00" must be greater than start /Local/Range/"\xba"/RangeDescriptor
W160831 21:46:19.568792 18270 storage/intent_resolver.go:344 could not GC completed transaction: end key /Local/Range/"\xba"/"rdsc\x00" must be greater than start /Local/Range/"\xba"/RangeDescriptor
I160831 21:46:19.590897 18222 kv/dist_sender.go:1036 application error: key range /Table/51/1-/Table/51/2 outside of bounds of range /Table/50-/Table/51
I160831 21:46:19.591237 18222 kv/dist_sender.go:771 reply error Scan [/Table/51/1,/Table/51/2): key range /Table/51/1-/Table/51/2 outside of bounds of range /Table/50-/Table/51
W160831 21:46:19.591922 18222 kv/dist_sender.go:816 key range /Table/51/1-/Table/51/2 outside of bounds of range /Table/50-/Table/51
I160831 21:46:19.613913 18222 sql/event_log.go:109 Event: "finish_schema_change", target: 51, info: {MutationID:1}
I160831 21:46:21.408214 18244 rpc/clock_offset.go:147 1 of 1 nodes are within the maximum offset of 50ms
I160831 21:46:23.039682 18175 kv/txn_coord_sender.go:251 [n1] txn coordinator: 4.42 txn/sec, 100.00/37.06/0.00/0.00 %cmmt/cmmt1pc/abrt/abnd, 91ms/66ms/265ms avg/σ/max duration, 0.2/0.4/1 avg/σ/max restarts (16 samples)
I160831 21:46:24.442465 18244 rpc/clock_offset.go:147 1 of 1 nodes are within the maximum offset of 50ms
I160831 21:46:27.487734 18244 rpc/clock_offset.go:147 1 of 1 nodes are within the maximum offset of 50ms
I160831 21:46:27.948211 18176 storage/raft_transport.go:217 stats:
qlen qmax qdropped client-sent client-recv server-sent server-recv
I160831 21:46:28.017169 18234 server/status/runtime.go:207 runtime stats: 645 MiB RSS, 59 goroutines, 74 MiB/5.2 MiB/94 MiB GO alloc/idle/total, 27 MiB/37 MiB CGO alloc/total, 0.00cgo/sec, 0.00/0.00 %(u/s)time, 0.00 %gc (547x)
I160831 21:46:28.117214 18175 kv/txn_coord_sender.go:251 [n1] txn coordinator: 3.23 txn/sec, 100.00/38.32/0.00/0.00 %cmmt/cmmt1pc/abrt/abnd, 402ms/1.247s/5.402s avg/σ/max duration, 0.2/0.4/1 avg/σ/max restarts (17 samples)
I160831 21:46:28.136463 18222 sql/event_log.go:109 [n1] Event: "alter_table", target: 51, info: {TableName:test Statement:ALTER TABLE t.test ADD COLUMN x DECIMAL DEFAULT (CAST('1.4' AS DECIMAL)) User:root MutationID:2}
I160831 21:46:28.403916 18222 sql/lease.go:269 publish (count leases): descID=51 name=test version=4 count=1
I160831 21:46:28.551619 18222 sql/lease.go:269 publish (count leases): descID=51 name=test version=4 count=1
I160831 21:46:28.748260 18222 sql/lease.go:269 publish (count leases): descID=51 name=test version=5 count=1
I160831 21:46:28.864134 18222 sql/lease.go:269 publish (count leases): descID=51 name=test version=5 count=1
I160831 21:46:28.887786 18297 kv/dist_sender.go:1036 application error: retry txn "sql/lease.go:228 LeaseStore.Release" id=74708c1d key=/Table/11/1/51/5/2016-08-31T21:51:28.989669Z/1 rw=true pri=0.03552814 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679988.790964060,1 orig=1472679988.786834986,0 max=1472679988.786834986,0 wto=false rop=false
I160831 21:46:28.888335 18297 kv/dist_sender.go:771 reply error EndTransaction [/Table/11/1/51/5/2016-08-31T21:51:28.989669Z/1,/Min): retry txn "sql/lease.go:228 LeaseStore.Release" id=74708c1d key=/Table/11/1/51/5/2016-08-31T21:51:28.989669Z/1 rw=true pri=0.03552814 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679988.790964060,1 orig=1472679988.786834986,0 max=1472679988.786834986,0 wto=false rop=false
I160831 21:46:28.888892 18297 internal/client/db.go:498 failed batch: retry txn "sql/lease.go:228 LeaseStore.Release" id=74708c1d key=/Table/11/1/51/5/2016-08-31T21:51:28.989669Z/1 rw=true pri=0.03552814 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679988.790964060,1 orig=1472679988.790964060,1 max=1472679988.836834986,0 wto=false rop=false
I160831 21:46:28.941441 18210 storage/replica_command.go:1357 [n1,s1,r2{/Table/11-/Table/12}] 3ac572c9 failed to push 74708c1d: priority
I160831 21:46:28.942026 18210 storage/replica_command.go:1363 [n1,s1,r2{/Table/11-/Table/12}] failed to push "sql/lease.go:228 LeaseStore.Release" id=74708c1d key=/Table/11/1/51/5/2016-08-31T21:51:28.989669Z/1 rw=true pri=0.03552814 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679988.790964060,1 orig=1472679988.786834986,0 max=1472679988.786834986,0 wto=false rop=false
I160831 21:46:28.970683 18222 kv/dist_sender.go:1036 application error: failed to push "sql/lease.go:228 LeaseStore.Release" id=74708c1d key=/Table/11/1/51/5/2016-08-31T21:51:28.989669Z/1 rw=true pri=0.03552814 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679988.790964060,1 orig=1472679988.786834986,0 max=1472679988.786834986,0 wto=false rop=false
I160831 21:46:28.971107 18222 kv/dist_sender.go:771 reply error PushTxn [/Table/11/1/51/5/2016-08-31T21:51:28.989669Z/1,/Min): failed to push "sql/lease.go:228 LeaseStore.Release" id=74708c1d key=/Table/11/1/51/5/2016-08-31T21:51:28.989669Z/1 rw=true pri=0.03552814 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679988.790964060,1 orig=1472679988.786834986,0 max=1472679988.786834986,0 wto=false rop=false
I160831 21:46:28.971392 18222 internal/client/db.go:498 failed batch: failed to push "sql/lease.go:228 LeaseStore.Release" id=74708c1d key=/Table/11/1/51/5/2016-08-31T21:51:28.989669Z/1 rw=true pri=0.03552814 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679988.790964060,1 orig=1472679988.786834986,0 max=1472679988.786834986,0 wto=false rop=false
I160831 21:46:28.971616 18222 storage/intent_resolver.go:102 [n1,s1] on Scan: failed to push "sql/lease.go:228 LeaseStore.Release" id=74708c1d key=/Table/11/1/51/5/2016-08-31T21:51:28.989669Z/1 rw=true pri=0.03552814 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679988.790964060,1 orig=1472679988.786834986,0 max=1472679988.786834986,0 wto=false rop=false
W160831 21:46:28.971925 18222 storage/store.go:2096 [n1,s1] conflicting intents on [/Table/11/1/51/5/2016-08-31T21:51:28.989669Z/1/0]: resolved? false
W160831 21:46:29.695575 18285 sql/pgwire/v3.go:176 unrecognized configuration parameter "client_encoding"
W160831 21:46:29.726257 18285 sql/pgwire/v3.go:176 unrecognized configuration parameter "datestyle"
W160831 21:46:29.726421 18285 sql/pgwire/v3.go:176 unrecognized configuration parameter "extra_float_digits"
I160831 21:46:30.004323 18160 storage/replica_command.go:1357 [n1,s1,r7{/Table/51-/Max}] 24765a86 pushed 04bb406c: priority
W160831 21:46:30.008728 18285 storage/store.go:2091 [n1,n1,s1] conflicting intents on [/Table/51/1/403/2/1]: resolved? false
I160831 21:46:30.160195 18222 kv/dist_sender.go:1036 application error: retry txn "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=04bb406c key=/Table/51/1/0/2/1 rw=true pri=0.03710520 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679989.782673648,1 orig=1472679989.060876119,0 max=1472679989.060876119,0 wto=false rop=false
I160831 21:46:30.160596 18222 kv/dist_sender.go:771 reply error EndTransaction [/Table/51/1/0/2/1,/Min): retry txn "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=04bb406c key=/Table/51/1/0/2/1 rw=true pri=0.03710520 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679989.782673648,1 orig=1472679989.060876119,0 max=1472679989.060876119,0 wto=false rop=false
I160831 21:46:30.161436 18222 internal/client/db.go:498 failed batch: retry txn "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=04bb406c key=/Table/51/1/0/2/1 rw=true pri=0.03710520 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679989.782673648,1 orig=1472679989.782673648,1 max=1472679989.110876119,0 wto=false rop=false
I160831 21:46:30.537990 18161 storage/replica_command.go:1357 [n1,s1,r7{/Table/51-/Max}] c138721d failed to push 04bb406c: priority
I160831 21:46:30.540732 18161 storage/replica_command.go:1363 [n1,s1,r7{/Table/51-/Max}] failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=04bb406c key=/Table/51/1/0/2/1 rw=true pri=0.03710520 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679989.782673648,1 orig=1472679989.060876119,0 max=1472679989.060876119,0 wto=false rop=false
I160831 21:46:30.561289 18285 kv/dist_sender.go:1036 application error: failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=04bb406c key=/Table/51/1/0/2/1 rw=true pri=0.03710520 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679989.782673648,1 orig=1472679989.060876119,0 max=1472679989.060876119,0 wto=false rop=false
I160831 21:46:30.561726 18285 kv/dist_sender.go:771 reply error PushTxn [/Table/51/1/0/2/1,/Min): failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=04bb406c key=/Table/51/1/0/2/1 rw=true pri=0.03710520 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679989.782673648,1 orig=1472679989.060876119,0 max=1472679989.060876119,0 wto=false rop=false
I160831 21:46:30.562024 18285 internal/client/db.go:498 failed batch: failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=04bb406c key=/Table/51/1/0/2/1 rw=true pri=0.03710520 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679989.782673648,1 orig=1472679989.060876119,0 max=1472679989.060876119,0 wto=false rop=false
I160831 21:46:30.562330 18285 storage/intent_resolver.go:102 [n1,n1,s1] on Scan: failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=04bb406c key=/Table/51/1/0/2/1 rw=true pri=0.03710520 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679989.782673648,1 orig=1472679989.060876119,0 max=1472679989.060876119,0 wto=false rop=false
W160831 21:46:30.562676 18285 storage/store.go:2096 [n1,n1,s1] conflicting intents on [/Table/51/1/208/2/1]: resolved? false
I160831 21:46:30.621427 18244 rpc/clock_offset.go:147 1 of 1 nodes are within the maximum offset of 50ms
I160831 21:46:30.909540 18160 storage/replica_command.go:1357 [n1,s1,r7{/Table/51-/Max}] c138721d failed to push 04bb406c: priority
I160831 21:46:30.910188 18160 storage/replica_command.go:1363 [n1,s1,r7{/Table/51-/Max}] failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=04bb406c key=/Table/51/1/0/2/1 rw=true pri=0.03710520 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679989.782673648,1 orig=1472679989.060876119,0 max=1472679989.060876119,0 wto=false rop=false
I160831 21:46:30.912326 18285 kv/dist_sender.go:1036 application error: failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=04bb406c key=/Table/51/1/0/2/1 rw=true pri=0.03710520 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679989.782673648,1 orig=1472679989.060876119,0 max=1472679989.060876119,0 wto=false rop=false
I160831 21:46:30.913002 18285 kv/dist_sender.go:771 reply error PushTxn [/Table/51/1/0/2/1,/Min): failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=04bb406c key=/Table/51/1/0/2/1 rw=true pri=0.03710520 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679989.782673648,1 orig=1472679989.060876119,0 max=1472679989.060876119,0 wto=false rop=false
I160831 21:46:30.913478 18285 internal/client/db.go:498 failed batch: failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=04bb406c key=/Table/51/1/0/2/1 rw=true pri=0.03710520 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679989.782673648,1 orig=1472679989.060876119,0 max=1472679989.060876119,0 wto=false rop=false
I160831 21:46:30.914121 18285 storage/intent_resolver.go:102 [n1,n1,s1] on Scan: failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=04bb406c key=/Table/51/1/0/2/1 rw=true pri=0.03710520 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679989.782673648,1 orig=1472679989.060876119,0 max=1472679989.060876119,0 wto=false rop=false
W160831 21:46:30.914801 18285 storage/store.go:2096 [n1,n1,s1] conflicting intents on [/Table/51/1/208/2/1]: resolved? false
I160831 21:46:31.416530 18160 storage/replica_command.go:1357 [n1,s1,r7{/Table/51-/Max}] 37412bf5 pushed 3d786b5b: priority
W160831 21:46:31.422614 18285 storage/store.go:2091 [n1,n1,s1] conflicting intents on [/Table/51/1/1106/2/1]: resolved? false
I160831 21:46:31.462883 18222 kv/dist_sender.go:1036 application error: retry txn "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=3d786b5b key=/Table/51/1/600/2/1 rw=true pri=0.03766012 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679991.413378476,1 orig=1472679991.061787050,0 max=1472679991.061787050,0 wto=false rop=false
I160831 21:46:31.463238 18222 kv/dist_sender.go:771 reply error EndTransaction [/Table/51/1/600/2/1,/Min): retry txn "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=3d786b5b key=/Table/51/1/600/2/1 rw=true pri=0.03766012 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679991.413378476,1 orig=1472679991.061787050,0 max=1472679991.061787050,0 wto=false rop=false
I160831 21:46:31.463953 18222 internal/client/db.go:498 failed batch: retry txn "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=3d786b5b key=/Table/51/1/600/2/1 rw=true pri=0.03766012 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679991.413378476,1 orig=1472679991.413378476,1 max=1472679991.111787050,0 wto=false rop=false
I160831 21:46:31.627643 18161 storage/replica_command.go:1357 [n1,s1,r7{/Table/51-/Max}] e03718b6 failed to push 3d786b5b: priority
I160831 21:46:31.628348 18161 storage/replica_command.go:1363 [n1,s1,r7{/Table/51-/Max}] failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=3d786b5b key=/Table/51/1/600/2/1 rw=true pri=0.03766012 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679991.413378476,1 orig=1472679991.061787050,0 max=1472679991.061787050,0 wto=false rop=false
I160831 21:46:31.630430 18285 kv/dist_sender.go:1036 application error: failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=3d786b5b key=/Table/51/1/600/2/1 rw=true pri=0.03766012 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679991.413378476,1 orig=1472679991.061787050,0 max=1472679991.061787050,0 wto=false rop=false
I160831 21:46:31.630718 18285 kv/dist_sender.go:771 reply error PushTxn [/Table/51/1/600/2/1,/Min): failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=3d786b5b key=/Table/51/1/600/2/1 rw=true pri=0.03766012 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679991.413378476,1 orig=1472679991.061787050,0 max=1472679991.061787050,0 wto=false rop=false
I160831 21:46:31.631004 18285 internal/client/db.go:498 failed batch: failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=3d786b5b key=/Table/51/1/600/2/1 rw=true pri=0.03766012 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679991.413378476,1 orig=1472679991.061787050,0 max=1472679991.061787050,0 wto=false rop=false
I160831 21:46:31.631239 18285 storage/intent_resolver.go:102 [n1,n1,s1] on Scan: failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=3d786b5b key=/Table/51/1/600/2/1 rw=true pri=0.03766012 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679991.413378476,1 orig=1472679991.061787050,0 max=1472679991.061787050,0 wto=false rop=false
W160831 21:46:31.631535 18285 storage/store.go:2096 [n1,n1,s1] conflicting intents on [/Table/51/1/773/2/1]: resolved? false
I160831 21:46:31.695527 18210 storage/replica_command.go:1357 [n1,s1,r7{/Table/51-/Max}] e03718b6 failed to push 3d786b5b: priority
I160831 21:46:31.696134 18210 storage/replica_command.go:1363 [n1,s1,r7{/Table/51-/Max}] failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=3d786b5b key=/Table/51/1/600/2/1 rw=true pri=0.03766012 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679991.413378476,1 orig=1472679991.061787050,0 max=1472679991.061787050,0 wto=false rop=false
I160831 21:46:31.698194 18285 kv/dist_sender.go:1036 application error: failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=3d786b5b key=/Table/51/1/600/2/1 rw=true pri=0.03766012 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679991.413378476,1 orig=1472679991.061787050,0 max=1472679991.061787050,0 wto=false rop=false
I160831 21:46:31.698525 18285 kv/dist_sender.go:771 reply error PushTxn [/Table/51/1/600/2/1,/Min): failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=3d786b5b key=/Table/51/1/600/2/1 rw=true pri=0.03766012 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679991.413378476,1 orig=1472679991.061787050,0 max=1472679991.061787050,0 wto=false rop=false
I160831 21:46:31.698749 18285 internal/client/db.go:498 failed batch: failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=3d786b5b key=/Table/51/1/600/2/1 rw=true pri=0.03766012 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679991.413378476,1 orig=1472679991.061787050,0 max=1472679991.061787050,0 wto=false rop=false
I160831 21:46:31.699011 18285 storage/intent_resolver.go:102 [n1,n1,s1] on Scan: failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=3d786b5b key=/Table/51/1/600/2/1 rw=true pri=0.03766012 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679991.413378476,1 orig=1472679991.061787050,0 max=1472679991.061787050,0 wto=false rop=false
W160831 21:46:31.699271 18285 storage/store.go:2096 [n1,n1,s1] conflicting intents on [/Table/51/1/773/2/1]: resolved? false
I160831 21:46:31.908510 18159 storage/replica_command.go:1357 [n1,s1,r7{/Table/51-/Max}] e03718b6 failed to push 3d786b5b: priority
I160831 21:46:31.909020 18159 storage/replica_command.go:1363 [n1,s1,r7{/Table/51-/Max}] failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=3d786b5b key=/Table/51/1/600/2/1 rw=true pri=0.03766012 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679991.413378476,1 orig=1472679991.061787050,0 max=1472679991.061787050,0 wto=false rop=false
I160831 21:46:31.910537 18285 kv/dist_sender.go:1036 application error: failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=3d786b5b key=/Table/51/1/600/2/1 rw=true pri=0.03766012 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679991.413378476,1 orig=1472679991.061787050,0 max=1472679991.061787050,0 wto=false rop=false
I160831 21:46:31.910764 18285 kv/dist_sender.go:771 reply error PushTxn [/Table/51/1/600/2/1,/Min): failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=3d786b5b key=/Table/51/1/600/2/1 rw=true pri=0.03766012 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679991.413378476,1 orig=1472679991.061787050,0 max=1472679991.061787050,0 wto=false rop=false
I160831 21:46:31.910951 18285 internal/client/db.go:498 failed batch: failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=3d786b5b key=/Table/51/1/600/2/1 rw=true pri=0.03766012 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679991.413378476,1 orig=1472679991.061787050,0 max=1472679991.061787050,0 wto=false rop=false
I160831 21:46:31.911133 18285 storage/intent_resolver.go:102 [n1,n1,s1] on Scan: failed to push "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=3d786b5b key=/Table/51/1/600/2/1 rw=true pri=0.03766012 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679991.413378476,1 orig=1472679991.061787050,0 max=1472679991.061787050,0 wto=false rop=false
W160831 21:46:31.911357 18285 storage/store.go:2096 [n1,n1,s1] conflicting intents on [/Table/51/1/773/2/1]: resolved? false
I160831 21:46:32.921474 18222 kv/dist_sender.go:1036 application error: retry txn "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=9aaf55be key=/Table/51/1/1200/2/1 rw=true pri=0.00274056 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679992.208770144,1 orig=1472679992.072585018,0 max=1472679992.072585018,0 wto=false rop=false
I160831 21:46:32.921728 18222 kv/dist_sender.go:771 reply error EndTransaction [/Table/51/1/1200/2/1,/Min): retry txn "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=9aaf55be key=/Table/51/1/1200/2/1 rw=true pri=0.00274056 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679992.208770144,1 orig=1472679992.072585018,0 max=1472679992.072585018,0 wto=false rop=false
I160831 21:46:32.932499 18222 internal/client/db.go:498 failed batch: retry txn "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=9aaf55be key=/Table/51/1/1200/2/1 rw=true pri=0.03021797 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679992.208770144,1 orig=1472679992.208770144,1 max=1472679992.122585018,0 wto=false rop=false
I160831 21:46:33.209554 18175 kv/txn_coord_sender.go:251 [n1] txn coordinator: 4.25 txn/sec, 100.00/56.44/0.00/0.00 %cmmt/cmmt1pc/abrt/abnd, 194ms/744ms/5.402s avg/σ/max duration, 0.1/0.3/1 avg/σ/max restarts (54 samples)
I160831 21:46:33.675644 18244 rpc/clock_offset.go:147 1 of 1 nodes are within the maximum offset of 50ms
I160831 21:46:35.628093 18222 sql/event_log.go:109 Event: "finish_schema_change", target: 51, info: {MutationID:2}
I160831 21:46:35.703454 18222 sql/lease.go:269 publish (count leases): descID=51 name=test version=6 count=1
I160831 21:46:35.763138 18222 sql/lease.go:269 publish (count leases): descID=51 name=test version=6 count=1
I160831 21:46:35.765596 18314 kv/dist_sender.go:1036 application error: retry txn "sql/lease.go:228 LeaseStore.Release" id=ce8905ff key=/Table/11/1/51/6/2016-08-31T21:50:45.103964Z/1 rw=true pri=0.09358898 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679995.748015092,1 orig=1472679995.719072762,0 max=1472679995.719072762,0 wto=false rop=false
I160831 21:46:35.765889 18314 kv/dist_sender.go:771 reply error EndTransaction [/Table/11/1/51/6/2016-08-31T21:50:45.103964Z/1,/Min): retry txn "sql/lease.go:228 LeaseStore.Release" id=ce8905ff key=/Table/11/1/51/6/2016-08-31T21:50:45.103964Z/1 rw=true pri=0.09358898 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679995.748015092,1 orig=1472679995.719072762,0 max=1472679995.719072762,0 wto=false rop=false
I160831 21:46:35.766498 18314 internal/client/db.go:498 failed batch: retry txn "sql/lease.go:228 LeaseStore.Release" id=ce8905ff key=/Table/11/1/51/6/2016-08-31T21:50:45.103964Z/1 rw=true pri=0.09358898 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679995.748015092,1 orig=1472679995.748015092,1 max=1472679995.769072762,0 wto=false rop=false
I160831 21:46:36.452625 18285 sql/event_log.go:109 [n1] Event: "alter_table", target: 51, info: {TableName:test Statement:ALTER TABLE t.test DROP pi User:root MutationID:3}
I160831 21:46:36.536912 18285 sql/lease.go:269 publish (count leases): descID=51 name=test version=7 count=1
I160831 21:46:36.685057 18285 sql/lease.go:269 publish (count leases): descID=51 name=test version=7 count=1
I160831 21:46:36.755126 18244 rpc/clock_offset.go:147 1 of 1 nodes are within the maximum offset of 50ms
I160831 21:46:36.758491 18319 kv/dist_sender.go:1036 application error: retry txn "sql/lease.go:228 LeaseStore.Release" id=6e39ecab key=/Table/11/1/51/7/2016-08-31T21:52:46.868957Z/1 rw=true pri=0.03961297 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679996.629004325,1 orig=1472679996.578265078,0 max=1472679996.578265078,0 wto=false rop=false
I160831 21:46:36.758834 18319 kv/dist_sender.go:771 reply error EndTransaction [/Table/11/1/51/7/2016-08-31T21:52:46.868957Z/1,/Min): retry txn "sql/lease.go:228 LeaseStore.Release" id=6e39ecab key=/Table/11/1/51/7/2016-08-31T21:52:46.868957Z/1 rw=true pri=0.03961297 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679996.629004325,1 orig=1472679996.578265078,0 max=1472679996.578265078,0 wto=false rop=false
I160831 21:46:36.759514 18319 internal/client/db.go:498 failed batch: retry txn "sql/lease.go:228 LeaseStore.Release" id=6e39ecab key=/Table/11/1/51/7/2016-08-31T21:52:46.868957Z/1 rw=true pri=0.03961297 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679996.629004325,1 orig=1472679996.629004325,1 max=1472679996.628265078,0 wto=false rop=false
I160831 21:46:36.864910 18160 storage/replica_command.go:1357 [n1,s1,r2{/Table/11-/Table/12}] 92fa8e16 failed to push 6e39ecab: priority
I160831 21:46:36.865614 18160 storage/replica_command.go:1363 [n1,s1,r2{/Table/11-/Table/12}] failed to push "sql/lease.go:228 LeaseStore.Release" id=6e39ecab key=/Table/11/1/51/7/2016-08-31T21:52:46.868957Z/1 rw=true pri=0.03961297 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679996.629004325,1 orig=1472679996.578265078,0 max=1472679996.578265078,0 wto=false rop=false
I160831 21:46:36.868281 18285 kv/dist_sender.go:1036 application error: failed to push "sql/lease.go:228 LeaseStore.Release" id=6e39ecab key=/Table/11/1/51/7/2016-08-31T21:52:46.868957Z/1 rw=true pri=0.03961297 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679996.629004325,1 orig=1472679996.578265078,0 max=1472679996.578265078,0 wto=false rop=false
I160831 21:46:36.869018 18285 kv/dist_sender.go:771 reply error PushTxn [/Table/11/1/51/7/2016-08-31T21:52:46.868957Z/1,/Min): failed to push "sql/lease.go:228 LeaseStore.Release" id=6e39ecab key=/Table/11/1/51/7/2016-08-31T21:52:46.868957Z/1 rw=true pri=0.03961297 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679996.629004325,1 orig=1472679996.578265078,0 max=1472679996.578265078,0 wto=false rop=false
I160831 21:46:36.869575 18285 internal/client/db.go:498 failed batch: failed to push "sql/lease.go:228 LeaseStore.Release" id=6e39ecab key=/Table/11/1/51/7/2016-08-31T21:52:46.868957Z/1 rw=true pri=0.03961297 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679996.629004325,1 orig=1472679996.578265078,0 max=1472679996.578265078,0 wto=false rop=false
I160831 21:46:36.870132 18285 storage/intent_resolver.go:102 [n1,s1] on Scan: failed to push "sql/lease.go:228 LeaseStore.Release" id=6e39ecab key=/Table/11/1/51/7/2016-08-31T21:52:46.868957Z/1 rw=true pri=0.03961297 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679996.629004325,1 orig=1472679996.578265078,0 max=1472679996.578265078,0 wto=false rop=false
W160831 21:46:36.873334 18285 storage/store.go:2096 [n1,s1] conflicting intents on [/Table/11/1/51/7/2016-08-31T21:52:46.868957Z/1/0]: resolved? false
I160831 21:46:36.992595 18285 sql/lease.go:269 publish (count leases): descID=51 name=test version=8 count=1
I160831 21:46:37.056705 18285 sql/lease.go:269 publish (count leases): descID=51 name=test version=8 count=1
I160831 21:46:37.119790 18331 kv/dist_sender.go:1036 application error: retry txn "sql/lease.go:228 LeaseStore.Release" id=027cf2fe key=/Table/11/1/51/8/2016-08-31T21:52:27.436897Z/1 rw=true pri=0.00159112 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679997.047556061,1 orig=1472679997.038666878,0 max=1472679997.038666878,0 wto=false rop=false
I160831 21:46:37.121712 18159 storage/replica_command.go:1357 [n1,s1,r2{/Table/11-/Table/12}] 528c6f20 pushed 027cf2fe: priority
I160831 21:46:37.124593 18331 kv/dist_sender.go:771 reply error EndTransaction [/Table/11/1/51/8/2016-08-31T21:52:27.436897Z/1,/Min): retry txn "sql/lease.go:228 LeaseStore.Release" id=027cf2fe key=/Table/11/1/51/8/2016-08-31T21:52:27.436897Z/1 rw=true pri=0.00159112 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679997.047556061,1 orig=1472679997.038666878,0 max=1472679997.038666878,0 wto=false rop=false
I160831 21:46:37.128653 18331 internal/client/db.go:498 failed batch: retry txn "sql/lease.go:228 LeaseStore.Release" id=027cf2fe key=/Table/11/1/51/8/2016-08-31T21:52:27.436897Z/1 rw=true pri=0.00614263 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679997.047556061,1 orig=1472679997.047556061,1 max=1472679997.088666878,0 wto=false rop=false
W160831 21:46:37.150472 18285 storage/store.go:2091 [n1,s1] conflicting intents on [/Table/11/1/51/8/2016-08-31T21:52:27.436897Z/1/0]: resolved? false
I160831 21:46:37.152152 18285 sql/lease.go:269 publish (count leases): descID=51 name=test version=8 count=1
I160831 21:46:37.163373 18331 kv/dist_sender.go:1036 application error: retry txn "sql/lease.go:228 LeaseStore.Release" id=027cf2fe key=/Table/11/1/51/8/2016-08-31T21:52:27.436897Z/1 rw=true pri=0.01542866 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679997.116920136,1 orig=1472679997.038666878,0 max=1472679997.038666878,0 wto=false rop=false
I160831 21:46:37.163732 18331 kv/dist_sender.go:771 reply error EndTransaction [/Table/11/1/51/8/2016-08-31T21:52:27.436897Z/1,/Min): retry txn "sql/lease.go:228 LeaseStore.Release" id=027cf2fe key=/Table/11/1/51/8/2016-08-31T21:52:27.436897Z/1 rw=true pri=0.01542866 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679997.116920136,1 orig=1472679997.038666878,0 max=1472679997.038666878,0 wto=false rop=false
I160831 21:46:37.164413 18331 internal/client/db.go:498 failed batch: retry txn "sql/lease.go:228 LeaseStore.Release" id=027cf2fe key=/Table/11/1/51/8/2016-08-31T21:52:27.436897Z/1 rw=true pri=0.01542866 iso=SERIALIZABLE stat=PENDING epo=2 ts=1472679997.116920136,1 orig=1472679997.116920136,1 max=1472679997.088666878,0 wto=false rop=false
I160831 21:46:37.940985 18176 storage/raft_transport.go:217 stats:
qlen qmax qdropped client-sent client-recv server-sent server-recv
I160831 21:46:37.950160 18285 kv/dist_sender.go:1036 application error: retry txn "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=a7978e7e key=/Table/51/1/0/1/1 rw=true pri=0.02041832 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679997.617031984,1 orig=1472679997.268494302,0 max=1472679997.268494302,0 wto=false rop=false
I160831 21:46:37.950565 18285 kv/dist_sender.go:771 reply error EndTransaction [/Table/51/1/0/1/1,/Min): retry txn "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=a7978e7e key=/Table/51/1/0/1/1 rw=true pri=0.02041832 iso=SERIALIZABLE stat=PENDING epo=0 ts=1472679997.617031984,1 orig=1472679997.268494302,0 max=1472679997.268494302,0 wto=false rop=false
I160831 21:46:37.951227 18285 internal/client/db.go:498 failed batch: retry txn "sql/backfill.go:372 (*SchemaChanger).truncateAndBackfillColumnsChunk" id=a7978e7e key=/Table/51/1/0/1/1 rw=true pri=0.02041832 iso=SERIALIZABLE stat=PENDING epo=1 ts=1472679997.617031984,1 orig=1472679997.617031984,1 max=1472679997.318494302,0 wto=false rop=false
I160831 21:46:38.016916 18234 server/status/runtime.go:207 runtime stats: 650 MiB RSS, 63 goroutines, 38 MiB/40 MiB/94 MiB GO alloc/idle/total, 29 MiB/39 MiB CGO alloc/total, 10889.91cgo/sec, 1.30/0.02 %(u/s)time, 0.00 %gc (5x)
panic: test timed out after 8m0s
goroutine 18334 [running]:
panic(0x1ae1fc0, 0xc4269f8910)
/usr/local/go/src/runtime/panic.go:500 +0x1ae
testing.startAlarm.func1()
/usr/local/go/src/testing/testing.go:918 +0x14e
created by time.goFunc
/usr/local/go/src/time/sleep.go:154 +0x78
goroutine 1 [chan receive]:
testing.(*T).Run(0xc420274cc0, 0x1c973b3, 0x14, 0x1f910d8, 0x933d01)
/usr/local/go/src/testing/testing.go:647 +0x56e
testing.RunTests.func1(0xc420274cc0)
/usr/local/go/src/testing/testing.go:793 +0xba
testing.tRunner(0xc420274cc0, 0xc420039d78)
/usr/local/go/src/testing/testing.go:610 +0xca
testing.RunTests(0x1f92618, 0x288ab00, 0x72, 0x72, 0x0)
/usr/local/go/src/testing/testing.go:799 +0x4bb
testing.(*M).Run(0xc420039f00, 0x35a6d40)
/usr/local/go/src/testing/testing.go:743 +0x130
github.com/cockroachdb/cockroach/sql_test.TestMain(0xc420039f00)
/go/src/github.com/cockroachdb/cockroach/sql/main_test.go:199 +0x13a
main.main()
github.com/cockroachdb/cockroach/sql/_test/_testmain.go:656 +0x1b6
goroutine 17 [syscall, 8 minutes, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1
goroutine 22 [chan receive]:
github.com/cockroachdb/cockroach/util/log.(*loggingT).flushDaemon(0x2beaf40)
/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:996 +0x85
created by github.com/cockroachdb/cockroach/util/log.init.1
/go/src/github.com/cockroachdb/cockroach/util/log/clog.go:577 +0xc3
goroutine 18180 [chan receive]:
github.com/cockroachdb/cockroach/util/netutil.MakeServer.func2()
/go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:100 +0x8e
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc426ad8940)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18244 [select]:
github.com/cockroachdb/cockroach/rpc.(*Context).runHeartbeat(0xc42717fd00, 0xc4208af800, 0xc4272004d0, 0xf, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/rpc/context.go:286 +0x813
github.com/cockroachdb/cockroach/rpc.(*Context).GRPCDial.func1.1.1()
/go/src/github.com/cockroachdb/cockroach/rpc/context.go:209 +0xb3
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc420ae4450)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18208 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:407 +0x395
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc420bb2200)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18174 [select]:
github.com/cockroachdb/cockroach/storage.(*StorePool).start.func1()
/go/src/github.com/cockroachdb/cockroach/storage/store_pool.go:348 +0x519
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc426ad84a0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18226 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:407 +0x395
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc420bb2260)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18240 [select]:
google.golang.org/grpc.(*addrConn).transportMonitor(0xc42078ba40)
/go/src/google.golang.org/grpc/clientconn.go:720 +0x7d7
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc42078ba40)
/go/src/google.golang.org/grpc/clientconn.go:479 +0x26e
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
/go/src/google.golang.org/grpc/clientconn.go:480 +0x647
goroutine 18189 [select]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).bootstrap.func1()
/go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:920 +0x5a6
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc4272005a0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18186 [chan receive]:
github.com/cockroachdb/cmux.(*muxListener).Accept(0xc426ad8a40, 0x1f92368, 0xc4200abea0, 0x28ac780, 0xc426d268f0)
<autogenerated>:9 +0xa4
google.golang.org/grpc.(*Server).Serve(0xc4200abea0, 0x28a4840, 0xc426ad8a40, 0x0, 0x0)
/go/src/google.golang.org/grpc/server.go:348 +0x1b8
github.com/cockroachdb/cockroach/server.(*Server).Start.func7()
/go/src/github.com/cockroachdb/cockroach/server/server.go:414 +0x8d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc426ad8be0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18213 [select]:
google.golang.org/grpc/transport.(*http2Server).controller(0xc4204a7ef0)
/go/src/google.golang.org/grpc/transport/http2_server.go:673 +0xa56
created by google.golang.org/grpc/transport.newHTTP2Server
/go/src/google.golang.org/grpc/transport/http2_server.go:139 +0xa91
goroutine 18187 [chan receive]:
github.com/cockroachdb/cmux.(*muxListener).Accept(0xc426ad8a00, 0x1f91710, 0xc420b46000, 0xc42007e230, 0x28ac780)
<autogenerated>:9 +0xa4
github.com/cockroachdb/cockroach/util/netutil.(*Server).ServeWith(0xc42007e230, 0xc420b46000, 0x28a4840, 0xc426ad8a00, 0xc42722c030, 0xdcbacd, 0xc420dc5f88)
/go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:117 +0x71
github.com/cockroachdb/cockroach/server.(*Server).Start.func8()
/go/src/github.com/cockroachdb/cockroach/server/server.go:422 +0x112
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc427039e90)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18246 [select]:
github.com/cockroachdb/cockroach/storage.(*idAllocator).start.func1()
/go/src/github.com/cockroachdb/cockroach/storage/id_alloc.go:124 +0xacc
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc4270b1900)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18214 [IO wait]:
net.runtime_pollWait(0x7f8cd9331e98, 0x72, 0x5f17c3)
/usr/local/go/src/runtime/netpoll.go:160 +0x5e
net.(*pollDesc).wait(0xc4208765a0, 0x72, 0x5ee078, 0x1b566a0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x5b
net.(*pollDesc).waitRead(0xc4208765a0, 0x2899b40, 0xc42007c0b0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x42
net.(*netFD).Read(0xc420876540, 0xc423e07400, 0x400, 0x400, 0x0, 0x2899b40, 0xc42007c0b0)
/usr/local/go/src/net/fd_unix.go:243 +0x1d9
net.(*conn).Read(0xc42007e428, 0xc423e07400, 0x400, 0x400, 0xc400000102, 0x638880, 0xc420028800)
/usr/local/go/src/net/net.go:173 +0x97
net.(*TCPConn).Read(0xc42007e428, 0xc423e07400, 0x400, 0x400, 0xc4270be260, 0xc4214cd540, 0xc4214cd540)
<autogenerated>:69 +0x73
crypto/tls.(*block).readFromUntil(0xc420ae5a10, 0x7f8cd92f3d30, 0xc42007e428, 0x5, 0xc42007e428, 0xc426a0554b)
/usr/local/go/src/crypto/tls/conn.go:472 +0x10d
crypto/tls.(*Conn).readRecord(0xc4270be000, 0x1f93017, 0xc4270be100, 0xc4214cd9a8)
/usr/local/go/src/crypto/tls/conn.go:574 +0x13d
crypto/tls.(*Conn).Read(0xc4270be000, 0xc420832000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1109 +0x186
bufio.(*Reader).fill(0xc4214e9b60)
/usr/local/go/src/bufio/bufio.go:97 +0x197
bufio.(*Reader).Read(0xc4214e9b60, 0xc420447178, 0x9, 0x9, 0x0, 0x0, 0xc4214cdb48)
/usr/local/go/src/bufio/bufio.go:209 +0x6ac
io.ReadAtLeast(0x2891e00, 0xc4214e9b60, 0xc420447178, 0x9, 0x9, 0x9, 0xc420cb7e98, 0xc42015b040, 0x0)
/usr/local/go/src/io/io.go:307 +0xb2
io.ReadFull(0x2891e00, 0xc4214e9b60, 0xc420447178, 0x9, 0x9, 0x1, 0xc423c4dcc0, 0x0)
/usr/local/go/src/io/io.go:325 +0x73
golang.org/x/net/http2.readFrameHeader(0xc420447178, 0x9, 0x9, 0x2891e00, 0xc4214e9b60, 0x0, 0x0, 0xc4254374a0, 0x0)
/go/src/golang.org/x/net/http2/frame.go:237 +0xa3
golang.org/x/net/http2.(*Framer).ReadFrame(0xc420447140, 0x0, 0x0, 0x0, 0x0)
/go/src/golang.org/x/net/http2/frame.go:464 +0x105
google.golang.org/grpc/transport.(*framer).readFrame(0xc42710fdd0, 0xc42037c150, 0xc42037c150, 0xc4214cdec0, 0x1)
/go/src/google.golang.org/grpc/transport/http_util.go:505 +0x53
google.golang.org/grpc/transport.(*http2Client).reader(0xc42043fc00)
/go/src/google.golang.org/grpc/transport/http2_client.go:921 +0x154
created by google.golang.org/grpc/transport.newHTTP2Client
/go/src/google.golang.org/grpc/transport/http2_client.go:203 +0xc1a
goroutine 18234 [select]:
github.com/cockroachdb/cockroach/server.(*Server).startSampleEnvironment.func1()
/go/src/github.com/cockroachdb/cockroach/server/server.go:626 +0x1b4
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc420bb24a0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18191 [select]:
github.com/cockroachdb/cockroach/storage.(*bookie).start.func1()
/go/src/github.com/cockroachdb/cockroach/storage/reservation.go:274 +0x4c8
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc426ad8ea0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18232 [select]:
github.com/cockroachdb/cockroach/server.(*Node).startGossip.func1()
/go/src/github.com/cockroachdb/cockroach/server/node.go:592 +0x545
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc4212e16b0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18220 [chan receive]:
database/sql.(*DB).connectionOpener(0xc426b49130)
/usr/local/go/src/database/sql/sql.go:730 +0x6e
created by database/sql.Open
/usr/local/go/src/database/sql/sql.go:493 +0x2af
goroutine 18157 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1()
/go/src/github.com/cockroachdb/cockroach/storage/store.go:2780 +0x184a
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc426a12070)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18188 [chan receive]:
github.com/cockroachdb/cockroach/gossip.(*server).start.func3()
/go/src/github.com/cockroachdb/cockroach/gossip/server.go:370 +0x94
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc426ad8c60)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18212 [IO wait]:
net.runtime_pollWait(0x7f8cd93326d8, 0x72, 0x5f17c3)
/usr/local/go/src/runtime/netpoll.go:160 +0x5e
net.(*pollDesc).wait(0xc421149480, 0x72, 0x5ee078, 0x1b566a0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x5b
net.(*pollDesc).waitRead(0xc421149480, 0x2899b40, 0xc42007c0b0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x42
net.(*netFD).Read(0xc421149420, 0xc4205e6c00, 0x400, 0x400, 0x0, 0x2899b40, 0xc42007c0b0)
/usr/local/go/src/net/fd_unix.go:243 +0x1d9
net.(*conn).Read(0xc420026398, 0xc4205e6c00, 0x400, 0x400, 0x6c, 0x0, 0x1f92e20)
/usr/local/go/src/net/net.go:173 +0x97
net.(*TCPConn).Read(0xc420026398, 0xc4205e6c00, 0x400, 0x400, 0x1, 0x2892380, 0xc42007c020)
<autogenerated>:69 +0x73
github.com/cockroachdb/cmux.(*MuxConn).Read(0xc426d268f0, 0xc4205e6c00, 0x400, 0x400, 0xc42131c5e0, 0xc4269bd318, 0xc4269bd318)
/go/src/github.com/cockroachdb/cmux/cmux.go:218 +0x183
crypto/tls.(*block).readFromUntil(0xc42073fb30, 0x7f8cd9333f80, 0xc426d268f0, 0x5, 0xc426d268f0, 0xc4205a4e31)
/usr/local/go/src/crypto/tls/conn.go:472 +0x10d
crypto/tls.(*Conn).readRecord(0xc42131c380, 0x1f93017, 0xc42131c480, 0x5f3b1f)
/usr/local/go/src/crypto/tls/conn.go:574 +0x13d
crypto/tls.(*Conn).Read(0xc42131c380, 0xc420822000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1109 +0x186
bufio.(*Reader).fill(0xc4214e9860)
/usr/local/go/src/bufio/bufio.go:97 +0x197
bufio.(*Reader).Read(0xc4214e9860, 0xc4204464b8, 0x9, 0x9, 0x10, 0x10, 0xc423b78da0)
/usr/local/go/src/bufio/bufio.go:209 +0x6ac
io.ReadAtLeast(0x2891e00, 0xc4214e9860, 0xc4204464b8, 0x9, 0x9, 0x9, 0x61c44a, 0xc420bc6b60, 0xc4204f3cb8)
/usr/local/go/src/io/io.go:307 +0xb2
io.ReadFull(0x2891e00, 0xc4214e9860, 0xc4204464b8, 0x9, 0x9, 0x0, 0xc421168730, 0xc4269bda00)
/usr/local/go/src/io/io.go:325 +0x73
golang.org/x/net/http2.readFrameHeader(0xc4204464b8, 0x9, 0x9, 0x2891e00, 0xc4214e9860, 0x0, 0xc400000000, 0xc4204f3cb8, 0x2895ec0)
/go/src/golang.org/x/net/http2/frame.go:237 +0xa3
golang.org/x/net/http2.(*Framer).ReadFrame(0xc420446480, 0x0, 0x2892380, 0xc42007c020, 0xc4204f3dd0)
/go/src/golang.org/x/net/http2/frame.go:464 +0x105
google.golang.org/grpc/transport.(*framer).readFrame(0xc42710fc20, 0xc4204f3da0, 0xc4204f3da0, 0xc4269bdcc0, 0x1)
/go/src/google.golang.org/grpc/transport/http_util.go:505 +0x53
google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc4204a7ef0, 0xc42710fd10)
/go/src/google.golang.org/grpc/transport/http2_server.go:260 +0x30a
google.golang.org/grpc.(*Server).serveStreams(0xc4200abea0, 0x28ac480, 0xc4204a7ef0)
/go/src/google.golang.org/grpc/server.go:424 +0x1d6
google.golang.org/grpc.(*Server).serveNewHTTP2Transport(0xc4200abea0, 0x28ac720, 0xc42131c380, 0x2898d80, 0xc420c54160)
/go/src/google.golang.org/grpc/server.go:411 +0x4f1
google.golang.org/grpc.(*Server).handleRawConn(0xc4200abea0, 0x28ac780, 0xc426d268f0)
/go/src/google.golang.org/grpc/server.go:388 +0x5c3
created by google.golang.org/grpc.(*Server).Serve
/go/src/google.golang.org/grpc/server.go:357 +0x20a
goroutine 18304 [IO wait]:
net.runtime_pollWait(0x7f8cc7dbb1a0, 0x72, 0x5f17c3)
/usr/local/go/src/runtime/netpoll.go:160 +0x5e
net.(*pollDesc).wait(0xc4262d5aa0, 0x72, 0x5ee078, 0x1b566a0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x5b
net.(*pollDesc).waitRead(0xc4262d5aa0, 0x2899b40, 0xc42007c0b0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x42
net.(*netFD).Read(0xc4262d5a40, 0xc420b5b800, 0x400, 0x400, 0x0, 0x2899b40, 0xc42007c0b0)
/usr/local/go/src/net/fd_unix.go:243 +0x1d9
net.(*conn).Read(0xc4200271c8, 0xc420b5b800, 0x400, 0x400, 0x102, 0x0, 0x0)
/usr/local/go/src/net/net.go:173 +0x97
net.(*TCPConn).Read(0xc4200271c8, 0xc420b5b800, 0x400, 0x400, 0xc4210eb3c0, 0x12f1e29, 0xc4210eb3c0)
<autogenerated>:69 +0x73
crypto/tls.(*block).readFromUntil(0xc420d8a6c0, 0x7f8cd92f3d30, 0xc4200271c8, 0x5, 0xc4200271c8, 0x0)
/usr/local/go/src/crypto/tls/conn.go:472 +0x10d
crypto/tls.(*Conn).readRecord(0xc42131ca80, 0x1f93017, 0xc42131cb80, 0xc420d2fc00)
/usr/local/go/src/crypto/tls/conn.go:574 +0x13d
crypto/tls.(*Conn).Read(0xc42131ca80, 0xc42144e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1109 +0x186
bufio.(*Reader).fill(0xc423e3ec00)
/usr/local/go/src/bufio/bufio.go:97 +0x197
bufio.(*Reader).Read(0xc423e3ec00, 0xc420ccb1a0, 0x5, 0x200, 0xc4210eb9b8, 0x61c44a, 0xc420bc6d00)
/usr/local/go/src/bufio/bufio.go:209 +0x6ac
io.ReadAtLeast(0x2891e00, 0xc423e3ec00, 0xc420ccb1a0, 0x5, 0x200, 0x5, 0x1f930a8, 0x4, 0x61c0c7)
/usr/local/go/src/io/io.go:307 +0xb2
io.ReadFull(0x2891e00, 0xc423e3ec00, 0xc420ccb1a0, 0x5, 0x200, 0x1b0ce20, 0x0, 0x30000628c0d)
/usr/local/go/src/io/io.go:325 +0x73
github.com/cockroachdb/pq.(*conn).recvMessage(0xc420ccb180, 0xc4200a72e0, 0x0, 0x1ae7620, 0x1)
/go/src/github.com/cockroachdb/pq/conn.go:953 +0xde
github.com/cockroachdb/pq.(*conn).recv1Buf(0xc420ccb180, 0xc4200a72e0, 0xc420ccb1a0)
/go/src/github.com/cockroachdb/pq/conn.go:1003 +0x47
github.com/cockroachdb/pq.(*conn).recv1(0xc420ccb180, 0xc4210ebc50, 0x1fb)
/go/src/github.com/cockroachdb/pq/conn.go:1024 +0xb0
github.com/cockroachdb/pq.(*conn).simpleExec(0xc420ccb180, 0x1c9eeea, 0x1a, 0xc4210ebd20, 0xc420dc74e0, 0x5ee799, 0x1b48ce0, 0x1c75440, 0xc420dc7401)
/go/src/github.com/cockroachdb/pq/conn.go:614 +0x2c0
github.com/cockroachdb/pq.(*conn).Exec(0xc420ccb180, 0x1c9eeea, 0x1a, 0x35a71a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/pq/conn.go:864 +0x158
database/sql.(*DB).exec(0xc426b49130, 0x1c9eeea, 0x1a, 0x0, 0x0, 0x0, 0x933d01, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/database/sql/sql.go:1036 +0x553
database/sql.(*DB).Exec(0xc426b49130, 0x1c9eeea, 0x1a, 0x0, 0x0, 0x0, 0xc420dc7748, 0x1460757, 0xc420119c30, 0x7f8cd9331278)
/usr/local/go/src/database/sql/sql.go:1010 +0xb0
github.com/cockroachdb/cockroach/sql_test.runSchemaChangeWithOperations.func1(0xc426b49130, 0x1c9eeea, 0x1a, 0xc4203a8180, 0xc4209bf930)
/go/src/github.com/cockroachdb/cockroach/sql/schema_changer_test.go:464 +0xb1
created by github.com/cockroachdb/cockroach/sql_test.runSchemaChangeWithOperations
/go/src/github.com/cockroachdb/cockroach/sql/schema_changer_test.go:469 +0x175
goroutine 18198 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func1()
/go/src/github.com/cockroachdb/cockroach/storage/store.go:1012 +0x2f9
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc4271c7c60)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18175 [runnable]:
github.com/codahale/hdrhistogram.(*iterator).next(0xc420487b50, 0x1)
/go/src/github.com/codahale/hdrhistogram/hdr.go:464
github.com/codahale/hdrhistogram.(*Histogram).Mean(0xc4239c2780, 0x0)
/go/src/github.com/codahale/hdrhistogram/hdr.go:160 +0xf6
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).startStats(0xc420119c30)
/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:236 +0x5c6
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).(github.com/cockroachdb/cockroach/kv.startStats)-fm()
/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:193 +0x42
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc427231bf0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18222 [select]:
github.com/cockroachdb/cockroach/storage.(*Replica).beginCmds(0xc420886600, 0x7f8cd9331278, 0xc4267cbad0, 0xc420c0a540, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/storage/replica.go:1095 +0x97e
github.com/cockroachdb/cockroach/storage.(*Replica).addWriteCmd(0xc420886600, 0x7f8cd9331278, 0xc4267cbad0, 0x147002a7d389e020, 0x0, 0x100000001, 0x1, 0x7, 0x0, 0xc4209d4480, ...)
/go/src/github.com/cockroachdb/cockroach/storage/replica.go:1411 +0x6b8
github.com/cockroachdb/cockroach/storage.(*Replica).Send(0xc420886600, 0x7f8cd9331278, 0xc4267cbad0, 0x147002a7d389e020, 0x0, 0x100000001, 0x1, 0x7, 0x0, 0xc4209d4480, ...)
/go/src/github.com/cockroachdb/cockroach/storage/replica.go:1001 +0x2a5
github.com/cockroachdb/cockroach/storage.(*Store).Send(0xc42062ea00, 0x7f8cd9331278, 0xc4267cb980, 0x147002a7d389e020, 0x0, 0x100000001, 0x1, 0x7, 0x0, 0xc4209d4480, ...)
/go/src/github.com/cockroachdb/cockroach/storage/store.go:2051 +0x8c3
github.com/cockroachdb/cockroach/storage.(*Stores).Send(0xc4268a0b00, 0x7f8cd9331278, 0xc4267cb8c0, 0x0, 0x0, 0x100000001, 0x1, 0x7, 0x0, 0xc4209d4480, ...)
/go/src/github.com/cockroachdb/cockroach/storage/stores.go:182 +0x24b
github.com/cockroachdb/cockroach/server.(*Node).Batch.func3()
/go/src/github.com/cockroachdb/cockroach/server/node.go:824 +0x3d4
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunTask(0xc420b46000, 0xc421228bd8, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:218 +0x10e
github.com/cockroachdb/cockroach/server.(*Node).Batch(0xc420bce000, 0x7f8cd92f3cc8, 0xc4205df260, 0xc420d32d48, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/server/node.go:836 +0x2cf
github.com/cockroachdb/cockroach/kv.(*grpcTransport).SendNext(0xc4203d9db0, 0xc4205df1a0)
/go/src/github.com/cockroachdb/cockroach/kv/transport.go:187 +0x3ec
github.com/cockroachdb/cockroach/kv.(*DistSender).sendToReplicas(0xc420db3c20, 0x7f8cd9331278, 0xc4267cb4d0, 0x2540be400, 0xb2d05e00, 0x0, 0x7, 0xc426a7a380, 0x1, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:1010 +0x203
github.com/cockroachdb/cockroach/kv.(*DistSender).sendRPC(0xc420db3c20, 0x7f8cd9331278, 0xc4267cb4d0, 0x7, 0xc426a7a380, 0x1, 0x1, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:379 +0x27c
github.com/cockroachdb/cockroach/kv.(*DistSender).sendSingleRange(0xc420db3c20, 0x7f8cd9331278, 0xc4267cb4d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc420a3fc80, ...)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:495 +0x1d9
github.com/cockroachdb/cockroach/kv.(*DistSender).sendChunk.func2(0xc421229bf0, 0xc421229a88, 0xc42122a0b0, 0xc420db3c20, 0xc42122a0a0, 0x8, 0xc423b67f38)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:763 +0x2eb
github.com/cockroachdb/cockroach/kv.(*DistSender).sendChunk(0xc420db3c20, 0x7f8cd9331278, 0xc4267cb4d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc420a3fc80, ...)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:764 +0x573
github.com/cockroachdb/cockroach/kv.(*DistSender).Send(0xc420db3c20, 0x7f8cd9331278, 0xc4267cb4d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc420a3fbc0, ...)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:606 +0x1a1
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).Send(0xc420119c30, 0x7f8cd9331278, 0xc4267cb4d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42049f7b8, ...)
/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:391 +0x68a
github.com/cockroachdb/cockroach/internal/client.(*txnSender).Send(0xc42049f760, 0x7f8cd9331238, 0xc42007c6f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42049f7b8, ...)
/go/src/github.com/cockroachdb/cockroach/internal/client/txn.go:52 +0x121
github.com/cockroachdb/cockroach/internal/client.(*DB).send(0xc42049f760, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/internal/client/db.go:495 +0x294
github.com/cockroachdb/cockroach/internal/client.(*Txn).send(0xc42049f760, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/internal/client/txn.go:681 +0x63d
github.com/cockroachdb/cockroach/internal/client.(*Txn).(github.com/cockroachdb/cockroach/internal/client.send)-fm(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/internal/client/txn.go:359 +0x7d
github.com/cockroachdb/cockroach/internal/client.sendAndFill(0xc42122b540, 0xc421236000, 0x0, 0xc4205df0e0)
/go/src/github.com/cockroachdb/cockroach/internal/client/db.go:422 +0x1ac
github.com/cockroachdb/cockroach/internal/client.(*Txn).Run(0xc42049f760, 0xc421236000, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/internal/client/txn.go:359 +0xfe
github.com/cockroachdb/cockroach/internal/client.(*Txn).CommitInBatch(0xc42049f760, 0xc421236000, 0x0, 0xc4200cc048)
/go/src/github.com/cockroachdb/cockroach/internal/client/txn.go:400 +0x27c
github.com/cockroachdb/cockroach/sql.(*tableUpdater).finalize(0xc4200cc090, 0x7f8cd9331338, 0xc4215b0d80, 0x0, 0xc420aab700)
/go/src/github.com/cockroachdb/cockroach/sql/tablewriter.go:161 +0x9e
github.com/cockroachdb/cockroach/sql.(*updateNode).Next(0xc4200cc000, 0x28aeb01, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/sql/update.go:292 +0xd2d
github.com/cockroachdb/cockroach/sql.countRowsAffected(0x28aebc0, 0xc4200cc000, 0x6, 0xc42122ba01, 0x28aebc0)
/go/src/github.com/cockroachdb/cockroach/sql/executor.go:655 +0xda
github.com/cockroachdb/cockroach/sql.(*Executor).execStmt(0xc4213a0000, 0x28a6800, 0xc4203d9630, 0xc4202da0a0, 0xc4203d9601, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/sql/executor.go:1148 +0x937
github.com/cockroachdb/cockroach/sql.(*Executor).execStmtInOpenTxn(0xc4213a0000, 0x28a6800, 0xc4203d9630, 0xc4202da0a0, 0x101, 0xc4202da028, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/sql/executor.go:1019 +0x33b
github.com/cockroachdb/cockroach/sql.(*Executor).execStmtsInCurrentTxn(0xc4213a0000, 0xc423b67600, 0x1, 0x1, 0xc4202da0a0, 0xc4202da028, 0x101, 0xc42122cd28, 0x62c99a, 0x20f7fe4, ...)
/go/src/github.com/cockroachdb/cockroach/sql/executor.go:761 +0x5a0
github.com/cockroachdb/cockroach/sql.runTxnAttempt(0xc4213a0000, 0xc4202da0a0, 0x1, 0xc4202da028, 0xc423b67610, 0xc423b67600, 0x1, 0x1, 0xcfbbfd, 0xecf59473d, ...)
/go/src/github.com/cockroachdb/cockroach/sql/executor.go:682 +0x202
github.com/cockroachdb/cockroach/sql.(*Executor).execRequest.func2(0xc42049f760, 0xc423b67610, 0x4000000000000000, 0x0)
/go/src/github.com/cockroachdb/cockroach/sql/executor.go:566 +0x19a
github.com/cockroachdb/cockroach/internal/client.(*Txn).Exec(0xc42049f760, 0xc4205d0101, 0xc4268a0540, 0xc4205deae0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/internal/client/txn.go:559 +0x229
github.com/cockroachdb/cockroach/sql.(*Executor).execRequest(0xc4213a0000, 0xc4202da000, 0xc4262d220f, 0x28, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/sql/executor.go:571 +0x619
github.com/cockroachdb/cockroach/sql.(*Executor).ExecuteStatements(0xc4213a0000, 0xc4202da000, 0xc4262d220f, 0x28, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/sql/executor.go:416 +0x197
github.com/cockroachdb/cockroach/sql/pgwire.(*v3Conn).executeStatements(0xc4203fe780, 0x7f8cd9331338, 0xc4215b0d80, 0xc4262d220f, 0x28, 0x0, 0x0, 0x0, 0x0, 0x10000c421529101, ...)
/go/src/github.com/cockroachdb/cockroach/sql/pgwire/v3.go:648 +0xba
github.com/cockroachdb/cockroach/sql/pgwire.(*v3Conn).handleSimpleQuery(0xc4203fe780, 0x7f8cd9331338, 0xc4215b0d80, 0xc4203fe7a8, 0x2d, 0x0)
/go/src/github.com/cockroachdb/cockroach/sql/pgwire/v3.go:327 +0x10a
github.com/cockroachdb/cockroach/sql/pgwire.(*v3Conn).serve(0xc4203fe780, 0xc42720a440, 0xc42720a440, 0x0)
/go/src/github.com/cockroachdb/cockroach/sql/pgwire/v3.go:279 +0x835
github.com/cockroachdb/cockroach/sql/pgwire.(*Server).ServeConn(0xc4268a0ac0, 0x28ac720, 0xc420be0e00, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/sql/pgwire/server.go:240 +0x98a
github.com/cockroachdb/cockroach/server.(*Server).Start.func8.1(0x28ac780, 0xc426b493f0)
/go/src/github.com/cockroachdb/cockroach/server/server.go:419 +0x89
github.com/cockroachdb/cockroach/util/netutil.(*Server).ServeWith.func1(0xc420b46000, 0xc42007e230, 0x28ac780, 0xc426b493f0, 0xc42722c030)
/go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:138 +0xc9
created by github.com/cockroachdb/cockroach/util/netutil.(*Server).ServeWith
/go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:140 +0x2f9
goroutine 18185 [chan receive]:
github.com/cockroachdb/cockroach/server.(*Server).Start.func6()
/go/src/github.com/cockroachdb/cockroach/server/server.go:407 +0xa6
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc426ad8bc0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18159 [chan receive]:
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1.1(0xc42133b8c0)
/go/src/github.com/cockroachdb/cockroach/storage/store.go:2692 +0x58
created by github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1
/go/src/github.com/cockroachdb/cockroach/storage/store.go:2695 +0xd8
goroutine 18181 [chan receive]:
github.com/cockroachdb/cockroach/server.(*Server).Start.func2()
/go/src/github.com/cockroachdb/cockroach/server/server.go:380 +0xae
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc426ad8ae0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18190 [select]:
github.com/cockroachdb/cockroach/gossip.(*Gossip).manage.func1()
/go/src/github.com/cockroachdb/cockroach/gossip/gossip.go:948 +0x416
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc4272005b0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18184 [chan receive]:
github.com/cockroachdb/cmux.(*muxListener).Accept(0xc426ad8b20, 0xc420df20c0, 0xc427039dd0, 0xc427039dd0, 0x289bf40)
<autogenerated>:9 +0xa4
crypto/tls.(*listener).Accept(0xc426ad8b80, 0xc420df2060, 0x1b257a0, 0x28784b0, 0x1bb01a0)
/usr/local/go/src/crypto/tls/tls.go:52 +0x56
net/http.(*Server).Serve(0xc426b37700, 0x289e080, 0xc426ad8b80, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2273 +0x249
github.com/cockroachdb/cockroach/server.(*Server).Start.func5()
/go/src/github.com/cockroachdb/cockroach/server/server.go:403 +0x8f
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc426ad8ba0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18182 [IO wait]:
net.runtime_pollWait(0x7f8cd9332318, 0x72, 0x5f17c3)
/usr/local/go/src/runtime/netpoll.go:160 +0x5e
net.(*pollDesc).wait(0xc42724fb10, 0x72, 0x5ee078, 0x1b566a0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x5b
net.(*pollDesc).waitRead(0xc42724fb10, 0x2899b40, 0xc42007c0b0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x42
net.(*netFD).accept(0xc42724fab0, 0x0, 0x2896d40, 0xc42147e060)
/usr/local/go/src/net/fd_unix.go:419 +0x2b8
net.(*TCPListener).accept(0xc42007e250, 0xc420023ed8, 0x60c393, 0xc420023ea8)
/usr/local/go/src/net/tcpsock_posix.go:132 +0x51
net.(*TCPListener).Accept(0xc42007e250, 0x1f8f0a0, 0xc4268a0cc0, 0xc42722c010, 0xc420023f60)
/usr/local/go/src/net/tcpsock.go:222 +0x50
github.com/cockroachdb/cmux.(*cMux).Serve(0xc4268a0cc0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cmux/cmux.go:124 +0xdd
github.com/cockroachdb/cockroach/server.(*Server).Start.func3()
/go/src/github.com/cockroachdb/cockroach/server/server.go:392 +0x47
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc426ad8b40)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18215 [select]:
google.golang.org/grpc/transport.(*http2Client).controller(0xc42043fc00)
/go/src/google.golang.org/grpc/transport/http2_client.go:999 +0x8c9
created by google.golang.org/grpc/transport.newHTTP2Client
/go/src/google.golang.org/grpc/transport/http2_client.go:233 +0x14bd
goroutine 18206 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:407 +0x395
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc420bb2140)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18161 [chan receive]:
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1.1(0xc42133b8c0)
/go/src/github.com/cockroachdb/cockroach/storage/store.go:2692 +0x58
created by github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1
/go/src/github.com/cockroachdb/cockroach/storage/store.go:2695 +0xd8
goroutine 18172 [chan receive]:
github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open.func1(0xc42005d860)
/go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:388 +0x64
created by github.com/cockroachdb/cockroach/storage/engine.(*RocksDB).Open
/go/src/github.com/cockroachdb/cockroach/storage/engine/rocksdb.go:389 +0xb61
goroutine 18210 [chan receive]:
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1.1(0xc42133b8c0)
/go/src/github.com/cockroachdb/cockroach/storage/store.go:2692 +0x58
created by github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1
/go/src/github.com/cockroachdb/cockroach/storage/store.go:2695 +0xd8
goroutine 18176 [select]:
github.com/cockroachdb/cockroach/storage.NewRaftTransport.func1()
/go/src/github.com/cockroachdb/cockroach/storage/raft_transport.go:172 +0x12f6
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc427231c50)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18229 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:407 +0x395
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc420bb22e0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18177 [select]:
github.com/cockroachdb/cockroach/sql.(*LeaseManager).RefreshLeases.func1()
/go/src/github.com/cockroachdb/cockroach/sql/lease.go:1144 +0x1204
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc427038de0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18242 [runnable]:
encoding/gob.(*Encoder).encodeStruct(0xc420acf680, 0xc420acf6b8, 0xc4200a6500, 0x1c5c020, 0xc42041dd10, 0x199)
/usr/local/go/src/encoding/gob/encode.go:312
encoding/gob.encOpFor.func4(0xc420330030, 0xc4208c9e80, 0x1c5c020, 0xc42041dd10, 0x199)
/usr/local/go/src/encoding/gob/encode.go:587 +0x117
encoding/gob.(*Encoder).encodeStruct(0xc420acf680, 0xc420acf6b8, 0xc4200a6540, 0x1c08280, 0xc42041dce0, 0x199)
/usr/local/go/src/encoding/gob/encode.go:334 +0x438
encoding/gob.encOpFor.func4(0xc420b56060, 0xc4208c99c0, 0x1c08280, 0xc42041dce0, 0x199)
/usr/local/go/src/encoding/gob/encode.go:587 +0x117
encoding/gob.(*Encoder).encodeStruct(0xc420acf680, 0xc420acf6b8, 0xc4200a64a0, 0x1c76140, 0xc422fea650, 0x199)
/usr/local/go/src/encoding/gob/encode.go:334 +0x438
encoding/gob.encOpFor.func4(0x0, 0xc4208c9980, 0x1c76140, 0xc422fea650, 0x199)
/usr/local/go/src/encoding/gob/encode.go:587 +0x117
encoding/gob.(*Encoder).encodeArray(0xc420acf680, 0xc420acf6b8, 0x1ac9b80, 0xc4262ad248, 0x197, 0xc420295210, 0x0, 0x2ee5, 0x0)
/usr/local/go/src/encoding/gob/encode.go:357 +0x2ca
encoding/gob.encOpFor.func1(0xc4201ef8f0, 0xc4208c9900, 0x1ac9b80, 0xc4262ad248, 0x197)
/usr/local/go/src/encoding/gob/encode.go:557 +0x171
encoding/gob.(*Encoder).encodeStruct(0xc420acf680, 0xc420acf6b8, 0xc4200a6320, 0x1bdaa80, 0xc4262ad1f0, 0x199)
/usr/local/go/src/encoding/gob/encode.go:334 +0x438
encoding/gob.(*Encoder).encode(0xc420acf680, 0xc420acf6b8, 0x1bdaa80, 0xc4262ad1f0, 0x199, 0xc420be8a80)
/usr/local/go/src/encoding/gob/encode.go:707 +0x258
encoding/gob.(*Encoder).EncodeValue(0xc420acf680, 0x1c695c0, 0xc4262ad1f0, 0x16, 0x0, 0x0)
/usr/local/go/src/encoding/gob/encoder.go:247 +0x6eb
encoding/gob.(*Encoder).Encode(0xc420acf680, 0x1c695c0, 0xc4262ad1f0, 0x0, 0x0)
/usr/local/go/src/encoding/gob/encoder.go:175 +0x73
github.com/cockroachdb/cockroach/kv.grpcTransportFactory.func1()
/go/src/github.com/cockroachdb/cockroach/kv/transport_race.go:94 +0x39c
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc4270eed50)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18209 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:407 +0x395
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc420bb2220)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18230 [select]:
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc420adfa00, 0xecf59472f, 0xc400bcef4c, 0x2be9b20, 0xc4268a0540, 0xc420b46000, 0xc4262a9b00, 0xc42062ec38)
/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:201 +0x299
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1.1(0xc4262a9b00, 0xc420df2e10)
/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:259 +0x124
github.com/cockroachdb/cockroach/storage.(*storeRangeSet).Visit(0xc420df2ed0, 0xc4208c0f40)
/go/src/github.com/cockroachdb/cockroach/storage/store.go:243 +0x453
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:261 +0x3a5
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc420bb2300)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18171 [IO wait]:
net.runtime_pollWait(0x7f8cd9331f58, 0x72, 0x5f17c3)
/usr/local/go/src/runtime/netpoll.go:160 +0x5e
net.(*pollDesc).wait(0xc420decfb0, 0x72, 0x5ee078, 0x1b566a0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x5b
net.(*pollDesc).waitRead(0xc420decfb0, 0x2899b40, 0xc42007c0b0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x42
net.(*netFD).Read(0xc420decf50, 0xc426286800, 0x400, 0x400, 0x0, 0x2899b40, 0xc42007c0b0)
/usr/local/go/src/net/fd_unix.go:243 +0x1d9
net.(*conn).Read(0xc42007ea48, 0xc426286800, 0x400, 0x400, 0x102, 0xc42628680d, 0xc426286805)
/usr/local/go/src/net/net.go:173 +0x97
net.(*TCPConn).Read(0xc42007ea48, 0xc426286800, 0x400, 0x400, 0xc42129c688, 0x12f1e29, 0xc42129c688)
<autogenerated>:69 +0x73
crypto/tls.(*block).readFromUntil(0xc4215477d0, 0x7f8cd92f3d30, 0xc42007ea48, 0x5, 0xc42007ea48, 0x0)
/usr/local/go/src/crypto/tls/conn.go:472 +0x10d
crypto/tls.(*Conn).readRecord(0xc420be1180, 0x1f93017, 0xc420be1280, 0xc426de9700)
/usr/local/go/src/crypto/tls/conn.go:574 +0x13d
crypto/tls.(*Conn).Read(0xc420be1180, 0xc421366000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1109 +0x186
bufio.(*Reader).fill(0xc4213ea7e0)
/usr/local/go/src/bufio/bufio.go:97 +0x197
bufio.(*Reader).Read(0xc4213ea7e0, 0xc420f056a0, 0x5, 0x200, 0xc42129cc80, 0x61c44a, 0xc42032ad00)
/usr/local/go/src/bufio/bufio.go:209 +0x6ac
io.ReadAtLeast(0x2891e00, 0xc4213ea7e0, 0xc420f056a0, 0x5, 0x200, 0x5, 0x1f930a8, 0x4, 0x61c0c7)
/usr/local/go/src/io/io.go:307 +0xb2
io.ReadFull(0x2891e00, 0xc4213ea7e0, 0xc420f056a0, 0x5, 0x200, 0x1b0ce20, 0x0, 0x30000628c0d)
/usr/local/go/src/io/io.go:325 +0x73
github.com/cockroachdb/pq.(*conn).recvMessage(0xc420f05680, 0xc42116f280, 0x0, 0x1ae7620, 0x1)
/go/src/github.com/cockroachdb/pq/conn.go:953 +0xde
github.com/cockroachdb/pq.(*conn).recv1Buf(0xc420f05680, 0xc42116f280, 0xc420f056a0)
/go/src/github.com/cockroachdb/pq/conn.go:1003 +0x47
github.com/cockroachdb/pq.(*conn).recv1(0xc420f05680, 0xc42129cf18, 0x1fb)
/go/src/github.com/cockroachdb/pq/conn.go:1024 +0xb0
github.com/cockroachdb/pq.(*conn).simpleExec(0xc420f05680, 0xc4267ca690, 0x28, 0xc42129cfe8, 0xc42129cf98, 0x5ee799, 0x1b48ce0, 0x1c75440, 0xc42129cf01)
/go/src/github.com/cockroachdb/pq/conn.go:614 +0x2c0
github.com/cockroachdb/pq.(*conn).Exec(0xc420f05680, 0xc4267ca690, 0x28, 0x35a71a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/pq/conn.go:864 +0x158
database/sql.(*DB).exec(0xc426b49130, 0xc4267ca690, 0x28, 0x0, 0x0, 0x0, 0x61c301, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/database/sql/sql.go:1036 +0x553
database/sql.(*DB).Exec(0xc426b49130, 0xc4267ca690, 0x28, 0x0, 0x0, 0x0, 0xc42129d1d8, 0x902642, 0x2878b60, 0x1c62240)
/usr/local/go/src/database/sql/sql.go:1010 +0xb0
github.com/cockroachdb/cockroach/sql_test.runSqlWithRetry(0xc4203a8180, 0xc426b49130, 0xc4267ca690, 0x28)
/go/src/github.com/cockroachdb/cockroach/sql/schema_changer_test.go:435 +0xa6
github.com/cockroachdb/cockroach/sql_test.runSchemaChangeWithOperations(0xc4203a8180, 0xc426b49130, 0xc420c6c5a0, 0x1c9eeea, 0x1a, 0xfa0, 0x3, 0xc423e5de60)
/go/src/github.com/cockroachdb/cockroach/sql/schema_changer_test.go:488 +0x5a0
github.com/cockroachdb/cockroach/sql_test.TestRaceWithBackfill(0xc4203a8180)
/go/src/github.com/cockroachdb/cockroach/sql/schema_changer_test.go:612 +0xb2a
testing.tRunner(0xc4203a8180, 0x1f910d8)
/usr/local/go/src/testing/testing.go:610 +0xca
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:646 +0x530
goroutine 18236 [select]:
github.com/cockroachdb/cockroach/server.(*Node).startWriteSummaries.func1()
/go/src/github.com/cockroachdb/cockroach/server/node.go:668 +0x32d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc420bb24c0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18285 [runnable]:
github.com/cockroachdb/cockroach/util/bufalloc.ByteAllocator.Copy(0xc423140000, 0x1678, 0x4000, 0x7f8cd482f4d0, 0x6, 0x6, 0x1, 0xc4268bdf00, 0x7f8cd482f4d0, 0x6, ...)
/go/src/github.com/cockroachdb/cockroach/util/bufalloc/byte_allocator.go:63 +0xfc
github.com/cockroachdb/cockroach/storage/engine.MVCCIterate(0x7f8cd9331278, 0xc4213cb7d0, 0x7f8cd9332a98, 0xc42005d860, 0xc420868a90, 0x2, 0x8, 0xc420868a98, 0x2, 0x2, ...)
/go/src/github.com/cockroachdb/cockroach/storage/engine/mvcc.go:1592 +0x73d
github.com/cockroachdb/cockroach/storage/engine.mvccScanInternal(0x7f8cd9331278, 0xc4213cb7d0, 0x7f8cd9332a98, 0xc42005d860, 0xc420868a90, 0x2, 0x8, 0xc420868a98, 0x2, 0x2, ...)
/go/src/github.com/cockroachdb/cockroach/storage/engine/mvcc.go:1464 +0x416
github.com/cockroachdb/cockroach/storage/engine.MVCCScan(0x7f8cd9331278, 0xc4213cb7d0, 0x7f8cd9332a98, 0xc42005d860, 0xc420868a90, 0x2, 0x8, 0xc420868a98, 0x2, 0x2, ...)
/go/src/github.com/cockroachdb/cockroach/storage/engine/mvcc.go:1486 +0x10c
github.com/cockroachdb/cockroach/storage.(*Replica).Scan(0xc420886600, 0x7f8cd9331278, 0xc4213cb7d0, 0x7f8cd9332b88, 0xc42005d860, 0x147002a7bdc94b30, 0x1, 0x100000001, 0x1, 0x7, ...)
/go/src/github.com/cockroachdb/cockroach/storage/replica_command.go:359 +0x1b1
github.com/cockroachdb/cockroach/storage.(*Replica).executeCmd(0xc420886600, 0x7f8cd9331278, 0xc4213cb7d0, 0x0, 0x0, 0x0, 0x7f8cd9332b88, 0xc42005d860, 0x0, 0x147002a7bdc94b30, ...)
/go/src/github.com/cockroachdb/cockroach/storage/replica_command.go:125 +0x2c55
github.com/cockroachdb/cockroach/storage.(*Replica).executeBatch(0xc420886600, 0x7f8cd9331278, 0xc4213cb7d0, 0x0, 0x0, 0x7f8cd9332b88, 0xc42005d860, 0x0, 0x147002a7bdc94b30, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/storage/replica.go:2684 +0x404
github.com/cockroachdb/cockroach/storage.(*Replica).addReadOnlyCmd(0xc420886600, 0x7f8cd9331278, 0xc4213cb7d0, 0x147002a7bdc94b30, 0x1, 0x100000001, 0x1, 0x7, 0x0, 0xc420e335c0, ...)
/go/src/github.com/cockroachdb/cockroach/storage/replica.go:1370 +0x2af
github.com/cockroachdb/cockroach/storage.(*Replica).Send(0xc420886600, 0x7f8cd9331278, 0xc4213cb7d0, 0x147002a7bdc94b30, 0x1, 0x100000001, 0x1, 0x7, 0x0, 0xc420e335c0, ...)
/go/src/github.com/cockroachdb/cockroach/storage/replica.go:1004 +0x51d
github.com/cockroachdb/cockroach/storage.(*Store).Send(0xc42062ea00, 0x7f8cd9331278, 0xc4213cb710, 0x147002a7bdc94b30, 0x1, 0x100000001, 0x1, 0x7, 0x0, 0xc420e335c0, ...)
/go/src/github.com/cockroachdb/cockroach/storage/store.go:2051 +0x8c3
github.com/cockroachdb/cockroach/storage.(*Stores).Send(0xc4268a0b00, 0x7f8cd9331278, 0xc4213cb680, 0x0, 0x0, 0x100000001, 0x1, 0x7, 0x0, 0xc420e335c0, ...)
/go/src/github.com/cockroachdb/cockroach/storage/stores.go:182 +0x24b
github.com/cockroachdb/cockroach/server.(*Node).Batch.func3()
/go/src/github.com/cockroachdb/cockroach/server/node.go:824 +0x3d4
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunTask(0xc420b46000, 0xc426838718, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:218 +0x10e
github.com/cockroachdb/cockroach/server.(*Node).Batch(0xc420bce000, 0x7f8cd92f3cc8, 0xc4239c21e0, 0xc4205ebf68, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/server/node.go:836 +0x2cf
github.com/cockroachdb/cockroach/kv.(*grpcTransport).SendNext(0xc424006500, 0xc4239c2120)
/go/src/github.com/cockroachdb/cockroach/kv/transport.go:187 +0x3ec
github.com/cockroachdb/cockroach/kv.(*DistSender).sendToReplicas(0xc420db3c20, 0x7f8cd9331278, 0xc4213cb320, 0x2540be400, 0xb2d05e00, 0x0, 0x7, 0xc427289da0, 0x1, 0x1, ...)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:1010 +0x203
github.com/cockroachdb/cockroach/kv.(*DistSender).sendRPC(0xc420db3c20, 0x7f8cd9331278, 0xc4213cb320, 0x7, 0xc427289da0, 0x1, 0x1, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:379 +0x27c
github.com/cockroachdb/cockroach/kv.(*DistSender).sendSingleRange(0xc420db3c20, 0x7f8cd9331278, 0xc4213cb320, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc420e33080, ...)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:495 +0x1d9
github.com/cockroachdb/cockroach/kv.(*DistSender).sendChunk.func2(0xc426839730, 0xc4268395c8, 0xc426839bf0, 0xc420db3c20, 0xc426839be0, 0x8, 0xc420868a98)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:763 +0x2eb
github.com/cockroachdb/cockroach/kv.(*DistSender).sendChunk(0xc420db3c20, 0x7f8cd9331278, 0xc4213cb320, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc420e33080, ...)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:764 +0x573
github.com/cockroachdb/cockroach/kv.(*DistSender).Send(0xc420db3c20, 0x7f8cd9331278, 0xc4213cb320, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc420e32fc0, ...)
/go/src/github.com/cockroachdb/cockroach/kv/dist_sender.go:606 +0x1a1
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).Send(0xc420119c30, 0x7f8cd9331278, 0xc4213cb320, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4226d2058, ...)
/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:391 +0x68a
github.com/cockroachdb/cockroach/internal/client.(*txnSender).Send(0xc4226d2000, 0x7f8cd9331238, 0xc42007c6f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4226d2058, ...)
/go/src/github.com/cockroachdb/cockroach/internal/client/txn.go:52 +0x121
github.com/cockroachdb/cockroach/internal/client.(*DB).send(0xc4226d2000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/internal/client/db.go:495 +0x294
github.com/cockroachdb/cockroach/internal/client.(*Txn).send(0xc4226d2000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/internal/client/txn.go:681 +0x63d
github.com/cockroachdb/cockroach/internal/client.(*Txn).(github.com/cockroachdb/cockroach/internal/client.send)-fm(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2710, ...)
/go/src/github.com/cockroachdb/cockroach/internal/client/txn.go:359 +0x7d
github.com/cockroachdb/cockroach/internal/client.sendAndFill(0xc42683b080, 0xc423b28600, 0x0, 0x1bd7780)
/go/src/github.com/cockroachdb/cockroach/internal/client/db.go:422 +0x1ac
github.com/cockroachdb/cockroach/internal/client.(*Txn).Run(0xc4226d2000, 0xc423b28600, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/internal/client/txn.go:359 +0xfe
github.com/cockroachdb/cockroach/sql/sqlbase.(*kvFetcher).fetch(0xc420c98680, 0x1, 0x61c0c7)
/go/src/github.com/cockroachdb/cockroach/sql/sqlbase/kvfetcher.go:210 +0x5b9
github.com/cockroachdb/cockroach/sql/sqlbase.(*kvFetcher).nextKV(0xc420c98680, 0x10, 0xc423b80e20, 0x7f8cd58423a8, 0x0, 0x1, 0xc420028800, 0x7f8cd942b000)
/go/src/github.com/cockroachdb/cockroach/sql/sqlbase/kvfetcher.go:249 +0x29f
github.com/cockroachdb/cockroach/sql/sqlbase.(*RowFetcher).NextKey(0xc420c98600, 0x68, 0x1, 0x1)
/go/src/github.com/cockroachdb/cockroach/sql/sqlbase/rowfetcher.go:194 +0x5a
github.com/cockroachdb/cockroach/sql/sqlbase.(*RowFetcher).StartScan(0xc420c98600, 0xc4226d2000, 0xc4213cb2c0, 0x1, 0x1, 0x0, 0x3, 0x4)
/go/src/github.com/cockroachdb/cockroach/sql/sqlbase/rowfetcher.go:183 +0x1e1
github.com/cockroachdb/cockroach/sql.(*SchemaChanger).truncateAndBackfillColumnsChunk.func1(0xc4226d2000, 0x2, 0x1)
/go/src/github.com/cockroachdb/cockroach/sql/backfill.go:317 +0xb7a
github.com/cockroachdb/cockroach/internal/client.(*DB).Txn.func1(0xc4226d2000, 0xc420868ab0, 0xa04b65c47e8e97a7, 0xa02ceb34e0f05abe)
/go/src/github.com/cockroachdb/cockroach/internal/client/db.go:459 +0x47
github.com/cockroachdb/cockroach/internal/client.(*Txn).Exec(0xc4226d2000, 0xc420860101, 0x0, 0xc420868aa0, 0x2892a40, 0xc423164240)
/go/src/github.com/cockroachdb/cockroach/internal/client/txn.go:559 +0x229
github.com/cockroachdb/cockroach/internal/client.(*DB).Txn(0xc421309110, 0x7f8cd9331238, 0xc42007c6f0, 0xc420449540, 0xc42007c6f0, 0xfff05ac205bc330)
/go/src/github.com/cockroachdb/cockroach/internal/client/db.go:460 +0x2fb
github.com/cockroachdb/cockroach/sql.(*SchemaChanger).truncateAndBackfillColumnsChunk(0xc421309100, 0x0, 0x0, 0x0, 0xc42691d6e0, 0x1, 0x1, 0x0, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/sql/backfill.go:372 +0x3da
github.com/cockroachdb/cockroach/sql.(*SchemaChanger).truncateAndBackfillColumns(0xc421309100, 0xc426ed5180, 0x0, 0x0, 0x0, 0xc42691d6e0, 0x1, 0x1, 0x63a1f0, 0xc42032ad00)
/go/src/github.com/cockroachdb/cockroach/sql/backfill.go:243 +0x541
github.com/cockroachdb/cockroach/sql.(*SchemaChanger).runBackfill(0xc421309100, 0xc426ed5180, 0x0, 0x726616)
/go/src/github.com/cockroachdb/cockroach/sql/backfill.go:162 +0x595
github.com/cockroachdb/cockroach/sql.(*SchemaChanger).runStateMachineAndBackfill(0xc421309100, 0xc426ed5180, 0xc420010460, 0xc42683cb68, 0xc426ed5101)
/go/src/github.com/cockroachdb/cockroach/sql/schema_changer.go:501 +0x83
github.com/cockroachdb/cockroach/sql.SchemaChanger.exec(0x300000033, 0x1, 0x2892700, 0xc420119c30, 0x3ff0000000000000, 0x2faf080, 0x12a05f200, 0x4000000000000000, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/sql/schema_changer.go:325 +0x508
github.com/cockroachdb/cockroach/sql.(*schemaChangerCollection).execSchemaChanges(0xc4262f2040, 0xc4213a0000, 0xc4262f20a0, 0xc422830120, 0x1, 0x1)
/go/src/github.com/cockroachdb/cockroach/sql/session.go:332 +0x482
github.com/cockroachdb/cockroach/sql.(*Executor).execRequest(0xc4213a0000, 0xc4262f2000, 0xc421113821, 0x1a, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/sql/executor.go:623 +0x8b1
github.com/cockroachdb/cockroach/sql.(*Executor).ExecuteStatements(0xc4213a0000, 0xc4262f2000, 0xc421113821, 0x1a, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/sql/executor.go:416 +0x197
github.com/cockroachdb/cockroach/sql/pgwire.(*v3Conn).executeStatements(0xc426d1f380, 0x7f8cd9331338, 0xc4207054c0, 0xc421113821, 0x1a, 0x0, 0x0, 0x0, 0x0, 0x10000c420eb0f01, ...)
/go/src/github.com/cockroachdb/cockroach/sql/pgwire/v3.go:648 +0xba
github.com/cockroachdb/cockroach/sql/pgwire.(*v3Conn).handleSimpleQuery(0xc426d1f380, 0x7f8cd9331338, 0xc4207054c0, 0xc426d1f3a8, 0x1f, 0x0)
/go/src/github.com/cockroachdb/cockroach/sql/pgwire/v3.go:327 +0x10a
github.com/cockroachdb/cockroach/sql/pgwire.(*v3Conn).serve(0xc426d1f380, 0xc422935ce0, 0xc422935ce0, 0x0)
/go/src/github.com/cockroachdb/cockroach/sql/pgwire/v3.go:279 +0x835
github.com/cockroachdb/cockroach/sql/pgwire.(*Server).ServeConn(0xc4268a0ac0, 0x28ac720, 0xc4270bee00, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/sql/pgwire/server.go:240 +0x98a
github.com/cockroachdb/cockroach/server.(*Server).Start.func8.1(0x28ac780, 0xc42419dc30)
/go/src/github.com/cockroachdb/cockroach/server/server.go:419 +0x89
github.com/cockroachdb/cockroach/util/netutil.(*Server).ServeWith.func1(0xc420b46000, 0xc42007e230, 0x28ac780, 0xc42419dc30, 0xc42722c030)
/go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:138 +0xc9
created by github.com/cockroachdb/cockroach/util/netutil.(*Server).ServeWith
/go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:140 +0x2f9
goroutine 18160 [chan receive]:
github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1.1(0xc42133b8c0)
/go/src/github.com/cockroachdb/cockroach/storage/store.go:2692 +0x58
created by github.com/cockroachdb/cockroach/storage.(*Store).processRaft.func1
/go/src/github.com/cockroachdb/cockroach/storage/store.go:2695 +0xd8
goroutine 18228 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:407 +0x395
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc420bb22c0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18333 [select]:
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).heartbeatLoop(0xc420119c30, 0x7f8cd9331278, 0xc4215a5ad0, 0xa04b65c47e8e97a7, 0xa02ceb34e0f05abe)
/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:619 +0x517
github.com/cockroachdb/cockroach/kv.(*TxnCoordSender).updateState.func2()
/go/src/github.com/cockroachdb/cockroach/kv/txn_coord_sender.go:888 +0x67
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask.func1(0xc420b46000, 0x2588771, 0x16, 0x379, 0xc4215a5aa0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:234 +0xa3
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunAsyncTask
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:235 +0xca
goroutine 18227 [select]:
github.com/cockroachdb/cockroach/storage.(*baseQueue).processLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/queue.go:407 +0x395
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc420bb2280)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18238 [IO wait]:
net.runtime_pollWait(0x7f8cd9331ad8, 0x72, 0x5f17c3)
/usr/local/go/src/runtime/netpoll.go:160 +0x5e
net.(*pollDesc).wait(0xc42724faa0, 0x72, 0x5ee078, 0x1b566a0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x5b
net.(*pollDesc).waitRead(0xc42724faa0, 0x2899b40, 0xc42007c0b0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x42
net.(*netFD).accept(0xc42724fa40, 0x0, 0x2896d40, 0xc421204680)
/usr/local/go/src/net/fd_unix.go:419 +0x2b8
net.(*TCPListener).accept(0xc42007e238, 0xc4200271c0, 0xc426b03ec8, 0x6166bf)
/usr/local/go/src/net/tcpsock_posix.go:132 +0x51
net.(*TCPListener).Accept(0xc42007e238, 0x1f8f0a8, 0xc4268a0c80, 0x28aca20, 0xc4200271c0)
/usr/local/go/src/net/tcpsock.go:222 +0x50
github.com/cockroachdb/cmux.(*cMux).Serve(0xc4268a0c80, 0x0, 0x0)
/go/src/github.com/cockroachdb/cmux/cmux.go:124 +0xdd
github.com/cockroachdb/cockroach/server.(*Server).Start.func11()
/go/src/github.com/cockroachdb/cockroach/server/server.go:497 +0x47
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc420bb2540)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18237 [select]:
github.com/cockroachdb/cockroach/sql.(*SchemaChangeManager).Start.func1()
/go/src/github.com/cockroachdb/cockroach/sql/schema_changer.go:705 +0x1a60
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc420bb24e0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18231 [select]:
github.com/cockroachdb/cockroach/server.(*Node).startComputePeriodicMetrics.func1()
/go/src/github.com/cockroachdb/cockroach/server/node.go:632 +0x2f3
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc420bb23e0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18178 [select]:
github.com/cockroachdb/cockroach/sql.NewExecutor.func1()
/go/src/github.com/cockroachdb/cockroach/sql/executor.go:288 +0x1c7
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc42139e720)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18155 [select]:
github.com/cockroachdb/cockroach/storage.(*bookie).start.func1()
/go/src/github.com/cockroachdb/cockroach/storage/reservation.go:274 +0x4c8
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc42147f480)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18183 [chan receive]:
github.com/cockroachdb/cmux.(*muxListener).Accept(0xc426ad8b00, 0xc420df2000, 0x1b257a0, 0x28784b0, 0x1bb01a0)
<autogenerated>:9 +0xa4
net/http.(*Server).Serve(0xc426b37780, 0x28a4840, 0xc426ad8b00, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2273 +0x249
github.com/cockroachdb/cockroach/server.(*Server).Start.func4()
/go/src/github.com/cockroachdb/cockroach/server/server.go:396 +0x54
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc426ad8b60)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18179 [chan receive]:
github.com/cockroachdb/cockroach/util/netutil.MakeServer.func2()
/go/src/github.com/cockroachdb/cockroach/util/netutil/net.go:100 +0x8e
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc426ad88c0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18199 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).startGossip.func2()
/go/src/github.com/cockroachdb/cockroach/storage/store.go:1032 +0x2f9
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc4271c7c70)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18197 [select]:
github.com/cockroachdb/cockroach/storage.(*Store).Start.func3()
/go/src/github.com/cockroachdb/cockroach/storage/store.go:936 +0x19d
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc420d5dbe0)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18235 [select]:
github.com/cockroachdb/cockroach/ts.(*poller).start.func1()
/go/src/github.com/cockroachdb/cockroach/ts/db.go:81 +0x1c7
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc4270ee500)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18207 [select]:
github.com/cockroachdb/cockroach/storage.(*replicaScanner).waitAndProcess(0xc420adfa80, 0xecf594729, 0xc439c5e4ec, 0x2be9b20, 0xc4268a0540, 0xc420b46000, 0xc420c68600, 0xc42062ec38)
/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:201 +0x299
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1.1(0xc420c68600, 0xc420df2e10)
/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:259 +0x124
github.com/cockroachdb/cockroach/storage.(*storeRangeSet).Visit(0xc420df3410, 0xc4268a1ec0)
/go/src/github.com/cockroachdb/cockroach/storage/store.go:243 +0x453
github.com/cockroachdb/cockroach/storage.(*replicaScanner).scanLoop.func1()
/go/src/github.com/cockroachdb/cockroach/storage/scanner.go:261 +0x3a5
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc420bb2160)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
goroutine 18173 [chan receive]:
github.com/cockroachdb/cockroach/rpc.NewContext.func1()
/go/src/github.com/cockroachdb/cockroach/rpc/context.go:126 +0x95
github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker.func1(0xc420b46000, 0xc421295b00)
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:190 +0x8b
created by github.com/cockroachdb/cockroach/util/stop.(*Stopper).RunWorker
/go/src/github.com/cockroachdb/cockroach/util/stop/stopper.go:191 +0x74
FAIL github.com/cockroachdb/cockroach/sql 480.300s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment