Skip to content

Instantly share code, notes, and snippets.

@tbg
Created October 5, 2019 02:17
Show Gist options
  • Save tbg/8acd0ea0193ba5a360fb5618d6d87812 to your computer and use it in GitHub Desktop.
Save tbg/8acd0ea0193ba5a360fb5618d6d87812 to your computer and use it in GitHub Desktop.
tschottdorf@gceworker-tschottdorf:~/go/src/github.com/cockroachdb/cockroach$ ./cockroach-linux-2.6.32-gnu-amd64 debug merge-logs artifacts/tpccbench/nodes\=9/cpu\=4/multi-region/run_1/*.logs/*.log --filter 'r231[^0-9]'
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:36.999231 2725 storage/replica_command.go:411 [n3,s3,r225/2:/Table/5{8/1/89/5…-9}] initiating a split of this range at key /Table/58/1/90/5244 [r231] (manual)
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:37.728773 2859 storage/replica_command.go:1583 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] change replicas (add [(n6,s6):4LEARNER] remove []): existing descriptor r231:/Table/5{8/1/90/5244-9} [(n8,s8):1, (n3,s3):2, (n5,s5):3, next=4, gen=33, sticky=1570230259.080228881,0]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:38.102603 2846 storage/replica_command.go:411 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] initiating a split of this range at key /Table/58/1/133/87285 [r233] (manual); delayed split for 0.2s to avoid Raft snapshot
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:38.387818 2859 storage/replica_raft.go:291 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] proposing ADD_REPLICA[(n6,s6):4LEARNER]: after=[(n8,s8):1 (n3,s3):2 (n5,s5):3 (n6,s6):4LEARNER] next=5
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:38.523186 2859 storage/store_snapshot.go:978 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] sending LEARNER snapshot c0d9b30c at applied index 15
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:38.523244 2972 storage/raft_snapshot_queue.go:125 [n3,raftsnapshot,s3,r231/2:/Table/5{8/1/90/5…-9}] skipping snapshot; replica is likely a learner in the process of being added: (n6,s6):4LEARNER
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:38.523297 2859 storage/store_snapshot.go:1021 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] streamed snapshot to (n6,s6):4LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.07s
tobias-1570199197-01-n12cpu4-geo-0007> I191004 22:04:38.558875 2584 storage/replica_raftstorage.go:794 [n6,s6,r231/4:{-}] applying LEARNER snapshot [id=c0d9b30c index=15]
tobias-1570199197-01-n12cpu4-geo-0007> I191004 22:04:38.566109 2584 storage/replica_raftstorage.go:815 [n6,s6,r231/4:/Table/5{8/1/90/5…-9}] applied LEARNER snapshot [total=7ms ingestion=4@6ms id=c0d9b30c index=15]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:38.599821 2859 storage/replica_command.go:1583 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] change replicas (add [(n6,s6):4VOTER_INCOMING] remove [(n5,s5):3VOTER_OUTGOING]): existing descriptor r231:/Table/5{8/1/90/5244-9} [(n8,s8):1, (n3,s3):2, (n5,s5):3, (n6,s6):4LEARNER, next=5, gen=34, sticky=1570230259.080228881,0]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:38.711874 2846 storage/replica_command.go:411 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] initiating a split of this range at key /Table/58/1/133/87285 [r234] (manual); delayed split for 0.2s to avoid Raft snapshot
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:39.702817 2859 storage/replica_raft.go:291 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] proposing ENTER_JOINT ADD_REPLICA[(n6,s6):4VOTER_INCOMING], REMOVE_REPLICA[(n5,s5):3VOTER_OUTGOING]: after=[(n8,s8):1 (n3,s3):2 (n5,s5):3VOTER_OUTGOING (n6,s6):4VOTER_INCOMING] next=5
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:39.770469 2859 storage/replica_command.go:1583 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] change replicas (add [] remove []): existing descriptor r231:/Table/5{8/1/90/5244-9} [(n8,s8):1, (n3,s3):2, (n5,s5):3VOTER_OUTGOING, (n6,s6):4VOTER_INCOMING, next=5, gen=35, sticky=1570230259.080228881,0]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:40.126266 2859 storage/replica_raft.go:291 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] proposing LEAVE_JOINT: after=[(n8,s8):1 (n3,s3):2 (n6,s6):4] next=5
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:40.194105 2859 storage/replica_command.go:1583 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] change replicas (add [(n5,s5):5LEARNER] remove []): existing descriptor r231:/Table/5{8/1/90/5244-9} [(n8,s8):1, (n3,s3):2, (n6,s6):4, next=5, gen=36, sticky=1570230259.080228881,0]
tobias-1570199197-01-n12cpu4-geo-0006> I191004 22:04:40.227671 179 storage/store_remove_replica.go:129 [n5,s5,r231/3:/Table/5{8/1/90/5…-9}] removing replica r231/3
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:40.261823 2846 storage/replica_command.go:411 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] initiating a split of this range at key /Table/58/1/133/87285 [r275] (manual)
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:41.095276 2859 storage/replica_raft.go:291 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] proposing ADD_REPLICA[(n5,s5):5LEARNER]: after=[(n8,s8):1 (n3,s3):2 (n6,s6):4 (n5,s5):5LEARNER] next=6
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:41.230308 2859 storage/store_snapshot.go:978 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] sending LEARNER snapshot 4ea5298f at applied index 29
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:41.230428 3305 storage/raft_snapshot_queue.go:125 [n3,raftsnapshot,s3,r231/2:/Table/5{8/1/90/5…-9}] skipping snapshot; replica is likely a learner in the process of being added: (n5,s5):5LEARNER
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:41.230467 2859 storage/store_snapshot.go:1021 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] streamed snapshot to (n5,s5):5LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.07s
tobias-1570199197-01-n12cpu4-geo-0006> I191004 22:04:41.266036 3391 storage/replica_raftstorage.go:794 [n5,s5,r231/5:{-}] applying LEARNER snapshot [id=4ea5298f index=29]
tobias-1570199197-01-n12cpu4-geo-0006> I191004 22:04:41.269223 3391 storage/replica_raftstorage.go:815 [n5,s5,r231/5:/Table/5{8/1/90/5…-9}] applied LEARNER snapshot [total=3ms ingestion=4@2ms id=4ea5298f index=29]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:41.302897 2859 storage/replica_command.go:1583 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] change replicas (add [(n5,s5):5VOTER_INCOMING] remove [(n6,s6):4VOTER_OUTGOING]): existing descriptor r231:/Table/5{8/1/90/5244-9} [(n8,s8):1, (n3,s3):2, (n6,s6):4, (n5,s5):5LEARNER, next=6, gen=37, sticky=1570230259.080228881,0]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:41.416927 2846 storage/replica_command.go:411 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] initiating a split of this range at key /Table/58/1/133/87285 [r276] (manual)
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:42.411161 2859 storage/replica_raft.go:291 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] proposing ENTER_JOINT ADD_REPLICA[(n5,s5):5VOTER_INCOMING], REMOVE_REPLICA[(n6,s6):4VOTER_OUTGOING]: after=[(n8,s8):1 (n3,s3):2 (n6,s6):4VOTER_OUTGOING (n5,s5):5VOTER_INCOMING] next=6
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:42.478803 2859 storage/replica_command.go:1583 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] change replicas (add [] remove []): existing descriptor r231:/Table/5{8/1/90/5244-9} [(n8,s8):1, (n3,s3):2, (n6,s6):4VOTER_OUTGOING, (n5,s5):5VOTER_INCOMING, next=6, gen=38, sticky=1570230259.080228881,0]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:42.746940 2859 storage/replica_raft.go:291 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] proposing LEAVE_JOINT: after=[(n8,s8):1 (n3,s3):2 (n5,s5):5] next=6
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:42.815456 2859 storage/replica_command.go:1583 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] change replicas (add [(n6,s6):6LEARNER] remove []): existing descriptor r231:/Table/5{8/1/90/5244-9} [(n8,s8):1, (n3,s3):2, (n5,s5):5, next=6, gen=39, sticky=1570230259.080228881,0]
tobias-1570199197-01-n12cpu4-geo-0007> I191004 22:04:42.849002 216 storage/store_remove_replica.go:129 [n6,s6,r231/4:/Table/5{8/1/90/5…-9}] removing replica r231/4
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:42.966205 2846 storage/replica_command.go:411 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] initiating a split of this range at key /Table/58/1/133/87285 [r277] (manual)
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:43.475129 2859 storage/replica_raft.go:291 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] proposing ADD_REPLICA[(n6,s6):6LEARNER]: after=[(n8,s8):1 (n3,s3):2 (n5,s5):5 (n6,s6):6LEARNER] next=7
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:43.610377 2859 storage/store_snapshot.go:978 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] sending LEARNER snapshot fe276774 at applied index 40
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:43.610437 3615 storage/raft_snapshot_queue.go:125 [n3,raftsnapshot,s3,r231/2:/Table/5{8/1/90/5…-9}] skipping snapshot; replica is likely a learner in the process of being added: (n6,s6):6LEARNER
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:43.610523 2859 storage/store_snapshot.go:1021 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] streamed snapshot to (n6,s6):6LEARNER: kv pairs: 14, log entries: 0, rate-limit: 8.0 MiB/sec, 0.07s
tobias-1570199197-01-n12cpu4-geo-0007> I191004 22:04:43.645786 2888 storage/replica_raftstorage.go:794 [n6,s6,r231/6:{-}] applying LEARNER snapshot [id=fe276774 index=40]
tobias-1570199197-01-n12cpu4-geo-0007> I191004 22:04:43.654051 2888 storage/replica_raftstorage.go:815 [n6,s6,r231/6:/Table/5{8/1/90/5…-9}] applied LEARNER snapshot [total=8ms ingestion=4@7ms id=fe276774 index=40]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:43.687993 2859 storage/replica_command.go:1583 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] change replicas (add [(n6,s6):6VOTER_INCOMING] remove [(n5,s5):5VOTER_OUTGOING]): existing descriptor r231:/Table/5{8/1/90/5244-9} [(n8,s8):1, (n3,s3):2, (n5,s5):5, (n6,s6):6LEARNER, next=7, gen=40, sticky=1570230259.080228881,0]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:44.239113 2846 storage/replica_command.go:411 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] initiating a split of this range at key /Table/58/1/133/87285 [r278] (manual)
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:44.836955 2859 storage/replica_raft.go:291 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] proposing ENTER_JOINT ADD_REPLICA[(n6,s6):6VOTER_INCOMING], REMOVE_REPLICA[(n5,s5):5VOTER_OUTGOING]: after=[(n8,s8):1 (n3,s3):2 (n5,s5):5VOTER_OUTGOING (n6,s6):6VOTER_INCOMING] next=7
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:44.904504 2859 storage/replica_command.go:1583 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] change replicas (add [] remove []): existing descriptor r231:/Table/5{8/1/90/5244-9} [(n8,s8):1, (n3,s3):2, (n5,s5):5VOTER_OUTGOING, (n6,s6):6VOTER_INCOMING, next=7, gen=41, sticky=1570230259.080228881,0]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:45.173068 2859 storage/replica_raft.go:291 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] proposing LEAVE_JOINT: after=[(n8,s8):1 (n3,s3):2 (n6,s6):6] next=7
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:45.241056 2859 storage/replica_command.go:1583 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] change replicas (add [(n5,s5):7LEARNER] remove []): existing descriptor r231:/Table/5{8/1/90/5244-9} [(n8,s8):1, (n3,s3):2, (n6,s6):6, next=7, gen=42, sticky=1570230259.080228881,0]
tobias-1570199197-01-n12cpu4-geo-0006> I191004 22:04:45.275421 190 storage/store_remove_replica.go:129 [n5,s5,r231/5:/Table/5{8/1/90/5…-9}] removing replica r231/5
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:45.900773 2859 storage/replica_raft.go:291 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] proposing ADD_REPLICA[(n5,s5):7LEARNER]: after=[(n8,s8):1 (n3,s3):2 (n6,s6):6 (n5,s5):7LEARNER] next=8
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:46.035829 2859 storage/store_snapshot.go:978 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] sending LEARNER snapshot 2a9e9fa1 at applied index 52
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:46.035869 3956 storage/raft_snapshot_queue.go:125 [n3,raftsnapshot,s3,r231/2:/Table/5{8/1/90/5…-9}] skipping snapshot; replica is likely a learner in the process of being added: (n5,s5):7LEARNER
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:46.035995 2859 storage/store_snapshot.go:1021 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] streamed snapshot to (n5,s5):7LEARNER: kv pairs: 17, log entries: 0, rate-limit: 8.0 MiB/sec, 0.07s
tobias-1570199197-01-n12cpu4-geo-0006> I191004 22:04:46.071807 4114 storage/replica_raftstorage.go:794 [n5,s5,r231/7:{-}] applying LEARNER snapshot [id=2a9e9fa1 index=52]
tobias-1570199197-01-n12cpu4-geo-0006> I191004 22:04:46.086889 4114 storage/replica_raftstorage.go:815 [n5,s5,r231/7:/Table/5{8/1/90/5…-9}] applied LEARNER snapshot [total=15ms ingestion=4@14ms id=2a9e9fa1 index=52]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:46.122749 2859 storage/replica_command.go:1583 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] change replicas (add [(n5,s5):7VOTER_INCOMING] remove [(n6,s6):6VOTER_OUTGOING]): existing descriptor r231:/Table/5{8/1/90/5244-9} [(n8,s8):1, (n3,s3):2, (n6,s6):6, (n5,s5):7LEARNER, next=8, gen=43, sticky=1570230259.080228881,0]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:46.236159 2846 storage/replica_command.go:411 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] initiating a split of this range at key /Table/58/1/133/87285 [r282] (manual); delayed split for 0.2s to avoid Raft snapshot
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:47.240187 2859 storage/replica_raft.go:291 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] proposing ENTER_JOINT ADD_REPLICA[(n5,s5):7VOTER_INCOMING], REMOVE_REPLICA[(n6,s6):6VOTER_OUTGOING]: after=[(n8,s8):1 (n3,s3):2 (n6,s6):6VOTER_OUTGOING (n5,s5):7VOTER_INCOMING] next=8
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:47.328431 2859 storage/replica_command.go:1583 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] change replicas (add [] remove []): existing descriptor r231:/Table/5{8/1/90/5244-9} [(n8,s8):1, (n3,s3):2, (n6,s6):6VOTER_OUTGOING, (n5,s5):7VOTER_INCOMING, next=8, gen=44, sticky=1570230259.080228881,0]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:47.600317 2859 storage/replica_raft.go:291 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] proposing LEAVE_JOINT: after=[(n8,s8):1 (n3,s3):2 (n5,s5):7] next=8
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:47.668777 2859 storage/replica_command.go:1583 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] change replicas (add [(n6,s6):8LEARNER] remove []): existing descriptor r231:/Table/5{8/1/90/5244-9} [(n8,s8):1, (n3,s3):2, (n5,s5):7, next=8, gen=45, sticky=1570230259.080228881,0]
tobias-1570199197-01-n12cpu4-geo-0007> I191004 22:04:47.701997 216 storage/store_remove_replica.go:129 [n6,s6,r231/6:/Table/5{8/1/90/5…-9}] removing replica r231/6
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:48.329131 2859 storage/replica_raft.go:291 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] proposing ADD_REPLICA[(n6,s6):8LEARNER]: after=[(n8,s8):1 (n3,s3):2 (n5,s5):7 (n6,s6):8LEARNER] next=9
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:48.468443 4534 storage/raft_snapshot_queue.go:125 [n3,raftsnapshot,s3,r231/2:/Table/5{8/1/90/5…-9}] skipping snapshot; replica is likely a learner in the process of being added: (n6,s6):8LEARNER
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:48.470659 2859 storage/store_snapshot.go:978 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] sending LEARNER snapshot 79ad20f1 at applied index 64
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:48.470787 2859 storage/store_snapshot.go:1021 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] streamed snapshot to (n6,s6):8LEARNER: kv pairs: 20, log entries: 0, rate-limit: 8.0 MiB/sec, 0.07s
tobias-1570199197-01-n12cpu4-geo-0007> I191004 22:04:48.506588 3222 storage/replica_raftstorage.go:794 [n6,s6,r231/8:{-}] applying LEARNER snapshot [id=79ad20f1 index=64]
tobias-1570199197-01-n12cpu4-geo-0007> I191004 22:04:48.516328 3222 storage/replica_raftstorage.go:815 [n6,s6,r231/8:/Table/5{8/1/90/5…-9}] applied LEARNER snapshot [total=10ms ingestion=4@9ms id=79ad20f1 index=64]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:48.550224 2859 storage/replica_command.go:1583 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] change replicas (add [(n6,s6):8VOTER_INCOMING] remove [(n5,s5):7VOTER_OUTGOING]): existing descriptor r231:/Table/5{8/1/90/5244-9} [(n8,s8):1, (n3,s3):2, (n5,s5):7, (n6,s6):8LEARNER, next=9, gen=46, sticky=1570230259.080228881,0]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:48.658905 2846 storage/replica_command.go:411 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] initiating a split of this range at key /Table/58/1/133/87285 [r379] (manual); delayed split for 0.2s to avoid Raft snapshot
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:49.669161 2859 storage/replica_raft.go:291 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] proposing ENTER_JOINT ADD_REPLICA[(n6,s6):8VOTER_INCOMING], REMOVE_REPLICA[(n5,s5):7VOTER_OUTGOING]: after=[(n8,s8):1 (n3,s3):2 (n5,s5):7VOTER_OUTGOING (n6,s6):8VOTER_INCOMING] next=9
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:49.736812 2859 storage/replica_command.go:1583 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] change replicas (add [] remove []): existing descriptor r231:/Table/5{8/1/90/5244-9} [(n8,s8):1, (n3,s3):2, (n5,s5):7VOTER_OUTGOING, (n6,s6):8VOTER_INCOMING, next=9, gen=47, sticky=1570230259.080228881,0]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:50.013705 2859 storage/replica_raft.go:291 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] proposing LEAVE_JOINT: after=[(n8,s8):1 (n3,s3):2 (n6,s6):8] next=9
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:50.082340 2859 storage/replica_command.go:1583 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] change replicas (add [(n5,s5):9LEARNER] remove []): existing descriptor r231:/Table/5{8/1/90/5244-9} [(n8,s8):1, (n3,s3):2, (n6,s6):8, next=9, gen=48, sticky=1570230259.080228881,0]
tobias-1570199197-01-n12cpu4-geo-0006> I191004 22:04:50.118941 167 storage/store_remove_replica.go:129 [n5,s5,r231/7:/Table/5{8/1/90/5…-9}] removing replica r231/7
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:50.735107 2846 storage/replica_command.go:411 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] initiating a split of this range at key /Table/58/1/133/87285 [r382] (manual)
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:50.747661 2859 storage/replica_raft.go:291 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] proposing ADD_REPLICA[(n5,s5):9LEARNER]: after=[(n8,s8):1 (n3,s3):2 (n6,s6):8 (n5,s5):9LEARNER] next=10
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:50.883047 4966 storage/raft_snapshot_queue.go:125 [n3,raftsnapshot,s3,r231/2:/Table/5{8/1/90/5…-9}] skipping snapshot; replica is likely a learner in the process of being added: (n5,s5):9LEARNER
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:50.883161 2859 storage/store_snapshot.go:978 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] sending LEARNER snapshot 92a74553 at applied index 77
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:50.883318 2859 storage/store_snapshot.go:1021 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] streamed snapshot to (n5,s5):9LEARNER: kv pairs: 23, log entries: 0, rate-limit: 8.0 MiB/sec, 0.07s
tobias-1570199197-01-n12cpu4-geo-0006> I191004 22:04:50.919092 4834 storage/replica_raftstorage.go:794 [n5,s5,r231/9:{-}] applying LEARNER snapshot [id=92a74553 index=77]
tobias-1570199197-01-n12cpu4-geo-0006> I191004 22:04:50.924849 4834 storage/replica_raftstorage.go:815 [n5,s5,r231/9:/Table/5{8/1/90/5…-9}] applied LEARNER snapshot [total=6ms ingestion=4@5ms id=92a74553 index=77]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:50.958515 2859 storage/replica_command.go:1583 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] change replicas (add [(n5,s5):9VOTER_INCOMING] remove [(n6,s6):8VOTER_OUTGOING]): existing descriptor r231:/Table/5{8/1/90/5244-9} [(n8,s8):1, (n3,s3):2, (n6,s6):8, (n5,s5):9LEARNER, next=10, gen=49, sticky=1570230259.080228881,0]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:51.969476 2846 storage/replica_command.go:411 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] initiating a split of this range at key /Table/58/1/133/87285 [r384] (manual)
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:52.069616 2859 storage/replica_raft.go:291 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] proposing ENTER_JOINT ADD_REPLICA[(n5,s5):9VOTER_INCOMING], REMOVE_REPLICA[(n6,s6):8VOTER_OUTGOING]: after=[(n8,s8):1 (n3,s3):2 (n6,s6):8VOTER_OUTGOING (n5,s5):9VOTER_INCOMING] next=10
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:52.137181 2859 storage/replica_command.go:1583 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] change replicas (add [] remove []): existing descriptor r231:/Table/5{8/1/90/5244-9} [(n8,s8):1, (n3,s3):2, (n6,s6):8VOTER_OUTGOING, (n5,s5):9VOTER_INCOMING, next=10, gen=50, sticky=1570230259.080228881,0]
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:52.405857 2859 storage/replica_raft.go:291 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] proposing LEAVE_JOINT: after=[(n8,s8):1 (n3,s3):2 (n5,s5):9] next=10
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:52.474422 5213 storage/replica_command.go:411 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] initiating a split of this range at key /Table/58/1/91/9863 [r405] (manual)
tobias-1570199197-01-n12cpu4-geo-0007> I191004 22:04:52.507614 220 storage/store_remove_replica.go:129 [n6,s6,r231/8:/Table/5{8/1/90/5…-9}] removing replica r231/8
tobias-1570199197-01-n12cpu4-geo-0003> I191004 22:04:53.076159 2846 storage/replica_command.go:411 [n3,s3,r231/2:/Table/5{8/1/90/5…-9}] initiating a split of this range at key /Table/58/1/133/87285 [r412] (manual)
tschottdorf@gceworker-tschottdorf:~/go/src/github.com/cockroachdb/cockroach$
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment