Skip to content

Instantly share code, notes, and snippets.

@tbg
Created November 15, 2018 15:04
Show Gist options
  • Save tbg/feae535d83db786283312eecdf6597c1 to your computer and use it in GitHub Desktop.
Save tbg/feae535d83db786283312eecdf6597c1 to your computer and use it in GitHub Desktop.
$ roachprod ssh tobias-test -- grep -h -R 'r2[42]' | cut -c 2- | sort | less -S
181115 13:34:13.273356 2449 storage/replica_command.go:306 [n1,s1,r47/1:/{Table/53/1/4…-Max}] initiating a split of this range at key /Table/53/1/5161764/5 [r22]
181115 13:34:13.807320 2504 storage/replica_command.go:306 [n1,s1,r22/1:/{Table/53/1/5…-Max}] initiating a split of this range at key /Table/53/1/5726787/5 [r23]
181115 13:34:13.872233 2513 storage/replica_command.go:449 [n1,merge,s1,r22/1:/Table/53/1/5{161764…-726787…}] initiating a merge of r23:/{Table/53/1/5726787/5-Max} [(n1,s1):1, (n3,s4):2, (n4,s3):3, next=4, gen=0
181115 13:34:14.084370 191 storage/store.go:2561 [n1,s1,r22/1:/Table/53/1/5{161764…-726787…}] removing replica r23/1
181115 13:34:14.084940 123 storage/store.go:2561 [n3,s4,r22/2:/Table/53/1/5{161764…-726787…}] removing replica r23/2
181115 13:34:14.087220 144 storage/store.go:2561 [n4,s3,r22/3:/Table/53/1/5{161764…-726787…}] removing replica r23/3
181115 13:34:14.162425 2496 storage/replica_command.go:306 [n1,s1,r22/1:/{Table/53/1/5…-Max}] initiating a split of this range at key /Table/53/1/5435170/5 [r24]
181115 13:34:14.559912 2482 storage/replica_command.go:306 [n1,s1,r24/1:/{Table/53/1/5…-Max}] initiating a split of this range at key /Table/53/1/5998983/4 [r26]
181115 13:34:15.220862 2575 storage/replica_command.go:306 [n1,s1,r24/1:/Table/53/1/5{435170…-998983…}] initiating a split of this range at key /Table/53/1/5453027/1 [r27]
181115 13:34:15.684731 2595 storage/store_snapshot.go:636 [n1,raftsnapshot,s1,r24/1:/Table/53/1/54{35170…-53027…}] sending Raft snapshot 84c162cb at applied index 26
181115 13:34:15.776163 2535 storage/replica_command.go:306 [n1,split,s1,r24/1:/Table/53/1/54{35170…-53027…}] initiating a split of this range at key /Table/53/1/5700579/5 [r28]
181115 13:34:15.777443 2535 storage/split_queue.go:160 [n1,split,s1,r24/1:/Table/53/1/54{35170…-53027…}] split saw concurrent descriptor modification; maybe retrying
181115 13:34:15.868994 2595 storage/store_snapshot.go:679 [n1,raftsnapshot,s1,r24/1:/Table/53/1/54{35170…-53027…}] streamed snapshot to (n3,s4):2: kv pairs: 9, log entries: 4, rate-limit: 8.0 MiB/sec, 191ms
181115 13:34:15.885740 1036 storage/replica_raftstorage.go:813 [n3,s4,r24/2:/{Table/53/1/5…-Max}] applying Raft snapshot at index 26 (id=84c162cb, encoded size=17949421, 1 rocksdb batches, 4 log entries)
181115 13:34:16.073472 1036 storage/replica_raftstorage.go:819 [n3,s4,r24/2:/Table/53/1/54{35170…-53027…}] applied Raft snapshot in 188ms [clear=0ms batch=0ms entries=180ms commit=8ms]
181115 13:37:21.402481 2964 storage/replicate_queue.go:266 [n3,replicate,s4,r22/2:/Table/53/1/5{161764…-435170…}] snapshot failed: (n2,s2):?: remote declined preemptive snapshot ff6ee1bb at applied index 31: sto
181115 14:17:11.570501 3552 storage/replica_command.go:306 [n1,s1,r20/1:/{Table/23-Max}] initiating a split of this range at key /Table/53/1/4181856/4 [r22]
181115 14:17:11.576687 3556 storage/replica_command.go:306 [n1,s1,r20/1:/{Table/23-Max}] initiating a split of this range at key /Table/53/1/1006755/5 [r24]
181115 14:23:09.254499 3511 storage/replica_command.go:306 [n1,s1,r35/1:/{Table/53/1/2…-Max}] initiating a split of this range at key /Table/53/1/3130789/2 [r22]
181115 14:23:09.511125 3702 storage/replica_command.go:449 [n1,merge,s1,r35/1:/Table/53/1/{285715…-313078…}] initiating a merge of r22:/{Table/53/1/3130789/2-Max} [(n1,s1):1, (n4,s3):2, (n2,s2):3, next=4, gen=0]
181115 14:23:09.603413 168 storage/store.go:2561 [n1,s1,r35/1:/Table/53/1/{285715…-313078…}] removing replica r22/1
181115 14:23:09.605160 151 storage/store.go:2561 [n2,s2,r35/4:/Table/53/1/{285715…-313078…}] removing replica r22/3
181115 14:23:09.608805 139 storage/store.go:2561 [n4,s3,r35/2:/Table/53/1/{285715…-313078…}] removing replica r22/2
181115 14:23:11.028392 3932 storage/replica_command.go:306 [n1,s1,r21/1:/Table/53/1/2{235969…-857157…}] initiating a split of this range at key /Table/53/1/2509123/4 [r24]
181115 14:23:12.087330 3969 storage/replica_command.go:306 [n1,s1,r24/1:/Table/53/1/2{509123…-857157…}] initiating a split of this range at key /Table/53/1/2584101/3 [r27]
181115 14:23:12.135409 3986 storage/store_snapshot.go:636 [n1,replicate,s1,r24/1:/Table/53/1/25{09123…-84101…}] sending preemptive snapshot 71a58e16 at applied index 14
181115 14:23:12.135602 4016 storage/store_snapshot.go:636 [n1,s1,r24/1:/Table/53/1/25{09123…-84101…}] sending preemptive snapshot a5dce9ba at applied index 14
181115 14:23:12.135628 3986 storage/store_snapshot.go:679 [n1,replicate,s1,r24/1:/Table/53/1/25{09123…-84101…}] streamed snapshot to (n3,s4):?: kv pairs: 8, log entries: 4, rate-limit: 2.0 MiB/sec, 1ms
181115 14:23:12.135778 4016 storage/store_snapshot.go:679 [n1,s1,r24/1:/Table/53/1/25{09123…-84101…}] streamed snapshot to (n3,s4):?: kv pairs: 8, log entries: 4, rate-limit: 2.0 MiB/sec, 0ms
181115 14:23:12.138658 1084 storage/replica_raftstorage.go:813 [n3,s4,r24/?:{-}] applying preemptive snapshot at index 14 (id=71a58e16, encoded size=2481, 1 rocksdb batches, 4 log entries)
181115 14:23:12.169677 1084 storage/replica_raftstorage.go:819 [n3,s4,r24/?:/Table/53/1/25{09123…-84101…}] applied preemptive snapshot in 31ms [clear=0ms batch=0ms entries=0ms commit=31ms]
181115 14:23:12.170474 4016 storage/replica_command.go:856 [n1,s1,r24/1:/Table/53/1/25{09123…-84101…}] change replicas (ADD_REPLICA (n3,s4):4): read existing descriptor r24:/Table/53/1/25{09123/4-84101/3} [(n1,s
181115 14:23:12.170572 3986 storage/replica_command.go:856 [n1,replicate,s1,r24/1:/Table/53/1/25{09123…-84101…}] change replicas (ADD_REPLICA (n3,s4):4): read existing descriptor r24:/Table/53/1/25{09123/4-84101
181115 14:23:12.218893 4016 storage/replica.go:3956 [n1,s1,r24/1:/Table/53/1/25{09123…-84101…}] proposing ADD_REPLICA((n3,s4):4): updated=[(n1,s1):1 (n4,s3):2 (n2,s2):3 (n3,s4):4] next=5
181115 14:23:12.330903 848 storage/replica_command.go:856 [n2,replicate,s2,r24/3:/Table/53/1/25{09123…-84101…}] change replicas (REMOVE_REPLICA (n1,s1):1): read existing descriptor r24:/Table/53/1/25{09123/4-841
181115 14:23:12.467927 848 storage/replica.go:3956 [n2,s2,r24/3:/Table/53/1/25{09123…-84101…}] proposing REMOVE_REPLICA((n1,s1):1): updated=[(n3,s4):4 (n4,s3):2 (n2,s2):3] next=5
181115 14:23:12.493192 3986 storage/replica_command.go:856 [n1,replicate,s1,r24/1:/Table/53/1/25{09123…-84101…}] change replicas (ADD_REPLICA (n3,s4):4): read existing descriptor r24:/Table/53/1/25{09123/4-84101
181115 14:23:12.523183 4160 storage/store.go:2561 [n1,replicaGC,s1,r24/1:/Table/53/1/25{09123…-84101…}] removing replica r24/1
181115 14:23:12.533124 4160 storage/replica.go:882 [n1,replicaGC,s1,r24/1:/Table/53/1/25{09123…-84101…}] removed 74716 (74710+6) keys in 10ms [clear=1ms commit=9ms]
181115 14:36:34.544906 4160 storage/replica_command.go:306 [n1,s1,r66/1:/{Table/53/6/9…-Max}] initiating a split of this range at key /Table/53/6/10347/5087009/2 [r22]
181115 14:36:34.544906 4160 storage/replica_command.go:306 [n1,s1,r66/1:/{Table/53/6/9…-Max}] initiating a split of this range at key /Table/53/6/10347/5087009/2 [r22]
181115 14:36:36.113690 4199 storage/replica_command.go:449 [n1,merge,s1,r66/1:/Table/53/6/{9507/2…-10347/…}] initiating a merge of r22:/{Table/53/6/10347/5087009/2-Max} [(n1,s1):1, (n4,s4):2, (n3,s2):3, next=4,
181115 14:36:36.113690 4199 storage/replica_command.go:449 [n1,merge,s1,r66/1:/Table/53/6/{9507/2…-10347/…}] initiating a merge of r22:/{Table/53/6/10347/5087009/2-Max} [(n1,s1):1, (n4,s4):2, (n3,s2):3, next=4,
181115 14:36:36.236569 179 storage/store.go:2561 [n1,s1,r66/1:/Table/53/6/{9507/2…-10347/…}] removing replica r22/1
181115 14:36:36.236569 179 storage/store.go:2561 [n1,s1,r66/1:/Table/53/6/{9507/2…-10347/…}] removing replica r22/1
181115 14:36:36.323915 144 storage/store.go:2561 [n3,s2,r66/5:/Table/53/6/{9507/2…-10347/…}] removing replica r22/3
181115 14:36:36.323915 144 storage/store.go:2561 [n3,s2,r66/5:/Table/53/6/{9507/2…-10347/…}] removing replica r22/3
181115 14:36:36.324193 160 storage/store.go:2561 [n4,s4,r66/4:/Table/53/6/{9507/2…-10347/…}] removing replica r22/2
181115 14:36:36.324193 160 storage/store.go:2561 [n4,s4,r66/4:/Table/53/6/{9507/2…-10347/…}] removing replica r22/2
181115 14:36:38.105905 4343 storage/replica_command.go:306 [n1,s1,r21/1:/Table/53/6/{8617/5…-9507/2…}] initiating a split of this range at key /Table/53/6/8667/3638725/7 [r24]
181115 14:36:38.105905 4343 storage/replica_command.go:306 [n1,s1,r21/1:/Table/53/6/{8617/5…-9507/2…}] initiating a split of this range at key /Table/53/6/8667/3638725/7 [r24]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment