Last active
July 10, 2021 00:28
-
-
Save mattlord/3afe4f920d9a59d39efcf84564bf4de2 to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The Pos column in the _vt.vreplication table is the vreplication equivalent of mysqld's @@global.gtid_executed value | |
The hard part is finding the source GTID you think you want to skip, verifying the contents of it and that it matches the error, and finally that it's safe to skip | |
You can find the source tablet for that errored vstream in the workflow output, e.g.: | |
... | |
{ | |
"Shard": "-", | |
"Tablet": "us_central1_a-3612345650", | |
"ID": 16, | |
"Bls": { | |
"keyspace": "foob", | |
"shard": "80-", | |
... | |
From there, you can find the mysqld instance, e.g.: | |
$ vtctl ListAllTablets | awk '$2 == "foob" && $3 == "80-" && $4 == "master" {print $6; exit}' | |
dbfoobhost:3306 | |
Then you can see that instance's server_uuid, e.g.: | |
[root@dbfoobhost:/home/mlord] $ mysql --defaults-file=/file/to/creds.cnf -BNe "select @@global.server_uuid" | |
70d613d1-678c-b18b-7704-64b6591be455 | |
And then come back to the workflow's vreplication record to find that UUID in the Pos field, e.g.: | |
"Pos": "MySQL56/70d613d1-678c-b18b-7704-64b6591be455:1-2646817,fb827d7a-696b-7ad1-988c-c77ff601c177:1-28" | |
The problematic GTID is likely the high value +1 -- so in this case 70d613d1-678c-b18b-7704-64b6591be455:2646818 -- but you can verify by finding that GTID in the source hosts's binlogs e.g.: | |
[root@dbfoobhost:/home/mlord] $ for file in /mysql/*-bin.*; do mysqlbinlog -vvv --base64-output=DECODE-ROWS ${file} | grep -A30 "70d613d1-678c-b18b-7704-64b6591be455:2646818" && echo "Found in ${file}"; done | |
Then you can verify that's the right GTID by looking at the values shown for the write in the binlog events and compare it to the error seen for that vreplication stream | |
Then you can work with the app team to be sure it's safe to skip it on the target side... | |
And if so, add that value to the Pos field on the target mysqld intance, e.g.: | |
[root@dbfoobhost:/home/mlord] $ mysql --defaults-file=/file/to/creds.cnf -e "UPDATE _vt.vreplication SET Pos='MySQL56/70d613d1-678c-b18b-7704-64b6591be455:1-2646818,fb827d7a-696b-7ad1-988c-c77ff601c177:1-28' WHERE id=16" |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment