recover veracrypt zfs pool after failed usb cable, without reboot

zfs-veracrypt-failed-usb-cable.txt

********************************************************************************
WARNING THESE ARE MY PARTICULAR CASE NOTES MADE PUBLIC SO I CAN EASILY FIND THEM
DO NOT BLINDLY PASTE WITHOUT UNDERSTANDING WHAT DO THESE COMMANDS DO
********************************************************************************



dmsetup table | grep veracrypt
veracrypt2: 0 3904923136 crypt aes-xts-plain64 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 256 8:34 256
dmsetup reload veracrypt2 --table "0 3904923136 error"
dmsetup suspend --noflush veracrypt2
dmsetup resume veracrypt2
(?)zpool clear backup-usb
remove from veracrypt UI
echo "0 3904923136 zero" | dmsetup reload veracrypt2
dmsetup suspend --noflush veracrypt2
dmsetup resume veracrypt2
dmsetup remove veracrypt2
re-setup dm table from veracrypt UI
zpool clear backup-usb
scrub will start automatically
errors will be found
export pool
reimport
scrub again

final working version:

# cat recover.sh
cd /dev/mapper
mv usb veracrypt2
dmsetup table
dmsetup info veracrypt2
#"Open count:        1" - we still have our hands tied at this stage
dmsetup suspend --noflush --nolockfs veracrypt2
dmsetup reload veracrypt2 --table "0 3904923136 error"
dmsetup resume veracrypt2
zpool clear backup-usb
dmsetup info veracrypt2
#You notice that "Open count:" is now 0 - hooray, we are free!
veracrypt -t -d /dev/disk/by-id/ata-ST2000DM001-1ER164_W4Z3NLD4-part2
ls -l /dev/$(dmesg -T | grep -e 'sd.:'|tail -1 | awk '{print $NF}') | awk '{print $5,$6}'|sed 's/,//g'|cut -d" " --output-delimiter=":" -f 1-2
read -s -n 1 -p "Press enter to continue, CTRL-C to cancel"
#start next line with empty space, bash will not remember it on history
 veracrypt --text -k "" --pim=0  --protect-hidden=no -p 'your password' --filesystem=none --slot=2 /dev/disk/by-id/ata-ST2000DM001-1ER164_W4Z3NLD4-part2 && mv veracrypt2 usb
zpool clear backup-usb
zpool clear backup-usb
#MUST BE RUN TWICE, it will sync changes from ram to disk and errors will go away!

what I recently did for simple dm-linear instead of veracrypt:

replace

veracrypt -t -d /dev/disk/by-id/ata-ST2000DM001-1ER164_W4Z3NLD4-part2

with dmsetup remove table-name

and

veracrypt --text -k "" --pim=0  --protect-hidden=no -p 'your password' --filesystem=none --slot=2 /dev/disk/by-id/ata-ST2000DM001-1ER164_W4Z3NLD4-part2 && mv veracrypt2 usb

with dmsetup create table (make sure it's a different name) then cd /dev/mapper, ln -s /dev/(the newly created)dm-X old-table-name

DON'T FORGET TO SCRUB TWICE in case of errors, I can't find the comment in some zfs issue where a contributor stated that.

scrub twice to make the error status go away openzfs/zfs#9705 (comment)

How do you force the zfs to use devmapper virtual device instead of real device?

when you import, you direct it to import from /dev/mapper
zpool import -d /dev/mapper