Last active
February 6, 2025 15:41
-
-
Save bobpaul/ab1a3fcfce58e25c327e8e74995019c7 to your computer and use it in GitHub Desktop.
btrfs scrub failing on 2 devices in 5device "mirror"
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# btrfs fi show /mnt/btrfs/ | |
Label: 'array-root' uuid: cce741b6-b864-4e9c-bccf-057d93694bb2 | |
Total devices 5 FS bytes used 10.50TiB | |
devid 1 size 3.64TiB used 3.32TiB path /dev/bcache3 | |
devid 4 size 3.64TiB used 3.32TiB path /dev/bcache0 | |
devid 5 size 12.73TiB used 10.03TiB path /dev/bcache1 | |
devid 6 size 3.64TiB used 3.32TiB path /dev/bcache2 | |
devid 7 size 2.73TiB used 2.41TiB path /dev/bcache4 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# # no problems were found, nothing was changed: | |
# | |
# btrfs rescue super-recover -v /dev/bcache0 | |
All Devices: | |
Device: id = 5, name = /dev/bcache1 | |
Device: id = 1, name = /dev/bcache3 | |
Device: id = 6, name = /dev/bcache2 | |
Device: id = 7, name = /dev/bcache4 | |
Device: id = 4, name = /dev/bcache0 | |
Before Recovering: | |
[All good supers]: | |
device name = /dev/bcache1 | |
superblock bytenr = 65536 | |
device name = /dev/bcache1 | |
superblock bytenr = 67108864 | |
device name = /dev/bcache1 | |
superblock bytenr = 274877906944 | |
device name = /dev/bcache3 | |
superblock bytenr = 65536 | |
device name = /dev/bcache3 | |
superblock bytenr = 67108864 | |
device name = /dev/bcache3 | |
superblock bytenr = 274877906944 | |
device name = /dev/bcache2 | |
superblock bytenr = 65536 | |
device name = /dev/bcache2 | |
superblock bytenr = 67108864 | |
device name = /dev/bcache2 | |
superblock bytenr = 274877906944 | |
device name = /dev/bcache4 | |
superblock bytenr = 65536 | |
device name = /dev/bcache4 | |
superblock bytenr = 67108864 | |
device name = /dev/bcache4 | |
superblock bytenr = 274877906944 | |
device name = /dev/bcache0 | |
superblock bytenr = 65536 | |
device name = /dev/bcache0 | |
superblock bytenr = 67108864 | |
device name = /dev/bcache0 | |
superblock bytenr = 274877906944 | |
[All bad supers]: | |
All supers are valid, no need to recover |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# btrfs check --readonly /dev/bcache4 | |
[1/8] checking log skipped (none written) | |
[2/8] checking root items | |
[3/8] checking extents | |
[4/8] checking free space cache | |
[5/8] checking fs roots | |
[6/8] checking only csums items (without verifying data) | |
[7/8] checking root refs | |
[8/8] checking quota groups skipped (not enabled on this FS) | |
Opening filesystem to check... | |
Checking filesystem on /dev/bcache4 | |
UUID: cce741b6-b864-4e9c-bccf-057d93694bb2 | |
found 11550182076419 bytes used, no error found | |
total csum bytes: 11248213256 | |
total tree bytes: 21745098752 | |
total fs tree bytes: 7784570880 | |
total extent tree bytes: 1561251840 | |
btree space waste bytes: 2209065715 | |
file data blocks allocated: 12056932474880 | |
referenced 11767166226432 | |
# |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# btrfs check --readonly --check-data-csum /dev/bcache0 | |
Opening filesystem to check... | |
Checking filesystem on /dev/bcache0 | |
UUID: cce741b6-b864-4e9c-bccf-057d93694bb2 | |
[1/8] checking log skipped (none written) | |
[2/8] checking root items | |
[3/8] checking extents | |
[4/8] checking free space cache | |
[5/8] checking fs roots | |
[6/8] checking csums against data | |
[7/8] checking root refs | |
[8/8] checking quota groups skipped (not enabled on this FS) | |
found 11550182072324 bytes used, no error found | |
total csum bytes: 11248213256 | |
total tree bytes: 21745098752 | |
total fs tree bytes: 7784570880 | |
total extent tree bytes: 1561251840 | |
btree space waste bytes: 2209065715 | |
file data blocks allocated: 12056932474880 | |
referenced 11767166226432 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# btrfs scrub start -BrdR /mnt/btrfs/ Starting scrub on devid 1 | |
Starting scrub on devid 4 | |
Starting scrub on devid 5 | |
Starting scrub on devid 6 | |
Starting scrub on devid 7 | |
ERROR: scrubbing /mnt/btrfs/ failed for device id 1: ret=-1, errno=5 (Input/output error) ERROR: scrubbing /mnt/btrfs/ failed for device id 5: ret=-1, errno=5 (Input/output error) | |
Scrub device /dev/bcache3 (id 1) canceled | |
Scrub started: Tue Jan 28 19:08:08 2025 | |
Status: aborted | |
Duration: 0:03:11 | |
data_extents_scrubbed: 0 | |
tree_extents_scrubbed: 1209000 | |
data_bytes_scrubbed: 0 | |
tree_bytes_scrubbed: 4952064000 | |
read_errors: 0 | |
csum_errors: 0 | |
verify_errors: 0 | |
no_csum: 0 | |
csum_discards: 0 | |
super_errors: 0 | |
malloc_errors: 0 | |
uncorrectable_errors: 0 | |
unverified_errors: 0 | |
corrected_errors: 0 | |
last_physical: 6171262976 | |
Scrub device /dev/bcache0 (id 4) done | |
Scrub started: Tue Jan 28 19:08:08 2025 | |
Status: finished | |
Duration: 7:23:24 | |
data_extents_scrubbed: 57117871 | |
tree_extents_scrubbed: 258621 | |
data_bytes_scrubbed: 3399622529024 | |
tree_bytes_scrubbed: 1059311616 | |
read_errors: 0 | |
csum_errors: 0 | |
verify_errors: 0 | |
no_csum: 1451224 | |
csum_discards: 0 | |
super_errors: 0 | |
malloc_errors: 0 | |
uncorrectable_errors: 0 | |
unverified_errors: 0 | |
corrected_errors: 0 | |
last_physical: 3991132307456 | |
Scrub device /dev/bcache1 (id 5) canceled | |
Scrub started: Tue Jan 28 19:08:08 2025 | |
Status: aborted | |
Duration: 0:02:32 | |
data_extents_scrubbed: 0 | |
tree_extents_scrubbed: 953596 | |
data_bytes_scrubbed: 0 | |
tree_bytes_scrubbed: 3905929216 | |
read_errors: 0 | |
csum_errors: 0 | |
verify_errors: 0 | |
no_csum: 0 | |
csum_discards: 0 | |
super_errors: 0 | |
malloc_errors: 0 | |
uncorrectable_errors: 0 | |
unverified_errors: 0 | |
corrected_errors: 0 | |
last_physical: 4090888192 | |
Scrub device /dev/bcache2 (id 6) done | |
Scrub started: Tue Jan 28 19:08:08 2025 | |
Status: finished | |
Duration: 6:55:00 | |
data_extents_scrubbed: 54834490 | |
tree_extents_scrubbed: 2295868 | |
data_bytes_scrubbed: 3421023117312 | |
tree_bytes_scrubbed: 9403875328 | |
read_errors: 0 | |
csum_errors: 0 | |
verify_errors: 0 | |
no_csum: 449898 | |
csum_discards: 0 | |
super_errors: 0 | |
malloc_errors: 0 | |
uncorrectable_errors: 0 | |
unverified_errors: 0 | |
corrected_errors: 0 | |
last_physical: 3991163305984 | |
Scrub device /dev/bcache4 (id 7) done | |
Scrub started: Tue Jan 28 19:08:08 2025 | |
Status: finished | |
Duration: 8:56:13 | |
data_extents_scrubbed: 38141401 | |
tree_extents_scrubbed: 505577 | |
data_bytes_scrubbed: 2363512324096 | |
tree_bytes_scrubbed: 2070843392 | |
read_errors: 0 | |
csum_errors: 0 | |
verify_errors: 0 | |
no_csum: 55206 | |
csum_discards: 0 | |
super_errors: 0 | |
malloc_errors: 0 | |
uncorrectable_errors: 0 | |
unverified_errors: 0 | |
corrected_errors: 0 | |
last_physical: 2991438626816 | |
# btrfs scrub status /mnt/btrfs/ | |
UUID: cce741b6-b864-4e9c-bccf-057d93694bb2 | |
Scrub started: Tue Jan 28 19:08:08 2025 | |
Status: aborted | |
Duration: 8:56:13 | |
Total to scrub: 21.01TiB | |
Rate: 272.87MiB/s | |
Error summary: no errors found | |
# #... I do find this quite misleading. Personally, I'd categorize this as a "usability bug". I understand that no | |
# # CHECKSUM errors were found, but certainly a structural error WAS found. | |
# # It seems like a non-zero exit status and an "Error summary" like "filesystem damaged, could not complete" would be good. | |
# # | |
# # I wasted time trying to figure out "what" was aborting the scrub, but the "no errors" left me rather non-plussed. | |
# # The invididual disk results (devid 1 and 5) are especially misleading while the scrub is still in progress. | |
# # while dmesg showed one had aborted, both stopped and immediately jumped to "100%" with a status of "running (history)". | |
# # No idication while watching scrub status for the scrub in progress that anything bad had happed. It certainly would have been | |
# # better if `btrfs scrub status -d` had shown 2 disks with a status of "aborted" while the other 3 were running. And again, | |
# # something other than "no errors" I think would have been appropriate... /soapbox |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[16558.355437] BTRFS info (device bcache3): first mount of filesystem cce741b6-b864-4e9c-bccf-057d93694bb2 | |
[16558.355454] BTRFS info (device bcache3): using crc32c (crc32c-intel) checksum algorithm | |
[16558.355462] BTRFS info (device bcache3): disk space caching is enabled | |
[16558.355465] BTRFS warning (device bcache3): space cache v1 is being deprecated and will be removed in a future release, please use -o space_cache=v2 | |
[16561.894309] BTRFS info (device bcache3): bdev /dev/bcache3 errs: wr 0, rd 94, flush 0, corrupt 9320, gen 0 | |
[16561.894323] BTRFS info (device bcache3): bdev /dev/bcache0 errs: wr 4, rd 56, flush 0, corrupt 21915, gen 0 | |
[16561.894328] BTRFS info (device bcache3): bdev /dev/bcache1 errs: wr 59, rd 1, flush 0, corrupt 17, gen 0 | |
[16561.894333] BTRFS info (device bcache3): bdev /dev/bcache2 errs: wr 1, rd 108, flush 1, corrupt 8078, gen 0 | |
[16561.894338] BTRFS info (device bcache3): bdev /dev/bcache4 errs: wr 524248893, rd 262954412, flush 1663, corrupt 9506114, gen 68115 | |
[16662.992224] BTRFS info (device bcache3): scrub: started on devid 4 | |
[16663.021845] BTRFS info (device bcache3): scrub: started on devid 5 | |
[16663.062588] BTRFS info (device bcache3): scrub: started on devid 1 | |
[16663.105192] BTRFS info (device bcache3): scrub: started on devid 7 | |
[16663.152500] BTRFS info (device bcache3): scrub: started on devid 6 | |
[16815.260152] BTRFS error (device bcache3): tree first key mismatch detected, bytenr=67541964967936 parent_transid=3164351 key expected=(67541791637504,168,17179873280) has=(67541791637504,168,4096) | |
[16815.262238] BTRFS error (device bcache3): tree first key mismatch detected, bytenr=67541964967936 parent_transid=3164351 key expected=(67541791637504,168,17179873280) has=(67541791637504,168,4096) | |
[16815.265664] BTRFS info (device bcache3): scrub: not finished on devid 5 with status: -5 | |
[16854.139103] BTRFS error (device bcache3): tree first key mismatch detected, bytenr=67541964967936 parent_transid=3164351 key expected=(67541791637504,168,17179873280) has=(67541791637504,168,4096) | |
[16854.143591] BTRFS error (device bcache3): tree first key mismatch detected, bytenr=67541964967936 parent_transid=3164351 key expected=(67541791637504,168,17179873280) has=(67541791637504,168,4096) | |
[16854.147465] BTRFS info (device bcache3): scrub: not finished on devid 1 with status: -5 | |
[23781.686897] perf: interrupt took too long (2526 > 2500), lowering kernel.perf_event_max_sample_rate to 78900 | |
[28342.160537] perf: interrupt took too long (3160 > 3157), lowering kernel.perf_event_max_sample_rate to 63000 | |
[41563.234606] BTRFS info (device bcache3): scrub: finished on devid 6 with status: 0 | |
[41740.662162] perf: interrupt took too long (3961 > 3950), lowering kernel.perf_event_max_sample_rate to 50400 | |
[43267.054362] BTRFS info (device bcache3): scrub: finished on devid 4 with status: 0 | |
[48836.426233] BTRFS info (device bcache3): scrub: finished on devid 7 with status: 0 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Tried
chunk-recover
with the intention of pressingn
on all questions, but was never asked. The output was longer than my scrollback buffer in tmux, but this is what I got:btrfs rescue chunk-recover -v /dev/bcache1 ... Chunk tree recovery failed