Created
November 19, 2012 16:31
-
-
Save rlaager/4111654 to your computer and use it in GitHub Desktop.
ZFS Resilvering Errors
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| For the c2t5000C5002623694Bd0 disk, we received a SMART predictive failure | |
| (fault.io.disk.predictive-failure) warning as well as | |
| fault.io.scsi.cmd.disk.dev.rqs.derr from fmadm. | |
| We then set out to replace the disk. | |
| Here is the zpool status before the replacement: | |
| # zpool status data | |
| pool: data | |
| state: ONLINE | |
| status: One or more devices has experienced an unrecoverable error. An | |
| attempt was made to correct the error. Applications are unaffected. | |
| action: Determine if the device needs to be replaced, and clear the errors | |
| using 'zpool clear' or replace the device with 'zpool replace'. | |
| see: http://illumos.org/msg/ZFS-8000-9P | |
| scan: resilvered 39.2G in 0h25m with 0 errors on Tue Nov 6 15:33:40 2012 | |
| config: | |
| NAME STATE READ WRITE CKSUM | |
| data ONLINE 0 0 0 | |
| mirror-0 ONLINE 0 0 0 | |
| c2t5000C5002623E7CBd0 ONLINE 0 0 0 | |
| c2t5000C50026244A37d0 ONLINE 0 0 0 | |
| c2t5000C50026248A1Fd0 ONLINE 0 0 0 | |
| mirror-1 ONLINE 0 0 0 | |
| c2t5000C50026247B97d0 ONLINE 0 0 0 | |
| c2t5000C5002624928Bd0 ONLINE 0 0 0 | |
| c2t5000C50026248D6Bd0 ONLINE 0 0 0 | |
| mirror-2 ONLINE 0 0 0 | |
| c2t5000C50026249623d0 ONLINE 0 0 0 | |
| c2t5000C50026227A6Fd0 ONLINE 0 0 0 | |
| c2t5000C5002623694Bd0 ONLINE 0 0 6 | |
| mirror-3 ONLINE 0 0 0 | |
| c2t5000C50026221947d0 ONLINE 0 0 0 | |
| c2t5000C5002624926Bd0 ONLINE 0 0 0 | |
| c2t5000C50026227FB7d0 ONLINE 0 0 0 | |
| spares | |
| c2t5000C50026225677d0 AVAIL | |
| errors: No known data errors | |
| We then ran: | |
| # zpool replace data c2t5000C5002623694Bd0 c2t5000C5003C624DEBd0 | |
| As expected, that started resilvering: | |
| # zpool status data | |
| pool: data | |
| state: ONLINE | |
| status: One or more devices is currently being resilvered. The pool will | |
| continue to function, possibly in a degraded state. | |
| action: Wait for the resilver to complete. | |
| scan: resilver in progress since Mon Nov 19 09:55:58 2012 | |
| 315M scanned out of 717G at 2.44M/s, 83h32m to go | |
| 75.5M resilvered, 0.04% done | |
| config: | |
| NAME STATE READ WRITE CKSUM | |
| data ONLINE 0 0 0 | |
| mirror-0 ONLINE 0 0 0 | |
| c2t5000C5002623E7CBd0 ONLINE 0 0 0 | |
| c2t5000C50026244A37d0 ONLINE 0 0 0 | |
| c2t5000C50026248A1Fd0 ONLINE 0 0 0 | |
| mirror-1 ONLINE 0 0 0 | |
| c2t5000C50026247B97d0 ONLINE 0 0 0 | |
| c2t5000C5002624928Bd0 ONLINE 0 0 0 | |
| c2t5000C50026248D6Bd0 ONLINE 0 0 0 | |
| mirror-2 ONLINE 0 0 0 | |
| c2t5000C50026249623d0 ONLINE 0 0 0 | |
| c2t5000C50026227A6Fd0 ONLINE 0 0 0 | |
| replacing-2 ONLINE 0 0 0 | |
| c2t5000C5002623694Bd0 ONLINE 0 0 6 | |
| c2t5000C5003C624DEBd0 ONLINE 0 0 0 (resilvering) | |
| mirror-3 ONLINE 0 0 0 | |
| c2t5000C50026221947d0 ONLINE 0 0 0 | |
| c2t5000C5002624926Bd0 ONLINE 0 0 0 | |
| c2t5000C50026227FB7d0 ONLINE 0 0 0 | |
| spares | |
| c2t5000C50026225677d0 AVAIL | |
| errors: No known data errors | |
| After a little while, something seems to have gone very wrong: | |
| # zpool status data | |
| pool: data | |
| state: DEGRADED | |
| status: One or more devices is currently being resilvered. The pool will | |
| continue to function, possibly in a degraded state. | |
| action: Wait for the resilver to complete. | |
| scan: resilver in progress since Mon Nov 19 10:00:24 2012 | |
| 19.4G scanned out of 717G at 12.8M/s, 15h32m to go | |
| 9.76G resilvered, 2.71% done | |
| config: | |
| NAME STATE READ WRITE CKSUM | |
| data DEGRADED 0 0 0 | |
| mirror-0 ONLINE 0 0 0 | |
| c2t5000C5002623E7CBd0 ONLINE 0 0 0 | |
| c2t5000C50026244A37d0 ONLINE 0 0 0 | |
| c2t5000C50026248A1Fd0 ONLINE 0 0 0 | |
| mirror-1 ONLINE 0 0 0 | |
| c2t5000C50026247B97d0 ONLINE 0 0 0 | |
| c2t5000C5002624928Bd0 ONLINE 0 0 0 | |
| c2t5000C50026248D6Bd0 ONLINE 0 0 0 | |
| mirror-2 DEGRADED 0 0 0 | |
| c2t5000C50026249623d0 ONLINE 0 0 0 | |
| c2t5000C50026227A6Fd0 ONLINE 0 0 0 | |
| replacing-2 DEGRADED 0 0 16 | |
| spare-0 DEGRADED 0 0 4 | |
| c2t5000C5002623694Bd0 DEGRADED 0 0 26 too many errors (resilvering) | |
| c2t5000C50026225677d0 ONLINE 0 0 14 (resilvering) | |
| c2t5000C5003C624DEBd0 ONLINE 0 0 24 (resilvering) | |
| mirror-3 ONLINE 0 0 0 | |
| c2t5000C50026221947d0 ONLINE 0 0 0 | |
| c2t5000C5002624926Bd0 ONLINE 0 0 0 | |
| c2t5000C50026227FB7d0 ONLINE 0 0 0 | |
| spares | |
| c2t5000C50026225677d0 INUSE currently in use |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment