2022-12-26

Dec 26 21:09:17 r720-01 kernel: mps1: mpssas_prepare_remove: Sending reset for target ID 21
Dec 26 21:09:17 r720-01 kernel: da12 at mps1 bus 0 scbus8 target 21 lun 0
Dec 26 21:09:17 r720-01 kernel: da12: <ATA Samsung SSD 850 3B6Q>  s/n S3PTNF0JA70159T      detached
Dec 26 21:09:17 r720-01 kernel: mps1: No pending commands: starting remove_device
Dec 26 21:09:18 r720-01 kernel: (da12:mps1:0:21:0): Periph destroyed
Dec 26 21:09:18 r720-01 ZFS[67733]: vdev state changed, pool_guid=1975810868733347630 vdev_guid=8716406602783665762
Dec 26 21:09:18 r720-01 ZFS[67737]: vdev is removed, pool_guid=1975810868733347630 vdev_guid=8716406602783665762
Dec 26 21:09:19 r720-01 kernel: da12 at mps1 bus 0 scbus8 target 21 lun 0
Dec 26 21:09:19 r720-01 kernel: da12: <ATA Samsung SSD 850 3B6Q> Fixed Direct Access SPC-4 SCSI device
Dec 26 21:09:19 r720-01 kernel: da12: Serial Number S3PTNF0JA70159T     
Dec 26 21:09:19 r720-01 kernel: da12: 600.000MB/s transfers
Dec 26 21:09:19 r720-01 kernel: da12: Command Queueing enabled
Dec 26 21:09:19 r720-01 kernel: da12: 476940MB (976773168 512 byte sectors)
Dec 26 21:09:19 r720-01 kernel: da12: quirks=0x8<4K>

[r720-01 dan ~] % zpool status tank_fast                                                                                            21:16:57
  pool: tank_fast
 state: DEGRADED
status: One or more devices has been removed by the administrator.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Online the device using zpool online' or replace the device with
	'zpool replace'.
  scan: scrub repaired 0B in 00:11:09 with 0 errors on Mon Dec 26 03:33:45 2022
config:

	NAME        STATE     READ WRITE CKSUM
	tank_fast   DEGRADED     0     0     0
	  mirror-0  ONLINE       0     0     0
	    da14p1  ONLINE       0     0     0
	    da13p1  ONLINE       0     0     0
	  mirror-1  DEGRADED     0     0     0
	    da11p1  ONLINE       0     0     0
	    da12p1  REMOVED      0     0     0

errors: No known data errors
[r720-01 dan ~] %                                                                                                                   21:17:17


[r720-01 dan ~] % sudo zpool online tank_fast da12p1                                                                                21:17:51
[r720-01 dan ~] % zpool status tank_fast                                                                                            21:17:55
  pool: tank_fast
 state: ONLINE
  scan: resilvered 7.20M in 00:00:01 with 0 errors on Mon Dec 26 21:17:55 2022
config:

	NAME        STATE     READ WRITE CKSUM
	tank_fast   ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    da14p1  ONLINE       0     0     0
	    da13p1  ONLINE       0     0     0
	  mirror-1  ONLINE       0     0     0
	    da11p1  ONLINE       0     0     0
	    da12p1  ONLINE       0     0     0

errors: No known data errors
[r720-01 dan ~] %                                                                                                                   21:18:01

2022-12-30

4 days later

Dec 30 04:56:12 r720-01 kernel: mps1: Controller reported scsi ioc terminated tgt 22 SMID 1810 loginfo 31120303
Dec 30 04:56:12 r720-01 kernel: mps1: Controller reported scsi ioc terminated tgt 22 SMID 486 loginfo 31120303
Dec 30 04:56:12 r720-01 kernel: (da13:mps1:0:22:0): WRITE(10). CDB: 2a 00 15 ca b4 80 00 00 30 00 
Dec 30 04:56:12 r720-01 kernel: (da13:mps1:0:22:0): CAM status: CCB request completed with an error
Dec 30 04:56:12 r720-01 kernel: (da13:mps1:0:22:0): Retrying command, 3 more tries remain
Dec 30 04:56:12 r720-01 kernel: (da13:mps1:0:22:0): WRITE(10). CDB: 2a 00 15 ca b4 58 00 00 28 00 
Dec 30 04:56:12 r720-01 kernel: (da13:mps1:0:22:0): CAM status: CCB request completed with an error
Dec 30 04:56:12 r720-01 kernel: (da13:mps1:0:22:0): Retrying command, 3 more tries remain
Dec 30 04:56:13 r720-01 kernel: (da13:mps1:0:22:0): WRITE(10). CDB: 2a 00 15 ca b4 80 00 00 30 00 
Dec 30 04:56:13 r720-01 kernel: (da13:mps1:0:22:0): CAM status: SCSI Status Error
Dec 30 04:56:13 r720-01 kernel: (da13:mps1:0:22:0): SCSI status: Check Condition
Dec 30 04:56:13 r720-01 kernel: (da13:mps1:0:22:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Dec 30 04:56:13 r720-01 kernel: (da13:mps1:0:22:0): Retrying command (per sense data)

Followed by

Dec 26 21:17:54 r720-01 ZFS[71257]: vdev state changed, pool_guid=1975810868733347630 vdev_guid=8716406602783665762

Which is when I did the zpool online above.