Crucible hang

gistfile1.txt

Here is the tail:

```
1685/1800 Read  block 3483  len 20480
1686/1800 Read  block 4648  len 28672
1687/1800 Write block  736  len 16384  data:  13  14  15  14
{"msg":" 26009 final dependency list []","v":0,"name":"crucible","level":30,"time":"2023-12-05T17:18:23.264983965Z","hostname":"EVT22200005","pid":8755,"client":"2","":"downstairs","session_id":"a7341b3f-853d-4687-a011-8b09ff9301ea"}
1688/1800 Write block 1746  len 32768  data:  13  13  14  13  14  13   8   8
{"msg":" 26010 final dependency list []","v":0,"name":"crucible","level":30,"time":"2023-12-05T17:18:23.265105144Z","hostname":"EVT22200005","pid":8755,"client":"2","":"downstairs","session_id":"a7341b3f-853d-4687-a011-8b09ff9301ea"}
1689/1800 Write block 2384  len 28672  data:  10  13  11   9   8   9  15
{"msg":"got repair ok in Closing { close_id: JobId(26003), repair_id: JobId(26004), noop_id: JobId(26005), reopen_id: JobId(26006), gw_repair_id: 25005, gw_noop_id: 25006 }","v":0,"name":"crucible","level":30,"time":"2023-12-05T17:18:23.265267511Z","hostname":"EVT22200005","pid":8755,"":"downstairs","":"downstairs","session_id":"a7341b3f-853d-4687-a011-8b09ff9301ea"}
{"msg":"Repair for extent 21 s:1 d:[ClientId(2)]","v":0,"name":"crucible","level":30,"time":"2023-12-05T17:18:23.265287769Z","hostname":"EVT22200005","pid":8755,"":"downstairs","":"downstairs","session_id":"a7341b3f-853d-4687-a011-8b09ff9301ea"}
{"msg":"RE:21 Wait for result from repair command 26004:25005","v":0,"name":"crucible","level":30,"time":"2023-12-05T17:18:23.265294432Z","hostname":"EVT22200005","pid":8755,"":"downstairs","":"downstairs","session_id":"a7341b3f-853d-4687-a011-8b09ff9301ea"}
{"msg":" 26004 final dependency list [JobId(26003)]","v":0,"name":"crucible","level":30,"time":"2023-12-05T17:18:23.265303068Z","hostname":"EVT22200005","pid":8755,"client":"2","":"downstairs","session_id":"a7341b3f-853d-4687-a011-8b09ff9301ea"}
1690/1800 Read  block 4210  len 16384
1691/1800 Read  block 3466  len 36864
1692/1800 Read  block 2193  len 32768
{"msg":"inserting repair IDs ExtentRepairIDs { close_id: JobId(26014), repair_id: JobId(26015), noop_id: JobId(26016), reopen_id: JobId(26017) }; got dep []","v":0,"name":"crucible","level":30,"time":"2023-12-05T17:18:23.266054128Z","hostname":"EVT22200005","pid":8755,"":"downstairs","":"downstairs","session_id":"a7341b3f-853d-4687-a011-8b09ff9301ea"}
{"msg":" 26018 final dependency list [JobId(26006), JobId(26017)]","v":0,"name":"crucible","level":30,"time":"2023-12-05T17:18:23.266070289Z","hostname":"EVT22200005","pid":8755,"client":"2","":"downstairs","session_id":"a7341b3f-853d-4687-a011-8b09ff9301ea"}
{"msg":"got repair ok in Repairing { repair_id: JobId(26004), noop_id: JobId(26005), reopen_id: JobId(26006), gw_noop_id: 25006 }","v":0,"name":"crucible","level":30,"time":"2023-12-05T17:18:23.311727854Z","hostname":"EVT22200005","pid":8755,"":"downstairs","":"downstairs","session_id":"a7341b3f-853d-4687-a011-8b09ff9301ea"}
{"msg":"RE:21 Wait for result from NoOp command 26005:25006","v":0,"name":"crucible","level":30,"time":"2023-12-05T17:18:23.311752851Z","hostname":"EVT22200005","pid":8755,"":"downstairs","":"downstairs","session_id":"a7341b3f-853d-4687-a011-8b09ff9301ea"}
{"msg":" 26005 final dependency list [JobId(26004)]","v":0,"name":"crucible","level":30,"time":"2023-12-05T17:18:23.311771607Z","hostname":"EVT22200005","pid":8755,"client":"2","":"downstairs","session_id":"a7341b3f-853d-4687-a011-8b09ff9301ea"}
{"msg":"got repair ok in Noop { noop_id: JobId(26005), reopen_id: JobId(26006) }","v":0,"name":"crucible","level":30,"time":"2023-12-05T17:18:23.311855786Z","hostname":"EVT22200005","pid":8755,"":"downstairs","":"downstairs","session_id":"a7341b3f-853d-4687-a011-8b09ff9301ea"}
{"msg":"RE:21 Wait for result from reopen command 26006","v":0,"name":"crucible","level":30,"time":"2023-12-05T17:18:23.311880152Z","hostname":"EVT22200005","pid":8755,"":"downstairs","":"downstairs","session_id":"a7341b3f-853d-4687-a011-8b09ff9301ea"}
{"msg":"got repair ok in Reopening { reopen_id: JobId(26006) }","v":0,"name":"crucible","level":30,"time":"2023-12-05T17:18:23.311955254Z","hostname":"EVT22200005","pid":8755,"":"downstairs","":"downstairs","session_id":"a7341b3f-853d-4687-a011-8b09ff9301ea"}
{"msg":"RE:22 repair extent with ids 26019,26020,26021,26022 deps:[JobId(26017), JobId(26018)]","v":0,"name":"crucible","level":30,"time":"2023-12-05T17:18:23.311977145Z","hostname":"EVT22200005","pid":8755,"":"downstairs","":"downstairs","session_id":"a7341b3f-853d-4687-a011-8b09ff9301ea"}
{"msg":" 26019 final dependency list [JobId(26017), JobId(26018)]","v":0,"name":"crucible","level":30,"time":"2023-12-05T17:18:23.311984229Z","hostname":"EVT22200005","pid":8755,"client":"2","":"downstairs","session_id":"a7341b3f-853d-4687-a011-8b09ff9301ea"}
{"msg":" 26022 final dependency list [JobId(26021)]","v":0,"name":"crucible","level":30,"time":"2023-12-05T17:18:23.31199547Z","hostname":"EVT22200005","pid":8755,"client":"2","":"downstairs","session_id":"a7341b3f-853d-4687-a011-8b09ff9301ea"}
{"msg":" 26023 final dependency list [JobId(26006), JobId(26009), JobId(26010), JobId(26018), JobId(26022)]","v":0,"name":"crucible","level":30,"time":"2023-12-05T17:18:23.577365845Z","hostname":"EVT22200005","pid":8755,"client":"2","":"downstairs","session_id":"a7341b3f-853d-4687-a011-8b09ff9301ea"}
```

```
       DS STATE 0        DS STATE 1        DS STATE 2    UPW   DSW BAKPR   NEW0  NEW1  NEW2    IP0   IP1   IP2     D0    D1    D2     S0    S1    S2
           active            active       live_repair      4    15     0      0     0     0      4     4     4     11    11     6      0     0     5
           active            active       live_repair      4    15     0      0     0     0      4     4     4     11    11     6      0     0     5
```