A bug caused by Implicit memory aliasing in go’s for loops caused the blob commitment values to be all set to the last value during encoding from go struct to a gRPC encoded value. This bug surfaced when a block that was received via a gRPC submission built using the erroneous code was the top or selected bid. The error seen would be mismatched KZG commitment.
A total of 16 slots were missed causing about 3.21 ETH missed block values.
All missed slots can be found here: https://gist.github.com/benhenryhunter/6feed90aff1a93106d629fe85a88d778
We released a fix to our regional relays and saw the bug resolved from our end but found out builders who previously did not use our library for gRPC began to use it. This caused a few more blocks to slip through requiring us to shut off getHeaders for the remainder of the night on the 14th.