On March 27-28 the Ethereum network suffered from extremely high rate of miss slots. Most of these slots were first relayed from the bloXroute relays. We identified that the bloXroute relays worked properly throughout the incident, publishing blocks and blobs correctly, however they propagated the blocks fast thru the BDN while the blobs sidecar propagated through the p2p more slowly (the sidecar is expected to propagate slower, and is allowed to be accepted until t=8 sec) this uncovered a specific CL behavior which caused clients to reject these blocks and cause missed slots. In the current Lighthouse version, the node is expecting the peer that first provided the block to also provide the blobs. The BDN does not propagate blobs and that caused the BDN connected consensus nodes to ignore blocks that were first received from the BDN. A recent release of the BDN improved the speed of gossiped blocks without blobs, relying on the rest of the p2p network to propagate blobs as needed which caused the significant increase of the missed slots. The BDN relies heavily on Lighthouse, which makes up the majority of our beacon nodes at bloXroute, due to its performance and speed. Post release we witnessed successful block propagation through our BDN and made the assumption this release was valid. This also showcased itself mainly on the bloXroute relays due to their tight coupling with the BDN. The BDNs speed of providing the beacon nodes with the block caused this behavior even in scenarios where other relays were publishing blocks that bloXroute did not have.
Throughout this time the bloXroute relays were providing blocks with blobs back to validators and also publishing blocks with blobs to our BDN and to our network of beacon nodes. These publish requests would return a 202 response due to the beacon nodes already seeing that block from the BDN.
This issue was able to be resolved after a series of tests were done isolating this issue to lighthouse’s behavior after seeing a block first through the BDN and then slowly migrating our relay away from using the BDN for block publishing and then disabling the BDN’s block propagation of any blocks containing blobs.
We had confirmed multiple different times that the blobs were included on all beacon node publishing via HTTP but the beacon nodes had all seen the blocks* from p2p first via the BDN so every response was a 202 due to already being seen and the beacon nodes would not use those requests containing the blobs.