Benjamin Hunter benhenryhunter

Overview

The bloXroute blockchain distribution network (BDN) encountered errors propagating partial blocks via p2p to Lighthouse nodes which caused all HTTP submissions with full payloads to be treated as duplicates. Due to this treatment many beacon nodes were in a state of waiting for blobs to be shared over p2p while blobs were available during HTTP submissions. The Lighthouse and bloXroute teams worked very closely together to identify the full scope of this problem and have provided more information below along with next steps for both bloXroute and Lighthouse teams.

Background

The bloXroute Blockchain Distribution Network (BDN) is heavily integrated with Lighthouse due to the high performance of Lighthouse. Out of this heavy reliance, specific BDN behavior uncovered some behavior in Lighthouse which caused block propagation errors during p2p gossip of blocks without p2p provision of blobs. In order to optimize global propagation time the BDN would gossip blocks over p2p to beacon node peers with

On March 27-28 the Ethereum network suffered from extremely high rate of miss slots. Most of these slots were first relayed from the bloXroute relays. We identified that the bloXroute relays worked properly throughout the incident, publishing blocks and blobs correctly, however they propagated the blocks fast thru the BDN while the blobs sidecar propagated through the p2p more slowly (the sidecar is expected to propagate slower, and is allowed to be accepted until t=8 sec) this uncovered a specific CL behavior which caused clients to reject these blocks and cause missed slots. In the current Lighthouse version, the node is expecting the peer that first provided the block to also provide the blobs. The BDN does not propagate blobs and that caused the BDN connected consensus nodes to ignore blocks that were first received from the BDN. A recent release of the BDN improved the speed of gossiped blocks without blobs, relying on the rest of the p2p network to propagate blobs as needed which caused the significa

Overview

A bug caused by Implicit memory aliasing in go’s for loops caused the blob commitment values to be all set to the last value during encoding from go struct to a gRPC encoded value. This bug surfaced when a block that was received via a gRPC submission built using the erroneous code was the top or selected bid. The error seen would be mismatched KZG commitment.

Impact

A total of 16 slots were missed causing about 3.21 ETH missed block values.

All missed slots can be found here: https://gist.github.com/benhenryhunter/6feed90aff1a93106d629fe85a88d778

We released a fix to our regional relays and saw the bug resolved from our end but found out builders who previously did not use our library for gRPC began to use it. This caused a few more blocks to slip through requiring us to shut off getHeaders for the remainder of the night on the 14th.

[
{
"slot": "8366820",
"parent_hash": "0x27dcb717d5921af93e2b56a82f546e1c11b33619238510cbe36e4d8c43f446a8",
"block_hash": "0x6868f8e474a163bb17f39d05847aa8e3f1b38db3cca0dfc9bd139db1da8dcde2",
"builder_pubkey": "0x8e6df6e0a9ca3fd89db2aa2f3daf77722dc4fbcd15e285ed7d9560fdf07b7d69ba504add4cc12ac999b8094ff30ed06c",
"proposer_pubkey": "0xb6d5424e28a738d002c96a19db7434fff22877272649e0ca38b579bb44398f3977f43af6c055414b7a71ec2bd7cb8480",
"proposer_fee_recipient": "0x72FDdC41CA177551Ce5949C75a92b945eFa04141",
"gas_limit": "30000000",

	[package]
	name = "serialize-gist"
	version = "0.1.0"
	edition = "2021"

	[dependencies]
	reqwest = { version = "0.12.5", features = ["json"] }
	serde = { version = "1.0", features = ["derive"] }
	serde_json = "1.0"
	mev-rs = { git = "https://github.com/ralexstokes/mev-rs", tag = "v0.4.0-alpha.6" }

	[
	{
	"slot": "8628587",
	"parent_hash": "0x4560468beeb3df67f7a68ee3be4bedbddf21ad70e34213d2b3c78b988ef96b64",
	"block_hash": "0x341f9d31fbd734ea5660de29ebd65e16996c89296049435388d2828edbf8c9d1",
	"builder_pubkey": "0xb26f96664274e15fb6fcda862302e47de7e0e2a6687f8349327a9846043e42596ec44af676126e2cacbdd181f548e681",
	"proposer_pubkey": "0xa99354d640de4482120cd1bb48df1c186a4d6f410ef6e23fa0031ab107a0c350746437f1f6b934c9b8eba282054a5783",
	"proposer_fee_recipient": "0x2ee5656c075bc756888C5a271FEd2386d55431E7",
	"gas_limit": "30000000",
	"gas_used": "13159575",

	func TraceExample(rootContext context.Context) {
	parentSpan := trace.SpanFromContext(rootContext)

	rootContextWithParentSpan = trace.ContextWithSpan(context.Background(), parentSpan)
	traceFunctionContext, traceFunctionSpan := s.tracer.Start(rootContextWithParentSpan, "traceFunction")

	childContext, childSpan := s.tracer.Start(traceFunctionContext, "childTrace")

	_, grandchildSpan := s.tracer.Start(childContext, "grandchildTrace")
	grandchildSpan.End()

	package main

	import (
	"context"
	"fmt"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"
	"google.golang.org/grpc/metadata"

	package main

	import (
	"context"
	"fmt"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"
	"google.golang.org/grpc/metadata"