Skip to content

Instantly share code, notes, and snippets.

@hi-ogawa

hi-ogawa/README-1-go-ethereum.md

Last active September 25, 2023 16:24
Show Gist options
  • Save hi-ogawa/3215def047fd933cf395d94f399b8377 to your computer and use it in GitHub Desktop.
Save hi-ogawa/3215def047fd933cf395d94f399b8377 to your computer and use it in GitHub Desktop.
Download ZIP
Ethereum ecosystem
Raw
README-1-go-ethereum.md

go-ethereum

Summary

  • debugger setup
  • main entrypoint
  • rpc handler
  • p2p/eth66 handler
  • new block execution
  • evm
    • byte code
    • accessible chain data (runtime environment)
  • (web3 compatible) contract code generation (see README-2-vyper.md)
    • vyper
    • constructor
    • transaction input
  • data persistence
    • key/value schema (cf. core/rawdb/schema.go)
    • database and trie implementation
  • eth2 PoS (see README-3-lighthouse.md)

references

misc

building

go build ./cmd/geth

./geth --goerli console

# e.g. to attach to a full node running in debugger
./geth --goerli attach
> web3.eth.getTransaction("0x14f2e5157d01ccb2d939e659ce9e6c2336327c467ad6b932f06c0c36271c54fb")

testing

go test ./cmd/geth -v -run TestConsoleWelcome

debugging

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "geth goerli",
      "type": "go",
      "request": "launch",
      "mode": "auto",
      "program": "${workspaceFolder}/cmd/geth",
      "args": ["--goerli"]
    },
    {
      "name": "geth ropsten",
      "type": "go",
      "request": "launch",
      "mode": "auto",
      "program": "${workspaceFolder}/cmd/geth",
      "args": ["--ropsten"]
    },
    {
      "name": "geth",
      "type": "go",
      "request": "launch",
      "mode": "auto",
      "program": "${workspaceFolder}/cmd/geth",
      "args": []
    }
  ]
}

components

#
# data structure
#

node.Node
  p2p.Server
    []p2p.Protocol (e.g. eth66)
    newTransport (newRLPX)
    enode.FairMix
    dialScheduler
  rpc.Server (IPC rpc handler (aka inprocHandler))
    serviceRegistry
      map[string]service (reflected object/method of e.g. PublicTransactionPoolAPI)
  httpServer (implements http.Handler)
    http.Server (go stdlib)
    rpcHandler (implements http.Handler)
      rpc.Server
  []Lifecycle (eth.Ethereum is registerd here)


eth.Ethereum
  EthAPIBackend (implements ethapi.Backend)
  p2p.Server (references node.Node's p2p.Server)
  ethdb.Database
  core.TxPool
  core.BlockChain
    ...
  consensus.Engine
  handler (handling eth66 logic)
    ...


handler
  chainSyncer
  Downloader (aka bodyQueue)
    peerSet
    queue
  BlockFetcher
  TxFetcher


core.BlockChain
  BlockValidator
  StateProcessor


StateDB

main entrypoint

main =>
  geth (as default cli) =>
    makeFullNode =>
      makeConfigNode =>
        node.New (aka stack) =>
          Node
          rpc.NewServer (for default inProcHandler) => RPCService
          p2p.Server
          newHTTPServer
      utils.RegisterEthService =>
        eth.New (construct `Ethereum`) =>
          Node.OpenDatabaseWithFreezer (use "chaindata" directory) =>
            rawdb.NewLevelDBDatabaseWithFreezer =>
              leveldb.New
              NewDatabaseWithFreezer
          core.SetupGenesisBlockWithOverride
          ethconfig.CreateConsensusEngine =>
            ethash.New (clique.New on goerli)
            beacon.New (wraps Ethash)
          core.NewBlockChain =>
            NewBlockValidator
            NewStateProcessor
            state.NewDatabaseWithConfig
            state.New (for current top block) => StateDB
          core.NewTxPool => ???
          newHandler (initialize eth66 protocol logic) =>
            fetcher.NewBlockFetcher (with anonymous functions validator (as verifyHeader) and inserter (as insertChain))
            fetcher.NewTxFetcher
            newChainSyncer
          EthAPIBackend
          Ethereum.Protocols => eth.MakeProtocols (register p2p.Protocol for eth66)
          eth.APIs =>
            e.g. ethapi.GetAPIs =>
              e.g. NewPublicTransactionPoolAPI =>
                PublicTransactionPoolAPI (implements GetTransactionByHash)
          Node.RegisterAPIs (update Node.rpcAPIs)
          Node.RegisterLifecycle(eth)
    startNode =>
      util.startNode =>
        Node.Attach => rpc.DialInProc =>
          newClient (with net.Pipe) => initClient => Client.dispatch (goro)
        Node.Start =>
          Node.openEndpoints =>
            p2p.Server.Start =>
              newRLPX (as p2p.Server.newTransport)
              p2p.Server.setupDialScheduler =>
                newDialScheduler (with SetupConn callback and discmix iterator) =>
                  dialScheduler.readNodes (from discmix) (goro)
                  dialScheduler.loop (goro)
              p2p.Server.run (goro)
            Node.startRPC =>
              Node.startInProc =>
                rpc.Server.RegisterName (for each Node.rpcAPIs) =>
                  serviceRegistry.registerName =>
                    via reflection, probe methods of "api" object (e.g. PublicTransactionPoolAPI)
                    and save it to serviceRegistry.services
              httpServer.enableRPC (if configured with http endpoint) =>
                rpc.NewServer
                RegisterApis => rpc.Server.RegisterName
                NewHTTPHandlerStack
              httpServer.start =>
                http.Server.Serve (go standard's http server)
          Ethereum.Start (as Lifecycle.start) =>
            handler.Start =>
              chainSyncer.loop (goro)
                BlockFetcher.Start => BlockFetcher.loop
                TxFetcher.Start
                ...

rpc handler entrypoint

(ipc server loop)
Client.dispatch =>
  (select loop)
    (on readOp) rpc.handler.handleMsg => rpc.startCallProc =>
      handleCallMsg => handleCall =>
        serviceRegistry.callback (find callback from serviceRegistry.services)
        runMethod => callback.call (via reflection, invoke e.g. PublicTransactionPoolAPI.GetTransactionByHash)


(method GetTransactionByHash)
PublicTransactionPoolAPI.GetTransactionByHash =>
  EthAPIBackend.getTransaction (as ethapi.Backend) =>
    rawdb.ReadTransaction => ???
  EthAPIBackend.HeaderByHash (find block header from hash) => ???
  newRPCTransaction


(method Call)
PublicBlockChainAPI.Call =>
  DoCall =>
    TransactionArgs.ToMessage
    EthAPIBackend.GetEVM
    ApplyMessage
  ExecutionResult.Return ()

eth66 handler entrypoint

(connection initialization)
dialScheduler.readNodes =>
  FairMix.Next (as enode.Iterator.Next)
  send Fairmix.Node to nodesIn


dialScheduler.loop =>
  (select loop)
    (on nodesIn) dialScheduler.startDial => newDialTask.run => dialTask.dial =>
      p2p.Server.SetupConn =>
        newRLPX (as newTransport) => ???
        SetupConn =>
          rlpxTransport.doEncHandshake => ???
          checkpoint => send checkpointAddPeer


(peer initialization)
p2p.Server.run =>
  (select loop)
    (on checkpointAddPeer)
      launchPeer =>
        newPeer (with p2p.server.Protocols registered in eth.New)
        runPeer (goro) =>
          Peer.run => Peer.startProtocols =>
            for each protocol in `Peer.running`
              Protocol.Run (coro)


(eth66)
p2p.Protocol.Run (defined in eth.MakeProtocols) =>
  NewPeer => ???
  ethHandler.runPeer => ethHandler.runEthPeer => handler.runEthPeer =>
    eth.Peer.Handshake (send/receive StatusMsg which includes total difficulty)
    Downloader.RegisterPeer =>
      downloader.peerSet.Register => ???
    eth.Handle => eth.handleMessage =>
      find handler from `eth66 map[uint64]msgHandler` (e.g. `handleNewBlock` for `NewBlockMsg`)

new block execution

  • via chain synchronisation by Downloader

    • send GetBlockHeaders
    • receive BlockHeaders
    • send GetBlockBodies
    • receive BlockBodies

  • via block propagation by BlockFetcher

    • NewBlock
#
# chain synchronization
#
(peer initialization)
handler.runEthPeer =>
  eth.Peer.Handshake (send/receive StatusMsg)
  Downloader.RegisterPeer => peerSet.registerPeer
  syncTransactions => ???
  Peer.RequestHeadersByNumber (for special checkpint or whitelist) =>
    GetBlockHeadersPacket66
    Peer.dispatchRequest
  ...


(chain synchronisation (send GetBlockHeaders and GetBlockBodies))
chainSyncer.loop =>
  ...
  (for loop)
    nextSyncOp =>
      peerSet.peerWithHighestTD (find "best" peer based on total difficulty from StatusMsg)
    startSync =>
      handler.doSync (goro) =>
        Downloader.Synchronise, synchronize => syncWithPeer =>
          findAncestor =>
            findAncestorSpanSearch =>
              fetchHeadersByNumber (blocking) => Peer.RequestHeadersByNumber (send GetBlockHeaders)
          spawnSync =>
            (concurrently as goroutines)

            fetchHeaders =>
              fetchHeadersByNumber
              (on receiving BlockHeaders)
              push `headerTask` to `headerQueue.headerProcCh`

            processHeaders =>
              (select headerProcCh)
                core.BlockChain.InsertHeaderChain => ...
                downloader.queur.Schedule

            fetchBodies => concurrentFetch =>
              (here Downloader itself as bodyQueue)
              bodyQueue.request => Peer.RequestBodies (send GetBlockBodies)
              (on receiving BlockBodies)
              bodyQueue.deliver => downloader.queue.DeliverBodies => deliver (push response to resultCache)

            processSnapSyncContent =>
              downloader.queue.Results (get response body from resultCache)
              commitSnapSyncData => ???
              importBlockResults => core.BlockChain.InsertChain => ...


#
# block propagation
#

(receive NewBlock)
handleNewBlock => ethHandler.Handle (with eth.NewBlockPacket) =>
  ethHandler.handleBlockBroadcast =>
    BlockFetcher.Enqueue (send BlockFetcher.inject and picked up by BlockFetcher.loop (see "new block execution" below))


BlockFetcher.loop =>
  (for loop)
    (if BlockFetcher.queue is non empty)
      BlockFetcher.importBlocks => ...
    (select on f.inject) BlockFetcher.enqueue (push to BlockFetcher.queue)


BlockFetcher.importBlocks =>
  validator (as verifyHeader) (defined in eth.newHandler) =>
    consensus.Engine.VerifyHeader => ...
  broadcastBlock (goro) => ???
  inserter (as insertChain) (defined in eth.newHandler) =>
    core.BlockChain.InsertChain => ...


#
# processing headers
#
core.BlockChain.InsertHeaderChain =>
  HeaderChain.ValidateHeaderChain =>
    Beacon.VerifyHeaders (as consensus.Engine.VerifyHeaders) =>
      (if not "PoSHeader", then delegate old engine)
        Ethash.VerifyHeaders => verifyHeaderWorker =>
          verifyHeader =>
            check expected "difficulty" based on its parent
            verifySeal (PoW verification)
            ...
      (otherwise)
        ???
  HeaderChain.InsertHeaderChain => writeHeadersAndSetHead =>
    WriteHeaders =>
      rawdb.WriteTd
      rawdb.WriteHeader


#
# processing blocks (use tests from `core/blockchain_tes.go` (e.g. `TestEIP1559Transition`) to debug into this function)
#
core.BlockChain.InsertChain => insertChain =>
  consensus.Engine.VerifyHeaders => ...
  newInsertIterator.next =>
    BlockValidator.ValidateBody =>
      VerifyUncles (who's uncle?)
      ...
  (for each block from newInsertIterator)
    state.New (for parent block) =>
      db.OpenTrie => trie.NewSecure => ???
    StateProcessor.Process => ...
    BlockValidator.ValidateState => ???
    writeBlockAndSetHead => ???


StateProcessor.Process =>
  NewEVMBlockContext
  vm.NewEVM
  (for each transaction in block)
    Transaction.AsMessage =>
      `Message` is a normalized structure of `Transaction` for EVM to process
      (e.g. "from" address is derived from ECDSA signatures via `Sender` function)
    StateDB.Prepare (initialize temporaries e.g. transaction hash and index)
    applyTransaction =>
      ApplyMessage => StateTransition.TransitionDb => ...
      return Receipt with e.g.
        ExecutionResult.UsedGas
        StateDB.GetLogs
        CreateBloom
        crypto.CreateAddress (for contract creation transaction)
        (here ExecutionResult.ReturnData is ignored, in contrast to the case of `PublicBlockChainAPI.Call`)
  Ethash.Finalize =>
    accumulateRewards => StateDB.AddValance (rewards for coinbase)
    StateDB.IntermediateRoot (for a block hash)


StateTransition.TransitionDb =>
  preCheck =>
    check e.g.
      - "from" nonce is valid
      - "from" is not a contract via `StateDB.GetCodeHash`
    buyGas (update sender's balance with fee = gas limit x gas price)
  (if contract create (i.e. "to" is nil))
    EVM.Create => EVM.create =>
      check collision of contract address
      StateDB.CreateAccount => createObject =>
        newObject (initialize stateObject)
        setStateObject
      NewContract (temporary "contract" to run the "code" from the transaction input bytes)
      EVMInterpreter.Run(contract, ...) => ...
      sanity check of the result (e.g. check MaxCodeSize)
      StateDB.SetCode (set interpreter's result as the actual code of the contract address)

  (otherwise)
    EVM.Call =>
      StateDB.CreateAccount (if not StateDB.Exist)
      Transfer ("from" =="value"==> "to")
      StateDB.getCode
      (if code exists)
        NewContract (temporary "contract" with the "code")
        EVMInterpreter.Run(contract, input, ...) (with the "input" from the transaction) => ...


EVMInterpreter.Run =>
  (loop)
    Contract.GetOp (opcode from program counter)
  • data structure
vm.EVM
  BlockContext
    Transfer
    GetHash
  TxContext
  StateDb
  EVMInterpreter
    Config
      JumpTable (mapping from opcodes to corresponding go functions (cf. core/vm/{jump_table,instructions}.go))

StateDB
  state.Database
  Trie
  map[common.Address]*stateObject

stateObject
  StateAccount (Nonce, Balance, Root, CodeHash)
  Trie (for storage)
  Code

eth2

See README-3-lighthouse.md.

Raw
README-2-vyper.md

vyper

https://github.com/vyperlang/vyper

example

misc

# building
pip install -e .[dev]

# testing
pytest tests/functional/context/validation -v -n0 --no-cov

# running compiler (e.g. Vyper to LLL)
vyper -f ir examples/tokens/ERC20.vy
  • .vscode/settings.json for pytest debugging
{
  "python.pythonPath": "${workspaceFolder}/.venv/bin/python",
  "python.testing.unittestEnabled": false,
  "python.testing.pytestEnabled": true,
  "python.testing.pytestArgs": [
    "-n0", "--no-cov", "-p", "no:forked"
  ]
}

compiler

  • data structure
CompilerData
  source_code
  Module (root of vyper ast)
  GlobalContext (vyper ast with metadata from top level definitions e.g. struct, event, ...)
  LLLnode
  assembly
  bytecode


VyperNode


LLLnode
  NodeType

generate_ast => parse_to_ast =>
  pre_parse (replace vyper special keywords (e.g. "event") with "class")
  parse (`ast.parse` from python stdlib)
  annotate_python_ast => ???
  get_node (python ast to vyper ast) => ???


generate_folded_ast => ???


generate_global_context => ???


validate_semantics =>
  ???
  • vyper ast => LLL
generate_lll_nodes => parse_tree_to_lll =>
  generate_lll_for_function (for init function) => ...
  parse_regular_functions =>
    (for each "regular" function)
      generate_lll_for_function => ...
    define "runtime" code with
      - func_init_lll (for contract entrypoint "if calldatasize < 4 then calldatacopy else goto fallback")
      - external functions ("with _calldata_method_id (mload 0) ...external functions...")
          (i.e. when "calldatasize >= 4", load "calldata" from memory to stack and see if any external functions gets invoked)
      - fallback function
      - internal functions
    emit code to return "runtime" code bytes (i.e. the actual contract code persisted on chain)


generate_lll_for_function =>
  FunctionSignature.from_definition
  _run_pass x 2 =>
    generate_lll_for_external_function (or generate_lll_for_internal_function) =>
      _generate_kwarg_handlers =>
        FunctionSignature.abi_signature_for_kwargs ("<name>(<type>, ..)" to hash into method id)
        emit code to jump to function body when `_calldata_method_id` is this function's method id
      _register_function_args =>
        (if "__init__" function (i.e. constructor))
          emit "~codelen" lll macro which will become "_sym_codeend" (in `_compile_to_assembly`)
          and then become actual deployed code size (in `assembly_to_evm`)
      parse_body => ...


old_codegen.parse_body => parse_stmt => Stmt =>
  handle each ast type via `parse_(ast type)` e.g.
    parse_AnnAssign =>
      old_codegen.Expr => ...
      ...
  • LLL => EVM assembly
compile_to_assembly => ???
Raw
README-3-lighthouse.md

lighthouse

Summary

TODO

  • consensus
    • fork choice
    • finality
      • non unique finalized checkpoint (i.e. |F(G)| > 1) ?
    • safety proof
    • liveness proof
    • skipping slot/epoch (empty blocks?)
      • is only for beacon hard fork?
    • FFG vote isn't necessary the ancestor of GHOST vote?
      • otherwise, no point of FFG vote at all?
    • it feels "justified pairs" is not well-defined...
    • how does fork occurs "naturally"?
      • there is only a single validator proposing a single block (otherwise, some validators are getting slashed).
      • but, for example, if the validator who is assigned to propose a block for the slot i didn't observe a proposal for i - 1, then the validator has to propose a block with the parent at i - 2. Meanwhile, there could be a part of network observing a proposed block at i - 1. At the point, for those in that part of the network, they would see "valid" child blocks for the single parent block i - 2.
  • reward

misc

# building
cargo build -p lighthouse

# running
cargo run -p lighthouse -- beacon_node --help

# testing
cargo test -p beacon_node -- http_server_genesis_state --exact --nocapture
  • .vscode/launch.json
{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "lldb",
      "request": "launch",
      "name": "Debug executable 'lighthouse'",
      "cargo": {
        "args": ["build", "--bin=lighthouse", "--package=lighthouse"],
        "filter": {
          "name": "lighthouse",
          "kind": "bin"
        }
      },
      "args": ["beacon_node", "--network", "pyrmont"],
      "cwd": "${workspaceFolder}"
    }
  ]
}

validator lifecycle

  • https://github.com/ethereum/consensus-specs/tree/dev/specs/phase0/validator.md

  • becoming a validator

  • synchronizing a chain

    • beacon chain
    • eth1 chain
      • interact with the deposit contract

  • block proposal

    • check if the next slot is assigned (frequency: uniformly selected out of all validators)
      • get_beacon_proposer_index
    • construct BeaconBlock
    • broadcast

  • attestation

    • check which slot is assigned (frequency: one slot for each epoch)
      • get_beacon_committee
    • construct Attestation
    • broadcast

  • attestation aggregation

    • check if is_aggregator
    • collect Attestation within the comittee
    • construct AggregateAndProof

data structure

beacon node (bn)

main => run => (spawn) ProductionBeaconNode::new => ...

ProductionBeaconNode::new =>
  ClientBuilder::new => ???
  ???

validator client (vc)

TODO

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment