Eclipse attacks occur when a node is not connected to any honest peers on the network, and instead its peer connections are controlled by an adversary. Once denied connectivity to the honest network, a victim can be attacked in numerous ways, such as with double-spends or funds loss on layer 2 systems (such as the Lightning Network). Mining nodes attacked in this way can be forced to waste hashpower or be commandeered in selfish mining attacks, or generally aid in causing forks on the network.
This document attempts to describe the mechanisms implemented in Bitcoin Core to mitigate eclipse attacks followed by open questions and areas of further research.
A restart-based eclipse attack occurs when the adversary is able to sufficiently saturate the victim's address manager with attacker IPs (a.k.a. addrman flooding), and then the victim restarts. If the attack succeeds, the victim will lose its current outgoing connections due to the restart and be more likely to make all of its connections to the adversary's addresses. Node restarts are common. They could be caused by natural behavior, a software upgrade (which could be predicted in the case of a widely-publicized bug), or could be triggered by an adversary in the presence of a remotely triggered vulnerability.
The Bitcoin peer-to-peer layer supports messages that allow nodes to gossip the addresses of other nodes on the network. This allows new nodes coming online to learn of other peers, and new listening nodes (that are able to take incoming connections) can have their address gossiped so that other nodes can connect to them later.
Bitcoin Core tracks addresses in an "addresses manager", known as 'AddrMan' and and defined by the class CAddrMan, was introduced by sipa in #787. AddrMan maintains known addresses (e.g. IP and onion) of bitcoin peers on the network. While the implementation has undergone several changes, namely increasing the number of buckets for each table, the essential design remains the same.
Note that a Bitcoin node's sybil resistance requires having at least one honest peer (providing connectivity to the honest network, and thus able to learn of newly mined blocks in some timely fashion).
When a node first comes online, it is at the mercy of the DNS seeds (or alternatively placing trust in the developers by using the hard-coded seeds included in each release), and the first connections it makes by querying for addresses. If fed bad addresses from the start, a fledgling node has little chance of collecting honest peers' addresses. Therefore, addrman assumes that the peer tables are unpoisoned before an attack is initiated.
Nodes store the IP addresses discovered by received ADDR messages in two tables: the "new" table and the "tried" table. The design goal of the address manager is to ensure that no attacker can fill the entire table with his nodes/addresses. A comment in AddrMan.h outlines both the address manager's design goals and the bucketing method.
Addresses that have not yet been tried go into 1024 "new" buckets. The address range (/16 for IPv4) of the source (the peer that gossiped the addr to us) determines which of the 64 buckets it randomly is assigned slots to by being cryptographic hashed. A single address can occur in up to 8 different buckets to increase selection chances of frequently seen addresses. When adding a new address to a full bucket, a randomly chosen entry (with a bias favoring less recently seen ones) is removed from it first. The new table stores the IP addresses that could potentially contain invalid or malicious addresses and could contain IPs that are not Bitcoin nodes or even unallocated IP addresses.
Addresses of nodes known to be an accessible bitcoin node go into 256 "tried" buckets. Each address range selects eight of these buckets at random. The actual bucket is chosen from one of these, based on the full address. Bucket selection is based on cryptographic hashing, using a randomly-generated 256-bit key. This key should remain private since an adversary with knowledge of a victim’s bucket selection could more easily target peer addresses to be evicted.
When adding a new good address to a full bucket in the "tried" table, a short-lived connection attempts to connect to a randomly chosen entry (with a bias favoring less recently tried ones). If the connection is successful, then the older address is not evicted back to the "new" buckets. The new address is stored in the "tried" table only if the connection fails. (For more, see Countermeasure #3: Test-before-evict.)
Peers are discovered from six sources:
- Loaded from peers.dat
A node can use its pre-existing knowledge of possible peers recorded in peers.dat.
- DNS Seeds
This is the most common method for a brand new Bitcoin node to discover peers for the first time. A node can use DNS seeds to populate the addrman when its addrman is empty or if passing in the -forcednsseed argument. DNS seeds are domains owned by Bitcoin contributors that do two things: (1) expose subdomains that DNS resolve to IP addresses of potential peers and (2) run minimal nodes capable of VERSION/VERACK handshakes and addr gossip (code). Usually, a node will query these seeds via the DNS protocol, which resolves to IP addresses independent from the bitcoin protocol.
If our node is behind a proxy (or if the seeder does not support service bit filtering over DNS), the seeders are treated as an AddrFetch connection. AddrFetch connections (previously referred to as “oneshot” connections) are short-lived connections that are established for the sole purpose of populating the addrman. Nodes will engage in the Bitcoin protocol handshake and addr gossip with the seeder. The connection will be disconnected soon after.
There are downsides to DNS seeding. DNS seeding leaks to the ISP that the node is a running bitcoin node. Also, DNS sources can provide useless or malicious addresses. Moreover, even if the DNS request results were cryptographically signed, developers are trusted to not attack nodes starting up in the future by, for example, relegating all newcomers to a separate network and taking advantage of them later.
- Fixed seeds
In the case that DNS seeds aren’t responding, as a fallback the node attempts to expand the peers in addrman via fixed seeds hardcoded in chainparamsseeds.h code. This hardcoded list contains addresses of recently active nodes on the network and is updated every release cycle. #18506 is an example of the process to update the fixed seeds. Connections to fixed seeds are established as ADDR_FETCH connections, used for addr collection and then disconnected.
- ADDR_FETCH connections via
-seednode
AddrFetch connections (previously referred to as “oneshot” connections) are short-lived connections that are established for the sole purpose of populating the addrman (source). These ADDR_FETCH connections can also be initiated by explicitly passing in -seednode addresses. or under certain circumstances with DNS seeds (i.e., here and here).
- Manual connection in
-connectmode
If the user runs the node in this configuration, bitcoind will only ever make outbound connections with nodes specified in -connect. For example, this might be useful in enterprise settings where internal nodes all only talk to a gateway node, which is responsible for staying in sync with the public Bitcoin network. Manual connections cannot be evicted and do not contribute to the limits of outbound-full-relay and outbound-block-relay.
- Manual connection with
-addnode
Bitcoind will make an outbound connection to a specified node, and engage in addr relay with the peer. As new peers are discovered, more outbound connections will be established. Manual connections cannot be evicted and do not contribute to the limits of outbound-full-relay and outbound-block-relay.
Once a node has peers established through any of the above seeding techniques, it engages in addr relay to grow its addrman.
Peers share data with each other by gossiping unsolicited addresses. After establishing a connection, the initiating node sends its own address to the new peer, which then forwards it to a couple more peers in the network. Additionally, nodes will resend their address to their peers on average once every 24 hours (source). Those peers will batch these in maximum sets of 10 (source) and send them to a limited group of their peers who haven't been relayed that address before (code).
*Note: this description of addr relay behavior only describes Bitcoin Core. There are no BIPs specifying addr relay, and therefore, the behavior of other Bitcoin software on the network is purely local policy.
GETADDR data provides another way to populate the addrman. After the initial version handshake, a node sends a GETADDR message requesting information about known active peers to help find potential nodes in the network (source). The receiving node proportionally samples addresses from the new and tried tables and responds with one or more (up to 1,000) addr messages with timestamps of 3 hours or less capped at 23% of its own addrman size (code).
Past research has exploited GETADDR messages by sending them to all possible nodes on the network, then comparing these messages to infer the possible edges of each node. The Coinscope paper took advantage of long-lived connections and used timestamps proliferated through the network to infer topology. This behavior was addressed in version 10.1 and further improved by #14897, which randomized the request order and introduced a bias toward outbound connections giving them a higher priority when a node chooses from whom it should request a transaction. #18991 cached responses to GETADDR to further prevent topology leaks.
AddrMan also tracks the timestamps of peers, currently connected or announced by other peers. A timestamp is kept for each address to keep track of when the node address was last seen. Nodes keep newly received addresses in memory and wait two hours before adding them to the peers.dat database. However, if the address received is in the database, the timestamp will be updated with the new one if it is more recent (source).
When successfully connecting to a peer, the CAddress's nTime is updated, which is gossiped to peers. net_processing also updates this timestamp when it disconnects from a peer to not leak information about currently connected peers (code).
The above timestamps should not be confused with the nLastSuccess timestamp on CAddrInfo that updates when we mark an address as Good.
Note that timestamps can be modified, and malicious nodes may change them or even set them in the future. As mentioned above, leaking timestamp information has been used to infer topology information by network spies. Recent discussions have questioned the usefulness of nTime timestamps, raising the possibility that they could be removed altogether.
The eclipse attack paper recommends 10 countermeasures. Deployed countermeasures are presented below in chronological order of when they were merged into Bitcoin Core.
Countermeasure 4 (feeler connections - #8282)
Heilman, et al. found that a large percentage of addresses in tried tables are stale IP addresses (ranging from 72% to 95% stale), which increases the risk of eclipse attacks.
To mitigate this, Bitcoin Core introduced feeler connections, which is a short-lived outbound connection that pings an address in a new table to check if it is online before transferring it to the tried table. Feelers confirm addresses are Bitcoin nodes by doing the VERSION/VERACK handshake before disconnecting. This test ensures that the tried table is replenished with a steady supply of recent online addresses.
To limit the feeler connections' network impact, nodes make one new connection at most every two minutes. Compared with other networking tasks that bitcoind performs, the bandwidth increase is very slight. Additionally, there is a random sleep of between 0 and 1000 milliseconds prior to making a feeler connection to avoid synchronization issues. To avoid threading issues, feeler connections are made in the same thread as non-feeler connections.
Countermeasure 3 (test-before-evict - #9037)
Before storing an address in its deterministically-chosen slot in a bucket in the tried table, check if there is an older address stored in that slot. If so, a short-lived connection, also referred to as a feeler connection, attempts to connect to the older address, and if the connection is successful, then the older address is not evicted. The new address is stored in the tried table only if the connection fails.
Another small side advantage of test-before-evict is that no more than ten addresses can be in the test buffer at once. Addresses are only cleared one at a time from the test buffer, so an attacker is forced to wait at least two minutes to insert a new address into the tried table after filling up the test buffer. This rate limits an attacker attempting to launch an eclipse attack.
Countermeasure 1 (Deterministic random eviction - #5941)
When adding an address to the new or tried tables, each address deterministically hashes to a single slot in a single bucket. Before this change, an attacker could increase the number of addresses stored by repeatedly inserting the same address in multiple rounds. This update gave each address a single fixed location in the new and tried tables, which become simple fixed-size arrays instead of sets and vectors.
Countermeasure 2 (Random selection - 214154e)
The Eclipse paper's attack exploited bitcoin core’s heavy bias towards initiating outgoing connections to addresses with fresh timestamps. This advantage was eliminated when addresses were selected at random from tried and new tables. With this change, a 50% exploitation success rate requires the adversary to fill 91.7% of the tried table.
Countermeasure 6 (More buckets - 1d21ba2)
As more buckets are added, an infrastructure attacker needs to increase the number of groups in order to expect to fill the same fraction of the tried table. More buckets were added, and the addresses saved per group were scaled. The last increase was in 2015. Revisiting the optimal number of buckets and considering larger new and tried tables may have to scale in proportion to the growth of the network. (See open question #7 for further discussion.)
A note from the eclipse paper is that this countermeasure is helpful only when the tried table already contains many legitimate addresses so that attacker owns a smaller fraction of the addresses in tried. However, if tried is mostly empty (or contains mostly stale addresses for nodes that are no longer online), the attacker will still own a large fraction of the tried table addresses, even though the number of tried buckets has increased.
Inspired by Tor entry guard rotation rates, the eclipse attack paper recommends adding connections that persist between restarts, known as anchor connections. These long-lasting connections make eclipse attack more expensive for eclipse attackers but at the cost of privacy concerns of leaking network topology or transaction sources.
Anchor connections were introduced by PR #17326 and merged into the 0.21 release. (Note that the eclipse attack paper suggested adding anchor connections for full connections, but this PR scopes anchor connections to block-relay-only connections to help with the privacy concerns stated above.)
Another tradeoff is that persistent long-lived connections would enable a snooping attacker to passively capture persistence of connections, potentially making topology inference more powerful. Strong persistence can contribute to network self-partitioning (e.g., longer distance links are less reliable, so they get culled, and eventually disconnected subgraphs might only connect to their own continent) (source). This added behavior should probably pair with a complementary behavior on the inbound side. As of now, about half the inbound slots are preserved for longest-connected peers. Half of those could be redirected to be preserved for peers with the longest-historically-connected time. Without some measure like this, persistent connection logic could be undermined by an attacker that fills the connection slots up on long-running static IPed nodes in order to cause the eviction of (or prevent connections from) the other hosts they hope to eclipse (source).
The eclipse attack paper recommended adding, "additional outgoing connections without risking that the network will run out of connection capacity."
Version 0.19 added two outbound block-relay-only connections via PR #15759, which do not relay or process transactions or addr messages. (See related discussion on the impact of address propagation through the network.)These add further robustness by increasing connectivity with minimal CPU/memory/bandwidth tradeoffs. Also, since fingerprinting transactions is a common technique to infer network topology, block-relay-only connections are much harder to observe than their transaction-relay counterparts.
When considering the addition of more outbound connections by default, there exists a fundamental tradeoff between resource minimization and robustness to peer misbehavior. Adding more connectivity to the network graph makes Bitcoin's network more robust (e.g., to eclipse or partition attacks), but at the cost of more resource utilization. The Erlay proposal presents a promising path to reduce the bandwidth required for these connections, which could allow more connections while maintaining resource requirements.
Currently, unsolicited ADDR messages of greater than 10 addresses are accepted but not relayed. A node could choose not to accept large unsolicited ADDR messages from incoming peers and only solicit ADDR messages from outgoing connections when its new table is near empty. This prevents adversarial incoming connections from flooding a victim's new table with useless or malicious addresses. The tradeoff would be the slower propagation of addresses from new nodes across the network.
A Bitcoin node can have all of its incoming connections come from the same IP address, making it far too easy for a single computer to monopolize a victim’s incoming connections during an eclipse attack or connection-starvation attack. The eclipse attack paper suggests a node accepts only a limited number of connections from the same IP address.
The current way incoming connections are diversified is by prioritizing incoming connections evictions from the largest netgroup. Keyed-netgroups is the mechanism by which each node uniquely divides the address space into netgroups. Asmap has been proposed as a way to further improve IP bucketing in addrman (issue: #16599 and PR: #16702, blog post). Instead of relying on /16 prefixes to diversify the connections every node creates, a node would rely on the IP to ASN mapping, if such a mapping is provided.
The eclipse attack can have several specific "signatures'' that make it detectable, including: (1) a flurry of short-lived incoming TCP connections from diverse IP addresses that send large ADDR messages containing "trash" IP addresses. (2) An attacker that suddenly connects a large number of nodes to the network. (3) As could one that uses eclipsing to decrease the network's mining power dramatically. Monitoring and anomaly detection systems that look for this behavior would be useful. They would, at the very least, force an eclipse attacker to attack at a lower rate or waste resources on overwriting new, rather than useless, IP addresses.
- Examining the rules for evicting an incoming connection
A DoS connection exhaustion attack is when an adversary fills all available incoming connection slots on the network, then overtakes the outbounds for a victim just starting up. A patch in 2015 fixed this by enabling new incoming connections to evict old incoming connections. However, this patch introduced the possibility of an attacker purposely evicting connections. For example, because a trusted outgoing connection is another node's untrusted incoming connection, an attacker with knowledge of a node's topology inference could evict this connection. Therefore, an attacker might be able to eclipse a node without reboots via connection eviction logic (source). Changes to incoming connection eviction logic to mitigate a connection exhaustion attack are described as follows:
-
Bitcoin Core in 2015:
- Bob has 116 incoming connections (125 total connections minus 8 outgoing connections minus 1 feeler connection).
- Alice makes an outgoing connection to Bob.
- Bob has 117 incoming connections
- Carol attempts to make an outgoing connection to Bob
- Carol's connection is rejected.
- Alice still has an outgoing connection to Bob
-
Bitcoin Core 2020:
- Bob has 114 incoming connections (125 total connections minus 8 outgoing full connections minus 2 block-only relay connections minus 1 feeler connection).
- Alice makes an outgoing connection to Bob.
- Bob has 115 incoming connections
- Carol attempts to make an outgoing connection to Bob
- There is a chance Alice's connection is evicted, and Carol's connection is established
- Alice loses her outgoing connection to Bob (sometimes)
- Carol now has an outgoing connection (sometimes)
The change described above presents a tradeoff. Nodes coming online are able to find receptive peers for their outgoing connections. However, an adversary with network topology inference is potentially more powerful under today's eviction ruleset (code) because an attacker could target the eviction of incoming connections of a victim node's peer (i.e., the victim node's outgoing connection).
Further research is needed to determine:
a) How feasible is it for incoming connections to be displaced by new incoming connections by an attacker? If feasible, this has serious consequences for someone who has knowledge of the network topology since they can target a victim's outbound connections via the peer's node.
b) For a connection exhaustion attack, at some point, the attacker may just be evicting itself and (only possibly) locking new connections out. How many IP addresses are needed to perform this connection exhaustion attack, and what netgroups or geographic locations would be required?
- Timestamps on addr messages:
The Coinscope paper used addr messages to infer network topology. Since then, changes have been made, including PR #18991, which introduced a mechanism to reduce the ability to scrape the address manager. However, there are likely more leaks and a smarter way to communicate timestamps. There may be the possibility of a spam attack using malicious timestamps as well. Some have proposed doing away with these timestamps all together. (See October 20th, 2020, P2P IRC meeting for more discussion.)
- Defending against bad actors when a node first comes online
As discussed in the discussion on Assumption of a Healthy Addrman from the Start, nodes are most vulnerable when they first come online. Can we harden the seeders? Should we use authentication? What are the vulnerabilities of the DNS caching?
- Effects on the network of outbound peer rotation
The effects of outbound peer rotation have both positive and negative effects on peers. Outbound peer rotation is good for transaction-relayed peers as it improves privacy and makes topology inference more difficult. However, outbound peer rotation is bad for block-only-relayed peers as it increases the risk of a node to be eclipsed.
Nodes make new full-relay connections when the tip is stale, which are disconnected after waiting a short time to see if we learn a new block. Also, #19858 makes eclipse attacks more difficult by regularly initiating outbound connections and staying connected long enough to sync headers and potentially learn of new blocks. If the node learns of a new block, it rotates out an existing block-relay peer in favor of the new peer. Since block-relay connections use minimal bandwidth, nodes can make these connections regularly and not just when the tip is stale.
Further discussion can be found at #4723, #15759, and the mailing list).
- Resources estimation given the current mitigations in place
Heilman, et al. cited ~9000 IP addresses were required to successfully complete an attack. Given the suggested countermeasures implemented as well as other changes deployed, a new model is needed to understand the resource requirements of successfully exploiting a victim.
- Heuristics for “quality” peers besides online presence
Should the feeler connection poll the chain tip of peers similar to the outbound peer rotation logic? What is the cost of adding this logic?
- The optimum bucket sizes and size of the new and tried tables
How is the growth of the network reflected in these bucket sizes and new and tried tables? Are there enough addresses to fill up these tables? Would smaller buckets lend themselves to a more defensible position?