The helium network started as a centralized network where the entire infrastructure (besides the hotpots) were deployed and managed by the Helium, inc. With the growth of the network, centralization starts to become an issue, where, as a completely decentralized network must work and grow even in the case of a failure of one of the main supporters (in this case, helium as a company).
At the current stage, most people, companies, and entities that interacts with the blockchain data utilizes the open API endpoints provided by helium, https://api.helium.io, which used to work smoothly until the network started to grow and the usage of the API skyrocketed causing multiple issues, from disrupting the wallet app in which most people use to monitor and manage their HNT.
This bounty comes in place to incentivize users to run their own infrastructure in order to reduce the current single-point of failure and load from the single API provided by helium, increase the decentralization and reliability of the network, and also offer more choices for users to build tools and analytics based on the blockchain data.
For the blockchain software, the server must run an instance of the blockchain-etl, where all the software requirementes are there. There is also a guide on running an ETL here.
This is the actual API server developed by Helium. The code interacts directly with the blockchain database and servers all kinds of information to the fetched by the internet. The code for this can be found here. The API should run behind a proxy server with SSL enabled (nginx, apache, let's encrypt, etc...)
The main requirement at this stage is a big and fast disk, such as a 2TB NvME disk on a dedicated machine. Virtual machines can run the ETL but they are usually not as fast as bare metal ones.
Choose a server with a good amount of ram (32GB+), multi-core (at least 16) and a lot of storage (minimum 2TB SSD). For a higher-end structure, ideally the server would be split into multiple servers:
- One server running just the blockchain-etl software with fast ssd disk with 512GB Disk.
- One server running the main server for the postgresql on a main/secondary setup.
- N amount of secondary servers running a copy of the postgresql database along with running the blockchain-http server serving the API data.
- A load-balancer serving the API data from the multiple secondary nodes.
Basides the suggestion above, you can run everything on the sama server, the blockchain-etl, and the blockchain-http. The downside is that this server will be able to handle less load and is less scalable. But can be an option to get started. Also any other permutation of the server options is possible (run the blockchain-etl on the same server as the main postgresql server, etc).
Server costs can go from $250 to $3000 per month, depending on the setup and provider.
- https://www.ovhcloud.com/en/bare-metal/
- https://www.vultr.com/products/bare-metal/
- https://www.hetzner.com/dedicated-rootserver/matrix-ax
- https://www.digitalcean.com if you want to use VPS with dedicated managed database and load balancing.
- https://aws.amazon.com/
- https://cloud.google.com/
Two questions, please: (1) What are the bandwidth requirements for the projected traffic? (2) Now that we're migrating to Solana, how will the API support requirements change? Thanks!