Evan Diewald evandiewald

Keybase proof

I hereby claim:

I am evandiewald on github.
I am diewaldo (https://keybase.io/diewaldo) on keybase.
I have a public key ASCrX-RpRZRJrn593biCTD84kXj4BNCJfaNnJPOGSWnmxgo

To claim this, I am signing this object:

DeWi Grant Bounty: Analytics-Friendly Data Dumps

Description

Open-source, easily-accessible datasets have been shown to spurn innovation in fields like machine learning and data visualization. While the DeWi Metabase platform is suitable for ad hoc analyses and dashboards, queries can be slow and rate-limited. On the other hand, blockchain-etl allows for direct programmatic access to the entire ledger, but extensive hardware requirements and the lengthy initial sync time present a significant barrier to entry.

This grant serves a gap in Helium’s existing data engineering infrastructure by producing periodic static data dumps of key tables over discrete time periods.

The offering will accelerate exploratory analyses related to anti-gaming, token flow, coverage mapping, data usage, and more by enabling data scientists to interact with manageable datasets locally, in the programming language of their choice.

Deliverables

Build a Lightweight Helium ETL with blockchain-node

Want to run your own analyses on Helium blockchain data, without the expense and complexity of a operating a full node? In this article, we'll explore blockchain-node, a lightweight block follower that can be leveraged for maintaining real-time datasets. In Part I, we'll introduce the benefits (and limitations) of this tool, hardware requirements, and usage patterns. Then, we'll walk through the core components of an ETL service that inserts transactions into an analytics-friendly SQL database.

Part I: What is blockchain-node?

From the blockchain-node README:

This is an Erlang application that is a Helium Blockchain node. It follows the blockchain and exposes functionality using a JSONRPC 2.0 API.

	class VsockStream:
	"""Client"""
	def __init__(self, conn_tmo=15):
	self.conn_tmo = conn_tmo
	self.parent_private_key = None
	self.parent_public_key = None
	self.enclave_private_key = None


	def connect(self, endpoint):

	class VsockListener:
	"""Server"""
	def __init__(self, conn_backlog=128):
	self.conn_backlog = conn_backlog
	self.files_received = [0, 0, 0] # --> [sym key, inference, pub key]

	def bind(self, port):
	"""Bind and listen for connections on the specified port"""
	self.sock = socket.socket(socket.AF_VSOCK, socket.SOCK_STREAM)
	self.sock.bind((socket.VMADDR_CID_ANY, port))

	FROM python:3.7.11-slim-buster
	# check on our entropy at initialization
	RUN cat /proc/sys/kernel/random/entropy_avail
	RUN apt-get update -y && apt-get install -y git curl gnupg build-essential libssl-dev libffi-dev python3-dev rng-tools openssh-client openssl haveged

	# start the rngd daemon
	RUN rngd -r /dev/urandom -o /dev/random
	RUN sleep 20
	RUN cat /proc/sys/kernel/random/entropy_avail

	from get_data import *
	import math
	import networkx as nx
	import h3
	import json
	from torch_geometric.utils import from_networkx
	from torch_geometric.data import Data
	import pickle

	import torch
	import torch.nn.functional as F
	from torch_geometric.nn import GCNConv, AGNNConv, SAGEConv, XConv
	from torch_geometric.data import Data, DataLoader, Dataset
	import pickle
	from torch.nn import Linear, ReLU, Flatten
	import random
	import numpy as np
	import matplotlib.pyplot as plt
	from get_data import get_city_details

	import requests
	import datetime
	import h3
	import networkx as nx
	import urllib
	import numpy as np
	import pandas as pd
	import time
	import random

	import requests
	import pandas as pd


	ACCOUNT = "B58_WALLET_ADDRESS"

	# UTC timestamps are given in ISO 8601 format
	MIN_TIME = "2022-02-01"
	MAX_TIME = "2022-03-01"