Skip to content

Instantly share code, notes, and snippets.

@rBurgett

rBurgett/cc_gateway_request_logging.md

Last active February 28, 2023 03:11
Show Gist options
  • Save rBurgett/ecfe4d558660cfd5e8e867223e0a001c to your computer and use it in GitHub Desktop.
Save rBurgett/ecfe4d558660cfd5e8e867223e0a001c to your computer and use it in GitHub Desktop.
Download ZIP
CC Gateway Request Logging
Raw
cc_gateway_request_logging.md

We have the first version of our internal relay stats manager live. There were a number of questions about how we collect and record this data, so I wanted to take a moment to give a little insight.

Our haproxy stack consists of three haproxy containers:

         TCP Server
           /    \
HTTP Server 0  HTTP Server 1
  • TCP Server - Despite the name, this server runs in HTTP mode. Its only job is to route traffic to the available HTTP Server 0 or 1.
  • HTTP Server 0 - Runs in HTTP mode. It holds the large front end routing tables and sends requests to the appropriate back end RPC servers.
  • HTTP Server 1 - Runs in HTTP. It is the exact same as HTTP Server 0 but is marked in TCP Server as a backup. If HTTP Server 0 temporarily dies for any reason, this one will automatically take over. But, its most common usage is to take over while HTTP Server 0 is being intentionally restarted when back end RCP servers change because haproxy's TCP runtime API does not provide a means to change back end servers on the fly.

All three servers pipe their logs to stdout which is piped into the cc-gateway application(s). Server notice logs are saved both locally and online for easy access and debugging. Server info logs (i.e. request logs) are only saved locally and we keep 40MB of each on hand.

While server info logs are not saved online en masse, we do parse in real time any requests that come into a valid route. Any requests that do not come into a valid route are rejected by the server and we do not save those as relays. During testing, we found it was possible to open a persistent TCP connection using a good host and then send in requests through that open socket to invalid backends. So, we count a request as a relay if all of the following are tru:

  1. The request came in via a valid host (e.g. 894334g9f403.0021.chains.cc.nodepilot.tech)
  2. The request was able to be routed to an appropritate backend
  3. The response status is something other than a 5xx error

All relays are counted in real time and then saved remotely every five minutes. With this data, we are currently able to see the exact amount of relays coming into every gateway, the exact relay counts per chain, and can look at it over any time period (in five minute intervals).

We have tested both by sending individual requests as well as by opening a persistent TCP socket and sending multiple requests through it. We found that whether the requests come from new sockets or persistent sockets, all are properly recorded by haproxy and properly counted by our system as individual requests.

We will soon make these numbers available to our providers in real time, but in the meantime we can provide them upon request. If you have any questions about routing or logging, just let us know. We are doing everything in our power to make sure that the only traffic that makes its way through our gateways is traffic from our users and that the relays/requests are being properly recorded for billing, metrics, and transparency with our providers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment