Skip to content

Instantly share code, notes, and snippets.

@robot56
Created August 14, 2018 06:18
Show Gist options
  • Save robot56/55b5d4a933c4b88dbbdff9385a3a6c5f to your computer and use it in GitHub Desktop.
Save robot56/55b5d4a933c4b88dbbdff9385a3a6c5f to your computer and use it in GitHub Desktop.
namespace nyc.realtime;
/*
Why require Protocol Buffers and Flatbuffers?
Because we interop with web-browsers, it was necessary to build
a communication format that works on both sides.
Although protocol buffers works in the browser, a key part in the
decision making process was that WebAssembly should be taken into account.
JSON proved to be very inefficent for Web<->Server communication because
we have to toss it around a few times between the backend server, the frontend
proxy and then to the user's web client. At the web client, as it turns out JSON
decoding took up rather large amounts of CPU usage because we send data so frequently.
Implementing protocol buffers in C++ works great: but with emscripten C++ binaries
can lead to monsterious binary sizes on the order of megabytes vs. kilobytes
when building with C because the C++ STL is gigantic.
Larger WASM binaries lead to slower page loads and more CPU usage on the
client's side (because JITing), so a C-only solution was the best bet,
and that's where flatbuffers came in with a super-fast C-only impl,
flatcc working great for our purpose.
Why is CPU usage such a large factor?
Because mobile devices where a large consideration (think battery usage!)
Not only on the CPU usage side either: by using a minimal binary format like flatbuffers
we shave 60% of the message size and save user's cell bandwith. Not to mention getting
data to the client faster. A win-win.
*/
/* == GTFS Feed Data == */
enum TripStatus : byte {
AtStation,
ApproachingStation,
EnrouteToStation
}
enum Direction : byte {
North,
South,
East,
West
}
// Frontend proxy is requesting a command from a backend server
// This message makes the backend aware of this so it can provide targetted
// info.
table ClientCommandProxyRequest {
user_address: int;
request_id: int; /* Int Hash used to identify response to this request */
payload: string; /* Raw JSON request payload */
}
table ClientCommandProxyResponse {
request_id: int; /* What request ID we're responding to */
format: byte; /* 1 - JSON, 2 - Flatbuffer */
payload: string; /* Raw JSON payload */
}
/* == NYC Subway-Specifc Feed Data */
// All Unix Timestamps are stored as an 32-bit integer because JavaScript does
// not support 64-bit integers. Will be fine for the next decade or so :)
// Structs
table NYCSubwayTrip {
gtfs_trip_id: string;
trip_id: string;
start_time: int; // don't use this -- get value from tripid instead
route_id: string;
assigned: bool;
direction: Direction;
}
table NYCSubwaySchedule {
scheduled_track: string;
actual_track: string;
arrival_time: int;
departure_time: int;
stop_id: string;
}
// Messages
table NYCSubwayStopUpdate {
timestamp: int;
trip_id: string;
current_status: TripStatus;
cumulative_arrival_delay: int;
cumulative_departure_delay: int;
on_schedule: bool;
// OPTIONAL: if current_status is `AtStation`
current_stop_id: string; // GTFS Stop ID
current_stop_departing_on: int;
// OPTIONAL: if current_status is `AtStation` and we have a next stop
next_stop_id: string; // GTFS Stop ID
next_stop_arriving_on: int;
next_stop_departing_on: int;
// OPTIONAL: if we have a last known stop
last_stop_id: string; // GTFS Stop ID
}
table NYCSubwayScheduleUpdate {
timestamp: int;
trip: NYCSubwayTrip;
schedule: [NYCSubwaySchedule];
}
// Sent only by request as this is an expensive operation
table NYCSubwayTrips {
timestamp: int;
trips: [NYCSubwayScheduleUpdate];
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment