Skip to content

Instantly share code, notes, and snippets.

@e-roux
Last active November 30, 2020 22:20
Show Gist options
  • Select an option

  • Save e-roux/d0d30ea93b90067a85cd9075fc60e092 to your computer and use it in GitHub Desktop.

Select an option

Save e-roux/d0d30ea93b90067a85cd9075fc60e092 to your computer and use it in GitHub Desktop.
Mémo: Apache Airflow

Apache Arrow

Flight

From https://www.dremio.com/understanding-apache-arrow-flight :

What Makes Apache Arrow Flight Fast?

No serialization/deserialization. The Apache Arrow memory representation is the same across all languages as well as on the wire (within Arrow Flight). As a result, the data doesn’t have to be reorganized when it crosses process boundaries.

Bulk operations. Flight operates on record batches without having to access individual columns, records or cells. For comparison, an ODBC interface involves asking for each cell individually. Assuming 1.5 million records, each with 10 columns, that’s 15 million function calls to get this data back into, say, Python.

Infinite parallelism. Flight is a scale-out technology, so for all practical purposes, the throughput is only limited by the capabilities of the client and server, as well as the network in between.

Efficient network utilization. Flight uses gRPC and HTTP/2 to transfer data, providing high network utilization,
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment