Skip to content

Instantly share code, notes, and snippets.

@gmcabrita

gmcabrita/fast-data-algorithms.md

Last active May 13, 2023 18:38
Show Gist options
  • Save gmcabrita/1198977838f5bcb6a2e99b458979887f to your computer and use it in GitHub Desktop.
Save gmcabrita/1198977838f5bcb6a2e99b458979887f to your computer and use it in GitHub Desktop.
Download ZIP
Raw
fast-data-algorithms.md

Fast data algorithms

From: https://jolynch.github.io/posts/use_fast_data_algorithms/

Application Common Bad Performance Choices Better Performance Choices Expected Performance Gain
Trusted data hashing md5, sha2, crc32 xxhash ~10x
Untrusted data hashing md5, sha2, sha1 blake3 ~10x
Fast compression snappy, gzip (zlib) lz4 10x over gzip~2x over snappy
Good compression gzip (zlib) zstd ~2-10x
Best compression xz (lzma) zstd -10+ ~2-10x
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment