Ubershader Archive Compression

compression_experiment.md

zstd is a C library that features very fast decompression (similar to snappy) and dictionary-based compression.

Asking zstd to perform best possible compression is slow, but decompressing the result is still fast.

It also has command tool:

brew install zstd

GL-only release build

(all sizes are decimal)

• Simply gzipping all the gltfio filamat files results in a 808K archive.
• zstd results in 516K file, but with a 112K dictionary, together these gzip down to 520K.

To repro, set your CMake options to exclude Vk and Mtl, make a release build, then do:

cd out/cmake-release/libs/gltfio
du -ch *.filamat                       # Lists the uncompressed files = 3.3M
zstd --train *.filamat -o uberdict     # Produces the dictionary = 112K
zstd --ultra -22 -D uberdict *.filamat # Produces zst files = 516K
zip -9 with_zstd.zip *.zst uberdict    # gzips the dict + zst files = 520K
zip -9 without_zstd.zip *.filamat      # Simple gzip = 808K

GL+Mtl+Vk release build

• Uncompressed gltfio filamat files total to 10M (!)
• Simply gzipping the filamat files results in 2.7M.
• The zstd+gzip procedure results in 1.8M.