float_compression.md

http://stackoverflow.com/a/35199727

A second method I have found which greatly reduces the size of compressed gzipped data is to first convert the data to the float16 (half-precision) format and back again to float32. This produces a lot of zeros in the output stream which can shrink file sizes by around 40-60 per cent after compression. One subtlety is that the maximum float16 value is rather low, so you may want to scale your data first, e.g. in python

import numpy as np
import math

input = np.array(...)

# format can only hold 65504 maximum, so we scale input data
log2max = int(math.log(np.nanmax(input), 2))
scale = 2**(log2max - 14)
scaled = input * (1./scale)

# do the conversion to float16
temp_float16 = np.array(scaled, dtype=np.float16)
# convert back again and rescale
output = np.array(temp_float16, dtype=np.float32) * scale

Some tests suggest that the mean absolute fractional difference between input and output for some data are around 0.00019 with a maximum of 0.00048. This is in line with the 2**11 precision of the mantissa.