Skip to content

Instantly share code, notes, and snippets.

@vtermanis

vtermanis/py_lz4framed_comp_perf.py

Last active December 21, 2016 17:47
Show Gist options
  • Save vtermanis/ddb42d56dcc8110623938ff4a643e09f to your computer and use it in GitHub Desktop.
Save vtermanis/ddb42d56dcc8110623938ff4a643e09f to your computer and use it in GitHub Desktop.
Download ZIP
Raw
py_lz4framed_comp_perf.py
#!/usr/bin/env python3
from time import monotonic
from lz4framed import compress
SHORT_INPUT = b'abcdefghijklmnopqrstuvwxyz0123456789'
LONG_INPUT = SHORT_INPUT * (10**5)
# Input a byte shorter than below does not trigger slowdown (for new lz4, not r131)
# LONG_INPUT = LONG_INPUT[:65550]
def comp(input_raw, level):
start = monotonic()
result = compress(input_raw, level=level)
end = monotonic()
print('% 5.3f, Level %2d, output size: %d' % ((end - start), level, len(result)))
def main():
print('Input length: %s' % len(LONG_INPUT))
for level in range(17):
comp(LONG_INPUT, level)
if __name__ == '__main__':
main()
@vtermanis
Copy link
Author

vtermanis commented Dec 21, 2016 •
edited
Loading

Output using lz4 r131 / py-lz4framed 0.9.3 (Python 3.5, Ubuntu 16.04 x64, AMD Phenom II 965):

Input length: 3600000
 0.001, Level  0, output size: 14889
 0.001, Level  1, output size: 14889
 0.001, Level  2, output size: 14889
 0.015, Level  3, output size: 14943
 0.017, Level  4, output size: 14943
 0.022, Level  5, output size: 14943
 0.033, Level  6, output size: 14943
 0.054, Level  7, output size: 14943
 0.095, Level  8, output size: 14943
 0.178, Level  9, output size: 14943
 0.344, Level 10, output size: 14943
 0.677, Level 11, output size: 14943
 1.193, Level 12, output size: 14943
 1.193, Level 13, output size: 14943
 1.193, Level 14, output size: 14943
 1.193, Level 15, output size: 14943
 1.193, Level 16, output size: 14943

@vtermanis
Copy link
Author

vtermanis commented Dec 21, 2016 •
edited
Loading

Output using lz4@70526a1 / py-lz4framed dev branch (same host & config as above):

Input length: 3600000
 0.001, Level  0, output size: 14889
 0.001, Level  1, output size: 14889
 0.001, Level  2, output size: 14889
 0.014, Level  3, output size: 14943
 0.016, Level  4, output size: 14943
 0.021, Level  5, output size: 14943
 0.029, Level  6, output size: 14943
 0.046, Level  7, output size: 14943
 0.079, Level  8, output size: 14943
 0.146, Level  9, output size: 14943
 0.966, Level 10, output size: 14943
 0.001, Level 11, output size: 15943
 51.467, Level 12, output size: 14943
 51.206, Level 13, output size: 14943
 51.183, Level 14, output size: 14943
 51.180, Level 15, output size: 14943
 51.180, Level 16, output size: 14943

Note: Level 11 seems to be incredibly fast also.

@vtermanis
Copy link
Author

Module build output for lz4 & py-lz4framed (same for both):

python3 setup.py build_ext -i
Warning: pypandoc module not found, will not convert Markdown to RST
running build_ext
building '_lz4framed' extension
creating build
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/lz4
creating build/temp.linux-x86_64-3.5/lz4framed
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -c lz4/lz4.c -o build/temp.linux-x86_64-3.5/lz4/lz4.o -Ilz4 -std=c99 -DXXH_NAMESPACE=PLZ4F_ -DVERSION="0.9.4" -DLZ4_VERSION="v1.7.4"
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -c lz4/lz4hc.c -o build/temp.linux-x86_64-3.5/lz4/lz4hc.o -Ilz4 -std=c99 -DXXH_NAMESPACE=PLZ4F_ -DVERSION="0.9.4" -DLZ4_VERSION="v1.7.4"
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -c lz4/lz4frame.c -o build/temp.linux-x86_64-3.5/lz4/lz4frame.o -Ilz4 -std=c99 -DXXH_NAMESPACE=PLZ4F_ -DVERSION="0.9.4" -DLZ4_VERSION="v1.7.4"
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -c lz4/xxhash.c -o build/temp.linux-x86_64-3.5/lz4/xxhash.o -Ilz4 -std=c99 -DXXH_NAMESPACE=PLZ4F_ -DVERSION="0.9.4" -DLZ4_VERSION="v1.7.4"
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -c lz4framed/py-lz4framed.c -o build/temp.linux-x86_64-3.5/lz4framed/py-lz4framed.o -Ilz4 -std=c99 -DXXH_NAMESPACE=PLZ4F_ -DVERSION="0.9.4" -DLZ4_VERSION="v1.7.4"
creating build/lib.linux-x86_64-3.5
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5/lz4/lz4.o build/temp.linux-x86_64-3.5/lz4/lz4hc.o build/temp.linux-x86_64-3.5/lz4/lz4frame.o build/temp.linux-x86_64-3.5/lz4/xxhash.o build/temp.linux-x86_64-3.5/lz4framed/py-lz4framed.o -o build/lib.linux-x86_64-3.5/_lz4framed.cpython-35m-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.5/_lz4framed.cpython-35m-x86_64-linux-gnu.so ->

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment