Skip to content

Instantly share code, notes, and snippets.

@nepsilon

nepsilon/how-to-split-a-file-into-smaller-chunks.md

Last active July 16, 2021 10:34
Show Gist options
  • Save nepsilon/57f4f46689b08b92b9bd1897c235f97d to your computer and use it in GitHub Desktop.
Save nepsilon/57f4f46689b08b92b9bd1897c235f97d to your computer and use it in GitHub Desktop.
Download ZIP
How to split a file into smaller chunks? — First published in fullweb.io issue #90
Raw
how-to-split-a-file-into-smaller-chunks.md

How to split a file into smaller chunks?

You may want to upload a 100GB file over an unstable network, or feed your scripts smaller inputs to load in RAM. In both cases, just splitting your file into smaller chunks is an easy solution.

Great news is Unix/Linux systems have the split utility already installed. And using it is simple as pie:

Cut a binary file into chunks of X bytes:

split -b X bigfile.avi

Cut a text file into chunks of N lines:

split -l N bigfile.csv

split being a Unix tool, you can actually pipe to it like this:

./gen_lotta_data.py | split -l 1000 -

It will save chunks into files named xaa, xab, xac, etc. Use the -a option to specify a filename suffix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment