Skip to content

Instantly share code, notes, and snippets.

@WillKoehrsen

WillKoehrsen/partition_with_chunking.py

Last active September 26, 2018 13:04
Show Gist options
  • Save WillKoehrsen/101b5fa8eb89eab4ed7fd2d78e30b795 to your computer and use it in GitHub Desktop.
Save WillKoehrsen/101b5fa8eb89eab4ed7fd2d78e30b795 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
partition_with_chunking.py
# Partition a large file that cannot fit into memory
for chunk in pd.read_csv(user_logs_file_path, chunksize = chunksize):
partition_by_hashing(chunk, name = 'logs', progress = None)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment