Skip to content

Instantly share code, notes, and snippets.

View gamame's full-sized avatar

Yusuke Hikime gamame

View GitHub Profile
@gamame
gamame / s3_multipart_upload.py
Created March 29, 2012 13:46 — forked from chrishamant/s3_multipart_upload.py
Example of Parallelized Multipart upload using boto
#!/usr/bin/env python
"""Split large file into multiple pieces for upload to S3.
S3 only supports 5Gb files for uploading directly, so for larger CloudBioLinux
box images we need to use boto's multipart file support.
This parallelizes the task over available cores using multiprocessing.
Usage:
s3_multipart_upload.py <file_to_transfer> <bucket_name> [<s3_key_name>]