Ali Arsanjani aarsanjani
aarsanjani
/ s3_file_combine.py
Created
December 8, 2019 19:14
— forked from bwicklund/s3_file_combine.py
S3 file Concatenation/Combination. S3 Spark file merge.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import argparse | |
| import boto3 | |
| import os | |
| import threading | |
| from fnmatch import fnmatch | |
| # S3 multi-part upload parts must be larger than 5mb | |
| MIN_S3_SIZE = 6000000 | |
| LOG_LEVEL = 'INFO' |