Skip to content

Instantly share code, notes, and snippets.

@holypriest
Created December 10, 2020 03:43
Show Gist options
  • Save holypriest/82f61b718e88f1291c38e221f6dd3094 to your computer and use it in GitHub Desktop.
Save holypriest/82f61b718e88f1291c38e221f6dd3094 to your computer and use it in GitHub Desktop.
Threads won't get you far...
from multiprocessing import Pool
import boto3
def move_to_glacier(f):
sess = boto3.session.Session(region_name='us-east-1')
s3res = sess.resource('s3')
copy_source = {
'Bucket': f[0],
'Key': f[1]
}
s3res.meta.client.copy(
CopySource=copy_source,
Bucket='my-destination-bucket',
Key=f[1],
ExtraArgs={'StorageClass': 'GLACIER'}
)
...
gazillion_files = [('my-source-bucket', 'file-000000000001'), ...]
p = Pool(8)
p.map(move_to_glacier, gazillion_files)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment