Find failed multipart uploads in all buckets in a region

mpu.py

#!/usr/bin/env python
#inventory of all multipart uploads in a region

import boto3
import os

#Not sure this applies here but keeping just in case
region = "us-east-1"
#The profile to use in the credentials or boto.config file, if used
profile_nm = 'foo'
#The env var to look in for the AWS KEY, if used
aws_key_env_var = 'AWS_KEY'
#The env var to look in for the secret key, if used
aws_secret_key_env_var = 'AWS_SECRET_KEY'
#First check for ENV VARs with AWS Credentials
#and make the connection
aws_key = os.environ.get(aws_key_env_var)  
aws_secret_key = os.environ.get(aws_secret_key_env_var)
if (aws_key and aws_secret_key):
    print("Signing in using ENV VAR credentials")
    aws_session = boto3.session.Session(aws_access_key_id=aws_key,aws_secret_access_key=aws_secret_key,region_name=region)
#If env vars don't exist, use the profile in the boto.config file
#If the env vars and profile both exist, the program will never look for the profile, and use the env vars.
else:
    print("Signing in using boto config credentials")
    aws_session = boto3.session.Session(region_name=region, profile_name=profile_nm)

s3 = aws_session.resource('s3')

tot_size = 0

bytes_per_gb = 1073741824
for bucket in s3.buckets.all():
    bucket_mpu_gb = 0
    #bucket = s3.Bucket(bn)
    for mpu in bucket.multipart_uploads.all():
        #print(mpu.initiated)
        mpu_size = 0
        for part in mpu.parts.all():
            mpu_size += part.size
        mpu_gb = mpu_size / bytes_per_gb
        print(bucket.name+","+str(mpu.initiated)+','+mpu.object_key+','+str(mpu_gb))
        #print("Total size of (presumably failed) multipart upload is",mpu_gb,"GB.")
        bucket_mpu_gb += mpu_gb
    print("bucket",bucket.name,bucket_mpu_gb,"GB")
    tot_size += bucket_mpu_gb
    
print("total GB in failed multipart uploads is",tot_size)

I wrapped one section in try/except as my billing bucket caused it to error out - either the parts disappeared while in the loop or something is odd about that upload. Anyway..

...

        mpu_size = 0
        try:
            for part in mpu.parts.all():
                mpu_size += part.size
                mpu_gb = mpu_size / bytes_per_gb
                   print(bucket.name+","+str(mpu.initiated)+','+mpu.object_key+','+str(mpu_gb))
            #print("Total size of (presumably failed) multipart upload is",mpu_gb,"GB.")
            bucket_mpu_gb += mpu_gb
        except:
            continue
    print("bucket",bucket.name,bucket_mpu_gb,"GB")

...

Thanks for the script, useful.

Thank you very much! I used this to make a slightly more refined "cli" version. https://gist.github.com/nijine/581ee63dde89cc56c7a32d3bdbcecbe5

As a note, if you have a credentials file (i.e. at ~/.aws/credentials) with your key and region details, you can skip all of the associated key manipulation above, i.e. between lines 7 and 25. My implementation would be a good example.

I also omitted passing over every bucket in the account, since I wanted information about specific parts in a specific bucket. You can of course tailor the cli to your needs.

@nijine you're welcome

actually this was probably running in Jenkins or something so it had to be like that because the ENV VARS are set based on what role you want to use or something (it's been a while so don't quote me on that)

@paulwakeford glad you find it useful and nice addition.

@jonathanwcrane ahh I see. I believe you can actually have boto-based utilities pick up environment variables automatically if they're named a certain way :). See here: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#environment-variables