Migrate from AWS S3 to Alibaba OSS

s3_oss_migrate.py

#!/usr/bin/python

"""
S3 to OSS Migration
This script will copy the contents of a S3
bucket to a Alibaba OSS bucket.
Depends on the boto3 and oss2 python libraries.
Based on a code from Tony Landis: https://gist.github.com/tony-landis/1720032
Author:     Vahid Ashrafian
License:    WTFPL

Usage:      Setup AWS credentials: https://boto3.readthedocs.io/en/latest/guide/quickstart.html
            Define the S3_* and CF_* settings below before running;
"""

import oss2
import boto3

S3_BUCKET       = ''

OSS_KEY         = ''
OSS_SECRET      = ''
OSS_BUCKET      = ''
OSS_ENDPOINT    = ''

# connect to s3
s3 = boto3.resource('s3')
s3_bucket = s3.Bucket(S3_BUCKET)

# connect to oss
oss_auth = oss2.Auth(OSS_KEY, OSS_SECRET)
oss_bucket = oss2.Bucket(oss_auth, OSS_ENDPOINT, OSS_BUCKET)
# setup temp files
tmp_file = '/tmp/%s' % S3_BUCKET

def handle(name):
    "try to do the copy"
    try:
        # check if the file already exists in oss
        try:
            oss_bucket.head_object(name)
            print 'The file already exists, go to next one...'
            return True
        except Exception:
            pass
        #get tmp file
        s3_bucket.download_file(name, tmp_file)

        # upload to oss
        with open(oss2.to_unicode(tmp_file), 'rb') as f:
            oss_bucket.put_object(name, f)

        return True
    except Exception:
        print ' retrying'
        return False

i = 0
"get all the keys"
rs = s3_bucket.objects.all()
for s3_key in rs:
    name = s3_key.key
    print "%i %s" % (i, name)
    done, tries = False, 0
    while done == False:
        #keep retrying, sometimes things time out
        done = handle(name)
    i+=1

print "All done!"

Hi, Vahid

I tried your code but getting one error for boto3 library while running it through "Alibaba Function Compute".

Error Detail: "ImportError: No module named boto3"

On Linux machine we can install boto3 package but how to use it in Alibaba function compute? Do you have any idea?

@gbhise No idea, I don't have experience with Alibaba function

Ok THank you.