Commands that I frequently use when using AWS S3 like services from command line

s3-cheat-sheet.md

Credentials

Tool	AWS Profile Support & CLI Argument	Custom Endpoint Support & Method	Set public-read ACL?
aws s3	✅ --profile <profile-name>	✅ --endpoint-url <url>	✅ --acl public-read
s3cmd	❌ No profile support, ✅ Env vars: AWS_ACCESS_KEY_ID, etc.	✅ Config-only: host_base, host_bucket in ~/.s3cfg	✅ --acl-public
s5cmd	✅ --profile <profile-name> or AWS_PROFILE	✅ --endpoint-url <url> or AWS_ENDPOINT_URL	✅ --acl public-read
s3-credentials	✅ --profile <profile-name>	❌ Not supported	❌ Not applicable
mc (MinIO)	❌ No profile support, ✅ Custom alias config	✅ mc alias set <alias> <endpoint> <key> <secret>	✅ mc anonymous set download <alias>/<bucket>

Switching between sets of credentials

Working with multiple sets of credentials (on multiple different S3 compatible hosts) can a real pain. The best solution I have found is to use a shared credentials file. This works both with s3cmd and within python (with xarray, fsspec etc).

You create a shared credentials file in ~/.aws/credentials like so:

[aws-lcd-dmi]
aws_access_key_id = 
aws_secret_access_key = 

[ewc-mllam]
# you're allowed comments too so that you can remember what different tokens are for
aws_access_key_id = 
aws_secret_access_key = 

You can then use this with s3cmd with:

AWS_CREDENTIAL_FILE=~/.aws/credentials AWS_PROFILE=aws-lcd-dmi s3cmd 

similarly in s3-credentials (I would never use anything else for creating credentials!) with:

AWS_CREDENTIAL_FILE=~/.aws/credentials AWS_PROFILE=aws-lcd-dmi s3-credentials whoami

and in python with:

ds = xr.open_zarr(f"s3://bucket-name/dataset.zarr/",
   consolidated=True,
   storage_options=dict(profile="aws-lcd-dmi")
)

Using non-AWS endpoints

Together with selecting your credentials profile (see above) you can also set the "endpoint url" which is the URL of the S3 compatible server you wish to connect to.

With the aws cli command to list your buckets for example:

aws s3 --profile ewc-mllam --endpoint-url https://object-store.os-api.cci1.ecmwf.int ls

and in python with:

ds = xr.open_zarr("s3://bucket-name/dataset.zarr/",
   consolidated=True,
   storage_options=dict(profile="ewc-mllam", endpoint_url="https://object-store.os-api.cci1.ecmwf.int"),
)