Skip to content

Instantly share code, notes, and snippets.

@nfelger
Created April 15, 2015 10:27
Show Gist options
  • Save nfelger/5edb27946505e480b6a0 to your computer and use it in GitHub Desktop.
Save nfelger/5edb27946505e480b6a0 to your computer and use it in GitHub Desktop.
Quick and dirty s3 speed tests with temporary STS credentials
#!/bin/bash
# Inspired by http://dl.getipaddr.net/ and http://curl.haxx.se/mail/archive-2014-10/0006.html
file=path/to/file
bucket=your-bucket
contentType="application/octet-stream"
dateValue=`date -R`
resource="/${bucket}/${file}"
s3Key=`curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2-app-role | grep 'AccessKeyId' | sed 's/.* "\([^"]*\).*/\1/'`
s3Secret=`curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2-app-role | grep 'SecretAccessKey' | sed 's/.* "\([^"]*\).*/\1/'`
token=`curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2-app-role | grep 'Token' | sed 's/.* "\([^"]*\).*/\1/'`
stringToSign="GET\n\n${contentType}\n${dateValue}\nx-amz-security-token:${token}\n${resource}"
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curlOut=$(curl -s -w "%{speed_download}" -o /dev/null -X GET -H "Host: ${bucket}.s3.amazonaws.com" -H "Date: ${dateValue}" -H "Content-Type: ${contentType}" -H "Authorization: AWS ${s3Key}:${signature}" -H "x-amz-security-token: ${token}" https://${bucket}.s3.amazonaws.com/${file})
# Output x-fer speed in MB/sec
echo "scale=2;$curlOut/1048576" | bc -q
#!/bin/bash
# Inspired by http://dl.getipaddr.net/ and http://tmont.com/blargh/2014/1/uploading-to-s3-in-bash
file=file-to-upload
bucket=your-bucket
pathInBucket=upload-tests/`date +%s`
contentType="application/octet-stream"
dateValue=`date -R`
resource="/${bucket}/${pathInBucket}"
s3Key=`curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2-app-role | grep 'AccessKeyId' | sed 's/.* "\([^"]*\).*/\1/'`
s3Secret=`curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2-app-role | grep 'SecretAccessKey' | sed 's/.* "\([^"]*\).*/\1/'`
token=`curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2-app-role | grep 'Token' | sed 's/.* "\([^"]*\).*/\1/'`
stringToSign="PUT\n\n${contentType}\n${dateValue}\nx-amz-security-token:${token}\n${resource}"
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curlOut=$(curl -s -w "%{speed_upload}" -o /dev/null -X PUT -T "${file}" -H "Host: ${bucket}.s3.amazonaws.com" -H "Date: ${dateValue}" -H "Content-Type: ${contentType}" -H "Authorization: AWS ${s3Key}:${signature}" -H "x-amz-security-token: ${token}" https://${bucket}.s3.amazonaws.com/${pathInBucket})
# Output x-fer speed in MB/sec
echo "scale=2;$curlOut/1048576" | bc -q
@vallamost
Copy link

Unless you need to speedtest single threaded workflows this isn't how you should speedtest against S3.

You should be using multi-threaded downloads and uploads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment