Upload Travis CI builds to S3

README.md

Upload Travis CI builds to S3

This will demonstrate how to upload build files from Travis CI to S3.

NOTE: Keys have been changed to protect the innocent.

Step 1: Create an S3 policy.

Create an S3 ACL policy, see s3_policy.json for an example.

Step 2: Generate the policy and signature.

Use gen_upload_policy.php to generate the base64 policy and signature. This requires your S3 secret key to generate the signature:

$ ./gen_upload_policy.php ./s3_policy.json lvbpI6ICIyMDA4LTErLTAxVDtyO+m202n03
S3_POLICY="eyAiZXhwaXshdGlvbpI6ICIyMDA4LTErLTAxVDtyOjAwOjAwLjAsMFoiLAogICJjb25kaXRpb25zPjogWwoJeyJidWNrZXQiOiAiczNwaG90b3MubW9hcHAubmV0IiB9LAogICAgWyJzdGFydHMtd2l0aCIsICIka2V5IiwgIkxpdmVTaG90cy8iXSwKICAgIHsiYWNsIjogInB1YmxpYy1yZWFkIiB9LAoJWyJlcSIsICIkQ29udGVudC1UeXBlIiwgImltYWdlL2pwZWciXSwKICBdCn0K"
S3_SIGNATURE="20uh08kU75ADHL49NyhYRgZW8BY="

You'll use the S3_POLICY and S3_SIGNATURE variables in your .travis.yml config file.

Step 3: Setup .travis.yml.

Take the variables from above and set them as environment variables in your .travis.yml. Also set the S3_ACCESS_KEY to your S3 access key (not the secret one) and S3_BUCKET to the name of the S3 bucket. See the example .travis.yml file for details.

Step 4: Create the upload script

Create an upload script that uploads files to the appropriate S3 bucket. See the example s3_upload.sh that uploads all *.jar files under the top-level directory target/ to S3.

Add the script location to the .travis.yml after_success hook.

dot.travis.yml

language: java

jdk:
  - openjdk6

env:
  global:
    - S3_POLICY="eyAiZXhwaXshdGlvbpI6ICIyMDA4LTErLTAxVDtyOjAwOjAwLjAsMFoiLAogICJjb25kaXRpb25zPjogWwoJeyJidWNrZXQiOiAiczNwaG90b3MubW9hcHAubmV0IiB9LAogICAgWyJzdGFydHMtd2l0aCIsICIka2V5IiwgIkxpdmVTaG90cy8iXSwKICAgIHsiYWNsIjogInB1YmxpYy1yZWFkIiB9LAoJWyJlcSIsICIkQ29udGVudC1UeXBlIiwgImltYWdlL2pwZWciXSwKICBdCn0K"
    - S3_SIGNATURE="20uh08kU75ADHL49NyhYRgZW8BY="
    - S3_ACCESS_KEY="2EO6H8MX1X8YWEA0V432"
    - S3_BUCKET="johnsmith"

after_success: ./scripts/s3upload.sh

gen_upload_policy.php

#!/usr/bin/env php
<?php

/*
 * From: http://raamdev.com/2008/amazon-s3-hmac-signatures-without-pear-or-php5/
 */

/*
 * Calculate HMAC-SHA1 according to RFC2104
 * See http://www.faqs.org/rfcs/rfc2104.html 
 */
function hmacsha1($key,$data) {
    $blocksize=64;
    $hashfunc='sha1';
    if (strlen($key)>$blocksize)
        $key=pack('H*', $hashfunc($key));
    $key=str_pad($key,$blocksize,chr(0x00));
    $ipad=str_repeat(chr(0x36),$blocksize);
    $opad=str_repeat(chr(0x5c),$blocksize);
    $hmac = pack(
                'H*',$hashfunc(
                    ($key^$opad).pack(
                        'H*',$hashfunc(
                            ($key^$ipad).$data
                        )
                    )
                )
            );
    return bin2hex($hmac);
}
 
/*
 * Used to encode a field for Amazon Auth
 * (taken from the Amazon S3 PHP example library)
 */
function hex2b64($str)
{
    $raw = '';
    for ($i=0; $i < strlen($str); $i+=2)
    {
        $raw .= chr(hexdec(substr($str, $i, 2)));
    }
    return base64_encode($raw);
}

if (count($argv) != 3) {
	echo "Usage: " . $argv[0] . " <S3 Policy File> <S3 secret key>\n";
	exit(1);
}

$policy = file_get_contents($argv[1]);
$secret = $argv[2];

/*
 * Base64 encode the Policy Document and then
 * create HMAC SHA-1 signature of the base64 encoded policy
 * using the secret key. Finally, encode it for Amazon Authentication.
 */
$base64_policy = base64_encode($policy);
$signature = hex2b64(hmacsha1($secret, $base64_policy));
echo "S3_POLICY=\"" . $base64_policy . "\"\nS3_SIGNATURE=\"" . $signature . "\"\n"
?>

s3_policy.json

{
    "expiration": "2015-01-01T12:00:00.000Z",
    "conditions": [
        {"acl": "private" },
        {"bucket": "johnsmith" },
        ["starts-with", "$key", "jars/"],
        ["content-length-range", 2048, 268435456],
        ["eq", "$Content-Type", "application/octet-stream"]
    ]
}

s3upload.sh

#!/bin/bash

# From:
# http://raamdev.com/2008/using-curl-to-upload-files-via-post-to-amazon-s3/

GIT_VERSION=`git describe`

for FILE in `ls target/*.jar`; do
	BASE=`basename $FILE`
	curl \
	    -F "key=jars/$GIT_VERSION/$BASE" \
	    -F "acl=private" \
	    -F "AWSAccessKeyId=$S3_ACCESS_KEY" \
	    -F "Policy=$S3_POLICY" \
	    -F "Signature=$S3_SIGNATURE" \
	    -F "Content-Type=application/octet-stream" \
	    -F "file=@$FILE" \
	    https://s3.amazonaws.com/$S3_BUCKET
	if [ $? -ne 0 ]; then
		exit 1
	fi
done

exit 0