damianmcdonald/README.md

A gist showing some common S3 and S3 Glacier actions using the API's listed below.

Actions

Create an S3 Bucket
Upload files to bucket choosing different storage classe
Download objects from S3
Restore objects from S3
Configure an S3 lifecycle policy
Create Glacier vault
Upload archive to vault
Request an inventory in Glacier
Request an archive retrieval in Glacier
Describe a Glacier job request
List Glacier jobs
Get Glacier job output
Sample security policy for Glacier Topic

APIs

API	Documentation
S3 API	https://docs.aws.amazon.com/cli/latest/reference/s3/
S3 API	https://docs.aws.amazon.com/cli/latest/reference/s3api/index.html#cli-aws-s3api
Glacier API	https://docs.aws.amazon.com/cli/latest/reference/glacier/index.html#cli-aws-glacier

Define some common variables

S3_BUCKETNAME=YOURBUCKETNAME${RANDOM}
S3_STANDARD_UPLOAD_1=koala.jpg
S3_STANDARD_UPLOAD_2=emu.jpg
S3_GLACIER_UPLOAD_1=kangaroo.jpg
S3_GLACIER_UPLOAD_2=galah.jpg
GLACIER_VAULT_NAME=YOURVAULTNAME
GLACIER_ARCHIVE=glacier-archive.zip
AWS_PROFILE=YOUR_AWS_CLI_PROFILE_NAME

Create an S3 Bucket

https://docs.aws.amazon.com/cli/latest/reference/s3/mb.html

aws s3 mb s3://${S3_BUCKETNAME} --profile ${AWS_PROFILE}

Upload files to bucket choosing different storage classes

https://docs.aws.amazon.com/cli/latest/reference/s3api/put-object.html

aws s3api put-object \
 --bucket ${S3_BUCKETNAME} \
 --storage-class STANDARD \
 --key ${S3_STANDARD_UPLOAD_1} \
 --body uploads/${S3_STANDARD_UPLOAD_1} \
 --profile ${AWS_PROFILE}

aws s3api put-object \
 --bucket ${S3_BUCKETNAME} \
 --storage-class ONEZONE_IA \
 --key ${S3_STANDARD_UPLOAD_2} \
 --body uploads/${S3_STANDARD_UPLOAD_2} \
 --profile ${AWS_PROFILE}

aws s3api put-object \
 --bucket ${S3_BUCKETNAME} \
 --storage-class GLACIER \
 --key ${S3_GLACIER_UPLOAD_1} \
 --body uploads/${S3_GLACIER_UPLOAD_1} \
 --profile ${AWS_PROFILE}

aws s3api put-object \
 --bucket ${S3_BUCKETNAME} \
 --storage-class DEEP_ARCHIVE \
 --key ${S3_GLACIER_UPLOAD_2} \
 --body uploads/${S3_GLACIER_UPLOAD_2} \
 --profile ${AWS_PROFILE}

Download objects from S3

https://docs.aws.amazon.com/cli/latest/reference/s3api/get-object.html

aws s3api get-object \
 --bucket ${S3_BUCKETNAME} \
 --key ${S3_STANDARD_UPLOAD_1} \
 --profile ${AWS_PROFILE} \
 downloads/${S3_STANDARD_UPLOAD_1}

Restore objects from S3

https://docs.aws.amazon.com/cli/latest/reference/s3api/restore-object.html

aws s3api restore-object \
 --bucket ${S3_BUCKETNAME} \
 --key ${S3_GLACIER_UPLOAD_1} \
 --restore-request Days=1 \
 --profile ${AWS_PROFILE}

Configure an S3 lifecycle policy

https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-lifecycle-configuration.html

aws s3api put-bucket-lifecycle-configuration \
 --bucket ${S3_BUCKETNAME} \
 --lifecycle-configuration file://json/lifecycle.json \
 --profile ${AWS_PROFILE}

Create Glacier vault

https://docs.aws.amazon.com/cli/latest/reference/glacier/create-vault.html

aws glacier create-vault \
 --account-id - \
 --vault-name ${GLACIER_VAULT_NAME} \
 --profile ${AWS_PROFILE}

Upload archive to vault

https://docs.aws.amazon.com/cli/latest/reference/glacier/upload-archive.html

aws glacier upload-archive \
 --account-id - \
 --vault-name ${GLACIER_VAULT_NAME} \
 --body uploads/${GLACIER_ARCHIVE} \
 --profile ${AWS_PROFILE}

Request an inventory in Glacier

https://docs.aws.amazon.com/cli/latest/reference/glacier/initiate-job.html

aws glacier initiate-job \
 --account-id - \
 --vault-name ${GLACIER_VAULT_NAME} \
 --job-parameters file://json/inventory-retrieval.json \
 --profile ${AWS_PROFILE}

Request an archive retrieval in Glacier

https://docs.aws.amazon.com/cli/latest/reference/glacier/initiate-job.html

aws glacier initiate-job \
 --account-id - \
 --vault-name ${GLACIER_VAULT_NAME} \
 --job-parameters file://json/archive-retrieval.json \
 --profile ${AWS_PROFILE}

Describe a Glacier job request

https://docs.aws.amazon.com/cli/latest/reference/glacier/describe-job.html

aws glacier describe-job \
 --account-id - \
 --vault-name ${GLACIER_VAULT_NAME} \
 --profile ${AWS_PROFILE} \
  --job-id JOB_ID

List Glacier jobs

https://docs.aws.amazon.com/cli/latest/reference/glacier/list-jobs.html

aws glacier list-jobs \
 --account-id - \
 --vault-name ${GLACIER_VAULT_NAME} \
 --profile ${AWS_PROFILE}

Get Glacier job output

https://docs.aws.amazon.com/cli/latest/reference/glacier/get-job-output.html

aws glacier get-job-output \
 --account-id - \
 --vault-name ${GLACIER_VAULT_NAME} \
 --job-id JOB_ID \
 --profile ${AWS_PROFILE} \
  glacier-jobs-out

Sample security policy for Glacier Topic

Limited policy

{
  "Version": "2008-10-17",
  "Id": "GlacierNotificationPolicy",
  "Statement": [
    {
      "Sid": "GlacierNotificationAllow",
      "Effect": "Allow",
      "Principal": {
        "Service": "glacier.amazonaws.com"
      },
      "Action": [
        "SNS:Publish"
      ],
      "Resource": "TOPIC_ARN"
     }
  ]
}

Default policy

{
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__default_statement_ID",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "SNS:Publish",
        "SNS:RemovePermission",
        "SNS:SetTopicAttributes",
        "SNS:DeleteTopic",
        "SNS:ListSubscriptionsByTopic",
        "SNS:GetTopicAttributes",
        "SNS:Receive",
        "SNS:AddPermission",
        "SNS:Subscribe"
      ],
      "Resource": "TOPIC_ARN",
      "Condition": {
        "StringEquals": {
          "AWS:SourceOwner": "AWS_ACCOUNT_ID"
        }
      }
    }
  ]
}

	{
	"Type": "archive-retrieval",
	"Format" : "JSON",
	"ArchiveId": "ADD_THE_ARCHIVE_ID",
	"Description": "Test example for retrieving Glacier archive",
	"SNSTopic": "ADD_SNS_TOPIC_ARN"
	}

	{
	"Type": "inventory-retrieval",
	"Format" : "JSON",
	"Description": "Inventory retrieval request",
	"SNSTopic": "ADD_SNS_TOPIC_ARN"
	}

	{
	"Rules": [
	{
	"ID": "Move files to Glacier",
	"Prefix": "/",
	"Status": "Enabled",
	"Transitions": [
	{
	"Days": 2,
	"StorageClass": "GLACIER"
	}
	]
	}
	]
	}