Skip to content

Instantly share code, notes, and snippets.

@nicolasdao

nicolasdao/aws_cli_manual.md

Last active January 15, 2024 07:39
Show Gist options
  • Save nicolasdao/edc30b5ad57f1e2147cf3b63bd76f6a9 to your computer and use it in GitHub Desktop.
Save nicolasdao/edc30b5ad57f1e2147cf3b63bd76f6a9 to your computer and use it in GitHub Desktop.
Download ZIP
AWS CLI manual. Keywords: aws cli aws-cli s3
Raw
aws_cli_manual.md

AWS CLI MANUAL

Table of Contents

Install (on MAC OSX)

Prerequisites: You must have python 2.6.5+ installed To check if you have python installed on your MAC, run which python. To check the python version, run python --version.

With brew (recommended)

The recommanded way on Mac:

brew install awscli
brew link --overwrite awscli

The second command creates an aws symlink so the aws command is available in your terminal. The --overwrite flag forces the creation in case there is already an existing AWS CLI installed.

This is the recommended way because you can later use brew upgrade awscli to easily update the CLI.

Manual install

  1. Download the AWS CLI: curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
  2. Unzip: unzip awscli-bundle.zip
  3. Install the AWS CLI: sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

Configure

Basic configuration

To use your AWS CLI you first need to configure it with an access key, access key secret and a region. Those details are stored on your local machine under ~/.aws/config and ~/.aws/credentials. To start the configuration, run the following command:

aws configure

or

aws configure --profile <your-specific-profile-name>

The first command is equivalent to aws configure --profile default, which overides the default profile.

You can generate access keys in your AWs console using the IAM service. As for the region, you'll find an exhaustive list in the Annexes under Regions.

The above command is nothing less than a helper to create 2 local files:

  • ~/.aws/credentials: Contains one or many profiles. Each profile is made of an access key and an access secret.
  • ~/.aws/config: Contains a set of configuration for each profile defined in the ~/.aws/credential.

In Windows, the .aws folder is also located under the User's folder (e.g., C:\Users\FrankJunior\.aws)

Example:

~/.aws/credentials

[default]
aws_access_key_id = <your-key-here>
aws_secret_access_key = <your-secret-key-here>

[neap]
aws_access_key_id = <your-key-here>
aws_secret_access_key = <your-secret-key-here>

~/.aws/config

[default]
output = json
region = ap-southeast-2

[profile neap]
output = json
region = us-east-1

Configuring the AWS CLI to use AWS SSO

Please refer to the AWS account management guide under the Configuring the AWS CLI to use SSO section.

Profiles

Command Description
aws configure list-profiles Lists all the configured profiles on this machine.

CloudWatch

Sending logs

Full API doc at https://docs.aws.amazon.com/cli/latest/reference/logs/put-log-events.html

First, you must acquire the next sequence token:

aws logs describe-log-streams --log-group-name <LOG GROUP NAME> --log-stream-name-prefix <LOG STREAM NAME>

This would not be required if the log entry was the first one in that log stream, which is obviously very rare.

The output of this command is a JSON similar to this:

{
	logStreams:[{
		/* ... */
		uploadSequenceToken: 323124326485473253783251473246753564562353624536,
		/* ... */
	}]
}
aws logs put-log-events --log-group-name <LOG GROUP NAME> --log-stream-name <LOG STREAM NAME> --log-events timestamp=1610585392937,message="hello world --sequence-token=323124326485473253783251473246753564562353624536"

IMPORTANT:

  • The default profile must have access to cloudwatch.
  • Both <LOG GROUP NAME> and <LOG STREAM NAME> must exist prior to sending the log message.

Elastic Beanstalk

Command Description
aws elasticbeanstalk list-available-solution-stacks Lists the available supported platforms (e.g., 64bit Amazon Linux 2 v5.6.4 running Node.js 14)
aws elasticbeanstalk describe-environments Lists the environments

IAM

Command Description
aws sts get-caller-identity Shows who we are.

Route 53

Command Description
aws route53 list-hosted-zones --profile YOUR_PROFILE List all the hosted zones.
aws route53 list-resource-record-sets --hosted-zone-id HOSTED_ZONE_ID --profile YOUR_PROFILE > yourdns.json Lists all the DNS records for a specific hosted zone ID.

List DNS records

aws route53 list-resource-record-sets --hosted-zone-id <HOSTED ZONE ID> --profile your-profile > yourdns.json

Migrate a domain from one AWS account to another

This operation requires 4 steps:

  1. Create a request to migrate that domain
  2. Create a new accept.json file with the migration request details
  3. Accept the migration transaction
  4. Check is operation status

1. Create a request to migrate that domain

aws route53domains transfer-domain-to-another-aws-account --domain-name <YOUR DOMAIN> --account-id <THE DESTINATION AWS ACCOUNT ID WITHOUT DASHES> --region us-east-1 --profile <THE PROFILE OF THE AWS ACCOUNT THAT CURRENTLY HOLDS THE DOMAIN>

IMPORTANT:

  • The --region us-east-1 flag. This is not a mistake. This API only supports that region, regardless of how your profile is set.
  • The output contains a important password. Copy this value as you need it in the next step.

2. Create a new accept.json file with the migration request details

You could technically skip that step and use the previous credentials explicitely on the command line, but the issue if that the password usually contains characters that must be escaped. If you wrongly escape those characters, the next step fais. Therefore, I personally found it easier to add those credentials in a accept.json JSON file as follow:

{
	"DomainName": "<YOUR DOMAIN>",
	"Password": "<RESPONSE PASSWARD>"
}

This can be done with this command:

echo '{ "DomainName": "<YOUR DOMAIN>", "Password": "<RESPONSE PASSWARD>" }' > accept.json

3. Accept the migration transaction

aws route53domains accept-domain-transfer-from-another-aws-account --cli-input-json file://accept.json --region us-east-1 --profile <THE PROFILE OF THE DESTINATION AWS ACCOUNT>

This command returns an operationId. Copy it to use it in the next step.

4. Check is operation status

aws route53domains get-operation-detail --operation-id "<OPERATION ID>" --region us-east-1 --profile <THE PROFILE OF THE DESTINATION AWS ACCOUNT>

S3

Basic S3 commands

Command Description
aws s3 ls Lists all buckets.
aws s3 sync s3://mybucket .(1) Downloads all the content of a bucket in the current directory ..
aws s3 sync ./myFolder s3://mybucket/myBackup(1) Uploads ./myFolder to s3://mybucket/myBackup.
aws s3 mb s3://your-universally-unique-bucket-name Creates a new bucket.
aws s3 cp your-local-file.txt s3://your-bucket-name Uploads file to a bucket.
aws s3 ls s3://your-universally-unique-bucket-name --recursive --human-readable --summarize List all files in the bucket.
aws s3 cp ./ur_file.txt s3://ur_other_bucket/ur_file.txt Copy paste your local file to S3
aws s3 cp s3://ur_bucket/ur_file.txt s3://ur_other_bucket/ur_file.txt Copy paste your file from S3 to S3

(1) aws s3 sync can only be used to sync folders.

Complex S3 commands

Delete many S3 buckets at once

aws s3api list-buckets --output yaml | grep -Eo "Name:\slu(.*?)$" | cut -c 7- | xargs -L1 aws s3api delete-bucket --bucket $1

Where:

  • aws s3api list-buckets --output yaml lists all the buckets in YAML (easier to apply regex on YAML).
  • grep -Eo "Name:\slu(.*?)$" filters the Name that match that regex.
  • cut -c 7- removes the first 7 characters (i.e., Name:) to isolate the bucket name.
  • xargs -L1 sends the output as the first argument for the following command so we can use $1.
  • aws s3api delete-bucket --bucket $1 deletes the bucket.

SSO

To learn more about AWS SSO, please refer to this document: Managing users access via AWS Single Sign-On.

Command Description
aws configure sso Creates a new SSO profile on this machine.
aws sso login --profile YOUR_PROFILE Renews a SSO session for a specific profile.

Troubleshooting

dyld: Library not loaded

Full message is similar to this:

dyld: Library not loaded: @executable_path/../.Python
  Referenced from: /usr/local/aws/bin/python2.7
  Reason: image not found

That's an AWS CLI bug. You must reinstall or update your CLI to the latest version.

To update, use:

brew upgrade awscli

To fully re-install, please refer to the previous With brew (recommended) section.

FAQ

How to configure the AWS CLI to use AWS SSO?

Please refer to the Configuring the AWS CLI to use AWS SSO section.

Annexes

Most Common Commands

Service Command Description
S3 aws s3 ls List all buckets.
S3 aws s3 mb s3://your-universally-unique-bucket-name Create a new bucket.
S3 aws s3 cp your-local-file.txt s3://your-bucket-name Upload file to a bucket.

Regions

Region name code Endpoint
US East (Ohio) us-east-2 rds.us-east-2.amazonaws.com
US East (N. Virginia) us-east-1 rds.us-east-1.amazonaws.com
US West (N. California) us-west-1 rds.us-west-1.amazonaws.com
US West (Oregon) us-west-2 rds.us-west-2.amazonaws.com
Asia Pacific (Hong Kong) ap-east-1 rds.ap-east-1.amazonaws.com
Asia Pacific (Mumbai) ap-south-1 rds.ap-south-1.amazonaws.com
Asia Pacific (Osaka-Local) ap-northeast-3 rds.ap-northeast-3.amazonaws.com
Asia Pacific (Seoul) ap-northeast-2 rds.ap-northeast-2.amazonaws.com
Asia Pacific (Singapore) ap-southeast-1 rds.ap-southeast-1.amazonaws.com
Asia Pacific (Sydney) ap-southeast-2 rds.ap-southeast-2.amazonaws.com
Asia Pacific (Tokyo) ap-northeast-1 rds.ap-northeast-1.amazonaws.com
Canada (Central) ca-central-1 rds.ca-central-1.amazonaws.com
China (Beijing) cn-north-1 rds.cn-north-1.amazonaws.com.cn
China (Ningxia) cn-northwest-1 rds.cn-northwest-1.amazonaws.com.cn
EU (Frankfurt) eu-central-1 rds.eu-central-1.amazonaws.com
EU (Ireland) eu-west-1 rds.eu-west-1.amazonaws.com
EU (London) eu-west-2 rds.eu-west-2.amazonaws.com
EU (Paris) eu-west-3 rds.eu-west-3.amazonaws.com
EU (Stockholm) eu-north-1 rds.eu-north-1.amazonaws.com
Middle East (Bahrain) me-south-1 rds.me-south-1.amazonaws.com
South America (Sao Paulo) sa-east-1 rds.sa-east-1.amazonaws.com
AWS GovCloud (US-East) us-gov-east-1 rds.us-gov-east-1.amazonaws.com
AWS GovCloud (US-West) us-gov-west-1 rds.us-gov-west-1.amazonaws.com

References

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment