cli.md

Useful AWS command line tools

awscli

The AWS Command Line Interface (CLI) is a unified tool to manage your AWS services.

Installation details

nb: See http://embedyoutube.org/ for embedding of youtube vids in markdown.

---

AWS awscli training

The best way to learn the awscli tool, is to use it, next, it's to watch the people that created the tool, show it off each year, at the AWS Re-invent conference. These videos are great.

AWS cli training video 2015

AWS cli training video 2015

---

youtube reinvent 2017

AWS Configure

the aws configure command is used to configure the credentials required to allow the cli to sign in to AWS services. More details here

---

Alias files

See https://github.com/awslabs/awscli-aliases/blob/master/alias for aws cli aliases

See here for a sample aws alias file

---

AWS Profiles

TODO

---

Specs

The jmespath specification is what the awscli tool uses to query / filter aws cli commands, using the --query parameter.

e.g.

## Retrieve the AWS roles for the `build` account, but only show the RoleName propery.
AWS_PROFILE=build \
  aws iam list-roles \
  --query Roles[].RoleName

---

Tools

jq - terminal

Command line tool, probably the most used, for querying / filtering json in the terminal. Does not necessarily follow the JMESPATH spec, nor functions.

Quick install:

brew install jq

# Windows install - chocolately
choco install jq

jq queries in the browser

---

jp - terminal

Command line tool, for using the SAME queries (follows the JMESPATH spec)as the awscli command. Quick Install:

brew tap jmespath/jmespath
brew install jp

Example - more at bottom of page

AWS_PROFILE=build aws sts get-caller-identity | jp -u 'Account'

The -u flag is used to ensure the output is NOT quoted. (similar to the raw, -r, flag in jq)

jp github pages

jmespath.org test queries online

---

jpterm - terminal

Command line tool, for using the same queries as the awscli command.

AWS_PROFILE=build aws sts get-caller-identity | jpterm

Use ctrl+p to swap between output modes. (e.g. Output the command OR the result set)

Further details on the jpterm github pages.

---

awscli - very useful tools

Escape an existing json file

Some aws commands require you to encode json, when passing commands.

e.g.

cmd

# Get some sample json
AWS_PROFILE=build aws sts get-caller-identity > id-example.json
cat id-example.json

output

{
    "Account": "1234567890",
    "UserId": "SOMEKEY:d911123",
    "Arn": "arn:aws:sts::1234567890:assumed-role/SomeUser/d911123"
}

encode with escape chars - use jp to_string method

jp -f trust-policy.json "to_string(@)"

output

"{\"Account\":\"972211316688\",\"Arn\":\"arn:aws:sts::1234567890:assumed-role/SomeUser/d911123\",\"UserId\":\"SOMEKEY:d911123:d911123\"}"

Nb: To save using an intermediate file, the file redirect operator can be used, use the following

jp -f <(AWS_PROFILE=build aws sts get-caller-identity)  "to_string(@)"

---

Example decoding of already escaped json string

To do the opposite (decode the json string), use the jq 'fromjson' command.

See the Policy property returned in the json

aws lambda get-policy --function-name my-lambda
{
    "Policy": "{\"Version\":\"2012-10-17\",\"Id\":\"default\",\"Statement\":[{\"Sid\":\"lambda-598abcdefg-90d1-41e6-83e2-90d5b0c6d08a\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"s3.amazonaws.com\"},\"Action\":\"lambda:InvokeFunction\",\"Resource\":\"arn:aws:lambda:ap-southeast-2:1234567890:function:somefunc\",\"Condition\":{\"StringEquals\":{\"AWS:SourceAccount\":\"1234567890\"},\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:s3:::some-bucket\"}}}]}",
    "RevisionId": "abcdefg-3f7a-4fa7-8863-88eca0b4c90c"
}

Now, decode the policy, using the jq fromjson method

aws lambda get-policy \
--function-name tmp_datalake_inventory \
--query "Policy" | jq '. | fromjson'

{
  "Version": "2012-10-17",
  "Id": "default",
  "Statement": [
    {
      "Sid": "lambda-598fbcde-90d1-41e6-83e2-90d5b0c6d08a",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "lambda:InvokeFunction",
      "Resource": "arn:aws:lambda:ap-southeast-2:12345678:function:somefunc",
      "Condition": {
        "StringEquals": {
          "AWS:SourceAccount": "1234567890"
        },
        "ArnLike": {
          "AWS:SourceArn": "arn:aws:s3:::some-bucket"
        }
      }
    }
  ]
}

---

Find all roles containing a given string, e.g. "pipeline"

    AWS_PROFILE=some-aws-profile-name \
    aws iam list-roles \
      --query 'Roles[?contains(RoleName, `pipeline`)].RoleName'

Examples

loop through each region showing uniq ec2 types

This command helps to locate any rogue instances, in other regions, that you may have forgotten about.

export ENV=some-aws-profile-name
for region in $(AWS_PROFILE="${ENV}" aws ec2 describe-regions --query 'Regions[].RegionName | sort(@)' --output text); \
do echo $region; \
    AWS_PROFILE="${ENV}" \
    aws ec2 describe-instances \
        --region "${region}" \
    --filter Name=instance-state-name,Values=running \
    --query 'Reservations[].Instances[].{InstanceType:InstanceType}' \
    | jq -r '.[].InstanceType' \
    | sort \
    | uniq -c \
    | sort -r;
done

ap-northeast-1
ap-northeast-2
ap-south-1
ap-southeast-1
ap-southeast-2
  24 t2.medium
   6 c4.8xlarge
   5 t2.micro
   3 c4.large
   2 m4.large
   1 m3.medium
ca-central-1
eu-central-1
eu-north-1
eu-west-1
eu-west-2
eu-west-3
sa-east-1
us-east-1
us-east-2
us-west-1
us-west-2

---

Match iam roles, with a prefix, e.g. pipeline-* in a given environment

Get INLINE ROLES

#Get INLINE ROLES
export ENV=some-aws-profile-name
for role in $(AWS_PROFILE=${ENV} aws iam list-roles --query "Roles[?contains(RoleName, 'ping')].RoleName" --output text)
do echo ${role}
  for policy in $(AWS_PROFILE="${ENV}" aws iam list-role-policies --role-name="${role}" --query PolicyNames --output text)
  do echo ${role}_${policy}
    AWS_PROFILE=${ENV} aws iam get-role-policy --role-name="${role}" --policy-name="${policy}" \
    > "ROLE-${role}-INLINE_ROLE_POLICY-${policy}.json"
  done
done

Get ATTACHED ROLE POLICIES

(and write them to a file)

export ENV=some-aws-profile-name
for role in $(AWS_PROFILE=${ENV} aws iam list-roles --query "Roles[?contains(RoleName, 'ping')].RoleName" --output text)
do echo ${role}
  for policy in $(AWS_PROFILE="${ENV}" aws iam list-attached-role-policies --role-name="${role}" --query "AttachedPolicies[].PolicyArn" --output text)
  do echo ${role}_${policy} - $(basename ${policy})
    AWS_PROFILE=${ENV} aws iam get-policy-version \
      --policy-arn "${policy}" \
      --version-id $(AWS_PROFILE=${ENV} aws iam get-policy --policy-arn="${policy}" --query "Policy.DefaultVersionId" --output text) \
      > "ROLE-${role}-ATTACHED_POLICY-$(basename ${policy}).json"
  done
done

---

diff 2 json files with different sort orders

# Diff json files with different formatting. Enables all sort of jq tricks, e.g. to diff only a subset of the content.
diff <(jq . --sort-keys ${file_1}) <(jq . --sort-keys ${file_2}) 

awslogs

https://github.com/jorgebastida/awslogs

AWS_PROFILE=someprofile \
   awslogs get /aws/codebuild/some-code-build-thing 
    --start='2h' \
    --filter-pattern "ami" > code-build-stuff-last-2-hours.txt

OR (for interactive tail)

AWS_PROFILE=someprofile awslogs get /aws/lambda/somelambda \
  --start='3h' \
  --filter-pattern="mysearchthing" \
  --query=message
  --watch