Tools of trade

README.md

Tools of trade

AWS

Cores and Threads per instance type

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/cpu-options-supported-instances-values.html

Get managed roles ARN

aws iam list-policies --query "Policies[][Arn]" --output text

Export credentials to shell

aws-export-session-token() {
  jq '.Credentials | .AccessKeyId, .SecretAccessKey, .SessionToken' -r | \
  paste -d '=' <(echo AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN | xargs -n1) - | \
  sed -e 's/^/export /' | sed -e 's/$/;/'
}

aws-export-creds() {
  paste -d '=' <(echo AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY | xargs -n1) - | \
  sed -e 's/^/export /' | sed -e 's/$/;/'
}

Example

# https://awscli.amazonaws.com/v2/documentation/api/latest/reference/configure/export-credentials.html
$(aws configure export-credentials --format env)

$(aws sts get-session-token | aws-export-session-token)

$(aws sts assume-role | aws-export-session-token)

$ cat <<'EOF' | aws-export-creds
some-access-key
some-secret-access-key
EOF
export AWS_ACCESS_KEY_ID=some-access-key;
export AWS_SECRET_ACCESS_KEY=some-secret-access-key;

$ cat <<'EOF' | aws-export-session-token
{
    "Credentials": {
        "AccessKeyId": "some-access-key",
        "SecretAccessKey": "some-secret-access-key",
        "SessionToken": "some-session-token",
        "Expiration": "2019-12-09T16:39:22Z"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "some-assumed-role",
        "Arn": "some-arn"
    }
}
EOF
export AWS_ACCESS_KEY_ID=some-access-key;
export AWS_SECRET_ACCESS_KEY=some-secret-access-key;
export AWS_SESSION_TOKEN=some-session-token;

AWS Flog Log search pattern

[version, account, eni, source, destination, srcport, destport, protocol, packets, bytes, windowstart, windowend, action, flowlogstatus]

ACM validation

dig TXT <record>

DNS

• dig +short <some.domain.name>
• nslookup <some.domain.name>
• host <some.domain.name>
• systemd-resolve <some.domain.name>
• dns-sd -G v4 <some.domain.name>

Differences/subtleties can be found, as always, on Archlinux Wiki. Also look at the manpages. And even this blog post series.

Load balancing

• What to consider when choosing one: https://cloud.google.com/load-balancing/docs/choosing-load-balancer#flow_chart

Log formats

• AWS Classic Load Balancer
• AWS Application Load Balancer
• AWS Network Load Balancer
• AWS VPC Flow
• Envoy Proxy
• Apache HTTP srever
• NGINX

TODO: Add examples

Reference tables

• Envoy response flags:
  • UF: Upstream connection failure in addition to 503 response code.
  • UC: Upstream connection termination in addition to 503 response code.
  • DC: Downstream connection termination.

Kubernetes

Add a user

NAME=`whoami`
NAMESPACE=default

kubectl create serviceaccount -n $NAMESPACE ${NAME}-service-account
kubectl create clusterrolebinding ${NAMESPACE}:${NAME}:cluster-admin --clusterrole=cluster-admin --serviceaccount=${NAMESPACE}:${NAME}-service-account

TOKEN=$(kubectl get secret \
   $(kubectl get serviceaccount ${NAME}-service-account \
       -n $NAMESPACE \
       -o jsonpath='{.secrets[0].name}') \
   -n $NAMESPACE \
   -o jsonpath='{.data.token}' | base64 --decode)

# Create a kubeconfig to the user based on yours
CONTEXT=$(kubectl config current-context)
kubectl config view --raw --minify > ./${NAME}-service-account-kubeconfig.yaml
kubectl config unset users
kubectl config set-credentials ${NAME}-token-user --token $TOKEN
kubectl config set-context $CONTEXT --user ${NAME}-token-user

Get pods on a node

kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=<node>

Get pods and IPs

kubectl get pods --all-namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.podIP}{"\n"}'

Get services and IPs

kubectl get services --all-namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.clusterIP}{"\n"}'

Get endpoints and IPs

kubectl get endpoints --all-namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.subsets[*].addresses[*].ip}{"\n"}'

Verbatim scripts as pod commands

command:
- /bin/sh
- -c
args:
- |-
  echo "your command goes here"
  echo "and here"

Pod without custom image

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: your-pod-script
data:
  install.sh: |-
    echo "Your content goes here"
  entrypoint.sh: |-
    echo "Your content goes here"
---
apiVersion: v1
kind: Pod
metadata:
  name: your-pod
spec:
  containers:
    - name: container
      image: alpine:latest
      command:
      - /bin/sh
      - -c
      args:
      - |-
        /bin/sh /src/install.sh
        /bin/sh /src/entrypoint.sh
      volumeMounts:
      - name: script-volume
        mountPath: /src
    volumes:
      - name: script-volume
        configMap:
          name: your-pod-script

Browser CRDs documentation

https://doc.crds.dev/

Validate KIAM config

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<role-arn>

What to monitor?

• https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/alerts
  Example: https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/alerts/resource_alerts.libsonnet#L110

Envoy

• https://www.envoyproxy.io/docs/envoy/v1.16.2/intro/life_of_a_request

Prometheus

List all metrics for a container / target

group by(__name__)({__name__!="", namespace="<NS>", container="<CONT>"})

Add pod's labels to a query

<query-with-pod-label> * on (pod) group_left(<labels-you-want>) kube_pod_labels

• * will match each instant-vector, and multiply the value by 1 (the value of each vector in kube_pod_labels)

Ex: count(kube_pod_container_status_running{container="istio-proxy"}) by (namespace, pod) * on (pod) group_left(label_app) kube_pod_labels

Docs:

• https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md
• https://github.com/kubernetes/kube-state-metrics/tree/master/docs#documentation

Istio

Default limits

From Envoy

• Connection (idle) timeout: 1 hour.
  The idle timeout is the time at which a downstream or upstream connection will be terminated if there are no active streams.
• Stream/request timeout: unspecified.
  The amount of time the connection manager will allow for the entire request stream to be received from the client. (1 HTTP/1 request = 1 HTTP/2 stream)
• Stream/request idle timeout: 5 minutes.
  The amount of time that the connection manager will allow a stream to exist with no upstream or downstream activity.
• Stream maximum duration: unspecified.
  The maximum time that a stream’s lifetime will span.
• Route timeout: 15 seconds.
  The amount of time that Envoy will wait for the upstream to respond with a complete response.
• Route idle timeout: unspecified.
  Overrides stream idle timeout for a route.
• Per try timeout: unspecified.
  Overrides request timeout for retries.
• TCP connection timeout: unspecified.
  The amount of time Envoy will wait for an upstream TCP connection to be established. Includes TLS handshake.
• TCP idle timeout: 1 hour.
  The amount of time that the TCP proxy will allow a connection to exist with no upstream or downstream activity.

More details: https://www.envoyproxy.io/docs/envoy/latest/faq/configuration/timeouts

TODO: How do they apply to upstream and downstream connections?

From Istio

• VirtualService request timeout (ISTIO_DEFAULT_REQUEST_TIMEOUT): 0s (disabled).
  (source)

Kafka

Validate an Avro schema

avro-tools canonical schema.avro.json - | jq

Generate random record

SCHEMA=schema.avro.json
avro-tools random --count 1 --schema-file "${SCHEMA}" - \
| avro-tools tojson --head 1 --pretty --reader-schema-file "${SCHEMA}" -

Security

Random string

export HISTIGNORE="&:random-string*"
random-string() {
  LEN=${1:-64}
  cat /dev/urandom \
    | LC_ALL=C tr -dc $'a-zA-Z0-9 !"#$%&()*+,-./:;<=>?@[\]^_`{|}~\'' \
    | fold -w "${LEN}" \
    | head -n 1
}

ALWAYS use the clipboard for sensitive information

It will build muscle memory to protect you from leaking secrets into the history file.

vault write secrets/aws-root-account password="$(pbpaste)"

Hide commands from history

# zsh
setopt HIST_IGNORE_SPACE

# bash
export HISTCONTROL=ignoreboth

And prepends your command with a space. Example:

# This will show
export SECRET_ON_HISTORY="foo"

# This wont show
 export SUPER_SECRET="blah"

history | tail -n 3

Alternatively, you can disable every command that matches a pattern:

hide() { eval $(printf "%q " "$@") }

# zsh
export HISTORY_IGNORE="&:hide*"

# bash
export HISTIGNORE="&:hide*"

# This wont show
hide vault write secret/

Alternatively, you can disable history for the rest of the session:

unset HISTFILE

shell-fu

# Print lines containing at least N occurences of Y
# (print log lines with more than 1 commas
$ cat logs.txt | awk -F, 'NF > 2'

Terraform

# Backup state
aws s3 cp s3://<state-bucket>/<state-file> ./

# Get latest version
V=$(aws s3api list-object-versions \
  --bucket <state-bucket> \
  --prefix <state-file> \
  --query 'Versions[0].VersionId' --text)

# Revert to previous version
aws s3api delete-object \
  --bucket <state-bucket> \
  --prefix <state-file> \
  --version-id "${V}"

Calculators

Sample size calculator

https://www.surveymonkey.com/mp/sample-size-calculator/

RegExp

https://regex101.com/

Debug JWT

https://jwt.io

SLA calculator

https://uptime.is/

A/B test calculators

https://www.evanmiller.org/ab-testing/

HTTP.md

Keep-Alive

time telnet www-org.example.com 80

GET / HTTP/1.1
Host: www-org.example.com
Connection: keep-alive

(wait)

time openssl s_client -connect www-org.example.com:443

GET / HTTP/1.1
Host: www-org.example.com
Connection: keep-alive

(wait)

LINUX-PERF.md

Linux performance analysis in 60s

uptime
dmesg | tail
vmstat 1
mpstat -P ALL 1
pidstat 1
iostat -xz 1
free -m
sar -n DEV 1
sar -n TCP,ETCP 1
top

uptime

• tasks (process/thread) wanting to run, and usually includes blocked.
• moving average: 1min, 5min, 15min

Look for saturation

uptime
21:34  up 10 days, 13:43, 2 users, load averages: 2.57 1.83 1.72

dmesg | tail

• last 10 system messages.

Look for errors

vmstat 1

virtual memory stat, 1 sec summary.

• r > CPUS -> saturation
• si or so -> saturation
• us, sy, wa, st -> CPU use and I/O saturation

mpstat -P ALL 1

CPU time breakdown

LINUX.md

Procfs

cat /proc/1/environ | tr '\000' '\n'

cat /proc/1/cmdline | tr '\000' '\n'

TOOLS.md

Tools

SSL

Certificate Transparency logs

https://censys.io/certificates

Display SSL handshake

DST="google.com"
echo "Q" | openssl s_client -connect ${DST}:443 -servername ${DST} -tlsextdebug

Obtain SSL certificate

DST="google.com"
echo "Q" | openssl s_client -connect ${DST}:443 -servername ${DST} -showcerts 2>/dev/null

Display SSL certificate

pbpaste | openssl x509 -text -noout

Curl and SSL

https://ec.haxx.se/usingcurl-tls.html

SSH

Keys in ssh-agent

ssh-add -l
ssh-add -L

Display signed user certificates

pbpaste | ssh-keygen -Lf -

Display fingerprint

# get the SHA256 and ascii art    
ssh-keygen -l -v -f /path/to/publickey

# get the MD5 for private key
openssl pkey -in /path/to/privatekey -pubout -outform DER | openssl md5 -c

# get the MD5 for public key
openssl pkey -in /path/to/publickey -pubin -pubout -outform DER | openssl md5 -c

https://developer.okta.com/blog/2019/10/21/illustrated-guide-to-oauth-and-oidc

https://search.censys.io/certificates?q=parsed.names%3A+example.dev