Playing with kubectl output

kubectl.md

Kubectl output options

Let's look at some basic kubectl output options.

Our intention is to list nodes (with their AWS InstanceId) and Pods (sorted by node).

We can start with:

kubectl get no

and

kubectl get po -o wide

Json and Jq

I've found the internal data structures easier to explore using the -o json output with jid and jq.

Once both jq and jid are installed (assuming OSX), we can quickly discover the data with the following command:

kubectl get no -o json | jid -q | pbcopy

This allows us to explore the json data interactively and keep our final jq query on the clipboard:

note: jid currently implements it's own query parser to allow powerfull autocompletion, the drawback is a lack of support for all the jq constructs (i.e.: we have to specify an index for array elements during discovery).

As can be seen in the recording: once done with jid, getting rid of the index on the items array in jq, did gave us the full listing.

jq gives us a lot more power for example:

Boxing the result into it's own array and constructing a new object combining several nested attributes gives us the following query:

kubectl get no -o json | jq -r '[.items[] | {name:.metadata.name, id:.spec.externalID, unschedulable:.spec.unschedulable}]'

Here is how the above query was built up using jid and jq:

Converting the json array into a tabular output with jq can be done using @tsv as follows:

kubectl get no -o json | jq -r '.items[] | select(.spec.unschedulable!=true) | [.metadata.name,.spec.externalID] | @tsv'

Jq also allows us to sort:

kubectl get po -o json | jq -r '.items | sort_by(.spec.nodeName)[] | [.spec.nodeName,.metadata.name] | @tsv'

The input for the sort_by command must be an array, we iterate the elements after the sorting.

Custom Columns and Sorting

If all we need is a nicely formatted, sorted tabular report, kubectl has built-in support for powerfull sorting:

kubectl get po -o wide --sort-by=.spec.nodeName

Using jid to list pods sorted by node:

The usage of Custom Columns with the knowledge of the data structure gained from jid, is also much easier:

kubectl get no -o=custom-columns=NAME:.metadata.name,AWS-INSTANCE:.spec.externalID,UNSCHEDULABLE:.spec.unschedulable

Note: apart from using grep, there is no easy way to filter.

Golang Templates

If we do not wish to use jq (or have no access to jq) need filtering and powerfull output control, we may use Kubectl's built-in support for golang templates (inline or from a template file on disk):

kubectl get no -o go-template='{{range .items}}{{if .spec.unschedulable}}{{.metadata.name}} {{.spec.externalID}}{{"\n"}}{{end}}{{end}}'
or
kubectl get no -o go-template="{{range .items}}{{if .spec.unschedulable}}{{.metadata.name}} {{.spec.externalID}}:{{end}}{{end}}" | tr ":" "\n"

I could not find an easy way to print newline characters with inline golang template, so used a trick printing colons and using tr to convert colons to newlines.

JSONPath

Golang templates can be complicated and verbose - an alternative, if you are more familiar with jq-style queries, or awscli, is to use JSONPath.

kubectl get no -o jsonpath="{.items[?(@.spec.unschedulable)].metadata.name}"

Internally, this seems tightly coupled to the golang templates.

Kubectl supports a superset of JSONPath, with a special range keyword to iterate over ranges, using the same trick to add newlines:

kubectl get no -o jsonpath="{range.items[?(@.spec.unschedulable)]}{.metadata.name}:{end}" | tr ":" "\n"

More examples of using jsonpath can be found in the Kubernetes tests for the JSONPath utility

@boomkap - sorry, I didn't use quota - did you try\ as escape character?

to check image used by containers in pod:

k get po -o custom-columns=name:.metadata.name,status:.status.phase,image:'.spec.containers[0].image'
name                 status    image
frontend             Running   1234567.dkr.ecr.ap-southeast-1.amazonaws.com/frontend:0.1.0-058d8d4
...

@so0k - does not work with \ as an escape character. Throws the following error

template: output1: bad character U+005c

@boomkap I think this what you are looking for, a command to output the total number of Pods running, CPU request/limits and Memory requests/limits for each projects in a cluster.

oc get quota --all-namespaces -o=custom-columns=Project:.metadata.namespace,TotalPods:.status.used.pods,TotalCPURequest:.status.used.requests'\.'cpu,TotalCPULimits:.status.used.limits'\.'cpu,TotalMemoryRequest:.status.used.requests'\.'memory,TotalMemoryLimit:.status.used.limits'\.'memory

🙂

kubectl get no -o json | jq -r '.items | sort_by(.status.capacity.memory)[]|[.metadata.name,.status.capacity.memory]| @TSV'

how to display in MB or GB, currently is displaying in kb. ?

Thanks.

save ConfigMap into files, per file each key,using key as filename.

cm_name=config
ns=default
files=$(kubectl get cm $cm_name -n $ns -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}')
for f in $(echo $files);do
  kubectl get cm $cm_name -n $ns -o go-template='{{range $k,$v := .data}}{{ if eq $k "'$f'"}}{{$v}}{{end}}{{end}}' > $f 2>/dev/null
done

Use jq to display Limits/Allocation Ratio for a given Kubernetes namespace quota:

kubectl get quota -n my-namespace -o json | jq -r "$(curl -ks https://gist.githubusercontent.com/berttejeda/e89da293d9b3ed8183948caac2bdd09c/raw/9dc2617baaa7f389ad8cbd92d3a7049169ad0cbc/k8s.quotas.jq)"

Sample output:

[
  {
    "namespace": "my-namespace",
    "quotaname": "default-12a3b",
    "limitsCpu": "360",
    "limitsCpu_used": "9",
    "limitsCpu_Allocation_Ratio": 2.5,
    "limitsMemory": "414Gi",
    "limitsMemory_used": "20Gi",
    "limitsMemory_Allocation_Ratio": 4.830917874396135,
    "limitsPod": "225",
    "limitsPods_used": "13",
    "limitsPod_Allocation_Ratio": 5.777777777777778
  }
]

Sum up all requested CPUs

k get pods -o json | jq -r '[.items[] | .spec.containers[].resources.requests.cpu | rtrimstr("m") | tonumber] | add'

I also found this deep dive on using Go template files to render the output of kubectl commands (oc builds on kubectl). There is nothing OpenShift about this, it would have been better named as "Kubectl and Go Templates workshop"

https://github.com/brandisher/openshift-and-gotemplates-workshop

I had to add a -A and select on top of what @Leo7654 recommended, to get across all namespaces, and filter out empty results.

k get pods -A -o json | jq -r '[.items[] | .spec.containers[].resources.requests.cpu | rtrimstr("m") | select(length > 0) | tonumber] | add'

I've got a large number of pods and would like to see the output sorted a certain way. I'd like to sort the output based on multiple fields when there's a tie.

Is this possible using --sort-by?

I'm currently using bash scripts to do this but was hoping that a newer version (my cluster is still at v1.18.10) of Kubernetes can do this natively.

@Guillaume-Mayer you should do the following command to fix that:

stdbuf -o0 kubectl logs -f mypod | jq

Hi Folks,

I am trying to fetch external LB names only but condition is not working as expected,
Can someone help me to get the data of external LB only.

kubectl get services --all-namespaces -o json | jq -r '.items[] | select(.spec.type=="LoadBalancer")| select (.spec | with_entries(select(.key | contains("loadBalancerSourceRanges")))) | [.metadata.name] | @TSV'

Getting all LB not running second condition

kubectl get services --all-namespaces -o json |jq '.items[*].metadata.annotations | with_entries(select(.key | startswith("networking.gke.io/"))) | map_values(fromjson)'

Thanks in advance

nodes sorted by lowest pod count using https://stackoverflow.com/questions/58201823/kubernetes-display-current-pods-vs-capacity-with-kubectl

kubectl get po -A -o json | jq -r '.items | sort_by(.spec.nodeName)[] | [.spec.nodeName,.metadata.name] | @csv' | awk -F, '{arr[$1]++}END{for (a in arr) print a, arr[a]}' | sort -nk2

Or using jq only

kubectl get po -A -o json | jq -r '[.items | sort_by(.spec.nodeName)[] | {"node":.spec.nodeName,"pod":.metadata.name}] | [group_by(.node)[] | {"node":.[0].node,"count":length}] | sort_by(.count)[] | [.node,.count] | @tsv'

worth mentioning you can also just hit enter while editing the command:

❯ k get nodes -lnode-role.kubernetes.io/master= -o go-template='{{ range .items }}
quote> {{ index .metadata.labels "kubernetes.io/hostname"}}{{end}}'

ip-10-0-1-1
ip-10-0-1-2
ip-10-0-1-3