patrick0057

Rancher 1.6 IPSec

Troubleshooting

If you are experiencing issues with containers communicating to each other in your Rancher 1.6 environment, your ipsec might be having some issues. In this article I will go over common troubleshooting steps and procedures to correct the problem.

• exec into one of your ipsec-router containers and run the following ipsec test

for i in `curl -s rancher-metadata/latest/self/service/containers/| cut -f1 -d=` ; do ping -c2 `curl -s curl rancher-metadata/latest/self/service/containers/$i/primary_ip` ; done

patrick0057

Overview

When using RKE 0.2.0 and newer, if you have restored a cluster with the incorrect rkestate file you will end up a state where your infrastructure pods will not start. This includes all pods in kube-system, cattle-system and ingress-nginx. As a result of these core pods not starting, all of your workload pods will be unable to function correctly. If you find yourself in this situation you can use the directions below to fix the cluster.

Recovery

1. Delete all service-account-token secrets in kube-system, cattle-system and ingress-nginx namespaces.

{
kubectl get secret -n cattle-system | awk '{ if ($2 == "kubernetes.io/service-account-token") system("kubectl -n cattle-system delete secret " $1) }'
kubectl get secret -n kube-system | awk '{ if ($2 == "kubernetes.io/service-account-token") system("kubectl -n kube-system delete secret " $1) }'

patrick0057

Below I'm archiving a slightly modified quote from Alena Prokharchyk in this Gist as it contains useful information that I would like to reference later.

Memory resources in Kubernetes are mesured in bytes, and can be expressed as an integer with one of these suffixes: E, P, T, G, M, K - decimal suffixes, or Ei, Pi, Ti, Gi, Mi, Ki - binary > suffixes (more commonly used for memory), or omit the suffix altogether (https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory). Lowercase "m" > is not a recommended suffix for memory.

Cluster.requested.memory is comprised of corresponding node.requested.memory. node.requested.memory is the sum of requested.memory of all the pods scheduled on this node.

While most of the nodes have memory with binary suffixes - memory": "2630Mi", two of them have a suffix "m":

node1.acme.com = 3731881984500m >

patrick0057

lssh function

Changelog

• 06.03.2020 - added zsh function that works for WSL
• 04.21.2020 - updated mac os x code to work with zsh and improved instructions.
• 01.02.2020 - added Windows Subsystem for Linux and broke out each OS into its own section for easy copy and paste
• 12.10.2019 - added support for Rancher 1.6 tar.gz files (requires gtar on mac)
• 12.06.2019 - made command lazier by not requiring user to paste the IP.

Description

Quick bash function to make my life easier when sshing into Rancher nodes. Make sure to update your default web browser download directory by modifying line 2 of the script. For mac: brew install findutils

patrick0057

go dumps for Rancher 2.x support

If you are experiencing random crashing of your Rancher 2.x pods or docker container, sometimes Rancher support will ask you to take a go routine dump or a memory dump. Below are the commmands you need to run inside of your Rancher container or pod.

#exec into Rancher pod or container first
mkdir dumps
curl localhost:6060/debug/pprof/goroutine -o dumps/goroutine
curl localhost:6060/debug/pprof/heap -o dumps/heap
curl localhost:6060/debug/pprof/threadcreate -o dumps/threadcreate
curl localhost:6060/debug/pprof/block -o dumps/block
curl localhost:6060/debug/pprof/mutex -o dumps/mutex

patrick0057

working with etcd keys, notes

restore snapshot to work with

docker run --rm -v $(pwd):/data -e ETCDCTL_API=3 -w /data quay.io/coreos/etcd etcdctl snapshot restore snapshot.db

run snapshot to work with

docker run --name etcd -d -v $(pwd):/data -e ETCDCTL_API=3 -w /data quay.io/coreos/etcd

dump keys to debug

patrick0057

Saving a copy of this script here for later use. I found this at the following stackoverflow post.

• https://stackoverflow.com/questions/21752067/counting-open-files-per-process

#!/bin/bash
cd /proc
for pid in [0-9]*
do
    echo "PID = $pid with $(ls /proc/$pid/fd/ | wc -l) file descriptors"
done | sort -rn -k5 | head | while read -r _ _ pid _ fdcount _