adleong

# Two backend pods, one always failing
# and another one returning OK response
# Slowcooker is used to generate traffic
# that will be routed via traffic split
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
spec:

adleong

# Step 1: Install Linkerd
# Step 2: Install Linkerd Viz (optional, but helpful for seeing what's going on)
# Step 3: Install Linkerd Jaeger
# Step 4: Install Nginx Ingress:

linkerd inject https://gist.githubusercontent.com/adleong/f7318d9282ab99ba12c3479405882e60/raw/713daaa51f035d0d484ea79daf77cc3aaded8010/yml | kubectl apply -f -
# Notice the tracing related config in the nginx config configmap in the aboe gist: https://gist.github.com/adleong/f7318d9282ab99ba12c3479405882e60#file-yml-L340

# Step 5: Install Emojivoto 

adleong

apiVersion: v1
kind: Namespace
metadata:
  name: emojivoto
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: emoji
  namespace: emojivoto

adleong

apiVersion: v1
kind: Namespace
metadata:
  labels:
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  name: ingress-nginx
---
apiVersion: v1
automountServiceAccountToken: true

adleong

---
apiVersion: policy.linkerd.io/v1beta1
kind: Server
metadata:
  namespace: emojivoto
  name: emoji-grpc
  labels:
    app.kubernetes.io/part-of: emojivoto
    app.kubernetes.io/name: emoji
    app.kubernetes.io/version: v11

adleong

---
apiVersion: policy.linkerd.io/v1beta1
kind: Server
metadata:
  name: proxy-admin
  labels:
    linkerd.io/extension: viz
spec:
  podSelector:
    matchExpressions:

adleong

#!/bin/bash

pods=$(kubectl get po -A -l linkerd.io/control-plane-ns -ojsonpath="{range .items[*]}{.metadata.name} {.metadata.namespace}{'\n'}{end}")

IFS=" "

while read name namespace; do
  tcp=$(kubectl debug -n $namespace $name --image=cr.l5d.io/linkerd/debug:stable-2.12.0 -it -- cat /proc/net/tcp)
  close_wait=$(echo $tcp | awk 'BEGIN {cnt=0} $4==08 {cnt++} END {print cnt}')
  fin_wait_2=$(echo $tcp | awk 'BEGIN {cnt=0} $4==05 {cnt++} END {print cnt}')

adleong

An unmeshed client (10.8.2.27:45090) sends an HTTP request with curl to a meshed server (10.8.2.26:5000)
the client gracefully closes the connection before receiving a response and the client OS sees the socket in state FIN-WAIT-2
the server OS sees the corresponding inbound proxy socket (10.8.2.26:4143) in state CLOSE-WAIT
it stays this way for around 85 seconds before disappearing from the conntrack table

adleong

Linkerd Stale Discovery Runbook

Identifying if you have stale endpoints

If you have a pod experiencing unexplained 503s, check the proxy logs from that pod. If you see connection errors to IP addresses which do not correspond to running pods, your Linkerd proxy likely has stale endpoints. The IP addresses with the connection errors likely correspond to pods which have been recently deleted.

adleong

/** Simple scala client */

import $ivy.`com.twitter::finagle-http:6.44.0`
import $ivy.`io.buoyant::finagle-h2:1.1.0`
import com.twitter.conversions.time._
import com.twitter.finagle.{Status => _, _}
import com.twitter.finagle.buoyant.h2._
import com.twitter.finagle.buoyant.H2
import com.twitter.io.Buf
import com.twitter.util._