statefulset label pattern

README.md

Tenant IPs

• tenant IPs are stored in ips.txt
• These are used to create the number of services and the number of replicas for a statefulset (1 tenantip = 1 pod)

Create Services

• Running ./update-service.sh a for the first time will create the services
• The files ips.txt contains the list of clusterIPs to set, which also creates a label and selector
• Passing an a or b will update the selector to use the pods in group a or b

Create Pods

• Create statefulsets
• Increase replicas for active set using patch
  • replicas are the number of tenatips in ips.txt
• For a/b deployment
  • Increaes replicas for inactive set using patch
  • Update labels for inactive group (see next step)
  • Update service selector to use group=${activegroup}

Update Labels

• Pods in a statefulset/deployment are applied labels, but not individual labsl which will allow for specific tenantIP data loading
• Deploy the statefulset first, and it will go into "init running" while it waits for a tenantip label to be applied
• The update-labels.sh script will read the IPs in ips.txt and apply a label to each pod with each one

Workflow

• Deploy service with activegroup a ./udpate-service.sh a kubectl apply -f service.yaml
• Deploy statefulset, by default they are set to 0 kubectl apply -f ./statefulSet.yaml
• scale up active statefulset to match the number of ips in ips.txt group=a; replicas=$(wc -l ./ips.txt | awk {'print $1'}); kubectl patch statefulsets analytics-server-${group} -p "{\"spec\":{\"replicas\":${replicas}}}"
• Update the running statefulset to read the ips.txt and apply a tenantIP to use for group a ./update-labels.sh a
• Pods will go from Init to Running and viewing the labels on the pod will show each one has a unique clusterIPs
• Verify services are going to correct pods kubectl get svc `kubectl describe svc analytics-server-172-20-123-221 | grep Endpoints`` - This will show that the Endopint is going to the pod with labels group=a and the tenantip=${clusterIP}

ips.txt

172.20.33.43
172.20.98.120
172.20.128.32
172.20.123.221
172.20.90.98

statefulSet.yaml

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: analytics-server
    environment: development
    group: a
  name: analytics-server-a
spec:
  podManagementPolicy: Parallel
  replicas: 0
  selector:
    matchLabels:
      app: analytics-server
  serviceName: analytics-server
  template:
    metadata:
      labels:
        app: analytics-server
        group: a
    spec:
      #init container to wait until the label tenantip is applied to a pod
      initContainers:
      - name: init-analytics-server
        image: debian:buster
        command:
          - bash
          - "-c"
          - |
            /bin/bash << 'EOF'
            until grep tenantip /etc/podinfo/labels; do
              sleep 1
            done

            EOF
        volumeMounts:
        - name: podinfo
          mountPath: /etc/podinfo
      containers:
      - name: analytics-server
        command: ["sh"]
        args: ['-c', "tail -f /dev/null"]
        image: debian:buster
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: 100Mi
          requests:
            cpu: 100m
        env:
        - name: KUBERNETES_SERVICE_IP
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['tenantip']
      volumes:
        - name: podinfo
          downwardAPI:
            items:
              - path: "labels"
                fieldRef:
                  fieldPath: metadata.labels
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: analytics-server
    environment: development
    group: b
  name: analytics-server-b
spec:
  podManagementPolicy: Parallel
  replicas: 0
  selector:
    matchLabels:
      app: analytics-server
  serviceName: analytics-server
  template:
    metadata:
      labels:
        app: analytics-server
        group: b
    spec:
      #init container to wait until the label tenantip is applied to a pod
      initContainers:
      - name: init-analytics-server
        image: debian:buster
        command:
          - bash
          - "-c"
          - |
            /bin/bash << 'EOF'
            until grep tenantip /etc/podinfo/labels; do
              sleep 1
            done

            EOF
        volumeMounts:
        - name: podinfo
          mountPath: /etc/podinfo
      containers:
      - name: analytics-server
        command: ["sh"]
        args: ['-c', "tail -f /dev/null"]
        image: debian:buster
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: 100Mi
          requests:
            cpu: 100m
        env:
        - name: KUBERNETES_SERVICE_IP
          valueFrom:
            fieldRef:
              fieldPath: metadata.labels['tenantip']
      volumes:
        - name: podinfo
          downwardAPI:
            items:
              - path: "labels"
                fieldRef:
                  fieldPath: metadata.labels

update-labels.sh

#!/bin/bash

case ${1} in
  a|b)
    activegroup=${1}
    podprefix="analytics-server-${activegroup}"
  ;;
  *)
    echo "An active group a or b is required"
    exit 1
  ;;
esac

echo "Reading pods with ${podprefix}"
for pod in $(kubectl get pods | grep ${podprefix} | awk {'print $1'}); do
  for tenantip in $(cat ./ips.txt); do
    if echo ${usedip} | grep ${tenantip} >/dev/null 2>&1; then
      :
    else
      echo "kubectl label pods ${pod} tenantip=${tenantip}"
      kubectl label pods ${pod} tenantip=${tenantip}
      usedip="${usedip} ${tenantip}"
      break
    fi
  done
done

update-service.sh

#!/bin/bash

case ${1} in
  a|b)
    activegroup=${1}
  ;;
  *)
    echo "An active group a or b is required"
    exit 1
  ;;
esac

echo "---
apiVersion: v1
kind: Service
metadata:
  name: analytics-server
  labels:
    app: analytics-server
spec:
  ports:
  - name: http
    port: 8888
    protocol: TCP
    targetPort: 8081
  selector:
    app: analytics-server
  sessionAffinity: None
  clusterIP: None" > service.yaml

for tenantip in $(cat ./ips.txt); do
  svcname=$(echo ${tenantip} | sed -e "s/\./-/g")
  echo "Updating analytics-server-${svcname} to active group ${activegroup} in service.yaml"
  echo "---
apiVersion: v1
kind: Service
metadata:
  name: analytics-server-${svcname}
  labels:
    app: analytics-server
    group: ${activegroup}
spec:
  ports:
  - name: http
    port: 8888
    protocol: TCP
    targetPort: 8081
  selector:
    app: analytics-server
    group: ${activegroup}
    tenantip: ${tenantip}
  clusterIP: ${tenantip}
  sessionAffinity: None" >> service.yaml
done