Manual test procedure for slow start mode

00-readme.md

Manual test procedure for slow start mode

This document describes manual test procedure for PR projectcontour/contour#4772

The test is executed by using k6. Additionally, it uses InfluxDB to record the performance test results and Grafana to visualize them.

Preparations

Spin up influxdb and grafana using docker-compose:

$ git clone https://github.com/grafana/k6
$ cd k6
$ docker-compose up influxdb grafana

Add new dashboard to Grafana

1. Navigate to http://localhost:3000/.
2. Click "Dashboards" icon in left panel, select + import.
3. Copy content of slowstart-grafana-panel.json into Import via panel json text box and click Load.
4. Click Import.

Deploy echoserver test application:

$ kubectl apply -f https://gist.githubusercontent.com/tsaarni/a22e3d69bf6c31ff0597e9f78454868a/raw/slowstart.yaml

The manifest (attached in this gist) deploys echoserver with 5 replicas. Each instance will respond to any HTTP request with a JSON document that looks like following:

$ http http://echoserver.127-0-0-101.nip.io/
HTTP/1.1 200 OK
content-encoding: gzip
content-type: application/json
...

{
    ...
    "host": "echoserver.127-0-0-101.nip.io",
    "ingress": "",
    "method": "GET",
    "namespace": "default",
    "path": "/",
    "pod": "echoserver-8679774579-q2bxm",
    ...
}

Note that the response document contains the name of the pod that processed the request.

Create test script for k6:

$ cat > slowstart.js <<EOF
import http from 'k6/http';
import exec from 'k6/execution';

export default function () {
    const res = http.get('http://echoserver.127-0-0-101.nip.io');
   if (res.status === 200) {
       exec.vu.tags['pod'] = res.json()['pod'];
   }
};
EOF

The script will send a request to echoserver and for every successful request, it will take the pod name and associate that as a tag to the test event recorded in InfluxDB. This way, we know which pod/replica responded to the request and we can visualize that in Grafana dashboard.

Execute traffic test

Start k6 with the test script:

$ docker run --rm -it --network host -e K6_OUT=influxdb=http://localhost:8086/k6 -v $PWD:/input:ro grafana/k6:latest run --vus 5 --duration 1h /input/slowstart.js

Observe that data about the traffic starts to cumulate in real-time on Grafana dashboard. The dashboard is updated automatically every 10 seconds but it can be manually refreshed by cliking the refresh button on right upper corner.

Kill one of the pods by runnign following on worker:

$ ps -ef|grep echoserver
65532    3741902 3741816 10 20:56 ?        00:01:47 /echoserver
65532    3741910 3741792 10 20:56 ?        00:01:47 /echoserver
65532    3742037 3741954 10 20:56 ?        00:01:47 /echoserver
65532    3742229 3742092 10 20:56 ?        00:01:47 /echoserver
65532    3742236 3742137 10 20:56 ?        00:01:47 /echoserver
tsaarni  3746917    9433  0 21:13 pts/1    00:00:00 grep --color echoserver
$ sudo kill 3741902

Observe in Grafana dashboard that one of the pods entered slow start mode for 10 seconds.

Example

In the following screenshot an echoserver process in pod represented by green color was killed. The number of requests goes down to 0 and increases gradually back to original number within slow start window which was configured to 10 seconds.

slowstart-grafana-panel.json

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "datasource",
          "uid": "grafana"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "target": {
          "limit": 100,
          "matchAny": false,
          "tags": [],
          "type": "dashboard"
        },
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 2,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "influxdb",
        "uid": "P6C0B0CFA7252F455"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "bars",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "normal"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 14,
        "w": 16,
        "x": 0,
        "y": 0
      },
      "id": 5,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "influxdb",
            "uid": "P6C0B0CFA7252F455"
          },
          "groupBy": [
            {
              "params": [
                "$__interval"
              ],
              "type": "time"
            },
            {
              "params": [
                "pod"
              ],
              "type": "tag"
            },
            {
              "params": [
                "null"
              ],
              "type": "fill"
            }
          ],
          "measurement": "http_reqs",
          "orderByTime": "ASC",
          "policy": "default",
          "query": "SELECT count(\"value\") FROM \"http_reqs\" WHERE $timeFilter GROUP BY time($__interval), \"pod\" fill(null)",
          "rawQuery": false,
          "refId": "A",
          "resultFormat": "time_series",
          "select": [
            [
              {
                "params": [
                  "value"
                ],
                "type": "field"
              },
              {
                "params": [],
                "type": "count"
              },
              {
                "params": [
                  10
                ],
                "type": "moving_average"
              }
            ]
          ],
          "tags": []
        }
      ],
      "title": "Requests per second",
      "type": "timeseries"
    }
  ],
  "refresh": "10s",
  "schemaVersion": 37,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-5m",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": [
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ]
  },
  "timezone": "",
  "title": "Slow start test",
  "uid": "Le2Ku9NMx",
  "version": 2,
  "weekStart": ""
}

slowstart.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: echoserver
  replicas: 5
  template:
    metadata:
      labels:
        app.kubernetes.io/name: echoserver
    spec:
      containers:
        - name: echoserver
          image: gcr.io/k8s-staging-ingressconformance/echoserver:v20210922-cec7cf2
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http-api
              containerPort: 3000
          # readinessProbe:
          #   httpGet:
          #     path: /health
          #     port: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: echoserver
spec:
  ports:
    - name: http
      port: 80
      targetPort: http-api
  selector:
    app.kubernetes.io/name: echoserver
---
apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: echoserver
spec:
  virtualhost:
    fqdn: echoserver.127-0-0-101.nip.io
  routes:
    - services:
        - name: echoserver
          port: 80
          slowStart:
            window: 10s