Skip to content

Instantly share code, notes, and snippets.

@danehans
Last active February 25, 2022 20:00
Show Gist options
  • Save danehans/2eb21e8ce284bea2c08eed428ebee87c to your computer and use it in GitHub Desktop.
Save danehans/2eb21e8ce284bea2c08eed428ebee87c to your computer and use it in GitHub Desktop.
Istio Load Testing User Guide

Istio Load Testing User Guide

This guide provides step-by-step instructions for using the setup_perf_cluster.sh load testing script. The script deploys a GKE cluster, an Istio mesh, a GCE VM and runs Fortio on the VM and within the mesh. Fortio is used to perform load testing.

Clone Istio

$ git clone https://github.com/istio/istio.git && cd istio

Prepare the Istio Deployment Manifest and Istio Client

Option A: Build the deployment manifest and istioctl binary:

$ ./install/updateVersion.sh

Follow the steps in the Developer Guide to build the Istio binaries.

Option B: Follow the quickstart guide to install the manifests and istioctl binary. Make sure istioctl in your path is the one matching the downloaded release. For instance, in ~/tmp/istio-0.4.0/ run:

$ ln -s $GOPATH/src/istio.io/istio/tools

Set Your Google Cloud Credentials.

$ export GOOGLE_APPLICATION_CREDENTIALS=/my/gce/creds.json

If you do not have a Google Cloud account, set one up.

Optional: Customize the Deployment

The setup_perf_cluster.sh script can be customized. View the script and modify the default variables if needed. For example, to update the default gcloud zone (us-east4-b):

$ ZONE=us-west1-a

Source the Script

$ source tools/setup_perf_cluster.sh

Note: setup_perf_cluster.sh can be used as a script or sourced and functions called interactively. Inside Google, you may need to rerun setup_vm_firewall multiple times.

Run the Functions

The output of source tools/setup_perf_cluster.sh provides a list of available functions or you can view the functions from within the setup_perf_cluster.sh script. The most common workflow is:

$ setup_all
Obtaining latest ubuntu xenial image name... (takes a few seconds)...
<SNIP>
### Running: istioctl create -n istio -f tools/cache_buster.yaml
Created config denier/istio/denyall at revision 881
Created config checknothing/istio/denyrequest at revision 882
Created config rule/istio/mixercachebuster at revision 883

The deployment is now complete. You can verify the deployment using standard kubectl commands:

$ kubectl get po --all-namespaces
NAMESPACE      NAME                                                   READY     STATUS    RESTARTS   AGE
fortio         fortio1-1966733334-xj5f6                               1/1       Running   0          8m
fortio         fortio2-3044850348-v5f74                               1/1       Running   0          8m
istio-system   istio-ca-1363003450-gvtmn                              1/1       Running   0          7m
istio-system   istio-ingress-1732553340-gv41r                         1/1       Running   0          7m
istio-system   istio-mixer-3192291716-psskv                           3/3       Running   0          8m
istio-system   istio-pilot-3663920167-4ns3g                           2/2       Running   0          7m
<SNIP>

You can now run the performance tests:

$ run_tests

The first test case uses the default loadbalancer and no Istio mesh or Istio Ingress Controller. The following command tells Fortio on the VM to run a load test against the Fortio echo server running in the Kubernetes cluster:

### Running: curl http://$VM_IP/fortio/?json=on&qps=-1&t=30s&c=48&load=Start&url=http://$K8S_FORTIO_EXT_IP:8080/echo

The following arguments are passed to the Fortio server running on the GCE VM:

Argument Description
json=on Sets output in json format
qps=-1 Requested queries per second to "max"
t=30s Requested duration to run load test
c=48 Number of connections/goroutine/threads
qps=-1 Requested queries per second to "max"
load=Start Tells Fortio to be a load generator
url=http://$K8S_FORTIO_EXT_IP:8080/echo The target to load test

The second test case uses the Fortio Ingress with no Istio mesh and the same arguments as the first test:

### Running: curl http://$VM_IP/fortio/?json=on&qps=-1&t=30s&c=48&load=Start&url=http://$NON_ISTIO_INGRESS/echo
$ kubectl get ing/fortio-ingress -n fortio
NAME             HOSTS     ADDRESS          PORTS     AGE
fortio-ingress   *         35.227.193.162   80        4h

The third test case uses the Istio Ingress with the same arguments as the first test:

### Running: curl http://$VM_IP/fortio/?json=on&qps=-1&t=30s&c=48&load=Start&url=http://$ISTIO_INGRESS/fortio1/echo

Compare the test results to understand the load differential between the 3 test cases.

@ldemailly
Copy link

looks like the ingress isn't routing to fortio

can you check the log of the ingress and of the proxy container for srv1 ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment