This guide provides step-by-step instructions for running Istio E2E Tests on Cisco Cloud Platform (CCP). The primary difference between running E2E tests on CCP and a standard E2E test environment, is the presence of the use_local_cluster argument. This argument instructs the E2E test framework to use type: NodePort instead of type: LoadBalancer for the Istio Ingress Service.
Until an Istio release is cut that includes commit a453938, E2E tests must be run from source. Docker images have been created and pushed to the danehans hub based on commit bfe1078, which includes the required commit (a453938). This will allow E2E test users to skip running Unit Tests, building/pushing Docker images for the Istio components, and provides a stable checkpoint for testing.
Most of the work for E2E involves preparing the host to run the tests. E2E tests can be run from any host that meets the following requirements:
- Network connectivity to the Kubernetes API server.
- Go installed.
- git installed.
- make installed.
- Environment variables set according to the Istio Developers Guide.
- Download the Istio source code, switch to the project directory and checkout commit bfe1078:
git clone https://github.com/istio/istio.git cd istio git reset --hard bfe10785be332491418ec81a5eddc3b4c19fcaf1 - The kubectl client installed.
- The kubectl client must have credentials to access the Kubernetes cluster. The credentials should be stored at
${HOME}/.kube/config. - Docker installed and running. Reference the Docker installation guide based on your Linux distro for details.
SSH to the node that will be used to run E2E tests:
ssh [email protected]
Verify access to the Kubernetes API Server:
kubectl cluster-info
Kubernetes master is running at https://$MY_K8S_API_SERVER:6443
<SNIP>
You need these permissions to create the necessary RBAC rules for Istio:
kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=admin
Build the istioctl client binaries. The binaries for Mac, Linux and Windows are stored in ${GOPATH}/out/linux_amd64/release/. This guide uses the Linux client named istioctl-linux:
make istioctl-all
The Istio Developers Guide uses the following environment variables:
export HUB="docker.io/$USER"
export TAG=$USER
These environment variables must be updated to use docker.io/danehans based on commit bfe1078:
export HUB="docker.io/danehans"
export TAG=bfe10785be332491418ec81a5eddc3b4c19fcaf1
When deploying the Istio Ingress to an environment such as CCP that does not use a cloud load-balancer, you must update the default Helm chart template to use type: NodePort and specify a static nodePort such as 32000:
git diff install/kubernetes/helm/istio/charts/ingress/templates/service.yaml
diff --git a/install/kubernetes/helm/istio/charts/ingress/templates/service.yaml b/install/kubernetes/helm/istio/charts/ingress/templates/service.yaml
index ae4a988..28bc384 100644
--- a/install/kubernetes/helm/istio/charts/ingress/templates/service.yaml
+++ b/install/kubernetes/helm/istio/charts/ingress/templates/service.yaml
@@ -12,10 +12,11 @@ spec:
{{- if .Values.service.loadBalancerIP }}
loadBalancerIP: "{{ .Values.service.loadBalancerIP }}"
{{- end }}
- type: LoadBalancer
+ type: NodePort
ports:
- port: 80
name: http
+ nodePort: 32000
- port: 443
name: https
selector:
Run the update script so the Istio manifests, Helm charts, etc. use the proper Docker Hub and Tag:
cd install && ./updateVersion.sh -a $HUB,$TAG
Run the e2e_simple test, passing in the appropriate E2E_ARGS arguments:
cd ..
make e2e_simple E2E_ARGS="--auth_enable --use_local_cluster --istioctl ${GOPATH}/out/linux_amd64/release/istioctl-linux"
Note: Cisco WSA (Web Proxy) prevents tests from passing. Do not run tests from a host that is connected to the Cisco internal network.
The E2E simple test should complete with an ok:
<SNIP>
2018-05-11T16:31:49.569305Z info Cleanup complete
ok istio.io/istio/tests/e2e/tests/simple 209.990s
By default, the E2E tests cleanup the test deployment. You can pass the --skip_cleanup argument to E2E_ARGS to avoid the post-test cleanup.
Daneyon great stuff
A couple comments & questions on the above.
It's not too difficult to run the tests against remote clusters without installing go, git or any istio source code. Unfortunately though there would be some changes required in the test code to properly allow this. It is probably something we should spend bit of time doing a some point. I have used this technique successfully create the environment remotely, although I didn't actually run any tests.
We will want a plan for how to handle the tests needing to run in a DMZ. That is going to be problematic in lots of environments. I am thinking we might want to push an option upstream that skips any tests that require external access. Let me know your thoughts.
I think we should extend these instructions to include some of the other tests, especially a multicluster test. Do you want me to provide you the updates or prefer that prepare a separate set of instructions.