Istio's current testing infrastructure is based on CircleCI. CircleCI automates the Istio pipeline from commit to deploy through:
- GitHub Integration: Every time an Istio developer submits a Pull Request (PR), CircleCI creates a build.
- Automated Testing: CircleCI automatically tests the build in a container or virtual machine.
- Notifications: The PR is updated with the status of the build.
- Automated Deployment: Passing builds are deployed to various environments.
CircleCI uses a YAML file to identify how you want your testing environment setup and what tests you want to run. On CircleCI 2.0, this file must be called config.yml and must be in a hidden folder called .circleci. Here is the Istio config.yaml (0.7 release).
TODO add background on config.yml components such as jobs, workflows, etc.. Workflows:
CircleCI jobs are run using make, a program building tool that aids in simplifying building program executables that may need various modules. To determine how the modules need to be compiled or recompiled together, make takes the help of user-defined makefiles.
Example Walkthrough:
- PR
- test job status
- test job config
- make sync target
- The make localTestEnv target will setup a local test environment for running Istio unit tests.
- The make test target runs unit tests for the Istio components.
Note: A benchmark test is run for nightly builds using this script.
Prow runs the "istio-perf-tests" test against daily builds and posts results here.
Here is the prow job for Istio perf testing in test-infra/prow/config.yaml:
- name: istio-perf-tests
agent: kubernetes
context: prow/istio-perf-tests.sh
always_run: true
rerun_command: "/test istio-perf-tests"
trigger: "(?m)^/(retest|test (all|istio-perf-tests))?,?(\\s+|$)"
labels:
preset-service-account: true
spec:
<<: *common_e2e_spec
- Unit tests: Should be done in specific modules for features implemented in a module. (PR to master)
- Integration tests: Each module should write integration tests that exercise their dependencies. Integration tests should be runnable by every contributor locally and should not need access to a cluster. Those tests will also be run when PRs are created on dependencies. Integration testing is the team's responsibility and therefore will not be planned in this document. (PR to master)
- Long Integration tests: Longer integration tests (>10 minutes) might only be triggered for qualification. (PR master to stable)
- End to end tests: Stored in istio/istio module where all components need to be tested together. (PR to master and PR master to stable)
- Performance Benchmarks: Benchmarks should be kept in modules. (PR master to stable)
- Performance Regression: Stored in istio/istio. (PR master to stable)
The framework provides a universal way to bring various Istio components into a test environment. Unlike e2e tests, integration tests usually don't need to start the whole package of istio and application services, instead, each integration test only needs several modules related to this test case. This integration test framework is built for this requirement. It's designed to be flexible enough for developers to pick the combination of components based on what behaviors are tested.
Fortio, mixer and proxy are components within the framework.
e2e contains Istio end-to-end tests and the associated test framework. E2E-bookInfoTest and E2E-smokeTest are automatically triggered in Before-Merge stage for every PR. The full suite of E2E tests are run in After-Merge stage only. Their results can be found in https://prow.istio.io/ and https://k8s-testgrid.appspot.com/istio
Fortio is used in the e2e to generate traffic for
Package dashboard provides testing of the grafana dashboards used in Istio to provide mesh monitoring capabilities. This pkg uses Fortio to generate traffic for Grafana monitoring.
This tests that a simple ingress with rewrite/dropping of the /fortio/ prefix works, as fortio only replies with "echo debug server ..." on the /debug uri. Note: Although this test uses Fortio, it is not a perf test.