End-to-end testing (or “E2E”) is the biggest, scariest, slowest, and most valuable type of testing around. They don’t interact at a code level, they interact like they’re a real user doing real things. They’re usually not going to cover every little thing, they’re more about ensuring critical paths through the ecosystem are supported, touching multiple applications and APIs to achieve that task.
The interactions are real, maybe a few config variables are using “Test” keys for sending emails and making payments, and maybe those are sandbox environments, but everything else is actually happening.
These sorts of tests are slow and hard to set up, they need to have real records created in the database and real users need to exist to do that. If the tests are run in a QA environment maybe they can do a big reset script to make all the APIs start from scratch, or its creating a new user every time - which can make the database huge if these tests run hourly.
E2E usually involves running the entire application, and also running all of its dependencies, and testing that real actions can be done through real interfaces. Because it’s testing the whole ecosystem, or certain chunks of it, the only difference between end-to-end testing is the entry-point and the tools used to initiate these tests.
Web Apps: tests are run in a pretending to be a human clicking around.
Mobile Apps: tests are run with a which will tap and swipe around like a real user.
APIs: test runners like or make a bunch of requests to loads of endpoints following key workflows, taking values from one response and using them for another.
Real APIs will be running in this test, if there are 10 APIs at the company and the E2E tests are being run on the mobile app, then you’ll have a similar or 10 API “dependencies” running using some tool like Docker or Kubernetes to maintain a testing environment. This can be complex to orchestrate if you’re not familiar with Docker, Kubernetes, or other DevOps practices, but it’s crucial for making sure your application actually works in the real world.
Alternatively, some E2E test suites run on staging, or maybe even production! 😎
Because the goal of E2E testing is to make sure multiple applications and APIs work when talking together, they really do not belong in a repository that is owned by one of those APIs. Instead, end-to-end testing is usually in another repository entirely, maybe owned by a QA team, or similar.
One caveat to that might be if you organization uses the pattern, in which case they’d be considered as a separate application or test suite from the other applications in that repo.
There are lots of E2E systems which are hosted Software-as-a-Service products and are totally separate from the source code.
Previous versions of Stoplight had a test runner that would allow for creating comprehensive E2E test suites, known as . We found that most people were using it to do two things:
Make sure a response has a HTTP 200/201/202 status code. Contract Testing Less than 1% of people were doing anything more than that, so we’re putting our efforts into making these use cases easier. Earlier we talked about Contract Testing with and its , and that fits in with end-to-end testing nicely. Just creating a bash script that makes some HTTP cURL calls through Prism, and you’ll have contract testing.
There are SaaS solutions like too, which will let you create tests for multiple APIs through a user interface, which expands who can contribute end-to-end tests at your organization.
Or you can get a little more advanced and use to create scenarios in YAML. This might not be as easy as creating tests in a UI, but is still more accessible for slightly less technical users than many of the other testing systems which force users to write JavaScript or other programming languages. Not only can Strest handle end-to-end testing with contract testing, but it’ll handle “stress testing” too.
So when do end-to-end tests run? You could run them on every single Pull Request, but they’re usually pretty slow and that might get expensive. Some people place them in the deployment pipeline, meaning a mobile app must pass its end to end suite before being published to the App Store. Similarly, an API might be end-to-end tested in a special testing or staging environment before its deployed to production. If continuous integration is being used it doesn’t really matter if these tests are slow, and the deployment will just bounce back if not accepted.
Many developers find having these external tests jarring at first, because they are mostly used to having their tests under their control in their repo. Having them in another system can feel like “an extra thing to do” because a big change to their application means they might need to go and update the end to end tests too, but that is actually a benefit not a bug. You want a system that’s outside of on API teams control.
When tests are owned by the API, the tests can be changed to show that the API is “all good”, but that might involve a change that would break expectations of other consumers. Having these tests under the control of a Software Testing or Quality Assurance team means these sort of accidental or unintentional breakages cannot slip through. If a breaking change is made to an API and the E2E testing is being run before deployments can go to production, then this breaking change will be caught safely.
Ok, two quick ones to go. Let’s knock em out.