Testing comes in many flavors depending on what you're specifically trying to test and how it accumulates in the tech stack:
- Unit Testing
- Integration Testing
- System Testing
- User Acceptance Testing (UAT)
- Regression Testing
There are others...but life is short
It's also important to understand how any particular test or test suite is executed:
- Manual Testing
- Automated Testing
Remember that some combinations are possible, while others are not:
- √ Automated Unit Testing
- √ Manual User Acceptance Testing
- √ Automated Regression Testing
- X
Manual Unit Testing - X
Automated User Acceptance Testing
The goal for any test in any methodology is to surface unexpected errors and protect your code from failing to deliver on it's intended purpose.
The main way this is accomplished is by mounting your test subject in a test harness and feeding it any and all input that it could be expected to encounter in a real usage scenario.
Inputs vary by test goal and harness type. While a successful unit test may involve nothing more than running a function with a series of values and comparing to expected output, complex system tests may need to simulate a user click and intercept and analyze attempted side effects such as XHR requests.
A test harness is the data, libraries, and environment required for a test suite to run. This term is borrowed from physical hardware testing, and can actually include specific hardware if that's an important feature of the test subject.
For unit testing, the harness can and should be low in complexity. High complexity for a unit test harness may be a sign that you need to break down units into smaller pieces or you need to write an integration test.
The test harness for System Testing or UAT may actually include hardware, such as computers with specific OS and resource configurations, specific mobile phones, etc.
Note that it is common for test suites to be much larger than the code they seek to test. Different pieces of code are being tested both explicitly and implicitly at many points as the test stack accumulates into higher level features.
All test suites should include tests that both succeed and fail. Neglecting to test for and handle failure scenarios means that you will definitely encounter those scenarios in production.
Using a trivial example:
function sum(a, b) {
return a + b;
}
// Positive assertion
console.assert(sum(2, 2) == 4, 'Expected 2 + 2 to equal 4');
// Negative assertion
console.assert(sum(2, 2) != 3, 'Yeah, and pigs can fly.');When writing any test for any suite it's important to first get your test to fail, then refactor both the code and test in order to get it to pass. Making sure that code fails test first avoids issues related to false positives; This is related to but distinct from negative testing.
Using another contrived example:
function sum(a, b) {
return 4;
}
// This passes, but if you stop here then you're in trouble...
console.assert(sum(2, 2) == 4, 'Expected 2 + 2 to equal 4');Don't need 100% to be successful
UNIT TESTING is a level of software testing where individual units/ components of a software are tested. The purpose is to validate that each unit of the software performs as designed. A unit is the smallest testable part of any software. It usually has one or a few inputs and usually a single output.1
Smallest testable unit of code, typically at the level of individual functions
Make sure interfaces between foundational components perform as expected when given diverse inputs
Copy this block of code into your browser console and watch it fail, then selectively un/comment an implementation inside the
sumfunction to make it succeed (i.e. no console error)
function sum(a, b) {
// return a + b; // this works
return a - b; // this fails
}
console.assert(sum(1, 1) == 2, 'do you even maths?');
INTEGRATION TESTING is a level of software testing where individual units are combined and tested as a group. The purpose of this level of testing is to expose faults in the interaction between integrated units. Test drivers and test stubs are used to assist in Integration Testing.2
Composed of units. If authoring exported modules in Node, this would test consuming several that are supposed to interact in a certain way.
The goal of integration testing is to validate the contract of your unit interfaces when paired together as they might be in your system.
Tests in your integration suite may look quite a bit like unit tests, with a helper library to smooth out framework specific concerns. These should be fast, but may have timing dependencies and accept or create side effects.
Mock interfaces could be required, which may make these tests slow to write, but generally fast to run.
A contrived example of an integration test
var getData = require('../db/interface.js');
var processData = require('../lib/processUser.js');
// Some code that may mock out the 'getData' call so no DB roundtrip is required in tests
var localUser = {
name: 'Alan',
uid: '12345'
}
describe('normalize user', function(){
it('should reformat user', function(done){
getData()
.then(processData)
.then((result)=>{
result.should.eql(localUser)
return;
})
.then(done);
.catch(done);
})
})SYSTEM TESTING is a level of software testing where a complete and integrated software is tested. The purpose of this test is to evaluate the system’s compliance with the specified requirements.3
System tests pull aggregate features together in the same form that will be encountered by users. This is where tools like PhantomJS and Selenium attempt to recreate a real execution environment, load your entire application, and interact with it through the same mechanisms as a user.
The goal of system testing is to validate that business requirements have been captured and function as intended.
System tests tend to be slower and more resource intensive than lower levels of testing. Expect to use 3rd party services that load specific OS/Browser versions in service of obtaining a faithful system test.
Containerization may also be a concept here, as is actual hardware.
Example7
const {Builder, By, Key, until} = require('selenium-webdriver');
(async function example() {
let driver = await new Builder().forBrowser('firefox').build();
try {
await driver.get('http://www.google.com/ncr');
await driver.findElement(By.name('q'));.sendKeys('webdriver', Key.RETURN);
await driver.wait(until.titleIs('webdriver - Google Search'), 1000);
} finally {
await driver.quit();
}
})();ACCEPTANCE TESTING is a level of software testing where a system is tested for acceptability. The purpose of this test is to evaluate the system’s compliance with the business requirements and assess whether it is acceptable for delivery.4
Watching a user test your product for the first time...
Test as written: "You turn the handle and walk through the door"
This scale of UAT is huge. It can be planned or unplanned. UAT happens whether you want it to or not whenever code is deployed into a production environment. It's up to the team to capture feedback both from analytics and direct end user interactions.
UAT in pre-production environments often will involve a staging environment, and an orchestrated event with solicitation for structured feedback.
The team should be prepared to defend design decisions at this level of testing.
Due to the subjective nature of UAT, you'll likely conflate testing along many dimensions. For example, you may want to test how users like the new call to action on your homepage, but you also get feedback about your homepage's loading performance.
The goal of UAT is to get as close to real-world usage of your application as possible by actually having people use it.
Small and informal UAT sessions can be conducted as hallway testing8
User acceptance testing is the slowest testing approach discussed by far. This does not mean that an individual UAT session is lengthly, but the level of effort involved in obtaining UAT test data is orders of magnitude higher than other forms of automated testing, with far less frequency and reproducibility
Regression testing is re-running functional and non-functional tests to ensure that previously developed and tested software still performs after a change. If not, that would be called a regression. Changes that may require regression testing include bug fixes, software enhancements, configuration changes, and even substitution of electronic components.6
Regression testing can exist at many scales. The simplest form of regression testing is to run other test suites again after a change to verify that features are only impacted as expected.
More common regression suites include fully manual testing of features that are expensive or impractical to automate. Testing how a mobile phone behaves when powered on requires a person to power that phone on.
Regression testing seeks to maintain consistent output. The inputs of a regression suite are almost always generated by a first run mechanism that then sets future expectations.
A visual regression suite, for example, would first capture screenshots of an interface then later compare new screenshots to the ones previously captured as a way of confirming that code changes did not result in interface changes.
Regression suites can be fast or slow, depending on the balance of manual vs. automated testing, the setup involved,and the complexity of the initialization data.
Speed is generally somewhere between system testing and UAT testing.
Example9
References
- Code katas http://codingdojo.org/
- Red Green Refactor: https://www.codecademy.com/articles/tdd-red-green-refactor
- X Unit Tests, 0 Integration Tests https://natooktesting.wordpress.com/2017/08/24/x-unit-tests-0-integration-tests/