The gold standard of software testing is formal verification, a mathematical proof that a body of code cannot fail with unspecified behavior. However, in practice formal verification proceeds at a slower pace than many teams prefer to operate. Fortunately, we have a productive compromise: Fuzzing.
Fuzzing significantly raises the quality bar of software components, closing subtle testing gaps that other approaches neglect to address. For example, validation, a handful of manual tests, and code coverage will still leave latent bugs in very many codebases. This is because the input/state space is many orders of magnitude larger than the number of lines of code, and scale exponentially with the bit width and number of variables.
Fuzzing is able to identify these kinds of bugs, by generating random test cases to more comprehensively evaluate the input and/or state space. Code coverage won't tell you when you have a division by zero bug, or a nil pointer bug, or an empty array bug. Fuzzing can tell you these things.
Even at 100% code coverage, <1% of the input/state space is tested. So don't obsess over the wrong thing.
Code coverage policies overtest relatively safe code sections and undertest core logic vital to the component's behavior. Code coverage assumes that each line is uniformally important, wasting human labor writing tests for less important code snippets. Whereas fuzzing naturally encourages developers to write fuzzing tests primarily for the most important or most fragile code snippets. There are even some abysmal code coverage policies that differentiate coverage thresholds between existing code vs changed code... Suffice to say, code coverage is a useful tool in its own right but the wrong hammer for most screws.
Like other programming languages, Go supports fuzzing to test the input/state space with random data.
First, implement a property test in a *_test.go file according to the testing.F.Fuzz interface.
Go's fuzzing framework has a fatal flaw: Unlike other fuzzing frameworks, it often skips zero values by default.
Although Go normally places an emphasis on intentional usage of zero values in data model design (e.g. false, 0, 0.0, rune(0), "", []byte{}, etc.), nonetheless the eggheads who designed the fuzzing framework completely forgot to include these. deeating the entire purpose of fuzzing.
As a workaround for the zero value omission, you will want to manually configure each fuzzing test by generating Add configurations, starting with assuming zero values for all of the arguments to the fuzzing test. This still omits the case where some of the arguments are zero and others are not zero... If you care a fig for quality, consider Rust with cargo-fuzz.
Finally, execute the test suite with fuzzing enabled:
$ go test -fuzz ./... -fuzztime <duration>For example, apply a duration of 1m for a relatively fast check of the input/state space, or longer durations for even more comprehensive checks.
To select a specific fuzzing test, replace ./... with the name of the fuzzing test function.
- libFuzzer (C/C++)
- QuickCheck (many programming languages)
- Mutation testing
- Rust + cargo-fuzz
- Proof assistant languages (for generating formal verification proofs)