DesignDocument.MD

Google Benchmark V2 Design Document

This document lays out the design goals for v2 of the Google Benchmark library, as well as the problems and use-cases they're intented to address.

Supporting Arbitrary User Input and Output

One problem with v1 of the library is its limitations in allowing users to provide arbitrary input to their benchmarks, as well as specifying arbitrary output as part of the results. Multiple disjoint attempts have been made in an attempt to improve this situation. As a result we have many different API's for specifying specific types of data; each too specific to be of general use.

Instead of adding new API's for each specific case, the goal of v2 is to create a single unified mechanism of handling and processing arbitrary user data throughout the library.

The solution in v2 is to make JSON the fundamental vocabulary through which users and the library communicate. In doing so the problem of handling arbitrary user types is solved. As long as the user can represent their data as JSON, the library can store, report, and otherwise handle it.

An example use case would look something like this:

struct TestCase { // Example user-defined input type
  bool expect_failure;
  std::vector<std::string> inputs;
  
  struct TestResult;
  TestResult evaluate_result(json const& counters);
};
void BM_Foo(State &st) {
  json const& input = st.GetInput();
  std::vector<int> args = st["args"];
  MyTestCase test_case = st["test_case"];
  std::map<std::string, Counter> counters_to_track = st["my_counters"];
  for (auto _ : st) { /* ... */ }
  /* ... process counters et al ... */
  st["my_counters"] = counters_to_track;
  st["my_results"] = test_case.evaluate_result(counters_to_track);
}
BENCHMARK(BM_Foo)->WithInput({
 {"name": "BM_Foo/WithCheapCounters"},
 {"my_counters", {
   {"accumulator", Counter(0.0)},
   {"average_counter", Counter(0.0, kAverage)}
 },
 {"test_case", TestCase{/*ExpectFailure*/false, /*Inputs*/{"abc", "meow", "woof"}}
})
->Range(1, 10);

One open question is how the library should best structure it's JSON reports. They should be relatively easy and intuative for users to navigate and manipulate. Suggestions welcome!

Secondary Custom Output Use Cases

Allowing users to specify arbitrary results as JSON solves only part of the problem. There is an additional need to allow users to do post-processing on their data. Either to compute statistics over a number of repetitions, or simply to adjust it based off of the measured time result. Therefore v2 should provide a mechanism allowing users access to the JSON results of a benchmark (or set of benchmarks) at the following times:

1. After every single run, once the timing results have been computed.
2. After running all repetitions of a single benchmark instance (ex. to compute statistics)
3. After running all benchmarks beloning to a single family (ex. to compute complexity)

I haven't decided exactly how to create this mechanism. The mechanism should be generic enough that the existing statistic/complexity computations can be implemented in terms of it. There are a couple possibilities:

1. Using user-specified callbacks functions to be registered and invoked for each aformentioned case. This would allow many different callback functions to be registered without affecting or overriding previously specified callbacks.

2. Using a virtual interface similar to that of Reporters, allowing the user to specify a custom "Reporter" object which would have its various virtual methods invoked at the appropriate time. For example:

struct BenchmarkCallbackHandler {
  // Process the result of a single repetition, which is possibly part of multiple runs.
  virtual void ActAfterRun(json& full_report, const json& repetition_result);
  // Process a benchmark following completion of all repetitions.
  virtual void ActAfterBenchmark(json& full_report);
  // Process a benchmark family after running all contained benchmark instances.
  virtual void ActAfterFamily(json& full_report);
};

My initial preference is choice #2, since having a callback object as opposed to multiple callback functions allows data to be stored, accumulated, and communicated between different callback points.

Comparing Benchmark Results

One thing that the current library does poorly is allowing the comparison of benchmark results. The new library API should treat this feature as a first class citizen, with the goal of allowing performance regression tests to be written using the library.

The most common need is the ability to compare the current performance of a benchmark to historical results for the same benchmark stored as JSON. However, the need to compare two distinct benchmark families can arise; either by executing both and performing a direct comparison, or by loading one or both of the benchmark results from previous JSON output.

• Comparing the result of a benchmark to previous results stored as JSON.
• Comparing two distinct benchmarks to each other (Either by running both, or via JSON reports)

Better Benchmark Creation and Managment

The classical v1 interface is built around globally registered benchmarks run from and controlled through the command line invocations. The benifit of this design is it's simplisity; both in overhead required to write a benchmark, but also run it.

However, such a system has it's flaws. Creating benchmarks as global object during static initialization has a number of limitations and draw backs. First, it prevents programatic generation and running of benchmarks (or at least makes it very difficult). Second,