Since C does not provide a "canonical" unit testing framework, we have to implement our own. The basic template for testing frameworks is called xUnit, which we'll follow.
For C, I guess the analogous xUnit patterns would be:
- Test runner: the
.cfile with themain()function running all (or some specified subset of) the test suites - Test suite: a collection of test cases
- Test case: the function which is a single unit test, i.e., has a single assert
Note, strictly speaking, a single .c file may contain multiple test suites,
though it seems that JUnit considers a single .java file to be a single test suite.
The test runner's output may be human readable, or a dump to an xml file, or something else (or some combination of these).
References:
Implementation. I actually implemented an xUnit-style framework for unit tests in my toy ML compiler/interpreter, see test.c and runner.c, as well as the associated header files. The implementation mostly follows the design discussed here, with some small, insignificant modifications.
For better or worse, JUnit seems to be the "modern standard" (although it is inspired from SUnit).
JUnit has XML output, the schema discussed in junit-xml-output.md,
which seems to work with the assumption that all code occurs in a class.
It would be nice to make a C unit-testing-framework produce JUnit consistent XML output, but that might be too hard.
So, it seems (unfortunately) there must be some global data. We need to keep track of the test suites for the runner.
Lets begin thinking about a test suite, which is a composite pattern consisting of a list of test cases. Since this is C, we should think of using a linked list of test cases. We can actually stick a lot of metadata onto the test_case_list.
struct property {
char *name;
char *value;
struct property *next;
};
struct test_suite {
const char *name;
struct test_case_list *cases;
struct property *properties;
time_t start_time;
time_t end_time;
int errors, failures;
const char *hostname;
int tests;
int failures;
int errors;
double time;
};
extern struct test_suite *current_suite;
struct test_suite* new_test_suite(char *name);
void free_test_suite(struct test_suite *suite);
void test_suite_setHostname(struct test_suite *suite, char *hostname);
void add_test_case(struct test_suite *suite, struct test_case *test_case);Now, we need to consider the test case. It should have a function pointer to a void test(void) signature function.
typedef enum { CASE_UNTESTED = -1, CASE_SUCCESS = 0, CASE_FAIL, CASE_ERROR, CASE_SKIPPED } test_case_status;
typedef enum { FAIL_NONE = 0, FAIL_FAILURE, FAIL_ERROR } fail_type;
struct fail_or_error {
fail_type type;
char *message; /* message specified by assert or error */
char *type; /* type of assert or error that occurred */
};
struct test_case {
void (*test_case)(void); /* the test case */
/* everything else is metadata */
const char *name;
const char *classname; /* junit compatiblity */
struct fail_or_error *failure;
double time;
time_t start_time;
time_t end_time;
test_case_status result;
};
struct test_case* new_test_case(void (*test_case)(void), char *name, char *classname);
void free_test_case(struct test_case *test);
void test_case_failure(struct test_case *test, char *message, char *type);
void test_case_error(struct test_case *test, char *message, char *type);
// TODO: handle milliseconds for ISO 8601 format?
inline char* test_case_time(struct test_case *test) {
if (CASE_UNTESTED == test->result) return "";
char buff[28];
strftime(buff, 20, "%Y-%m-%d %H:%M:%SZ", gmtime_r(&(test->start_time)));
return buf;
}A suite is a composite design pattern, i.e., a list of test_case objects. We have some rudimentary code for that:
struct test_case_list {
struct test_case_list *next; /* linked-list structure */
struct test_case *test;
char *systemOut;
char *systemErr;
};
struct test_case_list new_test_case_node(struct test_case *test_case);
void free_test_case_list(struct test_case_list *list);To let the test runner...run... we need to keep track of all the test suites. We simplify the design choice to just keep track of a composite test_suite. So we need, sadly, a global variable:
struct test_suite_list {
struct test_suite *suite;
struct test_suite_list *next;
};
extern struct test_suite_list *all_suites;
// boiler plate functions
struct test_suite_list* new_test_suite_list(struct test_suite *suite);
void free_suite_list(struct test_suite_list *list);
#define test_suite(name) /* current_suite = new_test_suite(#name); test_suite_list_addSuite(all_suites, current_suite); */
#define add_test(fn) fn; /* prototype the function */ \
add_test_case(current_suite, new_test_case((fn), #fn, __FILE__)); \
fn ()
/* example:
test_suite(my_awesome_test_suite)
add_test(math_makes_sense1) {
assertEqualInt(1, 1);
}
// more tests
*/We need to have a variety of assert macros, which takes 3 arguments (LHS, RHS, message) with the message have some default value if missing. Variadic macros make this easier, doing things like
#define assertEqualIntHelper(lhs, rhs, msg, ...) ((lhs) == (rhs) ? test_status = SUCCESS : (test_status = FAIL, fail_message = (msg)))
#define assertEqualInt(lhs, rhs, ...) assertEqualIntHelper(lhs, rhs, __VA_ARGS__, #lhs " != " #rhs)I suppose we could generalize it to permit
#define assertEqual(lhs, rhs, comparator, msg, ...) (comparator(lhs, rhs) ? /* etc. */ )
#define defaultEqualityComparator(lhs, rhs) ((lhs) == (rhs))The idea of variadic macros came from here.
We need some way to have a test case run. Each test_case is a command pattern, but we need to record the results of the assertEqual() macros in the suite.
A test suite, after running, should report a string to the user that looks like Tests run: 86, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.727 sec - (filename).
void test_case_run(struct test_case *test_case, struct test_suite *suite) {
test_result = CASE_UNTESTED;
// set test_case->start = ...
test_case->run();
// set test_case->end = ...
suite->tests++;
if (test_result == CASE_FAILURE) {
suite->failures++;
} else if (test_result == CASE_SKIPPED) {
suite->skipped++;
} else if (test_result == CASE_ERROR) {
suite->errors++;
} else if (test_result == CASE_SUCCESS) {
// huzzah!
} else {
// panic!!!
}
// print to file?
// print to screen?
}
void test_suite_run(struct test_suite *suite) {
time_t start = time();
struct test_case_list *caseIter = suite->cases;
struct test_case *test;
while(NULL != caseIter) {
test = caseIter->test_case;
test_case_run(test, suite);
caseIter = caseIter->next;
}
time_t end = time();
// print to file?
// print to screen?
}
void run_all_suites() {
if (NULL == all_suites) return;
struct suite_list *iter = all_suites;
struct test_suite *suite;
while (NULL != iter) {
suite = iter->suite;
test_suite_run(suite);
iter = iter->next;
}
}
void cleanup_all_suites() {
if (NULL == all_suites) return;
struct suite_list *iter = all_suites;
struct test_suite *suite = iter->suite;
struct test_suite *next;
while (NULL != iter) {
iter = iter->next;
next = iter->suite;
free_test_suite(suite);
suite = next;
}
}
int main() {
run_all_suites();
cleanup_all_suites();
return 0;
}The assertEqual() macro needs to set the test_case result status, and possibly create some kind of failure object. If we could update a TEST_NAME macro whenever a new test were defined, these macros would work:
#define testCaseForFailure() find_case(find_suite(__FILE__), TEST_NAME)
#define assertEqual(...) (... ? /* success */ : addFailure(testCaseForFailure(), msg))The correct structure, I think, should be to have the test_case::run() function implicitly pass along a pointer to the test_case object and use this implicitly in the assert() macros. The modified code looks like:
/* changes to test_case */
struct test_case {
void (*run)(struct test_case *this);
/* ... */
}
void test_case_run(struct test_case *test_case);
/* changes to test_suite + macros */
#define test_suite(name) /* { \
static struct test_suite *current_suite = new_test_suite(#name); \
test_suite_list_addSuite(all_suites, current_suite); } */
#define add_test(fn) void fn (struct test_case *this); /* prototype the function */ \
add_test_case(current_suite, new_test_case((fn), #fn, __FILE__)); \
void fn (struct test_case *this)
/* changes to assert macros, requires being inside a test_case() */
#define unlessError(result) (0 != errno ? (this->result = CASE_ERROR, errno=0) : this->result = result)
#define assertEqual(lhs, rhs, comparator, msg, ...) (comparator(lhs, rhs) ? unlessError(CASE_SUCCESS) : unlessError(CASE_FAILURE))char* timeISO(time_t time);
static void printIndent(FILE *stream, size_t layer) {
while((layer--) > 0) fprintf(stream, " ");
}
static void failure_or_error_printXML(struct fail_or_error *fail, FILE *stream) {
fprintf(stream, "<%s", (fail->type == FAIL_FAILURE ? "failure" : "error"));
if (NULL != fail->message)
fprintf(stream, " message=\"%s\"", fail->message);
if (NULL != faill->type)
fprintf(stream, " type=\"%s\"", fail->type);
fprintf(stream, "></%s>", (fail->type == FAIL_FAILURE ? "failure" : "error"));
}
void test_case_printXML(struct test_case *test_case, FILE *stream) {
indent(2);
fprintf(stream, "<testcase name=\"%s\" classname=\"%s\" time=\"%f\"",
test_case->name, test_case->classname, (test_case->end) - (test_case->start));
if (CASE_SUCCESS = test_case->result) {
fprintf(stream, "/>\n");
} else {
fprintf(stream, ">\n")
indent(3)
if (CASE_FAILURE == test_case->result) {
failure_or_error_printXML(test_case->failure, stream);
fprintf(stream, "\n");
} else (CASE_SKIPPED == test_case->result) {
fprintf(stream, "<skipped />\n");
} else if (CASE_ERROR == test_case->result) {
failure_or_error_printXML(test_case->failure, stream);
fprintf(stream, "\n");
}
indent(2);
fprintf(stream, "</testcase>");
}
}
void test_suite_printXML(struct test_suite *suite, FILE *stream) {
indent(1);
fprintf(stream, "<testsuite name=\"%s\" errors=\"%d\" skipped=\"%d\" tests=\"%d\" failures=\"%d\" time=\"%f\" timestamp=\"%s\">\n",
suite->name, suite->errors, suite->skipped, suite->failures, suite->tests, (suite->end) - (suite->start), timeISO(suite->start));
/* print the test cases */
struct test_case_list *iter = suite->cases;
while (NULL != iter) {
struct test_case *test_case = iter->test;
test_case_printXML(test_case, stream);
iter = iter->next;
}
/* close the tag */
indent(1);
fprintf("</testsuite>\n");
}
void test_suite_printToScreen(struct test_suite *suite) {
printf("Tests run: %d, Failures: %d, Errors: %d, Skipped: %d, Time elapsed: %f sec - %s",
suite->tests, suite->failures, suite->errors, suite->skipped, (suite->end) - (suite->start), suite->name);
}
The body of a
<failure>(and<error>) element is typically the stacktrace of the failure or error.