Based on a long-running conversation, mostly on slack with @oxinabox and @c42f, with a little bit @evizero. One issue that led to it is the desire to do XML reports of test runs, which exposes some issues with the extension mechanisms currently in place for Base.Test. I think to figure out how much rearchitecting is necessary we should answer a few questions.
oxinabox says DataStructures.jl has some testing-related hacks, we should investigate what they've had to do to see what might be needed in Base.Test