mutation-testing-survey.markdown

@simon_wc: Thanks for wanting to help me out! :D

1. Have you ever used a (JavaScript) mutation testing framework? If so, which framework(s) did you use and what are your experiences?

I haven't used a JavaScript framework; I've wanted one for a long time, after briefly experimenting with a Ruby one (before I almost entirely stopped using Ruby), but my searching has been fruitless until very recently. (It looks like Martin Koster recently started working on splitting some mutation-testing work out of a Grunt-specific project, though, and I'm following that with interest.

2. When searching for a framework to use, what would convince you to try it out?

Besides the necessity of it existing at all (i.e. it'd be worth experimenting with, to me, even if it went against every one of my instincts and methods, simply because I consider mut-testing crucial to efficient TDD), I think I'm most interested in it being universal: if given several options, I'd prefer to use a library that didn't require me to conform to its assumptions (about other tooling, such as what build-system I use; or about the code itself, such as formatting, indentation, stylization or naming, etceteras.)

In fact, that's important enough to me, that I'd choose to use a library that provided less- effective actual mutation intelligent, or one that was monumentally slow, over one that tried to impose opinions/assumptions about my code or my other tooling.

3. If you’d find a framework which requires you to use a certain CI tool such as Grunt, would that be a no-go for you? How about if the framework does not support a CI tool?

I think you mean ‘build system’, not ‘continuous integration’, but either way ... see above. Please don't make assumptions, if you can possibly avoid it! :D

(As for specifically supporting a particular build system ... if it happens to be one I use, that's cool, but I care very little; I am a competent developer, and can do the supporting work I need to to integrate a generic/universal/plain mutation-testing tool into the rest of my build system.)

4. If you could choose 3 of the following criteria, which ones would you choose? Could you rank the 3 you chose from most important (#1) to least important (#3)?

5. The framework should be extendable to support a new unit testing framework

6. The framework should include proper documentation

7. The framework should be extendable to support a new type of mutation

---

• The framework should be easy to use
• The framework should support many different types of mutations
• The framework should be able to test my code fast
• The framework should support many unit testing frameworks

Some commentary on my ordering: Other than the really important considerations I've detailed above, I'd like to mention that I do think extensibility, at least at first, is more important than built-in support for any particularly-large set of existing tools / build-systems. Adding those can come later, or can be provided by people like me.

As for documentation, I'd like to posit that good documentation (or at least, thorough and up-to-date documentation) is Item No. 1 when it comes to ‘ease of use.’ The two are nearly synonymous, actually, when it comes to developer tools.

Finally, speed is important, because mutation testing can be inherently slow ... but I'd rather see a basically-working product shipped sooner, and then see performance iterated on, than see any work you're doing wait for months or years until it's arbitrarily “fast enough.” (Personally, I'm not convinced mutation-testing is something that needs to run on every commit; it's something that can be invoked on occasion, when one is focused on specifically improving the test-suite.)

5. Are there any other criteria which are important to you? How important would these criteria be compared to the criteria in question 4?

You've about covered everything I think is important! (=

6. Do you have any questions, regards or advice?

If you're going to put work into this, develop it in public! I'd be delighted to contribute, if there's a way that I can. (I'm also sure I'm not the only one who feels that way ;)

Thanks for your thorough response! Do keep in mind that this is part of an internship and that it's not 100% certain that the tool will be open sourced :)