Mutation Testing - What and Why

Your codebase is large. You delete one line of code which you thought wasn't needed. All tests pass and it goes to production! Two days later, hundreds of production incidents come in - you then realise, that line was essential to a different part of code.

How do you assess the quality of your tests and coverage? Does the codebase have useful tests or just one that bring up the shiny coverage numbers?

Let's dive into Mutation Testing and how it can help.

How should I write unit tests?#

At some point in our careers, all of us have been guilty of writing a bad test - whether this be purposeful or accidental. We wrote an assertion either for the sake of writing an assertion, not knowing how to properly assert or even assuming that our assertions cover proper scenarios.

What harm can such a test do to the project? When you look at it in the micro level - close to nothing, it will live the rest of it's life being run by every developer working on the code. When you look at it on the bigger picture, it brings a very large hole in your project.

Unit tests are meant to be one of the first points for early feedback in the development cycle - if the logic you wrote is wrong, or the logic you wrote breaks something else, you need to know that as soon as possible, not after it causes damage.

Your goal, as a developer, is to write unit tests that are strict enough to define your business logic so that anyone coming and changing the code gets to know whether they should.

How do I know if my test suite is robust?#

This is where Mutation Tests come in. Mutation Tests deliberately make changes (or "mutations") to the code to simulate possible defects. It will then run your existing test suite against the mutant.

If the test suite passes, it's a bad sign and is referred to as a mutant that survived.

If the test suite fails, it's a good sign that your tests are robust enough to resist this change. This is referred to as a mutant that was killed.

The ideal scenario is for you to have no surviving mutants, but it's not always a great idea to write a test for each mutation - we'll look at why in the next section.

There are various types of mutations - deleting a line of code, changing the value of a string, flipping an arithmetic operation and many more! Combinations of mutations also exist. What types of mutations are checked and what each mutation is called will differ based on the tool being used.

What do I do with the results?#

The mutation test reports give you a picture of what parts of your code are tested thoroughly. It's your job as a developer on the codebase to assess the potential impact of a certain mutation causing chaos.

For example, If your code was a simple condition like

public bool IsSumLessThan5(int a, int b) {
    Console.WriteLine("Checking the value for sum...");
    if(a + b < 5) { return true; } 
    else { return false; }
}

and you have written a test

public void ShouldReturnTrueWhenSumLessThan5() {
    Assert.That(IsSumLessThan5(1,1), Is.EqualTo(true));
}

One mutant would easily call out that your test suite will survive if you change the addition operator into a multiplication operator. This will help you understand that you would need better values to be sent into the parameters.

At the same time, some other mutant will change the log message - this would survive because we haven't added an assertion for the log message, but this won't give you much value.

At the end of the day, mutation tests should be a way for you to identify what MIGHT be weak in your test suite. Always take it with a pinch of salt and analyze each surviving mutant thoroughly.

Great! I'm onboard. How do I start?#

Depending on the tech stack you use, there will be various mutation testing frameworks out there

If you are on the either JavaScript or C# family of frameworks, Stryker would be the most popular choice. Java applications tend to use the popular PiTest. Rust's most popular framework is Mutants.

If you're brave enough, you can always write your own mutation testing framework ;)

The most important thing to keep in mind is to always analyze the reports yourself and reason out what mutants need to be handled.

Till the next post, Ciao!