Mutation Testing

Introduction

Mutation testing is a suitable technique to determine the quality of test suites for a certain program. This testing technique is based on the creation of mutants, that is, versions of the original program with an intentionally introduced fault. These errors are inserted within the code through some defined rules called mutation operators. Mutation operators represent typical mistakes made by programmers when using a programming language and they produce a simple syntactic change in the program under test (PUT).

The mutation testing process starts with the generation of the mutants using the set of mutation operators. Then, those mutants are executed against the test suite created for the PUT in order to determine its quality. Test cases are supposed to produce the correct output when they are run on the original program. When the output of a mutant is different from the output of the original program, the mutant is classified as dead. Otherwise, the mutant is still alive and needs to be executed against the rest of the test cases to detect its modification.

A good set of test cases should be able to detect any changes generated affecting the program. Hence, if some mutants remain alive after the test suite execution, new test cases can be supplied to kill them. In this process, a mutation score is calculated to determine the test suite effectiveness distinguishing the mutants (see Equation 1 for the general calculation of mutation score); the goal is to increase it until all the mutants are killed. An equivalent mutant is produced when none of the test cases is able to kill it as the meaning of the program has not actually been modified. Equivalent mutants, test data generation and the expensive computational cost that this technique entails are the main drawbacks to a broader usage of mutation testing.

• MS: Mutation score

• P: PUT

• C: Test cases

• KM: Killed mutants

• TM: Total mutants

• EM: Equivalent mutants

Mutation testing is a white-box testing technique, i.e., it tests a program at the source code level. Therefore, the set of mutation operators and the overall technique should be developed around each programming language in particular; the correct choice of the set is one of the keys to successful mutation testing. Thus, a great variety of research studies devoted to the definition of mutation operators for specific programming languages and tools automating the generation of mutants can be found.

The purpose of the article is to look in depth at the development and the current state of mutation testing in order to widely make known this technique in the computer science research field. Next sections deal with the related work, the way that mutants are killed, the steps to accomplish in the mutation testing process, the approaches to evaluate mutation operators and the suggested techniques to improve the aforementioned problems. Finally, the C++ programming language will be focused as an example of the development of mutation testing.

Background

Mutation testing was originally proposed by Hamlet (1977) and DeMillo, Lipton and Sayward (1978) and its development has taken place in parallel with the appearance of the different programming languages (Offutt & Untch, 2001). As a result, in the early years, most of the works centered on procedural programming languages: Agrawal et al. (1989) defined a set of 77 mutation operators for C, the tool Mothra was developed including 22 operators to apply mutation testing to Fortran (King & Offutt, 1991) and Offutt, Voas, and Payne (1996) composed a set of 65 operators for the Ada language. The mutation operators for these procedural languages are known as traditional operators.