Mut-tools: mutation testing for C/C++ programs

Mut-tools are used to inject mutations to C/C++ programs. Mutations are injected by an extended LLVM/Clang compiler (version 6.0) that modifies the source code representation of the program on the fly. All mutations of a translation unit (i.e., a source code file) are included into a single source code representation that is translated to an object binary file. We call this as the all mutants form of the translation unit. In earlier research similar representations are called as program schemata or conditional mutations. The modified source code representation exists only in the memory of the compilation process.

As usual, an object binary file may represent an executable program (i.e., the program is composed of only one translation unit), or executable programs can be composed by link editing several binary object files together. In addition, libraries, both static and dynamic, can be built in the usual way. Ordinary object binary files and object binary files in the all mutants form can be mixed.

The idea in the all mutants form is that if one of the mutations is activated at program start, then the all mutants form behaves exactly the same way as a mutant containing only the activated mutation. If none of the mutations is activated, the all mutants form behaves exactly the same way as the original program. This makes management of the mutants easy as there is only one binary version instead of thousands of different binary versions each of which represents one single mutant.

Mut-tools are built on top of LLVM/Clang version 6.0. The main components are an LLVM/Clang plug-in module (`/usr/local/mut/lib/mutate.so') that injects mutations to the original C/C++ source code on the fly, and a run time library (`/usr/local/mut/lib/libmut.so') that is used to activate a specified mutant.

The user level programs (in directory `/usr/local/mut/bin')

mut-gen
mut-list
mut-run

hide the usage of the main components from end users.

The program `mut-gen' is used when a programmer wants to create binary object files in the all mutants form. The `mut-gen' program sets up the compilation environment so that the extended LLVM/Clang compiler injects mutants to the binary object file it creates.

The mutations injected to a binary program can be listed with the program `mut-list'. A mutant can be activated with the program `mut-run'.

A simple demonstration software can be found from the directory `/usr/local/mut/test/even'. The program takes one integer argument and prints whether the argument is even or odd. For demonstration purposes the source code is divided into two files (`main.cpp' and `even.cpp'). A simple makefile is provided, so the program `even' can be built with command

$ make even

The all mutants form can be built with command

$ mut-gen make even

Make sure that the directory `/usr/local/mut/bin' is in the PATH before any directory containing the Clang compiler (programs `clang' and `clang++').

In general, if you can build some (C/C++) software with command

$ build-command

then you can build the all mutants form of the software with command

$ mut-gen build-command

The injected mutations can be listed with command

$ mut-list even

A list of mutant selectors can listed with command

$ mut-list -s even

A mutant selector is composed of three parts: source code file name, mutation location number, and mutation index within the mutation location. For instance the selector `even.cpp:2:1' is valid for `even' program.

A mutant can be executed by activating the corresponding mutant selector at program start. This can be done with the `mut-run' program. For example

$ mut-run -s "even.cpp:2:1" ./even 3

In general, if you can execute the unmutated form of a test case with command

$ test-case args

then you can execute the test case with mutant `selector' activated with command

$ mut-run -s selector test-case args


All mutants of the `even' program on input `3' can be executed (in bash shell) as follows:

$ for m in $(mut-list -s even); do echo "Run $m"; mut-run -s "$m" ./even 3; done


In the research the main focus was on how mutation tools for C/C++ software can be integrated into development and test execution environments. Hence the current version injects only binary and unary operator replacement mutations.

The video shows how the all mutants form of the LLVM/Clang compiler infrastructure can be built with mut-tools, how the Clang unit test cases can be executed without activating any mutants, and how unit test cases can be executed with some mutants activated.

For further information, usage experience, development suggestions, and comments please don't hesitate to

contact `[email protected]'