Giter Club home page Giter Club logo

dumbmutate's Introduction

What is mutation testing?

Mutation testing is a way to test your automated tests. Imagine you have a unit-test or integration-test that 100% covers your code. Can you be sure your test actually would catch errors in your code? Mutation testing goes through your code and changes it slightly, like changing an operator (== into = for example), or a number (5 into a 0 for example). It then builds your code and tests it, hoping that your tests will fail (mutation killed). If the test passes the mutation is said to have survived, and the line is marked. For more explanation see Wikipedia: https://en.wikipedia.org/wiki/Mutation_testing

About dumbmutate

This is an as simple as possible implementation of this technique. There is no setup and no dependencies. Just download the binary and run it from the command-line. Works with any unit-testing framework (GoogleTest, Catch2, boosttest etc), or even without a complete framework as long as your code return a non-zero value when it fails. It is geared towards C-style languages but might work with other ones too.

Features:

  • Only command-line options needed, so could be part of a buildsystem for example. Has a threshold option to return error-code when mutations killed falls below a specified percentage.
  • Exports HTML line-by-line report and also surviving mutations to terminal, so you can go-to-code directly in your IDE.
  • Gives progress updates while running, so you can tell how long left to finish the file (in minutes).
  • Ability to mutate only part of file, i e from line 10 to line 20.
  • Only dependencies for building is CMake and a C++11 compatible compiler. No dependencies for the actual tool.

Simple example:

dumbmutate --mutate=FileToMutate.cpp --build=make --test=./test

Results will be shown both in terminal and in MutationResult.html

Demo (using doctest)

How to build dumbmutate:

cmake ./
make

Mutate specific part:

Mutate specific lines of FileToTest.cpp, line 10 to 20:

dumbmutate --mutate=FileToTest.cpp --build="make" --test="./test" --start=10 --end=20

Add threshold for percentage of killed mutations

Useful for example in a CI-solution where you could make dumbmutate return an error when the ratio of killed mutations vs survived falls below a set percentage. This example sets 80% as threshold and will return 0 when mutation-killratio is above this percentage.

dumbmutate --mutate=FileToTest.cpp --build="make" --test="./test" --threshold=80

dumbmutate's People

Contributors

maatuuli avatar ragnarda avatar sparist avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

dumbmutate's Issues

Require `#include <limits>` in cxxopts.hpp

On Ubuntu 22.04:

FAILED: CMakeFiles/dumbmutate.dir/Debug/Source/main.cpp.o 
/usr/bin/c++ -DCMAKE_INTDIR=\"Debug\"  -Werror -Wall -pedantic -Wextra -g -std=gnu++11 -MD -MT CMakeFiles/dumbmutate.dir/Debug/Source/main.cpp.o -MF CMakeFiles/dumbmutate.dir/Debug/Source/main.cpp.o.d -o CMakeFiles/dumbmutate.dir/Debug/Source/main.cpp.o -c /home/parallels/Builds/dumbmutate-build/src/Dumbmutate/Source/main.cpp
In file included from /home/parallels/Builds/dumbmutate-build/src/Dumbmutate/Source/main.cpp:2:
/home/parallels/Builds/dumbmutate-build/src/Dumbmutate/Source/cxxopts.hpp: In member function ‘void cxxopts::values::detail::SignedCheck<T, true>::operator()(bool, U, const string&)’:
/home/parallels/Builds/dumbmutate-build/src/Dumbmutate/Source/cxxopts.hpp:488:43: error: ‘numeric_limits’ is not a member of ‘std’
  488 |             if (u > static_cast<U>(-(std::numeric_limits<T>::min)()))
      |                                           ^~~~~~~~~~~~~~

How to run your demo as your gif did?

I'm sorry to bother you. I just start to learn mutate test and really unfamiliar to testing framework.
I download your project and just did what you did as gif. I found it was different between you gif 's main.cpp and your demo's main.cpp .So I change the file by using TEST_CASE and doctest .Then I found that the file failed to compile.
error message like this:
Undefined symbols for architecture x86_64:
"doctest_detail_test_suite_ns::getCurrentTestSuite()", referenced from:
___cxx_global_var_init.5 in main-6b9dae.o
"doctest::getContextOptions()", referenced from:
doctest::detail::Result doctest::detail::Expression_lhs<int const&>::operator==(int const&) in main-6b9dae.o
doctest::detail::Result doctest::detail::Expression_lhs<int const&>::operator!=(int const&) in main-6b9dae.o
......
can you help me? It must have something wrong about doctest using. I use macos system.
I knew it is really a low level issue. But I really want to learn and run your project. Thank you!

[doc] Explain mutation testing

Came here from Reddit post. I had no idea what was mutation testing until I found this repo.
It would be nice if you could add some resources, explaining what is mutation testing and why use it. At least a link to a blog post, for example
Edit: just noticed someone already asked that on Reddit. Well, now there's an issue to track progress :p

questions about using

  1. what testing frameworks can be used with ‘dumbmutate’?
  2. Is there a more detailed example? Or Can you explain the meaning of each parameter in detail?
    Thank you!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.