stryker-mutator / stryker-js Goto Github PK
View Code? Open in Web Editor NEWMutation testing for JavaScript and friends
Home Page: https://stryker-mutator.io
License: Apache License 2.0
Mutation testing for JavaScript and friends
Home Page: https://stryker-mutator.io
License: Apache License 2.0
We should use underscores in all module names
See if we need better option names. Discussion points:
If Stryker was unable to mutate the source code, the rest of the mutation testing process still continues (starting multiple test runners, logging the mutation score etc), while we can simply quit if there are no mutants.
TypeUtils was designed to add runtime checks to validate the types. We don't need it anymore as we moved to compile time checks with typescript.
The stryker mutation test process seems to hang for a few seconds before existing. This is due to the timeout set in the IsolatedTestRunnerAdapter
(it is never canceled)
We need a facade to start a test-runner in a child process. This way, every test runner can stay clear of starting child processes itself. The timeout can also be much easier implemented in this facade, because he can simply kill the child process if it takes too long (and start a new one).
This can be easily implemented using an adapter
pattern.
We should rewrite the karma test runner to keep the karma server open instead for the entire test run, instead of opening up a new node process for each mutation test run. This can be done using the way of working in the karma-coverage-per-test prototype.
We can maybe also add a new abstraction for starting child processes and handling them, so we don't need to recreate it at a later stage (server side unit tests).
For example:
node dist/src/Stryker.js -m test/sampleProject/src/*.js -f test/sampleProject/src/*.js,test/sampleProject/test/AddSpec.js,test/sampleProject/test/CircleSpec.js
Expected:
ERROR: One or more tests failed in the inial test run:
ReferenceError: Can't find variable: someGlobalVariableThatIsNotDeclared
at c:/z/github/infosupport/stryker/test/sampleProject/src/Error.js:9
Actual:
ERROR: One or more tests failed in the inial test run:
I discussed this feature with colleague @Jaapapa.
Basically we would like a sonar qube plugin. This plugin would output a single metric (at first): the mutation score. A sonar qube adminstrator would then be able to add a threshold to that metric, so when the mutation score falls below 80%: your sonar qube quality report is red.
To do this we should define these steps:
Right now, all code is being mutated. Code coverage has been collected before code is mutated. This makes it possible to only mutate code that has actually been tested. It is possible that only mutating tested code is slower than mutating everything due to the fact that code coverage has to be analysed.
This feature should not replace the current 'mutate everything' system. It should be an option (or the default, making 'mutate everything' an option).
As we moved to typescript, it would be relatively easy to move to attributes instead of getters-setters. If we need logic in the future, we'll replace the attribute with an ES 5 property.
For every Jasmine test that is ran, a new nodeJS process, a Karma server and a PhantomJS is started. This wastes a lot of time. By keeping the nodeJS process, the server and the browser running in between tests, tests could be ran much faster.
A proof of concept for this feature has been made by Nico which shows it is possible to keep the processes running while collecting code coverage for every test. The proof of concept can be found here.
In certain situations, the mutation testing process can get stuck. While this happens, several nodeJS, Karma and PhantomJS processes are still running.
If stryker freezes in a certain project, it will always freeze at that point. However, if another pc is used it can freeze at another point in the mutation run, or it does not freeze at all. In some cases stryker only freezes when the option individualTests
is turned on.
The issue can be fixed by enabling logging of nodeJS, Karma and PhantomJS processes. A temporary workaround has been added which reroutes the stdout of the child processes to an empty function. This enables the logging (which fixes the freezing) while not actually showing anything.
Tsd is depricated, we should move to typings
for our typescript typings.
escodegen applies formatting to the code. This usually doesn't matter, but in a few cases it does. This causes the mutated line of code to be shown incorrectly because line numbers have changed.
An example can be seen using the escodegen online demo by using the following code:
var arr = ['A', 'B'];
var obj = { name: 'Simon', age: 22 };
A possible solution is to ignore this feature of escodegen and search the AST for the code which should be on the specified line number. By generating this part of the AST and removing all line breaks, the correct line of code could be shown in the report.
When Karma errors in the initial test run it is treated as if there are no tests. When you turn off the silent
flag you can see the following output:
INFO: Running initial test run
using config {"browsers":["PhantomJS"],"frameworks":["jasmine"],"autoWatch":false,"singleRun":false,"plugins":["karma-jasmine","karma-phantomjs-launcher","karma-coverage"],"files":[...],"port":1234,"coverageReporter":{"type":"in-memory"},"preprocessors":{...},"reporters":["coverage"]}
24 03 2016 22:14:08.982:INFO [karma]: Karma v0.13.22 server started at http://localhost:1234/
24 03 2016 22:14:08.986:INFO [launcher]: Starting browser PhantomJS
24 03 2016 22:14:09.494:INFO [PhantomJS 1.9.8 (Windows 8 0.0.0)]: Connected on socket /#tOMWTOX3l_ZzzTYYAAAA with id 67783735
karma run done with 1
I've tested this using our internal KarmaDesigner project with the following arguments:
"program": "${workspaceRoot}/node_modules/stryker/dist/src/Stryker.js",
"stopOnEntry": false,
"args": ["-s","app/scripts/app.js,app/scripts/httpConfig.js,app/scripts/resources/ConditionRsc.js,app/scripts/resources/PageDtoRsc.js,app/scripts/resources/SurveyResultRsc.js,app/scripts/resources/SurveyRsc.js,app/scripts/resources/SurveySummariesPageRsc.js,app/scripts/routes.js,app/scripts/services/dragDropHelper.js,app/scripts/services/idGenerator.js,app/scripts/services/jsonHelper.js,app/scripts/services/localStorage.js,app/scripts/services/modalHelper.js,app/scripts/services/notificationStorage.js,app/scripts/services/random.js,app/scripts/services/surveyFind.js,app/scripts/services/surveyForEach.js,app/scripts/services/surveyStorage.js,app/scripts/services/tenantHelper.js","-o","test/spec/controllers/editFactRuleSet.test.js"],
For clarification: i deliberately not included the angular js sources in the otherFiles
list. I would expect Stryker to show me this (cannot call module on undefined
or something)
We should move styker-api, stryker-karma-runner, stryker-jasmine to seperate repositories before release 1
We should add dev-dependencies to them to add integration tests.
It seems that Karma-coverage is reporting more than expected. For example: the result of Add should be able to add two numbers
(see below).
I will bring forward the in-memory karma-coverage reporting to see if that solves this issue
EDIT Removed too long example:
Browser based Mocha tests are not yet supported. This feature could use the Karma test runner. See #10 for server sided Mocha support
It would be nice to have support for multiple severities for the console reporter such as info
or debug
. This gives the user the option to see as much or as little output as he/she wants.
We want a more clean the mutation api. It should be as easy as possible to create mutations.
We need a new api for Mutant. Right now: mutants handle their own file IO. The save the mutation to disk to a file of their choosing (using fileUtils.createFileInTempFolder
). There are 2 issues with this way of working.
I would suggest the following API per test mutation
Mutant
+ sourceFile: string; // path to original source file`
+ mutationPosition: Location; // location of the line to be mutated
+ substitude: string; // the piece of code to substituted (the actual 'mutation)
The orchestration of which source file corresponds to which source file per test runner should be done outside of the Mutant class
@simondel Do you agree?
All map
types in test_runner/Coverage.ts
are typed using a number indexer. For example:
export interface StatementMap {
[ref: number]: Location;
}
Although in practice, the ref is always a number, we should change it to be string. This because the usage is somewhat strange:
coveredFile.statementMap[parseInt(statementId)]
With parseInt
we convert the string to a number, but for the indexer []
it is converted back to a string. This is kind of pointless.
Entire statements could be removed to mutate code
Projects which include HTML files (such as AngularJS projects) are not properly supported. Support for these projects has to be added. This feature could be tested on an AngularJS project to ensure it works.
As we moved to typescript classes, it is easy to recognize private variables. We don't need the underscore _
prefix anymore.
If a test fails during the initial test run, the text: ERROR: One or more tests failed in the inial test run!
is logged. A user won't know which test failed, making it difficult to find the issue.
If a test fails during the initial test run, the user should see which test(s) caused the error.
Support has to be added for the QUnit test framework.
Integrate the new TestRunner
and TestFramework
apis in Stryker. Make sure to support the options we already had, maybe rename them.
For the TypeScript migration we should replace all constructor functions by actual typescript classes. We'll start doing this in the typescript-migration
branch and comment here when we convert each file.
Certain array elements could be removed as part of a mutation.
HTML reporting can make the final report of the mutation test more readable. This HTML report can be similar to a HTML code coverage report.
Right now, most promise reject's are silently ignored. Before we release v1, we should add error handling everywhere.
Server sided Mocha tests are not yet supported by stryker. See #11 for browser based Mocha test support.
The Typescript files should be compiled to JavaScript files during a NPM install
Can we get a contributor guide in the repo so people know how they can get their changes included?
I have a standard template ready to go: https://github.com/infosupport/contribute-md-template
If you like I can make a pull request for it as well :-)
I would like to add a way to read config using a config file instead of command line arguments. Very similar to what karma uses.
Note: the current command line arguments would NOT be replaced with a config file
Proposal:
stryker
you can supply a config file with --configFile
(or -c
?). stryker.conf.js
as a default.module.exports = function(config) {
config.set({
files: [
'lib/**/*.js',
'src/**/*.js',
'test/**/*.js'
],
mutate: [
'src/**/*.js'
],
// additional optional options with default values:
testFramework: 'jasmine',
testRunner: 'karma',
timeoutMs: 2000,
timeoutFactor: 1.5,
plugins: ['stryker-*'],
port: 9234
});
};
stryker-
.ConfigWriterFactory
.
stryker.conf.js
file. Indeed it has to, because that file defines the plugins to be loaded in the first place.stryker-karma-runner
could register an additional ConfigWriter
with the name 'karma'
to the ConfigWriterFactory
. That KarmaConfigWriter
can look for a property with the name karmaConfigFile
in the existing config. It can read that karma.conf.js
file and than choose to override some of the settings in the stryker.conf.js
file. For example override the files
based on the karma.conf.js
's files
key.Does everybody agree with this proposal?
stryker-cli
in the future where you would only accept a config file.*The base timeout is not saved with the properties of a test when it is placed in a queue. This timeout is read from the config when the test is ran. In many cases, the timeout has already been overwritten by the base timeout of another test.
As we want to transform our code to TypeScript, the first step will be to transform our code *.ts files. We should add those to source control instead of our js files. We should also add a compile step into our build system
We should not allow for implicit any type in TypeScript. Variables without type annotations are usually a mistake. After all type info is added we should enforce that we don't get implicit any-typed variables by setting noImplicitAny: true
in tsconfig.json.
We work on the typescript-migration branch. Please keep this todo list in sync:
Tests:
(User created) properties of objects could be removed to mutate code.
Right now: we always load the source
files before other
files. This approach will not work when you need to load library files.
For example: you need to load angular, after that your own source files, after that your test files. Karma solves this by having one big list of files. Thus you have fine grained control of the order of files.
I'm thinking of introducing the same principle for stryker. Instead of source files and other files, you'll have files
and source
files. The files
list also includes the source files and gives you fine grained control of the order of files. The source files
are the files we are going to mutate.
If the source
files contains a file which is not included in the files
list, we'll either have to give an error, or load the files before or after the 'normal' files.
@simondel what do you think?
It should be possible to configure which mutation types are and which aren't used. The importance of this feature increases as the number of mutation types increases.
See #29 , we should create the jasmine test framework in order to pick and choose tests.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.