stryker-mutator / stryker-handbook Goto Github PK

See stryker-mutator/mutation-testing-elements#7

Please do you have guideline how to write custom mutators ?
Outside the project or how to contribute one in the striker ?

Thanx a lot

The React guide has a dependency on Create-react-app. This is not mentioned in the guide.

The project type should also be different for a TypeScript project and we should also define how to start if you don't use create-react-app.

There should be a mutator that replaces numeric constants. I imagine the following variants:

• constant 0
• constant 1
• subtract/add 1

this should catch some off-by-one errors.

Example: It would detect that the following function:

export function lessThan100(x: number): boolean {
  if (x < 100) {
    return true;
  }
  return false;
}

Is not appropriately tested by the following testcases:

describe('lessThan100', () => {
  it('should detect which numbers are less than 100', () => {
    expect(lessThan100(100)).toEqual(false);
    expect(lessThan100(98)).toEqual(true);
  });
});

As I remember, switch mutator has been introduced a couple months ago, but I can't find it in list.

Am I missing something?

The stryker dashboard (https://dashboard.stryker-mutator.io) supports now hosts your full html report (although it is not fully advertised yet). We should implement this in Stryker, Stryker.NET and Stryker4s.

We can use this issue to discuss and align design choices across Stryker, Stryker.NET, and Stryker4s.

I'll keep this original post up to date with current design decisions.

Changelog

18-11

• Don't enforce the version name to be url encoded. You can use / instead of %2F.

13-11

• Remove "mutation-score-only-reports" again fromin the mutation testing report schema, instead allow report upload without object wrapper.

13-11

• Add "mutation-score-only-reports" in the mutation testing report schema

25-10

• Document 'mutation score'-only reports.

21-10

• Use Stryker settings rather than environment variables by default.
• Rename "repository slug" to "project"
• Rename "moduleName" to "module"
• Remove "provider". It is now part of the "project"
• Allow baseUrl to be overridden

Request

You can send a report with an HTTP PUT request, like so:

curl -X PUT \
  $BASE_URL$/api/reports/$PROJECT$/$VERSION$ \
  -H 'Content-Type: application/json' \
  -H 'Host: dashboard.stryker-mutator.io' \
  -H 'X-Api-Key: $API_KEY$ \
  -d '$REPORT_JSON$'

Multiple results per VERSION are also supported using this url: $BASE_URL$/api/reports/$PROJECT$/$VERSION$?module=$MODULE_NAME$

A report can also contain only a "mutation score" (without the files and mutants). This way, you will have a mutation score badge, but no other information is stored. In that case, the $REPORT_JSON$ should be in the format { "mutationScore": 42 }

The variables here are:

• BASE_URL: https://dashboard.stryker-mutator.io for production, https://stryker-dashboard-acceptance.azurewebsites.net/ for acceptance
• PROJECT: The name registered with the dashboard. It is in the form of gitProvider/organization/repository. At the moment the dashboard backend only supports github.com as a git provider, but we will also support gitlab.com/bitbucket.org, etc in the future. It can have an indefinite number of levels. Slashes (/) in this name are not escaped. For example github.com/stryker-mutator/stryker-net.
• VERSION: the version of the report. This should be filled with the branch name, git tag or git sha (although no validation is done). You can override a report of a specific version, like docker tags. Slashes in the version should not be encoded. For example, it's valid to use "feat/logging".
• API_KEY: The API key that you retrieved by enabling the report on the dashboard.stryker-mutator.io website.
• REPORT_JSON: A valid report according to the mutation testing report schema, or a mutation score only report in the form of { "mutationScore": 42 }
• MODULE: Optional. If you want to store multiple reports for a version, you can use this value to separate them logically. For example, in a mono-repo setup where each package (or project or module) delivers a report.

Reponse

{
  "href": "https://dashboard.stryker-mutator.io/reports/github.com/stryker-mutator/stryker-net"
}

The href you can communicate to the end-user.

Implementation suggestion

I think we should try to align on the implementation a bit. That way enabling stryker dashboard reports in your builds is the same across all implementations of Stryker. Current suggestion:

• Enabling the dashboard reporter is done by adding "dashboard" to the "reporters" configuration.
• Configuring the "project" is done via the dashboard.project configuration key.
• Configuring the "module is done via the dashboard.module configuration key.
• Configuring the "version" is done via the dashboard.version command-line variable. Configuration file might also be useful for consistency, but not encouraged as you will not be able to support different branches.
• Configuring the "baseUrl" is done via the dashboard.baseUrl configuration key. It defaults to "https://dashboard.stryker-mutator.io".
• Read the API key from the environment variable "STRYKER_DASHBOARD_API_KEY".
• If the "project" and/or "version" setting is missing it might be discovered from the environment variables of the build server. For example: Circle CI, Travis, Azure Devops, Github actions, Gitlab CI.

Open questions:

• STRYKER_DASHBOARD_API_KEY: Do we also want to provide it using a command-line variable?

After a short discussion with @nicojs yesterday I think it's best to open an issue.

Right now there's no discussed way of calculating what the score should be when there are 0 mutants. I'm not talking about showing the score as there's already a PR for this in elements: stryker-mutator/mutation-testing-elements#52.

We already have standardized ways of calculating metrics, and I think this is a hole in that standardized way that should be filled.

There's 3 options (I think) that all have a valid argument:

• 0, no mutants were killed
• 100, all mutants were killed
• NaN no sensible calculation that is a number can be made

Personally, I prefer NaN. Would like to hear your thoughts. cc: @nicojs @simondel @richardwerkman @Mobrockers

Currently the list of mutators says:

• Binary Operators
• Boolean Substitutions
• Logical operators
• Unary operators
• Update operators
• Remove conditionals
• Assignment mutator
• Array declarator
• String mutator
• Block statement
• Checked mutator
• Method mutator

The list should become (proposal):

• Arithmetic operator (+, -, *, /, %)
• Equality Operator (<, >, >=, <=, ==, !==)
• Logical operator (&&, ||)
• Boolean literal (true, false, !)
• Unary operator (+, -)
• Update operator (++, --)
• Conditional expression (remove condition)
• Assignment expression (+=, -=)
• Array declarator literal ([])
• String literal ("")
• Block statement ({})
• Checked statement (checked())
• Method expression (rename, remove method)

It is pretty inconsistant. Some have "mutator" in their name, others don't. Some say what the mutator does (Remove conditionals), others just describe what kind of operators they target.

I think we should make this list more consistant, and use a format for each mutator name.

Maybe all should be called mutator? Or all should describe what kind of statements/expressions they mutate. What do you guys think?

Is your feature request related to a problem? Please describe.
As of today, mutant naming is inconsistent in Stryker.Net (note that I probably contributed to that), there is a liberal of use of mutants and mutations as being one and the same.
On a similar topic, I have never been satisfied with the status 'CompileError' which sounds far too technical.

Describe the solution you'd like
I propose to adhere to stricter guidelines:

• a mutation should refer to individual mutations that Stryker generates thanks to mutators?
• a mutant should refer to a variant of the original project with at least one mutation.
• a mutated assembly is a variant of the original assembly integrating several mutations that can be individually activated.
• a mutation that cannot be tested because it triggers compilation error(s) should be flagged as 'Non viable*

In general, I propose we adhere to whatever de facto standard appears to exist. Pitest comes to mind, but I let you decide which source makes more sense if any.

Agreeing to these rules or some alternative would allow to review and clean up existing logging entries and messages to make them most consistent.
They should also be added to the contribution guidelines, of course.

In stryker-net we want to introduct a new feature: mutant levels. By setting a higher mutation level, more complex mutants are placed. All higher levels include the lower level mutations as well. Maybe we can think of universal levels for each stryker. So the levels mean the same thing across stryker.

My current proposal for the levels is:

This proposal is for mutators as a whole, but I can imagine that some mutations inside the mutator belong to another level than others. For example:
> to < basic
> to => advanced

Hi there.

First, I want to thank you for the amazing work you've done.

I'm making a stryker-json-reporter in order to dump the report in a json file and then consume it from another app for historical purposes.

Reading about plugins I came across the term PluginKind. This is probably a neat but, would you be alright if I rename all of it's singular and plural occurrences to PluginType instead?

https://github.com/stryker-mutator/stryker-handbook/blob/master/stryker/api/plugins.md

Cheers

The title error appears when attempting to build. I have numerous test projects which have worked without issue now I am suddenly seeing this error. Any help would be appreciated

• Switch mutator which is now part of the conditional expression mutator but does not seem to be documented at this time (#20)
• object literal mutator
• mutator of Array constructor missing new which seems to exist for JavaScript but not for TypeScript (stryker-mutator/stryker-js#1902) does not seem to be documented in https://github.com/stryker-mutator/stryker-handbook/blob/master/mutator-types.md#array-declaration

I can see, that this repository is not updated very often, and it mostly duplicates things we already have in documentation. Should we still care and maintain this project? Or we could find more important parts not included in docs and send there?

I'm look into mutators test frameworks. And found striker a good option, but in any place it talks aboult the paraller runners. There is anything that I'm missing?

https://github.com/stryker-mutator/stryker-handbook No FAQ and no Technical Reference links.
Yet technical reference is here - https://github.com/stryker-mutator/stryker-handbook/tree/master/stryker/technical-reference

Javascript &Typescript Getting Started links to empty md file as does Mutation Testing.