stryker-mutator / mutation-testing-elements Goto Github PK

Mull is interested in using mutation testing elements, see https://twitter.com/1101_debian/status/1113372029342842889

@AlexDenisov
I don't think we need to do any changes to the schema, do you agree? We would need to add C and C++ to the highlight js imports.

Dependabot can't resolve your JavaScript dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Error whilst updating prismjs in /packages/mutation-testing-elements/package-lock.json:
No matching version found for mutation-testing-metrics@^1.3.1.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

When the mutation-testing-elements report loads a language it does not have build-in syntax highlighting for, it will fall back on the autoloader. It will actually download the language highlighting on the spot. This works with the prism autoloader plugin.

However, there is currently no way to register a callback for when syntax highlighting is done. This is vital since we need to attach mutants to the properties of the mutant-components here:

The result is that the mutants will not be bound to the mutation-test-report-mutant elements which in turn makes the mutants invisible

See PrismJS/prism#2275

Thanks to @theofidry for pointing this out to me in infection/infection#1149

As a workaround, we should disable autoloading and just add languages whenever we want to support one. We'll be adding php shortly 🤝

Dependabot can't resolve your JavaScript dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Error whilst updating @types/prismjs in /packages/mutation-testing-elements/package-lock.json:
No matching version found for mutation-testing-metrics@^1.3.1.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

The readme for the schema is very short. It could be improved with some explanation about what it is and meant to be used for

Dependabot can't resolve your JavaScript dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Error whilst updating @types/prismjs in /packages/mutation-testing-elements/package-lock.json:
No matching version found for mutation-testing-metrics@^1.3.1.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Right now only files and folders are seperated. It would be nice if we could also mark a folder as a project.

Here we have two projects: SharedProject and TestApp. Right now it could be confusing to see the projects as folders. Maybe a different icon could help?

Where do we want to host additional information about the mutators? Like MutatorName? This is now missing entirely from the report, but it might be useful. For example, you can disable certain mutators by name, just as soon as you know the mutator name.

In the old report, we had a popup using bootstrap js:

I don't want to load the entire bootstrap js (it isn't "webpackable" and relies on JQuery 😢 ). So I would need to recreate it. At the same time, I'm not really happy with putting this info in a popup.

We also had a table with all the info:

I wasn't happy with that either. It can feel like an overload of info. It also took valuable space at the right side of the screen, resulting in a horizontal scrollbar for the code area.

What solution do we want?

We can do something like a drawer panel and fit the table in that. We could make it so that you can activate, or deactivate a mutant from there as well. https://www.w3schools.com/howto/tryit.asp?filename=tryhow_js_sidenav

Other ideas?

Mutation names get hard to read when they are long.

Example:

Maybe we could fit the popup to the content? Or we could make them larger by default?

Summary

I believe the original code should always be crossed-out when any of its mutants are "opened" (i.e. clicked) in the HTML reports. What seems to be happening instead is that only the last one of them actually triggers the text-decoration: line-through over its original code.

Example A (only mutant "3" uses strikethrough):

Example B (only mutant "8" uses strikethrough):

Stryker4s config

(not relevant / defaults)

Stryker4s environment

stryker4s 0.8.0

The schema has a free-form totals dictionary with header, value keypair. Since it is up to the report generator to decide the fields that should be available in the report, I would also like to be able to provide a description per column which could be displayed for example by hovering over the column header. We need this because we have some totals concepts such as caught mutants (killed by test vs timeout) vs killed mutants (killed by test failure).

Dependabot can't resolve your JavaScript dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Error whilst updating prismjs in /packages/mutation-testing-elements/package-lock.json:
No matching version found for mutation-testing-metrics@^1.3.1.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

The generated Mutation Report uses text colors that makes the text unreadable when using Dark Mode in Azure DevOps:

even though all the text is there (shown by selecting the text):

Describe the bug
My folder structure has multiple files, which has a folder name as a prefix.
E.g. here the folder BB is a prefix of both BB.cs and BBSomethingMore.cs.
When clicking BB.cs or BBSomethingMore.cs in the html report, I get a blank page.
When changing the folder name from BB to e.g. BB1, all three links works.

"files": {
  "BB\\CC.cs": {
    "language": "cs",
    "source": "FOO",
    "mutants": []
  },
  "BB.cs": {
    "language": "cs",
    "source": "BAR",
    "mutants": []
  },
  "BBSomethingMore.cs": {
    "language": "cs",
    "source": "BAZ",
    "mutants": []
  }
}

Logs

Expected behavior
I would expect all three links to work.

Desktop (please complete the following information):

• OS: Windows 10
• Type of project framework
• Framework Version net472
• Stryker Version beta 0.11.0

Additional context

Moved from stryker-mutator/stryker-net#521 for @jnyrup

Html report showing the behaviour:

mutation-report.html.txt

Dependabot can't resolve your JavaScript dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Error whilst updating prismjs in /packages/mutation-testing-elements/package-lock.json:
No matching version found for mutation-testing-metrics@^1.3.1.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

See:

Dependabot can't resolve your JavaScript dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Error whilst updating @types/prismjs in /packages/mutation-testing-elements/package-lock.json:
No matching version found for mutation-testing-metrics@^1.3.1.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

It'd be cool to have file type specific icons, similar to VS Code. We could use something like pretty-file-icons for it. Mapping icons by hand is probably not a very good idea.

Note: pretty-file-icons seems to be specific for https://www.flaticon.com, which I am not a huge fan of

This suite of mutation testing elements should work with a generic json result. We should reach a consensus on this and create a json-schema for it.

This original issue will be kept up-to-date with the discussion here.

Current proposal:

{    // The root report
  "schemaVersion": "1", // major version of this report
  "thresholds": {
    "high": 80,
    "low": 60
  },
  "files": {
    "src/module1/Example.cs": {
      "language": "cs", // Or any other highlightjs identifier, see https://highlightjs.org/static/demo/
      "source": "using System; using.....",
      "mutants": [{
        "id": "321321",  // A unique id, can be used to correlate this mutant with other reports
        "mutatorName": "BinaryMutator",
        "replacement": "-",
        "location": {
          "start": { "line": 4, "column": 3 }, // lines and columns both start at 1
          "end": { "line": 5, "column": 2 }
        },
        "status": "Killed" // or "Survived", "NoCoverage", "CompileError", "RuntimeError", "Timeout"
      }]
    }
  }
}

Discussion started here: stryker-mutator/stryker-js#925

We should support mutation-score-only reports. The JSON looks like this:

{
  "mutationScore": 42
}

The report can look like this:

See stryker-mutator/stryker-handbook#27 for the use case

It would be nice if we could show why a mutant was marked as killed by displaying the testrun output. That way users can see what test case killed the mutant.

This could also help by detecting false positives. When the mutation score is much higher than expected it could be seen that many mutants have the same testrun output (for example a bad unit test that fails randomly)

We could show the testrun output when clicking on the mutant.

Dependabot can't resolve your JavaScript dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Error whilst updating @types/prismjs in /packages/mutation-testing-elements/package-lock.json:
No matching version found for mutation-testing-metrics@^1.3.1.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

In stryker-net we have coverage analysis so we can determine which mutants are covered by which tests. Currently we do not allow a mutation testrun if you have failing unit tests because this would mark every mutant as killed. With coverage analysis we can disable all unit tests that are failing while also marking and skipping all mutants covered by those failing unit tests. We would like to have a new status in the schema to reflect a mutant that is covered by a test but for which we cannot determine if it's killed or not due to the covering unit test not working.

Dependabot can't resolve your JavaScript dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Error whilst updating prismjs in /packages/mutation-testing-elements/package-lock.json:
No matching version found for mutation-testing-metrics@^1.3.1.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Navigation in HTML report fails (blank screen) if source file name contains 0x60 ASCII character. Tested with Google Chrome 75.0 and Mozilla Firefox 67.0. Example report: mutation-report.zip

(moved from #2)
Preview:

(I think the image speaks for itself? 😁 )

Note that I included the path of root folder in header, similar to how it's currently displayed when opening a specific file.

As requested through stryker-net #458 we should add line numbers to the file view. This makes finding the mutation in the original source code easier.

As disussed in the Stryker for .NET repository, it would be helpful to have a .NET implementation of the mutation-testing-metrics package so we have a way to calculate the metrics with after running the stryker-net tool.

This issues is a starting point for discussing where to start with adding this additional package.

1. How to structure the layout? The whole repository is structured as a npm repo with the default paths. Do we want to group them in their own toolstack folders inside of mutation-testing-elements/packages/mutation-testing-metrics ?

I'll see if I can start something in the meanwhile that can be copied over :-)

Dependabot can't resolve your JavaScript dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Error whilst updating prismjs in /packages/mutation-testing-elements/package-lock.json:
No matching version found for mutation-testing-metrics@^1.3.1.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

PIT is interested in using mutation testing elements see https://twitter.com/0hjc/status/1113369365066473473

We need to support mutants that don't have an exact location or replacement. Only line number and mutator name.

Example JSON:

{ 
  "schemaVersion": "1.1",
   "thresholds": {
    "high": 80,
    "low": 60
  },
    "files": {
       "src/main/foo.java": {
           "language": "java",
           "source": "public class Progam {\n    public static void main(String[] args) {\n    System.out.println(\"bar\");\n}",
          "mutants": [
            {
              "id": "1",
              "mutatorName": "StringLiteral",
              "location": {
                "start": {
                  "line": 3
                }
              },
              "status": "Survived"
            }
          ]
       }

@hcoles
Is this json something PIT could produce? Also: mutation-testing-elements is opinionated: "mutatorName" is a free text field, but "status" isn't. Do you agree with our definitions and metrics? Documented here: https://github.com/stryker-mutator/stryker-handbook/blob/master/mutant-states-and-metrics.md

I've installed sbt via the msi (as instructed by the readme), yet npm test doesn't work.

@hugo-vrijswijk could you please help me out?

It would be a nice addition if we could also push the used configuration for the mutation run.

Why would we want this?

Well, when we configure for example configure the thresholds to break on 70% mutation score I expect the front end to do some nice displaying on warning statuses. When for the next test run I set the mutation score threshold to 10% all will be green I suppose. It's then hard to compare the reporting from one report to another.

The configuration would be a nice addition to make the report more meaningful in a long history.

Since we have coverage analysis, we can show the user which testcases cover which mutants. We would like for the json schema to have support for the coverage analysis mapping. It would look something like this in the resulting json:

{
  "coverage-mapping": [
    {"Unit Test 3": [3, 4, 56, 345]},
    {"Unit Test 4": [3, 8, 45, 113]},
    {"Unit Test 5": [15, 68, 99, 1545]}
  ]
}

The table of the mutation testing elements have fixed headers now, while the headers in the json schema are not fixed. This results in "useless" table columns. For example:

Stryker.NET does not support:

• No coverage
• Runtime errors
• Transpile errors

Thus "total undetected" is always the same as "killed" and can also be left out.

For us it would be ideal if the components would display only the columns that are passed with the json.

In a report generated by Stryker.NET, literal expressions look like killed mutations. This can be confusing.

As seen in this example the null and true are green, but also represent survived mutations.

This probably has to do with the syntax highlighting and I'm not sure if we can influence this much?

Dependabot can't resolve your JavaScript dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Error whilst updating prismjs in /packages/mutation-testing-elements/package-lock.json:
No matching version found for mutation-testing-metrics@^1.3.1.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

When document.baseURI is set, relative href's, even fragments, will honor this location.

For example, when the report is served from this location: http://localhost:4200/report/github.com/stryker-mutator/stryker/master with base set to /, it makes the #file.js href navigate to http://localhost:4200/#file.js instead of http://localhost:4200/report/github.com/stryker-mutator/stryker/master#file.js

Hi,

When I run 'dotnet stryker' on a test directory with a .Net framework project, I get the error:

[10:48:20 INF] Using C:\Dev\xxx\src\net45\FizzBuzz\FizzBuzz.Tests\FizzBuzz.Tests.csproj as project file [10:48:22 WRN] Testrunner set from DotnetTest to VsTest because IsTestProject property not set to true. This is only supported for vstest. [10:48:22 INF] Started initial build using msbuild.exe [10:48:22 ERR] An error occurred during the mutation test run System.ArgumentNullException: Value cannot be null. Parameter name: path at System.IO.Path.GetFullPath(String path) at Stryker.Core.Initialisation.InitialBuildProcess.InitialBuild(Boolean fullFramework, String projectPath, String solutionPath, String projectName) in D:\a\1\s\src\Stryker.Core\Stryker.Core\Initialisation\InitialBuildProcess.cs:line 33 at Stryker.Core.Initialisation.InitialisationProcess.Initialize(StrykerOptions options) in D:\a\1\s\src\Stryker.Core\Stryker.Core\Initialisation\InitialisationProcess.cs:line 47 at Stryker.Core.StrykerRunner.RunMutationTest(StrykerOptions options) in D:\a\1\s\src\Stryker.Core\Stryker.Core\StrykerRunner.cs:line 69 [10:48:22 INF] Time Elapsed 00:00:02.1496907

I am using the v0.11.0 (beta) on a very simple project.

Step to reproduce
1-at the sln level 'dotnet tool install -g fotnet-stryker'
2-at the test project level 'dotnet stryker'

Project is .Net 4.5.2.

What did I miss?

Thanks!

It would be nice to be able to skip though a file by using some arrows.

Kinda like clicking next search result or next conflict in a merge confict editor. That way we can quickly check for survived and/or not covered mutants without scrolling trough the whole file.

Something like (excuse my bad designer skills):

This is especially useful when there are few survived mutants present in a large file. It's like where's waldo at the moment 😅

Dependabot can't resolve your JavaScript dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Error whilst updating prismjs in /packages/mutation-testing-elements/package-lock.json:
No matching version found for mutation-testing-metrics@^1.3.1.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

On the dashboard, the sticky legenda falls under the navbar:

My proposal is to allow for the top property to be overridden using a CSS custom property https://lit-element.polymer-project.org/guide/styles#customprops

Dependabot can't resolve your JavaScript dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Error whilst updating @types/prismjs in /packages/mutation-testing-elements/package-lock.json:
No matching version found for mutation-testing-metrics@^1.3.1.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Dependabot can't resolve your JavaScript dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Error whilst updating prismjs in /packages/mutation-testing-elements/package-lock.json:
No matching version found for mutation-testing-metrics@^1.3.1.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Dependabot can't resolve your JavaScript dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Error whilst updating prismjs in /packages/mutation-testing-elements/package-lock.json:
No matching version found for mutation-testing-metrics@^1.3.1.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

For example: when showing the report on the dashboard, it needs to know which version to use.

Hi, there. Thanks for your excellent work, my team is going well with stryker recently, and we are trying to recommand it to other teams.
We've also met some trouble with the html report feature recently. That is: when the json data is too large, the whole report page would stuck.
For example, a js file which contains nearly 1300 mutants would stuck for about 15-20s or even more. Another file with 4800+ mutants(I know this file is unreasonably large.....) got stucked for too long that chrome suggest to close it because of no responding.
Now I try to display the ' on-loading' tag when some big file is loading, but if too large file(like I mentioned before) is clicked, the whole page would stuck and I have to close it.
I wonder if it is possible to add some feature such as lazy loading to solve this problem?
Thanks for you time any way.

Perhaps I do not understand how the schema works, but I'm pretty sure this position property is a duplicate of location and should be removed

We should try to come up with a software design for the web components that we want to create.

On a high level, I think we will have 3 components:

There are other smaller components, like breadcrumb, mutation totals table, etc.

I think we shouldn't rely on any library/framework for these components. Let's try it with plain web components.

Example:

This is a regression as of 1.1.0.

Previous version:

Both Stryker.NET as Stryker have (or will have) ways to ignore certain mutations (either by mutator name or other means). We want ignored mutations to also be visible in the mutation score. We should add this as a mutator state.

See stryker-mutator/stryker-js#1472

@Mobrockers what should the name be? I would prefer Ignored as it is really clear that the user choose to ignore it. Something like skipped could have any reason.