Giter Club home page Giter Club logo

playwright-coverage's Introduction

@bgotink/playwright-coverage Latest published version on NPM

Report coverage on playwright tests using v8 coverage, without requiring any instrumentation.

Usage

Install this package

yarn add -D @bgotink/playwright-coverage

Then add the reporter to your playwright configuration:

const config = {
  // ...

  reporter: [
    ['list'],
    [
      '@bgotink/playwright-coverage',
      /** @type {import('@bgotink/playwright-coverage').CoverageReporterOptions} */ {
        // Path to the root files should be resolved from, most likely your repository root
        sourceRoot: __dirname,
        // Files to ignore in coverage, useful
        // - if you're testing the demo app of a component library and want to exclude the demo sources
        // - or part of the code is generated
        // - or if you're running into any of the other many reasons people have for excluding files
        exclude: ['path/to/ignored/code/**'],
        // Directory in which to write coverage reports
        resultDir: path.join(__dirname, 'results/e2e-coverage'),
        // Configure the reports to generate.
        // The value is an array of istanbul reports, with optional configuration attached.
        reports: [
          // Create an HTML view at <resultDir>/index.html
          ['html'],
          // Create <resultDir>/coverage.lcov for consumption by tooling
          [
            'lcovonly',
            {
              file: 'coverage.lcov',
            },
          ],
          // Log a coverage summary at the end of the test run
          [
            'text-summary',
            {
              file: null,
            },
          ],
        ],
        // Configure watermarks, see https://github.com/istanbuljs/nyc#high-and-low-watermarks
        // watermarks: {},
      },
    ],
  ],
};

Now replace all calls to @playwright/test's test variable with a variant that tracks coverage. The easiest way to do this is by importing test from @bgotink/playwright-coverage instead.

-import {expect, test} from '@playwright/test';
+import {expect, test} from '@bgotink/playwright-coverage';

If you're already using a different test function, e.g. if you're using @ngx-playwright/test, you can add coverage tracking using the mixinFixtures function:

import {test as base} from '@ngx-playwright/test'; // or wherever your test function comes from
import {mixinFixtures as mixinCoverage} from '@bgotink/playwright-coverage';

export const test = mixinCoverage(base);

Now replace all usage of test with the function export defined there, and coverage will be tracked.

How does it work?

The fixtures registered in test or via mixinFixtures hook into created Pages to track javascript coverage with v8. The coverage data is added as attachment to every test.

Upon completion of all tests, the reporter merges all generated coverage files into one and then converts the v8 coverage format into the coverage format used by istanbul. The istanbul data is then passed into the reports of istanbul-reports.

Common issues

The HTML report shows errors saying the source files couldn't be read

This means the reporter is looking in the wrong place because playwright and the server process are using paths relative to a different working folder.

Try setting the sourceRoot folder. If you need more control over the actual path of the files, pass a rewritePath property in the options:

{
  sourceRoot: __dirname,

  /**
   * Modify the paths of files on which coverage is reported
   *
   * The input is an object with two properties:
   * - absolutePath
   * - relativePath
   * both are strings and they represent the absoslute and relative
   * path of the file as computed based on the source map.
   *
   * Return the rewritten path. If nothing is returned, `absolutePath`
   * is used instead.
   */
  rewritePath: ({absolutePath, relativePath}) => {
    return absolutePath;
  },
}

Coverage is empty

Did you perhaps use @playwright/test's own test function? If you don't use a test function created using mixinCoverage, coverage won't be tracked and the reporter won't have anything to report on.

Status

This project is very experimental. It has been proven to work on one angular application, i.e. with webpack with the unmodified configuration angular applies to it.

License

Licensed under the MIT license, see LICENSE.md.

playwright-coverage's People

Contributors

bgotink avatar hejiangtao-ecut avatar klathmon avatar mzedel avatar spenceryue avatar stovmascript avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

playwright-coverage's Issues

[Bug] Encountering error in Non Chromium browser test

Hi, thanks for developing this handy package!

I'm new to Playwright toolsets. Currently, I have my Playwright test cases running in all browser variants with the following dependencies:

But I'm getting this error in non Chromium browser.

 1) [firefox] › xxxx/xxx.e2e.ts:6:5 › <test-case-name>

    TypeError: coverage.startJSCoverage: Cannot read properties of null (reading 'startJSCoverage')

        at Object.page (/xxxx/node_modules/@bgotink/playwright-coverage/src/fixtures.ts:31:7)

I figure out that page.coverage.startJSCoverage object is never a null value in Non Chromium test. For now, I'm bypassing the error by patch the package with the following replacement. Not sure if it's the best way to resolve the error. If it's, I would love to help on submitting a pull request for the fix.

diff --git a/node_modules/@bgotink/playwright-coverage/lib/fixtures.js b/node_modules/@bgotink/playwright-coverage/lib/fixtures.js
index 2c957bd..8a31072 100644
--- a/node_modules/@bgotink/playwright-coverage/lib/fixtures.js
+++ b/node_modules/@bgotink/playwright-coverage/lib/fixtures.js
@@ -6,7 +6,7 @@ const data_1 = require("./data");
 const coverageFixtures = {
     collectCoverage: true,
     page: async ({ page, collectCoverage }, use, testInfo) => {
-        if (page.coverage == null || !collectCoverage) {
+        if (page.coverage == null || !collectCoverage || testInfo.project.use.defaultBrowserType !== 'chromium') {
             return use(page);
         }
         await page.coverage.startJSCoverage({

printsToStdio usage

Hello,
Custom reporters that do not implement printsToStdio function prevent pretty much any Playwright output besides their own.
Should printsToStdio: () => false be added to a reporter?
Printing summary in the end still works with this method added.

Add an option to rewrite paths

Hi, this is an awesome library!

I was trying to use it with one my projects, but all the paths have the webpack prefix _N_E on them, making it not work correctly:

image

image

If I had a way to rewrite those paths, I would be able to fix this issue.

How to solve the problem of source files not generating coverage due to the use of virtual paths?

Hello, I am developing a component and using the current library to detect coverage, but due to the use of @demo/component/radio to refer to the path ./src/radio/index.ts file in the document site's reference method, it cannot be resolved that @demo/component/radio points to the ./src/radio/index.ts file. If using ./src/radio/index.ts directly is not a problem, but it is not suitable for presentation as a component document. Can we define a sourcemap ourselves to point to this part of the reference? Or do you have any better suggestions to solve this problem?

TypeError [ERR_UNKNOWN_ENCODING]: Unknown encoding: base64url is thrown in vite dev server

How to reproduce:

  1. code example repo: https://github.com/stevez/playwright-test-coverage/tree/bgotink-coveage-lib-vite-integration
    git clone https://github.com/stevez/playwright-test-coverage.git
    git checkout bgotink-coveage-lib-vite-integration
  2. npm install
  3. npx playwright install
  4. npm run test

Then you will get following error:

(node:53635) UnhandledPromiseRejectionWarning: TypeError [ERR_UNKNOWN_ENCODING]: Unknown encoding: base64url
at fromString (buffer.js:454:13)
at Function.from (buffer.js:308:12)
at getSourceMap (/Users/s5474013/github/playwright-test-coverage/node_modules/@bgotink/playwright-coverage/lib/data.js:51:26)
at CoverageWorker._CoverageWorker_convert (/Users/s5474013/github/playwright-test-coverage/node_modules/@bgotink/playwright-coverage/lib/worker.js:74:123)
(Use node --trace-warnings ... to show where the warning was created)
(node:53635) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:53635) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:53635) UnhandledPromiseRejectionWarning: TypeError [ERR_UNKNOWN_ENCODING]: Unknown encoding: base64url
at fromString (buffer.js:454:13)
at Function.from (buffer.js:308:12)
at getSourceMap (/Users/s5474013/github/playwright-test-coverage/node_modules/@bgotink/playwright-coverage/lib/data.js:51:26)
at CoverageWorker._CoverageWorker_convert (/Users/s5474013/github/playwright-test-coverage/node_modules/@bgotink/playwright-coverage/lib/worker.js:74:123)
(node:53635) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:53635) UnhandledPromiseRejectionWarning: TypeError [ERR_UNKNOWN_ENCODING]: Unknown encoding: base64url
at fromString (buffer.js:454:13)
at Function.from (buffer.js:308:12)
at getSourceMap (/Users/s5474013/github/playwright-test-coverage/node_modules/@bgotink/playwright-coverage/lib/data.js:51:26)
at CoverageWorker._CoverageWorker_convert (/Users/s5474013/github/playwright-test-coverage/node_modules/@bgotink/playwright-coverage/lib/worker.js:74:123)
(node:53635) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 3)
(node:53635) UnhandledPromiseRejectionWarning: TypeError [ERR_UNKNOWN_ENCODING]: Unknown encoding: base64url
at fromString (buffer.js:454:13)
✓ [Chrome] › e2e/App2.test.ts:7:1 › use Turquoise as a background color (2s)
at getSourceMap (/Users/s5474013/github/playwright-test-coverage/node_modules/@bgotink/playwright-coverage/lib/data.js:51:26)
at CoverageWorker._CoverageWorker_convert (/Users/s5474013/github/playwright-test-coverage/node_modules/@bgotink/playwright-coverage/lib/worker.js:74:123)
(node:53635) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see h
✓ [Chrome] › e2e/App.test.ts:11:1 › use Red as a background color (477ms)

3 passed (4s)
Error in reporter TypeError [ERR_UNKNOWN_ENCODING]: Unknown encoding: base64url
at fromString (buffer.js:454:13)
at Function.from (buffer.js:308:12)
at getSourceMap (/Users/s5474013/github/playwright-test-coverage/node_modules/@bgotink/playwright-coverage/lib/data.js:51:26)
at CoverageWorker._CoverageWorker_convert (/Users/s5474013/github/playwright-test-coverage/node_modules/@bgotink/playwright-coverage/lib/worker.js:74:123)

====================================
I understand your library works in the webpack dev server: I have another branch confirmed this, even the coverage number is different with the number in the main branch: (https://github.com/stevez/playwright-test-coverage/tree/bgotink-coverage-lib),
But since I want to use the vite server, which source code are not bundled, I think this might cause the issue? I am not familiar with the v8 coverage format yet, maybe you can provide some insight?
My original plan was simple: in order to support coverage in vite, playwright tests just output the v8 format json file, then use c8 to generate the istanbul result -- it sounds easy, but the number is not correct, you can check my another branch for the demo:
https://github.com/stevez/playwright-test-coverage/tree/v8-coverage.

Either you can fix the issue to support vite, or any suggestion will be appreciated.

Incorrect path for covered files in Next.js & monorepo

Hello,
Trying to add coverage to Playwright using this package with my monorepo and Next.js (being in <root>/apps/next dir), I stumbled upon having only files from Next.js dir but no files from other packages directories (e.g. <root>/packages/components).
Digging down, I found that emitted coverage JSONs look something like this:

{
  "version": 3,
  "file": "../../packages/components/component.tsx",
  "mapping": "...",
  "sources": ["webpack://_N_E/../../packages/components/component.tsx?xxxx"],
  "sourcesContent": ["..."],
  "names": [...],
  "sourceRoot": ""
}

Eventually:

  • this JSON gets to v8-to-istanbul package
  • where the sources[0] is used as a path to resolve a filename (with cut webpack:// part)
  • which results to a join("", "_N_E/../../packages/components/component.tsx")
  • hence ../packages/components/component.tsx losing one level of double dots.

I'm not sure if I can manage this user-land, probably not. Using the same rewritePath on paths in source map could be a good call though passing a function to a worker is not an option I guess.

Let me know if I can give you more details on the issue.

Update:
I managed to fix this issue by making this diff:

# data.js
 {
   source,
-  sourceMap: {sourcemap: sourceMap},
+  sourceMap: {
+    ...sourceMap,
+    sourceRoot,
+    sources: await Promise.all(
+      sourceMap.sources.map((relativePath) =>
+        rewritePath({ relativePath })
+      ))
+  },
 },
 path => {

The rewritePath is passed with a proxy of a comlink:

// reporter.js
import { wrap, proxy } from "comlink";
...
const totalCoverage = await this.worker.getTotalCoverage(sourceRoot, this.exclude, proxy(this.rewritePath));

Cannot read property 'trimEnd' of null in v8-to-istanbul

When I run against my app I get the following exception:

TypeError: Cannot read property 'trimEnd' of null
    at new CovSource (/Users/scott/cxone/apps/cxone-hybrid-webapp-boilerplate/node_modules/v8-to-istanbul/lib/source.js:6:27)
    at /Users/scott/cxone/apps/cxone-hybrid-webapp-boilerplate/node_modules/v8-to-istanbul/lib/v8-to-istanbul.js:54:90
    at Array.map (<anonymous>)
    at V8ToIstanbul.load (/Users/scott/cxone/apps/cxone-hybrid-webapp-boilerplate/node_modules/v8-to-istanbul/lib/v8-to-istanbul.js:54:57)
    at convertToIstanbulCoverage (/Users/scott/cxone/apps/cxone-hybrid-webapp-boilerplate/node_modules/@bgotink/playwright-coverage/src/data.ts:183:5)
    at CoverageReporter.onEnd (/Users/scott/cxone/apps/cxone-hybrid-webapp-boilerplate/node_modules/@bgotink/playwright-coverage/src/reporter.ts:73:30)
    at Multiplexer.onEnd (/Users/scott/cxone/apps/cxone-hybrid-webapp-boilerplate/node_modules/@playwright/test/lib/test/reporters/multiplexer.js:73:7)
    at Runner._run (/Users/scott/cxone/apps/cxone-hybrid-webapp-boilerplate/node_modules/@playwright/test/lib/test/runner.js:363:7)

I will try to dig into it but wanted you to be aware. If you feel there's anything I can provide to help debug this then don't hesitate to let me know.

Hidden source Map

Is there a way to get coverage working with the hidden source map feature of angular ?
"sourceMap": { "hidden": true, "scripts": true, "styles": true }
In that case the browser does not know about the source map, but it could be attached.
Is there a way to attach the source map via playwright or your extension ?

Error: Error while reading @bgotink/playwright-coverage:

@bgotink I'm trying to imlplement your solution but when I execute the test it presents the following error:

(node:79038) [DEP0148] DeprecationWarning: Use of deprecated folder mapping "./" in the "exports" field module resolution of the package at .../node_modules/@playwright/test/package.json.
Update this package.json to use a subpath pattern like "./*".
(Use node --trace-deprecation ... to show where the warning was created)
Error: Error while reading @bgotink/playwright-coverage:
Cannot find module '.../@bgotink/playwright-coverage'

image

Can you help me please?

Which sourcemaps work?

We have inline sourcemaps, and that didn't work. I tried some other option but while it generated a report, that report contained 0 coverage information.

Every line is marked as being covered

After first having issues with zero coverage, now I have basically the opposite problem of #5 — every line is marked as covered.

Once I solved issues like #23 and providing rewritePath to fix any path issues, I saw code coverage information trickling in. However, it turns out that for some reason every single line in every single file is marked as covered, even though that can't be true.

=============================== Coverage summary ===============================
Statements   : 100% ( 8717/8717 )
Branches     : 100% ( 0/0 )
Functions    : 100% ( 0/0 )
Lines        : 100% ( 8717/8717 )

This is not necessarily a bug in this package, but I am opening this issue here to get some help with figuring out the root cause and how to address it.

From what I can tell, source map extraction etc. works fine, so maybe it's an issue within convertToIstanbulCoverage?

My WIP code can be found here: swissspidy/media-experiments#235

Get zero coverage in monorepo

Hello! I'm having some issues getting coverage in a monorepo where Playwright runs from a folder root/e2e-tests/ and tests a server located at root/frontend/. I start the server and follow the instructions from the Readme, but end up with:

================ Coverage summary ================
Statements   : Unknown% ( 0/0 )
Branches     : Unknown% ( 0/0 )
Functions    : Unknown% ( 0/0 )
Lines        : Unknown% ( 0/0 )
==================================================

I've tried various fixes, including pointing sourceRoot to the monorepo root and to the server folder.

I've had a similar issue setting trying to set up the coverage myself following https://playwright.dev/docs/api/class-coverage. Maybe I have a step missing? I think I don't need to instrument the server code using babel-plugin-istanbul, due to using v8 directly?

Source map can not be fully loaded when there is only one test case

I found an interested issue. It works fine when there are more than 1 test cases, however, it can not work if there is only one test case.

2 test cases, even if there is one empty test case, the result case be as expected:
image

1 test case, will encounter the following issue.
image

Could you kindly help to take a look?.
For reproducing: https://github.com/hsuehic/playwright-react-typescript
master branch for 2-test-cases scenario.
issue/cannot-with-one-test-case branch for 1-test-case scenario

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.