Run Lighthouse and Pa11y audits directly in Cypress test suites
For documentation about @cypress-audit
, make sure to check the documentation website.
⚡ Run Lighthouse and Pa11y audits directly in your E2E test suites
Home Page: https://mfrachet.github.io/cypress-audit/
License: MIT License
Run Lighthouse and Pa11y audits directly in Cypress test suites
For documentation about @cypress-audit
, make sure to check the documentation website.
Is it possible to save the lighthouse results in a JSON format? The raw report does not solve the case as there is too much information to look into.
From the documentation I see how to check scores, such as Performance and SEO.
I would like to also be able to assert on underlying metrics, such as First Contentful Paint etc. Is it possible with this tool?
Before creating an issue, make sure to have setup the projects with the 3 steps provided in the README. You can also verify the examples folder and check if there is already an application using a similar setup to yours.
What does not work?
Error in runner when adding const { lighthouse, prepareAudit } = require("@cypress-audit/lighthouse");
to cypress/plugins/index.js
How to reproduce?
add
const { lighthouse, prepareAudit } = require("@cypress-audit/lighthouse");
to
cypress/plugins/index.js
in terminal:
npx cypress open
Expected behavior
Runner should show available test suites
Environment (please complete the following information):
Any other information that may help fixing the issue?
This could be related to Optional chaining, not sure how to fix it since Node v16 and I think it should support it
In one of my projects, we have differents values with prod/dev project when using lighthouse. But we couldn't identify the reason for the number be lower when testing in prod. It should be able to get more descriptive infos, so we can fix it.
What does not work?
I have just upgraded Cypress for my automation project from 6.2.1 to 7.2.0, when I attempt to run the project with the command npm run cypress:open
I get the error below. When I check my index.ts file in the plugin folder, my import from cypress-audit gives the error File 'C:/REPO/AutomationProject/node_modules/cypress-audit/index.d.ts' is not a module.
I have deleted node_modules and installed multiple times. My cypress-audit version in my package.json is [email protected]
Can someone please look into this to see what the problem is?
How to reproduce?
Expected behaviour
The import should be successful and not throw an error
Environment (please complete the following information):
Your project looks really cool. I have an angular app using cypress via the Nrwl.io setup, but I'm stuck running this. I did the installation as suggested then tried to workaround the absence of @types/cypress-audit by faking an index.d.ts declaration but when I run I still get
src\support\index.ts
The error was:
Error: Webpack Compilation Error
E:/git/b2c/apps/appcheck-e2e/src/integration/app.spec.ts
ERROR in E:/git/b2c/apps/appcheck-e2e/src/integration/app.spec.ts(26,8):
TS2339: Property 'lighthouse' does not exist on type 'cy & EventEmitter'.
E:/git/b2c/apps/appcheck-e2e/src/integration/app.spec.ts
ERROR in E:/git/b2c/apps/appcheck-e2e/src/integration/app.spec.ts(27,8):
TS2339: Property 'pa11y' does not exist on type 'cy & EventEmitter'.
Any ideas?
Tim
Is there any way to run this in headless mode?
What I want to achieve is logging the results on cy dashboard during cypress run.
To take care of pa11y and to move cy.audit to cy.lighthouse
What does not work?
Lower performance when running llighthouse with cypress. I have a frontend app build with single-spa, this tool uses multiple scripts that comes from cdn.jsdelivr.net. When i run lighthouse inside of cypress i have a lower performance caused by the external scripts acording to lighthouse but when i run the test with chrome everything it's ok, i tried in multiple machines with differente enviroments and it's the same. I want to know if this is an expected behavior or there's something i'm missing
How to reproduce?
clone https://github.com/dapinto8/cypress-test, run npm run cypress:open
and then run the test lighthouse.spec.js
.
This repo is just a next js default app and also experiences lower performance
Expected behavior
Having a better performance
Screenshots / Animated Gifs
Performance ruuning lighthouse in chrome
Performance ruuning lighthouse in cypress
Opportunitties when lighthouse in cypress
Environment (please complete the following information):
npm audit reveals the following:
───────────────┬──────────────────────────────────────────────────────────────┐
│ Low │ Misinterpretation of malicious XML input │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package │ xmldom │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Patched in │ >=0.5.0 │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ cypress-audit [dev] │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path │ cypress-audit > lighthouse > jsonld > xmldom │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info │ https://npmjs.com/advisories/1650 │
What does not work?
New versions are not published to NPM
How to reproduce?
View the latest tag in NPM.
Expected behavior
Version 1.2.0
and 1.3.0
should be published in NPM
Screenshots / Animated Gifs
Environment (please complete the following information):
Any other information that may help fixing the issue?
NA
What does not work?
Trying to run examples/external-url locally I am getting a connection error.
How to reproduce?
Execute test in examples/external-url
Environment (please complete the following information):
Any other information that may help fixing the issue?
Read the comment here #62 but could not find a solution.
What does not work?
When running the cypress test command line, the test is skipped with the message 'Electron is not supported' and it stops as if the test is passed. But when I run the test with the cypress dashboard, the test fails as expected.
I don't know where to begin looking to fix this..
How to reproduce?
I only have a basic setup of cypress-audit:
plugin:
const { lighthouse, prepareAudit } = require("cypress-audit");
module.exports = (on, config) => {
on("before:browser:launch", (browser = {}, launchOptions) => {
prepareAudit(launchOptions);
});
on('task', {
lighthouse: lighthouse((lighthouseReport) => { // calling the function is important
console.log(lighthouseReport); // raw lighthouse reports
}),
log(message) {
console.log(message)
return null
},
table(message) {
console.table(message)
return null
}
})
}
integration:
it("should pass the audits", function () {
cy.visit('/')
const threshold = {
"performance": 100,
"accessibility": 100,
"best-practices": 100,
"seo": 100,
"pwa": 100
}
cy.lighthouse(threshold);
});
and run this with npm cypress run
Expected behavior
I expect when I run the test via terminal/command line to have the same result as when it's run from the dashboard.
Screenshots / Animated Gifs
Screenshot from the video output of the command line test:
Screenshot from the test in the cypress dashboard:
Environment (please complete the following information):
I am trying to run a very basic cypress audit test and seeing this issue Cannot destructure property 'errors' of 'object null' as it is null.
spec file
describe("External", () => {
it("check scores", () => {
cy.visit("https://google.com/");
cy.lighthouse({
accessibility: 50,
"best-practices": 50,
seo: 50,
pwa: 50,
});
});
});
chrome version - 89
cypress and cypress-audit version -
"cypress": "^6.4.0",
"cypress-audit": "^0.3.0",
index.js file
const { lighthouse, pa11y, prepareAudit } = require("cypress-audit");
module.exports = (on, config) => {
on("before:browser:launch", (browser = {}, launchOptions) => {
prepareAudit(launchOptions);
return launchOptions;
});
on("task", {
lighthouse: lighthouse((lighthouseReport) => {
console.log(lighthouseReport);
}), // calling the function is important
});
};
Will appreciate if you could share your thoughts on this. Thank you
What would you like to see in the library?
Get summary results (scores) inside a test
Why?
I generate an html report file and see the result in the log. But I need data from the log inside the test to process it and send it to the database.
Any other information that may help understanding the need
Test:
it('Lighthouse: Desktop', function () {
cy.viewport(1920, 1080).then(()=>{
cy.visit(pageUrl).then(()=>{
const lighthouseScore = {
performance: 0,
accessibility: 0,
seo: 0,
'best-practices': 0,
pwa: 0,
}
const lighthouseConfig = {
extends: 'lighthouse:default',
formFactor: 'desktop',
screenEmulation: {disabled: true}
}
cy.lighthouse(lighthouseScore, lighthouseConfig).then(()=>{
cy.get(databaseResult).then(()=>{
databaseResult['lighthouse_report'] = 'cypress/lighthouse/'+fileName+'lighthouse-report.html'
cy.log(
'cypress/lighthouse/'+fileName+'log.txt'
)
}).then(()=>{
cy.log(
`Lighthouse: report ready`
)
})
})
})
})
})
Plugins:
module.exports = (on, config) => {
function clearUrl(url){
url = url.replace("https://", '');
url = url.replace("http://", '');
url = url.replace("www.", '');
url = url.replaceAll("/", '_');
return url;
}
const fileName = clearUrl(config.env.PAGE_URL)
const lighthouseReportPath = config.projectRoot + '\\cypress\\lighthouse\\'+fileName+'lighthouse-report.html'
const lighthouseLogPath = config.projectRoot + '\\cypress\\lighthouse\\'+fileName+'log.txt'
/*cypress-audit*/
on("before:browser:launch", (browser = {}, launchOptions) => {
prepareAudit(launchOptions);
if (browser.name === 'chrome' && browser.isHeadless) {
launchOptions.args.push('--disable-gpu');
return launchOptions;
}
});
/*cypress-audit END*/
on("task", {
/*cypress-audit*/
lighthouse: lighthouse((lighthouseReport) => {
let lighthouseResult = JSON.stringify(lighthouseReport)
fs.writeFileSync(lighthouseLogPath, lighthouseResult, 'utf-8')
fs.writeFileSync(lighthouseReportPath, lighthouseReportGenerator.generateReport(lighthouseReport.lhr, 'html'))
console.log(lighthouseReport)
}),
pa11y: pa11y((pa11yReport) => {
console.log(pa11yReport)
})
});
/*cypress-audit END*/`
fs.writeFileSync(lighthouseLogPath, lighthouseResult, 'utf-8') - I get a lot of megabytes .txt file in which there is a huge amount of data
fs.writeFileSync(lighthouseReportPath, lighthouseReportGenerator.generateReport(lighthouseReport.lhr, 'html')) - i get html report
console.log(lighthouseReport) - I get what I need, but only AFTER running all the tests in the log.txt, during the test I cannot get this data from the file
I need to get data from log.txt and add it in the test to databaseResult array
What does not work?
We have a landing page -> 3rd party authentication (SiemensID wrapping auth0) -> the app.
We can use both UI login, or prog login. Prog login bypasses landing page and 3rd party authentication page by storing the token in session storage.
Landing page base url is the same as the app.
We login to the app, stabilize the URL. We start lighthouse. Lighthouse launches on a 2nd tab (perhaps expected). All evaluations are done against landing page instead of within the app.
In case we only run against the landing page ( we do not prog-login), the scores are the same. This proves lighthouse always runs against landing whether we are in the app or not.
How to reproduce?
Internal code, not able to share. Would be happy to have a Zoom meeting.
Expected behavior
Lighthouse executes for the application instead of the landing.
Screenshots / Animated Gifs
Video attached using cypress run --browser chrome
. The video does not show the 2nd tab we see on Test Runner. The 2nd tab shows the landing page.
Environment (please complete the following information):
Any other information that may help fixing the issue?
Add any other context about the problem here.
Whether we prog-login or UI-login, the 2nd tab where the audit is done is not aware of the session context -where the authentication token is stored. This seems like the gist. Changing the base url to the exact app page we want to run does not make a difference; again the 2nd tab is unaware of the session context and evaluates landing page.
This is our simple test.
describe('Lighthouse audit ', () => {
before(() => {
cy.progLogin(Cypress.env('USERNAME'), Cypress.env('PASSWORD'));
// we ensure we are in the appp
cy.location('pathname').should('contain', '/sites');
});
const thresholds = {
performance: 5,
accessibility: 50,
'best-practices': 50,
seo: 50,
pwa: 20
};
const desktopConfig = {
formFactor: 'desktop',
screenEmulation: { disabled: true }
};
it('should pass audit for empty site list', () => {
cy.lighthouse(thresholds, desktopConfig);
});
});
@mfrachet thought this might be useful to those hoping to use cypress-audit with an nx repo. Also I am having some issues with headless mode.
npm install cypress-audit --save-dev
const { preprocessTypescript } = require('@nrwl/cypress/plugins/preprocessor');
const { lighthouse, pa11y, prepareAudit } = require("cypress-audit");
module.exports = (on, config) => {
on("before:browser:launch", (browser = {}, launchOptions) => {
prepareAudit(launchOptions);
});
on("task", {
lighthouse: lighthouse(), // calling the function is important
pa11y: pa11y(), // calling the function is important
});
on('file:preprocessor', preprocessTypescript(config));
};
// Import commands.js using ES2015 syntax:
import './commands';
import 'cypress-audit/commands';
declare namespace Cypress {
interface Chainable<Subject> {
login(email: string, password: string): void;
lighthouse(...arg: any): any;
pa11y(...arg: any): any;
}
}
describe('todos', () => {
beforeEach(() => cy.visit('/'));
it('should use cypress audit', () => {
cy.lighthouse(
{
performance: 75,
accessibility: 95,
'best-practices': 93,
seo: 90
},
{
formFactor: 'desktop',
screenEmulation: {
mobile: false,
width: 1920,
height: 1080,
deviceScaleFactor: 1,
disabled: false,
},
});
});
it('should use pa11y audit', () => {
cy.pa11y();
})
});
ng e2e --watch
ng e2e
I'd like to clarify because I couldn't find information about this in the documentation or other issues, it is not possible to use this plugin together with lighthouse-ci, right?
My use cases are:
When I run the Perf testing for google page using the chrome lighthouse plugin the numbers are 100s and 90s. I do the same using the cypress-audit library using cy.lighthouse the numbers are like 40s and 50s.
Same goes with the other pages that Im currently working on my project. I was wondering as to why there's such a huge gap.
Prerequistes:
"cypress": "8.7.0",
"cypress-audit": "^1.1.0"
We are seeing obscure/strange page performance metrics within our company internal webpages using the \cypress-audit plugin with Cypress.
We recently did a POC with Puppeteer & the experience is fine. This looks to be an issue with the Cypress Runner.
STR:
ACTUAL:
Cypress-audit plugin metrics are very much incorrect, bloated. View the FCP & LCP numbers in the screengrabs below
EXPECTED:
I would expect the same (if not similar) metrics to the experience on the Google Chrome Browser
I keep on getting this error when running against this website. It works fine when I use this plugin using playwright.
https://github.com/abhinaba-ghosh/playwright-lighthouse
But, i use cypress much rather than playwright and i need these tests. Please look into it. i have made a repo so that it could be easy to reproduce the issue.Here is my repo
What does not work?
Values in window.sessionStorage
are not copied to the new tab when the Lighthouse audit is executed.
How to reproduce?
/myPage
) that requires a specific value in window.sessionStorage
in order to render its contentswindow.sessionStorage.setItem("my_item", "my_value")
) so that the page can be rendered properlycy.visit("/myPage")
)/myPage
(cy.lighthouse()
)For a runnable example, see https://github.com/olyhaa-aetna/simple-auth-example.
This repository has a sample app with two pages
welcomePage
- This page requires no authentication and has a login button. When the login button is clicked, a value is saved to the browser's sessionStorage
and we are redirected to /secondPage
.secondPage
- This page requires "authentication" (a value stored in sessionStorage
). If no such value is found, we will be redirected to welcomePage
.This repository has one Cypress test that will run Lighthouse on both of these two pages. Results are stored as html in cypress/test-results
. To run the test, execute npm run cy:open
and launch the test file.
Lighthouse is run for both pages, however, both the reports (found in cypress/test-results
) are only showing results for the welcomePage
. When the audit for secondPage
is run, the new tab is redirected to welcomePage
since no value is found in sessionStorage
. The audit is then run agains welcomePage
.
Expected behavior
I want to run Lighthouse on secondPage
.
The Cypress test performs the necessary Cypress commands to "log in" the user (which will save a value to sessionStorage
). We can see that the secondPage
is rendered properly in the original tab of Cypress viewer. When the new tab is opened to perform the Lighthouse audit, I want to see this secondPage
visible, and I want the Lighthouse report to reflect this secondPage
URL.
Instead, when the new tab is opened, we see the welcomePage
.
Environment:
Any other information that may help fixing the issue?
Two possible solutions (although I'm not sure how feasible they are):
sessionStorage
from the existing tab to the new tab.Lighthouse 6 is out: https://github.com/GoogleChrome/lighthouse
Tried to find this repo from https://www.npmjs.com/package/cypress-audit however the link is not there. My guess is this could be rectified by adding the repository
element to package.json but it might take more.
By the way, there is a broken link in the README here. Looks like there is a missing file under the docs folder.
Super useful plugin, thank you!
Seeing this error mentioned in https://githubmemory.com/repo/mfrachet/cypress-audit/issues/74
Hello,
first of all, thank you for the great plugin, we find it very useful for our project.
We added it to our synthetic monitoring tests written with Cypress; the tests are running against our production systems every 5 minutes and we're saving the speed-index score in Prometheus, so we have multiple data points to compare.
What we see is that the speed-index from the cypress-audit plugin is about significantly higher (1.5-2x in avg) of the score we get from the command-line utility (lighthouse 6.2.0) or the chrome devtools execution.
Our tests are running as cronjobs in Azure Kubernetes Cluster. What can be the reason for this difference in the score?
Thanks,
Adam
What does not work?
I'm attempting to add Lighthouse scores to our E2E job, however as soon as I add the file to our plugins, Cypress throws an error due to a missing file but I can't exactly figure out why this is happening.
**Message:** Your `pluginsFile` is invalid: `/Users/**/**/end_to_end/cypress/plugins/index.js`
It threw an error when required, check the stack trace below:
**Details:** Error: ENOENT: no such file or directory, open '/Users/**/**/end_to_end/node_modules/lighthouse/report/generator/../../flow-report/assets/standalone-flow-template.html'
at Object.openSync (node:fs:585:3)
at Object.readFileSync (node:fs:453:35)
at Object.<anonymous> (/Users/**/**/end_to_end/node_modules/lighthouse/report/generator/flow-report-assets.js:11:33)
at Module._compile (node:internal/modules/cjs/loader:1101:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
at Module.load (node:internal/modules/cjs/loader:981:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Module.require (node:internal/modules/cjs/loader:1005:19)
at require (node:internal/modules/cjs/helpers:102:18)
at Object.<anonymous> (/Users/**/**/end_to_end/node_modules/lighthouse/report/generator/report-assets.js:10:26)
at Module._compile (node:internal/modules/cjs/loader:1101:14)
at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
at Module.load (node:internal/modules/cjs/loader:981:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Module.require (node:internal/modules/cjs/loader:1005:19)
at require (node:internal/modules/cjs/helpers:102:18)
How to reproduce?
add const { lighthouse, prepareAudit } = require('@cypress-audit/lighthouse');
to /cypress/plugins/index.js
and run Cypress.
Expected behavior
What is to be expected is that the project works without errors.
Environment (please complete the following information):
12.3.1
102
9.5.2
1.3.0
Any other information that may help fixing the issue?
Add any other context about the problem here.
Hi, I did all the steps from readme and I am using plain JS but still I get the error like:
cy.task('lighthouse') failed with the following error:
> connect ECONNREFUSED 127.0.0.1:61216
my package.json:
"cypress": "^6.3.0",
"cypress-audit": "^0.3.0",
"cypress-file-upload": "^4.1.1",
"cypress-sql-server": "^1.0.0",
Maybe I am missing some necessary step.
I have
const { lighthouse, pa11y, prepareAudit } = require('cypress-audit');
and
on('before:browser:launch', (browser = {}, launchOptions) => {
prepareAudit(launchOptions);
});
and
lighthouse: lighthouse(), // calling the function is important
pa11y: pa11y(), // calling the function is important
in my plugins/index.js
And I have imported audit in support/commands:
import 'cypress-audit/commands';
Hello,
I see we can add threshold in the cypress.json in below way
"lighthouse": {
"performance": 85,
"accessibility": 80,
"best-practices": 85,
"seo": 85,
"pwa": 50
}
But is there a way that I can add configuration too in cypress.json rather than test level? like below
{"formFactor":"desktop",
"screenEmulation": {
"width": 1350,
"height": 940,
"deviceScaleRatio": 1,
"mobile": false,
"disable": false
}}
Thanks for the help
When running cypress-audit in a test as cy.lighthouse('www.google.com') the result related to performance is not matching the result of the lighthouse chrome plugin:
Test:
describe('using lightouse with cypress www.google.com', { taskTimeout: 120000 }, () => {
it('Should access www.google.com and perform lighthouse assessment', () => {
cy.visit('https://www.google.com')
cy.lighthouse({
performance: 1,
accessibility: 85,
'best-practices': 85,
//seo: 85,
//pwa: 30,
},
{
disableStorageReset: true ,
emulatedFormFactor: "desktop"
})
})
})
plugin implementation:
const { lighthouse, pa11y, prepareAudit } = require("cypress-audit");
module.exports = (on, config) => {
on("before:browser:launch", (browser = {}, launchOptions) => {
prepareAudit(launchOptions);
})
on("task", {
lighthouse: lighthouse((lighthouseReport) => {
console.log('LighthouseReport: '+lighthouseReport)
}), // calling the function is important
pa11y: pa11y((pa11yReport) => {
console.log(pa11yReport); // raw pa11y report
}), // calling the function is important
})
}
Hi,
This issue probably has been raised before here but I couldn't find any concrete workaround to fix this. I'm encountering this issue as well when I tried to integrate my Cypress test with cypress-audit plugin.
I'm getting the following error when I tried to run my tests using Cypress test runner tool or running the test in headless mode.
Here are the code snippet from the index.js file and my spec file
plugins/index.js
const del = require('del');
const fs = require('fs-extra');
const path = require('path');
const installLogsPrinter = require('cypress-terminal-report/src/installLogsPrinter');
const { lighthouse, pa11y, prepareAudit } = require('cypress-audit');
const getConfigurationByFile = (file) => {
const pathToConfigFile = `config/cypress.${file}.json`;
return !!file && fs.readJson(path.join(__dirname, '../', pathToConfigFile));
};
module.exports = (on, config) => {
installLogsPrinter(on, {
printLogsToConsole: 'onFail',
collectTestLogs: () => console.log(''),
});
on('before:browser:launch', (browser = {}, launchOptions) => {
prepareAudit(launchOptions);
return launchOptions;
});
on('after:spec', (spec, results) => {
if (results && results.video) {
// Do we have failures for any retry attempts?
const failures = results.tests.some((test) => {
return test.attempts.some((t) => t.state === 'failed');
});
if (!failures) {
// delete the video if the spec passed and no tests retried
return del(results.video);
}
}
});
on('task', {
lighthouse: lighthouse(),
pa11y: pa11y(),
});
return getConfigurationByFile(config.env.platform) || config;
};
My Spec file
describe.only('Lighthouse', () => {
it('should run performance audits', () => {
cy.visit('/');
cy.lighthouse();
});
});
I've also imported this line of code in my commands.ts file
import "cypress-audit/commands";
Cypress version : Tried both 7.7.0 and 6.9.1
Cypress-audit version : 1.0.0
Chrome version : 94
Mac OS Catalina 10.15.1
Before creating an issue, make sure to have setup the projects with the 3 steps provided in the README. You can also verify the examples folder and check if there is already an application using a similar setup to yours.
What does not work?
Report not able to be generated. Error being thrown is: "Cannot destructure property 'errors' of 'object null' as it is null," and it quotes line 41 at command-handler inside /src, where it reads: .then(({ errors, results })
How to reproduce?
Setting up a basic, essentially empty test, as follows.
describe("empty test", () => {
const retryOptions = {retries: {runMode: 0, openMode: 0}}
it('runs an empty test', retryOptions, () => {
cy.visit(*any url*)
cy.lighthouse()
})
})
index has what it's supposed to:
on('before:browser:launch', (browser, launchOptions) => {
prepareAudit(launchOptions);
return launchOptions
})
inside of on.task:
lighthouse: lighthouse(), // calling the function is important
pa11y: pa11y(), // calling the function is important
and in commands:
import "cypress-audit/commands";
Expected behavior
Audit will be generated
Environment (please complete the following information):
Any other information that may help fixing the issue?
My inclination is that this is perhaps more a problem with cypress and the way it handles port assignment, or maybe another thing I bring up a bit further down, but I'm still creating this issue in case other people are running into the same when trying to incorporate cypress-audit for the first time. Also, this is the console.log of the lighthouse object. Seems relevant that config and opts are both undefined:
{
"config": undefined
"opts": undefined
"url": "*any url*",
"thresholds": {
"performance": 100,
"accessibility": 100,
"best-practices": 100,
"seo": 100,
"pwa": 100
}
}
My last bit of context is that 'before:browser:launch' doesn't seem to be running- console.log-ing it, at least, doesn't show anything. Again, this points to cypress being the problem, but I'm just curious if this has become an issue for anyone else.
HI there according to PA11y documentation when running in CI there a config to pass so builds dont fail .
Ive got the following in my cypress script but seems to do nothing . What I am after to to perform the accessibility scan generate the reports but not fail the test if the violations are within the threshold . Currently with this config set I have 9 violations and the tests are failing instead of passing.
cy.pa11y({
runners: ['axe'],
threshold: 10,
standard: 'WCAG2AA'
});
Please is there a way we can do this with PA11y integrated with cypress
Line 13 in b8cce1e
When triggering lighthouse in my test... my mocked apis defined in cypress seems completely ignored in favor of 404 errors. The report is generated but I can see in the tab opened for lighthouse that something is wrong since I see error popups. My "fake login information" in the session storage seems to be passed along the lighthouse tab tho.
Note that all my apis are mocked so any ideas on how to resolve this?
Currently running lighthouse in a firefox test will fail gracefully with a simple log message, but Pa11y will fail the test with an error.
The behavior should be consistant I think:
Currently, I don't think there is a way for now, to interact with the page before it starts the audit
.
How can we add this functionality to cypress-audit?
I'm running pa11y on a page that contains an issue, but issue reported doesn't contain any details:
cy.pa11y - 1 accessibility violation was found
Issue: WCAG2A.Principle3.Guideline3_2.3_2_2.H32.2, # of occurrences: 1.
- undefined
- undefined
It looks like it's trying to output issue.description
and issue.link
when pa11y actually provides issue.context
, issue.message
and issue.selector
:
https://github.com/pa11y/pa11y#javascript-interface
Before creating an issue, make sure to have setup the projects with the 3 steps provided in the README. You can also verify the examples folder and check if there is already an application using a similar setup to yours.
What does not work?
A clear and concise description of what the bug is.
I have a very simple test here running against Google website.
it('test', () => {
cy.visit('https://www.google.com')
cy.lighthouse(
{
performance: 60,
accessibility: 90,
'best-practices': 80,
seo: 80,
},
{
formFactor: 'desktop',
screenEmulation: {
mobile: false,
disable: false,
width: 1350,
height: 940,
deviceScaleRatio: 1,
},
output: "html",
throttling: {
rttMs: 40,
throughputKbps: 11024,
cpuSlowdownMultiplier: 1,
requestLatencyMs: 0,
downloadThroughputKbps: 0,
uploadThroughputKbps: 0,
},
throttlingMethod: "simulate",
},
)
});
However, the report presented this saying the use of HTTP/1.1 while Cypress opened the new browser tab.
How to reproduce?
Steps to reproduce the behavior. A Github repository reproducing the error is very welcome.
As mentioned above by running Cypress Open Runner or Cypress headless
Expected behavior
A clear and concise description of what you expected to happen.
The report should show the HTTP/2 instead of HTTP/1.1
Screenshots / Animated Gifs
Add screenshots or screen recordings to help explain your problem.
Environment (please complete the following information):
Any other information that may help fixing the issue?
Add any other context about the problem here.
I am not really sure if this is the bug. However, I tried to look at many threads but I can't figure out this.
Using cypress to navigate though an app logging in or completing a form when a test hits cy.lighthouse a new tab is opened with a new session which causes the app to navigate back to the start page (due to security restrictions/anti forgery if a customer hits a page in the journey without a valid session they get sent to the "home page"), light house then runs against the home page rather than the required page, is there a way to run the audit against the page that cypress already has open?
Is the intended behaviour to open a new tab, if so what is the best way to handle session storage not being carried across to the new tab?
Hello 👋 I'm writing on behalf of the Cypress DX team. We wanted to notify you that some changes may need to be made to this plugin to ensure that it works in Cypress 10.
For more information, here is our official plugin update guide.
The docs say that we should be able to pass pa11y options to cy.pa11y()
, but the "standard" flag doesn't have any effect, and trying to set "runners" causes Cypress to crash. I believe this line might be responsible: https://github.com/mfrachet/cypress-audit/blob/master/src/a11y/task.js#L10
pa11yLib(url, { browser, runners: ["axe"], ...opts }).then((results) => {
Cypress 9.4.1
cypress-audit 1.1.0
Tell me how to save the test results in the log? I am trying to do like this:
it('Pa11y валидация', () => {
cy.viewport(1920, 1080)
cy.visit(pageUrl)
cy.reload(true)
cy.pa11y(pageUrl, {
"threshold": 9999
}).then((results) => {
let pa11yLog = JSON.stringify(results)
cy.writeFile('cypress/pa11y/'+fileName+'pa11y-report.json', pa11yLog)
});
cy.log(
`[${pageUrl}] Pa11y валидация - Done`
);
})
One setting I would love to know if it exists is a failure threshold: if less than the threshold, then succeed, othewise fail test.
Before creating an issue, make sure to have setup the projects with the 3 steps provided in the README. You can also verify the examples folder and check if there is already an application using a similar setup to yours.
What does not work?
If the config
in the module.exports
within plugins/index.ts
is typed (appropriately as Cypress.PluginConfig
, then TypeScript will give the following error:
Type '{}' is missing the following properties from type 'Browser': name, family, channel, displayName, and 5 more.
How to reproduce?
npm install --save-dev typescript cypress
).tsconfig.json
file to scan your Cypress files (most easily done by using "include": ["**/*.*"]
plugins/index.ts
file as specified in README.on
and config
variables as Cypress.PluginEvents
and Cypress.PluginConfigOptions
, respectivelytsc
or npx tsc
from your project's rootExpected behavior
browser
variable is correctly typed (or rather, there is documentation on how to correctly type this variable)
Environment (please complete the following information):
Any other information that may help fixing the issue?
I understand that this library does not explicitly support TypeScript, but at the very least it shouldn't cause any TS errors when attempting to use it with TS. Really just any documentation on how to type out that browser
object would probably solve the problem.
Additionally, type is not given for launchOptions
. This can be resolved by just declaring an any
type (sort of defeats the purpose of TS though 😂 ).
I tried setting browser = Cypress.browser
but I receive an INVALID_URL
error when running the test.
Any guidance would be greatly appreciated
Hi! First of all, thank you so much for this plugin. Been playing around with it and it's really great 👍
I noticed though that when I invoke cy.lighthouse()
, by default this runs the lighthouse test on mobile view. Please see attached image as reference
I tried passing in the lighthouse config but it didn't work
const thresholds = {
"performance": 50,
"accessibility": 80,
"best-practices": 50,
"seo": 60,
"pwa": 50
};
const lighthouseConfig = {
"ci": {
"collect": {
"settings": {
"chromeFlags": '--no-sandbox',
"emulatedFormFactor": 'desktop'
}
}
}
};
cy.lighthouse(thresholds, lighthouseConfig);
Any pointers on how to do it on desktop would be appreciated :) thank you!
I'm attempting to use audit to test out a11y and performance issues via cypress component testing, but as far as I can tell this currently isn't possible. With @cypress/react, instead of visiting a url you just mount a component which gets loaded into the browser. When trying to run cy.lighthouse on this mounted component, the blank page appears in the browser for about 10 seconds before all of the tests fail with 0 out of 100 results.
Is there anyway to have lighthouse run on these mounted components?
cy.task('lighthouse') failed with the following error:
> You probably have multiple tabs open to the same origin.
node_modules/cypress-audit/src/performances/commands.js:34:1
32 |
33 | cy.log("-------- cy.lighthouse --------");
> 34 | cy.task("lighthouse", {
| ^
35 | url,
36 | thresholds: thresholds || configThresholds || defaultThresholds,
37 | opts,
cypress: 6.0.1
cypress-audit: 0.2.3
Repo: https://github.com/katawaredev/permisisve-nextjs-template
(note: some of the packages there are not available in npm, if you need to run it I'll tweak it)
Reproducing:
"scripts": {
"prod": "run-s build start",
"test:e2e": "start-test prod 3000 'TZ=UTC cypress run'",
"e2e": "start-test prod 3000 'TZ=UTC cypress open'"
},
Running test:e2e
(headless mode) does not produce errors, but running e2e
(UI) does
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.