uktrade / cypress-image-diff Goto Github PK
View Code? Open in Web Editor NEWVisual regression test with cypress
License: MIT License
Visual regression test with cypress
License: MIT License
Rather than manually update the baselines, it would be nice to have an option to run tests and update baselines if we are sure that the new comparison images are correct.
Please, let me know if I can configure density, threshold, mask/exclusion areas, etc?
Thanks
Jeff
It was working perfectly fine with cypress version 8.7.0 and this package with version 1.14.4. I upgraded cypress to 9.0.0 and then I started to get the following error in the test runner.
Also, I upgraded "cypress-image-diff-js": "^1.15.5", but the error is still same.
The error:
The following error originated from your test code, not from Cypress.
> Expected viewportHeight to be a number. Instead the value was: "1440"
When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.
Cypress could not associate this error to any specific test.
& the cypress.json:
"viewportWidth": 1440,
"viewportHeight": 900,
Hello,
I've tried using your tool in Mocha Cypress and it did work and I liked the results and simplicity of use.
We use Cypress with Typescript and BDD/Cucumber. The I see run-time errors which I don't know how to fix.
my project structure looks like this
Keep in mind that because of Cucumber screenshot were generated but with specific name. Then I saw an error like this
Hi,
Is there currently a way to retrieve the image difference percentage ? I have my mochawesome reporter and I would like to display the percentage, but I am not finding a way to retrieve the return from the compareSnapshotCommand.
When I collect snapshots using cypress run
, the screenshots collected are collected at a default pixel density. However, when I collect snapshots using cypress open
on my MacBook, the screenshots collected are at double pixel density.
Is there a way to standardize this? Otherwise, when I run my tests using cypress open
they always fail because the comparison images are twice as wide as the baseline images that were generated with cypress run
.
Hello!
I have viewport sizes set in my cypress.json file, and as the Title says, even though I have them defined, this plugin overrides them and makes them bigger than I want them to.
This Issue is rather annoying, especially because there is no cross-platform way to set the Environment Variables and npm cannot find the "HEIGHT" command because it gets wrongly interpretet.
If possible I'd suggest checking the cypress.config before to see if there's things defined in it
It looks like screenshots are based on "scrolling the page", which doesn't work for pages with sticky
and fixed
CSS positioning, leading to hilariously wrong screenshots like this one:
Is there any way to tell the screenshooter to not scroll, but use the browser's built in "take a full page screenshot" function?
Hello,
Once I've started to use your tool it looks like it suppresses failed screenshot generation. Is it a way to still keep failed screenshots to be generated?
Thanks
Jeff
Hi there,
First of all, thank you for making this fantastic plugin!
Just a small question, I saw Cypress is in the dependency list in package.json
and the version is specified.
I'd like to ask is there a plan to move it to peerDependencies
, so users don't need to install two different Cypress versions (v10 and v9 in my case).
Thanks again!
Hi!
First of all, thx for this project! π
I am using Cypress version 7 (7.6.0
) and everytime I run npm install
two versions of cypress will be installed - 7.6.0
and 6.9.1
. I think that this line is the issue - you are referencing "cypress": "^6.1.0"
at package.json
.
Do you plan on updating this to "cypress": "^7.0.0"
(or remove the hard dep. at all)?
It seems that the project is working correctly with version 7 (I run cypress cache prune
after npm install
to only have version 7.6.0
installed).
Another thing - I am using 1.10.3
of this library, because the latest release including npm-force-resolutions
broke my project (fresh npm ci
brings an error package-lock.json
not found) - and I would be happy if you would not use that at all. Can you remove it - or why is it required?
Thank you in advance!
Both screenshot and report folders are hardcoded. There might be use cases where we would like to have these folder creation configurable to our needs.
I am not sure if this is intentional or not but as I was working with this package I noticed that if I run all my tests through the GUI from cypress open
by clicking run all integration tests, my screenshot file names get a hardcoded prefix added to them as seen in the attached screenshot All Integration Specs
. Is this an intention? Seems if I were to have CI errors and want to rerun locally through cypress open
to debug it would be best to standardize the file naming convention no matter how the test is run especially since running all tests through cypress run
follows the same file naming convention as running a singular suite through cypress open. Granted I know I'd most likely just run a singular suite to debug but still feel the naming convention should stay the same.
The intent of this enhancement is to hide elements that are dynamically rendered based on third party services or timestamp, causing visual regression tests to be flaky or increasing complexity to deal with these scenarios.
It is useful to make image inline if I runing testing in a CI. Then I just download the html file from artifacts without downloading all other files one by one.
I was wondering if there is a way to compare elements of a page that need to be fetched at runtime.
Currently I've experiencing the problem that the test immediately stops running after one of those elements differs from the previous screenshot. However I would like it to keep running the test, even if it partially fails, so that all screenshots are generated.
Example:
git clone https://github.com/zwarag/poc-differ
cd poc-differ
yarn install
npx cypress run
npx cypress run
Expected behaviour:
β
The first npx cypress run
creates 26 screenshots in poc-differ/cypress-visual-screenshots/baseline
β The second npx cypress run
creates 26 screenshots in poc-differ/cypress-visual-screenshots/diff
Current behaviour:
The first npx cypress run
creates 26 screenshots in poc-differ/cypress-visual-screenshots/baseline
The second npx cypress run
creates 1 screenshots in poc-differ/cypress-visual-screenshots/diff
Hi All,
I'm using the visual tool along with cypress assertions/validations, and I've noticed 2 things:
Please let me know if its an issue or I'm doing something wrong :)
1:
During my test it will execute some actions then assert, then visual-snapshot.
But if the assert failed - it should take a screenshot and store in folder, but when using the visual plugin - it doesnt.
I think its due to deleteScreenshot Task - it just deletes all screenshots instead of its own created one (I could be wrong)
When I disable getCompareSnapshotsPlugin(on, config)
the failed cypress screenshots return (but now visual testing wont work).
Outcome with getCompareSnapshotsPlugin:
Outcome without getCompareSnapshotsPlugin:
2:
When setup correctly and loading Cypress open (version 10.4.0), it decides to create a random base image, before anything is executed and I dont know why this is happening - I assume it should only generate base image during test execution.
Issue:
point 2 no longer an issue - turns out the image remained in comparison folder. Removing it, no longer generated a new one.
Let me know if more info is needed & thanks in advance
is there support or upgrade needed for this to work with cypress 10? Does anyone have an experience with this with cypress 10?
It would be nice to know how much of the code is being covered by tests so we could improve it as we go.
Is there a way to have this with this plugin? I've seen it with other approaches, and it seems really visually nice. Sometimes only the diff can be pretty difficult to read. Kinda like this:
Vs something like this (this example is probably the simplest using a single file and from cypress-image-snapshot plugin to be clear, but there are potentially a lot of ways to accomplish this I think)
The CSS :focus-visible state style on a button is sometimes applied and sometimes not when calling compareSnapshot. Which makes my test fail.
I tried with the cy.screenshot and seems to be always applied. Not sure what could cause this difference between
This plugin is only taking a screenshot of about a 1000px square section of the top left of the screen, which is a small portion of this long form I'm trying out (needs about 4000px height to render all of it without a scrollbar). Has anyone else noticed this? I'm setting the viewport like this and when I run the test, it displays with this viewport in the browser, but the snapshot command always only gets a fraction. I've also tried cy.compareSnapshot and cy.root().compareSnapshot... Is it obvious I'm doing something wrong? I'm doing this in a component test, btw.
Hi all,
i am new to cypress and this package and want to use it for ui testing. So far, the image-diff reacts correct, when i have a complete different page. But when i have same page, but only a different button it says failure treshhold 0.0.
Can the tool detect even those small changes?
And what information do you need to support in here?
Hint: The elements are inside an iFrame. Could that be an issue?
As titled thanks.
Would be nice, if this project had some templates for issue types
Hi,
I just got the following error after executing the basic test with this plug-in.
Script
describe('Visuals', () => { it('should compare screenshot of the entire page', () => { cy.visit('www.google.com') cy.compareSnapshot('home-page') }) })
Error -
`CypressError
cy.task('rp_screenshot') failed with the following error:
The task 'rp_screenshot' was not handled in the plugins file. The following tasks are registered: compareSnapshotsPlugin, copyScreenshot, generateReport
Fix this in your plugins file here:
C:\Users\l.wijenayake\OneDrive - ISM eGroup\Workspace\Cypress SCC\Cypress_Test_Automation_React\cypress\plugins\index.jsLearn more`
From the list of Cypress 10.0.0 breaking changes:
Generated screenshots and videos will still be created inside their respective folders (screenshotsFolder, videosFolder). However, the paths of generated files inside those folders will be stripped of any common ancestor paths shared between spec files found by the specPattern option and may no longer be deterministic. Addressed in #19319.
Steps to reproduce:
npm install
;cypress open
and migrate config files;npm run test:e2e
;folder-structure.cy-wholePage.png
is generated outside of folder-structure-test
folder.The majority of my false positives could be solved by adding some retry logic to the plugin. It would work something like this:
--update
cli arg passed, don't retry (nothing to compare).Ideally, T and N would be configurable, with default values of 1000 and 3, respectively.
This feature would also be somewhat idiomatic since Cypress has retry logic built into most assertions.
Is there a way to skip Error: Image difference greater than threshold
errors with this plugin?
The cypress-image-snapshot plugin provides this option.
https://github.com/jaredpalmer/cypress-image-snapshot
Thank you in advance.
Hi there,
First of all, thank you for making this fantastic plugin!
My issue is when compareSnapshot takes a snapshot of an element it takes a snapshot of the whole screen and then the image that is provided is cropped which only includes the element. I was expecting it to only capture an image of that element.
As I have a use case where I have different environments which have different functionality applied. My workaround is that I can include logic which considers a threshold depending on the environment however shouldn't the snapshot only capture an image of that element?
Example code
export function validateScreenshot(imageName: string, screenName: string) {
let url = window.location.href;
let threshold = url.includes("localhost") ? 0 : 0.16925600064703697;
cy.get(ScreenTestId[screenName]).compareSnapshot(imageName, threshold);
cy.task("generateReport");
}
No need to update the base image files for all tests.In actual scenario, just need to re-generate baseline image for specific test case
Would be nice to have the dependencies also for cypress ^8.5.0.
Now our pipe tries always install the new cypress version what causes some errors.
"^8.5.0 || ^9.0.0" would be fine.
Error: dest already exists.
at doRename (/test-app/node_modules/cypress-image-diff-js/node_modules/fs-extra/lib/move-sync/move-sync.js:25:34)
at Object.moveSync (/test-app/node_modules/cypress-image-diff-js/node_modules/fs-extra/lib/move-sync/move-sync.js:17:10)
at renameAndMoveFile (/test-app/node_modules/cypress-image-diff-js/dist/utils.js:60:23)
at Object.handler (/test-app/node_modules/cypress-image-diff-js/dist/plugin.js:138:34)
at invoke (/root/.cache/Cypress/9.1.0/Cypress/resources/app/packages/server/lib/plugins/child/run_plugins.js:22:16)
at /root/.cache/Cypress/9.1.0/Cypress/resources/app/packages/server/lib/plugins/util.js:45:14
at tryCatcher (/root/.cache/Cypress/9.1.0/Cypress/resources/app/packages/server/node_modules/bluebird/js/release/util.js:16:23)
at Function.Promise.attempt.Promise.try (/root/.cache/Cypress/9.1.0/Cypress/resources/app/packages/server/node_modules/bluebird/js/release/method.js:39:29)
at Object.wrapChildPromise (/root/.cache/Cypress/9.1.0/Cypress/resources/app/packages/server/lib/plugins/util.js:44:23)
at wrapChildPromise (/root/.cache/Cypress/9.1.0/Cypress/resources/app/packages/server/lib/plugins/child/run_plugins.js:112:10)
at execute (/root/.cache/Cypress/9.1.0/Cypress/resources/app/packages/server/lib/plugins/child/run_plugins.js:125:14)
at EventEmitter.<anonymous> (/root/.cache/Cypress/9.1.0/Cypress/resources/app/packages/server/lib/plugins/child/run_plugins.js:202:5)
at EventEmitter.emit (events.js:376:20)
at process.<anonymous> (/root/.cache/Cypress/9.1.0/Cypress/resources/app/packages/server/lib/plugins/util.js:19:22)
at process.emit (events.js:376:20)
at process.emit (/root/.cache/Cypress/9.1.0/Cypress/resources/app/node_modules/source-map-support/source-map-support.js:495:21)
at emit (internal/child_process.js:910:12)
at processTicksAndRejections (internal/process/task_queues.js:83:21)
cypress-image-diff-js: 1.15.8
cypress: 9.1.0
node 16.5.0
Error: ENOENT: no such file or directory, copyfile 'C:\repos\cypress\screenshots\503.spec.ts\503.spec.ts-#main article.png' -> 'C:\repos\cypress-visual-screenshots\comparison\503.spec.ts-#main > article.png'
The > char is a bad character in Windows filesystem.
Hi folks,
I'm trying to use the library to run image comparison tests, but the screenshots I'm getting are rather weirdly cropped.
For example, this is what my page looks normally in the browser:
But the snapshot taken by your library looks like this:
As you can see, the viewport is very narrow and weirdly cropped. I'm running the tests headless.
Tried Chrome and Firefox, no difference. I am settings the viewport manually in my tests:
cy.viewport(1920, 1080)
but it doesn't seem to have any effect.
I'd appreciate any help with this.
Thanks.
Hello. I tried to setup our project and added this plugin that uses baseline for visual testing. On my first run, instead of getting an image on my baseline
folder, I'm getting this error. Did I miss a step? I have the folder in our repository, same path as to which the error is stating.
cypress.config.js
const getCompareSnapshotsPlugin = require('cypress-image-diff-js/dist/plugin')
module.exports = defineConfig({
e2e: {
async setupNodeEvents(on, config) {
getCompareSnapshotsPlugin(on, config);
support/command.js
const compareSnapshotCommand = require('cypress-image-diff-js/dist/command')
compareSnapshotCommand()
support/e2e.js
require('./commands')
Script:
assertHeaderAndFooterSnapshots(headingName){
this.getHeadingHeader(headingName).eq(0)
.scrollIntoView()
.compareSnapshot('try', 0.1)
.should('exist')
this.getHeadingFooter(headingName).eq(0)
.scrollIntoView()
.compareSnapshot('try', 0.1)
.should('exist')
}
}
I also tried the one below but same result:
```
assertHeaderAndFooterSnapshots(headingName){
this.getHeadingHeader(headingName).eq(0)
.scrollIntoView()
.should('exist')
cy.compareSnapshot('try', 0.1)
this.getHeadingFooter(headingName).eq(0)
.scrollIntoView()
.should('exist')
cy.compareSnapshot('try', 0.1)
}
Is it possible to get snapshot testing working for component testing? on the install documents, it is mentioned that where {scheme} defaults to e2e
// cypress/support/index.js for Cypress versions below 10
// cypress/support/{scheme}.js for Cypress versions 10 and above, where {scheme} defaults to e2e
require('./commands')
is it possible to overwrite this {scheme} so it looks into component.js file for component testing as well?
When I configure the framework with component testing this is the error I am getting.
I tried to add compareSnapshotCommand() as a
Cypress.Commands.add('compareSnapshotCommand', compareSnapshotCommand());
but that did not work either.
I'm a new user. I don't see any report once the screenshots are generated including a diff.
The directory cypress-visual-report is created along with cypress-visual-screenshot structure that includes baseline, comparison and diff properly. However I have a diff screenshot generated, cypress-visual-report is left empty.
Could you advise and document that feature as I can read it was merged previously (reporting) ?
--> version 1.20.0
Thanks in advance for your kind reply
Hello, please check out following issue that I am facing:
These are my dependencies:
"auth0-js": "^9.18.0",
"chai": "^4.3.4",
"chai-exclude": "^2.1.0",
"cypress": "^9.6.0",
"cypress-image-diff-js": "^1.18.1",
"eslint": "^8.7.0",
"fs": "^0.0.1-security",
"mocha": "^10.0.0",
"mochawesome": "^7.1.3",
"prettier": "^1.19.1"
I have 0 vulnerabilities
But running my visual tests I get error: cy.compareSnapshot is not a function
Simple test:
it('Verify that Login page is loading correctly', () => {
cy.visit('/');
cy.wait(5000);
cy.compareSnapshot(${envName}-${Cypress.currentTest.title}
);
});
I have follow the steps not able to find this function.. Please help.
The error is saying property compareSnapshot does not exist on type cy.
// cypress/support/commands.js
const compareSnapshotCommand = require('cypress-image-diff-js/dist/command')
compareSnapshotCommand()
// cypress/support/index.js
require('./commands')
//plugins/indes.js
const cucumber = require('cypress-cucumber-preprocessor').default
const browserify = require('@cypress/browserify-preprocessor');
const resolve = require('resolve');
const getCompareSnapshotsPlugin = require('cypress-image-diff-js/dist/plugin')
module.exports = (on, config) => {
const options = {
...browserify.defaultOptions,
typescript: resolve.sync('typescript', { baseDir: config.projectRoot }),
};
on('file:preprocessor', cucumber(options));
getCompareSnapshotsPlugin(on, config)
};
What is wrong with my settings? Thanks.
Sometimes when I rerun my tests I get this error:
EPERM: operation not permitted, stat 'Z:\nidatracker\apps\nida-divi-e2e\cypress-visual-screenshots\comparison\section1.spec.ts-section1AllDataFilled.png'
at renameAndMoveFile (Z:\nidatracker\node_modules\cypress-image-diff-js\dist\utils.js:60:23)
at Object.handler (Z:\nidatracker\node_modules\cypress-image-diff-js\dist\plugin.js:170:34)
plugin.js (the second last line failed):
on('after:screenshot', function (details) {
// Change screenshots file permission so it can be moved from drive to drive
(0, _utils.setFilePermission)(details.path, 511);
(0, _utils.renameAndMoveFile)(details.path, _config["default"].image.comparison(details.name));
});
utils.js (the second line failed):
var renameAndMoveFile = function renameAndMoveFile(originalFilePath, newFilePath) {
_fsExtra["default"].moveSync(originalFilePath, newFilePath, {
overwrite: true
});
};
Has anybody faced this issue?
After some research, it seems like setting the permission is what causes this problem.
Because when an image already exists in cypress-visual-screenshots/comparison
a permission gets applied and it cannot be overwritten.
The workaround would be deleting the images in cypress-visual-screenshots/comparison
after each test...
I have cypress.json setting viewport as following:
{ "viewportWidth": 1440, "viewportHeight": 900 }
However screenshots taken are:
1280x768
1280x795
1280x1018
...
I was expecting to deal with width in cypress.json, or even to scroll horizontally.
Cypress version: 8.7.0
cypress-image-diff: 1.16.1
Running on MAC 12.1
Hi,
What's the intended way to use it in TS env?
VSCode warns Property 'compareSnapshot' does not exist on type 'Chainable<JQuery<HTMLCanvasElement>>'.ts(2339)
Though it is only a warn and runs well without types, it's annoyed to see everywhere.
I have to add types currently, follow here: https://docs.cypress.io/guides/tooling/typescript-support#Types-for-custom-commands
main
branch failed. π¨I recommend you give this issue a high priority, so other packages depending on you could benefit from your bug fixes and new features.
You can find below the list of errors reported by semantic-release. Each one of them has to be resolved in order to automatically publish your package. Iβm sure you can resolve this πͺ.
Errors are usually caused by a misconfiguration or an authentication problem. With each error reported below you will find explanation and guidance to help you to resolve it.
Once all the errors are resolved, semantic-release will release your package the next time you push a commit to the main
branch. You can also manually restart the failed CI job that runs semantic-release.
If you are not sure how to resolve this, here is some links that can help you:
If those donβt help, or if this issue is reporting something you think isnβt right, you can always ask the humans behind semantic-release.
An npm token must be created and set in the NPM_TOKEN
environment variable on your CI environment.
Please make sure to create an npm token and to set it in the NPM_TOKEN
environment variable on your CI environment. The token must allow to publish to the registry https://registry.npmjs.org/
.
Good luck with your project β¨
Your semantic-release bot π¦π
I use [email protected]
and [email protected]
.
I have a simple integration test in landing-page.spec.ts
:
describe('The landing page', () => {
it('is possible to make visual regression tests', () => {
cy.visit('/');
cy.get('.container').compareSnapshot('landing-page', 0.1);
});
});
If I start cypress via cypress run
I get a screenshot with the name landing-page.spec.ts-landing-page.png
.
If I start cypress via cypress open
and click on Run 1 integration spec
the resulting screenshot name is All Integration Specs-landing-page.png
.
I guess the relevant code is here
const specName = Cypress.spec.name
const testName = `${specName.replace('.js', '')}-${name}`
There are two issues:
Cypress.spec.name
doesn't seem to be the correct way to get the filename of the test.js
There also seems to be Cypress.currentTest.titlePath
.
The solution from cypress-plugin-snapshots seems to also not rely on the file name of the tests, but the - No it seems to have the same problem.describe
blocks?
function getTestTitle(test) {
return (test.parent && test.parent.title ? `${getTestTitle(test.parent)} > ` : '') + test.title;
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.