Giter Club home page Giter Club logo

magento2-cypress-testing-suite's Introduction

elgentos/magento2-cypress-testing-suite

A community-driven Cypress testing suite for Magento 2

This testing suite was announced in Peter Jaap his talk about Cypress testing in Magento 2 at Reacticon 4. Watch the talk for more information about Cypress testing.

image

A successful Cypress run which tests the store at https://hyva-demo.elgentos.io

image

homepage.spec.js tests running on https://hyva-demo.elgentos.io

You can see all runs of this testing suite in the public Cypress Dashboard for this project. See for example this succesfull run.

Table of Contents

Prerequisites

  • Magento 2.3.x / 2.4.x
  • Hyvä (1.1.17) or Luma-based theme
  • npm
  • An admin bearer token (see Setup)

Assumptions & limitations

Assumptions

  • Magento 2 runs in Single Store Mode

  • Default language is English

  • Viewport is 1920x1280, with support for mobile viewports

  • Sample Data is installed

  • Multi Source Inventory is not used

Not plug & play

This test suite is not plug & play for your store. A number of tests rely on Magento's default sample data. These tests will fail when you don't have the sample data. It is up to you to change the fixtures/selectors/tests to make them pass for your store.

Open source-only right now

We don't do Commerce builds over at elgentos, so we haven't spent time creating tests for Commerce-only functionality.

No 100% test coverage

We do not particularly strive for 100% test coverage. We have identified the most common and most revenue-dependent scenarios. For example, we do test viewing products, filtering categories, adding products to the cart etc, but we (currently) do not test the Email a Friend or Compare Products feature since these are rarely used in an average Magento store. We are perfectly willing to merge a PR with these tests of course.

No extensibility / inheritance of tests

You need to copy the whole suite into your project. We are open to suggestions on how to solve this, see Discussions.

The Hyvä checkout tests assume the React Checkout is used

The Hyvä checkout tests assume the Hyvä React Checkout. To skip checkout related Hyvä tests, set the environment variable CYPRESS_MAGENTO2_SKIP_CHECKOUT.
PR's are definitely welcome to improve the checkout related tests.

Progress

We are at 80%; 69 out of the proposed 86 tests are done.

Wording & naming are subject to change.

Spec file Group Test
account.spec.js Account test creation ✔️ it can create an account to log in with
Account activities ✔️ it creates an account to log in with and use for further testing
✔️ it can log out
✔️ it can show the account information page and display the name of the customer
✔️ it can change the password
✔️ it can change the name of the customer on the account information page
✔️ it can navigate to all customer account pages and displays the correct titles
✔️ it can navigate to order history and displays that there are no placed orders
✔️ it can add a new address
✔️ it can change an existing address
✔️ it can remove an address
✔️ it subscribe through the newsletter subscription page
✔️ it can add an address automatically when an order is placed
✔️ it can add a product to the wishlist of the logged in customer on a productpage and store it
✔️ it can edit the wishlist on the wishlist page
🔲 it can reset the password when it is forgotten
Do not login before these tests ✔️ it can login from cart without making changes to the cart
✔️ it can login from checkout
product-searches.spec.js Perform searches ✔️ it can perform search with multiple hits
✔️ it can find a single product
✔️ it can show a page for no search results when the searchterm cannot give any results
✔️ it can show suggestions when entering search terms
category.spec.js Category page tests ✔️ it can navigate to the category page and filter products on the color red
✔️ it can sort the products on price from lowest to highest
✔️ it can change the number of products to be displayed
✔️ it checks if the breadcrumb is displayed correctly
✔️ it checks if the pagination is working
✔️ it can switch from list to grid view
homepage.spec.js Home page tests ✔️ it can navigate to the homepage
✔️ it can perform search from homepage
✔️ it can open a category
✔️ it can show the header correctly and all links work
✔️ it can show the footer correctly and all links work
🔲 it can show the main section of the homepage correctly and all links work
✔️ it can subscribe to the newsletter
✔️ it can add products shown on the homepage to the cart when an add to cart button is present
🔲 it shows the cookie banner when cookies are not accepted yet (Vanilla Hyvä shows no cookie banner)
checkout.spec.js Checkout tests (guest) 🔲 it shows the correct products and quantities previously added to the cart
✔️ it shows correct prices like subtotal, VAT, shipping costs and total
✔️ it can see coupon discount in checkout
🔲 it can properly choose and use all listed payment methods
🔲 it can properly choose and use all listed shipping methods
🔲 it a conformation mail is send to the customer after placing the order
🔲 it an invoice is created and sent to the customer after placing the order
✔️ it after placing the order it is properly processed in the backend'
Checkout tests (logged in) 🔲 it can automatically fill in the name and address of the customer
✔️ it can find and order in the customer order history after having placed an order
cart.spec.js Cart tests ✔️ it can add a product to the cart
✔️ it can change the quantity in the cart
✔️ it can remove a product from the cart
✔️ it can add a coupon to the cart
✔️ it can delete an added coupon from the cart
✔️ it cannot add a non existing coupon
✔️ it displays the correct product prices and totals
🔲 it merges an already existing cart when a customer logs in
minicart.spec.js Minicart tests ✔️ it can open the cart slider by clicking on the cart icon in the header
✔️ it checks if the items and prices in the slider are displayed correctly
✔️ it can delete an item in the cart slider
✔️ it can change the quantity of an item in the cart slider
✔️ it can navigate to the cart with a link in the slider
✔️ it can navigate to the checkout with a link in the slider
product-page.spec.js Product page tests ✔️ it can display the title and image of the product
✔️ it shows the product price
✔️ it can configure the product when it is an configurable product
✔️ it can add the product to the cart
✔️ it can't add the product to the cart if it is a configurable product and no options are selected
✔️ it can add the product to the wishlist
✔️ it shows the correct breadcrumb
✔️ it can show reviews made by logged in customers
✔️ it can add a review when logged in
✔️ it can indicate if a product is in stock
✔️ it can't add a product to the cart when the product is out of stock (commented out, needs admin token in the cypress.env.json)
Bundle products test ✔️ it can render the product name
✔️ it can set the price to zero when every associated product qty is zero
✔️ it can calculate the price based on selected options
✔️ it can display selection quantities
✔️ it can add a bundled product to the cart
cms-page.spec.js CMS page tests 🔲 it shows the default 404 page on an non-existent route
🔲 it can open the default CMS page correctly
contact-form.spec.js Contact form tests ✔️ it shows the contact form correctly
🔲 it cannot submit a form when no valid email address is entered
✔️ it can submit the form when all validation passes
back-end.spec.js Back-end tests 🔲 it can login on the administration panel of the magento environment
🔲 it can show customer data
🔲 it processes orders and invoices correctly
🔲 it can edit an order

Installation

First, install Cypress and the dependencies in the root of your Magento 2 project:

npm ci

The easiest way to install the tests is to clone this repository and move the cypress folder into your project. As of right now, we do not provide a fallback mechanism for customizations to the tests, see Limitations.

git clone [email protected]:elgentos/magento2-cypress-testing-suite.git
mv magento2-cypress-testing-suite/cypress .
mv magento2-cypress-testing-suite/cypress.config.js .
rm -rf magento2-cypress-testing-suite
npm install cypress@^12.2.0 cypress-localstorage-commands@^2.2.2 cypress-tags@^1.1.2 typescript@^4.8.3

If you only need the Hyvä tests:

rm -rf cypress/{fixtures,page-objects,integration}/luma
rm -rf cypress/{integration}/luma-accessibility

If you only need the Luma tests;

rm -rf cypress/{fixtures,page-objects,integration}/hyva

Then edit the cypress.config.js file in the root of your project to update your baseUrl, the projectId and possibly some other defaults we've set:

const baseUrl =
  process.env.NODE_ENV === "develop"
    ? "http://cypress.magento2.localhost"
    : "https://example.com/";
    projectId: "8vuidn"

Also add these lines to your .gitignore to avoid cluttering your Git repo;

node_modules
cypress/screenshots
cypress/videos
cypress.env.json

Accessibility test installation

The accessibility test specs use cypress-axe and as such need extra items installed via npm.

npm install --save-dev cypress-axe
npm install --save-dev axe-core

Setup

Some tests are dependent on making changes in the database. This is done through the Magento 2 REST API. You will need to create an admin token for these tests. This is easily done using magerun2.

Get a list of all the admin users: magerun2 admin:user:list

If you don't have any admin users, create one; magerun2 admin:user:create --admin-user=john --admin-firstname=John --admin-lastname=Doe --admin-password=JohnDoe123 --admin-email="[email protected]"

Then create a token: magerun2 admin:token:create username_goes_here

You then can add the token to cypress.env.json as an environment variable:

{
  "MAGENTO2_ADMIN_TOKEN": "token_goes_here"
}

Overwrite the base url

{
  "MAGENTO2_BASE_URL": "https://demoshops.splendid-internet.de/magento/demoshop-magento2-daily"
}

Alternatively you can set it in your CI/CD variables by prefixing the environment variable name with CYPRESS_: CYPRESS_MAGENTO2_ADMIN_TOKEN: <token_goes_here>.

Tags

We use tags to discern between hot tests and cold tests. If you followed the installation instructions above the cypress-tags module is already installed.
Note: it is used in the cypress.config.js file in the setupNodeEvents callback.

Running

npx cypress run
# npx cypress open # if you want to use the GUI

Individual specs can be run using the following command:

npx cypress run --spec ./cypress/integration/path/to/some.spec.js

Hot and cold tests based on tags can be run using the following command:

CYPRESS_INCLUDE_TAGS=hot npx cypress run

Running against local environment

Set up your local base URL in cypress.config.js, or export CYPRESS_MAGENTO2_BASE_URL.
Then run Cypress with NODE_ENV=develop; npx cypress run.

Environment variables

Even though the test suite is intended to become part of a project, it is possible to change some behavior using environment variables.
This is useful for running the suite in different environments, for example, development, CI, or against production.

  • NODE_ENV if set to develop the development base URL configured in cypress.config.js will be used, and the default timeout is set to 10 seconds
  • CYPRESS_MAGENTO2_BASE_URL If set, this value will be used as the Magento 2 base_url. Otherwise, the base URL from cypress.config.js will be used.
  • CYPRESS_MAGENTO2_SPEC_PATTERN If set, only tests matching this glob pattern will be executed. Otherwise, the tests configured in cypress.config.js will be used.
  • CYPRESS_MAGENTO2_EXPORT_PATTERN If set, tests matching this glob pattern will be excluded.
  • CYPRESS_MAGENTO2_DEFAULT_TIMEOUT If set, used as the default timeout. Otherwise, the timeout defaults to 10 seconds if NODE_ENV is set to develop, or 4 seconds otherwise.
  • CYPRESS_MAGENTO2_ADMIN_TOKEN Used to authenticate against the Magento 2 API for setting up test fixtures.
  • CYPRESS_MAGENTO2_SKIP_CHECKOUT Set to a truthy value to skip any Hyvä tests that assume a Checkout is installed.
  • CYPRESS_MAGENTO2_SPEC_SUITE Set the test suite to run, if not set defaults to luma or hyva depending on response headers.

Any of these can also be configured in a cypress.env.json file without the CYPRESS_ prefix.

Running tests from modules

Spec files in extensions will be found by the default spec pattern at

  • app/code/**/Test/Cypress/hyva/**/*.spec.js or app/code/**/Test/Cypress/luma/**/*.spec.js
  • vendor/**/Test/Cypress/hyva/**/*.spec.js or vendor/**/Test/Cypress/luma/**/*.spec.js

Tests for different frontends can be supplied by using a different folder in Test/Cypress and setting the MAGENTO2_SPEC_SUITE config to that name. For example: app/code/Example/Module/Test/Cypress/vue/user/account.spec.js would be found by

CYPRESS_MAGENTO2_SPEC_SUITE=vue npx cypress run

If you do not want all tests to be run, regardless of the folder names, set MAGENTO2_SPEC_SUITE to an empty string.

Running accessibility tests

Accessibility spec files will not be found by the default spec pattern. This was desired as they should be seen as optional.

To run the accessibility tests locally you can update your cypress.env.json to include the following

{
    "MAGENTO2_SPEC_PATTERN": "cypress/integration/luma-accessibility/**/*.spec.js"
}

Videos

homepage.spec.js.mp4

homepage.spec.js tests running on https://hyva-demo.elgentos.io

product.spec.js.mp4

product.spec.js tests running on https://hyva-demo.elgentos.io

category.spec.js.mp4

category.spec.js tests running on https://hyva-demo.elgentos.io

product-searches.spec.js.mp4

product-searches.spec.js tests running on https://hyva-demo.elgentos.io

Contributing

We are very open to contributions! We would love to have mobile viewport support for Hyvä, tests for Commerce functionality, additional tests, code improvements, a fallback mechanism, etcetera etcetera. See the Issues tab for issues to pick up.

We will be updating this readme soon with extensive contribution guidelines, but here is a short summary:

  • Avoid creating global cy functions (Custom Commands), instead put functions in utils/helpers and import them
  • Avoid creating aliases that are only used once
  • Use cy.get() as much as possible, only use cy.contains() in specific cases - try to avoid it
  • Do not add assertions to page objects, move those to the spec files. Red flag; should() in a page object
  • Every test (an it() function) has to be able to run stand-alone; it should not depend on any other test. You can test this by add .only (see Cypress docs).

magento2-cypress-testing-suite's People

Contributors

andrewrmillar avatar danslo avatar dmanners avatar gregor-rw avatar knospe- avatar lefte avatar morgy93 avatar nisrinemach avatar norgeindian avatar peterjaap avatar rogoit avatar schmengler avatar shyambh avatar torhoehn avatar vinai avatar vladhorielov avatar zephanhazell avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

magento2-cypress-testing-suite's Issues

Add Mailosaur to tests

For some tests like reset password nieuws, newsletter subscription, order confirmation etc mails need te be received and read. For this purpose Mailosaur can be used. A mailosaur API-key and account will be needed, the API-key needs to be added to the cypress.json "env" object or to a cypress.env.json file. And the serverId needs to be added to the tests where mailosaur is used.

// cypress.json
"env": {
    "MAILOSAUR_API_KEY": "some-hash"
}
// sometest.spec.js
const serverId = 'SERVER_ID';
const testMail = `something@${serverId}.mailosaur.net`;

cy.mailodaurGetMessage(serverId, {
    sentTo: testEmail
}).then(email => {
    // Do stuff
})
  • Install mailosaur
  • Make sure cypress can read sent mails
  • Make sure it is optional since it requires an extra dependency

Tests

  • A confirmation email will be sent after placing the order
  • If payment is made an invoice will be sent

Example:

it('can receive a confirmation mail that an order has been placed', () => {
    // TODO: implement mailosaur
})

it('can receive an invoice through the mail when a payment is completed', () => {
    // TODO: implement mailosaur
})

Dependency of node "module" can not be used with "require()"

Describe the bug

The PR #98 breaks the config with

Your configFile is invalid: ...magento2-cypress-testing-suite/cypress.config.js

It threw an error when required, check the stack trace below:

Error [ERR_REQUIRE_ESM]: require() of ES Module ...magento2-cypress-testing-suite/node_modules/del/index.js from ...magento2-cypress-testing-suite/cypress.config.js not supported.
Instead change the require of index.js in /Volumes/CaseSensitive/Workspace/php-sites/mos-hyva-245/magento2-cypress-testing-suite/cypress.config.js to a dynamic import() which is available in all CommonJS modules.

The problem is that [email protected] is a node module, not a library (see "type": "module", in the del package.json).
This means it needs to be used with import.
The most simple fix is to use const del = import('del');
However, I suggest removing the dependency, since deleting folders and files is a trivial problem - node offers all that is needed out of the box (since fs is already imported):

return fs.rmdirSync(results.video, {recursive: true})

To Reproduce

git checkout main
git pull
npm ci
npx cypress run

Expected behavior
Cypress runs tests

Screenshots
Bildschirmfoto 2022-10-13 um 13 26 57

CMS pages

Tests

  • it can open the default CMS page correctly
  • it shows the default 404 page on an non-existent route

Alternative totals calculation in cart for USA locale display in format n,nnn.nn

Is your feature request related to a problem? Please describe.
Defined in cart.spec.js test 'displays the correct product prices and totals' the totals calculations seem off, or likely don;t cater for USA locale price display format of x,xxx.xx

The tests as is always fail not being able to compare price values/totals are correct, and it seems it only uses the thousands as a single int. In example below '2' from a total of 2,254.97 is definitely not equal to 48 (2 + 46) of the line items totals

image

Describe the solution you'd like
I have created an alternative test block, that properly works out the totals, as is, not just the first digits of the toatls, as what seems to be happening.
I did not do a MR, as I am unsure if these will work for other locales, and shouls likely just be in some form of docs for others to be able to use, and tweak to their needs. No need for others to work this out as well. Also, i am not a JS expert, so my code could likely be refined.

    it.only('displays the correct product prices and totals', () => {
        cy.visit(cart.url.product1Url)

        //check if product price matches with price in cart
        cy.get(cart.product.productPrice).then(($productPrice) => {
            const productPrice = Number($productPrice[0].textContent.replace(/[^0-9\.-]+/g, ""))
            Cart.addProductToCart(cart.url.product2Url)
            cy.visit(cart.url.cartUrl)
            cy.wait(3000)
            cy.get(cart.product1Price).then((itemPrice) => {
                let cartItemPrice = Number(itemPrice.text().replace(/[^0-9\.-]+/g, ""));
                cy.log(cartItemPrice)
                expect(productPrice).to.eq(cartItemPrice)
            })

            //change the qty value
            cy.get(cart.qtyInputField).eq(0).type('{backspace}2{enter}').then(($qty) => {
                const qty = parseInt($qty.val())
                cy.wait(3000)
                //check if qty * product subtotal displays the correct amount
                cy.get(cart.product1Subtotal).then(($subTotal) => {
                    const subTotal = Number($subTotal[0].textContent.replace(/[^0-9\.-]+/g, ""))
                    expect(productPrice * qty).to.equal(subTotal)

                })
            })
        })

        //check if the grand total is correct
        cy.get(cart.product1Subtotal).then(($total1) => {
            const subTotal1 = Number($total1[0].textContent.replace(/[^0-9\.-]+/g, "")).toFixed(2)
            cy.log(subTotal1)
            cy.get(cart.product2Subtotal).then(($total2) => {
                const subTotal2 = Number($total2[0].textContent.replace(/[^0-9\.-]+/g, "")).toFixed(2)
                cy.log(subTotal2)
                const calcTotal = Number(subTotal1) + Number(subTotal2)
                cy.log(calcTotal)
                cy.get(cart.grandTotal).then(($grandTotal) => {
                    const grandTotal = Number($grandTotal[0].textContent.replace(/[^0-9\.-]+/g, "")).toFixed(2)
                    cy.log(grandTotal)
                    expect(grandTotal).to.equal(calcTotal.toFixed(2))
                })
            })
        })
    })

image

Describe alternatives you've considered
See above

Additional context
None, see above

integration/hyva/homepage.spec.js import path to cart.json fixture is wrong

Describe the bug

Running:  homepage.spec.js
Oops...we found an error preparing this test file:
  > cypress/integration/hyva/homepage.spec.js
The error was:
Error: Webpack Compilation Error
./cypress/integration/hyva/homepage.spec.js
Module not found: Error: Can't resolve '../../../fixtures/hyva/selectors/cart.json' in '/data/web/public/cypress/integration/hyva'

To Reproduce
Steps to reproduce the behavior:

  1. Run the tests in the current main branch head

Expected behavior
Test can compile

Additional context
The path to the cart.json has one ../ to many at https://github.com/elgentos/magento2-cypress-testing-suite/blob/main/cypress/integration/hyva/homepage.spec.js#L6

Cypress version 10.0.0 no longer supports cypress.json.

Describe the bug
Using "cypress": "^10.0" will throw the following error: Cypress version 10.0.0 no longer supports cypress.json.

To Reproduce
Steps to reproduce the behaviour:

  1. Require "cypress": "^10.0"
  2. Run npx cypress run --spec ./cypress/integration/path/to/some.spec.js

Expected behaviour
Using the current version of cypress should work with the module

Additional context
A current workaround to this is to have a dependency on "cypress": "^9.7.0

Generic elements

Description.

Test all static content such as the header and footer are working correctly and that the homepage is working correctly.

Tests

  • Tests if the header is displayed correctly and if everything works correctly (do all links work)
  • Test if the footer is displayed correctly and if everything works correctly (do all links work)
  • Test if the homepage is correctly displayed and all links work correctly (all links work)
  • Test if the newsletter form, if any, is working correctly
  • If a product is displayed on the home page, can it be added to the shopping cart?

New input:

  • Contact form is working properly
  • Cookies are displayed correctly and work properly

Still working out:

  • Testing viewports mobile - desktop?
  • Wish list / wish list more extensive testing?

GUI not working inside docker (warden )

As we are using https://github.com/davidalger/warden for our projects, its not possible to use the GUI.
Running npx cypress open throws the following error:


[2051:0713/115838.687050:ERROR:bus.cc(397)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
[2051:0713/115838.687148:ERROR:bus.cc(397)] Failed to connect to the bus: Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory
[2051:0713/115838.688083:ERROR:bus.cc(397)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
[2051:0713/115838.688101:ERROR:bus.cc(397)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
[2202:0713/115838.728663:ERROR:sandbox_linux.cc(377)] InitializeSandbox() called with multiple threads in process gpu-process.
[2202:0713/115838.730793:ERROR:gpu_memory_buffer_support_x11.cc(44)] dri3 extension not supported.
[2051:0713/115838.850065:ERROR:bus.cc(397)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
[2051:0713/115838.862459:ERROR:bus.cc(397)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")

Is the GUI supported for cases like this?

Add Luma cypress tests

Tech notes

https://docs.cypress.io/guides/getting-started/installing-cypress

- run "npm install";
- run "npm install cypress --save-dev";
- run "npm install cypress-localstorage-commands --save-dev";
  • you will have folder cypress and cypress.json in root folder in your project:

  • run command "npx cypress open" to open app to select and run a test.

Contact form tests

Tests

  • it shows the contact form correctly
  • it cannot submit a form when no valid email address is entered
  • it can submit the form when all validation passes

Pull request #85 breaks test suite

Describe the bug

After merging #85 a number of tests stopped passing.
Reproduced on Magento 2.4.4 and 2.4.5 with sample data and Hyvä Theme 1.1.17.

More specifically:

Category page tests
    1) Can visit the category page and filters on color red

Perform searches
    1) Can find a single product
    2) Can see suggestions when entering search terms

Admin tests

Description.

Testing cache cleanup, indexing, tax, checking exceptions on all pages, etc.

Tests

  • Admin panel is working properly
  • Orders are added correctly after placing
  • Invoices are created automatically after placing the order and can be changed/viewed
  • Customer info can be viewed and adjusted
  • An order can be modified after placing the order

Tech note

  • Before orders come back to the backend the cron has to be run first
  • The admin login is best done via a fixture, this requires of course that an admin login has been created
    bin/magento admin:user:create --admin-user=admin --admin-password=test123 [email protected] --admin-firstname=John --admin-lastname=Doe

Accessibility tests with cypress-axe

We have been using cypress-axe https://github.com/component-driven/cypress-axe in our projects to not only test our system's functionality but also levels of accessibility. The test themselves are not so complex, most of the timte it's visit page and run the a11y test. We have also extended the logging to show more detailed information in our pipeline such as which elements are causing the issues and a detailed description.

Our current setup is a different test suite for accessibility as our normal test suite so that we can run them against different instances.

We would be more than happy to contribute this back to the repo if it would be interesting to have as part of the repository. Happy to discuss options and further ideas around accessibility checks.

Add missing tests

Add the tests missing from the repo README todo list

Cart

  • it merges an already existing cart when a customer logs in

MiniCart

  • it can open the cart slider by clicking on the cart icon in the header
  • it can change the quantity of an item in the cart slider

Homepage

  • it add products shown on the homepage to the cart when a add to cart button is present

CMS pages

  • it shows the default 404 page on an non-existent route
  • it can open the default CMS page correctly

Unify fixtures

There is some duplication throughout the repository. Fixture values are present in multiple fixture files, fixture files contain fixture values that do no relate to the test spec for that specific fixture.

There will also be errors that need fixing.

Also remove any possible routes that are still present in page-objects into fixtures

Definition of done

  • There are no duplicate values in the fixture files, both theme selector fixtures and regular fixtures
  • Fixtures contain the values for the tests related to that fixture (reuse some selectors in other specs)
  • The references to fixture file values from test specs all exist/point to the right fixture file
  • Move routes in page-object to fixtures
  • Order fixtures by alphabeth, for easier merging in the future
  • Make sure all references to the fixtures include the JSON file extension.

Update to Cypress 13

Would it be possible to update to Cypress version 13?
If so, what steps would be required to do that? Will be happy to do as much as I can, if someone tells me a bit, what to take care of.

Fix broken account test

Running:  account.spec.js                                                               (10 of 10)


  Account test creation
    ✓ Can create an account (4599ms)

  Account activities
    ✓ Can use API to login (16069ms)
    ✓ Can check your profile (4938ms)
    ✓ Can change password (16067ms)
    ✓ Can change the profile values (13030ms)
    ✓ Can view order history (5629ms)
    ✓ Can add an address ([143](https://github.com/elgentos/magento2-cypress-testing-suite/actions/runs/3117318267/jobs/5055893448#step:3:144)79ms)
    ✓ Can change an address (8038ms)
    ✓ Can remove an address (11938ms)
    ✓ Can change the newsletter subscription (5859ms)
    ✓ Can add a product the a wishlist (8703ms)
    ✓ Can edit the wishlist and remove item (11699ms)
    ✓ Can log out (5127ms)

  Customer cart
    1) Merges an already existing cart when a customer logs in

  Guest user test
    ✓ Can login from cart (1[150](https://github.com/elgentos/magento2-cypress-testing-suite/actions/runs/3117318267/jobs/5055893448#step:3:151)6ms)


  14 passing (3m)
  1 failing

  1) Customer cart
       Merges an already existing cart when a customer logs in:
     AssertionError: Timed out retrying after 4000ms: Expected to find element: `.cart .item .product-item-name a`, but never found it.
      at Context.eval (cypress/integration/hyva/user/account.spec.js:202:11)

See https://github.com/elgentos/magento2-cypress-testing-suite/actions/runs/3117318267/jobs/5055893448#step:3:157

Sensless default spec pattern

The default spec pattern is set to cypress/integration/**/*.spec.js (see MR).

This doesn't make any sense, since it will work neither with Luma nor with Hyvä.
The default value should be more useful.

Expected behavior
Ideally the matching integration tests are executed automatically, maybe by running a preflight check in cypress.config.js against the baseUrl and determining the theme.

Missing dependencies when running the module inside docker

We currently use https://github.com/davidalger/warden for our projects. Running npx cypress run --spec ./cypress/integration/path/to/some.spec.js leads to the following error:

Your system is missing the dependency: Xvfb

Install Xvfb and run Cypress again.

Read our documentation on dependencies for more information:

https://on.cypress.io/required-dependencies

If you are using Docker, we provide containers with all required dependencies installed.

----------

Error: spawn Xvfb ENOENT

----------

Platform: linux-x64 (CentOS - 8
)
Cypress Version: 10.3.0

I see two possible solutions to this:

  1. Require these packages directly into warden so that each installation will have them
  2. Update the README.md to include this so that people running into this issues should know how to fix it.

Default Hyvä cypress is failing

Describe the bug

When I run the cypress tests, several tests fail.
The failing tests:

  • Can add a coupon to the cart
  • Can delete an added coupon from the cart
  • Cannot add an invalid coupon
  • Can visit the category page and filters on color red
  • Can add a product to the wishlist when customer is logged in
  • Can open default CMS pages
  • Can find a single product
  • Can remove an address
  • Can edit the wishlist and remove item

To Reproduce
Steps to reproduce the behavior:

composer.json

    "require": {
        "hyva-themes/magento2-default-theme": "1.1.17",
        "hyva-themes/magento2-email-module": "^1.0",
        "hyva-themes/magento2-graphql-tokens": "^1.0",
        "hyva-themes/magento2-luma-checkout": "^1.1",
        "hyva-themes/magento2-reset-theme": "^1.1",
        "hyva-themes/magento2-theme-module": "1.1.17",
        "magento/product-community-edition": "^2.4",
        "markshust/magento2-module-disabletwofactorauth": "^2.0"
    },

    "require-dev": {
        "magento/magento-cloud-docker": "^1.3",
        "magento/module-bundle-sample-data": "100.4.*",
        "magento/module-catalog-rule-sample-data": "100.4.*",
        "magento/module-catalog-sample-data": "100.4.*",
        "magento/module-cms-sample-data": "100.4.*",
        "magento/module-configurable-sample-data": "100.4.*",
        "magento/module-customer-sample-data": "100.4.*",
        "magento/module-downloadable-sample-data": "100.4.*",
        "magento/module-grouped-product-sample-data": "100.4.*",
        "magento/module-msrp-sample-data": "100.4.*",
        "magento/module-offline-shipping-sample-data": "100.4.*",
        "magento/module-product-links-sample-data": "100.4.*",
        "magento/module-review-sample-data": "100.4.*",
        "magento/module-sales-rule-sample-data": "100.4.*",
        "magento/module-sales-sample-data": "100.4.*",
        "magento/module-swatches-sample-data": "100.4.*",
        "magento/module-tax-sample-data": "100.4.*",
        "magento/module-theme-sample-data": "100.4.*",
        "magento/module-widget-sample-data": "100.4.*",
        "magento/module-wishlist-sample-data": "100.4.*",
        "magento/sample-data-media": "100.4.*"
    },

cypress.config.js

const {defineConfig} = require('cypress');
const {tagify} = require('cypress-tags');
const fs = require('fs');
const envConfig = fs.existsSync('./cypress.env.json') ? require('./cypress.env.json') : {};

// Run "NODE_ENV=develop; npx cypress run" to run tests locally
const defaultBaseUrl = process.env.NODE_ENV === 'develop' ? 'https://hyva.playground.local' : 'https://hyva.playground.local';

// Sometimes our local envs are slow due to dev mode. Raising the timeout decreases flakiness
const defaultCommandTimeout = process.env.NODE_ENV === 'develop' ? 10000 : 4000;

// Customize to use either hyva, luma or custom specs
const defaultSpecPattern = [
    'cypress/integration/hyva/**/*.spec.js',
];

const baseUrl = process.env.CYPRESS_MAGENTO2_BASE_URL || envConfig.MAGENTO2_BASE_URL || defaultBaseUrl;
module.exports = defineConfig({
    projectId: "",
    e2e: {
        baseUrl: baseUrl,
        specPattern: process.env.CYPRESS_MAGENTO2_SPEC_PATTERN || envConfig.MAGENTO2_SPEC_PATTERN || defaultSpecPattern,
        specSuite: process.env.CYPRESS_MAGENTO2_SPEC_SUITE || envConfig.MAGENTO2_SPEC_SUITE || undefined,
        excludeSpecPattern: process.env.CYPRESS_MAGENTO2_EXCLUDE_PATTERN || envConfig.MAGENTO2_EXCLUDE_PATTERN || '',
        defaultCommandTimeout: parseInt(process.env.CYPRESS_MAGENTO2_DEFAULT_TIMEOUT || envConfig.MAGENTO2_DEFAULT_TIMEOUT || defaultCommandTimeout),
        videoUploadOnPasses: !!(process.env.CYPRESS_VIDEO_UPLOAD_ON_PASSES || envConfig.VIDEO_UPLOAD_ON_PASSES || false),
        watchForFileChanges: false,
        supportFile: 'cypress/support/index.js',
        viewportWidth: 1920,
        viewportHeight: 1080,

        env: {
            mobileViewportWidthBreakpoint: 768,
            mobileViewportWidthBreakpointHyva: 1024
        },

        setupNodeEvents(on, config) {

            on('after:spec', (spec, results) => {
                // If a retry failed, save the video, otherwise delete it to save time by not compressing it.
                if (results && results.video) {
                    // Do we have failures for any retry attempts?
                    const failures = results.tests.find(test => {
                        return test.attempts.find(attempt => {
                            return attempt.state === 'failed'
                        })
                    });
                    // Delete the video if the spec passed on all attempts
                    if (!failures) {
                        fs.existsSync(results.video) && fs.unlinkSync(results.video)
                    }
                }
            })

            on('file:preprocessor', tagify(config))

            const applySpecSuiteToSpecPattern = (config) => {
                // If the specSuite is an empty string, add a trailing / to $SPEC_SUITE to avoid // in the pattern
                const regex = config.specSuite === '' ? /\$SPEC_SUITE\//g : /\$SPEC_SUITE/g;
                const fn = (specPattern) => specPattern.replace(regex, config.specSuite)
                config.specPattern = typeof config.specPattern === 'string' ? fn(config.specPattern) : config.specPattern.map(fn);
                return config;
            }

            return config.specSuite === undefined
                ? new Promise((resolve, reject) => {
                    // Do a preflight request to determine which frontend test suite to run depending on the target theme
                    if (baseUrl.startsWith('https')) process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = '0';
                    const req = (baseUrl.startsWith('https') ? require('https') : require('http')).request(baseUrl);
                    req.on('response', res => {
                        config.specSuite = res.headers['x-built-with'] === 'Hyva Themes' ? 'hyva' : 'luma'
                        resolve(applySpecSuiteToSpecPattern(config));
                    })
                    req.on('error', reject);
                    req.end();
                })
                : applySpecSuiteToSpecPattern(config);
        }
    }
});

cypress.env.json

{
  "MAGENTO2_ADMIN_TOKEN": "eyJraWQiOiIxIiwiYWxnIjoiSFMyNTYifQ.eyJ1aWQiOjEsInV0eXBpZCI6MiwiaWF0IjoxNjcwOTIwNTA1LCJleHAiOjE2NzA5MjQxMDV9.TsFWBeGjERqDdcfZzHuWsWXGI22dMyZgt2XWsizA7ls",
  "MAGENTO2_SKIP_CHECKOUT": true,
  "MAGENTO2_DEFAULT_TIMEOUT": 10000
}

Order success page issue in Luma theme

Order success page not shown while executing the Checkout tests (can find and order in the customer order history after having placed an order). Instead its being redirected to the cart page. The order gets successfully placed.

The issue only happens for logged in scenario while automating with Cypress. During manual attempt and using selenium it works perfectly fine.

To Reproduce
Steps to reproduce the behavior:
Use any default luma instance. I tried in default Luma Magento 2.4.3
Run checkout tests >> can find and order in the customer order history after having placed an order

Expected behavior
Order success page should be shown and the order number verification against the order history section should pass.

Screenshots
image

Tests run against Hyvä demo site are failing

A number of tests are failing since the Hyvä demo site (https://demo.hyva.io/) is updated and some settings changed. For example; prices are now shown in Euro and the newsletter subscription box has been removed/deactivated.

Failing tests;

  • Can subscribe to newsletter
  • Can add a coupon to the cart
  • Can delete an added coupon from the cart
  • Cannot add an invalid coupon
  • Can set the price to zero when every associated product qty is zero
  • Can calculate the price based on selected options
  • Can visit the category page and filters on color red
  • Can see a price for the product
  • Can add a product to the wishlist when customer is logged in
  • Can see the correct product price and shipping costs
  • Can see coupon discount in checkout
  • Can find and order in the customer order history after having placed an order
  • Can open default CMS pages
  • Can find a single product
  • Can see suggestions when entering search terms
  • Can add an address automatically from saved address'
  • Can remove an address
  • Can edit the wishlist and remove item
  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✖  homepage.spec.js                         00:16        8        7        1        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  cart/cart.spec.js                        00:38        7        4        3        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  cart/minicart.spec.js                    00:13        7        7        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  catalog/bundle.spec.js                   00:16        5        3        2        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  catalog/category.spec.js                 00:11        6        5        1        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  catalog/product.spec.js                  00:23       13       11        2        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  checkout/checkout.spec.js                00:17        3        -        3        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  cms/cms-pages.spec.js                    00:05        2        1        1        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  search/product-searches.spec.js          00:13        4        2        2        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  user/account.spec.js                     01:57       17       12        5        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  9 of 10 failed (90%)                     04:33       72       52       20        -        -  

Add cypress-tags support

closes #26
Tagging of write tests (hot vs readonly ) CYPRESS_INCLUDE_TAGS=hot npx cypress run , if you run npx cypress run it should only run the non-hot tests

If we are going to add tag support the tech notes need to be added to the allround readme

Definition of done

  • Check all specs on potential hot tests
  • Add a script alias that runs only non hot tests
  • Add a stript alias that includes hot tests

Tech notes

Install tag support (uses typescript): npm install cypress-tags typescript --save-dev

Running the tests with tags:

  • Excluding 'hot' tests: npm run cypress
  • Only 'hot' tests: npm run cypress-hot

Make tests work on current (1.1.16) Hyvä theme out of the box

Describe the bug
The current test suite is not compatible with Hyvä 1.1.16 as some selectors need to be adjusted

To Reproduce
Steps to reproduce the behavior:

  1. Install a fresh Hyvä 1.1.16 on Magento 2.4.4
  2. Run the tests

Expected behavior
All tests pass

Desktop (please complete the following information):

  • I've observed the tests failing using Chrome and Electron, but I expect this to be the case on all browsers as many selectors simply no longer match.

Failing test

There are test failing in the cart spec because links in the cart page-object is linking to non existent JSON values.

Perhaps other spec files have the same issue.

Add tag support

Add support for tags in the test spec; https://github.com/annaet/cypress-tags#setup

Add tags to describe:

// cypress/integration/user/account.spec.js

describe(['hot'],'Account test creation', () => {
    it('Can create an account', () => {

or to the tests themselves:

// cypress/integration/products/product.spec.js

it(['hot'], 'Can\'t add an out of stock product to the cart', () => {

Tech notes

Install tag support (uses typescript):

npm install cypress-tags typescript --save-dev

Running the tests with tags by adding scripts to package.json:

  • Excluding 'hot' tests: npm run cypress
  • Only 'hot' tests: npm run cypress-hot
"scripts": {
    "cypress": "CYPRESS_EXCLUDE_TAGS=hot npx cypress run",
    "cypress-hot": "CYPRESS_INCLUDE_TAGS=hot npx cypress run"
}

Create pipeline for new PR

For each new PR the changes have to be verified manually. It would be handy for run a Github Actions pipeline which executes the Cypress tests. At least for Hyvä can be used as an endpoint http://demo.hyva.io/.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.