Command at bottom of sample readme showing how to run tests locally is jest, should be npm test/yarn test

Describe the bug

There's a small typo in the Playwright sample readme. On line 48 accessibility is spelled as accessibliity (note the l and i are switched).

Add instructions on how to restrict axe scans to include only failures corresponding to WCAG 2.0 A and AA rules:

use the AxeBuilder.withTags method as follows when you initialize Axe in your test code:

AxeBuilder(driver)
  .withTags(['wcag2a', 'wcag2aa'])

Participants were not sure about what the repo is trying to accomplish. One participant requested a high-level summary at the start of the page to give more context about what the user should expect to do and what the output will be. One user thought a short video describing the project would be a good idea for an overview.
Recommendation:
• Provide users with a high-level summary of the process and a table of contents to organize information
• Provide a clearer picture of what the end state looks like and step by step instructions to get there.

We integrated AxeCore into our test harness. Developers can easily specify a selector to run tests against their feature, however AxeCore appears to use document.querySelectorAll which cannot search in our features because we use Web Components and the Shadow DOM - even though we have the "open" property set. This means tests run against a whole page and not a specific feature.

Is there a way AxeCore can be updated to use xPath or another way to select from the DOM that works with Web Components?

Participants reviewing files that make up the sample project felt that,
• It would have been nice to start with a simpler scenario than including possible variations that are confusing and overwhelming
• Not sure how to get started
• Too heavy with data/ information
• Most of this can be done outside Azure pipelines
• Not confident that this is reliable and maintainable

Recommendation:
Simplify the sample project to the most basic scenario with options to build over it incrementally

One of our users in a user study told us after reading our Disclaimer text that they weren't sure they'd have continued implementing automated tests at all after having read it; they felt like they weren't convinced it would be valuable.

We discussed rewording the disclaimer to be more encouraging in #41 and offline with @ferBonnin; we want to maintain the spirit of the disclaimer and the pointer to Accessibility Insights, but we think there's room to try to have the disclaimer text be a bit more positive about automated testing still being valuable and worth doing as a first step, even if it won't catch all/most issues.

Please check whether the bug has already been filed.

TS lint on VSC can't find axe-webdriverjs
After clone the project and install the dependencies, was reading in the file index.test.ts and realize that import * as AxeWebdriverjs from 'axe-webdriverjs'; was showing this error

This is also happening after try to implement in my own project.

To Reproduce
Steps to reproduce the behavior:

1. Clone the project
2. Install dependencies
3. Go to index.test.ts with visual studio code
4. See error

Expected behavior
Don't see the error

Actual behavior
This error is show Could not find a declaration file for module 'axe-webdriverjs'. 'c:/Users/v-jogue/axe-pipelines-samples/typescript-selenium-webdriver-sample/node_modules/axe-webdriverjs/dist/index.js' implicitly has an 'any' type.
Try npm install @types/axe-webdriverjs if it exists or add a new declaration (.d.ts) file containing declare module 'axe-webdriverjs';

Screenshots or .GIF

Desktop (please complete the following information):

• OS: Windows 10
• VSC 1.41.1

It was not clear to one of the teams onboarding that they had to install the Scans viewer build tab extension to visualize failures in Azure DevOps build results page.

Recommendation:
Update to include this clearly as a step with link to Azure DevOps marketplace.

Issue:
When initally running the tests on my machine, I used the Developer Command Prompt for VS2017 as an admin. Once I ran 'yarn test', I received the following error message:

RUNS  tests/index.test.ts
(node:11356) UnhandledPromiseRejectionWarning: WebDriverError: unknown error: unable to discover open pages
(node:11356) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:11356) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

The error message wasn't helpful enough for me to find the issue and I was stuck trying to prevent my Chrome driver from crashing.

Eventually, I reopened the Command Prompt as a non-admin, ran 'yarn test' and had no issues.

Desired Behavior:
It would be better to have a more detailed error description for running the tests as an admin. Additionally, the ability to run the tests as admin would be ideal since our team usually runs our other Jest tests in that way already.

There are several mandatory files we require in this
repository. A pull request has been opened to add the
missing files. When the pr is merged this issue will be
closed automatically.

Merge this pull request

Participants were not sure about what the repo is trying to accomplish. One participant requested a high-level summary at the start of the page to give more context about what the user should expect to do and what the output will be. One user thought a short video describing the project would be a good idea for an overview.
Recommendation:
• Provide users with a high-level summary of the process and a table of contents to organize information
• Provide a clearer picture of what the end state looks like and step by step instructions to get there.

Clarify the README on the Sample page to answer a variety of questions asked by participants, including:
a. What is necessary?
b. Why do I need to know about SARIF?
c. What if I already use a similar tool?

Recommendation:
• Provide users context on what is the minimum tools they will need to install
• State clearly that this is a sample project for reference and the expectation is to update their existing tests to include automated accessibility scans

edit: As part of making the use of/action for each tool clear, we want to include making it clear why/how to install the sarif viewer build tab extension (originally part of both this issue and #39).

Note: Edited to merge issue #28 because the issues are closely related, and fixing #28 (by updating the 'Getting Started' section to have a tutorial with step by step instructions on how to use the sample) would require fixing this issue as well.

Describe the bug

Recently, we've been seeing about 30% of CI/PR builds show a failure during setup of the geckodriver<->Firefox connection in the firefox test job. We haven't root caused the issue or why it became markedly worse in the last 1-2 months; this bug tracks that work.

Is your feature request related to a problem? Please describe.

It would be nice if we could reset the test result baseline and exclude test failures from previous test runs. We used to have this feature with the old Chisel package, through DevOps configurations.

Describe the desired outcome

This feature maybe complicated, as it needs to allow test failure exclusions during run time, instead of exclude tests in code through CSS selectors and specific rules, which are already possible.

Describe alternatives you've considered

There isn't an exact alternative approach at this moment based on developers' feedback. We could exclude tests in code with selectors and rules, but that requires code changes every time a new issue comes up.

Additional context

To reset baseline this way was possible with old Chisel package and DevOps configurations

Describe the bug

The screenshot and its alternate text under "See it in action on your local machine" in typescript-selenium-webdriver-sample folder show jest command being run, while the code snippet directly above them instructs users to run yarn test or npm test.

To Reproduce
Steps to reproduce the behavior:

1. Go to typescript-selenium-webdriver-sample
2. Scroll down to "See it in action on your local machine" > "Run the tests!"
3. See screenshot and its alternate text

Expected behavior

The screenshot should show a yarn test or npm test command to match the code snippet, and its alternate text should also match.

Are you willing to submit a PR?

If asked, will you be willing to submit a PR to fix this bug? Yes

Update following sections of the C# sample to be inline with the typescript sample,

• Getting started
  • samplepagetests.cs to include instructions to use axe-sarif-converter to convert axe results to sarif format
  • azure-pipelines.yml to add Publish Build Artifacts task to paint the Scans tab
• Tools and libraries used
  • axe-sarif-converter to convert axe results to SARIF format
  • Sarif Viewer Build Tab to visualize the results in Azure Pipelines
• See it in action on Azure Pipelines
  • Reference to Sarif Viewer Build Tab
  • Screenshot of Scans tab in sample build
  • Link (off the screenshot) to the live demo of Scans tab on sample build

Is your feature request related to a problem? Please describe.

This is a feature request for a sample on how to integrate the axe-android checks in the CI/CD pipeline

Additional context

This has been a request from user feedback and pilot teams. Adding it here as a reference to group upvotes and comments. We also need to understand what type of E2E framework most teams use in their build pipelines. We have heard Espresso.
We would be happy to take a contribution on a team that has integrated these tests in their pipeline.

Is your feature request related to a problem? Please describe.

feature request - we are using testcafe for web automation. not found example of axe-pipelines on how to use for testcafer
https://github.com/DevExpress/testcafe

Describe the desired outcome

testcafe axe a11y automation sample to be provided

Participants were not sure,
• Why they would use the sample before trying on their own projects.
• Why they needed to integrate to Azure pipelines
o incremental build time
o Additional overhead in installing Azure DevOps extension
• Value of including automated checks compared to static analysis tools like ESLinkt and TSLint

Recommendation:
• Explain the benefits of the sample in the high-level overview
• Explain why an option to integrate with Azure pipelines can be beneficial, especially for visualizing failure details. How about including this as an optional step?

In the Selenium sample, the link embedded in ‘Screenshot of Scans tab in sample build’ graphic under section ‘See it in action in Azure Pipelines’ currently does not display scan results as intended.

Resolution:

1. Update the link embedded in the graphic to point to the correct page that displays failure details from the latest scan, as is here.
2. Update the screen shot (the link is embedded in), if necessary

Attachments:
Current screen.png: Screen shot of page that the link is pointing to currently
Scans results.png: Screen shot of the Scans tab with failure details (how this page should ideally look like)

Describe the desired outcome

I'd love to be able to add accessibility testing to my pipeline. I have a UWP app and UWP unit test project that contains UI Test. I'm able to run them in azure-pipelines using the WinAppDriver.

Describe alternatives you've considered

I tried following the sample for csharp against a web page. Using Selenium.Axe conflicts with the Microsoft.WinAppDriver.Appium.WebDriver that allows me to run UI tests (they depend on different versions of the Selenium.WebDriver, and the latest doesn't work for UWP).

Is your feature request related to a problem? Please describe.

My team is currently using the C# + Selenium option from https://github.com/microsoft/axe-pipelines-samples to run Accessibility Insights in our ADO build pipeline but would like to convert all our UIA to using C# + Playwright now that Microsoft has an official .NET port for Playwright (https://github.com/microsoft/playwright-dotnet).

Describe the desired outcome

Can an equivalent C# version of the npm @axe-core/playright package be created?

Additional context

Creating feature request on behalf of user

When participants initially landed on this page, they were distracted by links. None of the three participants successfully found the link to the sample on this page. One participant was confused by the name and thought it was a web driver, not a sample. Also, the immediate link to Axe was confusing. Two users immediately clicked on the link and were confused as to why they were being brought to this page. One user was confused and was not sure what the repo has to do with Accessibility Insights. One user realized after some time that the sample (typescript) was not relevant to his team (C#).

Recommendations:
• Update summary to introduce the solution better and establish connection with axe.
• Update axe link to point to axe-core repo. Clearly differentiate link to axe from link to the sample.
• Update summary to spell out which sample is relevant for what team.
• Change the name of the sample to something more obvious, and make the link to sample more prominent
• Consider having different pages for different samples

Describe the desired outcome
I'd love to be able to add accessibility testing to our application's engineering pipeline. Our application is native Win32 and the samples we've seen here are focus on apps that are implemented using web technologies.

We got a report from an internal customer we sent the sample to that, based on the existing selenium-based sample, they had added a Selenium dependency to get a11y testing in place even though they already had Puppeteer-based end to end tests in place.

Suggested resolution:
Update the Selenium sample readme to clarify that Even if a team is not using Selenium and are using a different browser automation framework, many of them already have specific open source libraries for integrating axe scans (puppeteer has axe-puppeteer, webdriverio has axe-webdriverio, cypress has cypress-axe). Most of the concepts from the Selenium sample would be the same, but they would want to use the axe integration library most relevant to the framework they are using.

Describe the bug

We received user feedback asking for more detailed guidance about how one might set up the localhost server described in index.test.ts's beforeEach method. Need to consider right balance of how much to explain/how general we can be.

I think in general, there’s going to be 2 basic types of option:

1. Do exactly like you suggested and make it part of the npm script. In particular, many projects already define a npm script like “serve” or “dev” to run a localhost server for manual testing purposes; it’s reasonable to reuse that with a command like "test": "npm run serve && jest"
2. Start the localhost server from within the test itself, usually in a beforeAll method or a Jest setupFiles entry.

Both will work; it’s mostly a matter of taste/which is easier to reuse for a particular project. We can probably add information along these lines to the motivating comment.

We've seen a few users attempt to apply the typescript-selenium-webdriver-sample to a project using Angular Protractor. This should work just fine if they replace the normal driver initialization step in the tests with

import { browser } from 'protractor';
import * as AxeBuilder from 'axe-webdriverjs';
const axeBuilder = AxeBuilder(browser.driver); // using protractor browser's underlying selenium driver, instead of a manually-initialized driver

...but it's not clear right now from the documentation that this is expected to be fine and work. It would be nice to make this option clear.

Describe the bug

We've had multiple reports of users trying to use the sample stumbling on getting ChromeDriver working. One user was unable to make progress without our input, and another wasn't able to get it to cooperate based on our instructions and instead ended up npm i chromedrivering based on external searches.

At minimum, we should make the installation steps/links more verbose/clear.

We should also consider suggesting the chromedriver package; the main disadvantage of using it (incompatibility risk with externally driven chrome/chromium updates) could be mitigated by showing how to do a version check and continuing to specifically override the Azure Pipelines case to use its preinstalled chromedriver.

We got a report from an internal customer we sent the sample to that, based on the existing selenium-based sample, they had added a Selenium dependency to get a11y testing in place even though they already had Puppeteer-based end to end tests in place.

It would be nice to have a sample using axe-puppeteer to prevent other teams doing the same

Few users have reached out for help on the support alias and reported this as an issue during interviews. The root cause is the exclusion of .withtags that isolates axe-core results to specified WCAG violations and the default behavior of Accessibility Insights for Web.

Recommended changes:

Update index.test.ts (Typescript sample), SamplePageTests.cs (C# sample) files to make .withtags more prominent by including,
.WithTags("wcag2a", "wcag2aa", "wcag21aa")
and accompanying instructions in the (respective) first test case:

• typescript sample: // This test case shows the most basic example: run a scan, fail the test if there are any failures.
• C# sample: // This test case shows the most basic example: run an accessibility scan on a page and assert that there are no violations.

Describe the bug

If you do something that causes index.test.ts's beforeAll to fail, the error output is verbose enough to be confusing, and masks the "real" error the user should be looking at. For example, if you don't have ChromeDriver available on your path, the error output shows the "ChromeDriver could not be found" error 3 times (should be once) and it also shows a spurious error ("TypeError: Cannot read property 'quit' of undefined"). Ideally, the output should just have the one copy of the one meaningful error.

(I thought jest-circus/runner was intended to cover this for us, but apparently it's not working like we'd expected)

To Reproduce
Steps to reproduce the behavior:

1. Ensure ChromeDriver is not on your path
2. Run yarn test
3. See console output

Are you willing to submit a PR?

If asked, will you be willing to submit a PR to fix this bug? Yes

Did you search for similar existing issues?

Yes