Giter Club home page Giter Club logo

accessibility-insights-action's Introduction

Product Logo Accessibility Insights Azure DevOps (ADO) extension

codecov

About

The Accessibility Insights Azure DevOps (ADO) extension helps integrate automated accessibility tests in your Azure pipeline. You can configure the ADO extension to scan a single page or crawl an entire site. This can work for static HTML files in your repository, a local web server, or remote URLs. It is currently only available for Microsoft internal teams.

The ADO extension uses the axe-core rules engine to scan websites and web apps for common accessibility issues.

While automated checks can detect some common accessibility problems such as missing or invalid properties, most accessibility problems can only be discovered through manual testing. The best way to evaluate web accessibility compliance is to complete an assessment using Accessibility Insights for Web.

For Microsoft internal teams, visit how to use the Azure DevOps (ADO) extension.

Contributing

To build and test, visit developer docs.

Visit CONTRIBUTING.md for more information about the contributor license agreement.

accessibility-insights-action's People

Contributors

adsathye avatar ahmedabdoortiga avatar asksep avatar brocktaylor7 avatar codeofdusk avatar davetryon avatar dbjorge avatar dependabot-preview[bot] avatar dependabot[bot] avatar devaradhanm avatar ferbonnin avatar github-actions[bot] avatar haonliu avatar jalkire avatar jgibson2019 avatar karanbirsingh avatar katydecorah avatar lamaks avatar lisli1 avatar manishsat avatar microsoft-github-operations[bot] avatar microsoft-github-policy-service[bot] avatar microsoftopensource avatar msftbot[bot] avatar pownkel avatar robgallo avatar sfoslund avatar thanyaleif avatar v-viyada avatar waabid avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

accessibility-insights-action's Issues

Jest configuration doesn't support pure-ESM packages

We've recently seen some dependabot updates to packages that are pure-ESM. This issue lists updates whose builds failed & whose update path wasn't immediately obvious - I didn't want to lose track of them. Looking closer & updating this description as we go

At first I thought this was because this repo targets Common JS; in general, trying to import ESM packages into CommonJS requires dynamic await() imports. I thought we'd need to follow the steps listed here: https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c

After trying the update locally, I noticed webpack overrides the require function so that overall index.js is still functional. The PR build failures come from jest tests with the error signature:

@accessibility-insights-action/shared: FAIL @accessibility-insights-action/shared src/ioc/setup-ioc-container.spec.ts
@accessibility-insights-action/shared:   ● Test suite failed to run
@accessibility-insights-action/shared:     Jest encountered an unexpected token
@accessibility-insights-action/shared:     Jest failed to parse a file. This happens e.g. when your code or its dependencies use non-standard JavaScript syntax, or when Jest is not configured to support such syntax.
@accessibility-insights-action/shared:     Out of the box Jest supports Babel, which will be used to transform your files into valid JS based on your Babel configuration.
@accessibility-insights-action/shared:     By default "node_modules" folder is ignored by transformers.
@accessibility-insights-action/shared:     Here's what you can do:
@accessibility-insights-action/shared:      • If you are trying to use ECMAScript Modules, see https://jestjs.io/docs/ecmascript-modules for how to enable it.
@accessibility-insights-action/shared:      • If you are trying to use TypeScript, see https://jestjs.io/docs/getting-started#using-typescript
@accessibility-insights-action/shared:      • To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config.
@accessibility-insights-action/shared:      • If you need a custom transformation specify a "transform" option in your config.
@accessibility-insights-action/shared:      • If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option.
@accessibility-insights-action/shared:     You'll find more details and examples of these config options in the docs:
@accessibility-insights-action/shared:     https://jestjs.io/docs/configuration
@accessibility-insights-action/shared:     For information about custom transformations, see:
@accessibility-insights-action/shared:     https://jestjs.io/docs/code-transformation
@accessibility-insights-action/shared:     Details:
@accessibility-insights-action/shared:     /home/runner/work/accessibility-insights-action/accessibility-insights-action/node_modules/get-port/index.js:1
@accessibility-insights-action/shared:     ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,jest){import net from 'node:net';
@accessibility-insights-action/shared:                                                                                       ^^^^^^

Jest has experimental support for ECMAScript modules. We can try changing its configuration to unblock these PRs. We can also choose to defer the PRs. We have a little more leeway since we aren't exporting a library of our own.

Allow exclusion of URL's from crawler

Problem

When using the Accessibility Insights Azure DevOps Task, the pipeline allows for the use of discovery patterns. However, this appears to be limited to adding URL and not excluding them.

Ask

I would like to see an additional parameter to exclude URLs via regex (or other options) in the same manner as the discovery pattern

Examples:

Exclude url’s that include query parameters (everything after /?)
Exclude url’s that are in the domain but fall into a specific subdomain or path (like example.com/blog/....)
I am sure there are other ideas as well :)

Problem with scanning local HTML files when using Windows agent

Problem

When using vmImage: "windows-latest" and following the instructions to Scan local HTML files in the ADO extension, the scan successfully completes but shows incorrect results. Currently the documentation says to use the following configuration:

inputs:
      siteDir: '$(System.DefaultWorkingDirectory)/website/root'
      scanUrlRelativePath: '/' # use '//' if Windows agent

The logs contain something that looks like:

[Trace][info] === Started local web server. Url: http://localhost:5858 Root directory: D:\a\1\s
[Trace][info] === Starting accessibility scanning of URL http://

More info

The extension is using url.resolve in scan-url-resolver.ts, which may be truncating the url that should be scanned. It appears also that the url.resolve method has been deprecated in node v11.0.0 and changed to legacy status in v14.17.0 and v15.13.0.

Note that even though this issue was found when using the ADO extension, it could possibly be an issue in the GitHub Action as well; scan-url-resolver.ts is in the shared package.

Workaround

A workaround for this is to include one trailing / to the siteDir variable and use just / for scanUrlRelativePath instead of the documented //. So it would look something like the following:

inputs:
      siteDir: '$(System.DefaultWorkingDirectory)/website/root/'
      scanUrlRelativePath: '/'

Possible solutions

Some options we might consider include:

  1. Changing the documentation to specify the configuration suggested in the workaround. Using this configuration seems to work for Linux and macOS agents as well, so there may be no need for system-specific notes.
  2. Improving path resolution in scan-url-resolver.ts to work for the currently suggested configuration
  3. A combination of 1 and 2
  4. Other ideas?

Add input to adjust the zoom level for the page(s) to be scanned

Is your feature request related to a problem? Please describe.

A user is interested in being able to increase the zoom level (400%) of their site during scan to help find issues at reflow.

Describe the desired outcome

A new input that would allow the user to define the zoom level that the action would then pass to ai-scan.

Describe alternatives you've considered

Additional context

It could also be helpful to investigate parameterizing the viewport height and width so that the user can also specify breakpoints that the scan should use.

Prior art: puppeteer/puppeteer#4419

Does this Private Extension work if we move ADO to Customer Tenant

Hi Team
This is listed as a Private Extension in Devops Marketplace , can the ADO continue to use it if that moves from the MS tenant to Customer tenant
If they are not available once ADO moves in the customer tenant , are there any alternative versions available for customers can continue to use them

It needs to be more obvious that siteDir and url are mutually exclusive inputs

Describe the bug

Several pilot teams onboarding to the Azure DevOps task have gotten confused about exactly which parameters they should be specifying when setting up the task. In particular, we've seen several pilot teams try to specify both siteDir and url inputs simultaneously, even though these parameters are mutually exclusive (specifying url causes siteDir, localhostPort, and scanUrlRelativePath to be ignored).

The task should do a better job of making it hard for a user to make this mistake. For users of ADO's graphical Pipeline definition UX (most of our pilot teams so far), I think the ideal would be to use task definition groups to enforce the parameters' mutually exclusivity as follows:

  • Add a new required option that the user must select to determine whether they want to use "our task hosts your site for you using a simple Express configuration" mode vs "you host your site in a separate task before this one runs" mode
    • siteDir, scanUrlRelativePath, and localhostPort should be moved to a group with is only visible when "we host your site for you" mode is selected. In this mode, siteDir should be required and the others should be optional. The guidance for each should clarify the exact expected format (particularly, siteDir should clarify if it accepts relative vs absolute paths).
    • url should only be visible (and should be required) when "you host your site separately" mode is selected

However, this wouldn't do anything to help users of yaml build definitions. For them, I'd suggest the following additions:

  • add an early parameter validation step before starting to scan which detects mutually exclusive parameters being set and fails fast with an actionable error message, rather than silently picking one set to ignore
  • Change the names of the inputs to make the grouping between them more clear and to make it easier for documentation to refer to the parameters as a group (eg, for error messages and the url documentation to explain what it's mutually exclusive with). Perhaps something like:
    • staticSiteDir (instead of siteDir)
    • staticSitePort (instead of localhostPort)
    • staticSiteUrlRelativePath (instead of scanUrlRelativePath)

Note that most of the above suggestions would be breaking changes; see #1063

To Reproduce
Steps to reproduce the behavior:

  1. create a test pipeline using the ADO task which specifies both siteDir and url
  2. run a build
  3. observe that the task silently ignores siteDir

CodePen repro example

n/a

Expected behavior

We should strive for our UX to be a "pit of success" - we should make it hard for users to misconfigure the product, especially in a way which misbehaves silently. It isn't enough to rely on users reading usage documentation.

Screenshots

n/a

Context (please complete the following information)

  • OS Name & Version: n/a
  • AI-Action Version & Environment: v1.1.1 production
  • Browser Version: n/a
  • Target Page: n/a

Are you willing to submit a PR?

Yes

Did you search for similar existing issues?

yes

Additional context

n/a

Missing reference docs for task/action inputs

Describe the bug

Right now, there are several inputs for both the ADO task and action which our usage docs never mention (eg, we never document how a user should decide whether or not to specify chromePath). We should go through every input and ensure that each of them has at least some usage documentation.

To Reproduce

n/a

CodePen repro example

n/a

Expected behavior

All inputs should be documented.

Screenshots

n/a

Context (please complete the following information)

  • OS Name & Version: n/a
  • AI-Action Version & Environment: v1.1.1 production
  • Browser Version: n/a
  • Target Page: n/a

Are you willing to submit a PR?

Yes

Did you search for similar existing issues?

Yes

Additional context

n/a

Emoji in ADO UploadSummary are shown as string values

Describe the bug

When the ADO extension completes a job and finds no issues, the summary contains emoji that are not being converted to images and remain as strings: :white_check_mark: and :point_right:.

To Reproduce
Steps to reproduce the behavior:

  1. Visit a recent pipeline, such as docs
  2. Review the summary to see that :white_check_mark: is a string and not show as ✅ and :point_right: is a strong and is not shown as 👉

CodePen repro example

n/a

Expected behavior

The emoji should be an image, not a string. However, since the emoji are decorative, removing them may be the ideal option.

Screenshots

Screenshot of the ADO summary where text shows emoji as string: :white_check_mark: No failures detected :point_right: Next step: Manually assess keyboard accessibility with Accessibility Insights Tab Stops)

Context (please complete the following information)

  • OS Name & Version:
  • AI-Action Version & Environment:
  • Browser Version:
  • Target Page:

Are you willing to submit a PR?

Yes

Did you search for similar existing issues?

Yes

Additional context

Reduce onboarding complexity by replacing dependency on yarn

Describe the bug

This issue reflects some conversation on our support thread and with @dbjorge

yarn is a an undocumented dependency of the Azure DevOps extension task. This is because the task installs its dependencies at runtime using the yarn package manager. While Microsoft hosted pool agents include yarn by default, some build environments don't have yarn installed. This increases onboarding complexity because users need to debug the error and add yarn to their pipeline themselves.

Users also need node >= 10.x. This is because the puppeteer install script (and perhaps others) run 'node install.js'. The ADO task execution handler runs node10 but installing node6 for example, may prefix the PATH and fail - puppeteer uses async syntax which isn't supported until 10.x.

Constraints

We install dependencies at runtime instead of pre-building them for a few reasons:

  • VSIX files have a maximum size limit of 50 MB. The documentation suggests a mitigation: "Fetch things at runtime or during install time rather than providing it within the package."
  • Some deps like puppeteer have different postinstall steps based on the OS platform; we might need to create and maintain several bundles

Current approach

We maintain a yarn.lock file that represents the required runtime dependencies. We assume yarn exists on the build agent and invoke yarn install towards the beginning of task execution.

Possible improvements

  • #1123
  • ADO documentation also suggests "Consider using the tool installer lib to pull tool dependencies at runtime. Using the lib offers benefits where the tool is cached by version so for private agents, it won't get downloaded every build. We made it a lib so it can be used outside of tool installer tasks. But, the task won't work in disconnected scenarios (no internet), which should be in the description / docs for the task." See if leveraging the tool installer lib (sample here) would help us.
  • Switch from yarn to npm for runtime-installation since npm will exist more often in node environments than yarn will. We don't use npm for our own local development yet because of issues like npm/cli#4232 (comment) - we need override behavior. If we take this approach, Dan suggests converting our yarn.lock file to package-lock.json using something like https://github.com/imsnif/synp.
  • ...others?

Use npm Codecov package

Use npm Codecov package instead of the Codecov action since action uses bash uploader and bash uploader might be affected by a security vulnerability

Improve Release pipeline support

Is your feature request related to a problem? Please describe.

Today, if a user tries to add the task to a release pipeline, they get a confusing error message when the task attempts to upload the results artifact. We have an internal FAQ on how to avoid the error, but it's not thrilling that the error message when you try it naively is unactionable.

Describe the desired outcome

Ideally, I think we'd detect that we're running in a release and automatically switch to using ##vso[task.uploadfile] rather than uploading an artifact

Describe alternatives you've considered

If there are issues with doing uploadfile, a smaller improvement would be to automatically detect that we're in a release and early-exit with a more actionable error message (similar to the FAQ, or maybe pointing to the FAQ) explaining what the user needs to configure.

Additional context

n/a

WCAG 1.1.1: Ensures <img> elements have alternate text or a role of none or presentation (img[width="1"])

Issue

Ensures <img> elements have alternate text or a role of none or presentation (image-alt)

Target application

Welcome to CityLights! [Inaccessible Home Page]

Element path

img[width="1"]

Snippet

<img src="./img/marker2_t.gif" width="1" height="30px">

How to fix

Fix any of the following:
  Element does not have an alt attribute
  aria-label attribute does not exist or is empty
  aria-labelledby attribute does not exist, references elements that do not exist or references elements that are empty
  Element has no title attribute
  Element's default semantics were not overridden with role="none" or role="presentation"

Environment

Microsoft Edge version 101.0.1210.47
This accessibility issue was found using Accessibility Insights for Web 2.32.0 (axe-core 4.4.1), a tool that helps find and fix accessibility issues. Get more information & download this tool at http://aka.ms/AccessibilityInsights.

crawler skip links nested within a shadow tree

Describe the bug

running the action on a website with links that are nested within shadow tree just skips visiting those links and only visits links nested within the light tree

CodePen repro example

it's not a repro but the actual site (preview channel) where this occurs and shadow links are skipped
https://vivid-j0vxkmsqmhn0.deno.dev/

Expected behavior

Shadowed links should be visited and checked

Screenshots

this gets visited
image

this doesn't
image

both are equivalent according to the accessibility tree view (devtools) and share the same computed properties
image

Are you willing to submit a PR?

Of course, with a little guidance

Did you search for similar existing issues?

yes, none to be found

Additional context

this bug might no be related to the GH action but to the underlying accessibility insights service.
forward to the relevant repo if need

index.js files are not checked in from a build machine

The GitHub action runs index.js. However, this file is not built & checked in from a build machine. Instead, the file is built on developer machines & checked in via the PR process. Changes in this file are difficult to review because the file is bundled via webpack; unrelated updates in third-party components can show up in the diff.

One approach to solve this problem is to build index.js in a pipeline and push it to a separate "release" repository or branch. Users would reference the "release" repository in consuming pipelines. This repository's source tree would only contain TypeScript files.

[ADO extension] Inconsistent/incorrect failure instance counts in PR comments

(Discovered during release validation)

Failure instance counts don't show as expected in ADO PR comments

  • They should show only a list of new failures when there is a baseline file that contains some of the failures found during a scan
  • When there is no baseline file, the total failure instance count shows the number of failure groups (i.e. when opening a report, this is the number of cards) instead of the total number of instances for each rule (if a given card in the report has x URLs listed, it should count x failure instances for that card, not 1 failure instance for only the card).

Emit a warning if the base URL leads to an empty page

We met with a potential user today who initially configured their ADO pipeline with a url parameter that was slightly off, resulting in the scanner seeing a blank page. The scan appeared successful, but wasn't really scanning anything meaningful.

It would be nice for the action to detect this case and warn about the probable misconfiguration. Maybe something along the lines of "empty body" and/or "4xx/5xx response code" to trigger a warning. It should probably either only apply to the base URL, be suppressable, or both.

Axecore update for Azure Devops extension

We are checking the ADO extension and noticed that the axecore has an old version: axe-core 4.3.2
how can we update it to the axe-core 4.4.1 or latest 4.4.2?
Thanks

Task log output is too verbose, difficult to read

Summary

The task log output from both the ADO extension and GitHub action include a ton of noise that users shouldn't need to parse through; the actual actionable information from a run is buried in the middle of the log and not highlighted meaningfully.

Pipelines and Actions both support specific syntax for stdout messages to enable different types of grouping/highlighting/etc; while designing a logger to support the suggestions in this issue, be sure to consider how to handle both systems' different syntax.

Actual vs expected

Actual output from an example run with a required baseline update (case 4 of ado-extension-staging-validation) - 169 lines
2021-11-02T19:50:08.9814274Z ##[section]Starting: [should fail] case 4: baseline missing failures
2021-11-02T19:50:08.9821025Z ==============================================================================
2021-11-02T19:50:08.9821509Z Task         : Accessibility Insights Azure DevOps Task
2021-11-02T19:50:08.9821883Z Description  : Scan accessibility issues in an Azure DevOps pipeline
2021-11-02T19:50:08.9822179Z Version      : 1.1.0
2021-11-02T19:50:08.9822455Z Author       : Accessibility Insights
2021-11-02T19:50:08.9822703Z Help         : 
2021-11-02T19:50:08.9822998Z ==============================================================================
2021-11-02T19:50:09.1991936Z installing runtime dependencies...
2021-11-02T19:50:09.3385233Z yarn install v1.22.17
2021-11-02T19:50:09.4468120Z [1/4] Resolving packages...
2021-11-02T19:50:09.4777135Z warning Lockfile has incorrect entry for "apify-shared@^0.5.0". Ignoring it.
2021-11-02T19:50:09.5120446Z warning Lockfile has incorrect entry for "socket.io@^2.3.0". Ignoring it.
2021-11-02T19:50:09.5122034Z warning Lockfile has incorrect entry for "[email protected]". Ignoring it.
2021-11-02T19:50:09.5385119Z warning Lockfile has incorrect entry for "apify-shared@^0.6.0". Ignoring it.
2021-11-02T19:50:09.5403086Z warning Lockfile has incorrect entry for "apify-shared@^0.1.45". Ignoring it.
2021-11-02T19:50:09.5796664Z warning Lockfile has incorrect entry for "http-signature@~1.2.0". Ignoring it.
2021-11-02T19:50:09.5822215Z warning Lockfile has incorrect entry for "[email protected]". Ignoring it.
2021-11-02T19:50:09.5824908Z warning Lockfile has incorrect entry for "nth-check@~1.0.1". Ignoring it.
2021-11-02T19:50:09.7566166Z warning accessibility-insights-scan > apify > socket.io > [email protected]: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
2021-11-02T19:50:09.7651204Z warning accessibility-insights-scan > apify > socket.io > socket.io-parser > [email protected]: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
2021-11-02T19:50:09.7692334Z warning accessibility-insights-scan > apify > socket.io > engine.io > [email protected]: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
2021-11-02T19:50:10.1510415Z warning Lockfile has incorrect entry for "axios@^0.21.1". Ignoring it.
2021-11-02T19:50:10.1511332Z warning Lockfile has incorrect entry for "marked@^2.0.0". Ignoring it.
2021-11-02T19:50:10.6201531Z [2/4] Fetching packages...
2021-11-02T19:50:10.8566642Z [3/4] Linking dependencies...
2021-11-02T19:50:10.8575211Z warning "accessibility-insights-report > [email protected]" has unmet peer dependency "@types/react@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8576645Z warning "accessibility-insights-report > [email protected]" has unmet peer dependency "@types/react-dom@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8586296Z warning "accessibility-insights-report > office-ui-fabric-react > @fluentui/[email protected]" has unmet peer dependency "@types/react@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8587662Z warning "accessibility-insights-report > office-ui-fabric-react > @fluentui/[email protected]" has unmet peer dependency "@types/react-dom@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8590382Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/[email protected]" has unmet peer dependency "@types/react@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8591727Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/[email protected]" has unmet peer dependency "@types/react-dom@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8594265Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/[email protected]" has unmet peer dependency "@types/react@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8595907Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/[email protected]" has unmet peer dependency "@types/react-dom@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8598361Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/[email protected]" has unmet peer dependency "@types/react@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8599922Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/[email protected]" has unmet peer dependency "@types/react-dom@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8603536Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/react-hooks > @fluentui/[email protected]" has unmet peer dependency "@types/react@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8605002Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/react-hooks > @fluentui/[email protected]" has unmet peer dependency "@types/react-dom@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8606870Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/styling > @fluentui/[email protected]" has unmet peer dependency "@types/react@>=16.8.0 <17.0.0".
2021-11-02T19:50:10.8609021Z warning "accessibility-insights-report > office-ui-fabric-react > @uifabric/styling > @fluentui/[email protected]" has unmet peer dependency "@types/react-dom@>=16.8.0 <17.0.0".
2021-11-02T19:50:12.4438052Z [4/4] Building fresh packages...
2021-11-02T19:50:12.4640699Z Done in 3.13s.
2021-11-02T19:50:14.0667002Z [Trace][info] === Starting accessibility scanning of URL http://localhost:5858
2021-11-02T19:50:14.0668807Z [Trace][info] === Chrome app executable: /usr/bin/google-chrome
2021-11-02T19:50:14.1113410Z INFO  PuppeteerCrawler:AutoscaledPool:Snapshotter: Setting max memory of this run to 1738 MB. Use the APIFY_MEMORY_MBYTES environment variable to override it.
2021-11-02T19:50:14.1138556Z INFO  PuppeteerCrawler:AutoscaledPool: state {"currentConcurrency":0,"desiredConcurrency":1,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"limitRatio":0.2,"actualRatio":null},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":null},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":null},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":null}}}
2021-11-02T19:50:14.1247887Z INFO  Launching Puppeteer {"args":["--disable-dev-shm-usage","--no-sandbox","--disable-setuid-sandbox","--js-flags=--max-old-space-size=8192","--no-sandbox","--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36"],"defaultViewport":{"width":1920,"height":1080,"deviceScaleFactor":1},"useChrome":true,"timeout":15000,"headless":true,"executablePath":"/usr/bin/google-chrome"}
2021-11-02T19:50:15.2863699Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:15.4885365Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:15.6899873Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:15.8998253Z Processing page http://localhost:5858/
2021-11-02T19:50:15.9069421Z Discovered 2 links on page http://localhost:5858/
2021-11-02T19:50:16.1213249Z Found 3 accessibility issues on page http://localhost:5858/
2021-11-02T19:50:17.1786109Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:17.3801515Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:17.5817595Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:17.7840499Z Processing page http://localhost:5858/linked1/
2021-11-02T19:50:17.7893439Z Discovered 2 links on page http://localhost:5858/linked1/
2021-11-02T19:50:18.9931340Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:19.1954982Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:19.3970696Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:19.5997987Z Processing page http://localhost:5858/linked2/
2021-11-02T19:50:19.6044738Z Discovered 1 links on page http://localhost:5858/linked2/
2021-11-02T19:50:20.7927508Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:20.9945812Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:21.1960170Z waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
2021-11-02T19:50:21.3989854Z Processing page http://localhost:5858/linked1/inner-page
2021-11-02T19:50:21.4034314Z Discovered 1 links on page http://localhost:5858/linked1/inner-page
2021-11-02T19:50:21.5728322Z Found 2 accessibility issues on page http://localhost:5858/linked1/inner-page
2021-11-02T19:50:21.5803110Z INFO  PuppeteerCrawler: All the requests from request list and/or request queue have been processed, the crawler will shut down.
2021-11-02T19:50:21.5824184Z INFO  PuppeteerCrawler: Final request statistics: {"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":1864,"requestsFinishedPerMinute":32,"requestsFailedPerMinute":0,"requestTotalDurationMillis":7454,"requestsTotal":4,"crawlerRuntimeMillis":7492,"requestsFinished":4,"requestsFailed":0,"retryHistogram":[4]}
2021-11-02T19:50:21.6436274Z Warning: Each child in a list should have a unique "key" prop.
2021-11-02T19:50:21.6437698Z 
2021-11-02T19:50:21.6439176Z Check the top-level render call using <ul>. See https://fb.me/react-warning-keys for more information.
2021-11-02T19:50:21.6442305Z     in li
2021-11-02T19:50:21.6442623Z     in SummaryReportDetailsSection
2021-11-02T19:50:21.6442929Z     in div
2021-11-02T19:50:21.6443185Z     in main
2021-11-02T19:50:21.6443472Z     in ContentSection
2021-11-02T19:50:21.6443757Z     in body
2021-11-02T19:50:21.6444016Z     in BodySection
2021-11-02T19:50:21.6444303Z     in ReportBody
2021-11-02T19:50:21.6780477Z [Trace][info] === Scan report saved successfully as /home/vsts/work/1/s/_accessibility-reports-case-4/index.html
2021-11-02T19:50:21.6785266Z Found 2 new violations compared to the baseline.
2021-11-02T19:50:21.6787446Z Found 0 cases where a previously baselined violation was fixed.
2021-11-02T19:50:21.6805177Z Saved new baseline file at /home/vsts/work/1/s/_accessibility-reports-case-4/missing-failures-5858.baseline
2021-11-02T19:50:21.6808175Z To update the baseline with these changes, either rerun with --updateBaseline or copy the updated baseline file to /home/vsts/work/1/s/dev/website-baselines/missing-failures-5858.baseline
2021-11-02T19:50:21.6849306Z [Exception] === { VError: An error occurred while scanning website page http://localhost:5858: Failed: The baseline file does not match scan results. If this is a PR, check the PR comments.
2021-11-02T19:50:21.6851053Z     at Logger.trackExceptionAny (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:132995:29)
2021-11-02T19:50:21.6852551Z     at Scanner.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133836:29)
2021-11-02T19:50:21.6853365Z     at Generator.throw (<anonymous>)
2021-11-02T19:50:21.6854334Z     at rejected (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133765:65)
2021-11-02T19:50:21.6855077Z   jse_shortmsg:
2021-11-02T19:50:21.6855923Z    'An error occurred while scanning website page http://localhost:5858',
2021-11-02T19:50:21.6856605Z   jse_cause:
2021-11-02T19:50:21.6857254Z    Error: Failed: The baseline file does not match scan results. If this is a PR, check the PR comments.
2021-11-02T19:50:21.6858444Z        at WorkflowEnforcer.failIfBaselineNeedsUpdating (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129208:19)
2021-11-02T19:50:21.6859691Z        at WorkflowEnforcer.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129192:18)
2021-11-02T19:50:21.6860497Z        at Generator.next (<anonymous>)
2021-11-02T19:50:21.6861449Z        at /home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129169:71
2021-11-02T19:50:21.6862197Z        at new Promise (<anonymous>)
2021-11-02T19:50:21.6863352Z        at __webpack_modules__../src/progress-reporter/enforcement/workflow-enforcer.ts.__awaiter (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129165:12)
2021-11-02T19:50:21.6864662Z        at WorkflowEnforcer.completeRun (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129190:16)
2021-11-02T19:50:21.6865847Z        at execute (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133401:45)
2021-11-02T19:50:21.6867034Z        at AllProgressReporter.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133419:27)
2021-11-02T19:50:21.6867841Z        at Generator.next (<anonymous>)
2021-11-02T19:50:21.6868774Z        at /home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133379:71
2021-11-02T19:50:21.6869520Z        at new Promise (<anonymous>)
2021-11-02T19:50:21.6870670Z        at __webpack_modules__.../shared/dist/progress-reporter/all-progress-reporter.js.__awaiter (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133375:12)
2021-11-02T19:50:21.6871973Z        at AllProgressReporter.execute (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133414:16)
2021-11-02T19:50:21.6875440Z        at AllProgressReporter.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133401:28)
2021-11-02T19:50:21.6876060Z        at Generator.next (<anonymous>)
2021-11-02T19:50:21.6876788Z        at /home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133379:71
2021-11-02T19:50:21.6877492Z        at new Promise (<anonymous>)
2021-11-02T19:50:21.6878432Z        at __webpack_modules__.../shared/dist/progress-reporter/all-progress-reporter.js.__awaiter (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133375:12)
2021-11-02T19:50:21.6879810Z        at AllProgressReporter.completeRun (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133399:16)
2021-11-02T19:50:21.6880811Z        at Scanner.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133833:48)
2021-11-02T19:50:21.6881387Z        at Generator.next (<anonymous>),
2021-11-02T19:50:21.6881761Z   jse_info: {},
2021-11-02T19:50:21.6882092Z   message:
2021-11-02T19:50:21.6882975Z    'An error occurred while scanning website page http://localhost:5858: Failed: The baseline file does not match scan results. If this is a PR, check the PR comments.' }
2021-11-02T19:50:21.6890015Z [Trace][info] === Accessibility scanning of URL http://localhost:5858 completed
2021-11-02T19:50:21.6897695Z Exception thrown in extension:  Error: Error: Failed: The baseline file does not match scan results. If this is a PR, check the PR comments.
2021-11-02T19:50:21.6898997Z     at WorkflowEnforcer.failIfBaselineNeedsUpdating (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129208:19)
2021-11-02T19:50:21.6900055Z     at WorkflowEnforcer.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129192:18)
2021-11-02T19:50:21.6900645Z     at Generator.next (<anonymous>)
2021-11-02T19:50:21.6901364Z     at /home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129169:71
2021-11-02T19:50:21.6901906Z     at new Promise (<anonymous>)
2021-11-02T19:50:21.6902835Z     at __webpack_modules__../src/progress-reporter/enforcement/workflow-enforcer.ts.__awaiter (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129165:12)
2021-11-02T19:50:21.6903915Z     at WorkflowEnforcer.completeRun (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129190:16)
2021-11-02T19:50:21.6904842Z     at execute (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133401:45)
2021-11-02T19:50:21.6905808Z     at AllProgressReporter.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133419:27)
2021-11-02T19:50:21.6906390Z     at Generator.next (<anonymous>)
2021-11-02T19:50:21.6907121Z     at /home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133379:71
2021-11-02T19:50:21.6907627Z     at new Promise (<anonymous>)
2021-11-02T19:50:21.6908559Z     at __webpack_modules__.../shared/dist/progress-reporter/all-progress-reporter.js.__awaiter (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133375:12)
2021-11-02T19:50:21.6909642Z     at AllProgressReporter.execute (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133414:16)
2021-11-02T19:50:21.6910636Z     at AllProgressReporter.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133401:28)
2021-11-02T19:50:21.6911216Z     at Generator.next (<anonymous>)
2021-11-02T19:50:21.6911925Z     at /home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133379:71
2021-11-02T19:50:21.6912449Z     at new Promise (<anonymous>)
2021-11-02T19:50:21.6913366Z     at __webpack_modules__.../shared/dist/progress-reporter/all-progress-reporter.js.__awaiter (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133375:12)
2021-11-02T19:50:21.6914634Z     at AllProgressReporter.completeRun (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133399:16)
2021-11-02T19:50:21.6915591Z     at Scanner.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133833:48)
2021-11-02T19:50:21.6916155Z     at Generator.next (<anonymous>)
2021-11-02T19:50:21.6916948Z     at WorkflowEnforcer.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129198:19)
2021-11-02T19:50:21.6917529Z     at Generator.next (<anonymous>)
2021-11-02T19:50:21.6918233Z     at /home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129169:71
2021-11-02T19:50:21.6918753Z     at new Promise (<anonymous>)
2021-11-02T19:50:21.6919810Z     at __webpack_modules__../src/progress-reporter/enforcement/workflow-enforcer.ts.__awaiter (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129165:12)
2021-11-02T19:50:21.6921026Z     at WorkflowEnforcer.failRun (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:129197:16)
2021-11-02T19:50:21.6921961Z     at execute (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133410:41)
2021-11-02T19:50:21.6922903Z     at AllProgressReporter.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133419:27)
2021-11-02T19:50:21.6923480Z     at Generator.next (<anonymous>)
2021-11-02T19:50:21.6924217Z     at /home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133379:71
2021-11-02T19:50:21.6924744Z     at new Promise (<anonymous>)
2021-11-02T19:50:21.6925646Z     at __webpack_modules__.../shared/dist/progress-reporter/all-progress-reporter.js.__awaiter (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133375:12)
2021-11-02T19:50:21.6926730Z     at AllProgressReporter.execute (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133414:16)
2021-11-02T19:50:21.6927711Z     at AllProgressReporter.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133410:24)
2021-11-02T19:50:21.6928289Z     at Generator.next (<anonymous>)
2021-11-02T19:50:21.6929013Z     at /home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133379:71
2021-11-02T19:50:21.6929520Z     at new Promise (<anonymous>)
2021-11-02T19:50:21.6930433Z     at __webpack_modules__.../shared/dist/progress-reporter/all-progress-reporter.js.__awaiter (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133375:12)
2021-11-02T19:50:21.6931512Z     at AllProgressReporter.failRun (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133409:16)
2021-11-02T19:50:21.6932485Z     at Scanner.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133837:48)
2021-11-02T19:50:21.6933035Z     at Generator.throw (<anonymous>)
2021-11-02T19:50:21.6933812Z     at rejected (/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js:133765:65)
2021-11-02T19:50:21.7300172Z ##[error]Exit code 1 returned from process: file name '/home/vsts/agents/2.194.0/externals/node10/bin/node', arguments '"/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js"'.
2021-11-02T19:50:21.7316103Z ##[section]Finishing: [should fail] case 4: baseline missing failures
An example of what I think this should look like - 46 lines, with grouping and highlighting
2021-11-02T19:50:08.9814274Z ##[section]Starting: [should fail] case 4: baseline missing failures
2021-11-02T19:50:08.9821025Z ==============================================================================
2021-11-02T19:50:08.9821509Z Task         : Accessibility Insights Azure DevOps Task
2021-11-02T19:50:08.9821883Z Description  : Scan accessibility issues in an Azure DevOps pipeline
2021-11-02T19:50:08.9822179Z Version      : 1.1.0
2021-11-02T19:50:08.9822455Z Author       : Accessibility Insights
2021-11-02T19:50:08.9822703Z Help         : 
2021-11-02T19:50:08.9822998Z ==============================================================================
2021-11-02T19:50:09.1991936Z ##[group]Installing runtime dependencies
2021-11-02T19:50:09.3385233Z yarn install v1.22.17
2021-11-02T19:50:09.4468120Z [1/4] Resolving packages...
2021-11-02T19:50:10.6201531Z [2/4] Fetching packages...
2021-11-02T19:50:10.8566642Z [3/4] Linking dependencies...
2021-11-02T19:50:12.4438052Z [4/4] Building fresh packages...
2021-11-02T19:50:12.4640699Z Done in 3.13s.
2021-11-02T19:50:12.4640699Z ##[endgroup]
2021-11-02T19:50:12.4640699Z ##[group]Scanning URL http://localhost:5858 for accessibility issues
2021-11-02T19:50:14.0667002Z ##[debug]Starting accessibility scanning of URL http://localhost:5858
2021-11-02T19:50:14.0668807Z ##[debug]Chrome app executable: /usr/bin/google-chrome
2021-11-02T19:50:14.1113410Z ##[debug]PuppeteerCrawler:AutoscaledPool:Snapshotter: Setting max memory of this run to 1738 MB. Use the APIFY_MEMORY_MBYTES environment variable to override it.
2021-11-02T19:50:14.1138556Z ##[debug]PuppeteerCrawler:AutoscaledPool: state {"currentConcurrency":0,"desiredConcurrency":1,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"limitRatio":0.2,"actualRatio":null},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":null},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":null},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":null}}}
2021-11-02T19:50:14.1247887Z ##[debug]Launching Puppeteer {"args":["--disable-dev-shm-usage","--no-sandbox","--disable-setuid-sandbox","--js-flags=--max-old-space-size=8192","--no-sandbox","--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36"],"defaultViewport":{"width":1920,"height":1080,"deviceScaleFactor":1},"useChrome":true,"timeout":15000,"headless":true,"executablePath":"/usr/bin/google-chrome"}
2021-11-02T19:50:15.8998253Z Processing page http://localhost:5858/
2021-11-02T19:50:15.9069421Z Discovered 2 links on page http://localhost:5858/
2021-11-02T19:50:16.1213249Z Found 3 accessibility issues on page http://localhost:5858/
2021-11-02T19:50:17.7840499Z Processing page http://localhost:5858/linked1/
2021-11-02T19:50:17.7893439Z Discovered 2 links on page http://localhost:5858/linked1/
2021-11-02T19:50:19.5997987Z Processing page http://localhost:5858/linked2/
2021-11-02T19:50:19.6044738Z Discovered 1 links on page http://localhost:5858/linked2/
2021-11-02T19:50:21.3989854Z Processing page http://localhost:5858/linked1/inner-page
2021-11-02T19:50:21.4034314Z Discovered 1 links on page http://localhost:5858/linked1/inner-page
2021-11-02T19:50:21.5728322Z Found 2 accessibility issues on page http://localhost:5858/linked1/inner-page
2021-11-02T19:50:21.5803110Z ##[debug]PuppeteerCrawler: All the requests from request list and/or request queue have been processed, the crawler will shut down.
2021-11-02T19:50:21.5824184Z ##[debug]PuppeteerCrawler: Final request statistics: {"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":1864,"requestsFinishedPerMinute":32,"requestsFailedPerMinute":0,"requestTotalDurationMillis":7454,"requestsTotal":4,"crawlerRuntimeMillis":7492,"requestsFinished":4,"requestsFailed":0,"retryHistogram":[4]}
2021-11-02T19:50:21.5824184Z ##[endgroup]
2021-11-02T19:50:21.6780477Z Scan report saved successfully as /home/vsts/work/1/s/_accessibility-reports-case-4/index.html
2021-11-02T19:50:21.6897695Z ##vso[task.logissue type=error;sourcepath=dev/website-baselines/missing-failures-5858.baseline]The baseline file does not match scan results.
2021-11-02T19:50:21.6897695Z ##[error]The baseline file does not match scan results.
2021-11-02T19:50:21.6785266Z Found 2 new violations compared to the baseline.
2021-11-02T19:50:21.6787446Z Found 0 cases where a previously baselined violation was fixed.
2021-11-02T19:50:21.6805177Z Saved new baseline file at /home/vsts/work/1/s/_accessibility-reports-case-4/missing-failures-5858.baseline
2021-11-02T19:50:21.6808175Z To update the baseline with these changes:
2021-11-02T19:50:21.6808175Z     1. Ensure you've published the output directory as an "accessibility reports" artifact, and
2021-11-02T19:50:21.6808175Z     2. Copy the updated baseline file from your artifact to dev/website-baselines/missing-failures-5858.baseline
2021-11-02T19:50:21.7300172Z ##[error]Exit code 1 returned from process: file name '/home/vsts/agents/2.194.0/externals/node10/bin/node', arguments '"/home/vsts/work/_tasks/accessibility-insights_aebf5565-d905-5999-8faf-a44d4db6c0fa/1.1.0/index.js"'.
2021-11-02T19:50:21.7316103Z ##[section]Finishing: [should fail] case 4: baseline missing failures

Specific suggested improvements

I think each of these would work fine as independent improvements, no need to wait until all of them are done to make incremental progress.

These are all written from the perspective of the ADO extension, but ideally any ## commands would want to be able to use GitHub Actions equivalents there.

  • Add ##[group]summary/##[endgroup] around the "Installing runtime dependencies" section (Fixed in #1040)
  • Add ##[group]summary/##[endgroup] around the "Running the scan" section (suggested name: Scanning URL http://the-url.com for accessibility issues) (Fixed in #1051)
  • Remove incorrect lockfile entries to fix the warning noise there
  • Add @types/react and @types/react-dom dependencies to suppress the noise from office-fabric (Fixed in #1038)
  • Intercept the log statements from apify/Puppeteer, and prepend ##[debug] instead of inconsistently using INFO and [Trace][info] === (Fixed in #1030 )
  • Either update apify to avoid the waitFor warning noise, or suppress it by intercepting Puppeteer's console output (Fixed in #1030 )
  • Intercept log statements from the report package and prepend ##[debug] (Fixed in #1030 )
  • Fix the Warning: Each child in a list should have a unique "key" prop. issue in the report package (Fixed in #1031 )
  • Drop the [Trace][info] === prefix from the "Scan report saved successfully" message (Fixed in #1030 )
  • Do not use exceptions to handle cases where the extension functions correctly but detects an accessibility/baseline problem. The stack trace is not useful to a user and it looks like we've crashed. (Fixed with #1046)
  • If we do encounter an exception, do not double-print exception stacks (Fixed with #1046)
  • For both "normal path" errors and exceptions, use ##[error] and/or ##vso[task.logissue type=error] to report it with appropriate highlighting (Fixed with #1046)
  • Particularly, for "baseline needs update" failures, use task.logissue's sourcepath argument to present the user-actionable error with error highlighting and file context. For example:
2021-11-02T19:50:21.6897695Z ##vso[task.logissue type=error;sourcepath=dev/website-baselines/missing-failures-5858.baseline]The baseline file does not match scan results.

(Fixed in #1075 )

  • Improve the action callout for the "baseline update required" case by replacing "If this is a PR, check the PR comments." with something that is actionable in the non-PR case, eg:
2021-11-02T19:50:21.6808175Z To update the baseline with these changes:
2021-11-02T19:50:21.6808175Z     1. Ensure you've published the output directory as an "accessibility reports" artifact, and
2021-11-02T19:50:21.6808175Z     2. Copy the updated baseline file from your artifact to dev/website-baselines/missing-failures-5858.baseline

(Fixed in #909)

ADO task does not fail on a11y issues in non-PR builds

The ADO build task doesn't fail on accessibility errors in our manually-triggered pipeline when failOnAccessibilityErrors is set to true. @karanbirsingh pointed out that the failOnAccessibilityError function only appears in the pull request comment creator, so it is expected to also not fail on errors in CI pipelines.

To reproduce:

  1. Create an ADO build pipeline with the ADO extension task and a url with known accessi
  2. set failOnAccessibilityError: true and url: https://www.washington.edu/accesscomputing/AU/before.html (or some other page with known a11y failures)
  3. Trigger the build manually. The logs will show that failures were found, but the build will not break.

Docs should include links to working examples/demos

One of our Azure DevOps pilot teams suggested that they would have liked to see our onboarding docs link to working demo/example pipelines, so they could get a better idea of what exactly a working configuration looks like. This is a good idea (we do this already in https://github.com/microsoft/axe-pipelines-samples repo) which shouldn't be too hard to add, and besides being good for documentation it would be nice to have as a verification step that production is working as expected.

Cannot add task in a classic pipeline without setting Authentication type

Describe the bug

A user reported that it's not possible to add the AI task to a classic pipeline without setting the Authentication Type. If you do not need authentication and are required to select the Authentication Type, then the task will fail with the error "--serviceAccountName, --serviceAccountPassword, and --authType all must be provided to scan authenticated pages."

To Reproduce
Steps to reproduce the behavior:

  1. Go to ADO
  2. Click "New Pipeline"
  3. Click "Use the classic editor"
  4. Click "Continue"
  5. Click "Empty job"
  6. Click the "+" On Agent Job 1
  7. Add the AI Task
  8. Click on the AI task and note that Authentication Type has an error "This setting is invalid. Select a valid option from the dropdown." and it's not possible to complete the task without selecting "Azure Active Directory" from the pipeline. The screenshot below illustrates this error. The dropdown has no other options.

CodePen repro example

n/a

Expected behavior

The user can submit an empty value for Authentication type if they do not require auth.

Screenshots

Included in reproduction steps.

Context (please complete the following information)

Classic ADO pipelines

Are you willing to submit a PR?

Yes

Did you search for similar existing issues?

Yes

Additional context

Scan output is not developer visible

Describe the bug

Output is not developer visible

Expected behavior

some parsing of the output results via test visualization or CLI output.

Are you willing to submit a PR?

no

Did you search for similar existing issues?

yes, might be related to #950

Additional context

User reported

Angle brackets in rule description not escaped in scan result markdown

Visit https://github.com/asksep/test-ai-action/runs/2657373953?check_suite_focus=true

Note the third failed instance on this page reads

1 × image-alt: Ensures elements have alternate text or a role of none or presentation

It should read

1 × image-alt: Ensures <img> elements have alternate text or a role of none or presentation

Viewing the page source finds a linked image between Ensures and elements:

1 × image-alt</strong>: Ensures <a target="_blank" rel="noopener noreferrer" href=""><img style="max-width:100%;"></a> elements

Running AI-Web on this page, entertainingly enough, spots an image-alt violation (the generated image has no alternate text or role) and a link-name violation (images in markdown are links to the image file). The empty linked image also creates an invisible tab stop.

Markdown generation should be reviewed to identify any other data or characters that need to be escaped.

Action does not pin the versions of dependencies installed at runtime

The first step of our composite action.yml is currently:

        - name: Install npm packages
          run: |
              echo 'Installing npm packages...'
              cd $GITHUB_ACTION_PATH/dist && npm install
              echo 'npm packages successfully installed'
          shell: bash

The npm install command isn't using a lockfile, which is problematic; it means we (and users) can be broken if any transitive dependency issues a broken update. This is true even if the user pins to a specific version of the action.

Note that converting dist/package.json to use non-caret versions is not sufficient (that doesn't solve the issue for transitive dependencies).

Easiest solution is probably to copy yarn.lock into /dist/ as part of the build and switch the install step to use yarn install --frozen-lockfile, but that might have performance implications; on my machine, this is much much slower than npm install. Need to perf test.

Accessibility-insights for azure devops not working and throw exception

Describe the bug

The accessibility task was working very well but suddnly it start report this error:
node:internal/modules/cjs/loader:1210
return process.dlopen(module, path.toNamespacedPath(filename));

image

Added a new task to install NodeJS version 16 but unfortunately we got a different error:
ERROR PuppeteerCrawler: handleRequestFunction failed, reclaiming failed request back to the list or queue {"url":"https://shopin3d-ppe.azurefd.net/en-us/store/shop-in-3d/surfaceprox.html","retryCount":1,"id":"HuyCQqyoK9MYjRm"}
Error: Navigation timed out after 60 seconds.
at handleRequestTimeout (D:\a_tasks\accessibility-insights_4811a442-2fd3-5aa8-ba1a-14cb7e24c113\3.2.0\node_modules\apify\build\crawlers\crawler_utils.js:19:11)
at PuppeteerCrawler._handleNavigationTimeout (D:\a_tasks\accessibility-insights_4811a442-2fd3-5aa8-ba1a-14cb7e24c113\3.2.0\node_modules\apify\build\crawlers\browser_crawler.js:418:54)
at PuppeteerCrawler._handleNavigation (D:\a_tasks\accessibility-insights_4811a442-2fd3-5aa8-ba1a-14cb7e24c113\3.2.0\node_modules\apify\build\crawlers\browser_crawler.js:401:18)
at runNextTicks (node:internal/process/task_queues:61:5)
at processTimers (node:internal/timers:499:9)
at async PuppeteerCrawler._handleRequestFunction (D:\a_tasks\accessibility-insights_4811a442-2fd3-5aa8-ba1a-14cb7e24c113\3.2.0\node_modules\apify\build\crawlers\browser_crawler.js:343:13)
at async wrap (D:\a_tasks\accessibility-insights_4811a442-2fd3-5aa8-ba1a-14cb7e24c113\3.2.0\node_modules@apify\timeout\index.js:73:27)
image

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '...'
  3. Scroll down to '...'
  4. See error

CodePen repro example

Expected behavior

Screenshots

Context (please complete the following information)

  • OS Name & Version:
  • Azure DevOps Extension Version & Environment:
  • Browser Version:
  • Target Page:

Are you willing to submit a PR?

Did you search for similar existing issues?

Additional context

Emit warning or error when the extension unknowingly scans the login page

Is your feature request related to a problem? Please describe.

It's possible that a user can set up the extension and receive a passing scan, when in reality the extension is scanning the login page. This can happen if the user isn't aware that their site requires authentication.

The only way to know that the extension scanned the login page is if you open the logs.

Example

I configured the extension to scan the url https://eng.ms/ without adding authentication.

The package report states that it scanned my desired target URL (https://eng.ms/) and passed:

Screenshot of the report package. It shows Target site https://eng.ms/. 1 total URLs scanned. 0 Failure instances were detected.

The logs, when not expanded, appear to show the same result; that my desired target URL (https://eng.ms/) and passed:

Screenshot of logs where Scanning URL https://eng.ms/ and No failures detected

However, when the log are expanded it reveals the Puppeteer actually processed the login page:

Screenshot of expanded log where Processing page  is "https://login.microsoftonline.com/"

Describe the desired outcome

The extension could emit a warning when there is a single scanned page and it's the login page. Ideally, this would be an error, but we'd need to consider teams who may need to scan the login page.

Describe alternatives you've considered

Additional context

ADO Extension: PR comments are missing 2 cases

We've identified the 2 following cases that are poorly represented in the PR comments that are created by ADO extension version 1.1.1:

Scenario 1: PR corrects some (but not all) violations

This occurs when:

  1. A baseline file exists
  2. The PR addresses some (but not all) of the violations in the baseline
  3. No new violations are introduced

The current code generates the "no new failures" PR message, which makes no mention of changes or updating the baseline file. A more appropriate PR comment would include verbiage about updating the baseline file, as well as a summary of what violations have been removed. These comments should help the PR author understand if the actual results of the PR match the anticipated results of the PR.

Scenario 2: PR modifies some violations without fixing them

This occurs when:

  1. A baseline file exists
  2. The PR includes a change that removes some existing violations but creates new ones. Since violations are specific to both the URL and HTML, this could happen if the HTML around a violation changes or if a page containing violations gets a different URL.

The current code generates the "new failures found" PR message, with no comment about the comments that have been fixed. It tells the user to update the PR, but makes no mention of the violations that no longer exist.

Action does not support being run multiple times from the same workflow

Describe the bug

The root cause of this is a variation of #631, but I wanted to file a separate issue since the symptom/use case is different.

Today, the action unilaterally adds a check run with a specific name ("Accessibility Checks") to report a summary of results, but if you have two different jobs in the same workflow both attempting to report results simultaneously, they can get confused and give you a confusing error when they both try to write to the same check, like in this example.

log output from a task failing in this way
Creating check run with status as in progress
Started local web server. Url: http://localhost:50381 Root directory: D:\a\accessibility-insights-action\accessibility-insights-action/dev/website-root
Scanning URL c:/Program%20Files/Git/
  Error: [Exception]ErrorWithCause: An error occurred while scanning website page c:/Program%20Files/Git/
      at Logger.trackExceptionAny (D:\a\accessibility-insights-action\accessibility-insights-action\packages\gh-action\dist\index.js:73815:29)
      at Scanner.<anonymous> (D:\a\accessibility-insights-action\accessibility-insights-action\packages\gh-action\dist\index.js:74874:29)
      at Generator.next (<anonymous>)
      at fulfilled (D:\a\accessibility-insights-action\accessibility-insights-action\packages\gh-action\dist\index.js:74799:58)
      at processTicksAndRejections (internal/process/task_queues.js:97:5)
  caused by: TypeError: Cannot read property 'baseUrl' of undefined
      at Scanner.getCombinedReportParameters (D:\a\accessibility-insights-action\accessibility-insights-action\packages\gh-action\dist\index.js:74887:60)
      at Scanner.<anonymous> (D:\a\accessibility-insights-action\accessibility-insights-action\packages\gh-action\dist\index.js:748[66](https://github.com/microsoft/accessibility-insights-action/runs/5726864137?check_suite_focus=true#step:6:66):55)
      at Generator.next (<anonymous>)
      at fulfilled (D:\a\accessibility-insights-action\accessibility-insights-action\packages\gh-action\dist\index.js:[74](https://github.com/microsoft/accessibility-insights-action/runs/5726864137?check_suite_focus=true#step:6:74)[79](https://github.com/microsoft/accessibility-insights-action/runs/5726864137?check_suite_focus=true#step:6:79)9:58)
      at processTicksAndRejections (internal/process/task_queues.js:97:5)
  Updating check run with status as failed
  Accessibility scanning of URL c:/Program%20Files/Git/ completed
  Error: Process completed with exit code 1.

To Reproduce
Steps to reproduce the behavior:

  1. Create a workflow which runs multiple jobs that each invoke the action. For example, a job with a strategy matrix:
jobs:
  example-job:
    strategy:
      matrix:
        os: ["ubuntu-latest", "windows-latest"]
    runs-on: ${{ matrix.os }}
    steps:
    # ...
    - name: Scan test site
      uses: accessibility-insights-action@v2
      with:
        repo-token: ${{ secrets.GITHUB_TOKEN }}
        site-dir: ${{ github.workspace }}/dev/website-root
  1. Run it
  2. Observe the failure above

Expected behavior

Running a scan from within a matrix strategy should "just work" with no special extra work from users.

Screenshots

n/a

Context (please complete the following information)

  • OS Name & Version: n/a
  • AI-Action Version & Environment: Dev build from main as of writing, @b6d3472
  • Browser Version: n/a
  • Target Page: n/a

Are you willing to submit a PR?

sure

Did you search for similar existing issues?

yes

Additional context

This will probably be solved incidentally by the same work that resolves #631

[ADO] potential v2.x breaking changes

This is a tracking issue with breaking changes that we would like to consider the next time we make an update to a new major version of the Azure DevOps extension.

  • Remove the Service Connection parameter (as part of dropping PR comment support) #1045
  • Embed the "upload output artifact" step into the task, rather than requiring that the user add it separately #1085
  • Options which users should rarely have to override should move to the bottom and be hidden by default under an "Advanced options" group which is collapsed by default. chromePath should go under here, and assuming we embed the "upload output artifact" step, so should outputDir #1092
  • failOnAccessibilityError should default to true instead of false
  • (see #1068) Modify the names/grouping of siteDir, scanUrlRelativePath, localhostPort, and url inputs to make it harder for users to accidentally configure mutually exclusive combinations of inputs #1122
    • Note: inputFile, inputUrls, and discoveryPatterns are not mutually exclusive with one another and should not be forced into separate groups like this. Their help text should be improved to clarify this, but improving their help text is non-breaking.

Abstract error message thrown when crawler encounters browser errors

Describe the bug

When the crawler encounters a browser error, it will often throw the following error:

[error][Exception]ErrorWithCause: An error occurred while scanning website page
....
caused by: TypeError: Cannot read properties of undefined (reading 'baseUrl')
    at Scanner.getCombinedReportParameters (/home/vsts/work/_tasks/accessibility-insights_4811a442-2fd3-5aa8-ba1a-14cb7e24c113/3.0.0/index.js:191027:60)
    at Scanner.<anonymous> (/home/vsts/work/_tasks/accessibility-insights_4811a442-2fd3-5aa8-ba1a-14cb7e24c113/3.0.0/index.js:191005:55)
    at Generator.next (<anonymous>)
    at fulfilled (/home/vsts/work/_tasks/accessibility-insights_4811a442-2fd3-5aa8-ba1a-14cb7e24c113/3.0.0/index.js:190928:58)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

I believe this is happening because the combinedScanResult in scanner.ts returns an errors array which does not get evaluated and elevated to the logger. As a result, the getCombinedReportParameters attempts to create the report metadata, but because it doesn't exist, it fails when it attempts to reach the baseUrl. The error message thrown is abstracted from the actual problem - the crawler isn't getting a page response.

To Reproduce

I created an example build

CodePen repro example

n/a

Expected behavior

The scanner evaluates the errors array and called this.logger.error() for each error. This will return an improved error message from the scanner: Unable to get a page response from the browser

Prior art from the service:

Screenshots

Context (please complete the following information)

  • OS Name & Version:
  • AI-Action Version & Environment: ADO@v3
  • Browser Version:
  • Target Page:

Are you willing to submit a PR?

Yes

Did you search for similar existing issues?

Additional context

@jlperkins and I debugged two different pipelines with this error over the past week.

Links to images being failed with `html-has-lang` and `image-alt` when scanning local files

Describe the bug

It seems like when you use the action to scan local files then links to image files are resulting in false positives.

My guess is that this could be because loading files from a directory does not provide MIME types like they would get if scanning over HTTP, or how the plain images are rendered by the browser.

To Reproduce

  1. Given a page that has a link to a local image from another image (eg. like a thumbnail image that has a link that lets you load the full image
  2. Run the scanner on the original page
  3. Notice errors are returned for the URL to the image.

eg.

Calling the action in a workflow like this:

      - uses: actions/checkout@v2

      - uses: ./.github/actions/jekyll-build-pages
        with:
          verbose: false

      - run: |
          sudo find -type f ! -regex ".*\.\(html\|svg\|gif\|css\|jpg\|png\)" -delete
        name: Remove non-HTML
        
        # https://github.com/microsoft/accessibility-insights-action
      - name: Scan for accessibility issues
        uses: microsoft/accessibility-insights-action@v2
        with:
            repo-token: ${{ secrets.GITHUB_TOKEN }}
            site-dir: ${{ github.workspace }}/_site
            scan-timeout: 6000000
            max-urls: 1500
            localhost-port: 12345
            scan-url-relative-path: /

and there's one or more pages that render HTML like this:

<a href="/images/fullimage.png"><img src="thumbnail.png"></a>

Expected behavior

Non-HTML pages should not be scanned

Screenshots

image
image

Context (please complete the following information)

  • OS Name & Version: ubuntu-latest
  • AI-Action Version & Environment: v2

Are you willing to submit a PR?

Depends how easy it would be to fix

Action is incompatible with `pull_request` triggers from forks/dependabot

Currently, if you trigger the action via a forked repo (or a native dependabot PR), the GITHUB_TOKEN provided to the workflow does not have write permissions for checks, so you get an error from the action when it receives a 403 response during its check creation/update requests.

The full error signature looks like this
[Exception] === VError: An error occurred while scanning website page undefined: Resource not accessible by integration
    at Logger.trackExceptionAny (/home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:70931:29)
    at Scanner.<anonymous> (/home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:71922:29)
    at Generator.throw (<anonymous>)
    at rejected (/home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:71853:65)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  jse_shortmsg: 'An error occurred while scanning website page undefined',
  jse_cause: RequestError [HttpError]: Resource not accessible by integration
      at /home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:5508:23
      at processTicksAndRejections (internal/process/task_queues.js:95:5) {
    status: 403,
    headers: {
      'access-control-allow-origin': '*',
      'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset',
      connection: 'close',
      'content-encoding': 'gzip',
      'content-security-policy': "default-src 'none'",
      'content-type': 'application/json; charset=utf-8',
      date: 'Tue, 25 May 2021 21:04:51 GMT',
      'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
      server: 'GitHub.com',
      'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
      'transfer-encoding': 'chunked',
      vary: 'Accept-Encoding, Accept, X-Requested-With',
      'x-content-type-options': 'nosniff',
      'x-frame-options': 'deny',
      'x-github-media-type': 'github.v3; format=json',
      'x-github-request-id': '0400:43D4:3336731:636CA48:60AD6673',
      'x-ratelimit-limit': '15000',
      'x-ratelimit-remaining': '14994',
      'x-ratelimit-reset': '1621979696',
      'x-ratelimit-resource': 'core',
      'x-ratelimit-used': '6',
      'x-xss-protection': '0'
    },
    request: {
      method: 'POST',
      url: 'https://api.github.com/repos/microsoft/accessibility-insights-action/check-runs',
      headers: {
        accept: 'application/vnd.github.v3+json',
        'user-agent': 'octokit-rest.js/18.5.3 octokit-core.js/3.4.0 Node.js/14.17.0 (linux; x64)',
        authorization: 'token [REDACTED]',
        'content-type': 'application/json; charset=utf-8'
      },
      body: '{"name":"Accessibility Checks","status":"in_progress","head_sha":"d396568bd3c95e53e11a3f79d34487240174ca31"}',
      request: { hook: [Function: bound bound register] }
    },
    documentation_url: 'https://docs.github.com/rest/reference/checks#create-a-check-run'
  },
  jse_info: {}
}
[Trace][info] === Updating check run with status as failed
[Trace][info] === Accessibility scanning of URL undefined completed
Exception thrown in action:  TypeError: Cannot read property 'id' of undefined
    at CheckRunCreator.<anonymous> (/home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:71360:50)
    at Generator.next (<anonymous>)
    at /home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:71293:71
    at new Promise (<anonymous>)
    at __webpack_modules__../src/progress-reporter/check-run/check-run-creator.ts.__awaiter (/home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:71289:12)
    at /home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:71356:37
    at CheckRunCreator.<anonymous> (/home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:71444:49)
    at Generator.next (<anonymous>)
    at /home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:71423:71
    at new Promise (<anonymous>)
    at __webpack_modules__../src/progress-reporter/progress-reporter.ts.__awaiter (/home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:71419:12)
    at CheckRunCreator.invoke (/home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:71443:16)
    at CheckRunCreator.<anonymous> (/home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:71356:24)
    at Generator.next (<anonymous>)
    at /home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:71293:71
    at new Promise (<anonymous>)
    at __webpack_modules__../src/progress-reporter/check-run/check-run-creator.ts.__awaiter (/home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:71289:12)
    at CheckRunCreator.failRun (/home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:71352:16)
    at /home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:71242:41
    at AllProgressReporter.<anonymous> (/home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:71249:23)
    at Generator.next (<anonymous>)
    at /home/runner/work/accessibility-insights-action/accessibility-insights-action/dist/index.js:71215:71
Error: Process completed with exit code 1.

Uploading a build artifact is allowed with the read-only token permissions, so the workaround I've seen other similar actions (eg, dorny/test-reporter's suggestion) use is to have the user include the action twice; once in a on: pull_request workflow that runs the scan and uploads the results as a build artifact, and a second time in a separate on: workflow_run workflow that triggers when the first workflow completes, runs using a more permissive GITHUB_TOKEN, and adds a check based on the build artifact from the on: pull_request results.

Ideally, we'd:

Surface Accessibility Reports Artifacts via Email

Describe the bug

Filing issue on behalf of user based on feedback:
"It would be awesome to be able to surface the accessibility report artifacts via email, like Azure pipeline notifications, so we don't have to keep digging for them."

Problem

Accessibility Insights ADO Extension does not have a way to surface reports via email.

Ask

Create a method to surface deduplicated Accessibility Reports Artifacts via email.

Examples:

Azure Pipeline Notifications

WCAG 1.1.1: Ensures <img> elements have alternate text or a role of none or presentation (img[width="1"])

Issue

Ensures <img> elements have alternate text or a role of none or presentation (image-alt)

Target application

Welcome to CityLights! [Inaccessible Home Page]

Element path

img[width="1"]

Snippet

<img src="./img/marker2_t.gif" width="1" height="30px">

How to fix

Fix any of the following:
  Element does not have an alt attribute
  aria-label attribute does not exist or is empty
  aria-labelledby attribute does not exist, references elements that do not exist or references elements that are empty
  Element has no title attribute
  Element's default semantics were not overridden with role="none" or role="presentation"

Environment

Microsoft Edge version 101.0.1210.47
This accessibility issue was found using Accessibility Insights for Web 2.32.0 (axe-core 4.4.1), a tool that helps find and fix accessibility issues. Get more information & download this tool at http://aka.ms/AccessibilityInsights.

ADO extension fails when run in a container: handleRequestFunction failed, { message: 'read ECONNRESET' }

Describe the bug

When run in a container, the ADO extension fails with:

 handleRequestFunction failed, reclaiming failed request back to the list or queue

It will sometimes include additional error messages, such as:

  • ProtocolError: Protocol error (Target.setAutoAttach): Target closed.
  • caused by: Error: "{ message: 'read ECONNRESET' }"

To Reproduce

You can reproduce the error by running the extension in a container. https://learn.microsoft.com/en-us/azure/devops/pipelines/process/container-phases?view=azure-devops#single-job

pool:
  vmImage: 'windows-2019'

container: mcr.microsoft.com/windows/servercore:ltsc2019

steps:
- script: set

- task: NodeTool@0
  inputs:
      versionSpec: '16.x'

- script: npm install [email protected] -g
  displayName: install yarn as a global dependency
    
- task: accessibility-insights.prod.task.accessibility-insights@3
  displayName: '[should fail] staging accessibility insights task against URL with known failures'
  inputs:
    url: 'https://www.washington.edu/accesscomputing/AU/before.html'

Repro example

Expected behavior

According to Running Puppeteer in Docker, Troubleshooting there are likely extra steps we should recommend to the user.

It's not clear to me if there's anything we can do on the extension side or if this can be helped with added documentation

Screenshots

Context (please complete the following information)

  • OS Name & Version:
  • Azure DevOps Extension Version & Environment:
  • Browser Version:
  • Target Page:

Are you willing to submit a PR?

Yes

Did you search for similar existing issues?

Additional context

ADO extension: baselining might be too sensitive to unrelated changes

The baseline support added to the ADO extension v1.1.0 is sometimes overly sensitive to non-accessibility-related changes in the page. In particular, if a baselined failure's axe-core-suggested HTML snippet ends up changing due to a PR in a way that doesn't impact whether it fails an accessibility rule or not, the baselining logic will currently see that as "one failure fixed, one new failure found", since it will consider the versions of the failure with different snippets as different.

This tradeoff was made intentionally in the interest of having baselining's idea of failure grouping be consistent with how failure grouping works already in accessibility-insights-service, but it's something that we predict might end up being unexpected for users. We don't have good data on exactly how often this is going to be hit in practice, so we'll use this tracking issue as a place to track potential user feedback on this topic.

If you're a user reading this that is frustrated by the behavior, we'd love if you could upvote this issue (with a thumbs-up reaction) and leave a comment letting us know the specific example that caused you trouble! (ideally, including a diff of the relevant section of your baseline)

Docs/help text for scanUrlRelativePath suggest bogus "//" value for Windows agents

Describe the bug

Our documentation/help text/input descriptions for Scan URL Relative Path inputs consistently suggests using a value of // on Windows, but this is incorrect. This input is used as a relative URL path passed to url.resolve; its behavior is not OS-specific, and on all systems, passing // will actually have the effect of forcing the system to use the totally bogus baseUrl of "http://".

We should:

  • Correct all references to this in task/action definitions and usage docs
  • Ensure that if the user does pass //, they get a reasonable error message
  • Update the unit tests for the scan-url-resolver to pin the behavior of this (and other incorrect use cases that a confused user might feasibly include, like paths with backslashes or absolute paths or query parameters or empty strings)

To Reproduce

  1. Try following the docs and passing // to the input in a siteDir-based configuration
  2. Observe that it tries to scan http://

CodePen repro example

n/a

Expected behavior

Our documentation shouldn't suggest the bogus parameter, and if the user tries passing it anyway, the error should be actionable

Screenshots

n/a

Context (please complete the following information)

  • OS Name & Version: windows-2016 standard agent
  • AI-Action Version & Environment: Azure DevOps extension v1.1.1 production
  • Browser Version: n/a
  • Target Page: n/a

Are you willing to submit a PR?

Yes

Did you search for similar existing issues?

Yes

Additional context

n/a

Help text from baselining should exclude unsupported --updateBaseline option

When running the ADO extension with baselining, we get the following message from the scanner:

To update the baseline with these changes, either rerun with --updateBaseline or copy the updated baseline file to /home/vsts/work/1/s/test.baseline

The --updateBaseline option makes sense when running the scanner CLI. It doesn't make sense when running the scanner from inside the ADO extension. We should have a way for the scanner to exclude the --updateBaseline help when running from the extension.

Action adds its check to the wrong workflow if it is re-run

In #628, I initially created the PR from a fork, and in our self-validation job, the action failed to add our custom check due to #629

I re-created the action from an upstream branch and selected "Re-run all jobs" from the checks UI, and the resulting re-run was able to add the custom check successfully as expected (since GITHUB_TOKEN has the appropriate checks write permissions when run from a non-forked repo), but the check was added to the first workflow instead of the rerun workflow like it should have been. This results in the corresponding PR (#628) showing "all checks succeeded" even though the accessibility checks include failures.

The action should instead detect which workflow it is running within when it is creating the checks (or, with the #629 fix, the workflow that the build artifact was added from) and add the results there.

Consider clarifying service account name and password labels

Describe the bug

Based on recent support emails, when using the task builder UI, it may not be clear to new users that service account name and password are only for authentication, unless you hover on the tooltip. The risk here is that if a user adds authentication, then the site will likely timeout if it cannot find the login, which causes additional friction to onboard users.

The screenshot below shows the UI builder, given the flow of the inputs there is little distinction given to the auth inputs:

Screenshot of UI builder. The labels for service account name and password do not mention authorization until you focus on the tooltip.

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '...'
  3. Scroll down to '...'
  4. See error

Expected behavior

A couple suggestions:

  • We could consider updating the labels in task.json to indicate that they are only for sites that require authentication. While wordy, it could improve clarification. For example the label "Service account name" could be updated to "Service account name (for sites that require authentication only)" or "Service account name (for sites that are behind a Microsoft login only)".

  • We could consider moving the auth inputs to its own group with the displayName: Authentication for sites behind Microsoft login (or something similar). Although, this would likely move the inputs towards the end of the form which may impact discoverability.

Are you willing to submit a PR?

Yes

Did you search for similar existing issues?

Additional context

Rename NOTICES.txt to NOTICE.txt

Our open-source guidance asks for our third-party notices to be in a file called NOTICE.txt. Ours is called NOTICES.txt due to a yarn bug that causes any yarn command to crash when a large file named NOTICE.* is present (see the note on #233). The file should probably be renamed when yarn releases a fix.

ADO Task: Error: Crawl base URL should not have any query parameters

Describe the bug

Our web site under test requires a valid query string to operate (otherwise it will error). Having this limitation makes it so the ADO task is unusable for our scenario.

To Reproduce
Steps to reproduce the behavior: Below is the yaml used to test, see the url with query strings

task: accessibility-insights@3
            displayName: Scan for accessibility issues
            inputs:
              hostingMode: 'dynamicSite'
              url: 'https://ourapp.contoso.com/?appid=1235&area=test' 
              serviceAccountName: $(serviceAccountName) # Key vault secret name for the service account name
              serviceAccountPassword: $(serviceAccountPassword) # Key vault secret name for the service password
              authType: 'AAD'
              singleWorker: true
              uploadOutputArtifact: true         

CodePen repro example

The above yaml outputs the error:

##[error][Exception]ErrorWithCause: An error occurred while scanning website page https://ourapp.contoso/?appid=1235&area=test
    at Logger.trackExceptionAny (D:\a\_tasks\accessibility-insights_4811a442-2fd3-5aa8-ba1a-14cb7e24c113\3.1.0\index.js:189142:29)
    at Scanner.<anonymous> (D:\a\_tasks\accessibility-insights_4811a442-2fd3-5aa8-ba1a-14cb7e24c113\3.1.0\index.js:190329:29)
    at Generator.next (<anonymous>)
    at fulfilled (D:\a\_tasks\accessibility-insights_4811a442-2fd3-5aa8-ba1a-14cb7e24c113\3.1.0\index.js:190244:58)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
caused by: Error: Crawl base URL should not have any query parameters. https://ourapp.contoso/?appid=1235&area=test

Expected behavior

Expected to run against the web site as it requires query parameters to run properly

Context (please complete the following information)

N/A

Are you willing to submit a PR?

Maybe

Did you search for similar existing issues?

Yes

Additional context

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.