Create tests for APG design pattern: menubar about aria-at HOT 17 OPEN

Community Group)
Although we agreed to start peer-review for the menubar at the meeting today, I noticed some of the issues raised by @mfairchild365 on his post Feb.23rd isn't yet fixed.
I'm gonna fix those asap and cue here once the menubar example is ready to review.

from aria-at.

For Test 27: Read disabled menu item in submenu in interaction mode, for VoiceOver, the second assertion is, "The name 'Smaller' is conveyed." I suspect this is a typo. Since the 'Larger' menu item is selected based on the Scripted Instructions, I think the assertion should read, "The name 'Larger' is conveyed."

from aria-at.

To clarify, is https://w3c.github.io/aria-at/review/menubar-editor.html up to date and ready to review?

If so, some initial feedback:

1. I'm glad to see this taking shape!
2. A lot of instructions list duplicate commands. "Enter, Space, Enter, Space". Is this intentional?
3. VoiceOver often seems to have an empty step 1 under instructions.
4. VoiceOver and NVDA seem to be missing commands for most tests
5. Some test titles have characters that can't render correctly on MacOS. For example "Change state of menu item radio�to�checked"
6. The word "label" is used instead of "name" in most places. I'm okay with either, but I think we should be consistent.
7. Would be helpful to see a description of what the setup script will do.

I also put together a matrix to help me identify at a high level which features, states, and modes were tested. I marked a few that I thought might be missing with a question mark. Maybe it would be helpful to generate a table like this for the review page? @mcking65 @spectranaut (not sure what the best structure of the table would be)

from aria-at.

Yes its up to date!

About the technical bugs: There is a PR open that will address the bug that made (2 - duplicate keys), and I can look into (5 - character rendering) today or tomorrow and plan to fix (7 - setup script description) today or tomorrow.

About the key commands: I think Jon and Yohta just didn't supply commands for VoiceOver and NVDA. Who can do that?

from aria-at.

Thank you @spectranaut - I can do it, but not before Wed.

from aria-at.

I've just finished commands for VoiceOver and NVDA (see 'Command' sheet for attached files),
Would you be able to encode them for review @spectranaut @mfairchild365 ?
Otherwise, I'll touch base with Jon and encode them once he's back next week.

Quick notes:

• Test No 1-3 may not be included in the final version of menubar testes. Please disregard these columns if needed.

NVDA key

• "Numpad_PLUS" and "Numpad_5" are currently not listed in the keys.mjs., which I marked in red.

200218_Menubar_NVDA_test_writing.xlsx

VO key

• There were 2 tasks I wasn't sure the corresponding command options to carry out the task. I marked them as "?" in column E.

200219_Menubar_VO_test_writing.xlsx

from aria-at.

@Yohta89 if you upload a spreadsheet I can add them. Can you reupload the spreadsheet that is already in the repository, so it's in the same format?

Edit: oh sorry I misread, I see you have added the commands! I'll work on encoding them.

from aria-at.

Thanks @spectranaut ! I attached the file when I edited the comment! Let me know if the format didn't fit.

from aria-at.

@Yohta89 -- I added the new commands to this checked in CSV file, and then used Jon's script to convert them to the commands.json format.

The commands can all be reviewed in the menubar testplan review page

from aria-at.

Once the PR #160 is merged, this should be ready for peer-review.

from aria-at.

From 2020-04-22 call; I'll document the existing behavior of native menu bars with JAWS. @Yohta89 @jongund will document NVDA and VO.

Sample of aspects to document:

• How roles are conveyed
• Is posinset information conveyed
• What states/properties are conveyed

from aria-at.

NVDA and Windows 10 System Menu on Firefox Browser

from aria-at.

JAWS 2020 and Windows 10 System Menu on Firefox Browser

Note: reading all of the roles resulted in equivalent information being conveyed as when navigating to it.

A look under the hood (what is in the a11y API)

I used Accessibility Insights to look at the accessibility properties of the Menu and compare them with SR output.

Menubar role

• Name: Application (not conveyed by JAWS)
• Role: MenuBar (not conveyed by JAWS)

When navigating to the menu bar, the name of the menu bar is not conveyed, and I think the role may be conveyed as "menu". See output above.

other roles states and properties

I wasn't able to find roles like menuitemradio or menuitemcheckbox or haspopup within the inspector.

Compare findings to test plan

Test 1, 2, 3, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26: - NA: reading mode is not supported for native menubars

• Test 5: Navigate to menu item in menubar using interaction mode
  • potential change: The two menubar assertions are not relevant here. Consider removing.
  • potential change: Add assertion for posinset
• Test 7: Navigate to menu item radio in submenu using interaction mode
  • Assertions do not match; In the native test, the role and state were not conveyed when in the unchecked state. I think this is a bug, and the assertions that we have are good.
• Test 9: Navigate to menu item checkbox in interaction mode
  • assertions match well
• Test 11: Change state of menu item radio to checked in interaction mode
  • Assertions do not match; The new state is not conveyed. Potential reason: the native menu was closed while the example menu is not.
• Test 13: Change state of menu item checkbox to checked in interaction mode
  • Assertions do not match; the new state is not conveyed. Potential reason: the native menu was closed while the example menu is not.
• Test 15: Change state of menu item checkbox to unchecked in interaction mode
  • Assertions do not match; the new state is not conveyed. Potential reason: the native menu was closed while the example menu is not.
• Test 17: Open submenu in interaction mode
  • assertions match. Note, the APG example doesn't have a similar second level 'sub menu'.
• Test 19: Close submenu in interaction mode
  • assertions do not match; only 'closing menus' was conveyed, likely because the native example closes the entire menu and focus is sent back to the open document, while in the APG example focus is sent back to the menuitem in the menubar.
• Test 21: Read menu item in menubar in interaction mode
  • assertions do not match; consider adding a position assertion, and removing assertions about the menubar
• Test 23: Read menu item radio button in submenu in interaction mode
  • Assertions do not match. Note that the native example does not have named groups. Instead, it uses submenus. Consider adding posinset assertions.
• Test 25: Read menu item checkbox in submenu in interaction mode
  • Assertions match
• Test 27: Read disabled menu item in submenu in interaction mode
  • Assertions match

Suggestions

1. Remove menubar related assertions from tests for navigating to and reading a menuitem in all modes. This is handled by test 3.
2. Add assertion for posinset and setsize to all reading and navigation tests in both modes. Reading mode might be optional for JAWS.
3. Keep assertions related to state changes when activating a menuitemcheckbox or menuitem radio, even though this does not match the behavior of the native example.
4. Keep assertions related to role and state of the unchecked menuitem radio, even though this does not match the behavior of the native example.
5. Keep assertions related to closing a menu, since the behavior of the native example is not the same. (native closes the entire menubar, while APG does not).
6. Keep optional assertions related role="group", even though this pattern is not found in the native example. The native example uses unnamed separators.

from aria-at.

For Test 23: Read menu item radio button in submenu in interaction mode, it should be noted that, for VoiceOver (on Safari, at least), while the group role and group name assertions would not be passed by this test, that information is available if the user navigates the menu using VO+Arrow. Not sure if that should be added as a different test, or somehow noted elsewhere, or if we just want to not worry about it, but I worry this may give a misleading impression about support for this structure on VoiceOver.

from aria-at.

@jongund @mcking65 I finished my review and comparison to a native menubar for JAWS. See the list of suggestions at the end of #54 (comment). I'm happy to answer any questions.

from aria-at.

@mfairchild365 commented:

1. Remove menubar related assertions from tests for navigating to and reading a menuitem in all modes. This is handled by test 3.

Agree, we only want to make assertions about role and name of the menubar in the tests related to navigating to the menubar itself.

It is worth noting that if you do not have JAWS loading from the system tray, and if you navigate to the native JAWS menubar in the main JAWS window by pressing "Alt", JAWS announces the menubar role. And, in the JAWS menubar, there is no position or setsize information like there is in the Firefox menubar. So, the behavior is different in different apps.

2. Add assertion for posinset and setsize to all reading and navigation tests in both modes. Reading mode might be optional for JAWS.

Have this now in #184

3. Keep assertions related to state changes when activating a menuitemcheckbox or menuitem radio, even though this does not match the behavior of the native example.

Since the menu closes, we can't really test state changes. Those have been removed. I wish the editor menubar would put focus in the text area when it closes. It would be more realistic that way. But, that is an APG issue.

4. Keep assertions related to role and state of the unchecked menuitem radio, even though this does not match the behavior of the native example.

Definitely agree here. Amazing this info is not communicated.

5. Keep assertions related to closing a menu, since the behavior of the native example is not the same. (native closes the entire menubar, while APG does not).

Agree that it is OK to keep them. Although, this is one where I can anticipate changing my mind after working with this more.

6. Keep optional assertions related role="group", even though this pattern is not found in the native example. The native example uses unnamed separators.

Yes, this is an important feature of ARIA; it would be good if this existed natively.

from aria-at.

I am going to go ahead and merge #184. I think it is really important to get the improvements to the ability to format instructions into all the examples.

That said, I still see need to make multiple changes to menubar. I will make them in a separate PR as suggested by @mfairchild365.

The biggest issue is that reading mode in JAWS and NVDA are so different that, in order to respect their perspectives on design, we will need to tailor assertions to each. This is a good test for demonstrating that even with very different approaches, different screen readers can fully support the ARIA necessary to make the widget understandable. It is not our job to say which presentation of accessibility semantics is more understandable; it is our job to find out whether the presentation made by each screen reader sufficiently represents the accessibility semantics in the user interface.

Then, there are some odd little issues like shift+tab does not cause a mode change for JAWS or NVDA because the text area also triggers interaction mode. So, for that test, we will test only Tab, not Shift+Tab. Or, we might be able to test shift+Tab, but it will need its own instructions. I am leaning toward not bothering with shift+Tab.

from aria-at.