Mapping abstract test instructions to concrete, screen-reader-specific instructions about aria-at HOT 10 CLOSED

I've taken a stab at creating a document that maps abstract test instructions to screen-reader-specific concrete instructions, starting with the Checkbox example.

Here is the document in Excel format:
Checkbox testing commands.xlsx

What the document does

The document gives a list of concrete, screen-reader-specific commands for testers to execute for each test for the Checkbox example.

So, for example, for "Read checkbox", it shows what concrete commands to press for JAWS, NVDA and VoiceOver in the different interaction modes.

Structure of the document

The document has three tabs:

1. Checkbox testing commands with JAWS
2. Checkbox testing commands with NVDA
3. Checkbox testing commands with VoiceOver macOS

Each tab contains one table that gives specific testing commands for that particular screen reader, for each screen reader mode, for each Checkbox test.

Where this data comes from

I've put together these commands from:

• Matt's 'Test Simplification Exploration' spreadsheet from early July
• Deque's Screen Reader Reference Guides

To avoid mistakes, I've also tested these commands in JAWS, NVDA and VoiceOver today.

Note: @Yohta89 's spreadsheets contain some specific test instructions for menubar. So they will come in handy for when we do the same for menubar.

Next steps

I've noticed that putting these instructions together does indeed take time!

I think that it'd be good to tackle this as several people. And I also want to make sure that the commands are accurate.

Before I and/or anyone else extends this work for menubar, I think that it'll be useful to review and feedback on the format and content of this document.

On top of that review, here are questions that have already come up:

Questions unrelated to any screen reader

1. Are we listing too many specific commands to test with? How many is enough?

2. Specifically, I've added a few more commands compared to Matt's 'Test Simplification Exploration' spreadsheet from early July, in an effort to be complete. But I'm imagining that we can reasonably assume some commands to produce the same output as some other, similar commands. So we should leave some commands out?

   • For example, I've added Control+Option+A for VoiceOver as a way to read a checkbox. This command will read the document from the current cursor position. Is it worth adding this instruction? Or will it generally produce the same output as when reaching elements using Control+Option+Right?

JAWS-related questions

1. Matt's 'Test Simplification Exploration' spreadsheet from early July said that the JAWS Virtual Cursor needed to be turned off (using Insert+Z) for some of these Interaction Mode instructions to work. (See cells I12, I13 and I14 in the 'Two-State Checkbox JAWS Tests' in Matt's document).

   I don't understand why JAWS' Virtual Cursor needs to be turned off. Why is that the case?

2. In Matt's 'Test Simplification Exploration' spreadsheet from early July, Up/Down was also listed as a command for reading a checkbox group in Interaction mode. (See cell I14 in the 'Two-State Checkbox JAWS Tests' in Matt's document).

   I don't understand how that would work, as I don't believe that the Up/Down arrows move JAWS' Virtual Cursor in Interaction/Forms mode. So I have removed these instructions for this specific case. Is that correct?

3. What’s the equivalent of Insert+Tab and Insert+Up on a laptop?

VoiceOver-related questions

1. My understanding is that testing commands with Quicknav on are always exactly the same as with QuickNav off, except there is no need to press Control+Option. Is that correct?

2. I also assume that VoiceOver's output will always be the same regardless of whether Quicknav is on or off. For example, I imagine that Control+Option+Command+C with Quicknav off will always produce the same result as Command+C with Quicknav on. Is that a reasonable assumption?

3. If it is indeed the case that Quicknav on and off are equivalent except for the need to press Control+Option, then I suggest that we do not repeat the instructions twice. At the moment, the document does have separate rows for Quicknav off and Quicknav on, but I suggest removing them. Instead, we could give instructions for Quicknav off only, and tell testers to make sure that Quicknav is off. What do you think?

4. Is there a VoiceOver command for reading the current item?

   I know that Control+Option+S reads the current sentence, but in the case of a checkbox, it reads the text content of the checkbox rather than the checkbox itself.

from aria-at.

Thank you so much for this work and detailed notes! Great work, and great questions. Let's discuss on the next call.

from aria-at.

JF, thank you for this comprehensive work!
I'm pretty sure it takes a lot of time and I agree with the next step you brought up.

Two quick responses to the questions regarding JAWS.

2. In Matt's 'Test Simplification Exploration' spreadsheet from early July, Up/Down was also listed as a command for reading a checkbox group in Interaction mode. (See cell I14 in the 'Two-State Checkbox JAWS Tests' in Matt's document).I don't understand how that would work, as I don't believe that the Up/Down arrows move JAWS' Virtual Cursor in Interaction/Forms mode. So I have removed these instructions for this specific case. Is that correct?

-This was also a tough call for me. But my interpretation for this was that JAWS announces group information when a user tried to get out from one of the checkbox menuitem.
*e.g. If my key focus on the Lettuce checkbox in interaction mode and press up arrow key twice, JAWS announces "Group starts, Sandwich Condiments".

> 3. What’s the equivalent of Insert+Tab and Insert+Up on a laptop?

-I don't have access to a laptop with JAWS for now so I haven't validated yet. But according to
the guide from keystroke document from freedom scientific it goes as follows:
Insert+Tab corresponds to CAPS LOCK+TAB
Insert+UP corresponds to CAPS LOCK+I on a laptop.

https://support.freedomscientific.com/Content/Documents/Manuals/JAWS/Keystrokes.pdf

from aria-at.

Questions unrelated to any screen reader

1. My opinion is yes, we can easily list and test with too many commands. I don't think it is realistic to test every single command for all screen readers. IMO, we should strive to test a representative sample of commands (perhaps those that are most commonly used). As the project evolves, there may be more opportunity to test more commands, but I'd submit that the efficiency/value ratio is higher when testing a few commands vs all commands.
2. That's a good point. I think we should come to a consensus on which commands to test with.

JAWS-related questions

1. That's a good question, we will have to ask Matt. I'd think that the automatic switching of modes would suffice, but perhaps not.
2. I can't speak for Matt, but perhaps it was to test automatic mode switching (press the arrow key twice to switch to reading mode)
3. See the response from @Yohta89

VoiceOver-related questions

1. not by default. I don't think we can test single key commands when using quick nav since the option to do so is off by default. In other words, by default, quick nav does not support single key navigation (such as J to move to the next form field). Additionally, command+c results in the OS 'copy' command. In my experience, quick nav being enabled only affects arrow key navigation (including rotor navigation) by default and the Apple documentation seems to agree.
2. I don't think that is a safe assumption. There is always room for bugs. However, I don't think it is reasonable to test every possible combination of commands.
3. Agree (but see my response for number 1), especially since single key commands is not a default option (but I would hesitate to call them equivalent from a testing standpoint).
4. VO+L is close; it reads the current line. It's not quite the same as 'read current item', since focus might be on a form field that is on the same line as other text. I'm not aware of an exact command for this.

from aria-at.

Thanks @Yohta89 and @mfairchild365.

Reading your comments I realised that I had confused VoiceOver's QuickNav with 'Control Option Lock'. The link you shared Michael was useful.

I do support the idea of testing with a smaller set of commands, at least to start with.

Heads up re. my limited availability for our Wednesday calls at the moment

Apologies for missing last week's call. I've just started work with a new client and a meeting ran over. I'm only about 60% confident I'll manage to join tomorrow, and I won't manage next week.

from aria-at.

Does the format of the document allow for programmatic parsing of the commands? (see Excel file attached in the second comment)

I'm imagining that the way that commands are listed might make them hard for a script to parse:

• There are several commands per cell, each preceded by a bullet point
• Within a cell, different commands are grouped under pseudo-headings, such as "Navigate to the checkbox from outside the checkbox group, using each of these methods:"
• None of that structure is semantic, it's just plain text (because of the Excel file format)

@spectranaut and @mfairchild365: is that an issue?

I'm not quite sure how we could store the commands in a way that is easier for a script to parse, while also making it easy to review for all of us.

I initially thought of putting the commands together in JSON format directly, but Matt rightly pointed out that that would make it harder for us to review the data.

from aria-at.

Thanks all for your feedback last week.

Updates since last week

Here's what I've done since last week:

• Updated the checkbox testing commands in line with your feedback
• Replaced the Excel document with an HTML document for better accessibility
• Added testing commands for menubar
  • I haven't double-checked the testing commends for menubar yet. I've just quickly copied and pasted the testing commands that @Yohta89 shared on August 19 in Issue 5, and added commands for VoiceOver.

Link to the testing command files

Testing commands for checkbox

• JAWS checkbox testing commands
• NVDA checkbox testing commands
• VoiceOver checkbox testing commands

Testing commands for menubar

• JAWS menubar testing commands
• NVDA menubar testing commands
• VoiceOver menubar testing commands

Next steps

• I believe that the testing commands for checkbox are ready to use for the prototype.
• The testing commands for menubar might need a bit of double-checking, because I haven't tested with JAWS or NVDA as I wrote the document. I just copied from Yohta's Excel document, and I'm not as familiar with JAWS and NVDA. I'll text and update the commands in the next couple of days. After that, it'd be useful to get your review @Yohta89 @mfairchild365.

Note: I'm not sure I'll be able to join our meeting tomorrow Nov 13.

from aria-at.

I've now double-checked the testing commands with NVDA and JAWS and made some small updates to the documents that the links above point to.

from aria-at.

I've been thinking about the reading the checkbox and checkbox grouping cases. I think the test harness needs to be redesigned given what JF has revealed, which is that there are two categories of commands in each of these cases -- where the categories are "reading the checkbox by navigating to it" and "reading the checkbox after the cursor is already on it".

Right now it is only possible to have one user instruction that describes both cases. I want to suggest something like this:

  verifyATBehavoir({
    mode: ["reading", "interaction"],
    specific_user_instruction: [
      {
	instruction: "navigate to the first checkbox",
	command_list: "navigate to checkbox",  // This returns all the commands for navigating to a checkbox
      },
      {
	instruction: "read the checkbox after the cursor is on the checkbox",
	command_list: "read checkbox", // This returns all the commands for reading a checkbox when the cursor is on the checkbox
      }
    ],
    assertions: [
      "The role 'checkbox' is spoken",
      "The name 'Lettuce' is spoken",
      "The state of the checkbox (not checked) is conveyed"
    ]
  });

This would result in the same UI as in #15 but instead you would see this in the test instructions broken out a bit more, maybe something like this:

Navigate to the first checkbox using each of the following JAWS controls:
    X / Shift+X
    Tab / Shift+Tab
    Up / Down
    Left / Right (with Smart Navigation on
    Control+Insert+X to see a list of checkboxes; then use the Up/Down arrows to select a checkbox; then press Enter to navigate to that checkbox

And, read the first checkbox after the cursor is on the checkbox using each of the following JAWS controls:
    Insert+Tab (or CapsLock+Tab)
    Insert+Up (or CapsLock+I)

Verify, after each of the above command executions, the following assertions are met...

I'm worried that this makes the test writing complicated to explain or learn. Maybe these two categories should have two different test files instead, or maybe we should just leave the test file as it is, with slightly confusing instructions (specifically, it currently says Then, navigate to the first checkbox using each of the following JAWS controls: ...Insert+Up (or CapsLock+I) with the cursor already on the checkbox)

from aria-at.

This work is complete, the menubar and checkbox tests have been written! :)

from aria-at.