Comments (17)
Attaching a complete draft of tests in a spreadsheet. The tests in this spreadsheet replace all the tests that are already in wpt format.
Combobox-autocomplete-both.xlsx
from aria-at.
While there is one combobox design pattern, we currently have 4 examples with different features. We will soon have a 5th. We probably want a separate issue for each example. I'm going to change the title of this issue to match the name of the combobox example we are working on now for the production system design phase. We will be testing only one combobox example during the design phase.
from aria-at.
I see this at https://w3c.github.io/aria-at/review/combobox-autocomplete-both.html - is it ready for a review?
from aria-at.
I see this at https://w3c.github.io/aria-at/review/combobox-autocomplete-both.html - is it ready for a review?
Nop, not updated from the spreadsheet yet.
Related thought ...
@spectranaut, do we have a way of specifying the sequence of tests in the plan for a pattern? At least for me, when writing and reading test plans, and even when reading results, the sequence really matters. In my spreadsheet, I tried to follow a logical sequence of operations and states that could roughly map to a flow for using combobox.
from aria-at.
I had to modify the spreadsheet you uploaded to work with Jon's script, here is the modified file:
tests.xlsx
Right now, we don't have a way to sequence tests. We should make a backlog test to consider it!
from aria-at.
Here is the test plan for review: https://w3c.github.io/aria-at/review/combobox-autocomplete-both.html
Things that still need to be done for these tests:
- setup scripts written
- some key commands aren't found, i think this is a data format problem
from aria-at.
Thanks spectranaut & Matt!
Made a high-level review for combobox, mainly from JAWS.
Review from test settings and consistency
I observed several concerns by comparing tests over multiple SR and the other widgets (menubar and checkbox). These are not specifically for combobox, but leave notes here for the record.
1-1.Test setting
-
NVDA key used in Tester instruction is not clear, such as "Put NVDA into Focus Mode using NVDA+Space" > I think we should use expression such as Insert + Space
-
Even though filtered the test radio button to VoiceOver , Test 1 shows the test applies to JAWS and NVDA.
-
The "Mode" used in the first list may be confusing for tasks involve mode switching > Use the term "Initial mode" or "Default mode"?
1-2.Inconsistency
-
"Applies to" information seems always 'Desktop Screen readers" in checkbox and menubar, while varies in combobox.
-
The name of tests were "Test number + Test name + in reading/interaction mode" in checkbox and menubar, while the test name didn't contain " in reading/interaction mode" in combobox.
Review from individual tests
The majority of concerns came from missing data probably from data formatting to me. Aside from missing data, I observed 3 concerns..
2-1.Command concerns from Individual test review
- For the tasks to read, Insert + UP (Say from cursor) could also be an option for key comamnds? (Test 21 - 26)
2-2.Language concerns from Individual test review
-
The scripted instruction reads "Set focus on 'States' button; Ensure combobox is empty and collapsed". But 'State' seems like a name/label to me, not button. Do we call 'button' even if it's not clickable? (Test 5)
-
I think we need Scripted Instruction to differentiate 10 & 11 (filled & empty), because mode and Tester Instructions of Test 10 and 11 are the same. (Test 10- 11)
2-3.Data format concerns from Individual test review
Test No. | Detail |
---|---|
Test 1 | No key commands are available. |
Test 2 | No assertions are available. |
Test 4 | No instructions, commands, assertions available. |
Test 9 | No key commands are available. |
Test 14 - 17 | First test instruction is empty/No key commands are available. |
Test 19 | First test instruction is empty/No key commands are available. |
Test 20 | Priority for second and third assertions were missing. |
Test 29 | First test instruction is empty/No key commands are available. |
As Michael mentioned in his post for menubar](#54 (comment)), having common matrix or grid would be preferable to ensure the quality in the long run.
from aria-at.
@Yohta89 wrote:
- NVDA key used in Tester instruction is not clear, such as "Put NVDA into Focus Mode using NVDA+Space" > I think we should use expression such as Insert + Space
Was wondering about this. I am fine with saying insert+space. Testers will be trained and can make their own translation as needed, e.g., use capslock if they have that enabled.
We will need to specify which defaults are OK to change, e.g., voice rate, keyboard, etc. But, it is acceptable that we provide only desktop layout key assignments in the instructions, since desktop is the default.
Kind of funny that we have that legacy hang over ... most people have laptops but desktop is the default.
- The "Mode" used in the first list may be confusing for tasks involve mode switching > Use the term "Initial mode" or "Default mode"?
I don't understand what you are talking about here.
- "Applies to" information seems always 'Desktop Screen readers" in checkbox and menubar, while varies in combobox.
Correct, and we will need to update checkbox and menubar accordingly. This has been done for checkbox now, I think. Each test needs the correct applies to value.
- The name of tests were "Test number + Test name + in reading/interaction mode" in checkbox and menubar, while the test name didn't contain " in reading/interaction mode" in combobox.
Seems redundant to specify it in two places. I wonder if this part of the name should be auto-generated.
- For the tasks to read, Insert + UP (Say from cursor) could also be an option for key comamnds? (Test 21 - 26)
No, insert+up does not apply in text inputs; it only reads the current line, not the current control.
- The scripted instruction reads "Set focus on 'States' button; Ensure combobox is empty and collapsed". But 'State' seems like a name/label to me, not button. Do we call 'button' even if it's not clickable? (Test 5)
Sorry, this assumes a later version of the combobox example that has not been merged yet. In that version, the "open" button is named "States" and it has aria-expanded on it.
from aria-at.
Thanks for the response Matt!
With regard to the following bullet point, I meant to address the lists of bullets points under the test title, such as "Mode: reading" or "Mode: interaction". Currently, this information shows the mode when testers started the task.
In the case of the mode-switching task, however, a mode can be switched from one mode to the other thorough a task. So I was wondering if we want to use the terminology such as "Initial mode" or "Default mode" instead of saying just "Mode", so that it's more self-describing.
- The "Mode" used in the first list may be confusing for tasks involve mode switching > Use the term "Initial mode" or "Default mode"?
from aria-at.
@Yohta89
With regard to the following bullet point, I meant to address the lists of bullets points under the test title, such as "Mode: reading" or "Mode: interaction". Currently, this information shows the mode when testers started the task.
In the case of the mode-switching task, however, a mode can be switched from one mode to the other thorough a task. So I was wondering if we want to use the terminology such as "Initial mode" or "Default mode" instead of saying just "Mode", so that it's more self-describing.
Oh, I understand your point now.
Since the test and assertions describe when a mode is expected to switch and how it is expected to switch, I don't think the extra words would necessarily add clarity.
There are very few tests where mode is expected to switch. In general, mode switching only applies to JAWS and NVDA, and only to a limited set of widgets. These are probably fewer than 0.5% of all desktop screen reader tests, and an even lower percentage of screen reader tests when we expand to cover mobile.
from aria-at.
@spectranaut, Attaching updated tests.xls with changes we discussed yesterday + a couple more tests.
tests.xlsx
from aria-at.
I'm going to try not to be involved in test writing unless it's really necessary (like when Jon was out last week!) and I haven't gotten to re-creating the tests.
@jongund, I just copied over your scripts in order to create these tests the first time. The csv files in the data/
directory are old, you will have to re-export the appropriate CSV files from the spreadsheet Matt uploaded in the previous comment. It will be easier to re-run these tests after the changes I outlined in this issue
Also, the setup scripts haven't been written yet for these tests!
from aria-at.
I had a chance to review some of the tests. I didn't have enough time to go through every test in the plan. Overall, this is a great set of tests! The notes/questions that I have are mostly about formatting and consistency than anything else.
Notes:
- Test names
- Should we use the present tense for test tiles ("Navigate to" instead of "Navigating to") (other plans do this)?
- should we include the mode in test tiles? Otherwise, we will see duplicate test titles in listings, and it will be hard to differentiate tests without looking at the review or starting a test run. This can be done either manually or automatically.
- These test names appear to include assertion-like language ("conveys combobox role; name; editability; value; and state"). Test names in other plans are centered on the task rather than the expectations. Should we do this same here?
- The words 'popup' and 'popup item' seem to be used in place of 'listbox' and 'option' roles. In some test names, the word 'option' is used instead of 'popup item'. Is there a reason for this, or would it be better to refer to the ARIA roles?
- Setup scripts
- test 'Navigating by line to filled; editable; expanded combobox conveys role; editability; value; and state' expects the target to be filled but is missing a setup script
- Re assertion 'Caret position is conveyed' - are you expecting numeric information about the position of the cursor, or would the character that the caret just moved in front of being announced pass? (I noticed that JAWS always announces 'blank' in the APG example)
- Carrot instructions for JAWS include MAC commands. Is that expected?
- ARIA roles, states, and properties
- I didn't see any assertions for
aria-controls
- is that expected? - I didn't see any assertions related to
aria-activedescendant
- is that expected?
- I didn't see any assertions for
from aria-at.
PAC are working on updates to this test plan to increase its coverage and consistency. As such, and given that a lot of information in this thread is now out-of-date, I'm closing this issue in favour of #340.
from aria-at.
Updates from the April 1, 2021 CG meeting: we should remove the files related to the older iteration of the test plan, and then add a prefix to the title of this issue to indicate that it is now "deprecated". In this case, deleting the files and entry in support.json
should suffice, because no data was gathered from testers that we want to keep (or at all).
from aria-at.
I've submitted PR #417 to remove the old files. The deprecated prefix can be added to this issue once it has been merged.
from aria-at.
Note: deprecated prefix should include the date.
from aria-at.
Related Issues (20)
- Feedback: "Navigate to the last unchecked radio button in a group" (Radio Group Example Using aria-activedescendant, Test 4) HOT 2
- Feedback: "Navigate to the last checked radio button in a group" (Radio Group Example Using aria-activedescendant, Test 8) HOT 1
- Feedback: "Navigate forwards to an unchecked radio button" (Radio Group Example Using aria-activedescendant, Test 10) HOT 2
- Feedback: Potentially out of scope test - "Navigate out of the start of a radio group in interaction mode" (Radio Group Example Using aria-activedescendant, Test 18)
- Should one tester be able to assign verdicts to AT responses collected by another? HOT 3
- Should it be possible for multiple testers to assign verdicts to a single set of AT responses? HOT 2
- Should verdict assignment depend on response verification? HOT 2
- Presenting results in a more intuitive and useful manner HOT 1
- Rethinking the wording for assertion verdicts HOT 3
- Disambiguating 'Test Plan Run' in the Working Mode
- Feedback: "Open a nested modal dialog using a link in reading mode" (Modal Dialog Example, Test 27) HOT 1
- Feedback: "Read information about a slider in interaction mode" (Color Viewer Slider, Test 8) HOT 2
- Explore including test cases outside APG HOT 3
- Establish explicit test plan life cycle in working mode HOT 1
- Feedback: "Increment a slider by one step in interaction mode" (Color Viewer Slider, Test 10)
- Feedback: "Navigate forwards to a slider" (Color Viewer Slider, Test 5) HOT 1
- System reporting conflicts where there are none apparent: "Open a menu in reading mode" (Navigation Menu Button, Test 10) HOT 1
- Feedback: "Open a menu in reading mode" (Navigation Menu Button, Test 10) HOT 1
- Inconsistent announcement of W3C Quick Links Menu "Open a menu in interaction mode" (Navigation Menu Button, Test 11) HOT 1
- Feedback: "Open a menu in interaction mode" (Navigation Menu Button, Test 11) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aria-at.