Comments (3)
Yes, it should be allowed. Allowing one tester to assign verdicts to AT responses collected by another can help properly segment and distribute work. This can reduce the effort required by individual contributors, as different testers can specialize in various aspects of the testing process. It also allows for more efficient utilization of resources and potentially faster turnaround times. Secondly, evaluating the AT responses and assigning verdicts requires a certain level of expertise and allowing testers with more experience and specialized knowledge to assign verdicts can only ensure that the results are accurate and reliable.
from aria-at.
We know that a human tester will one day need to assign verdicts to AT responses collected by the automated system.
Yes. However, to mitigate the need to do this more than is necessary, I believe we agreed the initial implementation of the automated system will assign verdicts when the AT response is identical to a previously analyzed response to the same command in the same test.
However, we have not discussed whether a human tester should be able to assign verdicts to AT responses collected by other human testers.
Admins have this capability. They can open a test plan run as another user and modify that user's results.
Even if we ignore automation for a moment, there are reasons to think this capability may be valuable. With it, we could:
- Reduce the effort required by individual contributors (by further segmenting and distributing the work)
In practice, I have found it difficult to look at the content in the response field and assign verdicts without experiencing the test case itself. Perhaps that could change with practice. Regardless, if my job is to assign verdicts to previously collected responses, I think I would always do a better job if I were to open the test page, run set up, and run the test for most of the commands.
That is to say, I think the primary value of this use case is that the AT responses are already recorded. Recording them is arduous and time consuming.
So, the question I have is whether we would ever want people to be response collectors if we have systems that can collect responses. The primary role of people would be to verify the system-recorded response is accurate and to assign the verdicts.
- Recognize the difference in authority/experience required for each step (i.e. by reserving verdict assignment for contributors with more expertise)
I don't anticipate much value from the use case of assigning some people to only collect responses. We currently have a mentoring model already built into the tool:
- A mix of senior and new testers are assigned to run a plan
- All record responses and assign verdicts
- If results are different, we have identified the places where new people might have a different understanding
I think this part of the approach is working well.
An understanding of these use cases (plus any others) will help us refine both ARIA-AT App and the nascent automation system.
This brings me to the question I posed in the title of this issue: should one tester be able to assign verdicts to AT responses collected by another?
We have a form of this in the admin role. I don't see much value in extending the current UI to support the additional use case of having some CG members only collect responses.
from aria-at.
We know that a human tester will one day need to assign verdicts to AT responses collected by the automated system.
Yes. However, to mitigate the need to do this more than is necessary, I believe we agreed the initial implementation of the automated system will assign verdicts when the AT response is identical to a previously analyzed response to the same command in the same test.
I do recall our discussion about verdict reuse, though I'm not sure if it needs to happen in the automation subsystem. Since reuse seems like an optimization which is orthogonal to the capability discussed here, I'll refrain from going into detail in this thread. (This will certainly come up as we continue to refine the design of the automation system, but if you'd like to continue the discussion right away, I'm happy to open a new issue.)
In practice, I have found it difficult to look at the content in the response field and assign verdicts without experiencing the test case itself. Perhaps that could change with practice. Regardless, if my job is to assign verdicts to previously collected responses, I think I would always do a better job if I were to open the test page, run set up, and run the test for most of the commands.
This reminds me of a discussion from a recent CG meeting. To the extent that "you had to be there" to explain the assignment of a verdict, that seems like a problem for the public's consumption of ARIA-AT's reports. If we could more concretely describe the kinds of situations for which the AT response data is insufficient, that might uncover improvements to ARIA-AT's processes/systems which ultimately promote transparency.
Do you think we could include an agenda item for the next CG meeting to ask the Testers present about their experience along these lines?
An understanding of these use cases (plus any others) will help us refine both ARIA-AT App and the nascent automation system. This brings me to the question I posed in the title of this issue: should one tester be able to assign verdicts to AT responses collected by another?
We have a form of this in the admin role. I don't see much value in extending the current UI to support the additional use case of having some CG members only collect responses.
Thanks!
from aria-at.
Related Issues (20)
- PR #997: V2 Test format build, *-commands.csv with "delete" key incorrectly reported as "not found" HOT 2
- Presentation Numbers in the V2 Test Format HOT 1
- v2 Test Format: Space-separated settings in AT_KEY-commands.csv HOT 1
- Do not show 0-priority assertions in the results collection form HOT 1
- Remove "Success Criteria" section from preview of the test collection form
- Preview of results collection form has title and instructions that are inconsistent with preview of test plan HOT 2
- Feedback: "Navigate backwards to a not pressed toggle button" (Toggle Button, Test 6, V23.12.06)
- Possibly all tests - Open test page button disabled if user navigates to another webpage in test page new window
- Feedback: "Navigate backwards to a pressed toggle button" (Toggle Button, Test 12, V23.12.14) HOT 2
- Feedback: "Navigate backwards to a pressed toggle button" (Toggle Button, Test 12, V23.12.14) HOT 4
- Feedback: Toggle Button Test plan, V23.12.14: Shouldn't role be 'Toggle Button' instead of 'button' HOT 2
- JAWS Changes Requested: "Trigger an alert" (Alert Example, Test 3, V23.12.06) HOT 3
- Define meaning of assertion priorities: "MUST", "SHOULD", and "MAY" HOT 5
- Tests are not correctly sequenced in preview
- Assertion tokens are causing test plan build to fail HOT 2
- Enhance V2 test format to allow 0 priority assertions in tests.csv HOT 5
- Proposal for new terminology for the phenomena we currently call "undesirable behaviors" or "Other behaviors with negative impact" HOT 2
- Feedback: "Activate a menu item" (Action Menu Button Example Using aria-activedescendant, Test 24)
- Feedback: "Read information about a menu item" (Action Menu Button Example Using aria-activedescendant, Test 16)
- Discuss options for heading text and levels on report pages HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aria-at.