Thank you @spectranaut for putting this together. This is wonderful. A few notes:

• Perhaps change 'setup code' to 'test code'. This seems a little clearer to me. 'setup instructions' is fine as is.
• I thought I'd have more feedback, but I think you covered most everything, described everything very well, and have good ideas for how to approach the task.
• Perhaps it would be good to document the vocabulary that will be used in the form to collect data as well.

from aria-at.

This document is complete.

from aria-at.

Deleting this wiki page as it is now obsolete. In the event a historical reference is ever needed, the content at time of deletion is pasted below.

This page describes and defines "a test" from the perspective of the "test harness". The "test harness" is an application which the tester will use to displays and save results from "a test".

Provide feedback in issue #10

Life cycle of test

1. Set up

Test author provides:

1. Example Code
   1. The "example code" is the html, aria, css and javascript that the AT will interact with during the test. The "example code" might be exactly the APG examples, or it might be custom configurations of html, aria, css and javascript.
2. Setup Instructions
   1. The "setup instructions" are instructions that a tester might need to perform in order to get the "example code" into the correct configuration in order to test a specific behavior of the assistive technology. For example, the instructions may be "Click the button to open the modal".

Role of test harness:

1. Displays setup up code in a browser to the tester.
   1. How we display the code depends on the harness, for example, it might be a popup, or iframe, or url link.
2. Displays set up instructions to the tester.

Role of tester:

1. Perform setup instructions
   1. The setup instructions could be performed using any method the tester prefers, for example, using the AT or a mouse.
2. (Optional) Provide option to record the AT output.

2. Execution

Test Author Provides:

1. Abstract Operating Instructions
   1. The "abstract operating instructions" are instructions that use abstracted language to describe how to operate any assistive technology, for example, "Navigate to the checkbox in reading mode".
2. A mapping between abstract operation instructions to specific operation instructions for every assistive technology being tested.
   1. Hopefully this mapping will be in separate database to decrease duplication. Some tests will re-use the same abstract instructions ("navigate in reading mode") where others might be unique to the test ("operate the checkbox")

Role of test harness:

1. Display specific operating instructions for the AT under test in a clear, human readable way.

Role of tester:

1. Perform the specific operations instructions using the assistive technology under test.

3. Expectations

Test Author Provides:

1. Test expectations
   1. "Test expectations" are clear observable that can be use to describe the behavior of any AT. For example "The checkbox's name is announced".

Role of test harness:

1. Provide form to record the result of the test

Role of tester:

1. Record the result of the test

Key components of a test

Each test contains at least the following parts. For clarity they are compared to the excel sheet of proposed tests in this comment on issue 5. In this excel sheet, each of these rows is considered a unique "test".

1. Example code

The APG example code or otherwise provided html, aria, css and javascript. In the excel sheet above, the example code is the navigation menubar example code for the first page of tests and the two-state checkbox example code for the second page of tests. There might be many tests that use the same example code.

2. Setup instructions

Set up instructions are not in a specific column for the examples in the excel sheet and potentially not needed for these particular example tests. The setup instructions should be understandable and performable by the tester. The tester should be able to follow the instructions however is best for them.

It is possible that "put the assistive technology in reading mode" is a setup instruction for all tests in "reading mode", and similarly for "interactive mode".

3. Abstract Operating Instructions

The abstract operating instructions are not in a specific column. They are similar to the "user task" column but will have to be more specific. They might have to be written in a structure object in order to be programmatically mapped to specific operating instructions. For example, the abstract operating instruction "Navigate to checkbox in reading more" might be structured like:

{
  abstract_operation: "Navigate to {object} in reading mode."
  operation_object: "checkbox"
}

The operation_object here can be different for different tests, for example, we might see the following for abstract operations for a menubar test:

{
  abstract_operation: "Navigate to {object} in reading mode."
  operation_object: "menubar"
}

4. Mapping between abstraction operating instructions and specific instructions for assistive technology

In the excel sheet example, the "Key Commands to Test" are the specific operating instructions for JAWS.

The mapping between abstraction operation instructions and specific operation instructions for a given assistive technology is one-to-many. There can be multiple commands to test for a specific abstract operating instruction (in the column "Key Commands To Test", you can see many key commands listed). For each specific operating instruction, the test expectations must be the same.

5. Test expectations

The "support expectations" column in the excel sheet. These expectation might be reworded to be a question. For example: "Are the checkbox role, checkbox's name and checkbox's boundaries announced for all ways of navigating?". For an inexperienced tested, the "checkbox's name" and "checkbox's boundaries" might not be clearly defined, we will need to be able to supply this information somewhere as well. Test expectations might also be a structure object with a list of the possible answers a tester could record.

from aria-at.