dequelabs / axrl Goto Github PK

There are a few classes and properties that have almost the same name, with only the capitalisation of the first letter as the same. Here are the ones I found:

• TestEngine & testEngine
• Remediation & remediation

This is too confusing, and should be avoided. We'll need to rename one of the two.

One of the limitations of JSON-LD 1.0 is that it does not allow ordered nested lists. Therefore a selector like this, is not permitted:

{
  "@context": {
    "selector": {
      "@id": "https://axrl.org/selector",
      "@type": "https://axrl.org/CSSSelector",
      "@container": "@list"
    }
  },
  "selector": [
    ".foo", [".bar", ".baz"]
  ]
}

Allowing this has been approved fir JSON-LD 1.1, which is currently in draft. I want to avoid using draft features. There are two ways to deal with this situation:

1. We abandon the current selector format in favour of a more verbose one
2. We do not declare in the context that the selector has an order, and adopt this once JSON-LD 1.1 is more stable.

I'm going for option 2, since it seems to be the least impactful. I'm going to keep this issue open as a record of that decision though, and a reminder to revisit this once JSON-LD 1.1 makes it to Candidate Recommendation.

The result of not defining this as an ordered list is that when expanding / flattening the JSON-LD, the nested array will be flattened ([['foo', 'bar'], 'baz'] becomes ['foo', 'bar', 'baz']). In practice, that shouldn't be an issue. Since iframes can't be shadow roots, there shouldn't be any confusion about what to do with that selector.

Feedback from Peter Durham:

The schema docs don't seem to distinguish between single-instance and multiple-instance properties, which sometimes makes the intent unclear (I haven’t dug into the details of JSON-LD yet so maybe it’s covered there? Looks like https://json-ld.org/ might be a good place for me to start learning more.)

Feedback from Peter Durham:

Would be helpful to have the any/all/none information [on Remediation] always be structured rather than be unstructured text, as that makes it easier to format for presentation.

The word "Node" means something in HTML, but node is meaningless in iOS and misleading in Android... conflicts with AccessibilityNodeInfo, frequently shortened to Node. However, the Node in question and an AccessibilityNodeInfo would not be the same thing.

Would prefer if we rename this to be more technology agnostic.

Alternatives: view, control

Priority: moderate

Feedback from Peter Durham:

Seems strange that remediation property of TestFailure can either be a Remediation (with a text property) or just text. Is this intended to cover the case of a failure having only one solution?

The current design is seriously in flux, making it difficult to see if all of the examples are up to date. It'd be good to have a validation function. Firstly, to validate the examples, but also it'd be good to be able to publish this to NPM for reuse in projects where we want to use AXRL.

Feedback from Peter Durham.

Remediation advice should not be a required property. In fact, in the example you gave, the remediation advice already feels like secondary information:

"description": "Web pages must have a title",

"remediation": "Add a title to the web page",

Feedback from Peter Durham:

HTMLPage Has a crawlerDepth property in the schema, is this describing the depth in the crawl at which the current page was found? In a crawl scenario, would we expect results in a single AXRL instance or spread across multiple instances?

Feedback from Peter Durham:

Would be helpful to have a property for a test case name, for named automated tests that exercise different scenarios on the same page.

Feedback from Peter Durham:

Might be helpful to have an id for test engines, so testEngineId + ruleId maps consistently to a unique rule (to allow consistent decoration of results, for example with links to relevant WCAG criteria)

Feedback from Peter Durham:

In the multiple test example, HTMLPage is used as the container, with a tests property not in the schema. Will there be a way to include tests of different pages in one structure?

Selector is a web specific as is the ability to grab "code", which to be honest, even code snippet is misleading here. This is an html tag. Regardless, both Selector and CodeSnippet are impossible in mobile.

I think the best solution for this is to use the @view solution you've used prior. results[0].@view and allow the contents of @view to be defined by the type. Or perhaps, that's already the way it works and I'm misunderstanding?

Priority: high

Feedback from Peter Durham:

Some scans could happen during development or builds; what are the implications for subject.release? (Presumably most data in subject is passed in to the test engine from the test runner?)

Should it be possible for someone to file a manual test report in axrl? For example, only the keyboard was needed, no special tools/engines and the axrl correctly maps to WCAG 2.1

Feedback from Peter Durham:

I could imagine reading in AXRL from two test runs and detecting differences (fixed items, new items). It would be interesting to then write out merged AXRL with when-first-seen and when-last-seen information. Sounds like we could do that by

Natural choices for JSON-LD and RDF data structures.

Use case: Tester1 submits a test result (/TestResultA). Tester1 links that test result to a test target (subjectOf:url). Tester 1 links the test result to the Tester1 object (testedBy:url). Later, Test 2 gets the same result using a different tool/technique. Tester2 can post their test result

• /TestResultB => validates => /TestResultA
• /TestResultB => toolUsed => ...
• /TestResultB => testedBy => Tester2
• /TestResultB => subjectOf => Target (previously created)

This makes queries against the target very straight-forward.

Feedback from Peter Durham:

It would be helpful to have the "id" for each rule (we use for tags and stats)

Feedback from Peter Durham:

This name seems awkward to me, something like TestRun might be clearer (or perhaps there's a different choice that makes a clearer distinction between Test in StateTest, TestSubject and Test in TestResult and TestTarget; the former is an invocation of the engine, and the latter is the result of the evaluation of a criterion)

Thinking about consuming, storing, and deduping this data: I don't think time is relevant. The ETag, last modified date, or SRI of the test subject, the test engine, and the client characteristics (e.g., preferred language, OS, high-contrast mode, etc.) is something hashable and repeatable. Date/time data will only make it harder to dedupe and should be stored as metadata with the record but not part of the test itself.

Feedback from Peter Durham:

Description looks to have been copy/pasted from TestResult

Feedback from Peter Durham.

Feedback from Peter Durham:

Didn't see a reference to this type; is this expected to be a screenshot of the whole test subject at the beginning of the test, or a screenshot showing/highlighting a particular test target in a test result, or both?