Measurability vs testability about silver HOT 27 CLOSED

Hi @ alastc and all,

You wrote:

We (the AGWG/W3C) setup Silver to:
“perform preliminary development of a new version of Accessibility Guidelines following a research-focused, user-centered design methodology to produce the most effective and flexible outcome.”

The requirements document isn't proposing a definate shift, it is reflecting the requirements that have come from the research and outlining potential approaches.

The charter says AG will:

…Incubate requirements for a major update to WCAG to address usability and conformance challenges…

I think that the sticking point may be the words "preliminary development" and "Incubate". We may not have a shared understanding of what those terms entail.

If it is not a "definite shift", I think something like @DavidMacDonald proposed paragraph clarifies it. David suggested:

"a thorough investigation will be conducted into widening the evaluation of the guidelines to include various models that would bring to play other considerations beyond pass/fail testing under a new term, measureability"

Then perhaps add that "Preliminary work in this regard has been initiated."

from silver.

Does not true/false not correspond to pass/fail?

It does now, but when you expand that to include measurable, a pass could also be:

• 90 or more out of 100 terms used are plain language.
• 8 / 10 score for readability.
• For alt text, that all instances on a ‘critical path’ pass, but not necessarily all on the page.
• That you have done an applicability assessment, and decided and documented that usability testing needs to be done.

Those are just ideas, the point is that true/false statements are only one possible method of passing, that’s kinda the point here.

The requirement for measurability in this doc means it is in scope for exploration and testing, not that there is a foregone conclusion.

from silver.

@DavidMacDonald: I appreciate your concerns and I think we have some confusion of language. "Testability" and "Measureability" don't have an intrinsic difference in meaning. We are using that language to draw a distinction between the WCAG 2.x reliance on statements that can be evaluated with a pass/fail test, and the Silver direction to allow a more broad way of testing, that can still be used in a regulatory environment. Some of the success criteria from COGA and Low Vision task forces were deferred to Silver because they didn't fit the WCAG 2.x structure of testable statements. Therefore, Silver has to have a new structure that will include the needs of people with disabilities that do not fit into WCAG 2.x.

The Silver Design Sprint involved lawyers, policy makers and regulators. The idea for the Conformance model that we are currently developing came from an accessibility-specialist lawyer in the US Office of Civil Rights. As we continue to develop the Silver prototypes, we will be user testing them with a number of people, including lawyers.

We need to do this to serve the needs of people with disabilities who were not served by WCAG 2.1. That's why it's a Requirement. We have a pretty good idea of how to do it, we already have experts (including lawyers) who have taken a critical look at the basic ideas, and we feel comfortable that we can make it a requirement.

from silver.

@jspellman

We are using that language to draw a distinction between the WCAG 2.x reliance on statements that can be evaluated with a pass/fail test, and the Silver direction to allow a more broad way of testing, that can still be used in a regulatory environment.

I agree that this is the intention of the Silver TF, and it is grounded in reputable recommendations. However, we don't have consensus at this point and have not ever discussed formally on the WG this foundational change in the evaluation of accessibility, and yet this requirements document is presenting it like it has been decided by the WG.

The Silver Design Sprint involved lawyers, policy makers and regulators.

I think that is amazing, but it still has never been discussed by the working group. A new conformance model has to be discussed, not just implemented in the Requirements doc as a fiat. I think the requirements document should say something like:

"a thorough investigation will be conducted into widening the evaluation of the guidelines to include various models that would bring to play other considerations beyond pass/fail testing under a new term, measureability"

from silver.

I agree that this is the intention of the Silver TF, and it is grounded in reputable recommendations. However, we don't have consensus at this point and have not ever discussed formally on the WG this foundational change in the evaluation of accessibility, and yet this requirements document is presenting it like it has been decided by the WG.

To that point, we specifically brought the Requirements draft to the Working Group with the expressed purpose of working to consensus on the points within it. Not just on the requirement around the conformance model, but all of the requirements included in the draft (and to identify requirements we might've left out!).

A new conformance model has to be discussed, not just implemented in the Requirements doc as a fiat.

Very much agreed! We haven't proposed a new conformance model, but we do know through the research that we've done that we need something different than the current conformance model's strict pass/fail requirement in order to (among other things) include guidelines around some needs of people with disabilities that can't go into WCAG because of it.

Through the research and prototyping that we've done so far, we've found that measurability addresses several things in one shift, but we haven't yet done the amount of prototyping and testing to know exactly what that would look like in terms of a conformance model. Before we do that work, though, we want to get consensus that we need to do that work, if that makes sense.

from silver.

To that point, we specifically brought the Requirements draft to the Working Group with the expressed purpose of working to consensus on the points within it.

I don't think the requirements doc is the proper context for this huge shift. I think that research and consensus building is a result of our work, not the beginning premise for it. If the TF has started that process on our behalf, that is amazing. We all want to find a viable way forward that builds on the success of WCAG.

The AGWG should look carefully at the research and the statements of the lawyers, and contact them for further clarification and articulation so that we thoroughly understand the issues and the proposals. This kind of a fundamental change is not something that should be decided by the AGWG over a 2 week CFC which marks the *beginning of our work" on the next major version.

My understanding of the job of a TF is to aid the WG in its mandate. This paragraph seems to turn it the other way around. My proposed change is something like:

"a thorough investigation will be conducted into widening the evaluation of the guidelines to include various models that would bring to play other considerations beyond pass/fail testing under a new term, measureability"

from silver.

Hi @DavidMacDonald,

Sorry for the duplication, I'd like to reply publically as well...

We (the AGWG/W3C) setup Silver to:
“perform preliminary development of a new version of Accessibility Guidelines following a research-focused, user-centered design methodology to produce the most effective and flexible outcome.”

The requirements document isn't proposing a definate shift, it is reflecting the requirements that have come from the research and outlining potential approaches.

The "Measurable Guidance" bit is under the heading "Opportunities", it is a possible avenue for the guidelines which needs to be explored.

True/false is one measurement (and Silver will likely include the majority of WCAG 2.x as a starting point), this is opening it up for other methods of testing. I don't see it as removing 'testable', but allowing for more methods of testing.

The prototypes will attempt to meet the requirements, i.e. try to build in an SC which uses measurability. It doesn’t say that none of the SCs will be testable, it is allowing for (some) SCs to be measurable. (And I’d like to allow for some that can be met by saying you’ve done a particular process, see pull request #24).

Multiple models / prototypes will be created tested, discussed. If you would like to create a prototype that tries to meet the requirements (e.g. covering COGA SCs) that only uses ‘testable’, please do join the Silver TF. It is both a TF and a community group for maxumum participation.

However I would note that in general, a consensus-process is not useful when you are exploring various ideas in a divergant fashion. That will be useful when it comes to narrowing down the ideas to what will work best, but we are not at that stage yet.

Cheers,

-Alastair

from silver.

What happens if after investigation that for some reason we can't make the wider conformance models work? Are we meeting our requirements doc if we have a conformance section that says "testable" pass/fail statements without wider means of measuring conformance. Are we now committed to going with the term "measurable" in the final, or in the first iteration of Silver, and are prevented from staying with "testable"?

from silver.

What happens if after investigation that for some reason we can't make the wider conformance models work?

Then we don't go with that. These requirements are for showing the scope and aim of the process, they do not decide the outcome.

I disagree with adding:

"a thorough investigation will be conducted into widening the evaluation of the guidelines...

Because that is what the whole process is! Not just on the point of conformance, but all the other 'opportunities' as well.

from silver.

from silver.

I believe @jspellman has in mind to do an overview of the process, to answer this and other questions along those lines. That would be a more suitable place, and this doc can be the requirements / scope. We can pick it up on the call tomorrow.

from silver.

David, is that the location that you had in mind?

I would integrate it into the bullet point on Measurability. It would like something like this:

1.2.2 Conformance Model
Investigation into Measurable Guidance: Certain accessibility guidance is quite clear and testable. Others, far less so. There are needs of people with disabilities, especially cognitive and low vision disabilities that are not well served by guidance that can only be tested by a pass/fail statement. A thorough investigation will be conducted into widening the evaluation of the guidelines to include various models that would bring to play other considerations beyond pass/fail testing under a new term, measureability. Multiple means of measurement may allow inclusion of more accessibility guidance.

from silver.

Hi @DavidMacDonald and all,

I think your latest proposed verbiage is a good compromise and could fit into the "Opportunities" section.

Given that, perhaps we should consider placing a note in the "3.1 Multiple ways to measure" section. That section lists the actual Requirements. Maybe something such as the following could work:

Editor's note:

Preliminary research indicates that different guidance has potential for different measurement. Silver is pursuing other solutions at this time. For example, guidance could possibly describe "minimal" and "better" methods of conforming to a requirement or allow user testing and usability testing as a method of conformance. We will relook at this requirement in a future version of this document.

from silver.

From the call today we were told:

• This is a first draft of the requirements document
• It will not be finalized for about a year
• There will be at least 2 more drafts
• If measurability doesn't work out due to industry pushback, implementation problems, legal complications, or evaluation problems, etc., we are not committing to measurability and we can return to testability without violating any previous decision, because a decision has not been made.

I think this issue should stay open until the WG formally makes a decision that this is the way the Guidelines will proceed.

from silver.

PS I think there are probably 2 changes to the paragraph that better reflect the intention of the Silver TF.

Measurable Guidance: Certain accessibility guidance is quite clear and <remove>measurable</remove><add>testable</add>. Others, far less so. There are needs of people with disabilities, especially cognitive and low vision disabilities that are not well served by guidance that can only be <remove>measured</remove><add>tested</add> by pass/fail statement. Multiple means of measurement allow inclusion of more accessibility guidance.
https://w3c.github.io/silver/requirements/index.html

from silver.

Hi David,

I think there is a confusion between 'testable' and the type of testable we are used to in WCAG 2.x

Something which is measurable is by definition testable. If you can measure it, that is a form of test.

What we're getting at is that only using binary answers from testing (true/false) is not sufficient across the board.

I agree that the paragraph doesn't quite make that clear though, how about:

Measurable Guidance: Certain accessibility guidance can be tested in a binary fashion, where a page either meets the requirement or does not meet it. However, there are some needs of people with disabilities, especially cognitive and low vision disabilities that are not well served by guidance that can only be measured by pass/fail statement. Multiple means of measurement allow inclusion of more accessibility guidance.

from silver.

Something which is measurable is by definition testable. If you can measure it, that is a form of test.

Does not true/false not correspond to pass/fail? To capture what Detlev said yesterday in the meeting.

We'll probably all agree on sites that are really bad at one end and really good at the other but there is a wide plateau in the middle which will be very hard to determine whether they pass or not.

This will be a challenge for any model that doesn't have a true/false set of tests, and I'll be interested in proposals that are advanced to overcome this.

An Example of this is:

• When is the language simple enough to pass, and when is it not simple enough to fail?

from silver.

90 or more out of 100 terms used are plain language.

would not the true statement be over 90 and false be under 90? (assuming the group could agree on what "plain" means)

8 / 10 score for readability.

the true statement would be 8 or more, false would be 7 or less. (assuming there is a score our there that works across languages???)

For alt text, that all instances on a ‘critical path’ pass, but not necessarily all on the page.

I hope we don't relax our alt text requirements. There has been reasonably unity about alt over the years and not much push back from industry.

That you have done an applicability assessment, and decided and documented that usability testing needs to be done.

"True" would be that it is documented, "false" is that it hasn't been documented.

One strategy to address issues that are difficult to test, could be that for SCs (or whatever they are called) that can't meet testability conformance are documented and published publicly linked from the home home page.

So the accessibility link at the bottom of the home page would go to a page that has statements something like this.

Here are the steps we took to simplify the language on this site

• Made an editorial pass to ensure a Gunning Fog index of 9 for non-technical pages.
• Conducted usability tests with 5 users with disabilities, one of which was blind, one of which had low vision, one who had red/green color blindness, and who had a learning disability, and one who had dexterity limitations.

The pass/fail condition is based on whether the statement is posted .... from there we can let public scrutiny be the determining factor.

from silver.

Hi @alastc and all,

Alastair wrote:

I believe @jspellman has in mind to do an overview of the process, to answer this and other questions along those lines.

That would be helpful. I ran across the document, "Draft of Silver relationship to W3C", in the Silver Google Drive.

But further clarification of process and relationships would be appreciated.

from silver.

I think we're just getting down to a language issue now, the examples above were just spit-balling, I wasn't saying we should change alt text.

The current SC are statements about the content that you answer as pass or fail. If we use a score (numbers, percentage etc.) you could build that into the statements and force it into a true/false approach, but you'd be missing an opportunity.

There is room for exploration of other methods. For example, if there were a bunch of SCs that are scored out of 10, you could then take an average and pass/fail on that basis.

The point is that we are not saying criteria are not testable, but we are exploring other options in how that reaches a pass/fail.

Secondarily on the process-approach point, security is a good parallel in terms of approach, and the ISO 27001 could provide some clues here. (I'm not saying we re-create that, it is large and difficult to acheive, but it might provide some methods we can try.) For example in ISO 27001 the bits that caught my attention were that you:

• Create an applicability statement, saying which aspects of the standand apply to you. (A bit like whether you use multimedia on the site.)
• Document what you are trying to acheive.
• What you need to acheive can be influenced by the size & resources of the organsation.
• There are a bunch of practical things to aim for (like the current SCs).
• You can optionally get a 3rd party to audit your documentation and results.

Anyone familiar with ISO 27001 will be able to poke holes or say that isn't quite right, I'm still investigating it with a new collegue who's been through it a couple of times.

However, that kind of framework would provide a way to include certain user-centred-design approaches such as usability testing & IA methods which would be very helpful for including COGA requirments and testability.

Anyway, the requirment is to try including SCs that are measurable, is there an objection to that now?

from silver.

I think we need language built in to allow us to back track to testability if necessary, rather than saying

"we have to make non testable models work, its in our requirements doc."

Particularly during transition releases of silver (after 2.x), of which there may be several with this 18 month dev cycle. If not, then I fall back to the assurances given during our meeting.

• This is a first draft of the requirements document
• It will not be finalized for about a year
• There will be at least 2 more drafts
• If measurability doesn't work out due to industry pushback, implementation problems, legal complications, or evaluation problems, etc., we are not committing to measurability and we can return to testability without violating any previous decision, because a decision has not been made.

I think this issue should stay open until the WG formally makes a decision that this is the way the Guidelines will proceed.

from silver.

from silver.

1. Silver is not getting rid of testing. Everything will be tested.
2. Silver wants to add additional tests that go beyond a true/false statement. The true/false statement is what kept many of the cognitive SC proposals out of 2.1. We HAVE to do better. That's why it's a requirement.
3. Silver is not removing pass/fail for a site. Silver needs a more flexible conformance than the WCAG 2.x model. The current model does not meet the needs of industry. For example, a big social media site that is updated thousands of times a second, can never meet the conformance of WCAG 2.0 because they can't test it the way WCAG 2.0 requires. In 2007, few people anticipated the boom of social media. WCAG 2.0 conformance wasn't designed for single page apps, dynamic content, or social media because they either didn't exist, or were in their infancy. Now, Silver has to have a conformance model that covers those situations and has flexibility to accommodate new technologies.
4. Silver has already done the investigation, now we are at the stage were we are proposing solutions and refining the details of those solutions. Then we will test Silver with users. Before AGWG has to charter Silver, we will have the data that shows that Silver is accepted by industry, lawyers, policy makers and all the other stakeholders we defined 18 months ago.

from silver.

Hi @jspellman ,

Jeanne wrote:

1. Silver is not…
2. Silver wants…
3. Silver is not…
4. Silver has…

When you speak of "Silver" do you mean the Silver Community Group Participants and Silver Task Force Participants?

Thanks,
Laura

from silver.

Hi @lauracarlson , we are running them as one group. We share the email list, the wiki, the action items, and the conference calls. The items I listed are a consensus of the active members of the joint group.

from silver.

Thank you, Jeanne.

from silver.

As discussed in the Silver meeting of 4 December 2020, this thread will be closed as being overly broad and then overtaken by events. If there are specifics that still need to be addressed, please start a new specific thread.

from silver.