[General Question] Are there plans to support individual element testing? about axe-windows HOT 14 CLOSED

@chingucoding, we're in full agreement. The unit test approach is a lot of work and is out of scope for our current plans. That said, it's something that others could do if they wanted to create an app to host the control and call into Axe.Windows--and in that case there's really no need to scope to anything smaller than the app, provided that the host app passes all Axe.Windows checks.

For the Config.Build route, our leanings are definitely toward a property bag to locate the starting element. If the test code already has a UIAutomation element, then the user could just pass the element's RuntimeId via the property bag, and we'd get the correct element every time. The tricky part is that the RuntimeId is dynamically assigned at runtime, so it can't be baked into a test like some other properties could be. This is the sort of feature that we'd really prefer to develop with a partner who will be actively using it, so that we can be sure that the solution meets a specific need that we can validate.

I'll leave this request with the "Needs investigation" status so people can upvote it if they wish. When we're ready to loop back to it, we can reach out to the upvoters to try to find partners to help define and implement the feature.

Thanks again for your interest in Axe.Windows!

from axe-windows.

Hi, @chingucoding! We discussed this today, and we love the idea of having a unit test library that would allow control authors to quickly and easily test their controls as part of their unit tests. As we discussed, that library would be doing a lot behind the scenes:

• Create a tiny app that allows the control to be hosted
• Start the app
• Run Axe.Windows against the app
• Capture the results

All of this is technically feasible, but it's not something that our team will be able to pick up for the foreseeable future. We'd be happy to help with integrating Axe.Windows into such a library (which could be in the Axe.Windows repo or in some other repo), but it's not something that our team currently has the bandwidth to take on. We'd certainly welcome PR's to try to solve this problem if you'd like to contribute in that way.

Thanks again for you interest in Axe.Windows!

from axe-windows.

Thanks for the update @DaveTryon ! I will definitely try to get something like this working just out of curiosity. If it seems feasible to use and works reliable enough, I'll create a PR and we can see where we go from there.

from axe-windows.

Hi, @chingucoding! We've discussed potential ways to use Axe.Windows to scan a subset of the UI Automation tree, but the sticking point has been how to specify where the scan should begin. There are a few potential solutions, but each one is very usage-specific. If you don't mind, here are a few questions that might help us figure out how feasible it would be to support this in your scenario:

1. Are you triggering the scan from the CLI or by calling the automation interface from managed code?
2. What parameter would you ideally use to identify the starting point of the scan? For example, could you pass in a Window object or a window handle or some set of UI Automation properties that would uniquely identify the starting point?

Thanks!

from axe-windows.

The team requires additional author feedback; please review their replies and update this issue accordingly. Thank you for contributing to Accessibility Insights!

from axe-windows.

I just noticed that there actually are two ways to interpret my issue. My initial idea was to have code behind testing, i.e. that I can pass actual WPF/UWP/WinUI elements to the scanner and have them scanned for accessibility issues. However that would require a lot of work and take some time, especially since this would be a huge difference to how axe-windows works now.
The reason why I would like that kind of scan is that starting apps takes times and in practice, interaction tests can be flaky. Creating just an element in code and calling a library however is fast and does usually not end up being flaky.

That said, there is also a point to be made about scanning just parts of an app, e.g. having to rescan the entire app just because I entered text or opened a different tab on an option panel is not ideal.

Q: Are you triggering the scan from the CLI or by calling the automation interface from managed code?

I am triggering scans from managed code as part of the interaction tests, e.g. after opening a page, I scan the whole app to find accessibility issues on said page.

What parameter would you ideally use to identify the starting point of the scan? For example, could you pass in a Window object or a window handle or some set of UI Automation properties that would uniquely identify the starting point?

This is a really good question. For CI purposes, providing a window or object handle does not sound ideal. Maybe we can look at how WinAppDriver and Selenium handle this? Those have different ways to get to a specific element, but the common ones are finding them by their name or automation ID. This could be a reasonable approach for axe; Users could specify the name/automation-ID and axe scans the first or all elements with that name/id?

from axe-windows.

Thanks for the extra input, @chingucoding! The code-behind testing is an interesting idea, but since Axe.Windows requires UI Automation, the actual implementation would probably be for Axe.Windows to have a platform-specific dummy app that hosts the control, then scans it. There's some upside there, but of course it also brings all of the complexity/reliability issues of creating the app, starting it, scanning it, shutting it down, and of course it would be testing the control in an "artificial" context that might not match its actual use.

For parameters, you're hitting exactly the same sorts of issues that we did:

Recognizing that there may be no "perfect" solution to this problem, we'd certainly welcome PR's to support target specification in the Config.Builder class in a way that works for their scenarios. We'd prefer to unblock some users if possible, but we also need to make sure that it's done in a way that won't limit targeting options that other users may need in the future.

We have at least one customer who scans the whole app, then walks the set of ScanResults for items that are children of a window with characteristics that they specify. It's unwieldy and certainly not a generalized solution, but it was the best available option for that customer.

from axe-windows.

The team requires additional author feedback; please review their replies and update this issue accordingly. Thank you for contributing to Accessibility Insights!

from axe-windows.

Another option how to do the code behind testing would be to provide an abstraction layer for specific UI frameworks such as UWP or WinUI 3 that maps from the controls or rather AutomationPeers to the internal A11Y objects that are used to validate controls. However that is no easy task and would probably require a lot of work.

Having a native app to start and place the control seems like a reasonable workaround, though I would argue that if the axe.windows where to do this under the hood, there is little benefit in not implementing that oneself to have more control over the entire scenario.

Regarding updating the Config.Build, I think there really is no silver bullet here. All possible ways have their downsides. Maybe we could allow the scanner to take the automation object and let users create the automation object just how they want to. So if you want to scan the entire app, you just pass the automation object of the app. If you want to scan a specific page, the user can get the page like they want and pass the automation object to the scanner.

from axe-windows.

This issue requires additional investigation by the Accessibility Insights team. When the issue is ready to be triaged again, we will update the issue with the investigation result and add "status: ready for triage". Thank you for contributing to Accessibility Insights!

from axe-windows.

Great, thanks! If there is anything I else can do, I am happy to help.

from axe-windows.

After a couple of hours, I found a big problem for my use case: Axe.Windows can not be used from inside the UWP container making unit testing UWP controls impossible since creating them has to be done inside UWP. So writing the test should be done inside the UWP app or would be very inconvenient. The advantage of the UWP test project is that it always has a window and we could have used that instead of having to start an app, wait for that and then close it. But well, the UIAutomation namespace can not be used from inside of UWP (beside a few exceptions listed here: https://docs.microsoft.com/en-us/uwp/win32-and-com/win32-apis#apis-from-uiautomationcoredll).

Should the limitation of non UWP sandboxed processes for scanning be pointed out somewhere?

For other UI frameworks, this kind of unit testing should definitely work since UWP apps are the only ones in a sandboxed and thus the only ones not being to able run the scanner. What are your thoughts on this @DaveTryon ? I could work on trying to get this to run with WPF or Winforms, but this would help next to nothing for me given that I only work on UWP projects. With WinUI 3, this whole unit testing thing could change though since people will be able to create a WinUI 3 desktop app that will be able to run the scanner and as such allow an easy way of unit testing controls.

from axe-windows.

@chingucoding, Ah, UWP sandboxing! That's something that I really hadn't thought about, but it makes sense in the broader context. It also brings up an interesting point, which is that a unit test library like we're discussing here would probably be best in a separate repository that consumes Axe.Windows via its NuGet package. That would allow the Axe.Windows repo to focus on accessibility tasks, and allow the test repo to focus on testing tasks. Our team could help with the accessibility aspects, but we're not really staffed with experts in test frameworks.

Given that you're mainly focused on UWP, a usable test framework would probably need to start the target app in the sandbox, then scan from a non-sandboxed app. The sandboxing may prevent the test framework from starting a non-sandboxed process, but if the non-sandboxed process is already running, the sandboxed process should be able to communicate with the non-sandboxed process to request the non-sandboxed process to initiate a scan. It might not be pretty, but it ought to be possible, even if the file system ended up being the communication channel between the two processes.

from axe-windows.

Yes, I agree with you, I think it makes sense to have that as a separate library with dependency on Axe.Windows. For WPF and other Win32 UI libraries, I think we can do something easy for testing (create UI objects, pass to library and scan them).

I was envisioning something like this:

var myControl = new MyControl();
var scanner = new Axe.Windows.ControlTesting.Scanner();

var results = scanner.Scan(myControl);

Assert.AreEqual(0, results.Length);

For UWP, I see an issue. Where should the code creating the UI objects? For UWP controls, they need to exist in a UWP context. As you already guessed, the UWP sandbox not allows people to easily start an external process. So the test would most likely have to start in a non UWP process, somehow create a UI object, start that and pass it to the library. The key issue is that the creation of the tested control seems a bit problematic. But given that you are the accessibility experts, I'll also ask on the WinUI repository if they have a better idea.

Thank you for your help so far!

from axe-windows.