marketcircle / axelements Goto Github PK

Due to race conditions, we sometimes have an issue where an element disappears between the time it gets added to the search queue and it is actually inspected. This is simply a byproduct of asynchronous actions taking place and the UI updating, so we need to handle it in AXElements.

I think a temporary solution for the time being will be to create a convention for understanding if an AX::Element instance is #dead? and check that before yielding to the given block during enumeration. Of course, this only makes the window smaller, since the element could die during the yield phase.

In the long term, AX::Element needs a convention for return values when the reference is dead. Possibly related to GH-37.

Something to think about.

Right now, to type out characters like £ or ¥ you need to hold the option key down and press the corresponding regular key.

We could support parsing those special characters just like we support parsing shifted (upper case) characters.

Allowing a search Qualifier to take a block could allow for more complex search filtering to occur.

It would be set in Qualifier#initialize, and need to be exposed in Element#search and anywhere else we use a qualifier.

The largest problem with the first generation of documentation is that it doesn't focus on a particular audience.

Should it target people new to AXElements? Ruby? Programming?

Would be great to be able to hold Command/Option/Control/Shift while clicking, e.g I want to select multiple things from a list.

Is this possible and will it reduce code size? Will it be faster or slower?

That is, for characters that require you to press and hold shift first.

There is currently a hack for handling upper case letters, but it is not extended for all symbols.

Also, need to get Command/Option/Control working.

This and the grapher are blocking the 0.6 release.

Technically, you can still use "\COMMAND+\s", but this is still an edge case worth handling.

Example of what it is now:

table.row( static_text: { value: 'blahblah' }).static_text( value: 'blahblah' )

Example of what it could be:

table.row_item( static_text: { value: 'blahblah' })  
=> static_text

The idea here is that instead of having to duplicate code, we would just be able to say that we want this element that is part of this row in one method.

This should make GH-25 and future improvements and features easier.

It should also allow adding an optimization phase to generate all caps strings properly.

Notifications suck. They are limited in what they offer by default, and wrapping them nicely is a pain.

Often time, you are waiting for some event to finish and the UI to be updated in some way. For this case, it might be better to just create some conveniences for polling the UI hierarchy to see if certain elements exist. The only problem with this, aside from the fact that it is a polling solution, is that it might be too CPU intensive the way that searching works right now.

Otherwise we would have to handle arbitrary key combinations by adding some other way to encode that information. I think it would be easier to just handle it by starting an escape sequence.

It makes irb output look weird to use inspect in that way. It also makes it hard to tell that he object is actually a qualifier.

I think #describe would be better.

Graphing basics are now done, it is just a matter of deciding how to encode properties of elements into the nodes.

If anyone has ideas or suggestions, feel free to add a comment.

• The node that currently has focus should have bold text for it's label
• Disabled nodes should be faded in some way, probably shading the node gray
• Use node shape to tell which nodes are controls (have actions) and which are not
• Node label should include the identifying attribute for the element

When the mouse needs to #move_to before preforming the click, there should be a check to verify that the #element_under_mouse is what is expected for the click before doing the click.

Not sure if this is important for release. But it will be useful to ship with.

In irb run the highlight command two times, the first time will respect the duration as you input it, the second time will just keep highlighting until you kill irb.

New docs may be needed. This still needs to be evaluated, I'm just creating this now so I don't keep forgetting.

Just something to consider.

Possibly call the API DSL#wait_until

Waits until something goes away. We need this sometimes, such as waiting for async searches to finish where we only know if the busy indicator goes away or stops.

FUUUUUUUUUUUUUUUU

So bear with me here. I'm not entirely sure if this is an AXElements thing, a minitest thing, a MacRuby thing...

If I add a failing test to any of AXElements' tests, I get the following at the end of the test run:

268 tests, 594 assertions, 2 failures, 0 errors, 7 skips
rake aborted!
Command failed with status (1): [/Library/Frameworks/MacRuby.framework/Vers...]
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/lib/ruby/1.9.2/rake.rb:993:in `block'
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/lib/ruby/1.9.2/rake.rb:1008:in `sh'
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/lib/ruby/1.9.2/rake.rb:1027:in `ruby'
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/lib/ruby/1.9.2/rake/testtask.rb:115:in `block'
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/lib/ruby/1.9.2/rake.rb:1110:in `verbose'
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/lib/ruby/1.9.2/rake/testtask.rb:100:in `block'
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/lib/ruby/1.9.2/rake.rb:632:in `block'
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/lib/ruby/1.9.2/rake.rb:629:in `execute'
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/lib/ruby/1.9.2/rake.rb:595:in `block'
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/lib/ruby/1.9.2/monitor.rb:201:in `synchronize'
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/lib/ruby/1.9.2/rake.rb:588:in `invoke_with_call_chain'
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/lib/ruby/1.9.2/rake.rb:581:in `invoke'
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/lib/ruby/1.9.2/rake.rb:2042:in `invoke_task'
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/lib/ruby/1.9.2/rake.rb:2020:in `block'
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/lib/ruby/1.9.2/rake.rb:2020:in `block'
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/lib/ruby/1.9.2/rake.rb:2059:in `standard_exception_handling'
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/lib/ruby/1.9.2/rake.rb:2014:in `top_level'
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/lib/ruby/1.9.2/rake.rb:1993:in `run'
/Library/Frameworks/MacRuby.framework/Versions/0.11/usr/bin/macrake:31:in `<main>'

It's pretty freaking annoying since it makes it harder to see what happened in the test run. Any idea what's going wrong?

Most core methods will only log errors when a method returns an error code. This is bad because accessibility will often not crash until a while after the first error. AXElements could raise an exception for certain error codes; possibly obsoleting the logging method.

The only thing you will ever do with such an object is unwrap it, so there is no point in not doing that in Core itself.

Similarly, there is no need to require items be wrapped when given to Core, we could wrap them in Core.

This still fits with hiding the CoreFoundation abstraction. The mechanism for wrapping already exists, #to_axvalue, and a similar mechanism could be added for unwrapping.

I consider this a bug as it causes Accessibility::Core to get mixed into the TopLevel when you load AXElements. This pollutes the hierarchy with stuff that shouldn't be there.

The only way to fix this is move the wait method from Core and expose the ability to unregister notifications through AXElements. I think this can be done without breaking backwards compatibility in DSL.

Since it is the only part of ActiveSupport that AXElements depends on, could we just bring in that part of Active Support and then depend on 2 fewer gems? Less gem dependencies is a big for MacRuby.

Just a thought. It might make testing much easier in some cases.

Such as "a+b+c". Right now the only way to make a key combination is to have a custom escape as the leading key in the combination.

Would be nice to add a method that can scroll through a given list until it finds a given element.

Would need to include a way for it to know the list is complete so it stops if not found.

For instance, holding the shift key down while typing a string of upper case letters.

This needs to happen entirely in the lexer, the event generator can already handle this.

At the moment, the API offers almost exactly the same feature set, but some APIs are differently named or available in a less OO manner.

We should adjust the latter case so that going between them is easier.

This lets us eliminate a lot of checks and possibly handle some AX API errors without raising exceptions.

This is just a TODO reminder for tomorrow. I also hope it will break apart some coupling between the AX and Accessibility namespaces.

Doh

I think there should be a special version of the read me that displays the links on github correctly (and maybe link to the pages in the ruby docs)

This is because when you see the sub tree in a search failure the subtree is being generated after the failure, so we will not catch timing based failures.

This means that the enumerator will have to cache all the children that it enumerates over.

We should be making use of AXUIElementCopyAttributeValues() in order to fetch children in batches. This should speed up searches over tables.

It may need to be implemented using proxies for children that, when sent messages, trigger a batch to be retrieved.

Descendent is the adjective usage: "the descendent y of x"

Descendant is the noun usage: "x is a descendant of y"

Assertions for minitest/unit exist, but why not also support minitest/spec?

We need a better way to move in menus. Moving mouse to the centre of objects causes issues with crossing over menu bounds, which leads to some strange and unpredictable mouse movements.

Poorly documented by Apple, and getting them to work properly has been a perpetual pain.

While polling is less efficient, the interface is easier to use and easier to maintain, and it exists now.

We talked about this over ichat one day. There is no safety for when invalid objects are called right now.

Why?

Because the current inspector operates at a lower level of abstraction than what AXElements prefers to use, users of AXElements need to be aware of the name translations that go on underneath. The accessibility inspector also has a weird way of setting the value for an object.

At the same time, we can't get rid of the inspector because it simply too helpful. But it could be more useful if it could help generate script code. We can't do this without the source code to the inspector and the source is mostly proprietary code (some of it is available as sample code).