acecentre / relaykeys Goto Github PK

"I can't get the device operations to work (even when connected to a tablet and working it lists no devices).

I have issues getting my connected target devices (android phone and an old Windows tablet) to reconnect. I can pair up and RelayKeys works fine, then turn off or take it out of range. When trying to re-establish connection later I have to force forget pairing and re-pair. I assume this is a behaviour of the targets but it happens on both. I have some BT mice and keyboards that reconnect smoothly so maybe it's a problem with the BLE Friend?"

Device management commands are available through the serial API but it would be useful to use these commands through the daemon/CLI. When the service is running it locks the port so to use the serial API you have to stop the service first and run up a serial terminal, use AT commands and then restart the service again. To control pairing mode or to switch target devices etc from a script or from a communication aid while the service is running it would be better to be able to use CLI calls.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

We need to create a simple installer & uninstaller of the service, cli and qt binaries - with a QT shortcut being available on desktop/program files shortcuts

Needs some thought this..

My suggestion:
We provide two Forms of mouse control:

1. Mouse free-flow window

The window acts like a glidepad. You move the mouse towards the top left - it moves the mouse on the second screen in the same way towards the top left of the screen.

2. Mouse control with buttons

A window is provided with buttons like so (credit to Dynavox for this.. I've basically pinched the idea but loads of people have replicated this in other ways):

Clicking on these buttons (or activating the relevant keyboard shorcut for each button) - activates the mouse. Note Continuous = Continually moves until user presses an Incremental or action button. Incremental = Moves the cursor in small amounts (5 px at a time for example).

A way of switching between paired device 1, 2, 3, 4 etc..

So I copy this text into the paste buffer running relayKeys on Windows:

https://paste.ee/p/FAEbB

And then we get this on the mac:

https://paste.ee/p/B6OJq

But whats really odd is that then, on the mac I cant type properly. Its like a Key is being held down.. Everything then comes to a halt. I can't send any more commands to relaykeys. Logfile isnt showing anything

So 3 Problems:

Numbers arent coming through. They are getting shifted.
Something is going wrong as its stopping the sending of text
A keypress is getting stuck down.

UPDATE

Looking at this - I'm wondering if something is going wrong with our board. Like a memory problem..

So relaykeys-cli is working but then it gets to a key and I'm getting a timeout on that key.. but its sending that one key over and over again..

https://www.dropbox.com/s/thcf8qakjtk113w/IMG_3240.MOV?dl=0
https://www.dropbox.com/s/2erxix87d8594iw/IMG_3241.MOV?dl=0

I then pull the board out, Plug it in again and it sends sending text - but its all garbled. Leave it a little bit longer and the text starts coming back in order again.

https://www.dropbox.com/s/p9me7qp2rahmkd4/IMG_3242.MOV?dl=0

@hosseinamin "This sounds like some sort of communication or memory issue. It can be an issue inside relaykeysd"

feature

Describe the solution you'd like
Use relaykeys on iOS and other OS that have no USB connection.

To do this feather needs to run in dual mode. Eg https://learn.adafruit.com/introducing-the-adafruit-nrf52840-feather/dual-roles-bleuart

We also need some example JS code that demonstrates the RPC sending code

Basically need a way of forcing the BT dongle to move from a bonded state to a pairing state

Describe the bug
Sending the documented \t sends a hash t to the connected pc

To Reproduce
relaykeys-cli.exe type:\t

Expected behavior
Should send tab

Additional context
Seems like another keyboard mapping problem

So - if I send LSHIFT,2 I get a @ - but actually it should be " - from a windows keyboard. What should we do about that?

https://docs.acecentre.org.uk/products/v/relaykeys/installation#wireless-mode

Step 4. is not easy for end users who dont want to get messy with Arduino. Is there any way we can do this on the fly? Maybe pressing and holding a USER_SW ?

Plug in a adafruit nrf52840 express to a i12. Go to the Grid - notice how the GEWA module can no longer be found. Take it out - it comes back... No idea.

All the vuepress stuff adds too much dependency crud.

On Windows the relaykeys daemon service cannot be stopped or restarted.

To Reproduce
Run services.msc as administrator. Select the running relaykeys daemon service. Click on either "restart" or "stop" and the dialog hangs for a long period and then fails, reporting error 1053. If the daemon is running it continues to function sending keys from the cli to the remote computer and does not terminate. A crashed daemon cannot be restarted this way (you need to terminate the process from task manager first).

Also, running relaykeysd-service-restart.bat as administrator does fails after a long pause.

Expected behavior
Clicking stop should end the service and stop relaying keys until start is clicked. Clicking restart should restart a failed daemon.

System info
Windows 10.
Bluefruit LE Friend.

Would like to come up with a way of a software program calling RelayKeys like an exe

e.g..

relayKeys.exe -pressKeys s,shift,ctrl
relayKeys.exe -mouseMove ..

I'm not to hot on how this should be done - so its called quickly - and the app doesnt just close. Is there a way the app can run in the background and these commands just be sent to the running thread to the serial port?

• If you use the relaykeys-qt app and go to add a device it times out too quickly - so you get an error even if you do successfully add one you get an error message
• Pairing with an iPhone works but its very laggy. It didn't use to be. Not sure why that is.
• Mouse - Not sure I can get this to work brilliantly. Maybe we need to capture the mouse within the qt app window rather than whole screen? Might help

So I can see we can emulate button press etc.. but can we do something like a drag function?

Symbols like Square root, Infinity, Degrees all need to be a type: command.
Also greek symbols (alpha, beta, gamma) etc.

When sending large text strings using the CLI paste command occasionally one of the characters is repeated 15-20 times.

Tested using the #76 pre-compiled installer with a Bluefruit BLE Friend connected to an Android phone. Also see non an old Window tablet. I copied approx 3kB text into the clipboard and sent it using the CLI paste command. All the text arrived on the target and the only problem was some additional repeating characters. Over the 3kB block this would occur around 2-3 times.

Example:
The source text might contain this "Lorem ipsum dolor sit amet, consectetur adipiscing elit." as part of a longer string. The target would end up looking like this: "Lorem ipsum dolor sit amet, consectetur adddddddddddddddddddddddddddddipiscing elit,"

Presumably this is caused by a delay receiving a key up command and autorepeat causing the repeats until the key up command finally arrives. It appears the key does arrive as the offending character does stop repeating. Perhaps this is a buffering issue?

Right now pyGame is not capturing keys like the Mac / Windows key. E.g. Win+Tab

I'm wondering if pyGame can do this or we need to move to pyHook e.g. http://code.activestate.com/recipes/553270-using-pyhook-to-block-windows-keys/

nonchars_key_map seems to be a US mapping. It needs to be localisable for different languages.
e.g In UK localisations:

• Can't type or paste '£' or '' in UK language environments.
• type:£ produces nothing. £ should be mapped to ("3", ["LSHIFT"]) but # is mapped to this (command type:# produces £ instead)
• type:# produces £. It should be mapped to NONUSHASH not ("3", ["LSHIFT"])
• type:\ produces nothing. It should be mapped to NON-US-BACKSLASH (0x64) but it's mapped to "BACKSLASH" which is the US keycode.
• NON-US-BACKSLASH (0x64) is not defined in blehid.py keymap

Not sure of the best way to do this. Either have multiple nonchars_key_maps and switch between them depending on a language setting, or read keymap from a file. The latter would allow translators to contribute language map files and the one to be used defined in the config file.

This is needed for iOS etc control. See AT+BLEHIDCONTROLKEY at https://learn.adafruit.com/introducing-adafruit-ble-bluetooth-low-energy-friend/ble-services and consumerKey here https://learn.adafruit.com/introducing-the-adafruit-nrf52840-feather/blehidadafruit

New install of latest pre-compiled windows installer fails to run.

Machine: Clean i13 with no previous installation of ReplayKeys. Windows 10 21H1.

Steps taken:
Downloaded Windows installer "RelayKeys.setup.exe". OK.
Ran installer successfully. OK.
Ran "RelayKeys" from start button, crashed immediately with the following report:
"Failed to execute script 'relaykeys-qt' due to unhandled exception: No Module named 'PyWinhook'
Traceback: file "relaykeys-qt.py", line 15, in ModuleNotFoundError.

In Services control panel the ReplayKeys service was present but not running. On clicking start it refused to start.

In README.md instructions for configuring Bluefruit LE Friend, after setting baud rate the final line in that section should read:

"Leave the device in CMD mode."

RelayKeys is sending it AT commands across the USB serial port so it still needs to be in command mode. I have tested this, it fails in DATA mode and works in CMD mode.

Is your feature request related to a problem? Please describe.
Concerned that now all keys are being picked up reliably

Describe the solution you'd like
Not sure if this helps - but maybe there is something about unicode v's scancode keys: https://docs.microsoft.com/en-us/windows-server/remote/remote-desktop-services/clients/mac-whatsnew#updates-for-version-1038

??

Describe the bug
When sending text with the paste or type commands text strings over about 20 to 40 characters can cause a failure. Sometimes it works for a few attempts but eventually locks up. If sending via relaykeys-cli.exe you can see the command window reports a number of successful key events and then starts timing out. Once it has started timing out it never recovers until the daemon service is restarted and the remote computer connection re-established.

I tried changing baudrate (device, cfg file & port settings together) e.g. 9600 but not convinced this helps.

workaround
Calling the cli with a small delay using the --delay argument seems to fix it. In my case 2mS improved things slightly but still not reliable. 5mS seems more reliable.

System Info:
Windows 10
Bluefruit LE Friend @115200 baud

Looks like some keys on this keyboard arent being detected..

See videos in here: https://www.dropbox.com/sh/6xf35n3qvjvbdh0/AAB0P8-FvUOGdZi6vhKjZ0m1a?dl=0

I think there are some missing keycodes &/or some language of keyboard differences.. Not sure

Support the GIDEI protocol

https://aacinstitute.org/aac-keys/

Not sure if, how this would even be useful

Using a Bluefruit BLE Friend. Pairing established and RelayKeys can type on target device OK. After a break in connection (e.g. separation or sleep) it does not successfully re-establish a working connection. The pairing must be deleted and re-paired to get it working again.

It appears a bluetooth connection is re-established but no keyboard input occurs. The server service & CLI report success sending characters but nothing arrives on the target. At this point AT+GAPGETCONN returns 1 indicating a GAP connection has been re-established, so some BLE layers auto-reconnect but the HID layer does not.

Have tried two Android target devices and a Linux laptop. On the Linux laptop BT GUI it shows the BLE Friend cycling through connect/disconnect. The Friend's blue LED flashes on/off with each cycle.

MINOR: These appear to be defined as CUSTOM~ and BACKSLASH but these non-standard names could cause confusion for people familiar with NON-US & NON_US_NUM, and for people doing other localisations. (Actually it's all confusing to the uninitiated anyway!) Even though I'm familiar with key names I did find BACKSLASH helpful, so could add NON-US & NON_US_NUM as aliases?

I think we need this at least. It needs a way of choosing different keyboard layouts/and OS systems for the host machine

Warning: Need to test this but think this is the case

An important use case is being able to latch down one or more modifiers before sending a character and have that character sent with the current set of modifiers. The modifier state should remain unchanged so that further characters can be sent before clearing or changing the modifers. However, keyevent and keypress both ignore existing modifiers when sending characters and reset them.

keyevent:None,LSHIFT,1
keyevent:A,None,1
keyevent:A,None,0
keyevent:None,LSHIFT,0

or
keyevent:None,LSHIFT,1
keypress, A
keyevent:None,LSHIFT,0

Expected behaviour for both sequences is they should write "A" on target, but observed behaviour is they write "a" on target.

Observed behaviour in more detail using an Android phone as target:
keyevent:None,LSHIFT,1 --> osk switches to capitals (ok)
keyevent:A,None,1 --> "a" {autorepeats if following command delayed}, osk switches to lower case (expected "A" and osk to remain in capitals)
keyevent:A,None,0 --> {terminates autorepeat} (ok)
keyevent:None,LSHIFT,0 --> osk switches back to capitals!!! (expected to return to lower case)

There is presumably an intention that keypress with an explicit modifier sends it's character with that state, but (1) how does this fit with any pre-set modifiers? And (2) it should as a minimum restore whatever the previous modifier state was.

The Adafruit Itsybitsy nrf52840 Express has an uncommitted switch onboard wired to D4. Optionally change file arduino_nRF52840.ino
to #define USER_SW 4 and then you don't have to wire an additional switch.

A useful configuration command for the BLE Friend. This should be inserted before the line "Turn on HID" so the ATZ command comes afterwards:

[Optional] Name your Friend to help you identify it when you are pairing the bluetooth connection.
AT+GAPDEVNAME=RelayKeys (or whatever name you want)

CRITICAL: On a communication aid using Windows the end user cannot restart the service because that command needs to be elevated. Even if the logged in user has admin rights if a restart batch file is called from a communication aid grid cell it is not elevated. Several of the methods that could elevate the call trigger the UAC admin login dialog which is not accessible to people with complex access needs.

To select next word typically you would do a ctrl+shift+right arrow. But how would someone do this with relaykeys cli ?

Configurable. Basically as user types a letter - or is in a shifted state window shows letter/whether modifier keys held down.

e.g. User presses 's'

| s

e.g. User presses Shift+s

| (shift image) + s

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe.
We have been the keystroke capture gui but now need a mouse capture GUI

Describe the solution you'd like
A Window that is resizable. Captures mouse movements within the window and left right clicks, drag etc. A bit like a virtual glide pad.

Ref #3

To Save a lot of lines - I think we need a way of the CLI dealing with a string of keyevent: - just not sure best way to do this.

Is your feature request related to a problem? Please describe.
A client who is trying relayKeys likes it - but is getting tired switching their view from one machine to another.

Describe the solution you'd like
He has asked whether we could display the last n characters sent.. I imagine we could build this into the qt version - but really its for the RPC protocol. So something like type:hello --viewlast 30

When then sending a command the rpc server would respond with OK - but also the string of chars sent.

Describe alternatives you've considered
Cant think of anything

so because the qt version cant capture windows key - im wondering about a button somewhere"Windows key" (LGUI) that would be like a toggle button - accessible by the keyboard.

Is your feature request related to a problem? Please describe.
VNC (Virtual Network Computing) is a lightweight protocol designed for implementing remote Keyboard Video Mouse systems. It is an ideal protocol for this use case, as there is an extensive ecosystem of clients and servers that already support this protocol. VNC supports transfer of video data, but does not require any responses to queries for it, making it actually quite simple to just ignore that part of the spec.

Describe the solution you'd like
Consider replacing the RPC server and firmware interface with an implementation of the VNC protocol. Substituting the line by line interface in the Arduino firmware with an implementation of the VNC protocol should be fairly straightforward. A sample implementation of VNC for the ESP8266 already exists as part of the ESP-VNC project (https://github.com/sensepost/esp-vnc/tree/7e770585c405288ff93fd42684a246627b5761e0/vnc), which could fairly easily be repurposed. While the current implementation expects a new TCP connection to initiate the protocol and reset the state machine, this implementation could begin when the USB Serial interface is opened.

Once that is done, the RPC Server implementation would collapse to "Listen on a TCP port, when a connection is made, open and lock the USB-Serial port, and simply copy bytes backwards and forwards", providing an adapter from a network port (expected by common VNC clients) to the USB-Serial port.

A user would then be able to make use of any suitable VNC client to emit keystrokes and mouse movements via the bluetooth interface.

Additionally, given that you are already familiar with Python, there is a command line VNC client implemented in Python (actually a simple frontend to a Python library) called VNCdo (https://github.com/sibson/vncdotool), which could provide an adapter between any specific functionality required by an individual, and the VNC server implementation. For example, this could be used to provide "macro" functions, where a single operation by the user results in a series of scripted keystrokes and mouse movements on the target.

VNC even includes support for copy and paste of client buffers to the server, which I have implemented as "Type out this block of text using the keyboard". Obviously, this is inappropriate for pasting of binary data, but can be a nice feature otherwise.

I would consider implementing this perhaps as a subfunction of the existing firmware, triggered by e.g. a custom AT command. "AT+VNC\n" "OK VNC" . This would still allow for things like AT commands to configure BlueTooth pairing, or an AT command to update a built-in keyboard map. And it would mean that any existing implementations using RPC-Service would continue to work unaltered.

Describe alternatives you've considered
I have implemented a similar sort of device using a "WiFi to USB keyboard" hardware, so I have some experience in this area.

All the other solutions I considered seemed to be more complex than simply using VNC. If using a custom protocol, one has to consider creating a client for it and dealing with things like special keys (e.g. Alt-Tab) which are interpreted locally to switch applications, when the desire was for that special key to be transmitted to the target.

An alternative to VNC might be Remote Desktop Protocol (RDP). I did not investigate that further as VNC was very simple to implement.

Additional context
Keyboard mapping (#20) could potentially be implemented by updating the mapKey() function with a different table.

Drag and drop (#21) is also then a solved problem, as it simply devolves to sending a mouse button press via VNC (with no immediate release event), moving the mouse, and eventually sending a button release event.

A way of toggling the sending of text on or off - and buffering (play/pause) the text in a window.

e.g. Something like

!+Ctrl+F8 (Toggle).
!+Ctrl+F7 (Off)
!+Ctrl+F9 (On)

Buffer/Pause:

Basically allows the person to cue the text up in a editing window - and when pressing "Play" sends the text in the window.

Mousebutton command implements click and doubleclick behaviours. Also need separate button press and button release (or down and up) behaviours so that drag and swipe operations can be performed like this:
mousebutton:L,press
mousemove:0,10
mousebutton:L,release

See also #21 how would drag work?

So on my client computer I can go to https://keycode.info - and press the apple/windows key - and I can see its detected.

On the machine running relaykeys I do a

relaykeys-cli.exe keypress:LMETA

And although I see success on the command line - I get no input on the client

relakeys window gets longer and longer.. when typing..

There is a repeated line of ??(0xa0) + ??(0xa0) that is shown and the window gets so long you cant find the close window

so need a command on the cli to type whatever is in the pasteboard..

e.g.. relaykeys-cli.exe --paste

(hint: https://github.com/AceCentre/CopyBuffer/blob/master/CopyBuffer.py#L12)

https://docs.acecentre.org.uk/products/v/relaykeys/installation#wireless-mode

Step 4. is not easy for end users who dont want to get messy with Arduino. Is there any way we can do this on the fly? Maybe pressing and holding a USER_SW ?