VNC Support about android-keyboard-gadget HOT 17 CLOSED

I have thought about it, remote functionality is even mentioned in the
plans in readme, but alas, I've got too little time to implement that. I
don't think VNC protocol will do any acceptable compression on camera feed,
I was going to make a separate client/server with ffmpeg or something. VNC
seems easier though, because I don't need to implement client. The only
issue is that client mouse cursor will be disconnected from the remote
cursor, because there's no way to determine it's position without doing
image recognition on camera feed.

On Thu, Jan 15, 2015 at 3:27 PM, lars18th [email protected] wrote:

Hi,

I feel that this project will be more interesting if we can use the
Android device as a PROXY for a physical hardware (like KVMIP server).

Here my proposal: Add VNC server tool.

To achieve this fast, in the project of UML (User Mode Linux), you can
found a very simple VNC server for a virtual shared memory framebuffer. In
this project the server isn't running with real vga, keyboard, mouse.

The source code is on this page:
http://intgat.tigress.co.uk/rmy/shmfb/index.html
If you download the source (uml_fb-0.1.1.tar.gz), inside you have the
"uml_vncfb.c" tool. This tool uses the well known LibVNCServer, and it's
very lightweight. In the source you have isolated the function that sends
the events of VGA, KEYBOARD and MOUSE. See this on the source:

• shmfb_command(SHMFB_KEY_EVENT, keymap[i].keycode, down, 0, 0);
• hmfb_command(SHMFB_BUTTON_EVENT, vnc_mousemap[i],((now >> i) & 1),
  x, y);
• shmfb_command(SHMFB_UPDATE_EVENT, 0, 0, 0, 0)

So, if you replace these three calls with the code of hid-gadget-test
utility you can implement a simple VNC server for this.

At time, the framebuffer can be a black window, but if you implement the
camera capture, then you can implement the full remote functionality.

Someone likes to implement this?

—
Reply to this email directly or view it on GitHub
#20.

from android-keyboard-gadget.

Hi,

I feel that VNC is the best option for several reasons:

1. It's a common protocol and a lot of clients exists.

2. This protocol focuses on KEYBOARD, MOUSE and VIDEO. The three targets of this project.

3. The VNC can support different compression algorithms, and some versions include MPEG video (also based on ffmpeg). Don't worry about this. Standard VNC video codecs focus on SCREEN, not real video, but the users like to point the camera to the screen!

4. Modern KVM-IP devices supports VNC or are native based on VNC. The aim of this project is replicate this functionality.

5. Using the source code pointed it's very easy to implement a testing-vnc-server that sends black video, but supports KEYBOARD and MOUSE. At time, this project is useless for remote use. With this simple "example" it's possible to use any VNC client to simulate a USB keyboard and mouse connected to any device. In this case only de video is missing. And this will be a huge improvement over the current status of the project, at the cost of some few code lines.

Why not try it?

from android-keyboard-gadget.

Okay, okay, but I'll likely use tightvncserver sources, because I need
PointerPos extension, also compression provided by it would be nice.

On Fri, Jan 16, 2015 at 10:11 AM, lars18th [email protected] wrote:

Hi,

I feel that VNC is the best option for several reasons:

1. It's a common protocol and a lot of clients exists.

2. This protocol focuses on KEYBOARD, MOUSE and VIDEO. The three targets
   of this project.

3. The VNC can support different compression algorithms, and some versions
   include MPEG video (also based on ffmpeg). Don't worry about this. Standard
   VNC video codecs focus on SCREEN, not real video, but the users like to
   point the camera to the screen!

4. Modern KVM-IP devices supports VNC or are native based on VNC. The aim
   of this project is replicate this functionality.

5. Using the source code pointed it's very easy to implement a
   testing-vnc-server that sends black video, but supports KEYBOARD and MOUSE.
   At time, this project is useless for remote use. With this simple "example"
   it's possible to use any VNC client to simulate a USB keyboard and mouse
   connected to any device. In this case only de video is missing. And this
   will be a huge improvement over the current status of the project, at the
   cost of some few code lines.

Why not try it?

—
Reply to this email directly or view it on GitHub
#20 (comment)
.

from android-keyboard-gadget.

Hi,

I hope you can prepare a simple test demo of VNCServer. The reason is this other project:
http://danman.eu/blog/reverse-engineering-lenkeng-hdmi-over-ip-extender/

This is my idea:

. COMPUTER ---HDMI------ (Video+Audio) --------> HDMI-2-IP ----> Android device ---> VNC
. . . . |-----USB--------------------------- (Keyboard+Mouse) --------------/

The HDMI-2-IP is really cheap (less than $/€100) and can send the HDMI compressed to the network in mulcast. Then any network device can capture the video (plus audio!) at ~15fps.

So, my suggestion is that your VNC server will be MODULAR to inject any video input using pipes (FIFO files). Perhaps you can read directly JPEG, or use VLC (or any other processor) to input raw data in you memory framebuffer.

If you feel that one Android device don't have sufficient power to achieve all these tasks, then my suggestion is to support BLACK video in the VNC Server. Then you can use a VNC PROXY to inject the video, as this example:

Keyboard+Mouse USB ---> Android VNC Server ---> VNC Proxy ---> Client

And the VNC Proxy integrates the JPEG-IP video stream inside the VNC connection.
The proxy can be based on: http://github.com/pmdumuid/rfbproxy

You like this (crazy) idea?

from android-keyboard-gadget.

Nah, $70 is not "really cheap", it is "not expensive". Raspberry PI type A
at $25 is really cheap. PI type B and Chromecast at $35 are just cheap.

Also setting up two devices instead of just one is way too complicated for
typical mobile users, I want to keep it simple, so I'll use camera.

Don't expect test demo anytime soon. Maybe in a month or two, also I'm no
fan of test demos, I intend to make it usable out of the box.

On Tue, Jan 20, 2015 at 1:51 PM, lars18th [email protected] wrote:

Hi,

I hope you can prepare a simple test demo of VNCServer. The reason is this
other project:
http://danman.eu/blog/reverse-engineering-lenkeng-hdmi-over-ip-extender/

This is my idea:

COMPUTER ---HDMI------ (Video+Audio) --------> HDMI-2-IP ----> Android
device ---> VNC
|-----USB--------------------------- (Keyboard+Mouse) --------------/

The HDMI-2-IP is really cheap (less than $/€100) and can send the HDMI
compressed to the network in mulcast. Then any network device can capture
the video (plus audio!) at ~15fps.

So, my suggestion is that your VNC server will be MODULAR to inject any
video input using pipes (FIFO files). Perhaps you can read directly JPEG,
or use VLC (or any other processor) to input raw data in you memory
framebuffer.

If you feel that one Android device don't have sufficient power to achieve
all this tasks, my suggestion is to support BLACK video in the VNC Server.
Then you can use a VNC PROXY to inject the video, as this example:

Keyboard+Mouse USB ---> Android VNC Server ---> VNC Proxy ---> Client

And the VNC Proxy integrates the JPEG-IP video stream inside the VNC
connection.

You like this (crazy) idea?

—
Reply to this email directly or view it on GitHub
#20 (comment)
.

from android-keyboard-gadget.

Yes, you're right!

However, a very simple vnc server (with black screen, it's say no-video) implemented using a simple library (perhaps based on the code I was commented) can be very useful. Please, think you on a remote software, running on another device, that likes to send Keyboard and/or Mouse commands over the USB. With this simple vnc server it's posible to use any Android device as a TUNNEL for send these commands. You don't need to create any new protocol to send this info. And the API it's the simple RFB protocol.

I hope you think on create a "test" version based on the simple "hid-gadget-test.c", and send command using shmfb_command() function present in "uml_vncfb.c".

But you can be confident, I will have patience. ;-)

from android-keyboard-gadget.

Stop sending me emails
On Jan 21, 2015 1:35 AM, "lars18th" [email protected] wrote:

Yes, you're right!

However, a very simple vnc server (with black screen, it's say no-video)
implemented using a simple library (perhaps based on the code I was
commented) can be very useful. Please, think you on a remote software,
running on another device, that likes to send Keyboard and/or Mouse
commands over the USB. With this simple vnc server it's posible to use any
Android device as a TUNNEL for send these commands. You don't need to
create any new protocol to send this info. And the API it's the simple RFB
protocol.

I hope you think on create a "test" version based on the simple
"hid-gadget-test.c", and send command using shmfb_command() function
present in "uml_vncfb.c".

But you can be confident, I will have patience. ;-)

—
Reply to this email directly or view it on GitHub
#20 (comment)
.

from android-keyboard-gadget.

STOP sending me emails
On Jan 16, 2015 9:46 AM, "Sergii Pylypenko" [email protected]
wrote:

Okay, okay, but I'll likely use tightvncserver sources, because I need
PointerPos extension, also compression provided by it would be nice.

On Fri, Jan 16, 2015 at 10:11 AM, lars18th [email protected]
wrote:

Hi,

I feel that VNC is the best option for several reasons:

1. It's a common protocol and a lot of clients exists.

2. This protocol focuses on KEYBOARD, MOUSE and VIDEO. The three targets
   of this project.

3. The VNC can support different compression algorithms, and some
   versions
   include MPEG video (also based on ffmpeg). Don't worry about this.
   Standard
   VNC video codecs focus on SCREEN, not real video, but the users like to
   point the camera to the screen!

4. Modern KVM-IP devices supports VNC or are native based on VNC. The
   aim
   of this project is replicate this functionality.

5. Using the source code pointed it's very easy to implement a
   testing-vnc-server that sends black video, but supports KEYBOARD and
   MOUSE.
   At time, this project is useless for remote use. With this simple
   "example"
   it's possible to use any VNC client to simulate a USB keyboard and mouse
   connected to any device. In this case only de video is missing. And this
   will be a huge improvement over the current status of the project, at
   the
   cost of some few code lines.

Why not try it?

—
Reply to this email directly or view it on GitHub
<
https://github.com/pelya/android-keyboard-gadget/issues/20#issuecomment-70220759>

.

—
Reply to this email directly or view it on GitHub
#20 (comment)
.

from android-keyboard-gadget.

Hi Pelya,

Related to the project of a VNC server... I try to execute this:

1. Install Xfvb and X11vnc in some Linux machine (can be your Android device or another computer).

2. Execute a virtual X11 session, for example with commands:

$ Xvfb :1 -screen 0 1024x768x24+32 &
$ DISPLAY=:1 blackbox & (if you like to execute some Window Manager)
$ DISPLAY=:1 x11vnc &

3. Then use any remote computer to connect to your VNC server.

4. Inside the X11 session execute command "xinput" for capture keyboard events. First list devices:

$ xinput --list
$ xinput test 'Virtual core XTEST keyboard'
$ xinput test 'Virtual core XTEST pointer'

Then if you capture the ouput of 'xinput test' then you can process it in the Android device. You can send it using Netcat and pipes; or you can execute xinput in Android and connect to the X11 using TCP/IP (I test to execute xinput remotely and it works). You only need the binary of xinput inside the Android device!

My idea is execute as a background of the X11 session the video capture of the remote screen. Then you don't need to implement any VCN server, only a new test-tool that can undesrtand the "xinput test" data.

What you think? Can you mix "xinput" and "hid-gadget-test" ?
Regards.

PD: Some examples of outputs running xinput on remote machine:

• Output of "xinput test 'Virtual core XTEST keyboard'":
  key press 111
  key release 111
  key press 36
  key release 36
• Output of "xinput test 'Virtual core XTEST pointer'":
  motion a[0]=641 a[1]=226
  motion a[0]=641 a[1]=225
  motion a[0]=646 a[1]=223

All working in realtime! No problem at all! Only need to pipe this output to your test-tool.
If you like to compile (static) the xinput tool, the source code is here:
http://cgit.freedesktop.org/xorg/app/xinput/tree/src
You only need a TCP socket to the X11 server to run the xinput in the Android device!

from android-keyboard-gadget.

Yes it's pissible, but kind of complicated.
I'll just cross-compile tightvncclient, I know how to do that and I've done
such things before, the only thing I need is two days of free time, which I
currently don't have.
On Jan 29, 2015 7:50 PM, "lars18th" [email protected] wrote:

Hi Pelya,

Related to the project of a VNC server... I try to execute this:

1. Install Xfvb and X11vnc in some Linux machine (can be your Android
   device or another computer).

2. Execute a virtual X11 session, for example with commands:

$ Xvfb :1 -screen 0 1024x768x24+32 &
$ DISPLAY=:1 blackbox & (if you like to execute some Window Manager)
$ DISPLAY=:1 x11vnc &

3. Then use any remote computer to connect to your VNC server.

4. Inside the X11 session execute command "xinput" for capture keyboard
   events. First list devices:

$ xinput --list
$ xinput test 'Virtual core XTEST keyboard'
$ xinput test 'Virtual core XTEST pointer'

Then if you capture the ouput of 'xinput test' then you can process it in
the Android device. You can send it using Netcat and pipes.

My idea is execute as a background of the X11 session the video capture of
the remote screen. Then you don't need to implement any VCN server, only a
new test-tool that can undesrtand the "xinput test" data.

What you think? Can you mix "xinput" and "hid-gadget-test" ?
Regards.

—
Reply to this email directly or view it on GitHub
#20 (comment)
.

from android-keyboard-gadget.

Hi Pelya,

I hope you found free time soon. Thank you for your comments!

Only one simple point for discuss: If you cross-compile the "xinput" tool, it will be a very good improvement. Perhaps you found quite complicated my proposal, but it's really simple. If we have "xinput" compiled for Android, we can do that in the Android shell:

$ DISPLAY:remote_ip:1 xinput test 'Virtual core XTEST keyboard' | hid-gadget-test /dev/hidg0 xkeyboard &
$ DISPLAY:remote_ip:1 xinput test 'Virtual core XTEST pointer' | hid-gadget-test /dev/hidg1 xmouse &

Then, all mouse/keyboard events in the remote X11 session will be processed by the "hid-gadget-test" tool. Please, note here the "xkeyboard" and "xmouse" parameters. These indicates to read standard input with xinput sintax.

Please, if you point to me about how to cross-compile the xinput tool then I'll try!
Regards.

PD: At time, I'll experiment with "Linux Deploy" and a Debian running in the Android device. With this I can try the semi-native xinput tool. Next step is modify "hid-gadget-test" to understand xinput sintax (at least for keyboard events, less complex to implement that for mouse).

from android-keyboard-gadget.

Hi Pelya,

Let me to explain my last test:

1. I do an install of Debian (ARMhf) using Linux Deploy. I access it using ssh. Install on it the xinput tool (sudo apt-get install xinput).

2. Execute in a remote PC the Xfvb and X11vnc.

3. From the Linux shell in the Android device I execute:

$ ssh -L 6002:127.0.0.1:6001 user@remote-pc

4. Then when you access to DISPLAY:2 in the Android device you're contacting with the X11 running with the Xfvb of the remote PC. I can access to this virtual X11 server using any VNC client.

5. In the Linux shell of Android device I can execute:

$ DISPLAY=:2 xinput test 'Virtual core XTEST keyboard'
$ DISPLAY=:2 xinput test 'Virtual core XTEST pointer'

And I see all keyboard and mouse events. So for now (and until we have an standalone version of xinput for Android) we can execute these commands from the Linux shell.
Please, can you write in the "Compilation" section a guide for compiling the "hid-gagdet-test"?

The next step is then do the pipe between the output of "xinput" and the input of "hid-gadget-test".
I hope you can found time to help me! Thank you for this very promissing tool.

PD: If you try to use the X11 VNC server running inside the Android device (Linux Deploy can execute one of this), you can't use the xinput tool. The reason is because the running VNC server is ThightVNC (Xvnc) and it don't includes the XInputExtension (you can check it with command "xdpyinfo"). So don't try to use this VNC server with xinput!

from android-keyboard-gadget.

Hi Pelya,

Related to my idea of implementing a VNC server in a different device, I like to explain my advances:

• If you like to remote send commands to /dev/hidgX, you can use this simple technique: Use SOCAT to create a PIPE in a remote machine that is connected to the character device. Then, in the remote machine you can execute ANY program that uses the device like it's local.

For example:

• In the shell of the Android device you can execute: "$ socat OPEN:/dev/hidg0 tcp-l:5000"
• In the shell of the remote (Linux) machine you can execute: "$ socat tcp:192.168.1.10:5000 PIPE:/tmp/hidg0"
• Then, you can execute the "hid-gadget-test" in the remote machine. As example: "$ echo f2 | hid-gadget-test /tmp/hidg0 keyboard" and this command sends a 'F2' keystroke.

Therefore, my idea is execute the SOCAT over a SSH connection to the Android device and "export" both devices: /dev/hidg0 & /dev/hidg1. Then it's possible to modify/compile/execute any tool in the remote machine. I'll try with the way of a virtual X11 session, capturing the keyboard and sending keypress to remote "/dev/hidg0". For the video, I like to execute root-window (background) video from VideoLAN or MPlayer. This video can be from a capture device or from the Android camera (several Apps for Video-2-IP exists).

If you need the native SOCAT binary for Android, you can found one here:
http://forum.xda-developers.com/showpost.php?p=51837955&postcount=28

Please, comment what you think. ;-)
Regards.

from android-keyboard-gadget.

I've fixed sources of USB Keyboard app, so you can compile it on generic
Linux - go to directory remote-client and type 'make', you'll need SDL 1.2,
SDL_ttf and SDL_image installed on your Linux.
It will still try to write to /dev/hidg0 and /dev/hidg1, it's up to you to
create appropriate pipes using mkfifo, and forward them to Android using
two socat commands.

On Sat, Feb 7, 2015 at 12:19 AM, lars18th [email protected] wrote:

Hi Pelya,

Related to my idea of implementing a VNC server in a different device, I
like to explain my advances:

• If you like to remote send commands to /dev/hidgX, you can use this
  simple technique: Use SOCAT to create a PIPE in a remote machine that is
  connected to the character device. Then, in the remote machine you can
  execute ANY program that uses the device like it's local.

For example:

In the shell of the Android device you can execute: "$ socat
OPEN:/dev/hidg0 tcp-l:5000"

In the shell of the remote (Linux) machine you can execute: "$ socat
tcp:192.168.1.10:5000 PIPE:/tmp/hidg0"

Then, you can execute the "hid-gadget-test" in the remote machine. As
example: "$ echo f2 | ./hid-gadget-test ./hidg0 keyboard" and this command
sends a keystroke.

Therefore, my idea is execute the SOCAT connection over a SSH connection
to the Android device and "export" both devices: /dev/hidg0 & /dev/hidg1.
Then it's possible to modify/compile/execute any tool in the remote
machine. I'll try with the way of a virtual X11 session, capturing the
keyboard and sending keypress to remote "/dev/hidg0". For the video, I like
to execute root-window (background) video from VideoLAN or MPlayer. This
video can be of a capture device or from the Android camera (several Apps
for Video-2-IP exists).

If you need the native SOCAT binary for Android, you can found one here:
http://forum.xda-developers.com/showpost.php?p=51837955&postcount=28

Please, comment what you think. ;-)
Regards.

—
Reply to this email directly or view it on GitHub
#20 (comment)
.

from android-keyboard-gadget.

Fixed in 1.10, camera size is hardcoded to 1280x720, maybe I'll do it cpnfigurable in the future.
You don't need all that xinput, socat, and mplayer stuff, there is just one button to turn it on.

from android-keyboard-gadget.

Could you advise where to find SDL_android.h ? Could not find this header among libsdl-dev packages.

remote-client # make
g++ -g -o usb-keyboard-client gfx.cpp vnc_server.cpp flash_kernel.cpp gui_settings.cpp vnc_keysyms.cpp input.cpp gui.cpp camera.cpp main.cpp touchpad.cpp scancodes.c -lSDL_ttf -lSDL_image `sdl-config --cflags` `sdl-config --libs` -I"/usr/lib/jvm/java-8-openjdk-amd64/include" -I"/usr/lib/jvm/java-8-openjdk-amd64/include/linux"
camera.cpp:2:25: fatal error: SDL_android.h: No such file or directory
compilation terminated.
Makefile:5: recipe for target 'usb-keyboard-client' failed
make: *** [usb-keyboard-client] Error 1

from android-keyboard-gadget.

from android-keyboard-gadget.