iPhone/Android support: touchscreen and viewport about novnc HOT 17 CLOSED

Any news about that ? iPad2's out; time to give it noVNC 2 as well :)

from novnc.

I'm currently in Nairobi, Kenya until the end of the week. Hopefully I'll get some time to work on this next week after I get back. Of course, there is also the opportunity for somebody else who is more experienced with mobile/touch screen web development (or simply highly motivated) to contribute ;-)

from novnc.

Any news on that ? I found https://github.com/jordansissel/fingerpoken which does all the 'mouse' handling, maybe somebody can mix fingerpoken and novnc ?

from novnc.

It's been a while in coming but I've finally had a crack at this.

The first commit (ad3f762) adds better mouse support. Prior to this commit, the only thing you could do on touch screen devices was left click. This change enables middle and right clicking support (using mouse button selectors in the upper right of the status bar). Also supported is dragging (using whichever button is selected). Also, if all mouse selectors are toggled off, then the mouse can be moved freely.

I tested this on an iPad 2 and it seems to work fairly well.

Virtual keyboard support will be needed to be able to do keyboard input. I'll probably look into the Mottie jQuery plugin for this: https://github.com/Mottie/Keyboard, http://mottie.github.com/Keyboard/

Also, for smaller screen devices (and for larger screen tablets connecting to higher resolution desktops), viewport support is needed. Also, this will help with performance for slower devices by only requesting and rendering the area of the screen the user is actually viewing. Polishing up and activating scaling support might be useful here too.

I actually found this page to be a better reference on touch events for both iOS and Android:
http://backtothecode.blogspot.com/2009/10/javascript-touch-and-gesture-events.html

from novnc.

I'll do keyboard input for mobile devices next week if thats cool? I got a good idea of how to do this using the mobile OS's native keyboard.

I had a look at your code and I can see how to easily hook it into includes/input.js.

from novnc.

Okay I have native keyboard support working on iOS and android devices. Need to cleanup before publishing. Give me another day and I'll upload it.

from novnc.

@snorkeyg. Looking forward to seeing it. Some of my research indicated that you can't programmatically activate the native keyboard without manually clicking in an input or textarea. My own though was to have a small input box but style it to look more like a button to activate the keyboard. What sort of mechanism did you settle on?

BTW, one thing I usually do when adding support for new functionality is to create a simple test page in the tests directory. For example, I've been working on the dynamic viewport support in tests/viewport.html and I added touch screen support to the tests/input.html file. That way I can test the support separate from the whole noVNC functionality.

from novnc.

Hi Kanaka,
That's exactly what I did. I have added a keyboard button to the mobile interface which sets focus to a hidden text box to bring up the keyboard. All key events are channeled through your key event handlers in input.js.

When the button is clicked to show the keyboard mobile browsers tend to zoom in on the text box. I have prevented this by disabling zooming before the text box focus and re-enabling afterwards. Also before the focus occurs I get the browsers current scroll position, and after the text box focus is done I immediately reset the browser scroll position.

All of this is down with a couple of helpers in the bottom of ui.js.

I have sent you a pull for my forks commits.

I'm working on a tablet friendly interface as well for the project I'm working on. Once done I'll commit this to my fork.

Cheers,
Chris

from novnc.

My current plan is just to disable zooming and scrolling all the time (you can see an example of that in tests/viewport.html). Web applications that act like native application usually don't allow zooming and handle the touch/pinch events directly themselves. I don't expect that I'll need to jquery to do that. For the most part I'm trying to avoid including jQuery wholesale so that it doesn't become a requirement for people that prefer other frameworks (mootools, scriptaculous, etc).

Hopefully I'll have some time to pull all the pieces together in the next couple of weeks. Thanks for your work on this.

from novnc.

@snorkeyg: sounds exciting; i was also trying to figure it out on iPad ... How is your work going on ? Can I help you test out ?

from novnc.

Hi Discofever,
As discussed in PM latest is in my fork. Main addition is keyboard support, and more mobile friendly GUI.

I'm currently working on proper pinch zooming support and scrolling.

For everyone else screenshots of what it looks like can be found here:
http://chrisgordon.com.au/?q=node/18

from novnc.

There a few minor things pending such as some better icons from @snorkeyg, but the mobile branch represents working mobile device support.

The mobile support includes a permanent control bar, mouse touch events (with a mouse button selection button), viewport clipping and dragging, ability to toggle the device keyboard, resizing of display elements for small screen devices, etc. Scaling support will eventually be implemented, but for now the viewport is clipped to the size of the physical display area and you pan/drag the viewport to view offscreen areas of the remote display.

It works quite well and is reasonable fast on an iPad. It's fairly slow and the visible area of the remote screen is very limited on my iPhone 3G. However, everything functions correctly on the iPhone, and if you had to use it to manage a remote server, it would do in a pinch.

I expect to merge the mobile branch to master within the week and to close this issue at that point.

from novnc.

Confirming it's working well, however having to use the mouse buttons is a little tricky and not that 'user friendly'.

Why not create some kind of 'virtual trackpad', a grey box with two buttons at the top (simulating the two buttons) and have this virtual trackpad be draggable on top of the VNC canvas.

Well, don't know how difficult that would be to 'code' (and i'm not the person for that), but maybe cool to have.

What do you think ?

from novnc.

@unityweb, I'm not sure I'm following your description exactly. You want a control element that has mouse button selectors that you can move around so that it is closer to what you are actually interacting with?

BTW, I'm sure there is probably some better model for mapping touch type devices to mouse/keyboard type remote systems. The current model is has fairly complete control and wasn't too difficult to implement (and should be pretty generic across touch devices).

One could certainly imagine a ways to make use of multi-touch for either viewport dragging/panning or for mouse button selection or both. However, the implementation would be significantly more complicated. For example, one model I've seen before is that a single tap is used for primary mouse button and a alternate mouse button was indicate by touching one finger to the display where the click should happen, tapping another finger (anywhere) and then removing the original finger. To do that with noVNC would require some sort of delayed mouse touch handling so that a mouse down and mouse up event are only sent after the app knows which mouse button to send.

The bigger issue in my mind is not mouse interaction, but keyboard interaction. That is less than optimal because often the keyboard hides the area of the display where the characters are being displayed. I don't have a good answer for that. If Apple had a mode that allowed the keyboard to be partially transparent that would be really nice. Implementing the keyboard directly is another option which would allow transparency, however, that means suddenly we have to worry about international keyboards in noVNC itself which I am also loathe to do.

I'm willing to explore options for more user friendly interfaces, however, I don't have much free time myself and my primary usage is from the desktop so ideas and initial designs and implementations will probably need to come from other people unless I suddenly find myself being paid to work on noVNC as my full time job.

from novnc.

@kanaka: sorry if i wasn't that clear, here's a quick illustration so you see what i meant.
http://imageshack.us/photo/my-images/64/novncidea.png/

I understand your your implementation but to me it seems a little bit overhead for the 'end user'. I agree with you that the keyboard is an issue, but not that big; from my experience and UI design, users tend to minimize the keyboard usage on mobile, think of typing a complete e-mail in a tiny iphone ? They'll prefer getting a 'real' keyboard and do the job. For me, this little improvement (with the virtual trackpad) would be a great intermediate solution (see how iSSH handles the X Vnc display over ssh tunnel, same simple but functionnal trackpad).

from novnc.

I've integrated and tested the updates from @snorkeyg and merged the mobile branch to master.

I consider this issue resolved.

@unityweb, can you open a new issue and add your suggestion to that? Thx

from novnc.

Hi guys,I have tested this on both android tabs and windows touch devices and cant get the keyboard to show - I dont suppose you have a simple working example I can view on my tablet to look at your setup, functionality, implementation and see how I can work it into our app?
Thanks a million
Jamie

from novnc.