Add "touchpad mode" with pinch-zoom for mobile view. #1835

bitbound · 2024-02-04T20:01:11Z

This PR adds a "touchpad mode" option for input when in mobile view. It includes pinch-zoom.

In touchpad mode, it will function similar to other remote control apps (RealVNC, TeamViewer, etc.). Touch movements will "push" the cursor in the same direction, and tap events will occur at the cursor's current location.

Long-press or two-tap will perform a right-click. Double-tap-and-drag will perform a click-and-drag with the left mouse button.

TouchpadMode.mp4

- Add touchpad.svg to images. - Add #noVNC_touchpad_button button to #noVNC_mobile_buttons div. - Add/modify css rules so margin applies to mobile buttons, not the surrounding div.

CendioOssman · 2024-02-05T10:29:31Z

This looks promising, thanks!

I've not looked at things in detail yet, but there are two things I noticed so far:

An easier zoom for mobile devices is something that is generally needed, so I don't think it should be conditional on being in touchpad mode.
Do we really need to be able to quickly toggle touchpad mode? I figured it was something that could be in the settings dialog, rather than taking up toolbar space.

bitbound · 2024-02-05T14:42:11Z

This looks promising, thanks!

Happy to contribute! This is an awesome project. Thanks for all your work on it.

An easier zoom for mobile devices is something that is generally needed, so I don't think it should be conditional on being in touchpad mode.

Should I create a separate option for this in settings? If so, any preference on what to call it? "Free Scale" is the first term that comes to my mind.

Do we really need to be able to quickly toggle touchpad mode?

That's totally up to you (and other maintainers). Since drag mode, default mode, and touchpad mode are mutually exclusive, my first thought was to put it here to easily see the relationship.

samhed · 2024-02-09T13:10:17Z

Nice!

I've found myself using this feature in both RealVNC and TeamViewer a lot, I love it.

Personally, I switch a lot between the modes. I would find it annoying to have it hidden in the settings.

LMattsson · 2024-02-09T13:12:40Z

If we don't want to populate the side bar too much, I see a few alternatives:

Can this replace view-drag-mode (hand icon)?
Could view-drag-mode and touchpad mode be combined into one button that a small panel, like the extra keyboard button?
- Alternatively, have a single button with 3 states.

samhed

Thank you for your patience.

We have now tested your changes and I must say the overall impression and feeling is very nice!

We have used a phone for testing, and the mapping between finger and mouse movements feels well calibrated. 🥇

We think this is a very impressive feature that will give our users a better
experience, especially if using small screens. 📱

Synaptics is, in our minds, the most well-known developer of trackpads. For reference, here's a list of “gestures” for the Synaptics driver on Linux:

https://man.archlinux.org/man/synaptics.4

However, we don't think we must support all the features they support. The most important thing is that your trackpad implementation doesn't conflict with theirs, since that's what most users expect from a trackpad.

Aside from testing your implementation, we also compared it with Microsoft's RDP Android app, and RealVNC's Android app:

In general, we think your implementation matches great with the “mouse mode” in Microsoft's RDP app. They both use the same gestures and actions. The feeling is very similar, that is an excellent thing.

There are two small differences when compared to Microsoft's RDP; Firstly, they have some inertia that makes the cursor stop softly after lifting the finger (it continues moving a little bit, slower and slower until coming to a complete stop). We're not sure whether we like this effect. Secondly, their center area is larger, the area where the cursor can move freely without moving the viewport.

Similar to noVNC, Microsoft's RDP also has a “touch mode”, which behaves like our standard mode. The gestures they use in the equivalent mode are listed here:

https://learn.microsoft.com/en-us/azure/virtual-desktop/users/client-features-android-chrome-os#use-touch-gestures-and-mouse-modes-in-a-remote-session
RealVNC only has this trackpad mode, no mode where touch positions are directly mapped to mouse movements. Your implementation matches their trackpad mode well in terms of available gestures and how they function. One major difference is that the cursor is always centered in RealVNC's variant. We like your implementation better.

RealVNC has the option of showing a “mouse button overlay”, see below screenshot.

While this might be useful in some cases, we don't think it is something we need
to include in the first version.

After testing the other apps, I noticed that Microsoft refers to their “standard mode” as “touch mode”, and their trackpad mode as “mouse mode”. I'm a bit worried that your “touchpad mode” could easily be confused with “touch mode”, which would be incorrect. What are your thoughts on renaming your new mode to “trackpad mode”?

Lastly, have you considered unit tests for your code?

samhed · 2024-02-29T15:47:51Z

app/styles/base.css

@@ -553,6 +556,12 @@ html {
 :root:not(.noVNC_connected) #noVNC_mobile_buttons {
  display: none;
 }
+
+#noVNC_mobile_buttons > .noVNC_button {


If you make this selector more specific, you can avoid the *:not(#noVNC_mobile_buttons) above.

I think #noVNC_control_bar > .noVNC_scroll > #noVNC_mobile_buttons > .noVNC_button { should work.

samhed · 2024-02-29T15:53:01Z

app/ui.js

+     * @param { "normal" | "info" | "warn" | "warning" | "error" } statusType
+     * @param {number} time
+     * @returns
+     */


This seems a bit out of place, we don't have these types of docstrings anywhere else.

samhed · 2024-02-29T15:54:39Z

app/ui.js

@@ -1061,8 +1070,10 @@ const UI = {
        UI.rfb.qualityLevel = parseInt(UI.getSetting('quality'));
        UI.rfb.compressionLevel = parseInt(UI.getSetting('compression'));
        UI.rfb.showDotCursor = UI.getSetting('show_dot');
+        UI.rfb.touchpadMode = WebUtil.readSetting('touchpad_mode', 'false') === 'true';


Doesn't UI.getSetting() work for you?

Should we default to touchpad_mode on very small screens?

samhed · 2024-02-29T15:56:38Z

app/ui.js

    },

    updateViewDrag() {
        if (!UI.connected) return;

        const viewDragButton = document.getElementById('noVNC_view_drag_button');

+        if (UI.rfb.dragViewport) {
+            UI.rfb.touchpadMode = false;


Good that you remembered adding this!

samhed · 2024-02-29T15:57:33Z

app/ui.js

+        if (!UI.rfb) return;
+
+        UI.rfb.touchpadMode = !UI.rfb.touchpadMode;
+        WebUtil.writeSetting('touchpad_mode', UI.rfb.touchpadMode);


UI.saveSetting()?

samhed · 2024-03-01T11:45:42Z

core/rfb.js

+                                this._sendMouse(cursorPos.x, cursorPos.y, 0x1);
+                            }
+                            break;
+                        }


I'm not sure that I understand this piece of code? Could you try to explain it?

samhed · 2024-03-01T11:48:46Z

core/rfb.js

@@ -1301,13 +1376,21 @@ export default class RFB extends EventTargetMixin {
                        break;
                    case 'drag':
                    case 'longpress':
-                        this._fakeMouseMove(ev, pos.x, pos.y);
+                        // In TouchpadMode, we want to move the cursor from its
+                        // current position, not to where the touch currently is.


These comments are helpful!

samhed · 2024-03-01T11:55:59Z

core/util/browser.js

+        // required for MS Surface
+        (navigator.maxTouchPoints > 0) ||
+        (navigator.msMaxTouchPoints > 0);
+}


So you've seen issues where this didn't update properly? Anyway, this should be a separate commit.

samhed · 2024-03-01T13:14:10Z

app/images/touchpad.svg

Could we make this icon a bit smaller? It seems larger than the other toolbar icons:

samhed · 2024-03-01T13:15:27Z

vnc.html

@@ -79,6 +79,9 @@ <h1 class="noVNC_logo" translate="no"><span>no</span><br>VNC</h1>
            <div id="noVNC_mobile_buttons">
                <input type="image" alt="Keyboard" src="app/images/keyboard.svg"
                    id="noVNC_keyboard_button" class="noVNC_button" title="Show Keyboard">
+
+                <input type="image" alt="Touchpad Mode" src="app/images/touchpad.svg"
+                    id="noVNC_touchpad_button" class="noVNC_button" title="Touchpad Mode">


Would you mind placing it above the keyboard button? Currently, it is in between the "Show Keyboard" button and the extra keys button:

samhed · 2024-03-01T15:41:45Z

We also have some future ideas for changes to the view drag mode (the “hand”). With trackpad mode in place, it would not be as critical anymore.

samhed · 2024-04-26T08:42:17Z

@bitbound are you interested in continuing working on this?

bitbound · 2024-04-28T14:35:25Z

@bitbound are you interested in continuing working on this?

I'm sorry. I forgot to update this PR.

I took my project in another direction (WebRTC) and won't be using noVNC anymore. I won't have time to finish this.

Of course, you or anyone else is free to take over and finish this if they want.

Cheers!

bitbound added 5 commits October 26, 2023 18:21

Add touchpad mobile button to the control bar.

c9692e5

- Add touchpad.svg to images. - Add #noVNC_touchpad_button button to #noVNC_mobile_buttons div. - Add/modify css rules so margin applies to mobile buttons, not the surrounding div.

Make touchpad button togglable.

810d294

Implement TouchpadMode gesture handling.

6175985

Adjust cursor coordinates by hotspot.

2c11e27

Fix linting errors.

183a7bc

LMattsson added feature ui/full labels Feb 9, 2024

samhed requested changes Mar 1, 2024

View reviewed changes

samhed linked an issue Apr 26, 2024 that may be closed by this pull request

Add simulated touchpad option for touchscreen devices #1764

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add "touchpad mode" with pinch-zoom for mobile view. #1835

Add "touchpad mode" with pinch-zoom for mobile view. #1835

bitbound commented Feb 4, 2024

CendioOssman commented Feb 5, 2024

bitbound commented Feb 5, 2024

samhed commented Feb 9, 2024

LMattsson commented Feb 9, 2024

samhed left a comment

samhed Feb 29, 2024

samhed Feb 29, 2024

samhed Feb 29, 2024

samhed Feb 29, 2024

samhed Feb 29, 2024

samhed Feb 29, 2024

samhed Mar 1, 2024

samhed Mar 1, 2024

samhed Mar 1, 2024

samhed Mar 1, 2024

samhed Mar 1, 2024

samhed commented Mar 1, 2024

samhed commented Apr 26, 2024 •

edited

bitbound commented Apr 28, 2024

Add "touchpad mode" with pinch-zoom for mobile view. #1835

Are you sure you want to change the base?

Add "touchpad mode" with pinch-zoom for mobile view. #1835

Conversation

bitbound commented Feb 4, 2024

CendioOssman commented Feb 5, 2024

bitbound commented Feb 5, 2024

samhed commented Feb 9, 2024

LMattsson commented Feb 9, 2024

samhed left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samhed commented Mar 1, 2024

samhed commented Apr 26, 2024 • edited

bitbound commented Apr 28, 2024

samhed commented Apr 26, 2024 •

edited