Add automatic clipboard support

[juanjoDiaz]

This PR adds support for automatic copy-pasting.
So when you are focused on the canvas and paste text it's pasted in the server and when you copy something in the server it's automatically copied to your local keyboard.

I've seen #1301 and #1174. But this takes a very different approach since it adds support for copy-pasting to noVNC's core. And I think that it's quite cleaner.

I've tested it in Safari, Firefox, and Chrome.
Unfortunately, the Clipboard API is only supported by Chrome and the paste event is broken (see https://bugs.chromium.org/p/chromium/issues/detail?id=634426).
So it just does nothing in Safari and Firefox, while in Chrome the copying of data works but not the pasting.

So, for now, this is not super useful but it should become better as browser support improves.

WDYT?

[CendioOssman]

Looks nice and clean, so a good start. But if browser support is too poor then I don't think we want to merge it yet.

I'm not sure how paste is supposed to work even with that chrome bug fixed. We intercept Ctrl+V, so what will be firing the paste event?

[juanjoDiaz]

This works in 2 ways:

1. The Clipboard class captures the ClipboardEvent on the canvas of type paste and paste the text content to the server using RFB's clipboardPasteFrom method.
2. The cut/copy events from the server that are handled by the RFB's _handle_server_cut_text method now also trigger a ClipboardEvent on the canvas of type copy. The Clipboard class captures the event and updates the local clipboard.

It's true that support varies. However, all browsers seem to be working on it and it's a change that doesn't break anything so I don't really see a reason to hold back.

[samhed]

According to MDN this is only supported on Chrome and Opera. Not Firefox, IE, Edge or Safari. I think that this can result in a lot of confused users.

Where can I find information about all browsers working on this?

[samhed]

ping @juanjoDiaz

[juanjoDiaz]

Sorry @samhed, totally missed this.
At the moment I've moved to work on very different stuff so I'm not actively looking into this.

You are right. It seems that:

• Firefox currently only considers the cliboard support for extensions: https://bugzilla.mozilla.org/show_bug.cgi?id=1344410
• Edge seems to support it according to Support for Clipboard API in Edge ckeditor/ckeditor4#468 but I haven't tested it.
• There are references in different blogs and unofficial channels of Safari working on this but I couldn't find anything official.

One thing to keep in mind is that this PR was mean to be just the foundation of a possible clipboard functionality and it just uses the Clipboard Async API if available.
It could be extended to fallback to the Sync API if available or to use browser-specific workarounds (like https://stackoverflow.com/questions/40147676/javascript-copy-to-clipboard-on-safari) if possible.
We could look into a library like https://github.com/zenorocha/clipboard.js (just to see the approaches that they use, since you guys don't like external dependencies) to maximize browser compatibility.
This page also seems to be useful for browser testing https://whatwebcando.today/clipboard.html

[samhed]

https://github.com/zenorocha/clipboard.js

The list of supported browsers there looks great. If they are able to achieve that so should we be! What's the state of this code at the moment?

[juanjoDiaz]

The async clipboard support (which is what's used in this PR) is now also supported in Safari: https://webkit.org/blog/10855/async-clipboard-api/

Firefox keeps having it as an open issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1619251

Just FYI.

[GirlBossRush]

@juanjoDiaz @samhed,

This PR looks like it would get noVNC much closer to an expected experience -- especially now that Chrome, Edge, and Safari support the async Clipboard API. If browser support is a blocker, I could follow up with a new PR that:

• Add an browser API check when defining the clipboard handler
• Log a console warning if either copy/paste event is unsuccessful
• Update the docs and example to include info about noVNC's clipboard API and its requirements

Thoughts?

[samhed]

I'm opposed to merging something that isn't supported by all major browsers, Firefox is important.

I don't have the time to investigate this personally right now, but https://github.com/zenorocha/clipboard.js claims support for Firefox as well, couldn't we do what they do?

[GirlBossRush]

@samhed from what I understand, Firefox's limitations are mostly around async clipboard events e.g. when noVNC wants to copy data to the user's clipboard without their input.

Clipboard.js depends on a "user originating" event to perform a clipboard copy. I suppose a stop-gap would be to offer a button in the UI to let the user retrieve the server's clipboard data.

[AdrianSanchezLopez]

Hi guys, this feature looks really neat, I was wondering if you have any ETA on this

[snickell]

@CendioOssman this is something we'd love to have for our novnc install, and I'm happy to do the dev work necessary if there's any requests you have.... any sense what you'd want to see here (besides no active merge conflicts and it actually tested and working) to consider a merge?

[CendioOssman]

What I'd like to see broad support in the browsers. I.e. this works on at least Chrome, Firefox and Safari (Edge would be nice too though).

Failing that, a clean fallback handling until missing pieces are in place in all browsers.

I haven't kept track of this so I'm afraid I don't know what the current state of the browsers is, or how well this code matches that. Some testing would be helpful to get a better picture.

[samhed]

No, we want this working across the board. If you want to get things moving more quickly, you can help out. We need to troubleshoot the Safari issue and find a fix.

[cebas]

No, we want this working across the board. If you want to get things moving more quickly, you can help out. We need to troubleshoot the Safari issue and find a fix.

I understand. I would like to help, but I'm a Linux user, so I can't help with Safari testing.

[LiuTangLei]

Very much looking forward to this feature can be launched as soon as possible

[cebas]

All checks have passed. We're closer to have clipboard support! Thanks @juanjoDiaz

[juanjoDiaz]

Hi all,

I've rebased and reverted the hacks that I tried for Safari.

This PR has been open now for 3.5 years. Browsers have evolved a bit, but the the problems not so much 😅

Firefox continues to not support reading from clipboard. They seem to be working on it but it will be based on the annoying "Paste" button as Safari is. See https://bugzilla.mozilla.org/show_bug.cgi?id=1770358

Safari continues to work based on the "Paste" button. So, with the current implementation based on the focus event. Whenever you focus on the canvas you get a "Paste" button. If you click on that button, then your data gets copied over. See https://webkit.org/blog/10855/async-clipboard-api/.

So, not much else that I can do here.
I could add a condition so the focus event is not used in Safari and Firefox to avoid the annoying "Paste" button. But that's about it.

[samhed]

Thanks for moving this forward!

I tested your new code, but I still can't make it work well on Safari on macOS. I do get the paste button more rarely now, but it still gets “stuck” occasionally. And even when the paste button shows, it doesn't paste things.

What I did:

1. Opened Safari and Notes next to each other on macOS
2. Connected to a TigerVNC server (linux host) using noVNC
3. Wrote some text in Notes and Copied it using ⌘-c
4. Verified successful copy by pasting in Notes using ⌘-v
5. Clicked in noVNC in Safari
6. Tried pasting to the remote using Ctrl-v, Ctrl-Shift-v, ⌘-v and middle-mouse => Nothing
7. Clicked outside the noVNC remote screen, but still in Safari, then clicked back in noVNC remote
8. I got a paste button that wouldn't go away, when I clicked it, it went away for half a second then reappeared
9. After spamming different buttons on the mouse for a while, it finally gave up showing the paste button (after ~15 seconds perhaps?)
10. When then I pasted using the middle-mouse to the remote, I get the copied text from Notes

I cleared all browser caches in Safari and made sure I pulled the latest version from your branch. Are we certain that the Safari issue with the paste button is properly fixed?

As described above, pasting appears to be possible in Safari on macOS (but still buggy). But copying from the remote to the macOS client doesn't work whatsoever for me. Have you gotten that to work on your end?

I also tested Safari on iPadOS, it still doesn't seem to work at all sadly. As I wrote in an earlier post, it seems it should be possible:

When testing this fiddle in Safari on an iPad things work well, both Copy and Paste:
https://jsfiddle.net/developit/c0aw4sua/

I also tested in both Chrome and Firefox on Fedora. Chrome works flawlessly, it's really nice! Firefox works really well regarding copy, just like you have said.

To summarize, I get the feeling this needs further polishing for Safari, both on macOS and iPadOS/iOS. If we get that working well, I believe we can merge this. We would have a good foundation ready for Firefox once their paste-feature gets released.

[juanjoDiaz]

Clicked outside the noVNC remote screen, but still in Safari, then clicked back in noVNC remote
I got a paste button that wouldn't go away, when I clicked it, it went away for half a second then reappeared
After spamming different buttons on the mouse for a while, it finally gave up showing the paste button (after ~15 seconds perhaps?)

This is the "onFocus" workaround.
We sync the clipboard during the focus event.
On Safari, that triggers the "paste" button.

This is why suggested disabling the pasting completely in Safari.

I also tested Safari on iPadOS, it still doesn't seem to work at all sadly. As I wrote in an earlier post, it seems it should be possible:

When testing this fiddle in Safari on an iPad things work well, both Copy and Paste:
https://jsfiddle.net/developit/c0aw4sua/

No, it doesn't.
It works if you copy using the "copy" button and paste using the "paste" button.
If you copy text for somewhere else and press the "paste" button, you get the annoying "paste" floating menu.

So, all in all, Chrome works so nicely because of the "focus" workaround.
Safari (and in the future Firefox seems like will follow the same approach) prevent this type of workarounds by showing the "paste" floating menu.

So the real question is: what do we do with pasting on Safari and Firefox?
They are trying to actively limit it so it only works if the user press a "paste" button and then the "paste" floating menu; i.e. we can only achieve this by adding a "paste" button on the noVNC UI. But it won't work as "natively" as Chrome does.

Wdyt?

[samhed]

Sorry, I wasn't clear. I have no problem at all with the concept of a floating “paste” button. I think it is an acceptable inconvenience when weighed against the safety issues Chrome users have to face.

When testing this fiddle in Safari on an iPad things work well, both Copy and Paste:
https://jsfiddle.net/developit/c0aw4sua/

No, it doesn't.
It works if you copy using the "copy" button and paste using the "paste" button.
If you copy text for somewhere else and press the "paste" button, you get the annoying "paste" floating menu.

What I meant is that it is possible to copy/paste stuff on an iPad using that jsfiddle, but not when using noVNC. I'm not even getting the paste button on iPad when using noVNC.

So the real question is: what do we do with pasting on Safari and Firefox?
They are trying to actively limit it so it only works if the user press a "paste" button and then the "paste" floating menu; i.e. we can only achieve this by adding a "paste" button on the noVNC UI. But it won't work as "natively" as Chrome does.

Wdyt?

Yeah, I think we need to embrace the floating paste button. But I would appreciate it if we could avoid another button in the noVNC UI.

I think the focus workaround needs further tweaks to not break things on Safari. The focus handler seems to trigger too much. My guess is that clicking the floating focus button briefly removes focus from the page (I have not verified this).

Do you think it would be possible to find a solution that automatically shows the floating paste button at the cursor only when we want it to?

[juanjoDiaz]

Sorry, I wasn't clear. I have no problem at all with the concept of a floating “paste” button. I think it is an acceptable inconvenience when weighed against the safety issues Chrome users have to face.

hehe we are not understanding each other here 😅

The focus handler seems to trigger too much. My guess is that clicking the floating focus button briefly removes focus from the page (I have not verified this).

That's it!
The problem is on how to sync the local and remote clipboard.
Due to security restrictions, all browsers require that you do it as part of a user action.
Which action should we use? Based on the discussion above, I opted for sync the clipboard during the focus event.
That works great in Chrome but means that every focus event on the canvas will show the "Paste" button.
And for some reason, the canvas seems to focus/blur quite often.

Should we use the click event?
That doesn't sound much better. Every user click will show the paste button.

Keypresses don't sound great either...

Which other event do we have that we can use?

So, it seems that adding a button to noVNC UI is the only working approach here... 😕

[samhed]

Thanks for clarifying!

Perhaps we can continue using "focus".. Can we detect if the browser is using the "paste" button approach? If we can detect it in a nice way, we could apply some sort of rate limiting or other kind of workaround to make it usable. What do you think?

[juanjoDiaz]

Hi!

Sorry, I didn't see your answer.

AFAIK, we can't detect if the browser is using the "paste" button other than detecting if the browser is chrome, firefox or safari.

[samhed]

In case we expect the "paste" button to appear on focus, perhaps we can approximately detect it by seeing if we immediately lose focus again?

The next steps should be to solve these two issues:

• limit the "paste" button spam
• ensure the "paste" button is visible even if remote screen covers the entire browser window

[spags-lacework]

Hi, is there any recent update on the status of this PR?

[I0x0I]

It seems this issue is resolved in a forked version kasmtech/noVNC