Virtual files for macOS

[claucambra]

Hi all,

I am very happy to announce we are soon releasing stable virtual files support on macOS. From now on the Nextcloud application will come in two flavours on macOS: standard, and virtual files. The standard version is the desktop client we have been releasing for some time now. The virtual files client supports both standard synchronisation and the new File Provider-based virtual files system provided by Apple on macOS.

No extra steps are required to enable virtual files folders in the virtual files client; once you install the client and configure an account (or use existing accounts), you should be able to access your files via entries in the “Locations” section of the Finder sidebar. You do not need to configure sync folders when configuring a new account; this configuration pertains only to standard synchronisation, not virtual files sync.

The virtual files client will replace the standard client when installed on your computer. Please note that the virtual files edition of the desktop client cannot support Finder integration on synchronised folders using standard sync. You are free to switch back to the standard sync client or switch over again to the virtual files client; there will be no data loss when installing one client over the other.

Please note that our current plans are to support both the standard sync client and the virtual files client concurrently. There are no plans at the moment to replace the standard client.

As always, your feedback is very welcome and I will be trying my best to go through your comments and address any bugs that come up.

Worthy of note for those of you who beta-tested the macOS VFS Beta package:

1. The macOS VFS Beta branding is now dropped, so you will need to reconfigure your virtual files accounts
2. There are new settings for virtual files in the client’s settings menu, including options to enable/disable virtual files for specific accounts, storage usage information and management, and the ability to create debug archives!
3. There are now file sharing options for virtual files! Right-click on a virtual file or folder, and click “Share options” to use this feature.
4. Fast sync is now available and enabled by default, making newly-configured accounts more quick to respond after initial configuration. You can disable this and allow the system to make a slow, full sync of the remote contents when it is convenient to do so.

Download 3.13.0 with virtual files support on macOS here:

https://github.com/nextcloud-releases/desktop/releases/download/v3.13.0/Nextcloud-3.13.0-macOS-vfs.pkg

[sopi20]

It looks like the fileprovider does not use dav_user (i.e. GUID) in the dav urls it is using, but is using username instead, which causes "Principal with name not found" errors in case where ldap auth is being used... any way how to fix this ?

[claucambra]

Noted, feel free to open an issue for this and I will tackle it soon

[claucambra]

I've opened an issue to track this: #6662

[sopi20]

Thank you very much.

[marcotrevisan]

Hi all, and congratulations @claucambra and Dev Team for this release!
I've just installed it and did a first run on my two instances (one of which is very populated).

Let me say I couldn't be happier on how fast and good it currently works (fast sync is on).
On a normal usage it really rocks. In a LAN environment it's finally superior to anything else I've tried so far. On a normal cloud instance it's as fast as the competition.
Previews work as expected. File editing works, saving works and does not seem to break stuff (including the locally open file).
The locking features seem to be still missing, but I knew this was still in progress... at the moment a locked file shows up with an additional lock icon, but no lock/unlock on file open/close is active.

Kudos @claucambra for such an achievement. Thanks Nextcloud for making it possible!
This implementation is without any doubt a game changer for MacOS users.

[BJKle]

Thank you so much. It's working on my own server.
But on a second server it is not working.
Does my administrator has to enable something?
In the finder window there is an error: "Nextcloud encountered an unexpected error. Items may be out of date."
But the log files are empty.
Always using the latest version (Server/Desktop)

[BJKle]

I must be stupid. I restarted a couple of times reinstalled the linked 3 times. Restarted,...
It still looks like the old version. (see pictures).
I even added a second account. No change.
Do I have to enable something in a config file?

[claucambra]

No changes to the config file should be necessary, it is possible that this is a bug -- could you send the desktop client's debug archive?

[BJKle]

where can i send it?
there are my usernames und server addresses in it and I don't want to leave here

[claucambra]

You may send it to claudio.cambra@nextcloud.com

[BJKle]

I got it to run now, after cleaning up the whole system and looking where extcloud somewhere.
Unfortunately VFS runs into an error:

2024-04-24 19:40:19.291322+0200 0x2357 Error 0x0 1110 0 FileProviderExt: [com.nextcloud.desktopclient.FileProviderExt:enumerator] 1 depth readFileOrFolder of url: https://myurl/remote.php/dav/files/Username did not complete successfully, received error: Principal with name Username not found
2024-04-24 19:40:19.291411+0200 0x2357 Error 0x0 1110 0 FileProviderExt: [com.nextcloud.desktopclient.FileProviderExt:enumerator] Finishing enumeration for user: Username https://myurl with serverUrl: https://myurl/remote.php/dav/files/Username with error Principal with name Username not found

My Admin has setup the users using LDAP. I login with "login name", but the "username" ist something cryptic. Trying to use the cryptic username fails "unknown user".

[strozzascotte]

Thank you very much and congratulations @claucambra!
I'm testing it right now and so far it's flawless.

[chaehni]

Why are there two clients if the new one supports standard and virtual sync mode?

[claucambra]

The standard client supports finder integration for standard sync folders, the virtual files client can't.

[chaehni]

I don't understand the "can't". Other cloud drives like Ondrive support Fimder integration. One can set a file to offline/cloud only directly on the files context menu. Or does Finder integration refer to something else?

[gradypark86]

Hi all, I uninstalled Release 3.12.3, restarted my mac, and installed Release 3.13.0-fileprovider-rc1 (macOS virtual files). Unfortunately, there is no entry in the “Locations” section of the Finder sidebar. I didn't set any folders in Standard file sync.

I'd like to provide logs to resolve the issue, is log files in /Users/username/Library/Preferences/Nextcloud/logs be helpful? When I click the Create debug archive button under Virtual file sync, the nextcloud client not responding, I have to force quit it in Activity Monitor. If there is anything I can offer, please let me know.

Nextcloud Server version: 28.0.4 - 28.0.4.1
MacOS version: 14.4.1
Log file: 20240423_1053_nextcloud.log

[claucambra]

Hi, sometimes Finder doesn't immediately respond to the creation of new locations. If you can't create a debug archive for the virtual files component, can you create one as normal for the desktop client?

Running log stream --debug --predicate 'process=="FileProviderExt" AND subsystem=="com.nextcloud.desktopclient.FileProviderExt"' in the terminal can also produce useful debug logging

[claucambra]

Okay, the issue, your server address has a port number suffixed with a colon, which NSFileProviderManager does not like. I will create an issue for this

[claucambra]

Tracking at #6661

[gradypark86]

I see, thank you very much @claucambra .

[gradypark86]

Great, I changed the port number to 443 and v3.13.0-fileprovider-rc1 runs normally. There are some small questions.

1. Enable virtual files is off by default and it's actully on. Files can sync normally.
   

2. After downloading a file, the modified time of the file will change to the current time.

3. There's no option such as "Always keep on this device".

[marcotrevisan]

Hi @claucambra, I found a small glitch in the UI (maybe it's already known...): sometimes clicking on the Settings menu item causes a complete freeze of the client's UI, I have to kill it or the dialog will stay open and frozen.
When this happens, the client seems unable to open the settings panel and the UI stays frozen with the the menu items open.

I hope this helps.
Kind regards

[marcotrevisan]

This won't cause issues to the FileProvider operation. I tested it and it fetches data, enumerates folders, ... all seems normal in Finder.

[claucambra]

Hi all, I've been going around opening pull requests for some of the issues faced here. Unfortunately, one new issue relates to autoupdating, which will not work correctly in the virtual files client's release candidate (RC1). Once the final, stable 3.13.0 release comes out you will be prompted to update and unfortunately you will receive a standard, non-virtual files client.

I've issued a fix for this but it cannot be retroactively applied to RC1, so you will have to manually re-download the 3.13.0 virtual files release when it comes out. Thankfully, this will be fixed going forward from 3.13.0.

[hrdwdmrbl]

Question: Is the plan to eventually sunset the "standard" version? The "virtual files" version would seem to be the future and superior version, but it seems like it might still be missing a couple features from the "standard" version.

I know there is also talk about an independent macOS desktop client. If those plans were (hypothetically) to not move forward, would the plan be to sunset the "standard" version eventually?

I know that these plans might still be in discussion, so if the current answer is unknown, that's okay too.

[claucambra]

We have now released 3.13.0 -- I have updated the download link in the top post :)

[smeingast]

I am having an issue uploading larger with the new fileprovider method. I am using a cloudflare tunnel and nginx proxy manager and for the standard sync procedure, I always managed to fix this by adjusting the nextcloud.cfg. Is this perhaps again an issue with the new client?

[chaehni]

I, too, have trouble uploading big files.

[smeingast]

It's strange. I had the new client (3.13.0) now running for several hours and at some point the big files showed up. However, syncing is still far from smooth. For instance, when I add many new files at once, these files take forever to be marked 'uploaded' in the finder, but I can see them already in the web interface of nextcloud.

[chaehni]

yes, same experience. At least they eventually get synced.

[marcotrevisan]

I just tried uploading a big file (~350 MB) from a slow connection and watched the web server log.
It worked, but It's not currently using the chunked upload mode (the log shows a single PUT line at the end of the upload), so you'll have to make sure your PHP and web server have large enough max request sizes and timeouts.

Also, please note that during the upload of a folder containing many files, the server will experience a certain amount of load (CPU load + requests from other clients due to the push notifications). Please check that too and if your server's CPU reaches 100% of all cores during the upload, be prepared for delays.... Make sure your php configuration allows for enough child processes. Server too busy means possible timeouts and FileProviderExt may also suffer from that.

@claucambra it might be worth checking how many push notifications will come out from the upload of a populated folder. Ideally it could be just one at the end of the upload, for the folder itself, but actually it might be one for each file/subfolder uploaded.
Thanks!

[chaehni]

I have an issue where files I try to free are not actually freed. I have now 250GB of storage taken in my folders but in Finder all files show up with a cloud icon. In the menu to delete the local files (it's awkward to access imo), the parent folder of the offending files shows up (but none of the actual files) and when I try to remove it and refresh the view it appears again. It seems I now have no way of deleteing local copies.

[marcotrevisan]

If you fully download a folder using the context menu in Finder, you should then find a "remove downloads" action.
Alternatively, in the "virtual file system" menu in the client settings you'll find the "evict local copies" button, once clicked you'll be presented with a list of all the files/folders and be able to evict the ones you like.

I would say it would be nice to have an "evict all" button too...

[chaehni]

Yes, I found the "evict local copies" button. I'd prefer such an option on partially downloaded folders, too. If I have 10 files downloaded in a folder I now have to decide between removing them individually or downloading potentially gigabytes of data only to then remove everything.

[marcotrevisan]

Yes that would be a very nice feature.
For the sake of completeness, there's also a built-in function in Apple's FileProvider framework, that when the local disk space is running low, it will evict materialized files automatically, I guess based on some sensible logic (like starting from the least recently used...).

[chaehni]

Good to know. I was not aware of that. Thanks for pointing it out.

[marcotrevisan]

I also noticed, using the current version you can uncheck the "use virtual files" flag, which will wipe your current local storage, and then re-enable it. Please be aware it won't ask you confirmation, so do it only if you haven't just added files to it...

[Mantany]

First of all, thank you for this great feature and the work of @claucambra 🎉
I would like to share my experience and solutions so far:

Installation:
My installation (upgrade from the "classic sync" version) did not go very well:
I uninstalled the "classic sync“ version with Appcleaner (removed all configurations, etc.) and installed the new version (3.13.0).
However, the new app only appeared for a few seconds and then closed by itself.

Workarounds:
It turned out that under the user Library => Application Support => FileProvider
there were some FileProviders that were already using my nextcloud server. By deleting the folders and reinstalling the nextcloud application, I was able to get it working.

I also had a problem with the location not showing up immediately in the Finder:
What helped me here is: open Finder, then Finder -> select Preferences in the top menu bar -> deselect iCloud storage, select iCloud storage again

Now its working very well, thought that it might be helpful to somebody having the same problems

[marcotrevisan]

Question: some days ago I added Italian translations in Transifex for the new features (and also some keys that are still untranslated) in the "client" resource.
Are they automatically merged or do they pass through validation? Just curious...
Thanks

[rakekniven]

@marcotrevisan

Are they automatically merged

Yes.

[2xsaiko]

I have a folder with a huge amount of small files in it (it's a Dwarf Fortress save folder, ~8500 files, 95MB in total according to the web UI). Looking at the folder in the Finder/trying to download them all hangs forever, or at least for longer than 20 minutes (looking at the Nextcloud UI, its state is Syncing, and the progress bar is currently at 20%).

I've had this synced with the standard file sync before and that worked fine.

[chaehni]

this also confuses me. The sync bar seems to not be indicative of any progress at all. Mine has been stuck on 30% since I installed the client but all my files seem to be synced.

[marcotrevisan]

Now I get what progress bar you mean. In my case it was so quick to go to 100% that I didn't remember it was there :-)

Perhaps the debug log is worth a look...

[2xsaiko]

Where can I find that? I looked for a log, and found one in Application Support/Nextcloud, but it doesn't seem to be used for the file provider since the latest entry was a few months old.

Can you get it via the "create debug archive" button?

[marcotrevisan]

Yes that's the button for that. I can't say wether or not it will contain the needed info because my client is working properly so far...

[2xsaiko]

Nope, it doesn't say anything. It's just really slow. https://gist.github.com/2xsaiko/882d3e2a1eb140dec2f2d6d9f2c019f1

edit: this is the folder: https://cloud.knifepoint.net/s/FB5CwYaZfXzQwzq
edit 2: to be clear, it did eventually finish reading the folder, this time it took maybe like 10-15 minutes.

[smeingast]

Just also quickly wanted to report that on both my Macs (iMac Pro and M1 MacBook Pro) I can't open the NextCloud VFS app anymore after I quit the app once. To make it work again, I have to uninstall, restart, and reinstall the client. Not sure what the cause is, but I am attaching the crash report.
Nextcloud-2024-05-07-152936.txt

[CSS-Noclip]

I'm using this feature and I find it great. Thank you very much for your efforts.
I want to ask if in the future there will be additional features for online use without having to download the file?
Thanks again for this great feature.

[claucambra]

Hi, yes, there will be. Sharing is working at the moment without having to download the target file/folder, but I am also working on locking, and more

[johannesvogel]

Hey! Unfortunately for me (on MacOS 14.4.1) the app immediately closes after opening. After finishing the configuration assistent I can see the popup for a split second before it closes down and I can't start Nextcloud again. I have a log that I could share, just let me know where I should send it!

[chaehni]

I had the same problem. What worked for me was installing the regular version and then install this release over it. I suspect that this release on its own doesn't write certain required files.

[johannesvogel]

@cyphar Thank you for the tip, but unfortunately that did not help :-/

[gradypark86]

Is the port number of your nextcloud server 443? I found that if it 's not 443 then the nextcloud vfs client will like you said.

[johannesvogel]

@gradypark86 yes, my instance uses a custom por! Then maybe that's the issue

[kklem0]

Right click option for making things online or offline available like dropbox would be REALLY appreciated since for me this is the whole point of having proper virtual files.

[kaorihinata]

Unfortunately, the file provider version attached above instantly segfaults for me. I've tried other suggestions such as installing over top of the old version, uninstalling everything, restarting, then installing it again, cleaning out old data from the Library directory, and nothing seems to allow it to start. No useful error messages either.

Nextcloud-2024-05-11-160221.ips.txt

[marcotrevisan]

The segfault issue is tracked in #6730

[marcotrevisan]

Hi all,
@claucambra I've noticed some glitch when I create a new sub-folder AND I move files from the parent folder into it very soon (i.e. before the cloud icon on the subfolder becomes dotted):

• sometimes my Finder's view of the folder will make the pasted files expire, but the files are in their correct position at server side. To "refresh" the view I can create an empty folder into the new folder, and soon after the moved files will also appear.
• other times, the files won't get moved and the newly created folder will show a "cloud with exclamation mark" icon. Finder shows the files in it, all with the same exclamation mark, and the server side won't show any file in the new subfolder. I'm unable to fully solve it. The exclamation mark on the subfolder will go away after renaming it, but I'm unable to do the same on the files. Tried moving them back, but the exclamation mark stays on.

In conclusion: at the moment it's advisable to wait until the cloud icon becomes dotted before adding files to a new folder.

[marcotrevisan]

EDIT: I was able to remove the exclamation marks in the second case, my steps were:

1. on Finder, move the files back to the parent folder;
2. rename them at server side;
3. wait for a while. You may try to speedup the process creating an empty folder and deleting it. However it will take a while.
4. the FIleProvider file names will "win" and the server side will then show the file names prior to their modification. You'll be able to see in the "activity tab" the two oppisite rename actions happened.

[marcotrevisan]

The recommendation stays valid however: when you create a new folder, please wait for its cloud icon to become "dotted" before doing stuff with it.

[hrdwdmrbl]

General question: Is searching with Finder supposed to work? I have found it to be non-functional.

[marcotrevisan]

Seems like it works on my mac. When I select the Nextcloud position in Finder and search for a known substring of my available file names, it shows them in the results. I guess it won't look into contents of non-materialized files.
Using Mac OS Sonoma 14.4.1 on Apple Silicon M1 (MBA 2020)

[claucambra]

So two things with search -- with fast enumeration enabled, it won't be able to provide files or folders in unexplored paths. And @marcotrevisan is right, AFAICT it won't provide content search for non-materialised files

[hrdwdmrbl]

Is fast-enumeration the same as "fast sync"?