QFieldSync does not sync edited QField project when copied from iPad/iPhone via Onedrive

[andreimirt]

Describe the issue

I just started testing out qfield on a demo project just to get the feel of it.
I created a QGIS project (3.23.13-ltr), then I created a single point layer as a new geopackage and added a single feature.
I set the layer for cable export as Offline editing (still don't know what's the difference between offline editing and copy, even after researching the documentation, but that's just a sidenote).
Then I exported the project to Onedrive so it can be moved to Qfield on ios.
I moved the project from the Onedrive folder to Qfield root and opened it in Qfield from there.
The project opened just fine so I added 1 feature to the layer without any issues.
I copied the feature back to a separate folder in Onedrive. This is where the first issue arises.
Then I tried to synchronize the QGIS project via Qfield plugin.
I was prompted by the "Handle unavailable layers" QGIS pop-up, which prompted me to point towards data.gpkg, basemap.gpkg file etc.
I pointed towards them in the imported project (apply) then I got prompted again (apply) and it seemed to work, I even got the "Open the original QGIS project pop-up of Qfield plugin".
The issue is that the layer was not synchronized and it only contained the original data.
So I tried to resynchronize again, but then I was prompted that this data has been synchronized already, even though the geopackage in the QGIS project does not contain the data recorded on Qfield (ios).

When I separately opened the project as it was copied from the tablet, the data.gpkg did in fact contain the new data, it's just that synchronization didn't update the QGIS desktop geopackage, but it did record somewhere that the data was synchronized, probably in the project file.

Reproduction steps

Steps to reproduce the behavior:

1. Cable export project via Onedrive to Qfield root folder on ios
2. Edit in Qfield
3. Copy the QField project using Files (ios app) from Qfield root to Onedrive <-- This is where the first issue arises
4. Use the Synchronize button of QFieldSync in Qgis Desktop
5. Handle unavailable layers by point to the correct file for each layer in the just imported folder <-- This situation is not properly handled by QFieldSync
6. Observe that even though no errors were given, the layers on desktop that were modified in QField were not synchronized <-- This is the result of 5. not being properly handled

Expected behavior

The layers on Desktop should be synchronized.

Observed behavior

The layers on Desktop were not synchronized without errors.

Screenshots and GIFs

Desktop (please complete the following information)

• Desktop OS: Windows 10
• Mobile OS:
• QGIS Version 3.28.13
• QFieldSync Version 4.6.1
• QField Version 3.0.5

Mobile (please complete the following information)

• Device: iPad Pro (4th)
• OS: iPadOS 17.1.1
• QField version: 3.0.5 - Amazonia (bfecdc v3.0.5)

Additional information

• Problem started happening recently, didn't happen in an older version of QField: [N/A]
• Problem can be reliably reproduced, doesn't happen randomly: [Yes]
• Problem happens with all files and projects, not only some files or projects: [Yes]

Workaround

Rename QField files to their expected filename (e.g. remove timestamps). In principle, they should match the filenames of the project prepared by QFieldSync for QField.

User's technical understanding of the problem

Getting straight to the point, the issue occurs because when File app on iPadOS copies files to Onedrive folder, it appends the date and time to each file name. So, e.g. data.gpkg becomes data 2023-12-13 07_40_17. On import the qgs project file from QField obviously contains references to filenames without the appended timestamp. At import I suppose some sort of open then join happens behind the scene. Since the open is done with the QGIS library, QGIS correctly identifies that the referenced files don't exist and offers the possibility to fix the issue by point to the correct files, which in this case it would be "data.gpkg" -> "data 2023-12-13 07_40_17". Unfortunately QFieldSync does not take into consideration the new filenames and I suppose it attempts to still merge data from the file with the expected filename of "data.gpkg", as recorded in the .qgs file. I noticed that a 0 bytes data.gpkg is created during synchronization. This indicates that QFieldSync tries to open the data.gpkg with a "Create if doesn't exist" flag, and then it syncs from that empty data.gpkg file, instead of syncing from the file indicated by QGIS'. This is probably why the synchronization doesn't fail at all, because the synchronization with the just created empty data.gpkg is successful, but clearly it is not the intended behavior.

Clearly, the main issue is not with QField, but with the Files app of Apple that for some reason understands "Copy" as "Copy and rename by adding a timestamp to each file name". The issue has been reported both to Apple (Files), and to Microsoft (Onedrive). Apple says that this is normal behavior for copying Files and Microsoft says that there might be a compatibility issue between Onedrive and Files. The issue has been reported for years and there is no solution in sight from there.

Nevertheless, QFieldSync should at minimum give an error and not synchronize, with a proper error message Could not find [expected file]. Synchronization FAILED!. This is probably easily achieved by removing the "create if doesn't exist" opening flag of the data.gpkg file at sync time. Moreover the issue should be documented, although I suppose this issue being public at least partially takes care of that, but a mention in the documentation that the filenames should not be modified would be welcomed.

But the proper solution is that the QFieldSync should take into account the new filenames as returned by QGIS open function that pops up the Handle unavailable layers dialog, and synchronize from there instead. Otherwise, having the "Handle unavailable layers" dialog popping up, pointing to the correct files, and then the now available layers to not be handled is highly confusing to the user.

I understand that this was not taken into consideration because who would modify filenames that are supposed to be used strictly by QField and QFieldSync. There is no conceivable use case that would require an intentional change of these filenames. Nevertheless, users are surprising as always, especially Apple users "Think differently".