apply_deltas: Work on TODOs // Refactor

[miili]

Dear all,

I have been cleaning up the mess left behind #570, reconstructing the 2000+ deltas, fixing the fids and sourcePks. Subsequently debugging and fixing the apply_deltas.py script.

In that process I found 8 TODOs, fixed obscure code, and improved on the performance cutting the runtime of 2000 deltas to 30%.

Here is a taste of what is deployed in production:

if new_feat_attrs:
    # `fid` is an extra field created during conversion to gpkg and makes this assert to fail.
    # if fields.size() < len(new_feat_attrs):
    #     raise DeltaException('The layer has less attributes than the provided by the delta')

    if new_feat_attrs:

This should never go through any review!

I am honestly shocked to see this historically-grown script in production. This is at the very heart of the operation! It and should have guards and rails everywhere.

Excuse my harsh tone here but I have been cleaning up the messed QGIS project for several days now, praying I didn't to any mistakes. For us, the customer, there are a lot of men-hours, expensive equipment and investment at stake.

Productive suggestions

... for a serious refactor

1. Create proper Python packages
2. Modern scaffolding (ruff, black, pre-commit, unittests).
3. Pylint, for a project of this scale and responsibility you need static code analysis!
4. Pydantic, remove all dicts.
5. Use asyncio otherwise the back end API won't scale properly and you will run into more problems in the future.
6. Deploy only built packages to Docker. Currently Docker and Python world are too interleaved.
7. Leverage GitHub environments for workflows.

[graymondon]

Thanks a lot for this your feedback.

We are experiencing the same problem right now, with people harvesting data on the field, some of them not integrated and deadlines to maintain with no ability to redo or postpone the job.

So, do you have any advice that could be useful :

• What to NOT do if u don't want to loose the data collected (even if u have to integrate them manually after)
• Is there a quickfix planned for this issue ? If so, will it be able to correct the problem retroactively (apply deltas that weren't applied)

Thank you in advance for your support.

[miili]

I downloaded the uploaded deltas through the QFieldCloud API (https://app.qfield.cloud/api/v1/projects/) and manually corrected all deltas.

Then I applied QFieldCloud's apply_deltas.py script on the local project. I had to bugfix apply_deltas.py, that is what motivated writing this frustrated issue.

Creating and vetting this workflow cost me 2+ days.

[graymondon]

Thanks, I'll try to do the same.

One last question : can you describe what has to be corrected manually before applying the script ? I cannot find the problem in the deltas downloaded from the API ?

[miili]

Check for differing fids. QField maintains a local and a remote fid, for our case there was a mismatch, so that the wrong feature was modified by QFieldCloud.