"Entity Too Large" when downloading a project using qfield. #872

Forstrevier · 2024-02-01T23:52:10Z

Hi,

Since the update to 0.24.0, when I try to download a project locally with qfield, I get the following error message (excerpt) during the process.
It is interesting to note that a directory with the ID 7b0ea8e2-c788-4d04-ac21-24196879a2f9 does not exist at all in the project file system. Even if I pretend to the system that there is no (last) package by deleting the last package ID and the timestamp in the project table and also removing the package directory in the file system. I still get the same error message.
Does anyone know what is going wrong here?
Thanks.

Feedback pre:

{
  "container_exit_code": 0,
  "error": "Requested \"http://<here-is-my-domain-server>/api/v1/packages/d1a3e564-00b6-4d7c-aa1d-b3383935508b/7b0ea8e2-c788-4d04-ac21-24196879a2f9/files/data.gpkg/\" and got \"413 Request Entity Too Large\":\nb'<html>\\r\\n<head><title>413 Request Entity Too Large</title></head>\\r\\n<body>\\r\\n<center><h1>413 Request Entity Too Large</h1></center>\\r\\n<hr><center>nginx/1.22.1</center>\\r\\n</body>\\r\\n</html>\\r\\n'",
  "error_class": "QfcRequestException",
  "error_origin": "container",
  "error_stack": [
    "  File \"/usr/src/app/qfc_worker/utils.py\", line 556, in run_workflow\n    return_values = step.method(**arguments)\n",
    "  File \"/usr/src/app/qfc_worker/utils.py\", line 255, in upload_package\n    client.upload_files(\n",
    "  File \"/usr/local/lib/python3.10/dist-packages/qfieldcloud_sdk/sdk.py\", line 246, in upload_files\n    raise err\n",
    "  File \"/usr/local/lib/python3.10/dist-packages/qfieldcloud_sdk/sdk.py\", line 232, in upload_files\n    self.upload_file(\n",
    "  File \"/usr/local/lib/python3.10/dist-packages/qfieldcloud_sdk/sdk.py\", line 288, in upload_file\n    return self._request(\n",
    "  File \"/usr/local/lib/python3.10/dist-packages/qfieldcloud_sdk/sdk.py\", line 811, in _request\n    raise QfcRequestException(response) from err\n"
  ],
  "error_type": "API_OTHER",
  "feedback_version": "2.0",

The text was updated successfully, but these errors were encountered:

suricactus · 2024-02-02T13:08:56Z

Can you try with a smaller offline layer if it works fine for you? Seems the offlined version of your data in data.gpkg is way too large. Just to confirm it works with smaller projects.

Do you have a large postgis layer?

If you revert back to the previous QFC version, does it work as expected?

Forstrevier · 2024-02-02T16:55:32Z

I used a different project as a test. In this project, the data.gpkg file went through without errors. Then the process got stuck on a file called "Luftbild RLP.tif" with the same error. This is a geo-tiff with a size of 93 MB.

I then excluded this tif file from the project. Then the project was downloaded from qfield without errors. The process was error-free according to qfc-log.

There are still crazy things that happen. In qfield the project is displayed as "locally available". But I can't call it up in qfield. Whenever I tap on the project, it is downloaded again and not started locally.

-- snip --

15:34:58.195 /usr/local/lib/python3.10/dist-packages/qfieldcloud_sdk/sdk.py INFO     Uploading file "data.gpkg"…
15:34:58.203 /usr/local/lib/python3.10/dist-packages/qfieldcloud_sdk/sdk.py INFO     Uploading file "Luftbild RLP.tif"…
::>>>::c056496d-9198-49ab-9506-f270ec9f6586 1

Feedback pre:

{
  "container_exit_code": 0,
  "error": "Requested \"http://<my_domain_server_here>/api/v1/packages/26a1791c-c704-4240-8eae-181da3933abb/53016345-3d7e-4ea8-a1e6-2536ea079647/files/Luftbild%20RLP.tif/\" and got \"413 Request Entity Too Large\":\nb'<html>\\r\\n<head><title>413 Request Entity Too Large</title></head>\\r\\n<body>\\r\\n<center><h1>413 Request Entity Too Large</h1></center>\\r\\n<hr><center>nginx/1.22.1</center>\\r\\n</body>\\r\\n</html>\\r\\n'",
  "error_class": "QfcRequestException",
  "error_origin": "container",
  "error_stack": [
    "  File \"/usr/src/app/qfc_worker/utils.py\", line 556, in run_workflow\n    return_values = step.method(**arguments)\n",
    "  File \"/usr/src/app/qfc_worker/utils.py\", line 255, in upload_package\n    client.upload_files(\n",
    "  File \"/usr/local/lib/python3.10/dist-packages/qfieldcloud_sdk/sdk.py\", line 246, in upload_files\n    raise err\n",
    "  File \"/usr/local/lib/python3.10/dist-packages/qfieldcloud_sdk/sdk.py\", line 232, in upload_files\n    self.upload_file(\n",
    "  File \"/usr/local/lib/python3.10/dist-packages/qfieldcloud_sdk/sdk.py\", line 288, in upload_file\n    return self._request(\n",
    "  File \"/usr/local/lib/python3.10/dist-packages/qfieldcloud_sdk/sdk.py\", line 811, in _request\n    raise QfcRequestException(response) from err\n"
  ],
  "error_type": "API_OTHER",
  "feedback_version": "2.0",

-- snap --

Forstrevier · 2024-02-02T17:22:43Z

One more important piece of information about the previously mentioned test project.

If the project is set to :

QGIS core Offline Editing (deprecated)
The data.gpkg file runs through without errors. The error "Request Entity Too Large" only occurs with the tif file.
Optimized Packager
The error "Request Entity Too Large" already occurs with the data.gpkg file

suricactus · 2024-02-02T17:28:58Z

Thanks for the input. So it seems the "Optimized packager" produces bigger offline package. It would be interesting to know a bit more about your vector layers? Is it single data source but multiple layers? Is it a pg behind it? Anything that will help us to identify the issue is welcomed. If you can share the data with me, would be even better.

suricactus · 2024-02-02T17:30:39Z

If you revert back to the previous QFC version, does it work as expected?

This is important bit would help me to identify the regression, is it working flawlessly, or you still get:

The data.gpkg file runs through without errors. The error "Request Entity Too Large" only occurs with the tif file.

suricactus · 2024-02-02T17:46:24Z

Can you please try one more thing for me? In the latest QFieldCloud version, please replace the nginx image from nginx:stable to nginx:1.23.4-bullseye.

Forstrevier · 2024-02-02T18:21:05Z

Hi, the change of image from nginx:stable to nginx.1.23.4-bullseye did not change the errors in either packaging mode.
I will take care of the other tests and information you requested.

Forstrevier · 2024-02-02T18:40:46Z

How can I send you the test project as a zip file? It doesn't seem to work in github.

Thanks for the input. So it seems the "Optimized packager" produces bigger offline package. It would be interesting to know a bit more about your vector layers? Is it single data source but multiple layers? Is it a pg behind it? Anything that will help us to identify the issue is welcomed. If you can share the data with me, would be even better.

suricactus · 2024-02-02T18:44:20Z

ivan at opengis.ch

Forstrevier · 2024-02-02T18:55:33Z

I have sent you a test project. And yes, almost all layers have pg behind them.

Forstrevier · 2024-02-02T21:34:03Z

If you revert back to the previous QFC version, does it work as expected?

This is important bit would help me to identify the regression, is it working flawlessly, or you still get:

The data.gpkg file runs through without errors. The error "Request Entity Too Large" only occurs with the tif file.

I have now tested the qfield download of the test project with qfieldcloud 0.23.2 and 0.23.0. The error "Request Entity Too Large" now also occurs with the previous versions. However, I have not reversed the database migration.

Forstrevier · 2024-02-02T22:29:13Z

I had now set up the following test scenario:

I used the test project from QGis to create a completely new qfc project via the qfc-extension. This worked flawlessly. The qfc-extension now creates its own directory locally. The qfc-extension then transferred exactly this directory to the qfc-server. Everything went smoothly and without errors.

Apparently the qfc-extension creates separate gpkg files for each pg-layer. Incidentally, there is no longer a file called data.gpkg in the directory.

Now I do a project downlaod from qfield (qfield is updated and the project is created from scratch with the qfc-extension of qgis). And that doesn't work again.

The error situation is the same as already described. Depending on the packager, the process gets stuck either at data.gpkg or at the tif file according to the log. Interestingly, as I said, there is no data.gpkg file in the project at all.

What's going wrong?

suricactus · 2024-02-03T11:21:08Z

It's not because of the DB migration. Most probably some of the underlying docker images is casuing this.

Forstrevier · 2024-02-03T19:58:42Z

I have now reached the point where I can download a test project consisting exclusively of pg layers with the old packager mode (deprecated) locally via qfield without errors.
The same project, set to the optimized new packager mode in qfc, still generates the known error during qfield download.

It is very strange that I cannot open a project downloaded locally from qfc without errors in qfield either. Tapping on it only starts the download again and again. But this phenomenon is possibly a topic for the qfield app area.

What is really annoying is the very spartan frontend introduced with qfc version 0.24.0. For example, if you want to add collaborators to a project, you can only create NEW users. It is not possible to select existing users, the userlist-dropdown field is empty.

I continue to look for a solution and a way out of this confused situation. My qfc server is currently unusable.

Forstrevier · 2024-02-05T22:41:17Z

It's not because of the DB migration. Most probably some of the underlying docker images is casuing this.

My images, containers, volumes are fine.
I have set up the following test scenario for verification purposes:

qfield-side several end devices with qfield 3.1.9 Borneo on Android.
qfc-side I went back to qfc 0.23.2, so that I have a better to use frontend.
qgis-installation at qfc I have changed from qgis:final-3_32_3 to qgis:final-3_34_3, so this warning also disappeared from the qfc log: QGSSTDERR WARNING Logged warning: Loading a file that was saved with a newer version of qgis (saved in 3.34.3-Prizren, loaded in 3.32.3-Lima). Problems may occur.
I did this by changing the Dockerfile in the docker-qgis folder. Apparently the qfc-update does not automatically update the QGis installation. libqfieldsync is now also up to date.
As a test I created a simple project, no postgres, just a pgkg file, point geometry, with 2 fields. Nothing else at all. The test project is attached here as a zip file.
The QGis-Version is 3.34.3-Prizren, the qfc-extension is the current one (4.7.0).

Test:

Creating the qfc project via the QGis extension - error-free.
Uploading the project files from QGis to qfc - error-free.
Downloading the project from qfield:
on the part of qfc - error-free
on the part of qfield - failed (previously no error was shown there, but instead of starting the project the download was carried out again and again, if I wanted to call up the project.).

I am also attaching the log file from qfc regarding the download of the qfield test project. For qfc the process was error-free. And nothing in the log suggests to me a reason why qfield thinks there is an error and the project does not start.

Based on the test results, I assume that either an error with qfc, libqfieldsync or qfield or a combination of all of these is the cause of the malfunction. In any case, it is not my docker.

testproject.log
testproject.zip

Forstrevier · 2024-02-22T20:04:40Z

Hi,
even with qfc version v0.25.0 the download of a project to qfield devices still does not work.
In a project with postgres layers, the already known error occurs when downloading the data.gpkg, documented in the qfc-log (see below). It is the only error.
With the simple test project already posted here, no error occurs in qfc-log, but qfield shows an error message and the project cannot be loaded in qfield.

The suspicion you expressed that my Docker image was not in order has already turned out to be wrong. I have already regenerated the images countless times. Everything is fine on my self-hosted qfc server, including the Docker images, containers and volumes.
When can we finally use QFC again so that it works? The trouble only started with version 0.24.x.

Regards,

19:51:46.436 /usr/local/lib/python3.10/dist-packages/qfieldcloud_sdk/sdk.py INFO Uploading file "data.gpkg"…
::>>>::3b470bce-5b9d-4ecf-94e7-32c3484c82c7 1

Feedback pre

{
"container_exit_code": 0,
"error": "Requested "http://qfc-server-adress here>/api/v1/packages/d1a3e564-00b6-4d7c-aa1d-b3383935508b/5eecf1dc-9ec2-40ea-919f-554739d08375/files/data.gpkg/" and got "413 Request Entity Too Large":\nb'\r\n<title>413 Request Entity Too Large</title>\r\n\r\n

413 Request Entity Too Large

\r\n

nginx/1.24.0\r\n\r\n\r\n'",
"error_class": "QfcRequestException",
"error_origin": "container",
"error_stack": [
" File "/usr/src/app/qfc_worker/utils.py", line 556, in run_workflow\n return_values = step.method(**arguments)\n",
" File "/usr/src/app/qfc_worker/utils.py", line 255, in upload_package\n client.upload_files(\n",
" File "/usr/local/lib/python3.10/dist-packages/qfieldcloud_sdk/sdk.py", line 246, in upload_files\n raise err\n",
" File "/usr/local/lib/python3.10/dist-packages/qfieldcloud_sdk/sdk.py", line 232, in upload_files\n self.upload_file(\n",
" File "/usr/local/lib/python3.10/dist-packages/qfieldcloud_sdk/sdk.py", line 288, in upload_file\n return self._request(\n",
" File "/usr/local/lib/python3.10/dist-packages/qfieldcloud_sdk/sdk.py", line 811, in _request\n raise QfcRequestException(response) from err\n"
],
"error_type": "API_OTHER",
"feedback_version": "2.0",

Forstrevier · 2024-02-22T23:24:52Z

Hi @suricactus

I have now completely reinstalled qfc on a server. I then transferred the 2 QGIS test projects (one with pg layers, the other a simple gpkg) to qfc.

The result:

the errors are exactly the same as already described.
what works now is logging in via username (previously only possible via email address) and a correctly displayed frontend. This indicates that there were already migration inconsistencies in previous updates of qfc.

Overall, despite reinstallation, qfc has been unusable for me as a self-hoster since version 0.24.x.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"Entity Too Large" when downloading a project using qfield. #872

"Entity Too Large" when downloading a project using qfield. #872

Forstrevier commented Feb 1, 2024 •

edited by suricactus

suricactus commented Feb 2, 2024

Forstrevier commented Feb 2, 2024 •

edited by suricactus

Forstrevier commented Feb 2, 2024

suricactus commented Feb 2, 2024

suricactus commented Feb 2, 2024

suricactus commented Feb 2, 2024 •

edited

Forstrevier commented Feb 2, 2024

Forstrevier commented Feb 2, 2024

suricactus commented Feb 2, 2024

Forstrevier commented Feb 2, 2024

Forstrevier commented Feb 2, 2024

Forstrevier commented Feb 2, 2024

suricactus commented Feb 3, 2024

Forstrevier commented Feb 3, 2024

Forstrevier commented Feb 5, 2024

Forstrevier commented Feb 22, 2024 •

edited

Forstrevier commented Feb 22, 2024

"Entity Too Large" when downloading a project using qfield. #872

"Entity Too Large" when downloading a project using qfield. #872

Comments

Forstrevier commented Feb 1, 2024 • edited by suricactus

suricactus commented Feb 2, 2024

Forstrevier commented Feb 2, 2024 • edited by suricactus

Forstrevier commented Feb 2, 2024

suricactus commented Feb 2, 2024

suricactus commented Feb 2, 2024

suricactus commented Feb 2, 2024 • edited

Forstrevier commented Feb 2, 2024

Forstrevier commented Feb 2, 2024

suricactus commented Feb 2, 2024

Forstrevier commented Feb 2, 2024

Forstrevier commented Feb 2, 2024

Forstrevier commented Feb 2, 2024

suricactus commented Feb 3, 2024

Forstrevier commented Feb 3, 2024

Forstrevier commented Feb 5, 2024

Forstrevier commented Feb 22, 2024 • edited

413 Request Entity Too Large

Forstrevier commented Feb 22, 2024

Forstrevier commented Feb 1, 2024 •

edited by suricactus

Forstrevier commented Feb 2, 2024 •

edited by suricactus

suricactus commented Feb 2, 2024 •

edited

Forstrevier commented Feb 22, 2024 •

edited