[BUG] Interrupted uploads do not resume #569
Labels
Comments
JakeSmarter
changed the title
|
🤦♂️ I have just spent over 4 hours uploading a sequence of 163 images worth 618,237,384 bytes. mapillary_tools is still not done because it is constantly resetting to either 0 or an arbitrary number. This is something that should have been completed in about 5 minutes. And, before you ask: no, my connection is stable. You see, this is exactly what I mean by “Mapillary being dysfunctional”; every upload tool from Mapillary is completely useless and unusable. Or, a horror to use and nobody seems to care. It is almost offending if you look at the amount of hoops I have to jump through in order to just contribute some meager amount of images voluntarily. Just because I am geek I have the patience and skill to deal with it. But, how do you expect any non‑technical people to be willing to contribute via this mess? |
|
Thanks for reporting and sorry for the inconvenience. This looks suspicious indeed. It seems the server returned offset=0 regardless it had data chunk uploaded there. Could you check if it work without a proxy? I can't reproduce it locally: |
|
If either of you need a reference point (debug logs) to work with I have been using the tools to upload BlackVue videos 24/7 for 2 weeks now. Something like 870GBytes, 180,000 mp4's so far that seems to be working okay. I have not been watching the logs closely, but only have maybe 20 of the ConnectionError similar to above. No doubt that I am uploading mp4's rather than jpg's has a lesser chance of failure, Drop me a note if you would like any analysis or dump, |
|
More logs…
Sure, this may be proxy related. However, I highly doubt it. Too me, the server looks like the most probable culprit because the offset is computed by the server. Unfortunately, I cannot bypass the proxy. Anyhow, proxy or not it should work because I cannot imagine a scenario where the server should respond a wrong offset over a proxy or the client to modify the offset. It is the server that keeps track of the upload session state. Btw, I have never had any issues with other online services accepting large (interrupted) uploads. I am not sure how you could reproduce this. But, it seems like these recurring messages are usually causing trouble: |
|
@gitne I suspect it's related to HTTP status 412 error, curious to see the payload. I'm trying to add a check on retires: if the offset fetched from the server does not move as expected, exit with full HTTP response printed out. I will make a new alpha release and let you know soon. |
Great, thank you! 😄 Just FYI: I have observed this behavior ever since I have started using version 0.8.0. |
|
Just a quick check @gitne: were you uploading this sequence in multiple processes/machines simultaneously? That might cause the offset to be inconsistent AFICS. |
No, single instance on one machine only. |
|
The fix is released here https://github.com/mapillary/mapillary_tools/releases/tag/v0.9.5a1 BTW it would be great if you could setup a local env here https://github.com/mapillary/mapillary_tools#development so we can test on branches without releasing binaries (faster iteration). |
|
This is what I got on my first attempt with v0.9.5a1: Next, I am simply going to retry and see how far it will go.
I will see what I can do. My best bet is to build a Flatpak because it provides a defined set of dependencies. This is going to help get reproducible results. 🤞 |
|
After retry I got this: The offset is apparently still incorrect on retry. The last offset was 184,550,620 and on retry it was 18,152,520. 😕 That’s like a magnitude lower! 😲 |
|
It seems that you were uploading zipfiles from What looks really strange though, is that the embedded md5sum in the zip filename I can't reproduce this issue neither unfortunately: # generate zipfiles in mapillary_public_uploads
python3 -m mapillary_tools.commands --verbose process_and_upload ~/ImageTestData/myimages/ --dry_run
# upload these zipfiles
python3 -m mapillary_tools.commands --verbose upload_zip ./mapillary_public_uploads
# Here is one of the logs:
# as you can see the md5sum in the filename and the log match
2022-10-14 10:30:25,780 - DEBUG - Sending upload_end via IPC: {'total_sequence_count': 3, 'sequence_idx': 2, 'file_type': 'zip', 'import_path': 'mapillary_public_uploads/mly_tools_6bfe61d03070c89f5d6fccdced76f0c7.zip', 'sequence_image_count': 2, 'entity_size': 6746255, 'md5sum': '6bfe61d03070c89f5d6fccdced76f0c7', 'upload_start_time': 1665768624.169763, 'upload_total_time': 1.478506088256836, 'offset': 6746255, 'retries': 0, 'upload_last_restart_time': 1665768624.301506, 'upload_first_offset': 0, 'chunk_size': 0, 'upload_end_time': 1665768625.7800121}
@gitne would you mind share us one of these zipfile for debugging via |
|
A bit context of how the md5sum is calculated for zipfiles (all images will be zipped before being uploaded). Assume we are uploading N images as a sequence: Once we got the |
|
I think, I may have solved the mystery behind those resetting offsets. And, it definitely IS server related/caused. It turns out that our ISP uses dynamic public IP addresses. Whenever the ISP shifts to a new public IP address the upload server counts this as a new upload session. Naturally, our proxy sits on the local network and thus is also affected by the public IP address change. So, apparently the upload server’s upload session is tied to a specific client (public) IP address. Well, lets just say that this is a bit sub‑optimal. Creating and assigning an upload session token free of any IP address per upload session is probably going to fix this issue and other similar scenarios where the upload client has no control over its public IP address. Migrating to an upload session token will not pose a security threat either because not only is this a widely adopted and commonly used method for authenticating an upload session but also a very secure method because the token is (only) a session secret with a very limited lifetime. The only thing the server has to be weary of is creating too many tokens per IP address and per unit of time. Fortunately, because of the intrinsic nature of a lifetime limited token the upload server does not need to store or keep track of issued tokens. Thus, you can implement load balancing and flooding protection in one go. I have tested this hypothesis by rerouting network traffic over a static public IP address node and I did not have a single interruption over GBs of data. So after all, I guess my gut feeling was right. |
|
Very interesting observation. Thanks for sharing.
The upload session is decided by the authorized user and the session_key in the url. I don't think IP changes upload sessions. If you upload a single file from IP A, then interrupt it, and then resume from IP B, it will result in the single file uploaded in the same bucket, instead of two buckets. Changing IP during uploading will interrupt and close the underlying TCP connection, which result in the data chunk partially uploaded. In this case the HTTP client should raise a network error, and retry with the new offset fetched from the server. However, what we observed from the logs is that the HTTP didn't get interrupted and the server even returned success (2xx), then the client continued uploading a few more chunks based on the calculated (expected) offset, until the HTTP 412 error responded. Could you run the test cli with/without static public IP address, and see what happens (especially the HTTP responses)? # Generate 2GB file
dd if=/dev/zero of=TEST_BIG_FILE count=4024000
# Upload it
# NOTE: The test cli does upload `TEST_BIG_FILE` to the server, but they won't be shown on your profile, and it will be deleted after a few days.
python3 -m tests.cli.upload_api_v4 --verbose TEST_BIG_FILE TEST_SESSION --user_name=YOUR_ACCOUNT |
|
Sorry for the long delay. Unfortunately, I was 🤧 incapacitated for the last week (no COVID). Anyway, I have run the test you have suggested, as usual over our proxy, no rerouting: Well, and then it crashed. I am not sure this is intended? Anyway, I hope this output is going to help you figure out what you wanted to know. |
|
So, I have run the test with rerouting. Btw, whenever you see a traceback in the log it means that I have manually resumed uploading, just so that it gets finished. I have interrupted here because I have noticed that the offset has been reset to 0 again. This log is actually my second attempt. I hope this gives you a clue. 😉 |
|
Thanks @gitne! The logs you shared above didn't reproduce this HTTP 412 errors. However, I've reproduced it locally by switching on/off VPN during uploading. I think we should retry on these 412 errors instead of exiting. The reason why it took long for you is because:
I guess we should revert #571, and let you configure the retry waiting time. |
I don’t know, just adding some waiting time will not solve the core issue, which is that the offset is reset to 0 on a new IP address. All I know is that other upload services/servers do not exhibit this issue whenever I upload via a web browser. This issue did not exist with Mapillary when the web uploader was still available either. So, in my understanding there is something wrong with the server not accepting a live upload token from a different IP. Could it be a Cloudflare, Cloudfront, or the like configuration issue? My best guess is that there is some sort of network security software in front of the upload server that sends error codes on client closed connections. And, when the client tries to reestablish an existing TLS session with the same (public) session key from a different IP address the network security software says “Nah! Multiple IP addresses cannot share a TLS (public) session key” (which is not completely devoid of some security logic but imho this matter should be left to TLS to deal with that situation). Let me see if I can perhaps reproduce a HTTP 412 error code with that upload test script. |
|
@gitne the offset reset is likely that when Note these logs: Do you get a different While fixing the issue with the team, the best we can do is:
Note on 1: it does not solve the problem, which is uploading the same data multiple times but it makes uploading faster |
Right, I have not thought of that; then I guess the upload token is not shared either.
If the above is true then this is indeed very reasonable. There is no need to pause.
Could be, I have not payed attention to this. I guess mainly because I could not figure out what that cryptic |
Indeed, the offset resets on |
|
v0.9.5 still resets to 0 on |
|
Yeah, the client can't do much with this issue, it has to be fixed at the server side. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Basic information
0.9.5Linux, probablyanyanySteps to reproduce behavior
Ctrl+Cand then resuming (while the upload session is still alive on the server) with exactly the same command should have the same effect.'offset': 0.It is unclear whether this is a client or server bug. My gut feeling tells me that it is a server bug but I may be wrong. The server may change the chunk size during upload depending on load. The client should adapt to this. The server should also always respond with the correct offset, independent of the current chunk size.
Expected behavior
Per sequence uploads should resume on the offset of the first incomplete chunk.
Actual behavior
Note that
'offset': 0, while it should have been'offset': 15171468because this is the offset of the first incomplete chunk.All of the above means that it is currently extremely difficult to upload large sequences on low bandwidth connections because as soon as the load on the server changes (which is very likely on long uploads), the chunk size changes and the client has to restart uploading from offset 0. This is a huge waste of resources!
The text was updated successfully, but these errors were encountered: