New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The exporting of resulting OSM files can potentially be sped up #37
Comments
Benchmarking the existing methodFirst, I must admit my current PC is at mid-high tier, and so things might be faster than average. But the point should still stand even for slower computers. Also, extra care must be taken because the files to be processed can be very large. Some details:
I run the command for 5 times. Measured time (average): 19.455 seconds |
One thing that sticks out when doing some detailed profiling:
It is actually the to_xml part which is slow, not the IO. It seems we may continue with some sort of multi-threading. |
Hmmm. We are already using Spawning new threads does not work due to Python's GIL, which effectively encourages single-threaded code. Playing around with multiprocessing did not bring much immediate results because we will need to do extra work to pass values into the subprocesses. This might be viable in the long term, but not something that can be done in a single day. If we are able to somehow utilize multi-processing effectively, then perhaps there will be a significant speedup. |
This just dropped a few days ago: https://www.bitecode.dev/p/whats-up-python-the-gil-removed-a THe removal of GIL in Python can be very useful to this speed up: instead of spawning difficult-to-control subprocesses to parallelize XML-to-string, we may finally have a easy-to-control multi-threaded XML-to-string process to speed up exporting. |
This requires confirmation later, but I noticed on this StackOverflow discussion:
https://stackoverflow.com/questions/44560655/python-writelines-and-write-huge-time-difference
I have used ogr2osm for a while, and I notice that it can be quite slow on larger files. Like, unusually slow.
It seems the exporting can be sped up. Will investigate later.
The text was updated successfully, but these errors were encountered: