Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Merge with already existing tiles #666

Open
cmahnke opened this issue Sep 21, 2023 · 2 comments
Open

[FEATURE] Merge with already existing tiles #666

cmahnke opened this issue Sep 21, 2023 · 2 comments

Comments

@cmahnke
Copy link
Contributor

cmahnke commented Sep 21, 2023

Is your feature request related to a problem? Please describe.
I'm currently looking into ways to build in a very constrained system, as a good measure, I want to be able to work with 4GB Ram and 30 GB disk max.
To achieve this goal I'm trying to implement an incremental build, for example for Germany. Its available either as a full dump, 3.9G or as 16 dumps, one for each state (See GeoFabrik page)

Describe the solution you'd like
I would like to see a functionality like tilemaker --merge. This way it should be possible to download each state separately, process and merge on the go, using a script to loop for example.
My expectation is that this would need less resources for lookup data structures etc for a single iteration, I don't mind if this would take considerably longer (5 times longer runtime would be acceptable, since in this scenario processing - CPU - is way cheaper than storage)

Describe alternatives you've considered
I've considered using tilemaker, but the memory management isn't very transparent, it's not even capable to give an estimation or hints how much memory operations might cost, it just fails. And using some sort of swap file don't really work either, it doesn't seem to use it efficiently.

Additional context
There has been significant ground work in #639, this added the --append flag, but also left out the mbtiles format for now.
The main issue here is certainly not, as a code comment suggests, the insertion, but merging of overlapping tiles.
Additionally it might be desirable to implement #536 to be able to merge with preexisting tiles in a directory structure.
And while we are at it, this can also be a possible way to implement #658.

@msbarry
Copy link
Contributor

msbarry commented Jan 7, 2024

Are you trying to build Germany? Or Europe? The least I've seen work for the planet in one go is 16gb although you might be able to get away with even less. All of Europe might work with as little as 4gb.

I don't think it would be too hard to support an mbtiles merge mode, just want to see if we can support your use case with a single pass of planetiler and no merging.

@cmahnke
Copy link
Contributor Author

cmahnke commented Jan 7, 2024

My goal is to generate parts of Europe (Germany alone is working) within 6 hours on a very constrained system. this system is just a runtime Frankenstein of a GitHub Runner (Free Tier)...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants