Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor overture stuff to be more generic #14

Open
cholmes opened this issue Aug 31, 2023 · 1 comment
Open

Refactor overture stuff to be more generic #14

cholmes opened this issue Aug 31, 2023 · 1 comment

Comments

@cholmes
Copy link
Collaborator

cholmes commented Aug 31, 2023

The main overture commands could likely be done fairly generic for any large geospatial file. It'd be great to evolve them to at least be 'tools', and perhaps even be their own package that 'open_buildings' would call / depend on. The overall flow of how the data is formatted is:

  • Add country_iso and quadkey columns to a directory of parquet files.
  • Create a duckdb database from all the files (this isn't actually a CLI / python script yet, as it's super easy - just create table from reading in the whole directory)
  • Write out individual parquet files based on country and iso, to the maximum size, with the appropriate rowgroup.

A more generic version of this would likely take input from more than parquet files (or at least have a command to convert to parquet files). And it would not be tied to the 'buildings' name.

@cholmes
Copy link
Collaborator Author

cholmes commented Oct 9, 2023

Note that the google versions of the overture scripts are likely 95% of the way there to being completely generic. They use the centroid of the geometry instead of relying on the bbox struct. The main thing to do is probably not use 'buildings' as the table name.

Making this completely generic likely warrants putting it in a new repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant