Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: reduce memory usage when applying geo operations on large input files #2

Open
4 of 9 tasks
theroggy opened this issue Feb 23, 2021 · 1 comment
Open
4 of 9 tasks
Labels
enhancement New feature or request

Comments

@theroggy
Copy link
Collaborator

theroggy commented Feb 23, 2021

Evade having out-of-memories, regardless of the size of the input files, eg. by:

  • limit batch size for geopandas based operations
  • give the user an option to specify batch size explicitly for cases where the heuristics don't work out
  • avoid using group_by clauses in queries, as this seems to use a lot of memory
    • erase/union/identity
    • export_by_location
    • clip
  • have more intelligent determination of the size of batches in general
  • check memory usage not only when starting an operation, but also eg. before the start of the processing of a new batch.
  • the sqlite cache size is now quite large, test if it can be reduced without impacting throughput (too much)
@theroggy theroggy added the enhancement New feature or request label May 20, 2021
@theroggy theroggy changed the title Improve resilience to decreasing memory availability during run Improve management of memory usage when applying geo operations on large input files Jul 19, 2021
@theroggy
Copy link
Collaborator Author

theroggy commented Feb 2, 2022

Memory usage for geopandas based operations was reduced here: 0ede76f

@theroggy theroggy changed the title Improve management of memory usage when applying geo operations on large input files reduce memory usage when applying geo operations on large input files Feb 29, 2024
@theroggy theroggy changed the title reduce memory usage when applying geo operations on large input files ENH: reduce memory usage when applying geo operations on large input files Feb 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant