Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split atmospheric products by variables and time #889

Draft
wants to merge 80 commits into
base: develop
Choose a base branch
from

Conversation

jlahovnik
Copy link
Collaborator

@jlahovnik jlahovnik commented Oct 17, 2023

  • adds the "split configuration" for wekeo, cop cds, cop ads and ecmwf providers
  • splitting is added for the cds and ecmwf api plugin and data request search plugin
  • products can be split by year or by month (currently one month configured for all providers)
  • splitting by additional parameters (e.g. leadtime hour) also possible
  • one asset per selected/available parameter is created
  • handling of constraints with either select boxes for year, month, day or date ranges available

introduces functions that will split a timespan into defined slices based on the configuration and given constraints
functions to create slices given by dates + some fixes
create one product per time slice and one asset per variable if config is done accordingly
fixes to make the download of ecmwf assets work + add date constraints support
fix format of day and month, do not split short timespans, check constraints before download
splitting of products by timespan and creating one asset per variable for cop_ads; also removes not existing request parameters
@github-actions
Copy link
Contributor

github-actions bot commented Oct 17, 2023

Test Results

       4 files  ±  0         4 suites  ±0   3m 33s ⏱️ ±0s
   447 tests +15     444 ✔️ +15    3 💤 ±0  0 ±0 
1 788 runs  +60  1 712 ✔️ +60  76 💤 ±0  0 ±0 

Results for commit e1a315f. ± Comparison against base commit a3f8296.

♻️ This comment has been updated with latest results.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 17, 2023

Code Coverage (Ubuntu)

File Coverage
All files 81%

Minimum allowed coverage is 70%

Generated by 🐒 cobertura-action against e1a315f

@github-actions
Copy link
Contributor

github-actions bot commented Oct 17, 2023

Code Coverage (Windows)

File Coverage
All files 76%

Minimum allowed coverage is 70%

Generated by 🐒 cobertura-action against e1a315f

@alambare-csgroup
Copy link
Collaborator

@jlahovnik can you rebase this PR and resolve the conflicts ?

@alambare-csgroup alambare-csgroup marked this pull request as draft November 17, 2023 13:58
introduces functions that will split a timespan into defined slices based on the configuration and given constraints
functions to create slices given by dates + some fixes
create one product per time slice and one asset per variable if config is done accordingly
@jlahovnik jlahovnik changed the title Configure product split params wekeo Split atmospheric products by variables and time Nov 21, 2023
@jlahovnik jlahovnik marked this pull request as ready for review November 21, 2023 14:30
@jlahovnik jlahovnik linked an issue Nov 21, 2023 that may be closed by this pull request
2 tasks
eodag/api/product/request_splitter.py Outdated Show resolved Hide resolved
eodag/rest/utils.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@sbrunato sbrunato left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a new cached_requests_get in eodag.utils and use it in request_splitter to speedup multiple requests to the same url

@jlahovnik jlahovnik marked this pull request as draft November 24, 2023 08:29
@sbrunato
Copy link
Collaborator

please add a new cached_requests_get in eodag.utils and use it in request_splitter to speedup multiple requests to the same url

done

Copy link
Collaborator

@sbrunato sbrunato left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please

  • rename product_data to product_type_conf
  • rename provider_product to provider_product_type
  • replace hard-coded "dataset" with a new provider setting: <provider>.search.metadata_mapping.providerProductType: $.dataset
  • rename constraints_param setting to constraints_entry
  • do not use any server_mode parameter, and directly use assets instead of downloadLinks in server mode
  • http://127.0.0.1:5000/catalogs/CAMS_GACF_MR/year/2021/items responds fast but http://127.0.0.1:5000/catalogs/CAMS_GACF_MR/items hangs forever
  • use HTTP_REQ_TIMEOUT for constraints request
  • in server mode, use {item}/download/{variable} pattern for assets href

@jlahovnik
Copy link
Collaborator Author

Please

* [x]  rename `product_data` to `product_type_conf`

* [x]  rename `provider_product` to `provider_product_type`

* [ ]  replace hard-coded `"dataset"` with a new provider setting: `<provider>.search.metadata_mapping.providerProductType: $.dataset`

* [x]  rename `constraints_param` setting to `constraints_entry`

* [ ]  do not use any `server_mode` parameter, and directly use `assets` instead of `downloadLinks` in server mode

* [ ]  http://127.0.0.1:5000/catalogs/CAMS_GACF_MR/year/2021/items responds fast but http://127.0.0.1:5000/catalogs/CAMS_GACF_MR/items hangs forever

* [x]  use `HTTP_REQ_TIMEOUT` for constraints request

* [ ]  in server mode, use `{item}/download/{variable}` pattern for assets `href`

renaming of variables done, timeout added
I don't understand the 3rd point. Where should that change be done?

@jlahovnik
Copy link
Collaborator Author

issue with http://127.0.0.1:5000/catalogs/CAMS_GACF_MR/items request is fixed

@sbrunato sbrunato added the pending Waiting for further information / action label Dec 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pending Waiting for further information / action
Projects
None yet
Development

Successfully merging this pull request may close these issues.

split atmostpheric products by variables and time
3 participants