Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alternative workflow for big data retrieval #384

Open
6 tasks
cristinamullin opened this issue Jan 9, 2024 · 1 comment
Open
6 tasks

Alternative workflow for big data retrieval #384

cristinamullin opened this issue Jan 9, 2024 · 1 comment

Comments

@cristinamullin
Copy link
Collaborator

cristinamullin commented Jan 9, 2024

Is your feature request related to a problem? Please describe.

Big data retrieval is challenging for WQP/TADA users. Consider including an example using a completely different alternative workflow for big data retrieval than what we have in TADA_BigDataRetrieval in DataDiscoveryRetrieval.R. Considering applying this different approach to an EPA use case in a new vignette: #362

Describe the solution you'd like

Matt Ross (Colorado State) suggested using targets.

USGS already created a targets vignette: https://waterdata.usgs.gov/blog/wqp-large-pull-targets/
Another USGS vignette: https://waterdata.usgs.gov/blog/large_sample_pull/
USGS target repo: https://github.com/DOI-USGS/ds-pipelines-targets-example-wqp

Describe alternatives you've considered

Still exploring...

Reminders for TADA contributors addressing this issue

New features should include all of the following work:

  • Create the function/code.

  • Document all code using comments to describe what is does.

  • Create tests in tests folder.

  • Create help file using roxygen2 above code.

  • Create working examples in help file (via roxygen2).

  • Add to appropriate vignette (or create new one).

@lkoenig-usgs
Copy link

Hello! 👋 I've periodically followed the progress on the TADA package and am really excited to see it come together. I created the USGS targets workflow with other USGS colleagues - if you have any questions or want to compare notes on our approach to big data pulls, please don't hesitate to reach out!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants