Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Q: Standards for download #1171

Open
jklenzing opened this issue Jan 17, 2024 · 3 comments
Open

Q: Standards for download #1171

jklenzing opened this issue Jan 17, 2024 · 3 comments

Comments

@jklenzing
Copy link
Member

Currently, we use inst.download somewhat inconsistently across the ecosystem. The question is whether the end date should be taken to be inclusive or exclusive. For example, if I want to download all of Jan 1982, should I invoke

inst.download(dt.datetime(1982, 1, 1), dt.datetime(1982, 1, 31))

or

inst.download(dt.datetime(1982, 1, 1), dt.datetime(1982, 2, 1))

This should be clarified in the docs before the next release.

@jklenzing jklenzing added this to the 3.2.0 Release milestone Jan 17, 2024
@jklenzing
Copy link
Member Author

Pinging @aburrell, @rstoneback, @JonathonMSmith. Forgot to bring this up at the meeting, but we can discuss via the issues.

@aburrell
Copy link
Member

Thoughts:

Right now the download docstring says stop is inclusive. This differentiates it from end_date, which is exclusive.

However, it is "standard" in pysat that if you only specify a starting time, you'll end up with one day of data. The way that time is handled within download and the way that stop is defined there, it will usually give you two days. This is not what I would expect, and is not great for nowcast/forecast files.

I think the way forward might be:

  1. Keep stop inclusive to avoid confusion
  2. Change the way time is handled in Instrument.download such that Instrument.download(start=stime) will only download one day of data
  3. Add end that is exclusive for convenience?

@rstoneback
Copy link
Collaborator

It has been awhile but I believe that stop is by default set to tomorrow due to UTC/local time issues and nowcast type data. Yeah, something like pysat tests would pass early in the day but not later when UTC was tomorrow and locally we were still on today.

Adding an end_date input would enable the functionality you want without breaking existing programs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants