Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add cache=True to ps.load_dataset #721

Open
raoulcollenteur opened this issue Apr 9, 2024 · 2 comments · May be fixed by #752
Open

Add cache=True to ps.load_dataset #721

raoulcollenteur opened this issue Apr 9, 2024 · 2 comments · May be fixed by #752
Assignees
@raoulcollenteur
Labels
enhancement Indicates improvement of existing features priority 1 normal, deal with in the foreseeable future
Milestone
1.6

Comments

@raoulcollenteur
Copy link
Member

A caching option should be added to the load-dataset method, because GH doesn't like it this way. GH Actions sometime fails now, so the tests should also be removed and only be executes when the master branch is updated.
@raoulcollenteur raoulcollenteur added the enhancement Indicates improvement of existing features label Apr 9, 2024
@raoulcollenteur raoulcollenteur self-assigned this Apr 9, 2024
@raoulcollenteur raoulcollenteur added this to the 1.5 milestone Apr 9, 2024
@martinvonk
Copy link
Collaborator

Can't we refer to the raw file ? Something like:

import pandas as pd

def pastas.load_datasset(name: Literal["collenteur_2021", "collenteur_2023", "vonk_2024"]) -> DataFrame:
   if name =collenteur_2021:
      urls = https://raw.githubusercontent.com/pastas/pastas-data/main/collenteur_2021/irregular_timesteps.csv
   elif ...
      ...
   return pd.read_csv(url, index_col=0, parse_dates=True)

@raoulcollenteur
Copy link
Member Author

No the problem is the download from GitHub, you can get the error: Exception: Error: 403. Reason: rate limit exceeded.

Which happened here: https://github.com/pastas/pastas/actions/runs/8541433512/job/23400748923

@martinvonk martinvonk added the priority 1 normal, deal with in the foreseeable future label Apr 12, 2024
@raoulcollenteur raoulcollenteur modified the milestones: 1.5, 1.6 Apr 17, 2024
@martinvonk martinvonk linked a pull request Apr 30, 2024 that will close this issue
Open
1 task
@martinvonk martinvonk linked a pull request Apr 30, 2024 that will close this issue
Caching load_dataset and adding read_csv_kwargs #752
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@raoulcollenteur raoulcollenteur

Labels
enhancement Indicates improvement of existing features priority 1 normal, deal with in the foreseeable future
Projects
None yet
Milestone
1.6
Development

Successfully merging a pull request may close this issue.

Caching load_dataset and adding read_csv_kwargs pastas/pastas
2 participants
@raoulcollenteur @martinvonk