Skip to content
This repository has been archived by the owner on Nov 14, 2023. It is now read-only.

Resuming from checkpoint? #260

Open
jonathanmiller2 opened this issue Dec 28, 2022 · 0 comments
Open

Resuming from checkpoint? #260

jonathanmiller2 opened this issue Dec 28, 2022 · 0 comments

Comments

@jonathanmiller2
Copy link

There doesn't seem to be any documentation for how to resume the train from the local_dir checkpoints.
Using the following provided example:

"""Example using an sklearn Pipeline with TuneGridSearchCV.
Example taken and modified from
https://scikit-learn.org/stable/auto_examples/compose/
plot_compare_reduction.html
"""

from tune_sklearn import TuneSearchCV
from tune_sklearn import TuneGridSearchCV
from sklearn.datasets import load_digits
from sklearn.pipeline import Pipeline
from sklearn.svm import LinearSVC
from sklearn.decomposition import PCA, NMF
from sklearn.feature_selection import SelectKBest, chi2

pipe = Pipeline([
    # the reduce_dim stage is populated by the param_grid
    ("reduce_dim", "passthrough"),
    ("classify", LinearSVC(dual=False, max_iter=10000))
])

N_FEATURES_OPTIONS = [2, 4, 8]
C_OPTIONS = [1, 10]
param_grid = [
    {
        "reduce_dim": [PCA(iterated_power=7), NMF()],
        "reduce_dim__n_components": N_FEATURES_OPTIONS,
        "classify__C": C_OPTIONS
    },
    {
        "reduce_dim": [SelectKBest(chi2)],
        "reduce_dim__k": N_FEATURES_OPTIONS,
        "classify__C": C_OPTIONS
    },
]

random = TuneSearchCV(pipe, param_grid, search_optimization="random", local_dir='checkpoints')
X, y = load_digits(return_X_y=True)
random.fit(X[:100], y[:100])
print(random.cv_results_)

Trying to load the generated checkpoint using normal Ray methods such as:

from ray.tune import Tuner
Tuner.restore('checkpoints/_Trainable_2022-12-28_11-47-41')

Yields:

RuntimeError: Could not find Tuner state in restore directory. Did you passthe correct path (including experiment directory?) Got: checkpoints/_Trainable_2022-12-28_11-47-41

What is the intended way of loading checkpoints?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant