Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement decorators to allow on-the-fly benchmarks #605

Open
svaiter opened this issue Jul 5, 2023 · 2 comments · May be fixed by #641
Open

Implement decorators to allow on-the-fly benchmarks #605

svaiter opened this issue Jul 5, 2023 · 2 comments · May be fixed by #641

Comments

@svaiter
Copy link

svaiter commented Jul 5, 2023

Idea would be to implement decorators @dataset @solver @objective to be able to generate a benchmark from a single .py

Typical use case: during numerical experiments dev, be able to compare few algorithms on simple dataset, without creating a full benchmark repo.

@mathurinm
Copy link
Contributor

Good idea, we could use temp_benchmark for that - and maybe decorators are not even needed. Can you provide an example of single .py file containing the syntax that you'd like to use ?

@svaiter
Copy link
Author

svaiter commented Jul 10, 2023

Something like that

import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import Lasso as Lasso_sklearn
from skglm import Lasso as Lasso_skglm
from benchopt import dataset, solver, objective, run_benchmark


@dataset(X=data.data, y=data.target)
def get_data():
    data = load_iris()
    return data.data, data.target


@solver(X=x, y=y, lmdb=lmdb, coef=lasso.coef_)
def solve_lasso_sklearn(X, y, lmbd):
    lasso = Lasso_sklearn(alpha=lmbd)
    lasso.fit(X, y)
    return lasso.coef_


@solver(X=x, y=y, lmdb=lmdb, coef=lasso.coef_)
def solve_lasso_skglm(X, y, lmbd):
    lasso = Lasso_skglm(alpha=lmbd)
    lasso.fit(X, y)
    return lasso.coef_


@objective(X=x, y=y, lmdb=lmdb, coef=coef)
def objective_lasso(X, y, lmbd, coef):
    return 0.5 * np.linalg.norm(y - X @ coef) ** 2 + lmbd * np.linalg.norm(coef, ord=1)


if __name__ == "__main__":
    run_benchmark(
        datasets=get_data,
        solvers=[solve_lasso_sklearn, solve_lasso_skglm],
        objectives=objective_lasso,
    )

@tomMoral tomMoral linked a pull request Aug 19, 2023 that will close this issue
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants