Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Callback Hooks for resample() / benchmark() #1013

Open
sebffischer opened this issue Apr 7, 2024 · 0 comments
Open

Callback Hooks for resample() / benchmark() #1013

sebffischer opened this issue Apr 7, 2024 · 0 comments

Comments

@sebffischer
Copy link
Sponsor Member

It would give users a lot of flexibility if we would provide some hooks to evaluate additional code during resample() and benchmark() (and maybe also batchmark(). One important use-case is to offer users a replacement for the holdout task, which we plan to remove soon. We have already made callbacks available in mlr3tuning, mlr3torch and bbotk etc..
Besides giving users the possibility to define their own callbacks, it would be great if we could also offer predefined callbacks for the most common things a user might want to do. Further, it would be great if we provided convenience functions that enable users to define custom callbacks without knowing too much about R6. mlr3tuning and mlr3torch also offer something like this.
Besides evaluating custom code, it should also be possible for these callbacks to store some information in the Result object, e.g. what is returned by a $get_state() method.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant