Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to track other metrics than solely the loss function. #629

Open
AMHermansen opened this issue Nov 14, 2023 · 1 comment
Open
Labels
feature New feature or request

Comments

@AMHermansen
Copy link
Collaborator

AMHermansen commented Nov 14, 2023

Is your feature request related to a problem? Please describe.
Currently we're the only value which is Logged to the progress bar and WandB is the loss value. However there are many situations in which the loss is either of little to no value, or you're more interested in another metric. Examples include when using a negative log-likelihood based loss function like (von-Mises Fisher, Gaussian, or CrossEntropy), where other quantities like RMSE, MAE, Accuracy, or AUC might be more relevant metrics to go by. Other examples could occur if a combination of multiple losses is used, then each component might be of interest.

Describe the solution you'd like
I would like to have the option to log additional more human-friendly quantities. There exists a library called torchmetrics which defines a plethora of metrics, which we probably can integrate with.
For implementation I think the easiest would be to have the option to add additional metrics to a Task, along with a name for the metric, and then the StandardModel, would process all metrics from all it's tasks and be responsible for feeding them the proper data and logging them during training/validation.

Describe alternatives you've considered
We could make our own implementation of a Metric like class, and use those, but I think it might be easier to use a library that already exists, especially since it is developed by lightning-ai and we probably won't run into issues with incompatible versions.

Additional context
Some of the metrics also define a plot method, which might be integrated in a way such that this would also solve #507.

@AMHermansen AMHermansen added the feature New feature or request label Nov 14, 2023
@RasmusOrsoe
Copy link
Collaborator

I think this would be a nice addition to the library, and I agree that tying metrics to Tasks is the right approach.

Just like we have default prediction names for tasks, we could add default metrics for each task that would automatically be logged by wandb. At run time, the users could overwrite these default values. That would be very elegant :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants