changing eval to include lm_harness args #74
Conversation
| print(f"Received job configuration:\n {config.model_dump_json(indent=2)}") | ||
|
|
||
| if config.tracking is not None: | ||
| with wandb_init_from_config( |
There was a problem hiding this comment.
We should still use our method wandb_init_from_config here, as it controls the naming convention for job types, and the resume argument.
If this is performed outside of the WandbLogger class, it will use the active run internally: https://github.com/EleutherAI/lm-evaluation-harness/blob/b177c82cb3ef4a8f03ff77341970ab9d77ffadb4/lm_eval/logging_utils.py#L99
However, we should remove the parameters=config.evaluator argument since logging the eval parameters is handled internally by the WandbLogger.
| @@ -82,7 +78,7 @@ def load_harness_model( | |||
| raise ValueError(f"Unexpected model config type: {type(config.model)}") | |||
|
|
|||
|
|
|||
| def load_and_evaluate( | |||
| def evaluate( | |||
There was a problem hiding this comment.
Can we revert the name here, or choose a different descriptive name? There is more than happening in this method than just evaluate.
Also, I think the results returned by this method are different than the results expected by wandb_logger.post_init. We may not need the get_numeric_metrics helper method above anymore?
There was a problem hiding this comment.
Edit: I see that you stopped calling get_numeric_metrics so the helper method above can be deleted
| @@ -4,6 +4,7 @@ | |||
| import torch | |||
| from lm_eval.models.huggingface import HFLM | |||
| from lm_eval.models.openai_completions import OpenaiCompletionsLM | |||
| from lm_eval.logging_utils import WandbLogger | |||
There was a problem hiding this comment.
Do we need a version bump on lm-eval package to include this? Should update the pyproject.toml file?
What's changing
How to test it
Related Jira Ticket
Additional notes for reviewers