Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support to ClearML logging #10

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Add support to ClearML logging #10

wants to merge 5 commits into from

Conversation

anmarques
Copy link
Member

This PR adds support to logging lm-evaluation-harness results to ClearML.

The project and task names must both be provided as arguments in order to enable logging. If the project/task exists, the results will be appended to it, otherwise the corresponding project/task will be created from scratch.

The logging adds:

  • an artifact with the complete json output from the harness
  • one scalar value for each of the metrics under the "results" key of the json output, pre-appended by the evaluation task name

For instance, for gsm8k, the following artifact is added:

{
    "config": {
        "batch_size": "32",
        "batch_sizes": [],
        "bootstrap_iters": 100000,
        "description_dict": {},
        "device": "cuda:7",
        "limit": null,
        "model": "sparseml",
        "model_args": "pretrained=dense_finetuning/dense_LR1e-4_E4_GC5_extended,dtype=bfloat16",
        "no_cache": true,
        "num_fewshot": 0
    },
    "results": {
        "gsm8k": {
            "acc": 0.0841546626231994,
            "acc_stderr": 0.0076470240466032045
        }
    },
    "versions": {
        "gsm8k": 0
    }
}

and the following values are added the the "Summary" section of the Scalars tab:

gsm8k_acc: 0.0841546626231994
gsm8k_acc_stderr: 0.0076470240466032045

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant