"Finetuning model" hangs

[noah-dbc]

Hello,

I've finetuned the NbAiLab/nb-roberta-base-scandi model with a corpus of article excerpts in Danish we have access to, and I wanted to use scandeval to see if it helped. So, I have written this little program:

$ cat article-model-scandeval.py
from transformers import AutoTokenizer
from transformers import AutoModelForMaskedLM
import datasets
from datasets import Dataset
import transformers
from scandeval import Benchmarker


print('loading model and tokenizer...')
model_path = '/data2/nots-model/my_other_mlm_model'
model = AutoModelForMaskedLM.from_pretrained(model_path)

benchmark = Benchmarker()
benchmark(model=model_path, language='da')

and when I run it, it gets stuck in the first finetuning (see below - it's been stuck for 10+ hours). Is there a way to get more information about what is going on?

Kind regards,

/Noah

$ python article-model-scandeval.py
loading model and tokenizer...
2024-03-22 08:29:25 ⋅ Flash attention has not been installed, so this will not be used. To install it, run `pip install -U wheel && FLASH_ATTENTION_SKIP_CUDA_BUILD=TRUE pip install flash-attn --no-build-isolation`. Alternatively, you can disable this message by setting the argument `use_flash_attention=False` in the `Benchmarker`.
2024-03-22 08:29:25 ⋅ Benchmarking /data2/nots-model/my_other_mlm_model on the truncated version of the Danish sentiment classification dataset AngryTweets
2024-03-22 08:29:28 ⋅ Loading model and tokenizer...
2024-03-22 08:29:29 ⋅ The model has 278,045,955 parameters, a vocabulary size of 250,002, and a maximum sequence length of 512.
Preprocessing data splits: 100%|████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:02<00:00,  4.39it/s]
Benchmarking:   0%|                                                                                                          | 0/10 [00:00<?, ?it/s]
Finetuning model: 0it [00:00, ?it/s]

[saattrupdan]

Hi! This looks like an old bug in ScandEval - are you working with the newest version of the package?

[noah-dbc]

Upgraded, and now I am getting another error. Just running with CLI here, the same happens with the program:

$ scandeval --model /data2/nots-model/abstracts_mlm_model --task sentiment-classification --language da
2024-03-22 13:13:20 ⋅ Flash attention has not been installed, so this will not be used. To install it, run `pip install -U wheel && FLASH_ATTENTION_SKIP_CUDA_BUILD=TRUE pip install flash-attn --no-build-isolation`. Alternatively, you can disable this message by setting the flag ``--no-use-flash-attention`.
2024-03-22 13:13:20 ⋅ Benchmarking /data2/nots-model/abstracts_mlm_model on the truncated version of the Danish sentiment classification dataset AngryTweets
Traceback (most recent call last):
  File "/home/nots/miniconda/envs/seval/bin/scandeval", line 8, in <module>
    sys.exit(benchmark())
             ^^^^^^^^^^^
  File "/home/nots/miniconda/envs/seval/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nots/miniconda/envs/seval/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/nots/miniconda/envs/seval/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nots/miniconda/envs/seval/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nots/ScandEval/src/scandeval/cli.py", line 282, in benchmark
    benchmarker(model=models)
  File "/home/nots/ScandEval/src/scandeval/benchmarker.py", line 693, in __call__
    return self.benchmark(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nots/ScandEval/src/scandeval/benchmarker.py", line 540, in benchmark
    record = self._benchmark_single(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nots/ScandEval/src/scandeval/benchmarker.py", line 644, in _benchmark_single
    dataset = self.dataset_factory.build_dataset(dataset_config)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nots/ScandEval/src/scandeval/dataset_factory.py", line 56, in build_dataset
    raise ValueError(f"Unknown dataset task: {dataset_config.task}")
ValueError: Unknown dataset task: Task(name='sentiment-classification', supertask='sequence-classification', metrics=[MetricConfig(name='mcc', pretty_name="Matthew's Correlation Coefficient", huggingface_id='matthews_correlation', results_key='matthews_correlation', compute_kwargs={}, postprocessing_fn=<function MetricConfig.<lambda>.<locals>.<lambda> at 0x7f16469e3920>), MetricConfig(name='macro_f1', pretty_name='Macro-average F1-score', huggingface_id='f1', results_key='f1', compute_kwargs={'average': 'macro'}, postprocessing_fn=<function MetricConfig.<lambda>.<locals>.<lambda> at 0x7f16469e39c0>)], labels=['negative', 'neutral', 'positive'])

[saattrupdan]

Thanks for that. Can you try downgrading to Python 3.10 and see if it persists?

[noah-dbc]

That seemed to do it, at least for that last bit. I'll try and backtrack from here. Thanks!

$ python --version
Python 3.10.0
(seval-310) [last: 0s]
(14:11:51) nots@ai-p301 ~/ScandEval (main)
$ pip list | grep -i scandeval
ScandEval                 12.3.2
(seval-310) [last: 0s]
(14:12:10) nots@ai-p301 ~/ScandEval (main)
$ python -c "from scandeval.utils import get_class_by_name; print(get_class_by_name(class_name='sequence-classification'))"
<class 'scandeval.sequence_classification.SequenceClassification'>

[saattrupdan]

Anytime 🙂

It is curious though. It would be nice to know for me if it was the Python version or if it was just the resetting of the virtual environment that did the trick. Could you try to re-do it for Python 3.11 again, from scratch, and see if the problem persists?

[noah-dbc]

Yeah, it does not persist. 🤯
Oh well. I'll try and continue with one of the envs that seem to work. Thanks again for the gentle push forward. 👍
Enjoy your weekend!

$ python -c "from scandeval.utils import get_class_by_name; print(get_class_by_name(class_name='sequence-classification'))"
<class 'scandeval.sequence_classification.SequenceClassification'>
(test-3118) [last: 8s]
(14:32:04) nots@ai-p301 ~/ScandEval (main)
$ pip list | grep -i scandeval
ScandEval                 12.3.2
(test-3118) [last: 1s]
(14:32:14) nots@ai-p301 ~/ScandEval (main)
$ python --version
Python 3.11.8
(test-3118) [last: 0s]

[saattrupdan]

Likewise! And hooray, good to know that 3.11 doesn't break the package 😅