Duplicate tokenizer entry in config.json auto_map section

[djliden]

❓ The question

The Hugging Face config.json files for the olmo models (and for the tokenizers) have the following repeated line under the auto_map AutoTokenizer key:

"AutoTokenizer": [
      "tokenization_olmo_fast.OLMoTokenizerFast",
      "tokenization_olmo_fast.OLMoTokenizerFast"
    ]

(e.g. link for olmo 7b config.json).

Couple of questions:

1. Is this intentional?
2. If not, is the config generated from somewhere in the GitHub codebase, or would the change need to be made on the Hugging Face side? (Happy to contribute if a change is needed)

This currently causes an AttributeError: 'list' object has no attribute 'split' error when trying to load the (peft) model with MLflow, which does not expect lists within auto_map. If it is expected that the AutoTokenizer entry is an array, I can pursue this from the MLflow side.

Thanks for looking!

[2015aroras]

We have very recently released transformers-integrated versions of the OLMo models on HF (e.g. https://huggingface.co/allenai/OLMo-1.7-7B-hf). It may be easier to try using these models, which will hopefully cause less issues.