qlora taining on qwen1.5-15b-chat

[wsp317]

训练qwen1.5-14b-chat，遇到下面的报错，transformers==4.38.2

RuntimeError(
"Unsloth: Tokenizer's pad_token cannot be = eos_token, and we couldn't find a\n"
"replacement of either <|reserved... or <|placeholder..."
)

[danielhanchen]

Oh that is an issue - the pad_token must be not the same as the eos_token, otherwise the finetune will be incorrect. I'll see if I can extend the tokenizer itself

[wsp317]

I change the pad_token from <|endoftext|> to <|im_end|> in qwen's tokenizer_config.json file, and the training seems work.

[danielhanchen]

@wsp317 I fixed it just then! Sorry on the delay! I

If you're on a local machine, please update Unsloth via

pip uninstall unsloth -y
pip install --upgrade --force-reinstall --no-cache-dir git+https://github.com/unslothai/unsloth.git

Colab and Kaggle is fine (just restart it)