[BUG] BOS token on textgen API? Does it show up for you?

[Ph0rk0z]

Environment

🐧 Linux

System

Ubuntu 22.04

Version

Git Staging

Desktop Information

No response

Describe the problem

Since all of the llama.cpp drama about the BOS token placement, I decided to check what's happening in my own gens.

I fired up textgen in verbose mode. Both exllamv2 loaders, HF and regular.

In the webui, if I tick "Add BOS token", it shows up in the prompt printed in the console. When I uncheck it, BOS token disappears as expected. At least while using chat-instruct.

Unfortunately, when using SillyTavern, every request I send has no BOS token in the ooba log. It doesn't matter if I check "add BOS" or not, the output is identical and only displays what I sent. I also see confirmation of my ST settings in the webui verbose output but I do not see add_bos_token. One thing I noticed is that I have both max_tokens and max_new_tokens, could be wrong but one of those is a chat completion only parameter.

Sooo.. if this wasn't TLDR, is this a bug on my particular system or is this happening for anyone else? Do you see add_bos_token:True in your verbose output and do you see the BOS token in the prompt?

Additional info

Please tick the boxes

• I have explained the issue clearly, and I included all relevant info
• I have checked that this issue hasn't already been raised
• I have checked the docs