TGI : Llama2 : Call tokenizer from TGI server for token counting in given string

[ansSanthoshM]

I have sucessfully setup TGI end point for Llama2 model.

Is there a way to call the tokenizer to count number of tokens in given string using TGI ?

Without using TGI, i used to count the number of token using below code, by directly pointing the directory of the Llama2 model weights.

from transformers import pipeline, AutoModelForCausalLM, LlamaTokenizer
tokenizer = LlamaTokenizer.from_pretrained(path to directory of the Llama2 model weights)
tokens = tokenizer.encode("This is test string")
num_tokens = len(tokens)
print("Num_tokens in input: " + str(num_tokens))

[ansSanthoshM]

Any thoughts on how to call tokenizer using HF TGI or lorax (LoRA eXchange) ?

[Narsil]

You can do it now. https://huggingface.github.io/text-generation-inference/#/Text%20Generation%20Inference/tokenize