How to call Hugging face TGI end point using liteLLM ?

[ansSanthoshM]

Hi,

I have been using Hugging face TGI end pointsas below. Now i am thinking to use use LiteLLM for load balancing between 3 TGI points.

Is there any example available to understand how to use TGI end points with LiteLLM for Generate call back of TGI server end points ?
Is there any example available to understand how to use TGI end points with LiteLLM for load balacning ?

from langchain.llms import HuggingFaceTextGenInference

llm= HuggingFaceTextGenInference(
	inference_server_url="http://50.23.21.235:8092",     
	temperature=0.4
	max_new_tokens=1000,
	streaming=True,
		)
response = llm(List 10 best use of LLMs?")

[ansSanthoshM]

i tried solution given in https://docs.litellm.ai/docs/providers/custom

import litellm
from litellm import completion 
import os
litellm.set_verbose=True
response = completion(
  model=  "custom/data/llama-2-70b-chat",
  messages=[{ "content": "Hello, how are you?","role": "user"}], 
  api_base="http://50.23.21.235:8092",
  temperature=0.4,
  max_tokens=1000,
  stream=True,
  )

print(response)

Getting error below

--------------------------------------------------------------------------

Request to litellm:
litellm.completion(model='custom/meta-llama/data/llama-2-70b-chat', messages=[{'content': 'Hello, how are you?', 'role': 'user'}], api_base='http://10.3.53.235:8082/', temperature=0.4, max_tokens=1000, stream=True)


self.optional_params: {}
kwargs[caching]: False; litellm.cache: None
UNMAPPED PROVIDER, ASSUMING IT'S OPENAI/AZUREs
Final returned optional params: {'temperature': 0.4, 'stream': True, 'max_tokens': 1000}
self.optional_params: {'temperature': 0.4, 'stream': True, 'max_tokens': 1000}

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Logging Details: logger_fn - None | callable(logger_fn) - False
Logging Details LiteLLM-Failure Call
self.failure_callback: []
JSONDecodeError                           Traceback (most recent call last)
File [d:\AIML\AIMLenv\lib\site-packages\requests\models.py:971](file:///D:/AIML/AIMLenv/lib/site-packages/requests/models.py:971), in Response.json(self, **kwargs)
    970 try:
--> 971     return complexjson.loads(self.text, **kwargs)
    972 except JSONDecodeError as e:
    973     # Catch JSON-related errors and raise as requests.JSONDecodeError
    974     # This aliases json.JSONDecodeError and simplejson.JSONDecodeError

File [d:\AIML\AIMLenv\lib\site-packages\simplejson\__init__.py:514](file:///D:/AIML/AIMLenv/lib/site-packages/simplejson/__init__.py:514), in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, use_decimal, allow_nan, **kw)
    510 if (cls is None and encoding is None and object_hook is None and
    511         parse_int is None and parse_float is None and
    512         parse_constant is None and object_pairs_hook is None
    513         and not use_decimal and not allow_nan and not kw):
--> 514     return _default_decoder.decode(s)
    515 if cls is None:

File [d:\AIML\AIMLenv\lib\site-packages\simplejson\decoder.py:386](file:///D:/AIML/AIMLenv/lib/site-packages/simplejson/decoder.py:386), in JSONDecoder.decode(self, s, _w, _PY3)
    385     s = str(s, self.encoding)
--> 386 obj, end = self.raw_decode(s)
    387 end = _w(s, end).end()

File [d:\AIML\AIMLenv\lib\site-packages\simplejson\decoder.py:416](file:///D:/AIML/AIMLenv/lib/site-packages/simplejson/decoder.py:416), in JSONDecoder.raw_decode(self, s, idx, _w, _PY3)
    415         idx += 3
--> 416 return self.scan_once(s, idx=_w(s, idx).end())
...
   (...)
   7643         ),  # stub the request
   7644     )

APIConnectionError: Expecting value: line 1 column 1 (char 0)
Output is truncated. View as a [scrollable element](command:cellOutput.enableScrolling?b1a148fb-af60-4c30-b747-5ee26096f859) or open in a [text editor](command:workbench.action.openLargeOutput?b1a148fb-af60-4c30-b747-5ee26096f859). Adjust cell output [settings](command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D)...

[ansSanthoshM]

I could make it work

## https://docs.litellm.ai/docs/providers/huggingface
import litellm
from litellm import completion 
import os
# litellm.set_verbose=True

'''
# e.g. Call 'llama-2-70b-chat' hosted on HF TGI end Point http://50.23.21.235:8092
You need to tell LiteLLM when you're calling Huggingface. 
This is done by adding the "huggingface/" prefix to model, example completion(model="huggingface/<model_name>",...).
'''
response = completion(
  model=  "huggingface/data/llama-2-70b-chat",
  messages=[{ "content":"What is capital of Egpty and list 3 attractions there?","role": "user"}], 
  api_base="http://50.23.21.235:8092",
  temperature=0.4,
  max_tokens=1000,
  # stream=True,
  )

print(response)
print(f"prompt_tokens : {response['usage']['prompt_tokens']}")
print(f"completion_tokens : {response['usage']['completion_tokens']}")
print(f"total_tokens : {response['usage']['total_tokens']}")
print(f"response time (sec) : {response['_response_ms']/1000}")
print(f"LLM Response : {response.choices[0]['message']['content']}")

Here is the output ;

ModelResponse(id='chatcmpl-d7f4b341-4c62-4792-997d-9045d8e2b805', choices=[Choices(finish_reason='eos_token', index=0, message=Message(content="\nThe capital of Egypt is Cairo. Here are three attractions in Cairo:\n\n1. The Pyramids of Giza: Located on the outskirts of Cairo, the Pyramids of Giza are considered to be one of the greatest architectural and engineering achievements of all time. The largest pyramid, the Great Pyramid of Khufu, is an absolute must-see when visiting Cairo.\n2. The Egyptian Museum: The Egyptian Museum in Cairo is home to the world's largest collection of ancient Egyptian artifacts, including mummies, sarcophagi, and other treasures. The museum's collection spans over 5,000 years of Egyptian history, making it a fascinating place to visit for anyone interested in ancient Egypt.\n3. Islamic Cairo: Islamic Cairo is a historic neighborhood in Cairo that is home to many important Islamic monuments, including mosques, madrasas, and mausoleums. The area is a great place to explore on foot, as the narrow streets are filled with vendors, cafes, and local shops. Some of the most notable attractions in Islamic Cairo include the Mosque of Muhammad Ali, the Al-Azhar Mosque, and the Khan el-Khalili market.", role='assistant', _logprob=-9.665692911283433))], created=1709040078, model='data/llama-2-70b-chat', object='chat.completion', system_fingerprint=None, usage=Usage(prompt_tokens=13, completion_tokens=233, total_tokens=246), _response_ms=18444.179)
prompt_tokens : 13
completion_tokens : 233
total_tokens : 246
response time (sec) : 18.444179
LLM Response : 
The capital of Egypt is Cairo. Here are three attractions in Cairo:

1. The Pyramids of Giza: Located on the outskirts of Cairo, the Pyramids of Giza are considered to be one of the greatest architectural and engineering achievements of all time. The largest pyramid, the Great Pyramid of Khufu, is an absolute must-see when visiting Cairo.
2. The Egyptian Museum: The Egyptian Museum in Cairo is home to the world's largest collection of ancient Egyptian artifacts, including mummies, sarcophagi, and other treasures. The museum's collection spans over 5,000 years of Egyptian history, making it a fascinating place to visit for anyone interested in ancient Egypt.
3. Islamic Cairo: Islamic Cairo is a historic neighborhood in Cairo that is home to many important Islamic monuments, including mosques, madrasas, and mausoleums. The area is a great place to explore on foot, as the narrow streets are filled with vendors, cafes, and local shops. Some of the most notable attractions in Islamic Cairo include the Mosque of Muhammad Ali, the Al-Azhar Mosque, and the Khan el-Khalili market.

[krrishdholakia]

Use the Router for load balancing - https://docs.litellm.ai/docs/routing