Discussions

• 1 You must be logged in to vote

  🙏

  Clarification on JSON_OUTPUT field in text-generation-launcher command.

  ansSanthoshM asked Feb 22, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  How do I use my fine-tuned version of Llava with TGI?

  Iven2132 asked Apr 23, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  Error for exllamav2_kernel, running TGI on Google Colab

  andychoi98 asked Apr 18, 2024 in Q&A · Unanswered

  

  

  4
• 3 You must be logged in to vote

  💬

  Timeline Support for Volta Architecture(Tesla V100)

  koffisam started Mar 27, 2024 in General

  

  

  0
• 1 You must be logged in to vote

  💬

  Will TGI Integrate Quanto?

  PenutChen started Mar 20, 2024 in General

  

  

  0
• 3 You must be logged in to vote

  🙏

  OTLP Endpoint Setup

  devidw asked Nov 8, 2023 in Q&A · Unanswered

  

  

  1
• 3 You must be logged in to vote

  💬

  Do you have any plan to expand the scope to CPU?

  mengniwang95 started Oct 8, 2023 in General

  

  

  1
• 1 You must be logged in to vote

  🙏

  Helm Chart available

  kjoth asked Feb 28, 2024 in Q&A · Unanswered

  

  

  0
• 2 You must be logged in to vote

  💬

  Multi-GPU Inference

  daz-williams started Jan 23, 2024 in General

  

  

  

  

  3
• 2 You must be logged in to vote

  🙏

  Does HF TGI support Multi Node -Multi GPU server set up ?

  ansSanthoshM asked Feb 14, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  Securing Specific Routes with Token Authentication in TGI on Kubernetes

  ahadrez asked Feb 10, 2024 in Q&A · Closed · Answered

  

  

  1
• 1 You must be logged in to vote

  💬

  Reliance on HuggingFace / Outage

  daz-williams started Feb 10, 2024 in General · Closed

  

  

  0
• 2 You must be logged in to vote

  🙏

  How to get client IP that sends request to TGI server ?

  ansSanthoshM asked Feb 7, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💡

  Do we have any Grafana Dashboard example on top of TGI metrics ? Could metrics be explained in more detail ?

  luk-cer started Oct 10, 2023 in Ideas · Closed

  

  

  

  

  13
• 1 You must be logged in to vote

  🙏

  Problem with using text-generation-inference newer then version 1.1.1 on Openshift

  ptrmayer asked Feb 1, 2024 in Q&A · Answered

  

  

  1
• 1 You must be logged in to vote

  🙏

  How to run docker on a DPO model

  tamanna-mostafa asked Jan 25, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  💬

  Integrate HF TGI with TEI

  daz-williams started Jan 14, 2024 in General · Closed

  

  

  1
• 1 You must be logged in to vote

  🙏

  TGI : Llama2 : Call tokenizer from TGI server for token counting in given string

  ansSanthoshM asked Jan 11, 2024 in Q&A · Answered

  

  

  2
• 1 You must be logged in to vote

  🙏

  License clarification for company-internal usage

  sa- asked Jan 11, 2024 in Q&A · Answered

  

  

  1
• 1 You must be logged in to vote

  🙏

  TGI : Llama2 : Counting input and generated tokens and token per second

  ansSanthoshM asked Jan 10, 2024 in Q&A · Answered

  

  

  4
• 1 You must be logged in to vote

  🙏

  TGI number of maximum total token to be handled by Llama2. How to increase from 2048 to 4096 ?

  ansSanthoshM asked Jan 9, 2024 in Q&A · Closed · Answered

  

  

  1
• 1 You must be logged in to vote

  🙏

  Can this TGI approach with Llama2 weights downloaded directly from Meta in .pth format ?

  ansSanthoshM asked Jan 8, 2024 in Q&A · Closed · Answered

  

  

  5
• 1 You must be logged in to vote

  💡

  Improve decode latency

  scse-l started Jan 3, 2024 in Ideas

  

  

  0
• 3 You must be logged in to vote

  🙏

  Why is my batch size maxing out at 20 when my estimated max is around 60?

  ColtonBehannon asked Dec 11, 2023 in Q&A · Unanswered

  

  

  

  

  2
• 4 You must be logged in to vote

  🙏

  Is torch.compile compatible or already integrated with TGI?

  dmorgan-clairvoyant-ai asked Dec 18, 2023 in Q&A · Unanswered

  

  

  0