Discussions

• 1 You must be logged in to vote

  💬

  Monitoring Dashboard for vLLM

  jasonacox started May 9, 2024 in General

  

  

  

  

  3
• 1 You must be logged in to vote

  🙏

  [Bug]: 使用vllm 部署用着用着 模型突然没有响应了 nvidia-smi多了个execute method

  lxb0425 asked May 8, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💬

  what is the difference between litellm and vllm?

  YunchaoYang started Apr 26, 2024 in General

  

  

  2
• 3 You must be logged in to vote

  🙏

  Question regarding the nearly double GPU memory consumption.

  Zhuqln asked Jun 25, 2023 in Q&A · Answered

  

  

  

  

  8
• 1 You must be logged in to vote

  🙌

  LLM Controller Interface for vLLM (LLM output format controller)

  Ksuriuri started May 3, 2024 in Show and tell

  

  

  2
• 1 You must be logged in to vote

  🙏

  Can vllm serving clients by using multiple model instances?

  aoyulong asked Jun 21, 2023 in Q&A · Answered

  

  

  

  

  2
• 1 You must be logged in to vote

  🙏

  Difficulty understanding sequence length and context length

  NicolasDrapier asked May 6, 2024 in Q&A · Closed · Unanswered

  

  

  1
• 1 You must be logged in to vote

  💬

  Offline Batched Inference response issue

  NihinduCG started May 6, 2024 in General

  

  

  0
• 1 You must be logged in to vote

  🙏

  Printing from CUDA code

  gpu2024 asked May 6, 2024 in Q&A · Unanswered

  

  

  0
• 3 You must be logged in to vote

  🙏

  Is there any way that vllm works with huggingface pipeline?

  linkfqy asked Mar 31, 2024 in Q&A · Closed · Unanswered

  

  

  1
• 1 You must be logged in to vote

  💬

  RuntimeError with CUDA 12.2 on Windows using vLLM and Llava: No Kernel Image Available

  OualidBougzime started Apr 26, 2024 in General · Closed

  

  

  1
• 1 You must be logged in to vote

  🙏

  Do you support streaming generating outputs?

  question Further information is requested

  ltz0120 asked Jun 24, 2023 in Q&A · Answered

  

  

  

  

  5
• 1 You must be logged in to vote

  🙏

  Has anyone tried to use some modules from VLLM to replace torch.nn.functional.scaled_dot_product_attention?

  leprodeveloper asked May 2, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Getting an assertion error on llama3 70b

  tsvisab asked May 1, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Issue Running LLaVA with vLLM Due to Tensor Size Mismatch

  OualidBougzime asked Apr 28, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  How to specify which gpu to use?

  Zhuqln asked Jul 24, 2023 in Q&A · Answered

  

  

  

  

  8
• 17 You must be logged in to vote

  💡

  /v1/embeddings please

  feature request

  yuhai-china started Jun 21, 2023 in Ideas

  

  

  

  

  18
• 1 You must be logged in to vote

  🙏

  how to use auth token to use llama2 in vLLM? and question about presence_penalty

  orellavie1212 asked Aug 9, 2023 in Q&A · Closed · Unanswered

  

  

  3
• 2 You must be logged in to vote

  🙏

  Unable to load mistralai/Mistral-7B-Instruct-v0.2. repository not found

  gningue asked Apr 27, 2024 in Q&A · Unanswered

  

  

  

  

  4
• 2 You must be logged in to vote

  🙏

  Conversation roles must alternate user/assistant/user/assistant/...

  silvacarl2 asked Dec 14, 2023 in Q&A · Closed · Unanswered

  

  

  

  

  13
• 1 You must be logged in to vote

  🙏

  How to support deploying two small models on a single GPU card for inference?

  RanchiZhao asked Apr 28, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  How do I find / build the Neuron docker image?

  yaronr asked Apr 26, 2024 in Q&A · Unanswered

  

  

  0
• 6 You must be logged in to vote

  💡

  Support for apple/OpenELM-3B-Instruct

  bks5881 started Apr 25, 2024 in Ideas

  

  

  0
• 1 You must be logged in to vote

  🙏

  how does vllm handle wrong tokens in speculative decoding?

  Tomorrowdawn asked Apr 23, 2024 in Q&A · Answered

  

  

  6
• 1 You must be logged in to vote

  🙏

  Setting a cache directory when hosting a model with vllm

  shaily99 asked Apr 23, 2024 in Q&A · Closed · Unanswered

  

  

  1