Pinned

1. AutoGPTQ/AutoGPTQ AutoGPTQ/AutoGPTQ Public

   An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

   Python 3.9k 396

2. vllm-project/vllm vllm-project/vllm Public

   A high-throughput and memory-efficient inference and serving engine for LLMs

   Python 20.1k 2.7k

3. sgl-project/sglang sgl-project/sglang Public

   SGLang is a structured generation language designed for large language models (LLMs). It makes your interaction with models faster and more controllable.

   Python 2.5k 160

4. Dao-AILab/flash-attention Dao-AILab/flash-attention Public

   Fast and memory-efficient exact attention

   Python 11.3k 994

5. flashinfer-ai/flashinfer flashinfer-ai/flashinfer Public

   FlashInfer: Kernel Library for LLM Serving

   Cuda 692 54

6. unslothai/unsloth unslothai/unsloth Public

   Finetune Llama 3, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory

   Python 10.6k 662