Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Upstream encoder/decoder support based on multiple blocktables #161

Draft
wants to merge 233 commits into
base: main
Choose a base branch
from
Draft

[WIP] Upstream encoder/decoder support based on multiple blocktables #161

wants to merge 233 commits into from

Commits on Feb 22, 2024

  1. Update comment (vllm-project#2934)

    @ronensc
    ronensc committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    d7f3964 View commit details
    Browse the repository at this point in the history

  2. Added early stopping to completion APIs (vllm-project#2939)

    @Maxusmusti
    Maxusmusti committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    5574081 View commit details
    Browse the repository at this point in the history

  3. Migrate MistralForCausalLM to LlamaForCausalLM (vllm-project#2868)

    @esmeetu
    esmeetu committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    344020c View commit details
    Browse the repository at this point in the history

  4. Use Llama RMSNorm custom op for Gemma (vllm-project#2974)

    @WoosukKwon
    WoosukKwon committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    95529e3 View commit details
    Browse the repository at this point in the history

  5. chore(vllm): codespell for spell checking (vllm-project#2820)

    @mspronesti
    mspronesti committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    93dc5a2 View commit details
    Browse the repository at this point in the history

  6. Optimize GeGLU layer in Gemma (vllm-project#2975)

    @WoosukKwon
    WoosukKwon committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    fd5dcc5 View commit details
    Browse the repository at this point in the history

  7. [FIX] Fix a bug in initializing Yarn RoPE (vllm-project#2983)

    @44670
    44670 committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    c530e2c View commit details
    Browse the repository at this point in the history

  8. Remove Flash Attention in test env (vllm-project#2982)

    @WoosukKwon
    WoosukKwon committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    6f32cdd View commit details
    Browse the repository at this point in the history

  9. Include tokens from prompt phase in counter_generation_tokens (vllm… 

    …-project#2802)
    @ronensc
    ronensc committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    4caf704 View commit details
    Browse the repository at this point in the history

  10. Fix nvcc not found in vlm-openai image (vllm-project#2781)

    @zhaoyang-star
    zhaoyang-star committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    57f0449 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2024

  1. [Fix] Fissertion on YaRN model len (vllm-project#2984)

    @WoosukKwon
    WoosukKwon committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    f7c1234 View commit details
    Browse the repository at this point in the history

Commits on Feb 25, 2024

  1. Port metrics from aioprometheus to prometheus_client (vllm-projec… 

    …t#2730)
    @hmellor
    hmellor committed Feb 25, 2024
    Configuration menu
    Copy the full SHA
    ef978fe View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2024

  1. Add LogProbs for Chat Completions in OpenAI (vllm-project#2918)

    @jlcmoore
    jlcmoore committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    70f3e8e View commit details
    Browse the repository at this point in the history

  2. Optimize Triton MoE Kernel (vllm-project#2979) 

    Co-authored-by: Cade Daniel <edacih@gmail.com>
    @pcmoritz @cadedaniel
    pcmoritz and cadedaniel committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    cfc15a1 View commit details
    Browse the repository at this point in the history

  3. [Minor] Remove gather_cached_kv kernel (vllm-project#3043)

    @WoosukKwon
    WoosukKwon committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    d6e4a13 View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2024

  1. [Minor] Remove unused config files (vllm-project#3039)

    @esmeetu
    esmeetu committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    d9f726c View commit details
    Browse the repository at this point in the history

  2. Don't use cupy when enforce_eager=True (vllm-project#3037)

    @esmeetu
    esmeetu committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    c1c0d00 View commit details
    Browse the repository at this point in the history

  3. Fix stablelm (vllm-project#3038)

    @esmeetu
    esmeetu committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    4dd6416 View commit details
    Browse the repository at this point in the history

  4. Support Orion model (vllm-project#2539) 

    Co-authored-by: zhangdacheng <zhangdacheng@ainirobot.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
    @dachengai @WoosukKwon
    3 people committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    48a8f4a View commit details
    Browse the repository at this point in the history

  5. fix get_ip error in pure ipv6 environment (vllm-project#2931)

    @Jingru
    Jingru committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    2410e32 View commit details
    Browse the repository at this point in the history

  6. [Minor] Fix type annotation in fused moe (vllm-project#3045)

    @WoosukKwon
    WoosukKwon committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    4bd18ec View commit details
    Browse the repository at this point in the history

  7. Support logit bias for OpenAI API (vllm-project#3027)

    @dylanwhawk
    dylanwhawk committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    e0ade06 View commit details
    Browse the repository at this point in the history

  8. [Minor] Fix StableLMEpochForCausalLM -> StableLmForCausalLM (vllm-pro… 

    …ject#3046)
    @WoosukKwon
    WoosukKwon committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    8b430d7 View commit details
    Browse the repository at this point in the history

  9. Enable GQA support in the prefix prefill kernels (vllm-project#3007) 

    Signed-off-by: Tao He <sighingnow@gmail.com>
    @sighingnow
    sighingnow committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    71bcaf9 View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2024

  1. multi-lora documentation fix (vllm-project#3064)

    @ElefHead
    ElefHead committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    a868310 View commit details
    Browse the repository at this point in the history

  2. Restrict prometheus_client >= 0.18.0 to prevent errors when importing… 

    … pkgs (vllm-project#3070)
    @AllenDou
    AllenDou committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    e46fa5d View commit details
    Browse the repository at this point in the history

  3. [Neuron] Support inference with transformers-neuronx (vllm-project#2569)

    @liangfu
    liangfu committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    3b7178c View commit details
    Browse the repository at this point in the history

  4. Add LoRA support for Gemma (vllm-project#3050)

    @WoosukKwon
    WoosukKwon committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    929b4f2 View commit details
    Browse the repository at this point in the history

Commits on Feb 29, 2024

  1. t5-small

    @js8544
    Jin Shang authored and js8544 committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    dd82ba3 View commit details
    Browse the repository at this point in the history

  2. fix

    @js8544
    js8544 committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    f2fd579 View commit details
    Browse the repository at this point in the history

  3. Add Support for 2/3/8-bit GPTQ Quantization Models (vllm-project#2330)

    @chu-tianxiang
    chu-tianxiang committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    01a5d18 View commit details
    Browse the repository at this point in the history

  4. Fix: AttributeError in OpenAI-compatible server (vllm-project#3018)

    @jaywonchung
    jaywonchung committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    a6d471c View commit details
    Browse the repository at this point in the history

  5. add cache_config's info to prometheus metrics. (vllm-project#3100)

    @AllenDou
    AllenDou committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    9289e57 View commit details
    Browse the repository at this point in the history

  6. Support starcoder2 architecture (vllm-project#3089)

    @sh0416
    sh0416 committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    bfdcfa6 View commit details
    Browse the repository at this point in the history

  7. lint

    @js8544
    js8544 committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    2fb6905 View commit details
    Browse the repository at this point in the history

  8. Fix building from source on WSL (vllm-project#3112)

    @aliencaocao
    aliencaocao committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    2c08ff2 View commit details
    Browse the repository at this point in the history

  9. [Fix] Don't deep-copy LogitsProcessors when copying SamplingParams (v… 

    …llm-project#3099)
    @njhill
    njhill committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    29a8d6a View commit details
    Browse the repository at this point in the history

  10. Add guided decoding for OpenAI API server (vllm-project#2819) 

    Co-authored-by: br3no <breno@veltefaria.de>
Co-authored-by: simon-mo <simon.mo@hey.com>
    @felixzhu555 @br3no @simon-mo
    3 people committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    703e42e View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2024

  1. Fix: Output text is always truncated in some models (vllm-project#3016)

    @HyperdriveHustle
    HyperdriveHustle committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    54d3544 View commit details
    Browse the repository at this point in the history

  2. Remove exclude_unset in streaming response (vllm-project#3143)

    @sh0416
    sh0416 committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    27ca23d View commit details
    Browse the repository at this point in the history

  3. docs: Add tutorial on deploying vLLM model with KServe (vllm-project#… 

    …2586)

Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>
    @terrytangyuan
    terrytangyuan committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    49d849b View commit details
    Browse the repository at this point in the history

  4. fix relative import path of protocol.py (vllm-project#3134) 

    Co-authored-by: huohuarong <huohuarong@zuoshouyisheng.com>
    @Huarong
    Huarong and Huarong committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    90fbf12 View commit details
    Browse the repository at this point in the history

  5. T5 enc/dec example file; linting/formatting

    @afeldman-nm
    afeldman-nm committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    be58c3b View commit details
    Browse the repository at this point in the history

  6. Integrate Marlin Kernels for Int4 GPTQ inference (vllm-project#2497) 

    Co-authored-by: Robert Shaw <114415538+rib-2@users.noreply.github.com>
Co-authored-by: alexm <alexm@neuralmagic.com>
    @robertgshaw2-neuralmagic @alexm-neuralmagic
    3 people committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    c0c2335 View commit details
    Browse the repository at this point in the history

  7. Bump up to v0.3.3 (vllm-project#3129)

    @WoosukKwon
    WoosukKwon committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    82091b8 View commit details
    Browse the repository at this point in the history

  8. native/vllm t5 comparison test

    @afeldman-nm
    afeldman-nm committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    70837fd View commit details
    Browse the repository at this point in the history

  9. merged upstream-main into enc_dec_t5

    @afeldman-nm
    afeldman-nm committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    42a6e2b View commit details
    Browse the repository at this point in the history

  10. allow user chose log level by --log-level instead of fixed 'info'. (v… 

    …llm-project#3109)

Co-authored-by: zixiao <shunli.dsl@alibaba-inc.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
    @AllenDou @simon-mo
    3 people committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    29e70e3 View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2024

  1. Merge branch 'upstream-main' into enc_dec_t5

    @afeldman-nm
    afeldman-nm committed Mar 2, 2024
    Configuration menu
    Copy the full SHA
    e3fd30d View commit details
    Browse the repository at this point in the history

  2. Merge pull request #1 from afeldman-nm/enc_dec_t5 

    T5 enc/dec example file; linting/formatting
    @js8544
    js8544 committed Mar 2, 2024
    Configuration menu
    Copy the full SHA
    db726e6 View commit details
    Browse the repository at this point in the history

  3. remove debug print statements

    @afeldman-nm
    afeldman-nm committed Mar 2, 2024
    Configuration menu
    Copy the full SHA
    43e920e View commit details
    Browse the repository at this point in the history

  4. silence warning; legacy=False for tokenizer; lint/format

    @afeldman-nm
    afeldman-nm committed Mar 2, 2024
    Configuration menu
    Copy the full SHA
    431f014 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'js8544_enc_dec_t5' into enc_dec_t5

    @afeldman-nm
    afeldman-nm committed Mar 2, 2024
    Configuration menu
    Copy the full SHA
    37fcf99 View commit details
    Browse the repository at this point in the history

  6. Reorder kv dtype check to avoid nvcc not found error on AMD platform (v… 

    …llm-project#3104)
    @cloudhan
    cloudhan committed Mar 2, 2024
    Configuration menu
    Copy the full SHA
    baee28c View commit details
    Browse the repository at this point in the history

  7. Merge pull request #2 from afeldman-nm/enc_dec_t5 

    Small PR for debug print statements
    @js8544
    js8544 committed Mar 2, 2024
    Configuration menu
    Copy the full SHA
    4bf056b View commit details
    Browse the repository at this point in the history

  8. Add Automatic Prefix Caching (vllm-project#2762) 

    Co-authored-by: ElizaWszola <eliza@neuralmagic.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
    @SageMoore @ElizaWszola @mgoin
    3 people committed Mar 2, 2024
    Configuration menu
    Copy the full SHA
    ce4f5a2 View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2024

  1. Add vLLM version info to logs and openai API server (vllm-project#3161)

    @jasonacox
    jasonacox committed Mar 3, 2024
    Configuration menu
    Copy the full SHA
    d65fac2 View commit details
    Browse the repository at this point in the history

  2. [FIX] Fix styles in automatic prefix caching & add a automatic prefix… 

    … caching benchmark (vllm-project#3158)
    @zhuohan123
    zhuohan123 committed Mar 3, 2024
    Configuration menu
    Copy the full SHA
    996d095 View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2024

  1. Make it easy to profile workers with nsight (vllm-project#3162) 

    Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
    @pcmoritz @ywang96
    pcmoritz and ywang96 committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    17c3103 View commit details
    Browse the repository at this point in the history

  2. [DOC] add setup document to support neuron backend (vllm-project#2777)

    @liangfu
    liangfu committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    d0fae88 View commit details
    Browse the repository at this point in the history

  3. [Minor Fix] Remove unused code in benchmark_prefix_caching.py (vllm-p… 

    …roject#3171)
    @gty111
    gty111 committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    901cf4c View commit details
    Browse the repository at this point in the history

  4. Add document for vllm paged attention kernel. (vllm-project#2978)

    @pian13131
    pian13131 committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    27a7b07 View commit details
    Browse the repository at this point in the history

  5. enable --gpu-memory-utilization in benchmark_throughput.py (vllm-proj… 

    …ect#3175)

Co-authored-by: zixiao <shunli.dsl@alibaba-inc.com>
    @AllenDou
    AllenDou and zixiao committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    9cbc7e5 View commit details
    Browse the repository at this point in the history

  6. [Minor fix] The domain dns.google may cause a socket.gaierror excepti… 

    …on (vllm-project#3176)

Co-authored-by: guofangze <guofangze@kuaishou.com>
    @ttbachyinsda
    ttbachyinsda and guofangze committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    76e8a70 View commit details
    Browse the repository at this point in the history

  7. Push logprob generation to LLMEngine (vllm-project#3065) 

    Co-authored-by: Avnish Narayan <avnish@anyscale.com>
    @Yard1 @avnishn
    Yard1 and avnishn committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    22de452 View commit details
    Browse the repository at this point in the history

  8. Add health check, make async Engine more robust (vllm-project#3015) 

    Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
    @Yard1 @zhuohan123
    Yard1 and zhuohan123 committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    ff578ca View commit details
    Browse the repository at this point in the history

  9. Fix the openai benchmarking requests to work with latest OpenAI apis (v… 

    …llm-project#2992)

Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
    @wangchen615 @ywang96
    wangchen615 and ywang96 committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    9a4548b View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2024

  1. [ROCm] enable cupy in order to enable cudagraph mode for AMD GPUs (vl… 

    …lm-project#3123)

Co-authored-by: lcskrishna <lollachaitanya@gmail.com>
    @hongxiayang @lcskrishna
    hongxiayang and lcskrishna committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    05af6da View commit details
    Browse the repository at this point in the history

  2. fix _make_tensor_with_pad args change which broke decoder scenario

    @afeldman-nm
    afeldman-nm committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    8a5060f View commit details
    Browse the repository at this point in the history

  3. fixed bug caused by non-handling of self.model_config is None in mode… 

    …l_runner.py
    @afeldman-nm
    afeldman-nm committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    29d6f44 View commit details
    Browse the repository at this point in the history

  4. remove commented-out print statements

    @afeldman-nm
    afeldman-nm committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    a4950ba View commit details
    Browse the repository at this point in the history

  5. small cleanup

    @afeldman-nm
    afeldman-nm committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    9c03760 View commit details
    Browse the repository at this point in the history

  6. Store eos_token_id in Sequence for easy access (vllm-project#3166)

    @njhill
    njhill committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    8999ec3 View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2024

  1. [Fix] Avoid pickling entire LLMEngine for Ray workers (vllm-project#3207 

    )

Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
    @njhill @Yard1
    njhill and Yard1 committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    2efce05 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #3 from afeldman-nm/enc_dec_t5 

    fix _make_tensor_with_pad args change which broke decoder scenarios
    @js8544
    js8544 committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    9f20ccf View commit details
    Browse the repository at this point in the history

  3. [Tests] Add block manager and scheduler tests (vllm-project#3108)

    @rkooo567
    rkooo567 committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    24aecf4 View commit details
    Browse the repository at this point in the history

  4. [Testing] Fix core tests (vllm-project#3224)

    @cadedaniel
    cadedaniel committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    a33ce60 View commit details
    Browse the repository at this point in the history

  5. Add tqdm dynamic_ncols=True (vllm-project#3242)

    @chujiezheng
    chujiezheng committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    4cb3b92 View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2024

  1. Add GPTQ support for Gemma (vllm-project#3200)

    @TechxGenus
    TechxGenus committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    d3c04b6 View commit details
    Browse the repository at this point in the history

  2. Update requirements-dev.txt to include package for benchmarking scrip… 

    …ts. (vllm-project#3181)

Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
    @wangchen615 @zhuohan123
    wangchen615 and zhuohan123 committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    cbf4c05 View commit details
    Browse the repository at this point in the history

  3. Separate attention backends (vllm-project#3005)

    @WoosukKwon
    WoosukKwon committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    2daf23a View commit details
    Browse the repository at this point in the history

  4. Measure model memory usage (vllm-project#3120)

    @mgoin
    mgoin committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    385da2d View commit details
    Browse the repository at this point in the history

  5. arg naming fix

    @afeldman-nm
    afeldman-nm committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    6d6dccd View commit details
    Browse the repository at this point in the history

  6. Possible fix for conflict between Automated Prefix Caching (vllm-proj… 

    …ect#2762) and multi-LoRA support (vllm-project#1804) (vllm-project#3263)
    @jacobthebanana
    jacobthebanana committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    8cbba46 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2024

  1. Fix auto prefix bug (vllm-project#3239)

    @ElizaWszola
    ElizaWszola committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    b35cc93 View commit details
    Browse the repository at this point in the history

  2. Connect engine healthcheck to openai server (vllm-project#3260)

    @njhill
    njhill committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    d2339d6 View commit details
    Browse the repository at this point in the history

  3. Feature add lora support for Qwen2 (vllm-project#3177)

    @whyiug
    whyiug committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    c59e120 View commit details
    Browse the repository at this point in the history

  4. [Minor Fix] Fix comments in benchmark_serving (vllm-project#3252)

    @gty111
    gty111 committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    1ece1ae View commit details
    Browse the repository at this point in the history

  5. [Docs] Fix Unmocked Imports (vllm-project#3275)

    @ywang96
    ywang96 committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    99c3cfb View commit details
    Browse the repository at this point in the history

  6. [FIX] Make flash_attn optional (vllm-project#3269)

    @WoosukKwon
    WoosukKwon committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    1cb0cc2 View commit details
    Browse the repository at this point in the history

  7. Move model filelocks from /tmp/ to ~/.cache/vllm/locks/ dir (vllm… 

    …-project#3241)
    @mgoin
    mgoin committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    c2c5e09 View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2024

  1. [FIX] Fix prefix test error on main (vllm-project#3286)

    @zhuohan123
    zhuohan123 committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    f48c679 View commit details
    Browse the repository at this point in the history

  2. [Speculative decoding 3/9] Worker which speculates, scores, and appli… 

    …es rejection sampling (vllm-project#3103)
    @cadedaniel
    cadedaniel committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    8437bae View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2024

  1. Enhance lora tests with more layer and rank variations (vllm-project#… 

    …3243)
    @tterrysun
    tterrysun committed Mar 10, 2024
    Configuration menu
    Copy the full SHA
    0bba88d View commit details
    Browse the repository at this point in the history

  2. [ROCM] Fix blockReduceSum to use correct warp counts for ROCm and CUDA ( 

    vllm-project#3262)
    @dllehr-amd
    dllehr-amd committed Mar 10, 2024
    Configuration menu
    Copy the full SHA
    e4a28e5 View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2024

  1. [BugFix] Fix get tokenizer when using ray (vllm-project#3301)

    @esmeetu
    esmeetu committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    9e8744a View commit details
    Browse the repository at this point in the history

  2. [Fix] Fix best_of behavior when n=1 (vllm-project#3298)

    @njhill
    njhill committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    4b59f00 View commit details
    Browse the repository at this point in the history

  3. Re-enable the 80 char line width limit (vllm-project#3305)

    @zhuohan123
    zhuohan123 committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    2f8844b View commit details
    Browse the repository at this point in the history

  4. [docs] Add LoRA support information for models (vllm-project#3299)

    @pcmoritz
    pcmoritz committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    657061f View commit details
    Browse the repository at this point in the history

  5. Add distributed model executor abstraction (vllm-project#3191)

    @zhuohan123
    zhuohan123 committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    4c92270 View commit details
    Browse the repository at this point in the history

  6. [ROCm] Fix warp and lane calculation in blockReduceSum (vllm-project#… 

    …3321)
    @kliuae
    kliuae committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    c9415c1 View commit details
    Browse the repository at this point in the history

  7. Support Mistral Model Inference with transformers-neuronx (vllm-proje… 

    …ct#3153)
    @DAIZHENWEI
    DAIZHENWEI committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    654865e View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2024

  1. Merge branch 'js8544_enc_dec_t5' into enc_dec_t5

    @afeldman-nm
    afeldman-nm committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    7035178 View commit details
    Browse the repository at this point in the history

  2. fixed attention_kernels.cu merge conflict; questions about ROCM

    @afeldman-nm
    afeldman-nm committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    dbec357 View commit details
    Browse the repository at this point in the history

  3. docs: Add BentoML deployment doc (vllm-project#3336) 

    Signed-off-by: Sherlock113 <sherlockxu07@gmail.com>
    @Sherlock113
    Sherlock113 committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    b0925b3 View commit details
    Browse the repository at this point in the history

  4. llm_engine.py conflict resolution; removed prefix caching code; Seque… 

    …nce constructor call takes is_encoder_decoder, eos_token_id, lora_request calls; set is_encoder_decoder field in constructor
    @afeldman-nm
    afeldman-nm committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    4b2a121 View commit details
    Browse the repository at this point in the history

  5. actually updated Sequence constructor to take i_encoder_decoder, eos_… 

    …token_id, lora_request arguments
    @afeldman-nm
    afeldman-nm committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    a93c17d View commit details
    Browse the repository at this point in the history

  6. xformers.py accept incoming changes; replace paged_attention function… 

    … with import of PagedAttentionImpl
    @afeldman-nm
    afeldman-nm committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    a62c3af View commit details
    Browse the repository at this point in the history

  7. saved changed to xformers woops

    @afeldman-nm
    afeldman-nm committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    c31921f View commit details
    Browse the repository at this point in the history

  8. attempt at fixing model_runner conflicts related to encoder/decoder &… 

    … prefix caching; low confidence of success
    @afeldman-nm
    afeldman-nm committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    0c78be9 View commit details
    Browse the repository at this point in the history

  9. encoder/decoder + prefix caching not supported; moved check from llm.… 

    …py to model_runner.py
    @afeldman-nm
    afeldman-nm committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    e25e6b8 View commit details
    Browse the repository at this point in the history

  10. refactoring, including: moved enc_dec_attention.py into vllm/model_ex… 

    …ecutor/layers/attention
    @afeldman-nm
    afeldman-nm committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    7f70d76 View commit details
    Browse the repository at this point in the history

  11. existing regressions pass (yay) but encoder/decoder example fails

    @afeldman-nm
    afeldman-nm committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    36c8291 View commit details
    Browse the repository at this point in the history

  12. fixed encoder/decoder reshape and cache bug, but paged attention call… 

    …s are still incorrect
    @afeldman-nm
    afeldman-nm committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    08f268a View commit details
    Browse the repository at this point in the history

  13. augmented paged attention with context_lens, max_context_len, block_t… 

    …ables arguments to override input_metadata values; tests still pass but enc/dec still fails
    @afeldman-nm
    afeldman-nm committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    b9b0600 View commit details
    Browse the repository at this point in the history

  14. linting/formatting fixes

    @afeldman-nm
    afeldman-nm committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    63e9dca View commit details
    Browse the repository at this point in the history

  15. Merge branch 'upstream-main' into enc_dec_t5_merge_upstream2

    @afeldman-nm
    afeldman-nm committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    4d7e5a8 View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2024

  1. Fixes vllm-project#1556 double free (vllm-project#3347)

    @br3no
    br3no committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    49a3c86 View commit details
    Browse the repository at this point in the history

  2. Add kernel for GeGLU with approximate GELU (vllm-project#3337)

    @WoosukKwon
    WoosukKwon committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    602358f View commit details
    Browse the repository at this point in the history

  3. [Fix] Fix quantization="gptq" when using Marlin (vllm-project#3319) 

    Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
    @DreamTeamWangbowen @WoosukKwon
    DreamTeamWangbowen and WoosukKwon committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    b167109 View commit details
    Browse the repository at this point in the history

  4. add hf_transfer to requirements.txt (vllm-project#3031)

    @RonanKMcGovern
    RonanKMcGovern committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    e221910 View commit details
    Browse the repository at this point in the history

  5. [Minor] Fix bias in if to remove ambiguity (vllm-project#3259)

    @hliuca
    hliuca committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    ba8dc95 View commit details
    Browse the repository at this point in the history

  6. [Minor Fix] Use cupy-cuda11x in CUDA 11.8 build (vllm-project#3256)

    @chenxu2048
    chenxu2048 committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    739c350 View commit details
    Browse the repository at this point in the history

  7. Add missing kernel for CodeLlama-34B on A/H100 (no tensor parallelism… 

    …) when using Multi-LoRA. (vllm-project#3350)
    @orsharir
    orsharir committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    ae0ccb4 View commit details
    Browse the repository at this point in the history

  8. Add batched RoPE kernel (vllm-project#3095)

    @tterrysun
    tterrysun committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    7e9bd08 View commit details
    Browse the repository at this point in the history

  9. Fix lint (vllm-project#3388)

    @Yard1
    Yard1 committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    c33afd8 View commit details
    Browse the repository at this point in the history

  10. [FIX] Simpler fix for async engine running on ray (vllm-project#3371)

    @zhuohan123
    zhuohan123 committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    eeab52a View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2024

  1. [Hotfix] [Debug] test_openai_server.py::test_guided_regex_completion (v… 

    …llm-project#3383)
    @simon-mo
    simon-mo committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    81653d9 View commit details
    Browse the repository at this point in the history

  2. allow user to chose which vllm's merics to display in grafana (vllm-p… 

    …roject#3393)
    @AllenDou
    AllenDou committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    a37415c View commit details
    Browse the repository at this point in the history

  3. [Kernel] change benchmark script so that result can be directly used;… 

    … tune moe kernel in A100/H100 with tp=2,4,8 (vllm-project#3389)
    @youkaichao
    youkaichao committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    8fe8386 View commit details
    Browse the repository at this point in the history

  4. Install flash_attn in Docker image (vllm-project#3396)

    @tdoublep
    tdoublep committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    06ec486 View commit details
    Browse the repository at this point in the history

  5. Add args for mTLS support (vllm-project#3410) 

    Co-authored-by: Daniel Clark <daniel.clark@ibm.com>
    @declark1
    declark1 and Daniel Clark committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    c17ca8e View commit details
    Browse the repository at this point in the history

  6. [issue templates] add some issue templates (vllm-project#3412)

    @youkaichao
    youkaichao committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    dfc7740 View commit details
    Browse the repository at this point in the history

  7. Fix assertion failure in Qwen 1.5 with prefix caching enabled (vllm-p… 

    …roject#3373)

Co-authored-by: Cade Daniel <edacih@gmail.com>
    @chenxu2048 @cadedaniel
    chenxu2048 and cadedaniel committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    54be8a0 View commit details
    Browse the repository at this point in the history

  8. fix marlin config repr (vllm-project#3414)

    @qeternity
    qeternity committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    b983ba3 View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2024

  1. Dynamically configure shared memory size for moe_align_block_size_ker… 

    …nel (vllm-project#3376)
    @akhoroshev
    akhoroshev committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    78b6c48 View commit details
    Browse the repository at this point in the history

  2. [Misc] add HOST_IP env var (vllm-project#3419) 

    Co-authored-by: Simon Mo <simon.mo@hey.com>
    @youkaichao @simon-mo
    youkaichao and simon-mo committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    b522c44 View commit details
    Browse the repository at this point in the history

  3. Add chat templates for Falcon (vllm-project#3420)

    @Dinghow
    Dinghow committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    21539e6 View commit details
    Browse the repository at this point in the history

  4. Add chat templates for ChatGLM (vllm-project#3418)

    @Dinghow
    Dinghow committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    253a980 View commit details
    Browse the repository at this point in the history

  5. Fix dist.broadcast stall without group argument (vllm-project#3408)

    @GindaChen
    GindaChen committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    429284d View commit details
    Browse the repository at this point in the history

  6. Fix tie_word_embeddings for Qwen2. (vllm-project#3344)

    @fyabc
    fyabc committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    a7c8716 View commit details
    Browse the repository at this point in the history

  7. [Fix] Add args for mTLS support (vllm-project#3430) 

    Co-authored-by: declark1 <daniel.clark@ibm.com>
    @declark1
    declark1 and declark1 committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    03d37f2 View commit details
    Browse the repository at this point in the history

  8. Fixes the misuse/mixuse of time.time()/time.monotonic() (vllm-project… 

    …#3220)

Signed-off-by: Tao He <sighingnow@gmail.com>
Co-authored-by: simon-mo <simon.mo@hey.com>
    @sighingnow @simon-mo
    sighingnow and simon-mo committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    14b8ae0 View commit details
    Browse the repository at this point in the history

  9. [Misc] add error message in non linux platform (vllm-project#3438)

    @youkaichao
    youkaichao committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    604f235 View commit details
    Browse the repository at this point in the history

  10. Fix issue templates (vllm-project#3436)

    @hmellor
    hmellor committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    a7af453 View commit details
    Browse the repository at this point in the history

  11. fix document error for value and v_vec illustration (vllm-project#3421)

    @laneeeee
    laneeeee committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    8fa7357 View commit details
    Browse the repository at this point in the history

  12. Asynchronous tokenization (vllm-project#2879)

    @Yard1
    Yard1 committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    fb96c1e View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2024

  1. Removed Extraneous Print Message From OAI Server (vllm-project#3440)

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    10585e0 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'upstream-main' into enc_dec_t5_merge_upstream2

    @afeldman-nm
    afeldman-nm committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    bb7a219 View commit details
    Browse the repository at this point in the history

  3. [Misc] PR templates (vllm-project#3413) 

    Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
    @youkaichao @zhuohan123
    youkaichao and zhuohan123 committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    413366e View commit details
    Browse the repository at this point in the history

  4. fixed bug introduced during formatting

    @afeldman-nm
    afeldman-nm committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    0b60121 View commit details
    Browse the repository at this point in the history

  5. fixed example

    @afeldman-nm
    afeldman-nm committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    d44257e View commit details
    Browse the repository at this point in the history

  6. Merge branch 'enc_dec_t5' into enc_dec_t5_merge_upstream2

    @afeldman-nm
    afeldman-nm committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    19c5c4b View commit details
    Browse the repository at this point in the history

  7. Fixes the incorrect argument in the prefix-prefill test cases (vllm-p… 

    …roject#3246)
    @sighingnow
    sighingnow committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    3123f15 View commit details
    Browse the repository at this point in the history

  8. Replace lstrip() with removeprefix() to fix Ruff linter warning (v… 

    …llm-project#2958)
    @ronensc
    ronensc committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    14e3f9a View commit details
    Browse the repository at this point in the history

  9. Fix Baichuan chat template (vllm-project#3340)

    @Dinghow
    Dinghow committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    cf6ff18 View commit details
    Browse the repository at this point in the history

  10. fix lint

    @simon-mo
    simon-mo committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    ad50bf4 View commit details
    Browse the repository at this point in the history

  11. [Misc] fix line length for entire codebase (vllm-project#3444)

    @simon-mo
    simon-mo committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    8e67598 View commit details
    Browse the repository at this point in the history

  12. Support arbitrary json_object in OpenAI and Context Free Grammar (vll… 

    …m-project#3211)
    @simon-mo
    simon-mo committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    120157f View commit details
    Browse the repository at this point in the history

  13. Fix setup.py neuron-ls issue (vllm-project#2671)

    @simon-mo
    simon-mo committed Mar 16, 2024
    Configuration menu
    Copy the full SHA
    6b78837 View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2024

  1. [Misc] Use dataclass for InputMetadata (vllm-project#3452) 

    Co-authored-by: youkaichao <youkaichao@126.com>
    @WoosukKwon @youkaichao
    WoosukKwon and youkaichao committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    abfc4f3 View commit details
    Browse the repository at this point in the history

  2. [CI] Shard tests for LoRA and Kernels to speed up (vllm-project#3445)

    @simon-mo
    simon-mo committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    93348d9 View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2024

  1. [Bugfix] Make moe_align_block_size AMD-compatible (vllm-project#3470)

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    9101d83 View commit details
    Browse the repository at this point in the history

  2. CI: Add ROCm Docker Build (vllm-project#2886)

    @simon-mo
    simon-mo committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    8c654c0 View commit details
    Browse the repository at this point in the history

  3. [Testing] Add test_config.py to CI (vllm-project#3437)

    @cadedaniel
    cadedaniel committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    482b0ad View commit details
    Browse the repository at this point in the history

  4. [CI/Build] Fix Bad Import In Test (vllm-project#3473)

    @robertgshaw2-neuralmagic
    robertgshaw2-neuralmagic committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    097aa0e View commit details
    Browse the repository at this point in the history

  5. [Misc] Fix PR Template (vllm-project#3478)

    @zhuohan123
    zhuohan123 committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    c0c17d4 View commit details
    Browse the repository at this point in the history

  6. Cmake based build system (vllm-project#2830)

    @bnellnm
    bnellnm committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    9fdf3de View commit details
    Browse the repository at this point in the history

  7. [Core] Zero-copy asdict for InputMetadata (vllm-project#3475)

    @Yard1
    Yard1 committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    49eedea View commit details
    Browse the repository at this point in the history

  8. [Misc] Update README for the Third vLLM Meetup (vllm-project#3479)

    @zhuohan123
    zhuohan123 committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    b30880a View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2024

  1. [Core] Cache some utils (vllm-project#3474)

    @Yard1
    Yard1 committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    b37cdce View commit details
    Browse the repository at this point in the history

  2. [Core] print error before deadlock (vllm-project#3459)

    @youkaichao
    youkaichao committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    6a9c583 View commit details
    Browse the repository at this point in the history

  3. [Doc] Add docs about OpenAI compatible server (vllm-project#3288)

    @simon-mo
    simon-mo committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    ef65dcf View commit details
    Browse the repository at this point in the history

  4. [BugFix] Avoid initializing CUDA too early (vllm-project#3487)

    @njhill
    njhill committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    7341c77 View commit details
    Browse the repository at this point in the history

  5. Update dockerfile with ModelScope support (vllm-project#3429)

    @ifsheldon
    ifsheldon committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    c614cfe View commit details
    Browse the repository at this point in the history

  6. merged upstream

    @afeldman-nm
    afeldman-nm committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    c2f97b6 View commit details
    Browse the repository at this point in the history

  7. [Doc] minor fix to neuron-installation.rst (vllm-project#3505)

    @jimburtoft
    jimburtoft committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    2a60c9b View commit details
    Browse the repository at this point in the history

  8. Revert "[Core] Cache some utils" (vllm-project#3507)

    @simon-mo
    simon-mo committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    cc63d03 View commit details
    Browse the repository at this point in the history

  9. [Doc] minor fix of spelling in amd-installation.rst (vllm-project#3506)

    @jimburtoft
    jimburtoft committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    63e8b28 View commit details
    Browse the repository at this point in the history

  10. rolled back some encoder/decoder changes

    @afeldman-nm
    afeldman-nm committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    0536ff5 View commit details
    Browse the repository at this point in the history

  11. Use lru_cache for some environment detection utils (vllm-project#3508)

    @simon-mo
    simon-mo committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    20478c4 View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2024

  1. [PREFIX CACHING FOLLOW UP] A bunch of fixes to block allocator perfor… 

    …mance when automatic prefix caching is disabled (vllm-project#3357)

Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
    @ElizaWszola @zhuohan123
    ElizaWszola and zhuohan123 committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    9474e89 View commit details
    Browse the repository at this point in the history

  2. [Core] Add generic typing to LRUCache (vllm-project#3511)

    @njhill
    njhill committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    4ad521d View commit details
    Browse the repository at this point in the history

  3. [Misc] Remove cache stream and cache events (vllm-project#3461)

    @WoosukKwon
    WoosukKwon committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    5ee1449 View commit details
    Browse the repository at this point in the history

  4. Abort when nvcc command is not found in the PATH (vllm-project#3527)

    @AllenDou
    AllenDou committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    84eaa68 View commit details
    Browse the repository at this point in the history

  5. Check for _is_cuda() in compute_num_jobs (vllm-project#3481)

    @bnellnm
    bnellnm committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    ba8ae1d View commit details
    Browse the repository at this point in the history

  6. [Bugfix] Fix ROCm support in CMakeLists.txt (vllm-project#3534)

    @jamestwhedbee
    jamestwhedbee committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    80e2548 View commit details
    Browse the repository at this point in the history

  7. [1/n] Triton sampling kernel (vllm-project#3186) 

    Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
    @Yard1 @ywang96
    Yard1 and ywang96 committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    426ec4e View commit details
    Browse the repository at this point in the history

  8. [1/n][Chunked Prefill] Refactor input query shapes (vllm-project#3236)

    @rkooo567
    rkooo567 committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    6e435de View commit details
    Browse the repository at this point in the history

  9. Migrate logits computation and gather to model_runner (vllm-proje… 

    …ct#3233)
    @esmeetu
    esmeetu committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    f1c0fc3 View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2024

  1. [BugFix] Hot fix in setup.py for neuron build (vllm-project#3537)

    @zhuohan123
    zhuohan123 committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    523e30e View commit details
    Browse the repository at this point in the history

  2. [PREFIX CACHING FOLLOW UP] OrderedDict-based evictor (vllm-project#3431) 

    Co-authored-by: rsnm2 <rshaw@neuralmagic.com>
Co-authored-by: Luka <luka@paperspace>
    @ElizaWszola @robertgshaw2-neuralmagic
    3 people committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    6ebd02b View commit details
    Browse the repository at this point in the history

  3. Fix 1D query issue from _prune_hidden_states (vllm-project#3539)

    @rkooo567
    rkooo567 committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    3bbff9e View commit details
    Browse the repository at this point in the history

  4. [🚀 Ready to be merged] Added support for Jais models (vllm-project#3183)

    @grandiose-pizza
    grandiose-pizza committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    4c07dd2 View commit details
    Browse the repository at this point in the history

  5. [Misc][Log] Add log for tokenizer length not equal to vocabulary size (… 

    …vllm-project#3500)
    @esmeetu
    esmeetu committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    8657323 View commit details
    Browse the repository at this point in the history

  6. [Misc] Bump up transformers to v4.39.0 & Remove StarCoder2Config (vll… 

    …m-project#3551)

Co-authored-by: Roy <jasonailu87@gmail.com>
Co-authored-by: Roger Meier <r.meier@siemens.com>
    @WoosukKwon @esmeetu @bufferoverflow
    3 people committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    c188ecb View commit details
    Browse the repository at this point in the history

  7. [BugFix] gemma loading after quantization or LoRA. (vllm-project#3553)

    @taeminlee
    taeminlee committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    b7050ca View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2024

  1. [Bugfix][Model] Fix Qwen2 (vllm-project#3554)

    @esmeetu
    esmeetu committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    ea5f14e View commit details
    Browse the repository at this point in the history

  2. [Hardware][Neuron] Refactor neuron support (vllm-project#3471)

    @zhuohan123
    zhuohan123 committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    e90fc21 View commit details
    Browse the repository at this point in the history

  3. [BugFix] Some fixes for custom allreduce kernels (vllm-project#2760)

    @hanzhi713
    hanzhi713 committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    f721096 View commit details
    Browse the repository at this point in the history

  4. merged in upstream-main

    @afeldman-nm
    afeldman-nm committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    7d4972c View commit details
    Browse the repository at this point in the history

  5. added cross_block_tables to SequenceGroupMetadata

    @afeldman-nm
    afeldman-nm committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    23a5da5 View commit details
    Browse the repository at this point in the history

  6. SequenceGroupMetadata: added cross_seq_data; optional along with cros… 

    …s_block_tables
    @afeldman-nm
    afeldman-nm committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    e32fb9c View commit details
    Browse the repository at this point in the history

  7. added block manager allocation of cross sequence block_tables

    @afeldman-nm
    afeldman-nm committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    ae1c368 View commit details
    Browse the repository at this point in the history

  8. scheduler schedule() support cross block-tables and cross sequences, … 

    …etc.
    @afeldman-nm
    afeldman-nm committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    691c2c1 View commit details
    Browse the repository at this point in the history

  9. LLMEngine can build a sequencegroup with cross sequences

    @afeldman-nm
    afeldman-nm committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    e240eb4 View commit details
    Browse the repository at this point in the history

  10. t5 Sampler does not pass vocab size to constructor; input_metadata.pr… 

    …ompt_lens is treated as a list in T5
    @afeldman-nm
    afeldman-nm committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    cbfba8e View commit details
    Browse the repository at this point in the history

  11. add_request now correctly swaps decoder_prompt, prompt in encoder/dec… 

    …oder mode; removed encoder/decoder argument of Sequence
    @afeldman-nm
    afeldman-nm committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    501551c View commit details
    Browse the repository at this point in the history

  12. Added cross_input_metadata field to InputMetadata

    @afeldman-nm
    afeldman-nm committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    08435e4 View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2024

  1. wip multi blocktable

    @afeldman-nm
    afeldman-nm committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    6e459a2 View commit details
    Browse the repository at this point in the history

  2. wip

    @afeldman-nm
    afeldman-nm committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    8e1ca33 View commit details
    Browse the repository at this point in the history

  3. plumbing dummy input metadata structures into model

    @afeldman-nm
    afeldman-nm committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    e097732 View commit details
    Browse the repository at this point in the history

  4. plumbed encoder/decoder input metadata all the way into t5

    @afeldman-nm
    afeldman-nm committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    2a44585 View commit details
    Browse the repository at this point in the history

Commits on Mar 26, 2024

  1. first pass at T5 encoder support

    @afeldman-nm
    afeldman-nm committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    91a4608 View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2024

  1. inefficient but effective & Attention-wrapper-compatible implementati… 

    …on of relative position encoding based on packed-variable-length-sequences
    @afeldman-nm
    afeldman-nm committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    d0c5e36 View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2024

  1. wip cross-attention

    @afeldman-nm
    afeldman-nm committed Mar 28, 2024
    Configuration menu
    Copy the full SHA
    3737d5b View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2024

  1. first pass at enc/dec support that runs e2e but doesn't produce corre… 

    …ct T5 inference result. Nothing is broken by this commit, unless there is a subsequent commit with changes in order to pass regression tests.
    @afeldman-nm
    afeldman-nm committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    38946ed View commit details
    Browse the repository at this point in the history

  2. to pass regression tests: removed debug prints

    @afeldman-nm
    afeldman-nm committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    3c39f55 View commit details
    Browse the repository at this point in the history

  3. wip vllm, examples => fp32

    @afeldman-nm
    afeldman-nm committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    4ec2fde View commit details
    Browse the repository at this point in the history

  4. works on bsz = 1

    @afeldman-nm
    afeldman-nm committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    38f55ed View commit details
    Browse the repository at this point in the history

  5. intermediate activations for prompt_run look right! Decoded token loo… 

    …ks wrong though. Added not_causal option for attn_bias to kernel interface contracts; also switched to batch size 1 to avoid incorrectness likely caused by packed-variable-sequence-length mask having zeroes rather than -inf's
    @afeldman-nm
    afeldman-nm committed Apr 1, 2024
    Configuration menu
    Copy the full SHA
    1aedc80 View commit details
    Browse the repository at this point in the history

Commits on Apr 2, 2024

  1. wip

    @afeldman-nm
    afeldman-nm committed Apr 2, 2024
    Configuration menu
    Copy the full SHA
    c1258b4 View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2024

  1. passing with t5-small

    @afeldman-nm
    afeldman-nm committed Apr 3, 2024
    Configuration menu
    Copy the full SHA
    0af1022 View commit details
    Browse the repository at this point in the history

Commits on Apr 4, 2024

  1. vLLM T5 matches nativegit status! fixes: decode-phase cross-input-met… 

    …adata has correct blocktable, slot_mapping=None, and correct (max) context length(s) (derived from prompt); decode-phase decoder self-attention relative position encoding mask has 1 x K geometry where 1 is the number of new tokens generated in a step and K is context length padded to the nearest multiple of block size, and also mask is reshuffled with contiguous (); ensured general correctness of cross-attention input_metadata; modified T5 example script to prevent HF/vLLM T5 instances from being length limited; net effect: batch-size 1 seems to work but batch-size >1 not supported
    @afeldman-nm
    afeldman-nm committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    9e8d234 View commit details
    Browse the repository at this point in the history

  2. refactoring out print statements

    @afeldman-nm
    afeldman-nm committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    f5242a0 View commit details
    Browse the repository at this point in the history

  3. fix to pass regression tests

    @afeldman-nm
    afeldman-nm committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    de0fd31 View commit details
    Browse the repository at this point in the history

  4. WIP google/flan-t5-xxxx

    @afeldman-nm
    afeldman-nm committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    5a67647 View commit details
    Browse the repository at this point in the history

  5. removed print statement

    @afeldman-nm
    afeldman-nm committed Apr 4, 2024
    Configuration menu
    Copy the full SHA
    ed05d47 View commit details
    Browse the repository at this point in the history

Commits on Apr 10, 2024

  1. batched enc/dec example

    @afeldman-nm
    afeldman-nm committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    d5a8b92 View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2024

  1. wip, trying prompt padding

    @afeldman-nm
    afeldman-nm committed Apr 12, 2024
    Configuration menu
    Copy the full SHA
    f555f5d View commit details
    Browse the repository at this point in the history

Commits on Apr 17, 2024

  1. bs >1 prefill works

    @afeldman-nm
    afeldman-nm committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    2c12b44 View commit details
    Browse the repository at this point in the history

  2. small change to examples

    @afeldman-nm
    afeldman-nm committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    dba02b2 View commit details
    Browse the repository at this point in the history

  3. fix to support case where num prompts != 2

    @afeldman-nm
    afeldman-nm committed Apr 17, 2024
    Configuration menu
    Copy the full SHA
    db201b6 View commit details
    Browse the repository at this point in the history