Whisper support

[gottlike]

Is support for Whisper on the roadmap? Something like https://github.com/ggerganov/whisper.cpp would be great.

[zhuohan123]

Supporting encoder-decoder models is in our roadmap as mentioned in #187. Feel free to join the discussion and potentially contribute!

[libratiger]

+1 for this feature

[silvacarl2]

+2 for this feature

[xtqxk]

+3 for this feature

[arun2728]

+4 for this feature

[SinanAkkoyun]

+555

[Swiffers]

+1

[hahazei]

+1

[binarycrayon]

monitoring

[afeldman-nm]

@zhuohan123 I am working on Whisper support.

[silvacarl2]

NO WAY!!!!!!!!!!!!!!!!!!! THAT WILL BE AWESOME!!!!!!!!!!!!!!!!!!!!!

[libratiger]

I am working on this PR, and will soon submit the draft.

[silvacarl2]

THIS IS GOING TO BE HUGE, THX!

[dbogunowicz]

Hey @libratiger, together with @afeldman-nm I am now working full-time on the same target. Would you like to sync? It would be more efficient to share knowledge, rather than develop the same thing in two silos.

[libratiger]

You're right. I've just discovered a discussion about T5 #187 (comment) , where there are differing opinions on the encoder-decoder model. Perhaps it will improve after that PR is merged?

[dbogunowicz]

@libratiger the current status is as follows: neural magic has finalized the original T5 PR, and we are now benchmarking the solution. In parallel, we are also developing support for Whisperer.

[JackZeng]

@dbogunowicz any update on this issue? looking forward

[dbogunowicz]

Hi! I am working on the Whisper on our team fork: neuralmagic#147
The status is: I am running the inference (both prompt prefill as well as autoregressive inference), but I get correctness issues, most likely caused by the erroneous attention mask implementation.

[junior-zsy]

@dbogunowicz I ran the feature/demian/Whisper branch to run the Whisper model and found an error message: vllm/worker/model_runner. py, line 477, in prepare_decode
Multi_modeal_input)
NameError: name 'multi_modal_input' is not defined, code execution cannot start

[dbogunowicz]

@junior-zsy fixed for now. Please remember, that we are still working on that PR, so it's pretty much in WiP state. Let me explicitly set the appropriate PR flag.

[junior-zsy]

@dbogunowicz Ok, thank you. Hope it can be used soon

[silvacarl2]

same here, this is going to be really cool!

[afeldman-nm]

@dbogunowicz thanks for your work on Whisper! Since there is clearly interest in this feature and its completion timeline, I want to add the context that Whisper support takes a dependency on encoder/decoder support -

Issue: #187
PR: #3117

which is also WIP (currently works partially but is not quite complete.) I expect to complete encoder/decoder support soon. JFYI for anyone interested in timelines.

[dwoodworth90]

+1

[afeldman-nm]

See the encoder/decoder support issue (#187) and new PR (#4289) for a status update on encoder/decoder support, which is a prereq for Whisper support.

[twicer-is-coder]

Hi, any update on serving faster-whisper via VLLM?

[afeldman-nm]

Hi, any update on serving faster-whisper via VLLM?

Hi @twicer-is-coder ,

Whisper (or any variant thereof) is high of the list of models to add once infrastructure support is in; you can see the roadmap for infrastructure support in this PR:

#4942