New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug] REST server doesn't work on V100 (SM70) - cudaErrorNoKernelImageForDevice (but chat works) #2296
Comments
Looks like the problem is in MLCEngine - this is a minimal reproducer (using the latest nightlies): from mlc_llm import MLCEngine
model = "HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC"
engine = MLCEngine(model)
# Run chat completion in OpenAI API.
for response in engine.chat.completions.create(
messages=[{"role": "user", "content": "What is the meaning of life?"}],
model=model,
stream=True,
):
for choice in response.choices:
print(choice.delta.content, end="", flush=True)
print("\n")
engine.terminate() |
I did confirm the script works on an A4000 RunPod instance, so this is definitely a bug related to pre-SM80 GPUs. I'm happy to help fix (chat works and performs well so this is clearly possible...) if someone gives me some guidance on where to start. I also tried rebuilding mlc-llm from source (but using the prebuilt mlc-ai nightly wheel) and it didn't help. |
likely we need to build without flashinfer (as it does not support pre-SM80) @Hzfengsy have some followup replacement that might help |
Thanks. What exactly do I need to rebuild without flashinfer? I tried explicitly disabling flashinfer (and cutlass) during model lib compilation but it didn't help. |
For now, we might need to build mlc-llm from source by setting flashinfer OFF https://llm.mlc.ai/docs/install/mlc_llm.html#option-2-build-from-source |
I tried that and it didn't help, I can go back and double check my build settings to make sure though. I did use the prebuilt mlc-ai wheel, could that be a problem? (mlc-llm built from source, mlc-ai from prebuilt wheel) |
ah yes we also need build tvm from source, since the flashinfer was packaged through runtime there |
I see, thanks - I'll give it a try. Does it make sense to provide prebuilt wheels that are built without flashinfer? Seems like especially Pascal users could benefit (used P40 is a popular hobby GPU for LLMs) |
I think we could update engine to have a runtime compatibility check, so we can leverage flashinfer wheel. However, looking a bit more, because there are so many compute compact vector that increases binary size, so we have to cut down old compute compact in a single wheel. So likely a separate build is needed. Likely build from source is best course of action as of now. We can consider have runtime option to turn off flashinfer |
Success, rebuilt TVM from source following the instructions in the docs (I had to install libzstd-dev through apt) and now MLCEngine works. |
When FlashInfer is disabled, what prefill algorithm is used? I noticed a pretty long prompt processing time on Llama-70B and was wondering if it internally used memory-efficient attention (xformers/Pytorch SDPA) or the naive algorithm. |
We use a TensorIr variant of flashinfer which normally was at 80 to 90 percent of flashinfer efficiency. Note this is for decode, still need to confirm prefill |
OK. Does the REST server store stats like the chat interface does? Would be useful to check prefill tokens per second, etc. for benchmarking. |
The stats is still something WIP, but indeed that is a great suggestion |
Also interested in this for running on Tesla P40s. |
It's pretty easy to run the build yourself, just make sure you have the right version of LLVM when building TVM or else you'll get confusing errors. |
馃悰 Bug
Not sure if this is an issue with my compilation settings or with MLC-LLM; I have an 8x V100 16GB SXM2 system (HPE XL270D gen10) and can build a library and quantized weights that work fine with the
mlc_llm chat
command, but the same files fail when passed tomlc_llm serve
after the server receives a request on the completions endpoint:This is code which sends the request:
I tried a number of combinations of engine mode and max_tokens thinking it might be a kernel for a particular batch size that was missing, but it seemed to have no effect. Maybe the server is trying to run some variant of FlashAttention/FlashInfer and failing because there are no Flash kernels for SM70?
The text was updated successfully, but these errors were encountered: