Upgrading to VLLM 0.4.1 - TypeError

[ProVega]

I recently upgraded to VLLM 0.4.1 and now get the following error. This looks internal to bento not my service (which is basically the default Llama 3 sample). here is my requirements.txt

accelerate==0.29.3 bentoml>=1.2.12 packaging==24.0 torch==2.2.1 transformers==4.40.0 vllm==0.4.1

2024-04-25T00:29:30-0600 [ERROR] [entry_service:bentovllm-llama3-8b-insruct-service:1] Initializing service error
Traceback (most recent call last):
File "/home/admin/.local/lib/python3.10/site-packages/_bentoml_sdk/service/factory.py", line 230, in call
instance = self.inner()
File "/home/admin/BentoVLLM/llama3-8b-instruct/bentovllm_openai/utils.py", line 77, in init
self.openai_serving_completion = OpenAIServingCompletion(
TypeError: OpenAIServingCompletion.init() got an unexpected keyword argument 'served_model'
2024-04-25T00:29:30-0600 [ERROR] [entry_service:bentovllm-llama3-8b-insruct-service:1] Traceback (most recent call last):
File "/home/admin/.local/lib/python3.10/site-packages/starlette/routing.py", line 732, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/usr/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/home/admin/.local/lib/python3.10/site-packages/bentoml/_internal/server/base_app.py", line 74, in lifespan
on_startup()
File "/home/admin/.local/lib/python3.10/site-packages/_bentoml_impl/server/app.py", line 313, in create_instance
self._service_instance = self.service()
File "/home/admin/.local/lib/python3.10/site-packages/_bentoml_sdk/service/factory.py", line 230, in call
instance = self.inner()
File "/home/admin/BentoVLLM/llama3-8b-instruct/bentovllm_openai/utils.py", line 77, in init
self.openai_serving_completion = OpenAIServingCompletion(
TypeError: OpenAIServingCompletion.init() got an unexpected keyword argument 'served_model'

2024-04-25T00:29:30-0600 [ERROR] [entry_service:bentovllm-llama3-8b-insruct-service:1] Application startup failed. Exiting.

[ProVega]

Looks like VLLM changed a parameter name. Here's my fix to get this project working with VLLM 0.4.1

--- a/llama3-8b-instruct/bentovllm_openai/utils.py
+++ b/llama3-8b-instruct/bentovllm_openai/utils.py
@@ -51,7 +51,7 @@ def openai_endpoints(
                         chat_template=None,
                     ):
                         super(OpenAIServingChat, self).__init__(
-                            engine=engine, served_model=served_model,
+                            engine=engine, served_model_names=served_model,
                             lora_modules=None,
                         )
                         self.response_role = response_role
@@ -75,7 +75,7 @@ def openai_endpoints(
                         return super()._load_chat_template(chat_template)

                 self.openai_serving_completion = OpenAIServingCompletion(
-                    engine=self.engine, served_model=served_model,
+                    engine=self.engine, served_model_names=served_model,
                 )

                 self.chat_template = chat_template
diff --git a/llama3-8b-instruct/requirements.txt b/llama3-8b-instruct/requirements.txt
index fce4600..194a74e 100644
--- a/llama3-8b-instruct/requirements.txt
+++ b/llama3-8b-instruct/requirements.txt
@@ -1,6 +1,6 @@
 accelerate==0.29.3
-bentoml>=1.2.11
+bentoml>=1.2.12
 packaging==24.0
-torch==2.1.2
-transformers==4.39.3
-vllm==0.4.0.post1
+torch==2.2.1
+transformers==4.40.0
+vllm==0.4.1
diff --git a/llama3-8b-instruct/service.py b/llama3-8b-instruct/service.py

[ProVega]

I'm not a git expert, but attempted to create a PR for you
#39

[larme]

@ProVega thanks for the finding! We need to update the same file in multiple folders. I will create another pr to fix this later.