Deprecation Warning for PyTorch Backend

[byteshiva]

Description

Running the command docker run --rm -it -p 3000:3000 ghcr.io/bentoml/openllm start facebook/opt-1.3b --backend pt results in a deprecation warning for the PyTorch backend.

Error

Error: PyTorch backend is deprecated and will be removed. Switch to vLLM.

Can you let me know how to fix it.

Resolution

• Update documentation to reflect deprecation.

Tasks

Beta Give feedback

1. Update documentation to reflect the deprecation of the PyTorch backend and provide guidance on transitioning to vLLM or another supported backend
2. Modify the Docker image to remove dependencies on the deprecated PyTorch backend
3. Test the modified Docker image to ensure it runs without any deprecation warnings.

[byteshiva]

found the fix

docker run --rm -it -p 3000:3000 ghcr.io/bentoml/openllm start facebook/opt-1.3b --backend vllm

Fix Available at:

• https://github.com/bentoml/OpenLLM?tab=readme-ov-file#-runtime-implementations
• ex. openllm start meta-llama/Llama-2-7b-chat-hf --backend vllm

[aarnphm]

just a headsup for 0.5 I'm removing pytorch backend, will work on CPU specific branch for better optimization.