This repository has been archived by the owner on Jan 24, 2024. It is now read-only.
Strong need for multiple models
in a single deployment
#263
Labels
enhancement
New feature or request
As mentioned in #179, users need multiple models. On a multi-GPU on-prem machine, I want to write a config file that's like:
Then users should be able to specify
"model": "<either_model>",
in their requests.I can start a PR if you want this feature. Let me know if you have any suggestions on the best way to load these models and keep them mostly separate from each other.
The text was updated successfully, but these errors were encountered: