Skip to content

balavenkatesh3322/model_deployment

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 

Repository files navigation

Maintenance GitHub GitHub GitHub

Model Deployment

logo

What is Model serving?

When it comes to deploying ML models, data scientists have to make a choice based on their use case. If they need a high volume of predictions and latency is not an issue, they typically perform inference in batch, feeding the model with large amounts of data and writing the predictions into a table. If they need predictions at low latency, e.g. in response to a user action in an app, the best practice is to deploy ML models as REST endpoints. These apps allows to send requests to an endpoint that’s always up and receive the prediction immediately.

Pre-trained Models


Library Name Description
Tensorflow Serving High-performant framework to serve Tensorflow models via grpc protocol able to handle 100k requests per second per core
TorchServe TorchServe is a flexible and easy to use tool for serving PyTorch models.
BentoML BentoML is an open source framework for high performance ML model serving
Clipper Model server project from Berkeley's Rise Rise Lab which includes a standard RESTful API and supports TensorFlow, Scikit-learn and Caffe models
Cortex Cortex is an open source platform for deploying machine learning models—trained with nearly any framework—as production web services.
Multi-Model-server Multi Model Server (MMS) is a flexible and easy to use tool for serving deep learning models trained using any ML/DL framework.
DeepDetect Machine Learning production server for TensorFlow, XGBoost and Cafe models written in C++ and maintained by Jolibrain
Aml-Deploy GitHub Action for deploying Machine Learning Models to Azure
MLOps MLOps empowers data scientists and app developers to help bring ML models to production.
ForestFlow Cloud-native machine learning model server.
Jina Cloud native search framework that supports to use deep learning/state of the art AI models for search.
KFServing Serverless framework to deploy and monitor machine learning models in Kubernetes - (Video)
NVIDIA TensorRT Inference Server TensorRT Inference Server is an inference microservice that lets you serve deep learning models in production while maximizing GPU utilization.
OpenScoring REST web service for scoring PMML models built and maintained by OpenScoring.io
Redis-AI A Redis module for serving tensors and executing deep learning models. Expect changes in the API and internals.
Seldon Core Open source platform for deploying and monitoring machine learning models in kubernetes - (Video)
model_server OpenVINO™ Model Server is a scalable, high-performance solution for serving machine learning models optimized for Intel® architectures. The server provides an inference service via gRPC enpoint or REST API