A fast, easy-to-use, production-ready inference server for computer vision supporting deployment of many popular model architectures and fine-tuned models.

The simplest way to serve AI/ML models in production

An open-source computer vision framework to build and deploy apps in minutes

Python + Inference - Model Deployment library in Python. Simplest model inference server ever.

A REST API for Caffe using Docker and Go

This is a repository for an nocode object detection inference API using the Yolov3 and Yolov4 Darknet framework.

This is a repository for an nocode object detection inference API using the Yolov4 and Yolov3 Opencv.

This is a repository for an object detection inference API using the Tensorflow framework.

Orkhon: ML Inference Framework and Server Runtime

Serving AI/ML models in the open standard formats PMML and ONNX with both HTTP (REST API) and gRPC endpoints

K3ai is a lightweight, fully automated, AI infrastructure-in-a-box solution that allows anyone to experiment quickly with Kubeflow pipelines. K3ai is perfect for anything from Edge to laptops.

ONNX Runtime Server: The ONNX Runtime Server is a server that provides TCP and HTTP/HTTPS REST APIs for ONNX inference.

Deploy DL/ ML inference pipelines with minimal extra code.

A standalone inference server for trained Rubix ML estimators.

Wingman is the fastest and easiest way to run Llama models on your PC or Mac.

Friendli: the fastest serving engine for generative AI

Advanced inference pipeline using NVIDIA Triton Inference Server for CRAFT Text detection (Pytorch), included converter from Pytorch -> ONNX -> TensorRT, Inference pipelines (TensorRT, Triton server - multi-format). Supported model format for Triton inference: TensorRT engine, Torchscript, ONNX

Fullstack machine learning inference template

Benchmark for machine learning model online serving (LLM, embedding, Stable-Diffusion, Whisper)

Inference Server Implementation from Scratch for Machine Learning Models