A high-throughput and memory-efficient inference and serving engine for LLMs

The most flexible way to serve AI/ML models in production - Build Model Inference Service, LLM APIs, Inference Graph/Pipelines, Compound AI systems, Multi-Modal, RAG as a Service, and more!

In this repository, I will share some useful notes and references about deploying deep learning-based models in production.

FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, FEDML Nexus AI (https://fedml.ai) is your generative AI platform at scale.

Standardized Serverless ML Inference Platform on Kubernetes

🏕️ Reproducible development environment

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

AICI: Prompts as (Wasm) Programs

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

Hopsworks - Data-Intensive AI platform with a Feature Store

The simplest way to serve AI/ML models in production

Model Deployment at Scale on Kubernetes 🦄️

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

A scalable inference server for models optimized with OpenVINO™

Python + Inference - Model Deployment library in Python. Simplest model inference server ever.

Learn to serve Stable Diffusion models on cloud infrastructure at scale. This Lightning App shows load-balancing, orchestrating, pre-provisioning, dynamic batching, GPU-inference, micro-services working together via the Lightning Apps framework.

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

FastAPI Skeleton App to serve machine learning models production-ready.

OneDiffusion: Run any Stable Diffusion models and fine-tuned weights with ease