#

llmops

Here are 267 public repositories matching this topic...

Scale3-Labs / langtrace

Langtrace 🔍 is an open-source, Open Telemetry based end-to-end observability tool for LLM applications, providing real-time tracing, evaluations and metrics for popular LLMs, LLM frameworks, vectorDBs and more.. Integrate using Typescript, Python. 🚀💻📊

open-source ai evaluations tracing openai gpt datasets observability open-telemetry llm prompt-engineering langchain llmops llm-framework

Updated May 13, 2024
TypeScript

BerriAI / litellm

Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs)

openai llm langchain llmops anthropic langchain-python

Updated May 13, 2024
Python

parea-ai / parea-sdk-py

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

metrics good-first-issue llm prompt-engineering generative-ai llmops llm-eval llm-tools llm-evaluation llm-evaluation-toolkit llms-benchmarking llm-evaluation-framework

Updated May 13, 2024
Python

phidatahq / phidata

Memory, knowledge and tools for LLMs

python aws ai developer-tools gpt-4 llm llmops

Updated May 13, 2024
Python

kolena

kolenaIO / kolena

Python client for Kolena's machine learning testing platform

testing machine-learning evaluation evaluation-metrics evaluation-framework mlops evaluate-models llmops

Updated May 13, 2024
Python

traceloop / openllmetry

Open-source observability for your LLM application, based on OpenTelemetry

python open-source monitoring metrics ml datascience help-wanted observability good-first-issue artifical-intelligence model-monitoring opentelemetry open-telemetry opentelemetry-python llm good-first-issues generative-ai llmops

Updated May 13, 2024
Python

phoenix

Arize-ai / phoenix

AI Observability & Evaluation

clustering umap mlops model-monitoring ml-monitoring ai-monitoring ml-observability ai-observability ai-roi model-observability llmops llm-eval

Updated May 13, 2024
Jupyter Notebook

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

amd cuda inference pytorch transformer llama gpt rocm model-serving mlops llm inferentia llmops llm-serving trainium

Updated May 13, 2024
Python

promptfoo / promptfoo

Test your prompts, models, and RAGs. Catch regressions and improve prompt quality. LLM evals for OpenAI, Azure, Anthropic, Gemini, Mistral, Llama, Bedrock, Ollama, and other local & private models with CI/CD integration.

testing ci evaluation ci-cd cicd prompts evaluation-framework rag llm prompt-engineering llmops prompt-testing llm-eval llm-evaluation llm-evaluation-framework

Updated May 13, 2024
TypeScript

microsoft / autogen

A programming framework for agentic AI. Discord: https://aka.ms/autogen-dc. Roadmap: https://aka.ms/autogen-roadmap

chat chatbot gpt chat-application agent-based-framework agent-oriented-programming gpt-4 chatgpt llmops gpt-35-turbo llm-agent llm-inference agentic llm-framework agentic-agi

Updated May 13, 2024
Jupyter Notebook

microsoft / aici

AICI: Prompts as (Wasm) Programs

rust ai wasm inference transformer language-model model-serving wasmtime llm llmops llm-serving llm-inference llm-framework

Updated May 13, 2024
Rust

Opla / opla

Empower Your Productivity with Local AI Assistants

llama gpt aiassistant opla ai-assistant llm generative-ai llmops llamacpp localai llm-inference local-ai llama2 aiagent ai-agent-front

Updated May 13, 2024
TypeScript

bentoml / OpenLLM

Run any open-source LLMs, such as Llama 2, Mistral, as OpenAI compatible API endpoint in the cloud.

Updated May 13, 2024
Python

DAGWorks-Inc / hamilton

Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.

Updated May 13, 2024
Jupyter Notebook

agenta

Agenta-AI / agenta

The all-in-one LLM developer platform: prompt management, evaluation, human feedback, and deployment all in one place.

prompt-toolkit rag human-annotation large-language-models llm prompt-engineering llms langchain llmops llama-index prompt-management llm-tools llm-framework llm-evaluation rag-evaluation

Updated May 13, 2024
Python

DAGWorks-Inc / burr

Build applications that make decisions (chatbots, agents, simulations, etc...). Monitor, persist, and execute on your own infrastructure.

state-management ai state-machine graphs visibility persistent-data-structure chatbot-framework dags burr mlops llms generative-ai llmops

Updated May 13, 2024
Python

trubrics-sdk

trubrics / trubrics-sdk

Product analytics for AI Assistants

machine-learning mlops streamlit ml-monitoring llm human-feedback llmops model-feedback

Updated May 13, 2024
Python

truera / trulens

Evaluation and Tracking for LLM Experiments

machine-learning neural-networks explainable-ml llm llmops

Updated May 13, 2024
Jupyter Notebook

langfuse / langfuse

🪢 Open source LLM engineering platform: Observability, metrics, evals, prompt management, playground, datasets. Integrates with LlamaIndex, Langchain, OpenAI SDK, LiteLLM, and more. 🍊YC W23

open-source playground monitoring analytics evaluation self-hosted ycombinator openai gpt observability large-language-models llm prompt-engineering langchain llmops llama-index prompt-management evals llm-evaluation

Updated May 13, 2024
TypeScript

Dana-Farber-AIOS / GPT4DFCI

A private and secure generative AI tool, based on GPT-4 and deployed for non-clinical use at Dana-Farber Cancer Institute

research operations gpt gpt-4 chatgpt llmops

Updated May 13, 2024
TypeScript

Improve this page

Add a description, image, and links to the llmops topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the llmops topic, visit your repo's landing page and select "manage topics."