Skip to content

Commit

Permalink
New getting started doc (#101)
Browse files Browse the repository at this point in the history
* new getting started doc

* add getting started doc
* add structure doc for Kitfile
* update readme to point to getting started

* Rework

* Moved structure doc into kitfile overview doc
* renamed getting-started.md to quick-start.md
* Updated overview to remove "getting started" header
* Updated  nav to add the quick start

* Final docs tweaks...

...hopefully
  • Loading branch information
bmicklea committed Mar 13, 2024
1 parent 48df819 commit 266ea7f
Show file tree
Hide file tree
Showing 8 changed files with 106 additions and 23 deletions.
19 changes: 10 additions & 9 deletions README.md
@@ -1,29 +1,26 @@
# Welcome to KitOps 🚀

## Unleashing the Power of Streamlined Collaboration for AI/ML Projects
## Streamlined Collaboration for AI/ML Projects

KitOps is your ultimate toolkit for transforming how you build, package, and deploy AI/ML models. Say goodbye to compatibility concerns and hello to smooth model sharing.
KitOps is your toolkit for transforming how you package, share, and deploy AI/ML models. Say goodbye to compatibility concerns and hello to smooth AI/ML collaboration.

KitOps is designed to enhance collaboration among data scientists, application developers, and SREs working on self-hosted AI/ML models. KitOps' ModelKits simplify packaging models with their dependencies, configurations, and environments. The ModelKit is portable and uses open standards for compatibility.
KitOps simplifies the handoffs between data scientists, application developers, and SREs working on self-hosted AI/ML models (including LLMs). KitOps' ModelKits create a unified package for models, their dependencies, configurations, and environments. The ModelKit is portable and uses open standards for compatibility with the tools you already use.

### What is in the box?

**ModelKit:** At the heart of KitOps is the ModelKit, an OCI-compliant packaging format that enables the seamless sharing of all necessary artifacts involved in the AI/ML model lifecycle. This includes datasets, code, configurations, and the models themselves. By standardizing the way these components are packaged, ModelKit facilitates a more streamlined and collaborative development process that is compatible with nearly any tool.

**Kitfile:** Complementing the ModelKit is the Kitfile, AI/ML project's blueprint, a YAML-based configuration file that simplifies the sharing of model, dataset, and code configurations. The Kitfile is designed with both ease of use and security in mind, ensuring that configurations can be efficiently packaged and shared without compromising on safety or governance.
**Kitfile:** Complementing the ModelKit is the Kitfile, your AI/ML project's blueprint. It's a YAML-based configuration file that simplifies the sharing of model, dataset, and code configurations. Kitfiles are designed with both ease of use and security in mind, ensuring that configurations can be efficiently packaged and shared without compromising on safety or governance.

**Kit CLI:** Your magic wand for AI/ML collaboration. The Kit CLI is a powerful tool that enables users to create, manage, run, and deploy ModelKits using Kitfiles. Whether you are packaging a new model for development or deploying an existing model into production, the Kit CLI provides the necessary commands and functionalities to streamline your workflow.

KitOps enhances the end-to-end lifecycle of AI/ML model management, making it as streamlined as managing containerized applications. It's about enhancing collaboration, streamlining processes, and unlocking a world of possibilities for AI/ML innovation.
**Kit CLI:** Your magic wand for AI/ML collaboration. The Kit CLI not only enables users to create, manage, run, and deploy ModelKits... it lets you pull only the pieces you need. Just need the serialized model for deployment? Use `unpack --model` or maybe you just want the training datasets? `unpack --datasets`. So, whether you are packaging a new model for development or deploying an existing model into production, the Kit CLI provides the flexibility and power to streamline your workflow.

## Quick Start with Kit

Dive into the world of KitOps with ease! Whether you're looking to streamline your AI/ML workflows or explore the power of ModelKits, getting started with Kit is straightforward.

### Running Kit with Pre-built Binaries


Get started with the Kit CLI by downloading a pre-built binary. Choose the `latest` [tagged version](https://github.com/jozu-ai/kitops/tags) for the most stable release, or explore the `next` tag for our development builds.
First, download the Kit CLI. Choose the `latest` [tagged version](https://github.com/jozu-ai/kitops/tags) for the most stable release, or explore the `next` tag for our development builds.

For installation instructions and selecting the right binary for your platform, please refer to our [Installation Guide](./docs/src/docs/cli/installation.md).

Expand Down Expand Up @@ -62,6 +59,10 @@ Or, for direct execution during development:
go run .
```

## Using Kit

The easiest way to get introduced to Kit is with our [quick start guide](./docs/src/docs/getting-started.md)

## Your Voice Matters

### Reporting Issues and Suggesting Features
Expand Down
1 change: 1 addition & 0 deletions docs/.vitepress/config.mts
Expand Up @@ -53,6 +53,7 @@ export default defineConfig({
text: 'Getting started',
items: [
{ text: 'Overview', link: '/docs/overview' },
{ text: 'Quick Start', link: '/docs/quick-start.md' },
{ text: 'Use Cases', link: '/docs/use-cases' },
{ text: 'Why KitOps?', link: '/docs/why-kitops' },
]
Expand Down
65 changes: 65 additions & 0 deletions docs/src/docs/kitfile/kf-overview.md
Expand Up @@ -21,3 +21,68 @@ Crafted with simplicity and efficiency in mind, the Kitfile organizes project de
By encapsulating the essence of your AI/ML project into a singular, version-controlled document, the Kitfile not only simplifies the packaging process but also enhances collaborative efforts. Whether you're sharing projects within your team or with the global AI/ML community, the Kitfile ensures that every artifact, from datasets to models, is accurately represented and easily accessible.

Embrace the Kitfile in your AI/ML projects to harness the power of structured packaging, efficient collaboration, and seamless artifact management. As the backbone of the KitOps ecosystem, the Kitfile is your first step towards simplifying AI/ML project management and achieving greater innovation.

## Kitfile Structure

The Kitfile defines the contents of your ModelKit. It is written in YAML and stored with the ModelKit. You can extract the Kitfile from any ModelKit with the Kit CLI:

```sh
kit unpack [registry/repo:tag] --config -d .
```

There are four main parts to a Kitfile:
1. ModelKit metadata in the `package` section
1. Path to the Jupyter notebook folder in the `code` section
1. Path to the serialized model in the `model` section
1. Path to the datasets in the `datasets` section (you can have multiple datasets in the same page)

Here's an example Kitfile:

```yaml
manifestVersion: v1.0.0

package:
authors:
- Jozu
description: Updated model to analyze flight trait and passenger satisfaction data
license: Apache-2.0
name: FlightSatML

code:
- description: Jupyter notebook with model training code in Python
path: ./notebooks

model:
description: Flight satisfaction and trait analysis model using Scikit-learn
framework: Scikit-learn
license: Apache-2.0
name: joblib Model
path: ./models/scikit_class_model_v2.joblib
version: 1.0.0

datasets:
- description: Flight traits and traveller satisfaction training data (tabular)
name: training data
path: ./data/train.csv
- description: validation data (tabular)
name: validation data
path: ./data/test.csv
```

The only mandatory parts of the Kitfile are:
* `manifestVersion`
* At least one of `code`, `model`, `or datasets` sections

A ModelKit can only contain one model, but multiple datasets or code bases are allowed. Also note that you can only use relative paths (no absolute paths) in your Kitfile. Right now you can only build ModelKits from files on your local system...but don't worry we're already working towards allowing you to reference remote files. For example, building a ModelKit from a local notebook and model, but a dataset hosted on DvC, S3, or anywhere else.

So a minimal ModelKit for distributing a pair of datasets might look like this:
```yaml
manifestVersion: v1.0.0

datasets:
- name: training data
path: ./data/train.csv
- description: validation data (tabular)
name: validation data
path: ./data/test.csv
```
1 change: 0 additions & 1 deletion docs/src/docs/kitfile/structure.md

This file was deleted.

8 changes: 3 additions & 5 deletions docs/src/docs/overview.md
@@ -1,6 +1,4 @@
# Getting started

## What is KitOps?
# What is KitOps?

KitOps is an innovative open-source project designed to enhance collaboration among data scientists, application developers, and SREs working on integrating or managing self-hosted AI/ML models.

Expand All @@ -16,9 +14,9 @@ KitOps is an innovative open-source project designed to enhance collaboration am

The primary goal of KitOps is to bridge the gaps between data science, software development, and operational deployment. By providing tools that standardize and simplify the sharing of AI/ML artifacts, KitOps drives greater speed, security, and collaboration for teams hosting their own AI/ML models.

_For application developers_ KitOps clears the path for application developers to harness AI/ML without getting entangled in its complexities. It lets developers concentrate on crafting exceptional applications, while KitOps handles the intricate AI/ML workflows. Whether integrating a new ML model or collaborating on novel features, KitOps accelerates the journey from idea to deployment.
_For application developers_ KitOps clears the path to use AI/ML with your existing tools and applications. No need to be an AI/ML expert, KitOps lets you concentrate on crafting exceptional applications, while it handles the packaging and sharing.

_For data scientists_ KitOps enables them to innovate in AI/ML without the usual infrastructure distractions. It simplifies dataset and model managementand sharing, fostering closer collaboration with developers. With KitOps, data scientists can spend more time experimenting and less time grappling with traditional software development tools.
_For data scientists_ KitOps enables you to innovate in AI/ML without the usual infrastructure distractions. It simplifies dataset and model managementand sharing, fostering closer collaboration with developers. With KitOps, you can spend more time experimenting and less time grappling with traditional software development tools.

Join us in shaping the future of AI/ML collaboration with KitOps.

Expand Down
@@ -1,16 +1,22 @@
# Getting Started with KitOps
# KitOps Quick Start

In this guide, we'll use ModelKits and the kit CLI to easily:
* Package up a model, notebook, and datasets into a single ModelKit you can share through your existing mechanisms
* Push the ModelKit package to a public or private registry
* Grab just the things you need from the ModelKit for testing, integration, local running, or deployment

## Before we start...

Make sure you've got the [Kit CLI setup](./cli/installation.md) on your machine.

We recommend pulling one of our [example ModelKits](https://github.com/orgs/jozu-ai/packages) for this quick start. From there you can try [writing a Kitfile](./kitfile/format.md) for your own AI/ML project.

## Preparing for packaging

The first step is to make a `Kitfile` - a YAML manifest for your ModelKit. There are four main parts to a Kitfile:
1. ModelKit metadata in the `package` section
1. Path to the Jupyter notebook folder in the `code` section
1. Path to the datasets in the `datasets` section (you can have multiple datasets in the same page)
1. Path to the datasets in the `datasets` section
1. Path to the serialized model in the `model` section

Here's an example Kitfile:
Expand Down
17 changes: 14 additions & 3 deletions docs/src/docs/use-cases.md
@@ -1,6 +1,17 @@
# Use Cases

## Use Case Context
KitOps removes friction from the handoffs between AI/ML and App/SRE teams:
* AI/ML teams can package their project, including models, code, datasets, and configuration, into a ModelKit that can be run by anyone, anywhere.
* Application teams can pull only what they need from a ModelKit to embed a model in their app, or generate a RESTful API for integration.
* SRE teams can pull the serialized model and validation dataset from a ModelKit and deploy it through their CI/CD pipelines.
* Governance teams can track and audit AI/ML assets with your existing processes and tools (we save everything as OCI-compatible artifacts).
* Security teams can integrate ModelKits into their SBOM processes.

To make the use cases a little more fun let's tell two stories:
1. Collaborating on an AI-enabled application for customers
2. Deploying a model for internal business usage

In both cases the context of the stories is the same.

Weyland-Yutani Corporation has been adding AI and ML models to its portfolio for both internal- and customer-facing deployment. There are three main groups that are involved in the development lifecycle for their AI/ML work:
* AI/ML Team, composed of data scientists, data engineers, and MLOps engineers. This specialized group builds, tunes, and validates the models that increasingly touch every aspect of the organization's work.
Expand All @@ -11,7 +22,7 @@ There has been a heated argument within Weyland-Yutani about how best to manage

...but all that's about to change! (Hint: the change involves KitOps...surprise!)

## Collaborating on an AI-Enabled Application for Production
## 1/ Collaborating on an AI-Enabled Application for Production

### AI/ML Team Collaboration

Expand Down Expand Up @@ -81,7 +92,7 @@ Once the deployment is complete across all galactic regions, Annika sends an upd

Thanks KitOps!

## Collaborating on an Internal Model
## 2/ Collaborating on an Internal Model

### AI/ML and SRE Team Collaboration

Expand Down
8 changes: 5 additions & 3 deletions docs/src/docs/why-kitops.md
@@ -1,14 +1,16 @@
# Why Use KitOps?

Because getting an LLM or other AI/ML model deployed safely to production is hard...and it shouldn't be.
Because hand-offs between AI/ML and app/SRE teams are painful.
Because getting an LLM or other AI/ML model deployed safely to production is hard.
Because they can be easy, and painless.

Jupyter notebooks are great, but extracting the model, datasets, and metadata from one is tricky. Similarly, ML-specific experimentation tools like MLFlow or Weights & Biases are excellent at training, but they save everything in proprietary formats that are confusing for software engineers and SREs.

Worse yet, none of these AI/ML tools are compatible with the toolchains organizations have successfully used for years with their applications.

With leaders demanding teams "add AI/ML" to their protfolios, many have fallen into a "throw it over the wall and hope it works" process that adds risk, delay, and frustration to self-hosting models.

**The goal of KitOps is to simplify the sharing of AI/ML models, datasets, code, and configuration so that they can be run anywhere.**
> **The goal of KitOps is to simplify the sharing of AI/ML models, datasets, code, and configuration so that they can be run anywhere.**
We're not trying to replace your other tools, we just want to improve the experience of packaging and sharing models.

Expand All @@ -33,4 +35,4 @@ Then `kit push` it to any OCI-compliant registry, even a private one.

Most people won't need everything, so just `kit unpack` from the remote registry to get just the model, only the datasets, or just the notebook. Or, if you need everything then a `kit pull` will grab everything.

Check out our [CLI command reference](./cli/cli-reference.md), or learn more about [ModelKits](./modelkit/intro.md).
Check out our [quick start](./quick-start.md), see the power and flexibility of our [CLI commands](./cli/cli-reference.md), or learn more about packaging your AI/ML project with [ModelKits](./modelkit/intro.md).

0 comments on commit 266ea7f

Please sign in to comment.