cml_introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d7849129-b971-47b3-b95c-8097a5ce0a47",
   "metadata": {},
   "source": [
    "# Continuous Machine Learning\n",
    "\n",
    "Als Data-Science-Team wollen wir so oft wie möglich zusammenarbeiten und unsere Arbeit integrieren. Um sicherzustellen, dass unsere Arbeit nicht nur lokal durchgeführt wird (\"aber es funktioniert auf meinem Rechner\") und mit der Arbeit der anderen kompatibel ist, möchten wir unsere Daten- und Trainingspipeline als Teil unserer kontinuierlichen Integrationspipeline (CI) ausführen.\n",
    "In diesem Showcase möchten wir zeigen, wie Sie eine einfache CI-Pipeline einrichten können, die über Github Actions läuft und das Modell für jede gepushte Commit-Gruppe trainiert. Als nettes Bonus-Feature werden wir cml.dev verwenden, um Trainingsmetriken an eine Pull-Anfrage zu senden.\n",
    "Haftungsausschluss\n",
    "Dieser Showcase basiert auf dem offiziellen [cml.dev](cml.dev)-Implementierungsbeispiel :)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a93dc3b1-38ee-410b-9faa-26d0037ee390",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "To follow this showcase, you will need\n",
    "- basic familiarity with git\n",
    "- a Github account"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5930bdf-d0f2-4296-8abd-29ca22123843",
   "metadata": {},
   "source": [
    "# Setup the repository\n",
    "\n",
    "1. Create a new Github repository via the Github UI (top-right corner, \"+\" symbol, \"New repository\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30df4d61-80eb-46de-bc63-83d8c7acb419",
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize git repo locally\n",
    "!git init"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cdb68cf-8556-4bc3-953f-83b63770be8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# set our author profile\n",
    "!git config user.email \"you@example.com\"\n",
    "!git config user.name \"Your Name\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21eaa66d-9290-49a0-9163-a6bee4a42b3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# add the Github repo as a remote to the local repository\n",
    "#!git remote add origin https://github.com/YOUR_GH_NAME/YOUR_GH_REPO.git"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7638cb9b-88d9-4be0-9e4a-1ad7feac470b",
   "metadata": {},
   "source": [
    "## Add the code to the repo\n",
    "We've already prepared a simple data science project. It consists of the file `get_data.py`, which imports some tabular data, as well as `train.py`, which trains a RandomForestClassifier. The `requirements.txt` contains a list of the python packages necessary to run our code. In addition, we also prepared a Github Actions workflow definition at `.github/workflows/pipeline.yaml`. Let's have a look at these files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87f6ea78-152d-4adb-9d7f-9787b18e7eda",
   "metadata": {},
   "outputs": [],
   "source": [
    "!git add get_data.py train.py requirements.txt .github/workflows/pipeline.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "decfcaa0-88ef-4a39-a607-f6d51dc4fed3",
   "metadata": {},
   "outputs": [],
   "source": [
    "!git commit -m \"Add core files\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cff2dbc0-5454-4548-97c8-8cfc6d55f962",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# we have to open a terminal for this, since it asks for input\n",
    "# !git push -u origin master"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fbdf14d-167e-48ae-b0ec-ce2310a5b550",
   "metadata": {},
   "source": [
    "View the pipeline running now automatically in your Github repo via Github Actions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0eeeab58-3e0d-45a5-a2d9-77e60542338b",
   "metadata": {},
   "source": [
    "## Let's change something\n",
    "Now that we established the CI pipeline, we want to change some hyperparameters and view the changes. We will create a new branch and push our changes to the new branch. Then, we will create a pull request.\n",
    "\n",
    "First, let's change the hyperparameter `depth` in our training file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "337da433-5e4d-4d76-8d06-bf5b1356f9a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "!git checkout -b changed-depth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc7367f8-2f57-4e29-81e4-8e1bb0ff9846",
   "metadata": {},
   "outputs": [],
   "source": [
    "!git add train.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f9dc40e-9af3-443a-848b-616162e9433d",
   "metadata": {},
   "outputs": [],
   "source": [
    "!git commit -m \"Try depth of 5\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5fa51e0-14e6-4c7d-bdcd-2fe4f84eca76",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we have to open a terminal for this, since it asks for input\n",
    "# !git push -u origin changed-depth"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c116df10-115f-44bd-a2c5-3bc4533a03dc",
   "metadata": {},
   "source": [
    "Open a pull request and wait for the pipeline to finish."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "725ff4e9-09d4-4b4c-9a03-990fed8b22f0",
   "metadata": {},
   "source": [
    "## Remarks\n",
    "\n",
    "Our showcase trained a model using a very basic python script. Of course, you can also combine this idea with more advanced pipelines and frameworks, such as dvc or mlflow.\n",
    "\n",
    "The more complicated and resource-intensive your pipeline gets (e.g. because your training requires GPUs), it might be worth looking into running the pipeline not directly in the Github runner, but remotely in an orchestrator such as Airflow or Kubeflow Pipelines."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}