README.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "You will need a kubernetes cluster available to run the workflow in.\n",
    "\n",
    "A kubeconfig file for that cluster should be uploaded to the to this jupyterlab instance\n",
    "and moved/renamed to the file `~/.kube/config` so that it will be used automatically.\n",
    "\n",
    "The kubeconfig file should be the config for the cluster-admin, or a config for an RBAC\n",
    "account for a specific namespace can be used (if argo/redis are already configured in the cluster).\n",
    "\n",
    "## Cluster setup\n",
    "\n",
    "`kubectl` is installed in this notebook, so as long as there is a valid kubeconfig file in the\n",
    "right location the commands below can run without any modifications.\n",
    "\n",
    "### Install argo in the cluster\n",
    "\n",
    "Argo is a workflow engine that will be processing our workflow file and orchestrating the creation of pods in the cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create namespace argo\n",
    "!kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo/v2.4.0/manifests/install.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For argo to work, it also needs [extra permissions](https://github.com/argoproj/argo/blob/dc54919/docs/workflow-rbac.md) for the service account that the workflow pods will be using (to monitor their own status).\n",
    "\n",
    "This will add the required permissions to the default service account in the default namespace:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl apply -f workflow-role.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For running in another namespace, the references to `namespace: default` in workflow-role.yaml should be changed.\n",
    "\n",
    "The argo client is already installed in this notebook, to install locally check the [releases page](https://github.com/argoproj/argo/releases).\n",
    "\n",
    "### Install redis\n",
    "\n",
    "For the live visualisation used in this analysis notebook, data is pushed from the workflow pods to a redis database and pulled into the notebook.\n",
    "\n",
    "The redis instance can run in the same cluster as the workflow, but because it will also need to be accessed from an external location (the notebook)\n",
    "there are some extra steps to take.\n",
    "\n",
    "The kubernetes service for the redis instance will be of type `NodePort` so that it can be accessed by the IP address of one of the cluster nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create ns redis\n",
    "!kubectl apply -f redis/redis.yaml -f redis/redis-svc.yaml\n",
    "\n",
    "REDIS_HOST = !kubectl -n redis get nodes -l 'node-role.kubernetes.io/master notin ()' -o jsonpath='{.items[0].status.addresses[0].address}'\n",
    "REDIS_PORT = !kubectl -n redis get svc redis -o jsonpath='{.spec.ports[0].nodePort}'\n",
    "\n",
    "!kubectl create secret generic redis-connection --from-literal=REDIS_HOST={REDIS_HOST[0]} --from-literal=REDIS_PORT={REDIS_PORT[0]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The workflow pods will be able to access the redis instance at `redis.redis.svc:6379`, and from the notebook redis will be accessible using the node IP and the service nodeport which are saved in the secret redis-connection.\n",
    "\n",
    "### Configure S3 storage\n",
    "\n",
    "Argo uses S3 for intermediate storage between steps in the workflow, and for the storing the final results.\n",
    "\n",
    "These commands should be run outside of the notebook.\n",
    "\n",
    "```bash\n",
    "$ S3_HOST=$(openstack catalog show s3 -f value -c endpoints | grep public | cut -d '/' -f3)\n",
    "$ ACCESS_KEY=$(openstack ec2 credentials create -f value -c access)\n",
    "$ SECRET_KEY=$(openstack ec2 credentials show $ACCESS_KEY -f value -c secret)\n",
    "\n",
    "$ kubectl create secret generic s3-cred --from-literal=accessKey=$ACCESS_KEY --from-literal=secretKey=$SECRET_KEY\n",
    "\n",
    "$ echo $S3_HOST\n",
    "s3.cern.ch\n",
    "```\n",
    "\n",
    "The `S3_HOST` value goes into the workflow yaml as a parameter.\n",
    "\n",
    "A bucket for artifact storage should be created in the S3 storage and its name\n",
    "should also be given as a parameter in the workflow. The bucket name has to be unique\n",
    "so this **must** be changed from the default value.\n",
    "\n",
    "## Start the workflow\n",
    "\n",
    "Everything is ready for the workflow to be run, head over to `masters-notebook.ipynb`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}