LFPy-example_test_suite_UNICORE.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example: LFPy test suite\n",
    "\n",
    "This example notebook submits a job to a chosen compute cluster (**JUSUF**), loads a **Singularity** container with a working **LFPy** installation and runs **LFPy**'s built in unit test suite (`LFPy.run_tests()`) before exiting. \n",
    "\n",
    "For more indepth info on using HPC backends via EBRAINS see https://wiki.ebrains.eu/bin/view/Collabs/using-supercomputers-from-the-collab/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use the pyunicore library\n",
    "!pip install pyunicore --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import modules\n",
    "import os\n",
    "import pyunicore.client as unicore_client\n",
    "import requests\n",
    "import json\n",
    "from time import sleep, time\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "unicore_client._HBP_REGISTRY_URL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create connection to supercomputer (e.g., JUSUF)\n",
    "tr = unicore_client.Transport(clb_oauth.get_token())\n",
    "r = unicore_client.Registry(tr, unicore_client._HBP_REGISTRY_URL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Valid choices for supercomputers are one of the keys in:\n",
    "r.site_urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create client\n",
    "supercomputer = 'JUSUF'\n",
    "try:\n",
    "    site_client = r.site(supercomputer)\n",
    "except KeyError:\n",
    "    site_client = unicore_client.Client(r.transport, 'https://zam2125.zam.kfa-juelich.de:9112/JUSUF/rest/core')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check connection to supercomputer\n",
    "headers = {}\n",
    "headers[\"Authorization\"] = \"Bearer \" + clb_oauth.get_token()\n",
    "headers['Accept'] = \"application/json\"\n",
    "rs = requests.get(url=site_client.site_url, headers=headers, verify=False)\n",
    "print(\"Status code %s \" % rs.status_code)\n",
    "print(\"Content-type %s \" % rs.headers['Content-Type'])\n",
    "reply = rs.json()\n",
    "# print(json.dumps(reply, indent = 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check connection to supercomputer\n",
    "# base = r.site_urls[supercomputer]\n",
    "headers = {}\n",
    "headers[\"Authorization\"] = \"Bearer \" + clb_oauth.get_token()\n",
    "headers['Accept'] = \"application/json\"\n",
    "rs = requests.get(url='https://zam2125.zam.kfa-juelich.de:9112/JUSUF/rest/core', headers=headers, verify=False)\n",
    "print(\"Status code %s \" % rs.status_code)\n",
    "print(\"Content-type %s \" % rs.headers['Content-Type'])\n",
    "reply = rs.json()\n",
    "# print(json.dumps(reply, indent = 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare singularity container\n",
    "See https://gitlab.version.fz-juelich.de/bvonstvieth_publications/container_userdoc_tmp for details. \n",
    "\n",
    "This step builds the singularity container. It is optional if the recipe has already been uploaded and built on the system. \n",
    "\n",
    "The procedure may be different on different HPC backends. \n",
    "The container can either way be built from the same recipe: https://raw.githubusercontent.com/LFPy/LFPydebian/main/mpich.Dockerfile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Turn the below Raw block into Code in order to run:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# build singularity container with LFPy using a Dockerfile. \n",
    "# This step only needs to be done once per user/project (possibly). \n",
    "build_job = {\n",
    "  \"Executable\" : \"\"\"module use $OTHERSTAGES\n",
    "module --force purge\n",
    "module load Stages/2020\n",
    "module load GCC Singularity-Tools\n",
    "\n",
    "#mkdir -p ~/.config/sib\n",
    "#cat > ~/.config/sib/settings.ini <<'EOF'\n",
    "#[config]\n",
    "#url_prefix=https://sbuild-hps.fz-juelich.de/\n",
    "#EOF\n",
    "\n",
    "wget https://raw.githubusercontent.com/LFPy/LFPy/master/Dockerfile\n",
    "sib upload ./Dockerfile lfpymaster\n",
    "sib build −−recipe−name lfpymaster −−blocking\n",
    "sib wait-for-completion\n",
    "\"\"\",\n",
    "    \n",
    "  \"Job type\": \"interactive\",\n",
    "}\n",
    "build_job = site_client.new_job(job_description=build_job, inputs=[])\n",
    "while build_job.is_running():\n",
    "    sleep(10)\n",
    "build_job.delete()  # delete job cleanly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare main simulation job\n",
    "\n",
    "This step combines in a single session the following:\n",
    "\n",
    "- download LFPy container\n",
    "- ask for resources (# nodes, # MPI processes, # seconds runtime)\n",
    "- execute simulation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create dictionary with job specification and define list of input files from this Collab\n",
    "# simulation_job = {\"Job type\": \"interactive\"}\n",
    "simulation_job = {}\n",
    "simulation_inputs = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Resources\n",
    "simulation_job['Resources'] = {\n",
    "    \"Queue\": \"batch\",\n",
    "    \"CPUs\": \"1\",\n",
    "    \"Runtime\": \"600\",\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# commands run on login node before job execution\n",
    "simulation_job[\"User precommand\"] = \"\"\"module use $OTHERSTAGES\n",
    "module --force purge\n",
    "module load Stages/2020\n",
    "module load GCC Singularity-Tools\n",
    "sib download --recipe-name lfpymaster\n",
    "\"\"\"\n",
    "simulation_job[\"RunUserPrecommandOnLoginNode\"] = \"true\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# - set some environment variables\n",
    "# - run the python code using interpreter embedded in container\n",
    "simulation_job[\"Executable\"] = \"\"\"module use $OTHERSTAGES\n",
    "module --force purge\n",
    "module load Stages/2020\n",
    "module load GCC Singularity-Tools\n",
    "unset DISPLAY  # matplotlib may look for a nonexistant display on compute node \n",
    "singularity exec lfpymaster.sif python3 -c \"import LFPy; LFPy.run_tests()\"\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# commands run after job is done\n",
    "simulation_job[\"User postcommand\"] = \"\" \n",
    "simulation_job[\"RunUserPostcommandOnLoginNode\"] = \"true\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "simulation_job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create job\n",
    "job = site_client.new_job(job_description=simulation_job, inputs=simulation_inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# wait while job is running\n",
    "while job.is_running():\n",
    "    sleep(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job.working_dir.listdir().keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# STDERR output (if any)\n",
    "stderr = job.working_dir.stat('stderr')\n",
    "pprint(stderr.raw().readlines())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# STDOUT output\n",
    "stdout = job.working_dir.stat('stdout')\n",
    "pprint(stdout.raw().readlines())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# kill job, clean up files on the remote\n",
    "job.delete()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}