4b_best_practices.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Workshop 4: cartopy and best practices\n",
    "\n",
    "# Part II: Best Practices\n",
    "\n",
    "Here I dump my entire accumulated wisdom upon you, not so much hoping that you know it all by the end, but that you know of the concepts and know what to search for. I realize that many lessons will be learned the hard way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Technical tips\n",
    "\n",
    "### jupyter lab tips\n",
    "\n",
    "|||\n",
    "|--- |--- |\n",
    "| `Option/Alt` + drag | multi-line selection |\n",
    "| `Ctrl/Cmd` + `X`/`C`/`V` | cut/copy/paste lines|\n",
    "| `Cmd` + `?`| comment out line |\n",
    "\n",
    "### Linux and SSH\n",
    "The science faculty has its own cluster: `gemini.science.uu.nl`. There you can perform heavier, longer computations.\n",
    "You can tunnel into the cluster by typing\n",
    "```\n",
    "ssh (your_solis_id)@gemini.science.uu.nl\n",
    "```\n",
    "you are then prompted for your password.\n",
    "\n",
    "Note that working `ssh` is not a very stable connection: `broken pipe` error. All active commands are interrupted if your connection is severed. One way around this is to use the `screen` functionality.\n",
    "\n",
    "Many advanced text editors let you work remotely. For example, Visual Studio Code has a `Remote-SSH` extension that lets you work on the remote machine as if it were local.\n",
    "\n",
    "Some useful commands when navigating the terminal and submitting jobs on gemini.\n",
    "\n",
    "| command              | effect |\n",
    "| ---                  | --- |\n",
    "| `pwd`                | print working directory, where  are you in the file system|\n",
    "| `cd`                 | change directory, without: to home directory; `..` for level up|\n",
    "| `ls`                 | list content of folder |\n",
    "| `cp`/`mv`            | copy/move file |\n",
    "| `rm`                 | remove file (CAUTION: this is permanent) |\n",
    "| `grep (word) (file)` | search for `word` in `file` |\n",
    "| `touch (file)`       | create empty `file` |\n",
    "| `top`                | monitor processes and system resources |\n",
    "| `Up(arrow)`          | previous command\n",
    "| `Ctrl + C`           | cancel |\n",
    "| `Ctrl + R`           | search command history |\n",
    "| `qsub (your_job.sh)` | submits `your_job.sh` bash script to queue* |\n",
    "| `qstat`              | check on your jobs in the batch queue |\n",
    "| `qdel (job-id)`      | deletes job with id `job-id` in queue |\n",
    "\n",
    "\\* There are 48 job slots (12 nodes with 4 cores) with 4Gb of memory per core\n",
    "\n",
    "### Project organization\n",
    "Once your projects contain more than a few results (e.g. for SOAC and your thesis), it is worthwhile organizing. A common structure has proven useful for most cases:\n",
    "```\n",
    "| project_name\n",
    "  README            (markdown or simple text file describing your project and its structure)\n",
    "  | data            (when working with big data this may be external to the project folder)\n",
    "    | raw_data      (never touch these files)  \n",
    "    | processed     (derived files)\n",
    "  | doc             (includes `requirements.txt` with python environment description)\n",
    "  | src             (all your [well-documented] code: .py, .ipynb, ... files)\n",
    "  | results         (all figures, maybe in subfolders)\n",
    "```\n",
    "A well organized project helps you current and future self as well as anyone else looking at the results. It is thus an important step for reproducibility.\n",
    "\n",
    "### colormaps\n",
    "Be conscious about the colors and colormaps you use! Colors can hide or emphasize data, which can be used to improve your presentation. Read this [short blog post](https://jakevdp.github.io/blog/2014/10/16/how-bad-is-your-colormap/) and learn why the `jet`colormap is bad.\n",
    "\n",
    "A main consideration for accessibility of your results must be the color blindness which afflicts quite a few people. See [ColorBrewer](https://colorbrewer2.org) for colors that work well together.\n",
    "\n",
    "The `cmocean` package adds some [beautiful, well-designed colormaps for oceanography](https://matplotlib.org/cmocean/) to the standard matplotlib colormaps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Fundemental programming guidelines\n",
    "\n",
    "### Understanding Python error messages\n",
    "\n",
    "Understanding Python errors can be daunting at first, especially if they are very long. But don;t despair, after some practive you become better at interpreting them and will find aathem helpful in pinning down the problem. The general idea is that Python shows you the steps from the line where you called the offending line all the way down to the line in the file that raised an error. Often, the most important part of the error message is located at the end.\n",
    "\n",
    "### DRY: Don't repeat yourself\n",
    "It is almost always a sign of bad programming if you have to repeat a line several times. It clutters the code and makes the code harder to maintain.\n",
    "\n",
    "### simplify code\n",
    "Instead of writing one huge function, __break your functions down into logical component functions__. This will save you many headaches when hunting for bugs.\n",
    "\n",
    "### coding style\n",
    "Python is very forgiving towards your code writing style. Just because it runs wihtout errors does not mean is well written, though.\n",
    "How to write good, readable python code is laid out in the __[PEP8 Style Guide for Python Code](https://pep8.org/)__. Read it and try to adhere to it.\n",
    "\n",
    "### reuse code\n",
    "Once you have iterated to stable code (and you want to share it across jupyter notebooks), you should put it in a separate `.py` file. You can __import functions from `.py` files__ simply as `from file import function`.\n",
    "\n",
    "### Defensive programming\n",
    "\n",
    "Defensive programming is a programming philosophy that tries to guard against errors and minimize time spent on solving bugs. The fundamental idea is that of __unit testing__: you break the code into the small into the small steps (functions) and then test whether they give the expected (known) results for simple test cases. One would write a function with known input and output. Unit testing can be automated and this is known as \"continuous integration (CI)\" (integrated in GitHub,for example).\n",
    "\n",
    "This approach works well for traditional software development with fixed goals, but it is not always suited to scientific programming as the goals shift with new knowledge.\n",
    "\n",
    "However, the principle of defensive programming is still very valuable. A __simple and easy-to-implement version of this defensive philosophy__ can be implemented by using the `assert` statement often (this is not exactly unit testing). It checks whether a statement is true and can raise a custom error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def calc_circumference(radius):\n",
    "    \"\"\" simple example function to calculate circle cirumference \"\"\"\n",
    "    return 2*np.pi*radius\n",
    "\n",
    "def calc_circumference2(radius):\n",
    "    \"\"\" simple example function to calculate circle cirumference \"\"\"\n",
    "    assert type(radius) in [float, int], 'radius must be a number'\n",
    "    return 2*np.pi*radius"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6.283185307179586"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this works as expected\n",
    "calc_circumference(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "can't multiply sequence by non-int of type 'float'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-3-59a092b91f47>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# this does not work and Python tells us why with its own error message\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mcalc_circumference\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'hello'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-1-34a4e286b023>\u001b[0m in \u001b[0;36mcalc_circumference\u001b[0;34m(radius)\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mcalc_circumference\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mradius\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m     \u001b[0;34m\"\"\" simple example function to calculate circle cirumference \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpi\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mradius\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mcalc_circumference2\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mradius\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: can't multiply sequence by non-int of type 'float'"
     ]
    }
   ],
   "source": [
    "# this does not work and Python tells us why with its own error message\n",
    "calc_circumference('hello')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "ename": "AssertionError",
     "evalue": "radius must be a number",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAssertionError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-4-e6932b9c995b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# this does not work and our message tells us why\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mcalc_circumference2\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'hello'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-1-34a4e286b023>\u001b[0m in \u001b[0;36mcalc_circumference2\u001b[0;34m(radius)\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mcalc_circumference2\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mradius\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m     \u001b[0;34m\"\"\" simple example function to calculate circle cirumference \"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m     \u001b[0;32massert\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mradius\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32min\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mfloat\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'radius must be a number'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     10\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpi\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mradius\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAssertionError\u001b[0m: radius must be a number"
     ]
    }
   ],
   "source": [
    "# this does not work and our message tells us why\n",
    "calc_circumference2('hello')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Back-up\n",
    "\n",
    "__Always back up your code and data!__ There is nothing more frustrating than having to rewrite code after you dropped your laptop or something crashed. Cloud services like _[SURFdrive](https://surfdrive.surf.nl)_ or Dropbox/OneDrive make this very easy. the advantage here is that is is __automated__ and does not rely on you remembering that you need to backup.\n",
    "\n",
    "### Version control\n",
    "Do you know this?\n",
    "\n",
    "![](figures/phd101212s.png)\n",
    "\n",
    "There is a better way: version control.\n",
    "\n",
    "Version control systems start with a base version of the document and then record changes you make each step of the way. You can think of it as a recording of your progress: you can rewind to start at the base document and play back each change you made, eventually arriving at your more recent version.\n",
    "\n",
    "![](https://osulp.github.io/git-beginner/fig/play-changes.svg)\n",
    "\n",
    "Once you think of changes as separate from the document itself, you can then think about “playing back” different sets of changes on the base document, ultimately resulting in different versions of that document. For example, two users can make independent sets of changes on the same document - these changes can be organized into separate “branches”, or groupings of work that can be shared.\n",
    "\n",
    "![](https://osulp.github.io/git-beginner/fig/versions.svg)\n",
    "\n",
    "Unless there are conflicts, you can even incorporate two sets of changes into the same base document, or “merged”.\n",
    "\n",
    "![](https://osulp.github.io/git-beginner/fig/merge.svg)\n",
    "\n",
    "__Key points:__\n",
    "- Version control is like an unlimited ‘undo’.\n",
    "- Version control also allows many people to work in parallel.\n",
    "- version control works well for human-readable files (e.g. .py, .txt, .tex), but not binary files (e.g. .docx, .png, ...) because it does line-by-line comparison.\n",
    "\n",
    "`git` is one implementation of a distributed version control system. You can `Github` is a company that let's you host repositories (version controlled folders) online. Everyone can create a free GitHub account and as a student you can create a free Pro account.\n",
    "\n",
    "The use of `git` and GitHub is requires its own tutorial, as there is a small learning up-front cost before you benefit from it. Much information in this section was talen from the [Software Carpentry tutorial](https://osulp.github.io/git-beginner/) which I recommend."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Open science, open access, reproducibility\n",
    "\n",
    "### Open Science\n",
    "From the [Open Science Wikipedia article](https://en.wikipedia.org/wiki/Open_science): \n",
    "> Open science is the movement to make scientific research (including publications, data, physical samples, and software) and its dissemination accessible to all levels of an inquiring society, amateur or professional.\n",
    "\n",
    "The fundamentl idea is that should be able to see all your steps for arriving at certain conclusions. In our context this specifically refers to making the (documented!) code available\n",
    "\n",
    "At Utrecht University, there is a society dedicated to Open Science: [Open Science Community Utrecht](https://openscience-utrecht.com/). \n",
    "\n",
    "### Open access\n",
    "The traditional publishing business model has been to charge readers for access to articles. This is less than ideal, as the public pays for the research and the results are behind a paywall. This hinders knowledge transfer and there is a growing movement to open access (i.e. make it free) to scientific knowledge.\n",
    "\n",
    "To publish open access usually costs money, as the publishers cannot earn money with selling the articles/journals. __The Dutch Universities have agreements with all major publishers to cover open access fees.__ Use this!\n",
    "\n",
    "### Licenses\n",
    "If you want to reuse any online content you must check for the license. If there is no license, you are legally not allowed to use it. This is why it's important that you include a license with your code if you want others to reuse it.\n",
    "\n",
    "### reproducible code\n",
    "When you publish your results (as a thesis or paper) you must __ensure that your code can reproduce all the results__. In jupyter notebooks you should check whether it runs completely from start to finish __without error__. The code must be documented. Ultimately, the clean version of __your code (and if possible raw data) should be uploaded to a permament repository__ such as [UU's own Yoda system](https://www.uu.nl/en/research/yoda), [Zenodo](https://zenodo.org/), [figshare](https://figshare.com/). It can then receivce a __digital object identifier number (DOI)__ and should be cited in your paper and __can be cited by others__.\n",
    "\n",
    "\n",
    "### virtual environments\n",
    "Virtual environments are custom python environments, with specific packages installed. So far you have likely used the `root` environment of your conda installation. This is fine for your course work. With conda you can easily create new environments in the GUI or the command line as such:\n",
    "```\n",
    "conda create -n my_new_env python\n",
    "```\n",
    "where `my_new_env` is the name of the environment. In the command line you would activate this environment with `conda activate my_new_env`. You will then see that name in parantheses in front of your prompt.\n",
    "\n",
    "In general, it is advised to create a new environment for every major project (like a thesis or a particular paper). This ensures that you know which packages + versions you used to do your calculations. You can then __export a list of all the packages used__ at the end of your project and save it with the rest of the code. Only this __ensures reproducibility__.\n",
    "\n",
    "You can view a list of your environments directly in the Anaconda GUI or byt typing `conda info --envs`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}