cwpk-57-publishing-interactive-notebooks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CWPK \\#57: Publishing Interactive Notebooks\n",
    "=======================================\n",
    "\n",
    "Notebooks are Only Interactive if You Share\n",
    "--------------------------\n",
    "\n",
    "<div style=\"float: left; width: 305px; margin-right: 10px;\">\n",
    "\n",
    "<img src=\"http://kbpedia.org/cwpk-files/cooking-with-kbpedia-305.png\" title=\"Cooking with KBpedia\" width=\"305\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "We first began publishing interactive electronic notebooks for this [*Cooking with Python and KBpedia*](https://www.mkbergman.com/cooking-with-python-and-kbpedia/) series in [**CWPK #16**](https://www.mkbergman.com/2345/cwpk-16-planning-the-project/), though the first installment using a notebook started with [**CWPK #14**](https://www.mkbergman.com/2342/cwpk-14-markdown-and-anatomy-of-a-notebook-file/). I had actually drafted all of the installments up to this one before that date was reached. Since one major purpose of this series was to provide hands-on training, I did not want to force those who wanted to experience some degree of interactivity to have to go through all of the steps to set up their own interactive environment. My hope was that a taste of direct involvement with the code and interactivity would itself encourage users to get more deeply involved to establish their own interactive environments.\n",
    "\n",
    "I had encountered for myself fully interactive notebooks prior to this point, ones where all I needed to do was to click to operate, so I knew there must be a way to make my own notebooks similarly available. In order to achieve my objective, as is true with so much of this series, I was forced to do the research and discover how I could set up such a thing.\n",
    "\n",
    "### A Survey of the Options\n",
    "In researching the options it was clear that a spectrum of choices existed. We have already discussed how we can create non-interactive mockups of an interactive notebook using the <code>nbconvert</code> option or converting a drafted notebook using [Pandoc](https://en.wikipedia.org/wiki/Pandoc) ([**CWPK #15**](https://www.mkbergman.com/2343/cwpk-15-using-notebooks-for-cwpk-documentation/)). My research surfaced some additional options to render a notebook page for general Web (HTML) display:\n",
    "\n",
    "#### Static Options\n",
    "1. <code>nbconvert</code>, but lose interactivity\n",
    "1. The Pandoc option\n",
    "1. Publish in other formats (PDF)\n",
    "1. View a non-interactive page via [nbviewer](https://nbviewer.jupyter.org/) by simply providing a URL, which works like <code>nbconvert</code>.\n",
    "\n",
    "These options are helpful, of course, but lack the full interactivity desired. \n",
    "\n",
    "#### Fully Interactive\n",
    "Systems that allow code cells to be run interactively are obviously more complex than nice rendering tools. My investigations turned up a number of online services, plus ways to set up own or private servers. From the standpoint of online services, here are the leading options:\n",
    "\n",
    "There is a Python option that does not provide complete interactivity, but simple interactions of certain aspects of certain notebook cells:\n",
    "\n",
    "1. <code>[nbinteract](https://www.nbinteract.com/index.html)</code> is a Python package that provides a command-line tool to generate interactive web pages from Jupyter notebooks\n",
    "\n",
    "Then, there are a series of online services:\n",
    "\n",
    "\n",
    "2. The [MyBinder](https://mybinder.org/) option, which uses a [JupyterHub](https://jupyterhub.readthedocs.io/en/latest/) server directly from a [Git](https://en.wikipedia.org/wiki/Git) repository\n",
    "1. Google's [Colaboratory](https://colab.research.google.com/notebooks/intro.ipynb), which provides a Google flavor on this approach\n",
    "1. Microsoft's [Azure Notebooks](https://notebooks.azure.com/), which provides a MS flavor on this approach\n",
    "1. There are other sites such as [Kaggle Kernels](https://www.kaggle.com/kernels), [CoCalc](https://cocalc.com/doc/jupyter-notebook.html), [nanoHUB](https://nanohub.org/whypublish/whypublishdeployjupyter), or [Datalore](https://datalore.jetbrains.com/) that also provide such services, some for a fee.\n",
    "\n",
    "The other interactive approach is to not use an established service, but to set up your own server. \n",
    "\n",
    "6. For private repositories, one can build on [BinderHub](https://binderhub.readthedocs.io/), the same technology used by MyBinder, and which runs on JupyterHub running on [Kubernetes](https://en.wikipedia.org/wiki/Kubernetes) for most of its functionality, or\n",
    "1. One can [run a public notebook server](https://jupyter-notebook.readthedocs.io/en/stable/public_server.html#notebook-public-server) based on Jupyter, though it is limited to a single access user at a time, or\n",
    "1. Set up one's own JupyterHub, similar to the BinderHub option but not limited to a Git repository.\n",
    "\n",
    "Frankly, most of the own-server options looked to be too much work simply to support my educational objectives for the **CWPK** series."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Chosen MyBinder Option\n",
    "I was very much committed to have an online service that would run my full stack. I chose to implement the MyBinder option because I could see it worked and was popular, it had close ties to Jupyter and rendered notebooks the same as when using locally, was free, and seemed to have strong backing and documentation. On the other hand, MyBinder has some weaknesses and poses some challenges. Some of the key ones I knew going in or identified as I began working with the system were:\n",
    "\n",
    "- As a hosted service that runs its applications in containers, it could take some minutes to get the online service active when used after a hiatus, necessary to reconfigure the container specific to the application and Python modules being used\n",
    "- It reportedly has memory limitation of 1-2 GB. Memory can be an issue with **CWPK** locally even at the 8 GB level\n",
    "- The service needed to run off of a Git repo. I had plans to better expose all aspects of the **CWPK** series and its supporting software on our existing public [GitHub repository](https://github.com/Cognonto); the Git requirement caused me to accelerate my exposure of this service\n",
    "- Though free now, each MyBinder application is a rather large consumer of resources. I have some concerns regarding the longer-term availability of the service\n",
    "- **CWPK** would have more than 60 interactive notebook pages, though I did see reference that performance issues may only arise due to multiple, concurrent use. Going in, I had no clue as to what the use factor might be for the service and whether this would pose a problem or not.\n",
    "\n",
    "Some of these issues deserve their own commentary below.\n",
    "\n",
    "### Setting Up the Environment\n",
    "Setting up a new instance at the MyBinder service is relatively straightforward. Here is the basic set-up screen on the [main page](https://mybinder.org/):\n",
    "\n",
    "<div style=\"margin: 10px auto; display: table;\">\n",
    "\n",
    "<img src=\"files/mybinder-1.png\" title=\"Setting Up a New MyBinder Project\" width=\"800\" alt=\"Setting Up a New MyBinder Project\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "<div style=\"margin: 10px auto; display: table; font-style: italic;\">\n",
    "\n",
    "Figure 1: Setting Up a New MyBinder Project\n",
    "\n",
    "</div>\n",
    "\n",
    "One must first have a Git repository available to start the service. One also needs to have completed an [environment configuration file](https://mybinder.readthedocs.io/en/latest/config_files.html) (<code>environment.yml</code> in our case with Python) and a project <code>README.md</code> at the root of the master branch on the repo. In our case, it is the **CWPK** repository on [GitHub](https://github.com/Cognonto/CWPK); we also indicate we are dealing with the master branch **(1)**. (I had some initial difficulty when I over-specified a link to an individual notebook page; removing this cleared things up.) These simple specifications create the URL that is the link to your formal online project **(2)**. Upon launch **(3)**, the build process is shown to screen **(4)**, which may take some minutes to build. The set of working input specs also provide the basis for generating a link badge you can use on your own Web sites **(5)**.\n",
    "\n",
    "Upon completion of a successful build, one is shown the standard Jupyter Notebook entry page.\n",
    "\n",
    "Here are two additional resources useful to setting up a MyBinder application for the first time:\n",
    "- [How to Use Binder for Reproducible Research](https://www.marsja.se/how-to-use-binder-python-for-reproducible-research/)\n",
    "- [Jupyter Notebooks Github Version Control: Prerequisites and Setup](https://www.lyasolisblog.ie/home/2020/4/6/jupyter-notebooks-github)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation Challenges\n",
    "Though set up is straightforward, there are some challenges in implementing MyBinder to accommodate specific **CWPK** needs. Here are some of the major areas I encountered, and some steps to address them.\n",
    "\n",
    "#### Importing Local Code\n",
    "As has become obvious in our series to date, Python is a highly configurable environment, with literally tens of thousands of packages to choose from to invoke needed functionality. The standard environment settings appear to do a good job of allowing new packages to be specified and imported into the MyBinder system. I had confidence these could be handled appropriately.\n",
    "\n",
    "My major concern related to **CWPK**'s own *cowpoke* package. At the time of starting this effort, this package was not commercial grade and was not registered on major distribution networks like [PyPi](https://pypi.org/) or [conda-forge](https://conda-forge.org/). When used locally, including *cowpoke* is not a big issue: we only need to include it in the local listing of site packages. But, once we rely on a cloud instance, how can we get that code into our online MyBinder system?\n",
    "\n",
    "The answer, it turns out, is to [package](https://the-hitchhikers-guide-to-packaging.readthedocs.io/en/latest/creation.html) our code as one would normally do for a commercial package, and then to [include a <code>setup.py</code>](https://stackoverflow.com/questions/1471994/what-is-setup-py) configuration file in our local specification. That enables us to invoke the package through the standard MyBinder environment configuration. See especially this [key reference](https://mybinder.readthedocs.io/en/latest/sample_repos.html) and this [stub for <code>setup.py</code>](https://github.com/navdeep-G/setup.py).\n",
    "\n",
    "#### Local Data Sharing and Organization\n",
    "Until this point, I had been developing and refining a local file directory structure in which to put different versions of KBpedia, other input files, and example outputs. This system was being developed logically from the perspective of a local file system.\n",
    "\n",
    "However, these files were local and not exposed for access to an online system like MyBinder. My first thought was to simply copy this structure to the GitHub repo. But the manual copying of files to a version control system is **NOT** efficient, and the directory structure itself did not appear suitable for a repo presence. Further, manually copying files presents an ongoing issue of keeping local and remote versions in sync. Moreover, as I began adding new daily installments to the GitHub repo, I could see in general that manual additions were not going to be sustainable.\n",
    "\n",
    "These realizations forced two decisions. First, I would need to re-think and re-organize my directory structures to accommodate both local and repo needs. The directory structure we have developed to date now reflects this re-organization. Second, as described in the next section, I needed to cease my manual use of GitHub and fully embrace it as a version control system."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fully Embracing GitHub\n",
    "I have to admit: Every time I try to work with version control with Git, I have been confused and frustrated with how to actually get anything done. I have, hopefully, progressed a bit beyond this point, but I would caution some of you looking to move into this area that you may have to overcome poor documentation and obfuscated instructions and commands.\n",
    "\n",
    "So, I will not divert this series to deal with how to properly set up a Git-based version control system. In brief, one needs to establish a Git repository, and on Windows set up a Git client and then (for me), set up [TortoiseGit](https://tortoisegit.org/) if you want to work directly in Windows and the File Explorer rather than from the command line. In the process of doing all of this, you will also need to set up a key-based access control system with PuTTy (puttygen) so that you can communicate securely between your remote instance and your local file system. These steps are more effectively described in the TortoiseGit [manual](https://tortoisegit.org/docs/tortoisegit/) or various [tutorials](https://www.mediawiki.org/wiki/Gerrit/TortoiseGit_tutorial) of one [aspect](https://www.softwaretestinghelp.com/tortoisegit-tutorial/) or [another](https://medium.com/fbdevclagos/working-with-tortoisegit-and-github-1bf67302f420). Installation, too, can be difficult with regard to [general aspects](https://sachingpta.gitlab.io/_posts/using-git-with-tortoisesvn.html) or the [PuTTy keys](https://dbanck.svbtle.com/github-windows-and-tortoisegit-part-1-installing-pulling).\n",
    "\n",
    "The reason, of course, for accepting this set-up complexity is being able to make changes either on a local version of code or data or a remote version of the same. This is the essence of the definition of keeping systems in 'sync'. After having worked with these systems now for some weeks, daily, I think I can offer some simple tips for how best to work with these version control systems, points which are not obvious from most written presentations:\n",
    "\n",
    "1. First, make sure both the remote and local sources are in sync (this is actually not such an easy point, and is often the point of failure and frustration. However, until this status of being in sync is met, none of the other points below are possible.) When working locally, it is good practice to 'pull' any changes first from the remote repository before you attempt to 'push' local changes back to it. If you run into problems at this initial point, you need to research and find a fix before moving on\n",
    "1. Remember that your version control can really only occur from the local side, where your TortoiseGit is installed. So, while changes may occur either in the remote or local repository, the control to keep things in sync will occur from the local side (TortoiseGit)\n",
    "1. Whether at the remote or local repository, make all needed changes there, including deleting files, adding files, or modifying files. Then, **commit** those changes to the repository at hand (local or remote). (On TortoiseGit locally, this is done via the 'Add' Explorer menu option for new files; use the 'Check for modifications' option for changed files.) Commitment to the repository at hand is needed before the version control system knows what has been formally modified\n",
    "1. Again, be cognizant of where the modifications have occurred, which in any case you will control from the local TortoiseGit. If the changes have been made locally, then 'push' those changes to the remote repository; if the changes have been made remotely, then 'pull' those changes changes back to the local. \n",
    "\n",
    "Always make sure that as any changes are made, at either side, they are synced to the system. In this way, you can be assured that your version control system is in a stable state, and you are free to make changes on either the local or remote side. Also know you can use GitHub for keeping multiple local instances (a desktop and a laptop in my case) in sync with the remote repository. Simply follow the above guidelines for each instance.\n",
    "\n",
    "#### Handling Styles (CSS)\n",
    "If you recall the discussion in [**CWPK #15**](https://www.mkbergman.com/2343/cwpk-15-using-notebooks-for-cwpk-documentation/), there is a difference in where custom styles can be set when viewing notebook pages locally versus whether they are called up directly from Python. Now, as we move to an online expression of these notebooks, we again raise the question of where online custom styles can be invoked.\n",
    "\n",
    "Perhaps in some expressions, where style overrrides can be invoked is a matter of little consequence. But this **CWPK** series has some specific styles in such things as pointing to warnings, pointing to online resources, etc. Having a consistent way to refer to these styles (presentation) means better efficiency. \n",
    "\n",
    "My hope had been that with MyBinder we had some identifiable means for providing such <code>custom.css</code> overrides as well. Though I can see links to such in page views, and there are hints online for how to actually modify styles, I was unable to find any means for effectively doing so. My suspicion is that online interactivity such as MyBinder is still in its infancy, and the degree of control that we expect in either local or remote environments is not yet mature.\n",
    "\n",
    "Thus, since I could find no way after many frustrating hours to provide my own specific styles, I had to make the reluctant decision to embed all such style changes in each individual notebook page. (What this means, effectively, is the specific statement of style attributes needs to be repeated each time as used [MyBinder does not support referring to a style name in a separate external file, which is the more efficient alternative].) This embedded approach is not efficient, but, like prior discussions about the use of relative addresses, sometimes being specific is the best way to ensure consistent treatment across environments.\n",
    "\n",
    "### Using MyBinder\n",
    "As time has gone on, I now have learned a general workflow that reflects these realities. In general, thus, with each new **CWPK** installment I try to:\n",
    "\n",
    "1. Draft all material in the Jupyter Notebook; make sure that version embodies all desired content and style changes and updates\n",
    "1. Inspect all link references and style definitions to make sure they are absolute, and not referenced\n",
    "1. Make sure all external files and images are moved and stored on the repository systems\n",
    "1. Post the updated file to the its current repository, and then commit it\n",
    "1. Push the updated file to the remote (or local) repository\n",
    "1. Convert the <code>*.ipynb</code> to HTML and post on my local blog\n",
    "1. Mix and stir again. \n",
    "\n",
    "Though it is not designed directly as such, it is also possible to analyze use of [MyBinder](https://archive.analytics.mybinder.org/) and gain [statistics of use](https://prometheus.io/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Documentation\n",
    "- A basic [Binder tutorial](https://github.com/alan-turing-institute/the-turing-way/blob/master/workshops/boost-research-reproducibility-binder/workshop-presentations/zero-to-binder-python.md).\n",
    "\n",
    "\n",
    " <div style=\"background-color:#ffecec; border:1px dotted #f5aca6; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "  <span style=\"font-weight: bold;\">NOTE:</span> This article is part of the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\" style=\"font-style: italic;\">Cooking with Python and KBpedia</a> series. See the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\"><strong>CWPK</strong> listing</a> for other articles in the series. <a href=\"http://kbpedia.org/\">KBpedia</a> has its own Web site. The <em>cowpoke</em> Python <a href=\"https://github.com/Cognonto/cowpoke\">code listing covering the series</a> is also available from GitHub.\n",
    "  </div>\n",
    "\n",
    "<div style=\"background-color:#ebf8e2; border:1px dotted #71c837; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "\n",
    "<span style=\"font-weight: bold;\">NOTE:</span> This <strong>CWPK \n",
    "installment</strong> is available both as an online interactive\n",
    "file <a href=\"https://mybinder.org/v2/gh/Cognonto/CWPK/master\" ><img src=\"https://mybinder.org/badge_logo.svg\" style=\"display:inline-block; vertical-align: middle;\" /></a> or as a <a href=\"https://github.com/Cognonto/CWPK\" title=\"CWPK notebook\" alt=\"CWPK notebook\">direct download</a> to use locally. Make sure and pick the correct installment number. For the online interactive option, pick the <code>*.ipynb</code> file. It may take a bit of time for the interactive option to load.</div>\n",
    "\n",
    "<div style=\"background-color:#feeedc; border:1px dotted #f7941d; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "<div style=\"float: left; margin-right: 5px;\"><img src=\"http://kbpedia.org/cwpk-files/warning.png\" title=\"Caution!\" width=\"32\" /></div>I am at best an amateur with Python. There are likely more efficient methods for coding these steps than what I provide. I encourage you to experiment -- which is part of the fun of Python -- and to <a href=\"mailto:mike@mkbergman.com\">notify me</a> should you make improvements.    \n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}