NSIDCLoginNotebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Cookie File and .netrc for NSDIC Login"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**If running from binder, this notebook can be skipped. It is only need to setup installation to run the notebooks locally and to setup QGIS for remote access.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook Purpose"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running this Notebook will create a cookiefile and create or append to an existing `~/.netrc` for use by [*gdal*](https://gdal.org/) and [*QGIS*](https://qgis.org/en/site/) to remotely access [GrIMP](https://nsidc.org/data/measures/grimp) files archived at [NSIDC](https://nsidc.org/). \n",
    "\n",
    "The cookie file should work indefinitely (unless your passwd changes). It can be refreshed by running this notebook again.\n",
    "\n",
    "Before attempting to run this notebook, users should have an [Earth Data Login](https://urs.earthdata.nasa.gov/), which requires a free registration.\n",
    "\n",
    "This notebook also constains instructions for the installation of the packages to run other GrIMP notebooks: [qgisRemoteNotebook](https://github.com/fastice/GrIMPNotebooks/blob/master/qgisRemoteNotebook.ipynb) and [GrIMPSubsetterNotebook](https://github.com/fastice/GrIMPNotebooks/blob/master/GrIMPSubsetterNotebook.ipynb)\n",
    "\n",
    "_**Once these steps are complete, users will be able to work with large volumes of [GrIMP](https://nsidc.org/data/measures/grimp) products (>TB).**_ In particular, users will be able to:\n",
    "* After setup and login, visualize GrIMP data in [*QGIS*](https://qgis.org/en/site/) (ARC-GIS opensource alternative) projects over even relatively slow network connections. Sample QGIS projects are located [here](https://github.com/fastice/GrIMPQGISProjects)\n",
    "* After setup, search for GrIMP products and create cutomized QGIS projects from the search results ([qgisRemoteNotebook](https://github.com/fastice/GrIMPNotebooks/blob/master/qgisRemoteNotebook.ipynb)).\n",
    "* After setup, subset GrIMP data in time and space to create regional time series (e.g., assemble all the monthly data for a particular glaciers and store the data in a single netCDF file)([GrIMPSubsetterNotebook](https://github.com/fastice/GrIMPNotebooks/blob/master/GrIMPSubsetterNotebook.ipynb)).\n",
    "\n",
    "Prior to performing any of these tasks, however, this use this notebook to a) download essential software needed by these notebooks, and b) setup the authetication files needed to run these other notebooks and programs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Started"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following instructions assume the presence of several python packages. The instructions below have been tested with a [conda](https://www.anaconda.com/products/individual) (Anaconda) install, which is the recommended approach especially for novice python users.  The conda-specific instructions should contain sufficient information for users proficient with other package managers (e.g., pip) to get this notebook up and running."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conda Install"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For users not already using [conda](https://www.anaconda.com/products/individual), this package manager can be downloaded and installed by following these [instructions](https://conda.io/projects/conda/en/latest/user-guide/install/index.html#regular-installation). Unfortunately, some of the standard anaconda releases do not work well with this notebook so it is _**strongly**_ recomended that all packages be downloaded from the [conda-forge](https://conda-forge.org/) channel. Because, mixing conda-forge with the regular conda packages can cause some package conflict issues, the cleanest approach is to create a new conda environment. (For those unfamiliar with conda, this means creating an environment that when activated runs a completely independent set of code). Using a seperate enviroment will both help ensure success in running this notebook as well avoiding problems with pre-existing conda environments (users with a conda-forge priority environment can probably use their existing environment).\n",
    "\n",
    "The `GrIMPNotebooks/binder` repository contains a file, [environment.yml](https://github.com/fastice/GrIMPNotebooks/blob/master/binder/environment.yml) that will create a new environment called `greenlandMapping` by executing the command: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`conda env create -f environment.yml`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This command creates a new a new environment and downloads the listed packages and their dependencies, totalling about 1GB. Some of these packages are not required for this notebook, but will be required by other GrIMP notebooks. Everytime this notebook is run, activate the environment via:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`conda activate greenlandMapping` (Note the name `greenlandMapping` can be changed prior to creating the evironment by editing the `environment.yml` file.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Juptyer lab can then be run from the command line (ideally from the directory where the notebook is located) as:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`jupyter lab NISDCLoginNotebook.ipynb`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If *jupyter lab* is run as an app, before opening it for the first time run the following procedure, which will make the enviroment accessible to jupyter lab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`python -m ipykernel install --user --name=greenlandMapping`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running this command, the kernel should be selectable from the upper right corner of the tool bar (it may be necessary to restart *jupyter*). This procedure only needs to run once."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trouble Shooting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above conda installation should ensure that the version of jupyter lab is >= 3.0. If an earlier version of jupyter lab is used, then the following extensions need to be enabled:\n",
    "* `jupyter labextension install @pyviz/jupyterlab_pyviz`\n",
    "* `jupyter labextension install @jupyter-widgets/jupyterlab-manager`\n",
    "* `jupyter labextension install @jupyterlab/server-proxy`\n",
    "\n",
    "Note without the extendsions nothing may appear broken(i.e., no error message), but the Notebook likely will not function correctly (i.e., there will be no response when buttons are pushed)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This next cell will import everything need for the login. If something above was not installed properly, its likely that an error will occur at this stage. If so, read the error message to assess which package is missing. If the problem is with `grimpfunc` revisit these [instructions](https://github.com/fastice/grimpfunc), otherwise check the Python/Conda [installation](#Conda-Install)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import grimpfunc.NASALogin as NASALogin\n",
    "import panel\n",
    "import os\n",
    "panel.extension()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the Login Cookie and update/create .netrc File"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section peforms the main task of setting up NSIDC authentication."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What this Process Will Do"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many of the GrIMP notebooks and QGIS projects use remote links to data archived at NSIDC. When a progam (e.g., *QGIS*) or a Notebook accesses these data it needs to authenticate via the user's [Earth Data Login](https://urs.earthdata.nasa.gov/). This Notebook will set up that authentication so that the process happens automatically when a Notebook or QGIS accesses those files. In particular, it will create a cookie file (`.grimp_download_cookiejar.txt`) in the user's home directory that [*GDAL*](https://gdal.org/) (one of the installed packages) will use to redirect the login appropriately for remote file access. It also creates a new or appends to an existing `.netrc` file in the user's home directory to include the login id and password (see [security risks](#Potential-Security-Risks) below). In principle, this notebook should only need to be run once to set thes files up, but can be rerun if needed. Further steps outlined below are required for programs to use this information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a valid cookie file and .netrc file already exists, this step will print a simple text message saying so and no further action is needed. If not, a widget will allow entry of NASA Earth Data credentials. If the login is successful, the status message will turn blue to indicate success. An orange message indicates the cookie file exists, but login credentials are needed for the .netrc file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "myLogin = NASALogin()\n",
    "#myLogin.resetCookie() # Uncomment to delete prior cookie (e.g., if expired or corrupted)\n",
    "myLogin.view()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Troubleshooting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If it appears that the credentials were entered correctly, but the status message does not indicate so, run the following cell, which will indicate login success as `True` and failure as `False`. It also checks whether a `.netrc` file exists. If still having problems, check the contents of the `.netrc` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Login status: {myLogin.check_cookie()}')\n",
    "print(f'.netrc exists: {os.path.exists(os.path.expanduser(\"~/.netrc\"))}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Potential Security Risks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.netrc` file that is created as part of this process is a potential security risk as it saves your username/password (only NASA Earth Data Login in this case) as clear text. The file is created as user only access (-rw-------), which prevents other users on the system from accessing it. If your acount is reasonably secure and you are using a password **_unique_** to this login, then this may be a reasonable risk (i.e, only your NASA Earth Data account could be compromised). If you are concerned, delete the `~/.netrc` after each use and then regenerate each time as needed with this notebook. At some point in the next year NASA may upgrade to a more secure [token based access](https://discourse.pangeo.io/t/earthdata-password-pop-up-box/1358/6)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## QGIS Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The steps above should have created the necessary files to use QGIS external to this notebook. Some additional steps are now needed to tell QGIS where to find these files. Once complete, QGIS should be able to access project files "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cookie Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a QGIS project includes remote */vsicurl* links to remote products at NSIDC GDAL needs to authenticate using the cookie file just described. So QGIS needs to know how to find it. Depending on how QGIS starts, it **_may_** read these directly from the users enviroment variables. In which case all the user has to do is specify them. For example,\n",
    "* `export GDAL_HTTP_COOKIEFILE=~/.grimp_download_cookiejar.txt` for bash or `setenv GDAL_HTTP_COOKIEFILE ~/.grimp_download_cookiejar.txt` for csh/tcsh.\n",
    "* `export GDAL_HTTP_COOKIEJAR=~/.grimp_download_cookiejar.txt` for bash or `setenv GDAL_HTTP_COOKIEJAR ~/.grimp_download_cookiejar.txt` for csh/tcsh.\n",
    "\n",
    "**_When run as an app (e.g., not from the commandline on a Mac or Pc), in many cases these variables should be set in QGIS using_** `Settings->Options->System->Environment` - see https://github.com/fastice/GrIMPNotebooks/blob/master/QGISVariableSettings.png.\n",
    "\n",
    "Note at least with conda installed *QGIS* called from an Ubuntu command line, the data load much quicker when the cookie is specified with user specified environment variables (as opposed to inside QGIS). As a result, it may be a good choice to select the `If Undefined` option under the Apply column when you enter the variables in QGIS. This will cause to use the system variables as the first and fastest option when they are available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample QGIS Projects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When QGIS opens a project with local or remote data, it opens every file to read the meta data. While fairly quick if the data are local, this process can take anywhere from a few seconds to several minutes depeinding on a) network bandwidth, and b) the number of the files in the project. Thus, as a first attempt it is best to try loading this project first ([QGISImageTest.qgs](https://github.com/fastice/GrIMPQGISProjects/blob/master/QGISImageTest.qgs)), since it only has five images. If it does not open with a couple of minutes, there is probably an authentication issue. If the project loads successfully, try [QGISImageMosaics.qgs](https://github.com/fastice/GrIMPQGISProjects/blob/master/QGISImageMosaics.qgs), which has more 300 SAR image mosaics of the Greenland coast at 25-m resolution. Both of these projects were generated automatically with [qgisRemoteNotebook](https://github.com/fastice/GrIMPNotebooks/blob/master/qgisRemoteNotebook.ipynb), which can create customized QGIS projects that mix velocity, image, and terminus position data.\n",
    "\n",
    "See **Performance Notes** in the [**qgisRemoteNotebook**](https://github.com/fastice/GrIMPNotebooks/blob/master/qgisRemoteNotebook.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### QGIS Troubleshooting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The setup procedure has been verified to work with 'QGIS' V3.16 and above. It did not work with at least one instance of V3.10. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In building the environment, a version of QGIS was installed, which while it can be opened via a command line call on Ubuntu 20.04 it does not appear to open properly on Mac Big Sur. Thus, some users will have greater success downloading and installing the app directly (it should also be more full featured than the conda version). Some of the notebooks, however, need to load the `QGIS` core libraries, which can be difficult to setup and import when QGIS is installed as an app. Thus, users should make sure the conda version of *QGIS* is also installed. Even if the conda version is not directly launch, the *QGIS* core routines will be imported in the other [notebooks](https://github.com/fastice/GrIMPNotebooks)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other Notebooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The login setup in this notebook is required for some of the other [notebooks](https://github.com/fastice/GrIMPNotebooks). Alternatively, these can be set as environment variables as described [above](#Cookie-Setup).\n",
    "\n",
    "By default the cookie file will reside in the users home directory with the name `.grimp_download_cookiejar.txt`. To point other notebooks to the cookie file from within a notebook, use a suitably modifed version of the command below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env = dict(GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/.grimp_download_cookiejar.txt'),\n",
    "            GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/.grimp_download_cookiejar.txt')) # Modify path as needed\n",
    "os.environ.update(env)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nisarCryoCalVal",
   "language": "python",
   "name": "nisarcryocalval"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}