JupyterHub_Integration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/jupyterhub_integration_header.png\">\n",
    "\n",
    "This tutorial demonstrates how to (a) pull Globus Auth tokens from the Jupyter Notebook Server environment, and (b) use those tokens to interact with different REST APIs secured with Globus Auth. The notebook implements a simple data flow. \n",
    "\n",
    "You launched this notebook using the Globus-enabled JupyterHub environment, so the following have already happened:\n",
    "1. You have established your identity by authenticating, with an institutional credential, ORCID, or similar\n",
    "1. You have granted consent to the issuance of tokens with certain scopes\n",
    "1. A notebook has been created, with access to those tokens\n",
    "\n",
    "<img src=\"img/jupyterhub_tokens.png\" alt=\"Steps followed prior to starting this notebook\" align=\"CENTER\" style=\"width: 85%;\"/>\n",
    "\n",
    "<strong>Notes</strong>\n",
    "1. You need to join the  [Tutorial Users Group](https://app.globus.org/groups/50b6a29c-63ac-11e4-8062-22000ab68755) in order to access the guest collection at the end of the tutorial.\n",
    "1. Our code here is pedantic, for clarity; much could be encapsulated in Python packages to simplify the notebook.\n",
    "1. Tokens are issued and stored in the JupyterHub database at login. They typically expire in 24 hours. As we do not provide a mechanism to handle refresh tokens, the simplest way to get new ones is to:\n",
    " - Stop your server (see the Control Panel)\n",
    " - Log out\n",
    " - Log back in\n",
    " - Start your server\n",
    " - Launch the notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# used to get the tokens\n",
    "import os\n",
    "import pickle\n",
    "import base64\n",
    "\n",
    "# much of what we deal with is JSON\n",
    "import json\n",
    "\n",
    "# we're going to make explicit HTTPS calls\n",
    "import requests\n",
    "\n",
    "# required to work with our example data\n",
    "import csv\n",
    "import datetime\n",
    "import matplotlib.pyplot as plt\n",
    "from io import StringIO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A. Get Tokens\n",
    "\n",
    "The Globus-enabled JupyterHub passes the tokens into the notebook environment `base64` encoded as a pickled Python dictionary assigned to the `GLOBUS_DATA` variable. We'll grab the variable and unpack it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get Globus Auth token data\n",
    "globus_token_data = os.getenv('GLOBUS_DATA')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now extract the pickled tokens\n",
    "pickled_tokens = base64.b64decode(globus_token_data)\n",
    "\n",
    "# Unpickle and get the dictionary\n",
    "tokens = pickle.loads(pickled_tokens)\n",
    "\n",
    "# Minimal sanity check, did we get the data type we expected?\n",
    "isinstance(tokens, dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introspect Tokens\n",
    "\n",
    "Depending on the JupyterHub configuration, there will be different numbers of tokens. For this tutorial, our identity token is a __[JSON Web Token (JWT)](https://tools.ietf.org/html/rfc7519)__. We also have tokens for different Resource Servers and scopes, including for retrieving our profile from Globus Auth; accessing the Eagle HTTPS server, and accessing the Globus Transfer service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(json.dumps(tokens, indent=4, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# B. Use Tokens\n",
    "\n",
    "Now we can talk to different servers. In this tutorial, we show how tokens can be passed as HTTP headers. Much of this can also be done with the __[Globus Python SDK](http://globus-sdk-python.readthedocs.io/en/stable/)__."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get User Information\n",
    "\n",
    "First, let's use our `auth.globus.org` token to get our __[OAuth2 user information](https://docs.globus.org/api/auth/reference/#get_or_post_v2_oauth2_userinfo_resource)__. We assemble the header with the appropriate access token, and do an HTTP `GET` on the resource."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# base URL for Globus Auth API\n",
    "auth_base_url = 'https://auth.globus.org/v2'\n",
    "\n",
    "# Create the header\n",
    "headers = {'Authorization':'Bearer '+ tokens['tokens']['auth.globus.org']['access_token']}\n",
    "\n",
    "# Get the user info as JSON\n",
    "user_info = requests.get(auth_base_url + '/oauth2/userinfo', headers=headers).json()\n",
    "\n",
    "# Look at the response\n",
    "print(json.dumps(user_info, indent=4, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Identity Information\n",
    "\n",
    "Using the __[Globus Auth API resource for identities](https://docs.globus.org/api/auth/reference/#v2_api_identities_resources)__, we perform a `GET` on a specific identity, our own, to examine its properties:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "identity = requests.get(auth_base_url + '/api/identities/' + user_info['sub'], headers=headers).json()\n",
    "print(json.dumps(identity, indent=4, sort_keys=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "# C. Implement a Simple Data Flow\n",
    "\n",
    "Our example below implements a simple, but common, data flow which includes the following steps, as illustrated below:\n",
    "\n",
    "1. Fetch some data from a remote location, in this case the Eagle data server at Argonne National Lab\n",
    "1. Plot the retrieved data\n",
    "1. Save the plot to a remote web server\n",
    "1. Share a link to data on the web server\n",
    "<img src=\"img/graph_plot_flow.png\" alt=\"Steps in the simple workflow\" align=\"LEFT\" style=\"width: 85%;\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Get CSV Data via HTTPS\n",
    "\n",
    "We replicate here some of the flow from the __[Modern Research Data Portal](https://mrdp.globus.org)__ design pattern and tutorial. In particular, we retrieve some weather data in a CSV file that has column names in the first row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GET the CSV from the publicly accessible HTTPS GCS collection\n",
    "url = 'https://g-fe1c1.fd635.8443.data.globus.org/portal/catalog/dataset_las/1952.csv'\n",
    "https_token = tokens['tokens']['a6f165fa-aee2-4fe5-95f3-97429c28bf82']['access_token']\n",
    "headers = {'X-Requested-With': 'XMLHttpRequest', 'Authorization': f'Bearer {https_token}'}\n",
    "resp = requests.get(url, headers=headers).text\n",
    "csv_rows = csv.DictReader(StringIO(resp))\n",
    "\n",
    "# inspect the header line\n",
    "print(','.join(csv_rows.fieldnames))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pull CSV data into lists for plotting\n",
    "dates = [] # from column DATE\n",
    "max_temps = [] # from column TMAX\n",
    "min_temps = [] # from column TMIN\n",
    "\n",
    "for row in csv_rows:\n",
    "    dates.append(datetime.date(int(row['DATE'][:4]), int(row['DATE'][4:6]), int(row['DATE'][6:])))\n",
    "    max_temps.append(int(row['TMIN']))\n",
    "    min_temps.append(int(row['TMAX']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Plot CSV Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we will save the generated plot to this file\n",
    "plot_filename = \"temp_plot.png\"\n",
    "\n",
    "# generate the plot\n",
    "plt.figure(figsize=(16,8))\n",
    "plt.plot(dates, min_temps, label = \"Min Temp\")\n",
    "plt.plot(dates, max_temps, label = \"Max Temp\")\n",
    "plt.xlabel('Date YYYY-MM')\n",
    "plt.ylabel('Temperature')\n",
    "plt.title('Maximum and Minimum Temperatures: ' + str(dates[0])[:4])\n",
    "plt.grid(True)\n",
    "plt.savefig(plot_filename)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Save Plot on Globus Collection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For these Globus Transfer API interactions, we use the Globus Transfer token. As the Globus Transfer API does not support __[`application/x-www-form-urlencoded`](https://docs.globus.org/api/transfer/overview/#document_formats)__ data, we are explicit about the JSON we pass.\n",
    "\n",
    "At this point, we will:\n",
    "\n",
    "1. Create a directory for our file\n",
    "1. `PUT` our plot there\n",
    "1. Generate a link to our plot and view it\n",
    "\n",
    "Note, the calls to Transfer can be made via the Globus SDK, with argument validation, etc. For this tutorial we're using a direct HTTP request for pedagogical purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# base URL for the Globus Transfer API\n",
    "transfer_base_url = 'https://transfer.api.globus.org/v0.10'\n",
    "\n",
    "# Globus Tutorials on ALCF Eagle\n",
    "collection_uuid = 'a6f165fa-aee2-4fe5-95f3-97429c28bf82'\n",
    "collection_url = 'https://g-fe1c1.fd635.8443.data.globus.org'\n",
    "collection_data_path = '/jupyter-integration/'\n",
    "\n",
    "# define headers for HTTP request\n",
    "headers = {'Authorization':'Bearer '+ tokens['tokens']['transfer.api.globus.org']['access_token'],\n",
    "          \"Content-Type\" : \"application/json\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get our username from JupyterHub (which includes a hash to avoid collision)\n",
    "username = os.getenv('JUPYTERHUB_USER')\n",
    "print(\"My user name is \" + username)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# call the Transfer API to make a directory\n",
    "mkdir_payload = {\"DATA_TYPE\": \"mkdir\", \"path\": collection_data_path + username}\n",
    "resp = requests.post(\n",
    "  transfer_base_url + '/endpoint/' + collection_uuid + '/mkdir',\n",
    "  headers=headers,\n",
    "  json=mkdir_payload)\n",
    "\n",
    "print(resp.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Upload Plot to a Guest Collection\n",
    "We now upload the image file to the Eagle data server at Argonne National Laboratory. This Globus collection supports HTTPS access, so we can `PUT` the plot image file there directly. If this was a large file (or many files) we might want to use a Globus Transfer request instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define headers for HTTP request\n",
    "headers = {'Authorization':'Bearer '+ tokens['tokens']['a6f165fa-aee2-4fe5-95f3-97429c28bf82']['access_token']}\n",
    "\n",
    "# pass file handle to the requests library for the PUT\n",
    "image_data = open(plot_filename, 'rb')\n",
    "\n",
    "# PUT the file to the guest collection\n",
    "resp = requests.put(collection_url + collection_data_path + username + '/' + plot_filename,\n",
    "                    headers=headers, data=image_data, allow_redirects=False)\n",
    "print(resp.status_code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Share Link to Plot File\n",
    "\n",
    "Let's look at a link to the file. Clicking the link will require you to authenticate to the Globus collection since your browser is a different client than this notebook server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(collection_url + collection_data_path + username + '/' + plot_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also look at the folders and permissions on the __[guest collection](https://app.globus.org/file-manager/collections/a6f165fa-aee2-4fe5-95f3-97429c28bf82)__."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}