SuaveDispatch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1><span style=\"color:red\">Welcome to the SuAVE Jupyter Notebook Server</span></h1>\n",
    "\n",
    "This is the Jupyter Notebook Dispatcher module of the SuAVE platform. This environment enables you to write and execute Python scripts to process and analyze data in SuAVE surveys and image galleries. In most included scripts, the derived data (secondary variables, image characteristics, predictive labels, etc.) can be added to SuAVE surveys for visual analysis.  \n",
    "\n",
    "Look several cells below for the types of operations supported by your selected JupyterHub.\n",
    "\n",
    "You can execute cells in sequence (by clicking 'Run', or pressing Shift-Enter). From this module (\"dispatcher\") you can launch other notebooks to perform computations, image processing,  modeling and statistical tasks.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example call parameters:\n",
    "SuaveDispatch.ipynb?surveyurl=https://suave-dev.sdsc.edu/main/file=zaslavsk_WBDH_Participants.csv&views=1110101&view=grid&user=zaslavsk&csv=zaslavsk_WBDH_Participants.csv&params=none&dzc=https://maxim.ucsd.edu/dzgen/lib-staging-uploads/063044157874ce7d5841c906fa6d8073/content.dzc&activeobject=null\n",
    "\n",
    "(one numeric, one incorrect numeric, no images)\n",
    "\n",
    "or\n",
    "\n",
    "?surveyurl=http://suave-dev.sdsc.edu/main/file=zaslavsk_Cyclops_Cave_Ceramic_Petrography.csv&views=1110101&view=grid&user=zaslavsk&csv=zaslavsk_Cyclops_Cave_Ceramic_Petrography.csv&params=none&dzc=https://maxim.ucsd.edu/dzgen/lib-staging-uploads/40fe2e6d801375611334d3fadd9b701c/content.dzc&activeobject=null\n",
    "\n",
    "(252 images, numeric and categorical, suave-dev) \n",
    "\n",
    "\n",
    "?surveyurl=http://suave-dev.sdsc.edu/main/file=zaslavsk_Kommos_Ceramic_Petrography.csv&views=1110101&view=grid&user=zaslavsk&csv=zaslavsk_Kommos_Ceramic_Petrography.csv&params=none&dzc=https://maxim.ucsd.edu/dzgen/lib-staging-uploads/4d862392cf30aa7d520443c52b0599d2/content.dzc&activeobject=null\n",
    "\n",
    "(352 images)\n",
    "\n",
    "or\n",
    "\n",
    "?user=zaslavsk&csv=zaslavsk_Cyclops_Cave.csv&params=none&dzc=https://maxim.ucsd.edu/dzgen/lib-staging-uploads/40fe2e6d801375611334d3fadd9b701c/content.dzc&activeobject=null&surveyurl=http://suave2.sdsc.edu/main/file=zaslavsk_Cyclops_Cave.csv&view=grid\n",
    "\n",
    "(252 images, numeric and categorical, suave2) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Accepting survey parameters from SuAVE \n",
    "\n",
    "The following cell will retrieve survey parameters from the URL. Execute it to continue "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "function getQueryStringValue (key)\n",
       "{  \n",
       "    return unescape(window.location.search.replace(new RegExp(\"^(?:.*[&\\\\?]\" + escape(key).replace(/[\\.\\+\\*]/g, \"\\\\$&\") + \"(?:\\\\=([^&]*))?)?.*$\", \"i\"), \"$1\"));\n",
       "}\n",
       "IPython.notebook.kernel.execute(\"survey_url='\".concat(getQueryStringValue(\"surveyurl\")).concat(\"'\"));\n",
       "IPython.notebook.kernel.execute(\"views='\".concat(getQueryStringValue(\"views\")).concat(\"'\"));\n",
       "IPython.notebook.kernel.execute(\"view='\".concat(getQueryStringValue(\"view\")).concat(\"'\"));\n",
       "IPython.notebook.kernel.execute(\"user='\".concat(getQueryStringValue(\"user\")).concat(\"'\"));\n",
       "IPython.notebook.kernel.execute(\"csv_file='\".concat(getQueryStringValue(\"csv\")).concat(\"'\")); \n",
       "IPython.notebook.kernel.execute(\"dzc_file='\".concat(getQueryStringValue(\"dzc\")).concat(\"'\")); \n",
       "IPython.notebook.kernel.execute(\"params='\".concat(getQueryStringValue(\"params\")).concat(\"'\")); \n",
       "IPython.notebook.kernel.execute(\"active_object='\".concat(getQueryStringValue(\"activeobject\")).concat(\"'\")); \n",
       "IPython.notebook.kernel.execute(\"full_notebook_url='\" + window.location + \"'\"); \n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%javascript\n",
    "function getQueryStringValue (key)\n",
    "{  \n",
    "    return unescape(window.location.search.replace(new RegExp(\"^(?:.*[&\\\\?]\" + escape(key).replace(/[\\.\\+\\*]/g, \"\\\\$&\") + \"(?:\\\\=([^&]*))?)?.*$\", \"i\"), \"$1\"));\n",
    "}\n",
    "IPython.notebook.kernel.execute(\"survey_url='\".concat(getQueryStringValue(\"surveyurl\")).concat(\"'\"));\n",
    "IPython.notebook.kernel.execute(\"views='\".concat(getQueryStringValue(\"views\")).concat(\"'\"));\n",
    "IPython.notebook.kernel.execute(\"view='\".concat(getQueryStringValue(\"view\")).concat(\"'\"));\n",
    "IPython.notebook.kernel.execute(\"user='\".concat(getQueryStringValue(\"user\")).concat(\"'\"));\n",
    "IPython.notebook.kernel.execute(\"csv_file='\".concat(getQueryStringValue(\"csv\")).concat(\"'\")); \n",
    "IPython.notebook.kernel.execute(\"dzc_file='\".concat(getQueryStringValue(\"dzc\")).concat(\"'\")); \n",
    "IPython.notebook.kernel.execute(\"params='\".concat(getQueryStringValue(\"params\")).concat(\"'\")); \n",
    "IPython.notebook.kernel.execute(\"active_object='\".concat(getQueryStringValue(\"activeobject\")).concat(\"'\")); \n",
    "IPython.notebook.kernel.execute(\"full_notebook_url='\" + window.location + \"'\"); "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Check if the passed parameters are correct "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<b><span style='color:red'>Verify survey parameters: </span></b>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Base Survey URL:  https://suave-net.sdsc.edu/main/file=joeykaminsky2_Tester_Netvis_2.csv\n",
      "Enabled Views:  \n",
      "Default View:  grid\n",
      "User ID:  joeykaminsky2\n",
      "Additional Parameters:  \n",
      "Data File:  joeykaminsky2_Tester_Netvis_2.csv\n",
      "Image Tile Collection URL:  https://dzgen.sdsc.edu/dzgen/lib-staging-uploads/6eb3af87e3c855ed01cdaad5591b4722/content.dzc\n",
      "Active Object:  null\n",
      "Jupyter Hub URL:  https://jupyter-suave.nrp-nautilus.io/user/jkaminsky@ucsd.edu/notebooks/jupyter-suave\n",
      "Local Tile Collection Path :  https://dzgen.sdsc.edu/dzgen/lib-staging-uploads/6eb3af87e3c855ed01cdaad5591b4722/content.dzc\n",
      "Full-size Image Path:  https://dzgen.sdsc.edu/dzgen/lib-staging-uploads/6eb3af87e3c855ed01cdaad5591b4722/full_images/\n",
      "Full-size Images Available\n"
     ]
    }
   ],
      "source": [
    "# Check if the parameters are correct\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "from urllib.parse import urljoin\n",
    "import webbrowser\n",
    "import ntpath\n",
    "import os\n",
    "from IPython.display import Markdown, display\n",
    "def printmd(string):\n",
    "    display(Markdown(string))\n",
    "\n",
    "url_partitioned = full_notebook_url.partition('/SuaveDispatch')\n",
    "base_url = url_partitioned[0];\n",
    "images_available = False\n",
    "\n",
    "\n",
    "if dzc_file == \"undefined\":\n",
    "    dzc_file = \"\"\n",
    "    localdzc = \"\" \n",
    "    full_images = \"full images not available on NFS storage\"\n",
    "if len(dzc_file) > 20:\n",
    "    if \"lib-staging-uploads\" in dzc_file:\n",
    "        localdzc = dzc_file.replace(\"https://maxim.ucsd.edu/dzgen/lib-staging-uploads\",\"/lib-nfs/dzgen\")\n",
    "        full_images = localdzc.replace(\"/content.dzc\",\"/full_images/\")\n",
    "    else:\n",
    "        localdzc = \"dzc not available on NFS storage\"\n",
    "        full_images = \"full images not available on NFS storage\"\n",
    "        images_available = False\n",
    "else:\n",
    "    localdzc = \"dzc not available on NFS storage\"\n",
    "    full_images = \"full images not available on NFS storage\"\n",
    "    images_available = False\n",
    "\n",
    "no_images = False\n",
    "try:\n",
    "    image_urls = []\n",
    "    response = requests.get(full_images)\n",
    "    soup = BeautifulSoup(response.text, 'html.parser')\n",
    "    image_links = soup.find_all('a', href=True)\n",
    "\n",
    "    for link in image_links:\n",
    "        image_url = link['href']\n",
    "\n",
    "        if not image_url.startswith(('http:', 'https:')):\n",
    "            full_image_url = urljoin(full_images, image_url)\n",
    "        else:\n",
    "            full_image_url = image_url\n",
    "\n",
    "        image_urls.append(full_image_url)\n",
    "\n",
    "except:\n",
    "    print(\"Image Scraping Issue\")\n",
    "    no_images = True\n",
    "    \n",
    "\n",
    "printmd(\"<b><span style='color:red'>Verify survey parameters: </span></b>\")\n",
    "\n",
    "print(\"Base Survey URL: \", survey_url)\n",
    "print(\"Enabled Views: \", views)\n",
    "print(\"Default View: \", view)\n",
    "print(\"User ID: \", user)\n",
    "print(\"Additional Parameters: \", params)\n",
    "print(\"Data File: \", csv_file)\n",
    "print(\"Image Tile Collection URL: \", dzc_file)\n",
    "print(\"Active Object: \", active_object)\n",
    "print(\"Jupyter Hub URL: \", base_url)\n",
    "print(\"Local Tile Collection Path : \", localdzc)\n",
    "print(\"Full-size Image Path: \", full_images)\n",
    "if len(image_urls) != 0 or no_images == False:\n",
    "    print(\"Full-size Images Available\")\n",
    "else:\n",
    "    print(\"Full-size Images Not Available\")\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2><span style=\"color:red\">2a. Optional: load notebooks from a different github repository</span></h2>\n",
    "\n",
    "<span style=\"color:red\">Skip this cell if you are already running notebooks from your repo  - don't clone you repo the second time!</span> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'myclone'...\n",
      "remote: Enumerating objects: 39, done.\u001b[K\n",
      "remote: Counting objects: 100% (39/39), done.\u001b[K\n",
      "remote: Compressing objects: 100% (27/27), done.\u001b[K\n",
      "remote: Total 39 (delta 9), reused 39 (delta 9), pack-reused 0\u001b[K\n",
      "Unpacking objects: 100% (39/39), 317.01 KiB | 1.80 MiB/s, done.\n",
      "Click the URL to open the new dispatcher :  https://jupyter-suave.nrp-nautilus.io/user/jkaminsky@ucsd.edu/notebooks/jupyter-suave/myclone/SuaveDispatch.ipynb?surveyurl=https://suave-net.sdsc.edu/main/file=joeykaminsky2_Tester_Netvis_2.csv&views=&view=grid&user=joeykaminsky2&csv=joeykaminsky2_Tester_Netvis_2.csv&dzc=https://dzgen.sdsc.edu/dzgen/lib-staging-uploads/6eb3af87e3c855ed01cdaad5591b4722/content.dzc&activeobject=null\n"
     ]
    }
   ],
   "source": [
    "!rm -rf myclone\n",
    "!git clone --depth 1 \"https://github.com/izaslavsky/jupyter-suave.git\" myclone\n",
    "url1 = ('{base_url}/SuaveDispatch.ipynb?'+'surveyurl=' + survey_url + '&' + 'views=' + views + '&' 'view=' + view + '&' + 'user='+user+'&'+'csv='+csv_file+'&'+'dzc='+dzc_file+\"&\"+'activeobject='+active_object).format(base_url=base_url +\"/myclone\")\n",
    "print('Click the URL to open the new dispatcher : ', url1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Retrieve the survey file for processing\n",
    "\n",
    "As before, Shift-Enter to run this cell and continue to the next one. This merely prepares the data for subsequent processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<b><span style='color:red'>Survey file retrieved. Run next cell to continue.</span></b>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import urllib3\n",
    "urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)\n",
    "\n",
    "csv_url = survey_url.split(\"main\")[0] + \"surveys/\" + csv_file\n",
    "\n",
    "# get the survey data file\n",
    "http = urllib3.PoolManager()\n",
    "r = http.request('GET', csv_url, preload_content=False)\n",
    "\n",
    "# place the file into temp_csvs \n",
    "path = \"temp_csvs/\" + csv_file \n",
    "with open(path, 'wb') as out:\n",
    "    while True:\n",
    "        data = r.read(1024)\n",
    "        if not data:\n",
    "            break\n",
    "        out.write(data)\n",
    "\n",
    "r.release_conn()\n",
    "printmd(\"<b><span style='color:red'>Survey file retrieved. Run next cell to continue.</span></b>\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Now, select a notebook to do some work\n",
    "\n",
    "Select a notebook, then continue to the next cell. Note that you will see only those operations that are supported on your selected hub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e1e2e0636f154ad5b04d4cee96e02dcc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(Dropdown(description='notebooks_menu', options=('1. Arithmetic Operations', '2. Descript…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<b><span style='color:red'>Select a Jupyter notebook and then run next cell</span></b>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "from ipywidgets import interact, interactive, fixed, interact_manual\n",
    "import ipywidgets as widgets\n",
    "import json\n",
    "\n",
    "from collections import OrderedDict\n",
    "\n",
    "nb_menu = OrderedDict()\n",
    "nb_menu_counter = 1\n",
    "\n",
    "menulist = [\n",
    "('Arithmetic Operations','arithmetic/SuaveArithmetic.ipynb','any'),\n",
    "('Descriptive Statistics','stats/DescriptiveStats.ipynb','any'),\n",
    "('Generate Contingency Tables','stats/Generate_Contingency_Tables.ipynb','any'),\n",
    "('Generate Factor Contributions','stats/Generate_Factor_Contributions.ipynb','any'),\n",
    "('Named Entity Recognition','tagger/NER.ipynb','any'),\n",
    "('Color Statistics','colors/ColorStats.ipynb','image'),\n",
    "('Classify Images','classify/ImageClassify.ipynb','image'),\n",
    "('Generate LeNet CNN Model v2','predict/PredictiveModel_v2.ipynb','image'),\n",
    "('Extend LeNet CNN Model','predict/ExtendModel.ipynb','image'),\n",
    "('Generate SVM Model','svm/SVMPredictiveModel.ipynb','image'),\n",
    "('Generate Aggregate Maps', 'maps/Generate_Aggregate_Maps_Suave.ipynb','any'),\n",
    "('Extend SVM Model','svm/ExtendSVM.ipynb','image'),\n",
    "('Generate SDG Dataset','SDG/GenerateSDGDataset.ipynb','largedataset'),\n",
    "('Explore with Holoviz','holoviz/holoviz.ipynb','any'),\n",
    "('Enhance Dataset','wrangling/qualgeoimage.ipynb','any'),\n",
    "('Knowledge Graph Query','kg/kg_query.ipynb','any'),\n",
    "('Annotate with NEMO','nemo/suave_nemo.ipynb','any'),\n",
    "('Transfer Learning','transfer_learning/transfer_learning.ipynb','image'),\n",
    "('Spatial Statistics','spatialstats/SpatialStats.ipynb','any'),\n",
    "('Network Analysis','networks/networks.ipynb','netvis')\n",
    "\n",
    "]\n",
    "netvis_url = None\n",
    "if len(image_urls) == 0:\n",
    "    print(\"ATTENTION!  This hub supports image-based processing, but full-size images are not available for this survey. \\n Full-size image operations are not available from this menu.\\n Please check that your dzgen link is correct.\\n If the dzgen link is correct and you are still having issues, please contact the admin at zaslavsk@sdsc.edu to re-generate images from image tiles.\")\n",
    "\n",
    "# For a setup where dzc's are only on NFS share, use this:\n",
    "\n",
    "for label, nb, nbtype in menulist:\n",
    "    if nbtype == 'any':\n",
    "        nb_menu[str(nb_menu_counter) + '. ' + label] = nb\n",
    "        nb_menu_counter +=1\n",
    "    elif nbtype == 'image':\n",
    "        if len(image_urls) != 0:\n",
    "            nb_menu[str(nb_menu_counter) + '. ' + label] = nb\n",
    "            nb_menu_counter +=1\n",
    "    elif nbtype == 'largedataset':\n",
    "        if os.path.isdir('/lib-nfs/largedatasets'):\n",
    "            nb_menu[str(nb_menu_counter) + '. ' + label] = nb\n",
    "            nb_menu_counter +=1\n",
    "    elif nbtype == 'netvis':\n",
    "        to_split = survey_url.split(\"file=\")[1].split(\".csv\")[0]\n",
    "        first = to_split.split(\"_\")[0]\n",
    "        second = \"_\".join(to_split.split(\"_\")[1:])\n",
    "\n",
    "        json_url = F\"https://suave-net.sdsc.edu/getSurveyDzc?user={first}&file={second}\"\n",
    "        request = requests.get(json_url)\n",
    "        json_file = json.loads(request.text)\n",
    "        if 'netvis' in json_file and json_file['netvis'] and 'url' in json_file['netvis'][0]:\n",
    "            netvis_url = json_file['netvis'][0]['url']\n",
    "            if netvis_url != \"\":\n",
    "                nb_menu[str(nb_menu_counter) + '. ' + label] = nb\n",
    "                nb_menu_counter +=1\n",
    "\n",
    "def f(notebooks_menu):\n",
    "    return notebooks_menu\n",
    "out = interact(f, notebooks_menu=nb_menu.keys());\n",
    "\n",
    "printmd(\"<b><span style='color:red'>Select a Jupyter notebook and then run next cell</span></b>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Open the selected notebook and pass survey parameters to it\n",
    "\n",
    "Once the URL is for the next notebook is constructed, click that URL to open it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "<b><span style='color:red'>Click the URL to open the selected notebook:</span></b>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://jupyter-suave.nrp-nautilus.io/user/jkaminsky@ucsd.edu/notebooks/jupyter-suave/operations/networks/networks.ipynb?surveyurl=https://suave-net.sdsc.edu/main/file=joeykaminsky2_Tester_Netvis_2.csv&views=&view=grid&user=joeykaminsky2&csv=joeykaminsky2_Tester_Netvis_2.csv&dzc=https://dzgen.sdsc.edu/dzgen/lib-staging-uploads/6eb3af87e3c855ed01cdaad5591b4722/content.dzc&activeobject=null\n"
     ]
    }
   ],
   "source": [
    "chosen_nb_name = nb_menu[out.widget.result]\n",
    "import webbrowser\n",
    "url1 = ('{base_url}/operations/{nb_name}?'+'surveyurl=' + survey_url + '&' + 'views=' + views + '&' 'view=' + view + '&' + 'user='+user+'&'+'csv='+csv_file+'&'+'dzc='+dzc_file+\"&\"+'activeobject='+active_object).format(base_url=base_url, nb_name=chosen_nb_name)\n",
    "\n",
    "printmd(\"<b><span style='color:red'>Click the URL to open the selected notebook:</span></b>\")\n",
    "print(url1)\n",
    "\n",
    "# webbrowser.open(url1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {
    "height": "286px",
    "width": "578px"
   },
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}