GreatLakes_Dashboard.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Great Lakes Models Dashboard\n",
    "Visualizing ocean model data using the [pyviz](pyviz.org) tools. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.098658Z",
     "start_time": "2019-04-17T16:48:32.016320Z"
    }
   },
   "outputs": [],
   "source": [
    "import xarray as xr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.106138Z",
     "start_time": "2019-04-17T16:48:32.013Z"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "#url = 'https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/LOOFS/fmrc/Aggregated_7_day_LOOFS_Fields_Forecast_best.ncd'\n",
    "url = 'http://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/LOOFS/MODELS/201903/glofs.loofs.fields.forecast.20190320.t06z.nc'\n",
    "ds = xr.open_dataset(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find all the data variables that depend on time (and are not time `bounds`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.110736Z",
     "start_time": "2019-04-17T16:48:32.016Z"
    }
   },
   "outputs": [],
   "source": [
    "ds['temp']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.116976Z",
     "start_time": "2019-04-17T16:48:32.018Z"
    }
   },
   "outputs": [],
   "source": [
    "ds = ds.set_coords(names=['lon','lat'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.119592Z",
     "start_time": "2019-04-17T16:48:32.021Z"
    }
   },
   "outputs": [],
   "source": [
    "rho_vars = []\n",
    "for var in ds.data_vars:\n",
    "    if len(ds[var].dims) > 0:\n",
    "        if 'time' in ds[var].dims[0] and (not 'validtime' in var) and ('nx' in ds[var].dims[-1]):\n",
    "            rho_vars.append(var)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.122209Z",
     "start_time": "2019-04-17T16:48:32.023Z"
    }
   },
   "outputs": [],
   "source": [
    "rho_vars"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the [pyviz](http://pyviz.org) tools we need"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.124725Z",
     "start_time": "2019-04-17T16:48:32.028Z"
    }
   },
   "outputs": [],
   "source": [
    "from cartopy import crs as ccrs\n",
    "import hvplot.xarray\n",
    "import holoviews as hv\n",
    "import geoviews as gv\n",
    "from geoviews import tile_sources as gvts\n",
    "import panel as pn\n",
    "from holoviews import streams\n",
    "import numpy as np\n",
    "import bokeh\n",
    "import random\n",
    "from itertools import cycle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create widget for variable selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.126923Z",
     "start_time": "2019-04-17T16:48:32.031Z"
    }
   },
   "outputs": [],
   "source": [
    "var_select = pn.widgets.Select(name='Model Variables:', options=rho_vars, \n",
    "                               value='temp')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create widget for basemap selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.129373Z",
     "start_time": "2019-04-17T16:48:32.034Z"
    }
   },
   "outputs": [],
   "source": [
    "base_map_select = pn.widgets.Select(name='Basemap:', options=gvts.tile_sources, value=gvts.OSM)                                                    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The `plot` function below creates the `hvplot` panel layout object.  We specify a basemap, pick the `quadmesh` plot type for the selected variable, and indicate we want to `rasterize` the plot so that we can render massive meshes in the browser. We also specify the `groupby` parameter as the list of dimensions that remains after we remove Y and X: `ds[var].dims[:-2]`, which automatically handles variables with either dimensions [T, Y, X] or [T, Z, Y, X].  We also specify which `bokeh` controls we want to be active by default:  the `wheel_zoom` and `pan` controls.\n",
    "\n",
    "We also change the default slider to a selection widget for the `time` dimension so that specific times are easy to select.  See https://stackoverflow.com/a/54912917/2005869"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.133042Z",
     "start_time": "2019-04-17T16:48:32.036Z"
    }
   },
   "outputs": [],
   "source": [
    "crs = ccrs.PlateCarree()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a stream to get location of tap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.138428Z",
     "start_time": "2019-04-17T16:48:32.043Z"
    }
   },
   "outputs": [],
   "source": [
    "clear_time_series = hv.streams.Stream.define('Clear_ts', clear=False)(transient=True)\n",
    "clear_points = hv.streams.Stream.define('Clear_points', clear=False)(transient=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.135774Z",
     "start_time": "2019-04-17T16:48:32.038Z"
    }
   },
   "outputs": [],
   "source": [
    "tap_mesh = streams.Tap(transient=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "colors = ['#60fffc','#6da252','#ff60d4','#ff9400','#f4e322','#229cf4','#af9862','#629baf','#7eed5a','#05040c','#e29ec8','#ff4300',]\n",
    "color_pool = cycle(colors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tapped_locs = {'Longitude':[],'Latitude':[],'color':[]}\n",
    "tapped_map = gv.Points([])\n",
    "\n",
    "def record_taps(x,y,clear):\n",
    "    global tapped_map,tapped_locs\n",
    "    if clear:\n",
    "        tapped_locs = {'Longitude':[],'Latitude':[],'color':[]}\n",
    "        tapped_map = gv.Points(tapped_locs).opts()\n",
    "        \n",
    "    if None not in [x,y]:\n",
    "        tapped_locs['Longitude'].append(x),tapped_locs['Latitude'].append(y),tapped_locs['color'].append(next(iter(color_pool)));\n",
    "        tapped_map = gv.Points(tapped_locs, vdims=[ 'color']).opts(color='color', line_color='black', marker='s')\n",
    "    return tapped_map\n",
    "\n",
    "\n",
    "taps_dmap = hv.DynamicMap(record_taps, streams=[tap_mesh,clear_points]).opts(size=8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.142022Z",
     "start_time": "2019-04-17T16:48:32.046Z"
    }
   },
   "outputs": [],
   "source": [
    "def plot(var=None, base_map=None):\n",
    "    base_map = base_map or base_map_select.value\n",
    "    var = var or var_select.value\n",
    "    mesh = ds[var][-24:,:,:].hvplot.quadmesh(x='lon', y='lat', rasterize=True, title=var,\n",
    "                                    width=600, height=400, crs=crs,\n",
    "                                    groupby=list(ds[var].dims[:-2]), cmap='jet')\n",
    "    \n",
    "    tap_mesh.source = mesh # Provide source to the tap stream\n",
    "    \n",
    "    overlay = (base_map * mesh.opts(alpha=0.7)).opts(active_tools=['wheel_zoom', 'pan']) * taps_dmap\n",
    "    widgets = {dim: pn.widgets.Select for dim in ds[var].dims[:-2]}\n",
    "    return pn.pane.HoloViews(overlay, widgets=widgets).layout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.146120Z",
     "start_time": "2019-04-17T16:48:32.050Z"
    }
   },
   "outputs": [],
   "source": [
    "def on_var_select(event):\n",
    "    var = event.obj.value\n",
    "    dashboard[-1] = plot(var=var)\n",
    "    clear_button.clicks +=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.148879Z",
     "start_time": "2019-04-17T16:48:32.052Z"
    }
   },
   "outputs": [],
   "source": [
    "def on_base_map_select(event):\n",
    "    base_map = event.obj.value\n",
    "    dashboard[-1] = plot(base_map=base_map)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function converts `lat lon` to `nx ny` for selection of data points from dataset. It finds closest `nx ny` for selected `lat lon` on the map.\n",
    "\n",
    "The index closest to the clicked point is the point with the minimum distance. Also a unit of latitude only equals a unit of longitude near the equator so we can apply an average factor `ds.lat.mean()`. The formula used below is for simplification and speed until we have a situation that demands something better. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.151360Z",
     "start_time": "2019-04-17T16:48:32.055Z"
    }
   },
   "outputs": [],
   "source": [
    "ts_base_map = hv.Curve([]) #ts_base: Time series base map\n",
    "\n",
    "def get_line_color():\n",
    "    try: color = tapped_locs['color'][-1] \n",
    "    except IndexError: color = 'red'\n",
    "    return color\n",
    "\n",
    "def create_time_series(x,y,clear):\n",
    "    \n",
    "    global ts_base_map\n",
    "    if clear:\n",
    "        ts_base_map = hv.Overlay()\n",
    "    \n",
    "    else:\n",
    "        if None not in [x,y]:\n",
    "            sel_lat = y \n",
    "            sel_lon = x \n",
    "\n",
    "            lat_mean=ds.lat.mean()*np.pi/180.\n",
    "            a = (ds.lat-sel_lat)**2 + ((ds.lon-sel_lon)*np.cos(lat_mean))**2\n",
    "\n",
    "            i,j = np.unravel_index(a.argmin(),a.shape)\n",
    "\n",
    "            time_series = ds.sel(nx =j,ny = i,sigma = 0.0)\n",
    "\n",
    "            ts_base_map = ts_base_map * time_series[var_select.value].hvplot(xlabel='time',\n",
    "                                                                             ylabel = f'{var_select.value}',\n",
    "                                                                             line_color =  get_line_color(),)\n",
    "        else:\n",
    "            ts_base_map  = ts_base_map * hv.Curve([])\n",
    "    \n",
    "\n",
    "    return ts_base_map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.153487Z",
     "start_time": "2019-04-17T16:48:32.058Z"
    }
   },
   "outputs": [],
   "source": [
    "var_select.param.watch(on_var_select, parameter_names=['value'])\n",
    "base_map_select.param.watch(on_base_map_select, parameter_names=['value']);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A dynamic map for time series plotting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.157070Z",
     "start_time": "2019-04-17T16:48:32.061Z"
    }
   },
   "outputs": [],
   "source": [
    "ts_map = hv.DynamicMap(lambda x,y,clear: create_time_series(x,y,clear), streams=[tap_mesh,clear_time_series])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.160143Z",
     "start_time": "2019-04-17T16:48:32.065Z"
    }
   },
   "outputs": [],
   "source": [
    "def clear_plot(arg=None):\n",
    "    clear_time_series.event(clear=True)\n",
    "    clear_points.event(clear=True)\n",
    "    tapped_locs['color'] = []\n",
    "    \n",
    "clear_button = pn.widgets.Button(name='Clear',width = 100)\n",
    "clear_button.param.watch(clear_plot, 'clicks')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.163119Z",
     "start_time": "2019-04-17T16:48:32.067Z"
    }
   },
   "outputs": [],
   "source": [
    "dashboard = pn.Column(var_select, base_map_select, plot(var_select.value))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.166254Z",
     "start_time": "2019-04-17T16:48:32.069Z"
    }
   },
   "outputs": [],
   "source": [
    "var_select.value = 'air_v' \n",
    "create_time_series(-77.77,43.66,True);\n",
    "create_time_series(-77.77,43.66,False);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-17T16:48:32.168193Z",
     "start_time": "2019-04-17T16:48:32.071Z"
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "pn.Column(dashboard,ts_map,\n",
    "          pn.Row(pn.Spacer(width = 300),clear_button))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tap anywhere on the mesh to get a time series map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clear_button.clicks +=1"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}