gridded_data_1_background.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Waterhackweek cyberseminar\n",
    "# Workflows for gridded climate datasets\n",
    "## Diana Gergel and Bart Nijssen\n",
    "### University of Washington\n",
    "### February 14, 2019"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Abstract:** Climate change, forecasting, satellite datasets, large model ensembles ... Large gridded datasets are everywhere in hydrology and earth science. While accessing and analyzing these datasets required some serious programming skills not so long ago, a number of toolkits are now available that let you easily access, ingest, analyze and display gridded climate datasets. In this webinar we’ll discuss one of the most common file formats used in our field for large data sets, the Network Common Data Format (NetCDF), and step through a Jupyter notebook to showcase python packages, such as xarray and cartopy, that can be used to examine them. No prior experience required, although we will build on some of the skills you have acquired in earlier webinars in the series."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GitHub repository\n",
    "\n",
    "The GitHub repository for this presentation can be found at: https://github.com/bartnijssen/gridded_data\n",
    "\n",
    "The README.md file in this repository explains how to configure the environment you need to run the sample notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outline\n",
    "\n",
    "* `gridded_data_1_background.ipynb` (this notebook)\n",
    "    + Introduction\n",
    "    + Gridded datasets\n",
    "    + NetCDF\n",
    "    + NetCDF conventions\n",
    "    + but enough background ...\n",
    "\n",
    "\n",
    "* `gridded_data_2_xarray_and_dask.ipynb`\n",
    "\n",
    "\n",
    "* `gridded_data_3_cartopy_and_seaborn.ipynb`\n",
    "\n",
    "\n",
    "* `gridded_data_4_rasterio.ipynb`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "By now you have already seen a large number of packages and tools that are available in python to access and analyze datasets that you may encounter in your research. Last week, Emilio Mayorga and Yifan Cheng discussed how to access and analyze time series data in a jupyter notebook.\n",
    "\n",
    "In this webinar, we will focus on gridded datasets. These datasets have at the very least an `x` and `y` coordinate (or a `latitude` and `longitude` coordinate or something similar). In most cases, they have more than that. Often they have a `time` coordinate and they may contain a `z` or `level` coordinate. They can also contain information about more than one variable.\n",
    "\n",
    "Clearly, not all spatial datasets are gridded datasets, but we will focus on this subset to get you started and to introduce some of the existing tools. In particular, in this webinar we will discuss the following python packages:\n",
    "* **xarray** - for manipulating labeled, N-dimensional arrays  \n",
    "* **cartopy** - for plotting spatial data on a map\n",
    "* **rasterio** - for reading and writing gridded raster datasets\n",
    "\n",
    "Along the way, we will briefly mention\n",
    "* **seaborn** - for statistical data visualization\n",
    "* **dask** - for manipulating very large datasets in a scalable manner\n",
    "\n",
    "The latter two are powerful packages in their own right, but are used \"_under the hood_\" by some of the other packages. For example, `dask` can be used by `xarray` without the user needing to invoke `dask` commands directly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gridded datasets\n",
    "\n",
    "As said, gridded datasets are a subset of spatial datasets, but they are very common in hydrology and geoscience. They are common enough that they form a useful example to discuss spatial analysis in general. In many cases, these datasets come from remote sensing or are used and produced by models. Over the years, a number of file formats have become de facto standards in geoscience. For example, the `NetCDF` format, which we will discuss in some detail, is widely used in earth system modeling. Similarly, `GeoTIFF` is commonly used as an interchange format for georeferenced raster imagery. Satellite datasets are often distributed using `HDF`. There are many good resources about each of these formats (see for example this [NASA web site](https://earthdata.nasa.gov/user-resources/standards-and-references/). Here we will focus on `NETCDF` files, which are commonly used (and produced) by models in atmospheric science and hydrology."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NetCDF\n",
    "\n",
    "`NetCDF` stands for \"_Network Common Data Form_\" and consists of a file format and a set of interfaces for array-oriented data access. These interfaces are encoded in data access libraries for a wide variety of computing languages, including python. These libraries support a machine-independent format for representing scientific data. `NetCDF` is developed and maintained by [Unidata](https://www.unidata.ucar.edu/software/netcdf/). \n",
    "\n",
    "`NetCDF` data is (from the [Unidata FAQ](https://www.unidata.ucar.edu/software/netcdf/docs/faq.html)):\n",
    "* **_Self-Describing_**: A netCDF file includes information about the data it contains.\n",
    "* **_Portable_**: A netCDF file can be accessed by computers with different ways of storing integers, characters, and floating-point numbers.\n",
    "* **_Scalable_**: A small subset of a large dataset may be accessed efficiently.\n",
    "* **_Appendable_**: Data may be appended to a properly structured netCDF file without copying the dataset or redefining its structure.\n",
    "* **_Sharable_**: One writer and multiple readers may simultaneously access the same netCDF file.\n",
    "* **_Archivable_**: Access to all earlier forms of netCDF data will be supported by current and future versions of the software.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you start using formats like `NetCDF`, it often takes a bit of getting used to. You cannot open the files and look at them without tools that are specifically designed to read them (i.e. the tools need to be use the `NetCDF` access routines). The same is true for `HDF` and `GeoTIFF` files and most non-ASCII files. But once you get used to this and once you start dealing with very large datafiles, the advantages of a format like `NetCDF` quickly become obvious.\n",
    "\n",
    "For example, datasets that you write to and read from a `NetCDF` file look **_exactly_** the same, independent of the hardware or operating system that was used to create them (\"_portable_\"). In addition, `NetCDF` allows you to include metadata directly in the `NetCDF` file itself and this metadata can be accessed in a standard manner (\"_self-describing_\"). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NetCDF conventions\n",
    "\n",
    "Just because you can include the metadata directly in the `NetCDF` file doesn't automatically mean that everyone will do that the same way. However, agreeing on standards for metadata can really facilitate data sharing and the construction of common tools. For this reason, a number of communities have developed `NetCDF` conventions for specifying metadata. Unidata maintains a [list](https://www.unidata.ucar.edu/software/netcdf/conventions.html) of a number of these conventions. \n",
    "\n",
    "The [CF conventions](http://cfconventions.org/), for example, are widely used in the _Climate and Forecast_ communities (hence the \"CF\"), and specify everything from how to format the time axis, how to specify variable names, and how to specify units. Software that is _CF-compliant_ can then rely on this convention to read files and perform standard analyses. For example, the CMIP5 (Coupled Model Intercomparison Project Phase 5) files are supposed to be CF-compliant, which makes it much easier to analyze output from a large number of different climate models in a standardized manner. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## but enough background ...\n",
    "\n",
    "Time to move on. The point of the webinar is to show you what some of these packages can actually do and it will be much easier to get a feel for `NetCDF` files if we start examining some of them. For that, we will use `xarray`, which is the main topic of the next notebook. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "whw_gridded_data",
   "language": "python",
   "name": "whw_gridded_data"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}