Chapter3_1_GDAL_answers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 Geospatial processing with `gdal`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "[GDAL](https://gdal.org) is the workhorse of geospatial processing. Basically, GDAL offers a common library to access a vast number of formats (if you want to see how vast, [check this](https://gdal.org/formats_list.html)). In addition to letting you open and convert obscure formats to something more useful, a lot of functionality in terms of processing raster data is available (for example, working with projections, combining datasets, accessing remote datasets, etc).\n",
    "\n",
    "For vector data, the counterpart to GDAL is OGR (which is now a part of the GDAL library anyway), which also supports [many vector formats](https://gdal.org/ogr_formats.html). The combination of both libraries is a very powerful tool to work with geospatial data, not only from Python, but from [many other popular computer languages](https://trac.osgeo.org/gdal/#GDALOGRInOtherLanguages).\n",
    "\n",
    "In this session, we will introduce the `gdal` geospatial module which can read a wide range of raster scientific data formats. We will also introduce the related `ogr` vector package.\n",
    "\n",
    "In pacticular, we will learn how to:\n",
    "\n",
    "* access and download NASA geophysical datasets (specifically, the MODIS LAI/FPAR product)\n",
    "* apply a vector mask to the dataset\n",
    "* apply quality control flags to the data\n",
    "* stack datasets into a 3D numpy dataset for further analysis, including interpolation of missing values\n",
    "* visualise the data\n",
    "* store the stacked dataset\n",
    "\n",
    "**These are all tasks that you will be required to do for the [part 1 formal assessment](Formal_assessment_part1.ipynb) of this course. You will however be using a different NASA dataset.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## 3.1 MODIS LAI product \n",
    "To introduce geospatial processing, we will use a dataset from the MODIS LAI product over the UK. \n",
    "\n",
    "You should note that the dataset you need to use for your assessed practical is a MODIS dataset with similar characteristics to the one in this example.\n",
    "\n",
    "The data product [MOD15](https://modis.gsfc.nasa.gov/data/dataprod/mod15.php) LAI/FPAR has been generated from NASA MODIS sensors Terra and Aqua data since 2002. We are now in dataset collection 6 (the data version to use).\n",
    "\n",
    "    LAI is defined as the one-sided green leaf area per unit ground area in broadleaf canopies and as half the total needle surface area per unit ground area in coniferous canopies. FPAR is the fraction of photosynthetically active radiation (400-700 nm) absorbed by green vegetation. Both variables are used for calculating surface photosynthesis, evapotranspiration, and net primary production, which in turn are used to calculate terrestrial energy, carbon, water cycle processes, and biogeochemistry of vegetation. Algorithm refinements have improved quality of retrievals and consistency with field measurements over all biomes, with a focus on woody vegetation.\n",
    "    \n",
    "We use such data to map and understand about the dynamics of terrestrial vegetation / carbon, for example, for climate studies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The raster data are arranged in tiles, indexed by row and column, to cover the globe:\n",
    "\n",
    "\n",
    "![MODIS tiles](https://www.researchgate.net/profile/J_Townshend/publication/220473201/figure/fig5/AS:277546596880390@1443183673583/The-global-MODIS-Sinusoidal-tile-grid.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 3.1.1**\n",
    "\n",
    "The pattern on the tile names is `hXXvYY` where `XX` is the horizontal coordinate and `YY` the vertical.\n",
    "\n",
    "\n",
    "* use the map above to work out the names of the two tiles that we will need to access data over the UK\n",
    "* set the variable `tiles` to contain these two names in a list\n",
    "\n",
    "For example, for the two tiles covering Madegascar, we would set:\n",
    "\n",
    "    tiles = ['h22v10','h22v11']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# do exercise here\n",
    "# ANSWER \n",
    "\n",
    "tiles = ['h17v03','h17v03']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1.1 NASA Earthdata access\n",
    "\n",
    "#### 3.1.1.1 Register at NASA Earthdata\n",
    "\n",
    "Before you attempt to do this section, you will need to register at [NASA Earthdata](https://urs.earthdata.nasa.gov/home).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have set up these notes so that you don't have to put your username and password in plain text. Instead, you need to enter your username and password when prompted by `cylog`. The password is stored in an encrypted file, although it can be accessed as plain text within your Python session.\n",
    "\n",
    "**N.B. using `cylog().login()` is only intended to work with access to NASA Earthdata and to prevent you having to expose your username and password in these notes**.\n",
    "\n",
    "`cylog().login()` returns the tuple `(username,password)` in plain text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import geog0111.nasa_requests as nasa_requests\n",
    "from geog0111.cylog import cylog\n",
    "%matplotlib inline\n",
    "\n",
    "url = 'https://e4ftl01.cr.usgs.gov/MOTA/MCD15A3H.006/2018.09.30/' \n",
    "        \n",
    "# grab the HTML information\n",
    "try:\n",
    "    html = nasa_requests.get(url).text\n",
    "    # test a few lines of the html\n",
    "    if html[:20] == '<!DOCTYPE HTML PUBLI':\n",
    "        print('this seems to be ok ... ')\n",
    "        print('use cylog().login() anywhere you need to specify the tuple (username,password)')\n",
    "except:\n",
    "    print('login error ... try entering your username password again')\n",
    "    print('then re-run this cell until it works')\n",
    "    cylog(init=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The NASA servers go down for weekly maintenance, usually on Wednesday afternoon (UK time), so you might not want to attempt this exercise then. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1.2 `gdal`\n",
    "\n",
    "We should now check to see if you have `gdal` properly installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gdal\n",
    "version = gdal.VersionInfo()  \n",
    "\n",
    "if int(version) >= 2020400:\n",
    "    print('gdal ok',version)\n",
    "else:\n",
    "    print('gdal problem',version,'2.2.4+ expected')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If there is a problem and you are on the geography system, we should be able to fix it for you. \n",
    "\n",
    "If you are not on the geography system, try running:\n",
    "    \n",
    "    conda env update -f environment.yml \n",
    "    \n",
    "before going any further. If an update occurs, shutdown and restart your notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Automatic downloading of NASA MODIS products"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In [this section](Chapter3_2_MODIS_download.ipynb), you will learn how to:\n",
    "\n",
    "* scan the directories (on the Earthdata server) where the MODIS data are stored\n",
    "* get the dataset filename for a given tile, date and product\n",
    "* get to URL associated with the dataset\n",
    "* use the URL to pull the dataset over to store in the local file system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 GDAL masking\n",
    "\n",
    "In [this section](Chapter3_3_GDAL_masking.ipynb) you will learn how to:\n",
    "\n",
    "* load locally stored files into gdal\n",
    "* select a particular dataset\n",
    "* form a virtual 'stitched' dataset from multiple files\n",
    "* apply a mask to the data from a vector boundary\n",
    "* crop the dataset "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4 GDAL stacking and interpolating\n",
    "\n",
    "In [this section](Chapter3_4_GDAL_stacking_and_interpolating.ipynb) you will learn how to:\n",
    "\n",
    "* generate a numpy time series of spatial data\n",
    "* interpolate/smooth the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.5 Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this session, we have learned to use some geospatial tools using GDAL in Python. A good set of [working notes on how to use GDAL](http://jgomezdans.github.io/gdal_notes/) has been developed that you will find useful for further reading, as well as looking at the [advanced](advanced.ipynb) section.\n",
    "\n",
    "We have also very briefly introduced dealing with vector datasets in `ogr`, but this was mainly through the use of a pre-defined function that will take an ESRI shapefile (vector dataset), warp this to the projection of a raster dataset, and produce a mask for a given layer in the vector file.\n",
    "\n",
    "If there is time in the class, we will develop some exercises to examine the datasets we have generated and/or to explore some different datasets or different locations.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 0,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}