notebooks/How_to_Access_GES_DISC_Data_Using_Python.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4da9684d-260e-49f4-a9ae-164738c17f3d",
   "metadata": {
    "id": "4da9684d-260e-49f4-a9ae-164738c17f3d"
   },
   "source": [
    "# How to Access GES DISC Data Using Python\n",
    "\n",
    "<p></p>\n",
    "\n",
    "<div style=\"background:#eeeeee; border:1px solid #cccccc;padding:5px 10px;\">Please, be very judicious when working on long data time series residing on a remote data server.<br />\n",
    "It is very likely that attempts to apply similar approaches on remote data, such as hourly data, for more than a year of data at a time, will result in a heavy load on the remote data server. This may lead to negative consequences, ranging from very slow performance that will be experienced by hundreds of other users, up to denial of service.</div>\n",
    "\n",
    "### Overview\n",
    "\n",
    "There are multiple ways to work with GES DISC data resources using Python. For example, the data can accessed using [techniques that rely on a native Python code](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html). \n",
    "\n",
    "Still, there are several third-party libraries that can further simplify the access. In the sections below, we describe four techniques that make use of Requests, Pydap, Xarray, and netCDF4-python libraries.\n",
    "\n",
    "### Prerequisites\n",
    "\n",
    "This notebook was written using Python 3.8, and requires these libraries and files:\n",
    "\n",
    "- `netrc` file with valid Earthdata Login credentials\n",
    "   - [How to Generate Earthdata Prerequisite Files](https://disc.gsfc.nasa.gov/information/howto?title=How%20to%20Generate%20Earthdata%20Prerequisite%20Files)\n",
    "- [requests](https://docs.python-requests.org/en/latest/) (version 2.22.0 or later)\n",
    "- [pydap](https://github.com/pydap/pydap) (we recommend using version 3.4.0 or later)\n",
    "- [xarray](https://docs.xarray.dev/en/stable/)\n",
    "- [netCDF4-python](https://github.com/Unidata/netcdf4-python) (we recommend using version 1.6.4)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "l0uyzq5VDk-j",
   "metadata": {
    "id": "l0uyzq5VDk-j"
   },
   "source": [
    "### Python Using 'Requests'\n",
    "\n",
    "'Requests' is a popular Python library that simplifies Python access to Internet-based resources. In the following code, we demonstrate how to use 'Requests' to access GES DISC data using cookies created by a host operating system.\n",
    "\n",
    "Download GES DISC data using the following Python3 code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "kc5XYhMmDtSE",
   "metadata": {
    "id": "kc5XYhMmDtSE"
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "# Set the URL string to point to a specific data URL. Some generic examples are:\n",
    "#   https://data.gesdisc.earthdata.nasa.gov/data/MERRA2/path/to/granule.nc4\n",
    "\n",
    "URL = 'your_URL_string_goes_here'\n",
    "\n",
    "# Set the FILENAME string to the data file name, the LABEL keyword value, or any customized name. \n",
    "FILENAME = 'your_file_string_goes_here'\n",
    "\n",
    "import requests\n",
    "result = requests.get(URL)\n",
    "try:\n",
    "    result.raise_for_status()\n",
    "    f = open(FILENAME,'wb')\n",
    "    f.write(result.content)\n",
    "    f.close()\n",
    "    print('contents of URL written to '+FILENAME)\n",
    "except:\n",
    "    print('requests.get() returned an error code '+str(result.status_code))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "oRj2te56DwTr",
   "metadata": {
    "id": "oRj2te56DwTr"
   },
   "source": [
    "### Python Using 'Pydap'\n",
    "\n",
    "A convenient access to GES DISC OPeNDAP resources can be also achieved with 'Pydap', a Python library that both provides an interface for Python programs to read from OPeNDAP servers and the netCDF4 Python module which uses the netCDF-C library to actually access data.\n",
    "\n",
    "Use the code below to access data on OPeNDAP servers ( [read more](https://pydap.readthedocs.io/en/latest/client.html#urs-nasa-earthdata) ):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "Wa-rQXw3D2hs",
   "metadata": {
    "id": "Wa-rQXw3D2hs"
   },
   "outputs": [],
   "source": [
    "from pydap.client import open_url\n",
    "from pydap.cas.urs import setup_session\n",
    "\n",
    "dataset_url = 'https://servername/opendap/path/file[.format[?subset]]'\n",
    "\n",
    "username = 'your_earthdata_username_goes_here'\n",
    "password = 'your_earthdata_password_goes_here'\n",
    "\n",
    "try:\n",
    "    session = setup_session(username, password, check_url=dataset_url)\n",
    "    dataset = open_url(dataset_url, session=session)\n",
    "except AttributeError as e:\n",
    "    print('Error:', e)\n",
    "    print('Please verify that the dataset URL points to an OPeNDAP server, the OPeNDAP server is accessible, or that your username and password are correct.')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "185538db",
   "metadata": {},
   "source": [
    "### Python using 'Xarray'\n",
    "\n",
    "If you wish to open datasets as Xarray data objects, you can simply pass in a dataset URL to the <code>open_dataset()</code> function. Depending on the subsetting service that you wish to access, different Earthdata authentication files may be required. Here, we demonstrate accessing a granule via OPeNDAP and THREDDS.\n",
    "\n",
    "#### OPeNDAP in Xarray:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "983bf113",
   "metadata": {},
   "outputs": [],
   "source": [
    "import xarray as xr\n",
    "\n",
    "# Reading a single granule URL:\n",
    "ds = xr.open_dataset('https://servername/opendap/path/file[.format[?subset]]')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c938f838",
   "metadata": {},
   "source": [
    "#### THREDDS in Xarray:\n",
    "\n",
    "Datasets that include <code>.ncml</code> aggregation, like some provided through THREDDS, may be useful for quickly subsetting multiple granules into a single data array.\n",
    "\n",
    "This operation requires a <code>.dodsrc</code> file in your root and working directories, and a <code>.netrc</code> file in your root directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89f27aa7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Subsetting a .ncml file URL:\n",
    "URL = 'https://servername/thredds/dodsC/path/dataset_Aggregation.ncml'\n",
    "\n",
    "try:\n",
    "    ds = xr.open_dataset(URL).sel(lat=slice(lat1,lat2),lon=slice(lon1,lon2),time=slice(time1,time2))\n",
    "except OSError as e:\n",
    "    print('Error', e)\n",
    "    print('Please check that your .dodsrc files are in their correct locations, or that your .netrc file has the correct username and password.')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "feb143f8",
   "metadata": {},
   "source": [
    "### Python using 'netCDF4-python'\n",
    "\n",
    "netCDF4-python is a Python library that uses the [netCDF-c](https://github.com/Unidata/netcdf-c) library to open and read netCDF4 files. It can be used to remotely access OPeNDAP netCDF4 granules, or locally downloaded netCDF4 granules."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c2625c5b",
   "metadata": {},
   "source": [
    "#### OPeNDAP in netCDF4-python:\n",
    "\n",
    "This step requires a `.netrc` file in your root directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62af191c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import netCDF4 as nc4\n",
    "\n",
    "nc = nc4.Dataset('https://servername/opendap/path/file[.nc4[?subset]]')"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Data_Access_Using_Python_Draft",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}