tutorial-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# sat-stac: Working with catalogs\n",
    "\n",
    "This notebook contains examples of using satstac to work with STAC catalogs.\n",
    "\n",
    "**Table of contents**\n",
    "\n",
    "- Working with existing catalogs\n",
    "- Creating new catalogs\n",
    "    - Adding catalogs to catalogs\n",
    "    - Adding collections to catalogs\n",
    "    - Adding items to collections\n",
    "- Views (sub-catalogs)\n",
    "\n",
    "\n",
    "The examples here use the [test catalog in the sat-stac repo](https://github.com/developmentseed/sat-stac/tree/master/test/catalog). The directory structure of the test catalog looks like the following, where the catalog.json files under the landsat-8-l1 and sentinel-2-l1c are `Collection`s, the rest of the catalog.json files are simple `Catalog`s, and the item.json files are `Item`s.\n",
    "\n",
    "**Test Catalog structure:**\n",
    "\n",
    "```\n",
    "catalog\n",
    "├── catalog.json\n",
    "└── eo\n",
    "    ├── catalog.json\n",
    "    ├── landsat-8-l1\n",
    "    │   ├── catalog.json\n",
    "    │   └── item.json\n",
    "    └── sentinel-2-l1c\n",
    "        ├── catalog.json\n",
    "        └── sentinel-2a\n",
    "            ├── catalog.json\n",
    "            └── item.json\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with existing catalogs<a name='existing' />\n",
    "\n",
    "Existing catalogs can be opened and traversed. Note that a Collection is also a type of Catalog. First, open up a catalog, in this case a root catalog."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from satstac import Catalog\n",
    "\n",
    "# open a root catalog\n",
    "cat = Catalog.open('test/catalog/catalog.json')\n",
    "\n",
    "# create pretty printer for later use\n",
    "import pprint\n",
    "pp = pprint.PrettyPrinter(indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The entire STAC tree can be traversed by following links. STAC links have a \"rel\" type, and several of the rel types are used to point to other members of the catalog:\n",
    "\n",
    "- **root**: All Catalogs, Collections and Items have a root link, even a root catalog (to itself).\n",
    "- **parent**: All Catalogs (and Collections), other than a root catalog, and Items have a parent link to it's parent catalog.\n",
    "- **child**: Catalogs (and Collections) may have child links that point to other Catalogs.\n",
    "- **collection**: Items have a link pointing to it's Collection.\n",
    "\n",
    "sat-stac makes traversing a Catalog easy by not just following the links, but instantiating them as objects of the proper type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Root Catalog:  stac-catalog\n",
      "Sub Catalog:  stac-catalog-eo\n",
      "Sub Catalog parent:  stac-catalog\n",
      "Sub Catalog children:\n",
      "     sentinel-s2-l1c\n",
      "     landsat-8-l1\n"
     ]
    }
   ],
   "source": [
    "# get first (and only in this case) sub-catalog\n",
    "subcat = [c for c in cat.children()][0]\n",
    "\n",
    "# print some IDs\n",
    "print(\"Root Catalog: \", cat.id)\n",
    "print(\"Sub Catalog: \", subcat.id)\n",
    "print(\"Sub Catalog parent: \", subcat.parent().id)\n",
    "\n",
    "# iterate through child catalogs of the sub-catalog\n",
    "print(\"Sub Catalog children:\")\n",
    "for child in subcat.children():\n",
    "    print('    ', child.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Being able to traverse a STAC catalog is useful, but it's more work to follow all the links in order to find anything. More useful is sat-stac's generator functions to get all other Catalogs, Collections, or Items that are within a Catalog in a recursive manner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<generator object Catalog.catalogs at 0x7f5b60def050>\n",
      "<generator object Catalog.collections at 0x7f5b60def0d0>\n",
      "<generator object Catalog.items at 0x7f5b60def1d0>\n"
     ]
    }
   ],
   "source": [
    "all_catalogs = cat.catalogs()\n",
    "all_collections = cat.collections()\n",
    "all_items = cat.items()\n",
    "\n",
    "print(all_catalogs)\n",
    "print(all_collections)\n",
    "print(all_items)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because these are generator functions they do not return a list of of objects, but a generator that can be iterated through with a for loop. Using generator functions is essential because a Catalog could contain a lot of items."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "**Catalogs**\n",
      "sentinel-s2-l1c\n",
      "sentinel-2a-catalog\n",
      "landsat-8-l1\n",
      "stac-catalog-eo\n",
      "\n",
      "**Collections**\n",
      "sentinel-s2-l1c\n",
      "landsat-8-l1\n",
      "\n",
      "**Items**\n",
      "S2B_25WFU_20200610_0_L1C\n",
      "LC08_L1TP_152038_20200611_20200611_01_RT\n"
     ]
    }
   ],
   "source": [
    "print('**Catalogs**')\n",
    "for c in cat.catalogs():\n",
    "    print(c.id)\n",
    "\n",
    "print('\\n**Collections**')\n",
    "for c in cat.collections():\n",
    "    print(c.id)\n",
    "    \n",
    "print('\\n**Items**')\n",
    "for i in cat.items():\n",
    "    print(i.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating new catalogs<a name='create' />\n",
    "\n",
    "New catalogs can be created with a STAC Catalog JSON (as a Python dictionary), or by using the create() function. The STAC version field (stac_version) will automatically be inserted into the Catalog based on the version of sat-stac set by the environment variable `STAC_VERSION`.\n",
    "\n",
    "STAC hierarchy links for the catalog (self, root, parent, child, collection, item) should not be provided, since they will be created automatically as the catalog is created and sub-catalogs, collections, or items are added. If hierachical links are provided they will be removed and replaced with new ones.\n",
    "\n",
    "The root keyword will specify this as the root catalog (there must be a root catalog)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mycat\n"
     ]
    }
   ],
   "source": [
    "# create a Catalog object with JSON\n",
    "cat_json = {\n",
    "    \"id\": \"mycat\",\n",
    "    \"description\": \"My shiny new STAC catalog\"\n",
    "}\n",
    "mycat = Catalog(cat_json)\n",
    "\n",
    "# or, use the create() function\n",
    "desc = 'My shiny new STAC catalog'\n",
    "mycat = Catalog.create(id='mycat', description=desc, root='https://my.cat')\n",
    "\n",
    "print(mycat.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When a new Catalog is created like this it does not yet exist on file. Since a hierarchical catalog doesn't make much sense unless it's stored somewhere, the Catalog will need to be saved before other Catalogs, Collections, or Items are added to it.\n",
    "\n",
    "The save() function can be used to save it as a new file by providing a filename. If a filename is not provided it will save it to the last saved filename if there is one, otherwise it will throw an error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mycat\n",
      "mycat/catalog.json\n",
      "mycat\n"
     ]
    }
   ],
   "source": [
    "# save as a root catalog\n",
    "mycat.save('mycat/catalog.json')\n",
    "\n",
    "print(mycat.id)\n",
    "# the filename is then stored with the object\n",
    "print(mycat.filename)\n",
    "print(mycat.path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding catalogs to catalogs<a name='addcat' />\n",
    "\n",
    "Once a Catalog has been saved, other Catalogs can be added to it, and they will automatically be saved and all the necessary links in all affected objects will be created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Child catalog filename before adding:  None\n",
      "Child catalog filename after adding:  mycat/mykitten/catalog.json\n"
     ]
    }
   ],
   "source": [
    "# add a new catalog to a root catalog\n",
    "cat_json = {\n",
    "    \"id\": \"mykitten\",\n",
    "    \"description\": \"A child catalog of my shiny new STAC catalog\"\n",
    "}\n",
    "\n",
    "kitten = Catalog(cat_json)\n",
    "print('Child catalog filename before adding: ', kitten.filename)\n",
    "\n",
    "mycat.add_catalog(kitten)\n",
    "print('Child catalog filename after adding: ', kitten.filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Catalog can have any number of child catalogs, and those can also have any number of child catalogs, and so on. These allow a data provider to partition the data in any way they wish. Data could be broken down by catalogs according to type, country, source, etc.\n",
    "\n",
    "### Adding collections to catalogs<a name='addcol' />\n",
    "\n",
    "Since a Collection is a Catalog, they can be added the same way. A Collection has additional fields over a normal Catalog that define the set of Items that fall under it. See the [Landsat example collection](https://github.com/developmentseed/sat-stac/blob/master/test/catalog/landsat-8-l1/catalog.json)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collection name:  landsat-8-l1\n",
      "Collection filename:  mycat/mykitten/landsat-8-l1/catalog.json\n",
      "\n",
      "**Collection links**\n",
      "[   {'href': '../../catalog.json', 'rel': 'root'},\n",
      "    {'href': '../catalog.json', 'rel': 'parent'}]\n"
     ]
    }
   ],
   "source": [
    "from satstac import Collection\n",
    "\n",
    "# open the Landsat collection\n",
    "collection = Collection.open('test/catalog/eo/landsat-8-l1/catalog.json')\n",
    "print('Collection name: ', collection)\n",
    "\n",
    "# add it to the child catalog created above\n",
    "kitten.add_catalog(collection)\n",
    "print('Collection filename: ', collection.filename)\n",
    "\n",
    "print('\\n**Collection links**')\n",
    "pp.pprint(collection._data['links'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding items to collections<a name='additem' />\n",
    "\n",
    "Since Items must belong in a Collection, they can be added to a Collection in a similar way that Collections or Catalogs are added to Catalogs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Item name:  landsat-8-l1\n",
      "Item filename:  mycat/mykitten/landsat-8-l1/LC08_L1TP_152038_20200611_20200611_01_RT.json\n",
      "\n",
      "**Item links**\n",
      "[   {'href': '../../catalog.json', 'rel': 'root'},\n",
      "    {'href': 'catalog.json', 'rel': 'parent'},\n",
      "    {'href': 'catalog.json', 'rel': 'collection'}]\n"
     ]
    }
   ],
   "source": [
    "from satstac import Item\n",
    "\n",
    "# open a Landsat item\n",
    "item = Item.open('test/catalog/eo/landsat-8-l1/item.json')\n",
    "print('Item name: ', collection)\n",
    "\n",
    "# add it to the collection created above\n",
    "collection.add_item(item)\n",
    "print('Item filename: ', item.filename)\n",
    "\n",
    "print('\\n**Item links**')\n",
    "pp.pprint(item._data['links'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Views (sub-catalogs)<a name='views' />\n",
    "\n",
    "It is often desirable to organize items by sub-catalogs according to some properties, rather than having all be together in it's Collection. For instance [Landsat on AWS](https://landsatonaws.com/) organizes data first by path, then row (path and row represent geographic location), then by date. This can be looked at as a series of catalogs, or a view of the Items.\n",
    "\n",
    "```\n",
    "root catalog\n",
    "├── landsat-8 catalog (collection)\n",
    "    └── path 170 catalog\n",
    "        ├── row 120 catalog\n",
    "            ├── 2018-10-31 catalog\n",
    "    ├── path 171 catalog\n",
    "        ├── row 120 catalog\n",
    "            ├── 2018-10-31 catalog\n",
    "```\n",
    "\n",
    "Luckily, with sat-stac, each of the sub-catalogs does not need to be created manually. When an Item is added to a Collection, a string pattern for the path and filename to the Item can be provided. Any property field in the Item (or Collection if using the [Commons extension](https://github.com/radiantearth/stac-spec/tree/master/extensions/commons)) can be provided, and it will be substituted for value of that property in the Item.\n",
    "\n",
    "In addition to the item's properties, there are two additional fields that may be used in the patterns:\n",
    "\n",
    "- id: The id of the item\n",
    "- date: The datetime property with the time portion stripped off\n",
    "\n",
    "The `path` provided indicates the sub-catalogs that will be used, while the `filename` provided indicates the relative filename of the Item to it's parent catalog. In this example `path` is `${landsat:path}/${landsat:row}` which means sub-catalogs are created for each Landsat 'path' which contains catalogs for each Landsat 'row'. Each Landsat 'row' catalog in turns contains `item` links with the name `${date}/${id}`.json."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Item filename:  mycat/mykitten/landsat-8-l1/107/18/2020-06-11/LC08_L1TP_152038_20200611_20200611_01_RT.json\n",
      "\n",
      "**Item links**\n",
      "[   {'href': '../../../../../catalog.json', 'rel': 'root'},\n",
      "    {'href': 'catalog.json', 'rel': 'parent'},\n",
      "    {'href': '../../../catalog.json', 'rel': 'collection'}]\n"
     ]
    }
   ],
   "source": [
    "# save \n",
    "filename = '${landsat:path}/${landsat:row}/${date}/${id}.json'\n",
    "\n",
    "collection.add_item(item, filename_template=filename)\n",
    "print('Item filename: ', item.filename)\n",
    "\n",
    "print('\\n**Item links**')\n",
    "pp.pprint(item._data['links'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the above code with the path set to path/row/date will result in a catalog tree with the following structure.\n",
    "\n",
    "```\n",
    "mycat/\n",
    "├── catalog.json\n",
    "└── mykitten\n",
    "    ├── catalog.json\n",
    "    └── landsat-8-l1\n",
    "        ├── 120\n",
    "        │   ├── 46\n",
    "        │   │   ├── 2018-10-12\n",
    "        │   │   │   └── LC08_L1GT_120046_20181012_20181012_01_RT.json\n",
    "        │   │   └── catalog.json\n",
    "        │   └── catalog.json\n",
    "        ├── catalog.json\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}