guides/catalog/06 Creating and Managing Vector Tables.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8035d092-56ac-4496-8b46-8e0863b545df",
   "metadata": {},
   "source": [
    "# Creating and Managing Vector Tables\n",
    "__________________\n",
    "\n",
    "This guide covers the basic methods for creating and managing tabular data. For full reference visit the [Vector API Documentation page](https://docs.descarteslabs.com/api/vector.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fbdc5b5-0201-45be-aa55-ce9afba4dcd0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import descarteslabs as dl\n",
    "from descarteslabs.vector import Feature, Table, models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba99c6ab-9c13-4bfd-a49d-b2fc55e9109a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import geopandas as gpd\n",
    "from pydantic import Field"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78af4130-0ac6-486d-8f1a-31576e63e05b",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Creating a Vector Table\n",
    "\n",
    "As with all other objects, tables must have an ID that is _unique to your organization_. For this example we will create a new table with our current user ID and share it with our organization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f67d925c-bad8-4e5a-8f41-aa0b8646195d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "auth = dl.auth.Auth()\n",
    "org = auth.payload[\"org\"]\n",
    "user_id = auth.namespace"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a2766d7-a51a-4bb5-b776-371c3b5a5e06",
   "metadata": {},
   "source": [
    "Next we'll read in a sample vector file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acccb67a-9789-4616-b66c-f81f431a53c0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gdf = gpd.read_parquet(\"data/countries.geoparquet\")\n",
    "gdf.plot(figsize=(10, 5))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22540369-f35f-440f-8d65-442269b648a0",
   "metadata": {},
   "source": [
    "And inspect its fields:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0a46039-383b-4b8b-9452-2f40616e1342",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gdf.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1d44992-5ae4-450f-9105-8c85903200d3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gdf.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a698477f-2389-4e03-9302-2bc3d9fcd63c",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Table Models\n",
    "\n",
    "Vector allows the user to define a custom schema for each table. Below is the list of predefined base [`model`](https://docs.descarteslabs.com/descarteslabs/vector/readme.html#module-descarteslabs.common.vector.models)s which should be inherited to define the schema:\n",
    "\n",
    "    models.VectorBaseModel # aspatial/tabular data only\n",
    "    models.PointBaseModel # data containing point geometries\n",
    "    models.MultiPointBaseModel # data containing multi-point geometries\n",
    "    models.PolygonBaseModel # data containing polygon geometries\n",
    "    models.MultiPolygonBaseModel # data containing multi-polygon geometries\n",
    "    models.LineBaseModel # data containing line geometries\n",
    "    models.MultiLineBaseModel # data containing multi-line geometries\n",
    "    \n",
    "__*Important Notes on Table Models*__:\n",
    "* All models automatically generate a required uuid column\n",
    "* If your data does not contain a geometry column (aspatial) the model should inherit from [`models.VectorBaseModel`](https://docs.descarteslabs.com/descarteslabs/vector/readme.html#descarteslabs.common.vector.models.VectorBaseModel)\n",
    "* If your data does contain a geometry column the appropriate model for the given geometry type must be selected, which will automatically generate a geometry field\n",
    "* If a multi-geometry model is selected, all geometries will be promoted to multi-part"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1317f45d-f500-4612-83b3-999dfddf4598",
   "metadata": {},
   "source": [
    "### Defining a Model's Fields\n",
    "Once the appropriate base model has been determined, a custom schema can be created by inheriting the base model. Column names and data types can then be attributed to the custom schema through simple Python data typing. \n",
    "\n",
    "If the table has a geometry column, a spatial index will be created automatically. To specify the creation of an index on another column, use [`pydantic.Field`](https://docs.pydantic.dev/latest/concepts/fields/), as in the below example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eee79c5a-4501-409d-94a0-a8edee8cc4b1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "class CountryModel(models.MultiPolygonBaseModel):\n",
    "    NAME: str = Field(json_schema_extra={\"index\": True})\n",
    "    REGION_UN: str\n",
    "    CONTINENT: str\n",
    "    POP_EST: float\n",
    "    LASTCENSUS: float"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05692611-226e-4c3d-ac42-2c99271e0884",
   "metadata": {},
   "source": [
    "Next we create our table by passing our model in to the [`Table.create()`](https://docs.descarteslabs.com/descarteslabs/vector/readme.html#descarteslabs.vector.Table.create) method. We can also share access control lists (ACLs) at this time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c24d2a8f-9f78-4307-9160-42317243f5b5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# create the table with the default generic feature model\n",
    "the_world = Table.create(\n",
    "    f\"the-world-{user_id}\", name=\"The World\", model=CountryModel, owners=[f\"org:{org}\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "368e6bc7-ec1a-4a14-a0de-18b7386babdd",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Adding to and Modifying Tables\n",
    "We can add a dataframe or geodataframe by [`Table.add()`](https://docs.descarteslabs.com/descarteslabs/vector/readme.html#descarteslabs.vector.Table.add). This will return a respective dataframe type with the new uuid field added:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fafd409-2ec1-4ff3-959e-ea6cc8d693e6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "borders = the_world.add(gdf)\n",
    "borders.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0a8e529-5b40-42da-8bae-cb9565ace3e3",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Features\n",
    "\n",
    "We can access and modify [`Feature`](https://docs.descarteslabs.com/descarteslabs/vector/readme.html#descarteslabs.vector.Feature)s directly by their uuid through the equivalent [`Feature.get()`](https://docs.descarteslabs.com/descarteslabs/vector/readme.html#descarteslabs.vector.Feature.get) or [`Table.get_feature()`](https://docs.descarteslabs.com/descarteslabs/vector/readme.html#descarteslabs.vector.Table.get_feature) methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a38b7e0-4bad-4564-a80c-98a60ff21fbf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "feat1 = Feature.get(f\"{the_world.id}:{borders.iloc[0].uuid}\")\n",
    "feat2 = the_world.get_feature(borders.iloc[0].uuid)\n",
    "assert feat1.values == feat2.values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d4a753d-f336-4033-9fb7-24a5a317d426",
   "metadata": {},
   "source": [
    "We can modify the feature and preserve the changes by [`Feature.save()`](https://docs.descarteslabs.com/descarteslabs/vector/readme.html#descarteslabs.vector.Feature.save):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "994575dc-43e3-4145-894f-19e08e619d11",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "feat1.values[\"geometry\"] = feat1.values[\"geometry\"].convex_hull\n",
    "feat1.values[\"POP_EST\"] = 1000\n",
    "feat1.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0f5bd0d-8ca5-4c3e-8525-bad03e435ea3",
   "metadata": {},
   "source": [
    "Retrieve the new feature to verify our changes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dac5e7b4-b134-441a-85cd-775a28b9268c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "feat1 = Feature.get(f\"{the_world.id}:{borders.iloc[0].uuid}\")\n",
    "feat1.values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ef5c120-c495-4608-8ffc-09bf490fe463",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Managing Access to Tables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56d0a19b-0756-460c-bf30-55eda4317a0d",
   "metadata": {},
   "source": [
    "We can also modify the description and other ACLs, making sure we call [`Table.save()`](https://docs.descarteslabs.com/descarteslabs/vector/readme.html#descarteslabs.vector.Table.save):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bbade47-7248-4d1d-ace7-d3bc24e79d3c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "the_world.description = \"Country boundaries for the world.\"\n",
    "the_world.readers.append(\"org:pga-tour\")\n",
    "the_world.save()\n",
    "print(the_world.description)\n",
    "print(the_world.readers)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6921144c-8d1b-4823-8027-10a9e7870451",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deleting Tables\n",
    "Finally, we can delete our table through [`Table.delete()`](https://docs.descarteslabs.com/descarteslabs/vector/readme.html#descarteslabs.vector.Table.delete):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3127cf01-e4a7-4938-869a-b127a27eb7cd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "the_world.delete()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "403ca3c9-f9ea-4ec0-b5e1-d6bc884b3dff",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}