Apache_Sedona_with_PySpark.nbconvert.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/groda/big_data/blob/master/Apache_Sedona_with_PySpark.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DNWKwG4TAcD2"
   },
   "source": [
    "\n",
    "<a href=\"https://github.com/groda/big_data\"><div><img src=\"https://github.com/groda/big_data/blob/master/logo_bdb.png?raw=true\" align=right width=\"90\" alt=\"Logo Big Data for Beginners\"></div></a>\n",
    "# Apache Sedona with PySpark\n",
    "\n",
    "Apache Sedona™ is\n",
    "\n",
    "> *a cluster computing system for processing large-scale spatial data. Sedona extends existing cluster computing systems, such as Apache Spark, Apache Flink, and Snowflake, with a set of out-of-the-box distributed Spatial Datasets and Spatial SQL that efficiently load, process, and analyze large-scale spatial data across machines.* ([https://sedona.apache.org/](https://sedona.apache.org/))\n",
    "\n",
    "To execute a basic Sedona demonstration using PySpark on Google Colab, we made a few minor adjustments. The Sedona notebook starts below at [Apache Sedona Core demo](#scrollTo=Apache_Sedona_Core_demo).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3AQNoWmX_B78"
   },
   "source": [
    "## Install Apache Sedona and PySpark\n",
    "\n",
    "To start with, we are going to install PySpark with Sedona following the instructions at: https://sedona.apache.org/latest-snapshot/setup/install-python/ but first we need to downgrade `shapely` because the version 2.0.2 that comes with Google Colab does not play well with the current version of Apache Sedona (see https://shapely.readthedocs.io/en/stable/migration.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zDoBDC3yAVvI"
   },
   "source": [
    "### Downgrade Shapely to version 1.7.1\n",
    "\n",
    "We need to install install any version of `shapely>=1.7.0` but smaller than `2.0`. We picked `1.7.1` because with 1.7.0 we got the error\n",
    "\n",
    "    geopandas 0.13.2 requires shapely>=1.7.1, but you have shapely 1.7.0 which is incompatible.\n",
    "\n",
    "Explanation for `pip -I`:\n",
    "\n",
    "- [`-I, --ignore-installed`](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-I)\n",
    "> Ignore the installed packages, overwriting them. This can break your system if the existing package is of a different version or was installed with a different package manager!\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:17:05.556748Z",
     "iopub.status.busy": "2024-03-24T21:17:05.556544Z",
     "iopub.status.idle": "2024-03-24T21:17:07.312002Z",
     "shell.execute_reply": "2024-03-24T21:17:07.311335Z"
    },
    "id": "_4nnDMo0Blfu",
    "outputId": "cf77f2ad-6e17-43da-8578-3ff535497efa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting shapely==1.7.1\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Downloading Shapely-1.7.1-cp38-cp38-manylinux1_x86_64.whl.metadata (34 kB)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading Shapely-1.7.1-cp38-cp38-manylinux1_x86_64.whl (1.0 MB)\r\n",
      "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/1.0 MB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r",
      "\u001b[2K   \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m31.8 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m27.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\r\n",
      "\u001b[?25h"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Installing collected packages: shapely\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully installed shapely-1.7.1\r\n",
      "\r\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.0.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\r\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!pip install -I shapely==1.7.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "76yI9hGZz_Ss"
   },
   "source": [
    "### Install Geopandas\n",
    "\n",
    "This step is only needed outside of Colab because on Google Colab `geopandas` is available by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:17:07.315415Z",
     "iopub.status.busy": "2024-03-24T21:17:07.314978Z",
     "iopub.status.idle": "2024-03-24T21:17:10.357671Z",
     "shell.execute_reply": "2024-03-24T21:17:10.356880Z"
    },
    "id": "KliMM6ka0Qee",
    "outputId": "170bc61d-9911-4929-8a51-7d82fd24d6aa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting geopandas==0.13.2\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Downloading geopandas-0.13.2-py3-none-any.whl.metadata (1.5 kB)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting fiona>=1.8.19 (from geopandas==0.13.2)\r\n",
      "  Downloading fiona-1.9.6-cp38-cp38-manylinux2014_x86_64.whl.metadata (50 kB)\r\n",
      "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/50.2 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.2/50.2 kB\u001b[0m \u001b[31m3.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\r\n",
      "\u001b[?25hRequirement already satisfied: packaging in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from geopandas==0.13.2) (24.0)\r\n",
      "Requirement already satisfied: pandas>=1.1.0 in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from geopandas==0.13.2) (1.5.3)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pyproj>=3.0.1 (from geopandas==0.13.2)\r\n",
      "  Downloading pyproj-3.5.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (28 kB)\r\n",
      "Requirement already satisfied: shapely>=1.7.1 in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from geopandas==0.13.2) (1.7.1)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: attrs>=19.2.0 in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from fiona>=1.8.19->geopandas==0.13.2) (23.2.0)\r\n",
      "Requirement already satisfied: certifi in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from fiona>=1.8.19->geopandas==0.13.2) (2024.2.2)\r\n",
      "Collecting click~=8.0 (from fiona>=1.8.19->geopandas==0.13.2)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Downloading click-8.1.7-py3-none-any.whl.metadata (3.0 kB)\r\n",
      "Collecting click-plugins>=1.0 (from fiona>=1.8.19->geopandas==0.13.2)\r\n",
      "  Downloading click_plugins-1.1.1-py2.py3-none-any.whl.metadata (6.4 kB)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting cligj>=0.5 (from fiona>=1.8.19->geopandas==0.13.2)\r\n",
      "  Downloading cligj-0.7.2-py3-none-any.whl.metadata (5.0 kB)\r\n",
      "Requirement already satisfied: six in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from fiona>=1.8.19->geopandas==0.13.2) (1.16.0)\r\n",
      "Requirement already satisfied: importlib-metadata in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from fiona>=1.8.19->geopandas==0.13.2) (7.1.0)\r\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from pandas>=1.1.0->geopandas==0.13.2) (2.9.0.post0)\r\n",
      "Requirement already satisfied: pytz>=2020.1 in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from pandas>=1.1.0->geopandas==0.13.2) (2024.1)\r\n",
      "Requirement already satisfied: numpy>=1.20.3 in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from pandas>=1.1.0->geopandas==0.13.2) (1.24.4)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: zipp>=0.5 in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from importlib-metadata->fiona>=1.8.19->geopandas==0.13.2) (3.18.1)\r\n",
      "Downloading geopandas-0.13.2-py3-none-any.whl (1.1 MB)\r\n",
      "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/1.1 MB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m33.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\r\n",
      "\u001b[?25hDownloading fiona-1.9.6-cp38-cp38-manylinux2014_x86_64.whl (15.7 MB)\r\n",
      "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/15.7 MB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[2K   \u001b[91m━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.8/15.7 MB\u001b[0m \u001b[31m173.1 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[2K   \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m\u001b[90m━━━━━━━━━━\u001b[0m \u001b[32m11.7/15.7 MB\u001b[0m \u001b[31m173.1 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r",
      "\u001b[2K   \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m \u001b[32m15.7/15.7 MB\u001b[0m \u001b[31m166.4 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m15.7/15.7 MB\u001b[0m \u001b[31m104.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\r\n",
      "\u001b[?25hDownloading pyproj-3.5.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\r\n",
      "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/7.8 MB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[2K   \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[90m╺\u001b[0m\u001b[90m━━━━━━━━━━━━\u001b[0m \u001b[32m5.3/7.8 MB\u001b[0m \u001b[31m159.4 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r",
      "\u001b[2K   \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m158.0 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m111.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\r\n",
      "\u001b[?25hDownloading click-8.1.7-py3-none-any.whl (97 kB)\r\n",
      "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/97.9 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m97.9/97.9 kB\u001b[0m \u001b[31m27.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\r\n",
      "\u001b[?25hDownloading click_plugins-1.1.1-py2.py3-none-any.whl (7.5 kB)\r\n",
      "Downloading cligj-0.7.2-py3-none-any.whl (7.1 kB)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Installing collected packages: pyproj, click, cligj, click-plugins, fiona, geopandas\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully installed click-8.1.7 click-plugins-1.1.1 cligj-0.7.2 fiona-1.9.6 geopandas-0.13.2 pyproj-3.5.0\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.0.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\r\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!pip install geopandas==0.13.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Vv5bhPBIDQ5G"
   },
   "source": [
    "### Install Apache Sedona and PySpark\n",
    "\n",
    "We can now install Apache Sedona together with PySpark (and Spark)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:17:10.360992Z",
     "iopub.status.busy": "2024-03-24T21:17:10.360730Z",
     "iopub.status.idle": "2024-03-24T21:17:12.103256Z",
     "shell.execute_reply": "2024-03-24T21:17:12.102426Z"
    },
    "id": "SOB7DNZ6AOio",
    "outputId": "5aa22c6b-d2eb-41ce-dd2e-f46a2dd2be69"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting apache-sedona[spark]\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Downloading apache_sedona-1.5.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.3 kB)\r\n",
      "Requirement already satisfied: attrs in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from apache-sedona[spark]) (23.2.0)\r\n",
      "Requirement already satisfied: shapely>=1.7.0 in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from apache-sedona[spark]) (1.7.1)\r\n",
      "Requirement already satisfied: pyspark>=2.3.0 in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from apache-sedona[spark]) (3.5.1)\r\n",
      "Requirement already satisfied: py4j==0.10.9.7 in /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages (from pyspark>=2.3.0->apache-sedona[spark]) (0.10.9.7)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading apache_sedona-1.5.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (155 kB)\r\n",
      "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/155.0 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m155.0/155.0 kB\u001b[0m \u001b[31m6.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\r\n",
      "\u001b[?25h"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Installing collected packages: apache-sedona\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully installed apache-sedona-1.5.1\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.0.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\r\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!pip install apache-sedona[spark]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:17:12.106458Z",
     "iopub.status.busy": "2024-03-24T21:17:12.106036Z",
     "iopub.status.idle": "2024-03-24T21:17:12.110100Z",
     "shell.execute_reply": "2024-03-24T21:17:12.109434Z"
    },
    "id": "ZoNNqcGaGrEh",
    "outputId": "aa79c622-24dd-413d-e86c-4e8f379e0b0b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: SPARK_HOME=\"/usr/local/lib/python3.10/dist-packages/pyspark\"\n"
     ]
    }
   ],
   "source": [
    "%env SPARK_HOME = \"/usr/local/lib/python3.10/dist-packages/pyspark\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:17:12.112649Z",
     "iopub.status.busy": "2024-03-24T21:17:12.112270Z",
     "iopub.status.idle": "2024-03-24T21:17:12.115773Z",
     "shell.execute_reply": "2024-03-24T21:17:12.115130Z"
    },
    "id": "z9fpbLrjHBeY",
    "outputId": "5dade95c-b20b-496f-a8d1-504ef69d100d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: PYTHONPATH=/usr/local/lib/python3.10/dist-packages/pyspark/python\n"
     ]
    }
   ],
   "source": [
    "%env PYTHONPATH = /usr/local/lib/python3.10/dist-packages/pyspark/python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:17:12.118406Z",
     "iopub.status.busy": "2024-03-24T21:17:12.118037Z",
     "iopub.status.idle": "2024-03-24T21:17:12.494374Z",
     "shell.execute_reply": "2024-03-24T21:17:12.493622Z"
    },
    "id": "d_YNI8fKiU3k",
    "outputId": "1cb25835-1990-44ed-8a65-2afeafc4dc32"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ERROR: unknown command \"info\"\r\n"
     ]
    }
   ],
   "source": [
    "!pip info pyspark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Sjlr4JO-Erv3"
   },
   "source": [
    "## Setup environment variables\n",
    "\n",
    "We need to set two environment variables:\n",
    "\n",
    "- `SPARK_HOME`\n",
    "- `PYTHONPATH`\n",
    "\n",
    "Once we have set `SPARK_HOME`, the variable `PYTHONPATH` is `$SPARK_HOME/python`.\n",
    "\n",
    "### Find Spark home\n",
    "\n",
    "There's an utility to find Spark home and I always forget how it's called exactly, what I remember is that it contains `\"find\"` and `\"spark\"`. Let us search for it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:17:12.497420Z",
     "iopub.status.busy": "2024-03-24T21:17:12.497194Z",
     "iopub.status.idle": "2024-03-24T21:18:31.194402Z",
     "shell.execute_reply": "2024-03-24T21:18:31.193639Z"
    },
    "id": "oK5vO9MUGQA6",
    "outputId": "c018213e-e7cc-4494-a45f-587f175d9eb7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "find: ‘/boot/efi’: Permission denied\r\n",
      "find: ‘/lost+found’: Permission denied\r\n",
      "find: ‘/proc/tty/driver’: Permission denied\r\n",
      "find: ‘/proc/1/task/1/fd’: Permission denied\r\n",
      "find: ‘/proc/1/task/1/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1/task/1/ns’: Permission denied\r\n",
      "find: ‘/proc/1/fd’: Permission denied\r\n",
      "find: ‘/proc/1/map_files’: Permission denied\r\n",
      "find: ‘/proc/1/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1/ns’: Permission denied\r\n",
      "find: ‘/proc/2/task/2/fd’: Permission denied\r\n",
      "find: ‘/proc/2/task/2/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/2/task/2/ns’: Permission denied\r\n",
      "find: ‘/proc/2/fd’: Permission denied\r\n",
      "find: ‘/proc/2/map_files’: Permission denied\r\n",
      "find: ‘/proc/2/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/2/ns’: Permission denied\r\n",
      "find: ‘/proc/3/task/3/fd’: Permission denied\r\n",
      "find: ‘/proc/3/task/3/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/3/task/3/ns’: Permission denied\r\n",
      "find: ‘/proc/3/fd’: Permission denied\r\n",
      "find: ‘/proc/3/map_files’: Permission denied\r\n",
      "find: ‘/proc/3/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/3/ns’: Permission denied\r\n",
      "find: ‘/proc/4/task/4/fd’: Permission denied\r\n",
      "find: ‘/proc/4/task/4/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/4/task/4/ns’: Permission denied\r\n",
      "find: ‘/proc/4/fd’: Permission denied\r\n",
      "find: ‘/proc/4/map_files’: Permission denied\r\n",
      "find: ‘/proc/4/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/4/ns’: Permission denied\r\n",
      "find: ‘/proc/5/task/5/fd’: Permission denied\r\n",
      "find: ‘/proc/5/task/5/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/5/task/5/ns’: Permission denied\r\n",
      "find: ‘/proc/5/fd’: Permission denied\r\n",
      "find: ‘/proc/5/map_files’: Permission denied\r\n",
      "find: ‘/proc/5/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/5/ns’: Permission denied\r\n",
      "find: ‘/proc/6/task/6/fd’: Permission denied\r\n",
      "find: ‘/proc/6/task/6/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/6/task/6/ns’: Permission denied\r\n",
      "find: ‘/proc/6/fd’: Permission denied\r\n",
      "find: ‘/proc/6/map_files’: Permission denied\r\n",
      "find: ‘/proc/6/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/6/ns’: Permission denied\r\n",
      "find: ‘/proc/7/task/7/fd’: Permission denied\r\n",
      "find: ‘/proc/7/task/7/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/7/task/7/ns’: Permission denied\r\n",
      "find: ‘/proc/7/fd’: Permission denied\r\n",
      "find: ‘/proc/7/map_files’: Permission denied\r\n",
      "find: ‘/proc/7/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/7/ns’: Permission denied\r\n",
      "find: ‘/proc/8/task/8/fd’: Permission denied\r\n",
      "find: ‘/proc/8/task/8/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/8/task/8/ns’: Permission denied\r\n",
      "find: ‘/proc/8/fd’: Permission denied\r\n",
      "find: ‘/proc/8/map_files’: Permission denied\r\n",
      "find: ‘/proc/8/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/8/ns’: Permission denied\r\n",
      "find: ‘/proc/10/task/10/fd’: Permission denied\r\n",
      "find: ‘/proc/10/task/10/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/10/task/10/ns’: Permission denied\r\n",
      "find: ‘/proc/10/fd’: Permission denied\r\n",
      "find: ‘/proc/10/map_files’: Permission denied\r\n",
      "find: ‘/proc/10/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/10/ns’: Permission denied\r\n",
      "find: ‘/proc/11/task/11/fd’: Permission denied\r\n",
      "find: ‘/proc/11/task/11/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/11/task/11/ns’: Permission denied\r\n",
      "find: ‘/proc/11/fd’: Permission denied\r\n",
      "find: ‘/proc/11/map_files’: Permission denied\r\n",
      "find: ‘/proc/11/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/11/ns’: Permission denied\r\n",
      "find: ‘/proc/12/task/12/fd’: Permission denied\r\n",
      "find: ‘/proc/12/task/12/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/12/task/12/ns’: Permission denied\r\n",
      "find: ‘/proc/12/fd’: Permission denied\r\n",
      "find: ‘/proc/12/map_files’: Permission denied\r\n",
      "find: ‘/proc/12/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/12/ns’: Permission denied\r\n",
      "find: ‘/proc/13/task/13/fd’: Permission denied\r\n",
      "find: ‘/proc/13/task/13/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/13/task/13/ns’: Permission denied\r\n",
      "find: ‘/proc/13/fd’: Permission denied\r\n",
      "find: ‘/proc/13/map_files’: Permission denied\r\n",
      "find: ‘/proc/13/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/13/ns’: Permission denied\r\n",
      "find: ‘/proc/14/task/14/fd’: Permission denied\r\n",
      "find: ‘/proc/14/task/14/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/14/task/14/ns’: Permission denied\r\n",
      "find: ‘/proc/14/fd’: Permission denied\r\n",
      "find: ‘/proc/14/map_files’: Permission denied\r\n",
      "find: ‘/proc/14/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/14/ns’: Permission denied\r\n",
      "find: ‘/proc/15/task/15/fd’: Permission denied\r\n",
      "find: ‘/proc/15/task/15/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/15/task/15/ns’: Permission denied\r\n",
      "find: ‘/proc/15/fd’: Permission denied\r\n",
      "find: ‘/proc/15/map_files’: Permission denied\r\n",
      "find: ‘/proc/15/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/15/ns’: Permission denied\r\n",
      "find: ‘/proc/16/task/16/fd’: Permission denied\r\n",
      "find: ‘/proc/16/task/16/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/16/task/16/ns’: Permission denied\r\n",
      "find: ‘/proc/16/fd’: Permission denied\r\n",
      "find: ‘/proc/16/map_files’: Permission denied\r\n",
      "find: ‘/proc/16/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/16/ns’: Permission denied\r\n",
      "find: ‘/proc/17/task/17/fd’: Permission denied\r\n",
      "find: ‘/proc/17/task/17/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/17/task/17/ns’: Permission denied\r\n",
      "find: ‘/proc/17/fd’: Permission denied\r\n",
      "find: ‘/proc/17/map_files’: Permission denied\r\n",
      "find: ‘/proc/17/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/17/ns’: Permission denied\r\n",
      "find: ‘/proc/18/task/18/fd’: Permission denied\r\n",
      "find: ‘/proc/18/task/18/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/18/task/18/ns’: Permission denied\r\n",
      "find: ‘/proc/18/fd’: Permission denied\r\n",
      "find: ‘/proc/18/map_files’: Permission denied\r\n",
      "find: ‘/proc/18/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/18/ns’: Permission denied\r\n",
      "find: ‘/proc/19/task/19/fd’: Permission denied\r\n",
      "find: ‘/proc/19/task/19/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/19/task/19/ns’: Permission denied\r\n",
      "find: ‘/proc/19/fd’: Permission denied\r\n",
      "find: ‘/proc/19/map_files’: Permission denied\r\n",
      "find: ‘/proc/19/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/19/ns’: Permission denied\r\n",
      "find: ‘/proc/20/task/20/fd’: Permission denied\r\n",
      "find: ‘/proc/20/task/20/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/20/task/20/ns’: Permission denied\r\n",
      "find: ‘/proc/20/fd’: Permission denied\r\n",
      "find: ‘/proc/20/map_files’: Permission denied\r\n",
      "find: ‘/proc/20/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/20/ns’: Permission denied\r\n",
      "find: ‘/proc/21/task/21/fd’: Permission denied\r\n",
      "find: ‘/proc/21/task/21/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/21/task/21/ns’: Permission denied\r\n",
      "find: ‘/proc/21/fd’: Permission denied\r\n",
      "find: ‘/proc/21/map_files’: Permission denied\r\n",
      "find: ‘/proc/21/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/21/ns’: Permission denied\r\n",
      "find: ‘/proc/22/task/22/fd’: Permission denied\r\n",
      "find: ‘/proc/22/task/22/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/22/task/22/ns’: Permission denied\r\n",
      "find: ‘/proc/22/fd’: Permission denied\r\n",
      "find: ‘/proc/22/map_files’: Permission denied\r\n",
      "find: ‘/proc/22/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/22/ns’: Permission denied\r\n",
      "find: ‘/proc/24/task/24/fd’: Permission denied\r\n",
      "find: ‘/proc/24/task/24/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/24/task/24/ns’: Permission denied\r\n",
      "find: ‘/proc/24/fd’: Permission denied\r\n",
      "find: ‘/proc/24/map_files’: Permission denied\r\n",
      "find: ‘/proc/24/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/24/ns’: Permission denied\r\n",
      "find: ‘/proc/25/task/25/fd’: Permission denied\r\n",
      "find: ‘/proc/25/task/25/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/25/task/25/ns’: Permission denied\r\n",
      "find: ‘/proc/25/fd’: Permission denied\r\n",
      "find: ‘/proc/25/map_files’: Permission denied\r\n",
      "find: ‘/proc/25/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/25/ns’: Permission denied\r\n",
      "find: ‘/proc/26/task/26/fd’: Permission denied\r\n",
      "find: ‘/proc/26/task/26/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/26/task/26/ns’: Permission denied\r\n",
      "find: ‘/proc/26/fd’: Permission denied\r\n",
      "find: ‘/proc/26/map_files’: Permission denied\r\n",
      "find: ‘/proc/26/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/26/ns’: Permission denied\r\n",
      "find: ‘/proc/27/task/27/fd’: Permission denied\r\n",
      "find: ‘/proc/27/task/27/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/27/task/27/ns’: Permission denied\r\n",
      "find: ‘/proc/27/fd’: Permission denied\r\n",
      "find: ‘/proc/27/map_files’: Permission denied\r\n",
      "find: ‘/proc/27/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/27/ns’: Permission denied\r\n",
      "find: ‘/proc/28/task/28/fd’: Permission denied\r\n",
      "find: ‘/proc/28/task/28/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/28/task/28/ns’: Permission denied\r\n",
      "find: ‘/proc/28/fd’: Permission denied\r\n",
      "find: ‘/proc/28/map_files’: Permission denied\r\n",
      "find: ‘/proc/28/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/28/ns’: Permission denied\r\n",
      "find: ‘/proc/30/task/30/fd’: Permission denied\r\n",
      "find: ‘/proc/30/task/30/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/30/task/30/ns’: Permission denied\r\n",
      "find: ‘/proc/30/fd’: Permission denied\r\n",
      "find: ‘/proc/30/map_files’: Permission denied\r\n",
      "find: ‘/proc/30/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/30/ns’: Permission denied\r\n",
      "find: ‘/proc/31/task/31/fd’: Permission denied\r\n",
      "find: ‘/proc/31/task/31/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/31/task/31/ns’: Permission denied\r\n",
      "find: ‘/proc/31/fd’: Permission denied\r\n",
      "find: ‘/proc/31/map_files’: Permission denied\r\n",
      "find: ‘/proc/31/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/31/ns’: Permission denied\r\n",
      "find: ‘/proc/32/task/32/fd’: Permission denied\r\n",
      "find: ‘/proc/32/task/32/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/32/task/32/ns’: Permission denied\r\n",
      "find: ‘/proc/32/fd’: Permission denied\r\n",
      "find: ‘/proc/32/map_files’: Permission denied\r\n",
      "find: ‘/proc/32/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/32/ns’: Permission denied\r\n",
      "find: ‘/proc/33/task/33/fd’: Permission denied\r\n",
      "find: ‘/proc/33/task/33/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/33/task/33/ns’: Permission denied\r\n",
      "find: ‘/proc/33/fd’: Permission denied\r\n",
      "find: ‘/proc/33/map_files’: Permission denied\r\n",
      "find: ‘/proc/33/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/33/ns’: Permission denied\r\n",
      "find: ‘/proc/34/task/34/fd’: Permission denied\r\n",
      "find: ‘/proc/34/task/34/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/34/task/34/ns’: Permission denied\r\n",
      "find: ‘/proc/34/fd’: Permission denied\r\n",
      "find: ‘/proc/34/map_files’: Permission denied\r\n",
      "find: ‘/proc/34/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/34/ns’: Permission denied\r\n",
      "find: ‘/proc/35/task/35/fd’: Permission denied\r\n",
      "find: ‘/proc/35/task/35/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/35/task/35/ns’: Permission denied\r\n",
      "find: ‘/proc/35/fd’: Permission denied\r\n",
      "find: ‘/proc/35/map_files’: Permission denied\r\n",
      "find: ‘/proc/35/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/35/ns’: Permission denied\r\n",
      "find: ‘/proc/36/task/36/fd’: Permission denied\r\n",
      "find: ‘/proc/36/task/36/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/36/task/36/ns’: Permission denied\r\n",
      "find: ‘/proc/36/fd’: Permission denied\r\n",
      "find: ‘/proc/36/map_files’: Permission denied\r\n",
      "find: ‘/proc/36/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/36/ns’: Permission denied\r\n",
      "find: ‘/proc/37/task/37/fd’: Permission denied\r\n",
      "find: ‘/proc/37/task/37/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/37/task/37/ns’: Permission denied\r\n",
      "find: ‘/proc/37/fd’: Permission denied\r\n",
      "find: ‘/proc/37/map_files’: Permission denied\r\n",
      "find: ‘/proc/37/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/37/ns’: Permission denied\r\n",
      "find: ‘/proc/38/task/38/fd’: Permission denied\r\n",
      "find: ‘/proc/38/task/38/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/38/task/38/ns’: Permission denied\r\n",
      "find: ‘/proc/38/fd’: Permission denied\r\n",
      "find: ‘/proc/38/map_files’: Permission denied\r\n",
      "find: ‘/proc/38/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/38/ns’: Permission denied\r\n",
      "find: ‘/proc/39/task/39/fd’: Permission denied\r\n",
      "find: ‘/proc/39/task/39/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/39/task/39/ns’: Permission denied\r\n",
      "find: ‘/proc/39/fd’: Permission denied\r\n",
      "find: ‘/proc/39/map_files’: Permission denied\r\n",
      "find: ‘/proc/39/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/39/ns’: Permission denied\r\n",
      "find: ‘/proc/40/task/40/fd’: Permission denied\r\n",
      "find: ‘/proc/40/task/40/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/40/task/40/ns’: Permission denied\r\n",
      "find: ‘/proc/40/fd’: Permission denied\r\n",
      "find: ‘/proc/40/map_files’: Permission denied\r\n",
      "find: ‘/proc/40/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/40/ns’: Permission denied\r\n",
      "find: ‘/proc/41/task/41/fd’: Permission denied\r\n",
      "find: ‘/proc/41/task/41/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/41/task/41/ns’: Permission denied\r\n",
      "find: ‘/proc/41/fd’: Permission denied\r\n",
      "find: ‘/proc/41/map_files’: Permission denied\r\n",
      "find: ‘/proc/41/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/41/ns’: Permission denied\r\n",
      "find: ‘/proc/42/task/42/fd’: Permission denied\r\n",
      "find: ‘/proc/42/task/42/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/42/task/42/ns’: Permission denied\r\n",
      "find: ‘/proc/42/fd’: Permission denied\r\n",
      "find: ‘/proc/42/map_files’: Permission denied\r\n",
      "find: ‘/proc/42/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/42/ns’: Permission denied\r\n",
      "find: ‘/proc/44/task/44/fd’: Permission denied\r\n",
      "find: ‘/proc/44/task/44/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/44/task/44/ns’: Permission denied\r\n",
      "find: ‘/proc/44/fd’: Permission denied\r\n",
      "find: ‘/proc/44/map_files’: Permission denied\r\n",
      "find: ‘/proc/44/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/44/ns’: Permission denied\r\n",
      "find: ‘/proc/45/task/45/fd’: Permission denied\r\n",
      "find: ‘/proc/45/task/45/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/45/task/45/ns’: Permission denied\r\n",
      "find: ‘/proc/45/fd’: Permission denied\r\n",
      "find: ‘/proc/45/map_files’: Permission denied\r\n",
      "find: ‘/proc/45/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/45/ns’: Permission denied\r\n",
      "find: ‘/proc/46/task/46/fd’: Permission denied\r\n",
      "find: ‘/proc/46/task/46/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/46/task/46/ns’: Permission denied\r\n",
      "find: ‘/proc/46/fd’: Permission denied\r\n",
      "find: ‘/proc/46/map_files’: Permission denied\r\n",
      "find: ‘/proc/46/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/46/ns’: Permission denied\r\n",
      "find: ‘/proc/48/task/48/fd’: Permission denied\r\n",
      "find: ‘/proc/48/task/48/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/48/task/48/ns’: Permission denied\r\n",
      "find: ‘/proc/48/fd’: Permission denied\r\n",
      "find: ‘/proc/48/map_files’: Permission denied\r\n",
      "find: ‘/proc/48/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/48/ns’: Permission denied\r\n",
      "find: ‘/proc/49/task/49/fd’: Permission denied\r\n",
      "find: ‘/proc/49/task/49/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/49/task/49/ns’: Permission denied\r\n",
      "find: ‘/proc/49/fd’: Permission denied\r\n",
      "find: ‘/proc/49/map_files’: Permission denied\r\n",
      "find: ‘/proc/49/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/49/ns’: Permission denied\r\n",
      "find: ‘/proc/50/task/50/fd’: Permission denied\r\n",
      "find: ‘/proc/50/task/50/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/50/task/50/ns’: Permission denied\r\n",
      "find: ‘/proc/50/fd’: Permission denied\r\n",
      "find: ‘/proc/50/map_files’: Permission denied\r\n",
      "find: ‘/proc/50/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/50/ns’: Permission denied\r\n",
      "find: ‘/proc/51/task/51/fd’: Permission denied\r\n",
      "find: ‘/proc/51/task/51/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/51/task/51/ns’: Permission denied\r\n",
      "find: ‘/proc/51/fd’: Permission denied\r\n",
      "find: ‘/proc/51/map_files’: Permission denied\r\n",
      "find: ‘/proc/51/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/51/ns’: Permission denied\r\n",
      "find: ‘/proc/52/task/52/fd’: Permission denied\r\n",
      "find: ‘/proc/52/task/52/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/52/task/52/ns’: Permission denied\r\n",
      "find: ‘/proc/52/fd’: Permission denied\r\n",
      "find: ‘/proc/52/map_files’: Permission denied\r\n",
      "find: ‘/proc/52/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/52/ns’: Permission denied\r\n",
      "find: ‘/proc/53/task/53/fd’: Permission denied\r\n",
      "find: ‘/proc/53/task/53/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/53/task/53/ns’: Permission denied\r\n",
      "find: ‘/proc/53/fd’: Permission denied\r\n",
      "find: ‘/proc/53/map_files’: Permission denied\r\n",
      "find: ‘/proc/53/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/53/ns’: Permission denied\r\n",
      "find: ‘/proc/54/task/54/fd’: Permission denied\r\n",
      "find: ‘/proc/54/task/54/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/54/task/54/ns’: Permission denied\r\n",
      "find: ‘/proc/54/fd’: Permission denied\r\n",
      "find: ‘/proc/54/map_files’: Permission denied\r\n",
      "find: ‘/proc/54/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/54/ns’: Permission denied\r\n",
      "find: ‘/proc/55/task/55/fd’: Permission denied\r\n",
      "find: ‘/proc/55/task/55/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/55/task/55/ns’: Permission denied\r\n",
      "find: ‘/proc/55/fd’: Permission denied\r\n",
      "find: ‘/proc/55/map_files’: Permission denied\r\n",
      "find: ‘/proc/55/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/55/ns’: Permission denied\r\n",
      "find: ‘/proc/56/task/56/fd’: Permission denied\r\n",
      "find: ‘/proc/56/task/56/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/56/task/56/ns’: Permission denied\r\n",
      "find: ‘/proc/56/fd’: Permission denied\r\n",
      "find: ‘/proc/56/map_files’: Permission denied\r\n",
      "find: ‘/proc/56/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/56/ns’: Permission denied\r\n",
      "find: ‘/proc/57/task/57/fd’: Permission denied\r\n",
      "find: ‘/proc/57/task/57/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/57/task/57/ns’: Permission denied\r\n",
      "find: ‘/proc/57/fd’: Permission denied\r\n",
      "find: ‘/proc/57/map_files’: Permission denied\r\n",
      "find: ‘/proc/57/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/57/ns’: Permission denied\r\n",
      "find: ‘/proc/58/task/58/fd’: Permission denied\r\n",
      "find: ‘/proc/58/task/58/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/58/task/58/ns’: Permission denied\r\n",
      "find: ‘/proc/58/fd’: Permission denied\r\n",
      "find: ‘/proc/58/map_files’: Permission denied\r\n",
      "find: ‘/proc/58/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/58/ns’: Permission denied\r\n",
      "find: ‘/proc/59/task/59/fd’: Permission denied\r\n",
      "find: ‘/proc/59/task/59/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/59/task/59/ns’: Permission denied\r\n",
      "find: ‘/proc/59/fd’: Permission denied\r\n",
      "find: ‘/proc/59/map_files’: Permission denied\r\n",
      "find: ‘/proc/59/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/59/ns’: Permission denied\r\n",
      "find: ‘/proc/60/task/60/fd’: Permission denied\r\n",
      "find: ‘/proc/60/task/60/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/60/task/60/ns’: Permission denied\r\n",
      "find: ‘/proc/60/fd’: Permission denied\r\n",
      "find: ‘/proc/60/map_files’: Permission denied\r\n",
      "find: ‘/proc/60/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/60/ns’: Permission denied\r\n",
      "find: ‘/proc/61/task/61/fd’: Permission denied\r\n",
      "find: ‘/proc/61/task/61/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/61/task/61/ns’: Permission denied\r\n",
      "find: ‘/proc/61/fd’: Permission denied\r\n",
      "find: ‘/proc/61/map_files’: Permission denied\r\n",
      "find: ‘/proc/61/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/61/ns’: Permission denied\r\n",
      "find: ‘/proc/62/task/62/fd’: Permission denied\r\n",
      "find: ‘/proc/62/task/62/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/62/task/62/ns’: Permission denied\r\n",
      "find: ‘/proc/62/fd’: Permission denied\r\n",
      "find: ‘/proc/62/map_files’: Permission denied\r\n",
      "find: ‘/proc/62/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/62/ns’: Permission denied\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "find: ‘/proc/63/task/63/fd’: Permission denied\r\n",
      "find: ‘/proc/63/task/63/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/63/task/63/ns’: Permission denied\r\n",
      "find: ‘/proc/63/fd’: Permission denied\r\n",
      "find: ‘/proc/63/map_files’: Permission denied\r\n",
      "find: ‘/proc/63/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/63/ns’: Permission denied\r\n",
      "find: ‘/proc/64/task/64/fd’: Permission denied\r\n",
      "find: ‘/proc/64/task/64/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/64/task/64/ns’: Permission denied\r\n",
      "find: ‘/proc/64/fd’: Permission denied\r\n",
      "find: ‘/proc/64/map_files’: Permission denied\r\n",
      "find: ‘/proc/64/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/64/ns’: Permission denied\r\n",
      "find: ‘/proc/65/task/65/fd’: Permission denied\r\n",
      "find: ‘/proc/65/task/65/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/65/task/65/ns’: Permission denied\r\n",
      "find: ‘/proc/65/fd’: Permission denied\r\n",
      "find: ‘/proc/65/map_files’: Permission denied\r\n",
      "find: ‘/proc/65/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/65/ns’: Permission denied\r\n",
      "find: ‘/proc/66/task/66/fd’: Permission denied\r\n",
      "find: ‘/proc/66/task/66/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/66/task/66/ns’: Permission denied\r\n",
      "find: ‘/proc/66/fd’: Permission denied\r\n",
      "find: ‘/proc/66/map_files’: Permission denied\r\n",
      "find: ‘/proc/66/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/66/ns’: Permission denied\r\n",
      "find: ‘/proc/67/task/67/fd’: Permission denied\r\n",
      "find: ‘/proc/67/task/67/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/67/task/67/ns’: Permission denied\r\n",
      "find: ‘/proc/67/fd’: Permission denied\r\n",
      "find: ‘/proc/67/map_files’: Permission denied\r\n",
      "find: ‘/proc/67/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/67/ns’: Permission denied\r\n",
      "find: ‘/proc/68/task/68/fd’: Permission denied\r\n",
      "find: ‘/proc/68/task/68/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/68/task/68/ns’: Permission denied\r\n",
      "find: ‘/proc/68/fd’: Permission denied\r\n",
      "find: ‘/proc/68/map_files’: Permission denied\r\n",
      "find: ‘/proc/68/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/68/ns’: Permission denied\r\n",
      "find: ‘/proc/69/task/69/fd’: Permission denied\r\n",
      "find: ‘/proc/69/task/69/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/69/task/69/ns’: Permission denied\r\n",
      "find: ‘/proc/69/fd’: Permission denied\r\n",
      "find: ‘/proc/69/map_files’: Permission denied\r\n",
      "find: ‘/proc/69/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/69/ns’: Permission denied\r\n",
      "find: ‘/proc/70/task/70/fd’: Permission denied\r\n",
      "find: ‘/proc/70/task/70/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/70/task/70/ns’: Permission denied\r\n",
      "find: ‘/proc/70/fd’: Permission denied\r\n",
      "find: ‘/proc/70/map_files’: Permission denied\r\n",
      "find: ‘/proc/70/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/70/ns’: Permission denied\r\n",
      "find: ‘/proc/71/task/71/fd’: Permission denied\r\n",
      "find: ‘/proc/71/task/71/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/71/task/71/ns’: Permission denied\r\n",
      "find: ‘/proc/71/fd’: Permission denied\r\n",
      "find: ‘/proc/71/map_files’: Permission denied\r\n",
      "find: ‘/proc/71/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/71/ns’: Permission denied\r\n",
      "find: ‘/proc/73/task/73/fd’: Permission denied\r\n",
      "find: ‘/proc/73/task/73/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/73/task/73/ns’: Permission denied\r\n",
      "find: ‘/proc/73/fd’: Permission denied\r\n",
      "find: ‘/proc/73/map_files’: Permission denied\r\n",
      "find: ‘/proc/73/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/73/ns’: Permission denied\r\n",
      "find: ‘/proc/74/task/74/fd’: Permission denied\r\n",
      "find: ‘/proc/74/task/74/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/74/task/74/ns’: Permission denied\r\n",
      "find: ‘/proc/74/fd’: Permission denied\r\n",
      "find: ‘/proc/74/map_files’: Permission denied\r\n",
      "find: ‘/proc/74/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/74/ns’: Permission denied\r\n",
      "find: ‘/proc/75/task/75/fd’: Permission denied\r\n",
      "find: ‘/proc/75/task/75/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/75/task/75/ns’: Permission denied\r\n",
      "find: ‘/proc/75/fd’: Permission denied\r\n",
      "find: ‘/proc/75/map_files’: Permission denied\r\n",
      "find: ‘/proc/75/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/75/ns’: Permission denied\r\n",
      "find: ‘/proc/76/task/76/fd’: Permission denied\r\n",
      "find: ‘/proc/76/task/76/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/76/task/76/ns’: Permission denied\r\n",
      "find: ‘/proc/76/fd’: Permission denied\r\n",
      "find: ‘/proc/76/map_files’: Permission denied\r\n",
      "find: ‘/proc/76/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/76/ns’: Permission denied\r\n",
      "find: ‘/proc/78/task/78/fd’: Permission denied\r\n",
      "find: ‘/proc/78/task/78/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/78/task/78/ns’: Permission denied\r\n",
      "find: ‘/proc/78/fd’: Permission denied\r\n",
      "find: ‘/proc/78/map_files’: Permission denied\r\n",
      "find: ‘/proc/78/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/78/ns’: Permission denied\r\n",
      "find: ‘/proc/79/task/79/fd’: Permission denied\r\n",
      "find: ‘/proc/79/task/79/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/79/task/79/ns’: Permission denied\r\n",
      "find: ‘/proc/79/fd’: Permission denied\r\n",
      "find: ‘/proc/79/map_files’: Permission denied\r\n",
      "find: ‘/proc/79/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/79/ns’: Permission denied\r\n",
      "find: ‘/proc/80/task/80/fd’: Permission denied\r\n",
      "find: ‘/proc/80/task/80/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/80/task/80/ns’: Permission denied\r\n",
      "find: ‘/proc/80/fd’: Permission denied\r\n",
      "find: ‘/proc/80/map_files’: Permission denied\r\n",
      "find: ‘/proc/80/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/80/ns’: Permission denied\r\n",
      "find: ‘/proc/81/task/81/fd’: Permission denied\r\n",
      "find: ‘/proc/81/task/81/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/81/task/81/ns’: Permission denied\r\n",
      "find: ‘/proc/81/fd’: Permission denied\r\n",
      "find: ‘/proc/81/map_files’: Permission denied\r\n",
      "find: ‘/proc/81/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/81/ns’: Permission denied\r\n",
      "find: ‘/proc/82/task/82/fd’: Permission denied\r\n",
      "find: ‘/proc/82/task/82/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/82/task/82/ns’: Permission denied\r\n",
      "find: ‘/proc/82/fd’: Permission denied\r\n",
      "find: ‘/proc/82/map_files’: Permission denied\r\n",
      "find: ‘/proc/82/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/82/ns’: Permission denied\r\n",
      "find: ‘/proc/83/task/83/fd’: Permission denied\r\n",
      "find: ‘/proc/83/task/83/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/83/task/83/ns’: Permission denied\r\n",
      "find: ‘/proc/83/fd’: Permission denied\r\n",
      "find: ‘/proc/83/map_files’: Permission denied\r\n",
      "find: ‘/proc/83/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/83/ns’: Permission denied\r\n",
      "find: ‘/proc/84/task/84/fd’: Permission denied\r\n",
      "find: ‘/proc/84/task/84/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/84/task/84/ns’: Permission denied\r\n",
      "find: ‘/proc/84/fd’: Permission denied\r\n",
      "find: ‘/proc/84/map_files’: Permission denied\r\n",
      "find: ‘/proc/84/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/84/ns’: Permission denied\r\n",
      "find: ‘/proc/85/task/85/fd’: Permission denied\r\n",
      "find: ‘/proc/85/task/85/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/85/task/85/ns’: Permission denied\r\n",
      "find: ‘/proc/85/fd’: Permission denied\r\n",
      "find: ‘/proc/85/map_files’: Permission denied\r\n",
      "find: ‘/proc/85/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/85/ns’: Permission denied\r\n",
      "find: ‘/proc/86/task/86/fd’: Permission denied\r\n",
      "find: ‘/proc/86/task/86/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/86/task/86/ns’: Permission denied\r\n",
      "find: ‘/proc/86/fd’: Permission denied\r\n",
      "find: ‘/proc/86/map_files’: Permission denied\r\n",
      "find: ‘/proc/86/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/86/ns’: Permission denied\r\n",
      "find: ‘/proc/87/task/87/fd’: Permission denied\r\n",
      "find: ‘/proc/87/task/87/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/87/task/87/ns’: Permission denied\r\n",
      "find: ‘/proc/87/fd’: Permission denied\r\n",
      "find: ‘/proc/87/map_files’: Permission denied\r\n",
      "find: ‘/proc/87/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/87/ns’: Permission denied\r\n",
      "find: ‘/proc/88/task/88/fd’: Permission denied\r\n",
      "find: ‘/proc/88/task/88/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/88/task/88/ns’: Permission denied\r\n",
      "find: ‘/proc/88/fd’: Permission denied\r\n",
      "find: ‘/proc/88/map_files’: Permission denied\r\n",
      "find: ‘/proc/88/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/88/ns’: Permission denied\r\n",
      "find: ‘/proc/89/task/89/fd’: Permission denied\r\n",
      "find: ‘/proc/89/task/89/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/89/task/89/ns’: Permission denied\r\n",
      "find: ‘/proc/89/fd’: Permission denied\r\n",
      "find: ‘/proc/89/map_files’: Permission denied\r\n",
      "find: ‘/proc/89/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/89/ns’: Permission denied\r\n",
      "find: ‘/proc/90/task/90/fd’: Permission denied\r\n",
      "find: ‘/proc/90/task/90/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/90/task/90/ns’: Permission denied\r\n",
      "find: ‘/proc/90/fd’: Permission denied\r\n",
      "find: ‘/proc/90/map_files’: Permission denied\r\n",
      "find: ‘/proc/90/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/90/ns’: Permission denied\r\n",
      "find: ‘/proc/91/task/91/fd’: Permission denied\r\n",
      "find: ‘/proc/91/task/91/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/91/task/91/ns’: Permission denied\r\n",
      "find: ‘/proc/91/fd’: Permission denied\r\n",
      "find: ‘/proc/91/map_files’: Permission denied\r\n",
      "find: ‘/proc/91/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/91/ns’: Permission denied\r\n",
      "find: ‘/proc/92/task/92/fd’: Permission denied\r\n",
      "find: ‘/proc/92/task/92/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/92/task/92/ns’: Permission denied\r\n",
      "find: ‘/proc/92/fd’: Permission denied\r\n",
      "find: ‘/proc/92/map_files’: Permission denied\r\n",
      "find: ‘/proc/92/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/92/ns’: Permission denied\r\n",
      "find: ‘/proc/93/task/93/fd’: Permission denied\r\n",
      "find: ‘/proc/93/task/93/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/93/task/93/ns’: Permission denied\r\n",
      "find: ‘/proc/93/fd’: Permission denied\r\n",
      "find: ‘/proc/93/map_files’: Permission denied\r\n",
      "find: ‘/proc/93/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/93/ns’: Permission denied\r\n",
      "find: ‘/proc/94/task/94/fd’: Permission denied\r\n",
      "find: ‘/proc/94/task/94/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/94/task/94/ns’: Permission denied\r\n",
      "find: ‘/proc/94/fd’: Permission denied\r\n",
      "find: ‘/proc/94/map_files’: Permission denied\r\n",
      "find: ‘/proc/94/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/94/ns’: Permission denied\r\n",
      "find: ‘/proc/95/task/95/fd’: Permission denied\r\n",
      "find: ‘/proc/95/task/95/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/95/task/95/ns’: Permission denied\r\n",
      "find: ‘/proc/95/fd’: Permission denied\r\n",
      "find: ‘/proc/95/map_files’: Permission denied\r\n",
      "find: ‘/proc/95/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/95/ns’: Permission denied\r\n",
      "find: ‘/proc/102/task/102/fd’: Permission denied\r\n",
      "find: ‘/proc/102/task/102/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/102/task/102/ns’: Permission denied\r\n",
      "find: ‘/proc/102/fd’: Permission denied\r\n",
      "find: ‘/proc/102/map_files’: Permission denied\r\n",
      "find: ‘/proc/102/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/102/ns’: Permission denied\r\n",
      "find: ‘/proc/104/task/104/fd’: Permission denied\r\n",
      "find: ‘/proc/104/task/104/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/104/task/104/ns’: Permission denied\r\n",
      "find: ‘/proc/104/fd’: Permission denied\r\n",
      "find: ‘/proc/104/map_files’: Permission denied\r\n",
      "find: ‘/proc/104/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/104/ns’: Permission denied\r\n",
      "find: ‘/proc/108/task/108/fd’: Permission denied\r\n",
      "find: ‘/proc/108/task/108/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/108/task/108/ns’: Permission denied\r\n",
      "find: ‘/proc/108/fd’: Permission denied\r\n",
      "find: ‘/proc/108/map_files’: Permission denied\r\n",
      "find: ‘/proc/108/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/108/ns’: Permission denied\r\n",
      "find: ‘/proc/109/task/109/fd’: Permission denied\r\n",
      "find: ‘/proc/109/task/109/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/109/task/109/ns’: Permission denied\r\n",
      "find: ‘/proc/109/fd’: Permission denied\r\n",
      "find: ‘/proc/109/map_files’: Permission denied\r\n",
      "find: ‘/proc/109/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/109/ns’: Permission denied\r\n",
      "find: ‘/proc/110/task/110/fd’: Permission denied\r\n",
      "find: ‘/proc/110/task/110/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/110/task/110/ns’: Permission denied\r\n",
      "find: ‘/proc/110/fd’: Permission denied\r\n",
      "find: ‘/proc/110/map_files’: Permission denied\r\n",
      "find: ‘/proc/110/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/110/ns’: Permission denied\r\n",
      "find: ‘/proc/111/task/111/fd’: Permission denied\r\n",
      "find: ‘/proc/111/task/111/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/111/task/111/ns’: Permission denied\r\n",
      "find: ‘/proc/111/fd’: Permission denied\r\n",
      "find: ‘/proc/111/map_files’: Permission denied\r\n",
      "find: ‘/proc/111/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/111/ns’: Permission denied\r\n",
      "find: ‘/proc/154/task/154/fd’: Permission denied\r\n",
      "find: ‘/proc/154/task/154/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/154/task/154/ns’: Permission denied\r\n",
      "find: ‘/proc/154/fd’: Permission denied\r\n",
      "find: ‘/proc/154/map_files’: Permission denied\r\n",
      "find: ‘/proc/154/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/154/ns’: Permission denied\r\n",
      "find: ‘/proc/176/task/176/fd’: Permission denied\r\n",
      "find: ‘/proc/176/task/176/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/176/task/176/ns’: Permission denied\r\n",
      "find: ‘/proc/176/fd’: Permission denied\r\n",
      "find: ‘/proc/176/map_files’: Permission denied\r\n",
      "find: ‘/proc/176/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/176/ns’: Permission denied\r\n",
      "find: ‘/proc/187/task/187/fd’: Permission denied\r\n",
      "find: ‘/proc/187/task/187/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/187/task/187/ns’: Permission denied\r\n",
      "find: ‘/proc/187/fd’: Permission denied\r\n",
      "find: ‘/proc/187/map_files’: Permission denied\r\n",
      "find: ‘/proc/187/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/187/ns’: Permission denied\r\n",
      "find: ‘/proc/190/task/190/fd’: Permission denied\r\n",
      "find: ‘/proc/190/task/190/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/190/task/190/ns’: Permission denied\r\n",
      "find: ‘/proc/190/fd’: Permission denied\r\n",
      "find: ‘/proc/190/map_files’: Permission denied\r\n",
      "find: ‘/proc/190/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/190/ns’: Permission denied\r\n",
      "find: ‘/proc/193/task/193/fd’: Permission denied\r\n",
      "find: ‘/proc/193/task/193/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/193/task/193/ns’: Permission denied\r\n",
      "find: ‘/proc/193/fd’: Permission denied\r\n",
      "find: ‘/proc/193/map_files’: Permission denied\r\n",
      "find: ‘/proc/193/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/193/ns’: Permission denied\r\n",
      "find: ‘/proc/194/task/194/fd’: Permission denied\r\n",
      "find: ‘/proc/194/task/194/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/194/task/194/ns’: Permission denied\r\n",
      "find: ‘/proc/194/fd’: Permission denied\r\n",
      "find: ‘/proc/194/map_files’: Permission denied\r\n",
      "find: ‘/proc/194/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/194/ns’: Permission denied\r\n",
      "find: ‘/proc/195/task/195/fd’: Permission denied\r\n",
      "find: ‘/proc/195/task/195/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/195/task/195/ns’: Permission denied\r\n",
      "find: ‘/proc/195/fd’: Permission denied\r\n",
      "find: ‘/proc/195/map_files’: Permission denied\r\n",
      "find: ‘/proc/195/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/195/ns’: Permission denied\r\n",
      "find: ‘/proc/196/task/196/fd’: Permission denied\r\n",
      "find: ‘/proc/196/task/196/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/196/task/196/ns’: Permission denied\r\n",
      "find: ‘/proc/196/task/200/fd’: Permission denied\r\n",
      "find: ‘/proc/196/task/200/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/196/task/200/ns’: Permission denied\r\n",
      "find: ‘/proc/196/task/202/fd’: Permission denied\r\n",
      "find: ‘/proc/196/task/202/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/196/task/202/ns’: Permission denied\r\n",
      "find: ‘/proc/196/task/203/fd’: Permission denied\r\n",
      "find: ‘/proc/196/task/203/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/196/task/203/ns’: Permission denied\r\n",
      "find: ‘/proc/196/task/204/fd’: Permission denied\r\n",
      "find: ‘/proc/196/task/204/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/196/task/204/ns’: Permission denied\r\n",
      "find: ‘/proc/196/task/205/fd’: Permission denied\r\n",
      "find: ‘/proc/196/task/205/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/196/task/205/ns’: Permission denied\r\n",
      "find: ‘/proc/196/task/206/fd’: Permission denied\r\n",
      "find: ‘/proc/196/task/206/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/196/task/206/ns’: Permission denied\r\n",
      "find: ‘/proc/196/fd’: Permission denied\r\n",
      "find: ‘/proc/196/map_files’: Permission denied\r\n",
      "find: ‘/proc/196/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/196/ns’: Permission denied\r\n",
      "find: ‘/proc/201/task/201/fd’: Permission denied\r\n",
      "find: ‘/proc/201/task/201/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/201/task/201/ns’: Permission denied\r\n",
      "find: ‘/proc/201/fd’: Permission denied\r\n",
      "find: ‘/proc/201/map_files’: Permission denied\r\n",
      "find: ‘/proc/201/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/201/ns’: Permission denied\r\n",
      "find: ‘/proc/249/task/249/fd’: Permission denied\r\n",
      "find: ‘/proc/249/task/249/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/249/task/249/ns’: Permission denied\r\n",
      "find: ‘/proc/249/fd’: Permission denied\r\n",
      "find: ‘/proc/249/map_files’: Permission denied\r\n",
      "find: ‘/proc/249/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/249/ns’: Permission denied\r\n",
      "find: ‘/proc/340/task/340/fd’: Permission denied\r\n",
      "find: ‘/proc/340/task/340/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/340/task/340/ns’: Permission denied\r\n",
      "find: ‘/proc/340/fd’: Permission denied\r\n",
      "find: ‘/proc/340/map_files’: Permission denied\r\n",
      "find: ‘/proc/340/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/340/ns’: Permission denied\r\n",
      "find: ‘/proc/390/task/390/fd’: Permission denied\r\n",
      "find: ‘/proc/390/task/390/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/390/task/390/ns’: Permission denied\r\n",
      "find: ‘/proc/390/fd’: Permission denied\r\n",
      "find: ‘/proc/390/map_files’: Permission denied\r\n",
      "find: ‘/proc/390/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/390/ns’: Permission denied\r\n",
      "find: ‘/proc/466/task/466/fd’: Permission denied\r\n",
      "find: ‘/proc/466/task/466/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/466/task/466/ns’: Permission denied\r\n",
      "find: ‘/proc/466/fd’: Permission denied\r\n",
      "find: ‘/proc/466/map_files’: Permission denied\r\n",
      "find: ‘/proc/466/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/466/ns’: Permission denied\r\n",
      "find: ‘/proc/469/task/469/fd’: Permission denied\r\n",
      "find: ‘/proc/469/task/469/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/469/task/469/ns’: Permission denied\r\n",
      "find: ‘/proc/469/fd’: Permission denied\r\n",
      "find: ‘/proc/469/map_files’: Permission denied\r\n",
      "find: ‘/proc/469/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/469/ns’: Permission denied\r\n",
      "find: ‘/proc/536/task/536/fd’: Permission denied\r\n",
      "find: ‘/proc/536/task/536/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/536/task/536/ns’: Permission denied\r\n",
      "find: ‘/proc/536/fd’: Permission denied\r\n",
      "find: ‘/proc/536/map_files’: Permission denied\r\n",
      "find: ‘/proc/536/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/536/ns’: Permission denied\r\n",
      "find: ‘/proc/537/task/537/fd’: Permission denied\r\n",
      "find: ‘/proc/537/task/537/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/537/task/537/ns’: Permission denied\r\n",
      "find: ‘/proc/537/fd’: Permission denied\r\n",
      "find: ‘/proc/537/map_files’: Permission denied\r\n",
      "find: ‘/proc/537/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/537/ns’: Permission denied\r\n",
      "find: ‘/proc/579/task/579/fd’: Permission denied\r\n",
      "find: ‘/proc/579/task/579/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/579/task/579/ns’: Permission denied\r\n",
      "find: ‘/proc/579/fd’: Permission denied\r\n",
      "find: ‘/proc/579/map_files’: Permission denied\r\n",
      "find: ‘/proc/579/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/579/ns’: Permission denied\r\n",
      "find: ‘/proc/585/task/585/fd’: Permission denied\r\n",
      "find: ‘/proc/585/task/585/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/585/task/585/ns’: Permission denied\r\n",
      "find: ‘/proc/585/task/589/fd’: Permission denied\r\n",
      "find: ‘/proc/585/task/589/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/585/task/589/ns’: Permission denied\r\n",
      "find: ‘/proc/585/fd’: Permission denied\r\n",
      "find: ‘/proc/585/map_files’: Permission denied\r\n",
      "find: ‘/proc/585/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/585/ns’: Permission denied\r\n",
      "find: ‘/proc/586/task/586/fd’: Permission denied\r\n",
      "find: ‘/proc/586/task/586/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/586/task/586/ns’: Permission denied\r\n",
      "find: ‘/proc/586/fd’: Permission denied\r\n",
      "find: ‘/proc/586/map_files’: Permission denied\r\n",
      "find: ‘/proc/586/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/586/ns’: Permission denied\r\n",
      "find: ‘/proc/588/task/588/fd’: Permission denied\r\n",
      "find: ‘/proc/588/task/588/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/588/task/588/ns’: Permission denied\r\n",
      "find: ‘/proc/588/fd’: Permission denied\r\n",
      "find: ‘/proc/588/map_files’: Permission denied\r\n",
      "find: ‘/proc/588/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/588/ns’: Permission denied\r\n",
      "find: ‘/proc/600/task/600/fd’: Permission denied\r\n",
      "find: ‘/proc/600/task/600/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/600/task/600/ns’: Permission denied\r\n",
      "find: ‘/proc/600/task/631/fd’: Permission denied\r\n",
      "find: ‘/proc/600/task/631/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/600/task/631/ns’: Permission denied\r\n",
      "find: ‘/proc/600/task/632/fd’: Permission denied\r\n",
      "find: ‘/proc/600/task/632/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/600/task/632/ns’: Permission denied\r\n",
      "find: ‘/proc/600/task/633/fd’: Permission denied\r\n",
      "find: ‘/proc/600/task/633/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/600/task/633/ns’: Permission denied\r\n",
      "find: ‘/proc/600/fd’: Permission denied\r\n",
      "find: ‘/proc/600/map_files’: Permission denied\r\n",
      "find: ‘/proc/600/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/600/ns’: Permission denied\r\n",
      "find: ‘/proc/603/task/603/fd’: Permission denied\r\n",
      "find: ‘/proc/603/task/603/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/603/task/603/ns’: Permission denied\r\n",
      "find: ‘/proc/603/task/645/fd’: Permission denied\r\n",
      "find: ‘/proc/603/task/645/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/603/task/645/ns’: Permission denied\r\n",
      "find: ‘/proc/603/task/655/fd’: Permission denied\r\n",
      "find: ‘/proc/603/task/655/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/603/task/655/ns’: Permission denied\r\n",
      "find: ‘/proc/603/task/660/fd’: Permission denied\r\n",
      "find: ‘/proc/603/task/660/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/603/task/660/ns’: Permission denied\r\n",
      "find: ‘/proc/603/task/661/fd’: Permission denied\r\n",
      "find: ‘/proc/603/task/661/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/603/task/661/ns’: Permission denied\r\n",
      "find: ‘/proc/603/task/663/fd’: Permission denied\r\n",
      "find: ‘/proc/603/task/663/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/603/task/663/ns’: Permission denied\r\n",
      "find: ‘/proc/603/task/778/fd’: Permission denied\r\n",
      "find: ‘/proc/603/task/778/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/603/task/778/ns’: Permission denied\r\n",
      "find: ‘/proc/603/task/1489/fd’: Permission denied\r\n",
      "find: ‘/proc/603/task/1489/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/603/task/1489/ns’: Permission denied\r\n",
      "find: ‘/proc/603/task/1490/fd’: Permission denied\r\n",
      "find: ‘/proc/603/task/1490/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/603/task/1490/ns’: Permission denied\r\n",
      "find: ‘/proc/603/task/1491/fd’: Permission denied\r\n",
      "find: ‘/proc/603/task/1491/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/603/task/1491/ns’: Permission denied\r\n",
      "find: ‘/proc/603/task/1493/fd’: Permission denied\r\n",
      "find: ‘/proc/603/task/1493/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/603/task/1493/ns’: Permission denied\r\n",
      "find: ‘/proc/603/task/1533/fd’: Permission denied\r\n",
      "find: ‘/proc/603/task/1533/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/603/task/1533/ns’: Permission denied\r\n",
      "find: ‘/proc/603/task/5828/fd’: Permission denied\r\n",
      "find: ‘/proc/603/task/5828/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/603/task/5828/ns’: Permission denied\r\n",
      "find: ‘/proc/603/task/6864/fd’: Permission denied\r\n",
      "find: ‘/proc/603/task/6864/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/603/task/6864/ns’: Permission denied\r\n",
      "find: ‘/proc/603/fd’: Permission denied\r\n",
      "find: ‘/proc/603/map_files’: Permission denied\r\n",
      "find: ‘/proc/603/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/603/ns’: Permission denied\r\n",
      "find: ‘/proc/616/task/616/fd’: Permission denied\r\n",
      "find: ‘/proc/616/task/616/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/616/task/616/ns’: Permission denied\r\n",
      "find: ‘/proc/616/fd’: Permission denied\r\n",
      "find: ‘/proc/616/map_files’: Permission denied\r\n",
      "find: ‘/proc/616/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/616/ns’: Permission denied\r\n",
      "find: ‘/proc/617/task/617/fd’: Permission denied\r\n",
      "find: ‘/proc/617/task/617/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/617/task/617/ns’: Permission denied\r\n",
      "find: ‘/proc/617/task/939/fd’: Permission denied\r\n",
      "find: ‘/proc/617/task/939/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/617/task/939/ns’: Permission denied\r\n",
      "find: ‘/proc/617/task/942/fd’: Permission denied\r\n",
      "find: ‘/proc/617/task/942/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/617/task/942/ns’: Permission denied\r\n",
      "find: ‘/proc/617/task/943/fd’: Permission denied\r\n",
      "find: ‘/proc/617/task/943/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/617/task/943/ns’: Permission denied\r\n",
      "find: ‘/proc/617/task/944/fd’: Permission denied\r\n",
      "find: ‘/proc/617/task/944/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/617/task/944/ns’: Permission denied\r\n",
      "find: ‘/proc/617/task/945/fd’: Permission denied\r\n",
      "find: ‘/proc/617/task/945/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/617/task/945/ns’: Permission denied\r\n",
      "find: ‘/proc/617/task/946/fd’: Permission denied\r\n",
      "find: ‘/proc/617/task/946/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/617/task/946/ns’: Permission denied\r\n",
      "find: ‘/proc/617/task/1151/fd’: Permission denied\r\n",
      "find: ‘/proc/617/task/1151/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/617/task/1151/ns’: Permission denied\r\n",
      "find: ‘/proc/617/task/1155/fd’: Permission denied\r\n",
      "find: ‘/proc/617/task/1155/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/617/task/1155/ns’: Permission denied\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "find: ‘/proc/617/task/1350/fd’: Permission denied\r\n",
      "find: ‘/proc/617/task/1350/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/617/task/1350/ns’: Permission denied\r\n",
      "find: ‘/proc/617/task/2688/fd’: Permission denied\r\n",
      "find: ‘/proc/617/task/2688/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/617/task/2688/ns’: Permission denied\r\n",
      "find: ‘/proc/617/fd’: Permission denied\r\n",
      "find: ‘/proc/617/map_files’: Permission denied\r\n",
      "find: ‘/proc/617/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/617/ns’: Permission denied\r\n",
      "find: ‘/proc/620/task/620/fd’: Permission denied\r\n",
      "find: ‘/proc/620/task/620/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/620/task/620/ns’: Permission denied\r\n",
      "find: ‘/proc/620/fd’: Permission denied\r\n",
      "find: ‘/proc/620/map_files’: Permission denied\r\n",
      "find: ‘/proc/620/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/620/ns’: Permission denied\r\n",
      "find: ‘/proc/622/task/622/fd’: Permission denied\r\n",
      "find: ‘/proc/622/task/622/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/622/task/622/ns’: Permission denied\r\n",
      "find: ‘/proc/622/fd’: Permission denied\r\n",
      "find: ‘/proc/622/map_files’: Permission denied\r\n",
      "find: ‘/proc/622/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/622/ns’: Permission denied\r\n",
      "find: ‘/proc/627/task/627/fd’: Permission denied\r\n",
      "find: ‘/proc/627/task/627/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/627/task/627/ns’: Permission denied\r\n",
      "find: ‘/proc/627/fd’: Permission denied\r\n",
      "find: ‘/proc/627/map_files’: Permission denied\r\n",
      "find: ‘/proc/627/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/627/ns’: Permission denied\r\n",
      "find: ‘/proc/628/task/628/fd’: Permission denied\r\n",
      "find: ‘/proc/628/task/628/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/628/task/628/ns’: Permission denied\r\n",
      "find: ‘/proc/628/task/682/fd’: Permission denied\r\n",
      "find: ‘/proc/628/task/682/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/628/task/682/ns’: Permission denied\r\n",
      "find: ‘/proc/628/task/684/fd’: Permission denied\r\n",
      "find: ‘/proc/628/task/684/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/628/task/684/ns’: Permission denied\r\n",
      "find: ‘/proc/628/task/689/fd’: Permission denied\r\n",
      "find: ‘/proc/628/task/689/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/628/task/689/ns’: Permission denied\r\n",
      "find: ‘/proc/628/task/690/fd’: Permission denied\r\n",
      "find: ‘/proc/628/task/690/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/628/task/690/ns’: Permission denied\r\n",
      "find: ‘/proc/628/task/700/fd’: Permission denied\r\n",
      "find: ‘/proc/628/task/700/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/628/task/700/ns’: Permission denied\r\n",
      "find: ‘/proc/628/task/755/fd’: Permission denied\r\n",
      "find: ‘/proc/628/task/755/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/628/task/755/ns’: Permission denied\r\n",
      "find: ‘/proc/628/task/800/fd’: Permission denied\r\n",
      "find: ‘/proc/628/task/800/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/628/task/800/ns’: Permission denied\r\n",
      "find: ‘/proc/628/task/815/fd’: Permission denied\r\n",
      "find: ‘/proc/628/task/815/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/628/task/815/ns’: Permission denied\r\n",
      "find: ‘/proc/628/fd’: Permission denied\r\n",
      "find: ‘/proc/628/map_files’: Permission denied\r\n",
      "find: ‘/proc/628/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/628/ns’: Permission denied\r\n",
      "find: ‘/proc/649/task/649/fd’: Permission denied\r\n",
      "find: ‘/proc/649/task/649/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/649/task/649/ns’: Permission denied\r\n",
      "find: ‘/proc/649/fd’: Permission denied\r\n",
      "find: ‘/proc/649/map_files’: Permission denied\r\n",
      "find: ‘/proc/649/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/649/ns’: Permission denied\r\n",
      "find: ‘/proc/673/task/673/fd’: Permission denied\r\n",
      "find: ‘/proc/673/task/673/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/673/task/673/ns’: Permission denied\r\n",
      "find: ‘/proc/673/fd’: Permission denied\r\n",
      "find: ‘/proc/673/map_files’: Permission denied\r\n",
      "find: ‘/proc/673/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/673/ns’: Permission denied\r\n",
      "find: ‘/proc/704/task/704/fd’: Permission denied\r\n",
      "find: ‘/proc/704/task/704/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/704/task/704/ns’: Permission denied\r\n",
      "find: ‘/proc/704/task/738/fd’: Permission denied\r\n",
      "find: ‘/proc/704/task/738/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/704/task/738/ns’: Permission denied\r\n",
      "find: ‘/proc/704/task/750/fd’: Permission denied\r\n",
      "find: ‘/proc/704/task/750/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/704/task/750/ns’: Permission denied\r\n",
      "find: ‘/proc/704/task/1079/fd’: Permission denied\r\n",
      "find: ‘/proc/704/task/1079/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/704/task/1079/ns’: Permission denied\r\n",
      "find: ‘/proc/704/task/1081/fd’: Permission denied\r\n",
      "find: ‘/proc/704/task/1081/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/704/task/1081/ns’: Permission denied\r\n",
      "find: ‘/proc/704/fd’: Permission denied\r\n",
      "find: ‘/proc/704/map_files’: Permission denied\r\n",
      "find: ‘/proc/704/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/704/ns’: Permission denied\r\n",
      "find: ‘/proc/718/task/718/fd’: Permission denied\r\n",
      "find: ‘/proc/718/task/718/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/718/task/718/ns’: Permission denied\r\n",
      "find: ‘/proc/718/fd’: Permission denied\r\n",
      "find: ‘/proc/718/map_files’: Permission denied\r\n",
      "find: ‘/proc/718/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/718/ns’: Permission denied\r\n",
      "find: ‘/proc/720/task/720/fd’: Permission denied\r\n",
      "find: ‘/proc/720/task/720/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/720/task/720/ns’: Permission denied\r\n",
      "find: ‘/proc/720/fd’: Permission denied\r\n",
      "find: ‘/proc/720/map_files’: Permission denied\r\n",
      "find: ‘/proc/720/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/720/ns’: Permission denied\r\n",
      "find: ‘/proc/763/task/763/fd’: Permission denied\r\n",
      "find: ‘/proc/763/task/763/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/763/task/763/ns’: Permission denied\r\n",
      "find: ‘/proc/763/fd’: Permission denied\r\n",
      "find: ‘/proc/763/map_files’: Permission denied\r\n",
      "find: ‘/proc/763/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/763/ns’: Permission denied\r\n",
      "find: ‘/proc/914/task/914/fd’: Permission denied\r\n",
      "find: ‘/proc/914/task/914/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/914/task/914/ns’: Permission denied\r\n",
      "find: ‘/proc/914/fd’: Permission denied\r\n",
      "find: ‘/proc/914/map_files’: Permission denied\r\n",
      "find: ‘/proc/914/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/914/ns’: Permission denied\r\n",
      "find: ‘/proc/925/task/925/fd’: Permission denied\r\n",
      "find: ‘/proc/925/task/925/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/925/task/925/ns’: Permission denied\r\n",
      "find: ‘/proc/925/fd’: Permission denied\r\n",
      "find: ‘/proc/925/map_files’: Permission denied\r\n",
      "find: ‘/proc/925/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/925/ns’: Permission denied\r\n",
      "find: ‘/proc/940/task/940/fd’: Permission denied\r\n",
      "find: ‘/proc/940/task/940/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/940/task/940/ns’: Permission denied\r\n",
      "find: ‘/proc/940/task/1010/fd’: Permission denied\r\n",
      "find: ‘/proc/940/task/1010/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/940/task/1010/ns’: Permission denied\r\n",
      "find: ‘/proc/940/task/1011/fd’: Permission denied\r\n",
      "find: ‘/proc/940/task/1011/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/940/task/1011/ns’: Permission denied\r\n",
      "find: ‘/proc/940/task/1012/fd’: Permission denied\r\n",
      "find: ‘/proc/940/task/1012/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/940/task/1012/ns’: Permission denied\r\n",
      "find: ‘/proc/940/task/1013/fd’: Permission denied\r\n",
      "find: ‘/proc/940/task/1013/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/940/task/1013/ns’: Permission denied\r\n",
      "find: ‘/proc/940/task/1014/fd’: Permission denied\r\n",
      "find: ‘/proc/940/task/1014/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/940/task/1014/ns’: Permission denied\r\n",
      "find: ‘/proc/940/task/1048/fd’: Permission denied\r\n",
      "find: ‘/proc/940/task/1048/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/940/task/1048/ns’: Permission denied\r\n",
      "find: ‘/proc/940/task/1049/fd’: Permission denied\r\n",
      "find: ‘/proc/940/task/1049/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/940/task/1049/ns’: Permission denied\r\n",
      "find: ‘/proc/940/task/1057/fd’: Permission denied\r\n",
      "find: ‘/proc/940/task/1057/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/940/task/1057/ns’: Permission denied\r\n",
      "find: ‘/proc/940/task/1140/fd’: Permission denied\r\n",
      "find: ‘/proc/940/task/1140/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/940/task/1140/ns’: Permission denied\r\n",
      "find: ‘/proc/940/fd’: Permission denied\r\n",
      "find: ‘/proc/940/map_files’: Permission denied\r\n",
      "find: ‘/proc/940/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/940/ns’: Permission denied\r\n",
      "find: ‘/proc/1043/task/1043/fd’: Permission denied\r\n",
      "find: ‘/proc/1043/task/1043/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1043/task/1043/ns’: Permission denied\r\n",
      "find: ‘/proc/1043/task/1363/fd’: Permission denied\r\n",
      "find: ‘/proc/1043/task/1363/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1043/task/1363/ns’: Permission denied\r\n",
      "find: ‘/proc/1043/task/1365/fd’: Permission denied\r\n",
      "find: ‘/proc/1043/task/1365/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1043/task/1365/ns’: Permission denied\r\n",
      "find: ‘/proc/1043/task/1368/fd’: Permission denied\r\n",
      "find: ‘/proc/1043/task/1368/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1043/task/1368/ns’: Permission denied\r\n",
      "find: ‘/proc/1043/task/1369/fd’: Permission denied\r\n",
      "find: ‘/proc/1043/task/1369/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1043/task/1369/ns’: Permission denied\r\n",
      "find: ‘/proc/1043/fd’: Permission denied\r\n",
      "find: ‘/proc/1043/map_files’: Permission denied\r\n",
      "find: ‘/proc/1043/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1043/ns’: Permission denied\r\n",
      "find: ‘/proc/1171/task/1171/fd’: Permission denied\r\n",
      "find: ‘/proc/1171/task/1171/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1171/task/1171/ns’: Permission denied\r\n",
      "find: ‘/proc/1171/fd’: Permission denied\r\n",
      "find: ‘/proc/1171/map_files’: Permission denied\r\n",
      "find: ‘/proc/1171/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1171/ns’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1588/fd’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1588/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1588/ns’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1589/fd’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1589/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1589/ns’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1590/fd’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1590/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1590/ns’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1591/fd’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1591/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1591/ns’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1592/fd’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1592/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1592/ns’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1593/fd’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1593/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1593/ns’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1594/fd’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1594/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1594/ns’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1595/fd’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1595/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1595/ns’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1596/fd’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1596/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1596/ns’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1597/fd’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1597/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1588/task/1597/ns’: Permission denied\r\n",
      "find: ‘/proc/1588/fd’: Permission denied\r\n",
      "find: ‘/proc/1588/map_files’: Permission denied\r\n",
      "find: ‘/proc/1588/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/1588/ns’: Permission denied\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "find: ‘/proc/5688/task/5688/fd’: Permission denied\r\n",
      "find: ‘/proc/5688/task/5688/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/5688/task/5688/ns’: Permission denied\r\n",
      "find: ‘/proc/5688/fd’: Permission denied\r\n",
      "find: ‘/proc/5688/map_files’: Permission denied\r\n",
      "find: ‘/proc/5688/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/5688/ns’: Permission denied\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "find: ‘/proc/5889/task/5889/fd’: Permission denied\r\n",
      "find: ‘/proc/5889/task/5889/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/5889/task/5889/ns’: Permission denied\r\n",
      "find: ‘/proc/5889/fd’: Permission denied\r\n",
      "find: ‘/proc/5889/map_files’: Permission denied\r\n",
      "find: ‘/proc/5889/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/5889/ns’: Permission denied\r\n",
      "find: ‘/proc/5902/task/5902/fd’: Permission denied\r\n",
      "find: ‘/proc/5902/task/5902/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/5902/task/5902/ns’: Permission denied\r\n",
      "find: ‘/proc/5902/fd’: Permission denied\r\n",
      "find: ‘/proc/5902/map_files’: Permission denied\r\n",
      "find: ‘/proc/5902/fdinfo’: Permission denied\r\n",
      "find: ‘/proc/5902/ns’: Permission denied\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "find: ‘/root’: Permission denied\r\n",
      "find: ‘/sys/kernel/tracing’: Permission denied\r\n",
      "find: ‘/sys/kernel/debug’: Permission denied\r\n",
      "find: ‘/sys/fs/pstore’: Permission denied\r\n",
      "find: ‘/sys/fs/bpf’: Permission denied\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "find: ‘/tmp/systemd-private-c674134577ce4a44a623048bcc29eaf1-systemd-resolved.service-sS9Umj’: Permission denied\r\n",
      "find: ‘/tmp/systemd-private-c674134577ce4a44a623048bcc29eaf1-systemd-logind.service-9bm7Tk’: Permission denied\r\n",
      "find: ‘/tmp/snap-private-tmp’: Permission denied\r\n",
      "find: ‘/tmp/systemd-private-c674134577ce4a44a623048bcc29eaf1-haveged.service-BuooXZ’: Permission denied\r\n",
      "find: ‘/tmp/systemd-private-c674134577ce4a44a623048bcc29eaf1-chrony.service-EAhkUW’: Permission denied\r\n",
      "find: ‘/etc/sudoers.d’: Permission denied\r\n",
      "find: ‘/etc/multipath’: Permission denied\r\n",
      "find: ‘/etc/ssl/private’: Permission denied\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "find: ‘/etc/polkit-1/localauthority’: Permission denied\r\n",
      "find: ‘/run/docker’: Permission denied\r\n",
      "find: ‘/run/containerd’: Permission denied\r\n",
      "find: ‘/run/libpod’: Permission denied\r\n",
      "find: ‘/run/containers’: Permission denied\r\n",
      "find: ‘/run/chrony’: Permission denied\r\n",
      "find: ‘/run/user/1001/systemd/inaccessible/dir’: Permission denied\r\n",
      "find: ‘/run/sudo’: Permission denied\r\n",
      "find: ‘/run/podman’: Permission denied\r\n",
      "find: ‘/run/multipath’: Permission denied\r\n",
      "find: ‘/run/cryptsetup’: Permission denied\r\n",
      "find: ‘/run/lvm’: Permission denied\r\n",
      "find: ‘/run/credentials/systemd-sysusers.service’: Permission denied\r\n",
      "find: ‘/run/systemd/propagate’: Permission denied\r\n",
      "find: ‘/run/systemd/unit-root’: Permission denied\r\n",
      "find: ‘/run/systemd/inaccessible/dir’: Permission denied\r\n",
      "find: ‘/run/lock/lvm’: Permission denied\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/hostedtoolcache/Python/3.8.18/x64/bin/find_spark_home.py\r\n",
      "/opt/hostedtoolcache/Python/3.8.18/x64/bin/__pycache__/find_spark_home.cpython-38.pyc\r\n",
      "/opt/hostedtoolcache/Python/3.8.18/x64/bin/find-spark-home\r\n",
      "/opt/hostedtoolcache/Python/3.8.18/x64/bin/find-spark-home.cmd\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pyspark/bin/find-spark-home\r\n",
      "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pyspark/bin/find-spark-home.cmd\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pyspark/find_spark_home.py\r\n",
      "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pyspark/__pycache__/find_spark_home.cpython-38.pyc\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "find: ‘/home/runneradmin’: Permission denied\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "find: ‘/mnt/lost+found’: Permission denied\r\n",
      "find: ‘/snap/core20/2182/etc/ssl/private’: Permission denied\r\n",
      "find: ‘/snap/core20/2182/root’: Permission denied\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "find: ‘/snap/core20/2182/var/cache/ldconfig’: Permission denied\r\n",
      "find: ‘/snap/core20/2182/var/cache/private’: Permission denied\r\n",
      "find: ‘/snap/core20/2182/var/lib/private’: Permission denied\r\n",
      "find: ‘/snap/core20/2182/var/lib/snapd/void’: Permission denied\r\n",
      "find: ‘/var/tmp/systemd-private-c674134577ce4a44a623048bcc29eaf1-chrony.service-OTwVd7’: Permission denied\r\n",
      "find: ‘/var/tmp/systemd-private-c674134577ce4a44a623048bcc29eaf1-systemd-resolved.service-ULVVxK’: Permission denied\r\n",
      "find: ‘/var/tmp/systemd-private-c674134577ce4a44a623048bcc29eaf1-systemd-logind.service-Ykpzu7’: Permission denied\r\n",
      "find: ‘/var/tmp/systemd-private-c674134577ce4a44a623048bcc29eaf1-haveged.service-6Zs7YT’: Permission denied\r\n",
      "find: ‘/var/log/private’: Permission denied\r\n",
      "find: ‘/var/log/chrony’: Permission denied\r\n",
      "find: ‘/var/log/azure/Microsoft.Azure.Extensions.CustomScript/events’: Permission denied\r\n",
      "find: ‘/var/spool/rsyslog’: Permission denied\r\n",
      "find: ‘/var/spool/cron/crontabs’: Permission denied\r\n",
      "find: ‘/var/cache/private’: Permission denied\r\n",
      "find: ‘/var/cache/apt/archives/partial’: Permission denied\r\n",
      "find: ‘/var/cache/apparmor/0012d2a9.0’: Permission denied\r\n",
      "find: ‘/var/cache/apparmor/b1676ef8.0’: Permission denied\r\n",
      "find: ‘/var/cache/pollinate’: Permission denied\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "find: ‘/var/cache/ldconfig’: Permission denied\r\n",
      "find: ‘/var/snap/lxd/common/lxd’: Permission denied\r\n",
      "find: ‘/var/lib/update-notifier/package-data-downloads/partial’: Permission denied\r\n",
      "find: ‘/var/lib/private’: Permission denied\r\n",
      "find: ‘/var/lib/mysql-files’: Permission denied\r\n",
      "find: ‘/var/lib/apt/lists/partial’: Permission denied\r\n",
      "find: ‘/var/lib/ubuntu-advantage/apt-esm/var/cache/apt/archives/partial’: Permission denied\r\n",
      "find: ‘/var/lib/ubuntu-advantage/apt-esm/var/lib/apt/lists/partial’: Permission denied\r\n",
      "find: ‘/var/lib/containerd’: Permission denied\r\n",
      "find: ‘/var/lib/php/sessions’: Permission denied\r\n",
      "find: ‘/var/lib/docker’: Permission denied\r\n",
      "find: ‘/var/lib/chrony’: Permission denied\r\n",
      "find: ‘/var/lib/snapd/cookie’: Permission denied\r\n",
      "find: ‘/var/lib/snapd/void’: Permission denied\r\n",
      "find: ‘/var/lib/containers’: Permission denied\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "find: ‘/var/lib/waagent’: Permission denied\r\n",
      "find: ‘/var/lib/snmp/cert_indexes’: Permission denied\r\n",
      "find: ‘/var/lib/mysql’: Permission denied\r\n",
      "find: ‘/var/lib/nginx/proxy’: Permission denied\r\n",
      "find: ‘/var/lib/nginx/body’: Permission denied\r\n",
      "find: ‘/var/lib/nginx/fastcgi’: Permission denied\r\n",
      "find: ‘/var/lib/nginx/uwsgi’: Permission denied\r\n",
      "find: ‘/var/lib/nginx/scgi’: Permission denied\r\n",
      "find: ‘/var/lib/postgresql/14/main’: Permission denied\r\n",
      "find: ‘/var/lib/mysql-keyring’: Permission denied\r\n",
      "find: ‘/var/lib/polkit-1’: Permission denied\r\n"
     ]
    }
   ],
   "source": [
    "!find / -name \"*find*spark*\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Nr_2rb6pFzF8"
   },
   "source": [
    "The script `/usr/local/bin/find_spark_home.py` is successful at finding Spark's home."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qxlEqcY9LRI4"
   },
   "source": [
    "#### Set `SPARK_HOME`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:18:31.197346Z",
     "iopub.status.busy": "2024-03-24T21:18:31.197084Z",
     "iopub.status.idle": "2024-03-24T21:18:33.827037Z",
     "shell.execute_reply": "2024-03-24T21:18:33.826244Z"
    },
    "id": "GCgU9MnaIZ_k",
    "outputId": "58d228a7-3a12-43f6-a7c7-eaedd18efe61"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stored 'output' (SList)\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "import os\n",
    "IN_COLAB = 'google.colab' in sys.modules\n",
    "if IN_COLAB:\n",
    "  output = !python /usr/local/bin/find_spark_home.py\n",
    "else:\n",
    "  output = !find / -name \"pyspark\" -type d 2>/dev/null|head -1\n",
    "# Store the output using %store\n",
    "%store output\n",
    "# get rid of extra quotation marks\n",
    "os.environ['SPARK_HOME'] = output[0].replace('\"', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:18:33.830068Z",
     "iopub.status.busy": "2024-03-24T21:18:33.829531Z",
     "iopub.status.idle": "2024-03-24T21:18:34.957412Z",
     "shell.execute_reply": "2024-03-24T21:18:34.956638Z"
    },
    "id": "olIKq1bOkMDy",
    "outputId": "6140ea63-9eca-4456-9c3c-a1a8db16f283"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: pyspark\r\n",
      "Version: 3.5.1\r\n",
      "Summary: Apache Spark Python API\r\n",
      "Home-page: https://github.com/apache/spark/tree/master/python\r\n",
      "Author: Spark Developers\r\n",
      "Author-email: dev@spark.apache.org\r\n",
      "License: http://www.apache.org/licenses/LICENSE-2.0\r\n",
      "Location: /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages\r\n",
      "Requires: py4j\r\n",
      "Required-by: \r\n"
     ]
    }
   ],
   "source": [
    "!pip show pyspark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ic1uHIKAK1ss"
   },
   "source": [
    "Verify that the correct `SPARK_HOME` has been set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:18:34.960496Z",
     "iopub.status.busy": "2024-03-24T21:18:34.960270Z",
     "iopub.status.idle": "2024-03-24T21:18:34.967602Z",
     "shell.execute_reply": "2024-03-24T21:18:34.966906Z"
    },
    "id": "BraJPa5KZY1y",
    "outputId": "8a434517-6d46-4bfd-edb3-f6a21ae93141"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pyspark'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.environ['SPARK_HOME']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:18:34.969878Z",
     "iopub.status.busy": "2024-03-24T21:18:34.969684Z",
     "iopub.status.idle": "2024-03-24T21:18:34.973868Z",
     "shell.execute_reply": "2024-03-24T21:18:34.973217Z"
    },
    "id": "sONAYA_DKOry",
    "outputId": "65c0353e-a303-4574-e044-7fe05e405920"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pyspark'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%env SPARK_HOME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gQ3zuxd9LKxA"
   },
   "source": [
    "#### Set `PYTHONPATH`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:18:34.976326Z",
     "iopub.status.busy": "2024-03-24T21:18:34.975944Z",
     "iopub.status.idle": "2024-03-24T21:18:34.978930Z",
     "shell.execute_reply": "2024-03-24T21:18:34.978288Z"
    },
    "id": "SOXsRzqNLAqx"
   },
   "outputs": [],
   "source": [
    "os.environ['PYTHONPATH'] = os.environ['SPARK_HOME'] + '/python'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HTzSh-RBLbsg"
   },
   "source": [
    "Check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:18:34.981428Z",
     "iopub.status.busy": "2024-03-24T21:18:34.981062Z",
     "iopub.status.idle": "2024-03-24T21:18:34.985155Z",
     "shell.execute_reply": "2024-03-24T21:18:34.984526Z"
    },
    "id": "Log-_SFWJScq",
    "outputId": "636e1c0a-282c-402b-90bc-8eac7f15db93"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pyspark/python'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%env PYTHONPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qi3MxEWpV9CL"
   },
   "source": [
    "## Download data\n",
    "\n",
    "In order to run, the Sedona notebook expects to find some specific files in the local folder `data`. Let us populate `data` with the files from the Sedona Github repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:18:34.987681Z",
     "iopub.status.busy": "2024-03-24T21:18:34.987305Z",
     "iopub.status.idle": "2024-03-24T21:19:16.018364Z",
     "shell.execute_reply": "2024-03-24T21:19:16.017662Z"
    },
    "id": "wnzi7JCOIA1v"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Cloning into 'sedona'...\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# it would be more efficient to just download the \"data\" folder and not the whole repo\n",
    "[ -d sedona ] || git clone https://github.com/apache/sedona.git\n",
    "\n",
    "cp -r sedona/binder/data ./"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qQVz6ckaScn8"
   },
   "source": [
    "Verify the presence of data in the designated `data` folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:16.021569Z",
     "iopub.status.busy": "2024-03-24T21:19:16.021129Z",
     "iopub.status.idle": "2024-03-24T21:19:16.146201Z",
     "shell.execute_reply": "2024-03-24T21:19:16.145555Z"
    },
    "id": "Rxy-t1JnRnf6",
    "outputId": "0cfc7552-1a33-407e-9a9d-bfc7bcd901b8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arealm-small.csv\t\tne_50m_airports\r\n",
      "county_small.tsv\t\tpolygon\r\n",
      "county_small_wkb.tsv\t\tprimaryroads-linestring.csv\r\n",
      "gis_osm_pois_free_1.cpg\t\tprimaryroads-polygon.csv\r\n",
      "gis_osm_pois_free_1.dbf\t\traster\r\n",
      "gis_osm_pois_free_1.prj\t\ttestPolygon.json\r\n",
      "gis_osm_pois_free_1.shp\t\ttestpoint.csv\r\n",
      "gis_osm_pois_free_1.shx\t\tzcta510-small.csv\r\n",
      "ne_50m_admin_0_countries_lakes\r\n"
     ]
    }
   ],
   "source": [
    "!ls data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Mk20FV3ZHj_U"
   },
   "source": [
    "# Apache Sedona Core demo\n",
    "\n",
    "The notebook is available at the following link:\n",
    "https://github.com/apache/sedona/blob/master/binder/ApacheSedonaCore.ipynb\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AcjGVcGzOjcK"
   },
   "source": [
    "```\n",
    "Licensed to the Apache Software Foundation (ASF) under one\n",
    "or more contributor license agreements.  See the NOTICE file\n",
    "distributed with this work for additional information\n",
    "regarding copyright ownership.  The ASF licenses this file\n",
    "to you under the Apache License, Version 2.0 (the\n",
    "\"License\"); you may not use this file except in compliance\n",
    "with the License.  You may obtain a copy of the License at\n",
    "  http://www.apache.org/licenses/LICENSE-2.0\n",
    "Unless required by applicable law or agreed to in writing,\n",
    "software distributed under the License is distributed on an\n",
    "\"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
    "KIND, either express or implied.  See the License for the\n",
    "specific language governing permissions and limitations\n",
    "under the License.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:16.149047Z",
     "iopub.status.busy": "2024-03-24T21:19:16.148638Z",
     "iopub.status.idle": "2024-03-24T21:19:16.515219Z",
     "shell.execute_reply": "2024-03-24T21:19:16.514502Z"
    },
    "id": "sK9oIz0FOjcN",
    "outputId": "a4ce9ce6-a014-4279-8d23-a9555a13574a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Skipping SedonaKepler import, verify if keplergl is installed\n",
      "Skipping SedonaPyDeck import, verify if pydeck is installed\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "from pyspark import StorageLevel\n",
    "import geopandas as gpd\n",
    "import pandas as pd\n",
    "from pyspark.sql.types import StructType\n",
    "from pyspark.sql.types import StructField\n",
    "from pyspark.sql.types import StringType\n",
    "from pyspark.sql.types import LongType\n",
    "from shapely.geometry import Point\n",
    "from shapely.geometry import Polygon\n",
    "\n",
    "from sedona.spark import *\n",
    "from sedona.core.geom.envelope import Envelope\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:16.518224Z",
     "iopub.status.busy": "2024-03-24T21:19:16.517760Z",
     "iopub.status.idle": "2024-03-24T21:19:35.209565Z",
     "shell.execute_reply": "2024-03-24T21:19:35.208910Z"
    },
    "id": "HzVqFLBtOjcO"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "https://artifacts.unidata.ucar.edu/repository/unidata-all added as a remote repository with the name: repo-1\n",
      "Ivy Default Cache set to: /home/runner/.ivy2/cache\n",
      "The jars for the packages stored in: /home/runner/.ivy2/jars\n",
      "org.apache.sedona#sedona-spark-3.4_2.12 added as a dependency\n",
      "org.datasyslab#geotools-wrapper added as a dependency\n",
      "uk.co.gresearch.spark#spark-extension_2.12 added as a dependency\n",
      ":: resolving dependencies :: org.apache.spark#spark-submit-parent-70afcabe-cd03-4a4a-80ca-78a1971f30ea;1.0\n",
      "\tconfs: [default]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ":: loading settings :: url = jar:file:/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pyspark/jars/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tfound org.apache.sedona#sedona-spark-3.4_2.12;1.5.1 in central\n",
      "\tfound org.apache.sedona#sedona-common;1.5.1 in central\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tfound org.apache.commons#commons-math3;3.6.1 in central\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tfound org.locationtech.jts#jts-core;1.19.0 in central\n",
      "\tfound org.wololo#jts2geojson;0.16.1 in central\n",
      "\tfound org.locationtech.spatial4j#spatial4j;0.8 in central\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tfound com.google.geometry#s2-geometry;2.0.0 in central\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tfound com.google.guava#guava;25.1-jre in central\n",
      "\tfound com.google.code.findbugs#jsr305;3.0.2 in central\n",
      "\tfound org.checkerframework#checker-qual;2.0.0 in central\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tfound com.google.errorprone#error_prone_annotations;2.1.3 in central\n",
      "\tfound com.google.j2objc#j2objc-annotations;1.1 in central\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tfound org.codehaus.mojo#animal-sniffer-annotations;1.14 in central\n",
      "\tfound com.uber#h3;4.1.1 in central\n",
      "\tfound net.sf.geographiclib#GeographicLib-Java;1.52 in central\n",
      "\tfound com.github.ben-manes.caffeine#caffeine;2.9.2 in central\n",
      "\tfound org.checkerframework#checker-qual;3.10.0 in central\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tfound com.google.errorprone#error_prone_annotations;2.5.1 in central\n",
      "\tfound org.apache.sedona#sedona-spark-common-3.4_2.12;1.5.1 in central\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\tfound commons-lang#commons-lang;2.6 in central\n",
      "\tfound org.scala-lang.modules#scala-collection-compat_2.12;2.5.0 in central\n",
      "\tfound org.beryx#awt-color-factory;1.0.0 in central\n",
      "\tfound org.datasyslab#geotools-wrapper;1.5.1-28.2 in central\n",
      "\tfound uk.co.gresearch.spark#spark-extension_2.12;2.11.0-3.4 in central\n",
      "\tfound com.github.scopt#scopt_2.12;4.1.0 in central\n",
      "downloading https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-3.4_2.12/1.5.1/sedona-spark-3.4_2.12-1.5.1.jar ...\n",
      "\t[SUCCESSFUL ] org.apache.sedona#sedona-spark-3.4_2.12;1.5.1!sedona-spark-3.4_2.12.jar (29ms)\n",
      "downloading https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/1.5.1-28.2/geotools-wrapper-1.5.1-28.2.jar ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t[SUCCESSFUL ] org.datasyslab#geotools-wrapper;1.5.1-28.2!geotools-wrapper.jar (228ms)\n",
      "downloading https://repo1.maven.org/maven2/uk/co/gresearch/spark/spark-extension_2.12/2.11.0-3.4/spark-extension_2.12-2.11.0-3.4.jar ...\n",
      "\t[SUCCESSFUL ] uk.co.gresearch.spark#spark-extension_2.12;2.11.0-3.4!spark-extension_2.12.jar (22ms)\n",
      "downloading https://repo1.maven.org/maven2/org/apache/sedona/sedona-common/1.5.1/sedona-common-1.5.1.jar ...\n",
      "\t[SUCCESSFUL ] org.apache.sedona#sedona-common;1.5.1!sedona-common.jar (22ms)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "downloading https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-common-3.4_2.12/1.5.1/sedona-spark-common-3.4_2.12-1.5.1.jar ...\n",
      "\t[SUCCESSFUL ] org.apache.sedona#sedona-spark-common-3.4_2.12;1.5.1!sedona-spark-common-3.4_2.12.jar (1157ms)\n",
      "downloading https://repo1.maven.org/maven2/org/locationtech/jts/jts-core/1.19.0/jts-core-1.19.0.jar ...\n",
      "\t[SUCCESSFUL ] org.locationtech.jts#jts-core;1.19.0!jts-core.jar(bundle) (28ms)\n",
      "downloading https://repo1.maven.org/maven2/org/wololo/jts2geojson/0.16.1/jts2geojson-0.16.1.jar ...\n",
      "\t[SUCCESSFUL ] org.wololo#jts2geojson;0.16.1!jts2geojson.jar (19ms)\n",
      "downloading https://repo1.maven.org/maven2/org/scala-lang/modules/scala-collection-compat_2.12/2.5.0/scala-collection-compat_2.12-2.5.0.jar ...\n",
      "\t[SUCCESSFUL ] org.scala-lang.modules#scala-collection-compat_2.12;2.5.0!scala-collection-compat_2.12.jar (21ms)\n",
      "downloading https://repo1.maven.org/maven2/org/apache/commons/commons-math3/3.6.1/commons-math3-3.6.1.jar ...\n",
      "\t[SUCCESSFUL ] org.apache.commons#commons-math3;3.6.1!commons-math3.jar (40ms)\n",
      "downloading https://repo1.maven.org/maven2/org/locationtech/spatial4j/spatial4j/0.8/spatial4j-0.8.jar ...\n",
      "\t[SUCCESSFUL ] org.locationtech.spatial4j#spatial4j;0.8!spatial4j.jar(bundle) (20ms)\n",
      "downloading https://repo1.maven.org/maven2/com/google/geometry/s2-geometry/2.0.0/s2-geometry-2.0.0.jar ...\n",
      "\t[SUCCESSFUL ] com.google.geometry#s2-geometry;2.0.0!s2-geometry.jar (23ms)\n",
      "downloading https://repo1.maven.org/maven2/com/uber/h3/4.1.1/h3-4.1.1.jar ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\t[SUCCESSFUL ] com.uber#h3;4.1.1!h3.jar (33ms)\n",
      "downloading https://repo1.maven.org/maven2/net/sf/geographiclib/GeographicLib-Java/1.52/GeographicLib-Java-1.52.jar ...\n",
      "\t[SUCCESSFUL ] net.sf.geographiclib#GeographicLib-Java;1.52!GeographicLib-Java.jar (20ms)\n",
      "downloading https://repo1.maven.org/maven2/com/github/ben-manes/caffeine/caffeine/2.9.2/caffeine-2.9.2.jar ...\n",
      "\t[SUCCESSFUL ] com.github.ben-manes.caffeine#caffeine;2.9.2!caffeine.jar (30ms)\n",
      "downloading https://repo1.maven.org/maven2/com/google/guava/guava/25.1-jre/guava-25.1-jre.jar ...\n",
      "\t[SUCCESSFUL ] com.google.guava#guava;25.1-jre!guava.jar(bundle) (44ms)\n",
      "downloading https://repo1.maven.org/maven2/com/google/code/findbugs/jsr305/3.0.2/jsr305-3.0.2.jar ...\n",
      "\t[SUCCESSFUL ] com.google.code.findbugs#jsr305;3.0.2!jsr305.jar (19ms)\n",
      "downloading https://repo1.maven.org/maven2/com/google/j2objc/j2objc-annotations/1.1/j2objc-annotations-1.1.jar ...\n",
      "\t[SUCCESSFUL ] com.google.j2objc#j2objc-annotations;1.1!j2objc-annotations.jar (18ms)\n",
      "downloading https://repo1.maven.org/maven2/org/codehaus/mojo/animal-sniffer-annotations/1.14/animal-sniffer-annotations-1.14.jar ...\n",
      "\t[SUCCESSFUL ] org.codehaus.mojo#animal-sniffer-annotations;1.14!animal-sniffer-annotations.jar (18ms)\n",
      "downloading https://repo1.maven.org/maven2/org/checkerframework/checker-qual/3.10.0/checker-qual-3.10.0.jar ...\n",
      "\t[SUCCESSFUL ] org.checkerframework#checker-qual;3.10.0!checker-qual.jar (19ms)\n",
      "downloading https://repo1.maven.org/maven2/com/google/errorprone/error_prone_annotations/2.5.1/error_prone_annotations-2.5.1.jar ...\n",
      "\t[SUCCESSFUL ] com.google.errorprone#error_prone_annotations;2.5.1!error_prone_annotations.jar (18ms)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "downloading https://repo1.maven.org/maven2/commons-lang/commons-lang/2.6/commons-lang-2.6.jar ...\n",
      "\t[SUCCESSFUL ] commons-lang#commons-lang;2.6!commons-lang.jar (21ms)\n",
      "downloading https://repo1.maven.org/maven2/org/beryx/awt-color-factory/1.0.0/awt-color-factory-1.0.0.jar ...\n",
      "\t[SUCCESSFUL ] org.beryx#awt-color-factory;1.0.0!awt-color-factory.jar (19ms)\n",
      "downloading https://repo1.maven.org/maven2/com/github/scopt/scopt_2.12/4.1.0/scopt_2.12-4.1.0.jar ...\n",
      "\t[SUCCESSFUL ] com.github.scopt#scopt_2.12;4.1.0!scopt_2.12.jar (20ms)\n",
      ":: resolution report :: resolve 8250ms :: artifacts dl 1899ms\n",
      "\t:: modules in use:\n",
      "\tcom.github.ben-manes.caffeine#caffeine;2.9.2 from central in [default]\n",
      "\tcom.github.scopt#scopt_2.12;4.1.0 from central in [default]\n",
      "\tcom.google.code.findbugs#jsr305;3.0.2 from central in [default]\n",
      "\tcom.google.errorprone#error_prone_annotations;2.5.1 from central in [default]\n",
      "\tcom.google.geometry#s2-geometry;2.0.0 from central in [default]\n",
      "\tcom.google.guava#guava;25.1-jre from central in [default]\n",
      "\tcom.google.j2objc#j2objc-annotations;1.1 from central in [default]\n",
      "\tcom.uber#h3;4.1.1 from central in [default]\n",
      "\tcommons-lang#commons-lang;2.6 from central in [default]\n",
      "\tnet.sf.geographiclib#GeographicLib-Java;1.52 from central in [default]\n",
      "\torg.apache.commons#commons-math3;3.6.1 from central in [default]\n",
      "\torg.apache.sedona#sedona-common;1.5.1 from central in [default]\n",
      "\torg.apache.sedona#sedona-spark-3.4_2.12;1.5.1 from central in [default]\n",
      "\torg.apache.sedona#sedona-spark-common-3.4_2.12;1.5.1 from central in [default]\n",
      "\torg.beryx#awt-color-factory;1.0.0 from central in [default]\n",
      "\torg.checkerframework#checker-qual;3.10.0 from central in [default]\n",
      "\torg.codehaus.mojo#animal-sniffer-annotations;1.14 from central in [default]\n",
      "\torg.datasyslab#geotools-wrapper;1.5.1-28.2 from central in [default]\n",
      "\torg.locationtech.jts#jts-core;1.19.0 from central in [default]\n",
      "\torg.locationtech.spatial4j#spatial4j;0.8 from central in [default]\n",
      "\torg.scala-lang.modules#scala-collection-compat_2.12;2.5.0 from central in [default]\n",
      "\torg.wololo#jts2geojson;0.16.1 from central in [default]\n",
      "\tuk.co.gresearch.spark#spark-extension_2.12;2.11.0-3.4 from central in [default]\n",
      "\t:: evicted modules:\n",
      "\torg.checkerframework#checker-qual;2.0.0 by [org.checkerframework#checker-qual;3.10.0] in [default]\n",
      "\tcom.google.errorprone#error_prone_annotations;2.1.3 by [com.google.errorprone#error_prone_annotations;2.5.1] in [default]\n",
      "\t---------------------------------------------------------------------\n",
      "\t|                  |            modules            ||   artifacts   |\n",
      "\t|       conf       | number| search|dwnlded|evicted|| number|dwnlded|\n",
      "\t---------------------------------------------------------------------\n",
      "\t|      default     |   25  |   25  |   25  |   2   ||   23  |   23  |\n",
      "\t---------------------------------------------------------------------\n",
      ":: retrieving :: org.apache.spark#spark-submit-parent-70afcabe-cd03-4a4a-80ca-78a1971f30ea\n",
      "\tconfs: [default]\n",
      "\t23 artifacts copied, 0 already retrieved (42471kB/73ms)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/03/24 21:19:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n"
     ]
    }
   ],
   "source": [
    "config = SedonaContext.builder() .\\\n",
    "    config('spark.jars.packages',\n",
    "           'org.apache.sedona:sedona-spark-3.4_2.12:1.5.1,'\n",
    "           'org.datasyslab:geotools-wrapper:1.5.1-28.2,'\n",
    "           'uk.co.gresearch.spark:spark-extension_2.12:2.11.0-3.4'). \\\n",
    "    config('spark.jars.repositories', 'https://artifacts.unidata.ucar.edu/repository/unidata-all'). \\\n",
    "    getOrCreate()\n",
    "\n",
    "sedona = SedonaContext.create(config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:35.214118Z",
     "iopub.status.busy": "2024-03-24T21:19:35.213404Z",
     "iopub.status.idle": "2024-03-24T21:19:35.218201Z",
     "shell.execute_reply": "2024-03-24T21:19:35.217457Z"
    },
    "id": "JI32GOnEOjcO"
   },
   "outputs": [],
   "source": [
    "sc = sedona.sparkContext"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OdQM3oaYOjcO"
   },
   "source": [
    "# Create SpatialRDD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X2umXxYfOjcO"
   },
   "source": [
    "## Reading to PointRDD from CSV file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j8Rgv--0OjcP"
   },
   "source": [
    "Suppose we want load the CSV file into Apache Sedona PointRDD\n",
    "```\n",
    "testattribute0,-88.331492,32.324142,testattribute1,testattribute2\n",
    "testattribute0,-88.175933,32.360763,testattribute1,testattribute2\n",
    "testattribute0,-88.388954,32.357073,testattribute1,testattribute2\n",
    "testattribute0,-88.221102,32.35078,testattribute1,testattribute2\n",
    "testattribute0,-88.323995,32.950671,testattribute1,testattribute2\n",
    "testattribute0,-88.231077,32.700812,testattribute1,testattribute2\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:35.222307Z",
     "iopub.status.busy": "2024-03-24T21:19:35.221617Z",
     "iopub.status.idle": "2024-03-24T21:19:35.568419Z",
     "shell.execute_reply": "2024-03-24T21:19:35.567552Z"
    },
    "id": "gusbYu1fOjcP"
   },
   "outputs": [],
   "source": [
    "point_rdd = PointRDD(sc, \"data/arealm-small.csv\", 1, FileDataSplitter.CSV, True, 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:35.573179Z",
     "iopub.status.busy": "2024-03-24T21:19:35.572858Z",
     "iopub.status.idle": "2024-03-24T21:19:35.579631Z",
     "shell.execute_reply": "2024-03-24T21:19:35.578886Z"
    },
    "id": "Ykc5FwouOjcP",
    "outputId": "4a981655-2238-4bb4-fd67-7c56584585a0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3000"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Getting approximate total count\n",
    "point_rdd.approximateTotalCount"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 121
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:35.583152Z",
     "iopub.status.busy": "2024-03-24T21:19:35.582665Z",
     "iopub.status.idle": "2024-03-24T21:19:35.678894Z",
     "shell.execute_reply": "2024-03-24T21:19:35.677945Z"
    },
    "id": "PdEjrrRHOjcP",
    "outputId": "fc0d5620-a4eb-4e37-e819-3e04e16ddcd4"
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"100.0\" height=\"100.0\" viewBox=\"-176.64696132 26.718666680000002 95.20719264000002 48.162659640000015\" preserveAspectRatio=\"xMinYMin meet\"><g transform=\"matrix(1,0,0,-1,0,101.59999300000001)\"><path fill-rule=\"evenodd\" fill=\"#66cc99\" stroke=\"#555555\" stroke-width=\"1.9041438528000003\" opacity=\"0.6\" d=\"M -173.120769,30.244859 L -173.120769,71.355134 L -84.965961,71.355134 L -84.965961,30.244859 L -173.120769,30.244859 z\" /></g></svg>"
      ],
      "text/plain": [
       "Envelope(-173.120769, -84.965961, 30.244859, 71.355134)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# getting boundary for PointRDD or any other SpatialRDD, it returns Enelope object which inherits from\n",
    "# shapely.geometry.Polygon\n",
    "point_rdd.boundary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:35.683037Z",
     "iopub.status.busy": "2024-03-24T21:19:35.682286Z",
     "iopub.status.idle": "2024-03-24T21:19:35.788320Z",
     "shell.execute_reply": "2024-03-24T21:19:35.787579Z"
    },
    "id": "MDV2-f2tOjcP",
    "outputId": "626448bb-0e07-4463-9830-7ec607546fcd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To run analyze please use function analyze\n",
    "point_rdd.analyze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 121
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:35.791901Z",
     "iopub.status.busy": "2024-03-24T21:19:35.791614Z",
     "iopub.status.idle": "2024-03-24T21:19:35.800695Z",
     "shell.execute_reply": "2024-03-24T21:19:35.800002Z"
    },
    "id": "puMA8o6POjcP",
    "outputId": "3b701a97-f5ca-4316-e345-35416125e9f8"
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"100.0\" height=\"100.0\" viewBox=\"-176.64696132 26.718666680000002 95.20719264000002 48.162659640000015\" preserveAspectRatio=\"xMinYMin meet\"><g transform=\"matrix(1,0,0,-1,0,101.59999300000001)\"><path fill-rule=\"evenodd\" fill=\"#66cc99\" stroke=\"#555555\" stroke-width=\"1.9041438528000003\" opacity=\"0.6\" d=\"M -173.120769,30.244859 L -173.120769,71.355134 L -84.965961,71.355134 L -84.965961,30.244859 L -173.120769,30.244859 z\" /></g></svg>"
      ],
      "text/plain": [
       "Envelope(-173.120769, -84.965961, 30.244859, 71.355134)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Finding boundary envelope for PointRDD or any other SpatialRDD, it returns Enelope object which inherits from\n",
    "# shapely.geometry.Polygon\n",
    "point_rdd.boundaryEnvelope"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:35.804429Z",
     "iopub.status.busy": "2024-03-24T21:19:35.803941Z",
     "iopub.status.idle": "2024-03-24T21:19:35.943331Z",
     "shell.execute_reply": "2024-03-24T21:19:35.942428Z"
    },
    "id": "hgOCi8lJOjcQ",
    "outputId": "fbf49d82-845e-4ddb-a178-c6d5817de5d6"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2996"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Calculate number of records without duplicates\n",
    "point_rdd.countWithoutDuplicates()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:35.947181Z",
     "iopub.status.busy": "2024-03-24T21:19:35.946879Z",
     "iopub.status.idle": "2024-03-24T21:19:35.952249Z",
     "shell.execute_reply": "2024-03-24T21:19:35.951539Z"
    },
    "id": "-l1BvhakOjcQ",
    "outputId": "f67bcf6c-e663-4fa3-ee71-bccd904b6806"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "''"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Getting source epsg code\n",
    "point_rdd.getSourceEpsgCode()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:35.955703Z",
     "iopub.status.busy": "2024-03-24T21:19:35.955209Z",
     "iopub.status.idle": "2024-03-24T21:19:35.960275Z",
     "shell.execute_reply": "2024-03-24T21:19:35.959501Z"
    },
    "id": "tiFEDeNdOjcQ",
    "outputId": "cc91916a-dcbf-442f-ff95-662975ddc04d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "''"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Getting target epsg code\n",
    "point_rdd.getTargetEpsgCode()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:35.963784Z",
     "iopub.status.busy": "2024-03-24T21:19:35.963318Z",
     "iopub.status.idle": "2024-03-24T21:19:36.115810Z",
     "shell.execute_reply": "2024-03-24T21:19:36.114660Z"
    },
    "id": "oip1bNogOjcQ",
    "outputId": "dfb01317-0542-4763-ee44-96b6e5eff05a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Spatial partitioning data\n",
    "point_rdd.spatialPartitioning(GridType.KDBTREE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "setaW5adOjcQ"
   },
   "source": [
    "## Operations on RawSpatialRDD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XeAQe3qqOjcQ"
   },
   "source": [
    "rawSpatialRDD method returns RDD which consists of GeoData objects which has 2 attributes\n",
    "<li> geom: shapely.geometry.BaseGeometry </li>\n",
    "<li> userData: str </li>\n",
    "\n",
    "You can use any operations on those objects and spread across machines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:36.119828Z",
     "iopub.status.busy": "2024-03-24T21:19:36.119535Z",
     "iopub.status.idle": "2024-03-24T21:19:36.985087Z",
     "shell.execute_reply": "2024-03-24T21:19:36.984177Z"
    },
    "id": "pNlkO6kYOjcR",
    "outputId": "d2a40cc1-dca5-413a-e4f0-d22c0c2be1a1"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "[Stage 8:>                                                          (0 + 1) / 1]\r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "                                                                                \r"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[Geometry: Point userData: testattribute0\ttestattribute1\ttestattribute2]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# take firs element\n",
    "point_rdd.rawSpatialRDD.take(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:36.988795Z",
     "iopub.status.busy": "2024-03-24T21:19:36.988327Z",
     "iopub.status.idle": "2024-03-24T21:19:37.270575Z",
     "shell.execute_reply": "2024-03-24T21:19:37.269899Z"
    },
    "id": "22UNTZiIOjcR",
    "outputId": "ec6cad29-26a3-44c6-8d0e-477397864945"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Geometry: Point userData: testattribute0\ttestattribute1\ttestattribute2,\n",
       " Geometry: Point userData: testattribute0\ttestattribute1\ttestattribute2,\n",
       " Geometry: Point userData: testattribute0\ttestattribute1\ttestattribute2,\n",
       " Geometry: Point userData: testattribute0\ttestattribute1\ttestattribute2,\n",
       " Geometry: Point userData: testattribute0\ttestattribute1\ttestattribute2]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# collect to Python list\n",
    "point_rdd.rawSpatialRDD.collect()[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:37.273720Z",
     "iopub.status.busy": "2024-03-24T21:19:37.273296Z",
     "iopub.status.idle": "2024-03-24T21:19:37.568885Z",
     "shell.execute_reply": "2024-03-24T21:19:37.568122Z"
    },
    "id": "_vz8jzj6OjcR",
    "outputId": "305812a3-9435-412f-d331-0d1f09544b41"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[111.08786851399313,\n",
       " 110.92828303170774,\n",
       " 111.1385974283527,\n",
       " 110.97450594034112,\n",
       " 110.97122518072091]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# apply map functions, for example distance to Point(52 21)\n",
    "point_rdd.rawSpatialRDD.map(lambda x: x.geom.distance(Point(21, 52))).take(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VJUwfnWbOjcR"
   },
   "source": [
    "## Transforming to GeoPandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HJT4cqunOjcR"
   },
   "source": [
    "## Loaded data can be transformed to GeoPandas DataFrame in a few ways"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ENst5TKKOjcR"
   },
   "source": [
    "### Directly from RDD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:37.573147Z",
     "iopub.status.busy": "2024-03-24T21:19:37.572494Z",
     "iopub.status.idle": "2024-03-24T21:19:37.579663Z",
     "shell.execute_reply": "2024-03-24T21:19:37.578908Z"
    },
    "id": "khvSdVtROjcR"
   },
   "outputs": [],
   "source": [
    "point_rdd_to_geo = point_rdd.rawSpatialRDD.map(lambda x: [x.geom, *x.getUserData().split(\"\\t\")])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:37.583248Z",
     "iopub.status.busy": "2024-03-24T21:19:37.582561Z",
     "iopub.status.idle": "2024-03-24T21:19:38.367448Z",
     "shell.execute_reply": "2024-03-24T21:19:38.366750Z"
    },
    "id": "h_aQLibwOjcR"
   },
   "outputs": [],
   "source": [
    "point_gdf = gpd.GeoDataFrame(\n",
    "    point_rdd_to_geo.collect(), columns=[\"geom\", \"attr1\", \"attr2\", \"attr3\"], geometry=\"geom\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:38.370266Z",
     "iopub.status.busy": "2024-03-24T21:19:38.370057Z",
     "iopub.status.idle": "2024-03-24T21:19:38.379692Z",
     "shell.execute_reply": "2024-03-24T21:19:38.379040Z"
    },
    "id": "Ht7qYNuwOjcS",
    "outputId": "a5f39f31-a0f7-414d-d3fe-17d7625cf0e4"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>geom</th>\n",
       "      <th>attr1</th>\n",
       "      <th>attr2</th>\n",
       "      <th>attr3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>POINT (-88.33149 32.32414)</td>\n",
       "      <td>testattribute0</td>\n",
       "      <td>testattribute1</td>\n",
       "      <td>testattribute2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>POINT (-88.17593 32.36076)</td>\n",
       "      <td>testattribute0</td>\n",
       "      <td>testattribute1</td>\n",
       "      <td>testattribute2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>POINT (-88.38895 32.35707)</td>\n",
       "      <td>testattribute0</td>\n",
       "      <td>testattribute1</td>\n",
       "      <td>testattribute2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>POINT (-88.22110 32.35078)</td>\n",
       "      <td>testattribute0</td>\n",
       "      <td>testattribute1</td>\n",
       "      <td>testattribute2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>POINT (-88.32399 32.95067)</td>\n",
       "      <td>testattribute0</td>\n",
       "      <td>testattribute1</td>\n",
       "      <td>testattribute2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         geom           attr1           attr2           attr3\n",
       "0  POINT (-88.33149 32.32414)  testattribute0  testattribute1  testattribute2\n",
       "1  POINT (-88.17593 32.36076)  testattribute0  testattribute1  testattribute2\n",
       "2  POINT (-88.38895 32.35707)  testattribute0  testattribute1  testattribute2\n",
       "3  POINT (-88.22110 32.35078)  testattribute0  testattribute1  testattribute2\n",
       "4  POINT (-88.32399 32.95067)  testattribute0  testattribute1  testattribute2"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "point_gdf[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2vW97p_iOjcS"
   },
   "source": [
    "### Using Adapter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:38.382412Z",
     "iopub.status.busy": "2024-03-24T21:19:38.381927Z",
     "iopub.status.idle": "2024-03-24T21:19:38.384921Z",
     "shell.execute_reply": "2024-03-24T21:19:38.384411Z"
    },
    "id": "HN0IIhZ8OjcS",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Adapter allows you to convert geospatial data types introduced with sedona to other ones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:38.387234Z",
     "iopub.status.busy": "2024-03-24T21:19:38.387037Z",
     "iopub.status.idle": "2024-03-24T21:19:38.482430Z",
     "shell.execute_reply": "2024-03-24T21:19:38.481604Z"
    },
    "id": "69dR65hgOjcS"
   },
   "outputs": [],
   "source": [
    "spatial_df = Adapter.\\\n",
    "    toDf(point_rdd, [\"attr1\", \"attr2\", \"attr3\"], sedona).\\\n",
    "    createOrReplaceTempView(\"spatial_df\")\n",
    "\n",
    "spatial_gdf = sedona.sql(\"Select attr1, attr2, attr3, geometry as geom from spatial_df\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:38.486647Z",
     "iopub.status.busy": "2024-03-24T21:19:38.485930Z",
     "iopub.status.idle": "2024-03-24T21:19:38.791942Z",
     "shell.execute_reply": "2024-03-24T21:19:38.791214Z"
    },
    "id": "TnGQMPUIOjcS",
    "outputId": "d3923b4f-12a4-43df-f874-8cba67e04bef"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------+--------------+--------------+----------------------------+\n",
      "|attr1         |attr2         |attr3         |geom                        |\n",
      "+--------------+--------------+--------------+----------------------------+\n",
      "|testattribute0|testattribute1|testattribute2|POINT (-88.331492 32.324142)|\n",
      "|testattribute0|testattribute1|testattribute2|POINT (-88.175933 32.360763)|\n",
      "|testattribute0|testattribute1|testattribute2|POINT (-88.388954 32.357073)|\n",
      "|testattribute0|testattribute1|testattribute2|POINT (-88.221102 32.35078) |\n",
      "|testattribute0|testattribute1|testattribute2|POINT (-88.323995 32.950671)|\n",
      "+--------------+--------------+--------------+----------------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "spatial_gdf.show(5, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:38.795646Z",
     "iopub.status.busy": "2024-03-24T21:19:38.795336Z",
     "iopub.status.idle": "2024-03-24T21:19:39.363878Z",
     "shell.execute_reply": "2024-03-24T21:19:39.363183Z"
    },
    "id": "IdL0awafOjcS",
    "outputId": "cafcd06d-0376-4b32-ff79-dd96936865e9"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>attr1</th>\n",
       "      <th>attr2</th>\n",
       "      <th>attr3</th>\n",
       "      <th>geom</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>testattribute0</td>\n",
       "      <td>testattribute1</td>\n",
       "      <td>testattribute2</td>\n",
       "      <td>POINT (-88.33149 32.32414)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>testattribute0</td>\n",
       "      <td>testattribute1</td>\n",
       "      <td>testattribute2</td>\n",
       "      <td>POINT (-88.17593 32.36076)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>testattribute0</td>\n",
       "      <td>testattribute1</td>\n",
       "      <td>testattribute2</td>\n",
       "      <td>POINT (-88.38895 32.35707)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>testattribute0</td>\n",
       "      <td>testattribute1</td>\n",
       "      <td>testattribute2</td>\n",
       "      <td>POINT (-88.22110 32.35078)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>testattribute0</td>\n",
       "      <td>testattribute1</td>\n",
       "      <td>testattribute2</td>\n",
       "      <td>POINT (-88.32399 32.95067)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            attr1           attr2           attr3                        geom\n",
       "0  testattribute0  testattribute1  testattribute2  POINT (-88.33149 32.32414)\n",
       "1  testattribute0  testattribute1  testattribute2  POINT (-88.17593 32.36076)\n",
       "2  testattribute0  testattribute1  testattribute2  POINT (-88.38895 32.35707)\n",
       "3  testattribute0  testattribute1  testattribute2  POINT (-88.22110 32.35078)\n",
       "4  testattribute0  testattribute1  testattribute2  POINT (-88.32399 32.95067)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gpd.GeoDataFrame(spatial_gdf.toPandas(), geometry=\"geom\")[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ilmR8RjIOjcS"
   },
   "source": [
    "### With DataFrame creation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:39.366810Z",
     "iopub.status.busy": "2024-03-24T21:19:39.366430Z",
     "iopub.status.idle": "2024-03-24T21:19:39.370087Z",
     "shell.execute_reply": "2024-03-24T21:19:39.369394Z"
    },
    "id": "LqgFLBZEOjcT"
   },
   "outputs": [],
   "source": [
    "schema = StructType(\n",
    "    [\n",
    "        StructField(\"geometry\", GeometryType(), False),\n",
    "        StructField(\"attr1\", StringType(), False),\n",
    "        StructField(\"attr2\", StringType(), False),\n",
    "        StructField(\"attr3\", StringType(), False),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:39.372703Z",
     "iopub.status.busy": "2024-03-24T21:19:39.372299Z",
     "iopub.status.idle": "2024-03-24T21:19:39.404925Z",
     "shell.execute_reply": "2024-03-24T21:19:39.404134Z"
    },
    "id": "tEO42a4DOjcT"
   },
   "outputs": [],
   "source": [
    "geo_df = sedona.createDataFrame(point_rdd_to_geo, schema, verifySchema=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:39.410063Z",
     "iopub.status.busy": "2024-03-24T21:19:39.408695Z",
     "iopub.status.idle": "2024-03-24T21:19:39.932265Z",
     "shell.execute_reply": "2024-03-24T21:19:39.931581Z"
    },
    "id": "PaIAqir1OjcT",
    "outputId": "89b2c8b0-f645-4ae0-c1be-660c5d7e60ea"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>geometry</th>\n",
       "      <th>attr1</th>\n",
       "      <th>attr2</th>\n",
       "      <th>attr3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>POINT (-88.33149 32.32414)</td>\n",
       "      <td>testattribute0</td>\n",
       "      <td>testattribute1</td>\n",
       "      <td>testattribute2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>POINT (-88.17593 32.36076)</td>\n",
       "      <td>testattribute0</td>\n",
       "      <td>testattribute1</td>\n",
       "      <td>testattribute2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>POINT (-88.38895 32.35707)</td>\n",
       "      <td>testattribute0</td>\n",
       "      <td>testattribute1</td>\n",
       "      <td>testattribute2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>POINT (-88.22110 32.35078)</td>\n",
       "      <td>testattribute0</td>\n",
       "      <td>testattribute1</td>\n",
       "      <td>testattribute2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>POINT (-88.32399 32.95067)</td>\n",
       "      <td>testattribute0</td>\n",
       "      <td>testattribute1</td>\n",
       "      <td>testattribute2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     geometry           attr1           attr2           attr3\n",
       "0  POINT (-88.33149 32.32414)  testattribute0  testattribute1  testattribute2\n",
       "1  POINT (-88.17593 32.36076)  testattribute0  testattribute1  testattribute2\n",
       "2  POINT (-88.38895 32.35707)  testattribute0  testattribute1  testattribute2\n",
       "3  POINT (-88.22110 32.35078)  testattribute0  testattribute1  testattribute2\n",
       "4  POINT (-88.32399 32.95067)  testattribute0  testattribute1  testattribute2"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gpd.GeoDataFrame(geo_df.toPandas(), geometry=\"geometry\")[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kO1EyFACOjcT"
   },
   "source": [
    "# Load Typed SpatialRDDs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w_0_hHNQOjcT"
   },
   "source": [
    "Currently The library supports 5 typed SpatialRDDs:\n",
    "<li> RectangleRDD </li>\n",
    "<li> PointRDD </li>\n",
    "<li> PolygonRDD </li>\n",
    "<li> LineStringRDD </li>\n",
    "<li> CircleRDD </li>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:39.935001Z",
     "iopub.status.busy": "2024-03-24T21:19:39.934745Z",
     "iopub.status.idle": "2024-03-24T21:19:40.290056Z",
     "shell.execute_reply": "2024-03-24T21:19:40.289289Z"
    },
    "id": "2m2k-n-LOjcT"
   },
   "outputs": [],
   "source": [
    "rectangle_rdd = RectangleRDD(sc, \"data/zcta510-small.csv\", FileDataSplitter.CSV, True, 11)\n",
    "point_rdd = PointRDD(sc, \"data/arealm-small.csv\", 1, FileDataSplitter.CSV, False, 11)\n",
    "polygon_rdd = PolygonRDD(sc, \"data/primaryroads-polygon.csv\", FileDataSplitter.CSV, True, 11)\n",
    "linestring_rdd = LineStringRDD(sc, \"data/primaryroads-linestring.csv\", FileDataSplitter.CSV, True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:40.293094Z",
     "iopub.status.busy": "2024-03-24T21:19:40.292877Z",
     "iopub.status.idle": "2024-03-24T21:19:40.562908Z",
     "shell.execute_reply": "2024-03-24T21:19:40.561994Z"
    },
    "id": "qRHPdLYdOjcb",
    "outputId": "ef9149b6-ad2c-46f0-fba0-ae478f802ba9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rectangle_rdd.analyze()\n",
    "point_rdd.analyze()\n",
    "polygon_rdd.analyze()\n",
    "linestring_rdd.analyze()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J2uTMMk1Ojcb"
   },
   "source": [
    "# Spatial Partitioning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "h3AC5CpdOjcc"
   },
   "source": [
    "Apache Sedona spatial partitioning method can significantly speed up the join query. Three spatial partitioning methods are available: KDB-Tree, Quad-Tree and R-Tree. Two SpatialRDD must be partitioned by the same way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:40.567408Z",
     "iopub.status.busy": "2024-03-24T21:19:40.567103Z",
     "iopub.status.idle": "2024-03-24T21:19:40.656579Z",
     "shell.execute_reply": "2024-03-24T21:19:40.655637Z"
    },
    "id": "2icbQf9NOjcc",
    "outputId": "a8f20089-8b5d-42bf-b6ad-ad23f57ee496"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "point_rdd.spatialPartitioning(GridType.KDBTREE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_ipdd7umOjcc"
   },
   "source": [
    "# Create Index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hEPaVPmJOjcc"
   },
   "source": [
    "Apache Sedona provides two types of spatial indexes, Quad-Tree and R-Tree. Once you specify an index type, Apache Sedona will build a local tree index on each of the SpatialRDD partition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:40.660644Z",
     "iopub.status.busy": "2024-03-24T21:19:40.660349Z",
     "iopub.status.idle": "2024-03-24T21:19:40.668103Z",
     "shell.execute_reply": "2024-03-24T21:19:40.667363Z"
    },
    "id": "3YIsx_55Ojcc"
   },
   "outputs": [],
   "source": [
    "point_rdd.buildIndex(IndexType.RTREE, True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LRHcULQDOjcc"
   },
   "source": [
    "# SpatialJoin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZnnDo119Ojcc"
   },
   "source": [
    "Spatial join is operation which combines data based on spatial relations like:\n",
    "<li> intersects </li>\n",
    "<li> touches </li>\n",
    "<li> within </li>\n",
    "<li> etc </li>\n",
    "\n",
    "To Use Spatial Join in GeoPyspark library please use JoinQuery object, which has implemented below methods:\n",
    "```python\n",
    "SpatialJoinQuery(spatialRDD: SpatialRDD, queryRDD: SpatialRDD, useIndex: bool, considerBoundaryIntersection: bool) -> RDD\n",
    "\n",
    "DistanceJoinQuery(spatialRDD: SpatialRDD, queryRDD: SpatialRDD, useIndex: bool, considerBoundaryIntersection: bool) -> RDD\n",
    "\n",
    "spatialJoin(queryWindowRDD: SpatialRDD, objectRDD: SpatialRDD, joinParams: JoinParams) -> RDD\n",
    "\n",
    "DistanceJoinQueryFlat(spatialRDD: SpatialRDD, queryRDD: SpatialRDD, useIndex: bool, considerBoundaryIntersection: bool) -> RDD\n",
    "\n",
    "SpatialJoinQueryFlat(spatialRDD: SpatialRDD, queryRDD: SpatialRDD, useIndex: bool, considerBoundaryIntersection: bool) -> RDD\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "icOL3ukTOjcc"
   },
   "source": [
    "## Example SpatialJoinQueryFlat PointRDD with RectangleRDD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:40.672026Z",
     "iopub.status.busy": "2024-03-24T21:19:40.671532Z",
     "iopub.status.idle": "2024-03-24T21:19:40.786361Z",
     "shell.execute_reply": "2024-03-24T21:19:40.785610Z"
    },
    "id": "unG1rYNZOjcc"
   },
   "outputs": [],
   "source": [
    "# partitioning the data\n",
    "point_rdd.spatialPartitioning(GridType.KDBTREE)\n",
    "rectangle_rdd.spatialPartitioning(point_rdd.getPartitioner())\n",
    "# building an index\n",
    "point_rdd.buildIndex(IndexType.RTREE, True)\n",
    "# Perform Spatial Join Query\n",
    "result = JoinQuery.SpatialJoinQueryFlat(point_rdd, rectangle_rdd, False, True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5A0s6CE5Ojcd"
   },
   "source": [
    "As result we will get RDD[GeoData, GeoData]\n",
    "It can be used like any other Python RDD. You can use map, take, collect and other functions  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:40.792138Z",
     "iopub.status.busy": "2024-03-24T21:19:40.790797Z",
     "iopub.status.idle": "2024-03-24T21:19:40.811952Z",
     "shell.execute_reply": "2024-03-24T21:19:40.811171Z"
    },
    "id": "8dw_JRafOjcd",
    "outputId": "d688a5ec-27e0-4fe6-846d-bdaf98a824a3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MapPartitionsRDD[63] at map at FlatPairRddConverter.scala:30"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:40.815531Z",
     "iopub.status.busy": "2024-03-24T21:19:40.815240Z",
     "iopub.status.idle": "2024-03-24T21:19:42.168430Z",
     "shell.execute_reply": "2024-03-24T21:19:42.167697Z"
    },
    "id": "7QKdzz9ZOjcd",
    "outputId": "565047a0-8f8c-4abd-a370-47731886c95c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[Geometry: Polygon userData: , Geometry: Point userData: ],\n",
       " [Geometry: Polygon userData: , Geometry: Point userData: ]]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.take(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:42.171501Z",
     "iopub.status.busy": "2024-03-24T21:19:42.170987Z",
     "iopub.status.idle": "2024-03-24T21:19:42.370220Z",
     "shell.execute_reply": "2024-03-24T21:19:42.369460Z"
    },
    "id": "cBMsFNe2Ojcd",
    "outputId": "166c9992-13e8-4603-c306-2fbd8cb0b815"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[Geometry: Polygon userData: , Geometry: Point userData: ],\n",
       " [Geometry: Polygon userData: , Geometry: Point userData: ],\n",
       " [Geometry: Polygon userData: , Geometry: Point userData: ]]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.collect()[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:42.374258Z",
     "iopub.status.busy": "2024-03-24T21:19:42.373763Z",
     "iopub.status.idle": "2024-03-24T21:19:43.431771Z",
     "shell.execute_reply": "2024-03-24T21:19:43.430984Z"
    },
    "id": "ZUPAfZ0wOjcd",
    "outputId": "7139d469-1c67-452f-dfc6-5ce6062bf81c"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "[Stage 44:=================================================>      (14 + 2) / 16]\r",
      "\r",
      "                                                                                \r"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.0, 0.0, 0.0, 0.0, 0.0]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# getting distance using SpatialObjects\n",
    "result.map(lambda x: x[0].geom.distance(x[1].geom)).take(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:43.434936Z",
     "iopub.status.busy": "2024-03-24T21:19:43.434441Z",
     "iopub.status.idle": "2024-03-24T21:19:44.285694Z",
     "shell.execute_reply": "2024-03-24T21:19:44.285005Z"
    },
    "id": "2ztnCLMnOjcd",
    "outputId": "7ad1f07a-7103-467a-b97d-2adcb496694b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.057069904940998895,\n",
       " 0.026651558685001447,\n",
       " 0.026651558685001447,\n",
       " 0.026651558685001447,\n",
       " 0.026651558685001447]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# getting area of polygon data\n",
    "result.map(lambda x: x[0].geom.area).take(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:44.289448Z",
     "iopub.status.busy": "2024-03-24T21:19:44.288765Z",
     "iopub.status.idle": "2024-03-24T21:19:44.292643Z",
     "shell.execute_reply": "2024-03-24T21:19:44.292119Z"
    },
    "id": "nt-r8k9VOjcd"
   },
   "outputs": [],
   "source": [
    "# Base on result you can create DataFrame object, using map function and build DataFrame from RDD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:44.295191Z",
     "iopub.status.busy": "2024-03-24T21:19:44.294814Z",
     "iopub.status.idle": "2024-03-24T21:19:44.298040Z",
     "shell.execute_reply": "2024-03-24T21:19:44.297487Z"
    },
    "id": "nALxyjQiOjce"
   },
   "outputs": [],
   "source": [
    "schema = StructType(\n",
    "    [\n",
    "        StructField(\"geom_left\", GeometryType(), False),\n",
    "        StructField(\"geom_right\", GeometryType(), False)\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:44.300548Z",
     "iopub.status.busy": "2024-03-24T21:19:44.300183Z",
     "iopub.status.idle": "2024-03-24T21:19:45.400063Z",
     "shell.execute_reply": "2024-03-24T21:19:45.399193Z"
    },
    "id": "UWqys-lqOjce",
    "outputId": "8b85bdf9-ea59-4306-fe79-cb47af321128"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+\n",
      "|           geom_left|          geom_right|\n",
      "+--------------------+--------------------+\n",
      "|POLYGON ((-87.285...|POINT (-87.186331...|\n",
      "|POLYGON ((-87.229...|POINT (-87.204033...|\n",
      "|POLYGON ((-87.229...|POINT (-87.204299...|\n",
      "|POLYGON ((-87.229...|POINT (-87.19351 ...|\n",
      "|POLYGON ((-87.229...|POINT (-87.18222 ...|\n",
      "+--------------------+--------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "[Stage 62:=================================================>      (14 + 2) / 16]\r",
      "\r",
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "# Set verifySchema to False\n",
    "spatial_join_result = result.map(lambda x: [x[0].geom, x[1].geom])\n",
    "sedona.createDataFrame(spatial_join_result, schema, verifySchema=False).show(5, True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:45.403014Z",
     "iopub.status.busy": "2024-03-24T21:19:45.402553Z",
     "iopub.status.idle": "2024-03-24T21:19:45.405897Z",
     "shell.execute_reply": "2024-03-24T21:19:45.405219Z"
    },
    "id": "pkY70z0lOjce"
   },
   "outputs": [],
   "source": [
    "# Above code produces DataFrame with geometry Data type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:45.408682Z",
     "iopub.status.busy": "2024-03-24T21:19:45.408244Z",
     "iopub.status.idle": "2024-03-24T21:19:45.428310Z",
     "shell.execute_reply": "2024-03-24T21:19:45.427604Z"
    },
    "id": "_BQVrY-IOjce",
    "outputId": "87a00dfd-cd77-47a2-f73a-48c63be79c8f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- geom_left: geometry (nullable = false)\n",
      " |-- geom_right: geometry (nullable = false)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sedona.createDataFrame(spatial_join_result, schema, verifySchema=False).printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5oID6P5kOjce"
   },
   "source": [
    "We can create DataFrame object from Spatial Pair RDD using Adapter object as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:45.431104Z",
     "iopub.status.busy": "2024-03-24T21:19:45.430649Z",
     "iopub.status.idle": "2024-03-24T21:19:47.436724Z",
     "shell.execute_reply": "2024-03-24T21:19:47.435979Z"
    },
    "id": "OyD3_m_zOjce",
    "outputId": "27fb6d7d-013e-480a-de6d-80e4a72c1fc7"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "[Stage 71:=================================================>      (14 + 2) / 16]\r",
      "\r",
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-----+--------------------+-----+\n",
      "|              geom_1|attr1|              geom_2|attr2|\n",
      "+--------------------+-----+--------------------+-----+\n",
      "|POLYGON ((-87.285...|     |POINT (-87.186331...|     |\n",
      "|POLYGON ((-87.229...|     |POINT (-87.204033...|     |\n",
      "|POLYGON ((-87.229...|     |POINT (-87.204299...|     |\n",
      "|POLYGON ((-87.229...|     |POINT (-87.19351 ...|     |\n",
      "|POLYGON ((-87.229...|     |POINT (-87.18222 ...|     |\n",
      "+--------------------+-----+--------------------+-----+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "[Stage 80:=================================================>      (14 + 2) / 16]\r",
      "\r",
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "Adapter.toDf(result, [\"attr1\"], [\"attr2\"], sedona).show(5, True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qAmXD2SKOjce"
   },
   "source": [
    "This also produce DataFrame with geometry DataType"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:47.440164Z",
     "iopub.status.busy": "2024-03-24T21:19:47.439747Z",
     "iopub.status.idle": "2024-03-24T21:19:48.422922Z",
     "shell.execute_reply": "2024-03-24T21:19:48.422059Z"
    },
    "id": "U7evFgqkOjce",
    "outputId": "530f8329-c296-47e6-8ccf-642bf0596817"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- geom_1: geometry (nullable = true)\n",
      " |-- attr1: string (nullable = true)\n",
      " |-- geom_2: geometry (nullable = true)\n",
      " |-- attr2: string (nullable = true)\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "[Stage 89:====================================================>   (15 + 1) / 16]\r",
      "\r",
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "Adapter.toDf(result, [\"attr1\"], [\"attr2\"], sedona).printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TldNHgcQOjcf"
   },
   "source": [
    "We can create RDD which will be of type RDD[GeoData, List[GeoData]]\n",
    "We can for example calculate number of Points within some polygon data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GlhD3mSvOjcf"
   },
   "source": [
    "To do that we can use code specified below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:48.425816Z",
     "iopub.status.busy": "2024-03-24T21:19:48.425448Z",
     "iopub.status.idle": "2024-03-24T21:19:48.463549Z",
     "shell.execute_reply": "2024-03-24T21:19:48.462733Z"
    },
    "id": "bl5mg9qdOjcf"
   },
   "outputs": [],
   "source": [
    "point_rdd.spatialPartitioning(GridType.KDBTREE)\n",
    "rectangle_rdd.spatialPartitioning(point_rdd.getPartitioner())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:48.466749Z",
     "iopub.status.busy": "2024-03-24T21:19:48.466312Z",
     "iopub.status.idle": "2024-03-24T21:19:48.506205Z",
     "shell.execute_reply": "2024-03-24T21:19:48.505426Z"
    },
    "id": "6S1gs7nDOjcf"
   },
   "outputs": [],
   "source": [
    "spatial_join_result_non_flat = JoinQuery.SpatialJoinQuery(point_rdd, rectangle_rdd, False, True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:48.509763Z",
     "iopub.status.busy": "2024-03-24T21:19:48.509475Z",
     "iopub.status.idle": "2024-03-24T21:19:48.513803Z",
     "shell.execute_reply": "2024-03-24T21:19:48.513112Z"
    },
    "id": "ZJiTacNkOjcf"
   },
   "outputs": [],
   "source": [
    "# number of point for each polygon\n",
    "number_of_points = spatial_join_result_non_flat.map(lambda x: [x[0].geom, x[1].__len__()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:48.517555Z",
     "iopub.status.busy": "2024-03-24T21:19:48.516833Z",
     "iopub.status.idle": "2024-03-24T21:19:48.520956Z",
     "shell.execute_reply": "2024-03-24T21:19:48.520260Z"
    },
    "id": "-EtrJVuROjcf"
   },
   "outputs": [],
   "source": [
    "schema = StructType([\n",
    "    StructField(\"geometry\", GeometryType(), False),\n",
    "    StructField(\"number_of_points\", LongType(), False)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:48.523808Z",
     "iopub.status.busy": "2024-03-24T21:19:48.523607Z",
     "iopub.status.idle": "2024-03-24T21:19:49.643046Z",
     "shell.execute_reply": "2024-03-24T21:19:49.642186Z"
    },
    "id": "mktoa5-HOjcf",
    "outputId": "170b19d6-9498-4dbb-86d9-831fe2337708"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+----------------+\n",
      "|            geometry|number_of_points|\n",
      "+--------------------+----------------+\n",
      "|POLYGON ((-86.860...|              12|\n",
      "|POLYGON ((-87.082...|              12|\n",
      "|POLYGON ((-86.749...|               4|\n",
      "|POLYGON ((-87.285...|              26|\n",
      "|POLYGON ((-87.105...|              15|\n",
      "|POLYGON ((-87.092...|               5|\n",
      "|POLYGON ((-86.697...|               1|\n",
      "|POLYGON ((-87.229...|               7|\n",
      "|POLYGON ((-86.816...|               6|\n",
      "|POLYGON ((-87.114...|              15|\n",
      "+--------------------+----------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sedona.createDataFrame(number_of_points, schema, verifySchema=False).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZosmctfPOjcf"
   },
   "source": [
    "# KNNQuery"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "utC2fQXwOjcg"
   },
   "source": [
    "Spatial KNNQuery is operation which help us find answer which k number of geometries lays closest to other geometry.\n",
    "\n",
    "For Example:\n",
    "    5 closest Shops to your home. To use Spatial KNNQuery please use object\n",
    "<b> KNNQuery </b> which has one method:\n",
    "```python\n",
    "SpatialKnnQuery(spatialRDD: SpatialRDD, originalQueryPoint: BaseGeometry, k: int,  useIndex: bool)-> List[GeoData]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sTcvQarvOjcg"
   },
   "source": [
    "### Finds 5 closest points from PointRDD to given Point"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:49.645934Z",
     "iopub.status.busy": "2024-03-24T21:19:49.645718Z",
     "iopub.status.idle": "2024-03-24T21:19:49.741091Z",
     "shell.execute_reply": "2024-03-24T21:19:49.740241Z"
    },
    "id": "n6BIfKk9Ojcg"
   },
   "outputs": [],
   "source": [
    "result = KNNQuery.SpatialKnnQuery(point_rdd, Point(-84.01, 34.01), 5, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:49.745699Z",
     "iopub.status.busy": "2024-03-24T21:19:49.745425Z",
     "iopub.status.idle": "2024-03-24T21:19:49.752211Z",
     "shell.execute_reply": "2024-03-24T21:19:49.751581Z"
    },
    "id": "i-SgOWD2Ojcg",
    "outputId": "a69d9f13-1aeb-4aed-e726-ab1767ee5143"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Geometry: Point userData: ,\n",
       " Geometry: Point userData: ,\n",
       " Geometry: Point userData: ,\n",
       " Geometry: Point userData: ,\n",
       " Geometry: Point userData: ]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Dft5_7AMOjcg"
   },
   "source": [
    "As Reference geometry you can also use Polygon or LineString object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:49.755745Z",
     "iopub.status.busy": "2024-03-24T21:19:49.755091Z",
     "iopub.status.idle": "2024-03-24T21:19:49.969657Z",
     "shell.execute_reply": "2024-03-24T21:19:49.968923Z"
    },
    "id": "C-G_a1iXOjcg"
   },
   "outputs": [],
   "source": [
    "polygon = Polygon(\n",
    "    [(-84.237756, 33.904859), (-84.237756, 34.090426),\n",
    "     (-83.833011, 34.090426), (-83.833011, 33.904859),\n",
    "     (-84.237756, 33.904859)\n",
    "    ])\n",
    "polygons_nearby = KNNQuery.SpatialKnnQuery(polygon_rdd, polygon, 5, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:49.973498Z",
     "iopub.status.busy": "2024-03-24T21:19:49.973193Z",
     "iopub.status.idle": "2024-03-24T21:19:49.978541Z",
     "shell.execute_reply": "2024-03-24T21:19:49.977846Z"
    },
    "id": "pYtK1T1oOjcg",
    "outputId": "d6b951cc-bca0-4b83-d42f-d8dde0ab7a8f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Geometry: Polygon userData: ,\n",
       " Geometry: Polygon userData: ,\n",
       " Geometry: Polygon userData: ,\n",
       " Geometry: Polygon userData: ,\n",
       " Geometry: Polygon userData: ]"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "polygons_nearby"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 0
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:49.982343Z",
     "iopub.status.busy": "2024-03-24T21:19:49.981664Z",
     "iopub.status.idle": "2024-03-24T21:19:49.987476Z",
     "shell.execute_reply": "2024-03-24T21:19:49.986574Z"
    },
    "id": "-4Mf5TP-Ojcg",
    "outputId": "2edfc5ae-f8d9-4225-c88e-12bb74d58a87"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'POLYGON ((-83.993559 34.087259, -83.993559 34.131247, -83.959903 34.131247, -83.959903 34.087259, -83.993559 34.087259))'"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "polygons_nearby[0].geom.wkt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SxP3ufPGOjch"
   },
   "source": [
    "# RangeQuery"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Air8I_XIOjch"
   },
   "source": [
    "A spatial range query takes as input a range query window and an SpatialRDD and returns all geometries that intersect / are fully covered by the query window.\n",
    "RangeQuery has one method:\n",
    "\n",
    "```python\n",
    "SpatialRangeQuery(self, spatialRDD: SpatialRDD, rangeQueryWindow: BaseGeometry, considerBoundaryIntersection: bool, usingIndex: bool) -> RDD\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:49.992257Z",
     "iopub.status.busy": "2024-03-24T21:19:49.990610Z",
     "iopub.status.idle": "2024-03-24T21:19:49.995332Z",
     "shell.execute_reply": "2024-03-24T21:19:49.994562Z"
    },
    "id": "7YQd4wvuOjch"
   },
   "outputs": [],
   "source": [
    "from sedona.core.geom.envelope import Envelope"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:49.998819Z",
     "iopub.status.busy": "2024-03-24T21:19:49.998149Z",
     "iopub.status.idle": "2024-03-24T21:19:50.013718Z",
     "shell.execute_reply": "2024-03-24T21:19:50.012930Z"
    },
    "id": "8WLf9PLbOjch"
   },
   "outputs": [],
   "source": [
    "query_envelope = Envelope(-85.01, -60.01, 34.01, 50.01)\n",
    "\n",
    "result_range_query = RangeQuery.SpatialRangeQuery(linestring_rdd, query_envelope, False, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:50.017574Z",
     "iopub.status.busy": "2024-03-24T21:19:50.016849Z",
     "iopub.status.idle": "2024-03-24T21:19:50.023018Z",
     "shell.execute_reply": "2024-03-24T21:19:50.022324Z"
    },
    "id": "9Wp7sOciOjch",
    "outputId": "ae9362ae-4ee1-4fa5-84aa-7b1053c17229"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MapPartitionsRDD[127] at map at GeometryRddConverter.scala:30"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_range_query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:50.026540Z",
     "iopub.status.busy": "2024-03-24T21:19:50.025827Z",
     "iopub.status.idle": "2024-03-24T21:19:50.083058Z",
     "shell.execute_reply": "2024-03-24T21:19:50.082158Z"
    },
    "id": "C1mittdHOjch",
    "outputId": "c93a70cd-cabb-4e56-b399-99f23a9b7778"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Geometry: LineString userData: ,\n",
       " Geometry: LineString userData: ,\n",
       " Geometry: LineString userData: ,\n",
       " Geometry: LineString userData: ,\n",
       " Geometry: LineString userData: ,\n",
       " Geometry: LineString userData: ]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_range_query.take(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:50.086549Z",
     "iopub.status.busy": "2024-03-24T21:19:50.086032Z",
     "iopub.status.idle": "2024-03-24T21:19:50.089475Z",
     "shell.execute_reply": "2024-03-24T21:19:50.088798Z"
    },
    "id": "Jv3ik2obOjch"
   },
   "outputs": [],
   "source": [
    "# Creating DataFrame from result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:50.093064Z",
     "iopub.status.busy": "2024-03-24T21:19:50.092307Z",
     "iopub.status.idle": "2024-03-24T21:19:50.096417Z",
     "shell.execute_reply": "2024-03-24T21:19:50.095635Z"
    },
    "id": "OF2xJfLCOjch"
   },
   "outputs": [],
   "source": [
    "schema = StructType([StructField(\"geometry\", GeometryType(), False)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:50.099702Z",
     "iopub.status.busy": "2024-03-24T21:19:50.099220Z",
     "iopub.status.idle": "2024-03-24T21:19:50.241728Z",
     "shell.execute_reply": "2024-03-24T21:19:50.240795Z"
    },
    "id": "7Y4ruCQ-Ojci",
    "outputId": "23bc8a1d-d16a-4ba7-a490-3b39b9d782cd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|            geometry|\n",
      "+--------------------+\n",
      "|LINESTRING (-72.1...|\n",
      "|LINESTRING (-72.4...|\n",
      "|LINESTRING (-72.4...|\n",
      "|LINESTRING (-73.4...|\n",
      "|LINESTRING (-73.6...|\n",
      "+--------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sedona.createDataFrame(\n",
    "    result_range_query.map(lambda x: [x.geom]),\n",
    "    schema,\n",
    "    verifySchema=False\n",
    ").show(5, True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jwiSJcuLOjci"
   },
   "source": [
    "# Load From other Formats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aLuQLIfJOjci"
   },
   "source": [
    "GeoPyspark allows to load the data from other Data formats like:\n",
    "<li> GeoJSON </li>\n",
    "<li> Shapefile </li>\n",
    "<li> WKB </li>\n",
    "<li> WKT </li>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:50.246011Z",
     "iopub.status.busy": "2024-03-24T21:19:50.245314Z",
     "iopub.status.idle": "2024-03-24T21:19:50.249535Z",
     "shell.execute_reply": "2024-03-24T21:19:50.248731Z"
    },
    "id": "vuiEDP9uOjci"
   },
   "outputs": [],
   "source": [
    "## ShapeFile - load to SpatialRDD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:50.253330Z",
     "iopub.status.busy": "2024-03-24T21:19:50.253054Z",
     "iopub.status.idle": "2024-03-24T21:19:50.368446Z",
     "shell.execute_reply": "2024-03-24T21:19:50.367724Z"
    },
    "id": "3bcx_gxAOjci"
   },
   "outputs": [],
   "source": [
    "shape_rdd = ShapefileReader.readToGeometryRDD(sc, \"data/polygon\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:50.371853Z",
     "iopub.status.busy": "2024-03-24T21:19:50.371368Z",
     "iopub.status.idle": "2024-03-24T21:19:50.376414Z",
     "shell.execute_reply": "2024-03-24T21:19:50.375751Z"
    },
    "id": "nIjo83o8Ojci",
    "outputId": "69c02c33-e22f-44a5-d6e0-fb509f3b4e7f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sedona.core.SpatialRDD.spatial_rdd.SpatialRDD at 0x7fe09675a310>"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shape_rdd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:50.379264Z",
     "iopub.status.busy": "2024-03-24T21:19:50.378807Z",
     "iopub.status.idle": "2024-03-24T21:19:50.503898Z",
     "shell.execute_reply": "2024-03-24T21:19:50.503111Z"
    },
    "id": "4y9EXCKfOjci",
    "outputId": "62c206c8-e84f-4532-c8f7-8075d519164d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|            geometry|\n",
      "+--------------------+\n",
      "|MULTIPOLYGON (((1...|\n",
      "|MULTIPOLYGON (((-...|\n",
      "|MULTIPOLYGON (((1...|\n",
      "|POLYGON ((118.362...|\n",
      "|MULTIPOLYGON (((-...|\n",
      "+--------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "Adapter.toDf(shape_rdd, sedona).show(5, True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:50.506537Z",
     "iopub.status.busy": "2024-03-24T21:19:50.506306Z",
     "iopub.status.idle": "2024-03-24T21:19:50.509338Z",
     "shell.execute_reply": "2024-03-24T21:19:50.508778Z"
    },
    "id": "LkVpdZgGOjci"
   },
   "outputs": [],
   "source": [
    "## GeoJSON - load to SpatialRDD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Mh1K7181Ojcj"
   },
   "source": [
    "```\n",
    "{ \"type\": \"Feature\", \"properties\": { \"STATEFP\": \"01\", \"COUNTYFP\": \"077\", \"TRACTCE\": \"011501\", \"BLKGRPCE\": \"5\", \"AFFGEOID\": \"1500000US010770115015\", \"GEOID\": \"010770115015\", \"NAME\": \"5\", \"LSAD\": \"BG\", \"ALAND\": 6844991, \"AWATER\": 32636 }, \"geometry\": { \"type\": \"Polygon\", \"coordinates\": [ [ [ -87.621765, 34.873444 ], [ -87.617535, 34.873369 ], [ -87.6123, 34.873337 ], [ -87.604049, 34.873303 ], [ -87.604033, 34.872316 ], [ -87.60415, 34.867502 ], [ -87.604218, 34.865687 ], [ -87.604409, 34.858537 ], [ -87.604018, 34.851336 ], [ -87.603716, 34.844829 ], [ -87.603696, 34.844307 ], [ -87.603673, 34.841884 ], [ -87.60372, 34.841003 ], [ -87.603879, 34.838423 ], [ -87.603888, 34.837682 ], [ -87.603889, 34.83763 ], [ -87.613127, 34.833938 ], [ -87.616451, 34.832699 ], [ -87.621041, 34.831431 ], [ -87.621056, 34.831526 ], [ -87.62112, 34.831925 ], [ -87.621603, 34.8352 ], [ -87.62158, 34.836087 ], [ -87.621383, 34.84329 ], [ -87.621359, 34.844438 ], [ -87.62129, 34.846387 ], [ -87.62119, 34.85053 ], [ -87.62144, 34.865379 ], [ -87.621765, 34.873444 ] ] ] } },\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:50.511983Z",
     "iopub.status.busy": "2024-03-24T21:19:50.511621Z",
     "iopub.status.idle": "2024-03-24T21:19:50.600739Z",
     "shell.execute_reply": "2024-03-24T21:19:50.599993Z"
    },
    "id": "g_a3q71ZOjcj"
   },
   "outputs": [],
   "source": [
    "geo_json_rdd = GeoJsonReader.readToGeometryRDD(sc, \"data/testPolygon.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:50.604443Z",
     "iopub.status.busy": "2024-03-24T21:19:50.603881Z",
     "iopub.status.idle": "2024-03-24T21:19:50.609272Z",
     "shell.execute_reply": "2024-03-24T21:19:50.608518Z"
    },
    "id": "LGrMegQ6Ojcj",
    "outputId": "faf4d1b1-a49a-4139-c881-dffce997ea0d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sedona.core.SpatialRDD.spatial_rdd.SpatialRDD at 0x7fe096768fd0>"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "geo_json_rdd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:50.612110Z",
     "iopub.status.busy": "2024-03-24T21:19:50.611909Z",
     "iopub.status.idle": "2024-03-24T21:19:50.917915Z",
     "shell.execute_reply": "2024-03-24T21:19:50.917361Z"
    },
    "id": "ZfH32XssOjcj",
    "outputId": "a773dbe0-962f-4c45-c1a4-4b44914b69f5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-------+--------+-------+--------+--------------------+------------+----+----+--------+\n",
      "|            geometry|STATEFP|COUNTYFP|TRACTCE|BLKGRPCE|            AFFGEOID|       GEOID|NAME|LSAD|   ALAND|\n",
      "+--------------------+-------+--------+-------+--------+--------------------+------------+----+----+--------+\n",
      "|POLYGON ((-87.621...|     01|     077| 011501|       5|1500000US01077011...|010770115015|   5|  BG| 6844991|\n",
      "|POLYGON ((-85.719...|     01|     045| 021102|       4|1500000US01045021...|010450211024|   4|  BG|11360854|\n",
      "|POLYGON ((-86.000...|     01|     055| 001300|       3|1500000US01055001...|010550013003|   3|  BG| 1378742|\n",
      "|POLYGON ((-86.574...|     01|     089| 001700|       2|1500000US01089001...|010890017002|   2|  BG| 1040641|\n",
      "|POLYGON ((-85.382...|     01|     069| 041400|       1|1500000US01069041...|010690414001|   1|  BG| 8243574|\n",
      "+--------------------+-------+--------+-------+--------+--------------------+------------+----+----+--------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "Adapter.toDf(geo_json_rdd, sedona).drop(\"AWATER\").show(5, True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:50.920802Z",
     "iopub.status.busy": "2024-03-24T21:19:50.920594Z",
     "iopub.status.idle": "2024-03-24T21:19:50.923535Z",
     "shell.execute_reply": "2024-03-24T21:19:50.922753Z"
    },
    "id": "wuJkRwbROjcj"
   },
   "outputs": [],
   "source": [
    "## WKT - loading to SpatialRDD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:50.926072Z",
     "iopub.status.busy": "2024-03-24T21:19:50.925869Z",
     "iopub.status.idle": "2024-03-24T21:19:50.984871Z",
     "shell.execute_reply": "2024-03-24T21:19:50.984129Z"
    },
    "id": "0d9owuZROjcj"
   },
   "outputs": [],
   "source": [
    "wkt_rdd = WktReader.readToGeometryRDD(sc, \"data/county_small.tsv\", 0, True, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:50.988837Z",
     "iopub.status.busy": "2024-03-24T21:19:50.988532Z",
     "iopub.status.idle": "2024-03-24T21:19:50.993165Z",
     "shell.execute_reply": "2024-03-24T21:19:50.992520Z"
    },
    "id": "XKALKBbeOjcj",
    "outputId": "e0b7e2d7-7626-4154-8028-518c9b8d6804"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sedona.core.SpatialRDD.spatial_rdd.SpatialRDD at 0x7fe0967a3c70>"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wkt_rdd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:50.997097Z",
     "iopub.status.busy": "2024-03-24T21:19:50.996628Z",
     "iopub.status.idle": "2024-03-24T21:19:51.010176Z",
     "shell.execute_reply": "2024-03-24T21:19:51.009210Z"
    },
    "id": "UUJfqy5EOjcj",
    "outputId": "7806b0df-cd77-4990-ca9e-b4ce8a27a646"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- geometry: geometry (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "Adapter.toDf(wkt_rdd, sedona).printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:51.014281Z",
     "iopub.status.busy": "2024-03-24T21:19:51.013759Z",
     "iopub.status.idle": "2024-03-24T21:19:51.211909Z",
     "shell.execute_reply": "2024-03-24T21:19:51.211066Z"
    },
    "id": "qjIgGZB2Ojck",
    "outputId": "8785e560-ad7c-419c-98e0-cdfc8cfe7d5e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|            geometry|\n",
      "+--------------------+\n",
      "|POLYGON ((-97.019...|\n",
      "|POLYGON ((-123.43...|\n",
      "|POLYGON ((-104.56...|\n",
      "|POLYGON ((-96.910...|\n",
      "|POLYGON ((-98.273...|\n",
      "+--------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "Adapter.toDf(wkt_rdd, sedona).show(5, True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:51.215746Z",
     "iopub.status.busy": "2024-03-24T21:19:51.215444Z",
     "iopub.status.idle": "2024-03-24T21:19:51.219066Z",
     "shell.execute_reply": "2024-03-24T21:19:51.218308Z"
    },
    "id": "9CzKWF_ZOjck"
   },
   "outputs": [],
   "source": [
    "## WKB - load to SpatialRDD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:51.222505Z",
     "iopub.status.busy": "2024-03-24T21:19:51.221819Z",
     "iopub.status.idle": "2024-03-24T21:19:51.263760Z",
     "shell.execute_reply": "2024-03-24T21:19:51.262962Z"
    },
    "id": "1-saFLNoOjck"
   },
   "outputs": [],
   "source": [
    "wkb_rdd = WkbReader.readToGeometryRDD(sc, \"data/county_small_wkb.tsv\", 0, True, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:51.267499Z",
     "iopub.status.busy": "2024-03-24T21:19:51.267100Z",
     "iopub.status.idle": "2024-03-24T21:19:51.387569Z",
     "shell.execute_reply": "2024-03-24T21:19:51.386758Z"
    },
    "id": "WFUhy3AtOjck",
    "outputId": "46d24d80-46b7-4165-cde1-eb52817ed7a9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|            geometry|\n",
      "+--------------------+\n",
      "|POLYGON ((-97.019...|\n",
      "|POLYGON ((-123.43...|\n",
      "|POLYGON ((-104.56...|\n",
      "|POLYGON ((-96.910...|\n",
      "|POLYGON ((-98.273...|\n",
      "+--------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "Adapter.toDf(wkb_rdd, sedona).show(5, True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xj-mt-60Ojck"
   },
   "source": [
    "## Converting RDD Spatial join result to DF directly, avoiding jvm python serde"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:51.391157Z",
     "iopub.status.busy": "2024-03-24T21:19:51.390863Z",
     "iopub.status.idle": "2024-03-24T21:19:51.442896Z",
     "shell.execute_reply": "2024-03-24T21:19:51.442052Z"
    },
    "id": "5DTG6EvFOjck"
   },
   "outputs": [],
   "source": [
    "point_rdd.spatialPartitioning(GridType.KDBTREE)\n",
    "rectangle_rdd.spatialPartitioning(point_rdd.getPartitioner())\n",
    "# building an index\n",
    "point_rdd.buildIndex(IndexType.RTREE, True)\n",
    "# Perform Spatial Join Query\n",
    "result = JoinQueryRaw.SpatialJoinQueryFlat(point_rdd, rectangle_rdd, False, True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:51.446806Z",
     "iopub.status.busy": "2024-03-24T21:19:51.446481Z",
     "iopub.status.idle": "2024-03-24T21:19:51.456616Z",
     "shell.execute_reply": "2024-03-24T21:19:51.455891Z"
    },
    "id": "Fs8XaQ6kOjck"
   },
   "outputs": [],
   "source": [
    "# without passing column names, the result will contain only two geometries columns\n",
    "geometry_df = Adapter.toDf(result, sedona)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:51.460323Z",
     "iopub.status.busy": "2024-03-24T21:19:51.460054Z",
     "iopub.status.idle": "2024-03-24T21:19:51.466765Z",
     "shell.execute_reply": "2024-03-24T21:19:51.466010Z"
    },
    "id": "qiWJjHz7Ojck",
    "outputId": "f5c12650-bce0-4d87-cacc-dcc82f943b41"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- leftgeometry: geometry (nullable = true)\n",
      " |-- rightgeometry: geometry (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "geometry_df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:51.470651Z",
     "iopub.status.busy": "2024-03-24T21:19:51.470348Z",
     "iopub.status.idle": "2024-03-24T21:19:51.934485Z",
     "shell.execute_reply": "2024-03-24T21:19:51.933674Z"
    },
    "id": "CUuhG1HHOjcl",
    "outputId": "963d7d66-c39f-44ef-8139-772dd2bdc563"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+\n",
      "|        leftgeometry|       rightgeometry|\n",
      "+--------------------+--------------------+\n",
      "|POLYGON ((-87.229...|POINT (-87.204033...|\n",
      "|POLYGON ((-87.229...|POINT (-87.204299...|\n",
      "|POLYGON ((-87.229...|POINT (-87.19351 ...|\n",
      "|POLYGON ((-87.229...|POINT (-87.18222 ...|\n",
      "|POLYGON ((-87.285...|POINT (-87.28468 ...|\n",
      "+--------------------+--------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "geometry_df.show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:51.938352Z",
     "iopub.status.busy": "2024-03-24T21:19:51.938031Z",
     "iopub.status.idle": "2024-03-24T21:19:52.078876Z",
     "shell.execute_reply": "2024-03-24T21:19:52.078034Z"
    },
    "id": "J_113xc7Ojcl",
    "outputId": "13894e68-ca5a-4fc4-ee26-f8bc7216c8d3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row(leftgeometry=<shapely.geometry.polygon.Polygon object at 0x7fe09671d3d0>, rightgeometry=<shapely.geometry.point.Point object at 0x7fe09671db50>)"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "geometry_df.collect()[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "S-eRk6AROjcl"
   },
   "source": [
    "## Passing column names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:52.084530Z",
     "iopub.status.busy": "2024-03-24T21:19:52.083086Z",
     "iopub.status.idle": "2024-03-24T21:19:52.102917Z",
     "shell.execute_reply": "2024-03-24T21:19:52.102139Z"
    },
    "id": "5CnT4RzeOjcl"
   },
   "outputs": [],
   "source": [
    "geometry_df = Adapter.toDf(result, [\"left_user_data\"], [\"right_user_data\"], sedona)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:52.106286Z",
     "iopub.status.busy": "2024-03-24T21:19:52.105723Z",
     "iopub.status.idle": "2024-03-24T21:19:52.295285Z",
     "shell.execute_reply": "2024-03-24T21:19:52.294564Z"
    },
    "id": "qfaNUcXQOjcl",
    "outputId": "fa42ffa3-5d2c-4aaa-f2f0-de165a56554d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------+--------------------+---------------+\n",
      "|        leftgeometry|left_user_data|       rightgeometry|right_user_data|\n",
      "+--------------------+--------------+--------------------+---------------+\n",
      "|POLYGON ((-87.229...|              |POINT (-87.204033...|           null|\n",
      "|POLYGON ((-87.229...|              |POINT (-87.204299...|           null|\n",
      "|POLYGON ((-87.229...|              |POINT (-87.19351 ...|           null|\n",
      "|POLYGON ((-87.229...|              |POINT (-87.18222 ...|           null|\n",
      "|POLYGON ((-87.285...|              |POINT (-87.28468 ...|           null|\n",
      "+--------------------+--------------+--------------------+---------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "geometry_df.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TzWjjT0UOjcl"
   },
   "source": [
    "# Converting RDD Spatial join result to DF directly, avoiding jvm python serde"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:52.298919Z",
     "iopub.status.busy": "2024-03-24T21:19:52.298621Z",
     "iopub.status.idle": "2024-03-24T21:19:52.306387Z",
     "shell.execute_reply": "2024-03-24T21:19:52.305736Z"
    },
    "id": "OB_t5UOcOjcl"
   },
   "outputs": [],
   "source": [
    "query_envelope = Envelope(-85.01, -60.01, 34.01, 50.01)\n",
    "\n",
    "result_range_query = RangeQueryRaw.SpatialRangeQuery(linestring_rdd, query_envelope, False, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:52.309670Z",
     "iopub.status.busy": "2024-03-24T21:19:52.309412Z",
     "iopub.status.idle": "2024-03-24T21:19:52.319956Z",
     "shell.execute_reply": "2024-03-24T21:19:52.319203Z"
    },
    "id": "58B9hoX_Ojcl"
   },
   "outputs": [],
   "source": [
    "# converting to df\n",
    "gdf = Adapter.toDf(result_range_query, sedona)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:52.325074Z",
     "iopub.status.busy": "2024-03-24T21:19:52.323774Z",
     "iopub.status.idle": "2024-03-24T21:19:52.401364Z",
     "shell.execute_reply": "2024-03-24T21:19:52.400630Z"
    },
    "id": "Rgna3MsmOjcl",
    "outputId": "f7412685-61ac-4712-cf53-a938170315f4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|            geometry|\n",
      "+--------------------+\n",
      "|LINESTRING (-72.1...|\n",
      "|LINESTRING (-72.4...|\n",
      "|LINESTRING (-72.4...|\n",
      "|LINESTRING (-73.4...|\n",
      "|LINESTRING (-73.6...|\n",
      "+--------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "gdf.show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:52.404773Z",
     "iopub.status.busy": "2024-03-24T21:19:52.404479Z",
     "iopub.status.idle": "2024-03-24T21:19:52.409149Z",
     "shell.execute_reply": "2024-03-24T21:19:52.408463Z"
    },
    "id": "Gzw-LrkgOjcm",
    "outputId": "924b5905-530a-4058-d6d0-3f36d9e5c230"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- geometry: geometry (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "gdf.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:52.412733Z",
     "iopub.status.busy": "2024-03-24T21:19:52.412187Z",
     "iopub.status.idle": "2024-03-24T21:19:52.422907Z",
     "shell.execute_reply": "2024-03-24T21:19:52.421982Z"
    },
    "id": "C5-BtLhSOjcm"
   },
   "outputs": [],
   "source": [
    "# Passing column names\n",
    "# converting to df\n",
    "gdf_with_columns = Adapter.toDf(result_range_query, sedona, [\"_c1\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:52.426750Z",
     "iopub.status.busy": "2024-03-24T21:19:52.425939Z",
     "iopub.status.idle": "2024-03-24T21:19:52.519530Z",
     "shell.execute_reply": "2024-03-24T21:19:52.518749Z"
    },
    "id": "r9ctqnn1Ojcm",
    "outputId": "a874be8b-00fd-4b9f-c4e5-a5f8006b4202"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+---+\n",
      "|            geometry|_c1|\n",
      "+--------------------+---+\n",
      "|LINESTRING (-72.1...|   |\n",
      "|LINESTRING (-72.4...|   |\n",
      "|LINESTRING (-72.4...|   |\n",
      "|LINESTRING (-73.4...|   |\n",
      "|LINESTRING (-73.6...|   |\n",
      "+--------------------+---+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "gdf_with_columns.show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2024-03-24T21:19:52.523021Z",
     "iopub.status.busy": "2024-03-24T21:19:52.522743Z",
     "iopub.status.idle": "2024-03-24T21:19:52.529022Z",
     "shell.execute_reply": "2024-03-24T21:19:52.528368Z"
    },
    "id": "h-xj5agIOjcm",
    "outputId": "30469257-a9ee-4711-8dbe-0527ff9e22f6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- geometry: geometry (nullable = true)\n",
      " |-- _c1: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "gdf_with_columns.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VIQE0WpbUQjD"
   },
   "source": [
    "# Summary\n",
    "\n",
    "We have shown how to install Sedona with Pyspark and run a basic example (source: https://github.com/apache/sedona/blob/master/binder/ApacheSedonaCore.ipynb) on Google Colab."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "authorship_tag": "ABX9TyM9JVOiZXxIFObMpyuhPLLD",
   "include_colab_link": true,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}