notebooks/format-preprocessing-times.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8c0bc123-e6c3-4c07-8e67-2c8928c43ac7",
   "metadata": {
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "# Preprocessing Time Analysis for Different File Formats\n",
    "\n",
    "## Introduction\n",
    "\n",
    "ICESat-2's rich hierarchical photon data, primarily stored in HDF5 formats, presents both advantages and challenges. While HDF5 files excel in supporting diverse scientific workflows due to their self-describing nature and ability to house heterogeneous data, their serialization structure requires users to load an entire ATL03 HDF5 file even when only a subset is desired. This contrasts with raster data formats like cloud-optimized GeoTIFFs, which allow efficient partial data access.\n",
    "\n",
    "In our quest to optimize data access, we explore various file formats and preprocessing techniques. This includes traditional methods and cutting-edge approaches such as using kerchunk, a library designed to enhance chunked data access in cloud environments.\n",
    "\n",
    "## Objective:\n",
    "\n",
    "In this section, we will analyze the preprocessing time taken for different file formats, including kerchunk, original h5, repacked, kerchunk of repacked, flatgeobuf and geoparquet. By benchmarking these times, we aim to support data providers considering the delivery of alternative data formats for HDF5 products.\n",
    "\n",
    "## Server Context\n",
    "\n",
    "The processing time will depend on the server you are using to generate the files. The server used to generate these benchmarks was one of the [CryoCloud](https://book.cryointhecloud.com/intro.html) JupyterHub instances. It is located in AWS us-west-2 and can be configured with different CPU and RAM options.\n",
    "\n",
    "### Memory Information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "b46958e0-8951-45da-85ae-844f80cd27c5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Memory: 498.38 GB\n",
      "Available Memory: 490.67 GB\n",
      "Used Memory: 4.48 GB\n",
      "Memory Percentage in use: 1.5%\n",
      "\n",
      "NOTE: JupyterHub is a multi-user server for Jupyter notebooks, and the above method gives the memory status of the whole server. If you have many users running tasks simultaneously, the available memory will be affected by all the tasks collectively.\n"
     ]
    }
   ],
   "source": [
    "import psutil\n",
    "\n",
    "# Get Memory usage\n",
    "memory_info = psutil.virtual_memory()\n",
    "\n",
    "print(f\"Total Memory: {memory_info.total / (1024**3):.2f} GB\")\n",
    "print(f\"Available Memory: {memory_info.available / (1024**3):.2f} GB\")\n",
    "print(f\"Used Memory: {memory_info.used / (1024**3):.2f} GB\")\n",
    "print(f\"Memory Percentage in use: {memory_info.percent}%\")\n",
    "print(\"\\nNOTE: JupyterHub is a multi-user server for Jupyter notebooks, and the above method gives the memory status of the whole server. If you have many users running tasks simultaneously, the available memory will be affected by all the tasks collectively.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4eb9742e-fcfa-4f52-96e7-584cf3b7e1bc",
   "metadata": {
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "### CPU Information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "ac1d93b1-e630-4fce-9886-2c8e29973958",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of logical CPUs: 64\n",
      "Number of physical CPUs: 32\n",
      "Current Frequency: 3106.564656249999Mhz\n",
      "Max Frequency: 0.0Mhz\n",
      "Min Frequency: 0.0Mhz\n"
     ]
    }
   ],
   "source": [
    "logical_cpus = psutil.cpu_count(logical=True)\n",
    "print(f\"Number of logical CPUs: {logical_cpus}\")\n",
    "\n",
    "# Number of physical CPUs (or cores)\n",
    "physical_cpus = psutil.cpu_count(logical=False)\n",
    "print(f\"Number of physical CPUs: {physical_cpus}\")\n",
    "# CPU Frequencies\n",
    "cpu_freq = psutil.cpu_freq()\n",
    "print(f\"Current Frequency: {cpu_freq.current}Mhz\")\n",
    "print(f\"Max Frequency: {cpu_freq.max}Mhz\")\n",
    "print(f\"Min Frequency: {cpu_freq.min}Mhz\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04f2d598-62e8-4065-99ad-9bb6633e60dc",
   "metadata": {
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "## Generating Kerchunk sidecar for ATL03 files\n",
    "\n",
    "The Kerchunk library creates an accompanying metadata file (much like motorbike sidecar which rides along with the primary file). A kerchunk sidecar contains the metadata, byte range locations, compression information, and other essential details that allow for efficient, chunked access to the main data file without needing to read the whole file."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97ced3d6-cd8f-4a65-87bf-b645f4ac3864",
   "metadata": {
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "## Setting Up the Environment for Kerchunk Processing with ATL03 Files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e251ba02-2877-4c25-b413-487364cf570e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "%pip install git+https://github.com/fsspec/kerchunk\n",
    "# You may need restart the kernel after installing kerchunk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c45309c4-5be9-48d6-a82c-d4a283293825",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from kerchunk.hdf import SingleHdf5ToZarr\n",
    "import fsspec\n",
    "from pathlib import Path\n",
    "\n",
    "import os\n",
    "import ujson\n",
    "import dask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c7392771-1742-478a-beb5-5d0aae628d66",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['nasa-cryo-scratch/h5cloud/original/ATL03_20200217204710_08110612_006_01.h5']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Initialize connection to the AWS S3 filesystem for reading files.\n",
    "fs_read_files = fsspec.filesystem('s3')\n",
    "\n",
    "# Create a list of all files present in the specified S3 directory. Here I am using one of the files.\n",
    "flist = fs_read_files.glob('s3://nasa-cryo-scratch/h5cloud/original/ATL03_20200217204710_08110612_006_01.h5')\n",
    "\n",
    "def gen_json(file_url):\n",
    "    \"\"\"\n",
    "    Generate a JSON representation of the chunked structure of an HDF5 file.\n",
    "\n",
    "    Args:\n",
    "    - file_url (str): URL to the HDF5 file hosted on S3 bucket.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Configuration for opening the file: Avoid caching to reduce memory usage.\n",
    "    so = dict(mode='rb', default_fill_cache=False, default_cache_type='first')\n",
    "    \n",
    "    # Initialize connection to the S3 filesystem.\n",
    "    fs = fsspec.filesystem('s3')\n",
    "    \n",
    "    # Initialize connection to the local filesystem for saving JSON outputs.\n",
    "    fs_local = fsspec.filesystem('')  \n",
    "    \n",
    "    # Open and process the HDF5 file from S3.\n",
    "    with fs.open(file_url, **so) as infile:\n",
    "        print(f\"Processing: {file_url}\")\n",
    "        \n",
    "        # Convert the HDF5 data structure for optimized cloud access.\n",
    "        h5chunks = SingleHdf5ToZarr(infile, file_url, inline_threshold=300)\n",
    "        \n",
    "        # Determine the name of the output JSON file based on the input file's name and directory.\n",
    "        variable = file_url.split('/')[-1].split('.')[0]\n",
    "        month = file_url.split('/')[2]\n",
    "        outf = f'{month}_{variable}.json'\n",
    "        \n",
    "        # Save the processed data as a JSON file locally.\n",
    "        with fs_local.open(f\"./{outf}\", 'wb') as f:\n",
    "            f.write(ujson.dumps(h5chunks.translate()).encode())\n",
    "\n",
    "# Display the list of files to be processed.\n",
    "flist\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d56d395a-b50f-457d-885c-5f59890ba7d5",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "### Function Internals:\n",
    "\n",
    "- The function sets up the S3 filesystem and a local filesystem.\n",
    "- It opens the provided HDF5 file from S3 and processes it with `SingleHdf5ToZarr`, which is designed to create an indexed file optimized for cloud access (Zarr).\n",
    "- The resulting representation is then saved as a JSON file locally in the `./kerchunked/` directory.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41c40c5e-594a-4a31-a86f-db5682b997de",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "### Preprocessing Time calculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "13e6e170-63c4-42cc-a160-7b4f9f875a84",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing: s3://nasa-cryo-scratch/h5cloud/original/ATL03_20200217204710_08110612_006_01.h5\n",
      "Time taken for preprocessing using Kerchunk: 1291.09 seconds\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "original_data = 's3://nasa-cryo-scratch/h5cloud/original/ATL03_20200217204710_08110612_006_01.h5'\n",
    "\n",
    "# file formats or files to process\n",
    "file_url = original_data\n",
    "\n",
    "# Start the timer\n",
    "start_time = time.time()\n",
    "\n",
    "# Call the preprocessing function\n",
    "kerchunked_json = gen_json(file_url)\n",
    "\n",
    "# End the timer and store the result\n",
    "end_time = time.time()\n",
    "preprocess_time_kerchunk = end_time - start_time\n",
    "\n",
    "print(f\"Time taken for preprocessing using Kerchunk: {preprocess_time_kerchunk:.2f} seconds\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd2b6a33-50fb-48a3-878a-6096be958ffc",
   "metadata": {
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "## Generating Geoparquet Data Samples from ATL03 Samples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e909721d-117a-4763-8553-e29277fa4eee",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "In this sub-section, the ICESat-2's ATL03 data samples are converted into the geoparquet format. Geoparquet is a columnar storage file format optimized for big data processing. By converting ATL03 samples to geoparquet, we can achieve more efficient data access and processing, especially in cloud-based environments."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3e7475c-4e0a-4ff6-b81b-0abd0d616139",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "### Using Sliderule\n",
    "\n",
    "SlideRule is a processing engine designed to work with ICESat-2 data. In this sub-section, we utilize SlideRule's capability to handle geoparquet by choosing the geoparquet export option. This will allow you to download the desired ATL03 data granules by specifying their IDs.\n",
    "\n",
    "SlideRule will save the granules to a bucket specified as the `output` parameter.\n",
    "\n",
    "#### Reference Guide\n",
    "\n",
    "For a detailed walkthrough of the process, especially the steps involving SlideRule, refer to the tutorial available at ICESat-2 Hackweek [SlideRule Tutorial](https://icesat-2-2023.hackweek.io/tutorials/sliderule/parquet-s3.html). This guide provides comprehensive instructions, tips, and best practices for working with ICESat-2 data and the geoparquet format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "878fe700-9058-4427-8d01-cd27e9234a33",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "os.environ['USE_PYGEOS'] = '0'\n",
    "from sliderule import icesat2, earthdata, io\n",
    "import boto3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "41be580c-49ca-43fc-bdcd-81e39ee7555a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Initialize the connection to SlideRule service\n",
    "icesat2.init(\"slideruleearth.io\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "a2f59106-578f-4a36-a6fa-c2930c25f2d4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Granule to process, we are using one granule to test but it can be more\n",
    "granule = 'ATL03_20190219140808_08110212_006_02.h5'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "bea7ab01-5be0-479a-8419-004b29e91643",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Fetch AWS credentials for accessing S3 resources and these credentials are needed for sliderule\n",
    "client = boto3.client('sts')\n",
    "with open(os.environ['AWS_WEB_IDENTITY_TOKEN_FILE']) as f:\n",
    "    TOKEN = f.read()\n",
    "\n",
    "response = client.assume_role_with_web_identity(\n",
    "    RoleArn=os.environ['AWS_ROLE_ARN'],\n",
    "    RoleSessionName=os.environ['JUPYTERHUB_CLIENT_ID'],\n",
    "    WebIdentityToken=TOKEN,\n",
    "    DurationSeconds=3600\n",
    ")\n",
    "\n",
    "ACCESS_KEY_ID = response['Credentials']['AccessKeyId']\n",
    "SECRET_ACCESS_KEY_ID = response['Credentials']['SecretAccessKey']\n",
    "SESSION_TOKEN = response['Credentials']['SessionToken']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "460f76e3-a443-443f-90e1-7e60f93c616c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Function to convert a given ATL03 granule to geoparquet format and save to S3\n",
    "def get_gpq(granule):\n",
    "    asset = \"icesat2\"\n",
    "    # Define the S3 path for the output geoparquet file\n",
    "    output = f\"s3://nasa-cryo-scratch/h5cloud/geoparquet/{granule}.gpq\"\n",
    "    \n",
    "    # Configuration parameters for the conversion process\n",
    "    params = {\n",
    "        \"output\": {\n",
    "            \"path\": output,\n",
    "            \"format\": \"parquet\",\n",
    "            \"open_on_complete\": False,\n",
    "            \"region\": \"us-west-2\",\n",
    "            \"credentials\": {\n",
    "                 \"aws_access_key_id\": ACCESS_KEY_ID,\n",
    "                 \"aws_secret_access_key\": SECRET_ACCESS_KEY_ID,\n",
    "                 \"aws_session_token\": SESSION_TOKEN\n",
    "             }\n",
    "        }\n",
    "    }\n",
    "    \n",
    "    # Convert ATL03 granule to geoparquet using the SlideRule service\n",
    "    status = icesat2.atl03s(parm=params, resource=granule, asset=asset)\n",
    "    \n",
    "    return status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "683de35c-9d83-4a46-86b6-5a227a6808bd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time taken for creating Geoparquet via SlideRule: 278.76 seconds\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "# Call the sliderul function\n",
    "result = get_gpq(granule)\n",
    "\n",
    "# End the timer and store the result\n",
    "end_time = time.time()\n",
    "preprocess_time_geoparquet_via_sliderule = end_time - start_time\n",
    "print(f\"Time taken for creating Geoparquet via SlideRule: {preprocess_time_geoparquet_via_sliderule:.2f} seconds\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "480d90af-93a6-423e-8fa2-9b95daeadfb5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://nasa-cryo-scratch/h5cloud/geoparquet/ATL03_20190219140808_08110212_006_02.h5.gpq\n"
     ]
    }
   ],
   "source": [
    "print(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "a9e8a1d5-72bd-437d-9933-0cd6f22eb21c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "ename": "ArrowInvalid",
     "evalue": "Could not open Parquet input source 'nasa-cryo-scratch/h5cloud/geoparquet/ATL03_20190219140808_08110212_006_02.h5.gpq': Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mArrowInvalid\u001b[0m                              Traceback (most recent call last)",
      "Cell \u001b[0;32mIn [28], line 3\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mgeopandas\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mgpd\u001b[39;00m\n\u001b[1;32m      2\u001b[0m \u001b[38;5;66;03m# Make sure it worked\u001b[39;00m\n\u001b[0;32m----> 3\u001b[0m \u001b[43mgpd\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mread_parquet\u001b[49m\u001b[43m(\u001b[49m\u001b[43mresult\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.10/site-packages/geopandas/io/arrow.py:560\u001b[0m, in \u001b[0;36m_read_parquet\u001b[0;34m(path, columns, storage_options, **kwargs)\u001b[0m\n\u001b[1;32m    558\u001b[0m path \u001b[38;5;241m=\u001b[39m _expand_user(path)\n\u001b[1;32m    559\u001b[0m kwargs[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124muse_pandas_metadata\u001b[39m\u001b[38;5;124m\"\u001b[39m] \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mTrue\u001b[39;00m\n\u001b[0;32m--> 560\u001b[0m table \u001b[38;5;241m=\u001b[39m \u001b[43mparquet\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mread_table\u001b[49m\u001b[43m(\u001b[49m\u001b[43mpath\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcolumns\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcolumns\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mfilesystem\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mfilesystem\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    562\u001b[0m \u001b[38;5;66;03m# read metadata separately to get the raw Parquet FileMetaData metadata\u001b[39;00m\n\u001b[1;32m    563\u001b[0m \u001b[38;5;66;03m# (pyarrow doesn't properly exposes those in schema.metadata for files\u001b[39;00m\n\u001b[1;32m    564\u001b[0m \u001b[38;5;66;03m# created by GDAL - https://issues.apache.org/jira/browse/ARROW-16688)\u001b[39;00m\n\u001b[1;32m    565\u001b[0m metadata \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.10/site-packages/pyarrow/parquet/core.py:2939\u001b[0m, in \u001b[0;36mread_table\u001b[0;34m(source, columns, use_threads, metadata, schema, use_pandas_metadata, read_dictionary, memory_map, buffer_size, partitioning, filesystem, filters, use_legacy_dataset, ignore_prefixes, pre_buffer, coerce_int96_timestamp_unit, decryption_properties, thrift_string_size_limit, thrift_container_size_limit)\u001b[0m\n\u001b[1;32m   2932\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m   2933\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mThe \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mmetadata\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m keyword is no longer supported with the new \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m   2934\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdatasets-based implementation. Specify \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m   2935\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124muse_legacy_dataset=True\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m to temporarily recover the old \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m   2936\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mbehaviour.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m   2937\u001b[0m     )\n\u001b[1;32m   2938\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m-> 2939\u001b[0m     dataset \u001b[38;5;241m=\u001b[39m \u001b[43m_ParquetDatasetV2\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m   2940\u001b[0m \u001b[43m        \u001b[49m\u001b[43msource\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   2941\u001b[0m \u001b[43m        \u001b[49m\u001b[43mschema\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mschema\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   2942\u001b[0m \u001b[43m        \u001b[49m\u001b[43mfilesystem\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mfilesystem\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   2943\u001b[0m \u001b[43m        \u001b[49m\u001b[43mpartitioning\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mpartitioning\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   2944\u001b[0m \u001b[43m        \u001b[49m\u001b[43mmemory_map\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmemory_map\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   2945\u001b[0m \u001b[43m        \u001b[49m\u001b[43mread_dictionary\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mread_dictionary\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   2946\u001b[0m \u001b[43m        \u001b[49m\u001b[43mbuffer_size\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mbuffer_size\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   2947\u001b[0m \u001b[43m        \u001b[49m\u001b[43mfilters\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mfilters\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   2948\u001b[0m \u001b[43m        \u001b[49m\u001b[43mignore_prefixes\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mignore_prefixes\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   2949\u001b[0m \u001b[43m        \u001b[49m\u001b[43mpre_buffer\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mpre_buffer\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   2950\u001b[0m \u001b[43m        \u001b[49m\u001b[43mcoerce_int96_timestamp_unit\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcoerce_int96_timestamp_unit\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   2951\u001b[0m \u001b[43m        \u001b[49m\u001b[43mthrift_string_size_limit\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mthrift_string_size_limit\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   2952\u001b[0m \u001b[43m        \u001b[49m\u001b[43mthrift_container_size_limit\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mthrift_container_size_limit\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   2953\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   2954\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mImportError\u001b[39;00m:\n\u001b[1;32m   2955\u001b[0m     \u001b[38;5;66;03m# fall back on ParquetFile for simple cases when pyarrow.dataset\u001b[39;00m\n\u001b[1;32m   2956\u001b[0m     \u001b[38;5;66;03m# module is not available\u001b[39;00m\n\u001b[1;32m   2957\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m filters \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.10/site-packages/pyarrow/parquet/core.py:2479\u001b[0m, in \u001b[0;36m_ParquetDatasetV2.__init__\u001b[0;34m(self, path_or_paths, filesystem, filters, partitioning, read_dictionary, buffer_size, memory_map, ignore_prefixes, pre_buffer, coerce_int96_timestamp_unit, schema, decryption_properties, thrift_string_size_limit, thrift_container_size_limit, **kwargs)\u001b[0m\n\u001b[1;32m   2475\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m single_file \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m   2476\u001b[0m     fragment \u001b[38;5;241m=\u001b[39m parquet_format\u001b[38;5;241m.\u001b[39mmake_fragment(single_file, filesystem)\n\u001b[1;32m   2478\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_dataset \u001b[38;5;241m=\u001b[39m ds\u001b[38;5;241m.\u001b[39mFileSystemDataset(\n\u001b[0;32m-> 2479\u001b[0m         [fragment], schema\u001b[38;5;241m=\u001b[39mschema \u001b[38;5;129;01mor\u001b[39;00m \u001b[43mfragment\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mphysical_schema\u001b[49m,\n\u001b[1;32m   2480\u001b[0m         \u001b[38;5;28mformat\u001b[39m\u001b[38;5;241m=\u001b[39mparquet_format,\n\u001b[1;32m   2481\u001b[0m         filesystem\u001b[38;5;241m=\u001b[39mfragment\u001b[38;5;241m.\u001b[39mfilesystem\n\u001b[1;32m   2482\u001b[0m     )\n\u001b[1;32m   2483\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m\n\u001b[1;32m   2485\u001b[0m \u001b[38;5;66;03m# check partitioning to enable dictionary encoding\u001b[39;00m\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.10/site-packages/pyarrow/_dataset.pyx:1345\u001b[0m, in \u001b[0;36mpyarrow._dataset.Fragment.physical_schema.__get__\u001b[0;34m()\u001b[0m\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.10/site-packages/pyarrow/error.pxi:144\u001b[0m, in \u001b[0;36mpyarrow.lib.pyarrow_internal_check_status\u001b[0;34m()\u001b[0m\n",
      "File \u001b[0;32m/srv/conda/envs/notebook/lib/python3.10/site-packages/pyarrow/error.pxi:100\u001b[0m, in \u001b[0;36mpyarrow.lib.check_status\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mArrowInvalid\u001b[0m: Could not open Parquet input source 'nasa-cryo-scratch/h5cloud/geoparquet/ATL03_20190219140808_08110212_006_02.h5.gpq': Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file."
     ]
    }
   ],
   "source": [
    "import geopandas as gpd\n",
    "# Make sure it worked\n",
    "gpd.read_parquet(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d934acb1-1c2c-4d30-bb3d-9448fac9894a",
   "metadata": {},
   "source": [
    "## Repacking h5 dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "f8661cf6-9a92-435b-b262-12c5162665ef",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error occurred while repacking\n",
      "Time taken to repack: 0.05 seconds\n"
     ]
    }
   ],
   "source": [
    "import subprocess\n",
    "\n",
    "def repack_hdf5(input_file, options=[]):\n",
    "    cmd = [\"h5repack\"] + options + [input_file, output_file]\n",
    "    subprocess.run(cmd)\n",
    "    \n",
    "# Start the timer\n",
    "start_time = time.time()\n",
    "\n",
    "# Usage\n",
    "input_file = \"s3://nasa-cryo-scratch/h5cloud/original/ATL03_20200217204710_08110612_006_01.h5\"\n",
    "output_file = \"h5repacked.h5\"\n",
    "repack_hdf5(input_file, options=[\"-S\", \"PAGE\", \"-G\", \"8000000\"])\n",
    "\n",
    "# End the timer and store the result\n",
    "end_time = time.time()\n",
    "preprocess_time_repacking = end_time - start_time\n",
    "print(f\"Time taken to repack: {preprocess_time_repacking:.2f} seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5ffb903-5d00-4ac0-81db-a74c82e4716b",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "## Generating Kerchunk sidecar h5repacked data\n",
    "\n",
    "In this sub-section, we create a Kerchunk sidecar metadata file specifically for an HDF5 file that has undergone the repacking process. This combination allows for the benefits of both techniques:\n",
    "\n",
    "* Efficient Data Structure: Thanks to h5repacking, the HDF5 file itself is optimized for faster read operations.\n",
    "* Optimized Access: The Kerchunk sidecar allows for chunked, efficient access, especially beneficial in cloud-based workflows. It means applications can pull specific chunks of data without downloading or reading the entire file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "216b7b5e-c2ea-48a3-ae5b-75c492d12b41",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Initialize connection to the AWS S3 filesystem for reading files.\n",
    "fs_read_files = fsspec.filesystem('s3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "00c25fdc-4d8c-472f-a611-fa95b231a0a9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "h5repack_data = 's3://nasa-cryo-scratch/h5cloud/h5repack/ATL03_20200217204710_08110612_006_01_repacked.h5'\n",
    "\n",
    "# file formats or files to process\n",
    "file_url = h5repack_data\n",
    "\n",
    "\n",
    "# Start the timer\n",
    "start_time = time.time()\n",
    "\n",
    "# Call the preprocessing function\n",
    "kerchunked_json = gen_json(file_url)\n",
    "\n",
    "# End the timer and store the result\n",
    "end_time = time.time()\n",
    "preprocess_time_kerchunk_repacked = end_time - start_time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "b99dd4d5-f504-49a7-92fb-9a6994d7d149",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time taken for generating a Kerchunk sidecar for an existing repacked file: 760.89 seconds\n"
     ]
    }
   ],
   "source": [
    "print(f\"Time taken for generating a Kerchunk sidecar for an existing repacked file: {preprocess_time_kerchunk_repacked:.2f} seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18582b26-17ef-4c50-8c8f-f678810af7f6",
   "metadata": {
    "tags": [],
    "user_expressions": []
   },
   "source": [
    "# Comparing times\n",
    "\n",
    "In this section, we combine all the preprocessing times into 1 dataframe to compare. Note that if considering a kerchunk sidecar for repacked HDF5 files, you would need to sum the preprocessing time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "97f137b7-b27e-477f-bf80-4839966c579c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "all_times = {\n",
    "    \"Kerchunk\": preprocess_time_kerchunk,\n",
    "    \"Repacking\": preprocess_time_repacking,\n",
    "    \"Kerchunk of Repacked\": preprocess_time_kerchunk_repacked,\n",
    "    \"Geoparquet\": preprocess_time_geoparquet_via_sliderule,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "8cd31bd5-6ebd-4b99-8b41-5e3688eeac5b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Kerchunk': 21.518,\n",
       " 'Repacking': 0.001,\n",
       " 'Kerchunk of Repacked': 12.682,\n",
       " 'Geoparquet': 4.646}"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_times_in_minutes = {}\n",
    "for key in all_times.keys():\n",
    "    all_times_in_minutes[key] = round(all_times[key]/60, 3)\n",
    "    \n",
    "all_times_in_minutes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "10734bd4-7629-4d2f-a4be-9aa04d0e9f2d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Kerchunk</th>\n",
       "      <th>Repacking</th>\n",
       "      <th>Kerchunk of Repacked</th>\n",
       "      <th>Geoparquet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>21.518</td>\n",
       "      <td>0.001</td>\n",
       "      <td>12.682</td>\n",
       "      <td>4.646</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Kerchunk  Repacking  Kerchunk of Repacked  Geoparquet\n",
       "0    21.518      0.001                12.682       4.646"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "all_times_df = pd.DataFrame(all_times_in_minutes, index=[0])\n",
    "all_times_df"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}