docs/tutorials/bigtable.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tce3stUlHN0L"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "tuOe1ymfHZPu"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qFdPvlXBOdUN"
      },
      "source": [
        "# Title"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MfBg1C5NB3X0"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/io/tutorials/bigtable\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/io/blob/master/docs/tutorials/bigtable.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/io/blob/master/docs/tutorials/bigtable.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View on GitHub</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/io/docs/tutorials/bigtable.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xHxb-dlhMIzW"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook represents the basic usage and features of the `tensorflow_io.bigtable` module. Make sure you are familiar with these topics before continuing:\n",
        "\n",
        "\n",
        "1. [Creating a GCP project.](https://console.cloud.google.com/cloud-resource-manager)\n",
        "2. [Installing the Cloud SDK for Bigtable](https://cloud.google.com/bigtable/docs/installing-cloud-sdk)\n",
        "3. [cbt tool overview](https://cloud.google.com/bigtable/docs/cbt-overview)\n",
        "4. [Using the emulator](https://cloud.google.com/bigtable/docs/emulator)\n",
        "\n",
        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MUXex9ctTuDB"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eOkl2R_V3wPW"
      },
      "outputs": [],
      "source": [
        "!pip install tensorflow-io"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Eh9oG2-2bwMn"
      },
      "source": [
        "**Note**: When executing the cell below, you will be asked to log in to google cloud."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-ZPPv_W9diZH"
      },
      "outputs": [],
      "source": [
        "!mkdir /tools/google-cloud-sdk/.install\n",
        "!gcloud --quiet components install beta cbt bigtable\n",
        "!gcloud init"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ACA47DqRcgt6"
      },
      "source": [
        "For the sake of this example, the bigtable emulator is used. If you have your bigtable instance set up and populated with values, skip these steps and go straight to the Quickstart section."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gMaPTsDzc5am"
      },
      "source": [
        "Start the emulator in the background."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6-tVA2lDg_H_"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import subprocess\n",
        "_emulator = subprocess.Popen(['/tools/google-cloud-sdk/bin/gcloud', 'beta', 'emulators', 'bigtable', 'start', '--host-port=127.0.0.1:8086'],\n",
        "                                      stdout=subprocess.DEVNULL,\n",
        "                                      stderr=subprocess.DEVNULL, bufsize=0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zec076IBc9Qi"
      },
      "source": [
        "Create a table"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B005n7wogepa"
      },
      "outputs": [],
      "source": [
        "%env BIGTABLE_EMULATOR_HOST=127.0.0.1:8086\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" createtable t1 families=cf1 splits=row-a,row-h,row-p,row-z\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" ls\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rfHIXohSdEuk"
      },
      "source": [
        "Populate table with values"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "84jqZk8JmZ9w"
      },
      "outputs": [],
      "source": [
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-a cf1:c1=A\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-b cf1:c1=B\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-c cf1:c1=C\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-d cf1:c1=D\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-e cf1:c1=E\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-f cf1:c1=F\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-g cf1:c1=G\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-h cf1:c1=H\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-i cf1:c1=I\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-j cf1:c1=J\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-k cf1:c1=K\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-l cf1:c1=L\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-m cf1:c1=M\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-n cf1:c1=N\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-o cf1:c1=O\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-p cf1:c1=P\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-q cf1:c1=Q\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-r cf1:c1=R\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-s cf1:c1=S\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-t cf1:c1=T\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-u cf1:c1=U\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-v cf1:c1=V\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-w cf1:c1=W\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-x cf1:c1=X\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-y cf1:c1=Y\n",
        "!cbt -project \"test-project\" -instance \"test-instance\" set t1 row-z cf1:c1=Z"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IqR2PQG4ZaZ0"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "import tensorflow_io as tfio\n",
        "import random\n",
        "\n",
        "random.seed(10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K1u49nEJYXwD"
      },
      "source": [
        "## Quickstart"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JzikHwm3jEns"
      },
      "source": [
        "First you need to create a client and a table you would like to read from."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iKaxP3ZGTDOd"
      },
      "outputs": [],
      "source": [
        "# If using your bigtable instance replace the project_id, instance_id \n",
        "# and the name of the table with suitable values.\n",
        "\n",
        "client = tfio.bigtable.BigtableClient(project_id=\"test-project\", instance_id=\"test-instance\")\n",
        "train_table = client.get_table(\"t1\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o2O_TOoSjvFi"
      },
      "source": [
        "Great! Now you can create a tensorflow dataset that will read the data from our\n",
        "table. \n",
        "\n",
        "To do that, you have to provide the type of the data you wish to read,\n",
        "list of column names in format `column_family:column_name`, and a row_set that\n",
        "you would like to read. \n",
        "\n",
        "To create a row_set use utility methods provided in `tensorflow.bigtable.row_set` and `tensorflow.bigtable.row_range` modules. Here a row_set containing all rows is created.\n",
        "\n",
        "Keep in mind that that bigtable reads values in lexicographical order, not the order they were put in. The rows were given random row-keys so they will be shuffled."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KlEnassJjivK"
      },
      "outputs": [],
      "source": [
        "row_set = tfio.bigtable.row_set.from_rows_or_ranges(tfio.bigtable.row_range.infinite())\n",
        "\n",
        "train_dataset = train_table.read_rows([\"cf1:c1\"],row_set, output_type=tf.string)\n",
        "\n",
        "for tensor in train_dataset:\n",
        "  print(tensor)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TacgaIGAkDNQ"
      },
      "source": [
        "That's it! Congrats!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yBUdy_20lA69"
      },
      "source": [
        "## Parallel read\n",
        "\n",
        "Our dataset supports reading in parallel from Bigtable. To do that, use the `parallel_read_rows` method and specify `num_parallel_calls` as an argument. When this method is called work is first split between workers based SampleRowKeys.\n",
        "\n",
        "**Note**: Keep in mind that when reading in parallel, the rows are not\n",
        "going to be read in any particular order."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7I4qWtE_lCZq"
      },
      "outputs": [],
      "source": [
        "for tensor in train_table.parallel_read_rows([\"cf1:c1\"],row_set=row_set, num_parallel_calls=2):\n",
        "  print(tensor)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SVvpg_wbnhX3"
      },
      "source": [
        "## Reading specific row_keys\n",
        "\n",
        "To read the data from Bigtable, you can specify a set of rows or a range or a\n",
        "combination of those.\n",
        "\n",
        "`read_rows` method expects you to provide a\n",
        "RowSet. You can construct a RowSet from specific row keys or RowRanges as follows:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Xu7L_YxClxFt"
      },
      "outputs": [],
      "source": [
        "row_range_below_300 = tfio.bigtable.row_range.right_open(\"row000\", \"row300\")\n",
        "\n",
        "my_row_set = tfio.bigtable.row_set.from_rows_or_ranges(row_range_below_300, \"row585\", \"row832\")\n",
        "print(my_row_set)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IfuTAF9Woxog"
      },
      "source": [
        "such row_set would contain a range of rows `[row000, row300)` and rows row585 and row832.\n",
        "\n",
        "you can also create a row_set from an infinite range, empty range or a prefix.\n",
        "You can also intersect it with a row_range."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IiAHxmmOow6R"
      },
      "outputs": [],
      "source": [
        "my_truncated_row_set = tfio.bigtable.row_set.intersect(my_row_set,\n",
        "                                         tfio.bigtable.row_range.right_open(\"row200\", \"row700\"))\n",
        "print(my_truncated_row_set)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z_t6FlohrkYk"
      },
      "source": [
        "## Specifying a version of a value\n",
        "\n",
        "Bigtable lets you keep many values in one cell with different timestamps. You\n",
        "can specify which version you want to pick using version filters. However, you\n",
        "can only retrieve a two dimensional vector using tensorflow.bigtable connector, so `latest` filter is always appended to the user specified version filter.\n",
        "Meaning, if more than one value for one cell goes through the provided filter,\n",
        "the newer shall be used.\n",
        "\n",
        "You can either use the `latest` filter passing the newest value, or you can\n",
        "specify a time range. The time range can be provided either as python datetime\n",
        "objects or a number representing seconds or microseconds since epoch."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "w21daM9IrIHa"
      },
      "outputs": [],
      "source": [
        "from datetime import datetime\n",
        "\n",
        "start = datetime(2020, 10, 10, 12, 0, 0)\n",
        "end = datetime(2100, 10, 10, 13, 0, 0)\n",
        "from_datetime = tfio.bigtable.filters.timestamp_range(start, end)\n",
        "from_posix_timestamp = tfio.bigtable.filters.timestamp_range(int(start.timestamp()), int(end.timestamp()))\n",
        "print(\"from_datetime:\", from_datetime)\n",
        "\n",
        "print(\"from_posix_timestamp:\", from_posix_timestamp)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "Tce3stUlHN0L"
      ],
      "name": "bigtable.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}