Microsoft19thCenturyBooks/load_JSON_files.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "colab": {
      "name": "load_JSON_files.ipynb",
      "provenance": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qH0ZKoc1c7SZ",
        "colab_type": "text"
      },
      "source": [
        "<div class=\"jumbotron jumbotron-fluid\">\n",
        "  <div class=\"container\">\n",
        "    <h1 class=\"display-4\">Load JSON files from a directory (and sub-directories)</h1>\n",
        "    <p class=\"lead\">Examples on how to load the full text from JSON files stored in a certain local directory (and sub-directories) of from a file downloaded from a URL.</p>\n",
        "  </div>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eRU6X0pqc7Sc",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/BL-Labs/Jupyter-notebooks-projects-using-BL-Sources/blob/master/Microsoft19thCenturyBooks/load_JSON_files.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oF2iCADMc7Se",
        "colab_type": "text"
      },
      "source": [
        "### In this Notebook:\n",
        "\n",
        " * Read JSON files containg full text: from local zip file or downloaded from a URL;\n",
        " * Loaded the contents of the several JSON files, even in several sub-directories, into a big DataFrame;\n",
        " * Add ALEPH SYS ID of the book (\"bookid\") and the Volume information extracted from the filename of each JSON file;\n",
        " * Exclude blocks of text that are empty;\n",
        " * Search for block of text that mention \"London\";\n",
        " * From those, search the ones who have the term \"love\";\n",
        " * Display a couple of those block of texts;\n",
        " * Retrieve the BookID and Volume to which those blocks of text belong to;\n",
        " * Link to their catalog record on BL Explorer.\n",
        "\n",
        "    \n",
        "#### Notes:\n",
        "\n",
        "    * If using BINDER or a did a git clone to your Jupyter Notebook LOCAL SERVER, if you want to clean the output from the previous run, from the main menu, choose \"Kernel\" > \"Restart and Clear Output\"; if using Google Colab, please open instead the mactching \"not_run\" notebook;\n",
        "    * If using BINDER or a did a git clone to your Jupyter Notebook LOCAL SERVER, skip cell 1, place the cursor on cell 2 and from the main menu, choose \"Cell\" > \"Run All Bellow\", if not executing cell-by-cell;\n",
        "    * If using Google Colab, with the cursor on the first cell, from the main menu, choose \"Runtime\" > \"Run after\", if not executing cell-by-cell.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H9IurVIjc7Sf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\"\"\"\n",
        "\n",
        "Only RUN this CELL if using Google COLAB\n",
        "To download the zip file (295MB), it could take from 10 to 30 minutes --- Not needed if using BINDER or a did a git clone to your Jupyter Notebook LOCAL SERVER, as you already have the data locally\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "!wget https://cld.pt/dl/download/b2d718a4-2aa2-496c-aad8-d889f27a4be2/dig19cbooksjsontext_sample.zip\n",
        "\n",
        "# Only RUN THE UNZIP ONCE, please\n",
        "!unzip dig19cbooksjsontext_sample.zip \n",
        "\n",
        "!mkdir data\n",
        "!mv dig19cbooksjsontext/ data/dig19cbooksjsontext/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pIKz0TOkc7Sp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Load the necessary modules / Libraries:\n",
        "\n",
        "import os, json\n",
        "import pandas as pd"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PJgg0PdEc7Sv",
        "colab_type": "text"
      },
      "source": [
        "### Let's check the files in this Directory:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2ZLKZh_rc7Sx",
        "colab_type": "code",
        "colab": {},
        "outputId": "34930d70-d6cc-41ca-e81b-344c634d8011"
      },
      "source": [
        "\"\"\"\n",
        "\n",
        "If all the files are under the same directory a simpler way to locate them would be:\n",
        "path_to_json = 'data/dig19cbooksjsontext/json/0037/'\n",
        "json_files = [pos_json for pos_json in os.listdir(path_to_json) if pos_json.endswith('.json')]\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "#this will locate all the JSON files inside the main Directory and any sub-Folder:\n",
        "path_to_json = 'data/dig19cbooksjsontext/'\n",
        "\n",
        "json_files = [os.path.join(root, name)\n",
        "             for root, dirs, files in os.walk(path_to_json)\n",
        "             for name in files\n",
        "             if name.endswith((\".json\"))] #If we needed to read several files extensions: if name.endswith((\".ext1\", \".ext2\"))\n",
        "\n",
        "print('Number of JSON files ready to be loaded: ' + str(len(json_files)))\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of JSON files ready to be loaded: 1080\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5Kyl9BxHc7S4",
        "colab_type": "code",
        "colab": {},
        "outputId": "0dac053b-f930-44d9-fff2-a80587b22cc6"
      },
      "source": [
        "print('Path to the first file: '+json_files[0])\n",
        "print('Path to the last file: '+json_files[len(json_files)-1])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Path to the first file: data/dig19cbooksjsontext/json/0050/005009643_01_text.json\n",
            "Path to the last file: data/dig19cbooksjsontext/json/0118/011836203_01_text.json\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1U4FzZiZc7S_",
        "colab_type": "text"
      },
      "source": [
        "### Files are ready to be loaded into a big DataFrame\n",
        "We will add ALEPH SYS ID of the book (\"bookid\") and the Volume information extracted from the filename of the JSON file.\n",
        "\n",
        "* A similar process can be used to append all the data into a single JSON, instead of loading the data from the JSON files into a DataFrame."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QGqTmV9vc7TA",
        "colab_type": "code",
        "colab": {},
        "outputId": "1e985c84-6046-48f1-bd6f-cae1dcf9e4c8"
      },
      "source": [
        "jsons_df = pd.DataFrame()\n",
        "#jsons_df = pd.DataFrame(columns=['indexglob', 'indexlocal', 'text','bookid','volume'])\n",
        "\n",
        "for index, js in enumerate(json_files):\n",
        "    jsons_df_ins = pd.read_json(json_files[index])  #if we were just loading from one Directory, first solution above, we would need to add the path to the file: jsons_df_ins = pd.read_json(path_to_json+json_files[index])\n",
        "    jsons_df_ins[\"bookid\"] = json_files[index][35:44]\n",
        "    jsons_df_ins[\"volume\"] = json_files[index][45:47]\n",
        "    jsons_df = jsons_df.append(jsons_df_ins, ignore_index=True)\n",
        "\n",
        "\n",
        "print('Blocks of text loaded into the DataFrame: ' + str(jsons_df.size))\n",
        "    "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Blocks of text loaded into the DataFrame: 1587840\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TwZD3Aouc7TH",
        "colab_type": "code",
        "colab": {},
        "outputId": "ce6caa35-8e40-4b57-cee7-fc068d3b8c45"
      },
      "source": [
        "print('First 20 blocks of text:')\n",
        "jsons_df.head(20)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "First 20 blocks of text:\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>bookid</th>\n",
              "      <th>volume</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td></td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td></td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td></td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td></td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>REPORT ONTHB GEOLOGY, MINERALOGY, BOTANY, AND ...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>6</td>\n",
              "      <td>Entered according to Act of Congress, in the y...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>7</td>\n",
              "      <td>To His Excellency, John davis, Esq.. Governor ...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>8</td>\n",
              "      <td></td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>9</td>\n",
              "      <td>INTRODUCTORY OR HISTORICAL NOTE. On the 3d of ...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>10</td>\n",
              "      <td>INTRODUCTION. IV Commonwealth, in connection w...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>11</td>\n",
              "      <td>INTRODUCTION. V such way and manner as he shal...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>12</td>\n",
              "      <td>VI INTRODUCTION. ' Resolved, that the said fiv...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>13</td>\n",
              "      <td>CONTENTS. PART I. ECONOMICAL GEOLOGY. Explanat...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>14</td>\n",
              "      <td>CONTENTS VI II METALS AND THEIR ORES. Iron, • ...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>15</td>\n",
              "      <td>CONTENTS IX Fall River: Pawtucket Falls : The ...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>16</td>\n",
              "      <td>X CONTENTS. t \\ S7 Organic Remains, - j°' Theo...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>17</td>\n",
              "      <td>contents X! Mineral Contents, - . . 340 Theory...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>18</td>\n",
              "      <td>XII contents Mineral Contents, _-..-- Theory o...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>19</td>\n",
              "      <td>REPORT PART I ECONOMICAL GEOLOGY. To His Excel...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>20</td>\n",
              "      <td>Economical Geology. 14 exhibits all that is im...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     0                                                  1     bookid volume\n",
              "0    1                                                     005009643     01\n",
              "1    2                                                     005009643     01\n",
              "2    3                                                     005009643     01\n",
              "3    4                                                     005009643     01\n",
              "4    5  REPORT ONTHB GEOLOGY, MINERALOGY, BOTANY, AND ...  005009643     01\n",
              "5    6  Entered according to Act of Congress, in the y...  005009643     01\n",
              "6    7  To His Excellency, John davis, Esq.. Governor ...  005009643     01\n",
              "7    8                                                     005009643     01\n",
              "8    9  INTRODUCTORY OR HISTORICAL NOTE. On the 3d of ...  005009643     01\n",
              "9   10  INTRODUCTION. IV Commonwealth, in connection w...  005009643     01\n",
              "10  11  INTRODUCTION. V such way and manner as he shal...  005009643     01\n",
              "11  12  VI INTRODUCTION. ' Resolved, that the said fiv...  005009643     01\n",
              "12  13  CONTENTS. PART I. ECONOMICAL GEOLOGY. Explanat...  005009643     01\n",
              "13  14  CONTENTS VI II METALS AND THEIR ORES. Iron, • ...  005009643     01\n",
              "14  15  CONTENTS IX Fall River: Pawtucket Falls : The ...  005009643     01\n",
              "15  16  X CONTENTS. t \\ S7 Organic Remains, - j°' Theo...  005009643     01\n",
              "16  17  contents X! Mineral Contents, - . . 340 Theory...  005009643     01\n",
              "17  18  XII contents Mineral Contents, _-..-- Theory o...  005009643     01\n",
              "18  19  REPORT PART I ECONOMICAL GEOLOGY. To His Excel...  005009643     01\n",
              "19  20  Economical Geology. 14 exhibits all that is im...  005009643     01"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eLdIgW06c7TP",
        "colab_type": "text"
      },
      "source": [
        "### Given that some blocks are empty, let's create a new DataFrame with only the blocks that have Text"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hk4dtHCIc7TQ",
        "colab_type": "code",
        "colab": {},
        "outputId": "4624caef-a0a0-4320-89bc-735d2c9e628c"
      },
      "source": [
        "#Drop blocks of text that are empty\n",
        "\n",
        "filter = jsons_df[1] != \"\"\n",
        "blocksText_df = jsons_df[filter]\n",
        "print('Blocks of text non empty: ' + str(blocksText_df.size))\n",
        "print('Excluded ' + str(jsons_df.size - blocksText_df.size) + ' that are empty')\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Blocks of text non empty: 1504596\n",
            "Excluded 83244 that are empty\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "enWUKLq-c7TX",
        "colab_type": "code",
        "colab": {},
        "outputId": "325aadcd-b15e-403b-a007-45a899de15ab"
      },
      "source": [
        "blocksText_df.head(20)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>bookid</th>\n",
              "      <th>volume</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>REPORT ONTHB GEOLOGY, MINERALOGY, BOTANY, AND ...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>6</td>\n",
              "      <td>Entered according to Act of Congress, in the y...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>7</td>\n",
              "      <td>To His Excellency, John davis, Esq.. Governor ...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>9</td>\n",
              "      <td>INTRODUCTORY OR HISTORICAL NOTE. On the 3d of ...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>10</td>\n",
              "      <td>INTRODUCTION. IV Commonwealth, in connection w...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>11</td>\n",
              "      <td>INTRODUCTION. V such way and manner as he shal...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>12</td>\n",
              "      <td>VI INTRODUCTION. ' Resolved, that the said fiv...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>13</td>\n",
              "      <td>CONTENTS. PART I. ECONOMICAL GEOLOGY. Explanat...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>14</td>\n",
              "      <td>CONTENTS VI II METALS AND THEIR ORES. Iron, • ...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>15</td>\n",
              "      <td>CONTENTS IX Fall River: Pawtucket Falls : The ...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>16</td>\n",
              "      <td>X CONTENTS. t \\ S7 Organic Remains, - j°' Theo...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>17</td>\n",
              "      <td>contents X! Mineral Contents, - . . 340 Theory...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>18</td>\n",
              "      <td>XII contents Mineral Contents, _-..-- Theory o...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>19</td>\n",
              "      <td>REPORT PART I ECONOMICAL GEOLOGY. To His Excel...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>20</td>\n",
              "      <td>Economical Geology. 14 exhibits all that is im...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>21</td>\n",
              "      <td>Division of the Report. 15 with any confidence...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>22</td>\n",
              "      <td>Economical Geology. 16 have endeavored to obta...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>23</td>\n",
              "      <td>Alluvium. 17 Alluvium. In this part of my Repo...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>24</td>\n",
              "      <td>Economical Geology. 18 Diluvium. This occupies...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>25</td>\n",
              "      <td>Tertiary Formations. 19 found any rocks that d...</td>\n",
              "      <td>005009643</td>\n",
              "      <td>01</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     0                                                  1     bookid volume\n",
              "4    5  REPORT ONTHB GEOLOGY, MINERALOGY, BOTANY, AND ...  005009643     01\n",
              "5    6  Entered according to Act of Congress, in the y...  005009643     01\n",
              "6    7  To His Excellency, John davis, Esq.. Governor ...  005009643     01\n",
              "8    9  INTRODUCTORY OR HISTORICAL NOTE. On the 3d of ...  005009643     01\n",
              "9   10  INTRODUCTION. IV Commonwealth, in connection w...  005009643     01\n",
              "10  11  INTRODUCTION. V such way and manner as he shal...  005009643     01\n",
              "11  12  VI INTRODUCTION. ' Resolved, that the said fiv...  005009643     01\n",
              "12  13  CONTENTS. PART I. ECONOMICAL GEOLOGY. Explanat...  005009643     01\n",
              "13  14  CONTENTS VI II METALS AND THEIR ORES. Iron, • ...  005009643     01\n",
              "14  15  CONTENTS IX Fall River: Pawtucket Falls : The ...  005009643     01\n",
              "15  16  X CONTENTS. t \\ S7 Organic Remains, - j°' Theo...  005009643     01\n",
              "16  17  contents X! Mineral Contents, - . . 340 Theory...  005009643     01\n",
              "17  18  XII contents Mineral Contents, _-..-- Theory o...  005009643     01\n",
              "18  19  REPORT PART I ECONOMICAL GEOLOGY. To His Excel...  005009643     01\n",
              "19  20  Economical Geology. 14 exhibits all that is im...  005009643     01\n",
              "20  21  Division of the Report. 15 with any confidence...  005009643     01\n",
              "21  22  Economical Geology. 16 have endeavored to obta...  005009643     01\n",
              "22  23  Alluvium. 17 Alluvium. In this part of my Repo...  005009643     01\n",
              "23  24  Economical Geology. 18 Diluvium. This occupies...  005009643     01\n",
              "24  25  Tertiary Formations. 19 found any rocks that d...  005009643     01"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GT5rpgYSc7Te",
        "colab_type": "text"
      },
      "source": [
        "### And now we can do whatever needed with this DataFrame\n",
        "\n",
        "#### E.g., let's look for blocks of text that mention London:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jzNnKElHc7Tf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "London_df = blocksText_df[blocksText_df[1].str.contains(\"London\")]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H4q7M44Qc7Tl",
        "colab_type": "code",
        "colab": {},
        "outputId": "c0c8f32b-ff6e-4355-a7f1-05459ddb1587"
      },
      "source": [
        "print(London_df)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "          0                                                  1     bookid  \\\n",
            "176     177  Origin of Diluvium. 171 ists,- that the land w...  005009643   \n",
            "183     184  178 Scientific Geology. Clay. I am not certain...  005009643   \n",
            "197     198  192 Scientific Geology. The conglomerate occup...  005009643   \n",
            "205     206  Scientific Geology. 200 duction of the green s...  005009643   \n",
            "207     208  202 Scientific Geology. with amber, and the in...  005009643   \n",
            "...     ...                                                ...        ...   \n",
            "395727   90  80 KAIRA : A DISTRICT IN FERTILE GUJARAT. No. ...  011834222   \n",
            "395777   16  IV PREFACE. Island of Madagascar, I prepared a...  011844350   \n",
            "396058    5  LIFE' IN THE MOFUSSIL; OR, THE CIVILIAN IN LOW...  011837920   \n",
            "396153  100  Life in the Mofussil. 92 considered the new pr...  011837920   \n",
            "396906  437  421 APPENDIX H tinge is much less extensive an...  011836203   \n",
            "\n",
            "       volume  \n",
            "176        01  \n",
            "183        01  \n",
            "197        01  \n",
            "205        01  \n",
            "207        01  \n",
            "...       ...  \n",
            "395727     01  \n",
            "395777     01  \n",
            "396058     01  \n",
            "396153     01  \n",
            "396906     01  \n",
            "\n",
            "[17674 rows x 4 columns]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aI7PNgCrc7Ts",
        "colab_type": "text"
      },
      "source": [
        "#### ... and of those, have the term \"love\":"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0GfWWKaLc7Tt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "London_love_df = London_df[London_df[1].str.contains(\"love\")]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lp1qYtQEc7Tz",
        "colab_type": "code",
        "colab": {},
        "outputId": "f20a037e-6111-472c-d14b-c6c7bdbe459c"
      },
      "source": [
        "print(London_love_df)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "          0                                                  1     bookid  \\\n",
            "4002      3  THE VINCENTS «coo» ■ - N the corner of Middles...  003798471   \n",
            "4356    341  CHATTO &> WIND US. PICCADILLY. 9 Faraday (Mich...  003761409   \n",
            "6132      9  CONTENTS. PAGE. Chapter the First. — The Metro...  003794928   \n",
            "6137     14  LONDON AND ITS ENVIRONS. 2 Till the last centu...  003794928   \n",
            "6153     30  LONDON AND ITS ENVIRONS. 18 III. THE ANCIENT C...  003794928   \n",
            "...     ...                                                ...        ...   \n",
            "393912  135  111 STREET COMMERCE. PIC-NICS. dency has to wr...  010878173   \n",
            "394022  245  LUCKNOW. 221 articles recommend them to the pu...  010878173   \n",
            "394788  291  A TRIP TO CASHMERE AND LADAK. By Cowley Lamber...  011837920   \n",
            "395539  214  194 ASHE PYEE. by an extraordinary helmet, 'so...  011833856   \n",
            "395564  239  219 MISCELLANEOUS RECORD. route to the amount ...  011833856   \n",
            "\n",
            "       volume  \n",
            "4002       01  \n",
            "4356       01  \n",
            "6132       01  \n",
            "6137       01  \n",
            "6153       01  \n",
            "...       ...  \n",
            "393912     01  \n",
            "394022     01  \n",
            "394788     02  \n",
            "395539     01  \n",
            "395564     01  \n",
            "\n",
            "[1261 rows x 4 columns]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ls6IMrmUc7T4",
        "colab_type": "text"
      },
      "source": [
        "#### Let's check one of the records, ID = 6132 (3rd in the list)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HSogRs44c7T5",
        "colab_type": "code",
        "colab": {},
        "outputId": "437619ed-745d-4c3b-cbce-e132cd2cc829"
      },
      "source": [
        "London_love_df[1][6153]\n",
        "\n",
        "# if using Google COLAB, comment the above and uncoment the bellow, please\n",
        "# London_love_df[1][186]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'LONDON AND ITS ENVIRONS. 18 III. THE ANCIENT CITY. REMAINS OF ROMAN LONDON. THE LONDON STONE. THE TOWER. — QUEEN VICTORIA\\'S KEYS. — ST. JOHN\\'S GATE. SHAKESPEARE\\'S HOUSE. The almost entire absence of ancient monuments in a city, the origin of wliich is so antiquated, is not one of the least surprises which London presents to the stranger. One fancies usually that the old \"City\" encloses a number of old (secular) edifices; that in those dark streets and obscure passages called \"lanes,\" new buildings would be the exception, not the rule. But this is an error. The dark houses which border the City ways are relatively modern, for, notwithstanding the respect whicli the English profess for all that savours of antiquity ; notwithstanding their love for old customs and ancient traditions (the word \"old\" is in English a term of affection), they know how to make allowance for increasing necessities and new wants, and sacrifice — not without hesitation, it is true, but nevertheless with determination — -all the ancient monuments which, useless or in the way, impede the development of their metropolis. It is for this reason tbat the picturesque houses, which formerly were visible in the City, have almost entirely disappeared, and are now replaced by massive edifices, in which iron, stone, and marble, are used profusely ; and while thus beautifying the City, the aspect of which is changing day by day, the Londoners are protecting it against destruction by fire, of which the risks in London are so great. As a matter of fact, at all times the City has had to contend against this formidable foe. Seven times before the terrible catastrophe of 1666, whicli destroyed 13,000 houses and 87 churches, the City has been the prey of the flames. One would have fancied that after such severe lessons, the civic authorities, rendered wiser by experience, would have taken measures to prevent the recurrence of similar calamities. But nothing came of it, and it is only within the last few years that the corporation has taken energetic measures, and given the impulse thanks to which the City is losing its picturesqueness, but is gaining in salubrity — and, let us confess it, in beauty. At the present time, when the pickaxe of the bouse demolisher is finishing the work begun by fire, there remain but a few scarce vestiges of ancient London, known ouly to archæologists and antiquaries, who deplore the approaching disappearance of oven these remains. Let us pass them quickly in review, and salute in these old ruins a past time of blood and barbarism, but a period not without glory and grandeur as well. Of the Roman enceinte, built in the year 306, there remain but a few fragments in the street called London Wall, near St. Alphage\\'s Church, in Bloomfield Street, Finsbury, and in St. Martin\\'s Court, on the South side of Ludgate Hill. Another very interesting relic, also dating from the Roman period, is the London Stone encased in the wall of St. Swithin\\'s Church, opposite Cannon Street Railway Station. All opinions agree in recognizing this block of granite as a \" mile-stone,\" placed by the Romans in the centre of the town, like that in the Forum at Rome, from which all distances were'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h471rShIc7T-",
        "colab_type": "text"
      },
      "source": [
        "(...) REMAINS OF ROMAN **LONDON**. THE **LONDON** STONE.... notwithstanding their **love** for old customs and ancient traditions ... \n",
        "\n",
        "or if using Google Colab example:\n",
        "\n",
        "(...) I am come down to take her to **London**, where (...) as the only true **love**r and friend she had (...)\n",
        "\n",
        "#### Retrieve the BookID and Volume to which this block of text belongs:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kZ_gDuA6c7T_",
        "colab_type": "code",
        "colab": {},
        "outputId": "5ecb1706-486c-4c22-d254-697b129a9503"
      },
      "source": [
        "London_love_df.loc[6153, ['bookid', 'volume']]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "bookid    003794928\n",
              "volume           01\n",
              "Name: 6153, dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mO8BxGZ7c7UF",
        "colab_type": "text"
      },
      "source": [
        "#### We can also check a random one (a sample of 1):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZMVD4DMrc7UG",
        "colab_type": "code",
        "colab": {},
        "outputId": "9e7c798b-2ce1-4bcf-9e38-aaa2329746c6"
      },
      "source": [
        "London_love_df[1].sample(1)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "386455    CHESTER AVARD.— AV. D. 222 Greencroft. This to...\n",
              "Name: 1, dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lkf7iZxjc7UN",
        "colab_type": "code",
        "colab": {},
        "outputId": "02fdcff6-d768-4425-8516-43cd7e8993e7"
      },
      "source": [
        "London_love_df[1][347374] # use the index, first number retrieved above -- when ran this time, it gave the index 347374"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"OF THE COUNTY OF DERBY. 351 gators in an undertaking, the object of which was the just enforcement of the laws. It gave him a severe pang, when he beheld among his judges, Valence, earl of Pembroke, , who had urged him, in the former insurrection against Piers de Gaveston, the king's former favourite to support by his rank and power the cause ofthe barons and ofthe people, against the tyranny and avarice of the king and his ministers. The tribunal, before which he stood, was not long in pronouncing his fate. He was sentenced to be drawn, hanged and quartered ; but the king, in consideration of his being so nearly related to himself, consented that he should be beheaded, and that the other practices attending the execution of a person condemned as a traitor should be omitted. He suf fered on the 23rd of March, 1322 ; and many of his followers perished also on the scaffold and the gibbet. — Few men of his elevated rank were, at that period, so loved and reverenced by the people. His grave was thronged by multitudes, who believed that miracles were wrought by his remains, and the hill where he suffered was the constant resort of thousands, who could not be kept away, until their devotion to his memory was in some degree satisfied by the erection of a church on the spot where he died. In the next reign he was actually canonized, and his pic ture being set up in St. Paul's cathedral in London, it was regarded as an object of adoration by the common people, who always mentioned him by the appellation of the good earl of Lancaster. Robert de Holand, who, we already have stated, was sent by the earl of Lancaster to raise troops, and to conduct them to the castle of Tutbury, seems to have been generally suspected of having betrayed the noble earl, his benefactor and patron. The particulars of his treachery are not related by the historians of that period, but we shall find that a strong belief in it entertained by the public was ultimately the occasion of his death. On receiving intelligence of the defeat and capture ofthe earl of Lancaster and his friends at Boroughbridge, he surrendered himself to the king at Derby. He was sent prisoner to Dover castle. The overthrow of the earl of Lancaster's party and the death of that excellent prince, left the avidity and tyranny of the two Spencers without control. Still they did not think their vengeance to be complete while Roger Mortimer was alive, who had been one of the most active partisans in the confederacy against them. He was in their power, and they procured his condemnation but the queen found means so effectually to intercede in his favour, that his execution was con tinually delayed. The people, disgusted at the great sacrifice of life on the scaffold, which had followed the battle of Boroughbridge, and irritated at the death of a prince so generally beloved by them, were kept in a state of excitation by the emissaries of the queen, who everywhere de claimed against the recent severities, and the Spencers were afraid to crush one whom they knew to be their most inveterate enemy, and whom they held in their grasp. Sir Stephen Segrave was constable of the tower at that time ; a brave soldier, who was not likely to betray a trust confided to him, however he might commiserate the condition of his prisoner. Segrave had a soldier from his own manor of Alspath in Warwickshire, named Gerard, who was little troubled with those scruples that are so intimately connected with military duties. To Gerard of Alspath the custody of Sir Roger de Mortimer had been confided, and through his means a correspond ence was carried on between Mortimer and the queen, and a plot was laid to facilitate the escape of the former from the tower and his passage into France. On the evening appointed for this purpose, Mortimer invited the governor to sup with him. The old warrior loved to indulge in a social glass, and his memory was stored with many of the occurrences of the Scots' wars, which he was fond of relating; and particularly his own capture at the fatal battle of Bannockburn, where he, at the head of a company of his brave tenantry (many of whom were from Bretby, Rosleston and his other manors in Derbyshire) had actually driven a division of the Scottish army from the field, when, being deserted by those who ought to have supported them, they were sur rounded by the enemy and made prisoners. The wine was potent, and in some that was particu larly relished by the governor there had been infused a soporific drug. He fell fast asleep, and Mortimer prepared to effect his escape. Some persons concerned in the plot were of opinion that they should avail themselves of this opportunity to make themselves masters ofthe tower, which would encourage an immediate insurrection of the Londoners : but Mortimer receiving a private intimation from the queen that one of the Spencers, with a body of men, was hastening to the\""
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wiy2XiIPc7UV",
        "colab_type": "text"
      },
      "source": [
        "(...) St. Paul's cathedral in **London**, it was regarded (...) at that period, so **love**d and reverenced by the people (...)\n",
        "\n",
        "#### And again, let's retrieve the BookID and Volume to which this block of text belongs:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zpHBAUwxc7UX",
        "colab_type": "code",
        "colab": {},
        "outputId": "92eb5b3f-ab57-4f22-a821-1430f1161cf0"
      },
      "source": [
        "London_love_df.loc[347374, ['bookid', 'volume']]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "bookid    004158128\n",
              "volume           01\n",
              "Name: 347374, dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MgOLhqf0c7Uc",
        "colab_type": "text"
      },
      "source": [
        "#### The bookid is ALEPH SYS number, so we can search it on BL Explorer:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eE_zMdxIc7Ud",
        "colab_type": "code",
        "colab": {},
        "outputId": "3c5f4e89-990e-496a-9823-c68b80a06059"
      },
      "source": [
        "print('Link to this book\\'s catalog record at BL explorer:')\n",
        "print('http://explore.bl.uk/primo_library/libweb/action/search.do?cs=frb&doc=BLL'+ London_love_df[\"bookid\"][347374] + '&dscnt=1&scp.scps=scope:(BLCONTENT)&frbg=&tab=local_tab&srt=rank&ct=search&mode=Basic&dum=true&tb=t&indx=1&vl(freeText0)='+ London_love_df[\"bookid\"][347374] + '&fn=search&vid=BLVU1')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Link to this book's catalog record at BL explorer:\n",
            "http://explore.bl.uk/primo_library/libweb/action/search.do?cs=frb&doc=BLL004158128&dscnt=1&scp.scps=scope:(BLCONTENT)&frbg=&tab=local_tab&srt=rank&ct=search&mode=Basic&dum=true&tb=t&indx=1&vl(freeText0)=004158128&fn=search&vid=BLVU1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lb6qzgjwxB-a",
        "colab_type": "text"
      },
      "source": [
        "### Now, try with your own search term:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sYQ_6crqwCRw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "my_term = input(\"What you would like to search for: \")\n",
        "my_term_df = blocksText_df[blocksText_df[1].str.contains(my_term)]\n",
        "print(my_term_df)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}