Hadoop_on_Colab.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Hadoop on Colab for Github.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "#Hadoop on Colab"
      ],
      "metadata": {
        "id": "2aZnBPqcrsqr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Content:\n",
        "\n",
        "1.   [Installing Java 8](#scrollTo=Kxt9UbTArwTC)\n",
        "2.   [Installing Secure Shell Server (SSHD)](#scrollTo=E1_rIFHNb1zk)\n",
        "3.   [Installing Hadoop 3.2.3](#scrollTo=qCeL0IBlrnoF)\n",
        "4.   [Running Hadoop in standalone mode](#scrollTo=xyNhcphwU326)\n",
        "5.   [Running Hadoop in Pseudo-distributed mode](#scrollTo=lEWV2YjJmR78)"
      ],
      "metadata": {
        "id": "E1_rIFHNb1zk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Introduction"
      ],
      "metadata": {
        "id": "uEg1qPOoyAvp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Hadoop is an open-source framework which is mainly used for storage purposes and maintaining and analyzing a large amount of data or datasets on the clusters of commodity hardware, which means it is actually a data management tool."
      ],
      "metadata": {
        "id": "aHBU-XQgO30O"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Hadoop mainly works on 3 different modes:\n",
        "\n",
        "*   Standalone Mode\n",
        "*   Pseudo-distributed Mode\n",
        "*   Fully-distributed Mode\n"
      ],
      "metadata": {
        "id": "zfwqfBB6O7cu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Standalone Mode**\n",
        "\n",
        "By default, Hadoop is configured to run in a non distributed mode. It runs as a single Java process. Instead of HDFS, this mode utilizes the local file system. This mode is useful for debugging and there isn't any need to configure core-site.xml, hdfs-site.xml, mapred-site.xml, masters & slaves. Stand-alone mode is usually the fastest mode in Hadoop.\n",
        "\n",
        "**Pseudo-distributed Mode**\n",
        "\n",
        "Hadoop can also run on a single node in a Pseudo-distributed mode. In this mode, each daemon runs on separate java processes. In this mode custom configuration is required (core-site.xml, hdfs-site.xml, mapred-site.xml). Here HDFS is utilized for input and output. This mode of deployment is useful for testing and debugging purposes.\n",
        "\n",
        "**Fully-distributed Mode**\n",
        "\n",
        "This is the production mode of Hadoop. In this mode typically one machine in the cluster is designated as NameNode and another as Resource Manager exclusively. These are masters. All other nodes act as Data Node and Node Manager. These are the slaves. Configuration parameters and environment need to be specified for Hadoop Daemons."
      ],
      "metadata": {
        "id": "saGpmRDDQoAX"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Installing Java 8"
      ],
      "metadata": {
        "id": "Kxt9UbTArwTC"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Hadoop is a java programming-based data processing framework\n",
        "\n",
        "OpenJDK is a development environment for building applications, applets, and components using the Java programming language."
      ],
      "metadata": {
        "id": "eXp-IUxoGXEH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Checking the installed Java version\n",
        "!java -version"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zwH_9BC1ZIET",
        "outputId": "d3fa5324-28a4-4608-f563-08d127785228"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "openjdk version \"11.0.15\" 2022-04-19\n",
            "OpenJDK Runtime Environment (build 11.0.15+10-Ubuntu-0ubuntu0.18.04.1)\n",
            "OpenJDK 64-Bit Server VM (build 11.0.15+10-Ubuntu-0ubuntu0.18.04.1, mixed mode, sharing)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Installing java 8 for better compatibility with Hadoop"
      ],
      "metadata": {
        "id": "KmHd4GpDWOjX"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "_UOxDVi9ceCm"
      },
      "outputs": [],
      "source": [
        "#Installing java 8\n",
        "!apt-get install openjdk-8-jdk-headless -qq > /dev/null\n",
        "# -q, quiet level 2: no output except for errors\n",
        "#> /dev/null on the end of any command where you want to redirect all the stdout into nothingness"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Switching java default version"
      ],
      "metadata": {
        "id": "HTZN58Mv1hlG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Switching java version to use as default (choose option 2)\n",
        "!update-alternatives --config java"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BSMxNJlea1K7",
        "outputId": "daa109a8-d3a1-4a3a-d29b-3f16fb68b6b0"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "There are 2 choices for the alternative java (providing /usr/bin/java).\n",
            "\n",
            "  Selection    Path                                            Priority   Status\n",
            "------------------------------------------------------------\n",
            "* 0            /usr/lib/jvm/java-11-openjdk-amd64/bin/java      1111      auto mode\n",
            "  1            /usr/lib/jvm/java-11-openjdk-amd64/bin/java      1111      manual mode\n",
            "  2            /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java   1081      manual mode\n",
            "\n",
            "Press <enter> to keep the current choice[*], or type selection number: 2\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java to provide /usr/bin/java (java) in manual mode\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Switching javac version to use as default (choose option 2)\n",
        "!update-alternatives --config javac"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L9_5cIQGlg2G",
        "outputId": "da3e4be2-e53c-4e23-84bc-99f9cfe9a659"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "There are 2 choices for the alternative javac (providing /usr/bin/javac).\n",
            "\n",
            "  Selection    Path                                          Priority   Status\n",
            "------------------------------------------------------------\n",
            "* 0            /usr/lib/jvm/java-11-openjdk-amd64/bin/javac   1111      auto mode\n",
            "  1            /usr/lib/jvm/java-11-openjdk-amd64/bin/javac   1111      manual mode\n",
            "  2            /usr/lib/jvm/java-8-openjdk-amd64/bin/javac    1081      manual mode\n",
            "\n",
            "Press <enter> to keep the current choice[*], or type selection number: 2\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/javac to provide /usr/bin/javac (javac) in manual mode\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Switching jps version to use as default (choose option 2)\n",
        "!update-alternatives --config jps"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Mo4Dnbf82vjj",
        "outputId": "0d9a8389-3a35-4be2-8d70-935d0c867549"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "There are 2 choices for the alternative jps (providing /usr/bin/jps).\n",
            "\n",
            "  Selection    Path                                        Priority   Status\n",
            "------------------------------------------------------------\n",
            "* 0            /usr/lib/jvm/java-11-openjdk-amd64/bin/jps   1111      auto mode\n",
            "  1            /usr/lib/jvm/java-11-openjdk-amd64/bin/jps   1111      manual mode\n",
            "  2            /usr/lib/jvm/java-8-openjdk-amd64/bin/jps    1081      manual mode\n",
            "\n",
            "Press <enter> to keep the current choice[*], or type selection number: 2\n",
            "update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/bin/jps to provide /usr/bin/jps (jps) in manual mode\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Checking Java default version\n",
        "!java -version"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AWunFxU8b9gh",
        "outputId": "f34e87a6-644b-4132-937d-6794d690ea62"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "openjdk version \"1.8.0_312\"\n",
            "OpenJDK Runtime Environment (build 1.8.0_312-8u312-b07-0ubuntu1~18.04-b07)\n",
            "OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Creating Java related environment variables"
      ],
      "metadata": {
        "id": "K4xYHnO72Gwo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The JAVA_HOME is an operating system environment variable points to the file system location where the JDK or JRE was installed."
      ],
      "metadata": {
        "id": "vBzOrjQ3nRlI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Finding the default Java path\n",
        "!readlink -f /usr/bin/java | sed \"s:bin/java::\""
      ],
      "metadata": {
        "id": "kXFJVGrIdYbW",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "af6bc8dd-3373-45eb-a679-4db8e3783eb0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/usr/lib/jvm/java-8-openjdk-amd64/jre/\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Importing os module\n",
        "import os\n",
        "#Creating environment variables\n",
        "os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\"\n",
        "os.environ[\"JRE_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64/jre\"\n",
        "os.environ[\"PATH\"] += \":$JAVA_HOME/bin:$JRE_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin\""
      ],
      "metadata": {
        "id": "0am0Ybf6cyvX"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Installing Secure Shell Server (SSHD)"
      ],
      "metadata": {
        "id": "KRWvrIpCr3-A"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We need to define a means for the master node to remotely access every node in our cluster.\n",
        "\n",
        "Hadoop uses passphrases SSH for the communication between the nodes."
      ],
      "metadata": {
        "id": "P2-dYIkaBb2d"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "SSH is a cryptographic network protocol for operating network services securely over an unsecured network.\n",
        "\n",
        "SSH utilizes standard public key criptography to create a pair of keys for user verification: one public and one private."
      ],
      "metadata": {
        "id": "F26UBlxmCe0f"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#It is good practice to purge before installation\n",
        "!apt-get purge openssh-server -qq"
      ],
      "metadata": {
        "id": "akOe9EEDsc0u"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Installing openssh-server\n",
        "!apt-get install openssh-server -qq > /dev/null"
      ],
      "metadata": {
        "id": "3TR6IHVTsqYJ"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Starting the server\n",
        "!service ssh start"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BSAhIFipdEg-",
        "outputId": "3e3a4b46-93a5-4c42-f108-719d12c19a4e"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " * Starting OpenBSD Secure Shell server sshd\n",
            "   ...done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The port number for SSH is 22 by default"
      ],
      "metadata": {
        "id": "qLGhrALUJBHb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!grep Port /etc/ssh/sshd_config"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KLChmkJQSph0",
        "outputId": "2c407eb7-377f-4eae-db75-9b97036d73fc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "#Port 22\n",
            "#GatewayPorts no\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The pseudo distributed mode is special case of fully the distributed mode, in which the single host is localhost (our machine). We need to make sure that to access to localhost and login we do not need to enter a password. Therefore, SSH needs to be set up to allow passwordless login for the Hadoop user. The simplest way to achive this is to generate a public-private key pair."
      ],
      "metadata": {
        "id": "uEVnYmY4Dk3b"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Creating a new rsa key pair with empty password\n",
        "!ssh-keygen -t rsa -P \"\" -f ~/.ssh/id_rsa"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GrVaVZ--s32h",
        "outputId": "9e7e1713-3af8-4025-f204-1bca4389f29f"
      },
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Generating public/private rsa key pair.\n",
            "Your identification has been saved in /root/.ssh/id_rsa.\n",
            "Your public key has been saved in /root/.ssh/id_rsa.pub.\n",
            "The key fingerprint is:\n",
            "SHA256:C4S0g7LR3HbtY6LnaRscpyqTTiHWv6fdQ0imcuI6Fpg root@b6c4a8758417\n",
            "The key's randomart image is:\n",
            "+---[RSA 2048]----+\n",
            "|    .            |\n",
            "| o + o .         |\n",
            "|o + * o .        |\n",
            "| +.. + +         |\n",
            "|+o..  B S        |\n",
            "|Eo +.= O +       |\n",
            "|  +.=.= o        |\n",
            "| ++. +++..       |\n",
            "|.o+o.+*o ..      |\n",
            "+----[SHA256]-----+\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Showing the public key\n",
        "!more /root/.ssh/id_rsa.pub"
      ],
      "metadata": {
        "id": "EaMOu0NJta2N"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Copying the key to autorized keys\n",
        "!cat $HOME/.ssh/id_rsa.pub>>$HOME/.ssh/authorized_keys\n",
        "#Changing the permissions on the key\n",
        "!chmod 0600 ~/.ssh/authorized_keys"
      ],
      "metadata": {
        "id": "s3pLGo6utsNC"
      },
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Conneting with the local machine\n",
        "!ssh -o StrictHostKeyChecking=no localhost uptime"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qz7fc93DaX7Z",
        "outputId": "a4ef257c-8c67-4c80-821e-24a57a93806b"
      },
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " 08:17:05 up 6 min,  0 users,  load average: 0.92, 0.67, 0.33\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Installing Hadoop 3.2.3"
      ],
      "metadata": {
        "id": "qCeL0IBlrnoF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Downloading Hadoop 3.2.3\n",
        "!wget -q https://archive.apache.org/dist/hadoop/common/hadoop-3.2.3/hadoop-3.2.3.tar.gz"
      ],
      "metadata": {
        "id": "t6w9exw9c11y"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Untarring the file\n",
        "!sudo tar -xzf hadoop-3.2.3.tar.gz\n",
        "#Removing the tar file\n",
        "!rm hadoop-3.2.3.tar.gz"
      ],
      "metadata": {
        "id": "Ngsl1XNPdOP0"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Standard location to install Hadoop are\n",
        "\n",
        "*   /usr/local\n",
        "*   /opt"
      ],
      "metadata": {
        "id": "PmoGNehEKW_X"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Copying the hadoop files to user/local\n",
        "!cp -r hadoop-3.2.3/ /usr/local/\n",
        "#-r copy directories recursively"
      ],
      "metadata": {
        "id": "bb1_Uve9dUX1"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Exploring hadoop-3.2.3/etc/hadoop directory\n",
        "!ls /usr/local/hadoop-3.2.3/etc/hadoop\n",
        "#we can see various configuration files of hadoop "
      ],
      "metadata": {
        "id": "YJqjMBenAqWf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4c741f23-39b2-4154-81e9-3f10d40c8e09"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "capacity-scheduler.xml\t\t  kms-log4j.properties\n",
            "configuration.xsl\t\t  kms-site.xml\n",
            "container-executor.cfg\t\t  log4j.properties\n",
            "core-site.xml\t\t\t  mapred-env.cmd\n",
            "hadoop-env.cmd\t\t\t  mapred-env.sh\n",
            "hadoop-env.sh\t\t\t  mapred-queues.xml.template\n",
            "hadoop-metrics2.properties\t  mapred-site.xml\n",
            "hadoop-policy.xml\t\t  shellprofile.d\n",
            "hadoop-user-functions.sh.example  ssl-client.xml.example\n",
            "hdfs-site.xml\t\t\t  ssl-server.xml.example\n",
            "httpfs-env.sh\t\t\t  user_ec_policies.xml.template\n",
            "httpfs-log4j.properties\t\t  workers\n",
            "httpfs-signature.secret\t\t  yarn-env.cmd\n",
            "httpfs-site.xml\t\t\t  yarn-env.sh\n",
            "kms-acls.xml\t\t\t  yarnservice-log4j.properties\n",
            "kms-env.sh\t\t\t  yarn-site.xml\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We need to configure a few things before running Hadoop. That is, we need to either add or modify few parameters in these configuration files to operate Hadoop in whichever mode we want to. "
      ],
      "metadata": {
        "id": "cx80yeBMBUDT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Configuring hadoop-env.sh file"
      ],
      "metadata": {
        "id": "0mFNgy1--JYr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "hadoop-env.sh is a bash script that containts environment variables that are used in the scripts to run Hadoop"
      ],
      "metadata": {
        "id": "hLjfw00rCIFq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Exploring hadoop-env.sh file\n",
        "!cat /usr/local/hadoop-3.2.3/etc/hadoop/hadoop-env.sh"
      ],
      "metadata": {
        "id": "-JiNs3H7FrmI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The only required enviroment variable is **JAVA_HOME**. All the others are optional.\n",
        "\n",
        "To specify the JAVA_HOME variable in hadoop-env.sh we need to uncomment the export line and update it with the actual directory.\n",
        "\n",
        "In this case it should look like this:\n",
        "\n",
        "`export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64`"
      ],
      "metadata": {
        "id": "3o9SROCEHl72"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Adding JAVA_HOME directory to hadoop-env.sh file\n",
        "!sed -i '/export JAVA_HOME=/a export JAVA_HOME=\\/usr\\/lib\\/jvm\\/java-8-openjdk-amd64' /usr/local/hadoop-3.2.3/etc/hadoop/hadoop-env.sh"
      ],
      "metadata": {
        "id": "TeCM3uiCXvF-"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Because it is convinient we create an environment variable that points to the Hadoop installation directory"
      ],
      "metadata": {
        "id": "sX6dPYTcMU5A"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Creating Hadoop home variable\n",
        "os.environ[\"HADOOP_HOME\"] = \"/usr/local/hadoop-3.2.3\""
      ],
      "metadata": {
        "id": "rNtD2mvkMs5x"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Configuring XML files\n",
        "\n",
        "The majority of Hadoop setting are contained in XML configuration files. These files are also known as **resources**.\n",
        "\n",
        "They have the following structure:\n",
        "\n",
        "\n",
        "```\n",
        "<configuration>\n",
        "...\n",
        "  <property>\n",
        "    <name>...</name>\n",
        "    <value>...</value>\n",
        "    <description>...</description>\n",
        "  </property>\n",
        "...\n",
        "</configuration>\n",
        "```\n",
        "\n"
      ],
      "metadata": {
        "id": "dCws8A6MOC1K"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The XLM file can contained any number of the property elements. Each property element defines a specific configuration name-value pair."
      ],
      "metadata": {
        "id": "-nBTuk9wO7ow"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Hadoop configuration is driven by two distict types of XLM configuration files:\n",
        "\n",
        "1. **Default** (read-only): core-default.xml, hdfs-default.xml, mapred-default.xml, yarn-default.xml. These files should never be modified.\n",
        "2. **Site specific** configuration files: core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml. These files are loaded from class path and their values are used to overwrite the corresponding values of the properties in the matching default configuration files.\n"
      ],
      "metadata": {
        "id": "szhRwlUOQ9D3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Exploring hadoop-3.2.3/etc/hadoop xml files\n",
        "!ls $HADOOP_HOME/etc/hadoop/*.xml"
      ],
      "metadata": {
        "id": "zaUWrl0lN203",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a5b15977-13bc-4fce-ea97-0d6511ae7e9f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/usr/local/hadoop-3.2.3/etc/hadoop/capacity-scheduler.xml\n",
            "/usr/local/hadoop-3.2.3/etc/hadoop/core-site.xml\n",
            "/usr/local/hadoop-3.2.3/etc/hadoop/hadoop-policy.xml\n",
            "/usr/local/hadoop-3.2.3/etc/hadoop/hdfs-site.xml\n",
            "/usr/local/hadoop-3.2.3/etc/hadoop/httpfs-site.xml\n",
            "/usr/local/hadoop-3.2.3/etc/hadoop/kms-acls.xml\n",
            "/usr/local/hadoop-3.2.3/etc/hadoop/kms-site.xml\n",
            "/usr/local/hadoop-3.2.3/etc/hadoop/mapred-site.xml\n",
            "/usr/local/hadoop-3.2.3/etc/hadoop/yarn-site.xml\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Each component in Hadoop is configured using an xml file\n",
        "\n",
        "*   core-site.xml: common properties\n",
        "*   hdfs-site.xml: HDFS properties\n",
        "*   mapred-site.xml: MapReduce properties\n",
        "*   yarn-site.xml: YARN properties\n",
        "\n",
        "By configuring these xml files accordingly Hadoop can be run in one of the three modes."
      ],
      "metadata": {
        "id": "SMbIY4UQr-Cc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Content of core-site.xml file\n",
        "!cat $HADOOP_HOME/etc/hadoop/core-site.xml"
      ],
      "metadata": {
        "id": "3X2yMzJRTTbz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "646bbba2-f2e8-4d3a-9732-aafb7f43df62"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n",
            "<?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?>\n",
            "<!--\n",
            "  Licensed under the Apache License, Version 2.0 (the \"License\");\n",
            "  you may not use this file except in compliance with the License.\n",
            "  You may obtain a copy of the License at\n",
            "\n",
            "    http://www.apache.org/licenses/LICENSE-2.0\n",
            "\n",
            "  Unless required by applicable law or agreed to in writing, software\n",
            "  distributed under the License is distributed on an \"AS IS\" BASIS,\n",
            "  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
            "  See the License for the specific language governing permissions and\n",
            "  limitations under the License. See accompanying LICENSE file.\n",
            "-->\n",
            "\n",
            "<!-- Put site-specific property overrides in this file. -->\n",
            "\n",
            "<configuration>\n",
            "</configuration>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "As we can see, no properties are set. They are empty by default. So there is nothing to overwrite and Hadoop runs with the default properties."
      ],
      "metadata": {
        "id": "LlK_OhXOTwLa"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Running Hadoop in standalone mode"
      ],
      "metadata": {
        "id": "j7iQ3JIqP9Av"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "With the default configuration properties Hadoop runs in a standalone mode (non distributed mode). That is, standalone mode (also knows as local mode) is the default mode for Hadoop.\n",
        "\n",
        "There are no deamons to run. Just a single java process\n",
        "\n",
        "Local filesystem and the local MapReduce job runner are used"
      ],
      "metadata": {
        "id": "xyNhcphwU326"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The command to run a Hadoop mapreduce program that is written in Java is:\n",
        "\n",
        "`$HADOOP_HOME/bin/hadoop jar <jar>`\n",
        "\n",
        "jar is Java archive tool that packages (and compresses) a set of files into a single archive."
      ],
      "metadata": {
        "id": "M5sq5u4YMWYy"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The default installation already has several MapReduce examples program that we can use."
      ],
      "metadata": {
        "id": "T0mlMcx90v3C"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Exploring mapreduce tools\n",
        "!ls $HADOOP_HOME/share/hadoop/mapreduce/*.jar"
      ],
      "metadata": {
        "id": "KDU32MA_1A3j",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e2f2be80-19c1-4c61-f62c-94ad64552a2b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.2.3.jar\n",
            "/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.2.3.jar\n",
            "/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.2.3.jar\n",
            "/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.2.3.jar\n",
            "/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.2.3.jar\n",
            "/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.3.jar\n",
            "/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.3-tests.jar\n",
            "/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.2.3.jar\n",
            "/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.2.3.jar\n",
            "/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.2.3.jar\n",
            "/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.3.jar\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Mapreduce examples"
      ],
      "metadata": {
        "id": "E1sibHPwHcIS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Exploring the examples of programs available\n",
        "!$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.3.jar"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TbNQV5Id1VEW",
        "outputId": "e592999a-d537-4d70-89ef-18e5e5b611f9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "An example program must be given as the first argument.\n",
            "Valid program names are:\n",
            "  aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.\n",
            "  aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.\n",
            "  bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.\n",
            "  dbcount: An example job that count the pageview counts from a database.\n",
            "  distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.\n",
            "  grep: A map/reduce program that counts the matches of a regex in the input.\n",
            "  join: A job that effects a join over sorted, equally partitioned datasets\n",
            "  multifilewc: A job that counts words from several files.\n",
            "  pentomino: A map/reduce tile laying program to find solutions to pentomino problems.\n",
            "  pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.\n",
            "  randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.\n",
            "  randomwriter: A map/reduce program that writes 10GB of random data per node.\n",
            "  secondarysort: An example defining a secondary sort to the reduce.\n",
            "  sort: A map/reduce program that sorts the data written by the random writer.\n",
            "  sudoku: A sudoku solver.\n",
            "  teragen: Generate data for the terasort\n",
            "  terasort: Run the terasort\n",
            "  teravalidate: Checking results of terasort\n",
            "  wordcount: A map/reduce program that counts the words in the input files.\n",
            "  wordmean: A map/reduce program that counts the average length of the words in the input files.\n",
            "  wordmedian: A map/reduce program that counts the median length of the words in the input files.\n",
            "  wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "wordcount"
      ],
      "metadata": {
        "id": "ky8_JvARHoVH"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "As description says, wordcount is a map/reduce program that counts the words in the input files."
      ],
      "metadata": {
        "id": "YOPq3tC91yRa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Usage of the wordcount MapReduce program \n",
        "!$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.3.jar wordcount"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0FjXG_lX2UUB",
        "outputId": "a51c67bd-7229-4968-d46f-14159f41bd57"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Usage: wordcount <in> [<in>...] <out>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The parameters are an input directory where the text to be analized is allocated and an output directory where the program is going to allocate its output "
      ],
      "metadata": {
        "id": "XkqKpjQl24wx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Dowloading text example to use as input\n",
        "!wget -q https://www.mirrorservice.org/sites/ftp.ibiblio.org/pub/docs/books/gutenberg/1/0/101/101.txt"
      ],
      "metadata": {
        "id": "fOHl7Iz84eyt"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Running MapReduce program wordcount\n",
        "#the output directory will be created automatically\n",
        "!$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.3.jar wordcount /content/101.txt /content/output"
      ],
      "metadata": {
        "id": "tgY1T-c45Vzg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5173bb77-c441-41c5-eab6-87feedfe6c5f"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2022-07-11 08:14:38,599 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties\n",
            "2022-07-11 08:14:38,707 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).\n",
            "2022-07-11 08:14:38,707 INFO impl.MetricsSystemImpl: JobTracker metrics system started\n",
            "2022-07-11 08:14:38,903 INFO input.FileInputFormat: Total input files to process : 1\n",
            "2022-07-11 08:14:38,944 INFO mapreduce.JobSubmitter: number of splits:1\n",
            "2022-07-11 08:14:39,138 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local310152804_0001\n",
            "2022-07-11 08:14:39,138 INFO mapreduce.JobSubmitter: Executing with tokens: []\n",
            "2022-07-11 08:14:39,315 INFO mapreduce.Job: The url to track the job: http://localhost:8080/\n",
            "2022-07-11 08:14:39,316 INFO mapreduce.Job: Running job: job_local310152804_0001\n",
            "2022-07-11 08:14:39,323 INFO mapred.LocalJobRunner: OutputCommitter set in config null\n",
            "2022-07-11 08:14:39,332 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2\n",
            "2022-07-11 08:14:39,332 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false\n",
            "2022-07-11 08:14:39,333 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter\n",
            "2022-07-11 08:14:39,378 INFO mapred.LocalJobRunner: Waiting for map tasks\n",
            "2022-07-11 08:14:39,379 INFO mapred.LocalJobRunner: Starting task: attempt_local310152804_0001_m_000000_0\n",
            "2022-07-11 08:14:39,414 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2\n",
            "2022-07-11 08:14:39,414 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false\n",
            "2022-07-11 08:14:39,445 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]\n",
            "2022-07-11 08:14:39,455 INFO mapred.MapTask: Processing split: file:/content/101.txt:0+678064\n",
            "2022-07-11 08:14:39,567 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)\n",
            "2022-07-11 08:14:39,567 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100\n",
            "2022-07-11 08:14:39,567 INFO mapred.MapTask: soft limit at 83886080\n",
            "2022-07-11 08:14:39,567 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600\n",
            "2022-07-11 08:14:39,567 INFO mapred.MapTask: kvstart = 26214396; length = 6553600\n",
            "2022-07-11 08:14:39,572 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer\n",
            "2022-07-11 08:14:39,834 INFO mapred.LocalJobRunner: \n",
            "2022-07-11 08:14:39,834 INFO mapred.MapTask: Starting flush of map output\n",
            "2022-07-11 08:14:39,834 INFO mapred.MapTask: Spilling map output\n",
            "2022-07-11 08:14:39,834 INFO mapred.MapTask: bufstart = 0; bufend = 1078906; bufvoid = 104857600\n",
            "2022-07-11 08:14:39,836 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 25792448(103169792); length = 421949/6553600\n",
            "2022-07-11 08:14:40,321 INFO mapreduce.Job: Job job_local310152804_0001 running in uber mode : false\n",
            "2022-07-11 08:14:40,322 INFO mapreduce.Job:  map 0% reduce 0%\n",
            "2022-07-11 08:14:40,482 INFO mapred.MapTask: Finished spill 0\n",
            "2022-07-11 08:14:40,498 INFO mapred.Task: Task:attempt_local310152804_0001_m_000000_0 is done. And is in the process of committing\n",
            "2022-07-11 08:14:40,503 INFO mapred.LocalJobRunner: map\n",
            "2022-07-11 08:14:40,503 INFO mapred.Task: Task 'attempt_local310152804_0001_m_000000_0' done.\n",
            "2022-07-11 08:14:40,514 INFO mapred.Task: Final Counters for attempt_local310152804_0001_m_000000_0: Counters: 18\n",
            "\tFile System Counters\n",
            "\t\tFILE: Number of bytes read=994673\n",
            "\t\tFILE: Number of bytes written=1178697\n",
            "\t\tFILE: Number of read operations=0\n",
            "\t\tFILE: Number of large read operations=0\n",
            "\t\tFILE: Number of write operations=0\n",
            "\tMap-Reduce Framework\n",
            "\t\tMap input records=13006\n",
            "\t\tMap output records=105488\n",
            "\t\tMap output bytes=1078906\n",
            "\t\tMap output materialized bytes=318005\n",
            "\t\tInput split bytes=86\n",
            "\t\tCombine input records=105488\n",
            "\t\tCombine output records=21438\n",
            "\t\tSpilled Records=21438\n",
            "\t\tFailed Shuffles=0\n",
            "\t\tMerged Map outputs=0\n",
            "\t\tGC time elapsed (ms)=0\n",
            "\t\tTotal committed heap usage (bytes)=258473984\n",
            "\tFile Input Format Counters \n",
            "\t\tBytes Read=678064\n",
            "2022-07-11 08:14:40,515 INFO mapred.LocalJobRunner: Finishing task: attempt_local310152804_0001_m_000000_0\n",
            "2022-07-11 08:14:40,515 INFO mapred.LocalJobRunner: map task executor complete.\n",
            "2022-07-11 08:14:40,519 INFO mapred.LocalJobRunner: Waiting for reduce tasks\n",
            "2022-07-11 08:14:40,519 INFO mapred.LocalJobRunner: Starting task: attempt_local310152804_0001_r_000000_0\n",
            "2022-07-11 08:14:40,530 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2\n",
            "2022-07-11 08:14:40,530 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false\n",
            "2022-07-11 08:14:40,531 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]\n",
            "2022-07-11 08:14:40,536 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@613ecef1\n",
            "2022-07-11 08:14:40,539 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!\n",
            "2022-07-11 08:14:40,576 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=2119434240, maxSingleShuffleLimit=529858560, mergeThreshold=1398826624, ioSortFactor=10, memToMemMergeOutputsThreshold=10\n",
            "2022-07-11 08:14:40,589 INFO reduce.EventFetcher: attempt_local310152804_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events\n",
            "2022-07-11 08:14:40,624 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local310152804_0001_m_000000_0 decomp: 318001 len: 318005 to MEMORY\n",
            "2022-07-11 08:14:40,628 INFO reduce.InMemoryMapOutput: Read 318001 bytes from map-output for attempt_local310152804_0001_m_000000_0\n",
            "2022-07-11 08:14:40,632 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 318001, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->318001\n",
            "2022-07-11 08:14:40,633 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning\n",
            "2022-07-11 08:14:40,634 INFO mapred.LocalJobRunner: 1 / 1 copied.\n",
            "2022-07-11 08:14:40,634 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs\n",
            "2022-07-11 08:14:40,643 INFO mapred.Merger: Merging 1 sorted segments\n",
            "2022-07-11 08:14:40,644 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 317994 bytes\n",
            "2022-07-11 08:14:40,683 INFO reduce.MergeManagerImpl: Merged 1 segments, 318001 bytes to disk to satisfy reduce memory limit\n",
            "2022-07-11 08:14:40,684 INFO reduce.MergeManagerImpl: Merging 1 files, 318005 bytes from disk\n",
            "2022-07-11 08:14:40,685 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce\n",
            "2022-07-11 08:14:40,685 INFO mapred.Merger: Merging 1 sorted segments\n",
            "2022-07-11 08:14:40,686 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 317994 bytes\n",
            "2022-07-11 08:14:40,686 INFO mapred.LocalJobRunner: 1 / 1 copied.\n",
            "2022-07-11 08:14:40,690 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords\n",
            "2022-07-11 08:14:40,873 INFO mapred.Task: Task:attempt_local310152804_0001_r_000000_0 is done. And is in the process of committing\n",
            "2022-07-11 08:14:40,875 INFO mapred.LocalJobRunner: 1 / 1 copied.\n",
            "2022-07-11 08:14:40,876 INFO mapred.Task: Task attempt_local310152804_0001_r_000000_0 is allowed to commit now\n",
            "2022-07-11 08:14:40,879 INFO output.FileOutputCommitter: Saved output of task 'attempt_local310152804_0001_r_000000_0' to file:/content/output\n",
            "2022-07-11 08:14:40,880 INFO mapred.LocalJobRunner: reduce > reduce\n",
            "2022-07-11 08:14:40,881 INFO mapred.Task: Task 'attempt_local310152804_0001_r_000000_0' done.\n",
            "2022-07-11 08:14:40,881 INFO mapred.Task: Final Counters for attempt_local310152804_0001_r_000000_0: Counters: 24\n",
            "\tFile System Counters\n",
            "\t\tFILE: Number of bytes read=1630715\n",
            "\t\tFILE: Number of bytes written=1732174\n",
            "\t\tFILE: Number of read operations=0\n",
            "\t\tFILE: Number of large read operations=0\n",
            "\t\tFILE: Number of write operations=0\n",
            "\tMap-Reduce Framework\n",
            "\t\tCombine input records=0\n",
            "\t\tCombine output records=0\n",
            "\t\tReduce input groups=21438\n",
            "\t\tReduce shuffle bytes=318005\n",
            "\t\tReduce input records=21438\n",
            "\t\tReduce output records=21438\n",
            "\t\tSpilled Records=21438\n",
            "\t\tShuffled Maps =1\n",
            "\t\tFailed Shuffles=0\n",
            "\t\tMerged Map outputs=1\n",
            "\t\tGC time elapsed (ms)=31\n",
            "\t\tTotal committed heap usage (bytes)=258473984\n",
            "\tShuffle Errors\n",
            "\t\tBAD_ID=0\n",
            "\t\tCONNECTION=0\n",
            "\t\tIO_ERROR=0\n",
            "\t\tWRONG_LENGTH=0\n",
            "\t\tWRONG_MAP=0\n",
            "\t\tWRONG_REDUCE=0\n",
            "\tFile Output Format Counters \n",
            "\t\tBytes Written=235472\n",
            "2022-07-11 08:14:40,881 INFO mapred.LocalJobRunner: Finishing task: attempt_local310152804_0001_r_000000_0\n",
            "2022-07-11 08:14:40,881 INFO mapred.LocalJobRunner: reduce task executor complete.\n",
            "2022-07-11 08:14:41,327 INFO mapreduce.Job:  map 100% reduce 100%\n",
            "2022-07-11 08:14:41,327 INFO mapreduce.Job: Job job_local310152804_0001 completed successfully\n",
            "2022-07-11 08:14:41,336 INFO mapreduce.Job: Counters: 30\n",
            "\tFile System Counters\n",
            "\t\tFILE: Number of bytes read=2625388\n",
            "\t\tFILE: Number of bytes written=2910871\n",
            "\t\tFILE: Number of read operations=0\n",
            "\t\tFILE: Number of large read operations=0\n",
            "\t\tFILE: Number of write operations=0\n",
            "\tMap-Reduce Framework\n",
            "\t\tMap input records=13006\n",
            "\t\tMap output records=105488\n",
            "\t\tMap output bytes=1078906\n",
            "\t\tMap output materialized bytes=318005\n",
            "\t\tInput split bytes=86\n",
            "\t\tCombine input records=105488\n",
            "\t\tCombine output records=21438\n",
            "\t\tReduce input groups=21438\n",
            "\t\tReduce shuffle bytes=318005\n",
            "\t\tReduce input records=21438\n",
            "\t\tReduce output records=21438\n",
            "\t\tSpilled Records=42876\n",
            "\t\tShuffled Maps =1\n",
            "\t\tFailed Shuffles=0\n",
            "\t\tMerged Map outputs=1\n",
            "\t\tGC time elapsed (ms)=31\n",
            "\t\tTotal committed heap usage (bytes)=516947968\n",
            "\tShuffle Errors\n",
            "\t\tBAD_ID=0\n",
            "\t\tCONNECTION=0\n",
            "\t\tIO_ERROR=0\n",
            "\t\tWRONG_LENGTH=0\n",
            "\t\tWRONG_MAP=0\n",
            "\t\tWRONG_REDUCE=0\n",
            "\tFile Input Format Counters \n",
            "\t\tBytes Read=678064\n",
            "\tFile Output Format Counters \n",
            "\t\tBytes Written=235472\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Exploring the created output directory\n",
        "#part-r-00000 contains the actual ouput\n",
        "!ls /content/output"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "51IBObej6BA6",
        "outputId": "ab4148bd-b64c-4d4d-b373-24eee2164e1a"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "part-r-00000  _SUCCESS\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Printing out first 50 lines\n",
        "!head -50 /content/output/part-r-00000"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ViiYYpJcUnjd",
        "outputId": "df202284-492e-4215-a435-1cfed035a7eb"
      },
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\"(d)\t1\n",
            "\"/H\"\t1\n",
            "\"0\"\t1\n",
            "\"02\"\t2\n",
            "\"1.\t1\n",
            "\"414\t3\n",
            "\"A\t2\n",
            "\"AT&T\t3\n",
            "\"AT&T's\t1\n",
            "\"Access\t1\n",
            "\"Acid\t3\n",
            "\"Ad-hocracy\"\t1\n",
            "\"Advanced\t1\n",
            "\"Agents\t1\n",
            "\"Al\t3\n",
            "\"All\t3\n",
            "\"American\t1\n",
            "\"An\t1\n",
            "\"And\t2\n",
            "\"Any\t1\n",
            "\"Are\t3\n",
            "\"Artificial\t1\n",
            "\"As\t1\n",
            "\"Assistant\t1\n",
            "\"Attctc\"\t2\n",
            "\"Auld\t1\n",
            "\"Autodesk,\"\t1\n",
            "\"BBS,\"\t2\n",
            "\"BIRTHPLACE\t1\n",
            "\"BRITS\t1\n",
            "\"Barry\t1\n",
            "\"Because\t1\n",
            "\"Before\t1\n",
            "\"Bell\t2\n",
            "\"Bell\"\t1\n",
            "\"BellSouth\t2\n",
            "\"Berkeley\t1\n",
            "\"Big\t1\n",
            "\"Biggest\t1\n",
            "\"Black\t3\n",
            "\"Blue\t1\n",
            "\"Bob\"\t1\n",
            "\"Bob,\t3\n",
            "\"Bob:\t1\n",
            "\"Bullet-N-Board.\"\t1\n",
            "\"Bureaucrat-ese\t1\n",
            "\"But\t4\n",
            "\"C-word.\"\t1\n",
            "\"CALIFORNIA\"\t1\n",
            "\"CC,\"\t1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Running Hadoop in Pseudo-distributed mode"
      ],
      "metadata": {
        "id": "5BUFo8gUbXER"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In Pseudo-distributed mode all the distributed components of Hadoop come into play. That is, all the Hadoop deamons that are responsible for distributed storage and distributed processing will run on the same machine.\n",
        "\n",
        "Master deamons:\n",
        "\n",
        "*   NameNode\n",
        "*   Resource Manager\n",
        "*   Standby NameNode\n",
        "\n",
        "Slave deamons:\n",
        "\n",
        "*   DataNode\n",
        "*   Node Manager\n"
      ],
      "metadata": {
        "id": "lEWV2YjJmR78"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Configuring XML files"
      ],
      "metadata": {
        "id": "h9KNyTmWppr3"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "As mentioned, by setting the properties in the **site** xml configuration files, we overwrite the corresponding properties in the **default** xml configuration files and, this way, we tell Hadoop which machines are in the cluster and where and how we want to run the Hadoop daemons"
      ],
      "metadata": {
        "id": "YjYGTIZRM43v"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The specific content that these files need to have to make Hadoop run in Pseudo-distributed mode can be found in the documentation of the release on the official website. For Hadoop 3.2.3 the website is:\n",
        "\n",
        "https://hadoop.apache.org/docs/r3.2.3/hadoop-project-dist/hadoop-common/SingleCluster.html"
      ],
      "metadata": {
        "id": "rkzy6KSdTGw6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Configuring core-site.xml"
      ],
      "metadata": {
        "id": "Karc-KZ0CLvg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Adding required property to core-site.xlm file\n",
        "!sed -i '/<configuration>/a\\\n",
        "  <property>\\n\\\n",
        "    <name>fs.defaultFS</name>\\n\\\n",
        "    <value>hdfs://localhost:9000</value>\\n\\\n",
        "  </property>' \\\n",
        "$HADOOP_HOME/etc/hadoop/core-site.xml"
      ],
      "metadata": {
        "id": "5aIR5y4hAZvI"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Content of core-site.xml after the editing\n",
        "!cat $HADOOP_HOME/etc/hadoop/core-site.xml"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wnmp-mDKPJVs",
        "outputId": "d8e20037-1236-4642-e664-568e40a537f6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n",
            "<?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?>\n",
            "<!--\n",
            "  Licensed under the Apache License, Version 2.0 (the \"License\");\n",
            "  you may not use this file except in compliance with the License.\n",
            "  You may obtain a copy of the License at\n",
            "\n",
            "    http://www.apache.org/licenses/LICENSE-2.0\n",
            "\n",
            "  Unless required by applicable law or agreed to in writing, software\n",
            "  distributed under the License is distributed on an \"AS IS\" BASIS,\n",
            "  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
            "  See the License for the specific language governing permissions and\n",
            "  limitations under the License. See accompanying LICENSE file.\n",
            "-->\n",
            "\n",
            "<!-- Put site-specific property overrides in this file. -->\n",
            "\n",
            "<configuration>\n",
            "<property>\n",
            "    <name>fs.defaultFS</name>\n",
            "    <value>hdfs://localhost:9000</value>\n",
            "  </property>\n",
            "</configuration>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Configuring hdfs-site.xml"
      ],
      "metadata": {
        "id": "mqG9Gg2sCP6X"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Adding required property to hdfs-site.xml file\n",
        "#Since we are running Hadoop in only one machine, a replication factor greater than 1 does not make sense \n",
        "!sed -i '/<configuration>/a\\\n",
        "  <property>\\n\\\n",
        "    <name>dfs.replication</name>\\n\\\n",
        "    <value>1</value>\\n\\\n",
        "  </property>' \\\n",
        "$HADOOP_HOME/etc/hadoop/hdfs-site.xml"
      ],
      "metadata": {
        "id": "s4lKHtfvDFPF"
      },
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Content of hdfs-site.xml after the editing\n",
        "!cat $HADOOP_HOME/etc/hadoop/hdfs-site.xml"
      ],
      "metadata": {
        "id": "jNRY6cvNC6Q3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "34c59ea1-41ea-472a-f0d4-f2383ae73bd0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n",
            "<?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?>\n",
            "<!--\n",
            "  Licensed under the Apache License, Version 2.0 (the \"License\");\n",
            "  you may not use this file except in compliance with the License.\n",
            "  You may obtain a copy of the License at\n",
            "\n",
            "    http://www.apache.org/licenses/LICENSE-2.0\n",
            "\n",
            "  Unless required by applicable law or agreed to in writing, software\n",
            "  distributed under the License is distributed on an \"AS IS\" BASIS,\n",
            "  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
            "  See the License for the specific language governing permissions and\n",
            "  limitations under the License. See accompanying LICENSE file.\n",
            "-->\n",
            "\n",
            "<!-- Put site-specific property overrides in this file. -->\n",
            "\n",
            "<configuration>\n",
            "<property>\n",
            "    <name>dfs.replication</name>\n",
            "    <value>1</value>\n",
            "  </property>\n",
            "\n",
            "</configuration>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Configuring mapred-site.xml"
      ],
      "metadata": {
        "id": "nxKVm6HDD3Aj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Adding required properties to mapred-site.xml file\n",
        "!sed -i '/<configuration>/a\\\n",
        "  <property>\\n\\\n",
        "    <name>mapreduce.framework.name</name>\\n\\\n",
        "    <value>yarn</value>\\n\\\n",
        "  </property>\\n\\\n",
        "  <property>\\n\\\n",
        "    <name>mapreduce.application.classpath</name>\\n\\\n",
        "    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>\\n\\\n",
        "  </property>' \\\n",
        "$HADOOP_HOME/etc/hadoop/mapred-site.xml"
      ],
      "metadata": {
        "id": "sKBkrA5wF3_E"
      },
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Content of mapred-site.xml after the editing\n",
        "!cat $HADOOP_HOME/etc/hadoop/mapred-site.xml"
      ],
      "metadata": {
        "id": "7YCJQSKOFwGj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "522c55e2-3076-498f-f45b-2809a6b6ad87"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<?xml version=\"1.0\"?>\n",
            "<?xml-stylesheet type=\"text/xsl\" href=\"configuration.xsl\"?>\n",
            "<!--\n",
            "  Licensed under the Apache License, Version 2.0 (the \"License\");\n",
            "  you may not use this file except in compliance with the License.\n",
            "  You may obtain a copy of the License at\n",
            "\n",
            "    http://www.apache.org/licenses/LICENSE-2.0\n",
            "\n",
            "  Unless required by applicable law or agreed to in writing, software\n",
            "  distributed under the License is distributed on an \"AS IS\" BASIS,\n",
            "  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
            "  See the License for the specific language governing permissions and\n",
            "  limitations under the License. See accompanying LICENSE file.\n",
            "-->\n",
            "\n",
            "<!-- Put site-specific property overrides in this file. -->\n",
            "\n",
            "<configuration>\n",
            "<property>\n",
            "    <name>mapreduce.framework.name</name>\n",
            "    <value>yarn</value>\n",
            "  </property>\n",
            "  <property>\n",
            "    <name>mapreduce.application.classpath</name>\n",
            "    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>\n",
            "  </property>\n",
            "\n",
            "</configuration>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Configuring yarn-site.xml"
      ],
      "metadata": {
        "id": "cCoXAPGLHj8b"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Adding required properties to yarn-site.xml file\n",
        "!sed -i '/<configuration>/a\\\n",
        "  <property>\\n\\\n",
        "    <description>The hostname of the RM.</description>\\n\\\n",
        "    <name>yarn.resourcemanager.hostname</name>\\n\\\n",
        "    <value>localhost</value>\\n\\\n",
        "  </property>\\n\\\n",
        "  <property>\\n\\\n",
        "    <name>yarn.nodemanager.aux-services</name>\\n\\\n",
        "    <value>mapreduce_shuffle</value>\\n\\\n",
        "  </property>\\n\\\n",
        "  <property>\\n\\\n",
        "    <name>yarn.nodemanager.env-whitelist</name>\\n\\\n",
        "    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>\\n\\\n",
        "  </property>' \\\n",
        "$HADOOP_HOME/etc/hadoop/yarn-site.xml"
      ],
      "metadata": {
        "id": "-76CUdTLIwCa"
      },
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Content of yarn-site.xml after the editing\n",
        "!cat $HADOOP_HOME/etc/hadoop/yarn-site.xml"
      ],
      "metadata": {
        "id": "qm0PsXooIj_h",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "eb0e3641-9963-4d09-9cf5-97a670137a87"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<?xml version=\"1.0\"?>\n",
            "<!--\n",
            "  Licensed under the Apache License, Version 2.0 (the \"License\");\n",
            "  you may not use this file except in compliance with the License.\n",
            "  You may obtain a copy of the License at\n",
            "\n",
            "    http://www.apache.org/licenses/LICENSE-2.0\n",
            "\n",
            "  Unless required by applicable law or agreed to in writing, software\n",
            "  distributed under the License is distributed on an \"AS IS\" BASIS,\n",
            "  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
            "  See the License for the specific language governing permissions and\n",
            "  limitations under the License. See accompanying LICENSE file.\n",
            "-->\n",
            "<configuration>\n",
            "<property>\n",
            "    <description>The hostname of the RM.</description>\n",
            "    <name>yarn.resourcemanager.hostname</name>\n",
            "    <value>localhost</value>\n",
            "  </property>\n",
            "  <property>\n",
            "    <name>yarn.nodemanager.aux-services</name>\n",
            "    <value>mapreduce_shuffle</value>\n",
            "  </property>\n",
            "  <property>\n",
            "    <name>yarn.nodemanager.env-whitelist</name>\n",
            "    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>\n",
            "  </property>\n",
            "\n",
            "<!-- Site specific YARN configuration properties -->\n",
            "\n",
            "</configuration>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Formatting the HDFS Filesystem"
      ],
      "metadata": {
        "id": "x5GRijGvFuQe"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Before HDFS can be used for the first time the file system must be formatted. The formatting process creates an empty file system by creating the storage directories and the initial versions of the NameNodes"
      ],
      "metadata": {
        "id": "QRBUEp4yKXEe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!$HADOOP_HOME/bin/hdfs namenode -format"
      ],
      "metadata": {
        "id": "EyB07ajYOBna",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e3e573de-ce76-493e-8b5e-671b7b3528a9"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "WARNING: /usr/local/hadoop-3.2.3/logs does not exist. Creating.\n",
            "2022-07-11 08:15:55,918 INFO namenode.NameNode: STARTUP_MSG: \n",
            "/************************************************************\n",
            "STARTUP_MSG: Starting NameNode\n",
            "STARTUP_MSG:   host = b6c4a8758417/172.28.0.2\n",
            "STARTUP_MSG:   args = [-format]\n",
            "STARTUP_MSG:   version = 3.2.3\n",
            "STARTUP_MSG:   classpath = /usr/local/hadoop-3.2.3/etc/hadoop:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/error_prone_annotations-2.2.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/checker-qual-2.5.2.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/j2objc-annotations-1.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/token-provider-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jetty-security-9.4.40.v20210413.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/kerb-util-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jackson-annotations-2.10.5.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/kerb-common-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jetty-servlet-9.4.40.v20210413.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/accessors-smart-2.4.7.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/httpcore-4.4.13.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/asm-5.0.4.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/curator-recipes-2.13.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/commons-beanutils-1.9.4.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/commons-compress-1.21.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jackson-core-2.10.5.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/httpclient-4.5.13.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jersey-core-1.19.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jetty-io-9.4.40.v20210413.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jersey-json-1.19.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/hadoop-annotations-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/commons-net-3.6.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jetty-webapp-9.4.40.v20210413.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/re2j-1.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jetty-server-9.4.40.v20210413.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/kerb-client-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/nimbus-jose-jwt-9.8.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/kerby-pkix-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jsch-0.1.55.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/commons-codec-1.11.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/javax.activation-api-1.2.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/json-smart-2.4.7.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jetty-util-ajax-9.4.40.v20210413.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/woodstox-core-5.3.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/snappy-java-1.0.5.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jetty-util-9.4.40.v20210413.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jetty-http-9.4.40.v20210413.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/dnsjava-2.1.7.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jetty-xml-9.4.40.v20210413.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/guava-27.0-jre.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/commons-text-1.4.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/curator-client-2.13.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/commons-configuration2-2.1.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/stax2-api-4.2.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jersey-servlet-1.19.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/commons-lang3-3.7.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jsr305-3.0.2.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/metrics-core-3.2.4.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/kerby-config-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/slf4j-api-1.7.25.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/animal-sniffer-annotations-1.17.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/spotbugs-annotations-3.1.9.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/failureaccess-1.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/kerb-server-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/zookeeper-3.4.14.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/commons-io-2.8.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/kerby-util-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/kerby-asn1-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jersey-server-1.19.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/avro-1.7.7.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/hadoop-auth-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/netty-3.10.6.Final.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jackson-databind-2.10.5.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/jul-to-slf4j-1.7.25.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/kerb-core-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/lib/curator-framework-2.13.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/hadoop-common-3.2.3-tests.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/hadoop-common-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/hadoop-nfs-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/common/hadoop-kms-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/error_prone_annotations-2.2.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/checker-qual-2.5.2.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/j2objc-annotations-1.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/token-provider-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jetty-security-9.4.40.v20210413.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/gson-2.2.4.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/kerb-util-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jackson-annotations-2.10.5.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/kerb-common-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jetty-servlet-9.4.40.v20210413.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/accessors-smart-2.4.7.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/httpcore-4.4.13.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/asm-5.0.4.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/curator-recipes-2.13.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-beanutils-1.9.4.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-compress-1.21.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jackson-core-2.10.5.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/httpclient-4.5.13.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jersey-core-1.19.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/kerb-crypto-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jetty-io-9.4.40.v20210413.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jersey-json-1.19.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/hadoop-annotations-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-net-3.6.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jetty-webapp-9.4.40.v20210413.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/re2j-1.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jetty-server-9.4.40.v20210413.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/kerb-client-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/nimbus-jose-jwt-9.8.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/kerby-pkix-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jsch-0.1.55.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-codec-1.11.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/javax.activation-api-1.2.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/audience-annotations-0.5.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-math3-3.1.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/json-smart-2.4.7.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jetty-util-ajax-9.4.40.v20210413.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/woodstox-core-5.3.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/snappy-java-1.0.5.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jetty-util-9.4.40.v20210413.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jetty-http-9.4.40.v20210413.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/dnsjava-2.1.7.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jetty-xml-9.4.40.v20210413.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/guava-27.0-jre.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-text-1.4.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/curator-client-2.13.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-configuration2-2.1.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/stax2-api-4.2.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/netty-all-4.1.68.Final.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jersey-servlet-1.19.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/kerb-simplekdc-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/kerb-identity-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-lang3-3.7.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jsr305-3.0.2.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/paranamer-2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/animal-sniffer-annotations-1.17.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/spotbugs-annotations-3.1.9.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/failureaccess-1.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/okio-1.6.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/kerb-server-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jettison-1.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/zookeeper-3.4.14.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-io-2.8.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/kerby-util-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jersey-server-1.19.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/avro-1.7.7.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/hadoop-auth-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/netty-3.10.6.Final.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/jackson-databind-2.10.5.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/lib/curator-framework-2.13.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/hadoop-hdfs-rbf-3.2.3-tests.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/hadoop-hdfs-3.2.3-tests.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/hadoop-hdfs-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/hadoop-hdfs-rbf-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/hadoop-hdfs-client-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/hadoop-hdfs-client-3.2.3-tests.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/hadoop-hdfs-native-client-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/hadoop-hdfs-native-client-3.2.3-tests.jar:/usr/local/hadoop-3.2.3/share/hadoop/hdfs/hadoop-hdfs-nfs-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/lib/junit-4.13.2.jar:/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.3-tests.jar:/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/bcprov-jdk15on-1.60.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/bcpkix-jdk15on-1.60.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/objenesis-1.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.10.5.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/jakarta.xml.bind-api-2.3.2.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/fst-2.50.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/guice-4.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/jackson-jaxrs-base-2.10.5.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.10.5.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/guice-servlet-4.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/snakeyaml-1.26.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/jakarta.activation-api-1.2.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/metrics-core-3.2.4.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/json-io-2.5.1.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/jersey-client-1.19.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/jersey-guice-1.19.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/java-util-1.9.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-server-router-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-submarine-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-services-api-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-server-tests-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-api-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-client-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-server-common-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-common-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-services-core-3.2.3.jar:/usr/local/hadoop-3.2.3/share/hadoop/yarn/hadoop-yarn-registry-3.2.3.jar\n",
            "STARTUP_MSG:   build = https://github.com/apache/hadoop -r abe5358143720085498613d399be3bbf01e0f131; compiled by 'ubuntu' on 2022-03-20T01:18Z\n",
            "STARTUP_MSG:   java = 1.8.0_312\n",
            "************************************************************/\n",
            "2022-07-11 08:15:55,986 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]\n",
            "2022-07-11 08:15:56,139 INFO namenode.NameNode: createNameNode [-format]\n",
            "Formatting using clusterid: CID-c4705c4b-d58e-42c5-93e3-f89d053af68b\n",
            "2022-07-11 08:15:56,927 INFO namenode.FSEditLog: Edit logging is async:true\n",
            "2022-07-11 08:15:56,972 INFO namenode.FSNamesystem: KeyProvider: null\n",
            "2022-07-11 08:15:56,974 INFO namenode.FSNamesystem: fsLock is fair: true\n",
            "2022-07-11 08:15:56,975 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false\n",
            "2022-07-11 08:15:56,986 INFO namenode.FSNamesystem: fsOwner             = root (auth:SIMPLE)\n",
            "2022-07-11 08:15:56,986 INFO namenode.FSNamesystem: supergroup          = supergroup\n",
            "2022-07-11 08:15:56,986 INFO namenode.FSNamesystem: isPermissionEnabled = true\n",
            "2022-07-11 08:15:56,986 INFO namenode.FSNamesystem: HA Enabled: false\n",
            "2022-07-11 08:15:57,062 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling\n",
            "2022-07-11 08:15:57,096 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000\n",
            "2022-07-11 08:15:57,097 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true\n",
            "2022-07-11 08:15:57,103 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000\n",
            "2022-07-11 08:15:57,103 INFO blockmanagement.BlockManager: The block deletion will start around 2022 Jul 11 08:15:57\n",
            "2022-07-11 08:15:57,105 INFO util.GSet: Computing capacity for map BlocksMap\n",
            "2022-07-11 08:15:57,105 INFO util.GSet: VM type       = 64-bit\n",
            "2022-07-11 08:15:57,107 INFO util.GSet: 2.0% max memory 2.8 GB = 57.8 MB\n",
            "2022-07-11 08:15:57,107 INFO util.GSet: capacity      = 2^23 = 8388608 entries\n",
            "2022-07-11 08:15:57,120 INFO blockmanagement.BlockManager: Storage policy satisfier is disabled\n",
            "2022-07-11 08:15:57,120 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false\n",
            "2022-07-11 08:15:57,128 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.9990000128746033\n",
            "2022-07-11 08:15:57,128 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0\n",
            "2022-07-11 08:15:57,128 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000\n",
            "2022-07-11 08:15:57,129 INFO blockmanagement.BlockManager: defaultReplication         = 1\n",
            "2022-07-11 08:15:57,129 INFO blockmanagement.BlockManager: maxReplication             = 512\n",
            "2022-07-11 08:15:57,129 INFO blockmanagement.BlockManager: minReplication             = 1\n",
            "2022-07-11 08:15:57,129 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2\n",
            "2022-07-11 08:15:57,129 INFO blockmanagement.BlockManager: redundancyRecheckInterval  = 3000ms\n",
            "2022-07-11 08:15:57,129 INFO blockmanagement.BlockManager: encryptDataTransfer        = false\n",
            "2022-07-11 08:15:57,129 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000\n",
            "2022-07-11 08:15:57,156 INFO namenode.FSDirectory: GLOBAL serial map: bits=29 maxEntries=536870911\n",
            "2022-07-11 08:15:57,156 INFO namenode.FSDirectory: USER serial map: bits=24 maxEntries=16777215\n",
            "2022-07-11 08:15:57,156 INFO namenode.FSDirectory: GROUP serial map: bits=24 maxEntries=16777215\n",
            "2022-07-11 08:15:57,156 INFO namenode.FSDirectory: XATTR serial map: bits=24 maxEntries=16777215\n",
            "2022-07-11 08:15:57,171 INFO util.GSet: Computing capacity for map INodeMap\n",
            "2022-07-11 08:15:57,171 INFO util.GSet: VM type       = 64-bit\n",
            "2022-07-11 08:15:57,171 INFO util.GSet: 1.0% max memory 2.8 GB = 28.9 MB\n",
            "2022-07-11 08:15:57,171 INFO util.GSet: capacity      = 2^22 = 4194304 entries\n",
            "2022-07-11 08:15:57,176 INFO namenode.FSDirectory: ACLs enabled? false\n",
            "2022-07-11 08:15:57,176 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true\n",
            "2022-07-11 08:15:57,176 INFO namenode.FSDirectory: XAttrs enabled? true\n",
            "2022-07-11 08:15:57,177 INFO namenode.NameNode: Caching file names occurring more than 10 times\n",
            "2022-07-11 08:15:57,183 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536\n",
            "2022-07-11 08:15:57,185 INFO snapshot.SnapshotManager: SkipList is disabled\n",
            "2022-07-11 08:15:57,191 INFO util.GSet: Computing capacity for map cachedBlocks\n",
            "2022-07-11 08:15:57,192 INFO util.GSet: VM type       = 64-bit\n",
            "2022-07-11 08:15:57,192 INFO util.GSet: 0.25% max memory 2.8 GB = 7.2 MB\n",
            "2022-07-11 08:15:57,192 INFO util.GSet: capacity      = 2^20 = 1048576 entries\n",
            "2022-07-11 08:15:57,205 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10\n",
            "2022-07-11 08:15:57,205 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10\n",
            "2022-07-11 08:15:57,205 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25\n",
            "2022-07-11 08:15:57,210 INFO namenode.FSNamesystem: Retry cache on namenode is enabled\n",
            "2022-07-11 08:15:57,211 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis\n",
            "2022-07-11 08:15:57,213 INFO util.GSet: Computing capacity for map NameNodeRetryCache\n",
            "2022-07-11 08:15:57,213 INFO util.GSet: VM type       = 64-bit\n",
            "2022-07-11 08:15:57,213 INFO util.GSet: 0.029999999329447746% max memory 2.8 GB = 887.0 KB\n",
            "2022-07-11 08:15:57,213 INFO util.GSet: capacity      = 2^17 = 131072 entries\n",
            "2022-07-11 08:15:57,247 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1272403714-172.28.0.2-1657527357236\n",
            "2022-07-11 08:15:57,271 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted.\n",
            "2022-07-11 08:15:57,327 INFO namenode.FSImageFormatProtobuf: Saving image file /tmp/hadoop-root/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression\n",
            "2022-07-11 08:15:57,463 INFO namenode.FSImageFormatProtobuf: Image file /tmp/hadoop-root/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 396 bytes saved in 0 seconds .\n",
            "2022-07-11 08:15:57,485 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0\n",
            "2022-07-11 08:15:57,527 INFO namenode.FSNamesystem: Stopping services started for active state\n",
            "2022-07-11 08:15:57,528 INFO namenode.FSNamesystem: Stopping services started for standby state\n",
            "2022-07-11 08:15:57,540 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.\n",
            "2022-07-11 08:15:57,541 INFO namenode.NameNode: SHUTDOWN_MSG: \n",
            "/************************************************************\n",
            "SHUTDOWN_MSG: Shutting down NameNode at b6c4a8758417/172.28.0.2\n",
            "************************************************************/\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Hadoop scripts"
      ],
      "metadata": {
        "id": "bJYwmqrZN3Ql"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Hadoop comes with scripts for running commands, and starting and stopping daemons across the whole cluster. These scripts can be found in the bin and sbin directories"
      ],
      "metadata": {
        "id": "0frkp5aMV7n0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Exploring Hadoop scripts available in sbin directory\n",
        "!ls $HADOOP_HOME/sbin"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UPDf1gXRN2DH",
        "outputId": "f7782b81-2ef5-4be4-8c76-de2890402000"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "distribute-exclude.sh\t start-all.sh\t      stop-balancer.sh\n",
            "FederationStateStore\t start-balancer.sh    stop-dfs.cmd\n",
            "hadoop-daemon.sh\t start-dfs.cmd\t      stop-dfs.sh\n",
            "hadoop-daemons.sh\t start-dfs.sh\t      stop-secure-dns.sh\n",
            "httpfs.sh\t\t start-secure-dns.sh  stop-yarn.cmd\n",
            "kms.sh\t\t\t start-yarn.cmd       stop-yarn.sh\n",
            "mr-jobhistory-daemon.sh  start-yarn.sh\t      workers.sh\n",
            "refresh-namenodes.sh\t stop-all.cmd\t      yarn-daemon.sh\n",
            "start-all.cmd\t\t stop-all.sh\t      yarn-daemons.sh\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Creating other necessary enviroment variables before starting nodes\n",
        "os.environ[\"HDFS_NAMENODE_USER\"] = \"root\"\n",
        "os.environ[\"HDFS_DATANODE_USER\"] = \"root\"\n",
        "os.environ[\"HDFS_SECONDARYNAMENODE_USER\"] = \"root\"\n",
        "os.environ[\"YARN_RESOURCEMANAGER_USER\"] = \"root\"\n",
        "os.environ[\"YARN_NODEMANAGER_USER\"] = \"root\""
      ],
      "metadata": {
        "id": "oN-jeeavWFa5"
      },
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Launching hdfs deamons\n",
        "!$HADOOP_HOME/sbin/start-dfs.sh"
      ],
      "metadata": {
        "id": "U6METYYLTQD4",
        "outputId": "ce525491-bd77-495b-da12-af037cfb1f01",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Starting namenodes on [localhost]\n",
            "Starting datanodes\n",
            "Starting secondary namenodes [b6c4a8758417]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Listing the running deamons\n",
        "!jps"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "phffzEs0xiql",
        "outputId": "8145a803-5ece-42aa-e5b0-da0cf6efdf9b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1956 DataNode\n",
            "1829 NameNode\n",
            "2316 Jps\n",
            "2156 SecondaryNameNode\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Launching yarn deamons\n",
        "#nohup causes a process to ignore a SIGHUP signal\n",
        "!nohup $HADOOP_HOME/sbin/start-yarn.sh"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "51Ytj29pLxeT",
        "outputId": "19a882f9-eb99-4023-d9c9-55e27cedeefe"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "nohup: ignoring input and appending output to 'nohup.out'\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Listing the running deamons\n",
        "!jps"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e8T7Q5OiUQ8R",
        "outputId": "162b6779-becf-41c5-8723-66d98ff1b823"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2441 SecondaryNameNode\n",
            "1481 ResourceManager\n",
            "2235 DataNode\n",
            "2110 NameNode\n",
            "2974 Jps\n",
            "2831 NodeManager\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Monitoring Hadoop cluster with hadoop admin commands"
      ],
      "metadata": {
        "id": "NMHP25Pda5I0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Report the basic file system information and statistics\n",
        "!$HADOOP_HOME/bin/hdfs dfsadmin -report"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F4CeeemSl01c",
        "outputId": "f30f37c2-5189-476d-d45a-792cbeb0c573"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Configured Capacity: 242548072448 (225.89 GB)\n",
            "Present Capacity: 198672994304 (185.03 GB)\n",
            "DFS Remaining: 198672969728 (185.03 GB)\n",
            "DFS Used: 24576 (24 KB)\n",
            "DFS Used%: 0.00%\n",
            "Replicated Blocks:\n",
            "\tUnder replicated blocks: 0\n",
            "\tBlocks with corrupt replicas: 0\n",
            "\tMissing blocks: 0\n",
            "\tMissing blocks (with replication factor 1): 0\n",
            "\tLow redundancy blocks with highest priority to recover: 0\n",
            "\tPending deletion blocks: 0\n",
            "Erasure Coded Block Groups: \n",
            "\tLow redundancy block groups: 0\n",
            "\tBlock groups with corrupt internal blocks: 0\n",
            "\tMissing block groups: 0\n",
            "\tLow redundancy blocks with highest priority to recover: 0\n",
            "\tPending deletion blocks: 0\n",
            "\n",
            "-------------------------------------------------\n",
            "Live datanodes (1):\n",
            "\n",
            "Name: 127.0.0.1:9866 (localhost)\n",
            "Hostname: e26563ead3bd\n",
            "Decommission Status : Normal\n",
            "Configured Capacity: 242548072448 (225.89 GB)\n",
            "DFS Used: 24576 (24 KB)\n",
            "Non DFS Used: 43858300928 (40.85 GB)\n",
            "DFS Remaining: 198672969728 (185.03 GB)\n",
            "DFS Used%: 0.00%\n",
            "DFS Remaining%: 81.91%\n",
            "Configured Cache Capacity: 0 (0 B)\n",
            "Cache Used: 0 (0 B)\n",
            "Cache Remaining: 0 (0 B)\n",
            "Cache Used%: 100.00%\n",
            "Cache Remaining%: 0.00%\n",
            "Xceivers: 1\n",
            "Last contact: Sun Jul 10 18:25:43 UTC 2022\n",
            "Last Block Report: Sun Jul 10 18:24:17 UTC 2022\n",
            "Num of Blocks: 0\n",
            "\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Monitoring Hadoop cluster with the browser interface"
      ],
      "metadata": {
        "id": "hK-i-QXpK_-9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import output"
      ],
      "metadata": {
        "id": "7BUnNhnijIS6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#The namenode posts the general report on port 9870\n",
        "output.serve_kernel_port_as_window(9870)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "id": "r1HfI7XWjtQa",
        "outputId": "f0a47b38-0e8c-4da9-e50f-e20e937c508f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "(async (port, path, text, element) => {\n",
              "    if (!google.colab.kernel.accessAllowed) {\n",
              "      return;\n",
              "    }\n",
              "    element.appendChild(document.createTextNode(''));\n",
              "    const url = await google.colab.kernel.proxyPort(port);\n",
              "    const anchor = document.createElement('a');\n",
              "    anchor.href = new URL(path, url).toString();\n",
              "    anchor.target = '_blank';\n",
              "    anchor.setAttribute('data-href', url + path);\n",
              "    anchor.textContent = text;\n",
              "    element.appendChild(anchor);\n",
              "  })(9870, \"/\", \"https://localhost:9870/\", window.element)"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Running wordcount in pseudo-distributed mode"
      ],
      "metadata": {
        "id": "HswIl9TN19RQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Dowloading text example to use as input (if it has not been donwloaded yet)\n",
        "!wget -q https://www.mirrorservice.org/sites/ftp.ibiblio.org/pub/docs/books/gutenberg/1/0/101/101.txt"
      ],
      "metadata": {
        "id": "bnZ4nS_qcNiL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Creating directory in HDFS\n",
        "!$HADOOP_HOME/bin/hdfs dfs -mkdir /word_count\n",
        "#Coping file from local file system to HDFS\n",
        "!$HADOOP_HOME/bin/hdfs dfs -put /content/101.txt /word_count"
      ],
      "metadata": {
        "id": "X0Rdrw-E610_"
      },
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Exploring Hadoop folder\n",
        "!$HADOOP_HOME/bin/hdfs dfs -ls /word_count"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NHDDarRG7BK1",
        "outputId": "c1cd8098-ce77-4b35-abc5-06eb4d554925"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found 1 items\n",
            "-rw-r--r--   1 root supergroup     678064 2022-07-10 18:27 /word_count/101.txt\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Running MapReduce program wordcount \n",
        "!$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.3.jar wordcount /word_count/101.txt /word_count/output/"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yhQOhKonnKsL",
        "outputId": "b2d7bcb2-e470-4c91-8ccb-ed2a7501444a"
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2022-07-11 08:18:39,387 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032\n",
            "2022-07-11 08:18:40,038 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1657527371635_0001\n",
            "2022-07-11 08:18:40,427 INFO input.FileInputFormat: Total input files to process : 1\n",
            "2022-07-11 08:18:40,563 INFO mapreduce.JobSubmitter: number of splits:1\n",
            "2022-07-11 08:18:41,208 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1657527371635_0001\n",
            "2022-07-11 08:18:41,209 INFO mapreduce.JobSubmitter: Executing with tokens: []\n",
            "2022-07-11 08:18:41,484 INFO conf.Configuration: resource-types.xml not found\n",
            "2022-07-11 08:18:41,485 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.\n",
            "2022-07-11 08:18:41,876 INFO impl.YarnClientImpl: Submitted application application_1657527371635_0001\n",
            "2022-07-11 08:18:41,978 INFO mapreduce.Job: The url to track the job: http://b6c4a8758417:8088/proxy/application_1657527371635_0001/\n",
            "2022-07-11 08:18:41,979 INFO mapreduce.Job: Running job: job_1657527371635_0001\n",
            "2022-07-11 08:18:52,239 INFO mapreduce.Job: Job job_1657527371635_0001 running in uber mode : false\n",
            "2022-07-11 08:18:52,240 INFO mapreduce.Job:  map 0% reduce 0%\n",
            "2022-07-11 08:18:58,339 INFO mapreduce.Job:  map 100% reduce 0%\n",
            "2022-07-11 08:19:04,453 INFO mapreduce.Job:  map 100% reduce 100%\n",
            "2022-07-11 08:19:05,477 INFO mapreduce.Job: Job job_1657527371635_0001 completed successfully\n",
            "2022-07-11 08:19:05,613 INFO mapreduce.Job: Counters: 54\n",
            "\tFile System Counters\n",
            "\t\tFILE: Number of bytes read=318005\n",
            "\t\tFILE: Number of bytes written=1108655\n",
            "\t\tFILE: Number of read operations=0\n",
            "\t\tFILE: Number of large read operations=0\n",
            "\t\tFILE: Number of write operations=0\n",
            "\t\tHDFS: Number of bytes read=678169\n",
            "\t\tHDFS: Number of bytes written=233636\n",
            "\t\tHDFS: Number of read operations=8\n",
            "\t\tHDFS: Number of large read operations=0\n",
            "\t\tHDFS: Number of write operations=2\n",
            "\t\tHDFS: Number of bytes read erasure-coded=0\n",
            "\tJob Counters \n",
            "\t\tLaunched map tasks=1\n",
            "\t\tLaunched reduce tasks=1\n",
            "\t\tData-local map tasks=1\n",
            "\t\tTotal time spent by all maps in occupied slots (ms)=4232\n",
            "\t\tTotal time spent by all reduces in occupied slots (ms)=3780\n",
            "\t\tTotal time spent by all map tasks (ms)=4232\n",
            "\t\tTotal time spent by all reduce tasks (ms)=3780\n",
            "\t\tTotal vcore-milliseconds taken by all map tasks=4232\n",
            "\t\tTotal vcore-milliseconds taken by all reduce tasks=3780\n",
            "\t\tTotal megabyte-milliseconds taken by all map tasks=4333568\n",
            "\t\tTotal megabyte-milliseconds taken by all reduce tasks=3870720\n",
            "\tMap-Reduce Framework\n",
            "\t\tMap input records=13006\n",
            "\t\tMap output records=105488\n",
            "\t\tMap output bytes=1078906\n",
            "\t\tMap output materialized bytes=318005\n",
            "\t\tInput split bytes=105\n",
            "\t\tCombine input records=105488\n",
            "\t\tCombine output records=21438\n",
            "\t\tReduce input groups=21438\n",
            "\t\tReduce shuffle bytes=318005\n",
            "\t\tReduce input records=21438\n",
            "\t\tReduce output records=21438\n",
            "\t\tSpilled Records=42876\n",
            "\t\tShuffled Maps =1\n",
            "\t\tFailed Shuffles=0\n",
            "\t\tMerged Map outputs=1\n",
            "\t\tGC time elapsed (ms)=165\n",
            "\t\tCPU time spent (ms)=3570\n",
            "\t\tPhysical memory (bytes) snapshot=518680576\n",
            "\t\tVirtual memory (bytes) snapshot=5187850240\n",
            "\t\tTotal committed heap usage (bytes)=529530880\n",
            "\t\tPeak Map Physical memory (bytes)=295497728\n",
            "\t\tPeak Map Virtual memory (bytes)=2596868096\n",
            "\t\tPeak Reduce Physical memory (bytes)=223182848\n",
            "\t\tPeak Reduce Virtual memory (bytes)=2590982144\n",
            "\tShuffle Errors\n",
            "\t\tBAD_ID=0\n",
            "\t\tCONNECTION=0\n",
            "\t\tIO_ERROR=0\n",
            "\t\tWRONG_LENGTH=0\n",
            "\t\tWRONG_MAP=0\n",
            "\t\tWRONG_REDUCE=0\n",
            "\tFile Input Format Counters \n",
            "\t\tBytes Read=678064\n",
            "\tFile Output Format Counters \n",
            "\t\tBytes Written=233636\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Exploring the created output directory\n",
        "#part-r-00000 contains the actual ouput\n",
        "!$HADOOP_HOME/bin/hdfs dfs -ls /word_count/output"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GZAT1WmQ3oGM",
        "outputId": "7c0a2e6b-ffd7-47da-b418-94295b318d02"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found 2 items\n",
            "-rw-r--r--   1 root supergroup          0 2022-07-10 18:29 /word_count/output/_SUCCESS\n",
            "-rw-r--r--   1 root supergroup     233636 2022-07-10 18:29 /word_count/output/part-r-00000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Printing out first 50 lines\n",
        "!$HADOOP_HOME/bin/hdfs dfs -cat /word_count/output/part-r-00000 | head -50"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "q37LukH09Jx1",
        "outputId": "bf5c42fc-f4c2-46d0-91b5-4f6a23045409"
      },
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\"(d)\t1\n",
            "\"/H\"\t1\n",
            "\"0\"\t1\n",
            "\"02\"\t2\n",
            "\"1.\t1\n",
            "\"414\t3\n",
            "\"A\t2\n",
            "\"AT&T\t3\n",
            "\"AT&T's\t1\n",
            "\"Access\t1\n",
            "\"Acid\t3\n",
            "\"Ad-hocracy\"\t1\n",
            "\"Advanced\t1\n",
            "\"Agents\t1\n",
            "\"Al\t3\n",
            "\"All\t3\n",
            "\"American\t1\n",
            "\"An\t1\n",
            "\"And\t2\n",
            "\"Any\t1\n",
            "\"Are\t3\n",
            "\"Artificial\t1\n",
            "\"As\t1\n",
            "\"Assistant\t1\n",
            "\"Attctc\"\t2\n",
            "\"Auld\t1\n",
            "\"Autodesk,\"\t1\n",
            "\"BBS,\"\t2\n",
            "\"BIRTHPLACE\t1\n",
            "\"BRITS\t1\n",
            "\"Barry\t1\n",
            "\"Because\t1\n",
            "\"Before\t1\n",
            "\"Bell\t2\n",
            "\"Bell\"\t1\n",
            "\"BellSouth\t2\n",
            "\"Berkeley\t1\n",
            "\"Big\t1\n",
            "\"Biggest\t1\n",
            "\"Black\t3\n",
            "\"Blue\t1\n",
            "\"Bob\"\t1\n",
            "\"Bob,\t3\n",
            "\"Bob:\t1\n",
            "\"Bullet-N-Board.\"\t1\n",
            "\"Bureaucrat-ese\t1\n",
            "\"But\t4\n",
            "\"C-word.\"\t1\n",
            "\"CALIFORNIA\"\t1\n",
            "\"CC,\"\t1\n",
            "cat: Unable to write to output stream.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Hadoop Streaming Using Python"
      ],
      "metadata": {
        "id": "vaGFzgmIzLLi"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Hadoop Streaming is a feature that comes with Hadoop and allows users or developers to use various different languages for writing **MapReduce** programs like Python, C++, Ruby, etc.\n",
        "\n",
        "The utility will create a Map/Reduce job, submit the job to an appropriate cluster, and monitor the progress of the job until it completes."
      ],
      "metadata": {
        "id": "RJXp5YpCzI22"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Exploring Hadoop utilities available\n",
        "!ls $HADOOP_HOME/share/hadoop/tools/lib/"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SuR_rvlR0eWs",
        "outputId": "5339163e-adb2-49ab-8be9-bce490eb1b15"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "aliyun-java-sdk-core-4.5.10.jar      hadoop-gridmix-3.2.3.jar\n",
            "aliyun-java-sdk-kms-2.11.0.jar\t     hadoop-kafka-3.2.3.jar\n",
            "aliyun-java-sdk-ram-3.1.0.jar\t     hadoop-openstack-3.2.3.jar\n",
            "aliyun-sdk-oss-3.13.0.jar\t     hadoop-resourceestimator-3.2.3.jar\n",
            "aws-java-sdk-bundle-1.11.901.jar     hadoop-rumen-3.2.3.jar\n",
            "azure-data-lake-store-sdk-2.2.9.jar  hadoop-sls-3.2.3.jar\n",
            "azure-keyvault-core-1.0.0.jar\t     hadoop-streaming-3.2.3.jar\n",
            "azure-storage-7.0.0.jar\t\t     ini4j-0.5.4.jar\n",
            "hadoop-aliyun-3.2.3.jar\t\t     jdom2-2.0.6.jar\n",
            "hadoop-archive-logs-3.2.3.jar\t     kafka-clients-2.8.1.jar\n",
            "hadoop-archives-3.2.3.jar\t     lz4-java-1.7.1.jar\n",
            "hadoop-aws-3.2.3.jar\t\t     ojalgo-43.0.jar\n",
            "hadoop-azure-3.2.3.jar\t\t     opentracing-api-0.33.0.jar\n",
            "hadoop-azure-datalake-3.2.3.jar      opentracing-noop-0.33.0.jar\n",
            "hadoop-datajoin-3.2.3.jar\t     opentracing-util-0.33.0.jar\n",
            "hadoop-distcp-3.2.3.jar\t\t     org.jacoco.agent-0.8.5-runtime.jar\n",
            "hadoop-extras-3.2.3.jar\t\t     wildfly-openssl-1.0.7.Final.jar\n",
            "hadoop-fs2img-3.2.3.jar\t\t     zstd-jni-1.4.9-1.jar\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Dowloading text example to use as input (if it has not been donwloaded yet)\n",
        "!wget -q https://www.mirrorservice.org/sites/ftp.ibiblio.org/pub/docs/books/gutenberg/1/0/101/101.txt"
      ],
      "metadata": {
        "id": "wk3DCuzh3hkw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Creating directory in HDFS\n",
        "!$HADOOP_HOME/bin/hdfs dfs -mkdir /word_count_with_python"
      ],
      "metadata": {
        "id": "DnDoNgGxj-kB"
      },
      "execution_count": 45,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Copying the file from local file system to Hadoop distributed file system (HDFS)\n",
        "!$HADOOP_HOME/bin/hdfs dfs -put /content/101.txt /word_count_with_python"
      ],
      "metadata": {
        "id": "ELSi8ljdCfyc"
      },
      "execution_count": 46,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Mapper"
      ],
      "metadata": {
        "id": "x92pOYK6zxZT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The mapper is an executable that reads all input records from a file/s and generates an output in the form of key-value pairs which works as input for the Reducer."
      ],
      "metadata": {
        "id": "tIceSP8y-crR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile mapper.py\n",
        "\n",
        "#!/usr/bin/env python\n",
        "\n",
        "#'#!' is known as shebang and used for interpreting the script\n",
        "\n",
        "# import sys because we need to read and write data to STDIN and STDOUT\n",
        "import sys\n",
        "  \n",
        "# reading entire line from STDIN (standard input)\n",
        "for line in sys.stdin:\n",
        "  # to remove leading and trailing whitespace\n",
        "  line = line.strip()\n",
        "  # split the line into words\n",
        "  words = line.split()\n",
        "    \n",
        "  # we are looping over the words array and printing the word\n",
        "  # with the count of 1 to the STDOUT\n",
        "  for word in words:\n",
        "    # write the results to STDOUT (standard output);\n",
        "    # what we output here will be the input for the\n",
        "    # Reduce step, i.e. the input for reducer.py\n",
        "    print('%s\\t%s' % (word, 1))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dlrxI4_0zypI",
        "outputId": "b1ef7deb-b6b0-42b1-92b5-91a5fefcf7de"
      },
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Writing mapper.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Reducer"
      ],
      "metadata": {
        "id": "9ssZYFBq_HtK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The reducer is an executable that reads all the intermediate key-value pairs generated by the mapper and generates a final output as a result of a computation operation like addition, filtration, and aggregation.\n",
        "\n",
        "Both the mapper and the reducer read the input from stdin (line by line) and emit the output to stdout."
      ],
      "metadata": {
        "id": "iu0p-9g7BEvr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile reducer.py\n",
        "\n",
        "#!/usr/bin/env python\n",
        "  \n",
        "from operator import itemgetter\n",
        "import sys\n",
        "  \n",
        "current_word = None\n",
        "current_count = 0\n",
        "word = None\n",
        "  \n",
        "# read the entire line from STDIN\n",
        "for line in sys.stdin:\n",
        "  # remove leading and trailing whitespace\n",
        "  line = line.strip()\n",
        "  # splitting the data on the basis of tab we have provided in mapper.py\n",
        "  word, count = line.split('\\t', 1)\n",
        "  # convert count (currently a string) to int\n",
        "  try:\n",
        "    count = int(count)\n",
        "  except ValueError:\n",
        "    # count was not a number, so silently\n",
        "    # ignore/discard this line\n",
        "    continue\n",
        "  \n",
        "  # this IF-switch only works because Hadoop sorts map output\n",
        "  # by key (here: word) before it is passed to the reducer\n",
        "  if current_word == word:\n",
        "    current_count += count\n",
        "  else:\n",
        "    if current_word: #to not print current_word=None\n",
        "      # write result to STDOUT\n",
        "      print('%s\\t%s' % (current_word, current_count))\n",
        "    current_count = count\n",
        "    current_word = word\n",
        "  \n",
        "# do not forget to output the last word if needed!\n",
        "if current_word == word:\n",
        "  print('%s\\t%s' % (current_word, current_count))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XHy3YQiN_Ogi",
        "outputId": "7edf564d-11d2-4c50-a737-670cef12dd6e"
      },
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Writing reducer.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Testing our MapReduce job locally (Hadoop does not participate here)\n",
        "!cat 101.txt | python mapper.py | sort -k1,1 | python reducer.py | head -50\n",
        "#We apply sorting after the mapper because it is the default operation in MapReduce architecture"
      ],
      "metadata": {
        "id": "0vpHJ3OA_4aI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d7648fbf-e78e-438d-e8cf-e09202c2ab5e"
      },
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "~~~~~~~~~~~~~~\t1\n",
            "~~~~~~~~~~~~~~~~~~~~~\t2\n",
            "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\t1\n",
            "=\t2\n",
            "_____________________________________\t1\n",
            "-\t39\n",
            "-----------\t1\n",
            "-------------\t1\n",
            "---------------\t1\n",
            "----------------------\t1\n",
            "----------------------------\t1\n",
            "---------------------------------\t1\n",
            "-----------------------------------\t1\n",
            ":\t1\n",
            ".\t271\n",
            ".'\t1\n",
            ".\"\t9\n",
            ".)\t8\n",
            "(.\t8\n",
            "@\t1\n",
            "**\t4\n",
            "***\t8\n",
            "*****\t2\n",
            "&\t10\n",
            "#\t31\n",
            "##\t2\n",
            "##!\t1\n",
            "+\t2\n",
            "+------------+\t2\n",
            "\"0\"\t1\n",
            "0.\t1\n",
            "00,\t1\n",
            "01,\t1\n",
            "\"02\"\t2\n",
            "02,\t1\n",
            "03,\t1\n",
            "04,\t1\n",
            "04,\"\t1\n",
            "05,\t1\n",
            "0-553-08058-X,\t1\n",
            "0-553-56370-X.\t1\n",
            "06-11-91\t53\n",
            "08-03-90\t1\n",
            "1\t4\n",
            "\"1.\t1\n",
            "($1\t1\n",
            "1,\t5\n",
            "1:\t1\n",
            "1.\t6\n",
            "1)\t1\n",
            "Traceback (most recent call last):\n",
            "  File \"reducer.py\", line 32, in <module>\n",
            "    print('%s\\t%s' % (current_word, current_count))\n",
            "BrokenPipeError: [Errno 32] Broken pipe\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Changing the permissions of the files\n",
        "!chmod 777 /content/mapper.py /content/reducer.py\n",
        "#Setting 777 permissions to a file or directory means that it will be readable, writable and executable by all users"
      ],
      "metadata": {
        "id": "3Vvg_0vcAD-8"
      },
      "execution_count": 50,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Running MapReduce programs\n",
        "!$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-3.2.3.jar \\\n",
        "  -input /word_count_with_python/101.txt \\\n",
        "  -output /word_count_with_python/output \\\n",
        "  -mapper \"python /content/mapper.py\" \\\n",
        "  -reducer \"python /content/reducer.py\""
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kVxn2MR01DSy",
        "outputId": "9792fe8b-eef4-4c47-d7b0-48c5ae76ef22"
      },
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "packageJobJar: [/tmp/hadoop-unjar4785058036501520475/] [] /tmp/streamjob5169551799259333114.jar tmpDir=null\n",
            "2022-07-11 08:22:15,394 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032\n",
            "2022-07-11 08:22:15,724 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032\n",
            "2022-07-11 08:22:16,087 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1657527371635_0002\n",
            "2022-07-11 08:22:16,442 INFO mapred.FileInputFormat: Total input files to process : 1\n",
            "2022-07-11 08:22:16,545 INFO mapreduce.JobSubmitter: number of splits:2\n",
            "2022-07-11 08:22:16,774 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1657527371635_0002\n",
            "2022-07-11 08:22:16,776 INFO mapreduce.JobSubmitter: Executing with tokens: []\n",
            "2022-07-11 08:22:17,117 INFO conf.Configuration: resource-types.xml not found\n",
            "2022-07-11 08:22:17,118 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.\n",
            "2022-07-11 08:22:17,215 INFO impl.YarnClientImpl: Submitted application application_1657527371635_0002\n",
            "2022-07-11 08:22:17,263 INFO mapreduce.Job: The url to track the job: http://b6c4a8758417:8088/proxy/application_1657527371635_0002/\n",
            "2022-07-11 08:22:17,264 INFO mapreduce.Job: Running job: job_1657527371635_0002\n",
            "2022-07-11 08:22:27,491 INFO mapreduce.Job: Job job_1657527371635_0002 running in uber mode : false\n",
            "2022-07-11 08:22:27,493 INFO mapreduce.Job:  map 0% reduce 0%\n",
            "2022-07-11 08:22:37,671 INFO mapreduce.Job:  map 50% reduce 0%\n",
            "2022-07-11 08:22:38,690 INFO mapreduce.Job:  map 100% reduce 0%\n",
            "2022-07-11 08:22:44,758 INFO mapreduce.Job:  map 100% reduce 100%\n",
            "2022-07-11 08:22:45,781 INFO mapreduce.Job: Job job_1657527371635_0002 completed successfully\n",
            "2022-07-11 08:22:45,905 INFO mapreduce.Job: Counters: 55\n",
            "\tFile System Counters\n",
            "\t\tFILE: Number of bytes read=1078912\n",
            "\t\tFILE: Number of bytes written=2873179\n",
            "\t\tFILE: Number of read operations=0\n",
            "\t\tFILE: Number of large read operations=0\n",
            "\t\tFILE: Number of write operations=0\n",
            "\t\tHDFS: Number of bytes read=682368\n",
            "\t\tHDFS: Number of bytes written=233636\n",
            "\t\tHDFS: Number of read operations=11\n",
            "\t\tHDFS: Number of large read operations=0\n",
            "\t\tHDFS: Number of write operations=2\n",
            "\t\tHDFS: Number of bytes read erasure-coded=0\n",
            "\tJob Counters \n",
            "\t\tKilled map tasks=1\n",
            "\t\tLaunched map tasks=2\n",
            "\t\tLaunched reduce tasks=1\n",
            "\t\tData-local map tasks=2\n",
            "\t\tTotal time spent by all maps in occupied slots (ms)=16968\n",
            "\t\tTotal time spent by all reduces in occupied slots (ms)=4462\n",
            "\t\tTotal time spent by all map tasks (ms)=16968\n",
            "\t\tTotal time spent by all reduce tasks (ms)=4462\n",
            "\t\tTotal vcore-milliseconds taken by all map tasks=16968\n",
            "\t\tTotal vcore-milliseconds taken by all reduce tasks=4462\n",
            "\t\tTotal megabyte-milliseconds taken by all map tasks=17375232\n",
            "\t\tTotal megabyte-milliseconds taken by all reduce tasks=4569088\n",
            "\tMap-Reduce Framework\n",
            "\t\tMap input records=13006\n",
            "\t\tMap output records=105488\n",
            "\t\tMap output bytes=867930\n",
            "\t\tMap output materialized bytes=1078918\n",
            "\t\tInput split bytes=208\n",
            "\t\tCombine input records=0\n",
            "\t\tCombine output records=0\n",
            "\t\tReduce input groups=21438\n",
            "\t\tReduce shuffle bytes=1078918\n",
            "\t\tReduce input records=105488\n",
            "\t\tReduce output records=21438\n",
            "\t\tSpilled Records=210976\n",
            "\t\tShuffled Maps =2\n",
            "\t\tFailed Shuffles=0\n",
            "\t\tMerged Map outputs=2\n",
            "\t\tGC time elapsed (ms)=426\n",
            "\t\tCPU time spent (ms)=4870\n",
            "\t\tPhysical memory (bytes) snapshot=782053376\n",
            "\t\tVirtual memory (bytes) snapshot=7762931712\n",
            "\t\tTotal committed heap usage (bytes)=755499008\n",
            "\t\tPeak Map Physical memory (bytes)=322330624\n",
            "\t\tPeak Map Virtual memory (bytes)=2587197440\n",
            "\t\tPeak Reduce Physical memory (bytes)=178343936\n",
            "\t\tPeak Reduce Virtual memory (bytes)=2589908992\n",
            "\tShuffle Errors\n",
            "\t\tBAD_ID=0\n",
            "\t\tCONNECTION=0\n",
            "\t\tIO_ERROR=0\n",
            "\t\tWRONG_LENGTH=0\n",
            "\t\tWRONG_MAP=0\n",
            "\t\tWRONG_REDUCE=0\n",
            "\tFile Input Format Counters \n",
            "\t\tBytes Read=682160\n",
            "\tFile Output Format Counters \n",
            "\t\tBytes Written=233636\n",
            "2022-07-11 08:22:45,906 INFO streaming.StreamJob: Output directory: /word_count_with_python/output\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Exploring the created output directory\n",
        "#part-r-00000 contains the actual ouput\n",
        "!$HADOOP_HOME/bin/hdfs dfs -ls /word_count_with_python/output"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SwBVQ8JcIJC3",
        "outputId": "d7eeade9-21a1-479f-ba9b-386a741a53fa"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found 2 items\n",
            "-rw-r--r--   1 root supergroup          0 2022-07-10 18:42 /word_count_with_python/output/_SUCCESS\n",
            "-rw-r--r--   1 root supergroup     233636 2022-07-10 18:42 /word_count_with_python/output/part-00000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Printing out first 50 lines\n",
        "!$HADOOP_HOME/bin/hdfs dfs -cat /word_count_with_python/output/part-00000 | head -50"
      ],
      "metadata": {
        "id": "y9R4KRfY7d9Q",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8ec1e71d-95aa-433f-c3f7-962b3cc6763d"
      },
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\"(d)\t1\n",
            "\"/H\"\t1\n",
            "\"0\"\t1\n",
            "\"02\"\t2\n",
            "\"1.\t1\n",
            "\"414\t3\n",
            "\"A\t2\n",
            "\"AT&T\t3\n",
            "\"AT&T's\t1\n",
            "\"Access\t1\n",
            "\"Acid\t3\n",
            "\"Ad-hocracy\"\t1\n",
            "\"Advanced\t1\n",
            "\"Agents\t1\n",
            "\"Al\t3\n",
            "\"All\t3\n",
            "\"American\t1\n",
            "\"An\t1\n",
            "\"And\t2\n",
            "\"Any\t1\n",
            "\"Are\t3\n",
            "\"Artificial\t1\n",
            "\"As\t1\n",
            "\"Assistant\t1\n",
            "\"Attctc\"\t2\n",
            "\"Auld\t1\n",
            "\"Autodesk,\"\t1\n",
            "\"BBS,\"\t2\n",
            "\"BIRTHPLACE\t1\n",
            "\"BRITS\t1\n",
            "\"Barry\t1\n",
            "\"Because\t1\n",
            "\"Before\t1\n",
            "\"Bell\t2\n",
            "\"Bell\"\t1\n",
            "\"BellSouth\t2\n",
            "\"Berkeley\t1\n",
            "\"Big\t1\n",
            "\"Biggest\t1\n",
            "\"Black\t3\n",
            "\"Blue\t1\n",
            "\"Bob\"\t1\n",
            "\"Bob,\t3\n",
            "\"Bob:\t1\n",
            "\"Bullet-N-Board.\"\t1\n",
            "\"Bureaucrat-ese\t1\n",
            "\"But\t4\n",
            "\"C-word.\"\t1\n",
            "\"CALIFORNIA\"\t1\n",
            "\"CC,\"\t1\n",
            "cat: Unable to write to output stream.\n"
          ]
        }
      ]
    }
  ]
}