Hadoop_Setting_up_a_Single_Node_Cluster.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "oUuQjW2oNMcJ",
        "qFfOrktMPq8M",
        "KE7kSYSXQYLf",
        "lGI4TNXPamMr",
        "RlgP1ytnRtUK",
        "KLmxLQeJSb4A",
        "kXbSKFyeMqr2",
        "k2-Fdp73cF0V",
        "xMrEiLB_VAeR",
        "CKRRbwDFv3ZQ",
        "G3KBe4R65bl1",
        "yVJA-3jSATGV",
        "BbosNo0TD3oH",
        "Sam22f-YT1xR",
        "IF6-Z5RotAcO"
      ],
      "toc_visible": true,
      "authorship_tag": "ABX9TyPAqughUbPLy2YH190+NViV",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/groda/big_data/blob/master/Hadoop_Setting_up_a_Single_Node_Cluster.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a href=\"https://github.com/groda/big_data\"><div><img src=\"https://github.com/groda/big_data/blob/master/logo_bdb.png?raw=true\" align=right width=\"90\"></div></a>\n",
        "\n",
        "# HDFS and MapReduce on a single-node Hadoop cluster\n",
        "<br>\n",
        "<br>\n",
        "\n",
        "In this tutorial/notebook we'll showcase the setup of a single-node cluster, following the guidelines outlined on https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html. Subsequently, we'll demonstrate the seamless execution of elementary HDFS and MapReduce commands.\n",
        "\n",
        "Upon downloading the software, several preliminary steps must be taken, including setting environment variables, generating SSH keys, and more. To streamline these tasks, we've consolidated them under the \"Prologue\" section.\n",
        "\n",
        "Upon completion of the prologue, we can launch a single-node Hadoop cluster on the current virtual machine.\n",
        "\n",
        "Following that, we'll execute a series of test HDFS commands and MapReduce jobs on the Hadoop cluster. These will be performed using a dataset sourced from a publicly available collection.\n",
        "\n",
        "Finally, we'll proceed to shut down the cluster.\n"
      ],
      "metadata": {
        "id": "oEF3qldGPj3T"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**TABLE OF CONTENTS**\n",
        "* **[Prologue](#scrollTo=oUuQjW2oNMcJ)**\n",
        "\n",
        " * [Check the available Java version](#scrollTo=qFfOrktMPq8M)\n",
        "\n",
        " * [Download core Hadoop](#scrollTo=KE7kSYSXQYLf)\n",
        "\n",
        "   * [Verify the downloaded file](#scrollTo=lGI4TNXPamMr)\n",
        "\n",
        " * [Configure `PATH`](#scrollTo=RlgP1ytnRtUK)\n",
        "\n",
        " * [Configure `core-site.xml` and `hdfs-site.xml`](#scrollTo=KLmxLQeJSb4A)\n",
        "\n",
        " * [Set environment variables](#scrollTo=kXbSKFyeMqr2)\n",
        "\n",
        " * [Setup localhost access via SSH key](#scrollTo=k2-Fdp73cF0V)\n",
        "\n",
        "   * [Install `openssh` and start server](#scrollTo=-Uxmv3RdUwiF)\n",
        "\n",
        "   * [Generate key](#scrollTo=PYKoSlaENuyG)\n",
        "\n",
        "   * [Check SSH connection to localhost](#scrollTo=FwA6rKpScnVi)\n",
        "\n",
        "* **[Launch a single-node Hadoop cluster](#scrollTo=V68C4cDySyek)**\n",
        "\n",
        "   * [Initialize the namenode](#scrollTo=HTDPwnVlSbHS)\n",
        "\n",
        "   * [Start cluster](#scrollTo=xMrEiLB_VAeR)\n",
        "\n",
        "* **[Run some simple HDFS commands](#scrollTo=CKRRbwDFv3ZQ)**\n",
        "\n",
        "* **[Run some simple MapReduce jobs](#scrollTo=G3KBe4R65bl1)**\n",
        "\n",
        "   * [Simplest MapReduce job](#scrollTo=yVJA-3jSATGV)\n",
        "\n",
        "   * [Another MapReduce example: filter a log file](#scrollTo=BbosNo0TD3oH)\n",
        "\n",
        "   * [Aggregate data with MapReduce](#scrollTo=Sam22f-YT1xR)\n",
        "\n",
        "* **[Stop cluster](#scrollTo=IF6-Z5RotAcO)**\n",
        "\n",
        "* **[Concluding remarks](#scrollTo=w5N7tb0HSbZB)**\n",
        "\n"
      ],
      "metadata": {
        "colab_type": "toc",
        "id": "hGm3LhVEWXr9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Prologue"
      ],
      "metadata": {
        "id": "oUuQjW2oNMcJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Check the available Java version\n",
        " Apache Hadoop 3.3 and upper supports Java 8 and Java 11 (runtime only). See: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions\n"
      ],
      "metadata": {
        "id": "qFfOrktMPq8M"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check if Java version is one of `8`, `11`"
      ],
      "metadata": {
        "id": "EuWqBiV89ryq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!java -version"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "C7X0EZaMPrsD",
        "outputId": "fb676d56-1b1f-4f7b-f1fa-11725c737ebd"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "openjdk version \"11.0.22\" 2024-01-16\n",
            "OpenJDK Runtime Environment (build 11.0.22+7-post-Ubuntu-0ubuntu222.04.1)\n",
            "OpenJDK 64-Bit Server VM (build 11.0.22+7-post-Ubuntu-0ubuntu222.04.1, mixed mode, sharing)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "JAVA_MAJOR_VERSION=$(java -version 2>&1 | grep -m1 -Po '(\\d+\\.)+\\d+' | cut -d '.' -f1)\n",
        "if [[ $JAVA_MAJOR_VERSION -eq 8 || $JAVA_MAJOR_VERSION -eq 11 ]]\n",
        " then\n",
        " echo \"Java version is one of 8, 11 ✓\"\n",
        " fi"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lABuOV124G4x",
        "outputId": "26e43b27-8aea-4168-9ca7-ddc0d1d1f83e"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Java version is one of 8, 11 ✓\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Find the variable for the environment variable `JAVA_HOME`"
      ],
      "metadata": {
        "id": "pWROofISgKKW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Find the path for the environment variable `JAVA_HOME`"
      ],
      "metadata": {
        "id": "uH4AGbkLP3iK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!readlink -f $(which java)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mCmk5GOqv0Y-",
        "outputId": "5307ce23-a53e-4172-fda4-b06643241114"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/usr/lib/jvm/java-11-openjdk-amd64/bin/java\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Extract JAVA_HOME from the Java path by removing the `bin/java` part in the end"
      ],
      "metadata": {
        "id": "rGHKH3Vu9Nwl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "JAVA_HOME=$(readlink -f $(which java) | sed 's/\\/bin\\/java$//')\n",
        "echo $JAVA_HOME"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Dd7en2Cv68ce",
        "outputId": "ddccfc2a-8e97-468b-9f54-be5349bc6e96"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/usr/lib/jvm/java-11-openjdk-amd64\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We're going to use `JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64`.\n",
        "\n",
        "Use `%env%` to set the variable for the current notebook session."
      ],
      "metadata": {
        "id": "Hck9zJ3kQK2X"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%env JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
      ],
      "metadata": {
        "id": "P88xcreEQBcx",
        "outputId": "34e70041-1087-4f82-ca0d-bdfb0315564d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "env: JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Download core Hadoop\n",
        "Download the latest stable version of the core Hadoop distribution from one of the download mirrors locations https://www.apache.org/dyn/closer.cgi/hadoop/common/.\n",
        "\n",
        "**Note** with the option `--no-clobber`, `wget` will not download the file if it already exists."
      ],
      "metadata": {
        "id": "KE7kSYSXQYLf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!wget --no-clobber https://dlcdn.apache.org/hadoop/common/stable/hadoop-3.4.0.tar.gz"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "54LqS5Rkgyli",
        "outputId": "cb9f9d6a-d504-479c-83b7-453af521fd61"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2024-05-05 15:48:28--  https://dlcdn.apache.org/hadoop/common/stable/hadoop-3.4.0.tar.gz\n",
            "Resolving dlcdn.apache.org (dlcdn.apache.org)... 151.101.2.132, 2a04:4e42::644\n",
            "Connecting to dlcdn.apache.org (dlcdn.apache.org)|151.101.2.132|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 965537117 (921M) [application/x-gzip]\n",
            "Saving to: ‘hadoop-3.4.0.tar.gz’\n",
            "\n",
            "hadoop-3.4.0.tar.gz 100%[===================>] 920.81M   204MB/s    in 4.1s    \n",
            "\n",
            "2024-05-05 15:48:52 (224 MB/s) - ‘hadoop-3.4.0.tar.gz’ saved [965537117/965537117]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Uncompress archive"
      ],
      "metadata": {
        "id": "um2CARkgg22j"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!tar xzf hadoop-3.4.0.tar.gz"
      ],
      "metadata": {
        "id": "C17WYI0mQRE8"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Verify the downloaded file\n",
        "\n",
        "(see https://www.apache.org/dyn/closer.cgi/hadoop/common/)"
      ],
      "metadata": {
        "id": "lGI4TNXPamMr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Download sha512 file"
      ],
      "metadata": {
        "id": "ATofMJRXhJ4K"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! wget --no-clobber https://dlcdn.apache.org/hadoop/common/stable/hadoop-3.4.0.tar.gz.sha512"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NhTinHLqCrFQ",
        "outputId": "e10b29b8-1b32-487c-eceb-a23b268fc771"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2024-05-05 15:49:12--  https://dlcdn.apache.org/hadoop/common/stable/hadoop-3.4.0.tar.gz.sha512\n",
            "Resolving dlcdn.apache.org (dlcdn.apache.org)... 151.101.2.132, 2a04:4e42::644\n",
            "Connecting to dlcdn.apache.org (dlcdn.apache.org)|151.101.2.132|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 160 [text/plain]\n",
            "Saving to: ‘hadoop-3.4.0.tar.gz.sha512’\n",
            "\n",
            "\r          hadoop-3.   0%[                    ]       0  --.-KB/s               \rhadoop-3.4.0.tar.gz 100%[===================>]     160  --.-KB/s    in 0s      \n",
            "\n",
            "2024-05-05 15:49:12 (10.0 MB/s) - ‘hadoop-3.4.0.tar.gz.sha512’ saved [160/160]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Compare"
      ],
      "metadata": {
        "id": "eL8FxjalhFAn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "A=$(sha512sum hadoop-3.4.0.tar.gz | cut - -d' ' -f1)\n",
        "B=$(cut hadoop-3.4.0.tar.gz.sha512 -d' ' -f4)\n",
        "printf \"%s\\n%s\\n\" $A $B\n",
        "[[ $A == $B ]] && echo \"True\""
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zL302M1OhFMH",
        "outputId": "043cbd94-7911-4ea1-b036-eebe15abcc0b"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "6f653c0109f97430047bd3677c50da7c8a2809d153b231794cf980b3208a6b4beff8ff1a03a01094299d459a3a37a3fe16731629987165d71f328657dbf2f24c\n",
            "6f653c0109f97430047bd3677c50da7c8a2809d153b231794cf980b3208a6b4beff8ff1a03a01094299d459a3a37a3fe16731629987165d71f328657dbf2f24c\n",
            "True\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Configure `PATH`\n",
        "\n",
        "Add the Hadoop folder to the `PATH` environment variable\n"
      ],
      "metadata": {
        "id": "RlgP1ytnRtUK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!echo $PATH"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "49xx-zSxIdxa",
        "outputId": "6a40406c-e06c-4ef4-a45b-bcd717897626"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/opt/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/google-cloud-sdk/bin\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "os.environ['HADOOP_HOME'] = os.path.join('/content', 'hadoop-3.4.0')\n",
        "os.environ['PATH'] = ':'.join([os.path.join(os.environ['HADOOP_HOME'], 'bin'), os.environ['PATH']])\n",
        "os.environ['JAVA_HOME'] = '/usr/lib/jvm/java-11-openjdk-amd64'"
      ],
      "metadata": {
        "id": "6V03we10Igek"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "print(os.environ)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Aif21X1ONvwH",
        "outputId": "78a2141b-cdeb-4db5-f4ce-c1460507625e"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "environ({'SHELL': '/bin/bash', 'NV_LIBCUBLAS_VERSION': '12.2.5.6-1', 'NVIDIA_VISIBLE_DEVICES': 'all', 'COLAB_JUPYTER_TRANSPORT': 'ipc', 'NV_NVML_DEV_VERSION': '12.2.140-1', 'NV_CUDNN_PACKAGE_NAME': 'libcudnn8', 'CGROUP_MEMORY_EVENTS': '/sys/fs/cgroup/memory.events /var/colab/cgroup/jupyter-children/memory.events', 'NV_LIBNCCL_DEV_PACKAGE': 'libnccl-dev=2.19.3-1+cuda12.2', 'NV_LIBNCCL_DEV_PACKAGE_VERSION': '2.19.3-1', 'VM_GCE_METADATA_HOST': '169.254.169.253', 'HOSTNAME': '4d8a89a36a38', 'LANGUAGE': 'en_US', 'TBE_RUNTIME_ADDR': '172.28.0.1:8011', 'COLAB_TPU_1VM': '', 'GCE_METADATA_TIMEOUT': '3', 'NVIDIA_REQUIRE_CUDA': 'cuda>=12.2 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526', 'NV_LIBCUBLAS_DEV_PACKAGE': 'libcublas-dev-12-2=12.2.5.6-1', 'NV_NVTX_VERSION': '12.2.140-1', 'COLAB_JUPYTER_IP': '172.28.0.12', 'NV_CUDA_CUDART_DEV_VERSION': '12.2.140-1', 'NV_LIBCUSPARSE_VERSION': '12.1.2.141-1', 'COLAB_LANGUAGE_SERVER_PROXY_ROOT_URL': 'http://172.28.0.1:8013/', 'NV_LIBNPP_VERSION': '12.2.1.4-1', 'NCCL_VERSION': '2.19.3-1', 'KMP_LISTEN_PORT': '6000', 'TF_FORCE_GPU_ALLOW_GROWTH': 'true', 'ENV': '/root/.bashrc', 'PWD': '/', 'TBE_EPHEM_CREDS_ADDR': '172.28.0.1:8009', 'COLAB_LANGUAGE_SERVER_PROXY_REQUEST_TIMEOUT': '30s', 'TBE_CREDS_ADDR': '172.28.0.1:8008', 'NV_CUDNN_PACKAGE': 'libcudnn8=8.9.6.50-1+cuda12.2', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'COLAB_JUPYTER_TOKEN': '', 'LAST_FORCED_REBUILD': '20240424', 'NV_NVPROF_DEV_PACKAGE': 'cuda-nvprof-12-2=12.2.142-1', 'NV_LIBNPP_PACKAGE': 'libnpp-12-2=12.2.1.4-1', 'NV_LIBNCCL_DEV_PACKAGE_NAME': 'libnccl-dev', 'TCLLIBPATH': '/usr/share/tcltk/tcllib1.20', 'NV_LIBCUBLAS_DEV_VERSION': '12.2.5.6-1', 'NVIDIA_PRODUCT_NAME': 'CUDA', 'COLAB_KERNEL_MANAGER_PROXY_HOST': '172.28.0.12', 'NV_LIBCUBLAS_DEV_PACKAGE_NAME': 'libcublas-dev-12-2', 'NV_CUDA_CUDART_VERSION': '12.2.140-1', 'COLAB_WARMUP_DEFAULTS': '1', 'HOME': '/root', 'LANG': 'en_US.UTF-8', 'COLUMNS': '100', 'CUDA_VERSION': '12.2.2', 'CLOUDSDK_CONFIG': '/content/.config', 'NV_LIBCUBLAS_PACKAGE': 'libcublas-12-2=12.2.5.6-1', 'NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE': 'cuda-nsight-compute-12-2=12.2.2-1', 'COLAB_RELEASE_TAG': 'release-colab_20240502-060125_RC00', 'KMP_TARGET_PORT': '9000', 'KMP_EXTRA_ARGS': '--logtostderr --listen_host=172.28.0.12 --target_host=172.28.0.12 --tunnel_background_save_url=https://colab.research.google.com/tun/m/cc48301118ce562b961b3c22d803539adc1e0c19/m-s-3oka6skupgz4n --tunnel_background_save_delay=10s --tunnel_periodic_background_save_frequency=30m0s --enable_output_coalescing=true --output_coalescing_required=true', 'NV_LIBNPP_DEV_PACKAGE': 'libnpp-dev-12-2=12.2.1.4-1', 'COLAB_LANGUAGE_SERVER_PROXY_LSP_DIRS': '/datalab/web/pyright/typeshed-fallback/stdlib,/usr/local/lib/python3.10/dist-packages', 'NV_LIBCUBLAS_PACKAGE_NAME': 'libcublas-12-2', 'COLAB_KERNEL_MANAGER_PROXY_PORT': '6000', 'CLOUDSDK_PYTHON': 'python3', 'NV_LIBNPP_DEV_VERSION': '12.2.1.4-1', 'NO_GCE_CHECK': 'False', 'PYTHONPATH': '/env/python', 'NV_LIBCUSPARSE_DEV_VERSION': '12.1.2.141-1', 'LIBRARY_PATH': '/usr/local/cuda/lib64/stubs', 'NV_CUDNN_VERSION': '8.9.6.50', 'SHLVL': '0', 'NV_CUDA_LIB_VERSION': '12.2.2-1', 'COLAB_LANGUAGE_SERVER_PROXY': '/usr/colab/bin/language_service', 'NVARCH': 'x86_64', 'NV_CUDNN_PACKAGE_DEV': 'libcudnn8-dev=8.9.6.50-1+cuda12.2', 'NV_CUDA_COMPAT_PACKAGE': 'cuda-compat-12-2', 'NV_LIBNCCL_PACKAGE': 'libnccl2=2.19.3-1+cuda12.2', 'LD_LIBRARY_PATH': '/usr/local/nvidia/lib:/usr/local/nvidia/lib64', 'COLAB_GPU': '', 'NV_CUDA_NSIGHT_COMPUTE_VERSION': '12.2.2-1', 'GCS_READ_CACHE_BLOCK_SIZE_MB': '16', 'NV_NVPROF_VERSION': '12.2.142-1', 'LC_ALL': 'en_US.UTF-8', 'COLAB_FILE_HANDLER_ADDR': 'localhost:3453', 'PATH': '/content/hadoop-3.4.0/bin:/opt/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/google-cloud-sdk/bin', 'NV_LIBNCCL_PACKAGE_NAME': 'libnccl2', 'COLAB_DEBUG_ADAPTER_MUX_PATH': '/usr/local/bin/dap_multiplexer', 'NV_LIBNCCL_PACKAGE_VERSION': '2.19.3-1', 'PYTHONWARNINGS': 'ignore:::pip._internal.cli.base_command', 'DEBIAN_FRONTEND': 'noninteractive', 'COLAB_BACKEND_VERSION': 'next', 'OLDPWD': '/', 'JPY_PARENT_PID': '82', 'TERM': 'xterm-color', 'CLICOLOR': '1', 'PAGER': 'cat', 'GIT_PAGER': 'cat', 'MPLBACKEND': 'module://ipykernel.pylab.backend_inline', 'ENABLE_DIRECTORYPREFETCHER': '1', 'USE_AUTH_EPHEM': '1', 'PYDEVD_USE_FRAME_EVAL': 'NO', 'JAVA_HOME': '/usr/lib/jvm/java-11-openjdk-amd64', 'HADOOP_HOME': '/content/hadoop-3.4.0'})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!echo $PATH"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GcM-idgZQqfV",
        "outputId": "e54866e0-fd61-4e79-8684-b76d41554e00"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/hadoop-3.4.0/bin:/opt/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/google-cloud-sdk/bin\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Configure `core-site.xml` and `hdfs-site.xml`\n",
        "\n",
        "Edit the file `etc/hadoop/core-site.xml` and `etc/hadoop/hdfs-site.xml` to configure pseudo-distributed operation.\n",
        "\n",
        "**`etc/hadoop/core-site.xml`**\n",
        "```\n",
        "<configuration>\n",
        "    <property>\n",
        "        <name>fs.defaultFS</name>\n",
        "        <value>hdfs://localhost:9000</value>\n",
        "    </property>\n",
        "</configuration>\n",
        "```\n",
        "\n",
        "**`etc/hadoop/hdfs-site.xml`**\n",
        "```\n",
        "<configuration>\n",
        "    <property>\n",
        "        <name>dfs.replication</name>\n",
        "        <value>1</value>\n",
        "    </property>\n",
        "</configuration>\n",
        "```"
      ],
      "metadata": {
        "id": "KLmxLQeJSb4A"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "echo -e \"<configuration> \\n\\\n",
        "    <property> \\n\\\n",
        "        <name>fs.defaultFS</name> \\n\\\n",
        "        <value>hdfs://localhost:9000</value> \\n\\\n",
        "    </property> \\n\\\n",
        "</configuration>\" >hadoop-3.4.0/etc/hadoop/core-site.xml\n",
        "\n",
        "echo -e \"<configuration> \\n\\\n",
        "    <property> \\n\\\n",
        "        <name>dfs.replication</name> \\n\\\n",
        "        <value>1</value> \\n\\\n",
        "    </property> \\n\\\n",
        "</configuration>\" >hadoop-3.4.0/etc/hadoop/hdfs-site.xml"
      ],
      "metadata": {
        "id": "_n2d2lqXSLU1"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check"
      ],
      "metadata": {
        "id": "5mdkNb-Cg9HW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "cat hadoop-3.4.0/etc/hadoop/hdfs-site.xml"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-ISxE4Gqg_LG",
        "outputId": "17b205bf-7cdb-46e5-dab1-6e7b39a02f51"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "<configuration> \n",
            "    <property> \n",
            "        <name>dfs.replication</name> \n",
            "        <value>1</value> \n",
            "    </property> \n",
            "</configuration>\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Set environment variables\n",
        "\n",
        "Add the following lines to the Hadoop configuration script `hadoop-env.sh`(the script is in `hadoop-3.4.0/sbin`).\n",
        "```\n",
        "export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64\n",
        "export HDFS_NAMENODE_USER=root\n",
        "export HDFS_DATANODE_USER=root\n",
        "export HDFS_SECONDARYNAMENODE_USER=root\n",
        "export YARN_RESOURCEMANAGER_USER=root\n",
        "export YARN_NODEMANAGER_USER=root\n",
        "```"
      ],
      "metadata": {
        "id": "kXbSKFyeMqr2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "echo -e \"export HDFS_NAMENODE_USER=root \\n\\\n",
        "export HDFS_DATANODE_USER=root \\n\\\n",
        "export HDFS_SECONDARYNAMENODE_USER=root \\n\\\n",
        "export YARN_RESOURCEMANAGER_USER=root \\n\\\n",
        "export YARN_NODEMANAGER_USER=root\" >> hadoop-3.4.0/etc/hadoop/hadoop-env.sh"
      ],
      "metadata": {
        "id": "2_vn-TGyPe9V"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Setup localhost access via SSH key\n",
        "\n",
        "We are going to allow passphraseless access to `localhost` with a secure key.\n",
        "\n",
        "SSH must be installed and sshd must be running in order to use the Hadoop scripts that manage remote Hadoop daemons.\n"
      ],
      "metadata": {
        "id": "k2-Fdp73cF0V"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "### Install `openssh` and start server\n",
        "\n",
        "I'm not sure why we need the option `StrictHostKeyChecking no`. This option tells the `ssh` server to allow key authentication only from known hosts, in particular it prevents a host from authenticating with key if the key has changed. I guess this option is needed since a new ssh key is generated every time one runs this notebook.\n",
        "\n",
        "Alternatively, one could just delete the file `~/.ssh/known_hosts` or else use `ssh-keygen -R hostname` to remove all keys belonging to hostname from the `known_hosts` file (see for instance [How to remove strict RSA key checking in SSH and what's the problem here?](https://serverfault.com/questions/6233/how-to-remove-strict-rsa-key-checking-in-ssh-and-whats-the-problem-here) or [Remove key from known_hosts](https://superuser.com/questions/30087/remove-key-from-known-hosts)). The option `ssh-keygen -R hostname` would be the most appropriate in a production setting where the file `~/.ssh/known_hosts` might contain other entries that you do not want to delete.\n"
      ],
      "metadata": {
        "id": "-Uxmv3RdUwiF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "apt-get update\n",
        "apt-get install openssh-server\n",
        "echo 'StrictHostKeyChecking no' >> /etc/ssh/ssh_config\n",
        "/etc/init.d/ssh restart"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yOxz683FNuYH",
        "outputId": "813dcfb6-26dd-4eac-df3e-04b3f8660a3d"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Get:1 https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/ InRelease [3,626 B]\n",
            "Get:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  InRelease [1,581 B]\n",
            "Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]\n",
            "Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease\n",
            "Get:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]\n",
            "Get:6 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  Packages [830 kB]\n",
            "Hit:7 https://ppa.launchpadcontent.net/c2d4u.team/c2d4u4.0+/ubuntu jammy InRelease\n",
            "Hit:8 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu jammy InRelease\n",
            "Hit:9 https://ppa.launchpadcontent.net/graphics-drivers/ppa/ubuntu jammy InRelease\n",
            "Hit:10 http://archive.ubuntu.com/ubuntu jammy-backports InRelease\n",
            "Hit:11 https://ppa.launchpadcontent.net/ubuntugis/ppa/ubuntu jammy InRelease\n",
            "Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1,077 kB]\n",
            "Get:13 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,756 kB]\n",
            "Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,037 kB]\n",
            "Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,371 kB]\n",
            "Fetched 7,306 kB in 3s (2,816 kB/s)\n",
            "Reading package lists...\n",
            "Reading package lists...\n",
            "Building dependency tree...\n",
            "Reading state information...\n",
            "The following additional packages will be installed:\n",
            "  libwrap0 ncurses-term openssh-sftp-server ssh-import-id\n",
            "Suggested packages:\n",
            "  molly-guard monkeysphere ssh-askpass ufw\n",
            "The following NEW packages will be installed:\n",
            "  libwrap0 ncurses-term openssh-server openssh-sftp-server ssh-import-id\n",
            "0 upgraded, 5 newly installed, 0 to remove and 46 not upgraded.\n",
            "Need to get 800 kB of archives.\n",
            "After this operation, 6,161 kB of additional disk space will be used.\n",
            "Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 openssh-sftp-server amd64 1:8.9p1-3ubuntu0.7 [38.9 kB]\n",
            "Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 libwrap0 amd64 7.6.q-31build2 [47.9 kB]\n",
            "Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 openssh-server amd64 1:8.9p1-3ubuntu0.7 [435 kB]\n",
            "Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 ncurses-term all 6.3-2ubuntu0.1 [267 kB]\n",
            "Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 ssh-import-id all 5.11-0ubuntu1 [10.1 kB]\n",
            "Preconfiguring packages ...\n",
            "Fetched 800 kB in 1s (1,100 kB/s)\n",
            "Selecting previously unselected package openssh-sftp-server.\r\n",
            "(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 121920 files and directories currently installed.)\r\n",
            "Preparing to unpack .../openssh-sftp-server_1%3a8.9p1-3ubuntu0.7_amd64.deb ...\r\n",
            "Unpacking openssh-sftp-server (1:8.9p1-3ubuntu0.7) ...\r\n",
            "Selecting previously unselected package libwrap0:amd64.\r\n",
            "Preparing to unpack .../libwrap0_7.6.q-31build2_amd64.deb ...\r\n",
            "Unpacking libwrap0:amd64 (7.6.q-31build2) ...\r\n",
            "Selecting previously unselected package openssh-server.\r\n",
            "Preparing to unpack .../openssh-server_1%3a8.9p1-3ubuntu0.7_amd64.deb ...\r\n",
            "Unpacking openssh-server (1:8.9p1-3ubuntu0.7) ...\r\n",
            "Selecting previously unselected package ncurses-term.\r\n",
            "Preparing to unpack .../ncurses-term_6.3-2ubuntu0.1_all.deb ...\r\n",
            "Unpacking ncurses-term (6.3-2ubuntu0.1) ...\r\n",
            "Selecting previously unselected package ssh-import-id.\r\n",
            "Preparing to unpack .../ssh-import-id_5.11-0ubuntu1_all.deb ...\r\n",
            "Unpacking ssh-import-id (5.11-0ubuntu1) ...\r\n",
            "Setting up openssh-sftp-server (1:8.9p1-3ubuntu0.7) ...\r\n",
            "Setting up ssh-import-id (5.11-0ubuntu1) ...\r\n",
            "Setting up libwrap0:amd64 (7.6.q-31build2) ...\r\n",
            "Setting up ncurses-term (6.3-2ubuntu0.1) ...\r\n",
            "Setting up openssh-server (1:8.9p1-3ubuntu0.7) ...\r\n",
            "\r\n",
            "Creating config file /etc/ssh/sshd_config with new version\r\n",
            "Creating SSH2 RSA key; this may take some time ...\r\n",
            "3072 SHA256:n/FC62orT6y/129QSaBZMhh9E6dZQdrlQ7+MlCBnwj8 root@4d8a89a36a38 (RSA)\r\n",
            "Creating SSH2 ECDSA key; this may take some time ...\r\n",
            "256 SHA256:ApmVZBETgQbbTXBkqs7R7W73vk6yzjx+MR0/Ce904s0 root@4d8a89a36a38 (ECDSA)\r\n",
            "Creating SSH2 ED25519 key; this may take some time ...\r\n",
            "256 SHA256:dvbJLevMXRu2eAzT9c6K73KShYI5OEwx4fXMur6KHeY root@4d8a89a36a38 (ED25519)\r\n",
            "invoke-rc.d: could not determine current runlevel\r\n",
            "invoke-rc.d: policy-rc.d denied execution of start.\r\n",
            "Created symlink /etc/systemd/system/sshd.service → /lib/systemd/system/ssh.service.\r\n",
            "Created symlink /etc/systemd/system/multi-user.target.wants/ssh.service → /lib/systemd/system/ssh.service.\r\n",
            "Processing triggers for man-db (2.10.2-1) ...\r\n",
            "Processing triggers for libc-bin (2.35-0ubuntu3.4) ...\r\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_0.so.3 is not a symbolic link\r\n",
            "\r\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc.so.2 is not a symbolic link\r\n",
            "\r\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc_proxy.so.2 is not a symbolic link\r\n",
            "\r\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind.so.3 is not a symbolic link\r\n",
            "\r\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_5.so.3 is not a symbolic link\r\n",
            "\r\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbb.so.12 is not a symbolic link\r\n",
            "\r\n",
            " * Restarting OpenBSD Secure Shell server sshd\n",
            "   ...done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Generate key\n",
        "Generate an SSH key that does not require a password.\n",
        "\n",
        "The private key is contained in the file `id_rsa` located in the folder `~/.ssh`.\n",
        "\n",
        "The public key is added to the file `~/.ssh/authorized_keys` in order to allow authentication with that key."
      ],
      "metadata": {
        "id": "PYKoSlaENuyG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "rm /root/.ssh/id_rsa\n",
        "ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa\n",
        "cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys\n",
        "chmod 0600 ~/.ssh/authorized_keys"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YHOjUaxHSsQD",
        "outputId": "98d1e76d-8506-4d2c-a913-dbb6fa5eabd4"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Generating public/private rsa key pair.\n",
            "Your identification has been saved in /root/.ssh/id_rsa\n",
            "Your public key has been saved in /root/.ssh/id_rsa.pub\n",
            "The key fingerprint is:\n",
            "SHA256:KqschEen2lo8Cxm92gSFFapZC0Pefve0smXSqBThAso root@4d8a89a36a38\n",
            "The key's randomart image is:\n",
            "+---[RSA 3072]----+\n",
            "| .o.             |\n",
            "|o+.              |\n",
            "|=o+...           |\n",
            "|+Xo+. .          |\n",
            "|BE*o + .S.       |\n",
            "| X .o o.= .      |\n",
            "|+ O ...+ *       |\n",
            "| O +.o. *        |\n",
            "|o =... .         |\n",
            "+----[SHA256]-----+\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "rm: cannot remove '/root/.ssh/id_rsa': No such file or directory\n",
            "Created directory '/root/.ssh'.\r\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Check SSH connection to localhost\n",
        "\n",
        "The following command should output \"hi!\" if the connection works."
      ],
      "metadata": {
        "id": "FwA6rKpScnVi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!ssh localhost \"echo hi!\""
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hqIRVxcfce0F",
        "outputId": "5ae35b16-1995-464c-eb74-b1e7ce275041"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.\r\n",
            "hi!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Launch a single-node Hadoop cluster"
      ],
      "metadata": {
        "id": "V68C4cDySyek"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Initialize the namenode"
      ],
      "metadata": {
        "id": "HTDPwnVlSbHS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!hdfs namenode -format -nonInteractive"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Q-aicnKKLVKQ",
        "outputId": "5f6c6289-9bea-451b-93d8-c599a53591a1"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "WARNING: /content/hadoop-3.4.0/logs does not exist. Creating.\n",
            "2024-05-05 15:49:34,850 INFO namenode.NameNode: STARTUP_MSG: \n",
            "/************************************************************\n",
            "STARTUP_MSG: Starting NameNode\n",
            "STARTUP_MSG:   host = 4d8a89a36a38/172.28.0.12\n",
            "STARTUP_MSG:   args = [-format, -nonInteractive]\n",
            "STARTUP_MSG:   version = 3.4.0\n",
            "STARTUP_MSG:   classpath = /content/hadoop-3.4.0/etc/hadoop:/content/hadoop-3.4.0/share/hadoop/common/lib/jsr305-3.0.2.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-handler-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/gson-2.9.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jul-to-slf4j-1.7.36.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/curator-client-5.2.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jetty-webapp-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-transport-classes-kqueue-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/kerb-admin-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-codec-http2-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/metrics-core-3.2.4.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-transport-rxtx-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-transport-sctp-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/animal-sniffer-annotations-1.17.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/snappy-java-1.1.10.4.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jersey-core-1.19.4.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/kerb-util-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jersey-servlet-1.19.4.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-handler-ssl-ocsp-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/kerb-server-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/kerb-simplekdc-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-transport-native-epoll-4.1.100.Final-linux-aarch_64.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jettison-1.5.4.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jetty-security-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-transport-native-unix-common-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-handler-proxy-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jetty-servlet-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/commons-io-2.14.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-codec-redis-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/commons-daemon-1.0.13.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-transport-native-kqueue-4.1.100.Final-osx-aarch_64.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/zookeeper-3.8.3.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/slf4j-api-1.7.36.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/kerby-pkix-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jackson-annotations-2.12.7.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/httpcore-4.4.13.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/commons-cli-1.5.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/avro-1.9.2.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/curator-recipes-5.2.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/commons-collections-3.2.2.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/commons-compress-1.24.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/kerb-client-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/failureaccess-1.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-transport-classes-epoll-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/commons-math3-3.6.1.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/token-provider-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-resolver-dns-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/checker-qual-2.5.2.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jersey-json-1.20.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jetty-http-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/bcprov-jdk15on-1.70.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-codec-haproxy-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/commons-codec-1.15.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/woodstox-core-5.4.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/httpclient-4.5.13.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-codec-http-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-common-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/hadoop-annotations-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/commons-text-1.10.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-codec-socks-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/kerb-common-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jetty-util-ajax-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jackson-core-2.12.7.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-codec-xml-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-codec-stomp-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-codec-smtp-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-resolver-dns-native-macos-4.1.100.Final-osx-x86_64.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-transport-udt-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/guava-27.0-jre.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-buffer-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jetty-xml-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-codec-memcache-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/commons-lang3-3.12.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/stax2-api-4.2.1.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jakarta.activation-api-1.2.1.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/commons-net-3.9.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/nimbus-jose-jwt-9.31.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/dnsjava-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/curator-framework-5.2.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jetty-server-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-resolver-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/commons-configuration2-2.8.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-codec-dns-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/kerby-util-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/hadoop-auth-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-resolver-dns-native-macos-4.1.100.Final-osx-aarch_64.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-resolver-dns-classes-macos-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jsch-0.1.55.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/audience-annotations-0.12.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/reload4j-1.2.22.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/re2j-1.1.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/commons-beanutils-1.9.4.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-transport-native-kqueue-4.1.100.Final-osx-x86_64.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jersey-server-1.19.4.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/kerby-config-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-codec-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jetty-io-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jline-3.9.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/kerby-asn1-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-transport-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/j2objc-annotations-1.1.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jetty-util-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/kerb-core-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-protobuf_3_21-1.2.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/commons-logging-1.2.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jsp-api-2.1.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-codec-mqtt-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/jackson-databind-2.12.7.1.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-guava-1.2.0.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/kerb-crypto-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-all-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/netty-transport-native-epoll-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/zookeeper-jute-3.8.3.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/kerby-xdr-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/common/lib/kerb-identity-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/common/hadoop-nfs-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/common/hadoop-common-3.4.0-tests.jar:/content/hadoop-3.4.0/share/hadoop/common/hadoop-kms-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/common/hadoop-registry-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/common/hadoop-common-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jsr305-3.0.2.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-handler-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/gson-2.9.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/curator-client-5.2.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jetty-webapp-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-transport-classes-kqueue-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/kerb-admin-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-codec-http2-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/metrics-core-3.2.4.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-transport-rxtx-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-transport-sctp-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/animal-sniffer-annotations-1.17.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/snappy-java-1.1.10.4.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jersey-core-1.19.4.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/kerb-util-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jersey-servlet-1.19.4.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-handler-ssl-ocsp-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/kerb-server-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/kerb-simplekdc-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-transport-native-epoll-4.1.100.Final-linux-aarch_64.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jettison-1.5.4.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jetty-security-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-transport-native-unix-common-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-handler-proxy-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jetty-servlet-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/commons-io-2.14.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-codec-redis-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-transport-native-kqueue-4.1.100.Final-osx-aarch_64.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/zookeeper-3.8.3.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/kerby-pkix-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jackson-annotations-2.12.7.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/httpcore-4.4.13.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/commons-cli-1.5.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/avro-1.9.2.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/curator-recipes-5.2.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/commons-compress-1.24.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/kerb-client-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/failureaccess-1.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-transport-classes-epoll-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/commons-math3-3.6.1.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/token-provider-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-resolver-dns-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/checker-qual-2.5.2.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jersey-json-1.20.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jetty-http-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-codec-haproxy-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/commons-codec-1.15.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/woodstox-core-5.4.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/httpclient-4.5.13.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/HikariCP-4.0.3.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-codec-http-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-common-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/hadoop-annotations-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/commons-text-1.10.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-codec-socks-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/kerb-common-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jetty-util-ajax-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jackson-core-2.12.7.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-codec-xml-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-codec-stomp-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-codec-smtp-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-resolver-dns-native-macos-4.1.100.Final-osx-x86_64.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-transport-udt-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/guava-27.0-jre.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-buffer-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jetty-xml-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-codec-memcache-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/commons-lang3-3.12.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/stax2-api-4.2.1.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jakarta.activation-api-1.2.1.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/commons-net-3.9.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/nimbus-jose-jwt-9.31.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/dnsjava-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/curator-framework-5.2.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jetty-server-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-resolver-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/commons-configuration2-2.8.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-codec-dns-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/kerby-util-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/hadoop-auth-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-resolver-dns-native-macos-4.1.100.Final-osx-aarch_64.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-resolver-dns-classes-macos-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jsch-0.1.55.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/audience-annotations-0.12.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/reload4j-1.2.22.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/re2j-1.1.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/commons-beanutils-1.9.4.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-transport-native-kqueue-4.1.100.Final-osx-x86_64.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jersey-server-1.19.4.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/kerby-config-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-codec-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jetty-io-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jline-3.9.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/kerby-asn1-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-transport-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/j2objc-annotations-1.1.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jetty-util-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/kerb-core-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/hadoop-shaded-protobuf_3_21-1.2.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/commons-logging-1.2.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-codec-mqtt-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/jackson-databind-2.12.7.1.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/hadoop-shaded-guava-1.2.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/kerb-crypto-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-all-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/netty-transport-native-epoll-4.1.100.Final.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/zookeeper-jute-3.8.3.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/kerby-xdr-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/lib/kerb-identity-2.0.3.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/hadoop-hdfs-client-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/hadoop-hdfs-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/hadoop-hdfs-rbf-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/hadoop-hdfs-rbf-3.4.0-tests.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/hadoop-hdfs-client-3.4.0-tests.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/hadoop-hdfs-nfs-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/hadoop-hdfs-native-client-3.4.0-tests.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/hadoop-hdfs-3.4.0-tests.jar:/content/hadoop-3.4.0/share/hadoop/hdfs/hadoop-hdfs-native-client-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar:/content/hadoop-3.4.0/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn:/content/hadoop-3.4.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/jackson-jaxrs-base-2.12.7.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/jetty-annotations-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.12.7.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/bcpkix-jdk15on-1.70.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/javax.websocket-api-1.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/javax.websocket-client-api-1.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.12.7.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/asm-tree-9.6.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/websocket-servlet-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/jetty-plus-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/jsonschema2pojo-core-1.0.2.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/guice-servlet-4.2.3.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/websocket-api-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/jetty-jndi-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/javax.inject-1.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/codemodel-2.6.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/javax-websocket-server-impl-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/websocket-server-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/jetty-client-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/websocket-client-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/jersey-client-1.19.4.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/asm-commons-9.6.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/javax-websocket-client-impl-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/bcutil-jdk15on-1.70.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/jakarta.xml.bind-api-2.3.2.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/guice-4.2.3.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/snakeyaml-2.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/fst-2.50.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/jna-5.2.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/websocket-common-9.4.53.v20231009.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/jersey-guice-1.19.4.jar:/content/hadoop-3.4.0/share/hadoop/yarn/lib/objenesis-2.6.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-services-core-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-server-common-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-api-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-applications-mawo-core-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-server-router-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-common-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-client-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-server-globalpolicygenerator-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-server-tests-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-registry-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.4.0.jar:/content/hadoop-3.4.0/share/hadoop/yarn/hadoop-yarn-services-api-3.4.0.jar\n",
            "STARTUP_MSG:   build = git@github.com:apache/hadoop.git -r bd8b77f398f626bb7791783192ee7a5dfaeec760; compiled by 'root' on 2024-03-04T06:35Z\n",
            "STARTUP_MSG:   java = 11.0.22\n",
            "************************************************************/\n",
            "2024-05-05 15:49:35,027 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]\n",
            "2024-05-05 15:49:35,223 INFO namenode.NameNode: createNameNode [-format, -nonInteractive]\n",
            "2024-05-05 15:49:36,250 INFO namenode.NameNode: Formatting using clusterid: CID-ad3519f3-0b30-4d58-b7b5-324c4aeedd0f\n",
            "2024-05-05 15:49:36,302 INFO namenode.FSEditLog: Edit logging is async:true\n",
            "2024-05-05 15:49:36,356 INFO namenode.FSNamesystem: KeyProvider: null\n",
            "2024-05-05 15:49:36,358 INFO namenode.FSNamesystem: fsLock is fair: true\n",
            "2024-05-05 15:49:36,359 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false\n",
            "2024-05-05 15:49:36,398 INFO namenode.FSNamesystem: fsOwner                = root (auth:SIMPLE)\n",
            "2024-05-05 15:49:36,398 INFO namenode.FSNamesystem: supergroup             = supergroup\n",
            "2024-05-05 15:49:36,398 INFO namenode.FSNamesystem: isPermissionEnabled    = true\n",
            "2024-05-05 15:49:36,398 INFO namenode.FSNamesystem: isStoragePolicyEnabled = true\n",
            "2024-05-05 15:49:36,398 INFO namenode.FSNamesystem: HA Enabled: false\n",
            "2024-05-05 15:49:36,475 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling\n",
            "2024-05-05 15:49:36,664 INFO blockmanagement.DatanodeManager: Slow peers collection thread shutdown\n",
            "2024-05-05 15:49:36,679 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit : configured=1000, counted=60, effected=1000\n",
            "2024-05-05 15:49:36,680 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true\n",
            "2024-05-05 15:49:36,684 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000\n",
            "2024-05-05 15:49:36,685 INFO blockmanagement.BlockManager: The block deletion will start around 2024 May 05 15:49:36\n",
            "2024-05-05 15:49:36,686 INFO util.GSet: Computing capacity for map BlocksMap\n",
            "2024-05-05 15:49:36,686 INFO util.GSet: VM type       = 64-bit\n",
            "2024-05-05 15:49:36,688 INFO util.GSet: 2.0% max memory 3.2 GB = 64.9 MB\n",
            "2024-05-05 15:49:36,688 INFO util.GSet: capacity      = 2^23 = 8388608 entries\n",
            "2024-05-05 15:49:36,720 INFO blockmanagement.BlockManager: Storage policy satisfier is disabled\n",
            "2024-05-05 15:49:36,720 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false\n",
            "2024-05-05 15:49:36,728 INFO blockmanagement.BlockManagerSafeMode: Using 1000 as SafeModeMonitor Interval\n",
            "2024-05-05 15:49:36,728 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.999\n",
            "2024-05-05 15:49:36,728 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0\n",
            "2024-05-05 15:49:36,728 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000\n",
            "2024-05-05 15:49:36,729 INFO blockmanagement.BlockManager: defaultReplication         = 1\n",
            "2024-05-05 15:49:36,729 INFO blockmanagement.BlockManager: maxReplication             = 512\n",
            "2024-05-05 15:49:36,729 INFO blockmanagement.BlockManager: minReplication             = 1\n",
            "2024-05-05 15:49:36,729 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2\n",
            "2024-05-05 15:49:36,729 INFO blockmanagement.BlockManager: redundancyRecheckInterval  = 3000ms\n",
            "2024-05-05 15:49:36,729 INFO blockmanagement.BlockManager: encryptDataTransfer        = false\n",
            "2024-05-05 15:49:36,729 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000\n",
            "2024-05-05 15:49:36,793 INFO namenode.FSDirectory: GLOBAL serial map: bits=29 maxEntries=536870911\n",
            "2024-05-05 15:49:36,793 INFO namenode.FSDirectory: USER serial map: bits=24 maxEntries=16777215\n",
            "2024-05-05 15:49:36,793 INFO namenode.FSDirectory: GROUP serial map: bits=24 maxEntries=16777215\n",
            "2024-05-05 15:49:36,793 INFO namenode.FSDirectory: XATTR serial map: bits=24 maxEntries=16777215\n",
            "2024-05-05 15:49:36,812 INFO util.GSet: Computing capacity for map INodeMap\n",
            "2024-05-05 15:49:36,812 INFO util.GSet: VM type       = 64-bit\n",
            "2024-05-05 15:49:36,812 INFO util.GSet: 1.0% max memory 3.2 GB = 32.5 MB\n",
            "2024-05-05 15:49:36,812 INFO util.GSet: capacity      = 2^22 = 4194304 entries\n",
            "2024-05-05 15:49:36,825 INFO namenode.FSDirectory: ACLs enabled? true\n",
            "2024-05-05 15:49:36,825 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true\n",
            "2024-05-05 15:49:36,825 INFO namenode.FSDirectory: XAttrs enabled? true\n",
            "2024-05-05 15:49:36,825 INFO namenode.NameNode: Caching file names occurring more than 10 times\n",
            "2024-05-05 15:49:36,832 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotFSLimit: 65536, maxSnapshotLimit: 65536\n",
            "2024-05-05 15:49:36,833 INFO snapshot.SnapshotManager: dfs.namenode.snapshot.deletion.ordered = false\n",
            "2024-05-05 15:49:36,836 INFO snapshot.SnapshotManager: SkipList is disabled\n",
            "2024-05-05 15:49:36,843 INFO util.GSet: Computing capacity for map cachedBlocks\n",
            "2024-05-05 15:49:36,843 INFO util.GSet: VM type       = 64-bit\n",
            "2024-05-05 15:49:36,843 INFO util.GSet: 0.25% max memory 3.2 GB = 8.1 MB\n",
            "2024-05-05 15:49:36,843 INFO util.GSet: capacity      = 2^20 = 1048576 entries\n",
            "2024-05-05 15:49:36,858 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10\n",
            "2024-05-05 15:49:36,858 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10\n",
            "2024-05-05 15:49:36,858 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25\n",
            "2024-05-05 15:49:36,864 INFO namenode.FSNamesystem: Retry cache on namenode is enabled\n",
            "2024-05-05 15:49:36,864 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis\n",
            "2024-05-05 15:49:36,867 INFO util.GSet: Computing capacity for map NameNodeRetryCache\n",
            "2024-05-05 15:49:36,867 INFO util.GSet: VM type       = 64-bit\n",
            "2024-05-05 15:49:36,868 INFO util.GSet: 0.029999999329447746% max memory 3.2 GB = 997.2 KB\n",
            "2024-05-05 15:49:36,868 INFO util.GSet: capacity      = 2^17 = 131072 entries\n",
            "2024-05-05 15:49:36,901 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1587440677-172.28.0.12-1714924176891\n",
            "2024-05-05 15:49:36,945 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted.\n",
            "2024-05-05 15:49:37,012 INFO namenode.FSImageFormatProtobuf: Saving image file /tmp/hadoop-root/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression\n",
            "2024-05-05 15:49:37,182 INFO namenode.FSImageFormatProtobuf: Image file /tmp/hadoop-root/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 399 bytes saved in 0 seconds .\n",
            "2024-05-05 15:49:37,228 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0\n",
            "2024-05-05 15:49:37,241 INFO blockmanagement.DatanodeManager: Slow peers collection thread shutdown\n",
            "2024-05-05 15:49:37,340 INFO namenode.FSNamesystem: Stopping services started for active state\n",
            "2024-05-05 15:49:37,341 INFO namenode.FSNamesystem: Stopping services started for standby state\n",
            "2024-05-05 15:49:37,348 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.\n",
            "2024-05-05 15:49:37,349 INFO namenode.NameNode: SHUTDOWN_MSG: \n",
            "/************************************************************\n",
            "SHUTDOWN_MSG: Shutting down NameNode at 4d8a89a36a38/172.28.0.12\n",
            "************************************************************/\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!echo -e \"export JAVA_HOME=$(readlink -f $(which java) | sed 's/\\/bin\\/java$//') \\n\\\n",
        "export HDFS_NAMENODE_USER=root \\n\\\n",
        "export HDFS_DATANODE_USER=root \\n\\\n",
        "export HDFS_SECONDARYNAMENODE_USER=root \\n\\\n",
        "export YARN_RESOURCEMANAGER_USER=root \\n\\\n",
        "export YARN_NODEMANAGER_USER=root\" >> hadoop-3.4.0/etc/hadoop/hadoop-env.sh"
      ],
      "metadata": {
        "id": "8cb7lXMptVHb"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Start cluster"
      ],
      "metadata": {
        "id": "xMrEiLB_VAeR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!$HADOOP_HOME/sbin/start-dfs.sh"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FXHowFfFEwAF",
        "outputId": "b95bf762-0e90-45ce-b0fe-5260e56e51c1"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Starting namenodes on [localhost]\n",
            "Starting datanodes\n",
            "Starting secondary namenodes [4d8a89a36a38]\n",
            "4d8a89a36a38: Warning: Permanently added '4d8a89a36a38' (ED25519) to the list of known hosts.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Run some simple HDFS commands"
      ],
      "metadata": {
        "id": "CKRRbwDFv3ZQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "# create directory \"my_dir\" in HDFS home\n",
        "hdfs dfs -mkdir /user\n",
        "hdfs dfs -mkdir /user/root # this is the \"home\" of user root on HDFS\n",
        "hdfs dfs -mkdir my_dir\n",
        "\n",
        "# upload file mnist_test.csv to my_dir\n",
        "hdfs dfs -put /content/sample_data/mnist_test.csv my_dir/\n",
        "\n",
        "# show contents of directory my_dir\n",
        "hdfs dfs -ls -h my_dir\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "73wuvOJTxX4O",
        "outputId": "bf3494b9-760f-4fe9-9b72-c3e1fd46a927"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found 1 items\n",
            "-rw-r--r--   1 root supergroup     17.4 M 2024-05-05 15:50 my_dir/mnist_test.csv\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Run some simple MapReduce jobs\n",
        "\n",
        "We'll employ the [streaming](https://hadoop.apache.org/docs/stable/hadoop-streaming/HadoopStreaming.html) library, which broadens our options by enabling the use of any programming language for both the mapper and/or the reducer.\n",
        "\n",
        "With this utility any executable or file containing code that the operating system can interpret and execute directly, can serve as mapper and/or reducer."
      ],
      "metadata": {
        "id": "G3KBe4R65bl1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Simplest MapReduce job\n",
        "\n",
        "This is a \"no-code\" example since we are going to use the existing Unix commands `cat` and `wc` respectively as mapper and as reducer. The result will show a line with three values: the counts of lines, words, and characters in the input file(s).\n",
        "\n",
        "Input folder is `/user/my_user/my_dir/`, output folder `/user/my_user/output_simplest`.\n",
        "\n",
        "**Note**: the output folder should not exist because it is created by Hadoop (this is in accordance with Hadoop's principle of not overwriting data)."
      ],
      "metadata": {
        "id": "yVJA-3jSATGV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now run the MapReduce job"
      ],
      "metadata": {
        "id": "o6ICcKO2jcHl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "\n",
        "hdfs dfs -rm -r output_simplest || hdfs namenode -format -nonInteractive\n",
        "mapred streaming \\\n",
        "  -input my_dir \\\n",
        "  -output output_simplest \\\n",
        "  -mapper /bin/cat \\\n",
        "  -reducer /usr/bin/wc"
      ],
      "metadata": {
        "id": "VDuQYWGi5b7J",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2eef7e2a-c0ac-4f84-dd35-b9e9560b1c6b"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "rm: `output_simplest': No such file or directory\n",
            "namenode is running as process 2009.  Stop it first and ensure /tmp/hadoop-root-namenode.pid file is empty before retry.\n",
            "2024-05-05 15:50:31,356 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties\n",
            "2024-05-05 15:50:31,530 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).\n",
            "2024-05-05 15:50:31,530 INFO impl.MetricsSystemImpl: JobTracker metrics system started\n",
            "2024-05-05 15:50:31,554 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!\n",
            "2024-05-05 15:50:32,133 INFO mapred.FileInputFormat: Total input files to process : 1\n",
            "2024-05-05 15:50:32,168 INFO mapreduce.JobSubmitter: number of splits:1\n",
            "2024-05-05 15:50:32,552 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1253376731_0001\n",
            "2024-05-05 15:50:32,552 INFO mapreduce.JobSubmitter: Executing with tokens: []\n",
            "2024-05-05 15:50:32,796 INFO mapreduce.Job: The url to track the job: http://localhost:8080/\n",
            "2024-05-05 15:50:32,798 INFO mapreduce.Job: Running job: job_local1253376731_0001\n",
            "2024-05-05 15:50:32,808 INFO mapred.LocalJobRunner: OutputCommitter set in config null\n",
            "2024-05-05 15:50:32,811 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter\n",
            "2024-05-05 15:50:32,819 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2\n",
            "2024-05-05 15:50:32,820 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false\n",
            "2024-05-05 15:50:32,911 INFO mapred.LocalJobRunner: Waiting for map tasks\n",
            "2024-05-05 15:50:32,920 INFO mapred.LocalJobRunner: Starting task: attempt_local1253376731_0001_m_000000_0\n",
            "2024-05-05 15:50:32,964 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2\n",
            "2024-05-05 15:50:32,965 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false\n",
            "2024-05-05 15:50:33,005 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]\n",
            "2024-05-05 15:50:33,018 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/root/my_dir/mnist_test.csv:0+18289443\n",
            "2024-05-05 15:50:33,081 INFO mapred.MapTask: numReduceTasks: 1\n",
            "2024-05-05 15:50:33,251 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)\n",
            "2024-05-05 15:50:33,251 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100\n",
            "2024-05-05 15:50:33,251 INFO mapred.MapTask: soft limit at 83886080\n",
            "2024-05-05 15:50:33,251 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600\n",
            "2024-05-05 15:50:33,251 INFO mapred.MapTask: kvstart = 26214396; length = 6553600\n",
            "2024-05-05 15:50:33,256 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer\n",
            "2024-05-05 15:50:33,260 INFO streaming.PipeMapRed: PipeMapRed exec [/bin/cat]\n",
            "2024-05-05 15:50:33,270 INFO Configuration.deprecation: mapred.work.output.dir is deprecated. Instead, use mapreduce.task.output.dir\n",
            "2024-05-05 15:50:33,275 INFO Configuration.deprecation: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir\n",
            "2024-05-05 15:50:33,276 INFO Configuration.deprecation: map.input.file is deprecated. Instead, use mapreduce.map.input.file\n",
            "2024-05-05 15:50:33,277 INFO Configuration.deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length\n",
            "2024-05-05 15:50:33,278 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id\n",
            "2024-05-05 15:50:33,278 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition\n",
            "2024-05-05 15:50:33,284 INFO Configuration.deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start\n",
            "2024-05-05 15:50:33,284 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap\n",
            "2024-05-05 15:50:33,285 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id\n",
            "2024-05-05 15:50:33,286 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id\n",
            "2024-05-05 15:50:33,287 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords\n",
            "2024-05-05 15:50:33,288 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name\n",
            "2024-05-05 15:50:33,516 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2024-05-05 15:50:33,518 INFO streaming.PipeMapRed: R/W/S=10/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2024-05-05 15:50:33,531 INFO streaming.PipeMapRed: Records R/W=73/1\n",
            "2024-05-05 15:50:33,546 INFO streaming.PipeMapRed: R/W/S=100/36/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2024-05-05 15:50:33,698 INFO streaming.PipeMapRed: R/W/S=1000/756/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2024-05-05 15:50:33,805 INFO mapreduce.Job: Job job_local1253376731_0001 running in uber mode : false\n",
            "2024-05-05 15:50:33,807 INFO mapreduce.Job:  map 0% reduce 0%\n",
            "2024-05-05 15:50:33,964 INFO streaming.PipeMapRed: R/W/S=10000/9962/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2024-05-05 15:50:33,966 INFO streaming.PipeMapRed: MRErrorThread done\n",
            "2024-05-05 15:50:33,970 INFO streaming.PipeMapRed: mapRedFinished\n",
            "2024-05-05 15:50:33,976 INFO mapred.LocalJobRunner: \n",
            "2024-05-05 15:50:33,976 INFO mapred.MapTask: Starting flush of map output\n",
            "2024-05-05 15:50:33,976 INFO mapred.MapTask: Spilling map output\n",
            "2024-05-05 15:50:33,976 INFO mapred.MapTask: bufstart = 0; bufend = 18319443; bufvoid = 104857600\n",
            "2024-05-05 15:50:33,976 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26174400(104697600); length = 39997/6553600\n",
            "2024-05-05 15:50:34,210 INFO mapred.MapTask: Finished spill 0\n",
            "2024-05-05 15:50:34,282 INFO mapred.Task: Task:attempt_local1253376731_0001_m_000000_0 is done. And is in the process of committing\n",
            "2024-05-05 15:50:34,290 INFO mapred.LocalJobRunner: Records R/W=73/1\n",
            "2024-05-05 15:50:34,290 INFO mapred.Task: Task 'attempt_local1253376731_0001_m_000000_0' done.\n",
            "2024-05-05 15:50:34,303 INFO mapred.Task: Final Counters for attempt_local1253376731_0001_m_000000_0: Counters: 23\n",
            "\tFile System Counters\n",
            "\t\tFILE: Number of bytes read=141937\n",
            "\t\tFILE: Number of bytes written=19219453\n",
            "\t\tFILE: Number of read operations=0\n",
            "\t\tFILE: Number of large read operations=0\n",
            "\t\tFILE: Number of write operations=0\n",
            "\t\tHDFS: Number of bytes read=18289443\n",
            "\t\tHDFS: Number of bytes written=0\n",
            "\t\tHDFS: Number of read operations=5\n",
            "\t\tHDFS: Number of large read operations=0\n",
            "\t\tHDFS: Number of write operations=1\n",
            "\t\tHDFS: Number of bytes read erasure-coded=0\n",
            "\tMap-Reduce Framework\n",
            "\t\tMap input records=10000\n",
            "\t\tMap output records=10000\n",
            "\t\tMap output bytes=18319443\n",
            "\t\tMap output materialized bytes=18359449\n",
            "\t\tInput split bytes=105\n",
            "\t\tCombine input records=0\n",
            "\t\tSpilled Records=10000\n",
            "\t\tFailed Shuffles=0\n",
            "\t\tMerged Map outputs=0\n",
            "\t\tGC time elapsed (ms)=31\n",
            "\t\tTotal committed heap usage (bytes)=423624704\n",
            "\tFile Input Format Counters \n",
            "\t\tBytes Read=18289443\n",
            "2024-05-05 15:50:34,304 INFO mapred.LocalJobRunner: Finishing task: attempt_local1253376731_0001_m_000000_0\n",
            "2024-05-05 15:50:34,305 INFO mapred.LocalJobRunner: map task executor complete.\n",
            "2024-05-05 15:50:34,311 INFO mapred.LocalJobRunner: Waiting for reduce tasks\n",
            "2024-05-05 15:50:34,316 INFO mapred.LocalJobRunner: Starting task: attempt_local1253376731_0001_r_000000_0\n",
            "2024-05-05 15:50:34,331 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2\n",
            "2024-05-05 15:50:34,331 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false\n",
            "2024-05-05 15:50:34,332 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]\n",
            "2024-05-05 15:50:34,341 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@e1ac71e\n",
            "2024-05-05 15:50:34,343 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!\n",
            "2024-05-05 15:50:34,374 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=2382574336, maxSingleShuffleLimit=595643584, mergeThreshold=1572499072, ioSortFactor=10, memToMemMergeOutputsThreshold=10\n",
            "2024-05-05 15:50:34,385 INFO reduce.EventFetcher: attempt_local1253376731_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events\n",
            "2024-05-05 15:50:34,474 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1253376731_0001_m_000000_0 decomp: 18359445 len: 18359449 to MEMORY\n",
            "2024-05-05 15:50:34,516 INFO reduce.InMemoryMapOutput: Read 18359445 bytes from map-output for attempt_local1253376731_0001_m_000000_0\n",
            "2024-05-05 15:50:34,519 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 18359445, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->18359445\n",
            "2024-05-05 15:50:34,521 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning\n",
            "2024-05-05 15:50:34,523 INFO mapred.LocalJobRunner: 1 / 1 copied.\n",
            "2024-05-05 15:50:34,523 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs\n",
            "2024-05-05 15:50:34,532 INFO mapred.Merger: Merging 1 sorted segments\n",
            "2024-05-05 15:50:34,532 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 18357610 bytes\n",
            "2024-05-05 15:50:34,624 INFO reduce.MergeManagerImpl: Merged 1 segments, 18359445 bytes to disk to satisfy reduce memory limit\n",
            "2024-05-05 15:50:34,625 INFO reduce.MergeManagerImpl: Merging 1 files, 18359449 bytes from disk\n",
            "2024-05-05 15:50:34,627 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce\n",
            "2024-05-05 15:50:34,628 INFO mapred.Merger: Merging 1 sorted segments\n",
            "2024-05-05 15:50:34,629 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 18357610 bytes\n",
            "2024-05-05 15:50:34,630 INFO mapred.LocalJobRunner: 1 / 1 copied.\n",
            "2024-05-05 15:50:34,632 INFO streaming.PipeMapRed: PipeMapRed exec [/usr/bin/wc]\n",
            "2024-05-05 15:50:34,637 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address\n",
            "2024-05-05 15:50:34,641 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps\n",
            "2024-05-05 15:50:34,718 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2024-05-05 15:50:34,719 INFO streaming.PipeMapRed: R/W/S=10/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2024-05-05 15:50:34,727 INFO streaming.PipeMapRed: R/W/S=100/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2024-05-05 15:50:34,768 INFO streaming.PipeMapRed: R/W/S=1000/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2024-05-05 15:50:34,815 INFO mapreduce.Job:  map 100% reduce 0%\n",
            "2024-05-05 15:50:35,042 INFO streaming.PipeMapRed: R/W/S=10000/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2024-05-05 15:50:35,045 INFO streaming.PipeMapRed: MRErrorThread done\n",
            "2024-05-05 15:50:35,047 INFO streaming.PipeMapRed: Records R/W=10000/1\n",
            "2024-05-05 15:50:35,048 INFO streaming.PipeMapRed: mapRedFinished\n",
            "2024-05-05 15:50:35,171 INFO mapred.Task: Task:attempt_local1253376731_0001_r_000000_0 is done. And is in the process of committing\n",
            "2024-05-05 15:50:35,177 INFO mapred.LocalJobRunner: 1 / 1 copied.\n",
            "2024-05-05 15:50:35,177 INFO mapred.Task: Task attempt_local1253376731_0001_r_000000_0 is allowed to commit now\n",
            "2024-05-05 15:50:35,206 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1253376731_0001_r_000000_0' to hdfs://localhost:9000/user/root/output_simplest\n",
            "2024-05-05 15:50:35,208 INFO mapred.LocalJobRunner: Records R/W=10000/1 > reduce\n",
            "2024-05-05 15:50:35,208 INFO mapred.Task: Task 'attempt_local1253376731_0001_r_000000_0' done.\n",
            "2024-05-05 15:50:35,209 INFO mapred.Task: Final Counters for attempt_local1253376731_0001_r_000000_0: Counters: 30\n",
            "\tFile System Counters\n",
            "\t\tFILE: Number of bytes read=36860867\n",
            "\t\tFILE: Number of bytes written=37578902\n",
            "\t\tFILE: Number of read operations=0\n",
            "\t\tFILE: Number of large read operations=0\n",
            "\t\tFILE: Number of write operations=0\n",
            "\t\tHDFS: Number of bytes read=18289443\n",
            "\t\tHDFS: Number of bytes written=26\n",
            "\t\tHDFS: Number of read operations=10\n",
            "\t\tHDFS: Number of large read operations=0\n",
            "\t\tHDFS: Number of write operations=3\n",
            "\t\tHDFS: Number of bytes read erasure-coded=0\n",
            "\tMap-Reduce Framework\n",
            "\t\tCombine input records=0\n",
            "\t\tCombine output records=0\n",
            "\t\tReduce input groups=10000\n",
            "\t\tReduce shuffle bytes=18359449\n",
            "\t\tReduce input records=10000\n",
            "\t\tReduce output records=1\n",
            "\t\tSpilled Records=10000\n",
            "\t\tShuffled Maps =1\n",
            "\t\tFailed Shuffles=0\n",
            "\t\tMerged Map outputs=1\n",
            "\t\tGC time elapsed (ms)=0\n",
            "\t\tTotal committed heap usage (bytes)=423624704\n",
            "\tShuffle Errors\n",
            "\t\tBAD_ID=0\n",
            "\t\tCONNECTION=0\n",
            "\t\tIO_ERROR=0\n",
            "\t\tWRONG_LENGTH=0\n",
            "\t\tWRONG_MAP=0\n",
            "\t\tWRONG_REDUCE=0\n",
            "\tFile Output Format Counters \n",
            "\t\tBytes Written=26\n",
            "2024-05-05 15:50:35,210 INFO mapred.LocalJobRunner: Finishing task: attempt_local1253376731_0001_r_000000_0\n",
            "2024-05-05 15:50:35,210 INFO mapred.LocalJobRunner: reduce task executor complete.\n",
            "2024-05-05 15:50:35,816 INFO mapreduce.Job:  map 100% reduce 100%\n",
            "2024-05-05 15:50:35,817 INFO mapreduce.Job: Job job_local1253376731_0001 completed successfully\n",
            "2024-05-05 15:50:35,831 INFO mapreduce.Job: Counters: 36\n",
            "\tFile System Counters\n",
            "\t\tFILE: Number of bytes read=37002804\n",
            "\t\tFILE: Number of bytes written=56798355\n",
            "\t\tFILE: Number of read operations=0\n",
            "\t\tFILE: Number of large read operations=0\n",
            "\t\tFILE: Number of write operations=0\n",
            "\t\tHDFS: Number of bytes read=36578886\n",
            "\t\tHDFS: Number of bytes written=26\n",
            "\t\tHDFS: Number of read operations=15\n",
            "\t\tHDFS: Number of large read operations=0\n",
            "\t\tHDFS: Number of write operations=4\n",
            "\t\tHDFS: Number of bytes read erasure-coded=0\n",
            "\tMap-Reduce Framework\n",
            "\t\tMap input records=10000\n",
            "\t\tMap output records=10000\n",
            "\t\tMap output bytes=18319443\n",
            "\t\tMap output materialized bytes=18359449\n",
            "\t\tInput split bytes=105\n",
            "\t\tCombine input records=0\n",
            "\t\tCombine output records=0\n",
            "\t\tReduce input groups=10000\n",
            "\t\tReduce shuffle bytes=18359449\n",
            "\t\tReduce input records=10000\n",
            "\t\tReduce output records=1\n",
            "\t\tSpilled Records=20000\n",
            "\t\tShuffled Maps =1\n",
            "\t\tFailed Shuffles=0\n",
            "\t\tMerged Map outputs=1\n",
            "\t\tGC time elapsed (ms)=31\n",
            "\t\tTotal committed heap usage (bytes)=847249408\n",
            "\tShuffle Errors\n",
            "\t\tBAD_ID=0\n",
            "\t\tCONNECTION=0\n",
            "\t\tIO_ERROR=0\n",
            "\t\tWRONG_LENGTH=0\n",
            "\t\tWRONG_MAP=0\n",
            "\t\tWRONG_REDUCE=0\n",
            "\tFile Input Format Counters \n",
            "\t\tBytes Read=18289443\n",
            "\tFile Output Format Counters \n",
            "\t\tBytes Written=26\n",
            "2024-05-05 15:50:35,832 INFO streaming.StreamJob: Output directory: output_simplest\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "If the `output` directory contains the empty file `_SUCCESS`, this means that the job was successful."
      ],
      "metadata": {
        "id": "UiZ6FH2gFfE5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check the output of the MapReduce job."
      ],
      "metadata": {
        "id": "kHPEoIIWFubx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!hdfs dfs -cat output_simplest/part-00000"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rB7FXYTbwNzm",
        "outputId": "5022e43e-32ad-4fdf-f8d5-c2293cc6c9b3"
      },
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  10000   10000 18299443\t\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The number of words is in this case equal to the number of lines because there are no word separators (empty spaces) in the file, so each line is a word."
      ],
      "metadata": {
        "id": "BDObCPW2F39S"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Another MapReduce example: filter a log file\n",
        "\n",
        "We're going to use a Linux logfile and look for the string `sshd` in a given position. The file stems from [Loghub](https://github.com/logpai/loghub), a freely available collection of system logs for AI-driven log analytics research.\n",
        "\n",
        "The mapper `mapper.py` filters the file for the given string `sshd` at field 4.\n",
        "\n",
        "The job has no reducer (option `-reducer NONE`). Note that without a reducer the sorting and shuffling phase after the map phase is skipped.\n"
      ],
      "metadata": {
        "id": "BbosNo0TD3oH"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Download the logfile `Linux_2k.log`:"
      ],
      "metadata": {
        "id": "iVdUuulwGzq5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!wget --no-clobber https://raw.githubusercontent.com/logpai/loghub/master/Linux/Linux_2k.log"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yJIm4SPZFPxy",
        "outputId": "ebdb1fa9-25ad-4700-99bd-317cfab80dee"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2024-05-05 15:50:40--  https://raw.githubusercontent.com/logpai/loghub/master/Linux/Linux_2k.log\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.108.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 216485 (211K) [text/plain]\n",
            "Saving to: ‘Linux_2k.log’\n",
            "\n",
            "\rLinux_2k.log          0%[                    ]       0  --.-KB/s               \rLinux_2k.log        100%[===================>] 211.41K  --.-KB/s    in 0.005s  \n",
            "\n",
            "2024-05-05 15:50:40 (43.8 MB/s) - ‘Linux_2k.log’ saved [216485/216485]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "hdfs dfs -mkdir input || true\n",
        "hdfs dfs -put Linux_2k.log input/ || true"
      ],
      "metadata": {
        "id": "M1WgyQE3MYWI"
      },
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Define the mapper"
      ],
      "metadata": {
        "id": "ILUOCdzEH3Gm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile mapper.py\n",
        "#!/usr/bin/env python\n",
        "import sys\n",
        "\n",
        "for line in sys.stdin:\n",
        "    # split the line into words\n",
        "    line = line.strip()\n",
        "    fields = line.split()\n",
        "    if (len(fields)>=5 and fields[4].startswith('sshd')):\n",
        "      print(line)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4-rraIUdfdj0",
        "outputId": "2bfe0a31-7876-4e57-f559-b44c852b98f4"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Writing mapper.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Test the script (after setting the correct permissions)"
      ],
      "metadata": {
        "id": "W8AxdFFPIuDo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!chmod 700 mapper.py"
      ],
      "metadata": {
        "id": "QwOk_y7egbGM"
      },
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Look at the first 10 lines"
      ],
      "metadata": {
        "id": "fhv95VzfIAnz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!head -10 Linux_2k.log"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9qf1dFqIKgoJ",
        "outputId": "40422b84-9d9f-4d86-9026-199cec0466d0"
      },
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Jun 14 15:16:01 combo sshd(pam_unix)[19939]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4 \r\n",
            "Jun 14 15:16:02 combo sshd(pam_unix)[19937]: check pass; user unknown\r\n",
            "Jun 14 15:16:02 combo sshd(pam_unix)[19937]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4 \r\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20882]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\r\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20884]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\r\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20883]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\r\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20885]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\r\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20886]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\r\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20892]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\r\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20893]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\r\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Test the mapper in the shell (not using MapReduce):"
      ],
      "metadata": {
        "id": "eQ09Y1AqR6Fy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!head -100 Linux_2k.log| ./mapper.py"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TJ0kDRsigCC2",
        "outputId": "ea8940b5-465f-4027-bfda-9387e813ca5e"
      },
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Jun 14 15:16:01 combo sshd(pam_unix)[19939]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4\n",
            "Jun 14 15:16:02 combo sshd(pam_unix)[19937]: check pass; user unknown\n",
            "Jun 14 15:16:02 combo sshd(pam_unix)[19937]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20882]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20884]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20883]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20885]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20886]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20892]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20893]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20896]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20897]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20898]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23397]: check pass; user unknown\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23397]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23395]: check pass; user unknown\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23395]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23404]: check pass; user unknown\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23404]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23399]: check pass; user unknown\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23399]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23406]: check pass; user unknown\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23406]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23396]: check pass; user unknown\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23394]: check pass; user unknown\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23407]: check pass; user unknown\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23394]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23403]: check pass; user unknown\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23396]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23407]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23403]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23412]: check pass; user unknown\n",
            "Jun 15 12:12:34 combo sshd(pam_unix)[23412]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4\n",
            "Jun 15 12:13:19 combo sshd(pam_unix)[23414]: check pass; user unknown\n",
            "Jun 15 12:13:19 combo sshd(pam_unix)[23414]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4\n",
            "Jun 15 12:13:20 combo sshd(pam_unix)[23416]: check pass; user unknown\n",
            "Jun 15 12:13:20 combo sshd(pam_unix)[23416]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4\n",
            "Jun 15 14:53:32 combo sshd(pam_unix)[23661]: check pass; user unknown\n",
            "Jun 15 14:53:32 combo sshd(pam_unix)[23661]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=061092085098.ctinets.com\n",
            "Jun 15 14:53:32 combo sshd(pam_unix)[23663]: check pass; user unknown\n",
            "Jun 15 14:53:32 combo sshd(pam_unix)[23663]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=061092085098.ctinets.com\n",
            "Jun 15 14:53:32 combo sshd(pam_unix)[23664]: check pass; user unknown\n",
            "Jun 15 14:53:32 combo sshd(pam_unix)[23664]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=061092085098.ctinets.com\n",
            "Jun 15 14:53:33 combo sshd(pam_unix)[23665]: check pass; user unknown\n",
            "Jun 15 14:53:33 combo sshd(pam_unix)[23665]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=061092085098.ctinets.com\n",
            "Jun 15 14:53:34 combo sshd(pam_unix)[23669]: check pass; user unknown\n",
            "Jun 15 14:53:34 combo sshd(pam_unix)[23669]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=061092085098.ctinets.com\n",
            "Jun 15 14:53:35 combo sshd(pam_unix)[23671]: check pass; user unknown\n",
            "Jun 15 14:53:35 combo sshd(pam_unix)[23671]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=061092085098.ctinets.com\n",
            "Jun 15 14:53:35 combo sshd(pam_unix)[23673]: check pass; user unknown\n",
            "Jun 15 14:53:35 combo sshd(pam_unix)[23673]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=061092085098.ctinets.com\n",
            "Jun 15 14:53:35 combo sshd(pam_unix)[23674]: check pass; user unknown\n",
            "Jun 15 14:53:35 combo sshd(pam_unix)[23674]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=061092085098.ctinets.com\n",
            "Jun 15 14:53:36 combo sshd(pam_unix)[23678]: check pass; user unknown\n",
            "Jun 15 14:53:36 combo sshd(pam_unix)[23678]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=061092085098.ctinets.com\n",
            "Jun 15 14:53:36 combo sshd(pam_unix)[23677]: check pass; user unknown\n",
            "Jun 15 14:53:36 combo sshd(pam_unix)[23677]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=061092085098.ctinets.com\n",
            "Jun 15 20:05:31 combo sshd(pam_unix)[24138]: check pass; user unknown\n",
            "Jun 15 20:05:31 combo sshd(pam_unix)[24138]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=d211-116-254-214.rev.krline.net\n",
            "Jun 15 20:05:31 combo sshd(pam_unix)[24137]: check pass; user unknown\n",
            "Jun 15 20:05:31 combo sshd(pam_unix)[24137]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=d211-116-254-214.rev.krline.net\n",
            "Jun 15 20:05:31 combo sshd(pam_unix)[24141]: check pass; user unknown\n",
            "Jun 15 20:05:31 combo sshd(pam_unix)[24141]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=d211-116-254-214.rev.krline.net\n",
            "Jun 15 20:05:31 combo sshd(pam_unix)[24140]: check pass; user unknown\n",
            "Jun 15 20:05:31 combo sshd(pam_unix)[24140]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=d211-116-254-214.rev.krline.net\n",
            "Jun 15 20:05:31 combo sshd(pam_unix)[24139]: check pass; user unknown\n",
            "Jun 15 20:05:31 combo sshd(pam_unix)[24139]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=d211-116-254-214.rev.krline.net\n",
            "Jun 17 19:43:13 combo sshd(pam_unix)[30565]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=211.46.224.253  user=guest\n",
            "Jun 17 20:29:26 combo sshd(pam_unix)[30631]: session opened for user test by (uid=509)\n",
            "Jun 17 20:34:57 combo sshd(pam_unix)[30631]: session closed for user test\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now run the MapReduce job on the pseudo-cluster"
      ],
      "metadata": {
        "id": "DXbgh5g7OraF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "\n",
        "hdfs dfs -rm -r output_filter\n",
        "\n",
        "mapred streaming \\\n",
        "  -file mapper.py \\\n",
        "  -input input \\\n",
        "  -output output_filter \\\n",
        "  -mapper mapper.py \\\n",
        "  -reducer NONE\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "G7SEzMC2OqWW",
        "outputId": "062817e1-5739-4411-fbce-79868b807ad0"
      },
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "packageJobJar: [mapper.py] [] /tmp/streamjob16384221810720665594.jar tmpDir=null\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "rm: `output_filter': No such file or directory\n",
            "2024-05-05 15:50:54,106 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.\n",
            "2024-05-05 15:50:56,651 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties\n",
            "2024-05-05 15:50:56,857 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).\n",
            "2024-05-05 15:50:56,857 INFO impl.MetricsSystemImpl: JobTracker metrics system started\n",
            "2024-05-05 15:50:56,882 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!\n",
            "2024-05-05 15:50:57,394 INFO mapred.FileInputFormat: Total input files to process : 1\n",
            "2024-05-05 15:50:57,414 INFO mapreduce.JobSubmitter: number of splits:1\n",
            "2024-05-05 15:50:57,742 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1455494043_0001\n",
            "2024-05-05 15:50:57,743 INFO mapreduce.JobSubmitter: Executing with tokens: []\n",
            "2024-05-05 15:50:58,142 INFO mapred.LocalDistributedCacheManager: Localized file:/content/mapper.py as file:/tmp/hadoop-root/mapred/local/job_local1455494043_0001_242b8a02-ef2e-4f18-ba70-08eb1d29141d/mapper.py\n",
            "2024-05-05 15:50:58,315 INFO mapreduce.Job: The url to track the job: http://localhost:8080/\n",
            "2024-05-05 15:50:58,317 INFO mapreduce.Job: Running job: job_local1455494043_0001\n",
            "2024-05-05 15:50:58,324 INFO mapred.LocalJobRunner: OutputCommitter set in config null\n",
            "2024-05-05 15:50:58,327 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter\n",
            "2024-05-05 15:50:58,333 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2\n",
            "2024-05-05 15:50:58,333 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false\n",
            "2024-05-05 15:50:58,404 INFO mapred.LocalJobRunner: Waiting for map tasks\n",
            "2024-05-05 15:50:58,409 INFO mapred.LocalJobRunner: Starting task: attempt_local1455494043_0001_m_000000_0\n",
            "2024-05-05 15:50:58,450 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2\n",
            "2024-05-05 15:50:58,450 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false\n",
            "2024-05-05 15:50:58,488 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]\n",
            "2024-05-05 15:50:58,501 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/root/input/Linux_2k.log:0+216485\n",
            "2024-05-05 15:50:58,534 INFO mapred.MapTask: numReduceTasks: 0\n",
            "2024-05-05 15:50:58,655 INFO streaming.PipeMapRed: PipeMapRed exec [/content/./mapper.py]\n",
            "2024-05-05 15:50:58,669 INFO Configuration.deprecation: mapred.work.output.dir is deprecated. Instead, use mapreduce.task.output.dir\n",
            "2024-05-05 15:50:58,682 INFO Configuration.deprecation: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir\n",
            "2024-05-05 15:50:58,684 INFO Configuration.deprecation: map.input.file is deprecated. Instead, use mapreduce.map.input.file\n",
            "2024-05-05 15:50:58,684 INFO Configuration.deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length\n",
            "2024-05-05 15:50:58,685 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id\n",
            "2024-05-05 15:50:58,685 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition\n",
            "2024-05-05 15:50:58,687 INFO Configuration.deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start\n",
            "2024-05-05 15:50:58,688 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap\n",
            "2024-05-05 15:50:58,688 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id\n",
            "2024-05-05 15:50:58,689 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id\n",
            "2024-05-05 15:50:58,689 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords\n",
            "2024-05-05 15:50:58,690 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name\n",
            "2024-05-05 15:50:58,879 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2024-05-05 15:50:58,880 INFO streaming.PipeMapRed: R/W/S=10/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2024-05-05 15:50:58,883 INFO streaming.PipeMapRed: R/W/S=100/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2024-05-05 15:50:58,906 INFO streaming.PipeMapRed: R/W/S=1000/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2024-05-05 15:50:58,919 INFO streaming.PipeMapRed: Records R/W=1611/1\n",
            "2024-05-05 15:50:58,949 INFO streaming.PipeMapRed: MRErrorThread done\n",
            "2024-05-05 15:50:58,965 INFO streaming.PipeMapRed: mapRedFinished\n",
            "2024-05-05 15:50:58,969 INFO mapred.LocalJobRunner: \n",
            "2024-05-05 15:50:59,069 INFO mapred.Task: Task:attempt_local1455494043_0001_m_000000_0 is done. And is in the process of committing\n",
            "2024-05-05 15:50:59,077 INFO mapred.LocalJobRunner: \n",
            "2024-05-05 15:50:59,077 INFO mapred.Task: Task attempt_local1455494043_0001_m_000000_0 is allowed to commit now\n",
            "2024-05-05 15:50:59,109 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1455494043_0001_m_000000_0' to hdfs://localhost:9000/user/root/output_filter\n",
            "2024-05-05 15:50:59,110 INFO mapred.LocalJobRunner: Records R/W=1611/1\n",
            "2024-05-05 15:50:59,111 INFO mapred.Task: Task 'attempt_local1455494043_0001_m_000000_0' done.\n",
            "2024-05-05 15:50:59,120 INFO mapred.Task: Final Counters for attempt_local1455494043_0001_m_000000_0: Counters: 21\n",
            "\tFile System Counters\n",
            "\t\tFILE: Number of bytes read=663\n",
            "\t\tFILE: Number of bytes written=718999\n",
            "\t\tFILE: Number of read operations=0\n",
            "\t\tFILE: Number of large read operations=0\n",
            "\t\tFILE: Number of write operations=0\n",
            "\t\tHDFS: Number of bytes read=216485\n",
            "\t\tHDFS: Number of bytes written=85436\n",
            "\t\tHDFS: Number of read operations=9\n",
            "\t\tHDFS: Number of large read operations=0\n",
            "\t\tHDFS: Number of write operations=3\n",
            "\t\tHDFS: Number of bytes read erasure-coded=0\n",
            "\tMap-Reduce Framework\n",
            "\t\tMap input records=2000\n",
            "\t\tMap output records=677\n",
            "\t\tInput split bytes=102\n",
            "\t\tSpilled Records=0\n",
            "\t\tFailed Shuffles=0\n",
            "\t\tMerged Map outputs=0\n",
            "\t\tGC time elapsed (ms)=21\n",
            "\t\tTotal committed heap usage (bytes)=432013312\n",
            "\tFile Input Format Counters \n",
            "\t\tBytes Read=216485\n",
            "\tFile Output Format Counters \n",
            "\t\tBytes Written=85436\n",
            "2024-05-05 15:50:59,120 INFO mapred.LocalJobRunner: Finishing task: attempt_local1455494043_0001_m_000000_0\n",
            "2024-05-05 15:50:59,121 INFO mapred.LocalJobRunner: map task executor complete.\n",
            "2024-05-05 15:50:59,323 INFO mapreduce.Job: Job job_local1455494043_0001 running in uber mode : false\n",
            "2024-05-05 15:50:59,324 INFO mapreduce.Job:  map 100% reduce 0%\n",
            "2024-05-05 15:50:59,327 INFO mapreduce.Job: Job job_local1455494043_0001 completed successfully\n",
            "2024-05-05 15:50:59,334 INFO mapreduce.Job: Counters: 21\n",
            "\tFile System Counters\n",
            "\t\tFILE: Number of bytes read=663\n",
            "\t\tFILE: Number of bytes written=718999\n",
            "\t\tFILE: Number of read operations=0\n",
            "\t\tFILE: Number of large read operations=0\n",
            "\t\tFILE: Number of write operations=0\n",
            "\t\tHDFS: Number of bytes read=216485\n",
            "\t\tHDFS: Number of bytes written=85436\n",
            "\t\tHDFS: Number of read operations=9\n",
            "\t\tHDFS: Number of large read operations=0\n",
            "\t\tHDFS: Number of write operations=3\n",
            "\t\tHDFS: Number of bytes read erasure-coded=0\n",
            "\tMap-Reduce Framework\n",
            "\t\tMap input records=2000\n",
            "\t\tMap output records=677\n",
            "\t\tInput split bytes=102\n",
            "\t\tSpilled Records=0\n",
            "\t\tFailed Shuffles=0\n",
            "\t\tMerged Map outputs=0\n",
            "\t\tGC time elapsed (ms)=21\n",
            "\t\tTotal committed heap usage (bytes)=432013312\n",
            "\tFile Input Format Counters \n",
            "\t\tBytes Read=216485\n",
            "\tFile Output Format Counters \n",
            "\t\tBytes Written=85436\n",
            "2024-05-05 15:50:59,334 INFO streaming.StreamJob: Output directory: output_filter\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check the result"
      ],
      "metadata": {
        "id": "iuZJ2ACzSTJi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!hdfs dfs -ls output_filter"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RhLA5HZEhfmT",
        "outputId": "743a7dab-4075-490b-84ec-c99ce0154957"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found 2 items\n",
            "-rw-r--r--   1 root supergroup          0 2024-05-05 15:50 output_filter/_SUCCESS\n",
            "-rw-r--r--   1 root supergroup      85436 2024-05-05 15:50 output_filter/part-00000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!hdfs dfs -cat output_filter/part-00000 |head"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ffi4RvXnPH14",
        "outputId": "91314072-7972-4a17-dbf2-419a262a9698"
      },
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Jun 14 15:16:01 combo sshd(pam_unix)[19939]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4\t\n",
            "Jun 14 15:16:02 combo sshd(pam_unix)[19937]: check pass; user unknown\t\n",
            "Jun 14 15:16:02 combo sshd(pam_unix)[19937]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4\t\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20882]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\t\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20884]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\t\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20883]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\t\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20885]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\t\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20886]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\t\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20892]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\t\n",
            "Jun 15 02:04:59 combo sshd(pam_unix)[20893]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=220-135-151-1.hinet-ip.hinet.net  user=root\t\n",
            "cat: Unable to write to output stream.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Aggregate data with MapReduce\n",
        "\n",
        "Following the example in [Hadoop Streaming/Aggregate package](https://hadoop.apache.org/docs/stable/hadoop-streaming/HadoopStreaming.html#Hadoop_Aggregate_Package)"
      ],
      "metadata": {
        "id": "Sam22f-YT1xR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile myAggregatorForKeyCount.py\n",
        "#!/usr/local/bin/python\n",
        "import sys\n",
        "\n",
        "def generateLongCountToken(id):\n",
        "    return \"LongValueSum:\" + id + \"\\t\" + \"1\"\n",
        "\n",
        "def main(argv):\n",
        "    line = sys.stdin.readline()\n",
        "    try:\n",
        "        while line:\n",
        "            line = line[:-1]\n",
        "            fields = line.split()\n",
        "            s = fields[4].split('[')[0]\n",
        "            print(generateLongCountToken(s))\n",
        "            line = sys.stdin.readline()\n",
        "    except \"end of file\":\n",
        "        return None\n",
        "\n",
        "if __name__ == \"__main__\":\n",
        "     main(sys.argv)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fMKEqUF1T-v9",
        "outputId": "31a2e256-a9a6-462b-be13-ab4eb9510285"
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Writing myAggregatorForKeyCount.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Set permissions"
      ],
      "metadata": {
        "id": "4b2S9K8FWDMz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!chmod 700 myAggregatorForKeyCount.py"
      ],
      "metadata": {
        "id": "35DP8K2_WDYO"
      },
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Test the mapper"
      ],
      "metadata": {
        "id": "r9M8lgxMVRYz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!head -20 Linux_2k.log| ./myAggregatorForKeyCount.py"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k-R7VNoTVRjL",
        "outputId": "2c13e07b-5dce-4072-bfcb-b5565761510a"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "LongValueSum:sshd(pam_unix)\t1\n",
            "LongValueSum:sshd(pam_unix)\t1\n",
            "LongValueSum:sshd(pam_unix)\t1\n",
            "LongValueSum:sshd(pam_unix)\t1\n",
            "LongValueSum:sshd(pam_unix)\t1\n",
            "LongValueSum:sshd(pam_unix)\t1\n",
            "LongValueSum:sshd(pam_unix)\t1\n",
            "LongValueSum:sshd(pam_unix)\t1\n",
            "LongValueSum:sshd(pam_unix)\t1\n",
            "LongValueSum:sshd(pam_unix)\t1\n",
            "LongValueSum:sshd(pam_unix)\t1\n",
            "LongValueSum:sshd(pam_unix)\t1\n",
            "LongValueSum:sshd(pam_unix)\t1\n",
            "LongValueSum:su(pam_unix)\t1\n",
            "LongValueSum:su(pam_unix)\t1\n",
            "LongValueSum:logrotate:\t1\n",
            "LongValueSum:su(pam_unix)\t1\n",
            "LongValueSum:su(pam_unix)\t1\n",
            "LongValueSum:sshd(pam_unix)\t1\n",
            "LongValueSum:sshd(pam_unix)\t1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Run the MapReduce job"
      ],
      "metadata": {
        "id": "vOEpMFvsVRtM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "\n",
        "hdfs dfs -rm -r output_aggregate\n",
        "\n",
        "mapred streaming \\\n",
        "  -file mapper.py \\\n",
        "  -file myAggregatorForKeyCount.py \\\n",
        "  -input input \\\n",
        "  -output output_aggregate \\\n",
        "  -mapper myAggregatorForKeyCount.py \\\n",
        "  -reducer aggregate\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XwxHJ7yyVR34",
        "outputId": "f995d90a-d1ce-493e-9ef7-8cda98c60e92"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "packageJobJar: [mapper.py, myAggregatorForKeyCount.py] [] /tmp/streamjob13488001389618729360.jar tmpDir=null\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "rm: `output_aggregate': No such file or directory\n",
            "2024-05-05 15:51:12,805 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.\n",
            "2024-05-05 15:51:14,518 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties\n",
            "2024-05-05 15:51:14,692 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).\n",
            "2024-05-05 15:51:14,692 INFO impl.MetricsSystemImpl: JobTracker metrics system started\n",
            "2024-05-05 15:51:14,716 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!\n",
            "2024-05-05 15:51:15,212 INFO mapred.FileInputFormat: Total input files to process : 1\n",
            "2024-05-05 15:51:15,237 INFO mapreduce.JobSubmitter: number of splits:1\n",
            "2024-05-05 15:51:15,579 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1844526749_0001\n",
            "2024-05-05 15:51:15,579 INFO mapreduce.JobSubmitter: Executing with tokens: []\n",
            "2024-05-05 15:51:16,012 INFO mapred.LocalDistributedCacheManager: Localized file:/content/mapper.py as file:/tmp/hadoop-root/mapred/local/job_local1844526749_0001_8c3e4266-2cef-4679-841e-ce1be9e2ce9a/mapper.py\n",
            "2024-05-05 15:51:16,052 INFO mapred.LocalDistributedCacheManager: Localized file:/content/myAggregatorForKeyCount.py as file:/tmp/hadoop-root/mapred/local/job_local1844526749_0001_33c29804-ab07-4044-8314-7c0eff25eeff/myAggregatorForKeyCount.py\n",
            "2024-05-05 15:51:16,197 INFO mapreduce.Job: The url to track the job: http://localhost:8080/\n",
            "2024-05-05 15:51:16,200 INFO mapreduce.Job: Running job: job_local1844526749_0001\n",
            "2024-05-05 15:51:16,208 INFO mapred.LocalJobRunner: OutputCommitter set in config null\n",
            "2024-05-05 15:51:16,211 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter\n",
            "2024-05-05 15:51:16,217 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2\n",
            "2024-05-05 15:51:16,217 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false\n",
            "2024-05-05 15:51:16,301 INFO mapred.LocalJobRunner: Waiting for map tasks\n",
            "2024-05-05 15:51:16,309 INFO mapred.LocalJobRunner: Starting task: attempt_local1844526749_0001_m_000000_0\n",
            "2024-05-05 15:51:16,349 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2\n",
            "2024-05-05 15:51:16,349 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false\n",
            "2024-05-05 15:51:16,388 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]\n",
            "2024-05-05 15:51:16,400 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/root/input/Linux_2k.log:0+216485\n",
            "2024-05-05 15:51:16,439 INFO mapred.MapTask: numReduceTasks: 1\n",
            "2024-05-05 15:51:16,530 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)\n",
            "2024-05-05 15:51:16,530 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100\n",
            "2024-05-05 15:51:16,530 INFO mapred.MapTask: soft limit at 83886080\n",
            "2024-05-05 15:51:16,530 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600\n",
            "2024-05-05 15:51:16,530 INFO mapred.MapTask: kvstart = 26214396; length = 6553600\n",
            "2024-05-05 15:51:16,558 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer\n",
            "2024-05-05 15:51:16,566 INFO streaming.PipeMapRed: PipeMapRed exec [/content/./myAggregatorForKeyCount.py]\n",
            "2024-05-05 15:51:16,575 INFO Configuration.deprecation: mapred.work.output.dir is deprecated. Instead, use mapreduce.task.output.dir\n",
            "2024-05-05 15:51:16,578 INFO Configuration.deprecation: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir\n",
            "2024-05-05 15:51:16,579 INFO Configuration.deprecation: map.input.file is deprecated. Instead, use mapreduce.map.input.file\n",
            "2024-05-05 15:51:16,580 INFO Configuration.deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length\n",
            "2024-05-05 15:51:16,583 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id\n",
            "2024-05-05 15:51:16,583 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition\n",
            "2024-05-05 15:51:16,586 INFO Configuration.deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start\n",
            "2024-05-05 15:51:16,587 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap\n",
            "2024-05-05 15:51:16,588 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id\n",
            "2024-05-05 15:51:16,588 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id\n",
            "2024-05-05 15:51:16,594 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords\n",
            "2024-05-05 15:51:16,595 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name\n",
            "2024-05-05 15:51:16,826 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2024-05-05 15:51:16,826 INFO streaming.PipeMapRed: R/W/S=10/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2024-05-05 15:51:16,829 INFO streaming.PipeMapRed: R/W/S=100/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2024-05-05 15:51:16,853 INFO streaming.PipeMapRed: R/W/S=1000/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2024-05-05 15:51:16,859 INFO streaming.PipeMapRed: Records R/W=1201/1\n",
            "2024-05-05 15:51:16,890 INFO streaming.PipeMapRed: MRErrorThread done\n",
            "2024-05-05 15:51:16,892 INFO streaming.PipeMapRed: mapRedFinished\n",
            "2024-05-05 15:51:16,896 INFO mapred.LocalJobRunner: \n",
            "2024-05-05 15:51:16,896 INFO mapred.MapTask: Starting flush of map output\n",
            "2024-05-05 15:51:16,896 INFO mapred.MapTask: Spilling map output\n",
            "2024-05-05 15:51:16,896 INFO mapred.MapTask: bufstart = 0; bufend = 48923; bufvoid = 104857600\n",
            "2024-05-05 15:51:16,896 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26206400(104825600); length = 7997/6553600\n",
            "2024-05-05 15:51:16,956 INFO mapred.MapTask: Finished spill 0\n",
            "2024-05-05 15:51:16,978 INFO mapred.Task: Task:attempt_local1844526749_0001_m_000000_0 is done. And is in the process of committing\n",
            "2024-05-05 15:51:16,987 INFO mapred.LocalJobRunner: Records R/W=1201/1\n",
            "2024-05-05 15:51:16,987 INFO mapred.Task: Task 'attempt_local1844526749_0001_m_000000_0' done.\n",
            "2024-05-05 15:51:16,999 INFO mapred.Task: Final Counters for attempt_local1844526749_0001_m_000000_0: Counters: 24\n",
            "\tFile System Counters\n",
            "\t\tFILE: Number of bytes read=1533\n",
            "\t\tFILE: Number of bytes written=721185\n",
            "\t\tFILE: Number of read operations=0\n",
            "\t\tFILE: Number of large read operations=0\n",
            "\t\tFILE: Number of write operations=0\n",
            "\t\tHDFS: Number of bytes read=216485\n",
            "\t\tHDFS: Number of bytes written=0\n",
            "\t\tHDFS: Number of read operations=5\n",
            "\t\tHDFS: Number of large read operations=0\n",
            "\t\tHDFS: Number of write operations=1\n",
            "\t\tHDFS: Number of bytes read erasure-coded=0\n",
            "\tMap-Reduce Framework\n",
            "\t\tMap input records=2000\n",
            "\t\tMap output records=2000\n",
            "\t\tMap output bytes=48923\n",
            "\t\tMap output materialized bytes=782\n",
            "\t\tInput split bytes=102\n",
            "\t\tCombine input records=2000\n",
            "\t\tCombine output records=30\n",
            "\t\tSpilled Records=30\n",
            "\t\tFailed Shuffles=0\n",
            "\t\tMerged Map outputs=0\n",
            "\t\tGC time elapsed (ms)=21\n",
            "\t\tTotal committed heap usage (bytes)=520093696\n",
            "\tFile Input Format Counters \n",
            "\t\tBytes Read=216485\n",
            "2024-05-05 15:51:16,999 INFO mapred.LocalJobRunner: Finishing task: attempt_local1844526749_0001_m_000000_0\n",
            "2024-05-05 15:51:16,999 INFO mapred.LocalJobRunner: map task executor complete.\n",
            "2024-05-05 15:51:17,004 INFO mapred.LocalJobRunner: Waiting for reduce tasks\n",
            "2024-05-05 15:51:17,010 INFO mapred.LocalJobRunner: Starting task: attempt_local1844526749_0001_r_000000_0\n",
            "2024-05-05 15:51:17,034 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2\n",
            "2024-05-05 15:51:17,035 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false\n",
            "2024-05-05 15:51:17,035 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]\n",
            "2024-05-05 15:51:17,041 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@54252d3b\n",
            "2024-05-05 15:51:17,043 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!\n",
            "2024-05-05 15:51:17,067 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=2382574336, maxSingleShuffleLimit=595643584, mergeThreshold=1572499072, ioSortFactor=10, memToMemMergeOutputsThreshold=10\n",
            "2024-05-05 15:51:17,084 INFO reduce.EventFetcher: attempt_local1844526749_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events\n",
            "2024-05-05 15:51:17,121 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1844526749_0001_m_000000_0 decomp: 778 len: 782 to MEMORY\n",
            "2024-05-05 15:51:17,126 INFO reduce.InMemoryMapOutput: Read 778 bytes from map-output for attempt_local1844526749_0001_m_000000_0\n",
            "2024-05-05 15:51:17,128 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 778, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->778\n",
            "2024-05-05 15:51:17,132 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning\n",
            "2024-05-05 15:51:17,134 INFO mapred.LocalJobRunner: 1 / 1 copied.\n",
            "2024-05-05 15:51:17,134 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs\n",
            "2024-05-05 15:51:17,145 INFO mapred.Merger: Merging 1 sorted segments\n",
            "2024-05-05 15:51:17,146 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 760 bytes\n",
            "2024-05-05 15:51:17,148 INFO reduce.MergeManagerImpl: Merged 1 segments, 778 bytes to disk to satisfy reduce memory limit\n",
            "2024-05-05 15:51:17,149 INFO reduce.MergeManagerImpl: Merging 1 files, 782 bytes from disk\n",
            "2024-05-05 15:51:17,149 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce\n",
            "2024-05-05 15:51:17,150 INFO mapred.Merger: Merging 1 sorted segments\n",
            "2024-05-05 15:51:17,150 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 760 bytes\n",
            "2024-05-05 15:51:17,151 INFO mapred.LocalJobRunner: 1 / 1 copied.\n",
            "2024-05-05 15:51:17,206 INFO mapreduce.Job: Job job_local1844526749_0001 running in uber mode : false\n",
            "2024-05-05 15:51:17,207 INFO mapreduce.Job:  map 100% reduce 0%\n",
            "2024-05-05 15:51:17,275 INFO mapred.Task: Task:attempt_local1844526749_0001_r_000000_0 is done. And is in the process of committing\n",
            "2024-05-05 15:51:17,281 INFO mapred.LocalJobRunner: 1 / 1 copied.\n",
            "2024-05-05 15:51:17,281 INFO mapred.Task: Task attempt_local1844526749_0001_r_000000_0 is allowed to commit now\n",
            "2024-05-05 15:51:17,307 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1844526749_0001_r_000000_0' to hdfs://localhost:9000/user/root/output_aggregate\n",
            "2024-05-05 15:51:17,309 INFO mapred.LocalJobRunner: reduce > reduce\n",
            "2024-05-05 15:51:17,309 INFO mapred.Task: Task 'attempt_local1844526749_0001_r_000000_0' done.\n",
            "2024-05-05 15:51:17,310 INFO mapred.Task: Final Counters for attempt_local1844526749_0001_r_000000_0: Counters: 30\n",
            "\tFile System Counters\n",
            "\t\tFILE: Number of bytes read=3129\n",
            "\t\tFILE: Number of bytes written=721967\n",
            "\t\tFILE: Number of read operations=0\n",
            "\t\tFILE: Number of large read operations=0\n",
            "\t\tFILE: Number of write operations=0\n",
            "\t\tHDFS: Number of bytes read=216485\n",
            "\t\tHDFS: Number of bytes written=326\n",
            "\t\tHDFS: Number of read operations=10\n",
            "\t\tHDFS: Number of large read operations=0\n",
            "\t\tHDFS: Number of write operations=3\n",
            "\t\tHDFS: Number of bytes read erasure-coded=0\n",
            "\tMap-Reduce Framework\n",
            "\t\tCombine input records=0\n",
            "\t\tCombine output records=0\n",
            "\t\tReduce input groups=30\n",
            "\t\tReduce shuffle bytes=782\n",
            "\t\tReduce input records=30\n",
            "\t\tReduce output records=30\n",
            "\t\tSpilled Records=30\n",
            "\t\tShuffled Maps =1\n",
            "\t\tFailed Shuffles=0\n",
            "\t\tMerged Map outputs=1\n",
            "\t\tGC time elapsed (ms)=0\n",
            "\t\tTotal committed heap usage (bytes)=520093696\n",
            "\tShuffle Errors\n",
            "\t\tBAD_ID=0\n",
            "\t\tCONNECTION=0\n",
            "\t\tIO_ERROR=0\n",
            "\t\tWRONG_LENGTH=0\n",
            "\t\tWRONG_MAP=0\n",
            "\t\tWRONG_REDUCE=0\n",
            "\tFile Output Format Counters \n",
            "\t\tBytes Written=326\n",
            "2024-05-05 15:51:17,310 INFO mapred.LocalJobRunner: Finishing task: attempt_local1844526749_0001_r_000000_0\n",
            "2024-05-05 15:51:17,310 INFO mapred.LocalJobRunner: reduce task executor complete.\n",
            "2024-05-05 15:51:18,209 INFO mapreduce.Job:  map 100% reduce 100%\n",
            "2024-05-05 15:51:18,210 INFO mapreduce.Job: Job job_local1844526749_0001 completed successfully\n",
            "2024-05-05 15:51:18,228 INFO mapreduce.Job: Counters: 36\n",
            "\tFile System Counters\n",
            "\t\tFILE: Number of bytes read=4662\n",
            "\t\tFILE: Number of bytes written=1443152\n",
            "\t\tFILE: Number of read operations=0\n",
            "\t\tFILE: Number of large read operations=0\n",
            "\t\tFILE: Number of write operations=0\n",
            "\t\tHDFS: Number of bytes read=432970\n",
            "\t\tHDFS: Number of bytes written=326\n",
            "\t\tHDFS: Number of read operations=15\n",
            "\t\tHDFS: Number of large read operations=0\n",
            "\t\tHDFS: Number of write operations=4\n",
            "\t\tHDFS: Number of bytes read erasure-coded=0\n",
            "\tMap-Reduce Framework\n",
            "\t\tMap input records=2000\n",
            "\t\tMap output records=2000\n",
            "\t\tMap output bytes=48923\n",
            "\t\tMap output materialized bytes=782\n",
            "\t\tInput split bytes=102\n",
            "\t\tCombine input records=2000\n",
            "\t\tCombine output records=30\n",
            "\t\tReduce input groups=30\n",
            "\t\tReduce shuffle bytes=782\n",
            "\t\tReduce input records=30\n",
            "\t\tReduce output records=30\n",
            "\t\tSpilled Records=60\n",
            "\t\tShuffled Maps =1\n",
            "\t\tFailed Shuffles=0\n",
            "\t\tMerged Map outputs=1\n",
            "\t\tGC time elapsed (ms)=21\n",
            "\t\tTotal committed heap usage (bytes)=1040187392\n",
            "\tShuffle Errors\n",
            "\t\tBAD_ID=0\n",
            "\t\tCONNECTION=0\n",
            "\t\tIO_ERROR=0\n",
            "\t\tWRONG_LENGTH=0\n",
            "\t\tWRONG_MAP=0\n",
            "\t\tWRONG_REDUCE=0\n",
            "\tFile Input Format Counters \n",
            "\t\tBytes Read=216485\n",
            "\tFile Output Format Counters \n",
            "\t\tBytes Written=326\n",
            "2024-05-05 15:51:18,228 INFO streaming.StreamJob: Output directory: output_aggregate\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check result"
      ],
      "metadata": {
        "id": "NkuYUkh5W0Je"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "hdfs dfs -ls output_aggregate\n",
        "hdfs dfs -cat output_aggregate/part-00000"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ET3KCfX1UC2u",
        "outputId": "55acd3ac-97b0-4875-995c-ef8448b00858"
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found 2 items\n",
            "-rw-r--r--   1 root supergroup          0 2024-05-05 15:51 output_aggregate/_SUCCESS\n",
            "-rw-r--r--   1 root supergroup        326 2024-05-05 15:51 output_aggregate/part-00000\n",
            "--\t1\n",
            "bluetooth:\t2\n",
            "cups:\t12\n",
            "ftpd\t916\n",
            "gdm(pam_unix)\t2\n",
            "gdm-binary\t1\n",
            "gpm\t2\n",
            "hcid\t1\n",
            "irqbalance:\t1\n",
            "kernel:\t76\n",
            "klogind\t46\n",
            "login(pam_unix)\t2\n",
            "logrotate:\t43\n",
            "named\t16\n",
            "network:\t2\n",
            "nfslock:\t1\n",
            "portmap:\t1\n",
            "random:\t1\n",
            "rc:\t1\n",
            "rpc.statd\t1\n",
            "rpcidmapd:\t1\n",
            "sdpd\t1\n",
            "snmpd\t1\n",
            "sshd(pam_unix)\t677\n",
            "su(pam_unix)\t172\n",
            "sysctl:\t1\n",
            "syslog:\t2\n",
            "syslogd\t7\n",
            "udev\t8\n",
            "xinetd\t2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Pretty-print table of aggregated data"
      ],
      "metadata": {
        "id": "Vj9qz8wSa1w0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "hdfs dfs -get output_aggregate/part-00000 result # download results file\n",
        "column -t result|sort -k2nr # sort by field 2 numerically in descending order"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Y8IYl4hAZhZm",
        "outputId": "d6c5d353-0442-47d5-c834-1e893416a90e"
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ftpd             916\n",
            "sshd(pam_unix)   677\n",
            "su(pam_unix)     172\n",
            "kernel:          76\n",
            "klogind          46\n",
            "logrotate:       43\n",
            "named            16\n",
            "cups:            12\n",
            "udev             8\n",
            "syslogd          7\n",
            "bluetooth:       2\n",
            "gdm(pam_unix)    2\n",
            "gpm              2\n",
            "login(pam_unix)  2\n",
            "network:         2\n",
            "syslog:          2\n",
            "xinetd           2\n",
            "--               1\n",
            "gdm-binary       1\n",
            "hcid             1\n",
            "irqbalance:      1\n",
            "nfslock:         1\n",
            "portmap:         1\n",
            "random:          1\n",
            "rc:              1\n",
            "rpcidmapd:       1\n",
            "rpc.statd        1\n",
            "sdpd             1\n",
            "snmpd            1\n",
            "sysctl:          1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Stop cluster\n",
        "\n",
        "When you're done with your computations, you can shut down the Hadoop cluster and stop the `sshd` service."
      ],
      "metadata": {
        "id": "IF6-Z5RotAcO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!./hadoop-3.4.0/sbin/stop-dfs.sh"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IoIYG5NlsIMv",
        "outputId": "2881e68f-c2ce-49d9-b587-0f6dfb11e329"
      },
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Stopping namenodes on [localhost]\n",
            "Stopping datanodes\n",
            "Stopping secondary namenodes [4d8a89a36a38]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Stop the `sshd` daemon"
      ],
      "metadata": {
        "id": "RGj96_e2ccZw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!/etc/init.d/ssh stop"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FUvKMpy6chQ5",
        "outputId": "977d8038-2951-4c5e-b6e4-ba2e770e28c2"
      },
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " * Stopping OpenBSD Secure Shell server sshd\n",
            "   ...done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Concluding remarks\n",
        "\n",
        "We have started a single-node Hadoop cluster and ran some simple HDFS and MapReduce commands.\n",
        "\n",
        "Even when running on a single machine, one can benefit from the parallelism provided by multiple virtual cores.\n",
        "\n",
        "Hadoop provides also a command-line utility (the CLI MiniCluster) to start and stop a single-node Hadoop cluster \"_without the need to set any environment variables or manage configuration files_\" (https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CLIMiniCluster.html). The [Hadoop MiniCluster](https://github.com/groda/big_data/blob/master/Hadoop_minicluster.ipynb) notebook serves as a guide for launching the Hadoop MiniCluster.\n",
        "\n",
        "While it can be useful to be able to start a Hadoop cluster with a single command, delving into the functionality of each component offers valuable insights into the intricacies of Hadoop architecture, thereby enriching the learning process.\n",
        "\n",
        "If you found this notebook helpful, consider exploring:\n",
        " - [Hadoop single-node cluster setup with Python](https://github.com/groda/big_data/blob/master/Hadoop_single_node_cluster_setup_Python.ipynb) similar to this but using Python in place of bash\n",
        " - [Setting up Spark Standalone on Google Colab](https://github.com/groda/big_data/blob/master/Hadoop_Setting_up_Spark_Standalone_on_Google_Colab.ipynb)\n",
        " - [Getting to know the Spark Standalone Architecture](https://github.com/groda/big_data/blob/master/Spark_Standalone_Architecture_on_Google_Colab.ipynb)\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "w5N7tb0HSbZB"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "qYyg1O7ysUs6"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}