hadoop_mapreduce.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "hadoop_mapreduce.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyNryPaSxVhByIGr0X0SJTxf"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "myPIGP-mwKBD",
        "colab_type": "text"
      },
      "source": [
        "#Hadoop Instalation Part\n",
        "Hadoop is a Java-based programming framework that supports the processing and storage of extremely large datasets on a cluster of inexpensive machines. It was the first major open source project in the big data playing field and is sponsored by the Apache Software Foundation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j9bT9M1yvyXG",
        "colab_type": "text"
      },
      "source": [
        "## Step 1:Installing Hadoop"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bijZAdD_cBMK",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "989d7a35-005b-4014-a0cf-ff458ddd46d7"
      },
      "source": [
        "!wget https://downloads.apache.org/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-09-10 05:21:56--  https://downloads.apache.org/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz\n",
            "Resolving downloads.apache.org (downloads.apache.org)... 88.99.95.219, 2a01:4f8:10a:201a::2\n",
            "Connecting to downloads.apache.org (downloads.apache.org)|88.99.95.219|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 500749234 (478M) [application/x-gzip]\n",
            "Saving to: ‘hadoop-3.3.0.tar.gz.1’\n",
            "\n",
            "hadoop-3.3.0.tar.gz 100%[===================>] 477.55M  10.4MB/s    in 47s     \n",
            "\n",
            "2020-09-10 05:22:44 (10.1 MB/s) - ‘hadoop-3.3.0.tar.gz.1’ saved [500749234/500749234]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Mj40txsTw6DZ"
      },
      "source": [
        "we’ll use the tar command with the -x flag to extract, -z to uncompress, -v for verbose output, and -f to specify that we’re extracting from a file"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nVce513-cBHm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!tar -xzvf hadoop-3.3.0.tar.gz"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JF-ze-YOdync",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#copy  hadoop file to user/local\n",
        "!cp -r hadoop-3.3.0/ /usr/local/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vh6Dqbbrwqpe",
        "colab_type": "text"
      },
      "source": [
        "## Step2:Configuring Hadoop’s Java Home\n",
        "Hadoop requires that you set the path to Java, either as an environment variable or in the Hadoop configuration file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_OUc19ZtcBG5",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "a3821595-1aa0-4f19-a6df-05c7f74c1ae1"
      },
      "source": [
        "#To find the default Java path\n",
        "!readlink -f /usr/bin/java | sed \"s:bin/java::\""
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/lib/jvm/java-11-openjdk-amd64/\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MxaPBWRKxXta",
        "colab_type": "text"
      },
      "source": [
        "To set java path, go to /usr/local/hadoop-3.3.0/etc/hadoop/hadoop-env.sh then\n",
        "\n",
        ". . .\n",
        "export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64/\n",
        " . . . "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Oj00rPPZyEWZ",
        "colab_type": "text"
      },
      "source": [
        "# Step 3:  Running Hadoop"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zhf-zK7NcBDF",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 952
        },
        "outputId": "4b567ac3-57ad-4992-9e1d-e5084ae61b46"
      },
      "source": [
        "#Running Hadoop\n",
        "!/usr/local/hadoop-3.3.0/bin/hadoop"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Usage: hadoop [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]\n",
            " or    hadoop [OPTIONS] CLASSNAME [CLASSNAME OPTIONS]\n",
            "  where CLASSNAME is a user-provided Java class\n",
            "\n",
            "  OPTIONS is none or any of:\n",
            "\n",
            "buildpaths                       attempt to add class files from build tree\n",
            "--config dir                     Hadoop config directory\n",
            "--debug                          turn on shell script debug mode\n",
            "--help                           usage information\n",
            "hostnames list[,of,host,names]   hosts to use in slave mode\n",
            "hosts filename                   list of hosts to use in slave mode\n",
            "loglevel level                   set the log4j level for this command\n",
            "workers                          turn on worker mode\n",
            "\n",
            "  SUBCOMMAND is one of:\n",
            "\n",
            "\n",
            "    Admin Commands:\n",
            "\n",
            "daemonlog     get/set the log level for each daemon\n",
            "\n",
            "    Client Commands:\n",
            "\n",
            "archive       create a Hadoop archive\n",
            "checknative   check native Hadoop and compression libraries availability\n",
            "classpath     prints the class path needed to get the Hadoop jar and the\n",
            "              required libraries\n",
            "conftest      validate configuration XML files\n",
            "credential    interact with credential providers\n",
            "distch        distributed metadata changer\n",
            "distcp        copy file or directories recursively\n",
            "dtutil        operations related to delegation tokens\n",
            "envvars       display computed Hadoop environment variables\n",
            "fs            run a generic filesystem user client\n",
            "gridmix       submit a mix of synthetic job, modeling a profiled from\n",
            "              production load\n",
            "jar <jar>     run a jar file. NOTE: please use \"yarn jar\" to launch YARN\n",
            "              applications, not this command.\n",
            "jnipath       prints the java.library.path\n",
            "kdiag         Diagnose Kerberos Problems\n",
            "kerbname      show auth_to_local principal conversion\n",
            "key           manage keys via the KeyProvider\n",
            "rumenfolder   scale a rumen input trace\n",
            "rumentrace    convert logs into a rumen trace\n",
            "s3guard       manage metadata on S3\n",
            "trace         view and modify Hadoop tracing settings\n",
            "version       print the version\n",
            "\n",
            "    Daemon Commands:\n",
            "\n",
            "kms           run KMS, the Key Management Server\n",
            "registrydns   run the registry DNS server\n",
            "\n",
            "SUBCOMMAND may print help when invoked w/o parameters or with -h.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uI-YBPIzcBCA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!mkdir ~/input\n",
        "!cp /usr/local/hadoop-3.3.0/etc/hadoop/*.xml ~/input"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6DuDJIsPcA98",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "d2e82548-75aa-4e40-a048-48928825d374"
      },
      "source": [
        "!ls ~/input"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "capacity-scheduler.xml\thdfs-rbf-site.xml  kms-acls.xml     yarn-site.xml\n",
            "core-site.xml\t\thdfs-site.xml\t   kms-site.xml\n",
            "hadoop-policy.xml\thttpfs-site.xml    mapred-site.xml\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RZi5zOGKyySH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!/usr/local/hadoop-3.3.0/bin/hadoop jar /usr/local/hadoop-3.3.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0.jar grep ~/input ~/grep_example 'allowed[.]*'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mtr0xWbfcA5J",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "8debaca6-e5f5-423f-e85f-0b3f0753d3e8"
      },
      "source": [
        "!cat ~/grep_example/*"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "22\tallowed.\n",
            "1\tallowed\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AQc3SKkOzNxn",
        "colab_type": "text"
      },
      "source": [
        "**Download 20newsgroups dataset available at** http://qwone.com/~jason/20Newsgroups."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vgEGfCcGOswd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!wget http://qwone.com/~jason/20Newsgroups/20news-18828.tar.gz\n",
        "\n",
        "!tar -xzvf 20news-18828.tar.gz"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oxCFNl3SQHDl",
        "colab_type": "text"
      },
      "source": [
        "#Hadoop Streaming"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bLRsjubgOs2p",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "69620c94-081b-45ac-dab6-ce32db7ef007"
      },
      "source": [
        "!find / -name 'hadoop-streaming*.jar'"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/hadoop-3.3.0/share/hadoop/tools/lib/hadoop-streaming-3.3.0.jar\n",
            "/usr/local/hadoop-3.3.0/share/hadoop/tools/sources/hadoop-streaming-3.3.0-sources.jar\n",
            "/usr/local/hadoop-3.3.0/share/hadoop/tools/sources/hadoop-streaming-3.3.0-test-sources.jar\n",
            "/content/hadoop-3.3.0/share/hadoop/tools/lib/hadoop-streaming-3.3.0.jar\n",
            "/content/hadoop-3.3.0/share/hadoop/tools/sources/hadoop-streaming-3.3.0-sources.jar\n",
            "/content/hadoop-3.3.0/share/hadoop/tools/sources/hadoop-streaming-3.3.0-test-sources.jar\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eu5IAGT2Os6D",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!chmod u+rwx /content/mapper.py\n",
        "!chmod u+rwx /content/reducer.py"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ru1lPx9yOsvO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "023b36ba-21fc-4290-e4db-1487084e5fb6"
      },
      "source": [
        "!/usr/local/hadoop-3.3.0/bin/hadoop jar /usr/local/hadoop-3.3.0/share/hadoop/tools/lib/hadoop-streaming-3.3.0.jar -input /content/20news-18828/alt.atheism/49960 -output /content/output -file /content/mapper.py  -file /content/reducer.py  -mapper 'python mapper.py'  -reducer 'python reducer.py'"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2020-09-05 15:03:48,933 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.\n",
            "packageJobJar: [/content/mapper.py, /content/reducer.py] [] /tmp/streamjob18407869961627485547.jar tmpDir=null\n",
            "2020-09-05 15:03:49,611 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties\n",
            "2020-09-05 15:03:49,758 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).\n",
            "2020-09-05 15:03:49,758 INFO impl.MetricsSystemImpl: JobTracker metrics system started\n",
            "2020-09-05 15:03:49,785 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!\n",
            "2020-09-05 15:03:49,912 INFO mapred.FileInputFormat: Total input files to process : 1\n",
            "2020-09-05 15:03:49,932 INFO mapreduce.JobSubmitter: number of splits:1\n",
            "2020-09-05 15:03:50,194 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local713819920_0001\n",
            "2020-09-05 15:03:50,194 INFO mapreduce.JobSubmitter: Executing with tokens: []\n",
            "2020-09-05 15:03:50,622 INFO mapred.LocalDistributedCacheManager: Localized file:/content/mapper.py as file:/tmp/hadoop-root/mapred/local/job_local713819920_0001_a88c2a90-6a94-4a97-8309-7a43c9913551/mapper.py\n",
            "2020-09-05 15:03:50,657 INFO mapred.LocalDistributedCacheManager: Localized file:/content/reducer.py as file:/tmp/hadoop-root/mapred/local/job_local713819920_0001_9a513f55-7ab1-42d4-b83b-6e6312e3b2de/reducer.py\n",
            "2020-09-05 15:03:50,747 INFO mapreduce.Job: The url to track the job: http://localhost:8080/\n",
            "2020-09-05 15:03:50,748 INFO mapreduce.Job: Running job: job_local713819920_0001\n",
            "2020-09-05 15:03:50,763 INFO mapred.LocalJobRunner: OutputCommitter set in config null\n",
            "2020-09-05 15:03:50,766 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter\n",
            "2020-09-05 15:03:50,777 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2\n",
            "2020-09-05 15:03:50,777 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false\n",
            "2020-09-05 15:03:50,843 INFO mapred.LocalJobRunner: Waiting for map tasks\n",
            "2020-09-05 15:03:50,847 INFO mapred.LocalJobRunner: Starting task: attempt_local713819920_0001_m_000000_0\n",
            "2020-09-05 15:03:50,887 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2\n",
            "2020-09-05 15:03:50,890 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false\n",
            "2020-09-05 15:03:50,919 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]\n",
            "2020-09-05 15:03:50,932 INFO mapred.MapTask: Processing split: file:/content/20news-18828/alt.atheism/49960:0+11599\n",
            "2020-09-05 15:03:50,949 INFO mapred.MapTask: numReduceTasks: 1\n",
            "2020-09-05 15:03:51,024 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)\n",
            "2020-09-05 15:03:51,024 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100\n",
            "2020-09-05 15:03:51,024 INFO mapred.MapTask: soft limit at 83886080\n",
            "2020-09-05 15:03:51,024 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600\n",
            "2020-09-05 15:03:51,024 INFO mapred.MapTask: kvstart = 26214396; length = 6553600\n",
            "2020-09-05 15:03:51,027 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer\n",
            "2020-09-05 15:03:51,037 INFO streaming.PipeMapRed: PipeMapRed exec [/usr/local/bin/python, mapper.py]\n",
            "2020-09-05 15:03:51,045 INFO Configuration.deprecation: mapred.work.output.dir is deprecated. Instead, use mapreduce.task.output.dir\n",
            "2020-09-05 15:03:51,045 INFO Configuration.deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start\n",
            "2020-09-05 15:03:51,046 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap\n",
            "2020-09-05 15:03:51,046 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id\n",
            "2020-09-05 15:03:51,047 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id\n",
            "2020-09-05 15:03:51,047 INFO Configuration.deprecation: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir\n",
            "2020-09-05 15:03:51,048 INFO Configuration.deprecation: map.input.file is deprecated. Instead, use mapreduce.map.input.file\n",
            "2020-09-05 15:03:51,049 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords\n",
            "2020-09-05 15:03:51,049 INFO Configuration.deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length\n",
            "2020-09-05 15:03:51,050 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id\n",
            "2020-09-05 15:03:51,050 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name\n",
            "2020-09-05 15:03:51,051 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition\n",
            "2020-09-05 15:03:51,082 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2020-09-05 15:03:51,083 INFO streaming.PipeMapRed: R/W/S=10/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2020-09-05 15:03:51,084 INFO streaming.PipeMapRed: R/W/S=100/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2020-09-05 15:03:51,757 INFO mapreduce.Job: Job job_local713819920_0001 running in uber mode : false\n",
            "2020-09-05 15:03:51,758 INFO mapreduce.Job:  map 0% reduce 0%\n",
            "2020-09-05 15:03:51,962 INFO streaming.PipeMapRed: Records R/W=293/1\n",
            "2020-09-05 15:03:52,097 INFO streaming.PipeMapRed: MRErrorThread done\n",
            "2020-09-05 15:03:52,097 INFO streaming.PipeMapRed: mapRedFinished\n",
            "2020-09-05 15:03:52,100 INFO mapred.LocalJobRunner: \n",
            "2020-09-05 15:03:52,100 INFO mapred.MapTask: Starting flush of map output\n",
            "2020-09-05 15:03:52,100 INFO mapred.MapTask: Spilling map output\n",
            "2020-09-05 15:03:52,100 INFO mapred.MapTask: bufstart = 0; bufend = 10685; bufvoid = 104857600\n",
            "2020-09-05 15:03:52,101 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26209920(104839680); length = 4477/6553600\n",
            "2020-09-05 15:03:52,130 INFO mapred.MapTask: Finished spill 0\n",
            "2020-09-05 15:03:52,142 INFO mapred.Task: Task:attempt_local713819920_0001_m_000000_0 is done. And is in the process of committing\n",
            "2020-09-05 15:03:52,145 INFO mapred.LocalJobRunner: Records R/W=293/1\n",
            "2020-09-05 15:03:52,145 INFO mapred.Task: Task 'attempt_local713819920_0001_m_000000_0' done.\n",
            "2020-09-05 15:03:52,152 INFO mapred.Task: Final Counters for attempt_local713819920_0001_m_000000_0: Counters: 17\n",
            "\tFile System Counters\n",
            "\t\tFILE: Number of bytes read=15021\n",
            "\t\tFILE: Number of bytes written=628522\n",
            "\t\tFILE: Number of read operations=0\n",
            "\t\tFILE: Number of large read operations=0\n",
            "\t\tFILE: Number of write operations=0\n",
            "\tMap-Reduce Framework\n",
            "\t\tMap input records=293\n",
            "\t\tMap output records=1120\n",
            "\t\tMap output bytes=10685\n",
            "\t\tMap output materialized bytes=12931\n",
            "\t\tInput split bytes=96\n",
            "\t\tCombine input records=0\n",
            "\t\tSpilled Records=1120\n",
            "\t\tFailed Shuffles=0\n",
            "\t\tMerged Map outputs=0\n",
            "\t\tGC time elapsed (ms)=28\n",
            "\t\tTotal committed heap usage (bytes)=353370112\n",
            "\tFile Input Format Counters \n",
            "\t\tBytes Read=11599\n",
            "2020-09-05 15:03:52,152 INFO mapred.LocalJobRunner: Finishing task: attempt_local713819920_0001_m_000000_0\n",
            "2020-09-05 15:03:52,152 INFO mapred.LocalJobRunner: map task executor complete.\n",
            "2020-09-05 15:03:52,157 INFO mapred.LocalJobRunner: Waiting for reduce tasks\n",
            "2020-09-05 15:03:52,159 INFO mapred.LocalJobRunner: Starting task: attempt_local713819920_0001_r_000000_0\n",
            "2020-09-05 15:03:52,171 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2\n",
            "2020-09-05 15:03:52,171 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false\n",
            "2020-09-05 15:03:52,176 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]\n",
            "2020-09-05 15:03:52,179 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@7c884daf\n",
            "2020-09-05 15:03:52,181 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!\n",
            "2020-09-05 15:03:52,199 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=2389914368, maxSingleShuffleLimit=597478592, mergeThreshold=1577343488, ioSortFactor=10, memToMemMergeOutputsThreshold=10\n",
            "2020-09-05 15:03:52,202 INFO reduce.EventFetcher: attempt_local713819920_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events\n",
            "2020-09-05 15:03:52,243 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local713819920_0001_m_000000_0 decomp: 12927 len: 12931 to MEMORY\n",
            "2020-09-05 15:03:52,247 INFO reduce.InMemoryMapOutput: Read 12927 bytes from map-output for attempt_local713819920_0001_m_000000_0\n",
            "2020-09-05 15:03:52,249 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 12927, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->12927\n",
            "2020-09-05 15:03:52,251 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning\n",
            "2020-09-05 15:03:52,253 INFO mapred.LocalJobRunner: 1 / 1 copied.\n",
            "2020-09-05 15:03:52,253 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs\n",
            "2020-09-05 15:03:52,260 INFO mapred.Merger: Merging 1 sorted segments\n",
            "2020-09-05 15:03:52,260 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 12914 bytes\n",
            "2020-09-05 15:03:52,269 INFO reduce.MergeManagerImpl: Merged 1 segments, 12927 bytes to disk to satisfy reduce memory limit\n",
            "2020-09-05 15:03:52,270 INFO reduce.MergeManagerImpl: Merging 1 files, 12931 bytes from disk\n",
            "2020-09-05 15:03:52,270 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce\n",
            "2020-09-05 15:03:52,270 INFO mapred.Merger: Merging 1 sorted segments\n",
            "2020-09-05 15:03:52,271 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 12914 bytes\n",
            "2020-09-05 15:03:52,272 INFO mapred.LocalJobRunner: 1 / 1 copied.\n",
            "2020-09-05 15:03:52,289 INFO streaming.PipeMapRed: PipeMapRed exec [/usr/local/bin/python, reducer.py]\n",
            "2020-09-05 15:03:52,294 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address\n",
            "2020-09-05 15:03:52,298 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps\n",
            "2020-09-05 15:03:52,325 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2020-09-05 15:03:52,325 INFO streaming.PipeMapRed: R/W/S=10/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2020-09-05 15:03:52,326 INFO streaming.PipeMapRed: R/W/S=100/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2020-09-05 15:03:52,346 INFO streaming.PipeMapRed: R/W/S=1000/0/0 in:NA [rec/s] out:NA [rec/s]\n",
            "2020-09-05 15:03:52,463 INFO streaming.PipeMapRed: Records R/W=1120/1\n",
            "2020-09-05 15:03:52,468 INFO streaming.PipeMapRed: MRErrorThread done\n",
            "2020-09-05 15:03:52,477 INFO streaming.PipeMapRed: mapRedFinished\n",
            "2020-09-05 15:03:52,478 INFO mapred.Task: Task:attempt_local713819920_0001_r_000000_0 is done. And is in the process of committing\n",
            "2020-09-05 15:03:52,480 INFO mapred.LocalJobRunner: 1 / 1 copied.\n",
            "2020-09-05 15:03:52,480 INFO mapred.Task: Task attempt_local713819920_0001_r_000000_0 is allowed to commit now\n",
            "2020-09-05 15:03:52,484 INFO output.FileOutputCommitter: Saved output of task 'attempt_local713819920_0001_r_000000_0' to file:/content/output\n",
            "2020-09-05 15:03:52,496 INFO mapred.LocalJobRunner: Records R/W=1120/1 > reduce\n",
            "2020-09-05 15:03:52,496 INFO mapred.Task: Task 'attempt_local713819920_0001_r_000000_0' done.\n",
            "2020-09-05 15:03:52,499 INFO mapred.Task: Final Counters for attempt_local713819920_0001_r_000000_0: Counters: 24\n",
            "\tFile System Counters\n",
            "\t\tFILE: Number of bytes read=40915\n",
            "\t\tFILE: Number of bytes written=649151\n",
            "\t\tFILE: Number of read operations=0\n",
            "\t\tFILE: Number of large read operations=0\n",
            "\t\tFILE: Number of write operations=0\n",
            "\tMap-Reduce Framework\n",
            "\t\tCombine input records=0\n",
            "\t\tCombine output records=0\n",
            "\t\tReduce input groups=787\n",
            "\t\tReduce shuffle bytes=12931\n",
            "\t\tReduce input records=1120\n",
            "\t\tReduce output records=787\n",
            "\t\tSpilled Records=1120\n",
            "\t\tShuffled Maps =1\n",
            "\t\tFailed Shuffles=0\n",
            "\t\tMerged Map outputs=1\n",
            "\t\tGC time elapsed (ms)=0\n",
            "\t\tTotal committed heap usage (bytes)=353370112\n",
            "\tShuffle Errors\n",
            "\t\tBAD_ID=0\n",
            "\t\tCONNECTION=0\n",
            "\t\tIO_ERROR=0\n",
            "\t\tWRONG_LENGTH=0\n",
            "\t\tWRONG_MAP=0\n",
            "\t\tWRONG_REDUCE=0\n",
            "\tFile Output Format Counters \n",
            "\t\tBytes Written=7698\n",
            "2020-09-05 15:03:52,499 INFO mapred.LocalJobRunner: Finishing task: attempt_local713819920_0001_r_000000_0\n",
            "2020-09-05 15:03:52,500 INFO mapred.LocalJobRunner: reduce task executor complete.\n",
            "2020-09-05 15:03:52,761 INFO mapreduce.Job:  map 100% reduce 100%\n",
            "2020-09-05 15:03:52,761 INFO mapreduce.Job: Job job_local713819920_0001 completed successfully\n",
            "2020-09-05 15:03:52,775 INFO mapreduce.Job: Counters: 30\n",
            "\tFile System Counters\n",
            "\t\tFILE: Number of bytes read=55936\n",
            "\t\tFILE: Number of bytes written=1277673\n",
            "\t\tFILE: Number of read operations=0\n",
            "\t\tFILE: Number of large read operations=0\n",
            "\t\tFILE: Number of write operations=0\n",
            "\tMap-Reduce Framework\n",
            "\t\tMap input records=293\n",
            "\t\tMap output records=1120\n",
            "\t\tMap output bytes=10685\n",
            "\t\tMap output materialized bytes=12931\n",
            "\t\tInput split bytes=96\n",
            "\t\tCombine input records=0\n",
            "\t\tCombine output records=0\n",
            "\t\tReduce input groups=787\n",
            "\t\tReduce shuffle bytes=12931\n",
            "\t\tReduce input records=1120\n",
            "\t\tReduce output records=787\n",
            "\t\tSpilled Records=2240\n",
            "\t\tShuffled Maps =1\n",
            "\t\tFailed Shuffles=0\n",
            "\t\tMerged Map outputs=1\n",
            "\t\tGC time elapsed (ms)=28\n",
            "\t\tTotal committed heap usage (bytes)=706740224\n",
            "\tShuffle Errors\n",
            "\t\tBAD_ID=0\n",
            "\t\tCONNECTION=0\n",
            "\t\tIO_ERROR=0\n",
            "\t\tWRONG_LENGTH=0\n",
            "\t\tWRONG_MAP=0\n",
            "\t\tWRONG_REDUCE=0\n",
            "\tFile Input Format Counters \n",
            "\t\tBytes Read=11599\n",
            "\tFile Output Format Counters \n",
            "\t\tBytes Written=7698\n",
            "2020-09-05 15:03:52,775 INFO streaming.StreamJob: Output directory: /content/output\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jiW5mPIcy_Jp",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "4e5d8d06-0965-4b2c-8204-503305e6fa49"
      },
      "source": [
        "!ls /content/output"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "part-00000  _SUCCESS\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CTFjOm59kc04",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "81c7ce84-4a64-47c2-d525-ab83c560e12a"
      },
      "source": [
        "!cat /content/output/part-00000"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "034529887x\t1\n",
            "0511211216\t1\n",
            "071\t5\n",
            "080182494x\t1\n",
            "0801834074\t1\n",
            "0877226423\t1\n",
            "0877227675\t1\n",
            "0908\t1\n",
            "0910309264\t1\n",
            "1\t1\n",
            "10\t1\n",
            "11\t1\n",
            "1266\t1\n",
            "1271\t1\n",
            "14\t1\n",
            "140195\t1\n",
            "14215\t1\n",
            "14226\t1\n",
            "142282197\t1\n",
            "17701900\t1\n",
            "1881\t1\n",
            "1977\t1\n",
            "1981\t1\n",
            "1986\t1\n",
            "1988\t1\n",
            "1989\t1\n",
            "1990\t1\n",
            "1992\t1\n",
            "20th\t1\n",
            "226\t1\n",
            "24hour\t1\n",
            "2568900\t1\n",
            "272\t1\n",
            "273\t1\n",
            "2nd\t1\n",
            "3005\t1\n",
            "316\t1\n",
            "372\t1\n",
            "3d\t1\n",
            "3nl\t1\n",
            "4\t1\n",
            "41\t2\n",
            "430\t2\n",
            "4581244\t1\n",
            "4679525\t1\n",
            "490\t1\n",
            "495\t2\n",
            "4rh\t1\n",
            "4rl\t1\n",
            "512\t2\n",
            "53701\t1\n",
            "541\t1\n",
            "59\t1\n",
            "608\t1\n",
            "664\t1\n",
            "700\t1\n",
            "702\t1\n",
            "7119\t1\n",
            "716\t1\n",
            "7215\t1\n",
            "7251\t1\n",
            "750\t1\n",
            "7723\t1\n",
            "787140195\t1\n",
            "787522973\t1\n",
            "831\t1\n",
            "8372475\t1\n",
            "88\t1\n",
            "880\t2\n",
            "8964079\t1\n",
            "8ew\t1\n",
            "91605\t1\n",
            "aah\t1\n",
            "aap\t2\n",
            "abortions\t1\n",
            "absurdities\t1\n",
            "accompanied\t1\n",
            "accounts\t1\n",
            "address\t1\n",
            "addresses\t2\n",
            "adulteries\t1\n",
            "aesthetics\t1\n",
            "african\t3\n",
            "africanamericans\t1\n",
            "agnostic\t1\n",
            "al\t1\n",
            "alien\t1\n",
            "allen\t2\n",
            "also\t3\n",
            "altatheism\t1\n",
            "altatheismarchivename\t1\n",
            "altatheismmoderated\t1\n",
            "alternate\t1\n",
            "alternative\t1\n",
            "although\t2\n",
            "america\t1\n",
            "american\t5\n",
            "americans\t2\n",
            "amherst\t1\n",
            "amongst\t1\n",
            "amusing\t1\n",
            "ancient\t1\n",
            "andor\t1\n",
            "another\t1\n",
            "anselm\t1\n",
            "anthology\t3\n",
            "anyone\t1\n",
            "appendix\t2\n",
            "approachable\t1\n",
            "archive\t1\n",
            "archivename\t1\n",
            "archives\t1\n",
            "archiveservermantiscouk\t1\n",
            "area\t2\n",
            "argues\t1\n",
            "arguments\t5\n",
            "articles\t1\n",
            "assassinated\t1\n",
            "assisted\t1\n",
            "association\t2\n",
            "assorted\t3\n",
            "atheism\t6\n",
            "atheismindex\t1\n",
            "atheismresources\t1\n",
            "atheist\t10\n",
            "atheisten\t2\n",
            "atheistic\t1\n",
            "atomic\t2\n",
            "atoms\t1\n",
            "atrocities\t1\n",
            "attempt\t1\n",
            "attempts\t1\n",
            "attention\t1\n",
            "atwood\t1\n",
            "atwoods\t1\n",
            "austin\t2\n",
            "authors\t1\n",
            "available\t2\n",
            "axiarchism\t1\n",
            "back\t1\n",
            "ball\t2\n",
            "ballantine\t1\n",
            "baltimore\t1\n",
            "bank\t1\n",
            "bantam\t1\n",
            "based\t2\n",
            "bay\t1\n",
            "beam\t1\n",
            "became\t1\n",
            "become\t1\n",
            "began\t1\n",
            "begins\t1\n",
            "belief\t2\n",
            "believed\t1\n",
            "beneath\t1\n",
            "berkeley\t1\n",
            "berlin\t2\n",
            "best\t1\n",
            "better\t1\n",
            "beyond\t1\n",
            "bible\t7\n",
            "biblebeliever\t1\n",
            "biblical\t1\n",
            "bibliography\t1\n",
            "bizarre\t1\n",
            "black\t2\n",
            "blueprints\t1\n",
            "book\t4\n",
            "books\t11\n",
            "box\t3\n",
            "brain\t2\n",
            "britain\t1\n",
            "british\t1\n",
            "bucherdienst\t1\n",
            "buffalo\t3\n",
            "bumper\t1\n",
            "bund\t1\n",
            "ca\t1\n",
            "cameron\t1\n",
            "canticle\t1\n",
            "canyon\t1\n",
            "card\t1\n",
            "cardiffs\t1\n",
            "carries\t1\n",
            "cars\t1\n",
            "case\t1\n",
            "catalog\t1\n",
            "cathedral\t1\n",
            "catholic\t1\n",
            "centuries\t1\n",
            "century\t1\n",
            "challenging\t1\n",
            "characters\t2\n",
            "charge\t1\n",
            "chilling\t1\n",
            "christ\t1\n",
            "christian\t4\n",
            "christianity\t4\n",
            "christians\t2\n",
            "church\t1\n",
            "clarendon\t1\n",
            "classical\t2\n",
            "claus\t1\n",
            "clerical\t1\n",
            "closed\t1\n",
            "cohen\t1\n",
            "coherence\t1\n",
            "compared\t1\n",
            "comply\t1\n",
            "comprehensive\t3\n",
            "compromise\t1\n",
            "conceived\t1\n",
            "concentrating\t1\n",
            "concept\t1\n",
            "concluded\t1\n",
            "conduit\t1\n",
            "congress\t2\n",
            "considering\t1\n",
            "considers\t1\n",
            "construct\t1\n",
            "containing\t1\n",
            "contains\t3\n",
            "contemporary\t1\n",
            "contempory\t1\n",
            "contradictions\t2\n",
            "contradicts\t1\n",
            "conway\t1\n",
            "copying\t1\n",
            "covering\t1\n",
            "craftsmen\t1\n",
            "creed\t2\n",
            "crimes\t1\n",
            "criticized\t1\n",
            "critique\t1\n",
            "critiques\t1\n",
            "d1000\t2\n",
            "d3000\t1\n",
            "darwin\t4\n",
            "davy\t1\n",
            "day\t1\n",
            "de\t2\n",
            "dead\t2\n",
            "death\t1\n",
            "december\t1\n",
            "decisively\t1\n",
            "defences\t1\n",
            "defining\t1\n",
            "deity\t2\n",
            "delight\t1\n",
            "deluxe\t1\n",
            "demand\t1\n",
            "demonstrates\t1\n",
            "der\t3\n",
            "derived\t1\n",
            "des\t1\n",
            "descartes\t1\n",
            "describe\t1\n",
            "description\t1\n",
            "designs\t3\n",
            "detailed\t1\n",
            "developments\t1\n",
            "devil\t1\n",
            "diary\t1\n",
            "dick\t3\n",
            "dictionary\t1\n",
            "die\t1\n",
            "diener\t1\n",
            "different\t2\n",
            "difficult\t1\n",
            "direct\t1\n",
            "directly\t1\n",
            "disch\t1\n",
            "dismissively\t1\n",
            "divine\t2\n",
            "doctors\t1\n",
            "dogmatic\t1\n",
            "doomsday\t2\n",
            "drive\t1\n",
            "droemerknaur\t1\n",
            "dull\t1\n",
            "dunkle\t1\n",
            "earth\t2\n",
            "earthers\t1\n",
            "east\t1\n",
            "easy\t1\n",
            "edgar\t1\n",
            "edition\t3\n",
            "editor\t1\n",
            "edmund\t1\n",
            "effect\t1\n",
            "emphasis\t1\n",
            "england\t1\n",
            "enlighting\t1\n",
            "erste\t1\n",
            "et\t1\n",
            "etc\t1\n",
            "ethical\t1\n",
            "ev\t2\n",
            "even\t1\n",
            "events\t1\n",
            "evil\t1\n",
            "evolution\t3\n",
            "examiner\t1\n",
            "examines\t1\n",
            "example\t1\n",
            "existence\t5\n",
            "exists\t3\n",
            "explicitly\t1\n",
            "expressed\t1\n",
            "faith\t2\n",
            "fallacies\t1\n",
            "fallible\t1\n",
            "faq\t1\n",
            "fate\t1\n",
            "fax\t2\n",
            "feet\t1\n",
            "fernwright\t1\n",
            "ffrf\t1\n",
            "fiction\t1\n",
            "fictitious\t1\n",
            "figmonetcomcom\t1\n",
            "files\t1\n",
            "filling\t1\n",
            "fired\t1\n",
            "first\t1\n",
            "fish\t6\n",
            "focusses\t1\n",
            "following\t1\n",
            "foote\t2\n",
            "forbids\t1\n",
            "formalistic\t1\n",
            "foundation\t2\n",
            "founded\t1\n",
            "france\t1\n",
            "francisco\t1\n",
            "freedom\t2\n",
            "freethinker\t1\n",
            "freethought\t2\n",
            "friend\t1\n",
            "fundamentalists\t2\n",
            "fuss\t1\n",
            "galactic\t1\n",
            "gem\t1\n",
            "george\t1\n",
            "german\t1\n",
            "germany\t4\n",
            "get\t3\n",
            "giant\t1\n",
            "glenn\t1\n",
            "gnostic\t1\n",
            "go\t1\n",
            "god\t13\n",
            "gods\t3\n",
            "goes\t1\n",
            "gold\t1\n",
            "gordon\t1\n",
            "gottes\t1\n",
            "great\t3\n",
            "group\t1\n",
            "grows\t1\n",
            "gw\t1\n",
            "hall\t1\n",
            "handbook\t1\n",
            "handmaids\t1\n",
            "handwaving\t1\n",
            "hanged\t1\n",
            "hannover\t1\n",
            "hardcover\t2\n",
            "haught\t1\n",
            "haughts\t1\n",
            "help\t1\n",
            "hero\t1\n",
            "hidden\t1\n",
            "high\t1\n",
            "history\t6\n",
            "holloway\t1\n",
            "hollywood\t1\n",
            "holy\t2\n",
            "hopkins\t1\n",
            "horrors\t2\n",
            "however\t1\n",
            "hrsg\t1\n",
            "humanism\t5\n",
            "humanist\t1\n",
            "hume\t1\n",
            "hunted\t1\n",
            "ibdk\t1\n",
            "ibka\t3\n",
            "idea\t2\n",
            "ie\t1\n",
            "ill\t1\n",
            "illustrated\t1\n",
            "immoralities\t2\n",
            "implicitly\t1\n",
            "imputation\t1\n",
            "includes\t3\n",
            "including\t2\n",
            "incoherent\t2\n",
            "inductive\t1\n",
            "information\t1\n",
            "informationen\t1\n",
            "ink\t1\n",
            "inside\t1\n",
            "intellectual\t1\n",
            "internationaler\t2\n",
            "invades\t1\n",
            "invasion\t1\n",
            "ironic\t1\n",
            "isbn\t5\n",
            "islington\t1\n",
            "j\t1\n",
            "james\t3\n",
            "joe\t1\n",
            "johns\t1\n",
            "journal\t2\n",
            "jr\t3\n",
            "justification\t2\n",
            "k\t2\n",
            "kant\t1\n",
            "kierkegaard\t1\n",
            "kind\t1\n",
            "king\t1\n",
            "kingdom\t1\n",
            "know\t1\n",
            "konfessionslosen\t2\n",
            "konfessionslosesn\t1\n",
            "kung\t1\n",
            "l\t1\n",
            "lambs\t1\n",
            "laser\t1\n",
            "lastmodified\t1\n",
            "late\t1\n",
            "laurel\t1\n",
            "leaving\t1\n",
            "legal\t1\n",
            "leibowitz\t2\n",
            "lelies\t1\n",
            "less\t1\n",
            "letters\t1\n",
            "library\t1\n",
            "life\t1\n",
            "like\t1\n",
            "lines\t1\n",
            "lion\t1\n",
            "listening\t1\n",
            "listing\t1\n",
            "lists\t1\n",
            "live\t1\n",
            "lives\t1\n",
            "living\t1\n",
            "london\t4\n",
            "looks\t1\n",
            "luxuries\t1\n",
            "lynn\t2\n",
            "mackie\t2\n",
            "mackies\t1\n",
            "madison\t1\n",
            "madness\t1\n",
            "magazine\t1\n",
            "mail\t2\n",
            "mailbased\t1\n",
            "mailing\t1\n",
            "mainly\t1\n",
            "mainstream\t1\n",
            "make\t1\n",
            "makes\t1\n",
            "making\t1\n",
            "man\t1\n",
            "mantiscouk\t1\n",
            "many\t3\n",
            "margaret\t1\n",
            "martin\t1\n",
            "martins\t1\n",
            "materialien\t1\n",
            "mathew\t2\n",
            "mathewmantiscouk\t1\n",
            "may\t1\n",
            "maze\t1\n",
            "md\t1\n",
            "men\t1\n",
            "met\t1\n",
            "michael\t1\n",
            "miller\t1\n",
            "mind\t1\n",
            "miracle\t2\n",
            "miz\t1\n",
            "mizvertrieb\t1\n",
            "monks\t1\n",
            "monthly\t1\n",
            "moral\t1\n",
            "morality\t1\n",
            "moulded\t1\n",
            "murder\t1\n",
            "music\t1\n",
            "must\t1\n",
            "mysteries\t1\n",
            "mysteriously\t1\n",
            "n1\t1\n",
            "n19\t1\n",
            "nation\t1\n",
            "national\t2\n",
            "necessarily\t1\n",
            "negative\t1\n",
            "neither\t1\n",
            "net\t2\n",
            "new\t4\n",
            "newer\t1\n",
            "newman\t1\n",
            "newsletter\t1\n",
            "nonbelief\t1\n",
            "nonexistence\t1\n",
            "nonfiction\t1\n",
            "norm\t2\n",
            "north\t1\n",
            "noteworthy\t1\n",
            "novel\t3\n",
            "novels\t2\n",
            "noyes\t1\n",
            "number\t2\n",
            "ny\t2\n",
            "obscure\t1\n",
            "observations\t1\n",
            "oceans\t1\n",
            "odd\t1\n",
            "often\t3\n",
            "old\t2\n",
            "older\t1\n",
            "one\t3\n",
            "ones\t1\n",
            "opinions\t1\n",
            "organization\t1\n",
            "organizations\t1\n",
            "origin\t1\n",
            "origins\t1\n",
            "outlawed\t1\n",
            "outstanding\t1\n",
            "oxford\t2\n",
            "pages\t4\n",
            "paid\t1\n",
            "pangborn\t1\n",
            "papal\t1\n",
            "paper\t3\n",
            "paperback\t1\n",
            "paperbacks\t1\n",
            "papsttums\t1\n",
            "paraphernalia\t1\n",
            "particular\t1\n",
            "particularly\t1\n",
            "passage\t1\n",
            "people\t6\n",
            "per\t1\n",
            "performed\t1\n",
            "period\t1\n",
            "persecution\t1\n",
            "persons\t1\n",
            "peter\t1\n",
            "philadelphia\t1\n",
            "philip\t2\n",
            "philips\t1\n",
            "philosophical\t3\n",
            "philosophy\t1\n",
            "pink\t1\n",
            "place\t1\n",
            "planet\t1\n",
            "plantinga\t1\n",
            "plastic\t1\n",
            "platinga\t1\n",
            "po\t3\n",
            "polished\t1\n",
            "politisches\t1\n",
            "popular\t1\n",
            "positions\t2\n",
            "positive\t1\n",
            "possibly\t1\n",
            "post\t2\n",
            "postfach\t3\n",
            "posthumous\t1\n",
            "postpaid\t1\n",
            "pothealer\t2\n",
            "pp\t1\n",
            "pregnant\t1\n",
            "premise\t1\n",
            "present\t2\n",
            "press\t8\n",
            "price\t1\n",
            "principal\t1\n",
            "probably\t1\n",
            "produce\t1\n",
            "prometheus\t5\n",
            "promoting\t1\n",
            "proof\t1\n",
            "property\t1\n",
            "publish\t4\n",
            "punished\t1\n",
            "push\t1\n",
            "quarterly\t1\n",
            "quickly\t1\n",
            "quite\t1\n",
            "quotations\t2\n",
            "r\t2\n",
            "radio\t1\n",
            "raise\t1\n",
            "rambling\t1\n",
            "range\t1\n",
            "ranges\t1\n",
            "rather\t2\n",
            "rational\t1\n",
            "rationalism\t1\n",
            "rationalist\t1\n",
            "read\t1\n",
            "reading\t1\n",
            "readings\t1\n",
            "reality\t1\n",
            "realm\t1\n",
            "reason\t1\n",
            "rebut\t1\n",
            "recent\t1\n",
            "red\t1\n",
            "refreshingly\t1\n",
            "refutations\t1\n",
            "refuting\t1\n",
            "rejected\t1\n",
            "relevance\t1\n",
            "religion\t6\n",
            "religious\t3\n",
            "rely\t1\n",
            "remained\t1\n",
            "remote\t1\n",
            "replacements\t1\n",
            "reply\t1\n",
            "resources\t4\n",
            "restatements\t1\n",
            "retroactively\t1\n",
            "returns\t1\n",
            "review\t1\n",
            "revised\t2\n",
            "revoked\t1\n",
            "richard\t1\n",
            "right\t2\n",
            "road\t2\n",
            "rosa\t2\n",
            "saint\t1\n",
            "san\t1\n",
            "santa\t2\n",
            "saying\t1\n",
            "sceptical\t1\n",
            "schizophrenic\t1\n",
            "scholarly\t1\n",
            "searches\t1\n",
            "second\t1\n",
            "secular\t3\n",
            "secularization\t1\n",
            "see\t2\n",
            "seems\t1\n",
            "seite\t1\n",
            "seldes\t1\n",
            "sell\t2\n",
            "send\t2\n",
            "series\t1\n",
            "server\t1\n",
            "set\t2\n",
            "sf\t1\n",
            "sheets\t1\n",
            "short\t2\n",
            "sidgwick\t1\n",
            "similarity\t1\n",
            "simple\t1\n",
            "sinful\t1\n",
            "single\t1\n",
            "small\t1\n",
            "society\t3\n",
            "somewhat\t3\n",
            "sort\t1\n",
            "south\t1\n",
            "spent\t1\n",
            "square\t1\n",
            "star\t1\n",
            "statements\t1\n",
            "states\t1\n",
            "stein\t1\n",
            "stick\t1\n",
            "stickers\t1\n",
            "stories\t2\n",
            "story\t2\n",
            "street\t2\n",
            "study\t1\n",
            "style\t1\n",
            "subject\t1\n",
            "subjects\t1\n",
            "substance\t1\n",
            "subtitled\t1\n",
            "summons\t1\n",
            "supposedly\t1\n",
            "suppressed\t1\n",
            "sure\t1\n",
            "swinburne\t6\n",
            "symbol\t1\n",
            "system\t1\n",
            "take\t1\n",
            "tale\t2\n",
            "technology\t1\n",
            "technologybased\t1\n",
            "telephone\t4\n",
            "temple\t2\n",
            "tendentious\t2\n",
            "terminally\t1\n",
            "terminology\t1\n",
            "theism\t3\n",
            "theists\t1\n",
            "theocracy\t1\n",
            "theres\t1\n",
            "theses\t1\n",
            "think\t1\n",
            "thomas\t1\n",
            "thoughtprovoking\t1\n",
            "thoughts\t1\n",
            "times\t2\n",
            "traces\t1\n",
            "translation\t1\n",
            "tries\t1\n",
            "trilogy\t1\n",
            "true\t1\n",
            "truth\t1\n",
            "try\t1\n",
            "turner\t1\n",
            "twisted\t1\n",
            "tx\t2\n",
            "uh\t1\n",
            "ultimate\t1\n",
            "ultimately\t1\n",
            "unable\t1\n",
            "unbelief\t2\n",
            "uncovering\t1\n",
            "und\t3\n",
            "unfortunately\t1\n",
            "united\t1\n",
            "university\t3\n",
            "unknown\t1\n",
            "unsupportable\t1\n",
            "upon\t1\n",
            "us\t3\n",
            "usa\t4\n",
            "usage\t1\n",
            "use\t1\n",
            "used\t2\n",
            "valis\t1\n",
            "values\t1\n",
            "various\t3\n",
            "version\t3\n",
            "versions\t1\n",
            "vicars\t1\n",
            "views\t1\n",
            "volume\t2\n",
            "walter\t1\n",
            "way\t2\n",
            "wc1r\t2\n",
            "well\t2\n",
            "western\t1\n",
            "whether\t1\n",
            "white\t1\n",
            "whose\t1\n",
            "wi\t1\n",
            "wide\t1\n",
            "wired\t1\n",
            "without\t4\n",
            "woman\t1\n",
            "womans\t1\n",
            "women\t1\n",
            "womens\t1\n",
            "word\t1\n",
            "work\t2\n",
            "works\t1\n",
            "world\t1\n",
            "worldview\t2\n",
            "worth\t1\n",
            "wp\t1\n",
            "write\t6\n",
            "writing\t1\n",
            "writings\t1\n",
            "written\t2\n",
            "wrote\t3\n",
            "york\t2\n",
            "youll\t1\n",
            "young\t1\n",
            "zeit\t1\n",
            "zur\t1\n",
            "ÿ\t1\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}