Feature Engineering--Adv. Data Science Capstone.ipynb

{
    "cells": [
        {
            "metadata": {},
            "cell_type": "markdown",
            "source": "### Feature Engineering--True and False News\nThe final step of feature engineering is to tokenize the text of the stories.  The raw data sequence of characters cannot be fed directly to the algorithms themselves as most of them expect numerical feature vectors with a fixed size rather than the raw text documents with variable length.\n\nI considered doing this using discrete Scikit-Learn modules, but the recently Tensorflow 2.1 adds support for a TextVectorization layer, and 2.3 adds experiment support for the new Keras Preprocessing Layers API. These layers allow you to package preprocessing logic inside the model for easier deployment \u2014 allowing the model to take raw strings, images, or rows from a table as input.  This module also includes a \n\nThe processing of each sample contains the following steps:\n\n1. Standardize each sample.  Lowercase all words and strip punctuation. \n\n2. Split each sample into substrings (usually words).\n\n3. Recombine substrings into tokens (usually ngrams). Options here include determining how many words to include in each token.  Text classification tasks typically  consider tokens of 1 or 2 works, but we may experiment with more than that.\n\n4. Index tokens (associate a unique int value with each token).\n\n5. Transform each sample using this index, either into a vector of ints or a dense float vector.  This layer includes the ability to set the length of the resulting vector, either truncating or padding the vector with zeroes so it will fit the size of our input layer.  It also has several output modes, including tf-idf which is weighting algorithm based on the frequency of words found on the dataset.\n\nFrom the Scikit-learn documentation: \n> The goal of using tf-idf instead of the raw frequencies of occurrence of a token in a given document is to scale down the impact of tokens that occur very frequently in a given corpus and that are hence empirically less informative than features that occur in a small fraction of the training corpus.  In a large text corpus, some words will be very present (e.g. \u201cthe\u201d, \u201ca\u201d, \u201cis\u201d in English) hence carrying very little meaningful information about the actual contents of the document. If we were to feed the direct count data directly to a classifier those very frequent terms would shadow the frequencies of rarer yet more interesting terms.\nIn order to re-weight the count features into floating point values suitable for usage by a classifier it is very common to use the tf\u2013idf transform.\n\nWe will consider varying output modes as we go forward. \n\nSince the TextVectorization layer will allow us to convert the text of our stories to integer tensors, so there is not much for us to do with feature engineering.  \n\n## Vocabulary Size\nThe default TextVectorization settings will retain all words found as part of our vocabulary.  In experimenting, we found some value in altering the vocabulary size.  \n\nBelow is the code we used to count words and find words that only appeared once, with the hypothesis that any word that appeared in only one article could not inform decisions on any other articles. The code and some further discussion is found below. "
        },
        {
            "metadata": {},
            "cell_type": "code",
            "source": "!pip install --upgrade numpy\n!pip install --upgrade pandas\n\n# we want tensorflow 2.3\n!pip install --upgrade tensorflow  ",
            "execution_count": 1,
            "outputs": [
                {
                    "output_type": "stream",
                    "text": "Collecting numpy\n\u001b[?25l  Downloading https://files.pythonhosted.org/packages/b1/9a/7d474ba0860a41f771c9523d8c4ea56b084840b5ca4092d96bdee8a3b684/numpy-1.19.1-cp36-cp36m-manylinux2010_x86_64.whl (14.5MB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 14.5MB 7.7MB/s eta 0:00:01\n\u001b[31mERROR: tensorflow 1.13.1 requires tensorboard<1.14.0,>=1.13.0, which is not installed.\u001b[0m\n\u001b[31mERROR: autoai-libs 1.10.5 has requirement pandas>=0.24.2, but you'll have pandas 0.24.1 which is incompatible.\u001b[0m\n\u001b[?25hInstalling collected packages: numpy\n  Found existing installation: numpy 1.15.4\n    Uninstalling numpy-1.15.4:\n      Successfully uninstalled numpy-1.15.4\nSuccessfully installed numpy-1.19.1\nCollecting pandas\n\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a1/c6/9ac4ae44c24c787a1738e5fb34dd987ada6533de5905a041aa6d5bea4553/pandas-1.1.1-cp36-cp36m-manylinux1_x86_64.whl (10.5MB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 10.5MB 7.6MB/s eta 0:00:01\n\u001b[?25hRequirement already satisfied, skipping upgrade: numpy>=1.15.4 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from pandas) (1.19.1)\nRequirement already satisfied, skipping upgrade: pytz>=2017.2 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from pandas) (2018.9)\nRequirement already satisfied, skipping upgrade: python-dateutil>=2.7.3 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from pandas) (2.7.5)\nRequirement already satisfied, skipping upgrade: six>=1.5 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from python-dateutil>=2.7.3->pandas) (1.12.0)\nInstalling collected packages: pandas\n  Found existing installation: pandas 0.24.1\n    Uninstalling pandas-0.24.1:\n      Successfully uninstalled pandas-0.24.1\nSuccessfully installed pandas-1.1.1\nCollecting tensorflow\n\u001b[?25l  Downloading https://files.pythonhosted.org/packages/97/ae/0b08f53498417914f2274cc3b5576d2b83179b0cbb209457d0fde0152174/tensorflow-2.3.0-cp36-cp36m-manylinux2010_x86_64.whl (320.4MB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 320.4MB 91kB/s s eta 0:00:01    |\u2588\u2588\u258b                             | 25.8MB 27.5MB/s eta 0:00:11     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258b                        | 76.5MB 39.8MB/s eta 0:00:07     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258d                       | 84.4MB 39.8MB/s eta 0:00:06     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588                       | 90.6MB 39.8MB/s eta 0:00:06     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e                 | 142.8MB 39.8MB/s eta 0:00:05     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258c               | 164.8MB 6.9MB/s eta 0:00:23     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a         | 227.4MB 9.4MB/s eta 0:00:10     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258d    | 274.4MB 8.4MB/s eta 0:00:06\n\u001b[?25hCollecting opt-einsum>=2.3.2 (from tensorflow)\n\u001b[?25l  Downloading https://files.pythonhosted.org/packages/bc/19/404708a7e54ad2798907210462fd950c3442ea51acc8790f3da48d2bee8b/opt_einsum-3.3.0-py3-none-any.whl (65kB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 71kB 18.9MB/s eta 0:00:01\n\u001b[?25hRequirement already satisfied, skipping upgrade: six>=1.12.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow) (1.12.0)\nRequirement already satisfied, skipping upgrade: termcolor>=1.1.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow) (1.1.0)\nCollecting scipy==1.4.1 (from tensorflow)\n\u001b[?25l  Downloading https://files.pythonhosted.org/packages/dc/29/162476fd44203116e7980cfbd9352eef9db37c49445d1fec35509022f6aa/scipy-1.4.1-cp36-cp36m-manylinux1_x86_64.whl (26.1MB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 26.1MB 8.8MB/s eta 0:00:011     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2589             | 15.3MB 35.4MB/s eta 0:00:01     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258d           | 16.6MB 35.4MB/s eta 0:00:01     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588  | 24.4MB 8.8MB/s eta 0:00:01\n\u001b[?25hCollecting tensorflow-estimator<2.4.0,>=2.3.0 (from tensorflow)\n\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e9/ed/5853ec0ae380cba4588eab1524e18ece1583b65f7ae0e97321f5ff9dfd60/tensorflow_estimator-2.3.0-py2.py3-none-any.whl (459kB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 460kB 12.8MB/s eta 0:00:01\n\u001b[?25hRequirement already satisfied, skipping upgrade: grpcio>=1.8.6 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow) (1.16.1)\nCollecting tensorboard<3,>=2.3.0 (from tensorflow)\n\u001b[?25l  Downloading https://files.pythonhosted.org/packages/e9/1b/6a420d7e6ba431cf3d51b2a5bfa06a958c4141e3189385963dc7f6fbffb6/tensorboard-2.3.0-py3-none-any.whl (6.8MB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 6.8MB 33.3MB/s eta 0:00:01    | 1.2MB 33.3MB/s eta 0:00:01\n\u001b[?25hCollecting google-pasta>=0.1.8 (from tensorflow)\n\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a3/de/c648ef6835192e6e2cc03f40b19eeda4382c49b5bafb43d88b931c4c74ac/google_pasta-0.2.0-py3-none-any.whl (57kB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 61kB 17.4MB/s eta 0:00:01\n\u001b[?25hCollecting gast==0.3.3 (from tensorflow)\n  Downloading https://files.pythonhosted.org/packages/d6/84/759f5dd23fec8ba71952d97bcc7e2c9d7d63bdc582421f3cd4be845f0c98/gast-0.3.3-py2.py3-none-any.whl\nRequirement already satisfied, skipping upgrade: absl-py>=0.7.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow) (0.7.0)\nRequirement already satisfied, skipping upgrade: wrapt>=1.11.1 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow) (1.11.1)\nCollecting keras-preprocessing<1.2,>=1.1.1 (from tensorflow)\n\u001b[?25l  Downloading https://files.pythonhosted.org/packages/79/4c/7c3275a01e12ef9368a892926ab932b33bb13d55794881e3573482b378a7/Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42kB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 51kB 20.2MB/s eta 0:00:01\n\u001b[?25hCollecting numpy<1.19.0,>=1.16.0 (from tensorflow)\n\u001b[?25l  Downloading https://files.pythonhosted.org/packages/b3/a9/b1bc4c935ed063766bce7d3e8c7b20bd52e515ff1c732b02caacf7918e5a/numpy-1.18.5-cp36-cp36m-manylinux1_x86_64.whl (20.1MB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 20.1MB 25.2MB/s eta 0:00:01\n\u001b[?25hCollecting astunparse==1.6.3 (from tensorflow)\n  Downloading https://files.pythonhosted.org/packages/2b/03/13dde6512ad7b4557eb792fbcf0c653af6076b81e5941d36ec61f7ce6028/astunparse-1.6.3-py2.py3-none-any.whl\nRequirement already satisfied, skipping upgrade: protobuf>=3.9.2 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow) (3.11.2)\nRequirement already satisfied, skipping upgrade: wheel>=0.26 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorflow) (0.32.3)\nCollecting h5py<2.11.0,>=2.10.0 (from tensorflow)\n\u001b[?25l  Downloading https://files.pythonhosted.org/packages/60/06/cafdd44889200e5438b897388f3075b52a8ef01f28a17366d91de0fa2d05/h5py-2.10.0-cp36-cp36m-manylinux1_x86_64.whl (2.9MB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 2.9MB 36.5MB/s eta 0:00:01\n\u001b[?25hCollecting google-auth-oauthlib<0.5,>=0.4.1 (from tensorboard<3,>=2.3.0->tensorflow)\n  Downloading https://files.pythonhosted.org/packages/7b/b8/88def36e74bee9fce511c9519571f4e485e890093ab7442284f4ffaef60b/google_auth_oauthlib-0.4.1-py2.py3-none-any.whl\nCollecting setuptools>=41.0.0 (from tensorboard<3,>=2.3.0->tensorflow)\n\u001b[?25l  Downloading https://files.pythonhosted.org/packages/b0/8b/379494d7dbd3854aa7b85b216cb0af54edcb7fce7d086ba3e35522a713cf/setuptools-50.0.0-py3-none-any.whl (783kB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 788kB 34.3MB/s eta 0:00:01\n\u001b[?25hRequirement already satisfied, skipping upgrade: werkzeug>=0.11.15 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorboard<3,>=2.3.0->tensorflow) (0.14.1)\nRequirement already satisfied, skipping upgrade: markdown>=2.6.8 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorboard<3,>=2.3.0->tensorflow) (3.0.1)\nRequirement already satisfied, skipping upgrade: requests<3,>=2.21.0 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from tensorboard<3,>=2.3.0->tensorflow) (2.21.0)\nCollecting tensorboard-plugin-wit>=1.6.0 (from tensorboard<3,>=2.3.0->tensorflow)\n\u001b[?25l  Downloading https://files.pythonhosted.org/packages/b6/85/5c5ac0a8c5efdfab916e9c6bc18963f6a6996a8a1e19ec4ad8c9ac9c623c/tensorboard_plugin_wit-1.7.0-py3-none-any.whl (779kB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 788kB 37.5MB/s eta 0:00:01\n\u001b[?25hCollecting google-auth<2,>=1.6.3 (from tensorboard<3,>=2.3.0->tensorflow)\n\u001b[?25l  Downloading https://files.pythonhosted.org/packages/63/7f/ef6bcf2cc0f50c7163afb94382aab67a6b278e1e447c2e3981aa281b9747/google_auth-1.21.0-py2.py3-none-any.whl (92kB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 102kB 23.5MB/s ta 0:00:01\n",
                    "name": "stdout"
                },
                {
                    "output_type": "stream",
                    "text": "\u001b[?25hCollecting requests-oauthlib>=0.7.0 (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<3,>=2.3.0->tensorflow)\n  Downloading https://files.pythonhosted.org/packages/a3/12/b92740d845ab62ea4edf04d2f4164d82532b5a0b03836d4d4e71c6f3d379/requests_oauthlib-1.3.0-py2.py3-none-any.whl\nRequirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow) (3.0.4)\nRequirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow) (2.8)\nRequirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow) (2020.6.20)\nRequirement already satisfied, skipping upgrade: urllib3<1.25,>=1.21.1 in /opt/conda/envs/Python36/lib/python3.6/site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow) (1.24.1)\nCollecting rsa<5,>=3.1.4; python_version >= \"3.5\" (from google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow)\n\u001b[?25l  Downloading https://files.pythonhosted.org/packages/1c/df/c3587a667d6b308fadc90b99e8bc8774788d033efcc70f4ecaae7fad144b/rsa-4.6-py3-none-any.whl (47kB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 51kB 18.2MB/s eta 0:00:01\n\u001b[?25hCollecting pyasn1-modules>=0.2.1 (from google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow)\n\u001b[?25l  Downloading https://files.pythonhosted.org/packages/95/de/214830a981892a3e286c3794f41ae67a4495df1108c3da8a9f62159b9a9d/pyasn1_modules-0.2.8-py2.py3-none-any.whl (155kB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 163kB 37.0MB/s eta 0:00:01\n\u001b[?25hCollecting cachetools<5.0,>=2.0.0 (from google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow)\n  Downloading https://files.pythonhosted.org/packages/cd/5c/f3aa86b6d5482f3051b433c7616668a9b96fbe49a622210e2c9781938a5c/cachetools-4.1.1-py3-none-any.whl\nCollecting oauthlib>=3.0.0 (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<3,>=2.3.0->tensorflow)\n\u001b[?25l  Downloading https://files.pythonhosted.org/packages/05/57/ce2e7a8fa7c0afb54a0581b14a65b56e62b5759dbc98e80627142b8a3704/oauthlib-3.1.0-py2.py3-none-any.whl (147kB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 153kB 33.3MB/s eta 0:00:01\n\u001b[?25hCollecting pyasn1>=0.1.3 (from rsa<5,>=3.1.4; python_version >= \"3.5\"->google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow)\n\u001b[?25l  Downloading https://files.pythonhosted.org/packages/62/1e/a94a8d635fa3ce4cfc7f506003548d0a2447ae76fd5ca53932970fe3053f/pyasn1-0.4.8-py2.py3-none-any.whl (77kB)\n\u001b[K     |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 81kB 20.6MB/s eta 0:00:01\n\u001b[31mERROR: tensorboard 2.3.0 has requirement grpcio>=1.24.3, but you'll have grpcio 1.16.1 which is incompatible.\u001b[0m\n\u001b[?25hInstalling collected packages: numpy, opt-einsum, scipy, tensorflow-estimator, pyasn1, rsa, pyasn1-modules, cachetools, setuptools, google-auth, oauthlib, requests-oauthlib, google-auth-oauthlib, tensorboard-plugin-wit, tensorboard, google-pasta, gast, keras-preprocessing, astunparse, h5py, tensorflow\n  Found existing installation: numpy 1.19.1\n    Uninstalling numpy-1.19.1:\n      Successfully uninstalled numpy-1.19.1\n  Found existing installation: scipy 1.2.0\n    Uninstalling scipy-1.2.0:\n      Successfully uninstalled scipy-1.2.0\n  Found existing installation: tensorflow-estimator 1.13.0\n    Uninstalling tensorflow-estimator-1.13.0:\n      Successfully uninstalled tensorflow-estimator-1.13.0\n  Found existing installation: setuptools 40.8.0\n    Uninstalling setuptools-40.8.0:\n      Successfully uninstalled setuptools-40.8.0\n  Found existing installation: gast 0.2.2\n    Uninstalling gast-0.2.2:\n      Successfully uninstalled gast-0.2.2\n  Found existing installation: Keras-Preprocessing 1.0.5\n    Uninstalling Keras-Preprocessing-1.0.5:\n      Successfully uninstalled Keras-Preprocessing-1.0.5\n  Found existing installation: astunparse 1.6.2\n    Uninstalling astunparse-1.6.2:\n      Successfully uninstalled astunparse-1.6.2\n  Found existing installation: h5py 2.9.0\n    Uninstalling h5py-2.9.0:\n      Successfully uninstalled h5py-2.9.0\n  Found existing installation: tensorflow 1.13.1\n    Uninstalling tensorflow-1.13.1:\n      Successfully uninstalled tensorflow-1.13.1\nSuccessfully installed astunparse-1.6.3 cachetools-4.1.1 gast-0.3.3 google-auth-1.21.0 google-auth-oauthlib-0.4.1 google-pasta-0.2.0 h5py-2.10.0 keras-preprocessing-1.1.2 numpy-1.18.5 oauthlib-3.1.0 opt-einsum-3.3.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 requests-oauthlib-1.3.0 rsa-4.6 scipy-1.4.1 setuptools-50.0.0 tensorboard-2.3.0 tensorboard-plugin-wit-1.7.0 tensorflow-2.3.0 tensorflow-estimator-2.3.0\n",
                    "name": "stdout"
                }
            ]
        },
        {
            "metadata": {},
            "cell_type": "code",
            "source": "import tensorflow as tf\nprint(\"Tensorflow version: \", tf.__version__)\nif not tf.__version__ == '2.3.0':\n    raise ValueError('please upgrade to TensorFlow 2.3, or restart your Kernel (Kernel->Restart & Clear Output)')",
            "execution_count": 2,
            "outputs": [
                {
                    "output_type": "stream",
                    "text": "Tensorflow version:  2.3.0\n",
                    "name": "stdout"
                }
            ]
        },
        {
            "metadata": {},
            "cell_type": "code",
            "source": "%matplotlib inline\nimport matplotlib\nimport matplotlib.pyplot as plt\nfrom pprint import pprint\nfrom time import time\nimport logging\nimport numpy as np\nimport pandas as pd\nimport string\nimport re\n\nfrom keras.utils import to_categorical\nfrom keras import models\nfrom keras import layers\n\nfrom tensorflow.keras.layers.experimental.preprocessing import TextVectorization\n\nfrom sklearn.model_selection import train_test_split\n\nfrom ibm_botocore.client import Config\nimport ibm_boto3",
            "execution_count": 3,
            "outputs": [
                {
                    "output_type": "stream",
                    "text": "Using TensorFlow backend.\n",
                    "name": "stderr"
                }
            ]
        },
        {
            "metadata": {},
            "cell_type": "code",
            "source": "#Get our data\n# @hidden_cell\n# The following code contains the credentials for a file in your IBM Cloud Object Storage.\n# You might want to remove those credentials before you share your notebook.\ncredentials_news = {\n    'IAM_SERVICE_ID': 'iam-ServiceId-32e8ee67-397c-4ff1-b69b-543172331f43',\n    'IBM_API_KEY_ID': 'Rx4FR4JSAueCnnIsoevsgYgOsuh8LCXtbkFpFpC0EmVU',\n    'ENDPOINT': 'https://s3-api.us-geo.objectstorage.service.networklayer.com',\n    'IBM_AUTH_ENDPOINT': 'https://iam.cloud.ibm.com/oidc/token',\n    'BUCKET': 'advanceddatasciencecapstone-donotdelete-pr-tqabpnbxebk8rm',\n    'FILE': 'dfTrueFalseNews.pkl'\n}\n\ndef download_file_cos(credentials,local_file_name,key):  \n    cos = ibm_boto3.client(service_name='s3',\n    ibm_api_key_id=credentials['IBM_API_KEY_ID'],\n    ibm_service_instance_id=credentials['IAM_SERVICE_ID'],\n    ibm_auth_endpoint=credentials['IBM_AUTH_ENDPOINT'],\n    config=Config(signature_version='oauth'),\n    endpoint_url=credentials['ENDPOINT'])\n    try:\n        res=cos.download_file(Bucket=credentials['BUCKET'],Key=key,Filename=local_file_name)\n    except Exception as e:\n        print(Exception, e)\n    else:\n        print('File Downloaded')\n\ndef upload_file_cos(credentials,local_file_name,key):  \n    cos = ibm_boto3.client(service_name='s3',\n    ibm_api_key_id=credentials['IBM_API_KEY_ID'],\n    ibm_service_instance_id=credentials['IAM_SERVICE_ID'],\n    ibm_auth_endpoint=credentials['IBM_AUTH_ENDPOINT'],\n    config=Config(signature_version='oauth'),\n    endpoint_url=credentials['ENDPOINT'])\n    try:\n        res=cos.upload_file(Filename=local_file_name, Bucket=credentials['BUCKET'],Key=key)\n    except Exception as e:\n        print(Exception, e)\n    else:\n        print(' File Uploaded')\n        \ndfNews = download_file_cos(credentials_news, \"dfTrueFalseNews.pkl\", \"dfTrueFalseNews.pkl\")",
            "execution_count": 4,
            "outputs": [
                {
                    "output_type": "stream",
                    "text": "File Downloaded\n",
                    "name": "stdout"
                }
            ]
        },
        {
            "metadata": {},
            "cell_type": "code",
            "source": "dfNews = pd.read_pickle('dfTrueFalseNews.pkl')\n#dfNews['truthvalue'] = pd.Categorical(dfNews['truthvalue'])\n\nprint (dfNews.shape, dfNews.columns, '\\n',  dfNews.dtypes)",
            "execution_count": 5,
            "outputs": [
                {
                    "output_type": "stream",
                    "text": "(1126, 3) Index(['text', 'source', 'truthvalue'], dtype='object') \n text          object\nsource        object\ntruthvalue    object\ndtype: object\n",
                    "name": "stdout"
                }
            ]
        },
        {
            "metadata": {},
            "cell_type": "code",
            "source": "dfNews.head()",
            "execution_count": 6,
            "outputs": [
                {
                    "output_type": "execute_result",
                    "execution_count": 6,
                    "data": {
                        "text/plain": "                                                           text  \\\ntech003legit  A Google computer victorious over the world's ...   \npolit11legit  White House keeps up sanctuary cities pressure...   \nbiz40legit    Why Silicon Valley isn't fighting to save the ...   \nedu10legit    Protesters Disrupt DeVos School Visit   Protes...   \ntech038legit  Solar-powered 'skin' could make prosthetics mo...   \n\n                         source truthvalue  \ntech003legit  MihalceaNewsLegit          1  \npolit11legit  MihalceaNewsLegit          1  \nbiz40legit    MihalceaNewsLegit          1  \nedu10legit    MihalceaNewsLegit          1  \ntech038legit  MihalceaNewsLegit          1  ",
                        "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>text</th>\n      <th>source</th>\n      <th>truthvalue</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>tech003legit</th>\n      <td>A Google computer victorious over the world's ...</td>\n      <td>MihalceaNewsLegit</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>polit11legit</th>\n      <td>White House keeps up sanctuary cities pressure...</td>\n      <td>MihalceaNewsLegit</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>biz40legit</th>\n      <td>Why Silicon Valley isn't fighting to save the ...</td>\n      <td>MihalceaNewsLegit</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>edu10legit</th>\n      <td>Protesters Disrupt DeVos School Visit   Protes...</td>\n      <td>MihalceaNewsLegit</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>tech038legit</th>\n      <td>Solar-powered 'skin' could make prosthetics mo...</td>\n      <td>MihalceaNewsLegit</td>\n      <td>1</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
                    },
                    "metadata": {}
                }
            ]
        },
        {
            "metadata": {},
            "cell_type": "code",
            "source": "indexes = dfNews.index.values\nx = dfNews['text'].values\ny = dfNews['truthvalue'].values\nprint(type(x), type(y), indexes[0:2])",
            "execution_count": 26,
            "outputs": [
                {
                    "output_type": "stream",
                    "text": "<class 'numpy.ndarray'> <class 'numpy.ndarray'> ['tech003legit' 'polit11legit']\n",
                    "name": "stdout"
                }
            ]
        },
        {
            "metadata": {},
            "cell_type": "code",
            "source": "X_train, X_test, y_train, y_test = train_test_split(x,y, test_size=0.2, random_state=42)\n\n# Once we have our handles, we format the datasets in a Keras-fit compatible\n# format: a tuple of the form (text_data, label).\ndef format_dataset(x, y):\n  return (x, y)\n\ntrain_dataset = list(map(format_dataset, X_train, y_train))\ntest_dataset = list(map(format_dataset, X_test, y_test))\n\n# We also create a dataset with only the textual data in it. This will be used\n# to build our vocabulary later on.\ntextL_dataset = list(map(lambda a:a, x))\n",
            "execution_count": 8,
            "outputs": []
        },
        {
            "metadata": {},
            "cell_type": "code",
            "source": "print (len(X_train), len(X_test), len(y_train), len(y_test), len(textL_dataset), '\\n',\ntype(X_train), type(X_test), type(y_train), type(y_test), type(textL_dataset))\n",
            "execution_count": 9,
            "outputs": [
                {
                    "output_type": "stream",
                    "text": "900 226 900 226 1126 \n <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'numpy.ndarray'> <class 'list'>\n",
                    "name": "stdout"
                }
            ]
        },
        {
            "metadata": {},
            "cell_type": "code",
            "source": "# move our numpy structures into Tensorflow datasets\nDataset = tf.data.Dataset\ntext_dataset = tf.data.Dataset.from_tensor_slices(textL_dataset)\n\nfeatures_dataset = Dataset.from_tensor_slices(X_train)\nlabels_dataset = Dataset.from_tensor_slices(list(y_train))\ntfds_train = Dataset.zip((features_dataset, labels_dataset))\n\nfeatures_test_dataset = Dataset.from_tensor_slices(X_test)\nlabels_test_dataset = Dataset.from_tensor_slices(list(y_test))\ntfds_test = Dataset.zip((features_test_dataset, labels_test_dataset))",
            "execution_count": 10,
            "outputs": []
        },
        {
            "metadata": {},
            "cell_type": "markdown",
            "source": "# Determine the optimum vocabulary size.  \nThe results for model 2 below showed that varying the vocabulary size was productive, with the optimum size seeming to be between 18000 and 25000.\nWe know there are a lot of junk words in our data, where spaces are missing and words appear only once in a story (ex: we find \"dyma davi d la pajti\" in our text, a French tranliteration of \"Dumas Davy de la Pailleterie\"). \n\nLet's count our words and see what the vocabulary size would be if we removed these."
        },
        {
            "metadata": {},
            "cell_type": "code",
            "source": "import re\nimport string\nimport statistics\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nfrom collections import Counter\ndef CleanUpPunctuation(pattern, rep, input_data):\n  lowercase = input_data.lower()\n  s = pattern.sub(lambda m: rep[re.escape(m.group(0))], lowercase)\n  return s\n\n\n#rep = {\"condition1\": \"\", \"condition2\": \"text\"} # define desired replacements here\nrep =  {re.escape(s):\"\" for i,s in enumerate(string.punctuation)}\n# use these three lines to do the replacement\npattern = re.compile(\"|\".join(rep.keys()))\n\nv = list(x)\ncnt = Counter()\nstorylength = []\nunique = []\nvocab = set()\nrow = 0\nfor a in v:\n    u = set()\n    a = CleanUpPunctuation(pattern, rep, a)\n    # split returns a list of words delimited by sequences of whitespace (including tabs, newlines, etc, like re's \\s) \n    alist = a.split()\n    storylen = len(alist)\n    storylength.append(storylen)\n    if storylen == 14722:\n        maxlenrow = row\n    if storylen == 27:\n        minlenrow = row\n    u = u.union((alist))\n    vocab = vocab.union(alist)\n    unique.append (len(u))\n    for word in alist:\n        cnt[word] += 1\n    row +=1\nprint (\"Average story length: {} Minimum story length: {} Maximum story length: {} Standard deviation: {} Total words in corpus: {} Vocab size: {} Maxstoryrow: {} Minstoryrow: {}\" \\\n       .format(statistics.mean(storylength), min(storylength), max(storylength), statistics.stdev(storylength), sum(storylength), len(vocab), indexes[maxlenrow], indexes[minlenrow]))\nplt.hist(storylength, 100)",
            "execution_count": 28,
            "outputs": [
                {
                    "output_type": "stream",
                    "text": "Average story length: 328.6305506216696 Minimum story length: 27 Maximum story length: 14722 Standard deviation: 624.980898772334 Total words in corpus: 370038 Vocab size: 26595 Maxstoryrow: celeb060legit Minstoryrow: celeb131fake\n",
                    "name": "stdout"
                },
                {
                    "output_type": "execute_result",
                    "execution_count": 28,
                    "data": {
                        "text/plain": "(array([552., 248., 168.,  57.,  33.,  16.,   6.,  11.,   3.,   6.,   4.,\n          4.,   1.,   2.,   1.,   1.,   2.,   0.,   1.,   1.,   0.,   0.,\n          0.,   0.,   1.,   1.,   0.,   1.,   2.,   1.,   1.,   0.,   0.,\n          0.,   1.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n          0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n          0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n          0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n          0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n          0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n          1.]),\n array([   27.  ,   173.95,   320.9 ,   467.85,   614.8 ,   761.75,\n          908.7 ,  1055.65,  1202.6 ,  1349.55,  1496.5 ,  1643.45,\n         1790.4 ,  1937.35,  2084.3 ,  2231.25,  2378.2 ,  2525.15,\n         2672.1 ,  2819.05,  2966.  ,  3112.95,  3259.9 ,  3406.85,\n         3553.8 ,  3700.75,  3847.7 ,  3994.65,  4141.6 ,  4288.55,\n         4435.5 ,  4582.45,  4729.4 ,  4876.35,  5023.3 ,  5170.25,\n         5317.2 ,  5464.15,  5611.1 ,  5758.05,  5905.  ,  6051.95,\n         6198.9 ,  6345.85,  6492.8 ,  6639.75,  6786.7 ,  6933.65,\n         7080.6 ,  7227.55,  7374.5 ,  7521.45,  7668.4 ,  7815.35,\n         7962.3 ,  8109.25,  8256.2 ,  8403.15,  8550.1 ,  8697.05,\n         8844.  ,  8990.95,  9137.9 ,  9284.85,  9431.8 ,  9578.75,\n         9725.7 ,  9872.65, 10019.6 , 10166.55, 10313.5 , 10460.45,\n        10607.4 , 10754.35, 10901.3 , 11048.25, 11195.2 , 11342.15,\n        11489.1 , 11636.05, 11783.  , 11929.95, 12076.9 , 12223.85,\n        12370.8 , 12517.75, 12664.7 , 12811.65, 12958.6 , 13105.55,\n        13252.5 , 13399.45, 13546.4 , 13693.35, 13840.3 , 13987.25,\n        14134.2 , 14281.15, 14428.1 , 14575.05, 14722.  ]),\n <a list of 100 Patch objects>)"
                    },
                    "metadata": {}
                },
                {
                    "output_type": "display_data",
                    "data": {
                        "text/plain": "<Figure size 432x288 with 1 Axes>",
                        "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAEShJREFUeJzt3W+MXNV5x/HvUztA/hXbeKGubXWhsaKQFwG6ok6pqhTShD9RTKUggaLEJY4sNTRKmlapKVLbSH1hkqpQ1IrECklNRAKUhGIBaYocorYv4mSdgIEA8eI4sDHCS/mTtlGq0Dx9MWft8XrWM7tzZ2f35PuRRnPuuWfmPnNm57d3752ZjcxEklSvXxp2AZKkwTLoJalyBr0kVc6gl6TKGfSSVDmDXpIqZ9BLUuUMekmqnEEvSZVbPuwCAFavXp2jo6PDLkOSlpS9e/c+n5kj3cYtiqAfHR1lfHx82GVI0pISET/sZZyHbiSpcga9JFXOoJekyhn0klQ5g16SKmfQS1LlDHpJqpxBL0mVM+glqXKL4pOx/Rjddt+R9sHtlw2xEklanNyjl6TKGfSSVDmDXpIqZ9BLUuUMekmqnEEvSZUz6CWpcga9JFXOoJekyhn0klQ5g16SKmfQS1LlDHpJqpxBL0mVM+glqXI9BX1EHIyIRyLioYgYL32rIuKBiNhfrleW/oiImyJiIiL2RcR5g3wAkqQTm8se/e9m5jmZOVaWtwG7M3MDsLssA1wCbCiXrcDNTRUrSZq7fg7dbAJ2lvZO4PK2/luz5ZvAiohY08d2JEl96DXoE/jXiNgbEVtL3xmZ+SxAuT699K8Fnmm77WTpO0ZEbI2I8YgYn5qaml/1kqSuev2fsRdk5qGIOB14ICKeOMHY6NCXx3Vk7gB2AIyNjR23XpLUjJ726DPzULk+DNwNnA88N31IplwfLsMngfVtN18HHGqqYEnS3HQN+oh4bUS8froNvAN4FNgFbC7DNgP3lPYu4P3l3TcbgZenD/FIkhZeL4duzgDujojp8V/MzH+JiG8Dd0bEFuBp4Ioy/n7gUmAC+AlwdeNVS5J61jXoM/MA8JYO/f8JXNShP4FrGqlOktQ3PxkrSZUz6CWpcga9JFXOoJekyhn0klQ5g16SKmfQS1LlDHpJqpxBL0mVM+glqXIGvSRVzqCXpMoZ9JJUOYNekipn0EtS5Qx6SaqcQS9JlTPoJalyBr0kVc6gl6TKGfSSVDmDXpIqZ9BLUuUMekmqnEEvSZUz6CWpcga9JFXOoJekyvUc9BGxLCK+GxH3luUzI2JPROyPiDsi4qTSf3JZnijrRwdTuiSpF3PZo/8I8Hjb8vXADZm5AXgR2FL6twAvZuYbgBvKOEnSkPQU9BGxDrgM+GxZDuBC4K4yZCdweWlvKsuU9ReV8ZKkIeh1j/5G4OPAz8vyacBLmflKWZ4E1pb2WuAZgLL+5TJekjQEXYM+It4FHM7Mve3dHYZmD+va73drRIxHxPjU1FRPxUqS5q6XPfoLgHdHxEHgdlqHbG4EVkTE8jJmHXCotCeB9QBl/anACzPvNDN3ZOZYZo6NjIz09SAkSbPrGvSZeW1mrsvMUeBK4OuZ+V7gQeA9Zdhm4J7S3lWWKeu/npnH7dFLkhZGP++j/zPgYxExQesY/C2l/xbgtNL/MWBbfyVKkvqxvPuQozLzG8A3SvsAcH6HMT8FrmigNklSA/xkrCRVzqCXpMoZ9JJUOYNekipn0EtS5Qx6SaqcQS9JlTPoJalyBr0kVc6gl6TKGfSSVDmDXpIqZ9BLUuUMekmqnEEvSZUz6CWpcga9JFXOoJekyhn0klQ5g16SKmfQS1LlDHpJqpxBL0mVM+glqXIGvSRVzqCXpMoZ9JJUOYNekipn0EtS5boGfUScEhHfioiHI+KxiPhE6T8zIvZExP6IuCMiTir9J5flibJ+dLAPQZJ0Ir3s0f8vcGFmvgU4B7g4IjYC1wM3ZOYG4EVgSxm/BXgxM98A3FDGSZKGpGvQZ8t/l8VXlUsCFwJ3lf6dwOWlvaksU9ZfFBHRWMWSpDnp6Rh9RCyLiIeAw8ADwFPAS5n5ShkyCawt7bXAMwBl/cvAaR3uc2tEjEfE+NTUVH+PQpI0q56CPjP/LzPPAdYB5wNv6jSsXHfae8/jOjJ3ZOZYZo6NjIz0Wq8kaY7m9K6bzHwJ+AawEVgREcvLqnXAodKeBNYDlPWnAi80Uawkae56edfNSESsKO1XA28HHgceBN5Thm0G7intXWWZsv7rmXncHr0kaWEs7z6ENcDOiFhG6xfDnZl5b0R8D7g9Iv4a+C5wSxl/C/CFiJigtSd/5QDqliT1qGvQZ+Y+4NwO/QdoHa+f2f9T4IpGqpMk9c1PxkpS5Qx6SaqcQS9JlTPoJalyBr0kVc6gl6TKGfSSVDmDXpIqZ9BLUuUMekmqnEEvSZUz6CWpcga9JFXOoJekyvXyffRLxui2+460D26/bIiVSNLi4R69JFXOoJekyhn0klQ5g16SKmfQS1LlDHpJqpxBL0mVM+glqXIGvSRVzqCXpMoZ9JJUOYNekipn0EtS5boGfUSsj4gHI+LxiHgsIj5S+ldFxAMRsb9cryz9ERE3RcREROyLiPMG/SAkSbPrZY/+FeBPMvNNwEbgmog4G9gG7M7MDcDusgxwCbChXLYCNzdetSSpZ12DPjOfzczvlPZ/AY8Da4FNwM4ybCdweWlvAm7Nlm8CKyJiTeOVS5J6Mqdj9BExCpwL7AHOyMxnofXLADi9DFsLPNN2s8nSJ0kagp6DPiJeB3wZ+Ghm/vhEQzv0ZYf72xoR4xExPjU11WsZkqQ56inoI+JVtEL+tsz8Sul+bvqQTLk+XPongfVtN18HHJp5n5m5IzPHMnNsZGRkvvVLkrro5V03AdwCPJ6Zf9u2ahewubQ3A/e09b+/vPtmI/Dy9CEeSdLC6+Wfg18AvA94JCIeKn1/DmwH7oyILcDTwBVl3f3ApcAE8BPg6kYrliTNSdegz8z/oPNxd4CLOoxP4Jo+65IkNcRPxkpS5Qx6SaqcQS9JlTPoJalyBr0kVc6gl6TKGfSSVLlePjC1JI1uu+9I++D2y4ZYiSQNl3v0klQ5g16SKmfQS1LlDHpJqpxBL0mVM+glqXIGvSRVzqCXpMoZ9JJUOYNekipn0EtS5Qx6SaqcQS9JlTPoJalyBr0kVc6gl6TKGfSSVDmDXpIqZ9BLUuUMekmqnEEvSZXrGvQR8bmIOBwRj7b1rYqIByJif7leWfojIm6KiImI2BcR5w2yeElSd73s0f8jcPGMvm3A7szcAOwuywCXABvKZStwczNlSpLmq2vQZ+a/AS/M6N4E7CztncDlbf23Zss3gRURsaapYiVJczffY/RnZOazAOX69NK/Fnimbdxk6TtORGyNiPGIGJ+amppnGZKkbpo+GRsd+rLTwMzckZljmTk2MjLScBmSpGnzDfrnpg/JlOvDpX8SWN82bh1waP7lSZL6Nd+g3wVsLu3NwD1t/e8v777ZCLw8fYhHkjQcy7sNiIgvAW8DVkfEJPCXwHbgzojYAjwNXFGG3w9cCkwAPwGuHkDNkqQ56Br0mXnVLKsu6jA2gWv6LUqS1Bw/GStJlTPoJalyBr0kVc6gl6TKGfSSVDmDXpIqZ9BLUuUMekmqnEEvSZUz6CWpcl2/AqEGo9vuO9I+uP2yIVYiSQvPPXpJqpxBL0mVM+glqXIGvSRVzqCXpMoZ9JJUOYNekipn0EtS5X4hPjDVrv3DU+AHqCTVzz16SaqcQS9JlTPoJalyBr0kVc6gl6TKGfSSVLlfuLdXzuR31UuqnXv0klS5gezRR8TFwN8By4DPZub2QWxnkNzTl1SLxvfoI2IZ8A/AJcDZwFURcXbT25Ek9WYQe/TnAxOZeQAgIm4HNgHfG8C2GjXz6xHmO8a/ACQtJoMI+rXAM23Lk8BvDmA7C6aXcO9nfLv2XxKz3c9cf5H0cj/zqbmXOjwEJh1voV8XkZnN3mHEFcA7M/ODZfl9wPmZ+eEZ47YCW8viG4En57nJ1cDz87ztQlkKNcLSqNMam7MU6lwKNcLw6vy1zBzpNmgQe/STwPq25XXAoZmDMnMHsKPfjUXEeGaO9Xs/g7QUaoSlUac1Nmcp1LkUaoTFX+cg3l75bWBDRJwZEScBVwK7BrAdSVIPGt+jz8xXIuKPgK/Renvl5zLzsaa3I0nqzUDeR5+Z9wP3D+K+O+j78M8CWAo1wtKo0xqbsxTqXAo1wiKvs/GTsZKkxcWvQJCkyi3ZoI+IiyPiyYiYiIhtC7zt9RHxYEQ8HhGPRcRHSv+qiHggIvaX65WlPyLiplLrvog4r+2+Npfx+yNi84DqXRYR342Ie8vymRGxp2zzjnLSnIg4uSxPlPWjbfdxbel/MiLe2XB9KyLiroh4oszpWxfjXEbEH5fn+9GI+FJEnDLsuYyIz0XE4Yh4tK2vsbmLiN+IiEfKbW6KiGiwzk+V53xfRNwdESva1nWco9le97M9D/3W2LbuTyMiI2J1WR7aXM5LZi65C62TvE8BZwEnAQ8DZy/g9tcA55X264Hv0/q6h08C20r/NuD60r4U+CoQwEZgT+lfBRwo1ytLe+UA6v0Y8EXg3rJ8J3BlaX8a+MPS/hDw6dK+ErijtM8uc3wycGaZ+2UN1rcT+GBpnwSsWGxzSeuDgD8AXt02h38w7LkEfgc4D3i0ra+xuQO+Bby13OarwCUN1vkOYHlpX99WZ8c54gSv+9meh35rLP3rab255IfA6mHP5bzmf6E21GjRrcn6WtvytcC1Q6znHuD3aH3oa03pWwM8WdqfAa5qG/9kWX8V8Jm2/mPGNVTbOmA3cCFwb/khe77tBXZkLssP81tLe3kZFzPnt31cA/X9Mq0AjRn9i2ouOfqJ71Vlbu4F3rkY5hIY5dgAbWTuyron2vqPGddvnTPW/T5wW2l3nCNmed2f6Ge6iRqBu4C3AAc5GvRDncu5XpbqoZtOX7OwdhiFlD/JzwX2AGdk5rMA5fr0Mmy2ehficdwIfBz4eVk+DXgpM1/psM0j9ZT1L5fxg6zzLGAK+Hy0Di99NiJeyyKby8z8EfA3wNPAs7TmZi+Lay6nNTV3a0t7kLVO+wCtvdz51Hmin+m+RMS7gR9l5sMzVi3muTzOUg36Tse2FvztQxHxOuDLwEcz88cnGtqhL0/Q34iIeBdwODP39lDLidYNss7ltP5cvjkzzwX+h9bhhtkMay5X0vpyvjOBXwVeS+sbWmfb5lDq7GKuNS1IrRFxHfAKcNt01xzrGUidEfEa4DrgLzqtnmMtQ82spRr0PX3NwiBFxKtohfxtmfmV0v1cRKwp69cAh0v/bPUO+nFcALw7Ig4Ct9M6fHMjsCIipj9D0b7NI/WU9acCLwy4zklgMjP3lOW7aAX/YpvLtwM/yMypzPwZ8BXgt1hcczmtqbmbLO2B1VpOVr4LeG+WYxrzqPN5Zn8e+vHrtH6xP1xeQ+uA70TEr8yjxoHP5Qkt1DGiJi+09gIP0HoSpk/KvHkBtx/ArcCNM/o/xbEnwT5Z2pdx7Imbb5X+VbSOT68slx8AqwZU89s4ejL2nzj2xNWHSvsajj2BeGdpv5ljT44doNmTsf8OvLG0/6rM46KaS1rfwPoY8Jqy7Z3AhxfDXHL8MfrG5o7WV5ps5OgJxEsbrPNiWl9fPjJjXMc54gSv+9meh35rnLHuIEeP0Q91Luf8uBZqQ40X3jrr/X1aZ+GvW+Bt/zatP7v2AQ+Vy6W0jhXuBvaX6+knOGj9M5angEeAsbb7+gAwUS5XD7Dmt3E06M+i9Q6AifICObn0n1KWJ8r6s9puf12p/0kafrcAcA4wXubzn8sLZNHNJfAJ4AngUeALJYiGOpfAl2idM/gZrb3GLU3OHTBWHu9TwN8z46R5n3VO0DqePf0a+nS3OWKW1/1sz0O/Nc5Yf5CjQT+0uZzPxU/GSlLlluoxeklSjwx6SaqcQS9JlTPoJalyBr0kVc6gl6TKGfSSVDmDXpIq9/8TDwN1w6/lGQAAAABJRU5ErkJggg==\n"
                    },
                    "metadata": {
                        "needs_background": "light"
                    }
                }
            ]
        },
        {
            "metadata": {},
            "cell_type": "code",
            "source": "# we have 26595 unique words in our vocabulary\nlen(cnt)",
            "execution_count": 12,
            "outputs": [
                {
                    "output_type": "execute_result",
                    "execution_count": 12,
                    "data": {
                        "text/plain": "26595"
                    },
                    "metadata": {}
                }
            ]
        },
        {
            "metadata": {},
            "cell_type": "code",
            "source": "cnt.most_common()[-5:-1]",
            "execution_count": 22,
            "outputs": [
                {
                    "output_type": "execute_result",
                    "execution_count": 22,
                    "data": {
                        "text/plain": "[('crosses', 1), ('biology', 1), ('growsandstates', 1), ('indianas', 1)]"
                    },
                    "metadata": {}
                }
            ]
        },
        {
            "metadata": {},
            "cell_type": "code",
            "source": "# we have 11,910 words that only appear once in any article.\n# 26,595 - 11,910 = 14685\ncntd = dict(cnt)\nsort_orders = sorted(cntd.items(), key=lambda x: x[1], reverse=False)\nsingles = []\nfor i in sort_orders:\n    if i[1] ==1:\n        singles.append(i[0])\nlen(singles)",
            "execution_count": 23,
            "outputs": [
                {
                    "output_type": "execute_result",
                    "execution_count": 23,
                    "data": {
                        "text/plain": "11910"
                    },
                    "metadata": {}
                }
            ]
        },
        {
            "metadata": {},
            "cell_type": "markdown",
            "source": "### Vocabulary Size\nOf the 26,595 words in our articles, we have 11,910 words used once, leaving 14,685 words found in more than one article.  Any analysis that depends on finding the same word multiple articles will not find any of these 11,910 words, so from that standpoint they are just noise.  However, embeddings that rely on analysis broader than word or ngram repetition, such as sentence structure, or by inferring parts of speech, will likely find meaning in words that appear once, so we will keep or remove them based on the vectorization we are using. "
        },
        {
            "metadata": {},
            "cell_type": "markdown",
            "source": "#### (Below is just kept as a note to myself on how to find a record based on its key value.)"
        },
        {
            "metadata": {},
            "cell_type": "code",
            "source": "i = dfNews.index.get_loc('biz01legit')\nprint (i)\n#dfNews.iloc[i:i+2]\ndfNews['text'][159]\n\n",
            "execution_count": 15,
            "outputs": [
                {
                    "output_type": "stream",
                    "text": "159\n",
                    "name": "stdout"
                },
                {
                    "output_type": "execute_result",
                    "execution_count": 15,
                    "data": {
                        "text/plain": "'Alex Jones Apologizes for Promoting \\'Pizzagate\\' Hoax  Alex Jones  a prominent conspiracy theorist and the host of a popular right-wing radio show  has apologized for helping to spread and promote the hoax known as Pizzagate. The admission on Friday by Mr. Jones  the host of \"The Alex Jones Show\" and the operator of the website Infowars  was striking. In addition to promoting the Pizzagate conspiracy theory  he has contended that the Sept. 11 attacks were inside jobs carried out by the United States government and that the 2012 shooting at Sandy Hook Elementary School in Newtown  Conn.  was a hoax concocted by those hostile to the Second Amendment. The Pizzagate theory  which posited with no evidence that top Democratic officials were involved with a satanic child pornography ring centered around Comet Ping Pong  a pizza restaurant in Washington  D.C.  grew in online forums before making its way to more visible venues  including Mr. Jones\\'s show. And its prominence after the election drew attention to the proliferation of false and misleading news  much of it politically charged  that circulated on platforms like Facebook  Twitter and YouTube.'"
                    },
                    "metadata": {}
                }
            ]
        }
    ],
    "metadata": {
        "kernelspec": {
            "name": "python3",
            "display_name": "Python 3.6",
            "language": "python"
        },
        "language_info": {
            "name": "python",
            "version": "3.6.9",
            "mimetype": "text/x-python",
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "pygments_lexer": "ipython3",
            "nbconvert_exporter": "python",
            "file_extension": ".py"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 1
}