experiments/text_generation_wikipedia_rnn/text_generation_wikipedia_rnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JL3ZPJPgC4wC"
   },
   "source": [
    "# Wikipedia Text Generation (using RNN LSTM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eoWGtkL8PPJ3"
   },
   "source": [
    "> - 🤖 See [full list of Machine Learning Experiments](https://github.com/trekhleb/machine-learning-experiments) on **GitHub**<br/><br/>\n",
    "> - ▶️ **Interactive Demo**: [try this model and other machine learning experiments in action](https://trekhleb.github.io/machine-learning-experiments/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bui0MyTjv1Mp"
   },
   "source": [
    "## Experiment overview\n",
    "\n",
    "In this experiment we will use character-based [Recurrent Neural Network](https://en.wikipedia.org/wiki/Recurrent_neural_network) (RNN) to generate a Wikipedia-like text based on the [wikipedia](https://www.tensorflow.org/datasets/catalog/wikipedia) TensorFlow dataset.\n",
    "\n",
    "![text_generation_wikipedia_rnn.png](../../demos/src/images/text_generation_wikipedia_rnn.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3XxEZuRNIzHH"
   },
   "source": [
    "_Inspired by [Text generation with an RNN](https://www.tensorflow.org/tutorials/text/text_generation)_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IDJctDhhEDTD"
   },
   "source": [
    "## Import dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Z2xZnxncD2Ep"
   },
   "outputs": [],
   "source": [
    "# Selecting Tensorflow version v2 (the command is relevant for Colab only).\n",
    "# %tensorflow_version 2.x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3119,
     "status": "ok",
     "timestamp": 1586973888852,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "SpueB6zADYgE",
    "outputId": "18c59941-ac1f-4d02-c824-8499f71a8a70"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python version: 3.7.6\n",
      "Tensorflow version: 2.1.0\n",
      "Keras version: 2.2.4-tf\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow_datasets as tfds\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import platform\n",
    "import time\n",
    "import pathlib\n",
    "import os\n",
    "\n",
    "print('Python version:', platform.python_version())\n",
    "print('Tensorflow version:', tf.__version__)\n",
    "print('Keras version:', tf.keras.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ciI_JnnNEGCw"
   },
   "source": [
    "## Download the dataset\n",
    "\n",
    "[Wikipedia](https://www.tensorflow.org/datasets/catalog/wikipedia) dataset contains cleaned articles of all languages. The datasets are built from the [Wikipedia dump](https://dumps.wikimedia.org/) with one split per language. Each example contains the content of one full Wikipedia article with cleaning to strip markdown and unwanted sections (references, etc.)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3103,
     "status": "ok",
     "timestamp": 1586973888853,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "q3PDebo50-Le",
    "outputId": "7ca9b4e2-97fb-439d-8a53-dda4e9204275"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['abstract_reasoning',\n",
       " 'aeslc',\n",
       " 'aflw2k3d',\n",
       " 'amazon_us_reviews',\n",
       " 'arc',\n",
       " 'bair_robot_pushing_small',\n",
       " 'big_patent',\n",
       " 'bigearthnet',\n",
       " 'billsum',\n",
       " 'binarized_mnist',\n",
       " 'binary_alpha_digits',\n",
       " 'c4',\n",
       " 'caltech101',\n",
       " 'caltech_birds2010',\n",
       " 'caltech_birds2011',\n",
       " 'cars196',\n",
       " 'cassava',\n",
       " 'cats_vs_dogs',\n",
       " 'celeb_a',\n",
       " 'celeb_a_hq',\n",
       " 'chexpert',\n",
       " 'cifar10',\n",
       " 'cifar100',\n",
       " 'cifar10_1',\n",
       " 'cifar10_corrupted',\n",
       " 'citrus_leaves',\n",
       " 'cityscapes',\n",
       " 'civil_comments',\n",
       " 'clevr',\n",
       " 'cmaterdb',\n",
       " 'cnn_dailymail',\n",
       " 'coco',\n",
       " 'coil100',\n",
       " 'colorectal_histology',\n",
       " 'colorectal_histology_large',\n",
       " 'cos_e',\n",
       " 'curated_breast_imaging_ddsm',\n",
       " 'cycle_gan',\n",
       " 'deep_weeds',\n",
       " 'definite_pronoun_resolution',\n",
       " 'diabetic_retinopathy_detection',\n",
       " 'dmlab',\n",
       " 'downsampled_imagenet',\n",
       " 'dsprites',\n",
       " 'dtd',\n",
       " 'duke_ultrasound',\n",
       " 'dummy_dataset_shared_generator',\n",
       " 'dummy_mnist',\n",
       " 'emnist',\n",
       " 'esnli',\n",
       " 'eurosat',\n",
       " 'fashion_mnist',\n",
       " 'flic',\n",
       " 'flores',\n",
       " 'food101',\n",
       " 'gap',\n",
       " 'gigaword',\n",
       " 'glue',\n",
       " 'groove',\n",
       " 'higgs',\n",
       " 'horses_or_humans',\n",
       " 'i_naturalist2017',\n",
       " 'image_label_folder',\n",
       " 'imagenet2012',\n",
       " 'imagenet2012_corrupted',\n",
       " 'imagenet_resized',\n",
       " 'imagenette',\n",
       " 'imdb_reviews',\n",
       " 'iris',\n",
       " 'kitti',\n",
       " 'kmnist',\n",
       " 'lfw',\n",
       " 'lm1b',\n",
       " 'lost_and_found',\n",
       " 'lsun',\n",
       " 'malaria',\n",
       " 'math_dataset',\n",
       " 'mnist',\n",
       " 'mnist_corrupted',\n",
       " 'movie_rationales',\n",
       " 'moving_mnist',\n",
       " 'multi_news',\n",
       " 'multi_nli',\n",
       " 'multi_nli_mismatch',\n",
       " 'newsroom',\n",
       " 'nsynth',\n",
       " 'omniglot',\n",
       " 'open_images_v4',\n",
       " 'oxford_flowers102',\n",
       " 'oxford_iiit_pet',\n",
       " 'para_crawl',\n",
       " 'patch_camelyon',\n",
       " 'pet_finder',\n",
       " 'places365_small',\n",
       " 'plant_leaves',\n",
       " 'plant_village',\n",
       " 'plantae_k',\n",
       " 'quickdraw_bitmap',\n",
       " 'reddit_tifu',\n",
       " 'resisc45',\n",
       " 'rock_paper_scissors',\n",
       " 'rock_you',\n",
       " 'scan',\n",
       " 'scene_parse150',\n",
       " 'scicite',\n",
       " 'scientific_papers',\n",
       " 'shapes3d',\n",
       " 'smallnorb',\n",
       " 'snli',\n",
       " 'so2sat',\n",
       " 'squad',\n",
       " 'stanford_dogs',\n",
       " 'stanford_online_products',\n",
       " 'starcraft_video',\n",
       " 'sun397',\n",
       " 'super_glue',\n",
       " 'svhn_cropped',\n",
       " 'ted_hrlr_translate',\n",
       " 'ted_multi_translate',\n",
       " 'tf_flowers',\n",
       " 'the300w_lp',\n",
       " 'titanic',\n",
       " 'trivia_qa',\n",
       " 'uc_merced',\n",
       " 'ucf101',\n",
       " 'vgg_face2',\n",
       " 'visual_domain_decathlon',\n",
       " 'voc',\n",
       " 'wider_face',\n",
       " 'wikihow',\n",
       " 'wikipedia',\n",
       " 'wmt14_translate',\n",
       " 'wmt15_translate',\n",
       " 'wmt16_translate',\n",
       " 'wmt17_translate',\n",
       " 'wmt18_translate',\n",
       " 'wmt19_translate',\n",
       " 'wmt_t2t_translate',\n",
       " 'wmt_translate',\n",
       " 'xnli',\n",
       " 'xsum']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# List all available datasets to see how the wikipedia dataset is called.\n",
    "tfds.list_builders()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "f2qtWwKA0-Li"
   },
   "source": [
    "[`tfds.load`](https://www.tensorflow.org/datasets/api_docs/python/tfds/load) is a convenience method that's the simplest way to build and load a [`tf.data.Dataset`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gsqkLYgZ0-Lj"
   },
   "outputs": [],
   "source": [
    "# Loading the wikipedia dataset.\n",
    "DATASET_NAME = 'wikipedia/20190301.en'\n",
    "# DATASET_NAME = 'wikipedia/20190301.uk'\n",
    "\n",
    "dataset, dataset_info = tfds.load(\n",
    "    name=DATASET_NAME,\n",
    "    data_dir='tmp',\n",
    "    with_info=True,\n",
    "    split=tfds.Split.TRAIN,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 496
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3058,
     "status": "ok",
     "timestamp": 1586973888854,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "V5nkTrRZ0-Ln",
    "outputId": "4f3b0eb0-84ef-4fb5-9087-72c096119542"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tfds.core.DatasetInfo(\n",
      "    name='wikipedia',\n",
      "    version=1.0.0,\n",
      "    description='Wikipedia dataset containing cleaned articles of all languages.\n",
      "The datasets are built from the Wikipedia dump\n",
      "(https://dumps.wikimedia.org/) with one split per language. Each example\n",
      "contains the content of one full Wikipedia article with cleaning to strip\n",
      "markdown and unwanted sections (references, etc.).\n",
      "',\n",
      "    homepage='https://dumps.wikimedia.org',\n",
      "    features=FeaturesDict({\n",
      "        'text': Text(shape=(), dtype=tf.string),\n",
      "        'title': Text(shape=(), dtype=tf.string),\n",
      "    }),\n",
      "    total_num_examples=5824596,\n",
      "    splits={\n",
      "        'train': 5824596,\n",
      "    },\n",
      "    supervised_keys=None,\n",
      "    citation=\"\"\"@ONLINE {wikidump,\n",
      "        author = \"Wikimedia Foundation\",\n",
      "        title  = \"Wikimedia Downloads\",\n",
      "        url    = \"https://dumps.wikimedia.org\"\n",
      "    }\"\"\",\n",
      "    redistribution_info=license: \"This work is licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.\",\n",
      ")\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(dataset_info)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3033,
     "status": "ok",
     "timestamp": 1586973888855,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "9A11MeAt0-Lr",
    "outputId": "749bc519-63ad-47af-ea41-7d6a83014172"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<DatasetV1Adapter shapes: {text: (), title: ()}, types: {text: tf.string, title: tf.string}>\n"
     ]
    }
   ],
   "source": [
    "print(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eKTy6YS5Gx-g"
   },
   "source": [
    "## Analyze the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3010,
     "status": "ok",
     "timestamp": 1586973888856,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "ldLeTmcM0-Lu",
    "outputId": "b36746a7-6016-44ca-ed40-d0fd36087e21"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of articles:  5824596\n"
     ]
    }
   ],
   "source": [
    "TRAIN_NUM_EXAMPLES = dataset_info.splits['train'].num_examples\n",
    "print('Total number of articles: ', TRAIN_NUM_EXAMPLES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3178,
     "status": "ok",
     "timestamp": 1586973889154,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "CHkz0ZHA0-Lz",
    "outputId": "16739986-fe0e-41d8-c57d-12abe90df9dd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First article \n",
      "======\n",
      "\n",
      "Title: \n",
      "------\n",
      "Joseph Greenberg\n",
      "\n",
      "Text: \n",
      "------\n",
      "Joseph Harold Greenberg (May 28, 1915 – May 7, 2001) was an American linguist, known mainly for his work concerning linguistic typology and the genetic classification of languages.\n",
      "\n",
      "Life\n",
      "\n",
      "Early life and education \n",
      "(Main source: Croft 2003)\n",
      "\n",
      "Joseph Greenberg was born on May 28, 1915 to Jewish parents in Brooklyn, New York. His first great interest was music. At the age of 14, he gave a piano concert in Steinway Hall. He continued to play the piano frequently throughout his life.\n",
      "\n",
      "After finishing high school, he decided to pursue a scholarly career rather than a musical one. He enrolled at Columbia University in New York. During his senior year, he attended a class taught by Franz Boas concerning American Indian languages. With references from Boas and Ruth Benedict, he was accepted as a graduate student by Melville J. Herskovits at Northwestern University in Chicago. During the course of his graduate studies, Greenberg did fieldwork among the Hausa people of Nigeria, where he learned the Hausa language. The subject of his doctoral dissertation was the influence of Islam on a Hausa group that, unlike most others, had not converted to it.\n",
      "\n",
      "During 1940, he began postdoctoral studies at Yale University. These were interrupted by service in the U.S. Army Signal Corps during World War II, for which he worked as a codebreaker and participated with the landing at Casablanca. Before leaving for Europe during 1943, Greenberg married Selma Berkowitz, whom he had met during his first year at Columbia University.\n",
      "\n",
      "Career\n",
      "After the war, Greenberg taught at the University of Minnesota before returning to Columbia University during 1948 as a teacher of anthropology. While in New York, he became acquainted with Roman Jakobson and André Martinet. They introduced him to the Prague school of structuralism, which influenced his work.\n",
      "\n",
      "During 1962, Greenberg relocated to the anthropology department of Stanford University in California, where he continued to work for the rest of his life. During 1965 Greenberg served as president of the African Studies Association. He received during 1996 the highest award for a scholar in Linguistics, the Gold Medal of Philology (http://insop.org/index.php?p=1_8_Ancient-Medal-Winners.).\n",
      "\n",
      "Contributions to linguistics\n",
      "\n",
      "Linguistic typology \n",
      "\n",
      "Greenberg's reputation rests partly on his contributions to synchronic linguistics and the quest to identify linguistic universals. During the late 1950s, Greenberg began to examine languages covering a wide geographic and genetic distribution. He located a number of interesting potential universals as well as many strong cross-linguistic tendencies.\n",
      "\n",
      "In particular, Greenberg conceptualized the idea of \"implicational universal\", which has the form, \"if a language has structure X, then it must also have structure Y.\" For example, X might be \"mid front rounded vowels\" and Y \"high front rounded vowels\" (for terminology see phonetics). Many scholars adopted this kind of research following Greenberg's example and it remains important in synchronic linguistics.\n",
      "\n",
      "Like Noam Chomsky, Greenberg sought to discover the universal structures on which human language is based. Unlike Chomsky, Greenberg's method was functionalist, rather than formalist. An argument to reconcile the Greenbergian and Chomskyan methods can be found in Linguistic Universals (2006), edited by Ricardo Mairal and Juana Gil .\n",
      "\n",
      "Many who are strongly opposed to Greenberg's methods of language classification (see below) acknowledge the importance of his typological work. During 1963 he published an article that was extremely influential: \"Some universals of grammar with particular reference to the order of meaningful elements\".\n",
      "\n",
      "Mass comparison \n",
      "\n",
      "Greenberg rejected the opinion, prevalent among linguists since the mid-20th century, that comparative reconstruction was the only method to discover relationships between languages. He argued that genetic classification is methodologically prior to comparative reconstruction, or the first stage of it: you cannot engage in the comparative reconstruction of languages until you know which languages to compare (1957:44).\n",
      "\n",
      "He also criticized the prevalent opinion that comprehensive comparisons of two languages at a time (which commonly take years to perform) could establish language families of any size. He argued that, even for 8 languages, there are already 4,140 ways to classify them into distinct families, while for 25 languages there are 4,749,027,089,305,918,018 ways (1957:44). For comparison, the Niger–Congo family is said to have some 1,500 languages. He thought language families of any size needed to be established by some scholastic means other than bilateral comparison. The theory of mass comparison is an attempt to demonstrate such means.\n",
      "\n",
      "Greenberg argued for the virtues of breadth over depth. He advocated restricting the amount of material to be compared (to basic vocabulary, morphology, and known paths of sound change) and increasing the number of languages to be compared to all the languages in a given area. This would make it possible to compare numerous languages reliably. At the same time, the process would provide a check on accidental resemblances through the sheer number of languages under review. The mathematical probability that resemblances are accidental decreases strongly with the number of languages concerned (1957:39).\n",
      "\n",
      "Greenberg used the premise that mass \"borrowing\" of basic vocabulary is unknown. He argued that borrowing, when it occurs, is concentrated in cultural vocabulary and clusters \"in certain semantic areas\", making it easy to detect (1957:39). With the goal of determining broad patterns of relationship, the idea was not to get every word right but to detect patterns. From the beginning with his theory of mass comparison, Greenberg addressed why chance resemblance and borrowing were not obstacles to its being useful. Despite that, critics consider those phenomena caused difficulties for his theory.\n",
      "\n",
      "Greenberg first termed his method \"mass comparison\" in an article of 1954 (reprinted in Greenberg 1955). As of 1987, he replaced the term \"mass comparison\" with \"multilateral comparison\", to emphasize its contrast with the bilateral comparisons recommended by linguistics textbooks. He believed that multilateral comparison was not in any way opposed to the comparative method, but is, on the contrary, its necessary first step (Greenberg, 1957:44). According to him, comparative reconstruction should have the status of an explanatory theory for facts already established by language classification (Greenberg, 1957:45).\n",
      "\n",
      "Most historical linguists (Campbell 2001:45) reject the use of mass comparison as a method for establishing genealogical relationships between languages. Among the most outspoken critics of mass comparison have been Lyle Campbell, Donald Ringe, William Poser, and the late R. Larry Trask.\n",
      "\n",
      "Genetic classification of languages\n",
      "\n",
      "The languages of Africa \n",
      "\n",
      "Greenberg is known widely for his development of a classification system for the languages of Africa, which he published as a series of articles in the Southwestern Journal of Anthropology from 1949 to 1954 (reprinted together as a book during 1955). He revised the book and published it again during 1963, followed by a nearly identical edition of 1966 (reprinted without change during 1970). A few more changes of the classification were made by Greenberg in an article during 1981.\n",
      "\n",
      "Greenberg grouped the hundreds of African languages into four families, which he dubbed Afroasiatic, Nilo-Saharan, Niger–Congo, and Khoisan. During the course of his work, Greenberg invented the term \"Afroasiatic\" to replace the earlier term \"Hamito-Semitic\", after showing that the Hamitic group, accepted widely since the 19th century, is not a valid language family. Another major feature of his work was to establish the classification of the Bantu languages, which occupy much of sub-Saharan Africa, as a part of the Niger–Congo language family, rather than as an independent family as many Bantuists had maintained.\n",
      "\n",
      "Greenberg's classification rested largely in evaluating competing earlier classifications. For a time, his classification was considered bold and speculative, especially the proposal of a Nilo-Saharan language family. Now, apart from Khoisan, it is generally accepted by African specialists and has been used as a basis for further work by other scholars.\n",
      "\n",
      "Greenberg's work on African languages has been criticised by Lyle Campbell and Donald Ringe, who do not believe that his classification is justified by his data; they request a reexamination of his macro-phyla by \"reliable methods\" (Ringe 1993:104). Harold Fleming and Lionel Bender, who are sympathetic to Greenberg's classification, acknowledge that at least some of his macrofamilies (particularly Nilo-Saharan and Khoisan) are not accepted completely by most linguists and may need to be divided (Campbell 1997). Their objection is methodological: if mass comparison is not a valid method, it cannot be expected to have brought order successfully out of the confusion of African languages.\n",
      "\n",
      "By contrast, some linguists have sought to combine Greenberg's four African families into larger units. In particular, Edgar Gregersen (1972) proposed joining Niger–Congo and Nilo-Saharan into a larger family, which he termed Kongo-Saharan. Roger Blench (1995) suggests Niger–Congo is a subfamily of Nilo-Saharan.\n",
      "\n",
      "The languages of New Guinea, Tasmania, and the Andaman Islands\n",
      "\n",
      "During 1971 Greenberg proposed the Indo-Pacific macrofamily, which groups together the Papuan languages (a large number of language families of New Guinea and nearby islands) with the native languages of the Andaman Islands and Tasmania but excludes the Australian Aboriginal languages. Its principal feature was to reduce the manifold language families of New Guinea to a single genetic unit. This excludes the Austronesian languages, which have been established as associated with a more recent migration of people.\n",
      "\n",
      "Greenberg's subgrouping of these languages has not been accepted by the few specialists who have worked on the classification of these languages. However, the work of Stephen Wurm (1982) and Malcolm Ross (2005) has provided considerable evidence for his once-radical idea that these languages form a single genetic unit. Wurm stated that the lexical similarities between Great Andamanese and the West Papuan and Timor–Alor families \"are quite striking and amount to virtual formal identity [...] in a number of instances.\" He believes this to be due to a linguistic substratum.\n",
      "\n",
      "The languages of the Americas\n",
      "\n",
      "Most linguists concerned with the native languages of the Americas classify them into 150 to 180 independent language families. Some believe that two language families, Eskimo–Aleut and Na-Dené, were distinct, perhaps the results of later migrations into the New World.\n",
      "\n",
      "Early on, Greenberg (1957:41, 1960) became convinced that many of the language groups considered unrelated could be classified into larger groupings. In his 1987 book Language in the Americas, while agreeing that the Eskimo–Aleut and Na-Dené groupings as distinct, he proposed that all the other Native American languages belong to a single language macro-family, which he termed Amerind.\n",
      "\n",
      "Language in the Americas has generated lively debate, but has been criticized strongly; it is rejected by most specialists of indigenous languages of the Americas and also by most historical linguists. Specialists of the individual language families have found extensive inaccuracies and errors in Greenberg's data, such as including data from non-existent languages, erroneous transcriptions of the forms compared, misinterpretations of the meanings of words used for comparison, and entirely spurious forms.\n",
      "\n",
      "Historical linguists also reject the validity of the method of multilateral (or mass) comparison upon which the classification is based. They argue that he has not provided a convincing case that the similarities presented as evidence are due to inheritance from an earlier common ancestor rather than being explained by a combination of errors, accidental similarity, excessive semantic latitude in comparisons, borrowings, onomatopoeia, etc.\n",
      "\n",
      "The languages of northern Eurasia \n",
      "\n",
      "Later in his life, Greenberg proposed that nearly all of the language families of northern Eurasia belong to a single higher-order family, which he termed Eurasiatic. The only exception was Yeniseian, which has been related to a wider Dené–Caucasian grouping, also including Sino-Tibetan.  During 2008 Edward Vajda related Yeniseian to the Na-Dené languages of North America as a Dené–Yeniseian family.\n",
      "\n",
      "The Eurasiatic grouping resembles the older Nostratic groupings of Holger Pedersen and Vladislav Illich-Svitych by including Indo-European, Uralic, and Altaic. It differs by including Nivkh, Japonic, Korean, and Ainu (which the Nostraticists had excluded from comparison because they are single languages rather than language families) and in excluding Afroasiatic. At about this time, Russian Nostraticists, notably Sergei Starostin, constructed a revised version of Nostratic. It was slightly larger than Greenberg's grouping but it also excluded Afroasiatic.\n",
      "\n",
      "Recently, a consensus has been emerging among proponents of the Nostratic hypothesis. Greenberg basically agreed with the Nostratic concept, though he stressed a deep internal division between its northern 'tier' (his Eurasiatic) and a southern 'tier' (principally Afroasiatic and Dravidian).\n",
      "\n",
      "The American Nostraticist Allan Bomhard considers Eurasiatic a branch of Nostratic, alongside other branches: Afroasiatic, Elamo-Dravidian, and Kartvelian. Similarly, Georgiy Starostin (2002) arrives at a tripartite overall grouping: he considers Afroasiatic, Nostratic and Elamite to be roughly equidistant and more closely related to each other than to any other language family. Sergei Starostin's school has now included Afroasiatic in a broadly defined Nostratic. They reserve the term Eurasiatic to designate the narrower subgrouping, which comprises the rest of the macrofamily. Recent proposals thus differ mainly on the precise inclusion of Dravidian and Kartvelian.\n",
      "\n",
      "Greenberg continued to work on this project after he was diagnosed with incurable pancreatic cancer and until he died during May 2001. His colleague and former student Merritt Ruhlen ensured the publication of the final volume of his Eurasiatic work (2002) after his death.\n",
      "\n",
      "Selected works by Joseph H. Greenberg\n",
      "\n",
      "Books \n",
      "  (Photo-offset reprint of the SJA articles with minor corrections.)\n",
      " \n",
      "  (Heavily revised version of Greenberg 1955. From the same publisher: second, revised edition, 1966; third edition, 1970. All three editions simultaneously published at The Hague by Mouton & Co.)\n",
      "  (Reprinted 1980 and, with a foreword by Martin Haspelmath, 2005.)\n",
      "\n",
      "Books (editor) \n",
      "  (Second edition 1966.)\n",
      "\n",
      "Articles, reviews, etc. \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      "  (Reprinted in Genetic Linguistics, 2005.)\n",
      " \n",
      "  (In second edition of Universals of Language, 1966: pp. 73–113.)\n",
      " \n",
      " \n",
      "  (Reprinted in Genetic Linguistics, 2005.)\n",
      "\n",
      "Bibliography\n",
      "\n",
      "Blench, Roger. 1995. \"Is Niger–Congo simply a branch of Nilo-Saharan?\" In Fifth Nilo-Saharan Linguistics Colloquium, Nice, 24–29 August 1992: Proceedings, edited by Robert Nicolaï and Franz Rottland. Cologne: Köppe Verlag, pp. 36–49.\n",
      "\n",
      "Campbell, Lyle. 1997. American Indian Languages: The Historical Linguistics of Native America. New York: Oxford University Press. .\n",
      "Campbell, Lyle. 2001. \"Beyond the comparative method.\" In Historical Linguistics 2001: Selected Papers from the 15th International Conference on Historical Linguistics, Melbourne, 13–17 August 2001, edited by Barry J. Blake, Kate Burridge, and Jo Taylor.\n",
      "Diamond, Jared. 1997. Guns, Germs and Steel: The Fates of Human Societies. New York: Norton. .\n",
      "\n",
      "Mairal, Ricardo and Juana Gil. 2006. Linguistic Universals. Cambridge–NY: Cambridge University Press. .\n",
      "\n",
      "Ross, Malcolm. 2005. \"Pronouns as a preliminary diagnostic for grouping Papuan languages.\" In Papuan Pasts: Cultural, Linguistic and Biological Histories of Papuan-speaking Peoples, edited by Andrew Pawley, Robert Attenborough, Robin Hide, and Jack Golson. Canberra: Pacific Linguistics, pp. 15–66.\n",
      "Wurm, Stephen A. 1982. The Papuan Languages of Oceania. Tübingen: Gunter Narr.\n",
      "\n",
      "See also\n",
      "\n",
      "Linguistic universal\n",
      "Monogenesis (linguistics)\n",
      "Nostratic languages\n",
      "\n",
      "References\n",
      "\n",
      "External links\n",
      "Joseph Greenberg at work; a portrait of himself\n",
      "\"What we all spoke when the world was young\" by Nicholas Wade, New York Times (February 1, 2000)\n",
      "Obituary from Stanford Report\n",
      "Memorial Resolution\n",
      "\"Joseph Harold Greenberg\" by William Croft (2003) (also: )\n",
      "\"Complete bibliography of the publications of Joseph H. Greenberg\" by William Croft (2003)\n",
      "\n",
      "Category:1915 births\n",
      "Category:2001 deaths\n",
      "Category:Linguists from the United States\n",
      "Category:American Africanists\n",
      "Category:American Jews in the military\n",
      "Category:Paleolinguists\n",
      "Category:Columbia University faculty\n",
      "Category:Stanford University Department of Anthropology faculty\n",
      "Category:Fellows of the American Academy of Arts and Sciences\n",
      "Category:Members of the United States National Academy of Sciences\n",
      "Category:United States Army personnel\n",
      "Category:American army personnel of World War II\n",
      "Category:Guggenheim Fellows\n",
      "Category:American expatriates in Nigeria\n",
      "Category:People from Brooklyn\n",
      "Category:Linguists of Na-Dene languages\n",
      "Category:Linguists of Eskimo–Aleut languages\n",
      "Category:Linguists of Hokan languages\n",
      "Category:Jewish scientists\n",
      "Category:Linguists of Papuan languages\n",
      "Category:Linguists of Amerind languages\n",
      "Category:Linguists of Andamanese languages\n",
      "Category:Linguists of Tasmanian languages\n",
      "Category:Linguists of Niger–Congo languages\n",
      "Category:Linguists of Afroasiatic languages\n",
      "Category:Linguistic Society of America presidents\n"
     ]
    }
   ],
   "source": [
    "print('First article','\\n======\\n')\n",
    "for example in dataset.take(1):\n",
    "    print('Title:','\\n------')\n",
    "    print(example['title'].numpy().decode('utf-8'))\n",
    "    print()\n",
    "\n",
    "    print('Text:', '\\n------')\n",
    "    print(example['text'].numpy().decode('utf-8'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GqpuKh9HMNnf"
   },
   "source": [
    "## Process the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FK_FFy7P0-L3"
   },
   "source": [
    "### Flatten the dataset\n",
    "\n",
    "Converting the dataset from the set of articles into the set of characters. We also are interested only in `text` of each article so we may drop the `title` along the way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3071,
     "status": "ok",
     "timestamp": 1586973889155,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "_AC6MHFC0-L3",
    "outputId": "22f5d674-3b5f-44b7-9b02-0e61855a4871"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[b'J' b'o' b's' ... b'n' b't' b's']\n",
      "\n",
      "\n",
      "[b'P' b'a' b'u' ... b'e' b'r' b's']\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def article_to_text(text):\n",
    "    return np.array([char for char in text.numpy().decode('utf-8')])\n",
    "\n",
    "# Converting each dataset item to a string ('text') instead of a dictionary ({'text', 'title'}).\n",
    "dataset_text = dataset.map(\n",
    "    lambda article: tf.py_function(func=article_to_text, inp=[article['text']], Tout=tf.string)\n",
    ")\n",
    "\n",
    "for text in dataset_text.take(2):\n",
    "    print(text.numpy())\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 357
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3964,
     "status": "ok",
     "timestamp": 1586973890071,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "oSsSbJbX0-L8",
    "outputId": "9e442c58-c3da-4ad8-d4c3-f297085ea312"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "J\n",
      "o\n",
      "s\n",
      "e\n",
      "p\n",
      "h\n",
      " \n",
      "H\n",
      "a\n",
      "r\n",
      "o\n",
      "l\n",
      "d\n",
      " \n",
      "G\n",
      "r\n",
      "e\n",
      "e\n",
      "n\n",
      "b\n"
     ]
    }
   ],
   "source": [
    "# Unbatch the text dataset into a more granular char dataset.\n",
    "# Now each dataset item is one character instead of a big piece of text.\n",
    "dataset_chars = dataset_text.unbatch()\n",
    "\n",
    "for char in dataset_chars.take(20):\n",
    "    print(char.numpy().decode('utf-8'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jHNkkXGz0-L_"
   },
   "source": [
    "### Generating vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 88
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 270306,
     "status": "ok",
     "timestamp": 1586974156434,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "B7rnYLeU0-MA",
    "outputId": "4312adbb-7dd5-44cc-9e4b-4a68efad7883"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unique characters: 621\n",
      "vocab:\n",
      "['\\t', '\\n', ' ', '!', '\"', '#', '$', '%', '&', \"'\", '(', ')', '*', '+', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '<', '=', '>', '?', '@', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', '[', ']', '^', '_', '`', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '{', '|', '}', '~', '\\xa0', '£', '§', '«', '®', '°', '±', '²', '·', '»', '¼', '½', '¿', 'Á', 'Å', 'Æ', 'Ç', 'É', 'Ë', 'Í', 'Î', 'Ó', 'Ö', '×', 'Ø', 'Ü', 'Þ', 'ß', 'à', 'á', 'â', 'ã', 'ä', 'å', 'æ', 'ç', 'è', 'é', 'ê', 'ë', 'ì', 'í', 'î', 'ï', 'ñ', 'ò', 'ó', 'ô', 'õ', 'ö', 'ø', 'ú', 'û', 'ü', 'ý', 'ā', 'ă', 'ą', 'Ć', 'ć', 'Č', 'č', 'đ', 'ė', 'ę', 'ě', 'ğ', 'ġ', 'Ħ', 'ī', 'İ', 'ı', 'ļ', 'Ł', 'ł', 'ń', 'ň', 'Ō', 'ō', 'ő', 'ř', 'Ś', 'ś', 'Ş', 'ş', 'Š', 'š', 'ţ', 'ū', 'ź', 'ż', 'Ž', 'ž', 'ơ', 'ư', 'ǔ', 'ș', 'ț', 'ɔ', 'ə', 'ɛ', 'ʷ', 'ʼ', 'ʿ', '˚', 'Ι', 'Π', 'α', 'β', 'ε', 'η', 'ι', 'κ', 'μ', 'ο', 'ρ', 'ς', 'τ', 'υ', 'χ', 'ψ', 'ό', 'Б', 'В', 'Д', 'Ж', 'З', 'И', 'К', 'Л', 'М', 'Н', 'О', 'П', 'С', 'У', 'Ф', 'Х', 'а', 'б', 'в', 'г', 'д', 'е', 'з', 'и', 'й', 'к', 'л', 'м', 'н', 'о', 'п', 'р', 'с', 'т', 'у', 'ф', 'х', 'ц', 'ч', 'ш', 'щ', 'ъ', 'ы', 'ь', 'ю', 'я', 'і', 'ј', 'ћ', 'ּ', 'א', 'ב', 'ג', 'ו', 'ט', 'י', 'ך', 'ל', 'מ', 'נ', 'ס', 'ע', 'פ', 'ץ', 'צ', 'ק', 'ר', 'ת', 'װ', '،', 'أ', 'إ', 'ا', 'ب', 'ة', 'ت', 'ج', 'ح', 'خ', 'د', 'ذ', 'ر', 'س', 'ش', 'ص', 'ط', 'ع', 'ف', 'ق', 'ل', 'م', 'ن', 'ه', 'و', 'ي', 'پ', 'ک', 'அ', 'ஆ', 'இ', 'க', 'ச', 'ட', 'ண', 'த', 'ந', 'ன', 'ப', 'ம', 'ர', 'ற', 'ல', 'ள', 'ழ', 'வ', 'ா', 'ி', 'ு', 'ே', 'ை', 'ொ', '்', 'ಡ', 'ಮ', 'ರ', 'ಲ', 'ಸ', 'ಿ', 'ก', 'ข', 'ค', 'ง', 'จ', 'ช', 'ฒ', 'ณ', 'ด', 'ต', 'ท', 'ธ', 'น', 'บ', 'ป', 'ผ', 'พ', 'ภ', 'ม', 'ย', 'ร', 'ฤ', 'ล', 'ว', 'ศ', 'ส', 'ห', 'อ', 'ะ', 'ั', 'า', 'ำ', 'ิ', 'ี', 'ื', 'ุ', 'ู', 'เ', 'แ', 'โ', 'ไ', '่', '้', '์', 'ზ', 'უ', 'ფ', 'ḩ', 'ḻ', 'ṟ', 'ṣ', 'ṭ', 'ạ', 'ễ', 'ệ', 'ὶ', '\\u200e', '–', '—', '‘', '’', '“', '”', '†', '‡', '•', '…', '′', '″', '₨', '€', '₱', '℃', 'ℓ', '№', '™', '→', '−', '≠', '♠', '♦', '➔', '\\u3000', 'り', 'ア', 'イ', 'ウ', 'カ', 'ク', 'コ', 'シ', 'ズ', 'ゼ', 'ソ', 'タ', 'チ', 'ッ', 'ツ', 'パ', 'ボ', 'マ', 'ム', 'ャ', 'ョ', 'リ', 'レ', 'ン', 'ー', '一', '三', '上', '下', '东', '中', '主', '义', '九', '乡', '事', '介', '伐', '会', '依', '俳', '僧', '光', '児', '全', '六', '兴', '其', '典', '况', '前', '加', '勇', '務', '化', '华', '単', '卡', '厂', '双', '句', '史', '号', '吉', '吹', '哈', '哪', '国', '國', '地', '坝', '坪', '堡', '大', '姚', '子', '孤', '安', '宗', '家', '寄', '寨', '射', '局', '屋', '岩', '島', '左', '巴', '師', '店', '庙', '延', '建', '得', '心', '怡', '排', '探', '教', '数', '文', '斌', '新', '方', '日', '昌', '明', '春', '昭', '普', '會', '本', '李', '村', '板', '桦', '概', '民', '水', '江', '法', '泥', '泽', '湾', '溪', '潭', '澤', '濟', '無', '燈', '界', '皮', '石', '砲', '磨', '禪', '站', '维', '置', '義', '羹', '育', '胜', '臨', '花', '茜', '莲', '華', '董', '薦', '薩', '虚', '街', '装', '覺', '解', '訪', '語', '话', '语', '赤', '転', '軽', '辞', '農', '达', '逆', '通', '造', '連', '道', '那', '鄉', '里', '野', '録', '镇', '长', '門', '陈', '食', '馬', '马', '鹿', '黄', '김', '준', '태', 'ﬂ', '）', '｜']\n"
     ]
    }
   ],
   "source": [
    "vocab = set()\n",
    "\n",
    "# Ideally we should take all dataset items into account here.\n",
    "for text in dataset_text.take(1000):\n",
    "    vocab.update([char.decode('utf-8') for char in text.numpy()])\n",
    "    \n",
    "vocab = sorted(vocab)\n",
    "\n",
    "print('Unique characters: {}'.format(len(vocab)))\n",
    "print('vocab:')\n",
    "print(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6dj4e-AGMaV4"
   },
   "source": [
    "### Vectorize the text\n",
    "\n",
    "Before feeding the text to our RNN we need to convert the text from a sequence of characters to a sequence of numbers. To do so we will detect all unique characters in the text, form a vocabulary out of it and replace each character with its index in the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 578
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 270275,
     "status": "ok",
     "timestamp": 1586974156436,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "xFFpuXfGMPq2",
    "outputId": "8c79b593-33d2-4761-e7cb-fb14f1f2cecc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  '\\t':   0,\n",
      "  '\\n':   1,\n",
      "  ' ' :   2,\n",
      "  '!' :   3,\n",
      "  '\"' :   4,\n",
      "  '#' :   5,\n",
      "  '$' :   6,\n",
      "  '%' :   7,\n",
      "  '&' :   8,\n",
      "  \"'\" :   9,\n",
      "  '(' :  10,\n",
      "  ')' :  11,\n",
      "  '*' :  12,\n",
      "  '+' :  13,\n",
      "  ',' :  14,\n",
      "  '-' :  15,\n",
      "  '.' :  16,\n",
      "  '/' :  17,\n",
      "  '0' :  18,\n",
      "  '1' :  19,\n",
      "  '2' :  20,\n",
      "  '3' :  21,\n",
      "  '4' :  22,\n",
      "  '5' :  23,\n",
      "  '6' :  24,\n",
      "  '7' :  25,\n",
      "  '8' :  26,\n",
      "  '9' :  27,\n",
      "  ':' :  28,\n",
      "  ';' :  29,\n",
      "  ...\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Map characters to their indices in vocabulary.\n",
    "char2index = {char: index for index, char in enumerate(vocab)}\n",
    "\n",
    "print('{')\n",
    "for char, _ in zip(char2index, range(30)):\n",
    "    print('  {:4s}: {:3d},'.format(repr(char), char2index[char]))\n",
    "print('  ...\\n}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 270255,
     "status": "ok",
     "timestamp": 1586974156438,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "lQB33zI7NkRo",
    "outputId": "e789efe1-b9c6-448c-91f8-02b6e985310c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['\\t' '\\n' ' ' '!' '\"' '#' '$' '%' '&' \"'\" '(' ')' '*' '+' ',' '-' '.' '/'\n",
      " '0' '1' '2' '3' '4' '5' '6' '7' '8' '9' ':' ';' '<' '=' '>' '?' '@' 'A'\n",
      " 'B' 'C' 'D' 'E' 'F' 'G' 'H' 'I' 'J' 'K' 'L' 'M' 'N' 'O' 'P' 'Q' 'R' 'S'\n",
      " 'T' 'U' 'V' 'W' 'X' 'Y' 'Z' '[' ']' '^' '_' '`' 'a' 'b' 'c' 'd' 'e' 'f'\n",
      " 'g' 'h' 'i' 'j' 'k' 'l' 'm' 'n' 'o' 'p' 'q' 'r' 's' 't' 'u' 'v' 'w' 'x'\n",
      " 'y' 'z' '{' '|' '}' '~' '\\xa0' '£' '§' '«' '®' '°' '±' '²' '·' '»' '¼'\n",
      " '½' '¿' 'Á' 'Å' 'Æ' 'Ç' 'É' 'Ë' 'Í' 'Î' 'Ó' 'Ö' '×' 'Ø' 'Ü' 'Þ' 'ß' 'à'\n",
      " 'á' 'â' 'ã' 'ä' 'å' 'æ' 'ç' 'è' 'é' 'ê' 'ë' 'ì' 'í' 'î' 'ï' 'ñ' 'ò' 'ó'\n",
      " 'ô' 'õ' 'ö' 'ø' 'ú' 'û' 'ü' 'ý' 'ā' 'ă' 'ą' 'Ć' 'ć' 'Č' 'č' 'đ' 'ė' 'ę'\n",
      " 'ě' 'ğ' 'ġ' 'Ħ' 'ī' 'İ' 'ı' 'ļ' 'Ł' 'ł' 'ń' 'ň' 'Ō' 'ō' 'ő' 'ř' 'Ś' 'ś'\n",
      " 'Ş' 'ş' 'Š' 'š' 'ţ' 'ū' 'ź' 'ż' 'Ž' 'ž' 'ơ' 'ư' 'ǔ' 'ș' 'ț' 'ɔ' 'ə' 'ɛ'\n",
      " 'ʷ' 'ʼ' 'ʿ' '˚' 'Ι' 'Π' 'α' 'β' 'ε' 'η' 'ι' 'κ' 'μ' 'ο' 'ρ' 'ς' 'τ' 'υ'\n",
      " 'χ' 'ψ' 'ό' 'Б' 'В' 'Д' 'Ж' 'З' 'И' 'К' 'Л' 'М' 'Н' 'О' 'П' 'С' 'У' 'Ф'\n",
      " 'Х' 'а' 'б' 'в' 'г' 'д' 'е' 'з' 'и' 'й' 'к' 'л' 'м' 'н' 'о' 'п' 'р' 'с'\n",
      " 'т' 'у' 'ф' 'х' 'ц' 'ч' 'ш' 'щ' 'ъ' 'ы' 'ь' 'ю' 'я' 'і' 'ј' 'ћ' 'ּ' 'א'\n",
      " 'ב' 'ג' 'ו' 'ט' 'י' 'ך' 'ל' 'מ' 'נ' 'ס' 'ע' 'פ' 'ץ' 'צ' 'ק' 'ר' 'ת' 'װ'\n",
      " '،' 'أ' 'إ' 'ا' 'ب' 'ة' 'ت' 'ج' 'ح' 'خ' 'د' 'ذ' 'ر' 'س' 'ش' 'ص' 'ط' 'ع'\n",
      " 'ف' 'ق' 'ل' 'م' 'ن' 'ه' 'و' 'ي' 'پ' 'ک' 'அ' 'ஆ' 'இ' 'க' 'ச' 'ட' 'ண' 'த'\n",
      " 'ந' 'ன' 'ப' 'ம' 'ர' 'ற' 'ல' 'ள' 'ழ' 'வ' 'ா' 'ி' 'ு' 'ே' 'ை' 'ொ' '்' 'ಡ'\n",
      " 'ಮ' 'ರ' 'ಲ' 'ಸ' 'ಿ' 'ก' 'ข' 'ค' 'ง' 'จ' 'ช' 'ฒ' 'ณ' 'ด' 'ต' 'ท' 'ธ' 'น'\n",
      " 'บ' 'ป' 'ผ' 'พ' 'ภ' 'ม' 'ย' 'ร' 'ฤ' 'ล' 'ว' 'ศ' 'ส' 'ห' 'อ' 'ะ' 'ั' 'า'\n",
      " 'ำ' 'ิ' 'ี' 'ื' 'ุ' 'ู' 'เ' 'แ' 'โ' 'ไ' '่' '้' '์' 'ზ' 'უ' 'ფ' 'ḩ' 'ḻ'\n",
      " 'ṟ' 'ṣ' 'ṭ' 'ạ' 'ễ' 'ệ' 'ὶ' '\\u200e' '–' '—' '‘' '’' '“' '”' '†' '‡' '•'\n",
      " '…' '′' '″' '₨' '€' '₱' '℃' 'ℓ' '№' '™' '→' '−' '≠' '♠' '♦' '➔' '\\u3000'\n",
      " 'り' 'ア' 'イ' 'ウ' 'カ' 'ク' 'コ' 'シ' 'ズ' 'ゼ' 'ソ' 'タ' 'チ' 'ッ' 'ツ' 'パ' 'ボ' 'マ'\n",
      " 'ム' 'ャ' 'ョ' 'リ' 'レ' 'ン' 'ー' '一' '三' '上' '下' '东' '中' '主' '义' '九' '乡' '事'\n",
      " '介' '伐' '会' '依' '俳' '僧' '光' '児' '全' '六' '兴' '其' '典' '况' '前' '加' '勇' '務'\n",
      " '化' '华' '単' '卡' '厂' '双' '句' '史' '号' '吉' '吹' '哈' '哪' '国' '國' '地' '坝' '坪'\n",
      " '堡' '大' '姚' '子' '孤' '安' '宗' '家' '寄' '寨' '射' '局' '屋' '岩' '島' '左' '巴' '師'\n",
      " '店' '庙' '延' '建' '得' '心' '怡' '排' '探' '教' '数' '文' '斌' '新' '方' '日' '昌' '明'\n",
      " '春' '昭' '普' '會' '本' '李' '村' '板' '桦' '概' '民' '水' '江' '法' '泥' '泽' '湾' '溪'\n",
      " '潭' '澤' '濟' '無' '燈' '界' '皮' '石' '砲' '磨' '禪' '站' '维' '置' '義' '羹' '育' '胜'\n",
      " '臨' '花' '茜' '莲' '華' '董' '薦' '薩' '虚' '街' '装' '覺' '解' '訪' '語' '话' '语' '赤'\n",
      " '転' '軽' '辞' '農' '达' '逆' '通' '造' '連' '道' '那' '鄉' '里' '野' '録' '镇' '长' '門'\n",
      " '陈' '食' '馬' '马' '鹿' '黄' '김' '준' '태' 'ﬂ' '）' '｜']\n"
     ]
    }
   ],
   "source": [
    "# Map character indices to characters from vacabulary.\n",
    "index2char = np.array(vocab)\n",
    "\n",
    "print(index2char)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 476
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 270802,
     "status": "ok",
     "timestamp": 1586974157036,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "DXUAlYmvN_Rj",
    "outputId": "2e7683da-5683-4d7a-b9d0-fffd7add372e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ORIGINAL CHARS: \n",
      "---\n",
      "J\n",
      "o\n",
      "s\n",
      "e\n",
      "p\n",
      "h\n",
      " \n",
      "H\n",
      "a\n",
      "r\n",
      "\n",
      "\n",
      "\n",
      "INDEXED CHARS: \n",
      "---\n",
      "44\n",
      "80\n",
      "84\n",
      "70\n",
      "81\n",
      "73\n",
      "2\n",
      "42\n",
      "66\n",
      "83\n",
      "80\n",
      "77\n",
      "69\n",
      "2\n",
      "41\n",
      "83\n",
      "70\n",
      "70\n",
      "79\n",
      "67\n"
     ]
    }
   ],
   "source": [
    "def char_to_index(char):\n",
    "    char_symbol = char.numpy().decode('utf-8')\n",
    "    char_index = char2index[char_symbol] if char_symbol in char2index else char2index['?']\n",
    "    return char_index\n",
    "\n",
    "dataset_chars_indexed = dataset_chars.map(\n",
    "    lambda char: tf.py_function(func=char_to_index, inp=[char], Tout=tf.int32)\n",
    ")\n",
    "\n",
    "print('ORIGINAL CHARS:', '\\n---')\n",
    "for char in dataset_chars.take(10):\n",
    "    print(char.numpy().decode())\n",
    "\n",
    "print('\\n\\n')    \n",
    "    \n",
    "print('INDEXED CHARS:', '\\n---')\n",
    "for char_index in dataset_chars_indexed.take(20):\n",
    "    print(char_index.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CHv5HhUuTQYS"
   },
   "source": [
    "## Create training sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rpdFJJc90-ML"
   },
   "outputs": [],
   "source": [
    "# The maximum length sentence we want for a single input in characters.\n",
    "sequence_length = 200"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 207
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 273784,
     "status": "ok",
     "timestamp": 1586974160086,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "Ap71VjB2Vuct",
    "outputId": "9b468ed6-8941-433b-f2e5-10b318d69ca9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'Joseph Harold Greenberg (May 28, 1915 – May 7, 2001) was an American linguist, known mainly for his work concerning linguistic typology and the genetic classification of languages.\\n\\nLife\\n\\nEarly life an'\n",
      "\n",
      "'d education \\n(Main source: Croft 2003)\\n\\nJoseph Greenberg was born on May 28, 1915 to Jewish parents in Brooklyn, New York. His first great interest was music. At the age of 14, he gave a piano concert '\n",
      "\n",
      "'in Steinway Hall. He continued to play the piano frequently throughout his life.\\n\\nAfter finishing high school, he decided to pursue a scholarly career rather than a musical one. He enrolled at Columbia'\n",
      "\n",
      "' University in New York. During his senior year, he attended a class taught by Franz Boas concerning American Indian languages. With references from Boas and Ruth Benedict, he was accepted as a graduat'\n",
      "\n",
      "'e student by Melville J. Herskovits at Northwestern University in Chicago. During the course of his graduate studies, Greenberg did fieldwork among the Hausa people of Nigeria, where he learned the Hau'\n",
      "\n",
      "'sa language. The subject of his doctoral dissertation was the influence of Islam on a Hausa group that, unlike most others, had not converted to it.\\n\\nDuring 1940, he began postdoctoral studies at Yale '\n",
      "\n",
      "'University. These were interrupted by service in the U.S. Army Signal Corps during World War II, for which he worked as a codebreaker and participated with the landing at Casablanca. Before leaving for'\n",
      "\n",
      "' Europe during 1943, Greenberg married Selma Berkowitz, whom he had met during his first year at Columbia University.\\n\\nCareer\\nAfter the war, Greenberg taught at the University of Minnesota before retur'\n",
      "\n",
      "'ning to Columbia University during 1948 as a teacher of anthropology. While in New York, he became acquainted with Roman Jakobson and André Martinet. They introduced him to the Prague school of structu'\n",
      "\n",
      "'ralism, which influenced his work.\\n\\nDuring 1962, Greenberg relocated to the anthropology department of Stanford University in California, where he continued to work for the rest of his life. During 196'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Generate batched sequences out of the char_dataset.\n",
    "sequences = dataset_chars_indexed.batch(sequence_length + 1, drop_remainder=True)\n",
    "\n",
    "# Sequences examples.\n",
    "for item in sequences.take(10):\n",
    "    print(repr(''.join(index2char[item.numpy()])))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Y8spPCfe-iTn"
   },
   "outputs": [],
   "source": [
    "# sequences shape:\n",
    "# - Each sequence of length 101\n",
    "#\n",
    "#    201     201          201\n",
    "# [(.....) (.....) ...  (.....)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HdcrcUs4Xxso"
   },
   "source": [
    "For each sequence, duplicate and shift it to form the input and target text. For example, say `sequence_length` is `4` and our text is `Hello`. The input sequence would be `Hell`, and the target sequence `ello`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9fxvXsP0XFDh"
   },
   "outputs": [],
   "source": [
    "def split_input_target(chunk):\n",
    "    input_text = chunk[:-1]\n",
    "    target_text = chunk[1:]\n",
    "    return input_text, target_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "454rWIQYXXRY"
   },
   "outputs": [],
   "source": [
    "dataset_sequences = sequences.map(split_input_target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 122
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 274570,
     "status": "ok",
     "timestamp": 1586974160943,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "Kuoh4tCdYCck",
    "outputId": "a8eb75f1-a9a1-4795-acd5-e2129e81e080"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input sequence size: 200\n",
      "Target sequence size: 200\n",
      "\n",
      "Input:\n",
      " 'Joseph Harold Greenberg (May 28, 1915 – May 7, 2001) was an American linguist, known mainly for his work concerning linguistic typology and the genetic classification of languages.\\n\\nLife\\n\\nEarly life a'\n",
      "\n",
      "Target:\n",
      " 'oseph Harold Greenberg (May 28, 1915 – May 7, 2001) was an American linguist, known mainly for his work concerning linguistic typology and the genetic classification of languages.\\n\\nLife\\n\\nEarly life an'\n"
     ]
    }
   ],
   "source": [
    "for input_example, target_example in dataset_sequences.take(1):\n",
    "    print('Input sequence size:', repr(len(input_example.numpy())))\n",
    "    print('Target sequence size:', repr(len(target_example.numpy())))\n",
    "    print()\n",
    "    print('Input:\\n', repr(''.join(index2char[input_example.numpy()])))\n",
    "    print()\n",
    "    print('Target:\\n', repr(''.join(index2char[target_example.numpy()])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cp0tl0sN807l"
   },
   "outputs": [],
   "source": [
    "# dataset shape:\n",
    "# - Each sequence is a tuple of 2 sub-sequences of length 100 (input_text and target_text)\n",
    "#\n",
    "#    200       200           200\n",
    "# /(.....)\\ /(.....)\\ ... /(.....)\\  <-- input_text\n",
    "# \\(.....)/ \\(.....)/     \\(.....)/  <-- target_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BDYHEJ0pY1ai"
   },
   "source": [
    "Each index of these vectors are processed as one time step. For the input at time step 0, the model receives the index for \"F\" and trys to predict the index for \"i\" as the next character. At the next timestep, it does the same thing but the RNN considers the previous step context in addition to the current input character."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 357
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 274533,
     "status": "ok",
     "timestamp": 1586974160945,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "C-0zpv53Y2o4",
    "outputId": "8f41e91e-3ecb-4086-bdbd-9c466564729b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step #0\n",
      "  input: 44 ('J')\n",
      "  expected output: 80 ('o')\n",
      "\n",
      "Step #1\n",
      "  input: 80 ('o')\n",
      "  expected output: 84 ('s')\n",
      "\n",
      "Step #2\n",
      "  input: 84 ('s')\n",
      "  expected output: 70 ('e')\n",
      "\n",
      "Step #3\n",
      "  input: 70 ('e')\n",
      "  expected output: 81 ('p')\n",
      "\n",
      "Step #4\n",
      "  input: 81 ('p')\n",
      "  expected output: 73 ('h')\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i, (input_idx, target_idx) in enumerate(zip(input_example[:5], target_example[:5])):\n",
    "    print('Step #{:1d}'.format(i))\n",
    "    print('  input: {} ({:s})'.format(input_idx, repr(index2char[input_idx])))\n",
    "    print('  expected output: {} ({:s})'.format(target_idx, repr(index2char[target_idx])))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1iDlp40lC5YB"
   },
   "source": [
    "## Split training sequences into batches\n",
    "\n",
    "We used `tf.data` to split the text into manageable sequences. But before feeding this data into the model, we need to shuffle the data and pack it into batches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 274506,
     "status": "ok",
     "timestamp": 1586974160945,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "eDq-wa5EC3wW",
    "outputId": "89c0b910-d022-46c0-b83b-3352b13b7a6f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<DatasetV1Adapter shapes: (<unknown>, <unknown>), types: (tf.int32, tf.int32)>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Batch size.\n",
    "BATCH_SIZE = 64\n",
    "\n",
    "# Buffer size to shuffle the dataset (TF data is designed to work\n",
    "# with possibly infinite sequences, so it doesn't attempt to shuffle\n",
    "# the entire sequence in memory. Instead, it maintains a buffer in\n",
    "# which it shuffles elements).\n",
    "BUFFER_SIZE = 100\n",
    "\n",
    "# How many items to prefetch before the next iteration.\n",
    "PREFETCH_SIZE = 10\n",
    "\n",
    "dataset_sequence_batches = dataset_sequences \\\n",
    "    .shuffle(BUFFER_SIZE) \\\n",
    "    .batch(BATCH_SIZE, drop_remainder=True) \\\n",
    "    .prefetch(PREFETCH_SIZE)\n",
    "\n",
    "dataset_sequence_batches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 306
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 478249,
     "status": "ok",
     "timestamp": 1586974364716,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "b_kYgvQGBO0U",
    "outputId": "0a456e52-5f7c-4f85-dea6-cdf4f42057fb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1st batch: input_text: tf.Tensor(\n",
      "[[84 80 79 ... 72 14  2]\n",
      " [84 66  2 ... 66 77 70]\n",
      " [69  2 70 ... 70 83 85]\n",
      " ...\n",
      " [84 71 70 ... 85 74 80]\n",
      " [ 1  1 39 ...  1  1 49]\n",
      " [83 66 79 ... 14  2 66]], shape=(64, 200), dtype=int32)\n",
      "\n",
      "1st batch: target_text: tf.Tensor(\n",
      "[[80 79 84 ... 14  2 19]\n",
      " [66  2 77 ... 77 70  2]\n",
      " [ 2 70 69 ... 83 85  2]\n",
      " ...\n",
      " [71 70 83 ... 74 80 79]\n",
      " [ 1 39 74 ...  1 49 86]\n",
      " [66 79 76 ...  2 66 84]], shape=(64, 200), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "for input_text, target_text in dataset_sequence_batches.take(1):\n",
    "    print('1st batch: input_text:', input_text)\n",
    "    print()\n",
    "    print('1st batch: target_text:', target_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UkDCH15v_2I6"
   },
   "outputs": [],
   "source": [
    "# dataset shape:\n",
    "# - 64 sequences per batch\n",
    "# - Each sequence is a tuple of 2 sub-sequences of length 100 (input_text and target_text)\n",
    "#\n",
    "#\n",
    "#     200       200           200             200       200           200\n",
    "# |/(.....)\\ /(.....)\\ ... /(.....)\\| ... |/(.....)\\ /(.....)\\ ... /(.....)\\|  <-- input_text\n",
    "# |\\(.....)/ \\(.....)/     \\(.....)/| ... |\\(.....)/ \\(.....)/     \\(.....)/|  <-- target_text\n",
    "#\n",
    "# <------------- 64 ---------------->     <------------- 64 ---------------->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ghB-VwLlD-Oz"
   },
   "source": [
    "## Build the model\n",
    "\n",
    "Use [tf.keras.Sequential](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential) to define the model. For this simple example three layers are used to define our model:\n",
    "\n",
    "- [tf.keras.layers.Embedding](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding): The input layer. A trainable lookup table that will map the numbers of each character to a vector with `embedding_dim` dimensions;\n",
    "- [tf.keras.layers.LSTM](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM): A type of RNN with size units=rnn_units (You can also use a GRU layer here.)\n",
    "- [tf.keras.layers.Dense](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense): The output layer, with vocab_size outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 425
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 478210,
     "status": "ok",
     "timestamp": 1586974364719,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "0cg8DlO3QjuT",
    "outputId": "c66429df-8d32-42ec-e8b9-2aea1157cb3f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tmp_input_array shape: (2, 8)\n",
      "tmp_input_array:\n",
      "[[5 8 2 5 2 8 0 9]\n",
      " [7 3 6 4 5 1 6 2]]\n",
      "\n",
      "tmp_output_array shape: (2, 8, 5)\n",
      "tmp_output_array:\n",
      "[[[-0.04438466 -0.0477155  -0.00650557  0.00578437 -0.02522211]\n",
      "  [-0.00249968  0.00477722  0.00990368 -0.02025222  0.008913  ]\n",
      "  [-0.04758141 -0.02608242  0.03385669  0.0057972  -0.00750101]\n",
      "  [-0.04438466 -0.0477155  -0.00650557  0.00578437 -0.02522211]\n",
      "  [-0.04758141 -0.02608242  0.03385669  0.0057972  -0.00750101]\n",
      "  [-0.00249968  0.00477722  0.00990368 -0.02025222  0.008913  ]\n",
      "  [-0.00630327 -0.04054337  0.04020781 -0.04592142  0.00964195]\n",
      "  [-0.03995473 -0.01548551 -0.00048856 -0.02285538 -0.02910516]]\n",
      "\n",
      " [[ 0.03945759  0.04857247 -0.00674983 -0.00098724  0.0119729 ]\n",
      "  [ 0.04236538  0.01068302 -0.04940217 -0.04754317  0.03218012]\n",
      "  [ 0.01865717 -0.02711406  0.04124454  0.02071499  0.03038916]\n",
      "  [ 0.00376308  0.00901873 -0.00290323  0.04185314  0.02587625]\n",
      "  [-0.04438466 -0.0477155  -0.00650557  0.00578437 -0.02522211]\n",
      "  [ 0.00597124 -0.00562616  0.03060223  0.04036254  0.02135176]\n",
      "  [ 0.01865717 -0.02711406  0.04124454  0.02071499  0.03038916]\n",
      "  [-0.04758141 -0.02608242  0.03385669  0.0057972  -0.00750101]]]\n"
     ]
    }
   ],
   "source": [
    "# Let's do a quick detour and see how Embeding layer works.\n",
    "# It takes several char indices sequences (batch) as an input.\n",
    "# It encodes every character of every sequence to a vector of tmp_embeding_size length.\n",
    "tmp_vocab_size = 10\n",
    "tmp_embeding_size = 5\n",
    "tmp_input_length = 8\n",
    "tmp_batch_size = 2\n",
    "\n",
    "tmp_model = tf.keras.models.Sequential()\n",
    "tmp_model.add(tf.keras.layers.Embedding(\n",
    "  input_dim=tmp_vocab_size,\n",
    "  output_dim=tmp_embeding_size,\n",
    "  input_length=tmp_input_length\n",
    "))\n",
    "# The model will take as input an integer matrix of size (batch, input_length).\n",
    "# The largest integer (i.e. word index) in the input should be no larger than 9 (tmp_vocab_size).\n",
    "# Now model.output_shape == (None, 10, 64), where None is the batch dimension.\n",
    "tmp_input_array = np.random.randint(\n",
    "  low=0,\n",
    "  high=tmp_vocab_size,\n",
    "  size=(tmp_batch_size, tmp_input_length)\n",
    ")\n",
    "tmp_model.compile('rmsprop', 'mse')\n",
    "tmp_output_array = tmp_model.predict(tmp_input_array)\n",
    "\n",
    "print('tmp_input_array shape:', tmp_input_array.shape)\n",
    "print('tmp_input_array:')\n",
    "print(tmp_input_array)\n",
    "print()\n",
    "print('tmp_output_array shape:', tmp_output_array.shape)\n",
    "print('tmp_output_array:')\n",
    "print(tmp_output_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "I7ZuvZHBD_pS"
   },
   "outputs": [],
   "source": [
    "# Length of the vocabulary in chars.\n",
    "vocab_size = len(vocab)\n",
    "\n",
    "# The embedding dimension.\n",
    "embedding_dim = 256\n",
    "\n",
    "# Number of RNN units.\n",
    "rnn_units = 1024"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-sojdDCAICWO"
   },
   "outputs": [],
   "source": [
    "def build_model(vocab_size, embedding_dim, rnn_units, batch_size):\n",
    "    model = tf.keras.models.Sequential()\n",
    "\n",
    "    model.add(tf.keras.layers.Embedding(\n",
    "      input_dim=vocab_size,\n",
    "      output_dim=embedding_dim,\n",
    "      batch_input_shape=[batch_size, None]\n",
    "    ))\n",
    "\n",
    "    model.add(tf.keras.layers.LSTM(\n",
    "      units=rnn_units,\n",
    "      return_sequences=True,\n",
    "      stateful=True,\n",
    "      recurrent_initializer=tf.keras.initializers.GlorotNormal()\n",
    "    ))\n",
    "\n",
    "    model.add(tf.keras.layers.Dense(vocab_size))\n",
    "  \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XoPwxyAPEg6z"
   },
   "outputs": [],
   "source": [
    "model = build_model(vocab_size, embedding_dim, rnn_units, BATCH_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 255
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 478161,
     "status": "ok",
     "timestamp": 1586974364722,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "iLnlZFgU55bQ",
    "outputId": "b299c133-c38b-4636-a478-6c601a5a4275"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (64, None, 256)           158976    \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (64, None, 1024)          5246976   \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (64, None, 621)           636525    \n",
      "=================================================================\n",
      "Total params: 6,042,477\n",
      "Trainable params: 6,042,477\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 422
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 478751,
     "status": "ok",
     "timestamp": 1586974365328,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "CcaO_rO_8-GH",
    "outputId": "80e5f385-1da2-41fb-c06b-7eb51fcd8b52"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa8AAAGLCAYAAABwVh0hAAAAAXNSR0IArs4c6QAAAERlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA6ABAAMAAAABAAEAAKACAAQAAAABAAABr6ADAAQAAAABAAABiwAAAADzcVENAABAAElEQVR4AeydBZjjRtKGO3RhZmZm5mRDm1wYNszMzMzMTBdmZma6MDMzM2+w/3rrUvplWbZlmBl7XfU8HkutVqv7a0+XunCwGOOVwckRcAQcAUfAEegcBN4ZTJhX7Jz+ek8dAUfAEXAEHIHw9OAOgiPgCDgCjoAj0GkIOPPqtBnz/joCjoAj4AgEZ17+I3AEHAFHwBHoOASGzPb4o48+Co8++mi22M8dAUfAEXAEHIE+QWDppZcOww8/fMmzy5gXjGu11VYrqeQnjoAj4Ag4Ao5AXyHw9ttvh8kmm6zk8WXMy666EaIh4d+OgCNgCIwzzjhhn332Cdtss40V+XcOAiuvvHIYZphhwqWXXppz1YuKIvD000+HOeaYI7e667xyYfFCR8ARcAQcgXZGwJlXO8+O980RcAQcAUcgFwFnXrmweKEj4Ag4Ao5AOyPgzKudZ8f75gg4Ao6AI5CLgDOvXFi80BFwBBwBR6CdEXDm1c6z431zBBwBR8ARyEWgoql8bm0vdAQcAUegSQRuvPHGsN1224Wzzjor9O/fv8nWBp3bd9lll/DHH3/ogAYbbLCw7777htFHH71kgIccckgYMGBAmGaaaUrKBw4cGM4444zwyCOPBNycttxyy7DYYouV1OHkwQcfDIcddli49dZbw+CD1793ufrqq8OFF14Yvv7667DooouqT/CMM84YTjvttDDqqKOGNddcs+yZO+64Y/j777+TclwtxhxzzOS80QNnXo0i5/c5Ao5AQwi8++674f333w989yWxyMMk2oVOPfVUdcS95pprwiijjBJGHnnkpGu///572GyzzcI888xTxrhuueWWsP3224cxxhgjnHfeeWHaaadN7ksffPfdd2GdddYJH374oTKTepnX2WefHa688kp96XjrrbcCTAlmeu2114YNNtggLL/88uGVV14JBx98cPqxYYcddgi//fabvrDccccdYfPNN28J86qf9ZZ0y08cAUfAEagPARZamBeLWF8ROwF2Jn/99VdfdSH3udNNN50yJ5zBhxzyf3sL+vjvf/87ULbFFluU3MdOaNlllw2TTDKJ7roqMS5u4l4YV6N0+umn6+5q0kknDUsssUTYc889tSnKhxtuuHDzzTeHyy+/XJlb+hkTTzxxmGqqqcKCCy6YLm762JlX0xB6A46AI1AvAhNNNFG9t7S0/l577RXuu+8+FbG1tOEeaOz8888PTz75ZNhvv/1KWmf3s/HGG4exxhorXHLJJWGIIYYouZ4+MVHfKquski6u65idHQzqzz//1Ptmm202/WZHBxFRZO+99w677757+Oabb7SsJ/848+pJdL1tR8ARKEPgtddeU30OYiiIxe/iiy8OhFRiZ3DyySerLmyjjTYKn332WXL/s88+G/bYY49w0kknBfRma621VphvvvlUh8Pu5L///W9Ye+21wxprrBEQvUFHHHGEnlPGdQjGdeSRR+oxiz8LLsR1dhTohdqFfvnlF2Va6LnY3aRp/fXXDz/88EPYeuutVfwJPr/++mu6ih6/8847Yf/99w8XXHBBQ3oua5Bd3jPPPJPsCF9//XW9lNatIZZkx3j44YfbbT33LXLfEhKZJpmVS8r8xBFwBBwBEBh77LGjMJeGwZAdQhQFv64xwjS0nUMPPTSOMMIIWiaGCFF2B1GYkp6LSEzr3H///VHe9LVMDAPiLLPMEoW5RY5Zr8RAQesdd9xxen7AAQfouYgHo+iKtOycc87RssceeyxOPfXUWia7r8g5JDsGLROxpp4382ellVaKYrxQVxOyc4nCpErusfX4iiuuKCn//vvvo+istL8iToxi2KHH//rXv+JRRx2V1JVdUpx33nmjMB4tW3XVVbWeGIYkdRo9WH311SPPll1WSRMSAT4OO+ywkWenSfRj+uyXX345XVz1+KmnntJ7JDBvtt5TvvPqufcCb9kRcAQyCLBbsl2PXWIntOKKK+opOyve8LGaQxxmu6CFF144CGPSOuiF2AEIMwrCeFRcheUi6Zy4liYMMoTRpYvC3HPPHUYccUTdrfTr10/PqbDbbrsFdoO2Eyu5qY9OMICAxh133JIeyKKuRhfok5544onw1Vdf6ffQQw+t48CIA8J4Aj1YM+LCkgf/c/LFF1+E6667LmBkgpVhmqaffnrdARIJvifJmVdPouttOwKOQBkCLLBZkjd1LUqbgMOIEIthqQZZPqeZZpopsRLEEIAUTogNX3rpJa1X9E/W0nC00UYLm2yySUss4Yr2oVa9V199VatkmReiVwhXgwknnFCP55xzzkQvhp4MMSgGFCeeeKJeb+UfdGgYiiDqzZJZSVrfs9dbde6m8q1C0ttxBByBliJQzQAh/SDL84TuLLsLSNfLHmeZV/Z6O5yj84JEHFjSHUzpoaweTESEWv7ee++FTTfdVHdsadP1F154Qa+zw8UAg+9G6N5771XdY9691id8wXqSnHn1JLretiPgCPQ4AvaGP+WUU6r4rOgDO4F5sbOEfvrpp5JhWTkiwzSNN954egpzwx3gzTff1I/VMStAdmTsNBtlXoh0p5hiCmu25BsRJmQ7wpKLLTxxsWELwfSmHAFHoPcREGOOgOk9kR7MN0oMGpKOmGl3OsoDjAtRo12zyogp24kYE5T1z8JMHfEpuj+LykE921ktsMACgUSO6AHTH9MtsjOzutwH1TP2c88993835fz99NNPtbQSc8u5paEiZ14NweY3OQKOQKMI2CL5wQcfJE2YSXya6eDIDGXFT4isbCdy1VVXhc8//1wjPSBaQ2yGMQZ+TyywONCa2AwfJWuTHYqYr6nRB/UwjDjzzDM1sgXhk9qFGA+RMJ577rmSLlGGLguGLNaFybXrr78+iEWoRr9ICgsc4H6AyLWIYQfuByONNFK47bbbcltmJzzDDDMEnJl7kpx59SS63rYj4AiUIIB12s4776xlN9xwgzq0svjefffdWobFH6Iwom/ghAvhu5VmYOiBZp55ZvXxIoYfDGrdddfVuuhbiPwAc6MNrO7wOSKWHkYPJtLCyIPdF/eL6bZa5BGCCYaW3sloo334B/EguquLLrpI+5buCpaSd911lzJprDEXX3xxZXIPPfSQMpd03VrH4AUjJNTTJ598UrU6Lxo///xzriMyc4YFKHEae5yyxvPmV5At93NHwBFwBJr182oGQYkyoT4/+HTh5yROslFEf7lNyu4u8T/Cp0kYU1m9L7/8MgpTLCmnrBXUKj8v+kKf8GeTYLoVuyZxIqMwlYrXi1wQcV8Us/oy/6zsvcLkoogis8V6vtNOO6lfmViIll1vtZ+XG2z0+OuBP8ARcARajQBiKz6VCNGhkenB7Ny+sbbLUl5Ztk5PnluopfQz6BP+Z+xY2XGaUUa6DrENmyF2m+xYiX9Yy8qTHev4449f9jj87hAlsvPLWkdSOS/6R1kjdRS0LfP6+OOPA2IFtv3LLLNM2GqrreoYVuWq8jqgW22iG9M+1jh5VkcoOZGR85l//vmTIJTtlM6BsRAGhzFYkMzKIy+/wv2EdsHHxpway2u1b8nRRx+tyuh0DxEb9UpomvRDc44fffRR9bFJX0JPgT7FfJrS1/y4NgImOmRtGBQJnRMiU4wtYFiIQ82/C10UYk2JXhFuv/12DdLbSgxYQ2hbInA01CwOy+gM6Vs2jQsho9Bz4njOCwf/By2h7N6uXcSGhGwhzIgMMh5zzDHZbjZ8TsgSeYuJYiYa5Q2jYjtiqRMFdH2+KHyTeieccIKWSe6cpKwvDkQxG+UtTPsijoINdQEs5J8kyptcRfFLQw03cBOiiHpJdB9R8ggpBvxOhIHHH3/8sd5mWlY/PQbEVGJYEOWtX/snLwlRFoiWPauvGuorsaG80UdCRzHPhB6SFBxRDDX6Coaaz21EbFizUalAmCTZnRWp2qt1xEgjVzTbbCc6MjwUIVwIzNlqYkuMFz2e/NW2x5iirrfeevr4dL12SefAtpxcOs0Q4+KNC+uglr0NNdAhFMWNpKdgB8NbnRGJ8CRGnp326nd2DEMNNVRYZJFF1OqKjhBYtadNh3t1wL38MIwT2M1+++23alBw/PHHa/ioXu5Gnz8Oh2yLYNHnnUl1gPWU33xvUtuKDQEhzTRaDQqLda0F267bt/Whr9M50B/y+xAtulkyT/1m22nm/mbSU1iMOnnD69N/6kpjMN2LhTZqBqduvpd0G3ycHAFDoGXMC/8CPvhcsGvB5JU3BBYVYmwRC2vXXXcNjz/+eLjpppv0rYlU4PgCIC/lg2KVnQ33pwlnQnRNl156qZpoWtqDdJ1Kz7c6ODL+5z//0eyts846q5rMZnVdKBTxYUA2y1sEQS+hdD1iiuFDAgPDhBUFK3oxTEzxu6AfjG+CCSZQ/QYJ5IyI0UZQUoKNwoDmmmuuQDwyPNFpr7eVxYwXXRfjIRgqLwtFxoP8mvnAx4ZgqZjxYrKLIpnU5ThPMueYRTN3yNGR2YOt+asw96SzyKanADeJMq73k8aB9hZaaCGDsNB3kTHQECkkJFq39huFN1EHcN4kZhs5icADJTk6Fn4Dl112WSANxEEHHaTjYr5OOeWUimMo1NlUJTDj/wC9BztIlOfLLbec1kCPZ35R/J8QGQGlORgT04+Fnd0IJNZp+lsEa5gnqTOWWmopvYafE/+L1GeeuIcXIf7vnByBjkIgK5NsROe1zz77RPEziA8//LDqk2Th15QD6CRkwYrCxFRWTRoDZMGSf0bD+ctCpXot+QeN8s+ldZCpm/mr/CNrmewOojC5uOiii0YRl2kZuiejas+njvyzRvknjsJkIn068MADtQ1R7lsTEdPa2WefPZKCQUQTERmubNG1nqUYaDSdgz1EFok4xxxzqF4GnGVB1PZFvKT6EatX9Bv5t/zYYiM6L4kKHUUsmuhkMCeGiqSnEMYTZfHTZzMvpH4QBqPnssOIsiPUtnorPYXhKAtz4THIy0yhFBvgYukmtHH5I9EO9PcrLx1ahH42L8UGFyV7rOLCb7Aa8b8jLzRRAqpGzIz5HxHmqebg3Ie5tCjvtS154Uiaoi56NbHw0jKwn3zyySM6WXkJiyJu03tIB3LPPfdE+sxvBv0Rv2/+V9GdFqW+0nkV7V+71OspnVe7jK+3+lFN58XOqITqZV7y9qr/ZPJWmLSz5JJL6j+IWJ9oGbb//MPIW15Sx4whxLM7KZM3RK1ni5AxL5TdRmIhqHVgiDCcWs8Xp0RlnuncOjAwsXqJaeYFA6SP+I4YsZBTZsyLcnwtKLNcRJTZWFh4jCT2lz7Dzm0SZDdhRVF2gNqWOAUmZfUcNMO87Dmy+9M+GPOivMh4bFEUq01rSvMrgY04h2qZWB5p25ZbiUIzsLDcSpTB0GFAacL/BsMamGwtyjIv6hcZAwyM/oo1aTRjC+YfxgzjgElB88wzj9bTk3/+ML/GvCjKGwPlRZmX7DD1GRgKQbxo0TdJ0qjn/DHsyE9lBP6ye7dTfTmUBIvJueiJtB0YH0T+KtrFSEd23jpGG2dyU5UD/I243z+OQW/+BljrMvRU02JDMqAiGpI3DRnL/wjva3n7CxbqxRSMFqeLWvKmqpURXRih9IMQbaT1Sumw+8svv7yK5DBlxzMecVe15yPqoR8oz41Q9NMXE2FRjlkqYhgLeEkZYj1IFkf95k896RwQVSIq5B5EOdAbb7yh3/zBKAXRFcEyzSQ2udhLB42Ox3Q46VxJ8gKg3v5pXIsOI40x91h6iqL3Z+uZObr9prhOio30nNgY+A3a85l/oi8gWkMchxizKFkbReun64Edv21E5ojWZeekl9Mx7TbYYAPNiEvfMNYhagT/fxtuuKHWFUtLFTuix7Q54H+D/0XE1AMHDkx8hOQFU0WH9YyPhyCyxJDKxJD6YP9ThgBuHBhVuTi2DJq6CtDrVwoe3DTzQgeA7T7J0eohZO5ZMmsVHOaqEX5X6CpgCLWeb/5hljbB2mWhscWGxGr4kOBfkSa7bt/pa7WO0ZekSd7AlYkRBgfmzsKJ/mwS0bUYI0/Xb7fj7Hjy+oeOkEUSnVO91AjG9T6jyBho034r9Y6jmTGw0PE8dKC8EKCHQpeaJhgyukL0gOi60CvjFEpcOshejLgu0o70rckx8wMVxSK58Z8D7kdn3Kg/ULa9QfWcl2bWOMepuRkmuHAlatpbDKdQFPgwkSxJyJJsUUvOzYKLFAi1no9jH4QRRiWyOgDFTqknCGYlIkeNudZPzH6Ju2bZSHmbHRSIxROTcealXmpm4a/3WbXq4zoA1TuORsbA701E9SpBYDeDGwfRFNI7xnR/eRnjtyQiRDUgIZ6dWYzyvwBJGKX0LXosonI1piq74AWOQIci0DTzMrERqbNFJpnAgOUUFluVKF23Up1K5SK3V58ZFpdazzexJNZwaSIVAgsthOgEMRXWd6I4T6pZugSrl1xo8IAtMCLQ8847LyAmQuxp/W+wyba6jXmBiIgCGVM28TFleZiy6CPesmvUg3gpqoea+U2ln4NoEbG1ibltHNYfnkN/07+LSmOo1SezbsUKlTZXX3117Qq7cyh7v4lTkToQVBZRohHiQawUsQJFHG1EP/FZNGtFK/dvR6CTEWiaeYnyWM1xCVNESmpk9bwdrrXWWonpruV3SS9iFuolLdO3iM84IqbJUl5ThpgEc2bEJog+aj0ffQCJ02BKiFmIeozOgMjHvI0ee+yx+kbKNUiU3YHF68UXX0ycgDF/t6jXtoClF4Ii6RzYhbLb4n4W+TvvvFNFn/WKW7WT//yxPhiW6WtFjxsdj7WPyTzEYisGK7pjAEOoN9JTgL0t8PY749lF5oR6ULUUG1zndw2xK8K1AFGQOcsyfpgDbgL0g/hulmKDc9vFWX9oBybN71CsXtXsHhGhMUheatDjmpwfs3nCmKUJkSD1xfIvLLHEEsklRI+IFWGCYv0bDjjgAHVlQGQtVqEqkjTdKy9STo5ARyMg/2AlVK+1ITfLLkutlwQItUIS3Udi+o25tJnKy5tsxMxXfFaSMiy2xL8mnnzyyRqqiDZEaR6xNMTiTN5EI6byWOZhFYZZ7wMPPFDS52rPp+Lzzz+fhJYRhhexhsRUWJTjkf6JrkEjKYsfWmKKjzWZ+JOpib0wwCi+WVF8ehLzeawVheFFeWPWcDX0m76Jsj1iDWZYiIgwYvEojDu5167ZdyOm7rKrjbLz1OfIW7+avWOyXZSw6BTfq8RcH4s6LCKLjkd2V/ps2dlGLBZxe8CyThbpki5ILD+16pTFNnKP+NpFMTSI4sOkz6MyLgiMgQ+WqcIMIlao4CMLcEl76RPCdjGPhqPslqIoyAuPwaKU8xvEbFyYrZrFi/FO+jFqkcdvl+dQ96yzzlLrQtwD+P0IMyobA78Xsya1/mHdKgYySX8px7IR61fxDYyYoWO6LgwpYnUo+lD9bWH1miXqiI9ctlhD9GBtKLopfQ6Y8nvkGcJoE7z4P5CoH4WsOdMPcVP5NBqVj91UvjI29VwxK+08a0PeFkuoEeZFAyw4xG6TN8rE7Lik4SZOaPtdCfkPA6hERZ4vzqZJXLBKcdHwmwEo2pMcN1F2FJUeWVe5JLtT5gYjlbdp9bnBHBqmLYr4xDeqrkb7sLIxL9IogBcMuhLJ7q5P01NU6pcxryIpNmgDhs/vAsr7XciupizFhlYu+Id4iOl2cV+A6WSJ3yi+Z9ViJRLjETcSsG8lOfMqhqYzr2I41apVjXm1zFJA3vB6LHYbbU8iVnnVqMjzEe0YIUrMI0QvZm1mptR59eopw3oS0RO6INPB2f0yeSoWIvoHEZmrEVadpleqVo/MsYg6q1HRtqq1wTWszwyvSnXNwIbrJh7L1s2LLpJXlr2vVefgwacapd03zDgiXb/Z/mJtaxa3tAtWabyEcSreiCZlp1z1/w3d16CkT03jPKgek8DRLK1Zz7AazUZoP0RcJIiTmTXowQ0ClQ2Gaawp6EOJF5olovuQ2QDjMbM8zdapdo5IG7ULqgoJGqFuJeiGMSAiKj7xRbO04447luiHEY3j5tEstYx5NduRQfl+dBD8KNELonsgRTb6OqzC0KWhQyFUUq3FxnyXamEFk8z+uLP3FG0re5+dm56N0EmVXgSsbrt+p8fQrn20frEw4TOEWwX6MhahbiQWZhb2nqKebr9av9Hj8yLIOoEFqfnHcg8W0ej3eWnJ/m+jh+W3wcsTxmCSUDL3MbhiiPN+wM7AXoRyK1YoxAoWy1gRm+tvEKYEMyU0HoZD+CmKhCnw8pymHXbYQa240ceSigrdfyuYV8vEhjLpTlUQQAeyxhprRHRpMrH6jU6NCAidROh3iISCDotxoDdMRxbplLF0WooN9FWIl9GboXvtK+pLsaG8BEZCqfEb7AmSF8m43377taTpRsSG6D/RQ2aJ8RIaDz13lsSyVP8PiUJUCxfsB/if5YNIul5Chysv2sltRH+hLfSvEBFbJHNCtMhKScV/DoTRaX2xss5eqnjeK2JDGYRTFQSw+OIjs6SRDprd+VR5VI9eYscocSX1Yw/Kczi3a+36ja8dKTbSZP5S6bJ2OSZALxaO4J8WJbZL/3qjH5Ui97fi2UT1wUKaYNvtRljxIqUh4Hea2IFj2Yvkg+Da/DYqkYn6CJDNzq4RYmeH47wwSf0NWgB1c+ZnHcBlih0Zok3cOnqSmjaV78nODYptI/LoVMZl88Ein/50IvOiz+kxtDPjMtwJ5dWJjAu3lG222SbgUI3rCtFxjPAHtSwRtqji+yZSCv1wHcpmH2CRxM2D8FgrrLBCEKMhdQ0g+g5i+BdeeEHvK9I+9/JiSdAAdMWEv4JZQNyPO0JfiWlx55HdoDKDrJ5VJCCKgQRxVlEqvn34qmYJtwiJqRouuOCChvRc1h76rmeeeSb5DVpgirRuDbEkv9HeyGbuzMtmxr8dAUeg5QgQMxSfM8nYoD5w7BJgTMSPhND1SlBjZWjEkoQIboC+ByZnPp4wKAujBgNEv4JBAzslfO1gMER4YeFkkaVdsVAu1D7GLRg4QITnwqHbjHNoG700ep2+IPRZEri7LJYkjJuXAgiMidvJToiXMOIqGqFvh6FINvokrqVdq/cbg6a0vo0UQqR8Mh9Z2gN/YsKiv+PZPUnOvHoSXW/bEehiBNjR4FDN4gnDIfYlkUTIz0YUEMRhEAtvmpBOZI2XCGKN1SrXEPlyTv48DBggDFrI9yc+oLpzwmmeZ0G12hc9XmIFjEEV7VMGsTBjqMBOry8IAwgoG7hbdEFqdAGmOLIT4IFvduf0GaYHYTyBAQfiwlaShbaDSWFlmKbpp59ed4DMf0+SM6+eRNfbdgS6GAFEcJhwG4MxKCwifSO7maylobmzpJkdptiQRda35xb5zrZv4bhaYh1XpAOZOhahJcu8bEeKC444zutd7BoRMUK8GCDyZHcEk281oUMjaWs644c9w6wkre9W3upvN5VvNaLeniPgCCgCGBRAWX8iy95gUfC1UsE/WeaSd1u7ZzfI63OlMnReEP6naTIdbVYPRkg2iBB6iFRhemnTddMFEn4MAwwLQ5Zuu8gxIdUsm0G2vvXJXFGy11t17syrVUh6O46AI1CCgO1WiCtq8S6pQN48yERzelLwTxHmNahkNwASyy8o0X5KELJyRIVpskAMMDd8uSQKi36sDrkDIXZk7CobZV7oLsUs3pot+bYYtbYjLLnYwhMXG7YQTG/KEXAE/h8B9FJQ1lIPfQ2EUQVkFpTpwN2YY0MswEYwLowA7JqVZ78tCg3O+lCR9o0p5qVEwjiir8gyG6QDmNMXjDOI1oP1n0XloNx2VuxuSfFE0t70Z8UVV6Sa7sysrhbIn3rGSfDpSmQBsisxt0r31VvuzKtexLy+I+AIFEIA5oF5PKbaaf0WYdAQWZmVWjPZB6wjWAVCGGqg78EqznZ7Rdq3HQthk2CuEnBZ2xOHW7XgI6RSXxB9R+ya1d9Rhi4L5n7UUUclXUPPyI4WX6t6CBEghhdFDDtwZcDykAwfeYSuiyhCGL/0JDnz6kl0vW1HoMsRwKoQ03gsDjeQEEI4AiPqYjdmVmroSPbcc09N1EroICzl8BNC7IjOxsRQmNezQ8KsnVx46TBIEuBaLRBhWljc3X///UmcyCLtY5HITobdCgu4pcEhLBOBBdK7m96cUsSD6K4uuugi7Uf62VhFkqeQXRDuCLwowOQeeuihmnE60+1wjFgSRshLBqb51Yj0PrwkmAgyXRc9Jyb8xGnsccrG5Wg0qny2HT93BByBQQ8BeavXTAj1joysA8JQqmZPELFV3dkHZHenIYdakd2A8FOyS4x8p4lsAfVSK8ND8Xxh9FF2hRW7QdaNbDqiipUrXABDeSGoGWZKmFwUUWRuK6Q0kt1iJPNBljw8VI+zc3+AI+AItBoBoqOzO6hGjWYfoE3EaM1mN6CNPFEXIs7eIgu1lH4ez8fXbOeddw6SMzDX2XiSGlk30u3lHbOzZPcrefaqhpniXna/ZnSTbotsB4gS2fllrSOplxf9I31/vcdubVgvYl7fEXAE2gYBM8fu5OwGBiZiVESqGFvAsIicYf5diDIRYS699NKaOonIFq0krBJpm9BajZAkDw7oB9FnZtO4IDLGGIR0LejKeEloBTnzagWK3oYj4Aj0KgJYHWKQYZaLm2yySSDlBvH+OpVq6ZrIlYUFZ0/ERkXnl41EUg+O6B9hTul8dHY/8Sd7gpx59QSq3qYj4Aj0KAKDSnaDekGqJRqtt71W1U/HPGxVm7XaceZVCyG/7gg4Am2JgEWZaMvOead6HIHWCB97vJv+AEfAEXAEHAFH4P8RcOb1/1j4kSPgCDgCjkCHIFBRbGj5djpkHN5NR8AR6AUEMOXGKTYb8qkXHt1Rj8BqEKs6X0ebmzayh1eiwXAkS1/EO9pCo6TL/dgRcASKI0D0AaIdkKE3zwKreEte0xFwBE455ZRAMOAUPV3GvFIX/dARcAQaRIDsu2T3hYlZGKQGm/LbHAFHoByBp13nVQ6KlzgCjoAj4Ai0OQLOvNp8grx7joAj4Ag4AuUIOPMqx8RLHAFHwBFwBNocAWdebT5B3j1HwBFwBByBcgSceZVj4iWOgCPgCDgCbY6AM682nyDvniPgCDgCjkA5As68yjHxEkfAEXAEHIE2R8CZV5tPkHfPEXAEHAFHoBwBZ17lmHiJI+AIOAKOQJsj4MyrzSfIu+cIOAKOgCNQjoAzr3JMvMQRcAQcAUegzRFw5tXmE+TdcwQcAUfAEShHwJlXOSZe4gg4Ao6AI9DmCDjzavMJ8u45Ao6AI+AIlCPgzKscEy9xBBwBR8ARaHMEnHm1+QR59xwBR8ARcATKEXDmVY6JlzgCjoAj4Ai0OQLOvNp8grx7joAj4Ag4AuUIOPMqx8RLHAFHwBFwBNocAWdebT5B3j1HwBFwBByBcgSceZVj4iWOgCPgCDgCbY6AM682nyDvniPgCDgCjkA5As68yjHxEkfAEXAEHIE2R8CZV5tPkHfPEXAEHAFHoBwBZ17lmHiJI+AIOAKOQJsj4MyrzSfIu+cIOAKOgCNQjoAzr3JMvMQRcAQcAUegzRFw5tXmE+TdcwQcAUfAEShHwJlXOSZe4gg4Ao6AI9DmCDjzavMJ8u45Ao6AI+AIlCPgzKscEy9xBBwBR8ARaHMEnHm1+QR59xwBR8ARcATKEXDmVY6JlzgCjoAj4Ai0OQJDtnn/vHuOQNsj8P3334cJJpggDBw4sKSvgw02WBhrrLFKyqabbrrw/PPPl5T5iSPgCNSPgO+86sfM73AEShAYeeSRQ79+/cJff/0V/vzzz+QTY0yOKef6gAEDSu71E0fAEWgMAWdejeHmdzkCJQisu+66Jed5JzCzNddcM++SlzkCjkCdCAwm/1Cxznu8uiPgCGQQ+PXXX8Noo41WJjpMV5tlllnCs88+my7yY0fAEWgMgad959UYcH6XI1CCwLDDDhtWXnnlMOSQ+Wpkytdff/2Se/zEEXAEGkfAmVfj2PmdjkAJAmuvvbbquEoK/zlB37X66qvnXfIyR8ARaAABZ14NgOa3OAJ5CPTv3z9gvJGlwQcfPCywwAJh3HHHzV7yc0fAEWgQAWdeDQLntzkCWQQQDWKQMdRQQ2UvuciwDBEvcASaQ8CZV3P4+d2OQAkCa621Vvjjjz9Kyth5oQ9zcgQcgdYh4MyrdVh6S46AigfHHnvsBIkhhhgiLLXUUmHUUUdNyvzAEXAEmkfAmVfzGHoLjkCCAFE11ltvvUR0+Pfff4d11lknue4HjoAj0BoE3M+rNTh6K45AgsBzzz0XZp11Vj0fZphhwtdffx2GG2645LofOAKOQNMIuJ9X0xB6A45ABgGckSebbDItXWmllZxxZfDxU0egFQjke1QWaPmVV14JL7/8coGaXsUR6D4E5phjjvDOO++EiSeeOFx11VXdB4CP2BEogEBTLiSEh2qEDjjgAMJK+ccx8N+A/wb8N+C/gYZ+AzfddFMj7Id7nmp45wVTJb2D774KvF54la5E4Lzzzgsbbrhh02PH4OOnn34K119/fdNtDcoNnHbaaUFeqsMXX3wxKA9zkBgb6YMIqdYMubVhM+j5vY5AFQRawbiqNO+XHIGuRsCZV1dPvw/eEXAEHIHORMCZV2fOm/faEXAEHIGuRsCZV1dPvw/eEXAEHIHORMCZV2fOm/faEXAEHIGuRsCZV1dPvw/eEXAEHIHORMCZV2fOm/faEagLgRtvvDFMMskk4c4776zrPq/cGAKHHHJIeO2118pufuqpp8LWW28dll12Wf0uq/BPwYMPPqgBnYmNWS/hKrD99tuHBRdcMKyyyirh8ssvDz///LM2gzvBZZddVm+TbVnfmVdbTot3yhFoLQLvvvtueP/99wPffU3iYNrXXeix5//+++9hgw02CGOMMUaYZpppkud88803YYsttggLL7xwmHTSScN1110XTj311OR6+uC7777TYM533HFHqJd5/fLLL2GhhRYK8803X7jiiivC1FNPrTnm+vXrp6l66Ns555wT9t133/QjO/LYmVdHTpt32hGoDwHexGFem2++eX03trj2Pffco47ELW62LZr766+/wr///e8wzjjjKKOyTv36669hscUWC2effXaQiBJhl112SbIOWJ30N0zuww8/TBcVPr777rvDm2++Gaaffvow3njjhcMOOyxMMcUUgR3fk08+qXE2b775Zt2NnXXWWYXbbceKzrzacVa8T45ADyAw0UQT9UCrxZuEeZKsk0V+UKTzzz9fGcR+++1XMjyYEZkG2O0suuiiJdeyJxdeeKFmIUDc1wix42O3dsMNNyS3zzbbbHrMjg4i08Hee+8ddt9998COsFPJmVenzpz32xGoAwH0LyyevP0bPf3002GvvfbSt3MW1y233FIX14suusiqhGeffTbsscce4aSTTgrozWA+iKR4ozcmtPPOO4c11lhDxVPc+Prrr4e1115by7bZZhtt6+2331YdDPoYwlxttNFGutBzcauttgqbbLKJ1uvUP4jrYFoDBgwoySLATgiGNOKII6oe6sUXXwwffPBB7jAJ5Lz//vuHCy64IJB9uxFibp5//vmw0047JbczHzCs+eefPykj5NiQQw4ZDj/88KSs4w4ajYooMcSixDZs9Ha/zxFwBAoiIIwgrrDCCgVrl1e75JJL4owzzqiBU+WNWytITNI477zzapmIl+IMM8wQ11133TjKKKNEWTjjE088Ee+///4ob+1aRzJBR0n1EoXpRI5loYvC7LStP/74I44++uhaZk8XsZe2M+GEE2rRZ599FoXhaR3Ru8T77rsvUvbjjz9GWUS1ruQ9s9sb+hYdUhxzzDEburfZm6688kodm+iZSpraYYcdtFySlEYRJ8ahhx5az+ecc8741ltvJXX//PNPnY+rr75ay1ZddVWtB7bN0KuvvqrtCJMqa2bppZeOEl8w8uzeJhGlar+aCczbGHvvOBbtHXYEuhcBdktHHnlkCQAE1bYdljCj8Nhjj+kOgR0VYqeHH35YjQuOO+44vY/6zzzzjCr7qcubPDqTjz76SN/gp5xyypL2J5hgAjVasMKxxx47zDTTTHqKwQIGBJSNMMII4bbbbgu33HJLGG200ax6x32TIgoad9xxS/p+++236/mBBx4YPv30UxUJbrzxxrrrZF7MIOPggw8O0047rVoHljTQ5MmZZ56piVHRs2UJvRj6OHbFnUjOvDpx1rzPjkCdCMgbf9kdFtVbdkdh+OGH1+swKcgis1s5jEd2D3ptqqmmCquttpqKDV966SUtq+ePtWP3LL744moWbued+C07HO12lnmZiBCGBYHnKaecooxddreB+/773/+qAcWJJ56odVr1R3ZtQXbd+sKBiDBLI488shZZ37PX2/28fETt3mPvnyPgCPQYAkMMMUShti1TtBkBFLrpn0pZ5lXPve1aF50X9K9//auki+OPP75a/w033HBJObtWsm2jD3vvvffCbrvtpjs2dl9GL7zwgh6ib8QIg+96CX3lkksuqTuvvHutTyKuzbvc9mXOvNp+iryDjkD7IWBv61lxYZGeDorMi90oRN61NFGO6Tq7rP79+yeXbIcmOkYVwVKHj5FZAeJgjDi1EebFrm/11Ve3Jsu+v/rqKy1j592J5GLDTpw177Mj0McIiDFHwPReDEG0JyaW+uGHH/RcDABUrGg6HQqNaf32229ax/7g2Jsts2ud8m04ZP2zLKfb448/XjIUrA6xQEQci9UnusP0Z8UVV9T67MxsF0aBGFeonqqksQonK6+8skbyqHBZdXBcww+sE8mZVyfOmvfZEagTAWMqpoPhdrH201a+//77pDV8sSB787cL9957b7KruOqqq8Lnn38eCIFkYjLbVWDyjvGFWMuFb7/9NnzyySdqYg8Tw2kWuvXWW9VpFmMQ9DLsTnj7T/fDntsp32K5qebtuBykCX8tdHpnnHFGokfEJB7mha8VDKwoYVyBuBYcH3rooaq3gT0vF4ssskjFeuyexcpUI35UrNTGF5x5tfHkeNccgVYgQBgifLEgnFdxTmUBxXkWYleA3gXDAQsbRGihc889V6/zB53OzDPPrD5e+IOdfvrpQUzrk+vsMNh9wNi4jq4F51ji62HNxk4MYxDbabCow/AoZ9cFE+O4UwkGvOmmm6oFZ3YcRNVYaqmlgpjHqx8YzExcjRTzesaLXx0xCtEzXnzxxVVvBVN0WR9//HFuPTHTVwvTPCvE3BvasVCAbogEfPfzagg5v8kRqA+BZv286ntaaW0JKaT+OPh0yc4oisNrlEW0tFLqTHZuUXZZWiILberK/w65VxhnSRvUy6tbdnONgr7086JrX375pfrAyc4yt6fyAhDFpD4247tFG6y9jLUW4TdH/TwSJ2b1KxMml3e5x8ta4eflBhvt+EbhfXIE2hCBkUYaKfCpRukQVGbNlq5P5Aj8vNKUVy99vVOOsQokggm7XHapJia1/uOagC9XM4RYErErQXtrUSW/OQLz4luH6NHEvrXaasfrHc+82BYjCkHOvswyy2iomVYALa8e4a677tIfCe1jCWQK53T7KFkJdMmH8Ct77rmnij8IxMmPlX71NTGWa665RsdA/+olwt4U0Udst912YfLJJ6+3ea1veOPUKdEXNCxRQw3l3JQ3R9lq9nwWhfR8U95Oc5ntd0+fmxl1JfFTTz+/09pHHIoBikSvCPyWCdLbSiLALu1ipdgIEc0ex2XakKgojTTRNvd0vM6LhQkGwdsICs1WEQpmlNsE28TiJ49x8SycOR955BHtg8V6414UsrwlcdyXxEI866yzqgKdH34jtNlmmwURiWh8O97a0Fsgv+fDmyT/rCeffLJGLW+kfe6BSRB/D10K8vhWUt4cZduvNN/tNJfZPvf0OQuchDfSx/Aih17LnJd7+tmd3P6aa64Zrr32Wn15bfU4tt1220BElEaJ/1fWq/QOudG2+vq+jmdec889twb5bDWQOGtiOUVOnmqOmyil11tvPX281eObnRrWPI0G2GzFeFh4EQtgFdYMEerH/EUI6UO0AAJ78kHpD8PhOkysUQIndm49YbabN0fZflaa73aZy2x/e+OcEE6PPvpoYjV4/PHHh7HGGqs3Ht3xz8Aq0CJYtNNgWM+GGmqodupSw33peLEhIzem0TAKVW5kUa3FgOy6fdNco9v6Kl2p+xL9Ib8QlmXNUi2T3oMOOqglsel6ai5tbuy7Eh5cz9Zph7ms1N+eLCcSBB8nR6AdEehV5oXoifhdiNNYDNdff/0kphliI0x1SR+w6667qvkuJqa86fFGjpIXeS0fHCJJrscbdZoQ25G24dJLL1WTUkvLYHWqPd/q4Hz5n//8RzPOIm7DCz0rMkQ8ecQRR+j2m7eYiSeeWG+3elxHlElcMYkSrcwV81b0YogTwIC0EIyPXQ3BUNOycUyXTzjhBBVX8qY011xzqQKYIKaI7FpJYE4aBkykycDaCHE/Oy+ToeMjRGQAfE3YlSJSRCRIPDxEKohhmSPGiZ6Q6AGGnT0fkS2BYwmhQwBR9G7pt/5ac1lrjuw5teY7by5xKkWHyHyg20CHQNoJxGpp83Ge0ZtzaWPyb0egKxAQptEQ1Wsqj3msKPOjOOtFkblGEUmoCa7oUPT5wgyibLO1jNQLK620Utx66601VYIs8JHw/cstt1wUfwmtI+KrxNxWGJqWyRtyFCYXJeFbFHGZlgkT0PZrPZ9KwjijMNUoTEZNTCUStLYh1lDaBn/E2TPOPvvscdlll42yOEcRDUYREWi9o446KopOIMqCrWke5AeUmMUeeuihURY7rScMKYpiN0ruHT2nLSOJ5q33yuKv7VsdMKDdRkj8bPQ54nFfdrv4/Og1eRkou5YukAy4Wo8UF8KI9YMprujRojCVKExQq4OH9Zm6zOWOO+6o3+AhkbQj4xc9WpJGI232K1Za+hwJn6PzyHxyn7wgqKk2D6k1l7XmyMZVa77z5rJIKhFrv1Vz2Zem8jaWTvjua1P5TsCoXfooL4X6f91MShQU5Q1RvcwLZiS6kuRZIktPFikrxPeAhUpk61YURa+iZZIMLymTHZuW4VMCGfMSq7CkjhgqaB0YIotZrefLDkuZp+wMkjbwkRDT4JhmXvvss4+2i7+LEYyJfsO8jGS3pGVpnw4bixiBWDVd+HmGEYxMRGdJjh0bB4t9o1SNecGAxLxXmW619o15MU4JNqof8hPJzlPHacyLNsCbejAe2ofIXUQZLzCyM9Oyd999V8tkx6Ln/IF5yS4sPvjgg1o2cODAOM8882g9XiagWnNZZI6KzjfPy86ljUV2hFFi2VElnnbaadpHiRqh5/xp1VzCvMAE/PxTHQP+dxyj6hi1Ez7NMK9eERtKwjkV96E7sPApiPgwq0a/IAuUytZNwWlxwgTkMPXUU/NVIi5DlAYRyiZtNUMsL6Pll19eRXJYIxIUE3Fjtedfdtllag6eDqeCqTt9sT7TNsYJRIq2QJyUIdaD0qKvaikorP/cQ9QBRFd4xHMP4jCwISoBz5CFm2pl4Xq0sAV/8AWpJ4stimj6ZsTc4tOSJkTCYAFO5mvCXCPuJQyQWUtNMskkOu8WksjawF+GyAwQmNA/ckgh4izyWyoyR0Xn2/qgnfnnT5FUIlRt5VzOMcccKk5P98OPSxHAzYEIH4j9ndobAYy7MPhqhnqFeb3xxhvaR/Qq6fTURTqepzA2axlCylQj9CmEubHAltWeb8nkWJzTxCJsTAkzYfxeFlhggXSV5LrVK7lY4yRroEDoGPQkmLij+8NiESK8TDsSjApcTd9VrY95DB2GVmQeaRdmUOu3VHSOisx3tbFkr2XnkeutnEscXokX6FQZAX4f6Lwdp8oYtcsVNiwdwbzMg15CzZRhJ6I5fZvGBLvVxMIK2S6p2vPNzBsfCJxS88jqoLC3nVJevWbK9tprr8DCyjdWguw4iDsn6debabZH78VQAYLpsptq9VxaVAfSb9T6LZkzba05srmsNt/NgtaJc9nsmP1+R6C3EOgVPy9ERlhmsaUnQZoRfkhYo6UjXds1+xbVgR3W/U2kBvyGxICj5vPNig9nzDSRgoB+QlgGIgbDAo006UbUgayelTfyTdvs7rBK3GabbdSikRTujezqij5fdFQ1q9aaBywMxaCmzMw83XCtNtJ108dENIcw+6/1W2IuisxRkflO96GR476Yy0b66fc4Ap2IQK8wLxxlMXdHl7PwwgsHMfYIpENAryFKfY3UAHiffvqpYpgORcRCDqXz5FgSNUyx04Q5thGxuzDJFgsk9XSv9Xx2D5hiw5TY6RDlAbN9dj7sDo899lhNA8E1CEdddFVE0jAnYMzfMe2GjCGkGXO1FBQ2TqI8I+Z8/vnnlXHB8BmLXdfG6/xjfchrAzNvdIGY61cjsbLTy2Cfzp4LNrwk8IKAjmvMMcdUnGBU6XlkIadueh6JkE15dh4pt3qIF9Bh8FvhRafIb6nIHBWdbwadnctq85hOJdITc1ltjvyaI9BVCMgi0xAJA6orqryIadTaUAw01BpIdhJqLi0Lmj4fKy0zlRcjiSh+UvHwww9PyjC7xnxcfIaiKPS1DREHqqWhLMpR/IwipvLimxVF16VWaw888EAytlrPp6IwDDXjlh+AWvxJWgc9F3+ySP8wEZc3+yi6qMQUHzNxrMEwscdqTpiNmsEzPtrBUg5zctk9RVH0axkWdSJi0/FThw+uA1jA4UpgZelvxpYeTzKwGgeYwou4TdukT5jbC0NO7sKKk+cwn5VIUj0k80Bd2sECU3RYJX0FF9FJRdH1aDlzjZUk5uVmPs/9WAsSXXuJJZZI7seClN8CcyxGLDrHWI9iVk9Ec/ud0Mdac1lrjsySsdZ8y4tJ2VzKC0qUsFjJ+Bgzrh9m0s9v2Nw/WjWXbipf6ZdZWu6m8qV4tPOZvLTq/1Az1oaDMUBZUOomMVsOV155ZbA38qINkCabXQ3iH9NJFb23Vj2GguUaYiPTk2TvKfJ8EugNP/zwGt4FA4C0c6y1h84ES0Z2juwoZEFP9DFWp5Fv8YlSC7sNNthADRRomz4TU5BdzQUXXNBIs1XvYTeFhV87EVaF9AsdGtjmUa25LDpHReY77/m1ylo1lyi2GSs7e6fKCIi7gkp1+J91qh8BYwWV/t/qb7HyHUhUsNolUIO4lFSuWPnK071ibZh+ProvcVxNF7XsGNAnkcWuGhV5fjqVQR7jon3EV2aZCKNrBWHiTWQNrKZgJmmTeszL77zzznDwwQfXXMRg3IjyilK7MS76zYtNrZebWnNZdI6KzHdRLK1ekbm0uv7dHggg/h8wYEDJ/x09Y6ElEzLGPSzwJNvMM+qSHb2K3wkSng0xVmSERONBVYF4HzE8EWnSbkNF2kDETQQf+oIFLGoaLLyzlr7oholsk7b0RV3BC0A2bUve+MkcTcBu4l3yUt0X1OvMqy8G2SnP5EcC8aNYccUVNewUFodY8aHPwzcJvWCaqeWNzfyQ8q55We8gUGQue6cnzT+FBbsn38Z7uv1aCLBD538On8rs/xZh3ghFxwveeeedV7awW9vogdkho6vFcKte5kUeMCRZZ511lkqm2LXDTDHcEjG7PabqN9IKEc+rrp8Xanag+L7xwegp3SfCt6HHTxN65SzjqjZ+mCuYSQCDmpuG9HNadiw/nIaoXp1XQw/psptkwYsSwy9KpHzVuaFTIrwS+jYRG3UZGp093FbOZV/qvMTIKorjfhLxpdWz0sr2G9F5oR8lnBy64Syhv5aFNqJ7pV41QudOXT7pqDrV7klfQ1cv/mlJ0cUXX6xtoRcuSoyB8Gvok+gv+lmLgCP+byXNoHeXl+FIlCL7CPMrqVNk/ETnwfZAdnwl99Y6aYXOq9fCQ9UajF8vRYDJlTe40kI/60gEmp3LvmReFvuykQW5yGS1sv1GmJdYsqqxlVi4lnRXUhppaDgMssS6tORa9kT00GqkRLzSRpkXTAqDLsMZgybawjioCLFWSBSWmGVAxIOlHULYGYn6QZ+FcZRYZFtxyXfR8dNfsV/Ql+6SBmqctIJ59YqpvIDnVCcCRBbpSVFNnd3x6k0g0BdzidjymGOOUR0OGcbJoI1hihGp6tdYYw2N8k8ZUfGFSWoZ/oUQTtb4GEK4huy99956jK8m+hKxVNWIFvK2r+Iq3C1whyGMl7VF9H2ILAw8jw/XoUrtc10Wc9XbaMUe+oMxFNkK0HOZ87s9iowX6I8kOLj+HzJmWXDtcvKNWJ+sChhSpcVySYWCB+i7nnnmGQ2hxi3MB5SnW9MLOX+IJoQeOE39+/fXUwl0nRTzu2AszAU2Anybc79VKjp+IuTwezFXIru/V75rMMiKl11sWBEav+AItBSBendeuI7gboI4TJTwkej2EgNUXUk4hnhjlpBe+lZunRV9jWZxwC0FwqVCYotqHTEA0nPxbYy4jsjiFMWISMXaEv1FjynDrQFC1M056wTEzgC3CcrMlSCvferabqxWpgPqpqnenZfomLQ/suinm9HsBcKI9BrBpw0nMQAqCb6NaE4MF6IwHr0fsR/js91TSaN1niCG5NkWxLrO25PqZHSg3+kdFuMWv1cNOG3jxBVH/EH1PvHP1N8BY6k2fnuIxI7Vcad3d3at0ncrdl4uNqyErpc7Am2CQL3MSwIZqy8eugwjUdjrAoPPn4Q202J8EFmg0oSYzJgX5YiiRAKQrhJhYNyHP6WJtsmyIDtM1dXCBG+//XatY8yLBizyvjEvyvLah/kWyXTA/Wmql3nRN8Zhfn/WlmVQIA2PLegs0PhyUp+USZDsuCKM26hVzOvzzz9XhiO7Vmu6oW+YkFgCql9qpQaYS9ItMS4JZq7Vio7f2kTkyv3pjBx2rdJ3K5iXiw0FdSdHYFBBQBYLDUaNmCidcQFLMizQRJcRXnrppbqGmxVfm2sIIbbsGvFDsT5DbNhs+/hpkkmgp02wLeh1WqQGMBapB5EbWRAg0T2piJFjSWmkok8s9jBLbzVhLo/vUzpLRiPPQCTKHFWLnoMpPSJRxKZYDcrLSKHxp/vDvQRLNzzT13ry2E3lexJdb9sR6GUEMI/GZNrSC9nj0U2IFauaTBOZP5uF3OrlfRuDyruWLjO/R8zGLe1N+nql46LtV7q/0XJ0XhD+gGkiXBqU1YPh2wSJCC5I1JkA08Pv0siyV6APxLSe70YIs3b0ic0QWciJ04pvGnNfjegrjBoneEKfFRl/tj3cc/BP602qPqre7Ik/yxFwBJpGAKbBYkX0F6LNiOgraZPYk1C9Uf+LMhd78yb6P9FRilLR9ou2V7SeZZsgekmarJw8gGkyZ3YWd3Yo7GL5GFlcS3Zk7B4bZV4ERiCgeKMEw5LQehpn1RhRrbb4TdBnGHKR8afbw08O45YZZpghXdzjxy427HGI/QGOQO8hwC7CItgQZSFNEmNTQ/LYdXsjZ+GBEDki9mNhNoKxUCbGCVZU8Vv0JyqqJCqEtZ0OzmxtFGnf+lTxYS24YNErLAi0NcmuFHEb1n9ifGHFSV5A8vmRcocXhPSHwAIQOzPbhdnN9Yzn3HPPtdtKvsEvz+IxXYmA3ptvvnnAudiYLdeJ+mGWn+n6HDO/7LpI4Mt8Fxl/ug0LVN0Mw023V/TYmVdRpLyeI9AhCIilny5CEsRaQxvRbXZh6HIwmTeRnplRo19isSOJIxH+MaknqSNMhgUQpkZsTRZVS+JJm4i3bNdC9gMxNNCoEDBQRGyE97rkkkv0PsJlmYhNDB60P7SR137RTAfc3wzRR8zb05nSaY8ydFmM/6ijjkoewQLPDoXoF/UQIkAwFz+wmrfhUkB4NzJJpAmmhVgWvB566KH0peSYmLFkgsc1A33Xuuuuqx/mmWOukk6kCwAAQABJREFUEbED5oS5vL1Y0D9EpIR6guodv+24SYnUq1TJGqRWOZY6RP92cgQcgZ5FoF5rQ3pDRAUszYjQgrm0vBWr2bq8ZSedxSoQk3pZcNTCUEITqfXfQgstpKbumIIL81FrQ3kjjxIjT60LJalrco8sqGoujjm5MKikbQ7EUECzD8guLIqvWcQhmD5tscUWmmmBOnnty2Kq7bPG1EP1WhvStuxS4vTTT59YTaafJzvJyPjAgygbYEnWhEqEtR1YZk3lMSGnnI/4U1W6XctxD6AeETbShPOxiPX0Gi4HWRLRXRSfreQ59jz7JvIGhBUoztCUYz05++yzq3sDGTOyVHT8EhZLcUr/trJtZc9bYW1YaiebfUKVc2deVcDxS45ACxFohHnxeJgPaV3E6bdqeDFM6mWXoT3ORpqgUAJFR8zXjYx54dOFOTZm8pUWLhGXJb5KLOosslnKts91yuqlRpgXz5FdURSxWsXHvfvuuzWjbFS8+Z8LEpM0StzAmmGmmAcRReY2JwYm+gLCOJsh5ox0QIzL5r1ae9XGz7jEMVpDUVVrI3utFczLDTbkFcTJERgUESCqeBEletqkPmthBy7Vsg4g4qqUfoh705kBTA9GeZry2s8rS9/TqmOeQ1BcIo5IvL8SPZE9gygUzRB6M8S1suPUSO/V2kLnZIY12XqIN9FdEWi3GWK+LJN4kXYqjR8Rs+Q81OgpK6ywQpGmWlrHmVdL4fTGHIFBHwEzic6GFOrUkaOLwmJu6aWXDiJWCxJVoqVDwSKRttEpNkMY3NC/ohaEzTyr1r3i6K6ZLwgjRQitviBnXn2Buj/TEehQBFg8zWABPyLJHq7xDyvlveuUYYq+Sv3geiKdkNgGBD7N0rbbbttsEy27n100Oc6yKVRa9oACDTnzKgCSV3EEHIH/IdCvX7/w6KOPlsDRDjuBkg41eGJO1g3e3lW3IZLuS8YF2M68uuon54N1BJpDADNsPk6OQF8j4H5efT0D/nxHwBFwBByBuhFw5lU3ZH6DI+AIOAKOQF8j0JTYkNAoRJJ2cgQcgZ5D4PHHH9fwTP6/Vh1jIkwQhslxqo5TO1wlJFWz1DDzwnoGG38nR8ARKEeAIK345ZCKhHQRzRDR4J1qI0Bsvd6Or1e7V14jDwEMPnAdyKajyatbqUyzzFW66OWOgCPQGAJ33323prKHiVkswcZa8rscAUcgB4GnXeeVg4oXOQKOgCPgCLQ3As682nt+vHeOgCPgCDgCOQg488oBxYscAUfAEXAE2hsBZ17tPT/eO0fAEXAEHIEcBJx55YDiRY6AI+AIOALtjYAzr/aeH++dI+AIOAKOQA4CzrxyQPEiR8ARcAQcgfZGwJlXe8+P984RcAQcAUcgBwFnXjmgeJEj4Ag4Ao5AeyPgzKu958d75wg4Ao6AI5CDgDOvHFC8yBFwBBwBR6C9EXDm1d7z471zBBwBR8ARyEHAmVcOKF7kCDgCjoAj0N4IOPNq7/nx3jkCjoAj4AjkIODMKwcUL3IEHAFHwBFobwScebX3/HjvHAFHwBFwBHIQcOaVA4oXOQKOgCPgCLQ3As682nt+vHeOgCPgCDgCOQg488oBxYscAUfAEXAE2hsBZ17tPT/eO0fAEXAEHIEcBJx55YDiRY6AI+AIOALtjYAzr/aeH++dI+AIOAKOQA4CzrxyQPEiR8ARcAQcgfZGwJlXe8+P984RcAQcAUcgBwFnXjmgeJEj4Ag4Ao5AeyPgzKu958d75wg4Ao6AI5CDgDOvHFC8yBFwBBwBR6C9EXDm1d7z471zBBwBR8ARyEHAmVcOKF7kCDgCjoAj0N4IOPNq7/nx3jkCjoAj4AjkIODMKwcUL3IEHAFHwBFobwScebX3/HjvHAFHwBFwBHIQcOaVA4oXOQKOgCPgCLQ3AkO2d/e8d45A+yPw/fffhwkmmCAMHDiwpLODDTZYGGussUrKpptuuvD888+XlPmJI+AI1I+A77zqx8zvcARKEBh55JFDv379wl9//RX+/PPP5BNjTI4p5/qAAQNK7vUTR8ARaAwBZ16N4eZ3OQIlCKy77rol53knMLM111wz75KXOQKOQJ0IDCb/ULHOe7y6I+AIZBD49ddfw2ijjVYmOkxXm2WWWcKzzz6bLvJjR8ARaAyBp33n1RhwfpcjUILAsMMOG1ZeeeUw5JD5amTK119//ZJ7/MQRcAQaR8CZV+PY+Z2OQAkCa6+9tuq4Sgr/OUHftfrqq+dd8jJHwBFoAAFnXg2A5rc4AnkI9O/fP2C8kaXBBx88LLDAAmHcccfNXvJzR8ARaBABZ14NAue3OQJZBBANYpAx1FBDZS+5yLAMES9wBJpDwJlXc/j53Y5ACQJrrbVW+OOPP0rK2HmhD3NyBByB1iHgzKt1WHpLjoCKB8cee+wEiSGGGCIstdRSYdRRR03K/MARcASaR8CZV/MYeguOQIIAUTXWW2+9RHT4999/h3XWWSe57geOgCPQGgTcz6s1OHorjkCCwHPPPRdmnXVWPR9mmGHC119/HYYbbrjkuh84Ao5A0wi4n1fTEHoDjkAGAZyRJ5tsMi1daaWVnHFl8PFTR6AVCOR7VLai5TZv49NPPw0PP/xwm/fSu9epCMwxxxzhnXfeCRNPPHG46qqrOnUY3u82R2C55ZYL7O67kbpWbHjTTTeF5Zdfvhvn3MfsCDgCgwgCH3/8cRhvvPEGkdHUNQwXGxKTjvCO/uk8DF588UX9tb/66qttOX/nnntu2/RrxBFHDOecc07b9Mf/35r7f3OpUQhubVgXs/fKjkBxBDbccMPilb2mI+AI1IWAM6+64PLKjoAj4Ag4Au2AgDOvdpgF74Mj4Ag4Ao5AXQg486oLLq/sCDgCjoAj0A4IOPNqh1nwPjgCjoAj4AjUhYAzr7rg8sqOgCPgCITEatOx6DsEutZJue8g9ye3EwKYbC+22GKBTMi33HJLO3Wtrfvy1ltvhYsuuigceOCBJf0cOHBgOOOMM8IjjzyiC/yWW26p+JZUkhNw32yzzcLCCy/cUOzHq6++Olx44YUaemvRRRcNq622Wphxxhmzj6l6/sMPP4QTTzwxPPjgg4EAyvRlp512CkMPPXTJfX/++WfYY489SrIFvPDCC+G0004L0047bUldTp566qlw3nnnhffff1+d1I8++mgd6/HHHx/GHHPMsvpe0CAC8iPqSrrxxhujQBbFz6srxz8oDFr8vHQOxc+r4eHIwhTHGGOMKI6eUbIdN9xOK26UIL6taCa3DfHziuLnlXut3kLxMYqLL754/Oyzz0puvfnmm+Pkk08e55577vjKK6+UXMueHHvssTp3e++9d/ZSzfOzzjpLny8RTOKdd94Zp59+em3r2muvrXmvVRDGpfcNP/zwcayxxtL7WQ+EgZX9DoRJJ9epw2fBBRe0ppJviWEZN9988yhxLKMwrPj7778n12644YYoIcPiu+++m5Q1c8Ac0A9xUm6mmU6+9ynegLqSnHl1/rS3gnmBwrfffhu///77PgUExrnIIotEmGlPUKuY12233RbHH3/8+OGHH5Z0U0Jg6WIqu9iaY5DAxfFf//qX1m+EeUnQ47jqqqsmz7/44ou1rSWWWCIpq3Ww++67R8m9pi+vYH799ddHSSKq7bA2pGnmmWeOl112WZSdVPL58ccf01XiL7/8EiWmZZTcbfGee+4puWYnZ599dpxqqqkijLNZcuYVn3Kdl7y+OHU3AqOMMkoYaaSR+hSEvfbaK9x3330qTuvTjlR5uOwkwtZbbx1kdxEmmGCCpCYixI033jjIDiZccsklKoJLLmYOiGhDihhEaI2S7JSD7PIC4jxottlm0+/vvvtOv2v9EcYRhMGEM888U+MCIjJcYYUVNO8a91rkFo7vuuuu8MYbbwRS3XDfRBNNpJ8RRhiBywltscUWgWwC++67b0CMmUcbbLBBkJeUcMwxx+Rd9rI6EXDmVSdgXn3QQoDFFP0J0d9ZWCAWQXmb1+zHssMIJ598cujfv3/YaKONgojKtA76Euqw6L399tvhgAMOCPPPP3+QHUFAHwL997//DWuvvXZYY401wjXXXKNlRxxxhJ5TxnUIxnXkkUfqMUxAdiN6zHXZTahORgv6+M/pp58eROwV1l133ZKerL/++gE8YGws8s8++2wA1zzaddddwzLLLBP69euXd7lQGfP1zDPPhCGH/J/K/vXXX9f70F0WpSuuuCJkGRBzDI077rhJMzAaxsJ8TTLJJPpNPME03X333ap/IwTX9ttvr8zvgw8+SFfRY/q7zTbbBBGZhs8//7zsuhfUiUCz29dOvd/Fhp06c//f72bFhl988UWUxJFRFhUVF/3xxx/a+KGHHhplYdOyaaaZJq6yyipxvvnm0/Nll11W62y33XZRonlr2aSTThrXXHPNuNBCC+k5ehT0MdBxxx2nZcLc9By9lhgqaJnpoB577LE49dRTa5nsviLnEKIt+XeOsiDqeTN/WiE2nG666aIYKJR0A3ErojL6Oc4448TRRx9djxELHnXUUSV10YnNPvvsqgt6+eWXtV4jYsOSRuVk9dVX12d/88032Ut1ne+4444qznzvvfeS+6688srIXM8111zJOGWnHoU5JXV22GEHHYswbu2HGHzo+ZxzzhllV5rU4+CJJ57Qa/zGmiEXG0bXeclbVTO/Ib+3DxFolnlZ11mYWHyNeVEuoi0tO//8862aKvZFvJicy+5B69xxxx1JmezOtAzFPXT77bfruTEvysRKTcuMeVEmKVQii1+aMABATwKTbZaaZV5gg05IRGIlXUG/A3aS+iVZ0FmgeR7lMCwI4w7ZuUQzrmkV85IdjDIc2dmW9KveE5iwWAJG2RlWvPX++++PIi7VcUlGiqQeLziM9aCDDtKyn376KcoOWsv4baUNgX7++Wct52WnGXLm5Tov+c05dTsCWdNo8MB0HpKFSb/5IzsPFY/99ttvWiY7LP0m+aTRPvvso4foP+olRG5pGm200cImm2zSFubV6LWEgZWI1Ojra6+9pl1G5DbhhBPqsew4wn777afHwvz1G5Gr7CRL8NQLTf7BXF52wyribaYp+jvTTDOFww47rGIzmNIjEiUrNjoz2UVrXRMRIvKF+F2ccsopAd2cMPJA1gMj7pWXgJIyu+bf9SHgfl714eW1uxgBFPu1iOSTIkZTvVmtutnrWeaVvd6X52JNp48XcWBJNzB2gViU0zTvvPPqqYjggpiwB9mdBjFpVwbGha+++kqvy85UmSK6QhJ41kv33ntvOOmkk+q9raQ+/moYZuCbZnq0kgqpExgSjFqsE1X/SS4tsb4Mb775ZgkGJIjkpQZ9GBgwdiNejGRXbaf+3SACzrwaBM5vcwTyEMAyjTfyKaecMu9y1bJ2Zl6Mh/6JSKxkDGL6refsMNJkCRJhbhg8iD4sXHrppUkVsxQEL4xgcPZthHlh4TjFFFMk7dZ7AMM6/PDDlckYI67Vxthjjx3YFZthBxjAvMDAjD5ow66n28ViE+OWGWaYodZj/HoNBNzasAZAftkRqAcBzN0hLOoge5MXnYqe88cWbhM7UQZjwNrRrlEGsdC1A2FJx64S68s0YaaOuA3rP8SKRmZxucACC6jF5UcffRTSH6JaQGIMoeWYkRvVM2YSflaiWu08//zzavZPZBVjtrR16623Jtah2baZI3ZdZGG3lw3L2/b444+XVMfkHtzAx8isVZthuNZWt3878+r2X4CPP2EQprsAEltk0kyHcD9QVuQjlqtaLsr4gI4HPZnpPxCfsYDh/8RCi7n5wQcfrPXxVbI2WTxFga/ZjqknESrUD4m39mp6GG2ol/7AiF566aXEpYDHIiIlxBKMWKwLk56wwLNDEQu+pKzIASLAUUcdNYiFZ83quB3gnyeO02V1a7WDDk+cwtXPC30X5v982DnxzTXEnTBnzOXtd0C7iEjTfmr0VSKOaFgsMa7Rvoi1qZrM4/bA/BuZ/mu55ZazIv9uEAFnXg0C57d1PgIwqwEDBiR+WfhkPf3007oIo6uAdtttNxUH4ZjLggdRL83A8AOTkEjKtDD+EKs0VcpTl4Vuzz33VHEbbfCWj5iKGHeIlUz3Q2w+3uSJBSiWeCpGQ8QEQ0vvaGizr0is6XRnyKKeJny20BnBdDFqYCHHYOWhhx6q2/kbsSSMkGd88skn6ceUHfOCwQuDmMiXXavWDnjiPyeRVXS+8dezD+NYaqmlVCyI4QUGKfimYYyCWBPxIEYbaVEgD7/pppv0PoxV+E2BgViY6u8n3TmeI2GimjYwSbfZtcfNmGt28r3u59XJs/e/vrfKVL5RJEQ0qGbPn376aRRH5SiMqGJTIsKK5ocki2dJ3Du76csvv4yYx6eJslaQvP23JLahWFOqz1OlPhG7T5hKpcuFysETfzIRoVatL0wuiiiyYp2i7VRsQC5gQi/ixci4eF4tIkwUcR2Z4yzRH/wHCUXVLLmpvJvKd+1Liw+8dQggOuNtWhx0KzaK6AhxGIQeDHPpLGHJhiFAmihrJ2IXidVlpRBHk0gUCsSFjRK7Ip5BuKVa1p3sVLH0y6N62sm738oQS6KzYlym47Jred9YEmJ8YrpOq8Mub8kllwyIOonK4tQ8Am5tWBBDRAWIMtBTEErI/FgK3t5r1eiXyeftocjvV1xxRTst+0ZMxT8V8nhi1kk0CV1cEZuQtgJxGaKxIoQ4BkW4yf65B/+etEI82w5iNNMxcQ09CYtFu5OJDgkXhNVbNxBiUHRM/A+wmO+8884tHTZiuaWXXlrDbDXTcKvaaaYPdi9+gfz/EUaLEFpOLUKg2e1rp95fr9hQrKM0TJDAHkWWXXjYRUQNhRsrUJFo33jv008+EsMt4tVficQqLBLOZp555omkbTj11FMjYW24V+Kw6W0WJYLoCkQREKW13kMdicsXhfFF+cfUCAySDykSYUCMEpI+7LLLLpUeryIZ66sEPY2SC6mQeIYG+0psiDiL8crbtY5RlPpRDDUqjrEdLrRKbGhj4XdNdHin2gjwe6mVIqZ2K6U1XGzo4aHqyuclynZdrIoyL8LC9GSai9Kf8/+fIVOHIYj45v8LKxwZo+OfwYh/NrGGiiuvvLIWXXfddWX5i4hjxzPE+sxui/vvv3/caqut9Fx2a3qdOoRUkmC3Sb30gZgZJ/XEyit9qeZxXzEvOkYalfSn3cOMtZp51Zwcr9CjCDjzcp2XrK3FKSvHrnVnX6W5QL8CWfiiav00CzpRSCfVTKdh1l5EC8BSrhZhYmw6CHQFtIM4DX8b0k9kCdGi5EkK//73v/WS9Ttbrx3PsTZLf8DIyRFwBHoPATeVbxJrnDZlx6ELMCFuRESmLVZKc4FZL+nDJaq0Onxiio3DIya0OECSXgM/IOK1oQuS17ekhz2RIsOcaTHRJtWEEVEDzG8H02HMw2uRZNHV9B5WD+X1tttuq6f4AmH6nSZ8njCt9mgDaVT82BFwBIog4MyrCEpV6rCoY22GEpsgqubxj0WRpLnQO/HAh0HhM4JzIkpbdiKUiYhOmRi7FknPoUyLSASSFkMZAQu8Ec6w+B9l/WzseiPfm266qVq44RcD82U8Zjwhqc4babLkHhEj6g6QXRyOukYwMsbWaoW/te/fjoAjMIgj0KOC2TZuvF6DDYZC6gv5OSQGGxJwU88POeSQZKT4/hjlpbnA34c2yAFlPj2mH5KdS+ILhF8J9dAJGVG/SIoMS1Mh5tt2a9Vvck9hsMHz+Ig4TFN5VLspT+eVrk8aD3xaIPJR0S75oMyABQMHMUHW66Sh4Lrpy7SwwB/TeVm//ft/85eHg7xgJfObd93LKmPXztiIpWuB/5RBsspTbiovv8xGiZTgwoQCaTAQHxKBAB1OmrK+Ieh1zD/FfHoQt6FPw4vffIEwFUePYuGDaNNSZKTbb8UxYyBA6RlnnBH22GMPjYiOOJHdoYU5auY5mL6LFaOGPCLCBCLRE044IYhlYjPNJvcSqsd0bUmhH5QgwK6e+IGVUtSXVPaTtkcAKU67uuv0FnjOvJpAGiaEmFAs9nShh3GxSEsiw6TVLPNKLmQO8nJKwdBwtuwNQvSJiA/Ggg8PgVZhyog8udYMEdBVst2q2PDoo4/W9OukQQe3VhA6uXTerVa0Oai1wUsIoYsQDTt1PgK8bHY782puVer830BTI8DAQsRt6pS71lprqVWd+P8EcgwZFWVeVr+3vom/hw5KUpiXPJLdpOnZ0H0R668VRHw4iGji6AYx5MjmhmrFc7wNR8AR6A4EnHk1Mc+Yl2NwgDk4jEDSumtQUSJqQzAuGBxGGbVIpNK1quj1WmkeqFSrLSJDsBMipBGWj0QjSBPhcGwnmA1AavVEd6WHlZ41cOBAzeNk12eeeeYk1xFMkfA/RrXasnr+7Qg4Ao6AIeDMy5Ao8G1+T+nQRxLBIrHOM7Pz+eefX1vLS3OBuIwFPR3CiWR9ZKpN50oiWjblxEQzQgcFM6mVIoNwTxCMzjLg8kzOsVYkXBR6LtI9wFwRFdo93HfWWWcFQtqgH8lLqpjulwQb5ZYyQiZP26T2MMItAEIUabo9zq2NdF3KnRwBR8ARqIiALGpdSfVaGxI2SeL+qcUW1niERZIdi1rmCSPQUEqie4linJBY1MluLMruSz+ETXr99dejpErQNrD+kriBUZhGlFiCiSWY6Js0lIzECEzKCEVEtGrCMslEJtaOeRMnpu9x5JFHTu4lhJE4K8estRnRMCD6Tr/FGCRK2vKIxSPPIPJGXpR0ESlG2UUl7QuDjqJP0bbsj4gEFRfakcCyUfRcdklDT2FdCYkxSpSYb0mYJfoo5vlantxQ5cCsDSUmY5VafgkEPMLGoPU78Agb8anBmNKKnG0QvkD+Hfys2EU0Gh0BcRciQXYYBGdFDJc1biBfE2VmWdgspLTXykjjpGHHIZmfAWJQRIrstiqJC5vtfyvvJzHijDPOqAGF3WCjOrJEPMHCc6ONNqpe0a92BAIYbJAclHWnWtDrjhhMY5182sWGjQGnd8GUMDogkgRpvbOMi0p5aS6aeGRLGRf9gHFB6OdgvlikdQLj0k77n15BgBBiRJHJIwmkrE73iJ4rRUznxQhneKLINEKI6cVXMOA0T9biyy+/XJNQNtIWfSGSDNFr8ujHH38M5513nlrCIt6+5pprynTIiN+JgkMGBSxdaQsxezXCUIm6pt9FnI9VsuRrq3abX6uGgExmV1K9YsOuBKnNB90OYkNZjHoUpVa136jYEPEUom4xsikZJw7zEu8ySooUFQtLxJSS6+mTY489VsXMe++9d7q40DEZESRSTRSGFXHIlVxf2hYBAKo9M69xAlabyNuCTqfrkUByscUW0zpiZRtvvfVWFXvvu+++STVhXHH66adXUbwYamlfZH2NEuYsigQmqZc+IICz+HBqXZ5hRBYHAgmIxMOKCn+72NCjytcVVb7wL8sr9goCfc28xPgliq9Nj421le03wrwk5FkU5+9Imp00oX9FPyqShkg0l2pE2hSRTjTMvFjgeQ5zbSRSDm1PRGdWVPMbxgIzEnWB3pvHvIwx8mJrhG4Y5kRaIEhy00Vxi9F1g+wLMERJLKp10vfZ/XyLZa9ep5008+IaEXNE+hFhivWQMy+PKl9tU+rXHIGKCBD5BN8+9J09QT3dfq0+E3sSMSDZBEhQmibcHIQpBdmRVI3YgT4Z0RgRUBolxO6I2oSJJU1gJQtJmp2krNYBIn2yF0iIstyqzKMwkjDuuOMGsxqmIj6JENFnhLkEYdYakAA9OVkTiGGKOBASBqvf6T8XXnih6pERd+YRUU94dqXM1Hn3eNn/EHCdl/8Sug4Bgh5Los0g4jA128fdwQi/vTXWWCOJ/iEWohqsmDLugYj8j/4FXQw+fRhBPPnkk+qKgF6HBY06kvct4DZBVAtJ+qn3NtM+DRAFxRZUbbCH/uCojgEPAaPThKsFCzJhztBDsWB/8MEH6SrJMY7pMIJ+/folZfUekNWbzNzpUGLMCczDXFLqbTOv/hNPPBEwhhKRYInuGsYt1rtBdm1qnMVvRWJ2ljTRv39/PYfxpUlihqqu8IILLihpM12HKDr8rkS0GnCjcaoDgXq2qoNSXdd5df5sNiI2FKdsFWNJWK9IYGXxPVORjjAYBQSxzuijj65lhhBiM0RX6C0g9D/ia6d15M053nfffVq23XbbRVlUtRwXBNwNFlpoIT3HXYEAyM20L8YE6lZAX9A51UP1ig0JojzttNOWPUIisuh4cAEhODNZuGW5KXGB4Kabb745zj777KqXwh2EOo3ovLIdwC2CtsRIInup0Lm8VOj9WbEhOjXazUuIKta3ek2YdO4zcI9BNMrvyQiR4rzzzhvFOESL+H3RflZsyEVhnHpN0iTZ7TW/XWzoYkP5PTl1CwLshniDR5SFJRkxF4888sggDCZcddVVQSLda4DkrGM2b99p9wTJUB2IQgIJk9KdBWXkLCNcGISY6dJLLw0PPPCA7sxwOudZvGk32j5v/KTeIbhxq1wvtLOZP7h/EHUlu5Ogmuh+tPaBBx6ozuW4VhA3kZ0nYlREfOwg2E2wCxV9UKb15k5x1J911lmTvHnNtfb/d5vVH5bDWbKydHACq4PlIeNkF8rvyQhrRGH+ah1pZZW+2e1BuH44FUfAxYbFsfKaHY4AIj7CVhmDseGYzqKRPGm4GKTJsleLQUNSTIBjCD1RvZRtH1Gn9bfetorWxzRedgi5zMtEhJZtgPGecsopytwRvcnOSJm1GDa0PFgyfbIwbLwEtJIQDUIw3yzxXCgvcwHBcXmRSUe9IWks5vy8zBQhsdhUJg92TsURaO0voPhzvaYj0OsIsChDKO/ThLMnhMN2vZRlLnn380bOM+sxMLB2irRvdVv1bSHF8gIns4CzK2PBNUL/BLNGH8ZuU/Leqe4IBgahS4LYtcEI0AGKqbuW1fPn2WefDUsuuaTuvOq5r0hdAlJDJGXNEr5fhDMTMWnJpYsuuijcddddmk4ozUzxaWPXyu7LyHSepBxiF893mtjdsYt1Ko6AM6/iWHnNDkdgzDHH1BGIvqAkT5m9USP6q5eKMBeYIm/0WXFhkWcVab9IO/XUoZ88N28hx6kd5sUuywwVaNtEjCziLPIwMSPEkBA4EJQZcVojzItdHwGle4IYFy8YGFmkCWaLSBHn/TQR4QLnZBh21qkfZg5GfIy++eYbPWRHhsg3zbyw7ET8OMMMM1h1/y6AQOkraIEbvIoj0KkIzD333Np1oh2kiSgREJZtkL1Fs6BAoj1Xc+a0SMmYSq3ICtwvBh18JSbYzbTPQlfkmfrABv9gSchuMR0o2ppCVwg9/vjjVqTfWB1y3y677BI++uijko/hLQYtWo55uBGMDZP6IiRGFhpEOq+uzVXetSJlMF+Js6mWjTBYI3ZM4L3aaqtZkdbBhQDdYzo0ExaJROQgjVAWA9qGxKgjsTy1Bu15ROlxKo6AM6/iWHnNDkcAs210Rrxdp/VbiLMQ5VjUe9tRYJLOAoWYi+j+ZBUQK1XdRdmixYIF8zvuuONK0KEehKEGhiDEXjQ9UaPtswtgh0DG7XRWgpIHt+gEUSoGBFk/NvyVwBCDFDNgAE+Yl1gTKgMr2gWYFiHJwPKhhx6qehv4I9ojI0KWJGC1ivUq+VKl65vOLk9ER//ZfR1xxBHJLcSD5KXHQl8heqYP7K7Qd+FKwIc55Tuvf0ljFQ5M17XccstVqOHFuQjUtMkcRCu4qXznT2wjpvKE6hGfrSg6hki0fszZCfcj6VgSQDCNl4C/ar6MebykiImEI8LsXZhUxAxa3sajKOq1jiyqSQQIYZBaxrW55ppLMxEQKR/zeqNG2+eZmKeT1YBx1EP1mspj1o/Z/5VXXln2GGE6UXzbIuMWhqGZCUS/k2RTyN5ANgVZfMpM5TH9J5sB18iwUI0kbY72R0SaZdUwMacNPoSQqkREx+B+6snOOa633npRfP5KqgsTjUTwIIKI6Nf0tyLiTq1DOKpJJpkkeZY9076JvFGJ+J1RT15AyqqI9auGiZIXhbJrlQrcVN7DQ3l4qEr/HR1Q3gjzsmGJEUGUbNLqe2Vl2W9StoioUIuJsZclFhsW+fSiY8yLxRafIp5TiRppn37k9aXSM6y8XubFfWIlqQzY2sh+EyYKpp+3IGfrVjqnjQMOOCCScqgW4dtG/TwCb/zSeLFoBcEEKz2rFe1bG/RbXCA0zJSVFfl25uV+XvIy5NSNCIgjcpBgquqnVWn8iKlMt5W2rrP6iJjw8+I7S5QhEuM5laiR9ulHXl8qPaOZcon1pyGQKoUuwkIO4wvT4TXyLNwHEL3iI1aLMHQwn6t0XcSp9JWwVYRsagUhysx7VivatjYQhWI9iZiSqCxO9SFQ/l9X3/1e2xFwBFIImC6FPEudTjBJnKJhLoQv6glCX4jOMWuxV8+zsOpbeumlAwYhnUIYgWDEIaLrRJ/WKX1vl366qXy7zIT3o6MRwLABgwyzXMTYg8WUxamTCeddgtGan1KrxyJZt5tukmC7lQLuNt14DzXAbhWjF3auTo0h4MyrMdz8LkegBAHEVVim8THCIm1QIESnkgdrUBhK24yB34szruamw5lXc/j53Y5AgkAzoq+kET9wBByBQgi4zqsQTF7JEXAEHAFHoJ0QcObVTrPhfXEEHAFHwBEohEDXiw3XXnvtlpnXFkLcK7UMAYsyITmmwkgjjdSydgfFhohmQYJJS2kyKI6xm8ZkwY67aczZsQ6GQ1y2sBvOiT9GfiUnR6AnECAQKz5MZFxudU6rnuivt9mZCJDfjIj3XUhPdy3z6sLJ9iH3IgJEG19iiSUCTKxLF5deRNsf1YUIPO06ry6cdR+yI+AIOAKdjoAzr06fQe+/I+AIOAJdiIAzry6cdB+yI+AIOAKdjoAzr06fQe+/I+AIOAJdiIAzry6cdB+yI+AIOAKdjoAzr06fQe+/I+AIOAJdiIAzry6cdB+yI+AIOAKdjoAzr06fQe+/I+AIOAJdiIAzry6cdB+yI+AIOAKdjoAzr06fQe+/I+AIOAJdiIAzry6cdB+yI+AIOAKdjoAzr06fQe+/I+AIOAJdiIAzry6cdB+yI+AIOAKdjoAzr06fQe+/I+AIOAJdiIAzry6cdB+yI+AIOAKdjoAzr06fQe+/I+AIOAJdiIAzry6cdB+yI+AIOAKdjoAzr06fQe+/I+AIOAJdiIAzry6cdB+yI+AIOAKdjoAzr06fQe+/I+AIOAJdiIAzry6cdB+yI+AIOAKdjoAzr06fQe+/I+AIOAJdiIAzry6cdB+yI+AIOAKdjoAzr06fQe+/I+AIOAJdiIAzry6cdB+yI+AIOAKdjoAzr06fQe+/I+AIOAJdiIAzry6cdB+yI+AIOAKdjoAzr06fQe+/I+AIOAJdiIAzry6cdB+yI+AIOAKdjoAzr06fQe+/I+AIOAJdiIAzry6cdB+yI+AIOAKdjoAzr06fQe+/I+AIOAJdiIAzry6cdB+yI+AIOAKdjsCQnT4A778j0NcIfP/992GCCSYIAwcOLOnKYIMNFsYaa6ySsummmy48//zzJWV+4gg4AvUj4Duv+jHzOxyBEgRGHnnk0K9fv/DXX3+FP//8M/nEGJNjyrk+YMCAknv9xBFwBBpDwJlXY7j5XY5ACQLrrrtuyXneCcxszTXXzLvkZY6AI1AnAoPJP1Ss8x6v7gg4AhkEfv311zDaaKOViQ7T1WaZZZbw7LPPpov82BFwBBpD4GnfeTUGnN/lCJQgMOyww4aVV145DDlkvhqZ8vXXX7/kHj9xBByBxhFw5tU4dn6nI1CCwNprr606rpLCf07Qd62++up5l7zMEXAEGkDAmVcDoPktjkAeAv379w8Yb2Rp8MEHDwsssEAYd9xxs5f83BFwBBpEwJlXg8D5bY5AFgFEgxhkDDXUUNlLLjIsQ8QLHIHmEHDm1Rx+frcjUILAWmutFf7444+SMnZe6MOcHAFHoHUIOPNqHZbekiOg4sGxxx47QWKIIYYISy21VBh11FGTMj9wBByB5hFw5tU8ht6CI5AgQFSN9dZbLxEd/v3332GdddZJrvuBI+AItAYB9/NqDY7eiiOQIPDcc8+FWWedVc+HGWaY8PXXX4fhhhsuue4HjoAj0DQC7ufVNITegCOQQQBn5Mkmm0xLV1ppJWdcGXz81BFoBQL5HpWtaLkN27jpppuqRkBowy57lzoUgTnmmCO88847YeKJJw5XXXVVh47Cu91JCOCKgUtGt1BXiQ2Z3M8++6xb5tbH6Qg4Al2EwDLLLBNuvvnmbhlx94kNTzrppEA4R/8Mehhst912+ubZLnN77rnntuXv7LLLLguY77cLTt6P5v8XuzH0mFsbdst7io+z1xHYcMMNe/2Z/kBHoFsQcObVLTPt43QEHAFHYBBCwJnXIDSZPhRHwBFwBLoFAWde3TLTPk5HwBFwBAYhBJx5DUKT6UNxBBwBR6BbEHDm1S0z7eN0BBwBR2AQQsCZ1yA0mT6U5hG48cYbwySTTBLuvPPO5hsbRFs45JBDwmuvvVY2uoEDB4YTTjghrLrqqmHAgAHhnnvuKatDAabxm266abj44otzr9cqvPrqq8Pyyy8f5p9//rDvvvuGF198sdYtFa9//PHHYdtttw3vvvtubp0nn3wy7L333gEfqsMOOyy33oMPPhhIRIqDMCbr3FON8sZ/2mmnBVwYnIoj4MyrOFZeswsQYBF7//33cxep3h4+i1w70e+//x422GCDMMYYY4RpppmmpGu33HJLmGGGGcLll18eDjrooACDWWyxxUrq2Mnxxx8f/vOf/+QyQKtT6fvss88OZ555ZjjxxBPDAQccEK677row00wz6Xele/LKP//887DDDjuEKaaYIpxyyinhp59+Kqt29913h3nnnTeMOOKI4dRTTw1vvvmmMsyPPvooqXveeeeFhRdeODz00EPhscceCxdeeKHeU+3lJ2/84HrOOecoM04a94PqCMg/SNfQOOOME8VJuWvG220DFSflKG+/TQ9bmFfTbTTbgCyccb/99mu2mdz75Q0/ipNy7rVKhX/++WdcdNFF4+67715WRcJfwWWjMKtIvWokQYvjv/71L60vO5pqVXOvScDjKDu75Jrs3rStJZZYIikrcvDMM89E2T3Gfv366f0vvPBCyW2ffvpplDQ2cbbZZkvKv//++yiMLApD07Jvv/02Svqb+MADD+j5Bx98EFdccUVtL31f0oAcVBv/r7/+GoWZRmHO6VsKHcuOL8rusFDdQaTSU77zqs7b/WoXIjDRRBP16ajZ+ZHU8q+//urTfqQffv7556s4TBhquji89dZbYeONNw5jjTVWuOSSSwL5yyqRLM6aHoadR6PEro8QSMIktQlhEvr93Xff1dUkUf+nnnrqMOGEE+bed8UVVwRhTiretAojjTRSWG211cKjjz4ann/++XDttdeGnXbaKSy00EJahbYQY0IvvfSSfqf/1Bo/GQgQUcoLQvjmm2/St/pxDgJdFZg3Z/xe5AiUIIAuh0UYBoZeBnr66afDNddcE0YYYYSw9NJLq9jq9ddfD0TQWHfddbXOs88+G1jwxhtvPNWZIT577733wrLLLquLEYv6zjvvHNCxkPML/QZtIGKDSbEoI756++23wyKLLBK++OKLcP3114dPPvkkbLnllmHOOecMW221VUB0h8itN+mXX34JMC30WNnULuh4fvjhBx0b4wIHRIrDDjtsWRd33XVX1R3JbqfsWtECxJFgMuSQ/1u6wBCqJKIs2m623q233qpFM844Y8ml6aefXs9vuOEGZdqjjz56yXWYKWXDDz98STknRcZP7jfqHX744eHoo48ua8MLUggMIlvIQsNwsWEhmDq2UrNiQ2FaURarEpHWyy+/rGIi+ZeJwpii6HWiMKw4yiijqOjtiSeeiPfff7+Kl6iDqElSosSNNtpIjykT5qOY/vHHH1EWNm3fQP7www+1HXlr1yIJHB3FMEDriB4k3nfffZGyH3/8McqCrXUlP5jd3tB3vWLDK6+8UvsjzLnkeYjRED8yRv63bGyIBY866qiSurJbirPPPnsU5hvBlHsaERuWNConq6++uj5bdirZS4XOmUv6khUbCgPWcnmZKGmH3wj1md88kh1gFCYeN9tss5LL9YxfXpCiMP+aItj0A1xsKLPi5Ah0KwKI6o488siS4U833XThoosu0jJhTIlSHsszsiQ//PDDqrA/7rjjtA71RZ+iyncU+IiCzjrrrICSn93ClFNOWdL+BBNMoLsuKxQdihogcD7ppJMGdimUseu77bbbAoYRo402mlXvle9XXnlFn0NWhjQ99dRTigFpX4SJh6+++kq/hx566LDbbrtpX6mPccQ222yj1oVDDTVUuommjtmdYrCBMQVz00qibSi7g7Rzu5595umnnx4QL+6///7JpXrHz+4OESO7cKfKCLjOqzI2fqULEWDhzZItWOg0TBwEk4JsEbNyLN8Qn0FTTTWV6kgQC+bpQLRSlT/WjlVZfPHFw1JLLWWnvfb96quv6rOyzMvM5fv375/ojhBvml4MPRkkuxQVnWYtFPViE3+w7EMsu/LKKzfRSv6tsrPWC7K7Kakgu2c9H3/88UvKORGDDRX1yU5VxcdWod7xjzzyyHqr4W7t+HcpAq7zKsXDzxyBQghUM0xIN2AZles1KKCNLPNKt9ubx+i8IBEHljzWFvisHgzzcgidH0YNd9xxR2A3gSECxA4Nuv322wPMAL8wknfWS/fee28gxVFPEDpPkomKuLakeTu3lxe7+PPPP6vvGboqmLlRI+M3PEU8bM34dw4CzrxyQPEiR6BVCNjbc1ZcWKT9dmFe7CChrC+UlSMyTBNGKxDMDfGX6MPCpZdemlQxS8E33nhDk8NOO+20DTEvLBzx0+oJok+iy1QGhlWiEQY3kBlucMzOGiflNdZYI4iui6KEGhm/MfdKlpBJ411+4GLDLv8B+PB7FgEWQN7izWrNrOSw0IMQS7H4oT8zMqb122+/WZF+Y2mYLSup0EMn1ncxLil5ApZ1iEnR8Zk4jQpi/KD1iDjBoo6+L/0hIgVE8lDKxTBFz/ljuCQFVQ5I9plHMEeYRjOEhSfE7jBNRM+AqcwzzzxazPzBsNiJ7bHHHklV8Nh8883rHj8NiI+ZttNTjDnpZIcfOPPq8An07rcWAVs80V8YibWfHop1nRVpFA5Osv44iLJshyLOu2qsQDglE7mZSGmTTTZRgwZEZvgTYf5NaCqYmO1cMNfGKAJjEBZDdjosnOl+JB3qwQPEgGReFgfbkqdQRqQL+izWhck1TPwxMtlxxx2TsiIHiAAxvFhllVVqVj/iiCPUMAIjljTBtBDVgiFRL2qRzXNWRAfDFkvGQEQP5gbCcIXIGWeccUai+9xll10CTBTGjtsEH+6DqWO40QixWydaCQY7TlUQSJtbDurHbio/aM9ws6by4mcVZeFTU2hZeKJYzEVMpcUIQcswCxcfnPjII49EWVi0TJTrUcL6RHkj13NhLtoGURgwHRfrsxLQMY03c3zqiiViFH1PFEfXKExKzaNldxVl8dP2ZNcWJXZfpIzfLyb6RHZohuo1ledZsouIIiqLwqjKHi27Sx0zYyDKBq4CIhIsq2cF4pulY8uayh966KFaLstVFPGcVc/93n777bUuETbShEuBWGPqtay5erqeOBrHlVZaKYruUusytmxEE8z6xQE7jjnmmBq9QvzvojDrpBlhbEl/6XP2wzjzqNL4qSshqLQdMXbJu7ViWTeayiO26Bpy5jVoT3WzzKsZdIx54dOF/xMLlIgDKzZJCCpjBKLsL6vHvWIwUNIG9fLqlt1co6AR5vXll1+q35rsBiu2LnEh1SetYoUCFwjLJPqmmj5OYCcix9wWxcAkStzDKCb0udfrLeTFgdBPvUESsUP9CnlmPdSNzMsNNqrsSv2SI9AIAoiLaomM0IMZmXWZnfONSC4rNsqrl76nJ4+JAIIIjSghM888cyLaTD9zEonG3wwhGt1zzz3DFltsUTXMFM9AL5hnrs41xJuIXLFybAUh8kVc29NEYF7EoIg7Tczc08/s5PadeRWcPXkLCnfddZf+QxAahgjTplgv2ESfVsPPiMUnS0TMRj+BPoWwRP5Pk0Wo2LnpTMwardhdnVULXRRGI4TIwpABK8JWEv9TtI0esBlCT0j/zJS/mbZ6616crYmWT78JL+VUGwE32KiNkdZAKY1yF8dL/Fc6iXExAEx/iSBBtAgU41iIoVQm4gFvejh7imw/YFzgVB8CLDik14B4wSHmoTkv19dS+9dec8011XfLHLdb2WMs9pplXPSH/FytjrjRynHmtcX/p+hS1TI177qXlSPgO69yTHJLcErFQoz8PbzZdRrR/7nnnjvMNddcunsUZbUGFrVxEB0ck1+iYuOcSUgjM+u2Ov6djwAhnIg0nqZOeutP97vIsTleF6nrdYoh0OroI8We2tm1nHnVOX/oIvh0KlXSxeBTQsRuUkXAoAmTRJw2p9oIEL+Qj5Mj4Aj0HgLOvGpgjZMpKSjIsMvCjvd7VmQolljq74KiGB2SWP4kMegIC0T+IcLE4BODD8xNN90UCMhKcFfTG+AnwnOIVkAQVoKdHnPMMUnvqj2DSq1Il0HAV9K4EyuOPqaZF/3mQ5BRnFMJvGox2IqkDKGPYIDvE5jw9k4q9yWXXJJLStWeYXX82xFwBBwBRaAec8xOr1uvqbwE/tTMqaQzwPz2wAMPVB8MsfpKoMCcefLJJ4/iuKj+PyJC0jr4/kD4rggz0jLSLIjSO84333x6LnqmpJ0FF1xQzXspEB2KmiXbxVrPqCddhugU9NnHHnusNV/yLQ6Zel1+HOrjxMV99vm/9s7eRYp1CeNz/gFTA0FMNDIUQRQEQUFh0UDERANBRNRMUOSiFwzESFDQxMRA0EADAw1URAM/4GwgCgpGgmBgYGr23udX91TzTm93T/d87NmZroLZmen3s6tnu7qqnqr6T9q9e3dSBvVEbIv8ZEkpc4wnbUqGMMfbt28t/kbCPymQ1OKBiGtyalrD+4x6/zeh8qP2tpbax4HKr6X9x15WcqCPUPmI81r5O7Aj3GQJQJWDuuiBACN4NRdeBDoSyOhE8CM3fmXg9kNJBebsWB54qLxsNhedBACxdoElijF5Se82awgokASzLcbXfRglvBhHICznIHBHUnFBC+TM41ykLVm7lyuXv8y+E+ip7BK29O3bt+2YCyjBn5Pg1kWArUAiSVkSrG+bNazjiD8ILwQre49X8KBvvwFVHBjxH7JQzX+H2VC/8CrS06ml4QE+7gTCirQxniaHDNNAXHHO+zHy1EkTM7/Ynz9/zBfiyKzcKQuyCpOkghENYURMj7QPSzNDdV3Wh9quQbmMaRB79vRGoA+VwcBy7wHwcCKDNufoaYr8/IiF8dIgnnXbUXfkgiOtDqZXzKdLS0sDZWKwKdus4WuPesd3Jw15VLdet4Nq4xpQuiNoMThATbN59sWPcxVCeNVwzQvwlZFV+Lvc50VWbAiEniLja2aqPpyX1GA+8qMBQybWA8HFj5GS4JOsUb1y81GSqpLYlD2BTKQ0PSCPrgjL/PxYkXxvCGtCDQ4ePGhlIxBa+PeUjWKsNarOhBiZacCtq+ZelGM8YN26dSv4tCgXVOdBkVLPRr9Ap9V4KvMLm2s8rckbCcaEeEqtI894QKbpMlEDCXBDG+Jmgmby8eNHi8UiOSygD5K8TmuNNvugD5okBKyeWBnWZz8ImDIBYmlLCERQjA8ePLDMCCQ4dSEzrTXa7iX6BQeCA/PPgRBeNdeQAF6IoNOcuAkTsAxhOkNzAEEnv03Rjfbjx49bUHNxsOEDQoC0O9Qnun//vpWQZw7Qd23XaFsuw/detZ179+4NyNaN9uJmSzftKYmqle/wcQJgFMUF/VjT+5kzZ6x2ExoY2h3zvn792rKyT2uNpvWjLTgQHFgsDoTwqrmeZElAmAhhZ7BwgngpO/7+/fsBWpXQelbKQiAB8wkJjTdQMlArXyHkoOWloyQ61FRSw9MKPXz4sOgnsIaN27lzp6VrGrVGl3IZQgfa3K4V4uMi2wbChSBsBBfxXpv+yVNH4DLw/0ePHpmpD78VsHyyddy4ccPmajo/LxmCdokJEgKSj2ZHQT/8hW3WsIHxJzgQHAgOOAcWCn8y4mS6QuVlxkvA28UrQ9yBsuO74pysfIXilZKXTZCz1PrJV5QoxQAyEbp+/XoSoMHalNA0ffjwwdqZkxfQesV2GcJPoI109uzZBGpItZCKrOOj1mhTLkPaTpJp0tb0tWWus3Up+QEiUjWZEudUJmDuqo9UjFUMWpJJ07q1LRkiYWfoSpCTJ06cSPv377fQAl+raQ3vM+o9oPKjOPT/dmnVid9r0OJwoI9Q+b+4fLqZ9YIEXx9cunTJcp91OWGK0YGiIygX9BwaWZlA6KGdYeZDU+lCmPIwR6KdkNgVkEgVcqhpDbRByH1kXdZv05efiQSVATnYn4NW2oyljyMv4SXJf8lSXqZJ11CNJ9Mi2xQhLK/dp+/4HalwzO8taDE4QDVqABskROgJLYfZsMWVpiqrZ5OoElxMge8L301XwcVYBBU3dCDnQL2rBNeoNRBasxJcrI2wYm8I566Ci/GePgleVgmuaazBHEH/DgdI6Pz169cVi4NSxSRN4mfe6+jNmzeWlabJJ1s3lgdKHlww15P5HuFMOMe4xIMWmWYAFB0+fHjw8uXLYirAS1evXh3s3bvX9nvt2jULdyk6ZB94GMMET5+cFANZ+JTz4/G5GwdCeHXjV/QODtRyYNZGjFnPX3tiDQ0AhXjq54Ekj2PE10ldLnzBxDCCYiX8o4pIF0ZYCPW3ugovLA6q4DxQ1poBfmNlfrGQE5njB/iCuxKQ861bt5oAJN4S4aPq0DYNMZesI1eAgY7YL5YcUpyV903ZJGIaEYDlMBP4Re0uQmyCxudACK/xeRcjgwMFB7h5cZOblSlu1vMXJ9LhA+cq36Xl50RQOSn9l/GC+nHkyDx//ryV3vH28jtjye05Dr148cJq6wH+QasnXygWAgRGVQhL0xoIKjREwEqEyFCmJCelerNCnJjnMH+DBqakEKhZhJ4T1wpLSl15ISwkmPfQEKneEDQeB0J4jce3GBUcGOIAT+CvXr0aCicY6jDhl1nPP872CDhHQFy+fHloOMKIjDNoFnv27BlqK38BwQviFnPfOITGh7BA03EicTSERteW8FcrzVsRrlIOskfrxXxIEgFM4LQTbC9wlS3x6dOnYinM/gh1zzJTNGQfmIPwkwsXLli4SNYUH1tyIIRXS0ZFt8XkAP4Nsvfj2yBEgTL0PFU7EX939OhRM0VxjGBtwA4cEzLUuiFYMCVB3AC5KUHE/l28eNGKfz558sTCCzA7oR2gtRAr53MRigARZ8fcvGiH6uanHd8L/qLVJsx1CC34lvta0YQQSPh+8UNxU6eIaxVRN+7KlSsD4gvr/LxV4/Jj8JPg/jzDDdcI4UCoSVsiKQD+LPxy+HS5dmiQOWGWxLed0759++wrYLCuhKmUmnlln1jXeXrbX08UvaGuUPneMGZBTrQrVF5P/Em5KpO0g0QoAVnzN27caOEDfIbkN0mKfbMwAWeTTFwGNVcuRzuk2D/Lsq+bSJL2lfiuVFgWUsExZSpJAvNYiACfOXb69GkbS+JivitG0L5LiyhCKbwyQdX8dNZTu42VkLCxbf9MAyqvvIi2tm7oQ8uqorQdlwBI/L+pLpx9JxyDBM5OQtemHTt2JJnq7JB8Q9YPfk9CX758sXkkEFpPoxyddj25DuzZr7dMfxY+0jQRIS30I7l2mQgjYU6VGCo3Fd8PHICnnvgAAAduSURBVDhgoTTwYxLqI1Q+NC/9uoL6yQFMNp8/f7a0VSRc5kkdUxjmJoLUASPwZLx58+YhBlGLLUdMEnCNpsETO0ABvgNUkGCycZiPCATHSU+QO1oBvo4fP36sMC0xh2cc8UWr5qeNmmr4lVzT8/6r8e65P8sah8r52PIkR/7586eZBNFGMS8S2I6JDwKxh09pXHOhTVLxB7MeQAn8bG0J/xj7ooYe9fTwafFOQVZ4nPuz8jnR1MjPiYbJ2HEIXx0aHmEoQd04EMKrG7+i94JwQE+5hk7j5ittqzgr4NbE9H379s0EW9HQ4gOCJyfPsE+qMW/bsmXL4MiRI2Y2RHB2IZ/Dx5CphKwoZP9fbZKGY0uWhZebCBFYEDwgswrCHoHAOMydgBXIbD9NAl3o6dV46GhLDvHHBEhlBIjsOO7L44Gmimjn2mIGHpc8BMf5Oe48fRwXwquPVz3O2YLNgT6XfS3c9NB0IM/o35ZdZeFSN84rFXQBFDBX2/nr1p3mcQ+KB1WX04YNG+xr7gdD03RtUua1wcmTJwcIPbQvtF9e5LuE8BHi9xuH8FMBW0fz6kKkKIPyPfNdZk3eBuy5TKp1Z3lPQSh2EZTleXxNTxNXbo/v9Rxo/3hSP0e0BAfmjgNkzOemg+nu+/fvQ2YfvwGvX7++03m1FS7+lI05sksZi7bzd9r0mJ3RICGv/ebTcBytFS3LwQy0uYaGoECY0YeXk+fARCNDo0SIdSW0PhI/dyU/F/acE9B7yIWbtwGjB2QBOKXc5n3avvv1d42v7bjop+QOwYTgQB85gMbg2kAZrYcPhGwn3u5P1vg4IEyOoAXdf8MxBAvHSPM1igTmMFMlfjaf2wt7MtbnaDO/72nUmtNuZ+9QOT4LXyGkHJ727n9AHeIXxMy2vLxsDw08OPjr0KFD1hUtx7UwH9v2HAWMsDgtH5e/w9MyetDbgdazL/ySeWCz72PXrl3e1ZCNp06dMj+YCzcanz59asmri44tP+AXhIhNC+rGgRBe3fgVvReIAwAqEDoUZgQyD6GF4QMBMo92BrkGgX8J5z1ZE37//m2QeiDwCBluZAg1QBkUFnVAA+Opy+YaCuVzyOhPACsCFNMUN3V8NYy7c+eOmdMYRyAr+4Gq5gecwJP/JD4Xm3yMP+wbk6tXEPcpAGBQ1ZvqA15FG0g8wgtgCefahW7evGnXYRSwg+uB7zKvfO7rILQw1cLDqryXnAf+N66jklP7MAtCRvsWotCOEQvG/GiO+LuOHTtmL34ffC6v7f6/JpMgWjgZPchCEtSRA5PAM+dtbEDl5+2KddtvV6g8s0v4JAEeDMoO7FlPwAZblxZVLA40Hki9/rUS8HghBdO2bduS0hJZdQFgzhI+SYLQXoo5sooAQtgVY3TzNGg4MGwJqGJuPkj4JPk+krSwpFizdPfuXduTgn2TtEDrWzW/buw2/3//gdkPTdrwZRpQeaaXBpKEliuqH/iSEhYWFkDYgYRO0o05yb+1op/3511VxO1cylB5ZbWw4/BeSavzIUOfpcEkCZUkU+zQcb7It5lkirR5qPhQR9KIE9eJ66psKfabkN/TulPZYdOmTcVe2E/+ompCToQxsBf68LtQfT8Locj7yGxq7QKE5IfH+txHqDxPi72hEF6LfanHEV5wBOEjzSAJBZekIdUySVpQcQNW4tcV/X79+pWIHXNy4UVMF7FECp5NuVD0frzLNJbk97FD3MC5WZapPD/tHOtK0xJerE3cmkxmlVsQqCNJA7VYucoOLQ8imASrt+vUNATes2YVcRwhr/yKVc1Dx1QcNqlG3dCxWXzhIYdYN0oaTUp9FF4B2NCjUVC/OUCqH0w3oyiH1DtKLB+Tx37lx/m8bt06e5WP+/fcnOZ+MG/z96r5q455/1m/szZxZmQhUa06M8vla+I3LOcHzNvbfMYHhQmXlFPllE3l8QA96gjzJn4pkumOImlYo7pM3I55+dmzZ2bGLCM2J568JxOE8OrJhY7TXH0OuK+DGm2LSviiCOZWpogBAcqybkz1VEEkMjd+xkkIEA77mxQdOMkefCwZ9vFXsh8qlweNx4EAbIzHtxgVHGjkADcmpUqyPs+fP7eMHQ5gaBw4h43yVw0eP35sCM1pb5/sJJMKLvZ07ty5AoAz7T12nQ9tFLh9rsl3nSP6DwahecWvIDgwAw6QJurdu3dDM6+Fp/6hDU3xiwdeT3HKhZ0qr3u2sCe5CicWwmsVmBxL9I8DwKl5BQUHggOz4UCYDWfD15g1OBAcCA4EB2bIgRBeM2RuTB0cCA4EB4IDs+FA78yGZDGoirKfDXtj1tXkAIlZSXZL1vageg6Q+YFsEsGneh7NWwt5GduEe8zbeTXtt1fCa2lpqVNp8CbGRdva40DXbOJr7wxWZ0eg3ALptjq8Xq1Vtm/fPiBHY5/oLyK7+3TCca7BgeBAcCA4MPccWA6f19xfwziB4EBwIDjQPw6E8OrfNY8zDg4EB4IDc8+BEF5zfwnjBIIDwYHgQP84EMKrf9c8zjg4EBwIDsw9B/4HyoiN6YkEsd0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.keras.utils.plot_model(\n",
    "    model,\n",
    "    show_shapes=True,\n",
    "    show_layer_names=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JpL9idwZV6fL"
   },
   "source": [
    "For each character the model looks up the embedding, runs the GRU one timestep with the embedding as input, and applies the dense layer to generate logits predicting the log-likelihood of the next character:\n",
    "\n",
    "![Model architecture](https://www.tensorflow.org/tutorials/text/images/text_generation_training.png)\n",
    "\n",
    "Image source: [Text generation with an RNN](https://www.tensorflow.org/tutorials/text/text_generation) notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Npruiy2RAPkt"
   },
   "source": [
    "## Try the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 678242,
     "status": "ok",
     "timestamp": 1586974564842,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "E4DCLA0GASL1",
    "outputId": "d6e39b67-d2da-46d2-c6a1-f62d81f64981"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(64, 200, 621) # (batch_size, sequence_length, vocab_size)\n"
     ]
    }
   ],
   "source": [
    "for input_example_batch, target_example_batch in dataset_sequence_batches.take(1):\n",
    "    example_batch_predictions = model(input_example_batch)\n",
    "    print(example_batch_predictions.shape, \"# (batch_size, sequence_length, vocab_size)\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MWebJXU9CEPd"
   },
   "source": [
    "To get actual predictions from the model we need to sample from the output distribution, to get actual character indices. This distribution is defined by the logits over the character vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 678227,
     "status": "ok",
     "timestamp": 1586974564843,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "Y4Jgo-iECFWI",
    "outputId": "22b015fd-bc24-4a4b-c60c-177856033131"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction for the 1st letter of the batch 1st sequense:\n",
      "tf.Tensor(\n",
      "[-2.96991039e-03  2.02196068e-04  5.34047745e-03 -2.94846855e-03\n",
      " -3.64167639e-03 -2.63241702e-04 -8.80502281e-04  7.99844624e-04\n",
      "  5.26232133e-03 -1.23821688e-03  2.34868471e-03 -1.18148176e-03\n",
      " -8.88869807e-04  5.29895071e-04 -2.79451575e-04  2.37091444e-06\n",
      "  1.92230579e-03 -1.01283449e-03 -1.99038046e-03 -5.02873491e-03\n",
      " -4.71714232e-03 -1.69062556e-03  3.95582151e-03  5.13770268e-04\n",
      " -2.62290076e-03  1.52810244e-03 -3.62532330e-03  4.52159438e-04\n",
      "  4.21410240e-03 -4.20481712e-03  2.28100177e-03 -1.53552578e-03\n",
      " -4.49910574e-03  2.41562491e-03 -2.41917069e-03  4.41195024e-03\n",
      " -2.03300873e-03 -3.13091557e-03  2.15532375e-03 -1.44562731e-03\n",
      " -8.98959697e-05 -2.03171140e-03  2.83075799e-03  3.00962362e-03\n",
      "  4.39736526e-03 -2.73631304e-04 -6.91650144e-04  1.73515151e-03\n",
      "  3.01030884e-03 -9.34766373e-04  1.50964432e-03  3.39096365e-03\n",
      "  3.16297705e-03  2.44869967e-04 -1.08598219e-03 -7.28239818e-03\n",
      " -2.05614767e-03  1.98792620e-03 -3.93527001e-03  3.37290025e-04\n",
      "  1.95513829e-03  2.91557517e-03  1.84102287e-03  4.72247927e-03\n",
      "  1.93154044e-03  7.20826583e-03 -8.27651296e-04  4.75527486e-03\n",
      "  1.38566690e-03  3.24708899e-03 -1.81622850e-03  1.36225706e-03\n",
      " -1.36198709e-04 -2.26765592e-03  2.54186074e-04 -2.58616405e-03\n",
      " -4.64894576e-03  3.89528414e-03  3.25334468e-03  5.27988188e-04\n",
      " -2.64238310e-03 -4.89778118e-03  3.21954163e-03  1.11659628e-03\n",
      " -3.06684617e-03  2.00703996e-03 -2.21931376e-03  2.23845663e-03\n",
      "  3.28445598e-03 -2.56485119e-03 -4.31890320e-03  2.41565611e-03\n",
      " -5.15983673e-04  3.64185032e-03  1.48760225e-03  1.20713573e-03\n",
      "  5.29383449e-03 -3.79023957e-04  2.40978692e-03  2.31299014e-03\n",
      " -4.91128303e-05  3.96255311e-03 -4.22925572e-04 -3.23428866e-03\n",
      " -2.07351381e-03  2.73143407e-03  1.88296579e-03 -3.68654262e-03\n",
      "  1.27331994e-03 -1.24076579e-03  6.32031867e-03 -2.48864666e-03\n",
      " -1.23163743e-03 -2.19777529e-03 -6.27300469e-04 -3.31383711e-03\n",
      " -1.13901915e-05 -2.82761734e-03 -3.86833388e-04 -1.73249340e-03\n",
      " -1.05973426e-03  5.09004341e-03  3.12257023e-03  1.97673100e-03\n",
      " -4.14281525e-03  2.05062283e-03  3.34373908e-06 -1.60101010e-03\n",
      "  4.21956182e-03 -4.85191122e-03  2.88532046e-03 -4.75326553e-04\n",
      " -2.57715257e-03  3.55798984e-03  2.37036962e-03  3.57332220e-03\n",
      "  2.17431062e-03 -1.53979345e-04 -4.33850288e-03  1.87864178e-03\n",
      " -1.86979759e-03 -1.36150082e-03  5.56515646e-04  1.42047508e-03\n",
      "  2.74778227e-03  3.35774850e-04 -2.53880257e-03  2.82756845e-03\n",
      "  2.44993693e-03 -3.19341733e-03  3.78998322e-03 -3.06723465e-04\n",
      "  1.50572485e-03 -9.95766837e-04  4.31416184e-03 -8.64217116e-04\n",
      "  1.43683166e-03  4.11726162e-03 -1.25944172e-03  4.96372813e-03\n",
      "  5.60203707e-03 -9.61561105e-04 -8.50761193e-04 -1.65662495e-03\n",
      "  1.30175764e-03 -4.40109987e-03 -1.54017517e-03  1.31750060e-03\n",
      "  4.35700268e-03  6.44978718e-04  3.13272886e-03 -4.42359271e-03\n",
      " -3.94872529e-03 -3.35123227e-03  4.19887435e-03  4.43855161e-03\n",
      "  3.22591141e-03  1.56010385e-04 -5.99237485e-03  1.03176723e-03\n",
      " -2.21929047e-03  3.54293408e-03 -1.23872038e-03  3.38202249e-03\n",
      " -1.98809942e-03  6.94049429e-03 -1.06861454e-03 -7.82860327e-04\n",
      "  5.16313594e-04  2.36987532e-03 -2.89311429e-05 -3.95814423e-04\n",
      "  4.56190342e-03 -1.23822247e-03  6.24573417e-03 -2.55804625e-03\n",
      " -1.04940555e-04  1.33907038e-03 -1.14532444e-03  2.81685661e-03\n",
      "  5.43777831e-04  4.64606658e-03  3.45542142e-03  3.82768194e-04\n",
      "  2.02233321e-03  1.85965083e-03 -4.97636292e-03  2.09314027e-03\n",
      " -5.81746921e-04 -1.73986098e-03  5.29615209e-05 -3.48408706e-04\n",
      "  6.21711044e-03 -3.84812290e-03 -1.18742208e-03  5.74179983e-04\n",
      "  9.04284534e-04 -1.36926793e-03 -1.08293397e-03  2.86536291e-04\n",
      "  5.69129945e-04 -1.80946407e-03 -5.48771955e-03  2.47454806e-03\n",
      " -2.21755286e-03  1.62935501e-03  1.63916254e-03 -3.32597556e-04\n",
      " -1.44371181e-03  9.67276050e-04 -1.20116572e-03  1.19435287e-03\n",
      " -1.70178805e-03 -3.87001270e-03 -1.19349419e-03 -1.01136998e-03\n",
      "  1.57997449e-04  1.95235421e-03 -1.90228014e-03 -3.79354716e-03\n",
      " -2.59016687e-03 -6.01103529e-05  1.70413696e-03  2.56037409e-03\n",
      "  1.15031702e-03  2.00827932e-03 -2.61775893e-03 -2.09540525e-03\n",
      "  2.69065960e-04 -3.63526307e-03  1.10269035e-03 -2.46130163e-03\n",
      "  1.82558456e-03 -5.70906699e-03  1.40825636e-03 -5.47343120e-03\n",
      "  1.49906403e-03 -1.50285801e-03  2.73680664e-04  1.48007460e-03\n",
      "  3.41087091e-03 -6.13138499e-03 -1.80945778e-03  2.28763092e-03\n",
      "  3.44680506e-03  2.71166908e-04  8.62840854e-04 -7.95917818e-04\n",
      " -1.36045180e-03  4.52237437e-04  7.40970951e-04 -3.14979139e-03\n",
      " -2.67863320e-03 -2.18148879e-03 -7.88994017e-04 -2.33359216e-03\n",
      "  3.23024672e-03  4.61527100e-03  3.90101760e-03  1.92098215e-03\n",
      " -3.53164040e-04 -6.38937391e-03 -3.20535246e-03 -1.51562714e-03\n",
      "  2.58783600e-03 -8.90760566e-05 -7.00501027e-04  4.46120463e-03\n",
      " -4.00476996e-03  3.15386383e-03 -3.74853914e-03  3.54121235e-04\n",
      " -3.16394540e-03 -1.55988208e-03 -2.88777007e-03 -1.30451447e-03\n",
      "  1.11281837e-03 -2.29158439e-04 -4.69365343e-03 -1.79625954e-03\n",
      "  4.65409551e-03  1.98075897e-04 -2.81246053e-03  1.92641513e-03\n",
      " -4.19606362e-03 -8.53825244e-04  1.13668293e-03  1.53516233e-03\n",
      "  4.45213215e-03 -9.88768414e-04  3.31520336e-04 -3.58278304e-03\n",
      "  3.44983581e-03  3.47022805e-03 -8.69183568e-04  8.29555676e-04\n",
      " -8.56739935e-05 -3.29165137e-04  2.72252737e-03 -2.37217173e-04\n",
      " -1.35505933e-03 -5.20083867e-03 -3.32186790e-03  5.31172240e-03\n",
      " -2.35710200e-03  5.52401273e-03  3.72211391e-04  5.33639104e-04\n",
      " -2.43864069e-03 -3.84261506e-03  1.45898527e-03  2.39854585e-03\n",
      " -4.71283449e-04  7.64812808e-04  1.33032037e-04 -1.38897519e-03\n",
      " -1.70430692e-03  9.96899325e-04  3.54808872e-05  2.56171334e-03\n",
      "  1.05264317e-03 -2.82326015e-04  1.81646517e-03 -2.33595213e-03\n",
      " -5.03519783e-03  1.10140303e-03 -2.84432247e-03 -2.00615404e-03\n",
      "  8.41872534e-04  6.67849928e-03 -5.83600253e-03 -4.24106652e-03\n",
      "  1.14184630e-03  2.95217102e-03  8.34879233e-04 -3.98391346e-03\n",
      "  3.25757754e-03 -7.84135307e-04  6.27445523e-04 -5.44302631e-03\n",
      "  3.13957781e-03 -3.02591873e-03 -2.06224294e-03 -8.65378068e-04\n",
      " -4.49024234e-03  5.93538024e-03  1.40665378e-03 -7.73238542e-04\n",
      " -7.91485480e-04 -5.07989712e-03 -4.39047441e-03 -6.28761947e-04\n",
      "  6.24953653e-04  5.42121101e-03  6.50510774e-05 -2.07748869e-03\n",
      " -1.37498835e-04 -3.86989280e-03  1.78228982e-03 -7.48617458e-04\n",
      " -1.48551865e-03 -7.57519389e-03  1.25740445e-03  3.24202410e-04\n",
      " -1.66424597e-03  3.90002830e-03 -8.55716702e-04 -3.19620734e-03\n",
      "  1.04847702e-03  1.12897437e-03  5.37774584e-04  4.69030929e-04\n",
      "  2.51115393e-03  3.55380587e-03  2.71298829e-03 -1.12220878e-03\n",
      " -7.17928517e-04 -5.92021132e-03  1.75648995e-04  1.48483738e-03\n",
      " -1.80276658e-03 -6.70493697e-04  1.44608214e-03 -7.93615938e-04\n",
      "  8.36108637e-04  1.00507773e-03  3.45401186e-03  3.91197857e-04\n",
      " -2.67160498e-03 -1.62438431e-04  4.58952691e-03 -1.47211296e-03\n",
      "  1.79704372e-03  4.59640333e-03  9.42048733e-04  2.82686495e-04\n",
      "  9.14397067e-04  3.60484119e-04  2.48183426e-03  3.25409998e-03\n",
      "  1.17904483e-03 -1.26637798e-03 -1.29893975e-04 -9.41952167e-04\n",
      " -4.60873917e-03 -2.93853390e-03  6.95107388e-04 -2.95422389e-04\n",
      "  5.65846916e-03  1.68879563e-03  3.46388901e-04 -7.11088069e-04\n",
      "  2.31424696e-03 -1.03079807e-03 -1.11149554e-03 -2.11489666e-03\n",
      " -3.50161269e-03  2.13855063e-03 -1.11947476e-03  1.50933955e-03\n",
      " -8.02038005e-04  1.07715710e-03 -2.15566857e-03  3.24162957e-03\n",
      " -5.65700233e-03  2.35174503e-03 -2.10525305e-03 -3.53414309e-03\n",
      "  1.07685325e-03 -1.47457980e-03  2.15876824e-03  7.34216243e-04\n",
      "  3.87710589e-03 -1.31599419e-03 -1.90719939e-03 -6.38154522e-03\n",
      "  1.56695000e-03 -1.64689089e-03 -9.93861700e-04  3.42892541e-04\n",
      " -4.98188473e-03 -3.21490457e-04 -2.57308129e-03  2.46718968e-03\n",
      "  6.55017793e-04 -4.55554621e-03  1.74585322e-03 -1.69836381e-03\n",
      "  2.10167514e-03  2.23360304e-03 -4.45178896e-03  1.87133299e-03\n",
      "  1.60568918e-04  1.32443936e-04  2.42036092e-03  6.58909208e-04\n",
      " -4.91692312e-03  2.87363742e-04 -1.09317084e-03 -5.35313226e-03\n",
      "  1.10405346e-03  1.37676660e-03 -4.65311634e-04 -1.21597422e-03\n",
      " -4.98747034e-03  1.53795874e-03 -1.83808943e-03  3.44327581e-03\n",
      "  8.32715072e-04  2.80157896e-04  3.36588779e-03 -2.38184677e-03\n",
      " -2.42060889e-03 -7.32985605e-03 -2.10792199e-03  1.89127750e-03\n",
      "  4.63969307e-04 -7.00481469e-05  4.24549542e-03 -4.59315022e-04\n",
      "  4.47517261e-03 -3.89564957e-04 -4.79762588e-04  1.35278446e-04\n",
      " -1.73517200e-03 -2.00542575e-03 -7.49243482e-04 -1.89404108e-03\n",
      " -2.37741228e-03  1.61092100e-03 -2.01807544e-03  5.82352094e-03\n",
      "  2.98094354e-03 -1.43182906e-03  4.14387044e-03 -1.60400930e-03\n",
      "  2.50670942e-03 -4.01832443e-03  3.09774280e-03 -2.00444483e-03\n",
      " -3.00918124e-04  3.17066116e-03  3.24714230e-04 -2.17295974e-03\n",
      " -5.12648025e-04 -5.16266003e-03 -2.77804467e-03  1.54582981e-03\n",
      " -5.44310175e-03 -3.54347925e-04 -1.64899079e-03 -5.02398331e-03\n",
      " -1.66997593e-03  6.73103798e-03 -2.24419008e-03 -2.70372396e-03\n",
      "  1.23207201e-03 -1.39791775e-03 -2.24590721e-03 -2.68167444e-03\n",
      " -6.05385052e-04  1.29042438e-03  2.37613777e-03  9.31537827e-04\n",
      " -5.84532274e-04  2.59521254e-03  4.43371711e-03 -5.75694116e-03\n",
      " -3.18599818e-03 -8.88234819e-04  1.28717162e-04 -2.76897848e-03\n",
      " -3.65862885e-04  2.70474306e-03  3.70543823e-03  5.91118075e-03\n",
      "  1.05132908e-03 -3.77161778e-03 -2.65625631e-03  8.57410312e-04\n",
      " -5.74532023e-05  3.38229514e-03  2.36529217e-04 -9.87916021e-04\n",
      "  1.98815018e-04 -3.57900467e-03  3.44835687e-03  5.63639542e-03\n",
      "  1.19890331e-03 -3.70563800e-03  8.23522161e-04 -3.22137121e-03\n",
      "  9.63578932e-04 -4.41345619e-05  2.51808111e-03 -2.12470093e-03\n",
      " -3.25000798e-03  2.89233401e-03 -8.26234813e-04 -1.61699392e-03\n",
      "  2.92486907e-03 -7.76923320e-04  8.76021804e-04 -1.70226162e-03\n",
      "  4.15711757e-03 -1.62895198e-03  4.77584108e-04 -8.71605764e-04\n",
      "  1.43936253e-04 -1.29298866e-03  9.00190964e-04  1.49092288e-04\n",
      "  2.68429215e-03  3.01266927e-03  3.24859563e-03  3.19268950e-03\n",
      "  1.95573724e-04  6.15902245e-04 -3.00159911e-03  1.44227070e-03\n",
      " -1.63919688e-03  1.42129988e-03 -1.46635727e-03 -1.37253653e-03\n",
      "  1.37668394e-04  2.78103747e-03  3.01332050e-03 -2.56322464e-03\n",
      "  1.98633084e-03 -6.50512637e-04  4.18016594e-03  1.70647725e-03\n",
      " -4.19194112e-05 -6.61098398e-03  5.50927129e-04  1.38886238e-03\n",
      " -5.23816736e-04  4.05704230e-03  1.09234464e-03  1.90317736e-03\n",
      "  3.31994961e-03], shape=(621,), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "print('Prediction for the 1st letter of the batch 1st sequense:')\n",
    "print(example_batch_predictions[0, 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 678211,
     "status": "ok",
     "timestamp": 1586974564844,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "0dOr0MwFHlRb",
    "outputId": "24a550a0-6a12-4e05-8f50-caf752888b58"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([[2 2 2 1 1]], shape=(1, 5), dtype=int64)\n"
     ]
    }
   ],
   "source": [
    "# Quick overview of how tf.random.categorical() works.\n",
    "\n",
    "# logits is 2-D Tensor with shape [batch_size, num_classes].\n",
    "# Each slice [i, :] represents the unnormalized log-probabilities for all classes.\n",
    "# In the example below we say that the probability for class \"0\" is low but the\n",
    "# probability for class \"2\" is much higher.\n",
    "tmp_logits = [\n",
    "  [-0.95, 0, 0.95],\n",
    "];\n",
    "\n",
    "# Let's generate 5 samples. Each sample is a class index. Class probabilities \n",
    "# are being taken into account (we expect to see more samples of class \"2\").\n",
    "tmp_samples = tf.random.categorical(\n",
    "    logits=tmp_logits,\n",
    "    num_samples=5\n",
    ")\n",
    "\n",
    "print(tmp_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 678151,
     "status": "ok",
     "timestamp": 1586974564845,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "JPzr0r4zCgS3",
    "outputId": "1ce0c5c8-addc-4777-f505-63663962d576"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([200, 1])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sampled_indices = tf.random.categorical(\n",
    "    logits=example_batch_predictions[0],\n",
    "    num_samples=1\n",
    ")\n",
    "\n",
    "sampled_indices.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 678118,
     "status": "ok",
     "timestamp": 1586974564846,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "YaA7DclID8dz",
    "outputId": "cfb93eca-f822-49e7-dc37-3365a80bebdc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(200,)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sampled_indices = tf.squeeze(\n",
    "    input=sampled_indices,\n",
    "    axis=-1\n",
    ").numpy()\n",
    "\n",
    "sampled_indices.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 678080,
     "status": "ok",
     "timestamp": 1586974564847,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "_ubGQ0gVENhB",
    "outputId": "398ecbd2-5973-448c-f6f2-2c48a40f73f1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([563, 288,   5, 200, 598, 455, 511, 317, 223, 118, 224,  92, 559,\n",
       "       399, 261, 554, 459, 472, 597, 220, 528, 177, 395, 350, 531, 410,\n",
       "       119, 246, 521, 431, 417, 513, 559, 505, 342, 329,  76, 507, 443,\n",
       "       317, 154,   6, 600, 280,  68, 456, 586, 245, 224, 410, 415,  74,\n",
       "       298, 275, 272, 297, 167,  75, 150, 159,  63, 344, 598, 205, 304,\n",
       "       230, 344,   8, 414, 309, 217, 431,  61, 498, 126, 382, 429, 332,\n",
       "       390, 229,  46, 574, 119,  65, 425, 440, 467, 207, 188,  87, 333,\n",
       "       509,   4, 205, 408, 122,  45, 283, 412, 206, 168,  76,  77, 479,\n",
       "       341, 595, 347, 410, 351, 309, 361, 465,  46, 173,  33, 526, 374,\n",
       "       330, 128, 240, 251, 546, 602, 464, 252, 377,  86, 230, 520, 367,\n",
       "       418, 329, 526, 262, 525, 115, 251, 465, 210, 190, 340, 322, 476,\n",
       "       606, 186, 164, 192, 561, 254, 610, 508,   3, 234, 214, 521, 165,\n",
       "        33, 564,  33,  85, 400,  85, 404, 616, 183, 177, 617, 189, 158,\n",
       "        24, 475, 464, 106,  82, 283, 595, 138, 306, 416, 121, 151, 480,\n",
       "       573, 310, 414, 341, 137, 181, 476, 350, 485, 534, 495, 459, 158,\n",
       "       532,  41, 573, 403, 390])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sampled_indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 122
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 678033,
     "status": "ok",
     "timestamp": 1586974564848,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "Gi9HOzw9EajS",
    "outputId": "177578de-a6be-417a-e528-d470494f59fb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input:\n",
      " 'University. These were interrupted by service in the U.S. Army Signal Corps during World War II, for which he worked as a codebreaker and participated with the landing at Casablanca. Before leaving fo'\n",
      "\n",
      "Next char prediction:\n",
      " '砲أ#˚造三射இИÖК{燈ễь溪中児通Д教Śṟจ斌‡×н延イ₱屋燈孤ರலk宗ツஇĆ$道פc上訪мК‡₨iذלטدıjýė^ಸ造εعСಸ&″نόイ[地âูりவზПL花×`♠タ会ιžvா寄\"ε”ÞKק…ηļkl前ಮ达ข‡ชنผ介LŌ?排ะளäзт概鄉事уำuС庙ฤ℃ல排ю怡Íт介οưಡத其镇żĦș皮х食家!аυ延ī?磨?tệt—준ţŚ태ơđ6兴事¼qק达îق€Üā加臨ه″ಮíŠ其จ単日哪中đ新G臨–ზ'\n"
     ]
    }
   ],
   "source": [
    "print('Input:\\n', repr(''.join(index2char[input_example_batch[0]])))\n",
    "print()\n",
    "print('Next char prediction:\\n', repr(''.join(index2char[sampled_indices])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 357
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 678007,
     "status": "ok",
     "timestamp": 1586974564848,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "b87e0lsYMTsv",
    "outputId": "e562488a-f4b7-4a2f-9f56-f2a6c7dc5737"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction #0\n",
      "  input: 55 ('U')\n",
      "  next predicted: 73 ('砲')\n",
      "\n",
      "Prediction #1\n",
      "  input: 79 ('n')\n",
      "  next predicted: 73 ('أ')\n",
      "\n",
      "Prediction #2\n",
      "  input: 74 ('i')\n",
      "  next predicted: 73 ('#')\n",
      "\n",
      "Prediction #3\n",
      "  input: 87 ('v')\n",
      "  next predicted: 73 ('˚')\n",
      "\n",
      "Prediction #4\n",
      "  input: 70 ('e')\n",
      "  next predicted: 73 ('造')\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i, (input_idx, sample_idx) in enumerate(zip(input_example_batch[0][:5], sampled_indices[:5])):\n",
    "    print('Prediction #{:1d}'.format(i))\n",
    "    print('  input: {} ({:s})'.format(input_idx, repr(index2char[input_idx])))\n",
    "    print('  next predicted: {} ({:s})'.format(target_idx, repr(index2char[sample_idx])))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LqcBufKEE_p6"
   },
   "source": [
    "## Train the model\n",
    "\n",
    "At this point the problem can be treated as a standard classification problem. Given the previous RNN state, and the input this time step, predict the class of the next character."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "l4s0-PvrFub5"
   },
   "source": [
    "### Attach an optimizer, and a loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 677987,
     "status": "ok",
     "timestamp": 1586974564849,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "UOEUUm6JE95a",
    "outputId": "c4b60238-ced1-4150-db09-fc056b409555"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction shape:  (64, 200, 621)  # (batch_size, sequence_length, vocab_size)\n",
      "scalar_loss:       6.430712\n"
     ]
    }
   ],
   "source": [
    "# An objective function.\n",
    "# The function is any callable with the signature scalar_loss = fn(y_true, y_pred).\n",
    "def loss(labels, logits):\n",
    "    return tf.keras.losses.sparse_categorical_crossentropy(\n",
    "      y_true=labels,\n",
    "      y_pred=logits,\n",
    "      from_logits=True\n",
    "    )\n",
    "\n",
    "example_batch_loss = loss(target_example_batch, example_batch_predictions)\n",
    "\n",
    "print(\"Prediction shape: \", example_batch_predictions.shape, \" # (batch_size, sequence_length, vocab_size)\")\n",
    "print(\"scalar_loss:      \", example_batch_loss.numpy().mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SXhJsB6eFgrJ"
   },
   "outputs": [],
   "source": [
    "adam_optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)\n",
    "model.compile(\n",
    "    optimizer=adam_optimizer,\n",
    "    loss=loss\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MK3Cf-xZFwL4"
   },
   "source": [
    "### Configure checkpoints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7Jet-3Ps0-Ny"
   },
   "outputs": [],
   "source": [
    "# %rm -rf tmp/checkpoints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LUhXnHPJFy5q"
   },
   "outputs": [],
   "source": [
    "# Directory where the checkpoints will be saved.\n",
    "checkpoint_dir = 'tmp/checkpoints'\n",
    "os.makedirs(checkpoint_dir, exist_ok=True)\n",
    "\n",
    "# Name of the checkpoint files\n",
    "checkpoint_prefix = os.path.join(checkpoint_dir, 'ckpt_{epoch}')\n",
    "\n",
    "checkpoint_callback=tf.keras.callbacks.ModelCheckpoint(\n",
    "    filepath=checkpoint_prefix,\n",
    "    save_weights_only=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oFg9MFJoGZWf"
   },
   "source": [
    "### Execute the training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AVk-pARPGaja"
   },
   "outputs": [],
   "source": [
    "EPOCHS=150\n",
    "STEPS_PER_EPOCH = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 5855654,
     "status": "ok",
     "timestamp": 1586982233796,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "y0rveBdAGeEz",
    "outputId": "1844d63b-85f6-4c40-97fd-dcc5b3a1c879"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:sample_weight modes were coerced from\n",
      "  ...\n",
      "    to  \n",
      "  ['...']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:sample_weight modes were coerced from\n",
      "  ...\n",
      "    to  \n",
      "  ['...']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train for 10 steps\n",
      "Epoch 1/50\n",
      "10/10 [==============================] - 183s 18s/step - loss: 4.4378\n",
      "Epoch 2/50\n",
      "10/10 [==============================] - 108s 11s/step - loss: 3.3721\n",
      "Epoch 3/50\n",
      "10/10 [==============================] - 104s 10s/step - loss: 3.2632\n",
      "Epoch 4/50\n",
      "10/10 [==============================] - 102s 10s/step - loss: 3.2935\n",
      "Epoch 5/50\n",
      "10/10 [==============================] - 136s 14s/step - loss: 3.1744\n",
      "Epoch 6/50\n",
      "10/10 [==============================] - 132s 13s/step - loss: 3.0984\n",
      "Epoch 7/50\n",
      "10/10 [==============================] - 129s 13s/step - loss: 3.0452\n",
      "Epoch 8/50\n",
      "10/10 [==============================] - 102s 10s/step - loss: 2.9947\n",
      "Epoch 9/50\n",
      "10/10 [==============================] - 141s 14s/step - loss: 2.9197\n",
      "Epoch 10/50\n",
      "10/10 [==============================] - 213s 21s/step - loss: 2.9320\n",
      "Epoch 11/50\n",
      "10/10 [==============================] - 282s 28s/step - loss: 3.0474\n",
      "Epoch 12/50\n",
      "10/10 [==============================] - 229s 23s/step - loss: 2.8073\n",
      "Epoch 13/50\n",
      "10/10 [==============================] - 165s 16s/step - loss: 2.7504\n",
      "Epoch 14/50\n",
      "10/10 [==============================] - 84s 8s/step - loss: 2.7144\n",
      "Epoch 15/50\n",
      "10/10 [==============================] - 80s 8s/step - loss: 2.6593\n",
      "Epoch 16/50\n",
      "10/10 [==============================] - 82s 8s/step - loss: 2.6027\n",
      "Epoch 17/50\n",
      "10/10 [==============================] - 82s 8s/step - loss: 2.5921\n",
      "Epoch 18/50\n",
      "10/10 [==============================] - 80s 8s/step - loss: 2.6757\n",
      "Epoch 19/50\n",
      "10/10 [==============================] - 81s 8s/step - loss: 2.6094\n",
      "Epoch 20/50\n",
      "10/10 [==============================] - 80s 8s/step - loss: 2.5761\n",
      "Epoch 21/50\n",
      "10/10 [==============================] - 81s 8s/step - loss: 2.4921\n",
      "Epoch 22/50\n",
      "10/10 [==============================] - 84s 8s/step - loss: 2.5235\n",
      "Epoch 23/50\n",
      "10/10 [==============================] - 81s 8s/step - loss: 2.5435\n",
      "Epoch 24/50\n",
      "10/10 [==============================] - 80s 8s/step - loss: 2.5429\n",
      "Epoch 25/50\n",
      "10/10 [==============================] - 81s 8s/step - loss: 2.5075\n",
      "Epoch 26/50\n",
      "10/10 [==============================] - 80s 8s/step - loss: 2.4761\n",
      "Epoch 27/50\n",
      "10/10 [==============================] - 81s 8s/step - loss: 2.4480\n",
      "Epoch 28/50\n",
      "10/10 [==============================] - 82s 8s/step - loss: 2.5827\n",
      "Epoch 29/50\n",
      "10/10 [==============================] - 79s 8s/step - loss: 2.4337\n",
      "Epoch 30/50\n",
      "10/10 [==============================] - 80s 8s/step - loss: 2.4442\n",
      "Epoch 31/50\n",
      "10/10 [==============================] - 81s 8s/step - loss: 2.4203\n",
      "Epoch 32/50\n",
      "10/10 [==============================] - 79s 8s/step - loss: 2.4016\n",
      "Epoch 33/50\n",
      "10/10 [==============================] - 81s 8s/step - loss: 2.4207\n",
      "Epoch 34/50\n",
      "10/10 [==============================] - 81s 8s/step - loss: 2.3306\n",
      "Epoch 35/50\n",
      "10/10 [==============================] - 81s 8s/step - loss: 2.4447\n",
      "Epoch 36/50\n",
      "10/10 [==============================] - 80s 8s/step - loss: 2.3742\n",
      "Epoch 37/50\n",
      "10/10 [==============================] - 91s 9s/step - loss: 2.3544\n",
      "Epoch 38/50\n",
      "10/10 [==============================] - 108s 11s/step - loss: 2.3882\n",
      "Epoch 39/50\n",
      "10/10 [==============================] - 105s 11s/step - loss: 2.3722\n",
      "Epoch 40/50\n",
      "10/10 [==============================] - 108s 11s/step - loss: 2.3662\n",
      "Epoch 41/50\n",
      "10/10 [==============================] - 108s 11s/step - loss: 2.2663\n",
      "Epoch 42/50\n",
      "10/10 [==============================] - 108s 11s/step - loss: 2.2274\n",
      "Epoch 43/50\n",
      "10/10 [==============================] - 108s 11s/step - loss: 2.3766\n",
      "Epoch 44/50\n",
      "10/10 [==============================] - 110s 11s/step - loss: 2.2903\n",
      "Epoch 45/50\n",
      "10/10 [==============================] - 109s 11s/step - loss: 2.2748\n",
      "Epoch 46/50\n",
      "10/10 [==============================] - 132s 13s/step - loss: 2.2947\n",
      "Epoch 47/50\n",
      "10/10 [==============================] - 108s 11s/step - loss: 2.2247\n",
      "Epoch 48/50\n",
      "10/10 [==============================] - 80s 8s/step - loss: 2.2167\n",
      "Epoch 49/50\n",
      "10/10 [==============================] - 80s 8s/step - loss: 2.2195\n",
      "Epoch 50/50\n",
      "10/10 [==============================] - 83s 8s/step - loss: 2.1841\n"
     ]
    }
   ],
   "source": [
    "tmp_dataset = dataset_sequence_batches.repeat()\n",
    "    \n",
    "history = model.fit(\n",
    "    x=tmp_dataset.as_numpy_iterator(),\n",
    "    epochs=EPOCHS,\n",
    "    steps_per_epoch=STEPS_PER_EPOCH,\n",
    "    callbacks=[\n",
    "        checkpoint_callback\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mLdnOyvzMggJ"
   },
   "outputs": [],
   "source": [
    "def render_training_history(training_history):\n",
    "    loss = training_history.history['loss']\n",
    "    plt.title('Loss')\n",
    "    plt.xlabel('Epoch')\n",
    "    plt.ylabel('Loss')\n",
    "    plt.plot(loss, label='Training set')\n",
    "    plt.legend()\n",
    "    plt.grid(linestyle='--', linewidth=1, alpha=0.5)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 617,
     "status": "ok",
     "timestamp": 1586982234407,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "4Ghveem_OQBV",
    "outputId": "b7e062f8-46ce-4669-e48a-9be5009d44e6"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXxU5dn/8c+VTJIJZBlIgIEECCZCCLKnuEAVkVp3LVB3bd1o1brUFZ+2Vv3VVtvncalSN1S0dam7Vq0VC26AIMgOQRIIkECAAUISYEgmuX9/ZBKSkIRkJpNJcl/v1ysvZs6cOef+5gy55mz3LcYYlFJK2Ssi3A1QSikVXloIlFLKcloIlFLKcloIlFLKcloIlFLKcloIlFLKcloIlFLKcloIlGqCiOSLyORwt0OpUNNCoJRSltNCoFQrich1IpIrIntE5AMR6eefLiLyqIjsFJESEVklIsf5XztLRNaKSKmIFIrIHeFNodRhWgiUagURmQT8CbgQ6AtsBl73v3w6cDIwGEj0z7Pb/9rzwC+MMfHAccDcdmy2Us1yhLsBSnUylwEvGGO+AxCRe4C9IpIGVADxQCaw2Bizrs77KoAsEVlhjNkL7G3XVivVDN0jUKp1+lG9FwCAMaaM6m/9KcaYucCTwExgp4g8KyIJ/lmnAmcBm0XkCxE5sZ3brVSTtBAo1TrbgIE1T0SkO5AEFAIYY/5qjBkLZFF9iOhO//RvjTHnA72B94A32rndSjVJC4FSzYsSEWfND/AacJWIjBKRGOCPwCJjTL6I/EBEjheRKGA/4AWqRCRaRC4TkURjTAVQAlSFLZFSDWghUKp5HwMH6/xMBH4HvA1sB9KBi/3zJgDPUX38fzPVh4z+4n/tCiBfREqAX1J9rkGpDkF0YBqllLKb7hEopZTltBAopZTltBAopZTltBAopZTlOt2dxcnJySYtLS2g9/p8PhyOThe5TdiaXXPbRXM3benSpR5jTK/GXut0v7G0tDSWLFkS0HsPHjxIbGxsG7eoc7A1u+a2i+Zumohsbuq1kB8aEpFIEVkmIh828trPRWSXiCz3/1wb6vYopZSqrz3OEdwCrGvm9X8aY0b5f2aFsiGbNzdZELs8W7Nrbrto7sCEtBCISCpwNhDSP/BKKaUCF+pzBI8Bd1HdNW9TporIycD3wK+NMVsbziAi04HpAKmpqeTk5NS+NnBgdf9fdSticnIyycnJ5Obm4vP5AHA6nQAUFRVRXFxcO296ejper5fCwsLaaW63G5fLVW89cXFxpKamUlBQQFlZWe30zMxMiouLKSoqqp2WkpKC0+kkLy+vdprL5cLtdpOfn4/X6wXA4XCQkZGBx+PB4/EElCktLa1FmWrW2ZUytWQ7eTwecnJyulSmlmynmtzhzGSMwefzERERQUREBBUVFbXziQgOhwOfz0fd3g2ioqKoqqqisrKydlpkZCQiUrtugIiICCIjI494P8Dq1aupqjrclVPNSdTG3t9YmyorK494vzHmiDa1V6aoqKhG21Q3U2VlJatXryYyMpLKysra5dfdTs0JWRcTInIOcJYx5gYRmQjcYYw5p8E8SUCZMeaQiPwCuMgYM6m55WZnZ5tATxZ7PB6Sk5MDem9nZ2t2zR0+mzZtIj4+nqSkJESkXdZZUVFBVFRUu6yrI6nJbYxh9+7dlJaWMmjQoHrziMhSY0x2Y+8P5aGh8cB5IpJP9QhOk0TkH3VnMMbsNsYc8j+dBYwNYXvC/h8jnGzNrrnDx+v1tmsRAKwsAnA4t4iQlJRUu+fXUiErBMaYe4wxqcaYNKp7Z5xrjLm87jwi0rfO0/No/qRy0HJzc0O5+A7N1uyaO7zaswgArf4D2FXUzR3I77zd7ywWkQdE5Dz/05tFZI2IrABuBn4eqvWuLypl1uId7NlfHqpVdGh1j0XaRHPbxdbelIPN3S6FwBjzec35AWPMvcaYD/yP7zHGDDPGjDTGnGqMyWl+SYHb5Cnj9ZXFbN93MFSrUEp1ELt372bUqFGMGjUKt9tNSkpK7fPy8pZ9GbzqqqtYv359s/PMnDmTV155pS2a3Cpz587lm2++abPldbo7iwOVEFt9DG3fwYqjzNk11Vw1ZRvNbZeIiOrvtklJSSxfvhyA++67j7i4OO6444568xpjMMbUvqehF1988ajru/HGG4NscWDmzp1LcnIyJ5xwAkCTGVrKmk7nEv2FoMTSQhBo/0ydnea2S0xMTLOv5+bmkpWVxWWXXcawYcPYvn0706dPJzs7m2HDhvHAAw/UzjthwgSWL1+Oz+fD5XIxY8YMRo4cyYknnsjOnTsB+O1vf8tjjz1WO/+MGTMYN24cQ4YMYcGCBQDs37+fqVOnkpWVxbRp08jOzq4tUnXdeeedZGVlMWLECO6++24AduzYwZQpU8jOzmbcuHF888035OXlMWvWLP7yl78watQoFixYcNTcR2PNHkGi5XsERUVFuN3ucDej3WnujuH+f61h7baSNl1mVr8Efn/usHrTWnL5aE5ODi+//DLZ2dVXUj700EP07NkTn8/HqaeeyrRp08jKyqr3nn379nHKKafw0EMPcdttt/HCCy8wY8aMI5ZtjGHx4sV88MEHPPDAA3zyySc88cQTuN1u3n77bVasWMGYMWOOeN+OHTv4+OOPWbNmDSJSe2/GzTffzF133cUJJ5xAfn4+55xzDqtXr+baa68lOTmZW2+9tcW5m2PdHoGthaDuTT820dx2aclJ8vT09NoiAPDaa68xZswYxowZw7p161i7du0R74mNjeXMM88EYOzYseTn5ze67ClTphwxz9dff83FF1cPaz1y5EiGDRt2xPt69uxJREQE1113He+++y7du3cH4LPPPuOXv/wlo0aN4oILLmDv3r0cPHjkec5gLw6wZo8gLsZBhNhbCJQKp4bf3MOp5o8swIYNG3j88cdZvHgxLpeLyy+/vNFLUKOjo2sf19z925iaQzTNzdOYqKgolixZwpw5c3jzzTd56qmn+PTTT2v3MOquPxSs2SMQEeKiI7QQKKVqlZSUEB8fT0JCAtu3b+c///lPm69j/PjxvPHGGwCsWrWq0T2O0tJSSkpKOOecc3j00UdZtmwZAJMnT2bmzJm189WcW4iPj6e0tLTN2mhNIQDo0d3JvoN2Xl+dnp4e7iaEhea2S2tPmo4ZM4asrCwyMzO58sorGT9+fJu36aabbqKwsJCsrCzuv/9+srKySExMrDfPvn37OPvssxk5ciSnnHIKjzzyCFB9eer8+fMZMWIEWVlZPPfccwCcf/75vPHGG4wePbpNThaHrK+hUAmmr6FzHv+SnvFOXr56XBu3quMrLS0lPr65vv+6Js0dPuvWrWPo0KHtus6aDtc6Ep/Ph8/nw+l0smHDBk4//XQ2bNjQpiOpNczd2O8+XH0NdTjR4rP20FDdHi5tornt0tKbxdpTWVkZ48ePZ+TIkUydOpVnnnmmzYfTDDa3NSeLAeKjI9lSZmchUEqFh8vlYunSpeFuRrOs2iOIi9GTxUq1p8526LkrCOR3blUh6NMzgX0HK6z8cHakm4vak+YOH6fTye7du9v1/5vt3VDXjEfQ2i5GrDo01McVR2WVYX95JXExVkXH5XKFuwlhobnDp2ZUtV27doW7KVZxOp2kpqa26j1W/TU8sG83UH1TmW2FoGa4Rtto7vCJioo6YpSsUOsIucMh2NxWHRqKj66Ou++AnidQSqkaVhWCuBh/IdATxkopVcuqQtDbFQfYWQji4uLC3YSw0Nx20dyBsaoQHDuw+gSKjWMStPbkUVehue2iuQNjVSHYv7f66gUb9wgKCgrC3YSw0Nx20dyBsaoQVJUfsLYr6rKysnA3ISw0t100d2CsKgQRIiTGRllZCJRSqilWFQJAC4FSSjVgVSHIzMy0thDYeJMNaG7baO7AWFUIiouLSbC0ENg6hq3mtovmDoxVhaCoqIjE2CgrLx8tKioKdxPCQnPbRXMHxqpCAHqOQCmlGrK2ENjYFbVSSjXGqkKQkpJCYmwUvirDgfLKcDenXaWkpIS7CWGhue2iuQNjVSFwOp0kxlYP4GDb4aHWDlTRVWhuu2juwFhVCPLy8qwtBHl5eeFuQlhobrto7sBYVQgAawuBUko1xbpCkKCFQCml6rFqvEaXy0VFtJ2FoCOMYRsOmtsumjswVhUCt9tNibe6ANh2U5nb7Q53E8JCc9tFcwcm5IeGRCRSRJaJyIeNvBYjIv8UkVwRWSQiaaFsS35+PnHRDiu7os7Pzw93E8JCc9tFcwemPc4R3AKsa+K1a4C9xpgM4FHg4VA2xOv1EhEhVvY35PV6w92EsNDcdtHcgQlpIRCRVOBsYFYTs5wPvOR//BZwmohIKNsE2s2EUkrVFeo9gseAu4CqJl5PAbYCGGN8wD4gKVSNcTiqT4nYWAhqsttGc9tFcwf4/jZqxxFE5BxgpzFmqYhMDHJZ04HpUD1Ic05OTu1rAwcOBGDz5s2105KTk0lOTiY3NxefzwdU33mXkZFBUVERjqpyivZ4ycnJIT09Ha/XS2FhYe373W43Lper3nri4uJITU2loKCg3rBwmZmZFBcX1+v9LyUlBafTWe8mD5fLhdvtJj8/v3Y3zuFwkJGRgcfjwePxBJQpLS2NoqKiet3QNpUJ6HKZWrKdcnJyulymo20nn89Xu4yukqkrbqe2zOTxeJrN1BwJVedrIvIn4ArABziBBOAdY8zldeb5D3CfMWahiDiAIqCXaaZR2dnZZsmSJQG1qeYXdeOr37Fuewlzb58Y0HI6o5rsttHcdtHcTRORpcaY7MZeC9mhIWPMPcaYVGNMGnAxMLduEfD7APiZ//E0/zwh6xa0pqLaOCZB3W8TNtHcdtHcgWn3O4tF5AEROc//9HkgSURygduAGe3RBu2KWimlDmuXMyvGmM+Bz/2P760z3Qv8tD3aUFdibBQVlYaDFZV0i7bz5JJSStWwqq+hmhMsNnY8V5PdNprbLpo7MFYVgho2FgKllGqKVYWg5lKr2kJwwJ5CUPcyM5tobrto7sBYVQhq6B6BUkodpoVAKaUsZ1UhqLnhwsbBaWy8yQY0t200d2CsLATxMQ5E7BqTQP+D2EVz20ULQSvk5uYCVHdF7bSr47ma7LbR3HbR3IGxqhDUdMQE9vVAWje7TTS3XTR3YKwqBHXZVgiUUqopVhUCp9NZ+9i2QlA3u000t100d2CsKgRpaWm1j20rBHWz20Rz20VzB8aqQlB3cIbqcYvtOZ54tIEpuirNbRfNHRirCkHdUYdqxiSwpSvqutltorntorkDY1UhqCsxNoryyiq8FU0Np6yUUnawuhCAXXcXK6VUY6wqBOnp6bWPbSsEdbPbRHPbRXMHxqpC4PV6ax/bVgjqZreJ5raL5g6MVYWgsLCw9rFthaBudptobrto7sBYVQjqsq0QKKVUU7QQaCFQSlnOqkLgdrtrH8c7q7uitqUQ1M1uE81tF80dGKsKgcvlqn0cESHExzisGZOgbnabaG67aO7AWFUIcnJy6j1P7GZPf0MNs9tCc9tFcwfGqkLQkG0dzymlVGO0EGghUEpZzqpCEBcXV++5TYWgYXZbaG67aO7AWFUIUlNT6z23qRA0zG4LzW0XzR0YqwpBQUFBvecJFhWChtltobntorkDY1UhKCsrq/c8MTaKcl8V3orKMLWo/TTMbgvNbRfNHRirCkFDenexUkppIQC0ECil7GZVIcjMzKz33KZC0DC7LTS3XTR3YKwqBA3H9awtBAe6fiHQsVztorntomMWt0JRUVG95zbtETTMbgvNbRfNHZiQFQIRcYrIYhFZISJrROT+Rub5uYjsEpHl/p9rQ9WexthUCJRSqimOEC77EDDJGFMmIlHA1yLyb2PMNw3m+6cx5lchbEeT4p1aCJRSKmSFwBhjgJqLW6P8PyZU62uJlJSUes8jI4R4p8OKQtAwuy00t100d2BCuUeAiEQCS4EMYKYxZlEjs00VkZOB74FfG2O2NrKc6cB0qL6Vum6XqwMHDgRg8+bNtdOSk5NJTk4mNzcXn88HgNPpJCUlhaKiononVhKdDnaXHqi3TLfbjcvlqjctLi6O1NRUCgoK6t28kZmZSXFxcb1jdCkpKTidTvLy8mqnuVwu3G43+fn5tQNNOxwOMjIy8Hg8eDyegDKlpaUdkSk9PR2v11tvHNPk5GTi4+O7VKaWbKfKykoiIyO7VKaWbKctW7YQGRnZpTK1ZDsNHDiwy2VqyXaqrKykT58+zWZqjlR/cQ8tEXEB7wI3GWNW15meBJQZYw6JyC+Ai4wxk5pbVnZ2tlmyZElA7cjJyTniMquz//oV7gQnz//8BwEts7NoLLsNNLddNHfTRGSpMSa7sdfa5aohY0wxMA84o8H03caYQ/6ns4Cx7dGeumzqeE4ppRoTyquGevn3BBCRWOBHQE6DefrWeXoesC5U7WmKFgKllO1CeY6gL/CS/zxBBPCGMeZDEXkAWGKM+QC4WUTOA3zAHuDnIWxPo+N62lIIdCxXu2huuwSbO5RXDa0ERjcy/d46j+8B7glVGxpyu91HTLOlEDSW3Qaa2y6aOzBW3Vmcn59/xLSE2CgOWdAVdWPZbaC57aK5A2NVIai5JKsuW+4ubiy7DTS3XTR3YKwqBI2xpRAopVRTWlQIRCRdRGL8jyeKyM01VwR1Jg7HkadEbCkEjWW3gea2i+YOTEv3CN4GKkUkA3gW6A+8GtSawyAjI+OIabZ0Rd1Ydhtobrto7sC0tBBUGWN8wE+AJ4wxd1J9eWinUvcW7Ro9ukUDsMmzv72b064ay24DzW0XzR2YlhaCChG5BPgZ8KF/WlRQaw6Dxn5Z/XvGMm5QT/46dwM7SrruiSb9D2IXzW2X9ioEVwEnAg8aYzaJyCDg70GtuYMQER6eOoJyXxW/e2817dH3klJKdSQtKgTGmLXGmJuNMa+JSA8g3hjzcIjb1m4GJXfnth8N5tO1O/h4lZ0jHCml7NXSq4Y+F5EEEekJfAc8JyKPhLZpba+m69bGXDNhEMNTEvn9B6vZu7+8HVvVPprL3pVpbrto7sC09NBQojGmBJgCvGyMOR6YHNSaOxhHZAQPTx1B8YEK/t+Ha8PdHKWUajctLQQOf0+hF3L4ZHGnU3dgisZk9UvghonpvLOskHnrdzY5n7eikoPlnatLiqNl76o0t100d2BaWggeAP4D5BljvhWRY4ANQa25g7pxUgbH9o7jN++sotRb/96CrXsO8MeP1zHuwc+4YOZ8yn1VYWqlUkq1nZaeLH7TGDPCGHO9//lGY8zU0DYtPGIckTw8bQTbS7z8+ZP1GGOYn+vhupeXcPJf5vH815sYnprI+h2lzF6wKdzNVUqpoLXovmQRSQWeAMb7J30F3GKMKQhVw0IhOTm5RfONGdCDq04axAvzNzE/z8PGXfvp2T2aGyamc/kJA+mbGMs1s7/l8c82cP6oFPokOEPc8uC1NHtXo7ntorkD06Ixi0VkDtVdStTcO3A5cJkx5kdBrT0AwYxZ3BoHyn38ZOYCHJHCz09K49yR/XBGRda+vnn3fn70yJecOdzN4xcfMeyCUkp1KG0xZnEvY8yLxhif/2c20KvNWthOcnNzWzxvt2gH//n1yXx08w/5aXb/ekUAYGBSd35xyjG8v3wbizbubuumtrnWZO9KNLddNHdgWloIdovI5SIS6f+5HOj4f/0a8Pl8bbq8GyZmkOKK5fcfrMFX2bFPHLd19s5Cc9tFcwempYXgaqovHS0CtgPTCPH4wp1BbHQkvztnKDlFpfzjGzsvW1NKdX4tvWposzHmPGNML2NMb2PMBUCnu2rI6Wz7k7o/Hubmh8cm839zvsdTdqjNl99WQpG9M9DcdtHcgQlmhLLbglpzGKSlpbX5MkWE3587DG9FJX/+JKfNl99WQpG9M9DcdtHcgQmmEEhQaw6DoqLQdCiX0TuOqycM4o0lBSzbsjck6whWqLJ3dJrbLpo7MMEUgk7XX3NxcXHIln3TpGPpkxDD795fjbei43U/EcrsHZnmtovmDkyzhUBESkWkpJGfUqBfUGvuYuJiHDxw/nGsLizhpteWdfiriJRSqkazhcAYE2+MSWjkJ94YY+co0c348TA39583jDlrd3DX2yupqup0O01KKQtZ9cc8PT095Ov42Ulp7DtYwSNzvifBGcXvz81CJPynU9oje0ekue2iuQNjVSHwer1ERYV+qOWbJmWw72AFz3+9CVe3KG6dPDjk6zya9sre0Whuu2juwARzsrjTKSwsbJf1iAi/OWso08am8thnG3jh6/D3Utpe2TsazW0XzR0Yq/YI2lNEhPDQlOGUeit44MO1JMZGMXVsaribpZRSR7Bqj6C9OSIjePzi0YzPSOKut1fy3JcbaUlvr0op1Z6sKgRut7vd1+mMiuTZK7KZPLQ3D368juteXkrxgfJ2b0c4sncEmtsumjswLRqPoCNpr/EI2poxhtkL8vnjx+voHe/kyUtHM3pAj3A3SyllibYYj6BLyMkJX19AIsJV4wfx1i9PQgR++vRCZn3VfoeKwpk9nDS3XTR3YKwqBB3ByP4uPrr5h5w2tDd/+Cj0h4oOllfyzBd5HKzQO52VUo3TQhAGibFRPH35WH5/bhZffL+TMx//ioV5oRnn57HPvudP/85hbl5pSJavlOr8QlYIRMQpIotFZIWIrBGR+xuZJ0ZE/ikiuSKySETSQtUegLi4uFAuvlVqDhW9c/14YqMiuXTWNzz07xzKfW33zX19USnP++9hWFrU/ieoO4KOtM3bk+a2S7C5Q7lHcAiYZIwZCYwCzhCRExrMcw2w1xiTATwKPBzC9pCa2vGu4x+emsiHN0/g4h/05+kv8pj61AI27ioLerlVVYbfvreKeKeD80b2Y2lBGQfLO16vqKHWEbd5e9Dcdgk2d8gKgalW8xctyv/T8Mzo+cBL/sdvAadJCDvmKSgoCNWig9It2sGfpozg6cvHsnXvAc7+69e8vnhLUCeS3/qugG/z93LPmUOZOjaVQ74qFm70tGGrO4eOus1DTXPbJdjcIb2zWEQigaVABjDTGLOowSwpwFYAY4xPRPYBSYCnwXKmA9OhuvLVPUM+cOBAADZvPjxmcHJyMsnJyeTm5tYO6ux0OvF6vRQVFdXruzs9PR2v11vvFm23243L5aq3nri4OFJTUykoKKCs7PA39szMTIqLi+sNDJGSkoLT6SQvL692msvlwu12k5+fj9frBcDhcJCRkYHH48Hj8ZDmgCfP7seTS0qY8c4q5q7K59aTehEh0mSmtLS0IzL17NufP360lqzeTo6LK8NXXkpMpDA3Zyf9ODxwTntkCmQ7NZYp0O3k8XgoKyvrUplasp3y8/Nr5+0qmVqynXw+X5fL1JLt5PF4cDqdzWZqljEm5D+AC5gHHNdg+mogtc7zPCC5uWWNHTvWBGrdunUBv7c9VVZWmb98kmMG3v2hufe9VaaqqqpV77/7rRXmmHs+Muu276uddvGTc82Jf/ys1cvq7DrLNm9rmtsuLckNLDFN/F1tl6uGjDHF/kJwRoOXCoH+ACLiABKB0Fw+04lERAi3nz6Y6344iJcWbuZ/P13f4vcu3byH17/dyjUTBpHpTqidPq5/d7bt87J+h149pJSqL5RXDfUSEZf/cSzwI6DhXQ8fAD/zP54GzPVXrpDIzMwM1aLbnIjwP2cN5ZJx/Zk5L4+/fZ571Pf4Kqv4zbur6Zfo5JbTjq332iWnjADgv+t2hqS9HVVn2uZtSXPbJdjcodwj6AvME5GVwLfAHGPMhyLygIic55/neSBJRHKB24AZIWxPpxvPVET4wwXDOW9kP/78yXr+vjC/2flnL8gnp6iUe88dRveY+qd/nMbLsH4JzMuxqxB0tm3eVjS3XYLNHbKTxcaYlcDoRqbfW+exF/hpqNrQUFFRES6Xq71W1yYiI4T/u3AkB8p9/O79NXSPcTBlzOFLxaqqDIXFB1mzrYRH53zPpMze/HhYnyOWU1RUxGmZvXlyXi5795fTo3t0e8YIm864zduC5rZLsLl1PIJOICoygicvHcPVs7/lzrdWsmFnGXvKysnZUcqGHaUc8N8fkBgbxf3nDWtyaMxTM3vz17m5fPH9Li4YndKeEZRSHZgWgk7CGRXJc1dmc+ULi3nq8zySukczxB3Phdn9GeKOZ3CfeDLd8UccEqprZKqLpO7R/DdnpxYCpVQtqwpBSkrn/uPXPcbBm784kX0HK1p9aCclJYWICGHikN7MWVuEr7IKR2TX72qqs2/zQGluuwSbu+v/JajD6XSGuwlBi4iQgI7v12Q/bWhvSrw+lm7ee5R3dA1dYZsHQnPbJdjcVhWCunfx2aYm+4Rjk3FECHPX23H1kK3bXHPbJdjcVhUCBQnOKMYN6slcy+4nUEo1TQuBhSZl9mbDzjK27jkQ7qYopToAqwqBjdcX16ibfVJmbwDmWnBzma3bXHPbJdjcVhUCt9sd7iaETd3sx/SKIy2pmxWFwNZtrrntEmxuqwpBfn5+uJsQNg2zT8rsw8KNuzlQ7gtPg9qJrdtcc9sl2NxWFYKaPr5t1DD7pMzelPuq+GL9rjC1qH3Yus01t12CzW1VIVCHjRvUk36JTu58ayWfrjnKoBVKqS7NqkLgcFh1I3U9DbNHOyJ46/qTOKZXd6b/fSmPzvmeqqqQ9QAeNrZuc81tl2BzSwi7/w+J7Oxss2TJknA3o8vwVlTym3dX8/Z3BUwe2odHLxpJvDMq3M1SSrUxEVlqjMlu7DWr9gjqjvlpm6ayO6Mi+d+fjuD352Yxb/1OLpg5n427yhqdtzOydZtrbrsEm1sLgSWayy4iXDV+EP+45nj2Hqjg/Cfn8/7ywi5xqMjWba657aKFQLWZE9OT+OBX40lL7s4try/n7Ce+5tM1RXS2w4dKqdbRQqDqSe3RjfduHM8j/lHRpv99Kec9OZ95OTu1ICjVRVl1svjgwYPExsa2cYs6h0Cy+yqreGdZIX/97wYK9h5kVH8Xd50xhJPSk0PUyrZn6zbX3HZpSW49WawC4oiM4MLs/sy9fSJ/mjKcnSVeLn1uEf/36Xoqu8D5A6VUNasKwebNmxfCPREAABgFSURBVMPdhLAJJnu0I4JLxg1g7h0TuSi7P0/MzeXnLy5mz/7yNmxhaNi6zTW3XYLNbVUhUMFxRkXy8LQRPDRlOIs27eHcJ75mxdbicDdLKRUkLQSq1S4eN4C3f3kSAD99eiGvLNrcJieSfZVV/OrV77h69reUeiuCXp5SqmWsKgTJyZ3nJGdba+vsw1MT+fCmCZyQnsRv3l3NHW+uxFtRGdQy//DROj5cuZ3P1+/kslmL2uTQk63bXHPbJdjcWggsEYrsPbpH8+LPf8DNpx3L298V8NOnF1JYfDCgZb28MJ/ZC/K5dsIgnrsym/VFpVz0zEKK9gXXq6Kt21xz20ULQSvk5uaGuwlhE6rskRHCbT8azKwrs8n37Oe8J75mYd7uVi3j8/U7ue+DNUwe2pt7zhrKaUP7MPuqcWwrPshPn1nAlt2BD6lp6zbX3HYJNrdVhcDn69qDsDQn1NknZ/XhvV+Nx9UtisufX8SL8ze16LzB+qJSfvXqMoa4E3j84tFERghQfZfzq9edQKnXx7SnF7C+qDSgdtm6zTW3XYLNbVUhUKGV3iuO924cz6lDenP/v9Zy+5srmj1v4Ck7xNWzv6VbdCTP/yyb7jH1u9Id2d/FG784EYALn1nIcr1CSamQsKoQOJ3OcDchbNore7wzimevGMuvJw/mne8Kmfb0Al5akM+X3+9i654DtTeieSsqmf7yEnbvP8Ssn2XTz9X4XZGD+8Tz1i9PIjE2ip8+vYA731xB7s6W945q6zbX3HYJNrdVXUyo9vXZ2h3c/fZKdte5+ifaEUFaUjciRMgpKuXpy8dwxnF9j7qsXaWHmDkvl9e/3cIhXxWnZ/Xh+okZjOrvCmUEpbqM5rqYsKoQFBUV4Xa727hFnUO4shtj2FV6iI2e/Wzy/2zctZ+CvQe49PgBXHliWquWt7vsELMX5PPSgnxKvD5OSk/ixlMzGJ/R+FUTnWWbv7W0gHXbS/jdOVltsrzOkrutae6mNVcIrBrXrbi42MoPCYQvu4jQO8FJ7wQnJxyTFPTykuJiuP30IfzilHReW7SFWV9v5LJZi/jTlOFcMm7AEfN3hm1efKCc+/+1hlKvj3NG9GX0gB7BL7MT5A4FzR0Yq84RqK4jLsbBdScfw5d3ncrJg3vxu/dWsyC3cw5K8vQXGyk75CMuxsHfPs8Ld3OUhazaI1BdT4wjkicvHc3Uvy3gl/9Yyns3jueYXnGtXo6vsooVBfvYuucAItX3R0SIECHVezXJcdGMGdADEWnT9u8s8TJ7wSYuGJXCwKRuPPbZBtYXlTLEHd+m61GqOVadI6ioqCAqys6B2bt69q17DnD+zPkkOB28e8N4enSPBprPvaPEyxff7+KL73fx9QYP+w4237/RyP4ubp18LBMH92qzgvDb91bx+uKtzL19IgmxDk56aC6nZ/XhsYtHB7Xcrr69m6K5mxaWcwQi0h94GegDGOBZY8zjDeaZCLwPbPJPescY80Co2uT1eq38kEDXz96/ZzeevWIslz63iOtfWcrLVx9PtCPiiNw7Sry88s1mPl27gxz/TWq942M4PasPpwzpRaY7Aag+yV1loMoYqoxhxdZ9zJyXy1UvfttmBWHz7v28vngrF4/rz4CkbgBcdvwAXpifz20/GlI7LRBdfXs3RXMHJpSHhnzA7caY70QkHlgqInOMMWsbzPeVMeacELajVmFhIZmZme2xqg7HhuzZaT15aOpwbntjBb97bzUPTR1em3t14T6e/3oTH67chq/KcPygnsw4M5NTBvci0x1/1D/ow/olMm1sKu98V8ATc/0FITWRWyYfy6lDegdUEB77bAOOSOHmScfWTrv2h8fw0oLNPPNlHg/+ZHirl1nDhu3dGM0dmJAVAmPMdmC7/3GpiKwDUoCGhUCpNjNlTCobd+3nyXm5HNOrOzHl+7nvy4V8s3EP3aMjuez4gVw1Po2BSd1bvexoRwQXjxvAlDHVBeHJeblcPXsJg/vEcc2EQZw/KgVnVGSLlpVTVMJ7ywuZfvIx9E44fDNQnwQn07JTeXNJAbecdmy915QKlXY5RyAiacCXwHHGmJI60ycCbwMFwDbgDmPMmkbePx2YDpCamjp2zpw5ta8NHDgQqD9CT3JyMsnJyeTm5tb2weF0OvF6vbhcLoqLD3dVkJ6ejtfrpbCwsHaa2+3G5XKRk5NTOy0uLo7U1FQKCgooKzt8Z2tmZibFxcUUFRXVTktJScHpdJKXd/gKEJfLhdvtJj8/H6+3ukdNh8NBRkYGHo8Hj+fwFS+tyZSWlkZRUdFRM3m9XkaNGtWlMjW1nfr1S+Hq5+fzed4+AHp1d3DdKRmcMTiRA8WH2xRspn6p/flo1Q6e/TKPTXvLSXRGcNGYvkyfNJTioq3NZrrvv9tZs6ucf9/wA8r27KyXqaQymon/O48pw1xcm50U0HZatGhRbY+UHXU7heKz5/P5SE5O7lKZWrKdPB4PmZmZzWbq27dv+G4oE5E44AvgQWPMOw1eSwCqjDFlInIW8Lgx5tjGllMjmJPFxcXFuFx23olqW/aD5ZU8Mmc9GT2jmDouHUdk6K6UNsawMG83s77exNycnUQ7IvjJqBSuPGkgw/olHjH/d1v2MuVvC7jj9MH8alLjH/dbXl/GZ2t3MH/GJFzdolvdJtu2dw3N3bSw3VksIlHAh8B/jDGPtGD+fCDbGNPkBeHaxYTqyHJ3lvHi/E28/V0B3ooqRqQmcsm4AZw7sh9xMQ6MMVz63CI27CzliztPPaKjvRo5RSWc8dhX/HryYG6Z3Ox3I6VapLlCELKvSVJ99ux5YF1TRUBE3P75EJFx/va0rjP7Vqi7G2cbW7O3d+6M3nE8+JPhLLpnMvedm0W5r4p73lnFuAc/Y8bbK3lpQT4LN+7mxlMzmiwCAJnuBCYP7cOLCzax/1DruxjW7W2XYHOH8qqh8cAVwCoRWe6f9j/AAABjzNPANOB6EfEBB4GLTWe7sUGpRiR2i+Ln4wfxs5PSWLa1mNcXb+H95dt4vaKSFFcslx5/ZHcYDd1wajpT/raD1xZv4dofHkOJt4J8f19NGz372bHPS3ZaD04f5iYxtv0umdx/yMdf/rOeC7P7k9Uvod3Wq0InlFcNfQ00e02dMeZJ4MlQtUGpcBMRxgzowZgBPfjdOVn8e1URQ9zxxDiOfnXRmAE9OPGYJB6d8z1Pf7ERT9mhOsuF+BgH/1yyld+8u5qTBydz9oi+TB7ah3hnaIvCo3O+Z/aCfN5bXsg/rjme41KOPA+iOherupiIi2t91wNdha3ZO1LueGcUF/6gf6veM+PMTP730/X0S4xlUK/uDEruzjHJ3enfsxsxjgiWby3mo5Xb+WjVdj5bV32ieuLgXoxxRxHX5wCpPZq+Kc0YQ96uMr7a4CFChCtPHHjU+yHWbNvHiwvyOfM4NysL9nHZrEW8cm3HKQYdaXu3p2BzW9XFhFJdVVWVYdnWvXy4cjsfr9rOjpLqvYeBSd0Yn5HM+PRkTkxPorLKMD/Xw1cbPMzP9VBU4q1dxh8uOI7LTxjY5DoqqwxTnlpA4d4D/Pe2iZR4K7j42W8oO+TrUMVANU7HI/ArKCggNTW1jVvUOdia3cbcxhi+WplHbmkkC/I8fLNxD2WHfIhAzX/3Ht2iOCkjmR9mJHNSejL3frCa+bke/vmLExnTRDfYf1+Yz+/eX8NjF43igtEpQHUfTxc/+w2l3gpeufYEhqeGtxjYuL2hZbl1PAK/ujd52MbW7DbmFhF6x/g4eWQGV08YREVlFSsL9rEg14MjMoIJGckM65dARMThw0CPXTSKc5/8mhv+8R3/umkCveJj6i1zZ4mXP3+yngkZyZw/ql/t9P49u/H69BO45LlvuGzWN/zj2uMZkdqy6/hLvRW8smgLKwuKGdYvkbEDezAy1UVsdMvuzm6Mjdsbgs9tVSFQykZRkRGMHdiDsQObHvDG1S2apy8fy5S/LeCm177jH9ccX+8mvAc+XMuhyir+3wXHHXEeoX4xWMR95w5j4pBeJMXFNFwNAHv2lzN7/iZm+0eZ65fo5ONV1XfdOiKEoX0TqotC/0TcCbEkx0WTFBeDKzaqXvEK1tptJbyxZCt3n5EZVPHpCrQQKKWA6o71/jSlutO+hz/J4TdnVw+b+cX3u/hw5XZ+PXkwg5Ib76MptUc3Xp9+IlfMWsTtb65ABIanJHLK4F6cMrgXo/q78JSV89xXG3l10RYOVlRyxjA3N5yazohUF3v3l7Ns616+21zM0s17eWPJVmYvyK+3jgiBnt2j6ZPg5N5zsjg+yBHv7vvXGhZv2oOn7BBPXDK6zcea6EysOkeglDq6e99fzcsLN/PkpaOZPLQPpz/6JY4I4d+3/vCol71WVhlWFe7jS/84D8u27KXKQLzTgbeikioD54/qx/WnpHNsn6YH3/FVVrHJs59dZYfYXVbO7rJD7N5fjqesnK827OKQr4r/3HoyPbu3vvsNgCX5e5j29EJG9nexYmsxd/54CDeemhHQsjoLPVnsZ2s/JGBvds3deuW+Ki557hvWbS/h9Kw+vLd8G69edzwnpSe3eln7DlQwP8/DVxt2ERvl4KrxafTvGfg4C1B9SOeCmfM5ZUgvnr1ibL1v8i3NffXsb1m2ZS/zZ0zinndW8f7ybTx7xVhOH9Y5xzsOtq8hq8Ysrtv7n21sza65Wy/aEcHfLhtDt2gH7y3fxpTRKQEVAai+w/qs4X3505QR3HtuVtBFACCrXwJ3nTGEOWt38Pq3W+u91pLca7eVMDdnJ1ePH0S3aAcPTx3BiNREfv3P5az3D1bU2QT7ObeqECilWqZPgpNnrhjLj4f14X/OHhru5hzh6vGDmJCRzAP/WsvGXa27YuapL/KIi3Fw5YlpADijInn2imy6xTi49uVv2bu/PAQt7ti0ECilGjV2YA+euSKb5Cau/gmniAjh/y4cSUxUBLe8vpxyX1WL3pfv2c9HK7dx2QkDSOx2uCsOd6KTZ68Yy46SQ9zwyndUVLZseV2FVYUgJSUl3E0IG1uza+6uq0+Ck4emjGBV4T4e++x74Oi5n/kyD0dkBNdMGHTEa6MH9OBPPxnOwo27eeBfa6msOvr5U2MM3+8o5cX5m/g2f09gQdpAsNvbqstHnU57h/2zNbvm7trOOM7NRdn9eeqLPE4Z3Isx/ZvuDbVon5e3lhZw0Q/60zu+8d/P1LGprN9RyrNfbuTdZYWM7J/I6P49GD3Axaj+LpLiYijxVrAg18MX3+/ii/W72LbPP5pYhPDkpaM547i+IcnanGC3t1WFIC8vz8qBrcHe7Jq767v33CwWbdrNr/+5nL+e5SZ75LBG55v11UaqDPzi5PRml3f3GZkcl5LIt5v2sGzrXp76Iq9276BfopMdpYeorDLExziYcGwyN5/Wi7EDe3D32yu58dVlPHEJnDW8fYtBsNvbqkKglOp6usc4eOzi0Ux9agF3fbKN/9e9D+Mz6l/ltHd/Oa8u3sL5I/sd9cqlyAjhvJH9OG9kdVcaB8srWVW4j2Vb9rKqcB8Dk7pxyuDejB7gIqrO3dcvXT2Oq178lpteW0aVMZwzol9Tq+hwtBAopTq9Uf1dPHXZGH77zgoum7WIiUN6MePMTDLd1YeKZi/I50B5Jb+c2PzeQGNioyMZN6gn4wb1bHa+eGcUs68ex9Uvfsstry+nylBbTDo6q04W23hjUQ1bs2tue5w+zM2bVw3nN2cN5bvNeznr8a+4660V5O0qY/aCfE7P6sPgZu5mbgtxMQ5evOoHZA/swa2vL+O9ZYUhXV+NYLe3VXcWK6XsUHygnJnzcnlpwWbK/ZeCvnfjeEb1b58CeaDcxzWzl7Bo027u+PEQRvfvQT+XE3eis0Wj04WCdjHhl5+fT1paWts2qJOwNbvmtkvD3Fv3HODROd8TGx3Jgz8Z3q5tOVheyfS/L+GrDZ5605PjoumbGEtSXDTlvioOlFfirajkYEUlB8orqaoynDuyH9dPTKdPQsuuBmrJ9tbxCPy8Xu/RZ+qibM2uue3SMHf/nt145KJRYWlLbHQkL101jvzd+9m+z8u24oNs3+dl+76DbCv2srusHGdUBPFOB30SYoiNiiQ22kGpt4K/f7OZVxdv4dJxA1pUEILd3lYVAqWUak8REcIxveI4plfrxhTesvsAM+fl8vdvNvPa4i1cevwArj8lnd4t3ENodTtDstQOyuGwt+7Zml1z26Wr5B6Q1I2Hp41g3u0TuWBUCi8v3MwP/zyPWV9tbHT+YHNbdY5AKaU6oy27D/DkvA1Myuwd8J3L2g21n8fjOfpMXZSt2TW3Xbpq7gFJ3fjztJFNFoFgc2shsISt2TW3XTR3YKwqBEoppY6khUAppSxnVSEYOHBguJsQNrZm19x20dyBsaoQKKWUOpJVhWDz5s3hbkLY2Jpdc9tFcwfGqkKglFLqSFoIlFLKcp3uzmIR2QUEuh+UDNh5obG92TW3XTR30wYaY3o19kKnKwTBEJElTd1i3dXZml1z20VzB0YPDSmllOW0ECillOVsKwTPhrsBYWRrds1tF80dAKvOESillDqSbXsESimlGtBCoJRSlrOmEIjIGSKyXkRyRWRGuNsTKiLygojsFJHVdab1FJE5IrLB/2+PcLYxFESkv4jME5G1IrJGRG7xT+/S2UXEKSKLRWSFP/f9/umDRGSR//P+TxGJDndbQ0FEIkVkmYh86H/e5XOLSL6IrBKR5SKyxD8tqM+5FYVARCKBmcCZQBZwiYhkhbdVITMbOKPBtBnAf40xxwL/9T/vanzA7caYLOAE4Eb/Nu7q2Q8Bk4wxI4FRwBkicgLwMPCoMSYD2AtcE8Y2htItwLo6z23JfaoxZlSdeweC+pxbUQiAcUCuMWajMaYceB04P8xtCgljzJfAngaTzwde8j9+CbigXRvVDowx240x3/kfl1L9xyGFLp7dVCvzP43y/xhgEvCWf3qXyw0gIqnA2cAs/3PBgtxNCOpzbkshSAG21nle4J9miz7GmO3+x0VAn3A2JtREJA0YDSzCguz+wyPLgZ3AHCAPKDbG+PyzdNXP+2PAXUCV/3kSduQ2wKcislREpvunBfU5d7Rl61THZ4wxItJlrxkWkTjgbeBWY0xJ9ZfEal01uzGmEhglIi7gXSAzzE0KORE5B9hpjFkqIhPD3Z52NsEYUygivYE5IpJT98VAPue27BEUAv3rPE/1T7PFDhHpC+D/d2eY2xMSIhJFdRF4xRjzjn+yFdkBjDHFwDzgRMAlIjVf9Lri5308cJ6I5FN9qHcS8DhdPzfGmEL/vzupLvzjCPJzbksh+BY41n9FQTRwMfBBmNvUnj4AfuZ//DPg/TC2JST8x4efB9YZYx6p81KXzi4ivfx7AohILPAjqs+PzAOm+WfrcrmNMfcYY1KNMWlU/3+ea4y5jC6eW0S6i0h8zWPgdGA1QX7OrbmzWETOovqYYiTwgjHmwTA3KSRE5DVgItXd0u4Afg+8B7wBDKC6C+8LjTENTyh3aiIyAfgKWMXhY8b/Q/V5gi6bXURGUH1yMJLqL3ZvGGMeEJFjqP6m3BNYBlxujDkUvpaGjv/Q0B3GmHO6em5/vnf9Tx3Aq8aYB0UkiSA+59YUAqWUUo2z5dCQUkqpJmghUEopy2khUEopy2khUEopy2khUEopy2khUKoBEan09+xY89NmHdWJSFrdnmGV6gi0iwmljnTQGDMq3I1Qqr3oHoFSLeTvB/7P/r7gF4tIhn96mojMFZGVIvJfERngn95HRN71jxWwQkRO8i8qUkSe848f8Kn/jmClwkYLgVJHim1waOiiOq/tM8YMB56k+k51gCeAl4wxI4BXgL/6p/8V+MI/VsAYYI1/+rHATGPMMKAYmBriPEo1S+8sVqoBESkzxsQ1Mj2f6kFgNvo7uCsyxiSJiAfoa4yp8E/fboxJFpFdQGrdLg78XWTP8Q8ggojcDUQZY/4Q+mRKNU73CJRqHdPE49ao2/dNJXquToWZFgKlWueiOv8u9D9eQHUPmACXUd35HVQPGXg91A4ek9hejVSqNfSbiFJHivWP+FXjE2NMzSWkPURkJdXf6i/xT7sJeFFE7gR2AVf5p98CPCsi11D9zf96YDtKdTB6jkCpFvKfI8g2xnjC3Ral2pIeGlJKKcvpHoFSSllO9wiUUspyWgiUUspyWgiUUspyWgiUUspyWgiUUspy/x/TaxvliyJDIwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "render_training_history(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "X-dhNP2OG2EM"
   },
   "source": [
    "## Generate text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SU_YfP6sG3NC"
   },
   "source": [
    "### Restore the latest checkpoint\n",
    "\n",
    "To keep this prediction step simple, use a batch size of 1.\n",
    "\n",
    "Because of the way the RNN state is passed from timestep to timestep, the model only accepts a fixed batch size once built.\n",
    "\n",
    "To run the model with a different `batch_size`, we need to rebuild the model and restore the weights from the checkpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 23,
     "status": "ok",
     "timestamp": 1586982234408,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "BlG8o3wiG6f2",
    "outputId": "944da6fc-cf4e-4833-e14e-2526aaf05445"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'tmp/checkpoints/ckpt_100'"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.train.latest_checkpoint(checkpoint_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "l7evN0LvH01P"
   },
   "outputs": [],
   "source": [
    "simplified_batch_size = 1\n",
    "\n",
    "restored_model = build_model(vocab_size, embedding_dim, rnn_units, batch_size=1)\n",
    "\n",
    "restored_model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))\n",
    "\n",
    "restored_model.build(tf.TensorShape([simplified_batch_size, None]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 255
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 60,
     "status": "ok",
     "timestamp": 1586982235202,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -120
    },
    "id": "Y3eduDtZI9zQ",
    "outputId": "461d4fb4-8059-444e-b602-9b9977336609"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_3\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_3 (Embedding)      (1, None, 256)            158976    \n",
      "_________________________________________________________________\n",
      "lstm_2 (LSTM)                (1, None, 1024)           5246976   \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (1, None, 621)            636525    \n",
      "=================================================================\n",
      "Total params: 6,042,477\n",
      "Trainable params: 6,042,477\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "restored_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "euNvAtr4JC3A"
   },
   "source": [
    "### The prediction loop\n",
    "\n",
    "The following code block generates the text:\n",
    "\n",
    "- It Starts by choosing a start string, initializing the RNN state and setting the number of characters to generate.\n",
    "\n",
    "- Get the prediction distribution of the next character using the start string and the RNN state.\n",
    "\n",
    "- Then, use a categorical distribution to calculate the index of the predicted character. Use this predicted character as our next input to the model.\n",
    "\n",
    "- The RNN state returned by the model is fed back into the model so that it now has more context, instead than only one character. After predicting the next character, the modified RNN states are again fed back into the model, which is how it learns as it gets more context from the previously predicted characters.\n",
    "\n",
    "![Prediction loop](https://www.tensorflow.org/tutorials/text/images/text_generation_sampling.png)\n",
    "\n",
    "Image source: [Text generation with an RNN](https://www.tensorflow.org/tutorials/text/text_generation) notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bOqdqGouJFf_"
   },
   "outputs": [],
   "source": [
    "# num_generate\n",
    "# - number of characters to generate.\n",
    "#\n",
    "# temperature\n",
    "# - Low temperatures results in more predictable text.\n",
    "# - Higher temperatures results in more surprising text.\n",
    "# - Experiment to find the best setting.\n",
    "def generate_text(model, start_string, num_generate = 1000, temperature=1.0):\n",
    "    # Evaluation step (generating text using the learned model)\n",
    "\n",
    "    # Converting our start string to numbers (vectorizing).\n",
    "    input_indices = [char2index[s] for s in start_string]\n",
    "    input_indices = tf.expand_dims(input_indices, 0)\n",
    "\n",
    "    # Empty string to store our results.\n",
    "    text_generated = []\n",
    "\n",
    "    # Here batch size == 1.\n",
    "    model.reset_states()\n",
    "    for char_index in range(num_generate):\n",
    "        predictions = model(input_indices)\n",
    "        # remove the batch dimension\n",
    "        predictions = tf.squeeze(predictions, 0)\n",
    "\n",
    "        # Using a categorical distribution to predict the character returned by the model.\n",
    "        predictions = predictions / temperature\n",
    "        predicted_id = tf.random.categorical(\n",
    "        predictions,\n",
    "        num_samples=1\n",
    "        )[-1,0].numpy()\n",
    "\n",
    "        # We pass the predicted character as the next input to the model\n",
    "        # along with the previous hidden state.\n",
    "        input_indices = tf.expand_dims([predicted_id], 0)\n",
    "\n",
    "        text_generated.append(index2char[predicted_id])\n",
    "\n",
    "    return (start_string + ''.join(text_generated))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Temperature: 0.2\n",
      "---\n",
      "Science is a species of the station of the season is a species of the company of the complete to the company of the company of the station of the company of the company of the station of the company of the company of the company of the company of the company of the company of the town of the company of the st\n",
      "\n",
      "\n",
      "Temperature: 0.4\n",
      "---\n",
      "Science is a restance of the color personal lines, and granting of the music of the company color in the forming the color players in the line in the color color and the color have a form the harpsic of the southern services in the control of the color of the competition of the focus of the come of the throug\n",
      "\n",
      "\n",
      "Temperature: 0.6\n",
      "---\n",
      "Science is a wingles of the city is with made end of the color and time. In the 106 million saw the moving public strains, and station in the strategy and church of his resistance of the Urderland for Commonther and Loya redistance of a color personal milital responsible reaching a MRSA victory of the New Yor\n",
      "\n",
      "\n",
      "Temperature: 0.8\n",
      "---\n",
      "Science is the town in the Ulive teams called to supporters. Louis named in the Color Business Scriptic Last got's successful in MAC Jument Renuary Phortigon, 1945 that conceptions. It is known as the United States in the Film Federation of a hamits resis, the medal) but combines for Otary Hally and fush the \n",
      "\n",
      "\n",
      "Temperature: 1.0\n",
      "---\n",
      "Science is Austrian ronum - RRLA, a Farsonti U.S. Robert Hair's; Douing violers. By NCR-Fold Avalbu)\n",
      "\n",
      " i  Liguten, nake (Roving (44)\n",
      "\n",
      "Arts early M1 A Jamy (2000) agrovio (1:00)The N9742Thie Peter Wing 1, 17; 3440 El-NWWA Antemphia retsider (1.332 (and may. A Qarround, it was Place (MR. St. national journalisy\n",
      "\n",
      "\n",
      "Temperature: 1.2\n",
      "---\n",
      "Science is the H.Shulftro, by Sz –40羹)\n",
      " Warger Ubay Australia\n",
      " Thua, Bing\".\n",
      "At the Southern 0s\" life or vocal services to wroting-Gebrareford 2018, 1597.\n",
      "\n",
      "Ploger\n",
      " Builnnels, capt\n",
      "Apartleto with its carling disbang-his limitti MRS 14 pizcryctor\n",
      "Category:Cornen\n",
      "\n",
      "Released Gamocre, MA)4, Politics, but herrisekrif\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "num_generate = 300\n",
    "temperatures = [0.2, 0.4, 0.6, 0.8, 1.0, 1.2]\n",
    "start_string = 'Science is'\n",
    "\n",
    "for temperature in temperatures:\n",
    "    print(\"Temperature: {}\".format(temperature))\n",
    "    print('---')\n",
    "    print(generate_text(restored_model, start_string, num_generate=num_generate, temperature=temperature))\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0hh80MqEO_XI"
   },
   "source": [
    "## Save the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VPE98xa8PA-u"
   },
   "outputs": [],
   "source": [
    "model_name = 'text_generation_wikipedia_rnn.h5'\n",
    "restored_model.save(model_name, save_format='h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WYP08xbbTNKp"
   },
   "source": [
    "## Converting the model to web-format\n",
    "\n",
    "To use this model on the web we need to convert it into the format that will be understandable by [tensorflowjs](https://www.tensorflow.org/js). To do so we may use [tfjs-converter](https://github.com/tensorflow/tfjs/tree/master/tfjs-converter) as following:\n",
    "\n",
    "```\n",
    "tensorflowjs_converter --input_format keras \\\n",
    "  ./experiments/text_generation_wikipedia_rnn/text_generation_wikipedia_rnn.h5 \\\n",
    "  ./demos/public/models/text_generation_wikipedia_rnn\n",
    "```\n",
    "\n",
    "You find this experiment in the [Demo app](https://trekhleb.github.io/machine-learning-experiments) and play around with it right in you browser to see how the model performs in real life."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "text_generation_wikipedia_rnn.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}