lectures/05_word-embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/563_banner.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Lecture 5: Word Embeddings, word2vec\n",
    "UBC Master of Data Science program, 2023-24\n",
    "\n",
    "Instructor: Varada Kolhatkar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lecture plan, imports, LOs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lecture plan \n",
    "\n",
    "- Introduction and motivation (~5 mins)\n",
    "- Summary of pre-watch videos (~20 mins)\n",
    "- Demo of word2vec skipgram (extremely inefficient version) on a toy corpus (~10 mins)\n",
    "- Q&A and T/F questions (~5 mins )\n",
    "- Break (~5 mins)\n",
    "- Training word2vec (~10 mins)\n",
    "- Word analogies, biases and stereotypes in word embeddings (~10 mins)\n",
    "- Q&A and T/F questions (~5 mins )\n",
    "- Final comments, summary, reflection (~5 mins) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Imports "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "import sys\n",
    "import time\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "sys.path.append(os.path.join(os.path.abspath(\".\"), \"code\"))\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from comat import CooccurrenceMatrix\n",
    "from preprocessing import MyPreprocessor\n",
    "from sklearn.decomposition import PCA, TruncatedSVD\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "# from support_functions import *\n",
    "\n",
    "plt.rcParams[\"font.size\"] = 16\n",
    "import matplotlib.cm as cm\n",
    "\n",
    "%matplotlib inline\n",
    "pd.set_option(\"display.max_colwidth\", 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Learning outcomes\n",
    "\n",
    "From this lecture, students are expected to be able to:\n",
    "\n",
    "- Explain the general idea of a vector space model.\n",
    "- Explain the difference between different word representations: one-hot encoding, term-term co-occurrence matrix representation, and word2vec representation. \n",
    "- Explain the skip-gram model at a high level.\n",
    "- Load and use pre-trained word embeddings to find word similarities and analogies.  \n",
    "- Train your own word vectors with `Gensim`.\n",
    "- Use word2vec models for word similarity and analogies.\n",
    "- Demonstrate biases in word2vec models and learn to watch out for such biases in pre-trained embeddings.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 1. Motivation and context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Do large language models, such as ChatGPT, \"understand\" your questions to some extent and provide useful responses?\n",
    "- What is required for a machine to \"understand\" language?\n",
    "  \n",
    "- Last week, we discussed representing numeric data with PCA and text with LSA or TruncatedSVD.\n",
    "  \n",
    "- So far we have been talking about sentence or document representations. \n",
    "- This week, we'll go one step back and talk about word representations. \n",
    "- Why? Because word is a basic semantic unit of text and in order to capture meaning of text it is useful to capture word meaning (e.g., in terms of relationships between words).  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Activity: Context and word meaning \n",
    "\n",
    "- Pair up with the person next to you and try to guess the meanings of two made-up words: **flibbertigibbet** and **groak**.\n",
    "\n",
    "\n",
    "> 1. The plot twist was totally unexpected, making it a **flibbertigibbet** experience.\n",
    "> 2. Despite its **groak** special effects, the storyline captivated my attention till the end.\n",
    "> 3. I found the character development rather **groak**, failing to evoke empathy.\n",
    "> 4. The cinematography is **flibbertigibbet**, showcasing breathtaking landscapes.\n",
    "> 5. A **groak** narrative that could have been saved with better direction.\n",
    "> 6. This movie offers a **flibbertigibbet** blend of humour and action, a must-watch.\n",
    "> 7. Sadly, the movie's potential was overshadowed by its **groak** pacing.\n",
    "> 8. The soundtrack complemented the film's theme perfectly, adding to its **flibbertigibbet** charm.\n",
    "> 9. It's rare to see such a **flibbertigibbet** performance by the lead actor.\n",
    "> 10. Despite high expectations, the film turned out to be quite **groak**.\n",
    "> 11. **Flibbertigibbet** dialogues and a gripping plot make this movie stand out.\n",
    "> 12. The film's **groak** screenplay left much to be desired.\n",
    "\n",
    "Attributions: Thanks to ChatGPT! \n",
    "\n",
    "- How did you infer the meaning of the words **flibbertigibbet** and **groak**?\n",
    "- Which specific words or phrases in the context helped you infer the meaning of these imaginary words? \n",
    "\n",
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What you did in the above activity is referred to as **distributional hypothesis**. \n",
    "\n",
    "<blockquote> \n",
    "    <p>You shall know a word by the company it keeps.</p>\n",
    "    <footer>Firth, 1957</footer>        \n",
    "</blockquote>\n",
    "\n",
    "<blockquote> \n",
    "If A and B have almost identical environments we say that they are synonyms.\n",
    "<footer>Harris, 1954</footer>    \n",
    "</blockquote>    \n",
    "\n",
    "Example: \n",
    "\n",
    "- The plot twist was totally unexpected, making it a **flibbertigibbet** experience.\n",
    "- The plot twist was totally unexpected, making it a **delightful** experience."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 1.2 Word representations: intro \n",
    "- A standard way to represent meanings of words is by placing them into a vector space.\n",
    "- Distances between words in the vector space indicate relationships between them.\n",
    "   \n",
    "<!-- <img src=\"img/t-SNE_word_embeddings.png\" width=\"600\" height=\"600\"> -->\n",
    "\n",
    "![](img/t-SNE_word_embeddings.png)\n",
    " \n",
    "(Attribution: [Jurafsky and Martin 3rd edition](https://web.stanford.edu/~jurafsky/slp3/))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Word meaning has been a favourite topic of philosophers for centuries. \n",
    "- An example from legal domain: [Are hockey gloves \"gloves, mittens, mitts\" or \"articles of plastics\"?](https://www.scc-csc.ca/case-dossier/info/sum-som-eng.aspx?cas=36258)\n",
    "\n",
    "<blockquote>\n",
    "Canada (A.G.) v. Igloo Vikski Inc. was a tariff code case that made its way to the SCC (Supreme Court of Canada). The case disputed the definition of hockey gloves as either \"gloves, mittens, or mitts\" or as \"other articles of plastic.\"\n",
    "</blockquote>\n",
    "\n",
    "![](img/hockey_gloves_case.png)\n",
    "<!-- <center>\n",
    "<img src=\"img/hockey_gloves_case.png\" width=\"500\" height=\"500\">\n",
    "</center>\n",
    " -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In ML and Natural Language Processing (NLP) we are interested in \n",
    "- Modeling word meaning that allows us to \n",
    "    - draw useful inferences to solve meaning-related problems \n",
    "    - find relationship between words, e.g., which words are similar, which ones have positive or negative connotations\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Example: Word similarity**\n",
    "\n",
    "- Suppose you are carrying out sentiment analysis. \n",
    "- Consider the sentences below. \n",
    "\n",
    "> S1: This movie offers a **flibbertigibbet** blend of humour and action, a must-watch.\n",
    "\n",
    "> S2: This movie offers a **delightful** blend of humour and action, a must-watch.\n",
    "\n",
    "- Here we would like to capture similarity between **flibbertigibbet** and **delightful** in reference to sentiment analysis task. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**How are word embeddings related to unsupervised learning?**\n",
    "\n",
    "- They are closely related to dimensionality reduction and extracting meaningful representations from raw data. \n",
    "- The word2vec algorithm is an unsupervised (or semi-supervised) method; we do not need any labeled data but we use running text as supervision signal.\n",
    "- We can build recommendation systems using word2vec algorithm (you'll explore this in the lab) which is the topic of next week.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 2. Word representations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Activity:  Brainstorm ways to represent words (~2 mins) \n",
    "\n",
    "- Suppose you are building a question answering system and you are given the following question and three candidate answers. \n",
    "- What kind of relationship between words would we like our representation to capture in order to arrive at the correct answer?  \n",
    "    \n",
    "<blockquote>       \n",
    "<p style=\"font-size:20px\"><b>Question:</b> How <b>tall</b> is Machu Picchu?</p>\n",
    "    <p style=\"font-size:20px\"><b>Candidate 1:</b> Machu Picchu is 13.164 degrees south of the equator.</p>    \n",
    "<p style=\"font-size:20px\"><b>Candidate 2:</b> The official height of Machu Picchu is 2,430 m.</p>\n",
    "<p style=\"font-size:20px\"><b>Candidate 3:</b> Machu Picchu is 80 kilometres (50 miles) northwest of Cusco.</p>    \n",
    "</blockquote> \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Let's explore different ways to represent words. \n",
    "- First, let's look at two simplistic word representations  \n",
    "    - One-hot representation\n",
    "    - Term-term co-occurrence matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 2.1 Simplest representation: One-hot representation of words \n",
    "\n",
    "- Example: Consider the sentence\n",
    "> How tall is Machu_Picchu ?\n",
    "\n",
    "- What is the one-hot representation for the word *tall*?\n",
    "    * Vocabulary size = 5 and index of the word *tall* = 1\n",
    "    * One-hot vector for *tall*: $\\begin{bmatrix} 0 & 1 & 0 & 0 & 0 \\end{bmatrix}$\n",
    "    \n",
    "- Build **vocabulary** containing all unique words in the corpus. \n",
    "- One-hot representation of a word is a vector of length $V$ such that the value at word index is 1 and all other indices is 0.    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def get_onehot_encoding(word, vocab):\n",
    "    onehot = np.zeros(len(vocab), dtype=\"float64\")\n",
    "    onehot[vocab[word]] = 1\n",
    "    return onehot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "\n",
    "def print_cosine_similarity(df, word1, word2):\n",
    "    \"\"\"\n",
    "    Returns similarity score between word1 and word2\n",
    "    Arguments\n",
    "    ---------\n",
    "    df    -- (pandas.DataFrame)\n",
    "        Dataframe containing word representations\n",
    "    word1 -- (array)\n",
    "        Representation of word1\n",
    "    word2 -- (array)\n",
    "        Representation of word2\n",
    "\n",
    "    Returns\n",
    "    --------\n",
    "    None. Returns similarity score between word1 and word2 with the given representation\n",
    "    \"\"\"\n",
    "    vec1 = df.loc[word1].values.reshape(1, -1)\n",
    "    vec2 = df.loc[word2].values.reshape(1, -1)\n",
    "    sim = cosine_similarity(vec1, vec2)\n",
    "    print(\n",
    "        \"The dot product between %s and %s: %0.2f and cosine similarity is: %0.2f\"\n",
    "        % (word1, word2, np.dot(vec1.flatten(), vec2.flatten()), sim[0][0])\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Vocabulary and one-hot encoding**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of the vocabulary: 12\n",
      "{'.': 0, '2,430': 1, '?': 2, 'height': 3, 'how': 4, 'is': 5, 'm': 6, 'machu_picchu': 7, 'of': 8, 'official': 9, 'tall': 10, 'the': 11}\n"
     ]
    }
   ],
   "source": [
    "corpus = (\n",
    "    \"\"\"how tall is machu_picchu ? the official height of machu_picchu is 2,430 m .\"\"\"\n",
    ")\n",
    "unique_words = list(set(corpus.split()))\n",
    "unique_words.sort()\n",
    "vocab = {word: index for index, word in enumerate(unique_words)}\n",
    "print(\"Size of the vocabulary: %d\" % (len(vocab)))\n",
    "print(vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>.</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2,430</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>?</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>height</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>how</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>is</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>m</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>machu_picchu</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>of</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>official</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tall</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>the</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               0    1    2    3    4    5    6    7    8    9    10   11\n",
       ".             1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0\n",
       "2,430         0.0  1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0\n",
       "?             0.0  0.0  1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0\n",
       "height        0.0  0.0  0.0  1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0\n",
       "how           0.0  0.0  0.0  0.0  1.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0\n",
       "is            0.0  0.0  0.0  0.0  0.0  1.0  0.0  0.0  0.0  0.0  0.0  0.0\n",
       "m             0.0  0.0  0.0  0.0  0.0  0.0  1.0  0.0  0.0  0.0  0.0  0.0\n",
       "machu_picchu  0.0  0.0  0.0  0.0  0.0  0.0  0.0  1.0  0.0  0.0  0.0  0.0\n",
       "of            0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  1.0  0.0  0.0  0.0\n",
       "official      0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  1.0  0.0  0.0\n",
       "tall          0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  1.0  0.0\n",
       "the           0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  1.0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = {}\n",
    "for word in vocab:\n",
    "    data[word] = get_onehot_encoding(word, vocab)\n",
    "ohe_df = pd.DataFrame(data).T\n",
    "ohe_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dot product between tall and height: 0.00 and cosine similarity is: 0.00\n",
      "The dot product between tall and official: 0.00 and cosine similarity is: 0.00\n"
     ]
    }
   ],
   "source": [
    "print_cosine_similarity(ohe_df, \"tall\", \"height\")\n",
    "print_cosine_similarity(ohe_df, \"tall\", \"official\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem with one-hot encoding**\n",
    "\n",
    "- We would like the word representation to capture the similarity between _tall_ and _height_ and so we would like them to have bigger dot product or bigger cosine similarity (normalized dot product). \n",
    "- The problem with one-hot representation of words is that there is no inherent notion of relationship between words and the dot product between similar and non-similar words is zero.\n",
    "\n",
    "$$ \n",
    "\\vec{tall}\\cdot\\vec{height} = 0\\\\ \n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Need a representation that captures relationships between words.**\n",
    "\n",
    "- We will be looking at two such representations.  \n",
    "    1. Sparse representation with **term-term co-occurrence matrix**\n",
    "    2. Dense representation with **word2vec skip-gram model**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 2.2 Term-term co-occurrence matrix\n",
    "\n",
    "- So far we have been talking about documents and we created document-term co-occurrence matrix (e.g., bag-of-words representation of text). \n",
    "- We can also do this with words. The idea is to go through a corpus of text, keeping a count of all of the words that appear in context of each word (within a window)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "**Visualizing word vectors and similarity**\n",
    "\n",
    "![](img/word_vectors_and_angles.png)\n",
    "<!-- <center>\n",
    "<img src=\"img/word_vectors_and_angles.png\" width=\"800\" height=\"800\">\n",
    "</center>\n",
    "     -->\n",
    "(Credit: Jurafsky and Martin 3rd edition)\n",
    "\n",
    "- The similarity is calculated using dot products between word vectors.\n",
    "    - Example: $\\vec{\\text{digital}}.\\vec{\\text{information}} = 0 \\times 1 + 1\\times 6 = 6$\n",
    "    - Higher the dot product more similar the words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "- The similarity is calculated using dot products between word vectors.\n",
    "    - Example: $\\vec{\\text{digital}}.\\vec{\\text{information}} = 0 \\times 1 + 1\\times 6 = 6$\n",
    "    - Higher the dot product more similar the words.\n",
    "\n",
    "- We can also calculate a normalized version of dot products. \n",
    "    $$similarity_{cosine}(w_1,w_2) = \\frac{w_1.w_2}{\\left\\lVert w_1\\right\\rVert_2 \\left\\lVert w_2\\right\\rVert_2}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "corpus = [\n",
    "    \"How tall is Machu Picchu?\",\n",
    "    \"Machu Picchu is 13.164 degrees south of the equator.\",\n",
    "    \"The official height of Machu Picchu is 2,430 m.\",\n",
    "    \"Machu Picchu is 80 kilometres (50 miles) northwest of Cusco.\",\n",
    "    \"It is 80 kilometres (50 miles) northwest of Cusco, on the crest of the mountain Machu Picchu, located about 2,430 metres (7,970 feet) above mean sea level, over 1,000 metres (3,300 ft) lower than Cusco, which has an elevation of 3,400 metres (11,200 ft).\",\n",
    "]\n",
    "sents = MyPreprocessor(corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tall</th>\n",
       "      <th>machu</th>\n",
       "      <th>picchu</th>\n",
       "      <th>13.164</th>\n",
       "      <th>degrees</th>\n",
       "      <th>south</th>\n",
       "      <th>equator</th>\n",
       "      <th>official</th>\n",
       "      <th>height</th>\n",
       "      <th>2,430</th>\n",
       "      <th>...</th>\n",
       "      <th>mean</th>\n",
       "      <th>sea</th>\n",
       "      <th>level</th>\n",
       "      <th>1,000</th>\n",
       "      <th>3,300</th>\n",
       "      <th>ft</th>\n",
       "      <th>lower</th>\n",
       "      <th>elevation</th>\n",
       "      <th>3,400</th>\n",
       "      <th>11,200</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tall</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>machu</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>picchu</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13.164</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>degrees</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         tall  machu  picchu  13.164  degrees  south  equator  official  \\\n",
       "tall     0     1      1       0       0        0      0        0          \n",
       "machu    1     0      5       1       0        0      0        1          \n",
       "picchu   1     5      0       1       1        0      0        0          \n",
       "13.164   0     1      1       0       1        1      0        0          \n",
       "degrees  0     0      1       1       0        1      1        0          \n",
       "\n",
       "         height  2,430  ...  mean  sea  level  1,000  3,300  ft  lower  \\\n",
       "tall     0       0      ...  0     0    0      0      0      0   0       \n",
       "machu    1       1      ...  0     0    0      0      0      0   0       \n",
       "picchu   1       2      ...  0     0    0      0      0      0   0       \n",
       "13.164   0       0      ...  0     0    0      0      0      0   0       \n",
       "degrees  0       0      ...  0     0    0      0      0      0   0       \n",
       "\n",
       "         elevation  3,400  11,200  \n",
       "tall     0          0      0       \n",
       "machu    0          0      0       \n",
       "picchu   0          0      0       \n",
       "13.164   0          0      0       \n",
       "degrees  0          0      0       \n",
       "\n",
       "[5 rows x 32 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm = CooccurrenceMatrix(\n",
    "    sents, window_size=2\n",
    ")  # Let's build term-term co-occurrence matrix for our text.\n",
    "comat = cm.fit_transform()\n",
    "vocab = cm.get_feature_names()\n",
    "df = pd.DataFrame(comat.todense(), columns=vocab, index=vocab, dtype=np.int8)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dot product between tall and height: 2.00 and cosine similarity is: 0.82\n",
      "The dot product between tall and official: 1.00 and cosine similarity is: 0.50\n"
     ]
    }
   ],
   "source": [
    "print_cosine_similarity(df, \"tall\", \"height\")\n",
    "print_cosine_similarity(df, \"tall\", \"official\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- We are getting non-zero cosine similarity now and we are able to capture some similarities between words now. \n",
    "- That said similarities do not make much sense in the toy example above because we're using a tiny corpus. \n",
    "- To find meaningful patterns of similarities between words, we need a large corpus. \n",
    "- Let's try a bit larger corpus and check whether the similarities make sense. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of sentences in the corpus:  709\n"
     ]
    }
   ],
   "source": [
    "import wikipedia\n",
    "from nltk.tokenize import sent_tokenize, word_tokenize\n",
    "\n",
    "corpus = []\n",
    "\n",
    "queries = [\n",
    "    \"Machu Picchu\",\n",
    "  # \"Everest\",\n",
    "    \"Sequoia sempervirens\",\n",
    "    \"President (country)\",\n",
    "    \"Politics Canada\",\n",
    "]\n",
    "\n",
    "for i in range(len(queries)):\n",
    "    sents = sent_tokenize(wikipedia.page(queries[i]).content)\n",
    "    corpus.extend(sents)\n",
    "print(\"Number of sentences in the corpus: \", len(corpus))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>machu</th>\n",
       "      <th>picchu</th>\n",
       "      <th>15th-century</th>\n",
       "      <th>inca</th>\n",
       "      <th>citadel</th>\n",
       "      <th>located</th>\n",
       "      <th>eastern</th>\n",
       "      <th>cordillera</th>\n",
       "      <th>southern</th>\n",
       "      <th>peru</th>\n",
       "      <th>...</th>\n",
       "      <th>comprehensive</th>\n",
       "      <th>overview</th>\n",
       "      <th>cbc</th>\n",
       "      <th>digital</th>\n",
       "      <th>archives</th>\n",
       "      <th>boondoggles</th>\n",
       "      <th>elephants</th>\n",
       "      <th>campaigning</th>\n",
       "      <th>compared</th>\n",
       "      <th>textbook</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>machu</th>\n",
       "      <td>0</td>\n",
       "      <td>79</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>picchu</th>\n",
       "      <td>79</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15th-century</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>inca</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>citadel</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>boondoggles</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>elephants</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>campaigning</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>compared</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>textbook</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4190 rows × 4190 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              machu  picchu  15th-century  inca  citadel  located  eastern  \\\n",
       "machu         0      79      1             5     1        2        1         \n",
       "picchu        79     0       1             6     3        1        1         \n",
       "15th-century  1      1       0             1     1        1        0         \n",
       "inca          5      6       1             0     1        1        1         \n",
       "citadel       1      3       1             1     0        1        1         \n",
       "...          ..     ..      ..            ..    ..       ..       ..         \n",
       "boondoggles   0      0       0             0     0        0        0         \n",
       "elephants     0      0       0             0     0        0        0         \n",
       "campaigning   0      0       0             0     0        0        0         \n",
       "compared      0      0       0             0     0        0        0         \n",
       "textbook      0      0       0             0     0        0        0         \n",
       "\n",
       "              cordillera  southern  peru  ...  comprehensive  overview  cbc  \\\n",
       "machu         0           1         2     ...  0              0         0     \n",
       "picchu        0           1         1     ...  0              0         0     \n",
       "15th-century  0           0         0     ...  0              0         0     \n",
       "inca          0           0         1     ...  0              0         0     \n",
       "citadel       1           0         0     ...  0              0         0     \n",
       "...          ..          ..        ..     ... ..             ..        ..     \n",
       "boondoggles   0           0         0     ...  0              0         1     \n",
       "elephants     0           0         0     ...  0              0         1     \n",
       "campaigning   0           0         0     ...  0              0         0     \n",
       "compared      0           0         0     ...  0              0         0     \n",
       "textbook      0           0         0     ...  0              0         0     \n",
       "\n",
       "              digital  archives  boondoggles  elephants  campaigning  \\\n",
       "machu         0        0         0            0          0             \n",
       "picchu        0        0         0            0          0             \n",
       "15th-century  0        0         0            0          0             \n",
       "inca          0        0         0            0          0             \n",
       "citadel       0        0         0            0          0             \n",
       "...          ..       ..        ..           ..         ..             \n",
       "boondoggles   0        1         0            1          0             \n",
       "elephants     1        1         1            0          0             \n",
       "campaigning   1        1         0            0          0             \n",
       "compared      0        0         0            0          0             \n",
       "textbook      1        0         0            0          0             \n",
       "\n",
       "              compared  textbook  \n",
       "machu         0         0         \n",
       "picchu        0         0         \n",
       "15th-century  0         0         \n",
       "inca          0         0         \n",
       "citadel       0         0         \n",
       "...          ..        ..         \n",
       "boondoggles   0         0         \n",
       "elephants     0         0         \n",
       "campaigning   0         0         \n",
       "compared      0         0         \n",
       "textbook      0         0         \n",
       "\n",
       "[4190 rows x 4190 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sents = MyPreprocessor(corpus)\n",
    "cm = CooccurrenceMatrix(sents)\n",
    "comat = cm.fit_transform()\n",
    "vocab = cm.get_feature_names()\n",
    "df = pd.DataFrame(comat.todense(), columns=vocab, index=vocab, dtype=np.int8)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dot product between tall and height: 18.00 and cosine similarity is: 0.33\n",
      "The dot product between tall and official: 0.00 and cosine similarity is: 0.00\n"
     ]
    }
   ],
   "source": [
    "print_cosine_similarity(df, \"tall\", \"height\")\n",
    "print_cosine_similarity(df, \"tall\", \"official\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 3. Dense representations "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The goal is to learn general purpose embeddings that are useful for common tasks involving text data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Sparse vs. dense word vectors**\n",
    "- Term-term co-occurrence matrix representation is long and sparse. \n",
    "    - length |V| is usually large (e.g., > 50,000) \n",
    "    - most elements are zero\n",
    "- OK because there are efficient ways to deal with sparse matrices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Learn short (~100 to 1000 dimensions) and dense vectors.  \n",
    "- Short vectors are usually easier to train with ML models (less weights to train).\n",
    "- They may generalize better.\n",
    "- In practice they work much better! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 3.1 What can we do with these word representations\n",
    "\n",
    "- Before looking at how to create dense word representations let's see how they look like and \n",
    "what can we do with them. \n",
    "- Below I am loading word vectors trained on Google News corpus. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# It'll take a while to run this when you try it out for the first time.\n",
    "import gensim.downloader as api\n",
    "\n",
    "google_news_vectors = api.load(\"word2vec-google-news-300\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of vocabulary:  3000000\n"
     ]
    }
   ],
   "source": [
    "print(\"Size of vocabulary: \", len(google_news_vectors))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- `google_news_vectors` above has 300 dimensional word vectors for 3,000,000 unique words from Google news. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Let's examine word vector for the word UBC. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.3828125 , -0.18066406,  0.10644531,  0.4296875 ,  0.21582031,\n",
       "       -0.10693359,  0.13476562, -0.08740234, -0.14648438, -0.09619141,\n",
       "        0.02807617,  0.01409912, -0.12890625, -0.21972656, -0.41210938,\n",
       "       -0.1875    , -0.11914062, -0.22851562,  0.19433594, -0.08642578],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "google_news_vectors[\"UBC\"][:20]  # Representation of the word UBC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(300,)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "google_news_vectors[\"UBC\"].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed it is a short and a dense vector! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Finding similar words**\n",
    "\n",
    "- Given word $w$, search in the vector space for the word closest to $w$ as measured by cosine distance. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('UVic', 0.788647472858429),\n",
       " ('SFU', 0.7588527202606201),\n",
       " ('Simon_Fraser', 0.7356573939323425),\n",
       " ('UFV', 0.6880434155464172),\n",
       " ('VIU', 0.6778583526611328),\n",
       " ('Kwantlen', 0.6771427989006042),\n",
       " ('UBCO', 0.6734487414360046),\n",
       " ('UPEI', 0.6731125116348267),\n",
       " ('UBC_Okanagan', 0.6709135174751282),\n",
       " ('Lakehead_University', 0.662250816822052)]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "google_news_vectors.most_similar(\"UBC\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('info', 0.7363681793212891),\n",
       " ('infomation', 0.680029571056366),\n",
       " ('infor_mation', 0.6733849048614502),\n",
       " ('informaiton', 0.6639008522033691),\n",
       " ('informa_tion', 0.6601257920265198),\n",
       " ('informationon', 0.633933424949646),\n",
       " ('informationabout', 0.6320980787277222),\n",
       " ('Information', 0.6186580657958984),\n",
       " ('informaion', 0.6093292236328125),\n",
       " ('details', 0.6063088178634644)]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "google_news_vectors.most_similar(\"information\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to extract all documents containing words similar to **information**, you could use this information.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Google News embeddings also support multi-word phrases. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('BC', 0.7640387415885925),\n",
       " ('Alberta', 0.7285022735595703),\n",
       " ('Ontario', 0.7031311392784119),\n",
       " ('Vancouver', 0.6976040005683899),\n",
       " ('Lower_Mainland', 0.6730169057846069),\n",
       " ('Saskatchewan', 0.6690970063209534),\n",
       " ('Manitoba', 0.6569437980651855),\n",
       " ('Canada', 0.6478375792503357),\n",
       " ('Kamloops', 0.6449971795082092),\n",
       " ('Nanaimo_BC', 0.6426822543144226)]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "google_news_vectors.most_similar(\"British_Columbia\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Finding similarity scores between words**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.27610135"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "google_news_vectors.similarity(\"Canada\", \"hockey\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0019627833"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "google_news_vectors.similarity(\"Japan\", \"hockey\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The similarity between height and tall is 0.473\n",
      "The similarity between height and official is 0.002\n",
      "The similarity between pineapple and mango is 0.668\n",
      "The similarity between pineapple and juice is 0.418\n",
      "The similarity between sun and robot is 0.029\n",
      "The similarity between GPU and hummus is 0.094\n"
     ]
    }
   ],
   "source": [
    "word_pairs = [\n",
    "    (\"height\", \"tall\"),\n",
    "    (\"height\", \"official\"),\n",
    "    (\"pineapple\", \"mango\"),\n",
    "    (\"pineapple\", \"juice\"),\n",
    "    (\"sun\", \"robot\"),\n",
    "    (\"GPU\", \"hummus\"),\n",
    "]\n",
    "for pair in word_pairs:\n",
    "    print(\n",
    "        \"The similarity between %s and %s is %0.3f\"\n",
    "        % (pair[0], pair[1], google_news_vectors.similarity(pair[0], pair[1]))\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are getting reasonable word similarity scores!! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 3.2 Creating dense representations\n",
    "\n",
    "![](img/word2vec.png)\n",
    "<!-- <center>\n",
    "<img src=\"img/word2vec.png\" width=\"600\" height=\"600\">\n",
    "</center>    \n",
    " -->\n",
    "There are two classes of approaches. \n",
    "\n",
    "- LSA (also referred to as count-based approaches)\n",
    "- word2vec (prediction-based approaches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "\n",
    "#### 3.2.1 (Optional) Dense embeddings with LSA\n",
    "\n",
    "- How can we get such dense word embeddings? \n",
    "- Can we use LSA to get such short and dense representations? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "from sklearn.decomposition import TruncatedSVD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "cm = CooccurrenceMatrix(sents)\n",
    "comat = cm.fit_transform()\n",
    "vocab = cm.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "lsa = TruncatedSVD(n_components=10, n_iter=7, random_state=42)\n",
    "lsa.fit(comat)\n",
    "embedding_matrix = lsa.transform(comat)\n",
    "lsa_rep_df = pd.DataFrame(embedding_matrix, index=vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>machu</th>\n",
       "      <td>59.744787</td>\n",
       "      <td>-56.048148</td>\n",
       "      <td>-9.476625</td>\n",
       "      <td>-2.125471</td>\n",
       "      <td>-0.931832</td>\n",
       "      <td>0.091583</td>\n",
       "      <td>0.016552</td>\n",
       "      <td>0.123792</td>\n",
       "      <td>-0.092774</td>\n",
       "      <td>0.092224</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>picchu</th>\n",
       "      <td>61.156957</td>\n",
       "      <td>57.356851</td>\n",
       "      <td>-10.068306</td>\n",
       "      <td>-2.102533</td>\n",
       "      <td>-1.328928</td>\n",
       "      <td>-1.133096</td>\n",
       "      <td>1.270330</td>\n",
       "      <td>-0.369837</td>\n",
       "      <td>0.059576</td>\n",
       "      <td>0.420707</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15th-century</th>\n",
       "      <td>1.425137</td>\n",
       "      <td>-0.004821</td>\n",
       "      <td>-0.271348</td>\n",
       "      <td>-0.060159</td>\n",
       "      <td>-0.064429</td>\n",
       "      <td>0.009745</td>\n",
       "      <td>-0.011915</td>\n",
       "      <td>-0.053462</td>\n",
       "      <td>0.043515</td>\n",
       "      <td>-0.009831</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>inca</th>\n",
       "      <td>9.689077</td>\n",
       "      <td>-0.487520</td>\n",
       "      <td>-0.900727</td>\n",
       "      <td>-0.141428</td>\n",
       "      <td>-0.458294</td>\n",
       "      <td>0.374853</td>\n",
       "      <td>-0.345878</td>\n",
       "      <td>-0.262844</td>\n",
       "      <td>0.506327</td>\n",
       "      <td>-0.231733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>citadel</th>\n",
       "      <td>2.995572</td>\n",
       "      <td>-1.302157</td>\n",
       "      <td>-0.468748</td>\n",
       "      <td>-0.161603</td>\n",
       "      <td>-0.264666</td>\n",
       "      <td>0.101359</td>\n",
       "      <td>-0.069382</td>\n",
       "      <td>-0.273455</td>\n",
       "      <td>-0.001200</td>\n",
       "      <td>0.171081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>boondoggles</th>\n",
       "      <td>0.023089</td>\n",
       "      <td>-0.000278</td>\n",
       "      <td>0.045347</td>\n",
       "      <td>0.017567</td>\n",
       "      <td>-0.036114</td>\n",
       "      <td>-0.016962</td>\n",
       "      <td>0.023338</td>\n",
       "      <td>0.042295</td>\n",
       "      <td>0.014989</td>\n",
       "      <td>-0.018444</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>elephants</th>\n",
       "      <td>0.006440</td>\n",
       "      <td>-0.000364</td>\n",
       "      <td>0.037552</td>\n",
       "      <td>-0.015651</td>\n",
       "      <td>-0.026522</td>\n",
       "      <td>-0.020554</td>\n",
       "      <td>0.013983</td>\n",
       "      <td>0.052387</td>\n",
       "      <td>0.015443</td>\n",
       "      <td>-0.012910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>campaigning</th>\n",
       "      <td>0.080149</td>\n",
       "      <td>0.005431</td>\n",
       "      <td>0.468662</td>\n",
       "      <td>-0.241442</td>\n",
       "      <td>-0.180483</td>\n",
       "      <td>0.207737</td>\n",
       "      <td>-0.182761</td>\n",
       "      <td>0.357888</td>\n",
       "      <td>0.178600</td>\n",
       "      <td>0.101775</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>compared</th>\n",
       "      <td>0.078030</td>\n",
       "      <td>0.005072</td>\n",
       "      <td>0.567311</td>\n",
       "      <td>-0.333567</td>\n",
       "      <td>-0.222730</td>\n",
       "      <td>0.195864</td>\n",
       "      <td>-0.187922</td>\n",
       "      <td>0.471011</td>\n",
       "      <td>0.309136</td>\n",
       "      <td>0.103632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>textbook</th>\n",
       "      <td>0.006850</td>\n",
       "      <td>-0.000662</td>\n",
       "      <td>0.051795</td>\n",
       "      <td>-0.027355</td>\n",
       "      <td>-0.036391</td>\n",
       "      <td>-0.021673</td>\n",
       "      <td>0.017683</td>\n",
       "      <td>0.069401</td>\n",
       "      <td>0.045774</td>\n",
       "      <td>-0.022885</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4190 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                      0          1          2         3         4         5  \\\n",
       "machu         59.744787 -56.048148 -9.476625  -2.125471 -0.931832  0.091583   \n",
       "picchu        61.156957  57.356851 -10.068306 -2.102533 -1.328928 -1.133096   \n",
       "15th-century  1.425137  -0.004821  -0.271348  -0.060159 -0.064429  0.009745   \n",
       "inca          9.689077  -0.487520  -0.900727  -0.141428 -0.458294  0.374853   \n",
       "citadel       2.995572  -1.302157  -0.468748  -0.161603 -0.264666  0.101359   \n",
       "...                ...        ...        ...        ...       ...       ...   \n",
       "boondoggles   0.023089  -0.000278   0.045347   0.017567 -0.036114 -0.016962   \n",
       "elephants     0.006440  -0.000364   0.037552  -0.015651 -0.026522 -0.020554   \n",
       "campaigning   0.080149   0.005431   0.468662  -0.241442 -0.180483  0.207737   \n",
       "compared      0.078030   0.005072   0.567311  -0.333567 -0.222730  0.195864   \n",
       "textbook      0.006850  -0.000662   0.051795  -0.027355 -0.036391 -0.021673   \n",
       "\n",
       "                     6         7         8         9  \n",
       "machu         0.016552  0.123792 -0.092774  0.092224  \n",
       "picchu        1.270330 -0.369837  0.059576  0.420707  \n",
       "15th-century -0.011915 -0.053462  0.043515 -0.009831  \n",
       "inca         -0.345878 -0.262844  0.506327 -0.231733  \n",
       "citadel      -0.069382 -0.273455 -0.001200  0.171081  \n",
       "...                ...       ...       ...       ...  \n",
       "boondoggles   0.023338  0.042295  0.014989 -0.018444  \n",
       "elephants     0.013983  0.052387  0.015443 -0.012910  \n",
       "campaigning  -0.182761  0.357888  0.178600  0.101775  \n",
       "compared     -0.187922  0.471011  0.309136  0.103632  \n",
       "textbook      0.017683  0.069401  0.045774 -0.022885  \n",
       "\n",
       "[4190 rows x 10 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lsa_rep_df  # word representations learned with LSA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dot product between tall and height: 6.53 and cosine similarity is: 1.00\n",
      "The dot product between tall and official: 0.02 and cosine similarity is: 0.00\n"
     ]
    }
   ],
   "source": [
    "print_cosine_similarity(lsa_rep_df, \"tall\", \"height\")\n",
    "print_cosine_similarity(lsa_rep_df, \"tall\", \"official\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### 3.2.2 word2vec\n",
    "\n",
    "- We can use LSA to get dense word embeddings.\n",
    "- In general, if we have a small dataset, embeddings extracted using LSA work better. \n",
    "- But an alternative and a more popular way to extract dense and short embeddings is using word2vec.  \n",
    "- word2vec is a family of algorithms to create dense word embeddings using neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**word2vec task**\n",
    "\n",
    "Remember fill in the blank puzzles in your highschool tests? \n",
    "\n",
    "> Add freshly squeezed ___ juice to your smoothie. \n",
    "1. pineapple \n",
    "2. scarf\n",
    "3. PCA\n",
    "4. earthquake \n",
    "\n",
    "Another slightly non-intuitive way to think about this is what would be the context words given the target word **pineapple**?    \n",
    "\n",
    "> Add freshly ___ **pineapple** ___ to your smoothie. \n",
    "\n",
    "word2vec learns meaningful word representations by learning to solve large number of such fill in the blank puzzles.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Based on this intuition, there are two primary algorithms \n",
    "    - Continuous bag of words (CBOW)\n",
    "    - **Skip-gram**\n",
    "- Two moderately efficient training methods \n",
    "    - Hierarchical softmax\n",
    "    - Negative sampling "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**word2vec: Skip-gram model**\n",
    "\n",
    "- We are going to talk about the inefficient Skip-gram model, as it's enough to get an intuition. \n",
    "- A neural network model to obtain short and dense representations of words. \n",
    "- A simple architecture with \n",
    "    - an input layer\n",
    "    - a linear hidden layer (without any activation function)\n",
    "    - an output layer with softmax "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- In skip-gram we work on an \"auxiliary\" supervised machine learning word prediction task of prediction context words given the target word.   \n",
    "- We train a neural network for this task and the learned weights are our word vectors. \n",
    "- So we are not actually interested in making prediction about the context words. \n",
    "- Our goal is to learn meaningful weights (meaningful representation of the input) in the process. \n",
    "\n",
    "<blockquote>\n",
    "    Add freshly squeezed$_{context}$ pineapple$_{target}$ juice$_{context}$ to your smoothie. \n",
    "</blockquote> \n",
    "\n",
    "![](img/target-context.png)\n",
    "\n",
    "<!-- <center>\n",
    "<img src=\"img/target-context.png\" width=\"300\" height=\"300\">\n",
    "</center>\n",
    " -->\n",
    "- So in the example above given the target word **pineapple**\n",
    "    - what's the probability that a randomly picked context word is **juice**. \n",
    "    - what's the probability that a randomly picked context word is **squeezed**. \n",
    "\n",
    "- Given a **target word** (i.e., center word) word, predict **context words** (i.e., surrounding words). \n",
    "- Note that we are using \"target\" in a different sense here compared to how we use it in supervised machine learning.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**A simple neural network architecture ([Source](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/))**\n",
    "\n",
    "- an input layer\n",
    "- a linear hidden layer (without any activation function)\n",
    "- an output layer with softmax\n",
    "\n",
    "![](img/word2vec_skipgram.png)\n",
    "\n",
    "<!-- <img src=\"img/word2vec_skipgram.png\" width=\"600\" height=\"800\"> -->\n",
    "\n",
    "<!-- <center>\n",
    "<img src=\"img/word2vec_skipgram.png\" width=\"500\" height=\"500\">\n",
    "</center>    \n",
    " -->\n",
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<!-- <img src=\"img/skipgram-juice.png\" width=\"1000\" height=\"1000\"> -->\n",
    "\n",
    "![](img/skipgram-juice.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/skipgram-squeezed.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Skip-gram objective**\n",
    "\n",
    "- Consider the conditional probabilities $p(w_c|w_t)$ and set the parameters $\\theta$ of $p(w_c|w_t; \\theta)$ so as to maximize the corpus probability. \n",
    "\n",
    "$$\\arg \\max\\limits_\\theta \\prod\\limits_{(w_c,w_t) \\in D} p(w_c|w_t;\\theta)$$\n",
    "\n",
    "\n",
    "- $w_t$ &rarr; target word\n",
    "- $w_c$ &rarr; context word\n",
    "- $D$ &rarr; the set of all target and context pairs from the text \n",
    "- $V$ &rarr; vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Model the conditional probability using softmax of the dot product.\n",
    "    * Higher the dot product higher the probability and vice-versa.     \n",
    "    \n",
    "\n",
    "$$P(w_c|w_t;\\theta) = \\frac{e^{w_c.w_t}}{\\sum\\limits_{\\substack{c' \\in V}} e^{w_{c'}.w_t}}$$\n",
    "\n",
    "- Substituting the conditional probability with the softmax of dot product: \n",
    "$$ \\arg \\max\\limits_\\theta \\prod\\limits_{(w_c,w_t) \\in D} P(w_c|w_t;\\theta) \\approx \\prod\\limits_{(w_c,w_t) \\in D} \\frac{e^{w_c.w_t}}{\\sum\\limits_{\\substack{c' \\in V}} e^{w_{c'}.w_t}}$$\n",
    "- Assumption: Maximizing this objective on a large corpus will result in meaningful embeddings for all words in the vocabulary. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Main hyperparameters of the model**\n",
    "\n",
    "- Dimensionality of the word vectors \n",
    "- Window size\n",
    "    * shorter window: more syntactic representation\n",
    "    * longer window: more semantic representation \n",
    "    * Mikolov et al. (2015) suggest setting this parameter in the range 5 to 20 for small training datasets and in the range 2 to 5 for large training datasets.    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**(Optional) Parameters to learn**\n",
    "\n",
    "- Given a corpus with vocabulary of size $V$, where a word $w_i$ is identified by its index $i \\in {1, ..., V}$, learn a vector representation for each $w_i$ by predicting the words that appear in its context. \n",
    "- Learn the following parameters of the model\n",
    "    - Suppose $V = 10,000$, $d = 300$, the number of parameters to learn are 6,000,000! \n",
    "\n",
    "$$\n",
    "\\theta = \n",
    "\\begin{bmatrix} aardvark_t\\\\\n",
    "                aback_t\\\\\n",
    "                \\dots\\\\\n",
    "                zymurgi_t\\\\\n",
    "                aardvark_c\\\\\n",
    "                aback_c\\\\                \n",
    "                \\dots\\\\\n",
    "                zymurgi_c\\\\                \n",
    "\\end{bmatrix} \\in R^{2dV}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><rb>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ❓❓ Questions for you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Exercise 5.1 Select all of the following statements which are **True** (iClicker)\n",
    "\n",
    "- (A) Word representation created by term-term co-occurrence matrix are long and sparse whereas the ones created by word2vec are short and dense. \n",
    "- (B) You could pass term-term co-occurrence matrix to `TruncatedSVD` or LSA to get short and dense representations. \n",
    "- (C) The word2vec algorithm does not require any manually labeled training data.\n",
    "- (D) When training a word2vec model, it is fine if we do poorly on the fake word prediction task because in the end we only care about the learned weights of the model.\n",
    "- (E) Given the following table (word 1, word 2) are more similar than (word 1, word 3) in terms of dot products. \n",
    "\n",
    "![](img/similarity_question.png)\n",
    "<!-- <img src=\"img/similarity_question.png\" width=\"500\" height=\"500\"> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{admonition} V's Solutions\n",
    ":class: tip, dropdown\n",
    "- A, B, C\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 4. More word2vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Skip-gram demo with toy data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**For the purpose of your lab or quiz, you do not have to understand the code in this demo.** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "toy_corpus = [\n",
    "    \"drink mango juice\",\n",
    "    \"drink pineapple juice\",\n",
    "    \"drink apple juice\",\n",
    "    \"drink squeezed pineapple juice\",\n",
    "    \"drink squeezed mango juice\",\n",
    "    \"drink apple tea\",\n",
    "    \"drink mango tea\",\n",
    "    \"drink mango water\",\n",
    "    \"drink apple water\",\n",
    "    \"drink pineapple water\",\n",
    "    \"drink juice\",\n",
    "    \"drink water\",\n",
    "    \"drink tea\",\n",
    "    \"play hockey\",\n",
    "    \"play football\",\n",
    "    \"play piano\",\n",
    "    \"piano play\",\n",
    "    \"play hockey game\",\n",
    "    \"play football game\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "sents = MyPreprocessor(toy_corpus)  # memory smart generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "EMBEDDING_DIM = 10\n",
    "CONTEXT_SIZE = 2\n",
    "EPOCHS = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "from word2vec_demo import *\n",
    "\n",
    "vocab = get_vocab(sents)\n",
    "word2idx = {w: idx for (idx, w) in enumerate(vocab)}\n",
    "idx2word = {idx: w for (idx, w) in enumerate(vocab)}\n",
    "\n",
    "vocab_size = len(vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['piano',\n",
       " 'juice',\n",
       " 'tea',\n",
       " 'apple',\n",
       " 'pineapple',\n",
       " 'drink',\n",
       " 'hockey',\n",
       " 'game',\n",
       " 'football',\n",
       " 'play',\n",
       " 'water',\n",
       " 'mango',\n",
       " 'squeezed']"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 5, 11],\n",
       "       [ 5,  1],\n",
       "       [11,  5],\n",
       "       [11,  1],\n",
       "       [ 1,  5],\n",
       "       [ 1, 11],\n",
       "       [ 5,  4],\n",
       "       [ 5,  1],\n",
       "       [ 4,  5],\n",
       "       [ 4,  1]])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx_pairs = create_input_pairs(sents, word2idx)    \n",
    "idx_pairs[:10] # training examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intel MKL WARNING: Support of Intel(R) Streaming SIMD Extensions 4.2 (Intel(R) SSE4.2) enabled only processors has been deprecated. Intel oneAPI Math Kernel Library 2025.0 will require Intel(R) Advanced Vector Extensions (Intel(R) AVX) instructions.\n"
     ]
    }
   ],
   "source": [
    "model = SkipgramModel(len(vocab), EMBEDDING_DIM)\n",
    "train_skipgram(model, idx_pairs, epochs=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from IPython.display import display"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "(function(root) {\n",
       "  function now() {\n",
       "    return new Date();\n",
       "  }\n",
       "\n",
       "  var force = true;\n",
       "  var py_version = '3.3.4'.replace('rc', '-rc.').replace('.dev', '-dev.');\n",
       "  var reloading = false;\n",
       "  var Bokeh = root.Bokeh;\n",
       "\n",
       "  if (typeof (root._bokeh_timeout) === \"undefined\" || force) {\n",
       "    root._bokeh_timeout = Date.now() + 5000;\n",
       "    root._bokeh_failed_load = false;\n",
       "  }\n",
       "\n",
       "  function run_callbacks() {\n",
       "    try {\n",
       "      root._bokeh_onload_callbacks.forEach(function(callback) {\n",
       "        if (callback != null)\n",
       "          callback();\n",
       "      });\n",
       "    } finally {\n",
       "      delete root._bokeh_onload_callbacks;\n",
       "    }\n",
       "    console.debug(\"Bokeh: all callbacks have finished\");\n",
       "  }\n",
       "\n",
       "  function load_libs(css_urls, js_urls, js_modules, js_exports, callback) {\n",
       "    if (css_urls == null) css_urls = [];\n",
       "    if (js_urls == null) js_urls = [];\n",
       "    if (js_modules == null) js_modules = [];\n",
       "    if (js_exports == null) js_exports = {};\n",
       "\n",
       "    root._bokeh_onload_callbacks.push(callback);\n",
       "\n",
       "    if (root._bokeh_is_loading > 0) {\n",
       "      console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n",
       "      return null;\n",
       "    }\n",
       "    if (js_urls.length === 0 && js_modules.length === 0 && Object.keys(js_exports).length === 0) {\n",
       "      run_callbacks();\n",
       "      return null;\n",
       "    }\n",
       "    if (!reloading) {\n",
       "      console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n",
       "    }\n",
       "\n",
       "    function on_load() {\n",
       "      root._bokeh_is_loading--;\n",
       "      if (root._bokeh_is_loading === 0) {\n",
       "        console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n",
       "        run_callbacks()\n",
       "      }\n",
       "    }\n",
       "    window._bokeh_on_load = on_load\n",
       "\n",
       "    function on_error() {\n",
       "      console.error(\"failed to load \" + url);\n",
       "    }\n",
       "\n",
       "    var skip = [];\n",
       "    if (window.requirejs) {\n",
       "      window.requirejs.config({'packages': {}, 'paths': {'jspanel': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/jspanel', 'jspanel-modal': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/modal/jspanel.modal', 'jspanel-tooltip': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/tooltip/jspanel.tooltip', 'jspanel-hint': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/hint/jspanel.hint', 'jspanel-layout': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/layout/jspanel.layout', 'jspanel-contextmenu': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/contextmenu/jspanel.contextmenu', 'jspanel-dock': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/dock/jspanel.dock', 'gridstack': 'https://cdn.jsdelivr.net/npm/gridstack@7.2.3/dist/gridstack-all', 'notyf': 'https://cdn.jsdelivr.net/npm/notyf@3/notyf.min'}, 'shim': {'jspanel': {'exports': 'jsPanel'}, 'gridstack': {'exports': 'GridStack'}}});\n",
       "      require([\"jspanel\"], function(jsPanel) {\n",
       "\twindow.jsPanel = jsPanel\n",
       "\ton_load()\n",
       "      })\n",
       "      require([\"jspanel-modal\"], function() {\n",
       "\ton_load()\n",
       "      })\n",
       "      require([\"jspanel-tooltip\"], function() {\n",
       "\ton_load()\n",
       "      })\n",
       "      require([\"jspanel-hint\"], function() {\n",
       "\ton_load()\n",
       "      })\n",
       "      require([\"jspanel-layout\"], function() {\n",
       "\ton_load()\n",
       "      })\n",
       "      require([\"jspanel-contextmenu\"], function() {\n",
       "\ton_load()\n",
       "      })\n",
       "      require([\"jspanel-dock\"], function() {\n",
       "\ton_load()\n",
       "      })\n",
       "      require([\"gridstack\"], function(GridStack) {\n",
       "\twindow.GridStack = GridStack\n",
       "\ton_load()\n",
       "      })\n",
       "      require([\"notyf\"], function() {\n",
       "\ton_load()\n",
       "      })\n",
       "      root._bokeh_is_loading = css_urls.length + 9;\n",
       "    } else {\n",
       "      root._bokeh_is_loading = css_urls.length + js_urls.length + js_modules.length + Object.keys(js_exports).length;\n",
       "    }\n",
       "\n",
       "    var existing_stylesheets = []\n",
       "    var links = document.getElementsByTagName('link')\n",
       "    for (var i = 0; i < links.length; i++) {\n",
       "      var link = links[i]\n",
       "      if (link.href != null) {\n",
       "\texisting_stylesheets.push(link.href)\n",
       "      }\n",
       "    }\n",
       "    for (var i = 0; i < css_urls.length; i++) {\n",
       "      var url = css_urls[i];\n",
       "      if (existing_stylesheets.indexOf(url) !== -1) {\n",
       "\ton_load()\n",
       "\tcontinue;\n",
       "      }\n",
       "      const element = document.createElement(\"link\");\n",
       "      element.onload = on_load;\n",
       "      element.onerror = on_error;\n",
       "      element.rel = \"stylesheet\";\n",
       "      element.type = \"text/css\";\n",
       "      element.href = url;\n",
       "      console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n",
       "      document.body.appendChild(element);\n",
       "    }    if (((window['jsPanel'] !== undefined) && (!(window['jsPanel'] instanceof HTMLElement))) || window.requirejs) {\n",
       "      var urls = ['https://cdn.holoviz.org/panel/1.3.8/dist/bundled/floatpanel/jspanel4@4.12.0/dist/jspanel.js', 'https://cdn.holoviz.org/panel/1.3.8/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/modal/jspanel.modal.js', 'https://cdn.holoviz.org/panel/1.3.8/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/tooltip/jspanel.tooltip.js', 'https://cdn.holoviz.org/panel/1.3.8/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/hint/jspanel.hint.js', 'https://cdn.holoviz.org/panel/1.3.8/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/layout/jspanel.layout.js', 'https://cdn.holoviz.org/panel/1.3.8/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/contextmenu/jspanel.contextmenu.js', 'https://cdn.holoviz.org/panel/1.3.8/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/dock/jspanel.dock.js'];\n",
       "      for (var i = 0; i < urls.length; i++) {\n",
       "        skip.push(urls[i])\n",
       "      }\n",
       "    }    if (((window['GridStack'] !== undefined) && (!(window['GridStack'] instanceof HTMLElement))) || window.requirejs) {\n",
       "      var urls = ['https://cdn.holoviz.org/panel/1.3.8/dist/bundled/gridstack/gridstack@7.2.3/dist/gridstack-all.js'];\n",
       "      for (var i = 0; i < urls.length; i++) {\n",
       "        skip.push(urls[i])\n",
       "      }\n",
       "    }    if (((window['Notyf'] !== undefined) && (!(window['Notyf'] instanceof HTMLElement))) || window.requirejs) {\n",
       "      var urls = ['https://cdn.holoviz.org/panel/1.3.8/dist/bundled/notificationarea/notyf@3/notyf.min.js'];\n",
       "      for (var i = 0; i < urls.length; i++) {\n",
       "        skip.push(urls[i])\n",
       "      }\n",
       "    }    var existing_scripts = []\n",
       "    var scripts = document.getElementsByTagName('script')\n",
       "    for (var i = 0; i < scripts.length; i++) {\n",
       "      var script = scripts[i]\n",
       "      if (script.src != null) {\n",
       "\texisting_scripts.push(script.src)\n",
       "      }\n",
       "    }\n",
       "    for (var i = 0; i < js_urls.length; i++) {\n",
       "      var url = js_urls[i];\n",
       "      if (skip.indexOf(url) !== -1 || existing_scripts.indexOf(url) !== -1) {\n",
       "\tif (!window.requirejs) {\n",
       "\t  on_load();\n",
       "\t}\n",
       "\tcontinue;\n",
       "      }\n",
       "      var element = document.createElement('script');\n",
       "      element.onload = on_load;\n",
       "      element.onerror = on_error;\n",
       "      element.async = false;\n",
       "      element.src = url;\n",
       "      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",
       "      document.head.appendChild(element);\n",
       "    }\n",
       "    for (var i = 0; i < js_modules.length; i++) {\n",
       "      var url = js_modules[i];\n",
       "      if (skip.indexOf(url) !== -1 || existing_scripts.indexOf(url) !== -1) {\n",
       "\tif (!window.requirejs) {\n",
       "\t  on_load();\n",
       "\t}\n",
       "\tcontinue;\n",
       "      }\n",
       "      var element = document.createElement('script');\n",
       "      element.onload = on_load;\n",
       "      element.onerror = on_error;\n",
       "      element.async = false;\n",
       "      element.src = url;\n",
       "      element.type = \"module\";\n",
       "      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",
       "      document.head.appendChild(element);\n",
       "    }\n",
       "    for (const name in js_exports) {\n",
       "      var url = js_exports[name];\n",
       "      if (skip.indexOf(url) >= 0 || root[name] != null) {\n",
       "\tif (!window.requirejs) {\n",
       "\t  on_load();\n",
       "\t}\n",
       "\tcontinue;\n",
       "      }\n",
       "      var element = document.createElement('script');\n",
       "      element.onerror = on_error;\n",
       "      element.async = false;\n",
       "      element.type = \"module\";\n",
       "      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",
       "      element.textContent = `\n",
       "      import ${name} from \"${url}\"\n",
       "      window.${name} = ${name}\n",
       "      window._bokeh_on_load()\n",
       "      `\n",
       "      document.head.appendChild(element);\n",
       "    }\n",
       "    if (!js_urls.length && !js_modules.length) {\n",
       "      on_load()\n",
       "    }\n",
       "  };\n",
       "\n",
       "  function inject_raw_css(css) {\n",
       "    const element = document.createElement(\"style\");\n",
       "    element.appendChild(document.createTextNode(css));\n",
       "    document.body.appendChild(element);\n",
       "  }\n",
       "\n",
       "  var js_urls = [\"https://cdn.bokeh.org/bokeh/release/bokeh-3.3.4.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-gl-3.3.4.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-3.3.4.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-3.3.4.min.js\", \"https://cdn.holoviz.org/panel/1.3.8/dist/panel.min.js\"];\n",
       "  var js_modules = [];\n",
       "  var js_exports = {};\n",
       "  var css_urls = [];\n",
       "  var inline_js = [    function(Bokeh) {\n",
       "      Bokeh.set_log_level(\"info\");\n",
       "    },\n",
       "function(Bokeh) {} // ensure no trailing comma for IE\n",
       "  ];\n",
       "\n",
       "  function run_inline_js() {\n",
       "    if ((root.Bokeh !== undefined) || (force === true)) {\n",
       "      for (var i = 0; i < inline_js.length; i++) {\n",
       "\ttry {\n",
       "          inline_js[i].call(root, root.Bokeh);\n",
       "\t} catch(e) {\n",
       "\t  if (!reloading) {\n",
       "\t    throw e;\n",
       "\t  }\n",
       "\t}\n",
       "      }\n",
       "      // Cache old bokeh versions\n",
       "      if (Bokeh != undefined && !reloading) {\n",
       "\tvar NewBokeh = root.Bokeh;\n",
       "\tif (Bokeh.versions === undefined) {\n",
       "\t  Bokeh.versions = new Map();\n",
       "\t}\n",
       "\tif (NewBokeh.version !== Bokeh.version) {\n",
       "\t  Bokeh.versions.set(NewBokeh.version, NewBokeh)\n",
       "\t}\n",
       "\troot.Bokeh = Bokeh;\n",
       "      }} else if (Date.now() < root._bokeh_timeout) {\n",
       "      setTimeout(run_inline_js, 100);\n",
       "    } else if (!root._bokeh_failed_load) {\n",
       "      console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n",
       "      root._bokeh_failed_load = true;\n",
       "    }\n",
       "    root._bokeh_is_initializing = false\n",
       "  }\n",
       "\n",
       "  function load_or_wait() {\n",
       "    // Implement a backoff loop that tries to ensure we do not load multiple\n",
       "    // versions of Bokeh and its dependencies at the same time.\n",
       "    // In recent versions we use the root._bokeh_is_initializing flag\n",
       "    // to determine whether there is an ongoing attempt to initialize\n",
       "    // bokeh, however for backward compatibility we also try to ensure\n",
       "    // that we do not start loading a newer (Panel>=1.0 and Bokeh>3) version\n",
       "    // before older versions are fully initialized.\n",
       "    if (root._bokeh_is_initializing && Date.now() > root._bokeh_timeout) {\n",
       "      root._bokeh_is_initializing = false;\n",
       "      root._bokeh_onload_callbacks = undefined;\n",
       "      console.log(\"Bokeh: BokehJS was loaded multiple times but one version failed to initialize.\");\n",
       "      load_or_wait();\n",
       "    } else if (root._bokeh_is_initializing || (typeof root._bokeh_is_initializing === \"undefined\" && root._bokeh_onload_callbacks !== undefined)) {\n",
       "      setTimeout(load_or_wait, 100);\n",
       "    } else {\n",
       "      root._bokeh_is_initializing = true\n",
       "      root._bokeh_onload_callbacks = []\n",
       "      var bokeh_loaded = Bokeh != null && (Bokeh.version === py_version || (Bokeh.versions !== undefined && Bokeh.versions.has(py_version)));\n",
       "      if (!reloading && !bokeh_loaded) {\n",
       "\troot.Bokeh = undefined;\n",
       "      }\n",
       "      load_libs(css_urls, js_urls, js_modules, js_exports, function() {\n",
       "\tconsole.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n",
       "\trun_inline_js();\n",
       "      });\n",
       "    }\n",
       "  }\n",
       "  // Give older versions of the autoload script a head-start to ensure\n",
       "  // they initialize before we start loading newer version.\n",
       "  setTimeout(load_or_wait, 100)\n",
       "}(window));"
      ],
      "application/vnd.holoviews_load.v0+json": "(function(root) {\n  function now() {\n    return new Date();\n  }\n\n  var force = true;\n  var py_version = '3.3.4'.replace('rc', '-rc.').replace('.dev', '-dev.');\n  var reloading = false;\n  var Bokeh = root.Bokeh;\n\n  if (typeof (root._bokeh_timeout) === \"undefined\" || force) {\n    root._bokeh_timeout = Date.now() + 5000;\n    root._bokeh_failed_load = false;\n  }\n\n  function run_callbacks() {\n    try {\n      root._bokeh_onload_callbacks.forEach(function(callback) {\n        if (callback != null)\n          callback();\n      });\n    } finally {\n      delete root._bokeh_onload_callbacks;\n    }\n    console.debug(\"Bokeh: all callbacks have finished\");\n  }\n\n  function load_libs(css_urls, js_urls, js_modules, js_exports, callback) {\n    if (css_urls == null) css_urls = [];\n    if (js_urls == null) js_urls = [];\n    if (js_modules == null) js_modules = [];\n    if (js_exports == null) js_exports = {};\n\n    root._bokeh_onload_callbacks.push(callback);\n\n    if (root._bokeh_is_loading > 0) {\n      console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n      return null;\n    }\n    if (js_urls.length === 0 && js_modules.length === 0 && Object.keys(js_exports).length === 0) {\n      run_callbacks();\n      return null;\n    }\n    if (!reloading) {\n      console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n    }\n\n    function on_load() {\n      root._bokeh_is_loading--;\n      if (root._bokeh_is_loading === 0) {\n        console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n        run_callbacks()\n      }\n    }\n    window._bokeh_on_load = on_load\n\n    function on_error() {\n      console.error(\"failed to load \" + url);\n    }\n\n    var skip = [];\n    if (window.requirejs) {\n      window.requirejs.config({'packages': {}, 'paths': {'jspanel': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/jspanel', 'jspanel-modal': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/modal/jspanel.modal', 'jspanel-tooltip': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/tooltip/jspanel.tooltip', 'jspanel-hint': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/hint/jspanel.hint', 'jspanel-layout': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/layout/jspanel.layout', 'jspanel-contextmenu': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/contextmenu/jspanel.contextmenu', 'jspanel-dock': 'https://cdn.jsdelivr.net/npm/jspanel4@4.12.0/dist/extensions/dock/jspanel.dock', 'gridstack': 'https://cdn.jsdelivr.net/npm/gridstack@7.2.3/dist/gridstack-all', 'notyf': 'https://cdn.jsdelivr.net/npm/notyf@3/notyf.min'}, 'shim': {'jspanel': {'exports': 'jsPanel'}, 'gridstack': {'exports': 'GridStack'}}});\n      require([\"jspanel\"], function(jsPanel) {\n\twindow.jsPanel = jsPanel\n\ton_load()\n      })\n      require([\"jspanel-modal\"], function() {\n\ton_load()\n      })\n      require([\"jspanel-tooltip\"], function() {\n\ton_load()\n      })\n      require([\"jspanel-hint\"], function() {\n\ton_load()\n      })\n      require([\"jspanel-layout\"], function() {\n\ton_load()\n      })\n      require([\"jspanel-contextmenu\"], function() {\n\ton_load()\n      })\n      require([\"jspanel-dock\"], function() {\n\ton_load()\n      })\n      require([\"gridstack\"], function(GridStack) {\n\twindow.GridStack = GridStack\n\ton_load()\n      })\n      require([\"notyf\"], function() {\n\ton_load()\n      })\n      root._bokeh_is_loading = css_urls.length + 9;\n    } else {\n      root._bokeh_is_loading = css_urls.length + js_urls.length + js_modules.length + Object.keys(js_exports).length;\n    }\n\n    var existing_stylesheets = []\n    var links = document.getElementsByTagName('link')\n    for (var i = 0; i < links.length; i++) {\n      var link = links[i]\n      if (link.href != null) {\n\texisting_stylesheets.push(link.href)\n      }\n    }\n    for (var i = 0; i < css_urls.length; i++) {\n      var url = css_urls[i];\n      if (existing_stylesheets.indexOf(url) !== -1) {\n\ton_load()\n\tcontinue;\n      }\n      const element = document.createElement(\"link\");\n      element.onload = on_load;\n      element.onerror = on_error;\n      element.rel = \"stylesheet\";\n      element.type = \"text/css\";\n      element.href = url;\n      console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n      document.body.appendChild(element);\n    }    if (((window['jsPanel'] !== undefined) && (!(window['jsPanel'] instanceof HTMLElement))) || window.requirejs) {\n      var urls = ['https://cdn.holoviz.org/panel/1.3.8/dist/bundled/floatpanel/jspanel4@4.12.0/dist/jspanel.js', 'https://cdn.holoviz.org/panel/1.3.8/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/modal/jspanel.modal.js', 'https://cdn.holoviz.org/panel/1.3.8/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/tooltip/jspanel.tooltip.js', 'https://cdn.holoviz.org/panel/1.3.8/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/hint/jspanel.hint.js', 'https://cdn.holoviz.org/panel/1.3.8/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/layout/jspanel.layout.js', 'https://cdn.holoviz.org/panel/1.3.8/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/contextmenu/jspanel.contextmenu.js', 'https://cdn.holoviz.org/panel/1.3.8/dist/bundled/floatpanel/jspanel4@4.12.0/dist/extensions/dock/jspanel.dock.js'];\n      for (var i = 0; i < urls.length; i++) {\n        skip.push(urls[i])\n      }\n    }    if (((window['GridStack'] !== undefined) && (!(window['GridStack'] instanceof HTMLElement))) || window.requirejs) {\n      var urls = ['https://cdn.holoviz.org/panel/1.3.8/dist/bundled/gridstack/gridstack@7.2.3/dist/gridstack-all.js'];\n      for (var i = 0; i < urls.length; i++) {\n        skip.push(urls[i])\n      }\n    }    if (((window['Notyf'] !== undefined) && (!(window['Notyf'] instanceof HTMLElement))) || window.requirejs) {\n      var urls = ['https://cdn.holoviz.org/panel/1.3.8/dist/bundled/notificationarea/notyf@3/notyf.min.js'];\n      for (var i = 0; i < urls.length; i++) {\n        skip.push(urls[i])\n      }\n    }    var existing_scripts = []\n    var scripts = document.getElementsByTagName('script')\n    for (var i = 0; i < scripts.length; i++) {\n      var script = scripts[i]\n      if (script.src != null) {\n\texisting_scripts.push(script.src)\n      }\n    }\n    for (var i = 0; i < js_urls.length; i++) {\n      var url = js_urls[i];\n      if (skip.indexOf(url) !== -1 || existing_scripts.indexOf(url) !== -1) {\n\tif (!window.requirejs) {\n\t  on_load();\n\t}\n\tcontinue;\n      }\n      var element = document.createElement('script');\n      element.onload = on_load;\n      element.onerror = on_error;\n      element.async = false;\n      element.src = url;\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      document.head.appendChild(element);\n    }\n    for (var i = 0; i < js_modules.length; i++) {\n      var url = js_modules[i];\n      if (skip.indexOf(url) !== -1 || existing_scripts.indexOf(url) !== -1) {\n\tif (!window.requirejs) {\n\t  on_load();\n\t}\n\tcontinue;\n      }\n      var element = document.createElement('script');\n      element.onload = on_load;\n      element.onerror = on_error;\n      element.async = false;\n      element.src = url;\n      element.type = \"module\";\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      document.head.appendChild(element);\n    }\n    for (const name in js_exports) {\n      var url = js_exports[name];\n      if (skip.indexOf(url) >= 0 || root[name] != null) {\n\tif (!window.requirejs) {\n\t  on_load();\n\t}\n\tcontinue;\n      }\n      var element = document.createElement('script');\n      element.onerror = on_error;\n      element.async = false;\n      element.type = \"module\";\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      element.textContent = `\n      import ${name} from \"${url}\"\n      window.${name} = ${name}\n      window._bokeh_on_load()\n      `\n      document.head.appendChild(element);\n    }\n    if (!js_urls.length && !js_modules.length) {\n      on_load()\n    }\n  };\n\n  function inject_raw_css(css) {\n    const element = document.createElement(\"style\");\n    element.appendChild(document.createTextNode(css));\n    document.body.appendChild(element);\n  }\n\n  var js_urls = [\"https://cdn.bokeh.org/bokeh/release/bokeh-3.3.4.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-gl-3.3.4.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-3.3.4.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-3.3.4.min.js\", \"https://cdn.holoviz.org/panel/1.3.8/dist/panel.min.js\"];\n  var js_modules = [];\n  var js_exports = {};\n  var css_urls = [];\n  var inline_js = [    function(Bokeh) {\n      Bokeh.set_log_level(\"info\");\n    },\nfunction(Bokeh) {} // ensure no trailing comma for IE\n  ];\n\n  function run_inline_js() {\n    if ((root.Bokeh !== undefined) || (force === true)) {\n      for (var i = 0; i < inline_js.length; i++) {\n\ttry {\n          inline_js[i].call(root, root.Bokeh);\n\t} catch(e) {\n\t  if (!reloading) {\n\t    throw e;\n\t  }\n\t}\n      }\n      // Cache old bokeh versions\n      if (Bokeh != undefined && !reloading) {\n\tvar NewBokeh = root.Bokeh;\n\tif (Bokeh.versions === undefined) {\n\t  Bokeh.versions = new Map();\n\t}\n\tif (NewBokeh.version !== Bokeh.version) {\n\t  Bokeh.versions.set(NewBokeh.version, NewBokeh)\n\t}\n\troot.Bokeh = Bokeh;\n      }} else if (Date.now() < root._bokeh_timeout) {\n      setTimeout(run_inline_js, 100);\n    } else if (!root._bokeh_failed_load) {\n      console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n      root._bokeh_failed_load = true;\n    }\n    root._bokeh_is_initializing = false\n  }\n\n  function load_or_wait() {\n    // Implement a backoff loop that tries to ensure we do not load multiple\n    // versions of Bokeh and its dependencies at the same time.\n    // In recent versions we use the root._bokeh_is_initializing flag\n    // to determine whether there is an ongoing attempt to initialize\n    // bokeh, however for backward compatibility we also try to ensure\n    // that we do not start loading a newer (Panel>=1.0 and Bokeh>3) version\n    // before older versions are fully initialized.\n    if (root._bokeh_is_initializing && Date.now() > root._bokeh_timeout) {\n      root._bokeh_is_initializing = false;\n      root._bokeh_onload_callbacks = undefined;\n      console.log(\"Bokeh: BokehJS was loaded multiple times but one version failed to initialize.\");\n      load_or_wait();\n    } else if (root._bokeh_is_initializing || (typeof root._bokeh_is_initializing === \"undefined\" && root._bokeh_onload_callbacks !== undefined)) {\n      setTimeout(load_or_wait, 100);\n    } else {\n      root._bokeh_is_initializing = true\n      root._bokeh_onload_callbacks = []\n      var bokeh_loaded = Bokeh != null && (Bokeh.version === py_version || (Bokeh.versions !== undefined && Bokeh.versions.has(py_version)));\n      if (!reloading && !bokeh_loaded) {\n\troot.Bokeh = undefined;\n      }\n      load_libs(css_urls, js_urls, js_modules, js_exports, function() {\n\tconsole.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n\trun_inline_js();\n      });\n    }\n  }\n  // Give older versions of the autoload script a head-start to ensure\n  // they initialize before we start loading newer version.\n  setTimeout(load_or_wait, 100)\n}(window));"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "if ((window.PyViz === undefined) || (window.PyViz instanceof HTMLElement)) {\n",
       "  window.PyViz = {comms: {}, comm_status:{}, kernels:{}, receivers: {}, plot_index: []}\n",
       "}\n",
       "\n",
       "\n",
       "    function JupyterCommManager() {\n",
       "    }\n",
       "\n",
       "    JupyterCommManager.prototype.register_target = function(plot_id, comm_id, msg_handler) {\n",
       "      if (window.comm_manager || ((window.Jupyter !== undefined) && (Jupyter.notebook.kernel != null))) {\n",
       "        var comm_manager = window.comm_manager || Jupyter.notebook.kernel.comm_manager;\n",
       "        comm_manager.register_target(comm_id, function(comm) {\n",
       "          comm.on_msg(msg_handler);\n",
       "        });\n",
       "      } else if ((plot_id in window.PyViz.kernels) && (window.PyViz.kernels[plot_id])) {\n",
       "        window.PyViz.kernels[plot_id].registerCommTarget(comm_id, function(comm) {\n",
       "          comm.onMsg = msg_handler;\n",
       "        });\n",
       "      } else if (typeof google != 'undefined' && google.colab.kernel != null) {\n",
       "        google.colab.kernel.comms.registerTarget(comm_id, (comm) => {\n",
       "          var messages = comm.messages[Symbol.asyncIterator]();\n",
       "          function processIteratorResult(result) {\n",
       "            var message = result.value;\n",
       "            console.log(message)\n",
       "            var content = {data: message.data, comm_id};\n",
       "            var buffers = []\n",
       "            for (var buffer of message.buffers || []) {\n",
       "              buffers.push(new DataView(buffer))\n",
       "            }\n",
       "            var metadata = message.metadata || {};\n",
       "            var msg = {content, buffers, metadata}\n",
       "            msg_handler(msg);\n",
       "            return messages.next().then(processIteratorResult);\n",
       "          }\n",
       "          return messages.next().then(processIteratorResult);\n",
       "        })\n",
       "      }\n",
       "    }\n",
       "\n",
       "    JupyterCommManager.prototype.get_client_comm = function(plot_id, comm_id, msg_handler) {\n",
       "      if (comm_id in window.PyViz.comms) {\n",
       "        return window.PyViz.comms[comm_id];\n",
       "      } else if (window.comm_manager || ((window.Jupyter !== undefined) && (Jupyter.notebook.kernel != null))) {\n",
       "        var comm_manager = window.comm_manager || Jupyter.notebook.kernel.comm_manager;\n",
       "        var comm = comm_manager.new_comm(comm_id, {}, {}, {}, comm_id);\n",
       "        if (msg_handler) {\n",
       "          comm.on_msg(msg_handler);\n",
       "        }\n",
       "      } else if ((plot_id in window.PyViz.kernels) && (window.PyViz.kernels[plot_id])) {\n",
       "        var comm = window.PyViz.kernels[plot_id].connectToComm(comm_id);\n",
       "        comm.open();\n",
       "        if (msg_handler) {\n",
       "          comm.onMsg = msg_handler;\n",
       "        }\n",
       "      } else if (typeof google != 'undefined' && google.colab.kernel != null) {\n",
       "        var comm_promise = google.colab.kernel.comms.open(comm_id)\n",
       "        comm_promise.then((comm) => {\n",
       "          window.PyViz.comms[comm_id] = comm;\n",
       "          if (msg_handler) {\n",
       "            var messages = comm.messages[Symbol.asyncIterator]();\n",
       "            function processIteratorResult(result) {\n",
       "              var message = result.value;\n",
       "              var content = {data: message.data};\n",
       "              var metadata = message.metadata || {comm_id};\n",
       "              var msg = {content, metadata}\n",
       "              msg_handler(msg);\n",
       "              return messages.next().then(processIteratorResult);\n",
       "            }\n",
       "            return messages.next().then(processIteratorResult);\n",
       "          }\n",
       "        }) \n",
       "        var sendClosure = (data, metadata, buffers, disposeOnDone) => {\n",
       "          return comm_promise.then((comm) => {\n",
       "            comm.send(data, metadata, buffers, disposeOnDone);\n",
       "          });\n",
       "        };\n",
       "        var comm = {\n",
       "          send: sendClosure\n",
       "        };\n",
       "      }\n",
       "      window.PyViz.comms[comm_id] = comm;\n",
       "      return comm;\n",
       "    }\n",
       "    window.PyViz.comm_manager = new JupyterCommManager();\n",
       "    \n",
       "\n",
       "\n",
       "var JS_MIME_TYPE = 'application/javascript';\n",
       "var HTML_MIME_TYPE = 'text/html';\n",
       "var EXEC_MIME_TYPE = 'application/vnd.holoviews_exec.v0+json';\n",
       "var CLASS_NAME = 'output';\n",
       "\n",
       "/**\n",
       " * Render data to the DOM node\n",
       " */\n",
       "function render(props, node) {\n",
       "  var div = document.createElement(\"div\");\n",
       "  var script = document.createElement(\"script\");\n",
       "  node.appendChild(div);\n",
       "  node.appendChild(script);\n",
       "}\n",
       "\n",
       "/**\n",
       " * Handle when a new output is added\n",
       " */\n",
       "function handle_add_output(event, handle) {\n",
       "  var output_area = handle.output_area;\n",
       "  var output = handle.output;\n",
       "  if ((output.data == undefined) || (!output.data.hasOwnProperty(EXEC_MIME_TYPE))) {\n",
       "    return\n",
       "  }\n",
       "  var id = output.metadata[EXEC_MIME_TYPE][\"id\"];\n",
       "  var toinsert = output_area.element.find(\".\" + CLASS_NAME.split(' ')[0]);\n",
       "  if (id !== undefined) {\n",
       "    var nchildren = toinsert.length;\n",
       "    var html_node = toinsert[nchildren-1].children[0];\n",
       "    html_node.innerHTML = output.data[HTML_MIME_TYPE];\n",
       "    var scripts = [];\n",
       "    var nodelist = html_node.querySelectorAll(\"script\");\n",
       "    for (var i in nodelist) {\n",
       "      if (nodelist.hasOwnProperty(i)) {\n",
       "        scripts.push(nodelist[i])\n",
       "      }\n",
       "    }\n",
       "\n",
       "    scripts.forEach( function (oldScript) {\n",
       "      var newScript = document.createElement(\"script\");\n",
       "      var attrs = [];\n",
       "      var nodemap = oldScript.attributes;\n",
       "      for (var j in nodemap) {\n",
       "        if (nodemap.hasOwnProperty(j)) {\n",
       "          attrs.push(nodemap[j])\n",
       "        }\n",
       "      }\n",
       "      attrs.forEach(function(attr) { newScript.setAttribute(attr.name, attr.value) });\n",
       "      newScript.appendChild(document.createTextNode(oldScript.innerHTML));\n",
       "      oldScript.parentNode.replaceChild(newScript, oldScript);\n",
       "    });\n",
       "    if (JS_MIME_TYPE in output.data) {\n",
       "      toinsert[nchildren-1].children[1].textContent = output.data[JS_MIME_TYPE];\n",
       "    }\n",
       "    output_area._hv_plot_id = id;\n",
       "    if ((window.Bokeh !== undefined) && (id in Bokeh.index)) {\n",
       "      window.PyViz.plot_index[id] = Bokeh.index[id];\n",
       "    } else {\n",
       "      window.PyViz.plot_index[id] = null;\n",
       "    }\n",
       "  } else if (output.metadata[EXEC_MIME_TYPE][\"server_id\"] !== undefined) {\n",
       "    var bk_div = document.createElement(\"div\");\n",
       "    bk_div.innerHTML = output.data[HTML_MIME_TYPE];\n",
       "    var script_attrs = bk_div.children[0].attributes;\n",
       "    for (var i = 0; i < script_attrs.length; i++) {\n",
       "      toinsert[toinsert.length - 1].childNodes[1].setAttribute(script_attrs[i].name, script_attrs[i].value);\n",
       "    }\n",
       "    // store reference to server id on output_area\n",
       "    output_area._bokeh_server_id = output.metadata[EXEC_MIME_TYPE][\"server_id\"];\n",
       "  }\n",
       "}\n",
       "\n",
       "/**\n",
       " * Handle when an output is cleared or removed\n",
       " */\n",
       "function handle_clear_output(event, handle) {\n",
       "  var id = handle.cell.output_area._hv_plot_id;\n",
       "  var server_id = handle.cell.output_area._bokeh_server_id;\n",
       "  if (((id === undefined) || !(id in PyViz.plot_index)) && (server_id !== undefined)) { return; }\n",
       "  var comm = window.PyViz.comm_manager.get_client_comm(\"hv-extension-comm\", \"hv-extension-comm\", function () {});\n",
       "  if (server_id !== null) {\n",
       "    comm.send({event_type: 'server_delete', 'id': server_id});\n",
       "    return;\n",
       "  } else if (comm !== null) {\n",
       "    comm.send({event_type: 'delete', 'id': id});\n",
       "  }\n",
       "  delete PyViz.plot_index[id];\n",
       "  if ((window.Bokeh !== undefined) & (id in window.Bokeh.index)) {\n",
       "    var doc = window.Bokeh.index[id].model.document\n",
       "    doc.clear();\n",
       "    const i = window.Bokeh.documents.indexOf(doc);\n",
       "    if (i > -1) {\n",
       "      window.Bokeh.documents.splice(i, 1);\n",
       "    }\n",
       "  }\n",
       "}\n",
       "\n",
       "/**\n",
       " * Handle kernel restart event\n",
       " */\n",
       "function handle_kernel_cleanup(event, handle) {\n",
       "  delete PyViz.comms[\"hv-extension-comm\"];\n",
       "  window.PyViz.plot_index = {}\n",
       "}\n",
       "\n",
       "/**\n",
       " * Handle update_display_data messages\n",
       " */\n",
       "function handle_update_output(event, handle) {\n",
       "  handle_clear_output(event, {cell: {output_area: handle.output_area}})\n",
       "  handle_add_output(event, handle)\n",
       "}\n",
       "\n",
       "function register_renderer(events, OutputArea) {\n",
       "  function append_mime(data, metadata, element) {\n",
       "    // create a DOM node to render to\n",
       "    var toinsert = this.create_output_subarea(\n",
       "    metadata,\n",
       "    CLASS_NAME,\n",
       "    EXEC_MIME_TYPE\n",
       "    );\n",
       "    this.keyboard_manager.register_events(toinsert);\n",
       "    // Render to node\n",
       "    var props = {data: data, metadata: metadata[EXEC_MIME_TYPE]};\n",
       "    render(props, toinsert[0]);\n",
       "    element.append(toinsert);\n",
       "    return toinsert\n",
       "  }\n",
       "\n",
       "  events.on('output_added.OutputArea', handle_add_output);\n",
       "  events.on('output_updated.OutputArea', handle_update_output);\n",
       "  events.on('clear_output.CodeCell', handle_clear_output);\n",
       "  events.on('delete.Cell', handle_clear_output);\n",
       "  events.on('kernel_ready.Kernel', handle_kernel_cleanup);\n",
       "\n",
       "  OutputArea.prototype.register_mime_type(EXEC_MIME_TYPE, append_mime, {\n",
       "    safe: true,\n",
       "    index: 0\n",
       "  });\n",
       "}\n",
       "\n",
       "if (window.Jupyter !== undefined) {\n",
       "  try {\n",
       "    var events = require('base/js/events');\n",
       "    var OutputArea = require('notebook/js/outputarea').OutputArea;\n",
       "    if (OutputArea.prototype.mime_types().indexOf(EXEC_MIME_TYPE) == -1) {\n",
       "      register_renderer(events, OutputArea);\n",
       "    }\n",
       "  } catch(err) {\n",
       "  }\n",
       "}\n"
      ],
      "application/vnd.holoviews_load.v0+json": "\nif ((window.PyViz === undefined) || (window.PyViz instanceof HTMLElement)) {\n  window.PyViz = {comms: {}, comm_status:{}, kernels:{}, receivers: {}, plot_index: []}\n}\n\n\n    function JupyterCommManager() {\n    }\n\n    JupyterCommManager.prototype.register_target = function(plot_id, comm_id, msg_handler) {\n      if (window.comm_manager || ((window.Jupyter !== undefined) && (Jupyter.notebook.kernel != null))) {\n        var comm_manager = window.comm_manager || Jupyter.notebook.kernel.comm_manager;\n        comm_manager.register_target(comm_id, function(comm) {\n          comm.on_msg(msg_handler);\n        });\n      } else if ((plot_id in window.PyViz.kernels) && (window.PyViz.kernels[plot_id])) {\n        window.PyViz.kernels[plot_id].registerCommTarget(comm_id, function(comm) {\n          comm.onMsg = msg_handler;\n        });\n      } else if (typeof google != 'undefined' && google.colab.kernel != null) {\n        google.colab.kernel.comms.registerTarget(comm_id, (comm) => {\n          var messages = comm.messages[Symbol.asyncIterator]();\n          function processIteratorResult(result) {\n            var message = result.value;\n            console.log(message)\n            var content = {data: message.data, comm_id};\n            var buffers = []\n            for (var buffer of message.buffers || []) {\n              buffers.push(new DataView(buffer))\n            }\n            var metadata = message.metadata || {};\n            var msg = {content, buffers, metadata}\n            msg_handler(msg);\n            return messages.next().then(processIteratorResult);\n          }\n          return messages.next().then(processIteratorResult);\n        })\n      }\n    }\n\n    JupyterCommManager.prototype.get_client_comm = function(plot_id, comm_id, msg_handler) {\n      if (comm_id in window.PyViz.comms) {\n        return window.PyViz.comms[comm_id];\n      } else if (window.comm_manager || ((window.Jupyter !== undefined) && (Jupyter.notebook.kernel != null))) {\n        var comm_manager = window.comm_manager || Jupyter.notebook.kernel.comm_manager;\n        var comm = comm_manager.new_comm(comm_id, {}, {}, {}, comm_id);\n        if (msg_handler) {\n          comm.on_msg(msg_handler);\n        }\n      } else if ((plot_id in window.PyViz.kernels) && (window.PyViz.kernels[plot_id])) {\n        var comm = window.PyViz.kernels[plot_id].connectToComm(comm_id);\n        comm.open();\n        if (msg_handler) {\n          comm.onMsg = msg_handler;\n        }\n      } else if (typeof google != 'undefined' && google.colab.kernel != null) {\n        var comm_promise = google.colab.kernel.comms.open(comm_id)\n        comm_promise.then((comm) => {\n          window.PyViz.comms[comm_id] = comm;\n          if (msg_handler) {\n            var messages = comm.messages[Symbol.asyncIterator]();\n            function processIteratorResult(result) {\n              var message = result.value;\n              var content = {data: message.data};\n              var metadata = message.metadata || {comm_id};\n              var msg = {content, metadata}\n              msg_handler(msg);\n              return messages.next().then(processIteratorResult);\n            }\n            return messages.next().then(processIteratorResult);\n          }\n        }) \n        var sendClosure = (data, metadata, buffers, disposeOnDone) => {\n          return comm_promise.then((comm) => {\n            comm.send(data, metadata, buffers, disposeOnDone);\n          });\n        };\n        var comm = {\n          send: sendClosure\n        };\n      }\n      window.PyViz.comms[comm_id] = comm;\n      return comm;\n    }\n    window.PyViz.comm_manager = new JupyterCommManager();\n    \n\n\nvar JS_MIME_TYPE = 'application/javascript';\nvar HTML_MIME_TYPE = 'text/html';\nvar EXEC_MIME_TYPE = 'application/vnd.holoviews_exec.v0+json';\nvar CLASS_NAME = 'output';\n\n/**\n * Render data to the DOM node\n */\nfunction render(props, node) {\n  var div = document.createElement(\"div\");\n  var script = document.createElement(\"script\");\n  node.appendChild(div);\n  node.appendChild(script);\n}\n\n/**\n * Handle when a new output is added\n */\nfunction handle_add_output(event, handle) {\n  var output_area = handle.output_area;\n  var output = handle.output;\n  if ((output.data == undefined) || (!output.data.hasOwnProperty(EXEC_MIME_TYPE))) {\n    return\n  }\n  var id = output.metadata[EXEC_MIME_TYPE][\"id\"];\n  var toinsert = output_area.element.find(\".\" + CLASS_NAME.split(' ')[0]);\n  if (id !== undefined) {\n    var nchildren = toinsert.length;\n    var html_node = toinsert[nchildren-1].children[0];\n    html_node.innerHTML = output.data[HTML_MIME_TYPE];\n    var scripts = [];\n    var nodelist = html_node.querySelectorAll(\"script\");\n    for (var i in nodelist) {\n      if (nodelist.hasOwnProperty(i)) {\n        scripts.push(nodelist[i])\n      }\n    }\n\n    scripts.forEach( function (oldScript) {\n      var newScript = document.createElement(\"script\");\n      var attrs = [];\n      var nodemap = oldScript.attributes;\n      for (var j in nodemap) {\n        if (nodemap.hasOwnProperty(j)) {\n          attrs.push(nodemap[j])\n        }\n      }\n      attrs.forEach(function(attr) { newScript.setAttribute(attr.name, attr.value) });\n      newScript.appendChild(document.createTextNode(oldScript.innerHTML));\n      oldScript.parentNode.replaceChild(newScript, oldScript);\n    });\n    if (JS_MIME_TYPE in output.data) {\n      toinsert[nchildren-1].children[1].textContent = output.data[JS_MIME_TYPE];\n    }\n    output_area._hv_plot_id = id;\n    if ((window.Bokeh !== undefined) && (id in Bokeh.index)) {\n      window.PyViz.plot_index[id] = Bokeh.index[id];\n    } else {\n      window.PyViz.plot_index[id] = null;\n    }\n  } else if (output.metadata[EXEC_MIME_TYPE][\"server_id\"] !== undefined) {\n    var bk_div = document.createElement(\"div\");\n    bk_div.innerHTML = output.data[HTML_MIME_TYPE];\n    var script_attrs = bk_div.children[0].attributes;\n    for (var i = 0; i < script_attrs.length; i++) {\n      toinsert[toinsert.length - 1].childNodes[1].setAttribute(script_attrs[i].name, script_attrs[i].value);\n    }\n    // store reference to server id on output_area\n    output_area._bokeh_server_id = output.metadata[EXEC_MIME_TYPE][\"server_id\"];\n  }\n}\n\n/**\n * Handle when an output is cleared or removed\n */\nfunction handle_clear_output(event, handle) {\n  var id = handle.cell.output_area._hv_plot_id;\n  var server_id = handle.cell.output_area._bokeh_server_id;\n  if (((id === undefined) || !(id in PyViz.plot_index)) && (server_id !== undefined)) { return; }\n  var comm = window.PyViz.comm_manager.get_client_comm(\"hv-extension-comm\", \"hv-extension-comm\", function () {});\n  if (server_id !== null) {\n    comm.send({event_type: 'server_delete', 'id': server_id});\n    return;\n  } else if (comm !== null) {\n    comm.send({event_type: 'delete', 'id': id});\n  }\n  delete PyViz.plot_index[id];\n  if ((window.Bokeh !== undefined) & (id in window.Bokeh.index)) {\n    var doc = window.Bokeh.index[id].model.document\n    doc.clear();\n    const i = window.Bokeh.documents.indexOf(doc);\n    if (i > -1) {\n      window.Bokeh.documents.splice(i, 1);\n    }\n  }\n}\n\n/**\n * Handle kernel restart event\n */\nfunction handle_kernel_cleanup(event, handle) {\n  delete PyViz.comms[\"hv-extension-comm\"];\n  window.PyViz.plot_index = {}\n}\n\n/**\n * Handle update_display_data messages\n */\nfunction handle_update_output(event, handle) {\n  handle_clear_output(event, {cell: {output_area: handle.output_area}})\n  handle_add_output(event, handle)\n}\n\nfunction register_renderer(events, OutputArea) {\n  function append_mime(data, metadata, element) {\n    // create a DOM node to render to\n    var toinsert = this.create_output_subarea(\n    metadata,\n    CLASS_NAME,\n    EXEC_MIME_TYPE\n    );\n    this.keyboard_manager.register_events(toinsert);\n    // Render to node\n    var props = {data: data, metadata: metadata[EXEC_MIME_TYPE]};\n    render(props, toinsert[0]);\n    element.append(toinsert);\n    return toinsert\n  }\n\n  events.on('output_added.OutputArea', handle_add_output);\n  events.on('output_updated.OutputArea', handle_update_output);\n  events.on('clear_output.CodeCell', handle_clear_output);\n  events.on('delete.Cell', handle_clear_output);\n  events.on('kernel_ready.Kernel', handle_kernel_cleanup);\n\n  OutputArea.prototype.register_mime_type(EXEC_MIME_TYPE, append_mime, {\n    safe: true,\n    index: 0\n  });\n}\n\nif (window.Jupyter !== undefined) {\n  try {\n    var events = require('base/js/events');\n    var OutputArea = require('notebook/js/outputarea').OutputArea;\n    if (OutputArea.prototype.mime_types().indexOf(EXEC_MIME_TYPE) == -1) {\n      register_renderer(events, OutputArea);\n    }\n  } catch(err) {\n  }\n}\n"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>*[data-root-id],\n",
       "*[data-root-id] > * {\n",
       "  box-sizing: border-box;\n",
       "  font-family: var(--jp-ui-font-family);\n",
       "  font-size: var(--jp-ui-font-size1);\n",
       "  color: var(--vscode-editor-foreground, var(--jp-ui-font-color1));\n",
       "}\n",
       "\n",
       "/* Override VSCode background color */\n",
       ".cell-output-ipywidget-background:has(\n",
       "    > .cell-output-ipywidget-background > .lm-Widget > *[data-root-id]\n",
       "  ),\n",
       ".cell-output-ipywidget-background:has(> .lm-Widget > *[data-root-id]) {\n",
       "  background-color: transparent !important;\n",
       "}\n",
       "</style>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.holoviews_exec.v0+json": "",
      "text/html": [
       "<div id='8c70989c-5f6f-4dd8-9a10-f7dd89e8d03f'>\n",
       "  <div id=\"e7071988-5275-4c2f-bb46-b0cad3495d05\" data-root-id=\"8c70989c-5f6f-4dd8-9a10-f7dd89e8d03f\" style=\"display: contents;\"></div>\n",
       "</div>\n",
       "<script type=\"application/javascript\">(function(root) {\n",
       "  var docs_json = {\"ed256e05-9118-43f0-9cec-f7ff32c5606d\":{\"version\":\"3.3.4\",\"title\":\"Bokeh Application\",\"roots\":[{\"type\":\"object\",\"name\":\"panel.models.browser.BrowserInfo\",\"id\":\"8c70989c-5f6f-4dd8-9a10-f7dd89e8d03f\"},{\"type\":\"object\",\"name\":\"panel.models.comm_manager.CommManager\",\"id\":\"6e17b2aa-4df9-4998-b768-113b8edd63d2\",\"attributes\":{\"plot_id\":\"8c70989c-5f6f-4dd8-9a10-f7dd89e8d03f\",\"comm_id\":\"c5feb2996df24c83bb183ef581fa1b20\",\"client_comm_id\":\"2afc53dec9df4e9b8cf43bf674052c91\"}}],\"defs\":[{\"type\":\"model\",\"name\":\"ReactiveHTML1\"},{\"type\":\"model\",\"name\":\"FlexBox1\",\"properties\":[{\"name\":\"align_content\",\"kind\":\"Any\",\"default\":\"flex-start\"},{\"name\":\"align_items\",\"kind\":\"Any\",\"default\":\"flex-start\"},{\"name\":\"flex_direction\",\"kind\":\"Any\",\"default\":\"row\"},{\"name\":\"flex_wrap\",\"kind\":\"Any\",\"default\":\"wrap\"},{\"name\":\"justify_content\",\"kind\":\"Any\",\"default\":\"flex-start\"}]},{\"type\":\"model\",\"name\":\"FloatPanel1\",\"properties\":[{\"name\":\"config\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}},{\"name\":\"contained\",\"kind\":\"Any\",\"default\":true},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"right-top\"},{\"name\":\"offsetx\",\"kind\":\"Any\",\"default\":null},{\"name\":\"offsety\",\"kind\":\"Any\",\"default\":null},{\"name\":\"theme\",\"kind\":\"Any\",\"default\":\"primary\"},{\"name\":\"status\",\"kind\":\"Any\",\"default\":\"normalized\"}]},{\"type\":\"model\",\"name\":\"GridStack1\",\"properties\":[{\"name\":\"mode\",\"kind\":\"Any\",\"default\":\"warn\"},{\"name\":\"ncols\",\"kind\":\"Any\",\"default\":null},{\"name\":\"nrows\",\"kind\":\"Any\",\"default\":null},{\"name\":\"allow_resize\",\"kind\":\"Any\",\"default\":true},{\"name\":\"allow_drag\",\"kind\":\"Any\",\"default\":true},{\"name\":\"state\",\"kind\":\"Any\",\"default\":[]}]},{\"type\":\"model\",\"name\":\"drag1\",\"properties\":[{\"name\":\"slider_width\",\"kind\":\"Any\",\"default\":5},{\"name\":\"slider_color\",\"kind\":\"Any\",\"default\":\"black\"},{\"name\":\"value\",\"kind\":\"Any\",\"default\":50}]},{\"type\":\"model\",\"name\":\"click1\",\"properties\":[{\"name\":\"terminal_output\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"debug_name\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"clears\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"copy_to_clipboard1\",\"properties\":[{\"name\":\"fill\",\"kind\":\"Any\",\"default\":\"none\"},{\"name\":\"value\",\"kind\":\"Any\",\"default\":null}]},{\"type\":\"model\",\"name\":\"FastWrapper1\",\"properties\":[{\"name\":\"object\",\"kind\":\"Any\",\"default\":null},{\"name\":\"style\",\"kind\":\"Any\",\"default\":null}]},{\"type\":\"model\",\"name\":\"NotificationAreaBase1\",\"properties\":[{\"name\":\"js_events\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"bottom-right\"},{\"name\":\"_clear\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"NotificationArea1\",\"properties\":[{\"name\":\"js_events\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}},{\"name\":\"notifications\",\"kind\":\"Any\",\"default\":[]},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"bottom-right\"},{\"name\":\"_clear\",\"kind\":\"Any\",\"default\":0},{\"name\":\"types\",\"kind\":\"Any\",\"default\":[{\"type\":\"map\",\"entries\":[[\"type\",\"warning\"],[\"background\",\"#ffc107\"],[\"icon\",{\"type\":\"map\",\"entries\":[[\"className\",\"fas fa-exclamation-triangle\"],[\"tagName\",\"i\"],[\"color\",\"white\"]]}]]},{\"type\":\"map\",\"entries\":[[\"type\",\"info\"],[\"background\",\"#007bff\"],[\"icon\",{\"type\":\"map\",\"entries\":[[\"className\",\"fas fa-info-circle\"],[\"tagName\",\"i\"],[\"color\",\"white\"]]}]]}]}]},{\"type\":\"model\",\"name\":\"Notification\",\"properties\":[{\"name\":\"background\",\"kind\":\"Any\",\"default\":null},{\"name\":\"duration\",\"kind\":\"Any\",\"default\":3000},{\"name\":\"icon\",\"kind\":\"Any\",\"default\":null},{\"name\":\"message\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"notification_type\",\"kind\":\"Any\",\"default\":null},{\"name\":\"_destroyed\",\"kind\":\"Any\",\"default\":false}]},{\"type\":\"model\",\"name\":\"TemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"BootstrapTemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"MaterialTemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]}]}};\n",
       "  var render_items = [{\"docid\":\"ed256e05-9118-43f0-9cec-f7ff32c5606d\",\"roots\":{\"8c70989c-5f6f-4dd8-9a10-f7dd89e8d03f\":\"e7071988-5275-4c2f-bb46-b0cad3495d05\"},\"root_ids\":[\"8c70989c-5f6f-4dd8-9a10-f7dd89e8d03f\"]}];\n",
       "  var docs = Object.values(docs_json)\n",
       "  if (!docs) {\n",
       "    return\n",
       "  }\n",
       "  const py_version = docs[0].version.replace('rc', '-rc.').replace('.dev', '-dev.')\n",
       "  function embed_document(root) {\n",
       "    var Bokeh = get_bokeh(root)\n",
       "    Bokeh.embed.embed_items_notebook(docs_json, render_items);\n",
       "    for (const render_item of render_items) {\n",
       "      for (const root_id of render_item.root_ids) {\n",
       "\tconst id_el = document.getElementById(root_id)\n",
       "\tif (id_el.children.length && (id_el.children[0].className === 'bk-root')) {\n",
       "\t  const root_el = id_el.children[0]\n",
       "\t  root_el.id = root_el.id + '-rendered'\n",
       "\t}\n",
       "      }\n",
       "    }\n",
       "  }\n",
       "  function get_bokeh(root) {\n",
       "    if (root.Bokeh === undefined) {\n",
       "      return null\n",
       "    } else if (root.Bokeh.version !== py_version) {\n",
       "      if (root.Bokeh.versions === undefined || !root.Bokeh.versions.has(py_version)) {\n",
       "\treturn null\n",
       "      }\n",
       "      return root.Bokeh.versions.get(py_version);\n",
       "    } else if (root.Bokeh.version === py_version) {\n",
       "      return root.Bokeh\n",
       "    }\n",
       "    return null\n",
       "  }\n",
       "  function is_loaded(root) {\n",
       "    var Bokeh = get_bokeh(root)\n",
       "    return (Bokeh != null && Bokeh.Panel !== undefined)\n",
       "  }\n",
       "  if (is_loaded(root)) {\n",
       "    embed_document(root);\n",
       "  } else {\n",
       "    var attempts = 0;\n",
       "    var timer = setInterval(function(root) {\n",
       "      if (is_loaded(root)) {\n",
       "        clearInterval(timer);\n",
       "        embed_document(root);\n",
       "      } else if (document.readyState == \"complete\") {\n",
       "        attempts++;\n",
       "        if (attempts > 200) {\n",
       "          clearInterval(timer);\n",
       "\t  var Bokeh = get_bokeh(root)\n",
       "\t  if (Bokeh == null || Bokeh.Panel == null) {\n",
       "            console.warn(\"Panel: ERROR: Unable to run Panel code because Bokeh or Panel library is missing\");\n",
       "\t  } else {\n",
       "\t    console.warn(\"Panel: WARNING: Attempting to render but not all required libraries could be resolved.\")\n",
       "\t    embed_document(root)\n",
       "\t  }\n",
       "        }\n",
       "      }\n",
       "    }, 25, root)\n",
       "  }\n",
       "})(window);</script>"
      ]
     },
     "metadata": {
      "application/vnd.holoviews_exec.v0+json": {
       "id": "8c70989c-5f6f-4dd8-9a10-f7dd89e8d03f"
      }
     },
     "output_type": "display_data"
    },
    {
     "data": {},
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.holoviews_exec.v0+json": "",
      "text/html": [
       "<div id='90920467-d36f-4899-8f12-2d496e2ce38f'>\n",
       "  <div id=\"dcf22597-1e5b-4e89-82db-88909f8750d8\" data-root-id=\"90920467-d36f-4899-8f12-2d496e2ce38f\" style=\"display: contents;\"></div>\n",
       "</div>\n",
       "<script type=\"application/javascript\">(function(root) {\n",
       "  var docs_json = {\"0372bee8-4baf-4349-9fbe-772561ed740a\":{\"version\":\"3.3.4\",\"title\":\"Bokeh Application\",\"roots\":[{\"type\":\"object\",\"name\":\"panel.models.layout.Column\",\"id\":\"90920467-d36f-4899-8f12-2d496e2ce38f\",\"attributes\":{\"name\":\"Column00119\",\"stylesheets\":[\"\\n:host(.pn-loading.pn-arc):before, .pn-loading.pn-arc:before {\\n  background-image: url(\\\"data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHN0eWxlPSJtYXJnaW46IGF1dG87IGJhY2tncm91bmQ6IG5vbmU7IGRpc3BsYXk6IGJsb2NrOyBzaGFwZS1yZW5kZXJpbmc6IGF1dG87IiB2aWV3Qm94PSIwIDAgMTAwIDEwMCIgcHJlc2VydmVBc3BlY3RSYXRpbz0ieE1pZFlNaWQiPiAgPGNpcmNsZSBjeD0iNTAiIGN5PSI1MCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjYzNjM2MzIiBzdHJva2Utd2lkdGg9IjEwIiByPSIzNSIgc3Ryb2tlLWRhc2hhcnJheT0iMTY0LjkzMzYxNDMxMzQ2NDE1IDU2Ljk3Nzg3MTQzNzgyMTM4Ij4gICAgPGFuaW1hdGVUcmFuc2Zvcm0gYXR0cmlidXRlTmFtZT0idHJhbnNmb3JtIiB0eXBlPSJyb3RhdGUiIHJlcGVhdENvdW50PSJpbmRlZmluaXRlIiBkdXI9IjFzIiB2YWx1ZXM9IjAgNTAgNTA7MzYwIDUwIDUwIiBrZXlUaW1lcz0iMDsxIj48L2FuaW1hdGVUcmFuc2Zvcm0+ICA8L2NpcmNsZT48L3N2Zz4=\\\");\\n  background-size: auto calc(min(50%, 400px));\\n}\",{\"type\":\"object\",\"name\":\"ImportedStyleSheet\",\"id\":\"2c2e3239-31bf-4cce-a22b-7f079d3a3df2\",\"attributes\":{\"url\":\"https://cdn.holoviz.org/panel/1.3.8/dist/css/loading.css\"}},{\"type\":\"object\",\"name\":\"ImportedStyleSheet\",\"id\":\"479aa1a6-74b5-4975-8090-f2408d553157\",\"attributes\":{\"url\":\"https://cdn.holoviz.org/panel/1.3.8/dist/css/listpanel.css\"}},{\"type\":\"object\",\"name\":\"ImportedStyleSheet\",\"id\":\"0fad62a6-e546-49b1-a0f4-37ed6d009db6\",\"attributes\":{\"url\":\"https://cdn.holoviz.org/panel/1.3.8/dist/bundled/theme/default.css\"}},{\"type\":\"object\",\"name\":\"ImportedStyleSheet\",\"id\":\"61e210a8-bb3d-4737-80c0-9c67cecfb7c7\",\"attributes\":{\"url\":\"https://cdn.holoviz.org/panel/1.3.8/dist/bundled/theme/native.css\"}}],\"margin\":0,\"align\":\"start\",\"children\":[{\"type\":\"object\",\"name\":\"panel.models.layout.Column\",\"id\":\"0dd12600-00de-4f82-a6fd-2d1c7d145fda\",\"attributes\":{\"name\":\"Column00124\",\"stylesheets\":[\"\\n:host(.pn-loading.pn-arc):before, .pn-loading.pn-arc:before {\\n  background-image: url(\\\"data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHN0eWxlPSJtYXJnaW46IGF1dG87IGJhY2tncm91bmQ6IG5vbmU7IGRpc3BsYXk6IGJsb2NrOyBzaGFwZS1yZW5kZXJpbmc6IGF1dG87IiB2aWV3Qm94PSIwIDAgMTAwIDEwMCIgcHJlc2VydmVBc3BlY3RSYXRpbz0ieE1pZFlNaWQiPiAgPGNpcmNsZSBjeD0iNTAiIGN5PSI1MCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjYzNjM2MzIiBzdHJva2Utd2lkdGg9IjEwIiByPSIzNSIgc3Ryb2tlLWRhc2hhcnJheT0iMTY0LjkzMzYxNDMxMzQ2NDE1IDU2Ljk3Nzg3MTQzNzgyMTM4Ij4gICAgPGFuaW1hdGVUcmFuc2Zvcm0gYXR0cmlidXRlTmFtZT0idHJhbnNmb3JtIiB0eXBlPSJyb3RhdGUiIHJlcGVhdENvdW50PSJpbmRlZmluaXRlIiBkdXI9IjFzIiB2YWx1ZXM9IjAgNTAgNTA7MzYwIDUwIDUwIiBrZXlUaW1lcz0iMDsxIj48L2FuaW1hdGVUcmFuc2Zvcm0+ICA8L2NpcmNsZT48L3N2Zz4=\\\");\\n  background-size: auto calc(min(50%, 400px));\\n}\",{\"id\":\"2c2e3239-31bf-4cce-a22b-7f079d3a3df2\"},{\"id\":\"479aa1a6-74b5-4975-8090-f2408d553157\"},{\"id\":\"0fad62a6-e546-49b1-a0f4-37ed6d009db6\"},{\"id\":\"61e210a8-bb3d-4737-80c0-9c67cecfb7c7\"}],\"margin\":0,\"align\":\"start\",\"children\":[{\"type\":\"object\",\"name\":\"Slider\",\"id\":\"3ff30394-f7ed-4f68-abb9-7eca8a65b137\",\"attributes\":{\"stylesheets\":[\"\\n:host(.pn-loading.pn-arc):before, .pn-loading.pn-arc:before {\\n  background-image: url(\\\"data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHN0eWxlPSJtYXJnaW46IGF1dG87IGJhY2tncm91bmQ6IG5vbmU7IGRpc3BsYXk6IGJsb2NrOyBzaGFwZS1yZW5kZXJpbmc6IGF1dG87IiB2aWV3Qm94PSIwIDAgMTAwIDEwMCIgcHJlc2VydmVBc3BlY3RSYXRpbz0ieE1pZFlNaWQiPiAgPGNpcmNsZSBjeD0iNTAiIGN5PSI1MCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjYzNjM2MzIiBzdHJva2Utd2lkdGg9IjEwIiByPSIzNSIgc3Ryb2tlLWRhc2hhcnJheT0iMTY0LjkzMzYxNDMxMzQ2NDE1IDU2Ljk3Nzg3MTQzNzgyMTM4Ij4gICAgPGFuaW1hdGVUcmFuc2Zvcm0gYXR0cmlidXRlTmFtZT0idHJhbnNmb3JtIiB0eXBlPSJyb3RhdGUiIHJlcGVhdENvdW50PSJpbmRlZmluaXRlIiBkdXI9IjFzIiB2YWx1ZXM9IjAgNTAgNTA7MzYwIDUwIDUwIiBrZXlUaW1lcz0iMDsxIj48L2FuaW1hdGVUcmFuc2Zvcm0+ICA8L2NpcmNsZT48L3N2Zz4=\\\");\\n  background-size: auto calc(min(50%, 400px));\\n}\",{\"id\":\"2c2e3239-31bf-4cce-a22b-7f079d3a3df2\"},{\"id\":\"0fad62a6-e546-49b1-a0f4-37ed6d009db6\"},{\"id\":\"61e210a8-bb3d-4737-80c0-9c67cecfb7c7\"}],\"margin\":[5,10],\"align\":\"start\",\"start\":0,\"end\":1000,\"value\":0,\"step\":100}}]}},{\"type\":\"object\",\"name\":\"Row\",\"id\":\"978c3fe7-0f25-466b-ac77-9892aa91d204\",\"attributes\":{\"name\":\"Row00123\",\"stylesheets\":[\"\\n:host(.pn-loading.pn-arc):before, .pn-loading.pn-arc:before {\\n  background-image: url(\\\"data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHN0eWxlPSJtYXJnaW46IGF1dG87IGJhY2tncm91bmQ6IG5vbmU7IGRpc3BsYXk6IGJsb2NrOyBzaGFwZS1yZW5kZXJpbmc6IGF1dG87IiB2aWV3Qm94PSIwIDAgMTAwIDEwMCIgcHJlc2VydmVBc3BlY3RSYXRpbz0ieE1pZFlNaWQiPiAgPGNpcmNsZSBjeD0iNTAiIGN5PSI1MCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjYzNjM2MzIiBzdHJva2Utd2lkdGg9IjEwIiByPSIzNSIgc3Ryb2tlLWRhc2hhcnJheT0iMTY0LjkzMzYxNDMxMzQ2NDE1IDU2Ljk3Nzg3MTQzNzgyMTM4Ij4gICAgPGFuaW1hdGVUcmFuc2Zvcm0gYXR0cmlidXRlTmFtZT0idHJhbnNmb3JtIiB0eXBlPSJyb3RhdGUiIHJlcGVhdENvdW50PSJpbmRlZmluaXRlIiBkdXI9IjFzIiB2YWx1ZXM9IjAgNTAgNTA7MzYwIDUwIDUwIiBrZXlUaW1lcz0iMDsxIj48L2FuaW1hdGVUcmFuc2Zvcm0+ICA8L2NpcmNsZT48L3N2Zz4=\\\");\\n  background-size: auto calc(min(50%, 400px));\\n}\",{\"id\":\"2c2e3239-31bf-4cce-a22b-7f079d3a3df2\"},{\"id\":\"479aa1a6-74b5-4975-8090-f2408d553157\"},{\"id\":\"0fad62a6-e546-49b1-a0f4-37ed6d009db6\"},{\"id\":\"61e210a8-bb3d-4737-80c0-9c67cecfb7c7\"}],\"margin\":0,\"align\":\"start\",\"children\":[{\"type\":\"object\",\"name\":\"panel.models.markup.HTML\",\"id\":\"040e0922-e6ee-455f-ad26-36aff8da39c3\",\"attributes\":{\"stylesheets\":[\"\\n:host(.pn-loading.pn-arc):before, .pn-loading.pn-arc:before {\\n  background-image: url(\\\"data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHN0eWxlPSJtYXJnaW46IGF1dG87IGJhY2tncm91bmQ6IG5vbmU7IGRpc3BsYXk6IGJsb2NrOyBzaGFwZS1yZW5kZXJpbmc6IGF1dG87IiB2aWV3Qm94PSIwIDAgMTAwIDEwMCIgcHJlc2VydmVBc3BlY3RSYXRpbz0ieE1pZFlNaWQiPiAgPGNpcmNsZSBjeD0iNTAiIGN5PSI1MCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjYzNjM2MzIiBzdHJva2Utd2lkdGg9IjEwIiByPSIzNSIgc3Ryb2tlLWRhc2hhcnJheT0iMTY0LjkzMzYxNDMxMzQ2NDE1IDU2Ljk3Nzg3MTQzNzgyMTM4Ij4gICAgPGFuaW1hdGVUcmFuc2Zvcm0gYXR0cmlidXRlTmFtZT0idHJhbnNmb3JtIiB0eXBlPSJyb3RhdGUiIHJlcGVhdENvdW50PSJpbmRlZmluaXRlIiBkdXI9IjFzIiB2YWx1ZXM9IjAgNTAgNTA7MzYwIDUwIDUwIiBrZXlUaW1lcz0iMDsxIj48L2FuaW1hdGVUcmFuc2Zvcm0+ICA8L2NpcmNsZT48L3N2Zz4=\\\");\\n  background-size: auto calc(min(50%, 400px));\\n}\",{\"id\":\"2c2e3239-31bf-4cce-a22b-7f079d3a3df2\"},{\"id\":\"0fad62a6-e546-49b1-a0f4-37ed6d009db6\"},{\"id\":\"61e210a8-bb3d-4737-80c0-9c67cecfb7c7\"}],\"width\":864,\"height\":576,\"min_width\":864,\"min_height\":576,\"margin\":[5,10],\"align\":\"start\",\"text\":\"&lt;img src=&quot;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA2AAAAJACAYAAADrSQUmAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAABYlAAAWJQFJUiTwAADdgUlEQVR4nOzdd1gU19cH8O8ubem9WSjSLGAXCyqosWCXiF3B3hB71JhETSJq7EasoNh7iYoNDGjEXlGxgAIqAoL03u77B+/Oj3V3YUGK6Pk8D4/LnTN37g4L7tk79wyPMcZACCGEEEIIIaTK8Wt6AIQQQgghhBDyvaAEjBBCSJl4PB7c3d1rehgVkpWVBU9PT5iYmEBOTg5mZmY1PaRK4+7uDh6PV9PDIIQQUg6UgBFCSA0JDg4Gj8cDj8eDj4+PxBgej4e+fftW88i+LatWrcLff/+NoUOHws/PDxs2bKjpIdVa58+fR4cOHaCqqgodHR24uroiMjKypodFCCG1inxND4AQQgiwZMkSjBw5EsrKyjU9lG9OQEAA7OzssHr16poeSq128uRJDB48GM2aNcPq1auRmpqKDRs2wMHBAffu3UOdOnVqeoiEEFIr0AwYIYTUsNatW+PDhw80M/P/CgsLkZWVVWn9xcXFQUdHp9L6+x7l5+djxowZqF+/Pv777z9MmzYNixYtwqVLlxAfH4+lS5fW9BAJIaTWoASMEEJq2JAhQ9CqVSusWrUKnz59KjNe2nosPz8/8Hg8BAcHc21Lly4Fj8dDWFgYZs2aBWNjY6iqqqJbt254+fIlgOKZjZYtW0JZWRlmZmbYsWOH1GMHBgaiXbt2UFFRgZGREWbOnInMzEyxuNTUVCxYsACWlpZQUlKCvr4+hg8fjjdv3kgcc2BgIP744w9YWFhAIBDg6NGjpZ6DgoICrFq1Co0bN4ZAIICuri4GDRqEJ0+eiPUdGRmJq1evcpd7ypIsHDlyBB07doS6ujpUVFTQtm1bHD9+XCxO+LOQ9bxERUVh9OjRMDQ0hJKSEiwsLPDzzz9LTDjT0tKwePFiNGrUiHuOHTt2xOHDh8ViU1NTMXXqVBgYGEAgEMDBwQG3b98WiWGMYcOGDWjatCnU1dWhoaEBGxsbjB8/Hvn5+aWej6tXr+LDhw+YMGEC1NTUuPbmzZvDyckJR44cKbMPQgghxSgBI4SQGsbj8bBq1SqkpqZi+fLlVXIMNzc3PH78GD///DPmzp2LW7duoWfPnti3bx+mT5+OgQMHYvXq1dDW1sbkyZNx/fp1sT4ePHiAgQMHon379lizZg06deqETZs2oX///igqKuLiUlNT0aFDB2zZsgV9+vTB33//DQ8PD/z7779o27YtoqOjxfqeN28eDh8+jIkTJ2Ljxo2wsbEp9fmMHDkSCxcuRL169bB69WpMmTIFQUFBaN++PR4+fAgA6Ny5M/bt2wc9PT00bNgQ+/btw759++Di4lJq37/88guGDRsGdXV1/PHHH1i5ciVUVFTg6uoKb2/vCp+X6Oho2Nvb4+jRoxg+fDjWr1+PVq1aYcWKFXB2dkZBQQEXm5KSgg4dOsDLywu2trb466+/8Msvv6BBgwY4d+6c2Bh69uyJ9+/f47fffsOiRYvw9OlT9O7dG+np6VzMn3/+idmzZ8PMzAyrVq3C6tWrMWjQINy8eRO5ubmlnpO7d+8CANq3by+2rV27dkhLS8OrV69K7YMQQsj/Y4QQQmpEUFAQA8BWr17NGGOse/fuTElJiUVFRXExAFifPn1E9gPA3NzcxPrbvXs3A8CCgoK4tiVLljAArG/fvqyoqIhr37hxIwPA1NTUWHR0NNf+8eNHpqSkxIYNGyZ2TADs1KlTIu2enp4MADt06JBIm0AgYI8ePRKJjYqKYurq6iJjF47Z2tqaZWZmSj5Rn7l8+TIDwIYMGSLynB4/fszk5ORYx44dReJNTU2Zo6OjTH3fv3+fAWCLFi0S2zZgwACmrq7O0tLSuLbynJcRI0YwAMzf318kdt68eQwA8/Hx4dqmTp3KALDt27eLjaOwsJB77ObmxgCwqVOnisQcPXqUAWDbtm3j2lq0aMEaNWpUxhmQzMPDgwFgYWFhYtu8vb0ZAHbp0qUK9U0IId8bmgEjhJCvxKpVq5CXl4dff/210vv29PQUKVfeqVMnAMCAAQNgYmLCtevr68PGxgbh4eFifdjY2GDgwIEibQsXLgQAnDp1CkDxZW4HDhxA586dUbduXSQmJnJfqqqqaNeuHS5fvizW99SpU6GioiLTcxEea/HixSLPqWnTpujbty+uX7+OhIQEmfr63IEDB8Dj8eDm5iYy9sTERPTv3x/p6em4efOmyD6ynJeioiKcOXMGLVq0QO/evUViFy1aBD6fLxJ7+PBhNGrUCBMnThQbI58v/l/37NmzRb7v2rUrAIj8HDU1NRETEyNxdrMswksklZSUxLYJBAKRGEIIIaWjBIwQQr4SLVq0wPDhw3HgwAGEhoZWat8NGjQQ+V5bWxsAYG5uLharra0tcS1ao0aNxNqMjY2hpaXFre1KSEjAp0+fcPnyZejr64t9BQQEID4+Xqwfa2trmZ9LZGQk+Hy+xPHY2tpyMRXx/PlzMMbQsGFDsbGPHz8eAMTGL+t5ycjIQJMmTcRidXR0YGxszMUmJiYiOTkZzZs3l/keX5//fHV1dQFA5Ofo5eUFgUCATp06oW7duhg5ciQOHjyIvLy8MvsXJseSLlXMyckRiSGEEFI6KkNPCCFfkT///BPHjx/HggULcOHChXLtW3IN0efk5OTK1c4YE2uTlgyUjBU+/uGHH7BgwQKp4/lced68SxpbZWGMgcfj4cKFC1LPzedJVHnOi6xjKK1fSWT5ObZv3x6vX7/GpUuXEBQUhKCgIBw8eBB//vknrl+/XmqlSGGJ+ZiYGLGEMyYmBgBQt25dmcdLCCHfM0rACCHkK2Jubo6pU6di48aNCAoKkhijo6ODpKQksfbPKwxWtrCwMLG22NhYpKamcjMw+vr60NLSQlpaGn744YcqGYeFhQUuXbqE58+fo2nTphLHKGlmTxZWVla4ePEiTExMJM5sSSLLeTEwMIC6ujqePXsmFpucnIzY2Fg0b94cQPE51NbWxqNHjyr0HEqjpqaGH3/8ET/++CMAYMuWLZg+fTp8fX0xf/58qfu1adMGAHDz5k2xn+utW7egoaFRrllMQgj5ntEliIQQ8pX55ZdfoKGhIXUGydraGjdv3hRZc5OcnIzdu3dX6bhevnyJ06dPi7StWrUKALg1UHw+HyNHjsSdO3cklm0HgI8fP37ROITHWrFihcgMz9OnT3HmzBl07NgR+vr6Fep79OjRAICff/4ZhYWFYtsljV3W89KvXz88fPgQFy9eFIlduXIlioqKMGjQIC52+PDhCAsLg6+vr9jxKjoDmJiYKNbWsmVLAJCY0Jfk6OgIY2Nj+Pj4ICMjg2t//PgxgoOD4erqCgUFhQqNixBCvjc0A0YIIV8ZPT09zJ8/X2oxDg8PD4waNQpdu3bF6NGjkZKSgp07d8LU1BRxcXFVNi47OzuMGjUKEydOhJWVFYKCgnD8+HE4Ojpi6NChXNzy5csREhKCIUOGYMiQIWjXrh0UFRURHR2N8+fPo1WrVvDz86vwOLp3744hQ4bg8OHDSE5ORt++fREXFwdvb28IBAJs2rSpwn23adMGy5Ytw5IlS9C8eXO4urqiTp06iI2Nxf3793H+/HmxNVOynhcvLy8EBARg4MCBmDZtGiwtLXHt2jUcOXIEnTt3hpubGxf7559/4t9//8WECRNw+fJldOzYEYwxPHz4EAUFBdi3b1+5n1ujRo3Qrl07tG3blntOO3bsgKKiIoYNG1bqvgoKCti4cSOGDh2KTp06YeLEiUhLS8P69euhr6+PZcuWlXs8hBDy3aqJ0ouEEELEy9CXlJmZyYyNjSWWoWeMsb/++ouZmJgwRUVF1rBhQ+br61tqGfrIyEiR/SMjIxkAtmTJErG+HR0dmampqUgb/r/0fUBAALO3t2cCgYAZGBgwDw8PkbLsJcf/+++/M1tbWyYQCJiamhpr2LAhmzBhArt16xYXJ2nMssjPz2crV65kDRs2ZIqKikxbW5sNGDCAhYaGisWWpwy90Llz51iPHj2YtrY2U1RUZPXq1WO9evViW7ZsEYkr73l58+YNGzVqFNPX12cKCgrM3NycLVq0SGIJ/uTkZDZ//nxmYWHBFBQUmI6ODuvYsSM7cuQIFyMsQy+JcGxCK1asYJ06dWL6+vrccxo8eDC7f/++zOfl7NmzrG3btkxZWZlpaWmxH3/8kUVERMi8PyGEEMZ4jFXhamZCCCHkGyYsWf8lM3qEEEK+L7QGjBBCCCGEEEKqCSVghBBCCCGEEFJNKAEjhBBCCCGEkGpCVRAJIYSQCqJl1IQQQsqLZsAIIYQQQgghpJpQAkZIFYuKigKPx8PSpUsrve+lS5eCx+MhKiqKa/Pz8wOPx0NwcHClHw8A3N3dwePxqqRvoHhGoX379hg5cqRIO4/Hg7u7e5UdtyxOTk4wMzOrseOX1/Hjx9GsWTMoKytX6euhLJJeo5VF0s9E0utz1qxZsLGxQX5+fqWPgRBCCCkvSsAIqaBr166hf//+MDMzg5KSEgwNDdG6dWvMnDkTb968qenhVavTp09XWoJ56NAh3L17t0oS1u/Fq1evMHz4cGhqamLz5s3Yt28fGjVqVGXHCw4OxtKlS5GSklJlx/gSCxcuxLt377B169aaHgrOnz+PDh06QFVVFTo6OnB1dUVkZKRM+zLGsH//fgwbNgyWlpZQUVGBiYkJ+vfvj9u3b0vcp6ioCOvXr0fDhg0hEAhQv359zJ07F5mZmRLjjx07xo1PXV0dnTp1wvnz5yXGxsfHY8qUKahfvz4UFRVhYmKCmTNnSn0d1Ma+165dCycnJxgbG0NJSQnGxsbo0qULTp06JbHvkrKysmBubg4ejwcPD48y47ds2QIejwcej4fExESRbS9fvsTIkSPRqFEjaGpqQkVFBQ0bNsScOXMQGxv7RX0DQEZGBry8vGBnZwd1dXXo6emhQ4cO8PPzk3iZbXnOd3leg8IxSvtavnx5mc+VkFqhJm9CRkhttWXLFgaANWjQgP32229s586dzMvLi40YMYKpq6uzY8eOcbFFRUUsOzub5efnV/o48vPzWXZ2NisqKuLaKnpjW1nl5eWx7OxskbbSbgZbXjY2NmzgwIFi7fjsprLVTdLNib9W27dvZwDKdYPdLyHtZs9lbftSkn4m0l6LY8eOZUZGRlXyeyirEydOMB6Px5o3b868vb2Zl5cXMzAwYMbGxiwmJqbM/bOzsxkA1rx5c7Z48WLm4+PD/vjjD1a3bl3G4/HYvn37xPbx9PRkANigQYPYjh072OzZs5m8vDzr0qULKywsFIlduXIlA8BatGjB1qxZw9auXctatGjBeDwe279/v0hsfHw8MzU1ZQoKCszDw4Nt27aNeXh4MAUFBda8eXOxG0vX1r6HDBnC3Nzc2KpVq5ivry9bvXo1s7e3ZwDY77//XurPa+7cuUxNTY0BYNOnTy81NiYmhmloaHDxCQkJItsDAwNZly5d2KJFi5i3tzfbvn078/DwYKqqqszY2JjFx8dXuO/CwkLWsWNHxufz2dixY9n27dvZ+vXruef5008/icSX53wzVr7X4L59+yR+WVhYMADs8ePHpZ5HQmoLSsAIKaf8/HympaXFTExMWGpqqtj2rKws9unTpxoYWbGqSMCKiopYenq61O2VlYAFBgYyAOzkyZNi2ygBk92yZcuqLOmRpDYkYP/++y8DwI4fP17p45BFXl4eq1OnDjMxMRH5XXr48CHj8/ls4sSJZfaRn5/PgoODxdrj4uKYrq4uMzAwEHlD+/TpU8bj8ZiLi4tI/KZNmxgAduDAAZE+FBUVma2tLcvLyxMZt62tLdPW1hb5ezdz5kwGgB08eFCk74MHDzIA7I8//qj1fUuTn5/PmjZtytTU1FhBQYHEmPv37zM5OTm2du1amRKwgQMHsubNm7NRo0ZJTJKkOXr0KAPAVq1aVeG+b9y4wQCwWbNmibTn5uYyc3NzpqmpybWV93yX5zUozbt37xifz2etW7cuM5aQ2oIuQSSknBITE5GSkoI2bdpAQ0NDbLuysjJ0dHS47yWtASvZdvToUTRv3hzKysqwtLTE7t27AQBv377F4MGDoaOjA3V1dYwaNQrp6ekix5J1fU16ejp++eUXtG3bFnp6elBSUoKlpSUWLlyIrKwskdjg4GDweDz4+fnB29sbjRs3hkAgwJo1awCIr7FxcnLCnj17AIhePuLn5wdPT0/weDyEh4eLjSk2Nhby8vIYP34813bs2DHIycmhR48eUp/LzZs34ejoCFVVVejp6WHChAnIyMgQiwsNDcWgQYOgq6sLgUCAxo0b46+//kJhYaFYbFxcHDw9PdGgQQMoKSnBwMAA3bt3R0BAQKnn9dOnT2jfvj00NTVx5coVrj0wMBA9evSAlpYWBAIBmjZtim3btons26xZM5iYmKCoqEis36NHj4LH42Hfvn2lHl8SHo+HJUuWAAB3+VPJdVJRUVEYPXo0DA0NoaSkBAsLC/z8889irwNZY93d3bFs2TKR40la85iZmQlPT08YGRlBWVkZbdu2FTlnQkeOHEH//v1hYmICJSUl6OnpYeDAgQgNDS33uSipc+fOUFVVxbFjx76on4q6evUqPnz4gAkTJkBNTY1rb968OZycnHDkyJEy16jJy8vD0dFRrN3Q0BCOjo74+PEjPn78yLUfOnQIjDHMmjVLJH7ixIlQUVHB/v37ubYbN24gLy8PI0eOhIKCAteuoKCAESNGIDk5Gf/88w/XHhQUBGVlZQwbNkyk76FDh0IgEHB/x2pz39LIy8ujbt26yMzMlPgzKywsxMSJE9GrVy+4uLiU2d+pU6dw5swZbN++HXJycmXGl2RqagoASE5OrnDfaWlpAIA6deqItCsqKkJPTw+qqqpcW3nPd3leg9Ls3r0bRUVFmDBhQpmxhNQWVIaekHIyNDSEmpoarl27hpcvX8LGxqbCfZ07dw7btm3DtGnToKOjA19fX4wbNw6Kior4+eef0bVrV3h5eeHu3bvYtWsXBAIBfHx8yn2cmJgY+Pj44Mcff8SIESMgLy+Pq1ev4q+//sLDhw9x6dIlsX02bNiAT58+YeLEiTAyMkL9+vUl9r148WIUFRXhv//+E0kYOnTogDZt2uDvv//Grl27sGLFCpH99uzZg8LCQpEE7OrVq2jSpInIf/glPXr0CH379sXYsWMxYsQIBAcHw9fXF3w+Hzt27ODi7t27B0dHRygoKGD69OkwMjLC2bNnsWDBAjx+/BgHDhzgYqOiouDg4ID4+HiMGTMGrVu3RmZmJm7duoXAwEB0795d4liioqLQs2dPpKen4+rVq2jevDkAYMeOHZgyZQratWuHxYsXQ1VVFQEBAZg6dSpev36N1atXAyh+AzJjxgwEBASgZ8+eIn3v2rULmpqaGDx4sMRjl2bfvn04efIkTp06hfXr10NPT497wx8dHQ17e3ukpqZi6tSpsLa2RnBwMFasWIGQkBBcuXIF8vLy5YqdPHky0tLSRI4HAE2bNhUZ15gxYyAnJ4cFCxYgPT0d27dvR69evXDhwgX88MMPXNzmzZuho6ODSZMmwcjICK9fv8aOHTvg4OCABw8ewMrKqtznBADk5OTQpk0bXL16Vab4jIwM5OTkyBQrEAhEkipJ7t69CwBo37692LZ27drh33//xatXr9CkSROZjvm59+/fQ1FREVpaWiLH5PP5sLe3Fxtv8+bNuTEBQG5uLgBARUVFrG9h261btzB69GguXiAQiBU84fP5UFZWxps3b5CYmAg9Pb1a23dJSUlJKCwsRGJiIo4dO4aLFy+iS5cuEAgEYsddv349Xrx4gRMnToht+1xaWho8PDwwefJk2NvbY8uWLaXG5+TkcK/NsLAwLFiwAADQu3fvCvdtb28PLS0t/PXXXzAzM0Pbtm2RnZ0NPz8/3L9/X+TDo/Ke7/K8BiVhjGH37t1QUVHB8OHDS40lpFap4Rk4QmqlNWvWMABMTk6OtWnThnl6erL9+/ez2NhYsdjIyEgGgC1ZskSsTUVFhUVFRXHtHz9+ZEpKSozH47G1a9eK9DNo0CCmoKAgcvmSpMu7JF2CmJubK3K5iNAvv/zCALDbt29zbUFBQQwA09bWlriuQNIlXqVdgti+fXtmbGwstvbGysqKNWrUiPu+oKCA8fl8NmjQIIn9AGA8Ho/dvHlTpL13795MXl5e5Lx06NCBycnJiawXKCoqYq6urgwACwwM5NqdnZ0ZAHbx4kWxY5a8nKvk5W4PHz5kRkZGzMbGRuTcf/jwgSkpKbHhw4eL9eXp6cn4fD6LiIhgjDGWkpLCVFRUmKurq0jc27dvGZ/PZ1OnTpV4HmQh7bK/ESNGMADM399fpH3evHkMAPPx8alQrCyXINrb27Pc3Fyu/d27d0xVVZU1bNhQJD4jI0Osj7CwMKaoqCh2TspzCSJjjI0fP54BYImJiRK3S+pHli9ZLo318PBgAFhYWJjYNm9vbwaAXbp0qcx+JPH392cA2OjRo0XabW1tmYGBgcR9hL8Lwp9JaGgoA8AGDBggFjtgwAAGgPXr149rc3FxYQDYw4cPRWIfPnzInRfhGsTa2ndJurq63HZ5eXk2ePBg9vHjR7G4N2/eMBUVFbZy5UrG2P/+1ku7BHHKlCnMyMiIpaSkMMb+97qTdgni33//LfLaMzMzk7juqrx9X7t2jVlbW4v0ra6uzk6dOiUSV97zXZ7XoCTCy9Ld3d2lxhBSG9EliIRUwNy5c3HmzBn06NEDYWFh2LRpE0aNGoV69eph/PjxEi/nkmTgwIHcJSQAoK+vDxsbG/D5fEyfPl0ktlOnTsjPz69QOW9FRUXucpGCggIkJycjMTGRm3mQVEFtzJgxMDAwKPexPjdp0iTExsbiwoULXNu1a9cQHh4uMvv16dMnFBUViVy++bn27dujXbt2Im1du3ZFQUEBd14+fvyIGzduoH///iKzMDweDz///DMAcBXMkpKScPHiRfTq1UtsFgoo/lT8c4GBgejcuTPMzMwQEhIicnnf8ePHkZubi/HjxyMxMVHkq1+/figqKuIuu9PU1ISrqyv++ecfkapkwsttSp6bylBUVIQzZ86gRYsWYp+WL1q0CHw+nzsv5YmV1ezZs6GoqMh9X69ePYwcORIvXrzA8+fPuXbh7CdjDGlpaUhMTOR+L6RV+pOVrq4uAIhcpifNTz/9hICAAJm+fvrppzL7E/5NUFJSEtsmnEWR9e9GSeHh4Rg9ejTq1q2LtWvXih1T0vEkHdPOzg7du3fHP//8g59++gnPnz/H8+fPsWDBAu53t+T4Zs2aBT6fjyFDhuD8+fN4+/YtLly4gKFDh3J/a2p73yWdPHkSly5dwq5du9C9e3dkZ2dzl+6VNHXqVJibm2POnDkSz3tJN27cwPbt27Fu3TpoamqWGQ8U/58REBCAU6dO4bfffoOWlhYSEhK+uG81NTXY2tpi3rx5OHnyJHx8fGBpaYkRI0aIXIpd3vNdntegJMIrPir77yEhNa6mM0BCaruCggIWGhrK1q9fz8zMzBgANmnSJG57aTNgv/zyi1h/jo6OrF69emLtwpmtkovwZZ0BY6z4U3Y7OzvG5/PFPsFftmwZFyecAfP29pb4fMs7A5aVlcU0NTVFPjEdM2YMU1RUFPkE+ePHjwwAGzdunMR+ALBRo0aJtX9+Xm7duiX13Obm5jI+n8+cnZ0ZY4zdvn2bAWCLFi2SeMySHB0dmUAgYAoKCqxZs2Zi1dIYY2zq1KllzpaUrJwWEhLCALD169czxopn6czMzFjz5s3LHE9pJL0u4uLipJ5DxhirW7cuNyNZnlhpx/t826NHj8S2bdiwgQFg586d49oePHjA+vTpw1RVVcXOnbm5ucj+5Z0Bmz9/PgPAnj9/LnF7VaqKGbA3b96w+vXrM11dXRYaGiq2vbyzD0lJSczFxYXxeDyRGZadO3cyAGKz00ePHmVGRkZcrJycHJs8eTIbNGgQA0Qr1tXWvqUZNmwYMzIyYklJSVzbvn37GI/HY//99x/XJm0GLDc3lzVu3Jh1795dpL2sGbDPPX78mCkqKjIvL68K9x0aGsoEAgHbunWrSHtmZiYzNzdnpqamIsVGynO+v2QGLCkpiSkpKYnNkhPyLaA1YIR8ITk5OdjZ2cHOzg4jR46ElZUV9uzZgy1btpS5oFra9tL2YxLuyVKWdevWYe7cuejRowc8PT1Rp04dKCoqIiYmBu7u7hILQUi6xr8ilJWVMWrUKGzfvh1xcXFQUVHB8ePH0b9/f+jr63Nxurq64PP5SEpKktqXLOelPOdHGCvrjaV1dHTQsmVL+Pv748CBA5g4caLE/vbu3QtjY2OJfTRo0IB73KFDB9ja2sLX1xezZs3ClStXEBUVhc2bN8v8HGRVkfNSmSSd48+P8/btW3Tu3BkaGhr49ddfYWNjA1VVVfB4PMyaNUtisZXyEL62Sr7upElNTUV2drZM/SorK5c5yyAscBATEyN2T7aYmBgAQN26dWU6HlC8BrFLly7IyMjAlStXYGdnJ/GYYWFhyM3NFZuFiImJgZ6ensispLa2Nk6cOIH4+Hi8evUKampqaNasGS5evAgAaNiwoUgfrq6ucHFxwZMnT5Ceng4bGxsYGBjA3t4e8vLysLS0rPV9S+Pm5obDhw/j5MmTGD9+PHJzczFnzhz07t0bRkZGiIiI4M4zUPx6ioiIgJ6eHrS0tODt7Y0XL15g7dq1XCwArtBSZGQk0tLSRP5eSNK0aVO0aNECW7ZswaJFiwCg3H2vX78eOTk5cHV1FelbRUUFffr0webNmxEVFQULC4tyn+/yvgZL2r9/P3dFASHfGkrACKlE+vr6sLCwwIMHD5CYmAhDQ8OaHhKA4sIMZmZmuHDhgshldcL/ML9UWQnMpEmT4O3tjb1790JTUxNZWVli/6ny+Xw0atRIYsXE8hC+qXj27JnYthcvXqCoqIiLsbKyAo/Hw8OHD2XqW0FBASdPnsTQoUMxefJk5OfnY9q0adx2YYEIPT09kcISpZk4cSJmzpyJO3fuwNfXFwKBACNHjpRp3/IwMDCAurq6xPOSnJyM2NhYrpBIeWIB2RLYsLAwscIcwksPhT+PU6dOISMjA2fOnEGXLl1EYj99+iT1UiZZRUREwMjIiLsUsTQzZ87kqnuWxc3NDX5+fqXGtGnTBkBxFc/PXxu3bt2ChoYGrK2tZTpedHQ0unTpgtTUVAQGBqJFixZSj3n58mXcuXMHnTp14tpzcnLw6NEjdO7cWeJ+hoaGIn+7hDfYlVToQU5OTuS1EBcXh4cPH8LR0VHihzi1te/PCZNzYVKfnZ2NhIQE+Pv7w9/fXyx+//792L9/P1avXo158+YhOjoaRUVFcHZ2lti/vb09VFVVZfrQITs7W+SDq/L2LUwSJVWILSgoEPm3JFnOd0VfgwDg6+sLBQUFjBkzRmoMIbUVJWCElFNWVhbu3r0rsRx0eHg4wsLCoKenJ9On7NVFTk4OPB5PZMahoKAAK1eurJT+hRXgkpKSJK7hatq0Kezt7bFr1y5oaGjAxMREYql5JycnbN26FWlpaRJL/MvCwMAAHTp0wNmzZ/H06VPY2toCKJ5tEVZiHDRoEIDiGS1nZ2ecP38egYGBYm+MGWNiyYWCggKOHj2KESNGYPr06cjPz8fMmTMBAEOGDMHPP/+MJUuWwMnJCcrKyiL7pqamQiAQiCQSo0ePxoIFC7B69WqcPXsWgwcPFqlkV1n4fD769euHgwcPcuvehFauXImioiLuvJQnFhD9+ZdcE1fS+vXr4eLiwn3a/f79exw8eBA2NjbcjJBwhvPzmbGdO3ciLi5OZL1keRUWFuLevXvo27evTPE//fQTRo0aJVPs5+W7JXF0dISxsTF8fHwwe/Zs7pw9fvwYwcHBGDt2rEhZ79TUVMTGxkJPT0+kIl90dDScnJyQnJyMgIAAtGrVSuoxhw4dCi8vL2zYsEHkze/OnTuRlZUlU6J/7949+Pj4wNHRER07diw1tqioCJ6enigsLMTixYtrfd+ZmZlgjIlVuCwsLIS3tzcAcGtSpd3iICEhAdOmTUOvXr0wfvx47kOIsWPHShyXt7c3goODsWvXLmhra3PtcXFxMDIyEosPCgrC06dP4eTkxLWVt+/GjRvj8uXL8PPzE1nPmJKSgn/++Qfa2trc7Jc00s53RV+D9+7dw+PHj+Hi4lIpa5EJ+dpQAkZIOWVlZcHJyQm2trbo1asXrKyswBjDixcvsHfvXuTk5MDb21tiAYeaMnjwYCxatAjOzs5wcXFBWloaDh48KPKG70u0a9cOmzdvxrRp09CnTx8oKCigbdu2MDc352ImTZrE3cdlyZIlEs+Pq6srvL29cfHiRQwZMqTC49m4cSMcHR3RqVMnrgz9uXPncOnSJYwYMQLdunXjYjdv3owOHTrA2dkZbm5uaNWqFbKzs3H79m2YmZlh1apVYv3Ly8vj0KFDUFBQwKxZs1BQUIC5c+eiXr162Lp1KyZMmIBGjRph9OjRMDU1RUJCAp48eYLTp08jLCxMJEnR1tbG4MGDufvhSLvXjZOTE65evYrIyEipSU5ZvLy8EBAQgIEDB2LatGmwtLTEtWvXcOTIEXTu3Blubm4VihW+CV2wYAFGjhwJgUAAW1tbLvkFihP+Tp06Yfjw4UhPT8e2bduQnZ2NTZs2cTHOzs5QUVHB6NGj4eHhAW1tbYSEhOD8+fOwsLCQ+Cm8rIKDg5GZmSl2mZU0jRs3RuPGjSt8vM8pKChg48aNGDp0KDp16oSJEyciLS0N69evh76+PncvNaFTp05h7NixWLJkCXdPtfT0dHTp0gVRUVGYMWMGXr58iZcvX4rs1717d25Wws7ODtOnT8fmzZvh4uKC3r174/nz59i0aRMcHR0xYsQIkX1//fVXhIeHw97eHpqamnjw4AF27dqFunXrit2TLiMjA/b29hg0aBDMzc2RmpqKQ4cO4f79+1i+fLnYDGZt7Ds8PByOjo4YPHgwbGxsoKOjg5iYGBw6dAgvX76Em5sbl1QoKChIvG2EsDiQhYWFyPZmzZqhWbNmYvHnzp0DAPTr108k8Z46dSpiY2PRtWtXmJqaIicnB/fv38fhw4ehrq4uUoClvH3PmjULe/fuxcKFC/HkyRM4ODggKSkJO3fuRGxsLLy9vbnbU5T3fJf3NSjk6+sLQPrfQ0JqvZpYeEZIbZafn8927drFhg0bxqytrZm6ujpTUFBgderUYYMGDWL//vuvSHxpRThKtglJKizAmOTiGrIW4SgoKGBeXl7MwsKCKSoqMhMTEzZ//nwWFhYmNg5hEY7du3dLfP6SihwUFhayuXPnsrp163JFPj7fPyMjg2loaDA+ny9Sev9zjRs3Zn379hVrh5Ry39KKjjx69IgNGDCAaWtrM0VFRdawYUO2atUqkcXkQu/fv2eTJ09m9evXZwoKCszAwIB1795dpFy9pJ9LYWEhdz5KLoK/fv06GzhwINPX12cKCgrM2NiYOTk5sTVr1rDs7Gyx41+7do0BYJaWlqyoqEjieWnZsiVTUVFhycnJEreXVFpRjDdv3rBRo0ZxYzM3N2eLFi2SWFSkPLGrVq1i5ubmTF5eXuQ1JRzL06dPmYeHBzM0NGRKSkqsTZs27PLly2L9XL16lTk4ODA1NTWmqanJevfuzZ48eSLx/JenCIe7uzszMjISux1CdTt79ixr27YtU1ZWZlpaWuzHH3/kbk1QkvB1LenvRmlfn/8eFBQUsDVr1jBra2umqKjI6tSpw2bPni1y2wahEydOsHbt2jFtbW2mpKTErKys2E8//STxNZebm8uGDh3KzMzMmJKSEtPW1mY9evSQeDuH2tp3QkICmzZtGmvatCnT1tZm8vLyTFdXl/3www9s//79Un9XSyqrDP3npBXKOHLkCOvduzerV68eU1JSYgKBgNnY2DAPDw8WHR39RX0zxlhERAQbM2YMq1u3LpOXl2fq6uqsU6dO7MSJE2Kx5TnfjJXvNcjY/wo31atXT+RWIIR8S3iMVcFqa0II+Uxubi6MjY3Rpk0biTd+Fjp8+DBGjRqFZ8+efdFNrmuTO3fuoG3btvDy8uIW0peUnJwMfX19LF68WGymhJQtLi4ODRo0wMqVK+Hp6VnTwyGEEPKd+3qukSKEfNMOHDiA5ORkTJ48udS4YcOGoU2bNt9VorF582YoKChg7NixErcHBgZCX18f8+fPr+aRfRtWrlyJevXqYerUqTU9FEIIIQQ0A0YIqVJnz55FdHQ0li5dCkNDQ4SGhpZZnv97kJmZibNnz+LZs2dYvnw5Jk2ahG3bttX0sAghhBBSxSgBI4RUKTMzM3z48AGtWrWCj48PmjRpUtND+ipERUXB3NwcampqcHZ2ho+PT4UrPxJCCCGk9qAEjBBCCCGEEEKqCa0BI4QQQgghhJBqQgkYIYQQQgghhFQTSsAIqSZRUVHg8XjcTVUr09KlS8Hj8bibfgKAn58feDwegoODK/14AODu7g4ej1clfQMAYwzt27fHyJEjRdp5PB7c3d2r7LhlcXJyqvCNkEnlio2NhYqKCvbs2VPTQyGEEEJkRgkYIV/o2rVr6N+/P8zMzKCkpARDQ0O0bt0aM2fOxJs3b2p6eNXq9OnTlZZgHjp0CHfv3q2ShJV8G4yNjTFlyhQsXrwYWVlZNT0cqc6fP48OHTpAVVUVOjo6cHV1RWRkpEz7Msawf/9+DBs2DJaWllBRUYGJiQn69++P27dvS9yHx+NJ/FJTU5MYf+zYMW586urq6NSpE86fPy8xNj4+HlOmTEH9+vWhqKgIExMTzJw5EykpKd9U3y9fvsTAgQOhra0NVVVVdOrUCf/++6/E2JKysrJgbm4OHo8HDw8PqXH+/v744YcfoK2tDRUVFVhbW4vFv3z5EiNHjkSjRo2gqakJFRUVNGzYEHPmzEFsbKxIbHW8TmQdNwBkZGTAy8sLdnZ2UFdXh56eHjp06AA/Pz+UVXpgy5Yt3FgSExNLjSWk1qqZ+z8T8m3YsmULA8AaNGjAfvvtN7Zz507m5eXFRowYwdTV1dmxY8e42KKiIpadnc3y8/MrfRz5+fksOzubFRUVcW27d+9mAFhQUFClH48xxvLy8lh2drZIm5ubG6usPys2NjZs4MCBYu0AmJubW6UcoyIcHR2ZqalpjR2fiIqMjGQ8Ho9t3ry5poci0YkTJxiPx2PNmzdn3t7ezMvLixkYGDBjY2MWExNT5v7Z2dkMAGvevDlbvHgx8/HxYX/88QerW7cu4/F4bN++fWL7AGCdOnVi+/btE/k6fPiwWOzKlSsZANaiRQu2Zs0atnbtWtaiRQvG4/HY/v37RWLj4+OZqakpU1BQYB4eHmzbtm3Mw8ODKSgosObNm7PMzMxvou+IiAimo6PDDAwMmJeXF/P29mbNmzdn8vLyLCAgoNSf19y5c5mamhoDwKZPny4xZunSpQwA69mzJ9u4cSPbuXMn+/XXX9mAAQNE4gIDA1mXLl3YokWLmLe3N9u+fTvz8PBgqqqqzNjYmMXHx3OxVf06Kc+4CwsLWceOHRmfz2djx45l27dvZ+vXr2f29vYMAPvpp5+knr+YmBimoaHBncOEhASpsYTUZpSAEVJB+fn5TEtLi5mYmLDU1FSx7VlZWezTp081MLJiVZGAFRUVsfT0dKnbKysBCwwMZADYyZMnxbZRAkY+17lzZ2ZnZ1fTwxCTl5fH6tSpw0xMTER+bx4+fMj4fD6bOHFimX3k5+ez4OBgsfa4uDimq6vLDAwMWGFhocg2WX9H4uLimKKiIrO1tWV5eXki47a1tWXa2toif9tmzpzJALCDBw+K9HPw4EEGgP3xxx+1vm/GGHN1dWV8Pp89fPiQa0tPT2cmJibM2tpa5IOuku7fv8/k5OTY2rVrpSZgAQEBDAD7/fffJfYhi6NHjzIAbNWqVVxbVb5OyjvuGzduMABs1qxZIu25ubnM3NycaWpqSt134MCBrHnz5mzUqFGUgJFvGl2CSEgFJSYmIiUlBW3atJF4/yZlZWXo6Ohw30taA1ay7ejRo2jevDmUlZVhaWmJ3bt3AwDevn2LwYMHQ0dHB+rq6hg1ahTS09NFjiVpDZgk6enp+OWXX9C2bVvo6elBSUkJlpaWWLhwodglXMHBweDxePDz84O3tzcaN24MgUCANWvWABBfA+bk5MStxSl5OYufnx88PT3B4/EQHh4uNqbY2FjIy8tj/PjxXNuxY8cgJyeHHj16SH0uN2/ehKOjI1RVVaGnp4cJEyYgIyNDLC40NBSDBg2Crq4uBAIBGjdujL/++guFhYVisXFxcfD09ESDBg2gpKQEAwMDdO/eHQEBAaWe10+fPqF9+/bQ1NTElStXuPbAwED06NEDWlpaEAgEaNq0qdjNlps1awYTExMUFRWJ9Xv06FHweDzs27ev1OMLf/5hYWGYNWsWjI2Noaqqim7duuHly5cAgJMnT6Jly5ZQVlaGmZkZduzYIdbPkSNH0L9/f5iYmEBJSQl6enoYOHAgQkNDxWLNzMzg5OSEFy9eoE+fPlBXV4empiYGDx6MuLg4sfjQ0FD06NEDqqqq0NXVhZubGxITEyWu6SsoKMCqVau415yuri4GDRqEJ0+eSHz+zs7OePLkCV68eFHqeapuV69exYcPHzBhwgSRy7qaN28OJycnHDlyBPn5+aX2IS8vD0dHR7F2Q0NDODo64uPHj/j48aPEffPy8iT+TgjduHEDeXl5GDlyJBQUFLh2BQUFjBgxAsnJyfjnn3+49qCgICgrK2PYsGEi/QwdOhQCgYD7m1Wb+87MzMSZM2fg5OSE5s2bc+1qamqYMGECXr16hbt374qdy8LCQkycOBG9evWCi4uL2HYhLy8vGBgYYNGiRQCKL9WT9LtfGlNTUwBAcnIy11aVr5PyjjstLQ0AUKdOHZF2RUVF6OnpQVVVVeJ+p06dwpkzZ7B9+3bIycmVOh5CajtKwAipIENDQ6ipqeHatWvcm9yKOnfuHDw9PeHi4oLVq1dDXV0d48aNw4EDB9CpUyeoq6vDy8sLQ4YMwYEDBzB79uwKHScmJgY+Pj5o3bo1fv31V6xbtw4tW7bEX3/9hUGDBkncZ8OGDVi5ciWGDRuGv//+G23btpUYt3jxYnTq1AkAsG/fPu6rc+fOmDx5MgBg165dYvvt2bMHhYWFIgnY1atX0aRJE6n/UT969Ah9+/ZFmzZtsG7dOnTv3h2+vr6YM2eOSNy9e/fQvn17BAUFYcqUKVi9ejXq1auHBQsWYMyYMSKxUVFRaNWqFbZs2QInJyesX78e8+fPh4aGBgIDA6Wc0eL9OnTogOjoaFy9ehXdunUDAOzYsQM9evRARkYGFi9ejHXr1sHCwgJTp07F/Pnzuf0nTpyId+/eSUzydu3axSU1snBzc8Pjx4/x888/Y+7cubh16xZ69uyJffv2Yfr06Rg4cCBWr14NbW1tTJ48GdevXxfZf/PmzeDxeJg0aRK8vb0xceJE/Pfff3BwcJCYPMfExMDJyQkmJiZYvXo1RowYgZMnT4qd2/DwcHTq1Ak3b96Ep6cnli1bhoSEBDg7O0t8HiNHjsTChQtRr149rF69GlOmTEFQUBDat2+Phw8fisW3b98eAGQqOFNUVITExESZv8r75rgk4Rt14fhKateuHdLS0vDq1asK9//+/XsoKipCS0tLbNvx48ehoqICdXV1GBgYYMaMGUhNTRWJyc3NBQCoqKiI7S9su3Xrlki8QCAQK77D5/OhrKyMN2/ecGt2amvfoaGhyM3NlfozAyAxAVu/fj1evHiBzZs3i20TyszMxLVr19C2bVv4+vqibt26UFdXh5qaGoYNG4b4+HiJ++Xk5CAxMRHv37/H5cuXub+nvXv3lnqskr70dVLecdvb20NLSwt//fUXjh07hrdv3+Lly5dYtGgR7t+/L3Fdb1paGjw8PDB58mTY29vL9LwIqdVqegqOkNpszZo1DACTk5Njbdq0YZ6enmz//v0sNjZWLDYyMpIBYEuWLBFrU1FRYVFRUVz7x48fmZKSEuPxeGzt2rUi/QwaNIgpKCiIXNK0ZMkSBoBFRkZybZIuQczNzRW5ZEfol19+YQDY7du3ubagoCAGgGlra4usNRCSdLlhaZcgtm/fnhkbG4utgbOysmKNGjXivi8oKGB8Pp8NGjRIYj8AGI/HYzdv3hRp7927N5OXlxc5Lx06dGBycnLs8ePHXFtRURFzdXVlAFhgYCDX7uzszACwixcvih2z5KU7JS9BfPjwITMyMmI2NjYi5/7Dhw9MSUmJDR8+XKwvT09PxufzWUREBGOMsZSUFKaiosJcXV1F4t6+fcv4fD6bOnWqxPNQkvDn37dvX5HLozZu3MgAMDU1NRYdHc21C19fw4YNE+knIyNDrO+wsDCmqKgoNg5TU1MGgB05ckSkfdq0aQwAe/78OdcmPN/Xr18XiR0yZIjYZVCXL19mANiQIUNEnsvjx4+ZnJwc69ixo9gY3717xwAwDw8PSadHhPB3Ttavkj/X8vLw8GAAWFhYmNg2b29vBoBdunSpQn37+/szAGz06NFi2+zt7dnq1avZqVOn2J49e9jQoUMZAGZnZyfy+xEaGsoAiK3hYYyxAQMGMACsX79+XJuLiwsDIHJpHmPFvwfC83X//v1a3ffx48cZALZlyxaxvp89e8YAsEWLFom0v3nzhqmoqLCVK1cyxv73Gvv8EsRHjx4xAExfX58pKSmxJUuWsJMnT7LZs2czPp/PGjVqJLYejTHG/v77b5HXpJmZmdg6N2kq43VSkXFfu3aNWVtbi4xbXV2dnTp1SuI4p0yZwoyMjFhKSgpj7H//l9AliORbRQkYIV/ozJkzzNnZmamqqnL/0cjJybFx48aJ/KdUWgI2YsQIsX6bNm3K5OTkWE5Ojkj7unXrGAD25MkTrk3WBKyk/Px8lpSUxBISElhwcDADwDZt2sRtFyZgM2fOlLh/eRMw4XjOnDnDtV29epUBYGvWrOHa4uPjGQA2fvx4if0AYB06dBBrFybDwvMi7EdSIid84yV8g/Tp0yfG4/FYr169JB6zJGECFhAQwNTV1Vm7du1YYmKiSMymTZu4BC8hIUHkS7iWYvv27Vy8m5sbU1RUFHmzsWzZMgaA3bt3r8wxCX/+ly9fFml/8OABA8BGjhwptk/Tpk1Zq1atJPZXVFTEUlNTuTHb2dmxli1bisSYmpqyOnXqiO0rfAN79uxZxlhxQq2qqsrs7e3FYm/fvi2WgE2dOpUBEEmahYRvrj9+/CjSnpWVxSVtZcnOzmYBAQEyf31eaKY8xo0bxwCw169fi23z9fVlAKS+IS3Nq1evmI6ODqtbt67YuZBm+fLlDAD7888/Rdq7d+/OALD58+ezsLAwFhYWxn766SemqKjIALBu3bpxsdeuXWN8Pp9ZWVkxf39/Fh0dzc6fP8+sra2ZgoICA8D++++/Wt333r17GQDm6+srdg5fv34t8W9iz549WZMmTbgPt6QlYP/99x/3f8TOnTtFtgl/hyUlfu/evWMBAQHs1KlT7LfffmPNmzdn69evF4v7XGW9Tioy7gcPHjAXFxc2b948dvLkSebj48NatGjBlJWVxf5OhYSEMB6PJ7JGjxIw8q2jBIyQSlJQUMBCQ0PZ+vXrmZmZGQPAJk2axG0vLQH75ZdfxPpzdHRk9erVE2sXJjIlF1yXJwHz9vZmdnZ2jM/ni33av2zZMi5OmIB5e3tLfL7lTcCysrKYpqamyKfWY8aMYYqKiiJvDj5+/MgAsHHjxknsBwAbNWqUWPvn5+XWrVtSz21ubi7j8/nM2dmZMfa/RODzT7YlcXR0ZAKBgCkoKLBmzZpJ/MRamESU9lVyMXtISAgDwL2pKioqYmZmZqx58+Zljoex//38hbNqQmW9vszMzETaHjx4wPr06SPyYYLwy9zcXCTW1NRU4myU8HXj5+fHGGMsNjZW6ifwKSkpYglYr169GJ/PlzhTu3jxYgaIztQyxlhmZiYDwIYOHSq2T1XLyspisbGxIl9ZWVmMsaqZAXvz5g2rX78+09XVZaGhoTLvl5eXxxQVFVn79u1F2pOSkpiLiwvj8XgiMyw7d+6U+AHG0aNHmZGRkciHTZMnT2aDBg0SS5xrY9/lnQHbt28f4/F4IsmhtATs3r17DADj8/liH6y9efNG5g8RHj9+zBQVFZmXl5fUmMp8nZR33KGhoUwgELCtW7eKxGZmZjJzc3NmamrKCgoKGGPFf4sbN27MunfvLhJLCRj51smDEFIp5OTkYGdnBzs7O4wcORJWVlbYs2cPtmzZUuaCYmnbS9uPlXEvFUnWrVuHuXPnokePHvD09ESdOnWgqKiImJgYuLu7S1zvImmdRUUoKytj1KhR2L59O+Li4qCiooLjx4+jf//+0NfX5+J0dXXB5/ORlJQktS9Zzkt5zo8wVtYbS+vo6KBly5bw9/fHgQMHMHHiRIn97d27F8bGxhL7aNCgAfe4Q4cOsLW1ha+vL2bNmoUrV64gKiqq1PUkkpT3dVTyHL19+xadO3eGhoYGfv31V9jY2EBVVRU8Hg+zZs2SuEj/S38Oks53RV7XwtdKydeRNIWFhUhISJC5b319/VKf55EjRzB27FiRtt27d8Pd3Z0rQhATE4NGjRqJxMTExAAA6tatK/NYoqKi0KVLF2RkZODKlSuws7OTeV8FBQXUqVNH7L5K2traOHHiBOLj4/Hq1SuoqamhWbNmuHjxIgCgYcOGIvGurq5wcXHBkydPkJ6eDhsbGxgYGMDe3h7y8vKwtLSs1X2X/Jl97vOfWW5uLubMmYPevXvDyMgIERERInGpqamIiIiAnp4etLS0UK9ePW7sSkpKIn0L/06ULKwhTdOmTdGiRQts2bKFK4pRUmW/Tso77vXr1yMnJweurq4isSoqKujTpw82b96MqKgoWFhYwNvbGy9evMDatWu58weAKzQVGRmJtLQ0kb+XhHwLKAEjpAro6+vDwsICDx48QGJiIgwNDWt6SACKi2OYmZnhwoUL4PP/V4NH+KblS5WVwAiLO+zduxeamprIysoSKb4BFC+Mb9SokcSiD+Uh/A/72bNnYttevHiBoqIiLsbKygo8Hk9igQdJFBQUcPLkSQwdOhSTJ09Gfn4+pk2bxm23srICAOjp6eGHH36Qqc+JEydi5syZuHPnDnx9fSEQCDBy5EiZ9q0Mp06dQkZGBs6cOYMuXbqIbPv06ZPYGy9ZGRgYQFVVVWKhGklVCy0sLHDp0iU8f/4cTZs2FdkWFhYGADA3NxdpF75xs7W1LXM87969E9u/NJGRkTAzM5O6vWfPnmIFVJo0aQIAaNOmDYDiip2fvw5u3boFDQ0NWFtbyzSO6OhodOnSBampqQgMDESLFi1kfg5AcSGH9+/fc4UkPmdoaCjyd0p4Q2NJhR7k5OREKgTGxcXh4cOHcHR0lPiBTW3q287ODkpKSrh586bY/sLCHq1btwYAZGdnIyEhAf7+/vD39xeL379/P/bv34/Vq1dj3rx5MDQ0hImJCd69e4esrCyR5/P+/XsAxb8vssjOzpb4IVVVvE7KO25hAiqp0mxBQYHIv9HR0SgqKpJakMfe3h6qqqplVmkkpLahKoiEVFBWVhauXr0qcVt4eDjCwsKgp6cn06fy1UVOTg48Hk9klqGgoAArV66slP6FpbalzV41bdoU9vb22LVrF3x9fWFiYiKx1LyTkxOeP3/OlTOuCAMDA3To0AFnz57F06dPuXbGGFasWAEAXOVHHR0dODs748KFCxIrHkqalVFQUMDRo0cxePBgTJ8+HRs3buS2DRkyBEpKSliyZAmys7PF9k1NTeUquQmNHj0aAoEAq1evxqlTp/Djjz9KrFpWVYSzPJ8/1507d0osK1+efp2dnXHnzh2EhISIbFu7dq1Y/MCBAwEAK1asEBnL06dPcebMGXTs2FHsd0r4xlhSGe7PGRkZISAgQOYvIyOjUvszNjbGDz/8IPIlnBVwdHSEsbExfHx8RN5APn78GMHBwXB1dRUpo56amooXL16IzVJFR0fDyckJycnJuHz5Mlq1aiV1PJ8+fZLY/uuvv6KgoAD9+vUr8xzdu3cPPj4+cHR0RMeOHUuNLSoqgqenJwoLC7F48eJa37eamhr69euH4OBgPH78mGvPyMiAj48PrKysuCp9qqqqOHbsmNjXli1bAAC9evXCsWPH0L9/f66f0aNHgzGG7du3i4xn69atAEQTR2m/d0FBQXj69KlYMl2Vr5PyjLtx48YAAD8/P5HYlJQU/PPPP9DW1oaFhQUAYOzYsRLPoZOTE4DiSrD79++X+jwIqa1oBoyQCsrKyoKTkxNsbW3Rq1cvWFlZgTGGFy9eYO/evcjJyYG3t7fITFNNGzx4MBYtWgRnZ2e4uLggLS0NBw8eFHkT+CXatWuHzZs3Y9q0aejTpw8UFBTQtm1bkRmHSZMmYcKECQCAJUuWSDw/rq6u8Pb2xsWLFzFkyJAKj2fjxo1wdHREp06dMH36dBgZGeHcuXO4dOkSRowYwZWMB4pLsHfo0AHOzs5wc3NDq1atkJ2djdu3b8PMzAyrVq0S619eXh6HDh2CgoICZs2ahYKCAsydOxf16tXD1q1bMWHCBDRq1AijR4+GqakpEhIS8OTJE5w+fRphYWEiMyva2toYPHgw92ZDeI4+5+TkhKtXr5Y5M1Nezs7OUFFRwejRo+Hh4QFtbW2EhITg/PnzsLCw4D6xrog///wTly5dQq9eveDh4YF69erB39+fuxSw5Mxp9+7dMWTIEBw+fBjJycno27cv4uLi4O3tDYFAgE2bNon17+/vDzs7O7HLziQRCAQyz0p+KQUFBWzcuBFDhw5Fp06dMHHiRKSlpWH9+vXQ19fHsmXLROJPnTqFsWPHYsmSJVyp7vT0dHTp0gVRUVGYMWMGXr58KTab2L17d24W6M8//8StW7fQpUsXmJiYICMjA+fPn0dQUBDatm2LGTNmiOz766+/Ijw8HPb29tDU1MSDBw+wa9cu1K1bV+z+cxkZGbC3t8egQYNgbm6O1NRUHDp0CPfv38fy5cvFZk5ra98rVqzAlStX0KNHD8yePRsaGhrYuXMnYmJi4O/vz71eFRQUJN4iQng/RgsLC7HtP/30E06cOIF58+bh1atXaNasGa5fv44DBw6ga9euGDp0KBc7depUxMbGomvXrjA1NUVOTg7u37+Pw4cPQ11dXeQDjKp+nZRn3LNmzcLevXuxcOFCPHnyBA4ODkhKSsLOnTsRGxsLb29vyMsXv/1s1qwZmjVrJnYOz507BwDo168f9PT0xLYTUutV+6ozQr4R+fn5bNeuXWzYsGHM2tqaqaurMwUFBVanTh02aNAg9u+//4rEl1aEo2SbUMly5yVJKq4haxGOgoIC5uXlxSwsLJiioiIzMTHhqoh9Pg5hMYXdu3dLfP6SCm4UFhayuXPnsrp163JFPj7fPyMjg2loaDA+ny9Sev9zjRs3Zn379hVrx2dFG0p7vowVl1AeMGAA09bWZoqKiqxhw4Zs1apV3CLwkt6/f88mT57M6tevzxQUFJiBgQHr3r27SLl6ST+XwsJC7nyUXBh//fp1NnDgQKavr88UFBSYsbExc3JyYmvWrJFYXe/atWsMALO0tBQpwV5Sy5YtmYqKCktOTubaJP38GSv/6+vq1avMwcGBqampMU1NTda7d2/25MkTibGmpqbM0dFRrF9pr5uHDx+ybt26MWVlZaatrc1Gjx7NLeD/vMR9fn4+W7lyJWvYsCFTVFRk2trabMCAARKLCURGRjIej8c2b94stu1rcfbsWda2bVumrKzMtLS02I8//ihWMIWx/72GJf2NKO2r5Gv+9OnTrEePHqxOnTpMSUmJqaiosGbNmrHly5dLfM2dOHGCtWvXjmlrazMlJSVmZWXFfvrpJ5HXl1Bubi4bOnQoMzMzY0pKSkxbW5v16NFD4q0banPfjBXffqF///5MU1OTKSsrMwcHBxYQECA1viRpRTiEEhIS2JQpU5ixsTFTUFBg5ubm7Oeffxb7+Rw5coT17t2b1atXjykpKTGBQMBsbGyYh4eHyG0lSh6zql4n5Rk3Y4xFRESwMWPGsLp16zJ5eXmmrq7OOnXqxE6cOCHTOaQiHORbx2OsAiueCSGkgnJzc2FsbIw2bdrg0qVLUuMOHz6MUaNG4dmzZ7CxsanGEdacO3fuoG3btvDy8pK4uD45ORn6+vpYvHix2OxJbXT//n20bt0aK1aswMKFCyvUx+zZs3Hs2DG8evWq0grGEEIIIVXp67k2ihDyXThw4ACSk5MxefLkUuOGDRuGNm3afBOJhqw2b94MBQUFsap6QoGBgdDX18f8+fOreWRf7vO1cIwx/PXXXwCKL42qiLi4OGzfvh3Lly+n5IsQQkitQTNghJBqcfbsWURHR2Pp0qUwNDREaGhomeX5vweZmZk4e/Ysnj17huXLl2PSpEnYtm1bTQ+r0tnY2KBr166ws7PjnvN///2HoUOH4vDhwzU9PEIIIaTaUAJGCKkWZmZm+PDhA1q1agUfHx+uVPf3LioqCubm5lBTU4OzszN8fHygoaFR08OqdD/99BPOnj2Ld+/eoaCgAObm5hg5ciQWLFhQaUVgCCGEkNqAEjBCCCGEEEIIqSa0BowQQkil4PF44PF43D18CCGEECKOEjBCCCGEEEIIqSaUgBFCCCGEEEJINaEEjBBCCCGEEEKqCSVghBBCCCGEEFJNKAEjhBBCCCGEkGpCZehrETMzM6SlpcHc3Lymh0II+Uakp6cjPDwcAGBkZIQ6deogOzsbHz9+RHp6OvLz8yEnJwdlZWXo6upCR0dHal8PHjwAAKipqcHa2lpiTE5ODlJTU5GRkYHs7Gzk5+cDAOTk5KCiogJNTU3o6uqCzxf/fJAxhqdPnyI/Px/y8vKwtbWVGFdSVlYWXrx4AQDQ1tamv5+EECKjyMhIaGhoICoqqqaH8s2Rr+kBENmlpaUhJyenpodBCPmGffr0CW/fvkXJz+YKCgqQnp6O9PR0JCUloUGDBmUmPtL6jo6OlritoKAAaWlpSEtLw8ePH9GgQQMoKyuLxPB4POjq6iIuLg4FBQVITU2FtrZ2qcdMTEzkHuvq6pZ7zIQQ8r2i95xVhxKwWkT4ye39+/dreCSEkG9FcHAwunTpAgBo2bIlAgICIC8vj9GjR6Nz586Qk5PD3bt34evri8zMTKSlpcHCwgLHjx8X64vH4wEAWrVqheDgYLHt27Ztw7Rp09CqVSt07twZNjY20NbWRlpaGqKjo3HkyBG8evUKubm5yMrKwo0bN6ClpSXSx/v372FmZobCwkJYWFggICBA6nPLysqCsbExAKBBgwZ49eoVN0ZCCCGla9WqVU0P4ZtFCRghhBAAwPnz56Guro7Lly+jXbt2XPuoUaPg4eEBJycnfPjwASdOnMCJEyfw448/lqv/Tp064fXr11IvA1y6dCnWrVuH+fPnIzo6Ghs3bsSSJUtEYurVq4fevXvj7NmzuHLlCiIjI6X2d+TIEaSlpQEAJkyYQMkXIYSQrwIV4SCEEMJZvXq1SPIlZGVlBV9fX+77NWvWlLvvJk2alLoGi8/nY968eejcuTMAYN++fRLjpkyZAqB4TVjJMX3Ox8cHACAvL4+xY8eWe7yEEEJIVaAEjBBCCIDiIhWlJSq9evVC48aNAQC3bt1CXFxclYyjQ4cOAIDXr1+LrOEqOQ5TU1MAwO7du1FYWCgWExYWhhs3bgAA+vXrByMjoyoZKyGEEFJelIARQggBUHyJoKKiYqkxXbt25R7fvXu3QscJDAzEhAkT0KxZM2hra0NeXh48Ho/7WrlyJRcbExMjtj+fz8fEiRMBAB8+fIC/v79YzM6dO7nHwlhCCCHka0AJGCGEEACApaVluWI+fPhQrv5TU1PRs2dPdO/eHb6+vggNDUVKSorEGSwh4Rquz40fPx4KCgoA/nepoVBubi53+WL9+vXRs2fPco2TEEIIqUpUhIMQQggAQEVFpcwYVVVV7nFGRka5+h88eDACAwMBAOrq6ujXrx+aN28OY2NjqKiocKXtDx8+jCNHjgCA1OTMyMgI/fv3x4kTJ3D+/HnExMSgbt26AIBTp07h06dPAIoTtYqUzCeEEEKqCiVghBBCABSXbS9LZmYm91hNTU3mvq9du8YlX82aNUNAQAD09fUlxoaEhMjU55QpU3DixAkUFhZi9+7d+OWXXwD87/JDPp+PcePGyTxGQgghpDrQx4KEEEIAABEREeWKqVOnjsx9C5MvAFi+fLnU5AuA1Js1f65bt27cJZG7du0CYwxv3rxBUFAQAMDZ2Rn169eXeYyEEEJIdaAEjBBCCADg+vXryMvLKzVGmNwAQJs2bWTuOz4+nntsYWEhNS4vL0/iTZwl4fF4mDRpEgAgMjISgYGB8PHxAWMMABXfIIQQ8nWiBIwQQggAICkpCXv27JG6/fLly3j27BkAoH379uUq7V5yfdnr16+lxm3duhUJCQky9zt27FgoKSlx+/r5+QEAjI2N0adPH5n7IYQQQqoLJWCEEEI48+bNk1he/vXr1yLrqebOnVuufkvOlv3+++/Izc0Vizl79iwWLlxYrn719PTw448/AiguvhEbGwugODGTl6dlzoQQQr4+9L8TIYR8o17FpyMkIhEZOQVQE8jDwVIP1obqUuN79+6NgIAAODg4wM3NDZ06dYKcnBzu3r0LX19fruqhi4sLl/TIatCgQahbty5iYmJw584dNG7cGOPHj0eDBg2QkpKC8+fP4+zZs1BRUYGLiwtOnjwpc9+TJ0/GwYMHue95PB4mTJhQrvERQggh1YUSMEII+caERCRi45Vw3IlMEttmb66Dmd2s4GCpJ7atTZs2GD58OCZMmAAfHx+x+2sBxUnagQMHyj0mZWVlHD9+HL1790ZycjLevHmDxYsXi8RoaWnhwIEDuHPnTrkSsM6dO6NRo0Z4/vw5AOCHH36Aubl5ucdICCGEVAe6BJEQQr4hR+6+xWjf2xKTLwC4E5mE0b63cfTuO4nbR40ahbt372LChAlo0KABBAIBdHR00LVrVxw4cAD+/v4QCAQVGlu7du3w+PFjeHh4wMLCAoqKitDU1IStrS0WLFiAx48fo3fv3hXq+4cffuAeU/ENQgghXzMeE5aLIl+9Vq1aAQDu379fwyMhhHyNQiISMdr3Nopk+KvO5wH7xrdF/vun6NKlCwBgyZIlWLp0adUOsgoUFRXBzMwM7969g76+Pt6/fw9FRcWaHhYhhNRq9L6z6tAMGCGEfCM2XgmXKfkCgCIGbLoSXrUDqib+/v549654Rm/s2LGUfBFCCPmqUQJGCCHfgFfx6VIvO5TmdmQS3iVlVdGIqkdhYSF+//13AIC8vDymTp1awyMihBBCSkdFOAgh5BsQEpFYof2exqRW8kiq3pMnTxATE4OkpCT4+fnh3r17AAB3d3eYmZnV7OAIIYSQMlACRggh34CMnIIK7ZedV1jJI6l6a9euFbthtJmZGVatWlVDIyKEEEJkR5cgEkLIN0BNULHP05QV5Sp5JNVHTk4O5ubmmDp1Km7dugUdHZ2aHhIhhBBSJpoBI4SQb4Ck+3rJYvyQvlg1o3YVw/Xz84Ofn19ND4MQQgipEJoBI4SQb4C1oTrszcs3A9TWXAfWhupVNCJCCCGESEIJGCGEfCNmdrMCnydbLJ8HeHazqtoBEUIIIUQMJWCEEPKNcLDUwwoXuzKTMD4PWOnStMKXLRJCCCGk4mgNGCGEfEOGtjFBPW0VbLoSjtsS7gvW1lwHnt2sKPkihBBCagglYIQQ8o1xsNSDg6UeXsWnIyQiERk5BVATyMPBUo/WfBFCCCE1jBIwQgj5RlkbqlPCRQghhHxlaA0YIYQQQgghhFQTSsAIIYQQQgghpJpQAkYIIYQQQggh1YQSMEIIIYQQQgipJpSAEUIIIYQQQkg1oQSMEEIIIYQQQqoJJWCEEEIIIYQQUk0oASOEEEIIIYSQakIJGCGEEEIIIYRUE/maHkBtxBjD7du3ERgYiBs3biAsLAwfP35EUVERtLW10aRJE3Tt2hXjxo2DkZFRTQ+XEEIIIYQQ8pXgMcZYTQ+iNjlz5gymTp2KDx8+lBkrEAjwxx9/YN68eZVy7FatWgEA7t+/Xyn9EUIIIYQQIgm976w6NANWTmFhYSLJl4WFBRwcHGBiYgJlZWVERkbi7NmziI+PR05ODubPn4/Y2FisXbu2BkdNCCGEEEII+RpQAlYB6urqmDhxIsaOHQtbW1ux7Tk5OZg1axa2b98OAFi3bh369u2LLl26VPdQCSGEEEIIIV8RKsJRTr1790ZUVBTWrl0rMfkCii893LZtG5ydnbm2zZs3V9cQCSGEEEIIIV8pSsDKqWnTptDR0ZEp1tPTk3t88+bNqhoSIYQQQgghpJagBKwKmZubc48/ffpUgyMhhBBCCCGEfA0oAatCMTEx3GN9ff0aHAkhhBBCCCHka0AJWBU6evQo97hTp041OBJCCCGEEELI14ASsCry7Nkz7N69m/t+0qRJNTgaQgghhBBCyNeAytBXgczMTIwaNQp5eXkAgEGDBpWrBL3wxnefe/78ORo1alQpYySEEEIIIYRUP5oBq2RFRUUYPXo0Hj16BACoV68edu7cWbODIoQQQgghhHwVaAasEjHGMHnyZJw6dQoAoKGhgdOnT0NXV7dc/dy/f19iu7SZMUIIIYQQQkjtQDNglWjGjBnw8fEBAKirq+PixYuUNBFCCCGEEEI4lIBVkhkzZsDb2xsAoKamhgsXLqB9+/Y1PCpCCCGEEELI14QSsEowY8YMbN68GQCgqqqK8+fPw8HBoYZHRQghhBBCCPna0BqwL+Th4cHNfKmoqMDf35/u+UUIIYQQQgiRiGbAvoCk5MvR0bGGR0UIIYQQQgj5WtEMWAVNnz4dW7ZsAVCcfJ07dw5OTk41OyhCCCGEEELIV41mwCpAUvJVnhstE0IIIYQQQr5PNANWTl5eXlzyBQDdunXD/fv3pd67q6ShQ4eifv36VTk8QgghhBBCyFeMErByevXqlcj3Z8+exdmzZ2Xat3Xr1pSAEUIIIYQQ8h2jSxAJIYQQQgghpJpQAlZOfn5+YIxV6IuKdBBCCCGEEPJ9owSMEEIIIYQQQqoJJWCEEEIIIYQQUk0oASOEEEIIIYSQakIJGCGEEEIIIYRUE0rACCGEEEIIIaSaUAJGCCGEEEIIIdWEEjBCCCGEEEIIqSaUgBFCCCGEEEJINaEEjBBCCCGEEEKqCSVghBBCCCGEEFJNKAEjhBBCCCGEkGpCCRghhBBCCCGEVBNKwAghhBBCCCGkmlACRgghhBBCCCHVhBIwQgghhBBCCKkmlIARQgghhBBCSDWhBIwQQgghhBBCqgklYIQQQgghhBBSTSgBI4QQQgghhJBqQgkYIYQQQgghhFQTSsAIIYQQQgghpJpQAkYIIYSQSldYWIitW7eic+fO0NXVhZycHHg8Hng8Hh49elTTw/tiUVFR3PNxd3ev6eFwli5dyo0rODhYYoxwu5OTU7WOjRBSTL6mB0AIIYSQb0tRUREGDBgAf3//mh5KuWzYsAEpKSnQ0tLCrFmzano4hJBvFCVghBBCCKlUp0+f5pIvU1NTTJ8+HWZmZlBQUAAAmJub1+TwpNqwYQOio6NhampKCRghpMpQAkYIIYSQSnX+/Hnu8eHDh9GuXbsaHA0hhHxdaA0YIYQQQirVu3fvuMctWrSowZEQQsjXhxIwQgghhFSq3Nxc7rGSklINjoQQQr4+lIARQggh5IsFBwdz1fWuXr3KtQvbhF9Lly4V2Y8xhqNHj2Lw4MEwMTGBQCCAlpYWmjZtijlz5iA8PFzmMTx//hwzZ86Era0tNDU1oaysDFNTUwwZMgSnTp2Sup+ZmRl4PB6io6MBANHR0WLj5vF48PPzK/X4UVFRmDNnDmxsbKCqqgodHR04ODhg69atKCwsLHXfgoICXLp0CXPnzkXHjh1hYGAARUVFqKurw9raGu7u7rh27ZrM54IQ8vWiNWCEEEIIqRHx8fEYNGgQbt68KdKem5uLJ0+e4MmTJ9i8eTN+//13LFy4sNS+lixZguXLl4slOm/fvsXbt29x7NgxODo64sSJE9DV1a3053L58mUMGTIEqampXFtWVhZu3LiBGzduYPfu3bhw4YLUY3fv3l1i2fj8/HyEh4cjPDwce/bsgZubG3bs2AFFRcVKfw6EkOpBCRghhBBCvpitrS03y/TLL7/g2bNnACA289SwYUMAQHp6Ojp37oxXr14BAIyNjTFu3Dg0adIEWVlZCAgIwLFjx5Cfn49FixahqKgIP//8s8RjL1q0CCtXrgQAyMnJYdiwYejatSuUlZXx5MkT7Nq1C/Hx8bh69Sq6du2KW7duQVlZmdt/x44dyMrKwqRJk5CQkAB9fX3s2LFD7DgtW7aUePzo6GgMHToUaWlpcHV1Rc+ePaGiooLQ0FD4+PggMTERd+/eRZ8+fXD9+nXIy4u//crOzoaamhq6deuGVq1awczMDAKBALGxsXj27BkOHDiAzMxM7NmzB1paWtiwYUNpPw5CyNeMkVqjZcuWrGXLljU9DEIIIaRUjo6ODAAr7W3GlClTuJiOHTuylJQUsZhLly4xgUDAADB5eXn26NEjsZgbN24wHo/HADBVVVV29epVsZhPnz6x1q1bc8ebN2+exDGZmpoyAMzU1LTM5xgZGcn1Jxzf6dOnxeLi4+NZkyZNuLg1a9ZI7C8wMJBlZWVJPV5iYiLr2LEjA8D4fD578+aNxLglS5ZwxwoKCpIYI9zu6OhY5vMk3y9631l1aA0YIYQQQqpVQkICdu/eDQDQ0NDAsWPHoKmpKRbXo0cP/PHHHwCK10itXr1aLGb16tVgjHGPO3fuLBajo6OD48ePQ0VFBQCwdetWpKSkVNbTAQDMmzcPAwYMEGs3MDDA4cOHIScnB6D4XmOS1oN169ZNZFbuc7q6utizZw+A4htdHzhwoJJGTgipbpSAEUIIIaRa+fv7c5US3dzcYGRkJDV22rRpUFdXBwD8888/IslLbm4ud88xXV1djB8/Xmo/pqamGD58OAAgMzMTly9f/uLnISQnJ1fqjZttbW3Rs2dPAMD79+9x9+7dCh2nQYMG3Lm6fft2hfoghNQ8SsAIIYQQUq3u3LnDPe7Ro0epsSoqKujYsSMAICMjA2FhYdy2x48fc4mck5NTmYUpSh6rMhOYJk2awNDQsNSYrl27co+lJWBpaWnYunUr+vXrBzMzM6ipqYlVYoyLiwNQnMgRQmonKsJBCCGEkGoVGxvLPba2ti4z3traGhcuXOD2tbOzq3A/ksbwpSwtLcsV8+HDB7HtQUFBGDFiBJdglSUtLU32ARJCviqUgBFCCCGkWqWnp3OPVVVVy4xXU1OTuG9l9fOlhGvLSlNyfBkZGSLbwsPD0adPH2RnZwMAbGxs4OzsDCsrK+jo6EAgEHCxwkqNZd1XjBDy9aIEjBBCCCHVSrimCyhej1WWkglLyX0rq58vlZWVVWZMyfGVTAQBYMWKFVzytXjxYvzxxx/g8XgS+5k4ceIXjJQQ8jWgNWCEEEIIqVbGxsbc4/Dw8DLjS8bUqVOn0vv5UhEREeWK+fzYgYGBAIorJv7+++9Sk6/09HQkJSV9wUgJIV8DSsAIIYQQUq3s7e25xwEBAaXGZmdn4/r16wCKZ44aNWrEbWvWrBmUlJQAAMHBwcjPzy+1r5KVD0uOQYjPL35bJCxrL6tnz54hPj6+1JigoCDucZs2bUS2Cfc1NzfnxiBJYGAgioqKyjU2QsjXhxIwQgghhFSrPn36cInTnj178PHjR6mxW7du5QpODBw4kLufFgAoKSmhT58+AIDExET4+flJ7efdu3c4dOgQgOL1WJKqLwovDZTlcsaSCgsLsWnTJqnbw8LCcOnSJQBAvXr1xBIw4RqyN2/eSE3+CgsL4eXlVa5xEUK+TpSAEUK+KsHBwVy55aVLlwIAXr16henTp8PKygoqKiqoU6cO+vXrhxs3bojt7+/vj759+6J+/foQCAQwNTXFtGnTSq0sVlBQgEuXLmHu3Lno2LEjDAwMoKioCHV1dVhbW8Pd3R3Xrl0rc+zu7u7c2KOiogAAly5dwsCBA1GvXj0oKSmhTp06cHV1lbkEdnp6On7//Xc0b94c6urq0NTURLNmzbBs2TJ8+vQJQHH5beFxy3L58mWMHj0aDRo0gIqKCtTV1dGwYUNMmTIF9+/fl2lMhHwpfX19jBs3DgCQkpKCIUOGSKzqd+XKFfzyyy8AAHl5ecybN08sZv78+dys0dy5cxESEiIWk5ycjMGDB3OJ1dSpU6GlpSUWZ25uDgD49OkT3r59W67ntHr1apw7d06sPSEhAcOGDUNBQQEAYNasWSJJJPC/GbGEhARs2LBBrI/8/HxMnDgR9+7dK9eYCCFfKUZqjZYtW7KWLVvW9DAIqVJBQUEMAAPAlixZwo4fP85UVFS4tpJfPB6P7dq1izHGWF5eHhs7dqzEOADMyMiIhYeHSzymk5OT1P1Kfrm5ubHc3FypY3dzc+NiX79+zaZOnSq1Lz6fz3x8fEo9F0+ePGH16tWT2oeJiQl7/Pgxc3R05NqkSU9PZ/369Sv1+fF4PObp6ckKCwtl+EkRIp0sr8m0tDRmbW3NxdWtW5f9+uuv7NChQ8zX15cNHz6c8fl8bvvy5cul9rVo0SIuTk5Ojo0aNYrt2rWLHTp0iC1evJgZGhpy25s2bcqysrIk9rNx40YurlWrVszHx4f5+/uzCxcusAsXLrD3799zsZGRkVysk5MT09LSYjwejw0ZMoT5+vqyQ4cOsUWLFjF9fX0uzt7enuXn54sd98yZMyK/i/3792fe3t7s8OHD7I8//mBWVlYMAOvSpQv3N8HU1FTic1iyZAnXT1BQkMQY4XZHR0ep55QQet9ZdSgBq0XoF4F8D0omYP369WNKSkpMXV2dzZo1i+3fv5/t3buXjRgxgvF4PO7NVnh4OPPw8ODeXP3111/syJEjbNOmTczW1pbrz8HBQeIx27Zty9TU1NiAAQPY77//zvbu3cuOHj3KNm7cyCZNmsRUVVW5PmbOnCl17CUTsOHDhzMAzNramv3555/s8OHDbOfOnczZ2ZmLUVRUZM+fP5fYV3x8vMibRisrK7Z8+XJ2+PBhtnXrVq4fCwsL1rJly1Lf7BYUFLCOHTtyMVpaWmzOnDls//79zM/Pj02YMIEpKipy2ydOnFjunxv59r2MS2O7rr9hmwJfsV3X37CXcWlSY2VJwBhjLC4ujrVr167UDwbk5eWZl5dXmeP79ddfmZycXKl9OTo6ssTERKl9pKeniySFn3/t3r2biy2ZgLm5ubGAgACmpaUldd82bdqUeuySSaSkLwcHB/bx40dmampKCRipFvS+s+pQAlaL0C8C+R6UTMCECUZ0dLRY3PLly7mYli1bMh6Px6ZOnSo2e5OZmcns7Oy42Nu3b4v1FRgYKPUTccYYS0xM5BIYPp/P3rx5IzGuZAIGgI0ZM0bip92enp5czNSpUyX2NWrUKC5mwIABLCcnRyzG19eXS0RLe7O7cuVKbruNjQ2LiYkRi3nw4AHT0dHh4s6ePSv1fJDvy/XwBOa67QYzXXBO7Mt12w12PTxBbB9ZEzDGGCsqKmJHjhxhgwYNYvXq1WNKSkpMQ0ODNWnShM2aNYu9evVK5rE+e/aMzZgxgzVu3Jipq6szJSUlVq9ePfbjjz+yEydOyNRHUlISW7x4MWvZsiXT1NQUmYUrLQETts2aNYtZW1szFRUVpqmpydq3b8+8vb0l/i343IULF1ifPn2Ynp4eU1BQYMbGxqxr165s586d3P6UgJHqQu87qw6PsXKW+iE1plWrVgBA6zTINy04OBhdunThvr958ybatWsnFpebmwt9fX3uZqp2dnZ48OAB5OXFb2948OBBjBw5EgDw+++/49dffy33uN68eQMLCwsAwB9//MGtSynJ3d0de/bsAQA0bNgQjx8/hqKiolhceno6DA0NkZ2djQYNGuD169ci2+Pi4lC/fn0UFBTAwMAA4eHh0NDQkDiukscEILaAPy8vDyYmJoiPj4e8vDwePHgAOzs7iX0dP34crq6uAAAHBweu8hz5fh25+xaLTj5BUSnvFPg8YKVLUwxpU7/6BkYIqXL0vrPqUBEOQshXq1WrVhKTL6C4+lnr1q257ydPniwx+QKAjh07co/DwsIqNJYGDRrAyMgIAGQqoDF16lSJyRdQfANY4dgjIyORk5Mjst3f359bsD927FipyRcAzJw5s9Rx3Lhxgytx7ezsLDX5AoDBgwfD0tISABASElJqZTry7QuJSCwz+QKAIgYsPBmKkIjE6hkYIYTUcpSAEUK+Wm3bti11u6GhIfdY0j19JMUlJydLjElLS8PWrVvRr18/mJmZQU1NjassKPwSVlJ8//59mWOXljgK1a1bF0DxjFVKSorItpKVzkrOBkrSokULaGpqSt1+584d7rGkstuf6969O/dY1kqN5Nu08Up4mcmXUBEDNl0p+0bIhBBCKAEjhHzFdHV1S90uvI9QWbEl4z6fbQKKb5BqY2ODadOm4dy5c4iOji71PkCSymV/Tk9Pr9TtpY3pw4cP3OMGDRqI7bt06VIuKQwODuZKZ0sSGxvLPba2ti5z3CVjSu5Lvi+v4tNxJzKpXPvcjkzCq/j0KhoRIYR8OyRfr0MIIV8B4b19Kju2pPDwcPTp0wfZ2dkAABsbGzg7O8PKygo6OjoQCARc7KRJk5CQkIDCwsIqGw8gehNY4Q1aS6Oqqip1m3CNXFlxQsIb0X6+L/m+VPRywpCIRFgbqlfyaAgh5NtCCRgh5Lu2YsUKLvlavHgx/vjjD6k3NJ44cWK1jKlkopSVlVVmfGmzderq6jLFCWVkZEjcl3xfMnIKqnU/Qgj5ntAliISQ71pgYCAAwMDAAL///rvU5Cs9PR1JSeW7JKui6tSpwz1+8+ZNmfGRkZFStxkbG3OPw8PLXqNTMqbkOMj3RU1Qsc9nK7ofIYR8TygBI4R814QVAs3NzUu9bDAwMBBFRUXVMqaS1R2DgoJKjQ0PD0dqaqrU7SWLkwQEBJR57JIxpRU2Id82B8vS1zBW9n6EEPI9oQSMEPJdE66xevPmjdg9tIQKCwvh5eVVbWPq06cPV1J/9+7dpRb9OHHiRKl9dejQgSuf7+/vX2oZ/pMnT3IzYB07doSBgUF5h06+EdaG6rA31ynXPm3NdWj9FyGEyIASMELINyktLQ0HDhzA+PHj0aJFC649JCQELVu2xNy5c/H69Wu0adMGAJCQkIANGzaI9GFmZgYejwdNTU2R0vCxsbHQ0dGBqqoqbG1tsXjx4jIvT3RycuIqFwIQmU1r2bIllJWVYW1tjZkzZyI/Px/Dhg0DAHz8+BFjxoxBbm6uxH4vX75c5rlwcHAAABQUFMDW1hYaGhpo2rQp5s6di6ioKABAaGgoJk+ezO2zcOHCMvsl37aZ3azAl3xFrhg+D/DsZlW1AyKEkG8EXaxNCPnm5OXlwcDAQGLSUlBQgIcPH+Lhw4fYtGkTxo8fz22bM2cOgoOD0bNnT+jq6nL358rMzET79u1x9+5dFBQUIC8vD3l5eQCAZ8+e4dmzZ/D19RW5dLA0ycnJuHTpksj3QPHlhOHh4di9eze2bduGgIAAxMfH459//oGdnR3c3d1hYWEhkgwaGxvD0NAQDx8+FFu/du/ePQwZMkRkjRhjDOnp6Xjy5AmePHmCjRs3okOHDrhz5w53viZOnIg+ffrI9FzIt8vBUg8rXOzKvBkznwesdGlKlx8SQoiMKAEjhFSLV/HpCIlIREZOAdQE8nCw1Kuyy5WKioqQm5uLOnXqoHv37txsD1B8Xy07OzucPXsWBQUF2L59O1xcXHDy5EkAwJkzZ3DmzBmR/hQVFZGdnY2CguIKb9ra2ti6dSuio6OxZ88ehIWFIT4+XqY1VgAwbtw4fPz4kft+06ZNKCoqwqFDh3D79m2kp6dj3Lhx8PPzw7x58xATE4Pw8HAsXrxYrK/ff/8du3fvBiBatfDmzZv44YcfuCqKjo6O+PjxI54/fy6yf2FhIf777z8AAI/Hw/Tp07Fx40aZngf59g1tY4J62irYdCUctyXcF6ytuQ48u1lR8kUIIeVACRghpEqFRCRi45VwiTd1tTfXwcwqePOmoKCACxcuoGfPntyskDABq1+/Pk6fPo3Q0FD07NkTcXFxePToEfz9/bFlyxbcvn0bqamp0NPTQ3JyMnJycpCXl4dHjx5BQ0MDaWlp0NDQwNChQwEAs2fPhru7Ow4ePMjNipXl9OnTMDc352am+vXrBzMzM3h6emLBggVYvXo1cnNz4eXlhWfPnmHDhg04efIk3rx5Ax6PB2VlZS6Bs7CwwKdPnwAAOjrFa3bS09MxdOhQZGVlQVVVFceOHYOzszMA4NKlS9i7dy9CQkIQFxeHvLw8MMYgLy+Pixcvolu3bpX0UyDfCgdLPThY6lXrhyiEEPIt4zFpq87JV6dVq1YAgPv379fwSAiRzZG7b2W+fGlIm/rVN7D/t3v3bowbNw4A8N9//6Fjx44i283MzBAdHQ0AaN++PUJCQiSWqc/JyUGjRo249VQvXryAjY2NSIyTkxOuXr3K9fv8+XORmzwLMcbg4OCAmzdvAgAuXryInj17isQsXboUy5YtAwCcPXsWAwYMQFFREfr3749//vkH69at4xLOvXv3YvTo0VLPwb///sslXcuXL8fPP/8sNZYQQsj3g953Vh0qwkEIqRIhEYllJl8AUMSAhSdDERKRWD0DK6FDhw7c49u3b5caO2fOHKn3CBMIBJg2bRr3/alTp0rta/r06RKTL6D4MsDZs2fL3Nc///zDFfTo0qULAGDfvn0AiteHjRw5stT9u3btyt3vS5aCHoQQQgj5MnQJIiGkSmy8El5m8iVUxIBNV8Ir/VLEqKgo7NmzB8HBwXjx4gVSUlKQk5MjMfb9+/el9tW1a1eZt9+9e7fU2LIu8yu5vazEcM+ePQCKy+mPHj0aqampCA0NBVCcgH2+nk0SNTU1ABBbH0YIIYSQykcJGCGk0r2KT5e45qs0tyOT8Co+vdLWlGzYsAELFy6UWr79c6Xda0tHR4dbXyWNpaUl9/jDhw8yx0o7npaWFlJSUvD48WOMGTMGrVq1grGxMYqKinD9+nUuNj8/HwDw119/QVdXF0+fPuVmxB48eIBBgwaVeqyShNUYCSGEEFJ1KAEjhFS6il5OGBKRWCkJ2IEDB0Qu4+vUqRMcHR1hZmYGdXV1KCoqAii+x5bw3leFhYVS+xPerLk0qqqq3OOMjIxSY2XtLyUlBYwx7Nu3j7us8HNycnLw8vLC9OnTAQCpqall9i2NMJkjhBBCSNWhBIwQUm5+fn4YO3YsgOJCFu7u7iLbM3IKytVfyvUDSA05hHGrAPOgIDg5OX3R+H777TcAgLy8PM6cOcNVAASKL0s0NzcHAAwYMECm/oSl3EuTmZnJPRZe0ldafyVLxpfWn7q6Otq1a4fIyEh8+vQJ6enpUFBQQHZ2NoDiIhsjRoyQeGx3d3euRD0hhBBCvg5UhIMQUunUBDX32c6bN2/w5s0bAMDAgQNFkq/PlTVTJZSUlISkpNIvqYyIiOAeC4tayBIr7XjCm0BbWVnh8uXLCA8PR1JSEvLz8/HTTz9JPVbdunW5x8+ePSv1OIQQQgipfpSAEUIqXU3elDU+Pp57bGFhUWpsWWu1SgoKCpJ5e5s2bUqN/ffff2XeXlZfn9PT00Pjxo0BFJcOfvfuXbn2J4QQQkjVogSMEFLprA3VYW9eetGKkrQ6jsSQbTfAGPviyw9Lrq96/fp1qbHh4eEy97t+/Xqp23Jzc7Flyxbu+7IKX2zZsqXU4iAlj+Xi4iLzGIXc3NwAAEVFRVi0aFG59yeEEEJI1aEEjBBSJWZ2swJf8m2zxPB5gGc3q0o5bqNGjbiCGP/88w/u3LkjNbagQPa1aiEhIfj555/x+b3rCwoKMGHCBERGRgIA+vbtK3YT5s+9efMGEydOFDs+Yww///wzbty4AQBo2rQpunfvLvMYhaZPnw5TU1MA/ytIkpeXJzU+LS0NmzZtQmBgYLmPRQghhJDyoSIchJAq4WCphxUudmXejJnPA1a6NJXpssVX8ekIiUhERk4B1ATycLDUE6uaqKioiMmTJ2PdunXIz89H586dMW7cOLRp0wYKCgoilwpaWFiUOUsGFK+zqlu3LlasWIHg4GAMHz4choaGiI6Oxp49e7i1Vtra2vD29i6zv4EDB2Lfvn14+PAh3NzcYGJigvj4eBw6dAg3b94EACgpKcHX11fqzZ9Lo6qqitOnT8PR0RFpaWnYsGEDjh49iiFDhqBp06bQ0NBAeno6IiMjcefOHQQFBSE3N1dqpUVCCCGEVB5KwAghYs6dO4ft27fj7t27SElJgZGRERwcHODh4YH27duXum/JKoNubm7Y98sarDx+A9fOHEB2xB0UpiWgKDcTmg7D0XP0DHh2s0LA/s3g2S8DULyW6vPLEI8HP4Brl1YAAFXbbtDrMxuFWalIf3AOLPI28lPjIcfjwcrKCq6urli8eDEePnzIJRZbt27F1q1bxcbarFkzqQlYQkICYmNjARSvFXN2dkZGRgZu3rzJJUklGRgYwN/fHyYmJqWfXBRXjkxISEBISAjmz58vtl1dXR2HDh1C69aty+xLmubNm+POnTsYPnw4Hj58iA8fPmDDhg1S45WUlKCnV3Nr9wghhJDvBSVgFZSYmIj79+9zX/fu3cPbt2+57ZLeRBLytSssLMT48eOxZ88ekfbo6GhER0fj8OHDWLFiBQwMDGTuM+P1fYT85Y60z27yO7qdKf6eXJzMBZSy/5G7bzHvyCORttzYcCSc/BOFGZ9E2h8+fIiHDx/i6NGjuHTpEk6cOIF9+/bh6dOnyMvLg5GREWxtbeHv7w8AUmeXoqKi0LNnT+6yPS0tLfj4+CAzMxObN2/G0aNH8ebNG+Tm5sLMzAwDBw7EvHnzyrxZs5CWlhaCgoKwc+dOHDhwAC9fvkRGRgbq1auH3r17Y/78+ahfv75MfZXGxsYG9+/fx9mzZ3Hy5EncvHkTcXFxyMzMhLq6OkxNTdGsWTN07doV/fv3h7a29hcfkxBCCCGlowSsArZt24apU6fW9DAIqXSenp5c8qWoqAg3Nzd07NgRfD4fd+7cga+vLxYsWICBAwfK1F9ERASGDBmCjIwMDB06FN26dYOGhgYiIyNFyqVLExKRiEUnn6DkFYyF6Qn4eHwZinLSodrYCUomTcFXFKDg0zuwsEtITU7Co0ePMG/ePOzdu1fsdzUqKopLwNTU1MTWdD158gS9evUSqZCoqakJoPjSvgULFmDBggUyPf/SKCgoYNq0aZg2bVq59126dCmWLl0qUyyPx0P//v3Rv3//ch+HEEIIIZWPErAKyMnJEWtTV1dHTk4O8vPza2BEhHy5//77j7tMT0tLC4GBgWjVqhW3fdSoUZg+fTqcnJxw+vRpmfoMCQmBmpoagoKC0Llz53KPaeOVcLH1YznRoeArqcJo5F9QqiNa7KKJ84+4sWYCUlJScPDgQaxcubLMe3KVdP36dfTr1w8pKSkQCARQV1dHQkJCucdNCCGEECINVUGsADU1NTg6OmLOnDk4cOAAXrx4gdTU1HK90SPka7N27VpuNmjDhg0iyZeQjY0NfHx8ytXv8uXLK5R8vYpPx51IyTc/1v5hsljyBQDP0gUY5j4BQPHllFeuXJH5eGfPnkWPHj2QkpICTU1NXLp0SaSkPSGEEEJIZaAZsAqYMGECJkyYUNPDIKTS5Obm4sKFCwCKi0mMGjVKamyfPn3QqFEjPH/+vMx+VVRUMH78+AqNKSQiUWI7X0UTqo0dpe6nbt6cexwWFibTsXbv3o2JEyeisLAQRkZGuHjxIpo1a1au8RJCCCGEyIJmwAghePz4MVdwwsnJCXJycqXGd+vWTaZ+W7Rowd2Tq7wyciTfo0vRyBI8vvTxKWr8r5Jf8meFPyRZtWoVxo0bh8LCQlhYWCAkJISSL0IIIYRUGZoBI4SIFJywtLQsM16WGAAyFdqQRk0g+c+TnLJGqftpqv3vskFJ6zVLOn36NFd0pFmzZrh06RIMDQ3LOVJCCCGEENlRAkYIQUZGBvdYlnVPss5qKSsrV3hMUm/MzCt94r6Vqeyl1AsK/jfLlp2djcLCQpHtUVFRMvdVluDg4ErrixBCCCG1F12CSAiBmpoa9zgrK6vM+MzMzKocDgDA2lAd9uay3VdLqK25Dhroq5Ud+P8GDx6MGTNmAABevXqFLl26iMwGEkIIIYRUNkrACCEiFTwjIiLKjJclpjLM7GYFvuR7JYvh8wDPblblPsamTZvg6ekJgJIwQgghhFQ9ugTxKySp/DcAPH/+HI0aNarm0ZDvQbNmzaCoqIi8vDxcvXoVhYWFpRbi+Pfff6tlXA6WeljhYod5u+JLjePzgJUuTeFgqYeoqIxSYyXZuHEjeDweNm7ciFevXsHJyQnBwcF0awlCCCGEVDqaASOEQElJCb179wYAxMfH4+DBg1JjL1y4IHN598owtI0J1g9tLnV7W3Md7BvfFkPa1P+i42zYsAGzZs0CAISHh8PJyQkxMTFf1CchhBBCyOdoBuwrdP/+fYnt0mbGCKkMc+fOxenTpwEAM2fOhJ2dHZo3by4SEx4eXuH7en2J1mb/WwvWvL4Whna3hppAHg6WerA2VK+046xfvx5AcTIWHh6OLl26ICgo6IuqORJCCCGElEQzYIQQAEDHjh0xbdo0AMX3z2rXrh0mT56Mffv24cCBA5g5cyZatGiB2NhYDBw4sMbGaWmghhndrDDWwbxSky+h9evXY/bs2QDAJWE0E0YIIYSQykIzYIR8417FpyMkIhEZOQVlzhpt2rQJGRkZ2Lt3L3Jzc7Fjxw7s2LGD287n8/HXX39BX1+fmy37Fq1btw48Hg/r1q3jLkcMDg6mmTBCCCGEfDFKwAj5RoVEJGLjlXDciUwS22ZvroOZ3azE7rUlJyeHPXv2wNXVFdu2bcOdO3eQlpYGQ0NDODg4YMaMGWjfvj38/Pyq6VnUnLVr14LH42Ht2rWIiIigJIwQQgghlYLHGGM1PYhvhZmZGaKjowEAQUFBcHJyqtT+hWvApK0RI0ToyN23WHTyCYpK+e0WVg780uIVhBBCCPn20PvOqkNrwAj5xoREJJaZfAFAEQMWngxFSERi9QyMEEIIIYRQAkbIt2bjlfAyky+hIgZsuhJetQMihBBCCCEcSsAI+Ya8ik+XuOarNLcjk/AqPr2KRkQIIYQQQkqiIhwVNHjwYLG2hIQE7vGSJUugr68vsn3IkCEYMmRIlY+NfL8qejlhSERilZR0J4QQQgghoigBq6ATJ06Uuv3atWtibba2tlU1HEIAABk5BdW6HyGEEEIIKR+6BJGQb4iaoGKfqVR0P0IIIYQQUj70rquCqHo/+Rp9fl+vqt6PEEIIIYSUD82AEfINsTZUh725Trn2aWuuQ+u/CCGEEEKqCSVghHxjZnazAp8nWyyfB3h2s6raARFCCCGEEA4lYIR8Yxws9bDCxa7MJIzPA1a6NKXLDwkhhBBCqhGtASPkGzS0jQnqaatg05Vw3JZwX7C25jrw7GZFyRchhBBCSDWjBIyQb5SDpR4cLPXwKj4dIRGJyMgpgJpAHg6WerTmixBCCCGkhlACRsg3ztpQnRIuQgghhJCvBK0BI4QQQgghhJBqQgkYIYQQQgghhFQTSsAIIYQQQkiFnTp1Cn369IGhoSEEAgHMzMwwatQo3L59GwDg5+cHHo8HHo8HPz8/sf2zs7Nx6tQpTJ8+HW3btoWuri4UFBSgqamJJk2aYOrUqXj8+HGZ43BycuKOAwBFRUXYtWsXnJycYGBgAFVVVdjZ2WH58uVIT08X2TcuLg6//vormjZtCg0NDWhqaqJz5844cuSIzOfh7du3WLx4Mezt7aGvrw9FRUUYGRmhe/fu2Lp1K/Ly8mTui3zbeIwxVtODILJp1aoVAOD+/fs1PBJCCCGEfO/y8/MxcuRIHDt2TOJ2OTk5rFq1Crq6uhg7diwAYPfu3XB3dxeJMzc3R1RUVJnHW7RoEby8vKRud3JywtWrVwEA6enpGDhwIK5cuSIxtkWLFrhy5Qq0tbVx8+ZNDBgwAAkJCRJj586dizVr1pQ6thUrVmDZsmXIzc2VGmNlZYVz587B2tq61L6+FvS+s+pQEQ5CCCEyiYqKgrm5OQDAzc1N4ifZlcXd3R179uwBAERGRsLMzKzKjkUIqZhJkyZxyZdAIIC7uzvat28POTk53Lt3D76+vpg3bx4GDx5caj/Z2dnQ0dFB9+7d0aJFC9StWxcKCgqIiYnBgwcPcPToUeTn52PFihUwMDDArFmzyhzb2LFjceXKFTg4OGDIkCEwMjJCdHQ0vL29ER0djYcPH2LWrFlYtmwZevbsiby8PEyYMAEdO3aEoqIi/vvvP+zcuRMFBQVYu3Ytevbsie7du0s81uzZs7FhwwYAgLq6OoYNGwZ7e3toamoiLi4Op0+fxr///ovw8HB07twZjx49gpGRUbnONfnGMFJrtGzZkrVs2bKmh0EI+U5FRkYyAAwAc3Nzq9Jjubm5cceKjIys0mMRQsovMDCQ+x3V09NjT548EYuJjIxkpqamXBwAtnv3brG4CxcusPz8fKnHioqKYg0bNmQAmLq6OktLS5MY5+joKHKs5cuXi8V8/PiR1alThwFgcnJyrFmzZkxfX589fvxYLHbv3r1cX87OzhKPefr0aS6mQ4cOLDY2VmLc9u3bubihQ4dKfa5fE3rfWXVoDRghhBBCCCmX9evXc483b94MW1tbsRgzMzOZZsp79eoFeXnpF2WZmppiy5YtAIovLfznn3/K7LNnz574+eefxdr19fXh4eEBACgsLMTjx4+xefNmNG3aVCx29OjRsLKyAgBcuXIFBQUFYjG//fYbAEBPTw9nz56VOrM1adIkjB49GgBw/PhxvHv3rsznQL5dlIARQgiRiZmZGRhjYIxV6eWHQPGifeGx6PJDQr4uOTk5uHz5MgDA2NgYrq6uUmOdnJwkJjfl1aFDB+6xsLhHaYRJliQODg7cY0NDw1IvkezYsSMAIC8vD69fvxbZ9vjxY4SGhgIAxo0bBx0dnVLHNGrUKADFiZ+0tWnk+0BrwAghhBBCiMweP36M/Px8AICjoyP4/NI/z3dycuISFWk+fvyIvXv34vLlywgLC0NycjKysrIkxr5//77MMbZt21bqNkNDQ+5xq1atSh1/ydjk5GSRbf/99x/3uKioCKdPny51TDExMdzj58+flxpLvm2UgBFCCCGEEJl9+PCBe9ygQYMy48uKOXLkCCZPnozU1FSZjp+WllZmjK6urtRtSkpKMsV9HpuTkyOyrWTlxjVr1pRZKbGkz5M58n2hSxAJIYTIJCoqirvHzudlpAFw25ycnMrsq6xYd3d3Lqas8tRZWVnYsmUL+vbti/r160NZWRnKyspo0KABXFxcsGPHjjLfsN28eRNTp05F48aNoaWlBYFAABMTEwwdOhT+/v5lPh9CvieZmZncYxUVlTLjVVVVpW67du0aRowYwSVfLVu2xE8//YTt27fj8OHDOHXqFPclVFhYWOYxy5qVK2+cJLImjJLQPcG+bzQDRgghpNa6ePEi3N3dER8fL7YtMjISkZGROHXqFG7evIndu3eLxWRmZmLChAk4fPiw2LZ3797h3bt3OHr0KPr06YNDhw5BXV29Sp4HIbVJyYRK2mWCJZVM2D63dOlSFBUVAQB27NiBiRMnlruPmqKmpsY9Dg4OhqOjYw2OhtQmlIARQgiplY4ePYoRI0Zwn4Y3bdoUP/74IywsLMDn8/Hu3TvcuHEDly5dAmNMbP/c3Fz88MMPuHXrFgDAxMQEw4cPR5MmTaCkpISIiAjs3bsXL1++hL+/PwYOHIiAgIAv+sSckG9BnTp1uMdv3rwpM15aTF5eHreOqnXr1lKTLwCIjo4u5yirXt26dbnHz549owSMyIwSMEIIIbVOZGQkxo0bh8LCQvD5fKxbtw6enp7g8XhiscnJyXj8+LFY+8KFC7nka8qUKdi4cSMUFRVFYubPn48JEyZg7969+Pfff7Fjxw5MmTKlap4UIbVEs2bNoKCggPz8fFy7dg1FRUWlfjARHBwssf3Tp09caXcLC4tSj3np0qUKj7eqlEy4Tp06hWnTptXgaEhtQh/jEUIIqXVWrFjBXZK0YMECzJw5U2LyBQDa2tpia81iY2O5+wp169YNW7duFUu+AEBBQQE+Pj5cEYF169ZV4rMgpHYSCATo0aMHgOKCHMeOHZMaGxwcLLUCYsn1Y5+XeC8pPT1d5L5jX4vWrVujSZMmAIDAwEAEBATU8IhIbUEJGCGEkFqlsLAQR44cAQCoq6tj0aJF5e7j6NGj3CL4uXPnlhqroKCAoUOHAgDCw8PLLApCyPdg1qxZ3GMPDw88ffpULCYqKkpiwR4hTU1N7kbH9+7dEym0IZSRkQFXV9ev8sbFPB4PK1as4L4fOnRomTN1z58/x9SpU6t6aOQrR5cgEkIIqVVCQ0O5qoZdunSpUGGMkvfv+fjxY5n37ylZMvr58+d0c2jyTXsVn46QiERk5BRATSAPB0s9WBuK/p798MMPcHd3h5+fHxITE9GmTRu4u7ujQ4cO4PP5uHfvHnbt2oW0tDQMHjwYx48fByBedXDGjBnw9PQEAAwePBgjR45Ex44doa6ujqdPn8LPzw8fPnzAmDFjsHfv3uo5AeXQr18//Pbbb/j999+RnJyMXr16oVOnTnB2doapqSnk5eWRlJSEZ8+e4erVq3jy5Ank5OSwdevWmh46qUGUgBFCCKlVSt6EtVGjRhXqo+QsVmmf0EtC9+8h36qQiERsvBKOO5FJYtvszXUws5sVHCz1uLYdO3YgIyMDx48fR05ODrZt24Zt27Zx2/l8PtasWQNNTU0uAfv8AxMPDw/cvn0bBw4cQFFREfbt24d9+/aJxAwYMADbtm37KhMwAFi2bBnq16+PuXPnIi0tDf/995/Ihzyfq1evXjWOjnyN6BJEQgghtUrJe3qVLANdHnT/HkJEHbn7FqN9b0tMvgDgTmQSRvvextG7/7sUUEFBAceOHcOJEyfQq1cv6OvrQ0lJCSYmJhg5ciRCQkIwd+5cfPr0idtHR0dHpF8ej4f9+/fj4MGD6NKlC7S0tKCoqIh69eqhb9++OHLkCE6fPg1lZeWqeeKVZMKECYiOjsbatWvRo0cP1KlTB0pKSlBSUoKRkRE6d+6M+fPn48qVKzJVjiTfNpoBI4QQUq2E9/ypKA0NDe5xRkZGhfoQJm7y8vLIzs6GvDz9d0i+XyERiVh08gmKxO/WIKKIAQtPhqKutrLITJiLiwtcXFyk7nfnzh3usZ2dncSY4cOHY/jw4aUeX9LtJEqSVm3xc2ZmZmX2JbR06VIsXbpUplgtLS3MmTMHc+bMkSmefL9oBowQQkilEFYRLGuGKDEx8YuOU/LynefPn1eoD+H9ewoKCvDq1asvGg8htd3GK+FlJl9CRQzYdCVc5r6joqJw7tw5AMXl6z+fASPke0QJGCGEkEqhpaUFoLgsdWlu3779Rcdp2rQpNwsWFBSE9PT0cvfx+f17CPlevYpPl3rZoTS3I5PwKj4dr1+/FlmT+bmYmBgMGjSI+1Bm8uTJXzRWQr4VlIARQgipFI0bNwYAREdHl7rGYdOmTV90HDk5Oe5SpfT0dJEy0LIaNmwYN2O3fv16xMXFfdGYCKmtQiIqNiMdEpGImzdvwtzcHN26dcPSpUuxb98+HD9+HNu3b8fYsWNhbW2NR48eAQDatWuHSZMmVeLICam9KAEjhBBSKXr16sU9XrBggcQ1Fr/99hsCAwO/+FgLFizg1nGtWrUKGzdulLqmIyUlBVevXhVpq1+/PmbMmAEA+PTpE3r27ImIiAipx2OM4cqVK1i+fPkXj52Qr0lGTsEX7VdQUIB///0Xy5Ytw5gxY+Dq6oopU6bAz88PWVlZAAAnJyf4+/tDTk6u0sZNSG1Gq44JIYRUinHjxuGvv/5CUlISjh8/jk6dOmHkyJHQ09PD27dvcfjwYdy7dw/Dhg3D4cOHv+hY5ubm8PX1xfDhw1FUVIRZs2Zh165dGDx4MCwtLcHj8RATE4ObN2/iwoULcHV1FbnsEABWrFiBR48e4cqVKwgNDUXjxo0xYMAAdO7cGUZGRsjPz0d8fDweP36MgIAAfPjwAd26dcPixYu/aOyEfE3UBBV7K6gmkEf//v1x4MABXLx4EQ8fPkRiYiKSkpKgqKgIQ0NDtG3bFsOGDUO/fv0qedSE1G6UgBFCCJHpxqtl0dfXx/79++Hi4oKcnByEhIQgJCREJKZv377w9fX94gQMAIYMGQIVFRWMHTsWiYmJCA0NRWhoqMTYz2/+ChSX0D5//jzmzp2Lrf/X3n3HRXHt/+N/LSC9C4qKBSmxxS4WVFCsGCMaFbvGGmNLYhKjxqj4UXO/Jt7EJGqs2K5dsFcQCxJEVGK7gopdMQhKUQRhfn/w27kL24HdpbyejwePDDvvOfOeBeK895w5Z9Uq5ObmYs+ePeJ6RYpIJ+8gqihkZzPU9jhbWxsMGzYMw4YNK+WsiCo2FmBERJWYtguvSkkkEoXt9e7dG1evXsW//vUvhIeH4/nz57Czs0OTJk0wduxYDB8+XOmxxfHRRx/h3r17WLduHQ4fPozr168jNTUVJiYmqFmzJlq0aIE+ffpg0KBBCo83NTXFb7/9hhkzZmD9+vU4ffo07t27h7S0NPFT/IYNG6Jjx4746KOPlE6hTVReeVW3gbebo1YTcbR1c9T6Axoi+h+JoOlCCGRwrVq1AgDExcUZOBMiqgh2xj5Uu/aPkQT4cUBTDG5TG4mJifDy8gJQMJvZ6tWr9ZQpEelS1J0UjFwfo9FU9EYSYMu4tsXuOaPyg/edusNJOIiIKiFtF16NupOCly9fiq9zLR+iisPHwwlLB3wIIzWd09IPZFh8EZUMhyASEVVCxVl4tUXa/2YSbNKkiY4yIyJDCGpTB64OllgRnogYBcMR27o5YrqSIclEpB0WYERElYw2C6/mvnyEnOS7OBpxB6HxRwAAFhYWCAgI0GWKRGQAPh5O8PFwKpVJeYhIORZgRESVjDYLr2bdOovXUdsLvfavf/0L9vb2pZwVEZUVXtVtWHAR6RCfASMiqmS0X3hVAiNzG3zQygcHDhwQFzAmIiIi7bEHjIioktFm4VX7jsNh33E4AGBW30bo6+Omq7SIiIgqBfaAERFVMiVZeJWIiIhKhgUYEVElI114VRtceJWIiKh0sAAjIqqEZvh7ql3zR8pIAkz399RtQkRERJUECzAiokqIC68SEREZBifhICKqpLjwKhERkf6xACMiqsS48CoREZF+sQAjIiIuvEpERKQnfAaMiIiIiIhIT1iAERERERER6QkLMCIiIiIiIj1hAUZERERERKQnLMCIiIiIiIj0hAUYERERERGRnrAAIyIiqqAkEgkkEgn8/PwMnQoREf3/WIARERERERHpCQswIiIiIiIiPTExdAJERESkG4IgGDoFIiIqgj1gREREREREesICjIiIiIiISE9YgBEREZUhkZGR4uyFCxYsAABcu3YNEydOhLu7OywsLODs7Ixu3bph+/btKtvSZBbE//73v1i2bBk+/vhj1K9fH5aWljAzM0ONGjXQq1cvrFy5EtnZ2Vrn/PDhQ8ycORMNGjSAlZUV7O3t0aFDB6xcuRLv37/X6L24desWZsyYgSZNmsDOzg4WFhaoW7cuBg8ejNDQUI3aICIqa/gMGBERURm2ZcsWTJgwAe/evRNfy87ORnh4OMLDw7Ft2zbs2bMH5ubmWre9efNmjB49WuG+58+f4/nz5zh+/DiWL1+OgwcPomHDhhq1e+zYMQwdOhSvXr0q9Hp0dDSio6MRFhaGgwcPwszMTGkb8+fPx+LFi5GXl1fo9YcPH+Lhw4fYvXs3fH19sXfvXlStWlWjvIiIygIWYERERGVUbGwslixZAgAYO3YsOnfuDGNjY8TGxmL9+vXIysrC4cOHMWLECOzZs0fr9t+8eQOJRIJWrVqhc+fO+OCDD+Dg4ID09HQ8ePAAO3fuREJCAu7evYvevXvj6tWrsLe3V9nm1atXsWzZMgiCgEmTJqF9+/YwMzPDpUuXsHr1amRlZeHkyZNYvHgxgoODFbYxe/Zs/PjjjwAAY2NjDBkyBF27doWFhQWuXbuGDRs2IDk5GWfOnEHXrl3x119/wcLCQuvrJyIyCIHKjZYtWwotW7Y0dBpERKRDp0+fFgCIXzY2NkJ0dLRcXEJCglCzZk0xbs+ePXIx0n2+vr4Kz3X9+nXh3r17SnPJy8sTli1bJrazYMECjXKuU6eOkJCQIBcXExMjmJiYCAAEBwcHITs7Wy7mwoULgkQiEQAIVlZWwpkzZ+RiXr58KbRu3Vo839dff630GoioeHjfqTt8BoyIiKgMW7ZsGdq1ayf3uqenJ9avXy9+/9NPP2ndduPGjeHm5qZ0v5GREb7++mt07twZQMFwSE1s3boVnp6ecq97e3sjKCgIAJCWloaLFy/KxUh7z6Tb0nPLcnR0xJ49e2BpaQkAWLVqldxwRyKisooFGBERURnl4OCATz/9VOn+Xr16oVGjRgCAv/76C8+fP9dJHh06dAAA3L17FykpKSpjW7RogU6dOind37VrV3H75s2bhfa9e/cOR44cAQBUrVoV48aNU9pO3bp1MXToUABAVlYWTpw4ofoiiIjKCBZgREREZVSnTp1gamqqMka2oImNjS3WeU6dOoXx48ejWbNmcHBwgImJiTiroUQiEZ/HAoAnT56obEtRb52sWrVqidtpaWmF9sXHx4uTjfj5+am99h49eojbMTExKmOJiMoKTsJBRERURnl4eGgV8/TpU63af/36NQYPHqxV71F6errK/U5OTir3y858WHR6+2fPnonbXl5eanORjZE9loioLGMBRkREVEZJn3FSxcrKStzOzMzUqv2BAwfi1KlTAAAbGxv07dsXzZs3R40aNWBpaQkjo4KBMjt27MDOnTsBQG5a+KKkxxRHRkaGuC17XcpYW1srPJaIqCxjAUZERFRGvXnzRm1MVlaWuC1bkKhz9uxZsfhq1qwZTp48CWdnZ4WxUVFRGrdbEjY2NuK27HUpI1twyh5LRFSW8RkwIiKiMurOnTtaxdSsWVPjtqXFFwAsXrxYafEFAA8ePNC43ZKoUaOGuJ2YmKg2XjZGm2snIjIkFmAldO/ePcyZMwfNmzeHo6MjLC0t4e7ujmHDhokzORERERXH+fPnkZOTozLm9OnT4nabNm00bjs5OVncdnd3VxqXk5ODyMhIjdstiWbNmonPiEVGRiI3N1dlvOyza97e3jrNjYiotLAAK4GVK1eiSZMmWLp0KeLj45GWloa3b9/i3r172L59O/r06YP+/furfWCZiIhIkdTUVGzatEnp/hMnTuDGjRsAgPbt28PFxUXjtmWfL7t7967SuFWrVuGff/7RuN2SMDMzQ58+fQAAKSkpCAkJURr76NEjbN++HUDB82KyMyISEZVlLMCKafXq1ZgyZQrevn0LAGjatCm+/fZbLFiwAP3794eJScHjdWFhYejfv7/aTzCJiIgU+frrrxVOL3/37l2MHTtW/H7mzJlatSvbWxYcHCxO/y7r4MGD+O6777Rqt6S++eYbcSKPmTNnKnz+LC0tDQMHDhSfE5s8eTLs7e31mSYRUbFxEo5iuHv3LmbMmCF+v3TpUrl/oK5cuYLevXsjOTkZERER+PnnnzF79mx9p0pERGVIQnIGou6kIDP7PazNTeDj4QSv6sonjwgICMDJkyfh4+OD0aNHo1OnTjA2NkZsbCzWr18vTkIxYMAAfPLJJ1rl0r9/f9SqVQtPnjzBxYsX0ahRI4wbNw7169fHq1evcOTIERw8eBCWlpYYMGAA9u3bV6Jr11S7du0wa9YsLF26FBkZGfD19cXQoUPRtWtXWFhY4Pr161i3bp04hLJp06YIDg7WS25ERKWBBVgx/PDDD2KP1rBhwxR+OtiiRQts3rwZPXv2BAD8+OOP/ISOiKiSirqTgl/DE3ExKVVun7ebI2b4e8LHQ379rDZt2mDo0KEYP3481q1bh3Xr1snFBAQEYNu2bVrnZGFhgT179iAgIABpaWm4d+8e5s6dWyjG3t4e27Ztw8WLF/VWgAHAkiVLYGJigiVLliAvLw9bt27F1q1b5eJ8fX2xd+9eWFhY6C03IqKS4hBELWVmZor/CEkkEvzwww9KY3v06IF27doBKFi4MiwsTB8pEhFRGbIz9iFGro9RWHwBwMWkVIxcH4NdsY8U7h8xYgRiY2Mxfvx41K9fH+bm5nB0dETXrl2xbds2HD58GObm5sXKrV27doiPj8fUqVPh7u4OU1NT2NnZoUmTJpg1axbi4+MREBBQrLZLKjg4GH///TemTZuGRo0awcbGBmZmZnB1dcUnn3yCvXv3IjIyElWrVjVIfkRExSURBEEwdBLlyb59+8RhHs2aNcPVq1dVxi9fvlwcl//xxx9j//79xT53q1atAABxcXHFboOIiPQn6k4KRq6PQb4G/9IaSYAt49oi9/F1dOnSBQAwf/58LFiwQLdJEhEpwPtO3WEPmJYuX74sbnfs2FFtfKdOncTtK1eu6CQnIiIqm34NT9So+AKAfAFYEa5+7SsiIirfWIBpSTrdLwB4enqqjffw8BC3Hz16xCnpiYgqiYTkDKXDDpWJSUrFo9Q3OsqIiIjKAhZgWnr+/Lm47erqqjbewcEBVlZW4veyC18SEVHFFXUnpVjHXX/yupQzISKisoSzIGopIyND3JYtrFSxtLQU1yqRPV4Z6Zjbom7duoWGDRtqdE4iIjKszOz3xTrubU5eKWdCRERlCXvAtCRdeBkATE1NNTpGdnaqN284tISIqDKwNi/eZ5wWpsalnAkREZUl7AHTkuxaI9K1wNTJzs4Wty0tLdXGK5ttRlnPGBERlT2K1vXSxLjBH+Ff0zhBMRFRRcUeMC3Z2NiI29JhherI9nrJHk9ERBWXV3UbeLs5anVMWzdHeFXnvxNERBUZCzAtubi4iNtPnjxRG//69etChVq1atV0khcREZU9M/w9YSTRLNZIAkz3Vz+7LhERlW8swLTUqFEjcTshIUFtfGLi/9Z0cXV1hZ2dnU7yIiKissfHwwlLB3yotggzkgA/Dmha7GGLRERUfvAZMC21bNlS3I6KilIbf+7cOXG7RYsWOsmJiIjKrqA2deDqYIkV4YmIUbAuWFs3R0z392TxRURUSbAA01LPnj1hbm6O7OxsxMfHIyEhAV5eXkrj9+zZI273799fHykSEVEZ4+PhBB8PJyQkZyDqTgoys9/D2twEPh5OfOaLiKiSYQGmJWtrawQGBmLHjh0QBAGLFi3Cli1bFMaeOnUKFy5cAFAw+UZgYKAeMyUiorLGq7oNCy4iokqOz4AVw6JFi1ClShUAwNatW7Fs2TK5mPj4eIwaNUr8ftasWXBwcNBbjkREREREVPZIBEHgYiPF8Mcff2Dq1Kni982aNUOvXr1gaWmJ+Ph4HDx4ELm5uQAAX19fHD9+HGZmZiU6p3QdMGXrhBERERERlQbed+oOhyAW05QpU5Cfn49vv/1WfB4sPj5eLq5v377YsmVLiYsvIiIiIiIq/zgEsQSmTZuG69evY9asWWjatCns7e1hbm6OevXqISgoCIcOHcKBAwc49TwREREREQHgEMRyhV3BRERERKQPvO/UHfaAERERERER6QkLMCIiIiIiIj1hAUZERERERKQnLMCIiIiIiIj0hAUYERERERGRnrAAIyIiIiIi0hMWYERERERERHrCAoyIiIiIiEhPWIARERERERHpCQswIiIiIiIiPWEBRkREREREpCcswIiIiIiIiPSEBRgREREREZGesAAjIiIiIiLSExZgREREREREesICjIiIiIiISE9YgBEREREREemJiaETICIiIlLk/v37CAkJAQD4+fnBz8/PoPkQEZUGFmBERERUJt2/fx8LFy4Uv2cBRkQVAYcgEhERERER6QkLMCIiIiIiIj1hAUZERERERKQnLMCIiIioTImMjIREIkGXLl3E1xYuXAiJRCL3pcjDhw8xd+5ceHt7w9nZGaampnBxcUH37t2xatUq5OTkqDz/+/fvcfz4ccycORMdO3ZEtWrVYGpqChsbG3h5eWHMmDE4e/ZsqV4zEVUeEkEQBEMnQZpp1aoVACAuLs7AmRAREelOZGRkoeJLlaK3MUuXLsXChQvx7t07pcd4enri0KFD8PLyUri/S5cuiIyMVHvu0aNHY82aNTA1NdUoV6LyhPedusNZEImIiKhMadKkCUJDQ3H9+nXMmzcPABAUFIQhQ4aoPO7LL7/EL7/8AgCwsbHBkCFD4O3tDTs7Ozx//hxhYWGIiIhAYmIiOnfujKtXr8LFxUWunbdv38La2hr+/v5o1aoV6tWrB3Nzczx79gw3btzAtm3bkJWVhU2bNsHe3l48JxGRJtgDVo7wkwgiIqpMZHvC5s+fjwULFiiN3b9/PwIDAwEAHTp0wN69exUWV2vWrMGkSZMAFBR1O3bskIsJDw9Hhw4dYGFhofBcL1++RGBgIM6fPw8jIyPcuXMHbm5uWl4dUdnG+07d4TNgREREVO798MMPAAAnJyccPHhQYfEFABMnTsTIkSMBAHv27MGjR4/kYvz9/ZUWXwBQtWpVbNq0CQCQn5+Pbdu2lTR9IqpEWIARERFRuRYfH4+///4bADB27Fg4OjqqjB8xYgQAIC8vD+Hh4cU6Z/369cUiLyYmplhtEFHlxGfAiIiIqFw7d+6cuJ2fn4+wsDCV8U+ePBG3b926pTAmPT0d27Ztw5EjR3Dt2jWkpKQgKytLYezjx4+1T5qIKi0WYERERFSu3b9/X9z+6aef8NNPP2l8bFpamtxrp0+fxrBhw/D8+XON2khPT9f4fERELMCIiIioXHv9+nWxjy26JlhiYiL69OmDt2/fAgA++OAD9O7dG56ennB0dIS5ubkYO3HiRPzzzz/Iy8sr9vmJqPJhAUZERETlmrW1tbgdGRkJX1/fYre1dOlSsfiaO3cuFi1apHTB5wkTJhT7PERUeXESDiIiIirXatWqJW7fuHGjRG2dOnUKAFCtWjUEBwcrLb4yMjKQmppaonMRUeXEAoyIiIjKJCOj/92mqFq2VLbHKzQ0tETnTE5OBgC4ubkVOn9Rp06dQn5+fonORUSVEwswIiIiKpNkhxYqm4EQAFq3bo3GjRsDKCiMTp48WexzWlpaAgDu3buntOjLy8vDkiVLin0OIqrcWIARERFRmeTm5iZuX758WWmcRCLB0qVLxe+DgoJw/PhxlW3funULkydPlnu9TZs2AIB//vkHv/zyi9z+3NxcTJgwAZcuXVKXPhGRQhJBVZ8+lSmtWrUCAMTFxRk4EyIiouJLSM5A1J0UZGa/h7W5CXw8nOBV3UZhbMuWLXHlyhUAwKRJk+Dv7w8bm//F9urVS9yeP38+goODxe87deqE3r17o27dujAxMUFqaipu3LiBM2fO4Nq1azA2Nsb79+8Lne/gwYP4+OOPxe8//vhj9OzZE1WrVkViYiI2b96MxMREdOnSBYmJiXj8+DHq1q1baCp8ooqA9526wwKsHOEfAhERlWdRd1Lwa3giLibJT17h7eaIGf6e8PFwKvT60aNH0bdvX6VTvRe9jVm3bh1mzpyp0dpcygqnOXPmFOpRK8rHxwehoaFo06YNHjx4wAKMKiTed+oOhyASERGRzu2MfYiR62MUFl8AcDEpFSPXx2BX7KNCr/fu3RtRUVEYNmwY3NzcYGFhofI848ePx4MHD/Dzzz+jR48eqFmzJszMzGBmZgYXFxd07twZ33zzDcLDw3Hv3j2FbSxZsgRHjx5Fnz594OTkhCpVqqBGjRro2rUr1q5di8jISDg7OxfvjSCiSo89YOUIP4kgIqLyKOpOCkauj0G+BnccRhJgy7i2cj1hRKRfvO/UHfaAERERkU79Gp6oUfEFAPkCsCI8UbcJEREZEAswIiIi0pmE5Aylww6ViUlKRUJyho4yIiIyLBZgREREpDNRd1L0ehwRUVnHAoyIiIh0JjP7vfqgUjyOiKisYwFGREREOmNtbqLX44iIyjoWYERERKQzxZ3NkLMgElFFxQKMiIiIdMarug283Ry1OqatmyO8qtvoKCMiIsNiAUZEREQ6NcPfE0YSzWKNJMB0f0/dJkREZEAswIiIiEinfDycsHTAh2qLMCMJ8OOAphx+SEQVGp9wJSIiIp0LalMHrg6WWBGeiBgF64K1dXPEdH9PFl9EVOGxACMiIiK98PFwgo+HExKSMxB1JwWZ2e9hbW4CHw8nPvNFRJUGCzAiIiLSK6/qNiy4iKjS4jNgREREREREesICjIiIiIiISE9YgBEREREREekJCzAiIiIiIiI9YQFGRERERESkJyzAiIiIiIiI9IQFGBERERERkZ6wACMiIiIiItITFmBERERERER6wgKMiIiIiIhIT1iAERERERER6QkLMCIiIiIiIj1hAUZERERERKQnLMCIiIiIiIj0hAUYERERERGRnrAAIyIiIiIi0hMWYERERERERHpiYugEyqOnT58iLi4OcXFxuHTpEuLi4vD8+XNxf1JSEurVq2e4BImIiIiIqExiAaal7777Dv/6178MnQYREREREZVDHIKopezsbLnXHB0dDZAJERERERGVN+wB05KDgwO6d++OVq1aiV9ubm6QSCSGTo2IiIiIiMo4FmBamj9/vqFTICIiIiKicopDEImIiIiIiPSEBRgREREREZGesAAjIiIiIiLSExZgREREREREesICjIiIiIiISE84C2IZ1KpVK4Wv37p1Cw0bNtRzNkREREREVFoqTAE2e/ZshIaGllp7CxcuRFBQUKm1R0REREREVGEKsGfPnuH27dul1l5aWlqptaWtuLg4ha8r6xkjIiIiIqLygc+AERERERER6UmF6QELCQlBSEiIodMgIiIiIiJSij1gREREREREesICjIiIiIionIqMjIREIoFEIsGCBQsMnU6pCAkJEa+pIo5wYwFGRERERESkJyzAiIiIiIiI9IQFGBERERERkZ5UmFkQ9eWff/7B5MmTVcZ8/vnnsLS0LPTa1KlT4efnp8PMiIiIiIiorGMBpqWsrCzs3btXZczRo0flXvvoo490lRIREREREZUTHIJIRERERFSBPHz4EDNnzkSDBg1gZWUFe3t7dOjQAStXrsT79+/VHn/r1i08evQIN2/ehJ2dHSwsLFC3bl0MHjwYoaGhWuUSERGBiRMnomHDhrC3t0eVKlXg7OyMTp064fvvv8fNmzeLe5mIjY2Fs7MzJBIJzMzMsGvXLrmY/Px87Nq1C0FBQXBzc4OlpSVsbGzQoEEDTJ48GdeuXVPY9q1bt8SZGPv166dRPj///LN4zOrVq5UHClRutGzZUmjZsqWh0yAiIiKiMuL06dMCAAGAMH/+fOHo0aOCvb29+FrRr+7duwvZ2dlK2/vhhx8EY2NjpccDEHx9fYWUlBSVeb148ULo1q2bynakX0Vt3LhR3Ldx40aF7Z84cUKwtrYWAAjW1tbCyZMn5WLu3LkjNG/eXOW5jYyMhHnz5ik8R+fOnQUAgrGxsfDkyROV1ysIgtCwYUMBgGBpaSm8fv1aaRyHIBIRERERVQBXr17FsmXLIAgCJk2ahPbt28PMzAyXLl3C6tWrkZWVhZMnT2Lx4sUIDg6WO3727Nn48ccfxe8dHBzw008/wcLCAteuXcOGDRuQnJyMM2fOoGvXrvjrr79gYWEh184///yDtm3bIikpCQBgZ2eHoUOHok2bNrC1tcXLly9x9epVHDp0CI8fP9b6Onfs2IFRo0YhNzcXzs7OOHLkCFq3bl0o5u7du2jXrh1SUlIAAG3btkW/fv3g5uaGvLw8XL58GSEhIUhNTcWiRYtgZGQkt47aZ599hrNnzyIvLw8bN27E3LlzleZ0/vx53Lp1CwAQFBQEW1tb5RegtpSjMoM9YEREREQkS7YHDIBQp04dISEhQS4uJiZGMDExEQAIDg4Ocr1gFy5cECQSiQBAsLKyEjw9PeXuO1++fCm0bt1aPNfXX3+tMKfevXuLMf7+/sLLly8VxuXn5wv79u2Te11VD9iKFSvEPOvWrSvcvn1b7vi8vDyhZcuWYu/Vhg0bFJ4/OTlZ7CEzMjISrl+/Xmj/u3fvBGdnZwGA4ObmJuTn5ytsRxAEYdSoUWLOFy5cUBonCILAZ8CIiIiIiCqIrVu3wtPTU+51b29vBAUFAQDS0tJw8eLFQvulPWfSbRsbG7k2HB0dsWfPHnG271WrVuHVq1eFYqKjo8UJ6Tw8PLB//344OjoqzFUikaB///4aX9v333+P6dOnQxAENG7cGFFRUfDy8pKLCwsLw+XLlwEA8+fPx6effqqwvWrVqmHnzp0wNjZGfn4+fv3110L7TU1NMWbMGABAUlISwsPDFbbz+vVr7NmzBwDQuHFjtG/fXuV1sAAjIiIiIqoAWrRogU6dOind37VrV3FbdvKLd+/e4ciRIwCAqlWrYty4cUrbqFu3LoYOHQqgYHbwEydOFNq/detWcfv777+HlZWVdhehQF5eHiZOnIjFixcDADp06IBz586hVq1aCuO3bNkCoKCAmjZtmsq2vby84O3tDQBy1wIAkyZNgkQiAQCsXbtWYRvbtm3DmzdvAAATJ05Uez18BoyIiIiIqAJo166dyv2yBUtaWpq4HR8fj3fv3gEA/Pz8YGpqqrKdHj16YP369QCAmJgYDB48WNx3/vx5AAW9W3379tXuAhTIzs7GoEGDxNkXAwICsHv3brk1d2WdO3cOQEEPV2RkpNpzGBsbAwAePHiAt2/fFnquzd3dHf7+/jh16hTCwsKQkpICJyenQsevW7cOAGBubo4RI0aoPR8LMCIiIiKiCqBoYVCUmZmZuJ2dnS1uP3v2TNxWNKSvKNkY2WMBiJNqVKtWTenQQ2189913eP36NQBgxIgR2LhxI0xMlJcwmZmZePnypZiLNkMcgYLCtOjEIp999hlOnTqFnJwcbN68GV999ZW4Ly4uDleuXAEAfPLJJxpdM4cgEhERERFVAEZGxbu1z8jIELc1GTJobW2t8FgASE9Pl4spCdl1y7KyssTn1JSRFmvFlZOTI/dav379UKNGDQD/6+2Skh2WOGHCBI3OwQKMiIiIiKgSk51wIysrS218ZmamwmMBiNOvy8aUxNKlS9GhQwcAQGhoKIKCgpCbm6s0Xrbw8/PzgyAIWn3Vq1dPrk0TExOMHTsWQMECzdJhlm/evMH27dsBFPQK+vr6anRNLMCIiIiIiCoxae8OACQmJqqNl42pWbNmoX2urq4AgBcvXiA1NbXEudnY2ODYsWOFirDBgwcrLcLs7OzEIuzmzZtqe8w0NXHiRLGHUdoLtnPnTrHHT9PeL4AFGBERERFRpdasWTPx+bDIyEiVPUxA4dkCpTMISklnYRQEAQcPHiyV/KRFmI+PD4CCaeZVFWGdO3cGUFAEXrhwoVRyqFOnDnr16gUA2L17N16/fi0OP6xSpQpGjx6tcVsswIiIiIiIKjEzMzP06dMHAJCSkoKQkBClsY8ePRKH3VlZWaFHjx6F9svOArh48WKNhjRqQlqEdezYEYDqIky2GJozZw7y8vJKJYfPPvsMQMHQw7lz5yI6OhoAEBgYCGdnZ43bYQFGRERERFTJffPNN+IQu5kzZyp8histLQ0DBw4Ui6rJkyfD3t6+UEy7du0QEBAAoGCoYmBgoNKhiIIg4MCBAxrnaG1tjaNHjxYqwgYNGiRXhA0cOBBt2rQBAJw9exbDhw+XmyxEVnZ2NjZt2oQdO3aoPH9AQABq164NAPjjjz/E17UZfghwGnoiIiIiokqvXbt2mDVrFpYuXYqMjAxkZGTA0dERGzduhIWFBa5fv45169YhOTkZANC0aVMEBwcrbGvTpk3w9vZGUlISTp06BXd3dwwZMgRt2rSBra0t0tLS8Pfff+PgwYN48OCBVs9pSYuwgIAAnDt3Dvv378egQYOwe/duVKlSBUDBbJB79+5F+/bt8eTJE+zcuRMnTpxAUFAQWrVqBXt7e7x58waPHj1CXFwcTp48iczMTCxatEjluY2NjTF+/HjMnz9ffM3NzQ3dunXTOH+ABRgRERERUZmTkJyBqDspyMx+D2tzE/h4OMGruo36A0tgyZIlMDExwZIlS5CXl4fU1FRx9j9Zvr6+2Lt3r9x6WVJOTk6Ijo5GUFAQzpw5g1evXmH16tVYvXq1XKxEItE6T9ki7OzZs9i/fz8GDhyIPXv2iEVY7dq1ERsbi5EjRyI8PBxpaWkKzy9lbGwMFxcXteceP348Fi1aJE6PP378eK2vgQUYEREREVEZEXUnBb+GJ+JikvywPW83R8zw94SPh+oFl0siODgYQ4YMgZ+fHzIyMlClShXk5OTA2dkZbdu2xbBhwzBgwAC17VSvXh2RkZE4evQotm/fjqioKCQnJyMnJwcODg5o2LAhunTpgmHDhhUrTysrKxw5ckQswg4cOICBAwdi9+7dMDU1BVAwu+OpU6dw5swZbN++HefPn8eTJ0+QkZEBKysruLq64sMPP4Sfn1+htb5UqVmzJho2bIhr167BxMQEn376qda5S4TSmpuRdK5Vq1YAClbcJiIiIqKKZWfsQ8zedw35Ku7OjSTAjwOaYnCb2jrNhfediiUmJsLLywtAwQLNYWFhWrfBSTiIiIiIiAws6k6K2uILAPIF4Lt9fyPqTop+EqNCZIcxTpo0qVhtsAAjIiIiIjKwX8MT1RZfUvkCsCJc/YLJVLqePn2KNWvWAAA8PT3FdcG0xWfAiIiIiIgMKCE5Q+EzX6rEJKUiITlD5xNzVHZnzpzBmzdv8PDhQyxfvlycnn/+/PnFmkAEYAFGRERERGRQxR1OGHUnhQWYjo0ePRoPHjwo9Fq/fv0wfPjwYrfJAoyIiIiIyIAys9/r9TjSnrm5Odzd3TF69GhMnz69RG2xACMiIiIiMiBr8+Ldkhf3ONLc/fv3S71NTsJBRERERGRAxV3XS5frgZHusAAjIiIiIjIgr+o28HZz1OqYtm6OfP6rnGIBRkRERERkYDP8PWGk4aR6RhJgur+nbhMinWEBRkRERERkYD4eTlg64EO1RZiRBPhxQFMOPyzH+OQeEREREVEZENSmDlwdLLEiPBExCtYFa+vmiOn+niy+yjkWYEREREREZYSPhxN8PJyQkJyBqDspyMx+D2tzE/h4OPGZrwqCBRgRERERURnjVd2GBVcFxWfAiIiIiIiI9IQFGBERERERkZ6wACMiIiIiItITFmBERERERER6wgKMiIiIiIhIT1iAERERERER6QkLMCIiIiIiIj1hAUZERERERKQnLMCIiIiIiIj0hAUYERERERGRnrAAIyIiIiIi0hMWYERERERERHrCAoyIiIiIiEhPWIARERERERHpCQswIiIiIiIiPWEBRkSVhkQigUQigZ+fn6FTIRkhISHizyYkJMTQ6RAREekUCzAiIiIiIiI9YQFGRERERESkJyaGToCISF8EQTB0CkRERFTJsQAjIiLSk6tXryIsLAwAEBgYiObNmxs0HyIi0j8WYERERHpy9epVLFy4EABQr149FmBERJUQnwEjojItMjJSnCFvwYIFAIBr165h4sSJcHd3h4WFBZydndGtWzds375dZVvqZkEcM2aMGHP//n0AwPHjxxEYGAhXV1eYmZmhZs2aGDRoEGJiYjS+hhs3buCrr75C8+bN4ejoCDMzM9SqVQsff/wxtm3bhvz8fJXHv337FqGhoZgyZQratm2LqlWrokqVKrCzs0Pjxo0xefJkxMfHq83Dz89PvD4AeP/+PVatWoWOHTvC2dkZFhYW8PLywowZM/Do0SOVbSl6r7Zt24Zu3bqhRo0aMDc3R7169TB+/HjcvHlTszdKA/n5+di1axeCgoLg5uYGS0tL2NjYoEGDBpg8eTKuXbtWauciIiLSCYHKjZYtWwotW7Y0dBpEenX69GkBgABAmD9/vrB582bBzMxMfK3oV58+fYS3b98qbEsa4+vrq3D/6NGjxZi7d+8KkydPVnoeIyMjYd26dSpzz83NFaZPny4YGRkpbQeA4O3tLTx79kxpO/Xq1VN5vPRr9uzZKvPx9fUVY1NTUwUfHx+lbdnY2AiHDx9W2pbse3X79m0hMDBQaVumpqYq36uNGzeKsRs3blQad+fOHaF58+Yq3wMjIyNh3rx5Kt8HQ9L0WomIDI33nbrDIYhEVG7ExsZiyZIlAICxY8eic+fOMDY2RmxsLNavX4+srCwcPnwYI0aMwJ49e0p0ru+//x7bt2+Hl5cXRo0aBQ8PD2RkZGDfvn04evQo8vPz8fnnn8PHxwcNGjSQO14QBAwePBihoaEAAGdnZwwdOhQtWrSAlZUVHjx4gJ07d+LSpUu4ePEi/P39ERsbC0tLS7m23r59C0dHR3Tv3h0tWrRArVq1UKVKFTx58gSXL1/Grl27kJubi6VLl6JatWr44osv1F7f2LFjERUVhUaNGmH06NGoW7cunj9/ju3btyMmJgYZGRkYMGAAzp8/j9atW6tsa9asWQgLC0PdunUxduxYfPDBB0hNTUVYWBhOnDiBnJwcTJgwAc7Ozvj44481+wEUcffuXbRr1w4pKSkAgLZt26Jfv35wc3NDXl4eLl++jJCQEKSmpmLRokUwMjISe0yJiIjKFENXgKQ5fhJBlZFsDxj+/56Z6OhoubiEhAShZs2aYtyePXvkYqT7NOkBAyCMGjVKyM3NlYubPn26GDN58mSFbf3yyy9iTGBgoPD69WuFcXPmzBHjZs2apTDm6NGjCvOQun//vtCgQQPx/UlPT1cYJ9sDBkAYNmyYkJOTUygmPz9f+Oabb8SYDz/8UMjPz5drq+h71aVLFyEjI0Mu7rfffhNjXFxcFMao6xXKy8sTWrZsKQAQjI2NhQ0bNhTaP27cOAGAIJFIhCZNmog9YdevXxdjli9fLp7D0tJSePfuncL36KuvvhLj/vvf/xbaFxcXJwQHBws9e/YUateuLZiZmQnm5uaCq6ur0K9fP2HLli3C+/fvFbYre42qvurWravweEEQhOvXrwtffvml0KxZM8HBwUEwNTUVatasKfTt21fYunWrkJeXp/TYpKQk8RyjR48WBEEQHj9+LMyZM0f48MMPBXt7e7GXmYhIEHjfqUsswMoR/iFQeTR//nzxxu/06dNaH1+0AFu9erXS2KNHj4px7dq1k9uvTQHWoEEDpTfp6enpgoWFhQBAqF+/vtz+t2/fCtWqVVPbjlSnTp0EAIKtra3S4ZPqREREiLlv2bJFYYxsAVavXj2l58rPzxfat28vxh47dkwuRva9srOzE5KTk5XmFhQUpPLnp64A27t3r7g/ODhYbv+WLVvE/b/88otgbGwsABAmTJggxnz88ceFfo/Onj2rMFdpoVejRo1Cry9YsECjAqpVq1bCkydPVF6jtgVYaQxlLVqAHTt2THBwcJBrgwUYEUnxvlN3OASRiMoNBwcHfPrpp0r39+rVC40aNcLNmzfx119/4fnz53BxcSnWuSZPngxTU1OF+2xsbNC6dWucO3cOSUlJyM7Ohrm5ubj/+PHjePHiBQBg+vTpStuRGjFiBM6dO4f09HT89ddfSicJUaVDhw7idkxMDEaMGKEyfsqUKYVyliWRSPDll18iOjoaABAaGoqePXuqzL9atWpK98+cORM7d+4U25o0aZLK3IrasmULAMDU1BTTpk2T29+lSxdx+/bt2/D29kZ0dDROnDgBoGDijrNnzxY65vTp0+jUqVOh1169eoWrV6/KtQkUDAM1MTFB+/bt4ePjAw8PD9ja2iI1NRVJSUnYunUrnjx5gri4OPTr1w8XLlxAlSpVxOO7du2K0NBQRERE4LfffgMATJs2DV27di10nqJDUIVSHMoqdefOHQwePBiZmZkICgqCv78/bG1tkZSUhFq1aik9joiISgcLMCIqNzp16qS2mOnatas4615sbCz69u1brHO1a9dO5X7pjaogCHj16lWhQu/cuXPidmZmprjukzJPnjwRt2/duqWwAHvx4gU2b96MEydO4ObNm0hLS8ObN28Utvf48WOV5wMAf39/jffHxsaWqK3WrVvD1tYW6enpattSRPp+VqtWDZGRkQpjatSogWfPnuHAgQNwc3MDADx48ABv377FzZs38erVKwBA+/btER0djdOnT+OHH34o1MbZs2fFGSmL/gw++eQTfPHFF0oL+uDgYHz77bf49ddfcenSJfznP//B6NGjxf116tRBnTp1xDwAoGXLlggMDFR57StWrBCLr8DAQGzatAm2traFYr7++mvMnTsXS5Yswc2bNxEcHIwff/xRaZtRUVGwtrbG6dOn0blzZ5XnJyKi0scCjIjKDQ8PD61inj59WuxzOTk5qdxvZmYmbmdnZxfaJ52WHQC+/fZbrc6blpYm99rOnTsxadIkvH79WqM20tPT1caoey8dHR1hb2+PV69eqX0f1bUlkUhQv359XL16FampqXj37l2h90+VzMxMvHz5EkBBYdm/f3+V8U+ePClU0KalpeH06dMACgq4zz//HNHR0YiOjpbruZTGAfI9YG3atFF5XlNTU/z88884cOAAkpKSsGXLlkIFWHFkZ2eLk840aNAAO3fuVPoBxOLFi3Hu3DmcO3cOq1atwoIFC5T2cErjWXwRERkGCzAiKjdUDauSsrKyErczMzOLfS4jo+Ivk6hpoaRITk5Ooe/Pnj2LYcOGiT0zLVu2RLdu3eDu7g47O7tChYy0OMnLy1N7Hk3fy1evXql9H4vzc9G0ACvJewkUvJ/SwqpLly7ikL93794hOjq6UKEl7V1zdXXVqNgvytjYGG3btkVSUhIuXrwIQRDENdeKQ1dDWS0tLTFu3Lhi50VERCXDAoyIyg1lQ+5kZWVlidvW1ta6TEcp2fPev38fdevWLXZbCxYsEIuvNWvWYMKECQrjZK9bE2/evIGNjY3KGGmb6t5HXf5cZGP9/PwK9VLJevbsGWrWrAkAmDBhAtasWQOgoBg9f/48gIICrGbNmvDy8kJCQgJOnz4tFmBpaWn4+++/xfMokp+fj7CwMOzduxdXrlzB06dPkZGRoXAh7YyMDKSnp8POzk7jay1KF0NZAYjPjxERkWEU/yNeoiLy8vKwZcsW9O3bF7Vr14a5uTksLCxQu3ZttGzZEhMnTsS+ffuU3qxlZWVhyZIlaNmyJWxtbWFnZ4emTZti/vz54to/fn5+kEgkSj9VVre/uLEPHz7E3Llz4e3tDWdnZ5iamsLFxQXdu3fHqlWr5HotlMnPz8euXbsQFBQENzc3WFpawsbGBg0aNMDkyZNx7do1pceOGTNGzFfTL9mhcEUdO3YMY8aMgaenJ2xsbGBpaQl3d3eMGTNGvGFVRxAEbN26Ff7+/nBycoKFhQXc3d0xadIk3LhxQ6M2tHHnzh2tYqQ35PomO5FBSd6HnJwc8Sa8devWSosvoOB5J22oey9TU1PF55XUvY/q2hIEAffu3QNQMLRR094vALCzsxOLsJs3b0IQBIVxNWrUgJeXFwAgIiJCfD0uLk4ckikttqT/lY07c+aMWEgVHX4IFAx/9Pb2xieffIL//Oc/uHXrFl6/fq2w+JLSZCioKkWHsvbv31/lV3BwsBivaCirFCfaICIyLPaAUalISUlBQECAwgfsHz9+jMePH+PKlStYu3YtQkND5R48T0xMRM+ePZGUlFTo9WvXruHatWvYsGEDDh48qMtLUGrp0qVYuHAh3r17V+j15ORkJCcn49SpU/j3v/+NQ4cOiTeAity9excDBw4UZ1mTdfv2bdy+fRtr1qzB3LlzC91IlbZ//vkHQ4YMKXTzKXXv3j3cu3cPmzZtwrhx47Bq1apCM7nJevPmDfr37y/ONCfbxpo1a7B582asXbu2VHM/f/48cnJyVA7Fku0hUffcjq74+vrijz/+AFAw619AQECx2nn58iXev38PAHB3d1cZe/z4ca3ajoiIQIsWLVTul1L3PkZERKh8Nku2CCrOz6Rz5844cuQIXrx4gQsXLsDHx0dhXJcuXZCQkIC7d+/i0aNHqF27tvj7IO35ksb9+eefuHjxIrKysmBlZaXy+a/c3Fz07NlTnNzFyckJH3/8MZo0aYLq1avD3NxcHLK6YsUKsS1NhoKqUppDWWVZWFgUu10iIio5FmBUKiZMmCAWXx4eHhg6dCi8vLxgYWGB9PR03L59G2fPnkVMTIzcsWlpaejatas4c5ubmxvGjh0LT09PpKamYt++fTh16hT69+8Pe3t7fV4WvvzyS/zyyy8ACqYeHzJkCLy9vWFnZ4fnz58jLCwMERERSExMROfOnXH16lWFs6TdvXsX7dq1E3vy2rZti379+sHNzQ15eXm4fPkyQkJCkJqaikWLFsHIyAgLFiwo1Mb06dPVzpiWlJSEmTNnQhAEVKlSRe7ZnNTUVLRv3x53794FADRq1AiDBg2Cl5cXjIyMcOPGDYSEhODx48dYv3493r9/j5CQEIXnGjx4sFh82djYYNy4cWjdujVyc3Nx5swZbN26FWPHjkX37t3VvMuaS01NxaZNm5T2BJ04cULscWrfvn2xp6AvqYCAADg5OSElJQWbN2/GF198gcaNG2vdjuzPT/ozUyQjIwP//ve/tWp75cqVmDp1qtLeKNn2BgwYoLKtrVu34ocffoCzs7PC/cuXL9e4LUVGjx6NI0eOAADmzJmDiIgIGBsby8X5+fnhzz//BFBQiI8aNarQ81+ycUBBYRUVFYUePXqIz3/VrVtXnEVRavv27WLx1b17d4SGhiodwrdt2zatr0+Z0hzKSkREZYgB1yAjLZXVBfGSk5PFBUJbt24tZGZmKo29f/++cP/+/UKvjRs3TlwEtGfPnkJWVpbccb/88ovcgqGKyC40q4662LCwMHF/hw4dlC5w+ueff4pxQUFBcvvz8vLExV2NjY2FDRs2KGwnOTlZaN68uQBAMDIyEq5fv672GmS9fv1aaNy4sZjL2rVr5WICAwPF/YsWLRLy8vLkYjIyMoQePXqIcUePHpWLkV34tk6dOsK9e/fkYs6fPy9YWVkV+pmVxkLMtra2wsWLF+Xi7ty5I9SqVUuM27Nnj1yMdJ8mCzEnJSWpzEtd7G+//VboPYqNjVXZ3sWLF4VvvvlG7nVPT0+xnX379sntz8jIEHr27FnoPVJ2fbK/8wCEkSNHCrm5uYVi8vPzhdmzZ4sxTZs2FfLz81VePwDB399f4d/+qlWrxBgXFxchIyNDLkbdQsx5eXlCmzZtCv2dpaeny8U9e/ZMjOnYsaOwZcsW8Xdw3bp1hWIbNmwoABC+++47ISUlRZBIJOIixUWNHDlSbFfd36Vsnop+L9Rdq6wpU6aIsYcPH1YZq07RhZiJiNQpq/edFQELsHKkrP4hREdHi/+wL1++XKtjX7x4IZiamgoAhKpVqwovX75UGjtw4EC9FmBNmzYVAAhOTk4q8xKE/92gGRsbCw8fPiy0b+/eveJ5goODVbZz+/ZtwdjYWAAgTJgwQe01SL1//17o3bu3eJ4vv/xSLiYuLk7cP3bsWJXtpaamCnZ2dgIAoXv37nL7pYUiACEyMlJpO7///nupFmABAQFClSpVhCpVqgjjx48XNm3aJGzdulWYMWOGYG1tLcYNGDBAYVv6LMAEQRBGjRolxkgkEqF3797CTz/9JOzYsUPYvn278NtvvwkTJ04U3N3dBQCCu7u7XBsrVqwQ2zAyMhJGjhwp/Pnnn8J//vMfYc6cOULNmjUFAIXOpUkBJi3GmzRpIixbtkzYuXOnsGLFCqF9+/ZijJmZmdLCUfb6pW3Vq1dPWLRokbBjxw5h9erVhQpDiUQi7N+/X2FbmhQlDx8+LFRgOzg4CJ999pmwdu1aYffu3cKmTZuE//u//yv0ezBhwgRxu+iHBJ9//rkAQGjbtm2hv9GQkBC5c8t+IPH27VuF+QlCwYcoVapUUfl7sWnTJnG/sg9jpHbt2iXGjh8/XmWsOizAiEhbZfW+syJgAVaOlNU/hPj4ePEf9nHjxml17IYNG8RjZ86cqTI2JiZGbwXY1atXxX3ffvut2raOHz+u9AZSenNqamoqpKWlqW1LegNct25dtbFSM2bMKFSkKOrZ+uKLL8SYmzdvqm1z6NCh4k14dna2+Pq9e/fEdlq1aqWyjezsbMHe3l5pAXb7ebqw4fw9YcWpBGHD+XvC7efyvRqyBdj8+fOFLVu2CGZmZoV+F2S/AgIClN4k67sAy8/PFxYtWqQyX9kvRXnl5+cLw4cPV3lcv379hDdv3mhVgKWlpQk+Pj5K27SxsREOHTqk0fXfvn1b6N+/v9K2TE1NFfbISmnaK/T06VPB399fo/cSKOgFU/a3tHv3bgEo+NBE9lqK9tALQuGeY1U9YLJ/Y8p+L2SLvd9++01pW4IgCJmZmYKTk5P4HmrbKy6LBRgRaaus3ndWBHwGjEqscePGqFmzJp4+fYr169dDEARMmDAB3t7eatdSkp20w9/fX2VsmzZtYGtrW+KZxTQhO/2zdOppVYpO/6yorWrVqonPmagifbblwYMHePv2rdoH5tesWYNff/0VQMHPYseOHQrfd2kepqam4qQfqkgnHXn37h3u3buHhg0bAtDuZ2ZmZoaOHTvi0KFDhV6PupOCX8MTcTEpVe4YbzdHzPD3hI+H4oWQR4wYgWbNmmHFihWIiIjA06dPYWlpiebNm2PcuHEYNmyYypz0SSKR4Pvvv8e4ceOwbt06hIeH4/bt20hNTYWRkRGcnJzQoEEDtG/fHgEBAWjXrp3CNrZu3Yo+ffpg7dq1uHLlCt68eYNq1aqhefPmGDlyJAYPHqx1bvb29jh9+jTWrl2Lbdu24fbt28jMzISrqysCAgLwzTffoHbt2hq1ZWpqin379mHbtm3YuHEjrl+/jrS0NLi4uKBbt26YOXMmGjVqpHWORdWoUQOnTp3CmTNnsH37dpw/fx5PnjxBRkYGrKys4OrqCjs7O0RHRwMA/vrrLwCKZzWUzoKal5cnPrfl5uam8DmrNm3aiP8PmDdvHvbs2SP3N7ZmzRqsWLFC7TXIPl92+fJllbFWVlaYP38+pk2bhpycHAQEBGDv3r1o3bq10mNiY2Oxe/du/L//9//U5kJERIbBAoxKzNjYGH/++Sc++eQT5OTkYMOGDdiwYQPs7e3Rvn17dOzYET179kSrVq3kjn369Km4rW7hU4lEgvr16yucRbC0yU7//NNPP+Gnn37S+FjZ6Z8zMzPx8uVLAAWzQaqaKU5ZW6oKsIiICEydOhUA4OzsjEOHDild20l6TTk5OcXKQ0qbn5mimJ2xDzF73zXkC4rjLyalYuT6GPw4oCkGt1FcAHz44YfFmmFRUDKFuVRISIjSSUdKElujRg3MmzcP8+bN0yhekaFDh2Lo0KEqY9RdX1FVqlTB559/js8//7zYeckaPnw4hg8frvVxY8aMwZgxYxTuS0jOQNSdFGRmv4e1uQl8PJzg6+sLX19fhfEvXrxA9erVAUCcQVJRAebk5IQmTZrg2rVrKuMAYOzYsViyZAmysrIQGhqKli1bYuTIkXB1dUVycjL27duHM2fOwMXFBR9++CFOnjyp9Fo//PBDVKtWDS9evMDWrVvh7OyMdu3aiX/nFhYWha5t6tSpiI2NxebNm/Hw4UN4e3ujV69e8Pf3h6urKwRBQEpKCq5du4bw8HDcvXsX7u7uLMCIiMowFmBUKj766CNcvHgRCxYswOHDh5Gbm4tXr17h6NGjOHr0KObOnYsmTZpg2bJl6NWrl3hcZmamuF10xj5F9LV4aGlN/1ySdoq2VVRiYiIGDhyI3NxcsQeiXr16SuNL65pK8jO79uQ1/n1defEllS8A3+37G7UcOF12ZVXcXtJq1aqhUaNG4qyFgPLCqkuXLoXW3lMW5+Ligm3btmHIkCHIzs5GfHw84uPjC8XUqlULoaGh4vIDypiYmGDRokWYNGkScnNz5QqlunXryq3fFxISAk9PT/zf//0f3r17J/5/VRlXV1eVORARkWFxIWYtvX//HhEREZg7dy66d+8OV1dXmJubw9LSErVr10afPn3w66+/iguYVibNmjVDaGgoXr58iaNHj2LevHnw9fUV15G6fv06AgICCk3TLDvNsrIFmmVlZWWVWr6qFlCVzSsyMhJCwfOSGn3J9orItuPn56dVO4IgKC2oXr16hb59+4o9U2vWrEHHjh1VXq80l3r16mmdh3Ta7qLXpO3PbG/cY7XFl1S+AKwIT9QsmCqUnbEPMXJ9jMLiC/hfL+mu2EcK98sWUu7u7kqHUhYtuGR/z4vq168fLl++jDFjxqB27dqoUqUKqlatilatWiE4OBjx8fEar3E2ceJEHDt2DIGBgXB1dVW7MLV0KGtSUhKCg4Ph6+sLFxcXmJqawtzcHK6urujWrRvmzZuH6OhojYY6ExGR4bAHTAtr1qzBnDlzxCFlRUkXHD5y5AgWLFiAFStWYOTIkXrO0vBsbGzQq1cvsafr5cuXWLx4Mf79739DEAR89dVXGDJkCIyNjVGzZk3xuDt37sDT01Npu4Ig4N69eyrPLXsjo27BXumaXIrUqlVL3L5x44bS4U7q2NnZwdraGpmZmbh58yYEQYBEIilWW1Lv37/HwIEDxWe4Zs2ahdGjR6s9rlatWnj16hUePXqE9PR02NraFuv8RX9m6sjG/Pd5OszraH6umKRUfORcsveLypeoOykqh6hKyfaSFu0J+/333/H777+rPVdgYKBWwzYbNmyIjRs3qozRdGhqz5490bNnT43PDZRsKKv0gxciIjI89oBp4fLly2LxJZFI0KhRI4wfPx7z589HcHAwRowYId7Uvnr1CqNGjcJvv/1myJTLhKpVq2L58uXig+MvXrxAYmJBz4a3t7cYFxERobKd2NhYtRNwyC7ULPusUlFpaWlISEhQul+24AoNDVV5TnU6d+4MoOC6L1y4UKK2gIIFmcPDwwEUfCq/ZMkSjY6TXlNeXh4OHjxY7PNr8zN79+4doqKiin0uALj+pGTDOKl8+TU8kb2kRERUobEA05KzszPmz5+Pu3fv4saNG1i7di0WLFiAefPmYcuWLXjw4AH69esnxn/99ddqZ5urLGSH00kfeu/Tp4/YSxUSElJosoeifv75Z7XnkJ1pTVVxsHLlSuTl5Snd37p1azRu3BgAcOrUKZUP1asj2zs1Z84cledVZ8WKFVi1ahWAgiGf27ZtUzvTpNSoUaPE7eDg4GIP56xXrx5atGgBALh06VKhGSOL2rBhg8qfqSbe5hT//aLyJSE5Q+mwQ2ViklKRkJyho4yIiIhKHwswLYwbNw7379/HggULCk0lLMve3h67d+9G06ZNARQMg1uzZo0+09S748eP49dff1U5ycOdO3fEIsba2hru7u4ACgpaaWGQkpKC4cOH4+3bt3LH//7779i1a5faXGQn+Fi6dKnCm/+jR49i0aJFKtuRSCRYunSp+H1QUBCOHz+u8phbt25h8uTJcq8PHDhQfDbk7NmzGD58ODIylN8wZmdnY9OmTdixY0eh148fP46vvvoKAFC9enUcOHBAq0lJ2rZti08++QQAkJCQgL59+yI5OVlp/Pv37xEWFoaVK1fK7Zs5c6a4PWrUKDx48EAuJjo6GrNmzdI4P2VatPMRn0VbsGBBidurzGSfZyypkJAQtc8qaivqjvJhwbo4joiIyBAkAgeF68Tq1avFm/H27duXytAz6TTucXFxJW5LU4qmgPaqXnia85CQEHz66acwNTVFly5d0LZtW9SvXx+WlpZISUlBbGwsdu3aJfa4fP/994UKoNTUVDRr1gyPHz8GANSvXx9jx46Fh4cH0tLSsG/fPpw8eRJubm6ws7MTp6FX9qvbuXNnsVemXr16+Oyzz1C/fn2kpqbi2LFj2L9/P7y8vGBpaYkrV66obEs6vFSqU6dO6N27N+rWrQsTExOkpqbixo0bOHPmDK5duwZjY2Oxd0/Wo0eP0L59e3G9MAcHBwQFBaFVq1awt7fHmzdv8OjRI8TFxeHkyZPIzMzEokWL8P3334tt2NnZiUMwZ82apXDNqKJ69OhRaKbC9PR0dOzYUZz5zdLSEgMHDkT79u3h5OSE7OxsPHv2DJcvX8aJEyeQmpoqrmFV1EcffYTDhw8DAGxtbTFu3Di0bt0aubm5OHv2LDZv3gwjIyN069YNR44cAQBUH7oE5nWaqs1b1okvO8v9zlHF9Ft4In4+qXxosDIzu3thmr/y50eJiEh7hrjvrCw4CYeOyPaQKZu0oyzTZgpo6RC4nJwcHD9+XGlPkUQiwfTp07Fw4cJCrzs6OiI8PBw9e/bE/fv3ce/evUKFB1AwrXJoaChmzJihNvdNmzahS5cuePDgAe7fv4/vvvuu0P4PPvgABw8exIQJE9S2tXDhQtSuXRszZ85Eeno6zp07p3LInbLpn2vXro3Y2FiMHDkS4eHhSEtLw+rVq5W2Y2xsDBcXl0KvyT7/9q9//Utt7gCQlJRUqHfC1tYWUVFRmDBhAnbu3Ik3b95g8+bN2Lx5s9I2ZCfdkLVz5070798fJ0+eRHp6Ov79738X2m9ubo5169YhMTFRLMAauNjivkaZF2jr5sjiqxKxNi/eP0nFPY6IiMgQOARRR6Q9HUDBMLvyRNspoEeOHIn4+HgsX74c/fr1g4eHB6ysrGBsbAw7Ozs0b94cU6dORVxcHH755ReFzyx5eXnh+vXrWLx4MZo3bw5ra2vY2NigSZMmmDdvHq5cuYJmzZpplL+bmxuuXLmCOXPmoFGjRrCwsICtrS1atGiBpUuX4tKlSypnWyxq/PjxePDgAX7++Wf06NEDNWvWhJmZGczMzODi4oLOnTvjm2++QXh4uMpZGmvUqIFTp04hMjISkyZNQuPGjWFvbw9jY2PY2tqiUaNGCAoKwqpVq/Do0SOMHz9e4xy1YWNjgx07duDy5cv44osv0KJFC1StWhUmJiawtraGp6cnAgMDsXz5cty9e7dQD6AsKysrHD9+HJs3b0aXLl3g6OgIc3Nz1K9fHxMmTMClS5fkFuX9pJUrjDSc1NBIAkxnr0alomhdL10eR0REZAgcgqgjvXr1EnuCvvvuu0LPExWXPrqCo+6kYOT6GI1mITOSAFvGtdXrzY+fnx/OnDkDQPmwQSrbdsY+VDvNuJEE+HFAUwxuo3j9Jqq4Bv8ZrdVEHG3dHLFzUnsdZkREVDlxCKLusAdMB06ePCkWXyYmJhg7dqyBM9Icp4AmXQtqUwdbxrVFWzdHhfvbujliy7i2LL4qqRn+nuwlJSKiCo0D50vZ8+fPCxVc06ZN02q4G/C/TxyKunXrFho2bFii/FQpyRTQfE6HtOHj4QQfDyeNJnmhysXHwwlLB3yocS8phx8SEVF5U2EKsNmzZ5d4wVxZCxcuRFBQkFbHvH37FoGBgeJsfs2bNy+VoYf6UpIpoHnTTMXhVd2GvzskJ6hNHbg6WGJFeCJiFHwo1NbNEdNlJgIiIiIqTypMAfbs2bNSXfBY28Vjc3JyMHDgQMTExAD436x9ZmZmWp9b2VhbZT1jpSUzW376dF0eR0SkDHtJiYiooqowBZgh5ebmYvDgweJU2zVq1EB4eHipLU6qL5wCmojKGvaSEhFRRVNh7pxDQkIQEhKi9/Pm5uYiKCgI+/fvBwC4uLjg9OnT8PLy0nsuJVUepoCOjIzU27mIiIiIiEobZ0EsAWnPl/TZs+rVqyMiIgIffPCBgTMrHq/qNvBWMjOdMlwol4iIiIhIcyzAiklafIWFhQEAqlWrhoiICJ3OUqgPnAKaiIiIiEh3WIAVQ9Hiy9nZGeHh4WjUqJFhEysF0img1RVhnAKaiIiIiEh7FeYZMH3Jzc3FoEGDxGe+nJ2dERERgSZNmhg4s9LDKaCJiIiIiHSDBZgWKkPxJcUpoImIiIiISh8LMC1MmTJFLL4AICAgAMeOHcOxY8fUHjtx4kTY2trqMj2d4BTQRERERESlhwWYFhISEgp9v2nTJo2PHThwYLkswIiIiIiIqPRwEg4iIiIiIiI9YQ+YFrgIMBERERERlQR7wIiIiIiIiPSEBRgREREREZGesAAjIiIiIiLSExZgREREREREesICjIiIiIiISE9YgBEREREREekJCzAiIiIiIiI9YQFGRERERESkJxJBEARDJ0GacXR0RHZ2Nho2bGjoVIiIiIioArt16xbMzc2Rmppq6FQqHBNDJ0Cas7W1NXQKVAK3bt0CABbQRODfA5EU/xaorDI3N+e9p46wB4xIT1q1agUAiIuLM3AmRIbHvweiAvxbIKp8+AwYERERERGRnrAAIyIiIiIi0hMWYERERERERHrCAoyIiIiIiEhPWIARERERERHpCWdBJCIiIiIi0hP2gBEREREREekJCzAiIiIiIiI9YQFGRERERESkJyzAiIiIiIiI9IQFGBERERERkZ6wACMiIiIiItITFmBERERERER6YmLoBIgIeP/+Pc6ePYvw8HBcvHgRt27dQkpKCoyMjFC1alU0bdoUPXr0wOjRo2Fvb2/odImK5d69e1i3bh2OHDmChw8fIjs7GzVq1EDbtm0xYsQIBAQEGDpFIp0SBAExMTE4deoULly4gJs3b+LFixfIz8+Hg4MDGjdujK5du2Ls2LFwcXExdLpEpCNciJnIwNasWYM5c+bg5cuXamPt7e2xYsUKjBw5Ug+ZEZWelStX4uuvv8bbt2+VxgQGBmLTpk2wtbXVY2ZE+nHgwAFMnjwZT58+VRtrbm6ORYsW4euvv9ZDZkSkb+wBIzKwy5cvi8WXRCJBw4YN0aFDB9SqVQvGxsZISEjAgQMHkJ6ejlevXmHUqFF49eoVpk2bZuDMiTSzevVqTJkyRfy+adOm6NWrFywtLREfH4+DBw/i/fv3CAsLQ//+/XH06FGYmpoaMGOi0nfz5s1CxZe7uzt8fHxQp04dWFhYICkpCQcPHkRycjKys7PxzTff4NmzZ/j5558NmDUR6QILMKIywNnZGZ9//jlGjx4NNzc3uf2vXr3CmDFjsH//fgDA119/jR49euCDDz7Qd6pEWrl79y5mzJghfr906VJ89913hWKuXLmC3r17Izk5GREREfj5558xe/ZsfadKpHM2NjaYMGECPv30UzRp0kRuf3Z2Nr744gv8+eefAIDly5fjo48+QpcuXfSdKhHpEIcgEhlYbGwsGjduDEtLS5Vxubm5aN26Nf7++28AwFdffcVPRqnMGz58OP7zn/8AAIYNG4Zt27YpjDtx4gR69uwJALC1tcWDBw/4vCNVKH///TdcXV3h6OioNjYgIABHjx4FAAwYMAB79+7VdXpEpEecBZHIwNq0aaO2+AKAKlWqYPLkyeL30dHRukyLqMQyMzOxb98+AAXDa3/44QelsT169EC7du0AAOnp6QgLC9NHikR607RpU42KLwCYPn26uM3/1xNVPCzAiMoR2eGJmkzaQWRIJ06cQHZ2NoCCm091Q2YHDRokboeGhuo0N6KyjP+vJ6rYWIARlSNPnjwRt52dnQ2YCZF6ly9fFrc7duyoNr5Tp07i9pUrV3SSE1F5wP/XE1VsLMCIypFdu3aJ27I3q0Rl0Y0bN8RtT09PtfEeHh7i9qNHj5Cenq6TvIjKOv6/nqhiYwFGVE6cPHkSx48fBwCYmJhg7NixBs6ISLXnz5+L266urmrjHRwcYGVlJX6fnJysk7yIyrIbN25g48aN4vcTJ040YDZEpAsswIjKgefPnxcquKZNm6ZRjwKRIWVkZIjbsoWVKrIT0sgeT1QZZGVlYcSIEcjJyQEA9O/fn1PQE1VAXAeMSIXZs2eX6mQACxcuRFBQkFbHvH37FoGBgXj8+DEAoHnz5li6dGmp5USkK2/fvhW3NV1Y2dzcXNx+8+ZNqedEVFbl5+dj5MiRuHr1KoCCXuO1a9caNiki0gkWYEQqPHv2DLdv3y619tLS0rSKz8nJwcCBAxETEwOg4B/k0NBQmJmZlVpORLpiYWEhbks/0VdHOmsiAI2WZyCqCARBwKRJk8QP/GxtbREWFoaqVasaODMi0gUOQSQqo3JzczF48GAcOXIEAFCjRg2Eh4ejXr16hk2MSEM2NjbidlZWlkbHyPZ6yR5PVJFNmzYN69atA1Dwe3/s2DG0atXKwFkRka6wB4xIhZCQEISEhOj9vLm5uQgKCsL+/fsBAC4uLjh9+jS8vLz0ngtRcbm4uIjbstNqK/P69etChVq1atV0khdRWTJt2jT88ccfAABra2scPXoU7du3N3BWRKRL7AEjKmOkPV/SoSjVq1dHRESE2kVsicqaRo0aidsJCQlq4xMTE8VtV1dX2NnZ6SQvorJi2rRp+P333wEUTFRz5MgR+Pj4GDgrItI19oARlSHS4issLAxAQQ9AREQEGjZsaNjEiIqhZcuW4nZUVJTa+HPnzonbLVq00ElORGXF1KlTxZ4vS0tLHD58mGt+EVUS7AEjKiOKFl/Ozs4IDw8v1ItAVJ707NlTnNUwPj5ebS/Ynj17xO3+/fvrNDciQ1JUfPn6+ho4KyLSFxZgRGVAbm4uBg0aVKj4ioiIQJMmTQybGFEJWFtbIzAwEEDBLG+LFi1SGnvq1ClcuHABQMEkBNLjiCqaKVOmFCq+Dh06BD8/P8MmRUR6xQKMyMCkxZd0wg0WX1SRLFq0CFWqVAEAbN26FcuWLZOLiY+Px6hRo8TvZ82aBQcHB73lSKQvU6ZMwcqVKwH8r/jiQstElY9EEATB0EkQVWYTJ04stNjm6NGjNS6+Jk6cCFtbW12lRlQq/vjjD0ydOlX8vlmzZujVqxcsLS0RHx+PgwcPIjc3FwDg6+uL48ePc607qnCWLFmCuXPnit/37dsXnTt31ujYoKAg1K5dW1epEZGesQAjMjA/Pz+cOXOmWMcmJSVxXTAqF3777Td8++23hRZaLqpv377YsmULZz+kCmnMmDHYtGlTsY49ffo0hykSVSAcgkhERDo3bdo0XL9+HbNmzULTpk1hb28Pc3Nz1KtXD0FBQTh06BAOHDjA4ouIiCo89oARERERERHpCXvAiIiIiIiI9IQFGBERERERkZ6wACMiIiIiItITFmBERERERER6wgKMiIiIiIhIT1iAERERERER6QkLMCIiIiIiIj1hAUZERERERKQnLMCIiIiIiIj0hAUYERERERGRnrAAIyIiIiIi0hMWYERERERERHrCAoyIiIiIiEhPWIARERERERHpCQswIiIiIiIiPWEBRkREREREpCcswIiIiIiIiPSEBRgREREREZGesAAjIiIiIiLSExZgREREREREesICjIiIiIiISE9YgBEREREREekJCzAiIiIiIiI9YQFGRERERESkJ/8fHpr4VuRJM18AAAAASUVORK5CYII=&quot;  style=&quot;max-width: 100%; max-height: 100%; object-fit: contain; width: 864px; height: 576px;&quot;&gt;&lt;/img&gt;\"}}]}}]}},{\"type\":\"object\",\"name\":\"panel.models.comm_manager.CommManager\",\"id\":\"feef3280-6e33-4c4e-bedf-0cd7f1bcac80\",\"attributes\":{\"plot_id\":\"90920467-d36f-4899-8f12-2d496e2ce38f\",\"comm_id\":\"fff249f013ed48ef9f8af8d21fb1329b\",\"client_comm_id\":\"aaf963baff594275bcd4bc5d0d2bb854\"}}],\"defs\":[{\"type\":\"model\",\"name\":\"ReactiveHTML1\"},{\"type\":\"model\",\"name\":\"FlexBox1\",\"properties\":[{\"name\":\"align_content\",\"kind\":\"Any\",\"default\":\"flex-start\"},{\"name\":\"align_items\",\"kind\":\"Any\",\"default\":\"flex-start\"},{\"name\":\"flex_direction\",\"kind\":\"Any\",\"default\":\"row\"},{\"name\":\"flex_wrap\",\"kind\":\"Any\",\"default\":\"wrap\"},{\"name\":\"justify_content\",\"kind\":\"Any\",\"default\":\"flex-start\"}]},{\"type\":\"model\",\"name\":\"FloatPanel1\",\"properties\":[{\"name\":\"config\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}},{\"name\":\"contained\",\"kind\":\"Any\",\"default\":true},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"right-top\"},{\"name\":\"offsetx\",\"kind\":\"Any\",\"default\":null},{\"name\":\"offsety\",\"kind\":\"Any\",\"default\":null},{\"name\":\"theme\",\"kind\":\"Any\",\"default\":\"primary\"},{\"name\":\"status\",\"kind\":\"Any\",\"default\":\"normalized\"}]},{\"type\":\"model\",\"name\":\"GridStack1\",\"properties\":[{\"name\":\"mode\",\"kind\":\"Any\",\"default\":\"warn\"},{\"name\":\"ncols\",\"kind\":\"Any\",\"default\":null},{\"name\":\"nrows\",\"kind\":\"Any\",\"default\":null},{\"name\":\"allow_resize\",\"kind\":\"Any\",\"default\":true},{\"name\":\"allow_drag\",\"kind\":\"Any\",\"default\":true},{\"name\":\"state\",\"kind\":\"Any\",\"default\":[]}]},{\"type\":\"model\",\"name\":\"drag1\",\"properties\":[{\"name\":\"slider_width\",\"kind\":\"Any\",\"default\":5},{\"name\":\"slider_color\",\"kind\":\"Any\",\"default\":\"black\"},{\"name\":\"value\",\"kind\":\"Any\",\"default\":50}]},{\"type\":\"model\",\"name\":\"click1\",\"properties\":[{\"name\":\"terminal_output\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"debug_name\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"clears\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"copy_to_clipboard1\",\"properties\":[{\"name\":\"fill\",\"kind\":\"Any\",\"default\":\"none\"},{\"name\":\"value\",\"kind\":\"Any\",\"default\":null}]},{\"type\":\"model\",\"name\":\"FastWrapper1\",\"properties\":[{\"name\":\"object\",\"kind\":\"Any\",\"default\":null},{\"name\":\"style\",\"kind\":\"Any\",\"default\":null}]},{\"type\":\"model\",\"name\":\"NotificationAreaBase1\",\"properties\":[{\"name\":\"js_events\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"bottom-right\"},{\"name\":\"_clear\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"NotificationArea1\",\"properties\":[{\"name\":\"js_events\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}},{\"name\":\"notifications\",\"kind\":\"Any\",\"default\":[]},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"bottom-right\"},{\"name\":\"_clear\",\"kind\":\"Any\",\"default\":0},{\"name\":\"types\",\"kind\":\"Any\",\"default\":[{\"type\":\"map\",\"entries\":[[\"type\",\"warning\"],[\"background\",\"#ffc107\"],[\"icon\",{\"type\":\"map\",\"entries\":[[\"className\",\"fas fa-exclamation-triangle\"],[\"tagName\",\"i\"],[\"color\",\"white\"]]}]]},{\"type\":\"map\",\"entries\":[[\"type\",\"info\"],[\"background\",\"#007bff\"],[\"icon\",{\"type\":\"map\",\"entries\":[[\"className\",\"fas fa-info-circle\"],[\"tagName\",\"i\"],[\"color\",\"white\"]]}]]}]}]},{\"type\":\"model\",\"name\":\"Notification\",\"properties\":[{\"name\":\"background\",\"kind\":\"Any\",\"default\":null},{\"name\":\"duration\",\"kind\":\"Any\",\"default\":3000},{\"name\":\"icon\",\"kind\":\"Any\",\"default\":null},{\"name\":\"message\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"notification_type\",\"kind\":\"Any\",\"default\":null},{\"name\":\"_destroyed\",\"kind\":\"Any\",\"default\":false}]},{\"type\":\"model\",\"name\":\"TemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"BootstrapTemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"MaterialTemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]}]}};\n",
       "  var render_items = [{\"docid\":\"0372bee8-4baf-4349-9fbe-772561ed740a\",\"roots\":{\"90920467-d36f-4899-8f12-2d496e2ce38f\":\"dcf22597-1e5b-4e89-82db-88909f8750d8\"},\"root_ids\":[\"90920467-d36f-4899-8f12-2d496e2ce38f\"]}];\n",
       "  var docs = Object.values(docs_json)\n",
       "  if (!docs) {\n",
       "    return\n",
       "  }\n",
       "  const py_version = docs[0].version.replace('rc', '-rc.').replace('.dev', '-dev.')\n",
       "  function embed_document(root) {\n",
       "    var Bokeh = get_bokeh(root)\n",
       "    Bokeh.embed.embed_items_notebook(docs_json, render_items);\n",
       "    for (const render_item of render_items) {\n",
       "      for (const root_id of render_item.root_ids) {\n",
       "\tconst id_el = document.getElementById(root_id)\n",
       "\tif (id_el.children.length && (id_el.children[0].className === 'bk-root')) {\n",
       "\t  const root_el = id_el.children[0]\n",
       "\t  root_el.id = root_el.id + '-rendered'\n",
       "\t}\n",
       "      }\n",
       "    }\n",
       "  }\n",
       "  function get_bokeh(root) {\n",
       "    if (root.Bokeh === undefined) {\n",
       "      return null\n",
       "    } else if (root.Bokeh.version !== py_version) {\n",
       "      if (root.Bokeh.versions === undefined || !root.Bokeh.versions.has(py_version)) {\n",
       "\treturn null\n",
       "      }\n",
       "      return root.Bokeh.versions.get(py_version);\n",
       "    } else if (root.Bokeh.version === py_version) {\n",
       "      return root.Bokeh\n",
       "    }\n",
       "    return null\n",
       "  }\n",
       "  function is_loaded(root) {\n",
       "    var Bokeh = get_bokeh(root)\n",
       "    return (Bokeh != null && Bokeh.Panel !== undefined)\n",
       "  }\n",
       "  if (is_loaded(root)) {\n",
       "    embed_document(root);\n",
       "  } else {\n",
       "    var attempts = 0;\n",
       "    var timer = setInterval(function(root) {\n",
       "      if (is_loaded(root)) {\n",
       "        clearInterval(timer);\n",
       "        embed_document(root);\n",
       "      } else if (document.readyState == \"complete\") {\n",
       "        attempts++;\n",
       "        if (attempts > 200) {\n",
       "          clearInterval(timer);\n",
       "\t  var Bokeh = get_bokeh(root)\n",
       "\t  if (Bokeh == null || Bokeh.Panel == null) {\n",
       "            console.warn(\"Panel: ERROR: Unable to run Panel code because Bokeh or Panel library is missing\");\n",
       "\t  } else {\n",
       "\t    console.warn(\"Panel: WARNING: Attempting to render but not all required libraries could be resolved.\")\n",
       "\t    embed_document(root)\n",
       "\t  }\n",
       "        }\n",
       "      }\n",
       "    }, 25, root)\n",
       "  }\n",
       "})(window);</script>"
      ],
      "text/plain": [
       "Column\n",
       "    [0] Column\n",
       "        [0] IntSlider(end=1000, step=100)\n",
       "    [1] Row\n",
       "        [0] Matplotlib(Figure, name='interactive00118')"
      ]
     },
     "execution_count": 36,
     "metadata": {
      "application/vnd.holoviews_exec.v0+json": {
       "id": "90920467-d36f-4899-8f12-2d496e2ce38f"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import panel as pn\n",
    "from panel import widgets\n",
    "from panel.interact import interact\n",
    "import matplotlib\n",
    "\n",
    "pn.extension()\n",
    "\n",
    "def f(n_epochs):\n",
    "    fig = plt.figure(figsize=(6, 4))    \n",
    "    return plot_embeddings(fig, n_epochs, model, idx_pairs, vocab, word2idx)\n",
    "\n",
    "#interact(f, angle=widgets.IntSlider(start=0, end=182, step=1, value=0))\n",
    "interact(f, n_epochs=widgets.IntSlider(start=0, end=1000, step=100, value=0))#.embed(max_opts=200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Skip-gram before the weights are trained**\n",
    "<img src=\"img/word2vec-toy-epochs0.png\" width=\"1000\" height=\"1000\">\n",
    "\n",
    "<!-- <center>\n",
    "<img src=\"img/word2vec-toy-epochs0.png\" width=\"1000\" height=\"1000\">\n",
    "</center>     -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "**Skip-gram after 900 epochs**\n",
    "\n",
    "<img src=\"img/word2vec-toy-epochs900.png\" width=\"1000\" height=\"1000\">\n",
    "\n",
    "<!-- <center>\n",
    "<img src=\"img/word2vec-toy-epochs900.png\" width=\"1000\" height=\"1000\">\n",
    "</center>     -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 4.2 Training word2vec embeddings\n",
    "\n",
    "- [Original C code](https://code.google.com/archive/p/word2vec/) \n",
    "- [GitHub version of the code](https://github.com/tmikolov/word2vec)\n",
    "- [Gensim](https://radimrehurek.com/gensim/), an open source Python library that provides a Python interface for word2vec family of algorithms\n",
    "\n",
    "You can install it as follows: \n",
    "\n",
    "```\n",
    "> conda activate 563\n",
    "> conda install conda-forge::gensim\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['cat', 'say', 'meow'], ['dog', 'say', 'woof']]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences = [[\"cat\", \"say\", \"meow\"], [\"dog\", \"say\", \"woof\"]]\n",
    "sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need a corpus to be preprocessed and stored as \n",
    "- A list of lists. \n",
    "- Each inner list corresponds to a sequence of tokens from a sentence. In other words, each sentence is tokenized and stored as a list of tokens. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the word vector of the word _cat_. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models import Word2Vec\n",
    "\n",
    "model = Word2Vec(sentences, vector_size=10, window=5, min_count=1) # train word2vec model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.0960355 ,  0.05007293, -0.08759586, -0.04391825, -0.000351  ,\n",
       "       -0.00296181, -0.0766124 ,  0.09614743,  0.04982058,  0.09233143],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv[\"cat\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What's the most similar word to the word _cat_? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('say', -0.22418658435344696),\n",
       " ('dog', -0.3207966387271881),\n",
       " ('meow', -0.36627137660980225),\n",
       " ('woof', -0.5381841659545898)]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.wv.most_similar(\"cat\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try it out on our small wiki corpus. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "709"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Machu Picchu is a 15th-century Inca citadel located in the Eastern Cordillera of southern Peru on a 2,430-meter (7,970 ft) mountain ridge.',\n",
       " 'Often referred to as the \"Lost City of the Incas\", it is the most familiar icon of the Inca Empire.',\n",
       " 'It is located in the Machupicchu District within Urubamba Province above the Sacred Valley, which is 80 kilometers (50 mi) northwest of Cusco.',\n",
       " 'The Urubamba River flows past it, cutting through the Cordillera and creating a canyon with a tropical mountain climate.']"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "sents = MyPreprocessor(corpus)\n",
    "wiki_model = Word2Vec(sents, vector_size=100, window=5, min_count=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.00726288, -0.00653676, -0.00611133,  0.00270797,  0.0062954 ,\n",
       "       -0.01246749,  0.00360739,  0.01890011, -0.00831644, -0.00525167],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wiki_model.wv[\"tree\"][0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('ft', 0.5620871186256409),\n",
       " ('redwood', 0.5433863401412964),\n",
       " ('state', 0.49551403522491455),\n",
       " ('trees', 0.49252673983573914),\n",
       " ('picchu', 0.4924555718898773),\n",
       " ('parliament', 0.4915319085121155),\n",
       " ('provinces', 0.4913058280944824),\n",
       " ('constitution', 0.4784410297870636),\n",
       " ('president', 0.45981258153915405),\n",
       " ('also', 0.4576851427555084)]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wiki_model.wv.most_similar(\"tree\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- This is good. But if you want good and meaningful representations of words you need to train models on a large corpus such as the whole Wikipedia, which is computationally intensive. \n",
    "- So instead of training our own models, we use the **pre-trained embeddings**. These are the word embeddings people have trained embeddings on huge corpora and made them available for us to use. \n",
    "\n",
    "Let's try out Google news pre-trained word vectors. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 4.3 Pre-trained embeddings\n",
    "\n",
    "- Training embeddings is computationally expensive\n",
    "- For typical corpora, the vocabulary size is greater than 100,000.  \n",
    "- If the size of embeddings is 300, the number of parameters of the model is $2 \\times 30,000,000$. \n",
    "- So people have trained embeddings on huge corpora and made them available.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "A number of pre-trained word embeddings are available. The most popular ones are:  \n",
    "\n",
    "- [word2vec](https://code.google.com/archive/p/word2vec/)\n",
    "    * trained on several corpora using the word2vec algorithm \n",
    "- [wikipedia2vec](https://wikipedia2vec.github.io/wikipedia2vec/pretrained/)\n",
    "    * pretrained embeddings for 12 languages \n",
    "- [GloVe](https://nlp.stanford.edu/projects/glove/)\n",
    "    * trained using [the GloVe algorithm](https://nlp.stanford.edu/pubs/glove.pdf) \n",
    "    * published by Stanford University \n",
    "- [fastText pre-trained embeddings for 294 languages](https://fasttext.cc/docs/en/pretrained-vectors.html) \n",
    "    * trained using [the fastText algorithm](http://aclweb.org/anthology/Q17-1010)\n",
    "    * published by Facebook    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try Google News pre-trained embeddings.  \n",
    "\n",
    "- You can download pre-trained embeddings from their original source. \n",
    "- `Gensim` provides an api to conveniently load them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "google_news_vectors = api.load(\"word2vec-google-news-300\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 4.4 Success of word2vec\n",
    "\n",
    "- This analogy example often comes up when people talk about word2vec, which was used by the authors of this method. \n",
    "- **MAN : KING :: WOMAN : ?**\n",
    "    - What is the word that is similar to **WOMAN** in the same sense as **KING** is similar to **MAN**?\n",
    "- Perform a simple algebraic operations with the vector representation of words.\n",
    "    $\\vec{X} = \\vec{\\text{KING}} − \\vec{\\text{MAN}} + \\vec{\\text{WOMAN}}$\n",
    "- Search in the vector space for the word closest to $\\vec{X}$ measured by cosine distance.\n",
    "\n",
    "<!-- ![](img/word_analogies1.png) -->\n",
    "\n",
    "<center>\n",
    "<img src=\"img/word_analogies1.png\" width=\"400\" height=\"400\">\n",
    "</center>\n",
    "    \n",
    "(Credit: Mikolov et al. 2013)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def analogy(word1, word2, word3, model=google_news_vectors):\n",
    "    \"\"\"\n",
    "    Returns analogy word using the given model.\n",
    "\n",
    "    Parameters\n",
    "    --------------\n",
    "    word1 : (str)\n",
    "        word1 in the analogy relation\n",
    "    word2 : (str)\n",
    "        word2 in the analogy relation\n",
    "    word3 : (str)\n",
    "        word3 in the analogy relation\n",
    "    model :\n",
    "        word embedding model\n",
    "\n",
    "    Returns\n",
    "    ---------------\n",
    "        pd.dataframe\n",
    "    \"\"\"\n",
    "    print(\"%s : %s :: %s : ?\" % (word1, word2, word3))\n",
    "    sim_words = model.most_similar(positive=[word3, word2], negative=[word1])\n",
    "    return pd.DataFrame(sim_words, columns=[\"Analogy word\", \"Score\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "man : king :: woman : ?\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Analogy word</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>queen</td>\n",
       "      <td>0.711819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>monarch</td>\n",
       "      <td>0.618967</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>princess</td>\n",
       "      <td>0.590243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>crown_prince</td>\n",
       "      <td>0.549946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>prince</td>\n",
       "      <td>0.537732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>kings</td>\n",
       "      <td>0.523684</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Queen_Consort</td>\n",
       "      <td>0.523595</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>queens</td>\n",
       "      <td>0.518113</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>sultan</td>\n",
       "      <td>0.509859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>monarchy</td>\n",
       "      <td>0.508741</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Analogy word     Score\n",
       "0  queen          0.711819\n",
       "1  monarch        0.618967\n",
       "2  princess       0.590243\n",
       "3  crown_prince   0.549946\n",
       "4  prince         0.537732\n",
       "5  kings          0.523684\n",
       "6  Queen_Consort  0.523595\n",
       "7  queens         0.518113\n",
       "8  sultan         0.509859\n",
       "9  monarchy       0.508741"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogy(\"man\", \"king\", \"woman\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Montreal : Canadiens :: Vancouver : ?\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Analogy word</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Canucks</td>\n",
       "      <td>0.821327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Vancouver_Canucks</td>\n",
       "      <td>0.750401</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Calgary_Flames</td>\n",
       "      <td>0.705471</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Leafs</td>\n",
       "      <td>0.695783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Maple_Leafs</td>\n",
       "      <td>0.691617</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Thrashers</td>\n",
       "      <td>0.687503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Avs</td>\n",
       "      <td>0.681716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Sabres</td>\n",
       "      <td>0.665307</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Blackhawks</td>\n",
       "      <td>0.664625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Habs</td>\n",
       "      <td>0.661023</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Analogy word     Score\n",
       "0  Canucks            0.821327\n",
       "1  Vancouver_Canucks  0.750401\n",
       "2  Calgary_Flames     0.705471\n",
       "3  Leafs              0.695783\n",
       "4  Maple_Leafs        0.691617\n",
       "5  Thrashers          0.687503\n",
       "6  Avs                0.681716\n",
       "7  Sabres             0.665307\n",
       "8  Blackhawks         0.664625\n",
       "9  Habs               0.661023"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogy(\"Montreal\", \"Canadiens\", \"Vancouver\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Toronto : UofT :: Vancouver : ?\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Analogy word</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>SFU</td>\n",
       "      <td>0.579245</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>UVic</td>\n",
       "      <td>0.576921</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>UBC</td>\n",
       "      <td>0.571431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Simon_Fraser</td>\n",
       "      <td>0.543464</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Langara_College</td>\n",
       "      <td>0.541347</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>UVIC</td>\n",
       "      <td>0.520495</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Grant_MacEwan</td>\n",
       "      <td>0.517273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>UFV</td>\n",
       "      <td>0.514150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Ubyssey</td>\n",
       "      <td>0.510421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Kwantlen</td>\n",
       "      <td>0.503807</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Analogy word     Score\n",
       "0  SFU              0.579245\n",
       "1  UVic             0.576921\n",
       "2  UBC              0.571431\n",
       "3  Simon_Fraser     0.543464\n",
       "4  Langara_College  0.541347\n",
       "5  UVIC             0.520495\n",
       "6  Grant_MacEwan    0.517273\n",
       "7  UFV              0.514150\n",
       "8  Ubyssey          0.510421\n",
       "9  Kwantlen         0.503807"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogy(\"Toronto\", \"UofT\", \"Vancouver\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gauss : mathematician :: Bob_Dylan : ?\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Analogy word</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>singer_songwriter_Bob_Dylan</td>\n",
       "      <td>0.520782</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>poet</td>\n",
       "      <td>0.501191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Pete_Seeger</td>\n",
       "      <td>0.497143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Joan_Baez</td>\n",
       "      <td>0.492307</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>sitarist_Ravi_Shankar</td>\n",
       "      <td>0.491968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>bluesman</td>\n",
       "      <td>0.490930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>jazz_musician</td>\n",
       "      <td>0.489593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Joni_Mitchell</td>\n",
       "      <td>0.487740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Billie_Holiday</td>\n",
       "      <td>0.486664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Johnny_Cash</td>\n",
       "      <td>0.485722</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  Analogy word     Score\n",
       "0  singer_songwriter_Bob_Dylan  0.520782\n",
       "1  poet                         0.501191\n",
       "2  Pete_Seeger                  0.497143\n",
       "3  Joan_Baez                    0.492307\n",
       "4  sitarist_Ravi_Shankar        0.491968\n",
       "5  bluesman                     0.490930\n",
       "6  jazz_musician                0.489593\n",
       "7  Joni_Mitchell                0.487740\n",
       "8  Billie_Holiday               0.486664\n",
       "9  Johnny_Cash                  0.485722"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogy(\"Gauss\", \"mathematician\", \"Bob_Dylan\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "USA : pizza :: India : ?\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Analogy word</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>vada_pav</td>\n",
       "      <td>0.554463</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>jalebi</td>\n",
       "      <td>0.547090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>idlis</td>\n",
       "      <td>0.540039</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>pav_bhaji</td>\n",
       "      <td>0.526046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>dosas</td>\n",
       "      <td>0.521772</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>samosa</td>\n",
       "      <td>0.520700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>idli</td>\n",
       "      <td>0.516858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>pizzas</td>\n",
       "      <td>0.516199</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>tiffin</td>\n",
       "      <td>0.514347</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>chaat</td>\n",
       "      <td>0.508534</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Analogy word     Score\n",
       "0  vada_pav     0.554463\n",
       "1  jalebi       0.547090\n",
       "2  idlis        0.540039\n",
       "3  pav_bhaji    0.526046\n",
       "4  dosas        0.521772\n",
       "5  samosa       0.520700\n",
       "6  idli         0.516858\n",
       "7  pizzas       0.516199\n",
       "8  tiffin       0.514347\n",
       "9  chaat        0.508534"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogy(\"USA\", \"pizza\", \"India\") # Just for fun"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So you can imagine these models being useful in many meaning-related tasks. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/word2vec-country-capitals.png)\n",
    "\n",
    "(Credit: Mikolov et al. 2013)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Examples of semantic and syntactic relationships**\n",
    "\n",
    "\n",
    "<img src=\"img/word_analogies2.png\" width=\"800\" height=\"800\">\n",
    "\n",
    "(Credit: Mikolov 2013)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.5 Implicit biases and stereotypes in word embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "man : computer_programmer :: woman : ?\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Analogy word</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>homemaker</td>\n",
       "      <td>0.562712</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>housewife</td>\n",
       "      <td>0.510505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>graphic_designer</td>\n",
       "      <td>0.505180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>schoolteacher</td>\n",
       "      <td>0.497949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>businesswoman</td>\n",
       "      <td>0.493489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>paralegal</td>\n",
       "      <td>0.492551</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>registered_nurse</td>\n",
       "      <td>0.490797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>saleswoman</td>\n",
       "      <td>0.488163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>electrical_engineer</td>\n",
       "      <td>0.479773</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>mechanical_engineer</td>\n",
       "      <td>0.475540</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Analogy word     Score\n",
       "0  homemaker            0.562712\n",
       "1  housewife            0.510505\n",
       "2  graphic_designer     0.505180\n",
       "3  schoolteacher        0.497949\n",
       "4  businesswoman        0.493489\n",
       "5  paralegal            0.492551\n",
       "6  registered_nurse     0.490797\n",
       "7  saleswoman           0.488163\n",
       "8  electrical_engineer  0.479773\n",
       "9  mechanical_engineer  0.475540"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogy(\"man\", \"computer_programmer\", \"woman\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/eva-srsly.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Embeddings reflect gender stereotypes present in broader society.\n",
    "- They may also amplify these stereotypes because of their widespread usage. \n",
    "- See the paper [Man is to Computer Programmer as Woman is to ...](http://papers.nips.cc/paper/6228-man-is-to-computer-programmer-as-woman-is-to-homemaker-debiasing-word-embeddings.pdf)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the modern embeddings are de-biased for some obvious biases. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 4.6 Other popular methods to get embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**[fastText](https://fasttext.cc/)**\n",
    "\n",
    "- NLP library by Facebook research  \n",
    "- Includes an algorithm which is an extension to word2vec\n",
    "- Helps deal with unknown words elegantly\n",
    "- Breaks words into several n-gram subwords \n",
    "- Example: trigram sub-words for *berry* are *ber*, *err*, *rry*\n",
    "- Embedding(*berry*) = embedding(*ber*) + embedding(*err*) + embedding(rry)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "**(Optional) [GloVe: Global Vectors for Word Representation](https://nlp.stanford.edu/projects/glove/)**\n",
    "- Starts with the co-occurrence matrix\n",
    "    - Co-occurrence can be interpreted as an indicator of semantic proximity of words\n",
    "- Takes advantage of global count statistics    \n",
    "- Predicts co-occurrence ratios\n",
    "- Loss based on word frequency"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ❓❓ Questions for you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 5.2 Select all of the following statements which are **True** (iClicker)\n",
    "\n",
    "- (A) Suppose you learn word embeddings for a vocabulary of 20,000 words using word2vec. Then each dense word embedding associated with a word is of size 20,000 to make sure that we capture the full range of meaning of that word.\n",
    "- (B) If you try to get word vector for a word outside the vocabulary of your word2vec model, it will throw an error. \n",
    "- (C) A model trained with fastText is likely to contain representation for words even when they are not in the vocabulary. \n",
    "- (D) Suppose I have a HUGE corpus of biomedical text, which contains many domain-specific words and abbreviations. It makes sense to train our own embeddings to get word representations of the words in the corpus rather than using pre-trained Google News embeddings.  \n",
    "- (E) Suppose my corpus gets updated very often, and the new text tends to contain many unknown words. It makes sense to either train fastText or use pre-trained embeddings trained using fastText."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{admonition} V's Solutions!\n",
    ":class: tip, dropdown\n",
    "- B, C, D, E\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Final comments, summary, and reflection "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Word embeddings \n",
    "\n",
    "- Word embeddings give a representation of individual words in a vector space so that \n",
    "    * Distance between words in this vector space indicate the relationship between them. \n",
    "- Word representations\n",
    "    * One-hot encoding \n",
    "        - not able to capture similarities between words \n",
    "    * Term-term co-occurrence matrix\n",
    "        - long and sparse representations\n",
    "    * word2vec\n",
    "        - short and dense representations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### word2vec\n",
    "\n",
    "- word2vec is a recent alternative to LSA. \n",
    "- One of the most widely used and influential algorithms in the last decade. \n",
    "- Skipgram model predicts surrounding words (context words) given a target word. \n",
    "- We are not interested in the prediction task itself but we are interested in the learned weights which we use as word embeddings.  \n",
    "- Freely available code and pre-trained models. \n",
    "- Generally do not train these models yourself. It's very likely that there are pre-trained embedddings available for the domain of your interest. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Pre-trained embeddings \n",
    " \n",
    "- Available for many different languages in a variety of domains.  \n",
    "- Finally beware of biases and stereotypes encoded in these word embeddings! \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Resources\n",
    "\n",
    "- [Embedding Projector](https://projector.tensorflow.org/)\n",
    "- [Word embedding workshop by Rachel Thomas](https://www.youtube.com/watch?v=25nC0n9ERq4)\n",
    "- [Distributed representations of words and phrases and their compositionality](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)\n",
    "- [Efficient estimation of word representations in vector space](https://arxiv.org/pdf/1301.3781.pdf)\n",
    "- [Linguistic regularities in continuous space word representations](https://www.aclweb.org/anthology/N13-1090)\n",
    "- [Enriching Word Vectors with Subword Information](http://aclweb.org/anthology/Q17-1010)\n",
    "\n",
    "## Fun tools\n",
    "[wevi: word embedding visual inspector](https://ronxin.github.io/wevi/)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [conda env:563]",
   "language": "python",
   "name": "conda-env-563-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}