8.0-using-word-embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 單詞嵌入(word embeddings)\n",
    "\n",
    "![word-embeddings](http://techblog.gumgum.com/media/288/download/word-emb-pipeline-1024x321.png?v=1)\n",
    "\n",
    "將向量與單詞關聯的另一種流行且強大的方式是使用密集的“單詞向量(word vectors)”，也稱為“單詞嵌入(word embeddings)”。\n",
    "\n",
    "雖然通過one-hot編碼獲得的向量是二元，稀疏（大部分由零構成）而且維度會變成非常的高（與詞彙表中單詞的數量相同, 也就是如果有1000個不重覆的單詞, 那每個單詞就會變成1000維的向量），然而“單詞嵌入(word embeddings)”卻可以用低維度的浮點向量\n",
    "（即“密集”向量，而不是稀疏向量）來表示相同的一個單詞。\n",
    "\n",
    "與透過one-hot編碼獲得的單詞向量不同地方在於“單詞嵌入(word embeddings)”是從數據中學習得來的。通常在處理非常大的詞彙表時，我們通常會看到256維，512維或1024維的“單詞嵌入(word embeddings)”層。\n",
    "\n",
    "另一方面，one-hot編碼單詞通常導致20,000維或更高維的向量（假設我們需要處理 20,000個詞彙的詞彙表的任務）。因此從某種角度來說，“單詞嵌入(word embeddings)”可以用將更多信息包裝到更少的維度中。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![word embeddings vs. one hot encoding](https://s3.amazonaws.com/book.keras.io/img/ch6/word_embeddings.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這裡有兩種獲得“單詞嵌入(word embeddings)”的方法：\n",
    "\n",
    "*把“單詞嵌入(word embeddings)”與我們關心的主要任務模型（例如文檔分類或情感預測）一起設計在模型的結構中一起訓練共同學習。 在這樣的設計裡，我們將從隨機詞向量開始，然後學習我們的詞向量(word vector)，就像學習神經網絡的權重一樣。\n",
    "\n",
    "*加載預訓練好的“單詞嵌入(word embeddings)”到訓練模型中，這種嵌入(embeddings)是使用不同於我們正在嘗試解決的機器學習任務的機器學習任務所預先計算與訓練出來的。\n",
    "這些被稱為“預先訓練的詞嵌入”。\n",
    "\n",
    "我們來看看兩者。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 利用嵌入層(embedding layer)來學習單詞嵌入(word embeddings)\n",
    "\n",
    "將單詞與密集向量相關聯的最簡單方法是選擇一種隨機向量(vector at random)。這種方法的問題是，這樣的嵌入空間(embedding space)將沒有什麼結構的關聯性：例如，\"accurate\"和\"exact\"這兩個字可能會是完全不同的嵌入(embeddings)，深度神經網絡很難理解這樣一個嘈雜的，非結構化的嵌入空間。\n",
    "\n",
    "為了得到更抽象一點：不同詞向量(word vectors)之間的幾何關係應該反映這些詞之間的語義關係。單詞嵌入(word embeddings)意味著將人類語言映射到幾何空間(geometric space)。例如，在一個合理的嵌入空間中，我們期望\n",
    "同義詞被嵌入到相似的單詞向量中，並且通常我們會期望任意兩個之間的幾何距離（例如L2距離）\n",
    "單詞向量與相關單詞的語義距離相關（單詞意味著非常不同的事物將嵌入到點中\n",
    "彼此相距很遠，而相關的詞會更接近）。即使超出了距離，我們可能會希望在嵌入空間中__方向__\n",
    "是有意義的。\n",
    "\n",
    "在現實世界的詞嵌入空間中，有意義的幾何變換的常見例子是“性別向量(gender vectors)”和“複數向量(plural vector)”。例如，通過給“國王(king)”向量添加“女性(female)”向量，會獲得“女王(queen)”向量。通過添加“複數(plural)矢量”，可以獲得“國王們(kings)”。\n",
    "字嵌入空間通常具有數千個這樣的可解釋和潛在有用的向量。\n",
    "\n",
    "![word2vec](http://ox5l2b8f4.bkt.clouddn.com/images/%E8%AF%8D%E5%90%91%E9%87%8F%E6%80%BB%E7%BB%93%E7%AC%94%E8%AE%B0%EF%BC%88%E7%AE%80%E6%B4%81%E7%89%88%EF%BC%89/3.jpg)\n",
    "\n",
    "是否有一些“完美”的單詞嵌入空間可以完美地映射人類語言，並可用於任何自然語言處理\n",
    "任務？ 或許吧，但無論如何，我們還沒有訓練出這樣類型的東西出來。另外，不同的語言有著不同的結構，因為語言是特定文化和特定背景的反映。\n",
    "\n",
    "那麼從實務的角度來說，到底什麼樣的詞嵌入空間才算的上是一個好的詞嵌入？ 它很大程度上取決於我們想要進行的任務：一個“電影評論情感分析模型”的的詞嵌入空間可能與“法律文件分類模型”的詞嵌入空間看起來有很大不同，因為某些語義關係的重要性會因任務或要解決的問題而有所差異。\n",
    "\n",
    "為每個新任務學習一個新的詞嵌入空間是合理的。值得慶幸的是，反向傳播(backpropagation)使得這件事非常地簡單，而Keras讓這件事再更容易了, 因為keras讓單詞嵌入(word embeddings)的訓練變成學習神經圖層的權重。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.1.2'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Embedding\n",
    "\n",
    "# 嵌入層( Embedding layer)的構建至少需要兩個參數：\n",
    "# 可能的符標(token)的數量，這裡是 1000(1 + maximum word index),\n",
    "# 和嵌入(embedding)的維度，這裡是 64。\n",
    "\n",
    "embedding_layer = Embedding(1000, 64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "嵌入層(embedding layer)最好的理解是: 將整數索引（代表特定單詞）映射到密集向量的字典物件中。輸入整數，它查找一個內部字典，並返回相關的向量。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "嵌入層以`(samples, sequence_length)`的整數2D張量作為輸入，其中每筆資料是一個整數序列（sequence of integers)。它也可以嵌入不同長度的序列，例如，我們可以將這些序列輸入到可能具有形狀（32,10）（一批32個長度10）或（64,15）的批次之上的嵌入層中（一批64個長度15）。雖然批處理中的所有序列必須具有相同的長度（因為我們需要將它們打包為單個張量），所以比其他序列短的序列應該用零填充，並且應該截斷較長的序列。\n",
    "\n",
    "這個嵌入層會返回一個形狀為`（samples，sequence_length，embedding_dimensionality）`的3D浮點張量。然後這樣的3D張量由RNN層或1D卷積層處理。\n",
    "\n",
    "當我們實例化一個`Embedding`層的時候，它的權重（它的內部符標向量字典）最初是隨機的，就像任何其他神經層。在訓練過程中，這些單詞向量將通過反向傳播逐漸調整，將嵌入空間構建成一些下游模式可以利用的東西。一旦經過充分訓練，我們的嵌入空間將顯示出許多結構(一種專門針對我們正在訓練模型的特定問題結構)。\n",
    "\n",
    "讓我們將這個想法應用於IMDB電影評論情緒預測任務。讓我們快速準備數據。我們會將電影評論限制在最常見的10,000個單詞中，並且僅在20個單詞後刪減評論。我們的網絡將簡單地學習10,000個單詞中的每一個的8維嵌入，將輸入整數序列（2D整數張量）轉化為嵌入序列（3D浮點張量），將張量平坦化為2D，並訓練一個單獨的`Dense`層來進行分類。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import imdb\n",
    "from keras import preprocessing\n",
    "\n",
    "# 要考慮作為特徵的單詞數\n",
    "max_features = 10000\n",
    "\n",
    "# 在此單詞數量之後剪切文本\n",
    "maxlen = 20\n",
    "\n",
    "# 將數據加載為整數列表\n",
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n",
    "\n",
    "# 這將我們的整數列表變成一個2D整個張量 (samples, maxlen)\n",
    "x_train = preprocessing.sequence.pad_sequences(x_train, maxlen=maxlen) # 不夠長的補'0', 太長的裁剪掉\n",
    "x_test = preprocessing.sequence.pad_sequences(x_test, maxlen=maxlen) # 不夠長的補'0', 太長的裁剪掉"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 20)\n",
      "(25000, 20)\n"
     ]
    }
   ],
   "source": [
    "print(x_train.shape)\n",
    "print(x_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_2 (Embedding)      (None, 20, 8)             80000     \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 160)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 161       \n",
      "=================================================================\n",
      "Total params: 80,161\n",
      "Trainable params: 80,161\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 2s 86us/step - loss: 0.6759 - acc: 0.6049 - val_loss: 0.6398 - val_acc: 0.6810\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 1s 71us/step - loss: 0.5657 - acc: 0.7427 - val_loss: 0.5467 - val_acc: 0.7204\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 1s 72us/step - loss: 0.4752 - acc: 0.7808 - val_loss: 0.5113 - val_acc: 0.7384\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 1s 71us/step - loss: 0.4263 - acc: 0.8078 - val_loss: 0.5008 - val_acc: 0.7454\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 2s 77us/step - loss: 0.3930 - acc: 0.8257 - val_loss: 0.4981 - val_acc: 0.7538\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 1s 72us/step - loss: 0.3668 - acc: 0.8396 - val_loss: 0.5014 - val_acc: 0.7532\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 1s 73us/step - loss: 0.3435 - acc: 0.8532 - val_loss: 0.5052 - val_acc: 0.7522\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 1s 72us/step - loss: 0.3223 - acc: 0.8655 - val_loss: 0.5132 - val_acc: 0.7486\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 1s 74us/step - loss: 0.3023 - acc: 0.8765 - val_loss: 0.5213 - val_acc: 0.7490\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 1s 74us/step - loss: 0.2839 - acc: 0.8861 - val_loss: 0.5302 - val_acc: 0.7464\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Flatten, Dense\n",
    "\n",
    "model = Sequential()\n",
    "\n",
    "# 我們為嵌入層指定最大輸入長度，以便稍後將嵌入式輸入平坦化\n",
    "# 參數：\n",
    "#      符標(token)的數量，這裡是 1000\n",
    "#      嵌入(embedding)的維度，這裡是 8\n",
    "model.add(Embedding(10000, 8, input_length=maxlen))\n",
    "\n",
    "# 在嵌入層之後，我們的張量形狀轉換成為 `(samples, maxlen, 8)`.\n",
    "\n",
    "# 我們將3D嵌入張量變成2D張量形狀 `(samples, maxlen * 8)`\n",
    "model.add(Flatten())\n",
    "\n",
    "# 我們添加一個二元分類層\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])\n",
    "model.summary()\n",
    "\n",
    "history = model.fit(x_train, y_train,\n",
    "                    epochs=10,\n",
    "                    batch_size=32,\n",
    "                    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們的驗證準確性達到了76％，考慮到我們只查看每篇評論的前20個單詞，這樣的結果算是相當不錯的了。但要注意的是，僅僅將嵌入的序列扁平展開並在頂部訓練單個`Dense`神經層會產生出一種模型，該模型將單獨處理輸入序列中的每個單詞，而不考慮單詞間關係和語句結構。在嵌入序列之上添加循環圖層(recurrent layer)或一維卷積圖層(1D convolutional layer)來學習將每個序列作為一個整體考慮在內的特徵會更好。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用預先訓練的詞嵌入(pre-trained word embeddings)\n",
    "\n",
    "有時候，我們只有很少的訓練數據可供使用，因此無法單獨使用我們的數據來學習適合特定任務的詞彙嵌入。那該怎麼辦？\n",
    "\n",
    "我們可以加載預先訓練好的詞嵌入(pre-trained word embeddings)。在自然語言處理中使用預先訓練的單詞嵌入背後的原理與在圖像分類中使用預先訓練的模型非常相似：我們沒有足夠的數據來自己學習真正強大的模型，但我們需要相當通用的特徵，即常見的視覺特徵或語義特徵。在這種情況下，重覆使用在不同問題上學習到的特徵是有意義的。\n",
    "\n",
    "一般都會使用詞語出現（word occurrency)統計量來計算這種詞嵌入，但也有人用神經網絡來訓練詞嵌入。 最著名和成功的詞嵌入方案之一：2013年由Mikolov在谷歌開發的Word2Vec算法。\n",
    "\n",
    "我們可以使用Keras幾種己經預先訓練好字詞嵌入模型，並在載入到Keras`Embedding`層開始使用。 Word2Vec就是其中之一。另一個流行的詞嵌入是2014年由斯坦福大學研究人員開發的名為“GloVe”的詞嵌入。它是一種基於對詞共現統計矩陣進行因式分解的嵌入技術。它的開發者從Kikipedia或Common Crawl數據獲得的數百萬英文符標產出預先訓練的詞嵌入。\n",
    "\n",
    "讓我們來看看如何開始在Keras模型中使用GloVe嵌入。對於Word2Vec嵌入或任何可以下載的其他單詞嵌入數據庫，同樣的方法當然是有效的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 從原始文本到單詞嵌入(word embeddings)\n",
    "\n",
    "我們將使用類似於我們剛剛練習的模型-將句子透過嵌入層的轉換成向量序列，將它們壓扁打平並在頂部訓練一個`Dense`層來進行分類。但是這一次我們會使用預先訓練的詞嵌入來完成它，而不是使用Keras中打包的預標記IMDB數據，我們將從頭開始下載原始文本數據。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 下載IMDB數據\n",
    "\n",
    "首先，前往`http://ai.stanford.edu/~amaas/data/sentiment/`並下載原始IMDB數據集（如果URL不再有效，只需去Google\"IMDB dataset\"）。解壓縮它。\n",
    "\n",
    "現在讓我們將各個訓練用的評論放到一個字符串列表中，每個評論一個字符串，然後把評論標籤(positive / negative)放入一個`labels`列表中："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "imdb_dir = 'data/aclImdb' # IMDB解壓後的路徑\n",
    "train_dir = os.path.join(imdb_dir, 'train') # 訓練資料的路徑\n",
    "\n",
    "labels = []\n",
    "texts = []\n",
    "\n",
    "# 迭代檔案目錄來產生訓練資料\n",
    "for label_type in ['neg', 'pos']:\n",
    "    dir_name = os.path.join(train_dir, label_type)\n",
    "    for fname in os.listdir(dir_name):\n",
    "        if fname[-4:] == '.txt':\n",
    "            f = open(os.path.join(dir_name, fname),encoding='utf8')\n",
    "            texts.append(f.read())\n",
    "            f.close()\n",
    "            if label_type == 'neg':\n",
    "                labels.append(0)\n",
    "            else:\n",
    "                labels.append(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000\n",
      "25000\n"
     ]
    }
   ],
   "source": [
    "print(len(labels))\n",
    "print(len(texts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 對文本數據進行標記(tokenize)\n",
    "\n",
    "讓我們將我們收集的文本進行向量化，並準備一個訓練和驗證的分割。因為預先訓練的詞嵌入對於只有很少的訓練數據可用的問題特別有用（否則，針對特定任務所訓練的詞嵌入表現可能會優於它們），我們將添加以下內容：我們將訓練數據限制在前200樣本。因此，我們將在學習僅僅200個例子後，對電影評論進行分類。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 88582 unique tokens.\n",
      "Shape of data tensor: (25000, 100)\n",
      "Shape of label tensor: (25000,)\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "import numpy as np\n",
    "\n",
    "maxlen = 100  # 每一筆評論我們將裁減成100個字\n",
    "training_samples = 200  # 我們將使用200個樣本進行訓練\n",
    "validation_samples = 10000  # 我們用10000個樣本來進行驗證\n",
    "max_words = 10000  # 我們只考慮數據集中前10,000個單詞\n",
    "\n",
    "tokenizer = Tokenizer(num_words=max_words)\n",
    "tokenizer.fit_on_texts(texts)\n",
    "sequences = tokenizer.texts_to_sequences(texts)\n",
    "\n",
    "word_index = tokenizer.word_index\n",
    "print('Found %s unique tokens.' % len(word_index))\n",
    "\n",
    "data = pad_sequences(sequences, maxlen=maxlen)\n",
    "\n",
    "labels = np.asarray(labels)\n",
    "print('Shape of data tensor:', data.shape)\n",
    "print('Shape of label tensor:', labels.shape)\n",
    "\n",
    "# 將數據分成訓練集和驗證集\n",
    "\n",
    "# 但首先，進行數據順序洗牌\n",
    "indices = np.arange(data.shape[0])\n",
    "np.random.shuffle(indices)\n",
    "data = data[indices]\n",
    "labels = labels[indices]\n",
    "\n",
    "x_train = data[:training_samples]\n",
    "y_train = labels[:training_samples]\n",
    "x_val = data[training_samples: training_samples + validation_samples]\n",
    "y_val = labels[training_samples: training_samples + validation_samples]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 下載GloVe單詞嵌入(word embeddings)\n",
    "\n",
    "前往`https://nlp.stanford.edu/projects/glove/`（你可以在這裡了解更多關於GloVe算法的資訊）。它是一個名為`glove.6B.zip`的822MB zip文件，包含400,000個字的100維嵌入向量。解壓縮它。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 單詞嵌入(word embeddings)預處理\n",
    "\n",
    "讓我們解析解壓後的文件（它是一個`txt`文件）來構建一個索引映射詞（作為字符串）到它們的向量表示（作為數字向量）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 400000 word vectors.\n"
     ]
    }
   ],
   "source": [
    "glove_dir = 'model/glove' # GloVe解壓後的路徑\n",
    "\n",
    "embeddings_index = {}\n",
    "f = open(os.path.join(glove_dir, 'glove.6B.100d.txt'), encoding='utf8')\n",
    "\n",
    "for line in f:\n",
    "    values = line.split()\n",
    "    word = values[0]\n",
    "    coefs = np.asarray(values[1:], dtype='float32')\n",
    "    embeddings_index[word] = coefs\n",
    "f.close()\n",
    "\n",
    "print('Found %s word vectors.' % len(embeddings_index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "現在讓我們建立一個嵌入矩陣，我們將能夠加載進到一個Keras的`Embedding`層。它必須是一個numpy的2D矩陣`(max_words, embedding_dim)`，其中每個條目`i`包含我們引用詞索引（在標記化期間構建）中索引'i'的單詞的嵌入_dim維向量。請注意，索引'0'不應該代表任何單詞或標記 - 它是一個佔位符。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_dim = 100 # 代表每一個單詞會用一個100維的向量來表示\n",
    "\n",
    "embedding_matrix = np.zeros((max_words, embedding_dim))\n",
    "for word, i in word_index.items():\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if i < max_words:\n",
    "        if embedding_vector is not None:\n",
    "            # 在嵌入索引中找不到的詞的向量裡的值將全是零。\n",
    "            embedding_matrix[i] = embedding_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 定義一個模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_3 (Embedding)      (None, 100, 100)          1000000   \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 10000)             0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 32)                320032    \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 1,320,065\n",
      "Trainable params: 1,320,065\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Embedding, Flatten, Dense\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_words, embedding_dim, input_length=maxlen))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 將GloVe嵌入加載到模型中\n",
    "\n",
    "嵌入層具有單一權重矩陣：2D浮點數矩陣，其中每個條目“i”是與索引“i”相關聯的單詞向量。讓我們將我們準備好的GloVe矩陣加載到我們的模型中的`Embedding`層："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.layers[0].set_weights([embedding_matrix]) # 載入嵌入層權重矩陣\n",
    "model.layers[0].trainable = False # 設定這個嵌入層不可以被訓練"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們凍結嵌入層（我們將它的`trainable`屬性設置為`False`）： 模型訓練的過程中不應該更新己經預先訓練好的部分，以避免忘記已經知道的部分。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 訓練和評估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 200 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "200/200 [==============================] - 1s 6ms/step - loss: 1.6335 - acc: 0.5250 - val_loss: 0.7301 - val_acc: 0.5077\n",
      "Epoch 2/10\n",
      "200/200 [==============================] - 1s 5ms/step - loss: 0.8242 - acc: 0.6350 - val_loss: 1.2378 - val_acc: 0.5074\n",
      "Epoch 3/10\n",
      "200/200 [==============================] - 1s 5ms/step - loss: 0.4997 - acc: 0.7550 - val_loss: 2.0810 - val_acc: 0.4936\n",
      "Epoch 4/10\n",
      "200/200 [==============================] - 1s 5ms/step - loss: 0.7441 - acc: 0.7550 - val_loss: 0.6971 - val_acc: 0.5699\n",
      "Epoch 5/10\n",
      "200/200 [==============================] - 1s 5ms/step - loss: 0.3083 - acc: 0.8600 - val_loss: 0.6890 - val_acc: 0.5692\n",
      "Epoch 6/10\n",
      "200/200 [==============================] - 1s 5ms/step - loss: 0.1583 - acc: 0.9900 - val_loss: 1.0931 - val_acc: 0.5082\n",
      "Epoch 7/10\n",
      "200/200 [==============================] - 1s 5ms/step - loss: 0.1409 - acc: 0.9550 - val_loss: 1.0605 - val_acc: 0.5142\n",
      "Epoch 8/10\n",
      "200/200 [==============================] - 1s 5ms/step - loss: 0.0569 - acc: 1.0000 - val_loss: 0.8752 - val_acc: 0.5516\n",
      "Epoch 9/10\n",
      "200/200 [==============================] - 1s 5ms/step - loss: 0.0278 - acc: 1.0000 - val_loss: 0.9471 - val_acc: 0.5572\n",
      "Epoch 10/10\n",
      "200/200 [==============================] - 1s 5ms/step - loss: 0.0267 - acc: 1.0000 - val_loss: 3.0603 - val_acc: 0.5073\n"
     ]
    }
   ],
   "source": [
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['acc'])\n",
    "history = model.fit(x_train, y_train,\n",
    "                    epochs=10,\n",
    "                    batch_size=32,\n",
    "                    validation_data=(x_val, y_val))\n",
    "model.save_weights('pre_trained_glove_model.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "讓我們繪製不同訓練週期的表現："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x27f8c15f400>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x27f90b80940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "plt.plot(epochs, acc, label='Training acc')\n",
    "plt.plot(epochs, val_acc, label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, label='Training loss')\n",
    "plt.plot(epochs, val_loss, label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "毫不意外的是，由於訓練樣本很少，這個模型很快就開始過度擬合。由於相同的原因，驗證準確度也有很大的差距。\n",
    "\n",
    "請注意，每個人看到的結果可能會有所不同：由於我們的培訓樣本數量非常少，因此性能高度依賴於我們挑選的那200個樣本，因為我們隨機挑選了它們。\n",
    "\n",
    "我們還可以嘗試訓練相同的模型，而不加載預先訓練的詞嵌入並且不凍結嵌入層。在這種情況下，我們將學習特定嵌入，當有大量數據可用時，它通常比預先訓練的字詞嵌入更強大。但在我們的案例中，我們只有200個培訓樣本。讓我們試試看："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_4 (Embedding)      (None, 100, 100)          1000000   \n",
      "_________________________________________________________________\n",
      "flatten_3 (Flatten)          (None, 10000)             0         \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 32)                320032    \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 1,320,065\n",
      "Trainable params: 1,320,065\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Train on 200 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "200/200 [==============================] - 2s 8ms/step - loss: 0.6952 - acc: 0.4400 - val_loss: 0.6949 - val_acc: 0.5157\n",
      "Epoch 2/10\n",
      "200/200 [==============================] - 1s 6ms/step - loss: 0.5022 - acc: 0.9800 - val_loss: 0.7054 - val_acc: 0.5065\n",
      "Epoch 3/10\n",
      "200/200 [==============================] - 1s 6ms/step - loss: 0.2890 - acc: 0.9850 - val_loss: 0.7013 - val_acc: 0.5203\n",
      "Epoch 4/10\n",
      "200/200 [==============================] - 1s 6ms/step - loss: 0.1180 - acc: 1.0000 - val_loss: 0.7169 - val_acc: 0.5149\n",
      "Epoch 5/10\n",
      "200/200 [==============================] - 1s 6ms/step - loss: 0.0523 - acc: 1.0000 - val_loss: 0.7147 - val_acc: 0.5283\n",
      "Epoch 6/10\n",
      "200/200 [==============================] - 1s 6ms/step - loss: 0.0261 - acc: 1.0000 - val_loss: 0.7249 - val_acc: 0.5270\n",
      "Epoch 7/10\n",
      "200/200 [==============================] - 1s 6ms/step - loss: 0.0141 - acc: 1.0000 - val_loss: 0.7209 - val_acc: 0.5383\n",
      "Epoch 8/10\n",
      "200/200 [==============================] - 1s 6ms/step - loss: 0.0082 - acc: 1.0000 - val_loss: 0.7390 - val_acc: 0.5267\n",
      "Epoch 9/10\n",
      "200/200 [==============================] - 1s 6ms/step - loss: 0.0049 - acc: 1.0000 - val_loss: 0.7281 - val_acc: 0.5384\n",
      "Epoch 10/10\n",
      "200/200 [==============================] - 1s 6ms/step - loss: 0.0030 - acc: 1.0000 - val_loss: 0.7474 - val_acc: 0.5319\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Embedding, Flatten, Dense\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_words, embedding_dim, input_length=maxlen))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.summary()\n",
    "\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['acc'])\n",
    "history = model.fit(x_train, y_train,\n",
    "                    epochs=10,\n",
    "                    batch_size=32,\n",
    "                    validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xt8XHWd//HXJ7emuTRpkzQtSe+UXqgFSiko4eKiLMiliChUWQVEBC2IP/2tiPwAEVxXXNYLLAuiqGtt7YIIuAWU2hXwAi2XFmgTepWGZtI0bZNJmnu+vz/OmWQyzWWSJp3MzPv5eMwjc/nOmc+cJO858z3f8z3mnENERBJLSqwLEBGR4adwFxFJQAp3EZEEpHAXEUlACncRkQSkcBcRSUAK9wRmZqlm1mBmU4ezbSyZ2bFmNuzjd83sQ2a2K+x2hZmdEU3bIbzWI2Z261CfLxKNtFgXIN3MrCHsZhbQAnT4tz/vnFsxmOU55zqAnOFumwycc3OGYzlmdi1wpXPu7LBlXzscyxbpj8J9FHHOdYWrv2V4rXPu+b7am1mac679aNQmMhD9PY4u6paJI2Z2t5n92sxWmlkQuNLM3m9mfzOzg2ZWZWY/NLN0v32amTkzm+7f/qX/+DNmFjSzv5rZjMG29R8/38zeMbM6M/uRmf3ZzK7qo+5oavy8mW0zswNm9sOw56aa2b+bWa2ZbQfO62f93GZmqyLue8DM7vOvX2tmW/z3s93fqu5rWZVmdrZ/PcvM/suv7W3g5F5ed4e/3LfN7GL//vcB9wNn+F1e+8LW7Z1hz7/ef++1ZvZbM5sczboZzHoO1WNmz5vZfjMLmNk/h73O//PXSb2ZbTCzY3rrAjOzl0K/Z399vuC/zn7gNjObbWbr/Peyz19veWHPn+a/xxr/8R+YWaZf87ywdpPN7JCZFfT1fmUAzjldRuEF2AV8KOK+u4FW4CK8D+axwCnAqXjfwmYC7wDL/fZpgAOm+7d/CewDFgPpwK+BXw6h7UQgCCz1H/s/QBtwVR/vJZoanwTygOnA/tB7B5YDbwOlQAHwgvdn2+vrzAQagOywZe8FFvu3L/LbGPAPQBOw0H/sQ8CusGVVAmf7178H/C8wHpgGbI5o+wlgsv87+aRfQ7H/2LXA/0bU+UvgTv/6uX6NJwKZwH8Af4xm3QxyPecB1cCXgDHAOGCJ/9jXgY3AbP89nAhMAI6NXNfAS6Hfs//e2oEbgFS8v8fjgHOADP/v5M/A98Lez1v++sz225/uP/YwcE/Y63wFeCLW/4fxfIl5Abr08YvpO9z/OMDzvgr8t3+9t8D+z7C2FwNvDaHtNcCLYY8ZUEUf4R5ljaeFPf4b4Kv+9RfwuqdCj30kMnAilv034JP+9fOBd/pp+zvgi/71/sL93fDfBfCF8La9LPct4AL/+kDh/nPg22GPjcPbz1I60LoZ5Hr+J2BDH+22h+qNuD+acN8xQA2XAev962cAASC1l3anAzsB82+/AVw63P9XyXRRt0z82R1+w8zmmtn/+F+z64G7gMJ+nh8Iu36I/nei9tX2mPA6nPffWNnXQqKsMarXAv7eT70AvwKW+dc/CXTthDazC83sZb9b4iDeVnN/6ypkcn81mNlVZrbR71o4CMyNcrngvb+u5Tnn6oEDQElYm6h+ZwOs5ynAtj5qmIIX8EMR+fc4ycxWm9l7fg0/i6hhl/N23vfgnPsz3reAMjNbAEwF/meINQnqc49HkcMAH8LbUjzWOTcOuB1vS3okVeFtWQJgZkbPMIp0JDVW4YVCyEBDNX8NfMjMSvG6jX7l1zgWeAz4F7wuk3zg91HWEeirBjObCTyI1zVR4C+3PGy5Aw3b3IPX1RNaXi5e9897UdQVqb/1vBuY1cfz+nqs0a8pK+y+SRFtIt/fv+KN8nqfX8NVETVMM7PUPur4BXAl3reM1c65lj7aSRQU7vEvF6gDGv0dUp8/Cq/5O2CRmV1kZml4/bhFI1TjauBmMyvxd659rb/GzrlqvK6DR4EK59xW/6ExeP3ANUCHmV2I1zccbQ23mlm+eccBLA97LAcv4GrwPueuxdtyD6kGSsN3bEZYCXzWzBaa2Ri8D58XnXN9fhPqR3/r+SlgqpktN7MMMxtnZkv8xx4B7jazWeY50cwm4H2oBfB23Kea2XWEfRD1U0MjUGdmU/C6hkL+CtQC3zZvJ/VYMzs97PH/wuvG+SRe0MsRULjHv68An8HbwfkQ3pbriPID9HLgPrx/1lnA63hbbMNd44PAWuBNYD3e1vdAfoXXh/6rsJoPAl8GnsDbKXkZ3odUNO7A+waxC3iGsOBxzm0Cfgi84reZC7wc9tw/AFuBajML714JPf9ZvO6TJ/znTwU+FWVdkfpcz865OuDDwMfwduC+A5zlP3wv8Fu89VyPt3Mz0+9u+xxwK97O9WMj3ltv7gCW4H3IPAU8HlZDO3AhMA9vK/5dvN9D6PFdeL/nVufcXwb53iVCaOeFyJD5X7P3AJc5516MdT0Sv8zsF3g7ae+MdS3xTgcxyZCY2Xl4X7Ob8YbSteNtvYoMib//YinwvljXkgjULSNDVQbswPu6fh5wiXaAyVCZ2b/gjbX/tnPu3VjXkwjULSMikoC05S4ikoBi1udeWFjopk+fHquXFxGJS6+++uo+51x/Q4+BGIb79OnT2bBhQ6xeXkQkLpnZQEdpA+qWERFJSAp3EZEEpHAXEUlACncRkQSkcBcRSUADhruZ/dTM9prZW308bv5ptraZ2SYzWzT8ZYqIyGBEs+X+M/o5byXe2W5m+5fr8GbxExGRGBpwnLtz7gXzT5rch6XAL/zpQf/mz3k92TlXNUw1yijV2t7J32sb2V7TwPaaRlraDjvBjoj04px5xZwwJX9EX2M4DmIqoeeptir9+w4Ld3+y/+sApk4d6IQ6MlocPNTqBfjeUJB7Yf7u/kN0dHbPTWQjff4nkQQxcVxmXIR7b//Svc5G5px7GO9EACxevFgzlo0iHZ2OPQeb2FbTwPa9Xnhv96/XNrZ2tctIS2FmYTbzJ4/jooWTmTUxh1lFOcwozCZ7jGaQFhkthuO/sZKe55csxTtxg4xCh1rb2VHT3ZUSCvCd+xppae/sajchO4NZRdl8eH4xs4pymDUxm2OLcikZP5bUFG2ii4x2wxHuTwHLzWwVcCpQp/722HLOUdPQclg3yva9Dbx3sKmrXYrBlAlZzCrK4czjiphVlM2sohxmFuUwITsjhu9ARI7UgOFuZiuBs4FCM6vEO0diOoBz7j+BNcBHgG3AIeDqkSpWvOBuae+koaWdhuZ2Glraee9g02F94sHm9q7nZGWkMqsoh1Omj+eKoildXSnTCrLITO/rRPQiEs+iGS2zbIDHHfDFYasoATnnaGrr6ArjxpYOgi1tNLZ00NDSRkNLB41hYd3Y4v0Mv+619a6H78QMVzxuDLOKcrjkxBJvK9wP8cl5mZj2dookFe0BG6LtNQ0893aAhmYvgIN+EHvBHbruBXZjazt95HEPKQbZY9LI8S+h6xNzM/3rqeRkph3WpnhcJrOKssnNTB/5Ny4icUHhPkT/+kw5v99cTVqKeYGb4QduZhp5Y9Mpyc/sEdB9Xs9MI3tMKjlj0hibnqotbBEZFgr3IdoSqOeC903m/k+epEAWkVFHE4cNQUNLO7v3NzF3Uq6CXURGJYX7EFQEggDMnTwuxpWIiPRO4T4EXeE+KTfGlYiI9E7hPgTlgXpyxqRRkj821qWIiPRK4T4E5YEgxxXnkKLD8EVklFK4D5JzjvKqevW3i8iopnAfpEB9M/XN7epvF5FRTeE+SOX+ztQ5xQp3ERm9FO6DVF4VGimjbhkRGb0U7oNUEahncl4meVmax0VERi+F+yCVB4LqbxeRUU/hPgit7Z1sr2lgjrpkRGSUU7gPwo59DbR1OG25i8iop3AfhO45ZRTuIjK6KdwHoTwQJC3FmFmYE+tSRET6pXAfhPKqeo6dmENGmlabiIxuSqlBqAgEmaP+dhGJAwr3KNUdamNPXbMOXhKRuKBwj1JFteZwF5H4oXCPUkWgHkDdMiISFxTuUdoSCDIuM43JeZmxLkVEZEAK9yhVBILMnTROJ8QWkbigcI+Cc84Ldx28JCJxIqpwN7PzzKzCzLaZ2S29PD7NzNaa2SYz+18zKx3+UmOn8kATDS3t6m8XkbgxYLibWSrwAHA+MB9YZmbzI5p9D/iFc24hcBfwL8NdaCyFTtChkTIiEi+i2XJfAmxzzu1wzrUCq4ClEW3mA2v96+t6eTyuhUbKHKezL4lInIgm3EuA3WG3K/37wm0EPuZf/yiQa2YFR17e6FAeCFI6fiy5mTpBh4jEh2jCvbfhIS7i9leBs8zsdeAs4D2g/bAFmV1nZhvMbENNTc2gi42Vcn+kjIhIvIgm3CuBKWG3S4E94Q2cc3ucc5c6504CvuHfVxe5IOfcw865xc65xUVFRUdQ9tHT3NbBzn2N6m8XkbgSTbivB2ab2QwzywCuAJ4Kb2BmhWYWWtbXgZ8Ob5mxs21vAx2dTsMgRSSuDBjuzrl2YDnwHLAFWO2ce9vM7jKzi/1mZwMVZvYOUAzcM0L1HnUVGikjInEoLZpGzrk1wJqI+24Pu/4Y8NjwljY6lAfqyUhLYXpBdqxLERGJmo5QHUB5IMjsiTmkpWpViUj8UGINQCfoEJF4pHDvx/7GVvYGW5inYZAiEmcU7v0o1xzuIhKnFO79KK/SSBkRiU8K935UBIJMyM6gKHdMrEsRERkUhXs/yquDzCnO1Qk6RCTuKNz70NnpeEcn6BCROKVw78O7+w/R1Nah/nYRiUsK9z6ERspoNkgRiUcK9z6UB4KY6QQdIhKfFO59KK8KMm1CFmMzUmNdiojIoCnc+1BRrRN0iEj8Urj3oqm1g121jToyVUTilsK9F+9UB3EO5mkYpIjEKYV7L0In6JijbhkRiVMK915sCdSTmZ7C1AlZsS5FRGRIFO69qAh40w6kpmjaARGJTwr3CM45ynWCDhGJcwr3CDUNLexvbNUwSBGJawr3CKGdqZpTRkTimcI9QugEHeqWEZF4pnCPUB4IUpQ7hoIcnaBDROKXwj1CeaBeXTIiEvcU7mHaOzrZurdB4S4icU/hHmZX7SFa2zt1ZKqIxD2Fe5juE3Roy11E4ltU4W5m55lZhZltM7Nbenl8qpmtM7PXzWyTmX1k+EsdeRWBIKkpxrETc2JdiojIERkw3M0sFXgAOB+YDywzs/kRzW4DVjvnTgKuAP5juAs9GrZUBZlekEVmuk7QISLxLZot9yXANufcDudcK7AKWBrRxgGhjuo8YM/wlXj0VFTXM3ey+ttFJP5FE+4lwO6w25X+feHuBK40s0pgDXBjbwsys+vMbIOZbaipqRlCuSOnoaWd3fubmKtzpopIAogm3HubGtFF3F4G/Mw5Vwp8BPgvMzts2c65h51zi51zi4uKigZf7QjqmnZAW+4ikgCiCfdKYErY7VIO73b5LLAawDn3VyATKByOAo8WzSkjIokkmnBfD8w2sxlmloG3w/SpiDbvAucAmNk8vHAfXf0uAygP1JMzJo2S/LGxLkVE5IgNGO7OuXZgOfAcsAVvVMzbZnaXmV3sN/sK8Dkz2wisBK5yzkV23Yxq5YEgxxXnkKITdIhIAkiLppFzbg3ejtLw+24Pu74ZOH14Szt6nHOUV9VzwcJjYl2KiMiw0BGqQKC+mfrmduZNVn+7iCQGhTtelwzAHA2DFJEEoXCn+wQdOrWeiCQKhTtQEahncl4meVnpsS5FRGRYKNzxumV0Wj0RSSRJH+6t7Z1sr2lQl4yIJJSkD/cd+xpo63A6MlVEEkrSh3v3nDIKdxFJHEkf7uWBIGkpxsxCnaBDRBKHwr2qnmMn5pCRlvSrQkQSSNInWoVGyohIAkrqcK871MaeumaFu4gknKQO94pqb2fqPA2DFJEEk9zhHqgH0Ja7iCScpA73LYEg4zLTmJyXGetSRESGVVKHe0UgyNxJ4zDTCTpEJLEkbbg75zRSRkQSVtKGe+WBJhpa2nVkqogkpKQN99AJOjSnjIgkoqQN99BImeN09iURSUBJG+7lgSCl48eSm6kTdIhI4knqcNcc7iKSqJIy3JvbOti5r1H97SKSsJIy3LftbaCj02kYpIgkrKQM99AJOuZpGKSIJKikDPfyQD0ZaSlML8iOdSkiIiMiqnA3s/PMrMLMtpnZLb08/u9m9oZ/ecfMDg5/qcOnPBBk9sQc0lKT8rNNRJJA2kANzCwVeAD4MFAJrDezp5xzm0NtnHNfDmt/I3DSCNQ6bCoCQcpmF8a6DBGRERPNpusSYJtzbodzrhVYBSztp/0yYOVwFDcS9je2sjfYopEyIpLQogn3EmB32O1K/77DmNk0YAbwxz4ev87MNpjZhpqamsHWOizK/SNTNcZdRBJZNOHe23y4ro+2VwCPOec6envQOfewc26xc25xUVFRtDUOq/IqzSkjIokvmnCvBKaE3S4F9vTR9gpGcZcMeP3tE7IzKModE+tSRERGTDThvh6YbWYzzCwDL8CfimxkZnOA8cBfh7fE4VVeHWROca5O0CEiCW3AcHfOtQPLgeeALcBq59zbZnaXmV0c1nQZsMo511eXTcx1djreCQQ1h7uIJLwBh0ICOOfWAGsi7rs94vadw1fWyHh3/yGa2jrU3y4iCS+pjuIJjZSZo5EyIpLgkizcg5jBccU5sS5FRGREJVe4VwWZNiGLrIyoeqNEROJWUoV7RbVO0CEiySFpwr2ptYNdtY2aw11EkkLShPs71UGc05GpIpIckibcQyfomDtZ3TIikviSJty3BOrJTE9h6oSsWJciIjLikibcKwLetAOpKZp2QEQSX1KEu3OO8kBQO1NFJGkkRbjXNLSwv7FVwyBFJGkkRbh37UzVlruIJImkCPfQCTrULSMiySI5wj0QpCh3DAU5OkGHiCSHJAn3enXJiEhSSfhwb+/oZOveBoW7iCSVhA/3XbWHaG3v1BzuIpJUEj7cQyfo0Ja7iCSThA/3ikCQFINjJ+oEHSKSPBI+3LdUBZlRmE1memqsSxEROWoSPtwrqus1E6SIJJ2EDveGlnZ2729ibrH620UkuSR0uGsOdxFJVskR7hopIyJJJqHDvTxQT3ZGKiX5Y2NdiojIUZXg4e7N4Z6iE3SISJKJKtzN7DwzqzCzbWZ2Sx9tPmFmm83sbTP71fCWOXjOOcqr6nVkqogkpbSBGphZKvAA8GGgElhvZk855zaHtZkNfB043Tl3wMwmjlTB0QrUN1Pf3M68yepvF5HkE82W+xJgm3Nuh3OuFVgFLI1o8zngAefcAQDn3N7hLXPwyv2dqXM0DFJEklA04V4C7A67XenfF+444Dgz+7OZ/c3MzuttQWZ2nZltMLMNNTU1Q6s4SqETdOjUeiKSjKIJ9972RrqI22nAbOBsYBnwiJnlH/Yk5x52zi12zi0uKioabK2DUhGoZ3JeJnlZ6SP6OiIio1E04V4JTAm7XQrs6aXNk865NufcTqACL+xjJjRSRkQkGUUT7uuB2WY2w8wygCuApyLa/Bb4IICZFeJ10+wYzkIHo7W9k+01DeqSEZGkNWC4O+fageXAc8AWYLVz7m0zu8vMLvabPQfUmtlmYB3wf51ztSNV9EB27GugrcPpyFQRSVoDDoUEcM6tAdZE3Hd72HUH/B//EnPdc8oo3EUkOSXkEarlgSBpKcbMQp2gQ0SSU2KGe1U9s4pyyEhLyLcnIjKghEy/ikBQXTIiktQSLtzrDrWxp65ZwyBFJKklXLhXVHs7U+dpGKSIJLHEC/dAPYC23EUkqSVcuG8JBMnNTGNyXmasSxERiZmEC/eKQJB5k8ZhphN0iEjySqhwd85RoTllREQSK9wrDzTR0NKuYZAikvQSKtxDJ+jQnDIikuwSKtxDI2WO09mXRCTJJVS4lweClI4fS26mTtAhIskt4cJdXTIiIgkU7s1tHezc16gTdIiIEOV87vFg294GOjqdhkFK8urshIYA1FVC3W7vZ+shyCuBvCmQPxXySiFtTKwrPTpaGrz1cPBd71JXCWaQmQ9jx8PY/J7Xx46HjByvTQJImHAPnaBjnoZBSqJqbewZ3HWVcHB39331e6CzbeDl5EyC/Cl+4E8JC37/9pg4+B9yDpoO+OHtB3iPIN/tPR4uxd8X1986SkmDzDwv6DPzu0O/vw+EULv0sSP3focgccK9OkhGWgrTC7JjXYrI4HV2QuPe7qA+GBbgoTBv2t/zOZYK447xtsanLPF+5vlhnVfqXdLHQv17/vJCy33X+1n1BpT/Djpaey43M98P/am9fwhkFYz81q1z0LC3O7C7gnt39/XWhp7PSc/urrP0lJ4150+F7Ile3W2HvOBvOuj9bD7Y9/VDtVC7vft+XN81p2VG/4FQfLz3uxtBCRPuW6rqObYoh7TUhNmNIEPh3Oj8Wt16yA/Zd8NCOxTcu6HuvcO3KDNy/YAq9cKqK7z94M6dDKlR/AuPn+5dehP6UAkP/VB47t8BO//US4hmddfS25Z/7mRISe2/ps4OCFaFBfa7PbfA6yqhvbnnczLzvNcZPwNmnBkW3P4HUdaE6H73GdneJa904LaR66qlvo8PBP/DIvx6XSVUv+Vdbw32XNYF98Epnx3c6w9SwoR7RSBI2ezCWJchR0NbMxzY5YVP6HJgp/fz4G5wHd7Xa0v1fqakepd+b6eBpUTcTu1u2+ftNEhJOfz1Oju8MA8F1aGI88VbiheCeaVQshjmX9IzvPOneGE20lJSIHeSd5lyyuGPR3Z/9Lb1H/neUtL8bxRhW/5mYc//u9+F1N7zedlFXtvi42HO+d3PD31wZMZ4sERKir/lnT/453a0QXNdd/DnTxn++iIkRLjvb2xlb7BFwyATSUtDd2B3XXZ6l/r36PH1eEweFMyEkpPh+I964dLZ4YWH6/R+dt3u8K93+Nfb+7jtP7e9eYDndoY9FrYsS4Fxk71QOmZRxJZuaKs7Do7HMPO2iLMmwOQTem8T2hdw2Nb/btj5greF7pwf+FNgyqmHb3XnlUJG1tF9b0dTajpkF3qXoyQhwr3cPzI16YZBdnZ4/1Qdbd5OsMxxXr/faOyW6E3TgZ6hHR7kjXt7ts0u8r6OTy+DCTP9ywzv59jx8fOeE1FGNhTN8S696Wjzwj0t4+jWleQSI9yrEnhOmc5OqK/0durs3w61oQDc7nVNRO4MS0mDMeO6wz50vcd9odt93Jc5bni2Kp2Dxn2Hd52ELpGjGXKP8cL6uH/sGd7jZ8T+K7kMXTx8Q0lA8RfuO1+Erb+HnGKvnzCnmP1/P8jUrA6KcuJ0y6CzE4J7ugN8/w4/xLd7W7QdLd1t08Z6gVd4nNcvOWGWt4OrpQ5agtBc7/1sqe++Xb8HWiq8+5rroxsul5bZy4dC+IdFxIdCRo63td21Je7/DN+RZCne1/EJM7zuk64t8JmQPy2xv5aPIm1tbVRWVtLc3DxwY4mZzMxMSktLSU8f2odj/IV79Vvw8kM9Au+r/oVvj4WciX7oT/TG8+YUQ26x9zN0yS6KbpTBcHLO63vcvyNsK3x7dwi2N3W3TR3jb7XOgtkf9n4WzPJ+5k72duwcibbmsA+A+l4+FOrDbod9UOzf6d/2P0hc5+HLTkmH8dO8wJ72ge4t7wkzvT5WfTWPucrKSnJzc5k+fbpOajNKOeeora2lsrKSGTNmDGkZ8Rfup90Ap17vDTlq2EtnfRX//PPn+fAUxz9OMwhWe0fp7dvqbeU3H+xlIeYFfG/Bn1vsfyj4HxIZgxg3Hxqb2xXc4QG+wxtfG5Ka4Q1PmzALZv2DF36hAB9XcuQB3p/0TO+SUzT0ZTjnvZ/wD4HsQm/H2EDD4CSmmpubFeyjnJlRUFBATU3NkJcRf+EO3s6zseNh7HjetVIea23klBPfB6dMPbxtews0VHuhGwz416v963u9D4K9W7z7Iodmgdfd0CP4i7u7hJzrGeT7d/YcE5yS1h3gM87sGeDxHoJm3eOFmRzramSQFOyj35H+jqIKdzM7D/gBkAo84pz7TsTjVwH3Au/5d93vnHvkiCqLUmikzJy+Rsqkjek+Qq0/nZ3eDr6GQM/g7/pQ2AuBN6FhrddNEWKpfjfELJh2ut+FMtMP8ClHv/tHRIQowt3MUoEHgA8DlcB6M3vKObc5oumvnXPLR6DGfpUHgpjBccU5R7aglBTILvAuxcf337a10dvSd8770NBoAJGo1dbWcs455wAQCARITU2lqMjrInzllVfIyBh4v8zVV1/NLbfcwpw5fQy/BB544AHy8/P51Kc+NTyFx5loNiuXANucczsAzGwVsBSIDPeYKK8KMm1CFlkZR3ELOSPb62IRkUErKCjgjTfeAODOO+8kJyeHr371qz3aOOdwzpHSx76nRx99dMDX+eIXv3jkxcaxaBKxBNgddrsSOLWXdh8zszOBd4AvO+d2RzYws+uA6wCmTh2gmyRKFdXB5Dt4SWSYfPPpt9m8p37ghoMw/5hx3HHRAN9+e7Ft2zYuueQSysrKePnll/nd737HN7/5TV577TWampq4/PLLuf322wEoKyvj/vvvZ8GCBRQWFnL99dfzzDPPkJWVxZNPPsnEiRO57bbbKCws5Oabb6asrIyysjL++Mc/UldXx6OPPsoHPvABGhsb+fSnP822bduYP38+W7du5ZFHHuHEE0/sUdsdd9zBmjVraGpqoqysjAcffBAz45133uH666+ntraW1NRUfvOb3zB9+nS+/e1vs3LlSlJSUrjwwgu55557hmXdDkY0QzJ669WPnBrtaWC6c24h8Dzw894W5Jx72Dm32Dm3OPQ17Eg0tXawq7ZRc7iLJIjNmzfz2c9+ltdff52SkhK+853vsGHDBjZu3Mgf/vAHNm8+vMOgrq6Os846i40bN/L+97+fn/70p70u2znHK6+8wr333stdd90FwI9+9CMmTZrExo0bueWWW3j99dd7fe6XvvQl1q9fz5tvvkldXR3PPvssAMuWLePLX/4yGzdu5C9/+QsTJ07k6aef5plnnuGVV15h48aNfOUrXxmmtTM40Wy5VwLhs9yUAnvCGzjnwmepyugvAAALhUlEQVQO+jHwr0de2sDeqQ7iXIIemSpyFAxlC3skzZo1i1NO6Z7AbOXKlfzkJz+hvb2dPXv2sHnzZubPn9/jOWPHjuX8888H4OSTT+bFF1/sddmXXnppV5tdu3YB8NJLL/G1r30NgBNOOIHjj+99faxdu5Z7772X5uZm9u3bx8knn8xpp53Gvn37uOiiiwDvoCOA559/nmuuuYaxY7353SdMmDCUVXHEogn39cBsM5uBNxrmCuCT4Q3MbLJzrsq/eTGwZVir7EPoBB1zJ6tbRiQRZGd3H1eydetWfvCDH/DKK6+Qn5/PlVde2etRteE7YFNTU2lv72VIMzBmzJjD2jjXz/zsvkOHDrF8+XJee+01SkpKuO2227rq6G24onNuVAw1HbBbxjnXDiwHnsML7dXOubfN7C4zu9hvdpOZvW1mG4GbgKtGquBwWwL1ZKanMHWCDlsXSTT19fXk5uYybtw4qqqqeO6554b9NcrKyli9ejUAb775Zq/dPk1NTaSkpFBYWEgwGOTxxx8HYPz48RQWFvL0008D3sFhhw4d4txzz+UnP/kJTU3eUef79+8/bJlHQ1RDTJxza4A1EffdHnb968DXh7e0gVUEgswpziU1JfafkiIyvBYtWsT8+fNZsGABM2fO5PTTTx/217jxxhv59Kc/zcKFC1m0aBELFiwgL6/nPPoFBQV85jOfYcGCBUybNo1TT+0eT7JixQo+//nP841vfIOMjAwef/xxLrzwQjZu3MjixYtJT0/noosu4lvf+taw1z4Qi+ZryUhYvHix27Bhw5Cf75zj5Luf50PzJvLdy/qYZ1pEDrNlyxbmzZsX6zJGhfb2dtrb28nMzGTr1q2ce+65bN26lbS00XHwYW+/KzN71Tm3eKDnjo53MAQ1DS3sb2zt+8hUEZEBNDQ0cM4559De3o5zjoceemjUBPuRitt3EdqZOk8jZURkiPLz83n11VdjXcaIiNuzSYdO0KEx7iIih4vfcA8EKcodQ0HOmFiXIiIy6sRxuNfr4CURkT7EZbi3d3SydW+Dwl1EpA9xGe67ag/R2t6pkTIicejss88+7ICk73//+3zhC1/o93k5Od603nv27OGyyy7rc9kDDbH+/ve/z6FD3WdF+8hHPsLBg72dsS2+xWW4h07QoS13kfizbNkyVq1a1eO+VatWsWzZsqief8wxx/DYY48N+fUjw33NmjXk5+cPeXmjVVwOhawIBEkxOHbiEZ6gQyTZPXOLd4ax4TTpfXD+d/p8+LLLLuO2226jpaWFMWPGsGvXLvbs2UNZWRkNDQ0sXbqUAwcO0NbWxt13383SpUt7PH/Xrl1ceOGFvPXWWzQ1NXH11VezefNm5s2b13XIP8ANN9zA+vXraWpq4rLLLuOb3/wmP/zhD9mzZw8f/OAHKSwsZN26dUyfPp0NGzZQWFjIfffd1zWr5LXXXsvNN9/Mrl27OP/88ykrK+Mvf/kLJSUlPPnkk10Tg4U8/fTT3H333bS2tlJQUMCKFSsoLi6moaGBG2+8kQ0bNmBm3HHHHXzsYx/j2Wef5dZbb6Wjo4PCwkLWrl07jL+EOA33LVVBZhRmk5kex+cgFUlSBQUFLFmyhGeffZalS5eyatUqLr/8csyMzMxMnnjiCcaNG8e+ffs47bTTuPjii/uciOvBBx8kKyuLTZs2sWnTJhYtWtT12D333MOECRPo6OjgnHPOYdOmTdx0003cd999rFu3jsLCwh7LevXVV3n00Ud5+eWXcc5x6qmnctZZZzF+/Hi2bt3KypUr+fGPf8wnPvEJHn/8ca688soezy8rK+Nvf/sbZsYjjzzCd7/7Xf7t3/6Nb33rW+Tl5fHmm96H6IEDB6ipqeFzn/scL7zwAjNmzBiR+WfiMtwrqutZWJp4X6NEjrp+trBHUqhrJhTuoa1l5xy33norL7zwAikpKbz33ntUV1czadKkXpfzwgsvcNNNNwGwcOFCFi5c2PXY6tWrefjhh2lvb6eqqorNmzf3eDzSSy+9xEc/+tGumSkvvfRSXnzxRS6++GJmzJjRdQKP8CmDw1VWVnL55ZdTVVVFa2srM2bMALwpgMO7ocaPH8/TTz/NmWee2dVmJKYFjrs+94aWdnbvb2JusfrbReLVJZdcwtq1a7vOshTa4l6xYgU1NTW8+uqrvPHGGxQXF/c6zW+43rbqd+7cyfe+9z3Wrl3Lpk2buOCCCwZcTn/zbIWmC4a+pxW+8cYbWb58OW+++SYPPfRQ1+v1NgXw0ZgWOO7CPTTtgI5MFYlfOTk5nH322VxzzTU9dqTW1dUxceJE0tPTWbduHX//+9/7Xc6ZZ57JihUrAHjrrbfYtGkT4E0XnJ2dTV5eHtXV1TzzzDNdz8nNzSUYDPa6rN/+9rccOnSIxsZGnnjiCc4444yo31NdXR0lJSUA/Pzn3SejO/fcc7n//vu7bh84cID3v//9/OlPf2Lnzp3AyEwLHLfhPk8n6BCJa8uWLWPjxo1cccUVXfd96lOfYsOGDSxevJgVK1Ywd+7cfpdxww030NDQwMKFC/nud7/LkiVLAO+sSieddBLHH38811xzTY/pgq+77jrOP/98PvjBD/ZY1qJFi7jqqqtYsmQJp556Ktdeey0nnXRS1O/nzjvv5OMf/zhnnHFGj/782267jQMHDrBgwQJOOOEE1q1bR1FREQ8//DCXXnopJ5xwApdffnnUrxOtuJvy9/dvB/jvVyt56MqTSdE87iKDpil/40dSTfl77vGTOPf43neuiIiIJ+66ZUREZGAKd5EkFKvuWInekf6OFO4iSSYzM5Pa2loF/CjmnKO2tpbMzMwhLyPu+txF5MiUlpZSWVlJTU1NrEuRfmRmZlJaWjrk5yvcRZJMenp615GRkrjULSMikoAU7iIiCUjhLiKSgGJ2hKqZ1QD9Txwx+hUC+2JdxCii9dFN66InrY+ejmR9THPOFQ3UKGbhngjMbEM0hwEnC62PbloXPWl99HQ01oe6ZUREEpDCXUQkASncj8zDsS5glNH66KZ10ZPWR08jvj7U5y4ikoC05S4ikoAU7iIiCUjhPgRmNsXM1pnZFjN728y+FOuaYs3MUs3sdTP7XaxriTUzyzezx8ys3P8beX+sa4olM/uy/3/ylpmtNLOhT3UYZ8zsp2a218zeCrtvgpn9wcy2+j/Hj8RrK9yHph34inNuHnAa8EUzmx/jmmLtS8CWWBcxSvwAeNY5Nxc4gSReL2ZWAtwELHbOLQBSgSv6f1ZC+RlwXsR9twBrnXOzgbX+7WGncB8C51yVc+41/3oQ75+3JLZVxY6ZlQIXAI/EupZYM7NxwJnATwCcc63OuYOxrSrm0oCxZpYGZAF7YlzPUeOcewHYH3H3UuDn/vWfA5eMxGsr3I+QmU0HTgJejm0lMfV94J+BzlgXMgrMBGqAR/1uqkfMLDvWRcWKc+494HvAu0AVUOec+31sq4q5YudcFXgbisDEkXgRhfsRMLMc4HHgZudcfazriQUzuxDY65x7Nda1jBJpwCLgQefcSUAjI/S1Ox74/clLgRnAMUC2mV0Z26qSg8J9iMwsHS/YVzjnfhPremLodOBiM9sFrAL+wcx+GduSYqoSqHTOhb7JPYYX9snqQ8BO51yNc64N+A3wgRjXFGvVZjYZwP+5dyReROE+BGZmeH2qW5xz98W6nlhyzn3dOVfqnJuOt6Psj865pN0yc84FgN1mNse/6xxgcwxLirV3gdPMLMv/vzmHJN7B7HsK+Ix//TPAkyPxIjrN3tCcDvwT8KaZveHfd6tzbk0Ma5LR40ZghZllADuAq2NcT8w45142s8eA1/BGmb1OEk1FYGYrgbOBQjOrBO4AvgOsNrPP4n34fXxEXlvTD4iIJB51y4iIJCCFu4hIAlK4i4gkIIW7iEgCUriLiCQghbuISAJSuIuIJKD/D4L3PJ/bwopyAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x27f90f46630>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl4VPXZ//H3nclGFhJIgiwJOwJhx4hYooBSBZWlFhXUx2r1ofpz3ypaq5XaumDdqa21WqsitVgLrtQqj2C1SEDWsAoIYQ2BBEIgyST3748zCZOQZRISTjK5X9c118w55zvn3DOBzznzPZuoKsYYY4JLiNsFGGOMaXgW7sYYE4Qs3I0xJghZuBtjTBCycDfGmCBk4W6MMUHIwt1USUQ8IpIvIp0bsq2bRKSniDT4sb8iMkZEtvkNbxCRcwJpW49lvSIiD9T3/TXM91ER+UtDz9e4J9TtAkzDEJF8v8EooBAo8Q3/TFXfqsv8VLUEiGnoti2BqvZuiPmIyA3A1ao6ym/eNzTEvE3ws3APEqpaHq6+LcMbVPXf1bUXkVBV9Z6K2owxp551y7QQvp/dfxORt0XkMHC1iJwtIv8VkVwR2S0iz4tImK99qIioiHT1Db/pm/6xiBwWka9FpFtd2/qmjxORjSKSJyIviMh/ROTaauoOpMafichmETkoIs/7vdcjIs+ISI6IfAeMreH7eVBE5lQaN0tEnva9vkFE1vk+z3e+rerq5pUlIqN8r6NE5A1fbWuBM6pY7hbffNeKyATf+AHAi8A5vi6v/X7f7a/83n+j77PniMg/RaRDIN9NbURkkq+eXBH5XER6+017QER2icghEVnv91mHi8hy3/i9IjIz0OWZRqCq9giyB7ANGFNp3KNAETAeZ6XeCjgTOAvnF1x3YCNwi699KKBAV9/wm8B+IA0IA/4GvFmPtu2Aw8BE37S7gGLg2mo+SyA1zgPigK7AgbLPDtwCrAWSgQRgkfNPvsrldAfygWi/ee8D0nzD431tBDgPOAoM9E0bA2zzm1cWMMr3+ing/4A2QBcgs1Lby4EOvr/Jlb4aTvNNuwH4v0p1vgn8yvf6Al+Ng4FI4PfA54F8N1V8/keBv/he9/XVcZ7vb/SA73sPA/oB3wPtfW27Ad19r5cCU32vY4Gz3P6/0JIftuXesnypqu+raqmqHlXVpaq6RFW9qroFeBkYWcP756pqhqoWA2/hhEpd214CrFDVeb5pz+CsCKoUYI2PqWqeqm7DCdKyZV0OPKOqWaqaAzxew3K2AGtwVjoAPwRyVTXDN/19Vd2ijs+Bz4Aqd5pWcjnwqKoeVNXvcbbG/Zf7jqru9v1NZuOsmNMCmC/AVcArqrpCVY8B04GRIpLs16a676YmU4D5qvq572/0ONAaZyXrxVmR9PN17W31fXfgrKR7iUiCqh5W1SUBfg7TCCzcW5Yd/gMi0kdEPhSRPSJyCJgBJNbw/j1+rwuoeSdqdW07+tehqoqzpVulAGsMaFk4W5w1mQ1M9b2+EmelVFbHJSKyREQOiEguzlZzTd9VmQ411SAi14rISl/3Ry7QJ8D5gvP5yuenqoeAg0AnvzZ1+ZtVN99SnL9RJ1XdANyN83fY5+vma+9reh2QCmwQkW9E5KIAP4dpBBbuLUvlwwD/iLO12lNVWwMP4XQ7NKbdON0kAIiIUDGMKjuZGncDKX7DtR2q+TdgjG/LdyJO2CMirYC5wGM4XSbxwL8CrGNPdTWISHfgJeAmIME33/V+863tsM1dOF09ZfOLxen+2RlAXXWZbwjO32wngKq+qaojcLpkPDjfC6q6QVWn4HS9/Q54V0QiT7IWU08W7i1bLJAHHBGRvsDPTsEyPwCGish4EQkFbgeSGqnGd4A7RKSTiCQA99XUWFX3Al8CrwEbVHWTb1IEEA5kAyUicglwfh1qeEBE4sU5D+AWv2kxOAGejbOeuwFny73MXiC5bAdyFd4GrheRgSISgROyi1W12l9Cdah5goiM8i37Xpz9JEtEpK+IjPYt76jvUYLzAf5HRBJ9W/p5vs9WepK1mHqycG/Z7gZ+gvMf9484W66NyhegVwBPAzlAD+BbnOPyG7rGl3D6xlfj7OybG8B7ZuPsIJ3tV3MucCfwHs5Oyck4K6lAPIzzC2Ib8DHwV7/5rgKeB77xtekD+PdTfwpsAvaKiH/3Stn7P8HpHnnP9/7OOP3wJ0VV1+J85y/hrHjGAhN8/e8RwJM4+0n24PxSeND31ouAdeIcjfUUcIWqFp1sPaZ+xOnyNMYdIuLB6QaYrKqL3a7HmGBhW+7mlBORsSIS5/tp/0ucIzC+cbksY4KKhbtxQzqwBeen/VhgkqpW1y1jjKkH65YxxpggZFvuxhgThFy7cFhiYqJ27drVrcUbY0yztGzZsv2qWtPhw4CL4d61a1cyMjLcWrwxxjRLIlLbmdaAdcsYY0xQsnA3xpggZOFujDFByMLdGGOCkIW7McYEIQt3Y4wJQhbuxhgThFw7zt0YY1qMoiOQsxn2b4Kc7+D0C6FjIHc8rD8Ld2OMaQilJZC73Qnx8iDfBPs3w+Fdfg0FohMt3I0xzdjRXDi4FbyFEJ3khFpEa5DGvptjIyo44Avuzb7w9r0+sAVK/O5NEhkHCb2g+0hI6AmJvZzntt0hrFWjl2nhbow5OQUH4MBWJ9wOfOd79j0Kck5s7wmHqEQn6KOTjod+5eEo33B41Kn/TN5Cp/7yLXC/rfGjB463CwmDtt2cED/9Qie8E3o5QR6V4OpKzMLdGFMzVV+AVxHeOd/BsVy/xgJxyU7g9R0PbXv4tlQj4UgOHMn2PfZDwX7ndc4mZ7i4oOrlh0VXCn/fc1Sl4egkJ1BDwwP/XId3V+w+Kdsaz90O6nf715j2TmCnTjge3gk9Ib4LeJpmjDbNqowxp5aqE7T+wV0W3ge2QmHe8bYS4gvwHtD/x054lz3adHWCvD6Kjjghf8QX+mXhXzZ8JBsO7YLdq5zXpcVVzycyrmLY+78+eqDijs3iI8ffFxYFCT2g41AYeIUT4gk9nBCPbF2/z+QiC3dj6qu0FI4edLoeCnyhVPa68LDT/RAaAaGRlR4RAT5HQkgDHq2sCvl7KwV3WZBvhaLDx9uKB+I7O4GdfKbznODbCo/v7NTY0MKjnUebLoF9lmN5zvdd/msgu9KvA9/KascSp52WAuLUn9gLuoyAxJ7Hu1Jad2ze+wIqsXA3pkzxMSeYC3J8QX3gxNAuOHC8S+HowYo/3f15Inw7107yTmchYRUDP6yOKwjvMV9/uK9P3H9LNSTU6VZo2x26/KDiFnh8Z/CEnVztjUkEWsU7j4QetbcvLXH+XuEx9f9l0cxYuJv6KymGvCzI2wF5O52fySGhJz48YRDi8Q2H+Z49vvFVtfVND/Gf7qnbVpWq0xfsH8bloZ1T6bUvtIvyq56XhECrtr6dfAmQ1BuiRzivo3zjov1eRyU4AaIKpV4nYIuPOc/ewkrPVY2r7dnvfYWHnS3U6tp6wp2ukrbdods5vvDu5nSpxKU02f7iBhficf5+LUgL+cuaeik+Crk7IG+785y73Qny3B3O8+Hd1W+5NobyoPdbWVReGYg4h98dPeAEa1XCoo6HcHSi78iGRIjyC/CyozmiEiAyvn7dIyJOfZ4wiIg9uc9eH6rOoyG7dkyzYeHekh3LOx7UudtPDO8j2RXbiwfiOkFcZ+h2rvPTPS4F4lN8W4HhTqCWljhb8aVe51HiPf66tNg33ets+de7vdfvPSXHp2mJE8YVQjqh4la2G4fWuUEkqPqQTd0EFO4iMhZ4DvAAr6jq45WmPwOM9g1GAe1UNb4hC3WdqvMTuGC/s9OmYL+zZVt5B5h/n2dYq+PDIaGn9j+aqtPlUFVol22F+x8BAU6dcclOULcf4Avtzs5zfGeI7eBsIRtjmrxaw11EPMAs4IdAFrBUROaramZZG1W906/9rcCQRqi1YZWW+vpkc/yOud1fMbwrD/uffVZXElLFiqCaHWH+K4WqjqCo/N5jece7Tsq2wvOyTjxuODzWCen4FOg8/PgWd3wX53V0km3pGRMkAtlyHwZsVtUtACIyB5gIZFbTfirwcMOUVwelJZWObvA7yqHK4RznJ3xVwmOP7yBr3Qk6DPLrg/Xriw1r5dtxVcMOr7LXNe5Q8z2O5R4fX7l9IEddRCU4YZ3UG3r+8PgWd1nXSWS8hbcxLUQg4d4J2OE3nAWcVVVDEekCdAM+r2b6NGAaQOfOnetUaJm1i+dxIGMu6R1ByoL6SLZzmFN1AVjeB5t4/LjdykHtP9wYx/CeDFWnT7mqFUfxMWdnXXyKc4ywMcYQWLhXtalX3WbkFGCuatWbxKr6MvAyQFpaWr0OAC7Zt57U3IXkFifSJqmjs5XadUQ1QZ3kHAHRlI/XDYSIc0p1aDjQ/M6UM8aceoGEexaQ4jecDOyqpu0U4OaTLaomAy79Of+bfz6LN2Xz8XXn0D0ppjEXZ4wxzVIgB8AuBXqJSDcRCccJ8PmVG4lIb6AN8HXDlnjCcvjNj/oTERrCfe+uorT0JM8ANMaYIFRruKuqF7gFWACsA95R1bUiMkNEJvg1nQrMUdVGT9vTWkfyy0tSWbrtIK9/va2xF2eMMc2OnIIsrlJaWppmZGTU+/2qynV/WcqSLQdYcMe5dE5oISemGGNaNBFZpqpptbVrtucliwi//dEAPCFi3TPGGFNJsw13gI7xrfjFxX35eksOs7/Z7nY5xhjTZDTrcAeYcmYK6T0TeeyjdWQdrOZOLsYY08I0+3AXER67dAAK3P+P1bi1D8EYY5qSZh/uAClto7h/XB8Wb9rPOxk7an+DMcYEuaAId4CrzurC8O5tefSDdezOO+p2OcYY46qgCfeQEOGJHw/EW6o8YN0zxpgWLmjCHaBLQjT3XtibhRuy+cfynW6XY4wxrgmqcAe49gddSevShkfeX8u+Q8fcLscYY1wRdOEeEiI8OXkghd5SfvHPNdY9Y4xpkYIu3AG6J8Vw9wWn82nmXt5ftdvtcowx5pQLynAHuD69O4NS4nl43hr25xe6XY4xxpxSQRvunhDhqckDOVJYwsPz1rpdjjHGnFJBG+4AvU6L5fYxvfhw9W4+Xm3dM8aYliOowx1g2rnd6d+pNb+ct4YDR4rcLscYY06JoA/3ME8IMycPIu9oMY+8b90zxpiWIejDHaBvh9bcPLon81bs4tPMvW6XY4wxjS6gcBeRsSKyQUQ2i8j0atpcLiKZIrJWRGY3bJkn7/+N6kmf9rH84r3V5BUUu12OMcY0qlrDXUQ8wCxgHJAKTBWR1EptegH3AyNUtR9wRyPUelLCQ0N46rJB5BwpYsYHmW6XY4wxjSqQLfdhwGZV3aKqRcAcYGKlNv8LzFLVgwCquq9hy2wY/TvFcdPIHry7PIuFG5pkicYY0yACCfdOgP9F0rN84/ydDpwuIv8Rkf+KyNiqZiQi00QkQ0QysrOz61fxSbr1/J6cfloM97+7mkPHrHvGGBOcAgl3qWJc5Qu2hAK9gFHAVOAVEYk/4U2qL6tqmqqmJSUl1bXWBhER6mHm5EHsO3yM3364zpUajDGmsQUS7llAit9wMrCrijbzVLVYVbcCG3DCvkkalBLP/57bnTlLd7B4kzu/IIwxpjEFEu5LgV4i0k1EwoEpwPxKbf4JjAYQkUScbpotDVloQ7tzzOl0T4pm+ruryS/0ul2OMcY0qFrDXVW9wC3AAmAd8I6qrhWRGSIywddsAZAjIpnAQuBeVc1prKIbQmSYh5mTB7Ir7yhPfLze7XKMMaZBiVvXO09LS9OMjAxXlu1vxvuZvPqfrbz9v8M5u0eC2+UYY0yNRGSZqqbV1q5FnKFak3sv7E2XhCjue3cVBUXWPWOMCQ4tPtxbhXt44scD2X6ggJkLNrhdjjHGNIgWH+4Aw7sncM3ZXfjLV9vI2HbA7XKMMeakWbj73De2D53iW/Hzuas4VlzidjnGGHNSLNx9oiNCeeLHA9my/whPf7rR7XKMMeakWLj7GdEzkanDOvPK4i18u/2g2+UYY0y9WbhX8sBFfWjfOpJ7rXvGGNOMWbhXEhsZxmM/Hsjmffm88Pkmt8sxxph6sXCvwsjTk7jsjGT+8MUWVmfluV2OMcbUmYV7NR68JJXEmHDunbuSIm+p2+UYY0ydWLhXI65VGL/90QDW7znMrIWb3S7HGGPqxMK9Buf3PY0fDenErIWbydx1yO1yjDEmYBbutXjoklTio5zumeIS654xxjQPFu61aBMdzqOT+rF21yH++MV3bpdjjDEBsXAPwNj+Hbh4YAee/2wzG/cedrscY4yplYV7gGZM6EdMZCj3/n0lXuueMcY0cRbuAUqIieCRCf1YmZXHK19udbscY4ypkYV7HVwysAMX9juNpz/dyOZ9+W6XY4wx1Qoo3EVkrIhsEJHNIjK9iunXiki2iKzwPW5o+FLdJyL8elJ/osI9/HzuSkpK3blFoTHG1KbWcBcRDzALGAekAlNFJLWKpn9T1cG+xysNXGeT0S42kofHp7J8ey6v/ce6Z4wxTVMgW+7DgM2qukVVi4A5wMTGLatpmzS4E+f3acfv/rWRXblH3S7HGGNOEEi4dwJ2+A1n+cZV9mMRWSUic0UkpaoZicg0EckQkYzs7Ox6lNs0iAi/mtCPUlV+8+E6t8sxxpgTBBLuUsW4yp3N7wNdVXUg8G/g9apmpKovq2qaqqYlJSXVrdImJqVtFDeP7smHq3fz5ab9bpdjjDEVBBLuWYD/lngysMu/garmqGqhb/BPwBkNU17TNu3c7nRJiOLh+WvsypHGmCYlkHBfCvQSkW4iEg5MAeb7NxCRDn6DE4AW0VcRGebh4fGpfJd9xHauGmOalFrDXVW9wC3AApzQfkdV14rIDBGZ4Gt2m4isFZGVwG3AtY1VcFNzXp/TGNO3Hc99tok9ecfcLscYYwAQVXeO1U5LS9OMjAxXlt3QtucUMOaZL7iwX3temDrE7XKMMUFMRJapalpt7ewM1QbQOSGKm0b24P2Vu/jqO9u5aoxxn4V7A7lpVA9S2rbi4Xlr7brvxhjXWbg3kMgwDw9d0o9N+/J5/attbpdjjGnhLNwb0Ji+7RjdO4ln/72JfYds56oxxj0W7g1IRHh4fD+KvKU89vF6t8sxxrRgFu4NrGtiNNPO7c573+7km60H3C7HGNNCWbg3gptH96RTfCsemrfG7tpkjHGFhXsjaBXu4ZeX9GX9nsO88d/v3S7HGNMCWbg3kgv7teecXok8/a+NZB8urP0NxhjTgCzcG4mI8MiEfhzzlvC47Vw1xpxiFu6NqHtSDDec0513l2ex7HvbuWqMOXUs3BvZref1pENcJL/851q756ox5pSxcG9kUeGhPHhxKpm7D/HWEtu5aow5NSzcT4GLBrRnRM8EnlqwgZx827lqjGl8Fu6nQNnO1YKiEp74xHauGmMan4X7KdKzXSzXp3fjnYwslm8/6HY5xpggZ+F+Ct16fi9Oax3BQ/PW2M5VY0yjCijcRWSsiGwQkc0iMr2GdpNFREWk1ruEtEQxEaH84uJU1uw8xNvfbHe7HGNMEKs13EXEA8wCxgGpwFQRSa2iXSzO/VOXNHSRwWT8wA6c3T2BmQs2cOBIkdvlGGOCVCBb7sOAzaq6RVWLgDnAxCra/Rp4ErALmddARHhkYj+OFHqZucB2rhpjGkcg4d4J2OE3nOUbV05EhgApqvpBTTMSkWkikiEiGdnZ2XUuNlicflos1/6gK3OW7mDljly3yzHGBKFAwl2qGFe+N1BEQoBngLtrm5GqvqyqaaqalpSUFHiVQej2Mb1IjHF2rpbazlVjTAMLJNyzgBS/4WRgl99wLNAf+D8R2QYMB+bbTtWaxUaG8YuL+rIyK4+/Zeyo/Q3GGFMHgYT7UqCXiHQTkXBgCjC/bKKq5qlqoqp2VdWuwH+BCaqa0SgVB5GJgzsyrFtbnvxkPbkFtnPVGNNwag13VfUCtwALgHXAO6q6VkRmiMiExi4wmJWduXromJeZCza4XY4xJoiEBtJIVT8CPqo07qFq2o46+bJajr4dWvM/w7vw+tfbmHJmZwYkx7ldkjEmCNgZqk3AnT88nYTocH5pO1eNMQ3Ewr0JiGsVxvRxfVmxI5e5y7LcLscYEwQs3JuIS4d04owubXj8k/XkFRS7XY4xppmzcG8iQkKEGRP7kVtQxO8+tZ2rxpiTY+HehPTrGMfVw7vw5n+/Z+2uPLfLMcY0YxbuTczdP+xNm6hwHpq3FlXbuWqMqR8L9yYmLiqM+8b2Ydn3B/nH8p1ul2OMaaYs3JugyWckMzglnsc+Xs+hY7Zz1RhTdxbuTVBIiPDrif3JOVLIM59udLscY0wzZOHeRA1IjuPKYZ3569ffs37PIbfLMcY0MxbuTdi9F/amdWQoD/3Tdq4aY+rGwr0Ji48K5+dj+/DNtgPMW7Gr9jcYY4yPhXsTd0VaCoOS4/jNR+s4bDtXjTEBsnBv4pwzV/uzP7+Q5/69ye1yjDHNhIV7MzAoJZ4pZ6bw2lfb2Lj3sNvlGGOaAQv3ZuLeC/sQGxnKQ/PW2M5VY0ytLNybibbR4dxzQW/+u+UA76/a7XY5xpgmzsK9GZk6rDP9O7XmNx9mkl/odbscY0wTFlC4i8hYEdkgIptFZHoV028UkdUiskJEvhSR1IYv1Xh8O1f3Hirkhc9s56oxpnq1hruIeIBZwDggFZhaRXjPVtUBqjoYeBJ4usErNQAM7dyGy9OS+fOXW9m8z3auGmOqFsiW+zBgs6puUdUiYA4w0b+BqvqfHx8N2B6/RnTf2D5EhXt4eL6duWqMqVog4d4J2OE3nOUbV4GI3Cwi3+Fsud9W1YxEZJqIZIhIRnZ2dn3qNUBCTAR3X9Cb/2zO4aPVe9wuxxjTBAUS7lLFuBM2F1V1lqr2AO4DHqxqRqr6sqqmqWpaUlJS3So1FVx1Vmf6dmjNox9mcsR2rhpjKgkk3LOAFL/hZKCmC53MASadTFGmdqGeEH49sR+7847xnO1cNcZUEki4LwV6iUg3EQkHpgDz/RuISC+/wYsBS5tTIK1rW6YOS+FPi7ewaKN1cxljjqs13FXVC9wCLADWAe+o6loRmSEiE3zNbhGRtSKyArgL+EmjVWwqeOiSfvRqF8Odf1vBvkPH3C7HGNNEiFtHW6SlpWlGRoYryw42m/YeZvyLXzIkpQ1v3nAWnpCqdpMYY4KBiCxT1bTa2tkZqkGg12mxzJjYn6+35PDC59YjZoyxcA8al52RzKVDOvHcZ5v46rv9bpdjjHGZhXuQEBF+Pak/3RKjuX3OCvbnF7pdkjHGRRbuQSQ6IpRZVw7l0NFi7vzbCkpL7exVY1oqC/cg07dDax4e34/Fm/bz0hffuV2OMcYlFu5BaOqwFC4Z2IHf/WsD32w94HY5xhgXWLgHIRHhsUsHkNI2itve/pYDR4rcLskYc4pZuAep2MgwZl05lANHirjn7yut/92YFsbCPYj17xTHLy7uy+fr9/HKl1vcLscYcwpZuAe5a87uwth+7Xnykw0s337Q7XKMMaeIhXuQExGemDyQ9nGR3Dr7W/IKit0uyRhzCli4twBxrcJ48cqh7D10jHvmrrS7NxnTAli4txCDU+KZPq4Pn2bu5S9fbXO7HGNMI7Nwb0GuT+/GmL7t+O1H61iVlet2OcaYRmTh3oKICE9dNoikmAhumf0th45Z/7sxwcrCvYWJjwrnhSuHsDP3KNPfXWX978YEKQv3FuiMLm2554LefLR6D28u2e52OcaYRhBQuIvIWBHZICKbRWR6FdPvEpFMEVklIp+JSJeGL9U0pJ+d252Rpyfx6w8yWbsrz+1yjDENrNZwFxEPMAsYB6QCU0UktVKzb4E0VR0IzAWebOhCTcMKCRGevnwQbaLCuGX2t+QXet0uyRjTgALZch8GbFbVLapaBMwBJvo3UNWFqlrgG/wvkNywZZrGkBATwXNThvB9zhF+8d5q6383JogEEu6dgB1+w1m+cdW5Hvj4ZIoyp87w7gncMeZ05q3YxTsZO2p/gzGmWQgk3KWKcVVu4onI1UAaMLOa6dNEJENEMrKzswOv0jSqm0f3ZETPBB6ev5YNew67XY4xpgEEEu5ZQIrfcDKwq3IjERkD/AKYoKpV3sBTVV9W1TRVTUtKSqpPvaYReEKEZ64YTExEGDfPXk5BkfW/G9PcBRLuS4FeItJNRMKBKcB8/wYiMgT4I06w72v4Mk1jaxcbyXNTBvNddj4PzVvrdjnGmJNUa7irqhe4BVgArAPeUdW1IjJDRCb4ms0EYoC/i8gKEZlfzexMEzaiZyK3ju7J3GVZvLssy+1yjDEnQdw6QiItLU0zMjJcWbapnreklCtfWcKanXnMvyWdnu1i3C7JGONHRJapalpt7ewMVVNBqCeE56cMITLMwy2zl3OsuMTtkowx9WDhbk7QPi6Spy8fxPo9h3nk/Uy3yzHG1IOFu6nSqN7tuHFkD97+ZjvzV55wcJQxpomzcDfVuvuC0zmjSxvuf3cVW/cfcbscY0wdWLibaoV5Qnh+6hBCPSHW/25MM2PhbmrUKb4Vv7tsEGt3HeK3H61zuxxjTIAs3E2txqSexvXp3fjr19/z8erdbpdjjAmAhbsJyH1j+zAoOY6fv7uK7TkFtb/BGOMqC3cTkPDQEF68cigAt769nCJvqcsVGWNqYuFuApbSNoqZkweyMiuPJz5Z73Y5xpgaWLibOhnbvwM/ObsLf/5yK59m7nW7HGNMNSzcTZ3df1Ff+nVszT1/X8nO3KNul2OMqYKFu6mzyDAPs64cSkmpcuvs5RSXWP+7MU2Nhbupl66J0fz20gEs357L7/610e1yjDGVWLibepswqCNTh3XmD198x8INdo8WY5oSC3dzUh4en0qf9rHc/c5Kdhyw49+NaSos3M1JiQzz8OKVQyksLmHM01/w6AeZ7M+v8ha6xphTyMLdnLSe7WL45I5zGT+oI6/+ZyvnPrmQmQvWk1dQ7HZpxrRYAYW7iIwVkQ0isllEplcx/VwRWS4iXhGZ3PBlmqYupW0UT102iE/vGsn5fU9j1sLvSH/yc57/bBOHj1nIG3Oq1XpfhpPcAAAUT0lEQVQPVRHxABuBHwJZwFJgqqpm+rXpCrQG7gHmq+rc2hZc1T1Ui4uLycrK4tixY3X7FMYVkZGRJCcnExYWdsK0dbsP8cynG/lX5l7aRIVx48geXHN2V1qFe1yo1JjgEeg9VEMDmNcwYLOqbvHNeA4wESgPd1Xd5pt2Ugc8Z2VlERsbS9euXRGRk5mVaWSqSk5ODllZWXTr1u2E6X07tObla9JYuSOXpz/dyGMfr+eVL7dyy+ieTBmWQkSohbwxjSmQbplOwA6/4SzfuDoTkWkikiEiGdnZ2SdMP3bsGAkJCRbszYCIkJCQUOuvrEEp8bz+02H8/caz6Z4YzcPz1zJ65v8x55vtdvKTMY0okHCvKmlr7suphqq+rKppqpqWlJRU9cIs2JuNuvytzuzaljnThvPm9WfRrnUk0/+xmjFPf8F732ZRUlqvf07GmBoEEu5ZQIrfcDJgd0w2dSYipPdK5L3/9wP+/JM0osJDufNvK7nw2UV8tHo3pRbyxjSYQMJ9KdBLRLqJSDgwBZjfuGW5Iycnh8GDBzN48GDat29Pp06dyoeLiooCmsd1113Hhg0bamwza9Ys3nrrrYYomfT0dFasWNEg8zpVRITz+57Gh7em8/urnGvE/7+3ljP+xS/5fP1eatvJb4ypXa07VFXVKyK3AAsAD/Cqqq4VkRlAhqrOF5EzgfeANsB4EXlEVfs1auWNICEhoTwof/WrXxETE8M999xToY2qoqqEhFS9XnzttddqXc7NN9988sUGgZAQ4aIBHbiwX3vmr9zJM59u4qd/yWBI53juuaA3P+hh+1+Mqa9AjpZBVT8CPqo07iG/10txumsazCPvryVz16GGnCWpHVvz8Pi6r3M2b97MpEmTSE9PZ8mSJXzwwQc88sgjLF++nKNHj3LFFVfw0EPO15Gens6LL75I//79SUxM5MYbb+Tjjz8mKiqKefPm0a5dOx588EESExO54447SE9PJz09nc8//5y8vDxee+01fvCDH3DkyBGuueYaNm/eTGpqKps2beKVV15h8ODB1db55ptv8sQTT6CqTJgwgd/+9rd4vV6uu+46VqxYgaoybdo0brvtNp555hn+9Kc/ERYWxoABA3jzzTfr/b2eLE+I8KMhyVwysCNzl2Xx/GebuOqVJQzv3pZ7LuhNWte2rtVmTHMVULgbyMzM5LXXXuMPf/gDAI8//jht27bF6/UyevRoJk+eTGpqaoX35OXlMXLkSB5//HHuuusuXn31VaZPP+EcMFSVb775hvnz5zNjxgw++eQTXnjhBdq3b8+7777LypUrGTp0aI31ZWVl8eCDD5KRkUFcXBxjxozhgw8+ICkpif3797N69WoAcnNzAXjyySf5/vvvCQ8PLx/ntjBPCFOHdeZHQzox55vtvLjwOyb/4WtGnp7E3ReczsDkeLdLNKbZaLLhXp8t7MbUo0cPzjzzzPLht99+mz//+c94vV527dpFZmbmCeHeqlUrxo0bB8AZZ5zB4sWLq5z3pZdeWt5m27ZtAHz55Zfcd999AAwaNIh+/Wr+PpYsWcJ5551HYmIiAFdeeSWLFi3ivvvuY8OGDdx+++1cdNFFXHDBBQD069ePq6++mokTJzJp0qQ6fhuNKzLMw7UjunHFmZ3569fbeOmL75jw4n+4sN9p3PnD0+nTvrXbJRrT5Nm1ZQIUHR1d/nrTpk0899xzfP7556xatYqxY8dWebx3eHh4+WuPx4PX661y3hERESe0qetOxeraJyQksGrVKtLT03n++ef52c9+BsCCBQu48cYb+eabb0hLS6OkpKROyzsVWoV7+NnIHiz++WjuHHM6X23OYdxzi7nt7W/Zkp3vdnnGNGkW7vVw6NAhYmNjad26Nbt372bBggUNvoz09HTeeecdAFavXk1mZmaN7YcPH87ChQvJycnB6/UyZ84cRo4cSXZ2NqrKZZddVr6foKSkhKysLM477zxmzpxJdnY2BQVN93K9sZFh3D6mF4vvG81NI3vwaeZexjz9Bff+3S4zbEx1mmy3TFM2dOhQUlNT6d+/P927d2fEiBENvoxbb72Va665hoEDBzJ06FD69+9PXFxcte2Tk5OZMWMGo0aNQlUZP348F198McuXL+f6669HVRERnnjiCbxeL1deeSWHDx+mtLSU++67j9jY2Ab/DA0tPiqcn4/tw0/Tu/HS/33HG//9nn+u2MkVZ6Zwy+hetI+LdLtEY5qMWi8c1liqunDYunXr6Nu3ryv1NDVerxev10tkZCSbNm3iggsuYNOmTYSGNq31sZt/sz15x3hx4Sb+tnQHISL8z/Au3DiqB4kxEa7UY8yp0JAXDjMuyM/P5/zzz8fr9aKq/PGPf2xywe629nGRPDppAD87twfPf7aJV/+zlbeWbGdol3gGJsczsFMcA1Pi6RgXacfLmxbH0qKJio+PZ9myZW6X0SyktI1i5mWDuHFUD17/ahvLtx/kT4u24PVdziAxJpwBneKcwE92npNibeveBDcLdxM0eiTFMGNifwCOFZewfs9hVmXlsiorj1VZuXyxMZuyy9d0iIssD/qByXEM6BRHfFR4DXM3pnmxcDdBKTLMw+CUeAanHD/x6Uihl7W7DpUH/uqdeSxYu7d8epeEqOPdOclx9OsUR0yE/RcxzZP9yzUtRnREKMO6tWVYt+OXM8grKGbNrjxWZuWyOiuP5d8f5P2VzkVPRaBnUgwDkuMYlBzPgOQ4Uju0JjLMbjRimj4Ld9OixUWFMaJnIiN6JpaP259fyOqs44G/aON+/rF8JwChIULv9rHlXToDOsXRu30sYR47ZcQ0LRbufkaNGsX999/PhRdeWD7u2WefZePGjfz+97+v9n0xMTHk5+eza9cubrvtNubOPfEWsqNGjeKpp54iLa36I5ieffZZpk2bRlRUFAAXXXQRs2fPJj7+5K6pUt0VLk3VEmMiGN2nHaP7tAOcs3/3HDrGyh15rN7pdOl8tHoPb3/j3KAsPDSE1A6tGZQcx4DkeAYlx9E9KQZPiB2hY9xj4e5n6tSpzJkzp0K4z5kzh5kzZwb0/o4dO1YZ7IF69tlnufrqq8vD/aOPPqrlHeZUEBE6xLWiQ1wrxvZvDziBv/1AASuz8lidlcvKrDz+viyL17/+HoCI0BBOax1JUmwESTERznPZwzfcrnUECdERhIfaVr9peE033D+eDntWN+w82w+AcY9XO3ny5Mk8+OCDFBYWEhERwbZt29i1axfp6enk5+czceJEDh48SHFxMY8++igTJ06s8P5t27ZxySWXsGbNGo4ePcp1111HZmYmffv25ejRo+XtbrrpJpYuXcrRo0eZPHkyjzzyCM8//zy7du1i9OjRJCYmsnDhQrp27UpGRgaJiYk8/fTTvPrqqwDccMMN3HHHHWzbto1x48aRnp7OV199RadOnZg3bx6tWrWq9jOuWLGCG2+8kYKCAnr06MGrr75KmzZteP755/nDH/5AaGgoqampzJkzhy+++ILbb78dcAJu0aJFzeJM1lNBROiSEE2XhGgmDOoIQEmpsiU7n1VZeazfc4jsw4Vk5xeyZX8+/92aQ25BcZXzahMVdkLwJ8VG0C42ssL4+KgwO17fBKzphrsLEhISGDZsGJ988gkTJ05kzpw5XHHFFYgIkZGRvPfee7Ru3Zr9+/czfPhwJkyYUO1/tpdeeomoqChWrVrFqlWrKlyy9ze/+Q1t27alpKSE888/n1WrVnHbbbfx9NNPs3DhwvIrO5ZZtmwZr732GkuWLEFVOeussxg5ciRt2rRh06ZNvP322/zpT3/i8ssv59133+Xqq6+u9jNec801vPDCC4wcOZKHHnqIRx55hGeffZbHH3+crVu3EhERUX4J4KeeeopZs2YxYsQI8vPziYy00/tr4gkRep0WS6/Tql4BFnpLyMkvckLfF/xlr/cdPkb24UKWbT/IvkOFFHpPvHl4mEdIjKm09R9b+VeBs0JoFW47fVu6phvuNWxhN6ayrpmycC/bWlZVHnjgARYtWkRISAg7d+5k7969tG/fvsr5LFq0iNtuuw2AgQMHMnDgwPJp77zzDi+//DJer5fdu3eTmZlZYXplX375JT/60Y/Kr0x56aWXsnjxYiZMmEC3bt3Kb+Dhf8ngquTl5ZGbm8vIkSMB+MlPfsJll11WXuNVV13FpEmTyi8BPGLECO666y6uuuoqLr30UpKTG/R+LC1ORKiHjvGt6Bhf/S8rcP6t5Rd6T1gJ7Dt8fGWwO+8Yq3bmkZNfSFW3no2JCCUpNoK20eFEhXuIDg8lKuL4c1RYKNERHqLC/Z7DPbQK9xAdEVrhPeGeEPvF0Aw13XB3yaRJk7jrrrvK77JUtsX91ltvkZ2dzbJlywgLC6Nr165VXubXX1X/IbZu3cpTTz3F0qVLadOmDddee22t86np+j9llwsG55LB/t0/dfHhhx+yaNEi5s+fz69//WvWrl3L9OnTufjii/noo48YPnw4//73v+nTp0+95m8CJyLERoYRGxlG96SYGtuWlCoHjhSVrwT2HTpW4RfBwYIiDh/zsu9QIUeKvBQUlXCk0FvlL4PqhIaIE/r+KwjfSsAZf+JKIqpS28gwD+GhIc7Dc/w5rOzZI7YCaWABhbuIjAWew7mH6iuq+nil6RHAX4EzgBzgClXd1rClnhoxMTGMGjWKn/70p0ydOrV8fF5eHu3atSMsLIyFCxfy/fff1zifc889l7feeovRo0ezZs0aVq1aBTiXC46OjiYuLo69e/fy8ccfM2rUKABiY2M5fPjwCd0y5557Ltdeey3Tp09HVXnvvfd444036vzZ4uLiaNOmDYsXL+acc87hjTfeYOTIkZSWlrJjxw5Gjx5Neno6s2fPJj8/n5ycHAYMGMCAAQP4+uuvWb9+vYV7E+MJkfIumbrwlpRSUFxCQWEJBX6hX1BU4qwECkvKVwYFRV6O+NodKSqhwNcu+3DhCW1LqvoZEaDy0A91wr58BeAJISLUf1rFFUSF8aEnTgvzbxMiePweoSEhlYZPfB0aEkJICOVtQ0MEj0fwSMV2TW3lVGu4i4gHmAX8EMgClorIfFX1v8D49cBBVe0pIlOAJ4ArGqPgU2Hq1KlceumlzJkzp3zcVVddxfjx40lLS2Pw4MG1htxNN93Eddddx8CBAxk8eDDDhg0DnLsqDRkyhH79+p1wueBp06Yxbtw4OnTowMKFC8vHDx06lGuvvbZ8HjfccANDhgypsQumOq+//nr5DtXu3bvz2muvUVJSwtVXX01eXh6qyp133kl8fDy//OUvWbhwIR6Ph9TU1PK7SpnmL9QTQmtPCK0jwxpsnqpKobeUo0UlFX4lHCksodBbQpG3lKKS0grPxWXD3lKKStT3XEKxV09oW+QtpbC4lMPHvH7vOf5cXPZc4s6VbkOk4gogpMIKouLwHWNOZ7xvR3xjqfWSvyJyNvArVb3QN3w/gKo+5tdmga/N1yISCuwBkrSGmdslf4OD/c1MU1NaqhSXHl8hFPutNLylirdEKSlVStR5rjhcWj7sLVVKVSsMl6hSUlLqvC4f1orD5fMsrXIZ3lJlypkpnNMrqV6fryEv+dsJ2OE3nAWcVV0bVfWKSB6QAOyvVNQ0YBpA586dA1i0McbUTUiIEBHiISK0ZR8xFMjZE1V1JFXeIg+kDar6sqqmqWpaUlL91lrGGGNqF0i4ZwEpfsPJwK7q2vi6ZeKAA/UpyK07Q5m6s7+VMU1XIOG+FOglIt1EJByYAsyv1GY+8BPf68nA5zX1t1cnMjKSnJwcC41mQFXJycmxE5uMaaJq7XP39aHfAizAORTyVVVdKyIzgAxVnQ/8GXhDRDbjbLFPqU8xycnJZGVlkZ2dXZ+3m1MsMjLSTmwypolqUjfINsYYU7NAj5axy9EZY0wQsnA3xpggZOFujDFByLU+dxHJBmq+QEvTl0ilE7VaOPs+jrPvoiL7Pio6me+ji6rWeqKQa+EeDEQkI5AdGy2FfR/H2XdRkX0fFZ2K78O6ZYwxJghZuBtjTBCycD85L7tdQBNj38dx9l1UZN9HRY3+fVifuzHGBCHbcjfGmCBk4W6MMUHIwr0eRCRFRBaKyDoRWSsit7tdk9tExCMi34rIB27X4jYRiReRuSKy3vdv5Gy3a3KTiNzp+3+yRkTeFpEWcylREXlVRPaJyBq/cW1F5FMR2eR7btMYy7Zwrx8vcLeq9gWGAzeLSKrLNbntdmCd20U0Ec8Bn6hqH2AQLfh7EZFOwG1Amqr2x7mybL2uGttM/QUYW2ncdOAzVe0FfOYbbnAW7vWgqrtVdbnv9WGc/7yd3K3KPSKSDFwMvOJ2LW4TkdbAuTiXwUZVi1Q1192qXBcKtPLdyCeKE2/2E7RUdREn3rhoIvC67/XrwKTGWLaF+0kSka7AEGCJu5W46lng50Cp24U0Ad2BbOA1XzfVKyIS7XZRblHVncBTwHZgN5Cnqv9ytyrXnaaqu8HZUATaNcZCLNxPgojEAO8Cd6jqIbfrcYOIXALsU9VlbtfSRIQCQ4GXVHUIcIRG+tndHPj6kycC3YCOQLSIXO1uVS2DhXs9iUgYTrC/par/cLseF40AJojINmAOcJ6IvOluSa7KArJUteyX3FycsG+pxgBbVTVbVYuBfwA/cLkmt+0VkQ4Avud9jbEQC/d6EBHB6VNdp6pPu12Pm1T1flVNVtWuODvKPlfVFrtlpqp7gB0i0ts36nwg08WS3LYdGC4iUb7/N+fTgncw+/jfc/onwLzGWEit91A1VRoB/A+wWkRW+MY9oKofuViTaTpuBd7y3VB+C3Cdy/W4RlWXiMhcYDnOUWbf0oIuRSAibwOjgEQRyQIeBh4H3hGR63FWfpc1yrLt8gPGGBN8rFvGGGOCkIW7McYEIQt3Y4wJQhbuxhgThCzcjTEmCFm4G2NMELJwN8aYIPT/AXX9M8HeEyjRAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x27f9112b940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "plt.plot(epochs, acc, label='Training acc')\n",
    "plt.plot(epochs, val_acc, label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, label='Training loss')\n",
    "plt.plot(epochs, val_loss, label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在這個例子中，預先訓練的詞嵌入確實勝過了聯合學習的詞嵌入。如果我們增加訓練樣本的數量，那麼情形很快就會改變。\n",
    "\n",
    "最後，讓我們在測試數據上評估模型。首先，我們需要標記測試數據："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_dir = os.path.join(imdb_dir, 'test') # 測試資料的路徑\n",
    "\n",
    "labels = []\n",
    "texts = []\n",
    "\n",
    "for label_type in ['neg', 'pos']:\n",
    "    dir_name = os.path.join(test_dir, label_type)\n",
    "    for fname in sorted(os.listdir(dir_name)):\n",
    "        if fname[-4:] == '.txt':\n",
    "            f = open(os.path.join(dir_name, fname), encoding='utf8')\n",
    "            texts.append(f.read())\n",
    "            f.close()\n",
    "            if label_type == 'neg':\n",
    "                labels.append(0)\n",
    "            else:\n",
    "                labels.append(1)\n",
    "\n",
    "sequences = tokenizer.texts_to_sequences(texts)\n",
    "x_test = pad_sequences(sequences, maxlen=maxlen)\n",
    "y_test = np.asarray(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們來加載並評估第一個模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/25000 [==============================] - 3s 105us/step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[3.108985914062932, 0.50156]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.load_weights('pre_trained_glove_model.h5')\n",
    "model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們得到了50.156%的測試準確性(跟用猜的準確性也差不多)。它證明要使用少量訓練樣本來訓練一個好的模型是很困難的！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 參考: \n",
    "* [fchollet: deep-learning-with-python-notebooks (原文)](https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/6.1-using-word-embeddings.ipynb)\n",
    "* [Keras官網](http://keras.io/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MIT License\n",
    "\n",
    "Copyright (c) 2017 François Chollet\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
    "of this software and associated documentation files (the \"Software\"), to deal\n",
    "in the Software without restriction, including without limitation the rights\n",
    "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
    "copies of the Software, and to permit persons to whom the Software is\n",
    "furnished to do so, subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice shall be included in all\n",
    "copies or substantial portions of the Software.\n",
    "\n",
    "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
    "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
    "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
    "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
    "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
    "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
    "SOFTWARE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}