1.c-lstm-learn-alphabetic-seq.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 循環神經網絡 LSTM (長短期記憶)來學習字母表順序"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "很多人在看過RNN或LSTM的原理說明後, 對於RNN神經網絡在序列資料的學習與應用上很難一開始就理解。在本文中，我們將開發和比較幾種不同的LSTM神經網絡模型。\n",
    "\n",
    "![lstm-abc](https://pro.guidesocial.be/images/thumbs/580x387/arton24101.jpg?fct=1456434296)\n",
    "\n",
    "我們將要使用深度學習來學習英文26個字母出現的順序。也就是說，給定一個英文字母表的某一個字母，來讓神經網絡預測下一個可能會出現的字母。\n",
    "\n",
    "> ABCDEFGHIJKLMNOPQRSTUVWXYZ\n",
    "\n",
    "> 例如: \n",
    "\n",
    "> 給 J -> 預測 K\n",
    "\n",
    "> 給 X -> 預測 Y\n",
    "\n",
    "這是一個簡單的序列預測問題，一旦被理解，就可以推廣到其他序列預測問題，如時間序列預測和序列分類。\n",
    "\n",
    "![lstm-many-to-one](https://i.stack.imgur.com/QCnpU.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型 1. 用LSTM學習一個字符到一個字符映射"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP1. 匯入 Keras 及相關模組"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import numpy\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import LSTM\n",
    "from keras.utils import np_utils\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "# 給定隨機的種子, 以便讓大家跑起來的結果是相同的\n",
    "numpy.random.seed(7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP2. 準備資料\n",
    "\n",
    "我們現在可以定義我們的數據集，字母表(alphabet)。為了便於閱讀，我們使用大寫字母來定義字母表。\n",
    "\n",
    "我們需要將字母表的每個字母映射到數字以便使用人工網絡來進行訓練。我們可以通過為字符創建字母索引的字典來輕鬆完成此操作。\n",
    "我們還可以創建一個反向查找，將預測轉換回字符以供以後使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定義序列數據集\n",
    "alphabet = \"ABCDEFGHIJKLMNOPQRSTUVWXYZ\"\n",
    "\n",
    "# 創建字符映射到整數（0 - 25)和反相的查詢字典物件\n",
    "char_to_int = dict((c, i) for i, c in enumerate(alphabet))\n",
    "int_to_char = dict((i, c) for i, c in enumerate(alphabet))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "字母對應到數字編號: \n",
      " {'O': 14, 'T': 19, 'M': 12, 'Y': 24, 'E': 4, 'I': 8, 'P': 15, 'X': 23, 'R': 17, 'V': 21, 'W': 22, 'S': 18, 'H': 7, 'Z': 25, 'D': 3, 'J': 9, 'B': 1, 'A': 0, 'Q': 16, 'C': 2, 'G': 6, 'L': 11, 'K': 10, 'F': 5, 'U': 20, 'N': 13}\n",
      "\n",
      "\n",
      "數字編號對應到字母: \n",
      " {0: 'A', 1: 'B', 2: 'C', 3: 'D', 4: 'E', 5: 'F', 6: 'G', 7: 'H', 8: 'I', 9: 'J', 10: 'K', 11: 'L', 12: 'M', 13: 'N', 14: 'O', 15: 'P', 16: 'Q', 17: 'R', 18: 'S', 19: 'T', 20: 'U', 21: 'V', 22: 'W', 23: 'X', 24: 'Y', 25: 'Z'}\n"
     ]
    }
   ],
   "source": [
    "# 打印看一下\n",
    "print(\"字母對應到數字編號: \\n\", char_to_int)\n",
    "print(\"\\n\")\n",
    "\n",
    "print(\"數字編號對應到字母: \\n\", int_to_char)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP3. 準備訓練用資料\n",
    "\n",
    "現在我們需要創建我們的輸入(X)和輸出(y)來訓練我們的神經網絡。我們可以通過定義一個輸入序列長度，然後從輸入字母序列中讀取序列。\n",
    "例如，我們使用輸入長度1.從原始輸入數據的開頭開始，我們可以讀取第一個字母“A”，下一個字母作為預測“B”。我們沿著一個字符移動並重複，直到達到“Z”的預測。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A -> B\n",
      "B -> C\n",
      "C -> D\n",
      "D -> E\n",
      "E -> F\n",
      "F -> G\n",
      "G -> H\n",
      "H -> I\n",
      "I -> J\n",
      "J -> K\n",
      "K -> L\n",
      "L -> M\n",
      "M -> N\n",
      "N -> O\n",
      "O -> P\n",
      "P -> Q\n",
      "Q -> R\n",
      "R -> S\n",
      "S -> T\n",
      "T -> U\n",
      "U -> V\n",
      "V -> W\n",
      "W -> X\n",
      "X -> Y\n",
      "Y -> Z\n"
     ]
    }
   ],
   "source": [
    "# 準備輸入數據集\n",
    "seq_length = 1\n",
    "dataX = []\n",
    "dataY = []\n",
    "for i in range(0, len(alphabet) - seq_length, 1):\n",
    "    seq_in = alphabet[i:i + seq_length]\n",
    "    seq_out = alphabet[i + seq_length]\n",
    "    dataX.append([char_to_int[char] for char in seq_in])\n",
    "    dataY.append(char_to_int[seq_out])\n",
    "    print(seq_in, '->', seq_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP4. 資料預處理\n",
    "我們需要將NumPy數組重塑為LSTM網絡所期望的格式，也就是: (samples, time_steps, features)。\n",
    "同時我們將進行資料的歸一化(normalize)來讓資料的值落於0到1之間。並對標籤值進行one-hot的編碼。\n",
    "\n",
    "\n",
    "> ABCDEFGHIJKLMNOPQRSTUVWXYZ\n",
    "\n",
    "> 例如: \n",
    "\n",
    "> 給 J -> 預測 K\n",
    "\n",
    "> 給 X -> 預測 Y\n",
    "\n",
    "\n",
    "目標訓練張量結構: (samples, time_steps, features) -> (n , **1**, **1** )\n",
    "\n",
    "請特別注意, 這裡的1個字符會變成1個時間步裡頭的1個element的\"feature\"向量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X shape:  (25, 1, 1)\n",
      "y shape:  (25, 26)\n"
     ]
    }
   ],
   "source": [
    "# 重塑 X 資料的維度成為 (samples, time_steps, features)\n",
    "X = numpy.reshape(dataX, (len(dataX), seq_length, 1))\n",
    "\n",
    "# 歸一化\n",
    "X = X / float(len(alphabet))\n",
    "\n",
    "# one-hot 編碼輸出變量\n",
    "y = np_utils.to_categorical(dataY)\n",
    "\n",
    "print(\"X shape: \", X.shape) # (25筆samples, \"1\"個時間步長, 1個feature)\n",
    "print(\"y shape: \", y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP5. 建立模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "lstm_1 (LSTM)                (None, 32)                4352      \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 26)                858       \n",
      "=================================================================\n",
      "Total params: 5,210\n",
      "Trainable params: 5,210\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# 創建模型\n",
    "model = Sequential()\n",
    "model.add(LSTM(32, input_shape=(X.shape[1], X.shape[2])))\n",
    "model.add(Dense(y.shape[1], activation='softmax'))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP6. 定義訓練並進行訓練"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/500\n",
      " - 2s - loss: 3.2660 - acc: 0.0000e+00\n",
      "Epoch 2/500\n",
      " - 0s - loss: 3.2582 - acc: 0.0000e+00\n",
      "Epoch 3/500\n",
      " - 0s - loss: 3.2551 - acc: 0.0400\n",
      "Epoch 4/500\n",
      " - 0s - loss: 3.2524 - acc: 0.0400\n",
      "Epoch 5/500\n",
      " - 0s - loss: 3.2495 - acc: 0.0400\n",
      "Epoch 6/500\n",
      " - 0s - loss: 3.2470 - acc: 0.0400\n",
      "Epoch 7/500\n",
      " - 0s - loss: 3.2440 - acc: 0.0400\n",
      "Epoch 8/500\n",
      " - 0s - loss: 3.2411 - acc: 0.0400\n",
      "Epoch 9/500\n",
      " - 0s - loss: 3.2378 - acc: 0.0400\n",
      "Epoch 10/500\n",
      " - 0s - loss: 3.2348 - acc: 0.0400\n",
      "...\n"
     ]
    },    
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [      
      "Epoch 490/500\n",
      " - 0s - loss: 1.7054 - acc: 0.7200\n",
      "Epoch 491/500\n",
      " - 0s - loss: 1.7041 - acc: 0.7600\n",
      "Epoch 492/500\n",
      " - 0s - loss: 1.7029 - acc: 0.8800\n",
      "Epoch 493/500\n",
      " - 0s - loss: 1.7021 - acc: 0.7600\n",
      "Epoch 494/500\n",
      " - 0s - loss: 1.7024 - acc: 0.8800\n",
      "Epoch 495/500\n",
      " - 0s - loss: 1.6992 - acc: 0.7600\n",
      "Epoch 496/500\n",
      " - 0s - loss: 1.7001 - acc: 0.8000\n",
      "Epoch 497/500\n",
      " - 0s - loss: 1.6995 - acc: 0.6800\n",
      "Epoch 498/500\n",
      " - 0s - loss: 1.6994 - acc: 0.7600\n",
      "Epoch 499/500\n",
      " - 0s - loss: 1.7001 - acc: 0.8000\n",
      "Epoch 500/500\n",
      " - 0s - loss: 1.6963 - acc: 0.8400\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x2145c550>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "model.fit(X, y, epochs=500, batch_size=1, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP7. 評估模型準確率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Accuracy: 92.00%\n"
     ]
    }
   ],
   "source": [
    "# 評估模型的性能\n",
    "scores = model.evaluate(X, y, verbose=0)\n",
    "print(\"Model Accuracy: %.2f%%\" % (scores[1]*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP8. 預測結果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['A'] -> B\n",
      "['B'] -> C\n",
      "['C'] -> D\n",
      "['D'] -> E\n",
      "['E'] -> F\n",
      "['F'] -> G\n",
      "['G'] -> H\n",
      "['H'] -> I\n",
      "['I'] -> J\n",
      "['J'] -> K\n",
      "['K'] -> L\n",
      "['L'] -> M\n",
      "['M'] -> N\n",
      "['N'] -> O\n",
      "['O'] -> P\n",
      "['P'] -> Q\n",
      "['Q'] -> R\n",
      "['R'] -> S\n",
      "['S'] -> T\n",
      "['T'] -> U\n",
      "['U'] -> V\n",
      "['V'] -> W\n",
      "['W'] -> Y\n",
      "['X'] -> Z\n",
      "['Y'] -> Z\n"
     ]
    }
   ],
   "source": [
    "# 展示模型預測能力\n",
    "for pattern in dataX:\n",
    "    # 把26個字母一個個拿進模型來預測會出現的字母\n",
    "    x = numpy.reshape(pattern, (1, len(pattern), 1))\n",
    "    x = x / float(len(alphabet))\n",
    "    \n",
    "    prediction = model.predict(x, verbose=0)\n",
    "    index = numpy.argmax(prediction) # 機率最大的idx\n",
    "    result = int_to_char[index] # 看看預測出來的是那一個字母\n",
    "    seq_in = [int_to_char[value] for value in pattern]\n",
    "    print(seq_in, \"->\", result) # 打印結果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們可以看到，\"序列資料的預測\"這個問題對於網絡學習確實是困難的。\n",
    "原因是，在以上的範例中的LSTM單位沒有任何上下文的知識(時間歩長只有\"1\")。每個輸入輸出模式以隨機順序(shuffle)出現到人工網網絡上，而且Keras的LSTM網絡內步狀態(state)會在每個訓練循環(epoch)後被重置(reset)。\n",
    "\n",
    "接下來，讓我們嘗試提供更多的順序資訊來讓LSTM學習。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型 2. LSTM 學習三個字符特徵窗口(Three-Char Feature Window)到一個字符映射\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP1. 準備訓練用資料"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ABC -> D\n",
      "BCD -> E\n",
      "CDE -> F\n",
      "DEF -> G\n",
      "EFG -> H\n",
      "FGH -> I\n",
      "GHI -> J\n",
      "HIJ -> K\n",
      "IJK -> L\n",
      "JKL -> M\n",
      "KLM -> N\n",
      "LMN -> O\n",
      "MNO -> P\n",
      "NOP -> Q\n",
      "OPQ -> R\n",
      "PQR -> S\n",
      "QRS -> T\n",
      "RST -> U\n",
      "STU -> V\n",
      "TUV -> W\n",
      "UVW -> X\n",
      "VWX -> Y\n",
      "WXY -> Z\n"
     ]
    }
   ],
   "source": [
    "# 準備輸入數據集\n",
    "seq_length = 3 # 這次我們要準備3個時間步長的資料\n",
    "dataX = []\n",
    "dataY = []\n",
    "for i in range(0, len(alphabet) - seq_length, 1):\n",
    "    seq_in = alphabet[i:i + seq_length] # 3個字符\n",
    "    seq_out = alphabet[i + seq_length]\n",
    "    dataX.append([char_to_int[char] for char in seq_in])\n",
    "    dataY.append(char_to_int[seq_out])\n",
    "    print(seq_in, '->', seq_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP2. 資料預處理\n",
    "\n",
    "\n",
    "> ABCDEFGHIJKLMNOPQRSTUVWXYZ\n",
    "\n",
    "> 例如: \n",
    "\n",
    "> 給 HIJ -> 預測 K\n",
    "\n",
    "> 給 EFG -> 預測 H\n",
    "\n",
    "目標訓練張量結構: (samples, time_steps, features) -> (n , **1**, **3** )\n",
    "\n",
    "請特別注意, 這裡的三個字符會變成一個有3個element的\"feature\" vector。因此在準備訓練資料集的時候, 1筆訓練資料只有\"1\"個時間步, 裡頭存放著\"3\"個字符的資料\"features\"向量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X shape:  (23, 1, 3)\n",
      "y shape:  (23, 26)\n"
     ]
    }
   ],
   "source": [
    "# 重塑 X 資料的維度成為 (samples, time_steps, features)\n",
    "X = numpy.reshape(dataX, (len(dataX), 1, seq_length))  # <-- 特別注意這裡\n",
    "\n",
    "# 歸一化\n",
    "X = X / float(len(alphabet))\n",
    "\n",
    "# 使用one hot encode 對Y值進行編碼\n",
    "y = np_utils.to_categorical(dataY)\n",
    "\n",
    "print(\"X shape: \", X.shape)\n",
    "print(\"y shape: \", y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP3. 建立模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "lstm_2 (LSTM)                (None, 32)                4608      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 26)                858       \n",
      "=================================================================\n",
      "Total params: 5,466\n",
      "Trainable params: 5,466\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# 創建模型\n",
    "model = Sequential()\n",
    "model.add(LSTM(32, input_shape=(X.shape[1], X.shape[2]))) # <-- 特別注意這裡\n",
    "model.add(Dense(y.shape[1], activation='softmax'))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP4. 定義訓練並進行訓練"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/500\n",
      " - 2s - loss: 3.2753 - acc: 0.0435\n",
      "Epoch 2/500\n",
      " - 0s - loss: 3.2629 - acc: 0.0000e+00\n",
      "Epoch 3/500\n",
      " - 0s - loss: 3.2556 - acc: 0.0000e+00\n",
      "Epoch 4/500\n",
      " - 0s - loss: 3.2487 - acc: 0.0435\n",
      "Epoch 5/500\n",
      " - 0s - loss: 3.2420 - acc: 0.0435\n",
      "Epoch 6/500\n",
      " - 0s - loss: 3.2355 - acc: 0.0435\n",
      "Epoch 7/500\n",
      " - 0s - loss: 3.2294 - acc: 0.0435\n",
      "Epoch 8/500\n",
      " - 0s - loss: 3.2214 - acc: 0.0435\n",
      "Epoch 9/500\n",
      " - 0s - loss: 3.2142 - acc: 0.0435\n",
      "Epoch 10/500\n",
      " - 0s - loss: 3.2056 - acc: 0.0435\n",
      "...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [      
      "Epoch 490/500\n",
      " - 0s - loss: 1.6114 - acc: 0.7826\n",
      "Epoch 491/500\n",
      " - 0s - loss: 1.6089 - acc: 0.7826\n",
      "Epoch 492/500\n",
      " - 0s - loss: 1.6108 - acc: 0.8261\n",
      "Epoch 493/500\n",
      " - 0s - loss: 1.6091 - acc: 0.7826\n",
      "Epoch 494/500\n",
      " - 0s - loss: 1.6057 - acc: 0.7826\n",
      "Epoch 495/500\n",
      " - 0s - loss: 1.6060 - acc: 0.7826\n",
      "Epoch 496/500\n",
      " - 0s - loss: 1.6058 - acc: 0.8261\n",
      "Epoch 497/500\n",
      " - 0s - loss: 1.6045 - acc: 0.7826\n",
      "Epoch 498/500\n",
      " - 0s - loss: 1.6042 - acc: 0.7826\n",
      "Epoch 499/500\n",
      " - 0s - loss: 1.6006 - acc: 0.8696\n",
      "Epoch 500/500\n",
      " - 0s - loss: 1.6011 - acc: 0.7826\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x235b8d68>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "model.fit(X, y, epochs=500, batch_size=1, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP5. 評估模型準確率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Accuracy: 82.61%\n"
     ]
    }
   ],
   "source": [
    "# 評估模型的性能\n",
    "scores = model.evaluate(X, y, verbose=0)\n",
    "print(\"Model Accuracy: %.2f%%\" % (scores[1]*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP6. 預測結果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['A', 'B', 'C'] -> D\n",
      "['B', 'C', 'D'] -> E\n",
      "['C', 'D', 'E'] -> F\n",
      "['D', 'E', 'F'] -> G\n",
      "['E', 'F', 'G'] -> H\n",
      "['F', 'G', 'H'] -> I\n",
      "['G', 'H', 'I'] -> J\n",
      "['H', 'I', 'J'] -> K\n",
      "['I', 'J', 'K'] -> L\n",
      "['J', 'K', 'L'] -> M\n",
      "['K', 'L', 'M'] -> N\n",
      "['L', 'M', 'N'] -> O\n",
      "['M', 'N', 'O'] -> P\n",
      "['N', 'O', 'P'] -> Q\n",
      "['O', 'P', 'Q'] -> R\n",
      "['P', 'Q', 'R'] -> S\n",
      "['Q', 'R', 'S'] -> T\n",
      "['R', 'S', 'T'] -> U\n",
      "['S', 'T', 'U'] -> W\n",
      "['T', 'U', 'V'] -> X\n",
      "['U', 'V', 'W'] -> Z\n",
      "['V', 'W', 'X'] -> Z\n",
      "['W', 'X', 'Y'] -> Z\n"
     ]
    }
   ],
   "source": [
    "# 展示一些模型預測\n",
    "for pattern in dataX:\n",
    "    x = numpy.reshape(pattern, (1, 1, len(pattern)))\n",
    "    x = x / float(len(alphabet))\n",
    "    prediction = model.predict(x, verbose=0)\n",
    "    index = numpy.argmax(prediction)\n",
    "    result = int_to_char[index]\n",
    "    seq_in = [int_to_char[value] for value in pattern]\n",
    "    print(seq_in, \"->\", result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們可以看到，\"模型#2\"相比於\"模型#1\"在預測的表現上只有小幅提升。這個簡單的問題，即使使用window方法，我們仍然無法讓LSTM學習到預測正確的字母出現的順序。\n",
    "\n",
    "以上的範例也是一個誤用LSTM網絡的糟糕的張量結構。事實上，字母序列是一個特徵的\"時間步驟(timesteps)\"，而不是單獨特徵的一個時間步驟。我們已經給了網絡更多的上下文，但是沒有更多的順序上下文(context)。\n",
    "\n",
    "下一範例中，我們將以\"時間步驟(timesteps)\"的形式給出更多的上下文(context)。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型 3. LSTM 學習三個字符的時間步驟窗口(Three-Char Time Step Window)到一個字符的映射"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP1. 準備訓練用資料"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ABC -> D\n",
      "BCD -> E\n",
      "CDE -> F\n",
      "DEF -> G\n",
      "EFG -> H\n",
      "FGH -> I\n",
      "GHI -> J\n",
      "HIJ -> K\n",
      "IJK -> L\n",
      "JKL -> M\n",
      "KLM -> N\n",
      "LMN -> O\n",
      "MNO -> P\n",
      "NOP -> Q\n",
      "OPQ -> R\n",
      "PQR -> S\n",
      "QRS -> T\n",
      "RST -> U\n",
      "STU -> V\n",
      "TUV -> W\n",
      "UVW -> X\n",
      "VWX -> Y\n",
      "WXY -> Z\n"
     ]
    }
   ],
   "source": [
    "seq_length = 3\n",
    "dataX = []\n",
    "dataY = []\n",
    "for i in range(0, len(alphabet) - seq_length, 1):\n",
    "    seq_in = alphabet[i:i + seq_length]\n",
    "    seq_out = alphabet[i + seq_length]\n",
    "    dataX.append([char_to_int[char] for char in seq_in])\n",
    "    dataY.append(char_to_int[seq_out])\n",
    "    print(seq_in, '->', seq_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP2. 資料預處理\n",
    "\n",
    "\n",
    "> ABCDEFGHIJKLMNOPQRSTUVWXYZ\n",
    "\n",
    "> 例如: \n",
    "\n",
    "> 給 HIJ -> 預測 K\n",
    "\n",
    "> 給 EFG -> 預測 H\n",
    "\n",
    "目標訓練張量結構: (samples, time_steps, features) -> (n , **3**, **1** )\n",
    "\n",
    "準備訓練資料集的時候要把資料的張量結構轉換成, 1筆訓練資料有\"3\"個時間步, 裡頭存放著\"1\"個字符的資料\"features\"向量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 重塑 X 資料的維度成為 (samples, time_steps, features)\n",
    "X = numpy.reshape(dataX, (len(dataX), seq_length, 1))  # <-- 特別注意這裡\n",
    "\n",
    "# 歸一化\n",
    "X = X / float(len(alphabet))\n",
    "\n",
    "# 使用one hot encode 對Y值進行編碼\n",
    "y = np_utils.to_categorical(dataY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP3. 建立模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "lstm_3 (LSTM)                (None, 32)                4352      \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 26)                858       \n",
      "=================================================================\n",
      "Total params: 5,210\n",
      "Trainable params: 5,210\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# 創建模型\n",
    "model = Sequential()\n",
    "model.add(LSTM(32, input_shape=(X.shape[1], X.shape[2]))) # <-- 特別注意這裡\n",
    "model.add(Dense(y.shape[1], activation='softmax'))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP4. 定義訓練並進行訓練"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/500\n",
      " - 2s - loss: 3.2632 - acc: 0.0000e+00\n",
      "Epoch 2/500\n",
      " - 0s - loss: 3.2500 - acc: 0.0000e+00\n",
      "Epoch 3/500\n",
      " - 0s - loss: 3.2425 - acc: 0.0435\n",
      "Epoch 4/500\n",
      " - 0s - loss: 3.2357 - acc: 0.0000e+00\n",
      "Epoch 5/500\n",
      " - 0s - loss: 3.2285 - acc: 0.0000e+00\n",
      "Epoch 6/500\n",
      " - 0s - loss: 3.2207 - acc: 0.0435\n",
      "Epoch 7/500\n",
      " - 0s - loss: 3.2125 - acc: 0.0435\n",
      "Epoch 8/500\n",
      " - 0s - loss: 3.2036 - acc: 0.0435\n",
      "Epoch 9/500\n",
      " - 0s - loss: 3.1919 - acc: 0.0435\n",
      "Epoch 10/500\n",
      " - 0s - loss: 3.1821 - acc: 0.0435\n",
      "...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [      
      "Epoch 490/500\n",
      " - 0s - loss: 0.2457 - acc: 1.0000\n",
      "Epoch 491/500\n",
      " - 0s - loss: 0.2387 - acc: 1.0000\n",
      "Epoch 492/500\n",
      " - 0s - loss: 0.2394 - acc: 1.0000\n",
      "Epoch 493/500\n",
      " - 0s - loss: 0.2384 - acc: 1.0000\n",
      "Epoch 494/500\n",
      " - 0s - loss: 0.2416 - acc: 1.0000\n",
      "Epoch 495/500\n",
      " - 0s - loss: 0.2385 - acc: 1.0000\n",
      "Epoch 496/500\n",
      " - 0s - loss: 0.2380 - acc: 1.0000\n",
      "Epoch 497/500\n",
      " - 0s - loss: 0.2331 - acc: 1.0000\n",
      "Epoch 498/500\n",
      " - 0s - loss: 0.2341 - acc: 1.0000\n",
      "Epoch 499/500\n",
      " - 0s - loss: 0.2371 - acc: 1.0000\n",
      "Epoch 500/500\n",
      " - 0s - loss: 0.2325 - acc: 1.0000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x261e7ac8>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "model.fit(X, y, epochs=500, batch_size=1, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP5. 評估模型準確率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Accuracy: 100.00%\n"
     ]
    }
   ],
   "source": [
    "# 評估模型的性能\n",
    "scores = model.evaluate(X, y, verbose=0)\n",
    "print(\"Model Accuracy: %.2f%%\" % (scores[1]*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP6. 預測結果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['A', 'B', 'C'] -> D\n",
      "['B', 'C', 'D'] -> E\n",
      "['C', 'D', 'E'] -> F\n",
      "['D', 'E', 'F'] -> G\n",
      "['E', 'F', 'G'] -> H\n",
      "['F', 'G', 'H'] -> I\n",
      "['G', 'H', 'I'] -> J\n",
      "['H', 'I', 'J'] -> K\n",
      "['I', 'J', 'K'] -> L\n",
      "['J', 'K', 'L'] -> M\n",
      "['K', 'L', 'M'] -> N\n",
      "['L', 'M', 'N'] -> O\n",
      "['M', 'N', 'O'] -> P\n",
      "['N', 'O', 'P'] -> Q\n",
      "['O', 'P', 'Q'] -> R\n",
      "['P', 'Q', 'R'] -> S\n",
      "['Q', 'R', 'S'] -> T\n",
      "['R', 'S', 'T'] -> U\n",
      "['S', 'T', 'U'] -> V\n",
      "['T', 'U', 'V'] -> W\n",
      "['U', 'V', 'W'] -> X\n",
      "['V', 'W', 'X'] -> Y\n",
      "['W', 'X', 'Y'] -> Z\n"
     ]
    }
   ],
   "source": [
    "# 讓我們擷取3個字符轉成張量結構 shape:(1,3,1)來進行infer\n",
    "for pattern in dataX:\n",
    "    x = numpy.reshape(pattern, (1, len(pattern), 1))\n",
    "    x = x / float(len(alphabet))\n",
    "    prediction = model.predict(x, verbose=0)\n",
    "    index = numpy.argmax(prediction)\n",
    "    result = int_to_char[index]\n",
    "    seq_in = [int_to_char[value] for value in pattern]\n",
    "    print(seq_in, \"->\", result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由\"模型#3\"的表現來看, 當我們以\"時間步驟(timesteps)\"的形式給出更多的上下文(context)來訓練LSTM模型時, 這時候循環神經網絡在序列資料的學習的效果就可以發揮出它的效用。\n",
    "\n",
    "\"模型#3\"在驗證的結果可達到100%的預測準確度(在這個很簡單的26個字母的順序預測的任務上)!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型 4. LSTM學習可變長度字符輸入到單字符輸出\n",
    "\n",
    "讓我們建立一個模型，來接受\"變動字母序列(variable-length)\"的輸入來預測下一個字母。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP1. 準備訓練用資料\n",
    "\n",
    "為了簡化，我們將定義一個最大輸入序列長度(比如說\"5\", 代表輸入的序列可以是 1 ~ 5)，以加速訓練。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "UVWXY -> Z\n",
      "UVWXY -> Z\n",
      "EFGH -> I\n",
      "GHIJ -> K\n",
      "EFGH -> I\n",
      "DEF -> G\n",
      "CDEFG -> H\n",
      "OP -> Q\n",
      "X -> Y\n",
      "TU -> V\n",
      "IJK -> L\n",
      "LMNOP -> Q\n",
      "T -> U\n",
      "UVWX -> Y\n",
      "X -> Y\n",
      "H -> I\n",
      "EFGHI -> J\n",
      "QRSTU -> V\n",
      "EFG -> H\n",
      "RSTU -> V\n",
      "QRST -> U\n",      
      "QR -> S\n",
      "JK -> L\n",
      "GHI -> J\n",
      "KL -> M\n",
      "BCDE -> F\n",
      "AB -> C\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KLMNO -> P\n",
      "UVWXY -> Z\n",
      "EFGH -> I\n",
      "FG -> H\n",
      "DEF -> G\n",
      "STU -> V\n",
      "FGHI -> J\n",
      "OP -> Q\n",
      "FGHIJ -> K\n",
      "LMNOP -> Q\n",
      "DEF -> G\n",
      "W -> X\n",
      "KLMN -> O\n",
      "WXY -> Z\n",
      "PQRST -> U\n",
      "LMNOP -> Q\n",
      "PQ -> R\n",
      "FGHI -> J\n",
      "QRS -> T\n",
      "CDEFG -> H\n",
      "VW -> X\n",
      "DEF -> G\n",
	  "..."
     ]
    }
   ],
   "source": [
    "# 準備訓練資料\n",
    "num_inputs = 1000\n",
    "max_len = 5 # 最大序列長度\n",
    "dataX = []\n",
    "dataY = []\n",
    "for i in range(num_inputs):\n",
    "    start = numpy.random.randint(len(alphabet)-2)\n",
    "    end = numpy.random.randint(start, min(start+max_len,len(alphabet)-1))\n",
    "    sequence_in = alphabet[start:end+1]\n",
    "    sequence_out = alphabet[end + 1]\n",
    "    dataX.append([char_to_int[char] for char in sequence_in])\n",
    "    dataY.append(char_to_int[sequence_out])\n",
    "    print(sequence_in, '->', sequence_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP2. 資料預處理\n",
    "因為輸入序列的長度會在1到max_len之間變動，因此需要以\"0\"來填充(padding)。在這裡，我們使用Keras內附的pad_sequences（）函數並設定使用左側（前綴）填充。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 將訓練資料轉換為陣列和並進行序列填充（如果需要）\n",
    "X = pad_sequences(dataX, maxlen=max_len, dtype='float32') # <-- 注意這裡\n",
    "\n",
    "# 重塑 X 資料的維度成為 (samples, time_steps, features)\n",
    "X = numpy.reshape(X, (X.shape[0], max_len, 1)) # <-- 特別注意這裡\n",
    "\n",
    "# 歸一化\n",
    "X = X / float(len(alphabet))\n",
    "\n",
    "# 使用one hot encode 對Y值進行編碼\n",
    "y = np_utils.to_categorical(dataY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP3. 建立模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "lstm_4 (LSTM)                (None, 32)                4352      \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 26)                858       \n",
      "=================================================================\n",
      "Total params: 5,210\n",
      "Trainable params: 5,210\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# 創建模型\n",
    "batch_size = 1\n",
    "model = Sequential()\n",
    "model.add(LSTM(32, input_shape=(X.shape[1], 1))) # <-- 注意這裡\n",
    "model.add(Dense(y.shape[1], activation='softmax'))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP4. 定義訓練並進行訓練"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/500\n",
      " - 7s - loss: 3.0984 - acc: 0.0770\n",
      "Epoch 2/500\n",
      " - 6s - loss: 2.8499 - acc: 0.1180\n",
      "Epoch 3/500\n",
      " - 7s - loss: 2.5159 - acc: 0.1990\n",
      "Epoch 4/500\n",
      " - 6s - loss: 2.2225 - acc: 0.2440\n",
      "Epoch 5/500\n",
      " - 6s - loss: 2.0490 - acc: 0.2880\n",
      "Epoch 6/500\n",
      " - 5s - loss: 1.9284 - acc: 0.3050\n",
      "Epoch 7/500\n",
      " - 5s - loss: 1.8142 - acc: 0.3530\n",
      "Epoch 8/500\n",
      " - 6s - loss: 1.7206 - acc: 0.3940\n",
      "Epoch 9/500\n",
      " - 6s - loss: 1.6397 - acc: 0.4070\n",
      "Epoch 10/500\n",
      " - 6s - loss: 1.5607 - acc: 0.4340\n",
      "...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [      
      "Epoch 490/500\n",
      " - 5s - loss: 0.2087 - acc: 0.9550\n",
      "Epoch 491/500\n",
      " - 5s - loss: 0.0767 - acc: 0.9910\n",
      "Epoch 492/500\n",
      " - 5s - loss: 0.0779 - acc: 0.9900\n",
      "Epoch 493/500\n",
      " - 5s - loss: 0.0790 - acc: 0.9920\n",
      "Epoch 494/500\n",
      " - 5s - loss: 0.0812 - acc: 0.9860\n",
      "Epoch 495/500\n",
      " - 5s - loss: 0.0815 - acc: 0.9870\n",
      "Epoch 496/500\n",
      " - 5s - loss: 0.0811 - acc: 0.9840\n",
      "Epoch 497/500\n",
      " - 6s - loss: 0.1914 - acc: 0.9610\n",
      "Epoch 498/500\n",
      " - 5s - loss: 0.1258 - acc: 0.9730\n",
      "Epoch 499/500\n",
      " - 5s - loss: 0.0752 - acc: 0.9940\n",
      "Epoch 500/500\n",
      " - 5s - loss: 0.0786 - acc: 0.9890\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x261918d0>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "model.fit(X, y, epochs=500, batch_size=batch_size, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP5. 評估模型準確率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Accuracy: 98.50%\n"
     ]
    }
   ],
   "source": [
    "# 評估模型的性能\n",
    "scores = model.evaluate(X, y, verbose=0)\n",
    "print(\"Model Accuracy: %.2f%%\" % (scores[1]*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP6. 預測結果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['T', 'U', 'V', 'W', 'X'] -> Y\n",
      "['B', 'C'] -> D\n",
      "['G', 'H', 'I'] -> J\n",
      "['Q', 'R', 'S', 'T'] -> U\n",
      "['D', 'E', 'F'] -> G\n",
      "['I', 'J', 'K'] -> L\n",
      "['G', 'H', 'I'] -> J\n",
      "['K', 'L', 'M'] -> N\n",
      "['N'] -> O\n",
      "['A', 'B', 'C', 'D', 'E'] -> F\n",
      "['X'] -> Y\n",
      "['A', 'B', 'C', 'D'] -> E\n",
      "['V'] -> W\n",
      "['Q', 'R', 'S', 'T', 'U'] -> V\n",
      "['B', 'C', 'D', 'E', 'F'] -> G\n",
      "['R'] -> S\n",
      "['W'] -> X\n",
      "['A'] -> B\n",
      "['E', 'F', 'G'] -> H\n",
      "['T'] -> U\n"
     ]
    }
   ],
   "source": [
    "# 讓我們擷取1~5個字符轉成張量結構 shape:(1,5,1)來進行infer\n",
    "for i in range(20):\n",
    "    pattern_index = numpy.random.randint(len(dataX))\n",
    "    pattern = dataX[pattern_index]\n",
    "    x = pad_sequences([pattern], maxlen=max_len, dtype='float32')\n",
    "    x = numpy.reshape(x, (1, max_len, 1))\n",
    "    x = x / float(len(alphabet))\n",
    "    prediction = model.predict(x, verbose=0)\n",
    "    index = numpy.argmax(prediction)\n",
    "    result = int_to_char[index]\n",
    "    seq_in = [int_to_char[value] for value in pattern]\n",
    "    print(seq_in, \"->\", result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "我們可以看到，雖然這個網絡模型沒有從生成的序列資料中完全學習到英文字母表的順序，但它表現相當的好。如果需要, 我們可以對這個模型進行進一歩的優化與調整，比如更多的訓練循環(more epochs)或更大的網絡(larger network)，或兩者。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 參考:\n",
    "* Jason Brownlee - \"[Understanding Stateful LSTM Recurrent Neural Networks in Python with Keras](https://machinelearningmastery.com/understanding-stateful-lstm-recurrent-neural-networks-python-keras/)\"\n",
    "\n",
    "* Keras官網 - [Recurrent Layer](https://keras.io/layers/recurrent/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}