notebooks/Model Example.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "from sklearn.metrics import accuracy_score, f1_score\n",
    "\n",
    "from bella.parsers import semeval_14\n",
    "from bella.helper import read_config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to use the pre-trained models\n",
    "\n",
    "A lot of the models trained in the model zoo took a long time to train espically the Neural Network models which include LSTM, TDLSTM, and TCLSTM. In this notebook we are going to show you how to:\n",
    "1. Load the models\n",
    "2. Use the models to predict on some of the test data\n",
    "\n",
    "First we need the path to the model zoo:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LSTM SemEval 14 Restaurant.h5\n",
      "LSTM SemEval 14 Restaurant.pkl\n",
      "LSTM YouTuBean.h5\n",
      "LSTM YouTuBean.pkl\n",
      "LSTM SemEval 14 Laptop.h5\n",
      "LSTM SemEval 14 Laptop.pkl\n",
      "LSTM Mitchel.h5\n",
      "LSTM Mitchel.pkl\n",
      "TDLSTM YouTuBean.h5\n",
      "TDLSTM YouTuBean.pkl\n",
      "LSTM Dong Twitter.h5\n",
      "LSTM Dong Twitter.pkl\n",
      "TCLSTM YouTuBean.h5\n",
      "TCLSTM YouTuBean.pkl\n",
      "TDLSTM Mitchel.h5\n",
      "TDLSTM Mitchel.pkl\n",
      "TDLSTM Dong Twitter.h5\n",
      "TDLSTM Dong Twitter.pkl\n",
      "LSTM Election Twitter.h5\n",
      "LSTM Election Twitter.pkl\n",
      "TDLSTM SemEval 14 Restaurant.h5\n",
      "TDLSTM SemEval 14 Restaurant.pkl\n",
      "TCLSTM Mitchel.h5\n",
      "TCLSTM Mitchel.pkl\n",
      "TDLSTM SemEval 14 Laptop.h5\n",
      "TDLSTM SemEval 14 Laptop.pkl\n",
      "TDLSTM Election Twitter.h5\n",
      "TDLSTM Election Twitter.pkl\n",
      "TCLSTM SemEval 14 Laptop.h5\n",
      "TCLSTM SemEval 14 Laptop.pkl\n",
      "TCLSTM Dong Twitter.h5\n",
      "TCLSTM Dong Twitter.pkl\n",
      "TCLSTM Election Twitter.h5\n",
      "TCLSTM Election Twitter.pkl\n",
      "TCLSTM SemEval 14 Restaurant.h5\n",
      "TCLSTM SemEval 14 Restaurant.pkl\n",
      "Target Dependent Mitchel\n",
      "Target Dependent Plus Mitchel\n",
      "Target Dependent Election Twitter\n",
      "Target Dependent Plus Election Twitter\n",
      "Target Dependent YouTuBean\n",
      "TDParse Mitchel\n",
      "Target Dependent Plus YouTuBean\n",
      "TDParsePlus Mitchel\n",
      "TDParse SemEval 14 Laptop\n",
      "Target Dependent SemEval 14 Restaurant\n",
      "TDParse Election Twitter\n",
      "Target Dependent Plus SemEval 14 Restaurant\n",
      "TDParsePlus Election Twitter\n",
      "Target Dependent Dong Twitter\n",
      "Target Dependent Plus Dong Twitter\n",
      "Target Dependent SemEval 14 Laptop\n",
      "Target Dependent Plus SemEval 14 Laptop\n",
      "TDParse YouTuBean\n",
      "TDParsePlus YouTuBean\n",
      "TDParse SemEval 14 Restaurant\n",
      "TDParsePlus SemEval 14 Restaurant\n",
      "TDParse Dong Twitter\n",
      "TDParsePlus Dong Twitter\n",
      "TDParsePlus SemEval 14 Laptop\n"
     ]
    }
   ],
   "source": [
    "##\n",
    "#  ADD YOUR CONFIG FILE PATH HERE \n",
    "##\n",
    "CONFIG_FP = Path('..', 'config.yaml')\n",
    "model_zoo_folder = Path(read_config('model_zoo_folder', CONFIG_FP))\n",
    "\n",
    "\n",
    "for model_file in model_zoo_folder.iterdir():\n",
    "    print(model_file.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the model zoo you should be able to see the list of models as shown above. The models in there can be broken up into different methods which are the following: \n",
    "1. Target Dependent\n",
    "2. Target Dependent Plus\n",
    "3. TDParse\n",
    "4. TDParse Plus\n",
    "5. LSTM\n",
    "6. TDLSTM\n",
    "7. TCLSTM \n",
    "\n",
    "Each one of these methods was trained on the datasets which can be found in the [Dataset notebook](./datasets.ipynb). Therefore we have a model for each method on each dataset, if you would like to see how these models were trained look at the following Mass Evaluation notebooks: \n",
    "1. [Target Dependent methods](./Mass%20Evaluation%20-%20Target%20Dependent.ipynb)\n",
    "2. [TDParse methods](./Mass%20Evaluation%20-%20TDParse.ipynb)\n",
    "3. [LSTM methods](./Mass%20Evaluation%20LSTM.ipynb)\n",
    "\n",
    "We are going to show how to load a model from each of the methods on the [SemEval 2014 Resturant dataset](http://alt.qcri.org/semeval2014/task4/index.php?id=data-and-tools).\n",
    "\n",
    "We are going to load one model at a time as each model can take:\n",
    "1. A long time to load\n",
    "2. Take up a lot of memory\n",
    "\n",
    "For each model we will:\n",
    "1. Load it\n",
    "2. Predict the sentiment of the test data on the SemEval 2014 Resturant dataset\n",
    "3. Evaluate the performance using Accuracy and Macro F1 score.\n",
    "\n",
    "But before going through each model we need to load the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "CONFIG_FP = Path('..', 'config.yaml')\n",
    "\n",
    "train_data = semeval_14(read_config('semeval_2014_rest_train', CONFIG_FP))\n",
    "test_data = semeval_14(read_config('semeval_2014_rest_test', CONFIG_FP))\n",
    "y_test = test_data.sentiment_data()\n",
    "dataset_name = 'SemEval 14 Restaurant'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the test data has been loaded all the cells grouped by models can be run independently.\n",
    "\n",
    "## Target Dependent and Target Dependent Plus Models\n",
    "\n",
    "These two methods come from [Vo and Zhang's paper](https://www.ijcai.org/Proceedings/15/Papers/194.pdf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Target Dependent scores:\n",
      "Macro F1 score: 56.15868320061148\n",
      "Accuracy: 73.83928571428572\n"
     ]
    }
   ],
   "source": [
    "from bella.models.target import TargetDep\n",
    "\n",
    "\n",
    "target_dep_model_path = Path(model_zoo_folder, \n",
    "                             f'{TargetDep.name()} {dataset_name}')\n",
    "# Loads the model\n",
    "target_dep = TargetDep.load(target_dep_model_path)\n",
    "\n",
    "# Predicts the class on the test data\n",
    "target_dep_preds = target_dep.predict(test_data.data())\n",
    "# Evaluates the results\n",
    "target_dep_f1 = f1_score(y_test, target_dep_preds, \n",
    "                         average='macro') * 100\n",
    "target_dep_acc = accuracy_score(y_test, \n",
    "                                target_dep_preds) * 100\n",
    "print(f'Target Dependent scores:\\nMacro F1 score: {target_dep_f1}\\n'\n",
    "      f'Accuracy: {target_dep_acc}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Target Dependent Plus scores:\n",
      "Macro F1 score: 57.71114226900386\n",
      "Accuracy: 74.64285714285714\n"
     ]
    }
   ],
   "source": [
    "from bella.models.target import TargetDepPlus\n",
    "\n",
    "target_dep_plus_model_path = Path(model_zoo_folder, \n",
    "                                  f'{TargetDepPlus.name()} {dataset_name}')\n",
    "# Loads the model\n",
    "target_dep_plus = TargetDepPlus.load(target_dep_plus_model_path)\n",
    "# Predicts the class on the test data\n",
    "target_dep_plus_preds = target_dep_plus.predict(test_data.data())\n",
    "# Evaluates the results\n",
    "target_dep_plus_f1 = f1_score(y_test, target_dep_plus_preds, \n",
    "                              average='macro') * 100\n",
    "target_dep_plus_acc = accuracy_score(y_test, \n",
    "                                     target_dep_plus_preds) * 100\n",
    "print(f'Target Dependent Plus scores:\\nMacro F1 score: {target_dep_plus_f1}\\n'\n",
    "      f'Accuracy: {target_dep_plus_acc}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TDParse and TDParse Plus Models\n",
    "\n",
    "These two methods come from [Wang et al.'s paper](https://www.aclweb.org/anthology/E17-1046). Just a note that these models use the Tweebo dependency parser."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TDParse scores:\n",
      "Macro F1 score: 59.63851089722315\n",
      "Accuracy: 73.48214285714286\n"
     ]
    }
   ],
   "source": [
    "from bella.models.tdparse import TDParse\n",
    "\n",
    "tdparse_model_path = Path(model_zoo_folder, \n",
    "                          f'{TDParse.name()} {dataset_name}')\n",
    "# Loads the model\n",
    "tdparse = TDParse.load(tdparse_model_path)\n",
    "# Predicts the class on the test data\n",
    "tdparse_preds = tdparse.predict(test_data.data())\n",
    "# Evaluates the results\n",
    "tdparse_f1 = f1_score(y_test, tdparse_preds, average='macro') * 100\n",
    "tdparse_acc = accuracy_score(y_test, tdparse_preds) * 100\n",
    "print(f'TDParse scores:\\nMacro F1 score: {tdparse_f1}\\n'\n",
    "      f'Accuracy: {tdparse_acc}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TDParse Plus scores:\n",
      "Macro F1 score: 61.107803501564874\n",
      "Accuracy: 76.16071428571428\n"
     ]
    }
   ],
   "source": [
    "from bella.models.tdparse import TDParsePlus\n",
    "\n",
    "\n",
    "tdparse_plus_model_path = Path(model_zoo_folder, \n",
    "                               f'{TDParsePlus.name()} {dataset_name}')\n",
    "# Loads the model\n",
    "tdparse_plus = TDParsePlus.load(tdparse_plus_model_path)\n",
    "# Predicts the class on the test data\n",
    "tdparse_plus_preds = tdparse_plus.predict(test_data.data())\n",
    "# Evaluates the results\n",
    "tdparse_plus_f1 = f1_score(y_test, tdparse_plus_preds, \n",
    "                           average='macro') * 100\n",
    "tdparse_plus_acc = accuracy_score(y_test, \n",
    "                                  tdparse_plus_preds) * 100\n",
    "print(f'TDParse Plus scores:\\nMacro F1 score: {tdparse_plus_f1}\\n'\n",
    "      f'Accuracy: {tdparse_plus_acc}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LSTM, TDLSTM and TCLSTM Models\n",
    "\n",
    "These three methods come from [Tang et al.'s paper](https://aclweb.org/anthology/C16-1311). \n",
    "\n",
    "We also show how you can visualise the model. This requires pydot and graphviz to be installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LSTM scores:\n",
      "Macro F1 score: 46.36196276283449\n",
      "Accuracy: 71.07142857142857\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/andrew/Envs/Bella/lib/python3.6/site-packages/sklearn/metrics/classification.py:1135: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"308pt\" viewBox=\"0.00 0.00 340.00 308.00\" width=\"340pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 304)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"white\" points=\"-4,4 -4,-304 336,-304 336,4 -4,4\" stroke=\"none\"/>\n",
       "<!-- 140600800479496 -->\n",
       "<g class=\"node\" id=\"node1\"><title>140600800479496</title>\n",
       "<polygon fill=\"none\" points=\"31,-253 31,-299 301,-299 301,-253 31,-253\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"100.5\" y=\"-272.3\">text_input: InputLayer</text>\n",
       "<polyline fill=\"none\" points=\"170,-253 170,-299 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"197.5\" y=\"-283.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"170,-276 225,-276 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"197.5\" y=\"-260.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"225,-253 225,-299 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"263\" y=\"-283.8\">(None, 78)</text>\n",
       "<polyline fill=\"none\" points=\"225,-276 301,-276 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"263\" y=\"-260.8\">(None, 78)</text>\n",
       "</g>\n",
       "<!-- 140600800478208 -->\n",
       "<g class=\"node\" id=\"node2\"><title>140600800478208</title>\n",
       "<polygon fill=\"none\" points=\"-1,-169 -1,-215 333,-215 333,-169 -1,-169\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"90\" y=\"-188.3\">embedding_layer: Embedding</text>\n",
       "<polyline fill=\"none\" points=\"181,-169 181,-215 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"208.5\" y=\"-199.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"181,-192 236,-192 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"208.5\" y=\"-176.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"236,-169 236,-215 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"284.5\" y=\"-199.8\">(None, 78)</text>\n",
       "<polyline fill=\"none\" points=\"236,-192 333,-192 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"284.5\" y=\"-176.8\">(None, 78, 50)</text>\n",
       "</g>\n",
       "<!-- 140600800479496&#45;&gt;140600800478208 -->\n",
       "<g class=\"edge\" id=\"edge1\"><title>140600800479496-&gt;140600800478208</title>\n",
       "<path d=\"M166,-252.593C166,-244.118 166,-234.297 166,-225.104\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"169.5,-225.096 166,-215.096 162.5,-225.096 169.5,-225.096\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 140600800478600 -->\n",
       "<g class=\"node\" id=\"node3\"><title>140600800478600</title>\n",
       "<polygon fill=\"none\" points=\"30.5,-85 30.5,-131 301.5,-131 301.5,-85 30.5,-85\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"90\" y=\"-104.3\">lstm_layer: LSTM</text>\n",
       "<polyline fill=\"none\" points=\"149.5,-85 149.5,-131 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"177\" y=\"-115.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"149.5,-108 204.5,-108 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"177\" y=\"-92.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"204.5,-85 204.5,-131 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"253\" y=\"-115.8\">(None, 78, 50)</text>\n",
       "<polyline fill=\"none\" points=\"204.5,-108 301.5,-108 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"253\" y=\"-92.8\">(None, 50)</text>\n",
       "</g>\n",
       "<!-- 140600800478208&#45;&gt;140600800478600 -->\n",
       "<g class=\"edge\" id=\"edge2\"><title>140600800478208-&gt;140600800478600</title>\n",
       "<path d=\"M166,-168.593C166,-160.118 166,-150.297 166,-141.104\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"169.5,-141.096 166,-131.096 162.5,-141.096 169.5,-141.096\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 140600800477984 -->\n",
       "<g class=\"node\" id=\"node4\"><title>140600800477984</title>\n",
       "<polygon fill=\"none\" points=\"54.5,-1 54.5,-47 277.5,-47 277.5,-1 54.5,-1\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"100.5\" y=\"-20.3\">output: Dense</text>\n",
       "<polyline fill=\"none\" points=\"146.5,-1 146.5,-47 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"174\" y=\"-31.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"146.5,-24 201.5,-24 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"174\" y=\"-8.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"201.5,-1 201.5,-47 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"239.5\" y=\"-31.8\">(None, 50)</text>\n",
       "<polyline fill=\"none\" points=\"201.5,-24 277.5,-24 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"239.5\" y=\"-8.8\">(None, 3)</text>\n",
       "</g>\n",
       "<!-- 140600800478600&#45;&gt;140600800477984 -->\n",
       "<g class=\"edge\" id=\"edge3\"><title>140600800478600-&gt;140600800477984</title>\n",
       "<path d=\"M166,-84.5931C166,-76.1177 166,-66.2974 166,-57.104\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"169.5,-57.0958 166,-47.0959 162.5,-57.0959 169.5,-57.0958\" stroke=\"black\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from bella.models.tdlstm import LSTM\n",
    "from IPython.display import SVG\n",
    "from keras.utils.vis_utils import model_to_dot\n",
    "\n",
    "lstm_model_path = Path(model_zoo_folder, \n",
    "                       f'{LSTM.name()} {dataset_name}')\n",
    "lstm = LSTM.load(lstm_model_path)\n",
    "\n",
    "sentiment_mapper = {0: -1, 1: 0, 2: 1}\n",
    "\n",
    "lstm_preds = lstm.predict(test_data.data_dict())\n",
    "lstm_preds = [sentiment_mapper[pred] for pred in lstm_preds]\n",
    "# Evaluates the results\n",
    "lstm_f1 = f1_score(y_test, lstm_preds, average='macro') * 100\n",
    "lstm_acc = accuracy_score(y_test, lstm_preds) * 100\n",
    "print(f'LSTM scores:\\nMacro F1 score: {lstm_f1}\\n'\n",
    "      f'Accuracy: {lstm_acc}')\n",
    "SVG(model_to_dot(lstm.model, show_shapes=True, \n",
    "                 show_layer_names=True).create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TDLSTM scores:\n",
      "Macro F1 score: 57.681236460102625\n",
      "Accuracy: 74.375\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"392pt\" viewBox=\"0.00 0.00 763.00 392.00\" width=\"763pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 388)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"white\" points=\"-4,4 -4,-388 759,-388 759,4 -4,4\" stroke=\"none\"/>\n",
       "<!-- 140600850973920 -->\n",
       "<g class=\"node\" id=\"node1\"><title>140600850973920</title>\n",
       "<polygon fill=\"none\" points=\"34.5,-337 34.5,-383 329.5,-383 329.5,-337 34.5,-337\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"116.5\" y=\"-356.3\">left_text_input: InputLayer</text>\n",
       "<polyline fill=\"none\" points=\"198.5,-337 198.5,-383 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"226\" y=\"-367.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"198.5,-360 253.5,-360 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"226\" y=\"-344.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"253.5,-337 253.5,-383 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"291.5\" y=\"-367.8\">(None, 66)</text>\n",
       "<polyline fill=\"none\" points=\"253.5,-360 329.5,-360 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"291.5\" y=\"-344.8\">(None, 66)</text>\n",
       "</g>\n",
       "<!-- 140600850973528 -->\n",
       "<g class=\"node\" id=\"node3\"><title>140600850973528</title>\n",
       "<polygon fill=\"none\" points=\"-0.5,-253 -0.5,-299 364.5,-299 364.5,-253 -0.5,-253\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"102.5\" y=\"-272.3\">left_embedding_layer: Embedding</text>\n",
       "<polyline fill=\"none\" points=\"205.5,-253 205.5,-299 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"233\" y=\"-283.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"205.5,-276 260.5,-276 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"233\" y=\"-260.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"260.5,-253 260.5,-299 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"312.5\" y=\"-283.8\">(None, 66)</text>\n",
       "<polyline fill=\"none\" points=\"260.5,-276 364.5,-276 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"312.5\" y=\"-260.8\">(None, 66, 300)</text>\n",
       "</g>\n",
       "<!-- 140600850973920&#45;&gt;140600850973528 -->\n",
       "<g class=\"edge\" id=\"edge1\"><title>140600850973920-&gt;140600850973528</title>\n",
       "<path d=\"M182,-336.593C182,-328.118 182,-318.297 182,-309.104\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"185.5,-309.096 182,-299.096 178.5,-309.096 185.5,-309.096\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 140600850973752 -->\n",
       "<g class=\"node\" id=\"node2\"><title>140600850973752</title>\n",
       "<polygon fill=\"none\" points=\"418,-337 418,-383 720,-383 720,-337 418,-337\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"503.5\" y=\"-356.3\">right_text_input: InputLayer</text>\n",
       "<polyline fill=\"none\" points=\"589,-337 589,-383 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"616.5\" y=\"-367.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"589,-360 644,-360 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"616.5\" y=\"-344.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"644,-337 644,-383 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"682\" y=\"-367.8\">(None, 71)</text>\n",
       "<polyline fill=\"none\" points=\"644,-360 720,-360 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"682\" y=\"-344.8\">(None, 71)</text>\n",
       "</g>\n",
       "<!-- 140600850973584 -->\n",
       "<g class=\"node\" id=\"node4\"><title>140600850973584</title>\n",
       "<polygon fill=\"none\" points=\"382.5,-253 382.5,-299 755.5,-299 755.5,-253 382.5,-253\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"489.5\" y=\"-272.3\">right_embedding_layer: Embedding</text>\n",
       "<polyline fill=\"none\" points=\"596.5,-253 596.5,-299 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"624\" y=\"-283.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"596.5,-276 651.5,-276 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"624\" y=\"-260.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"651.5,-253 651.5,-299 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"703.5\" y=\"-283.8\">(None, 71)</text>\n",
       "<polyline fill=\"none\" points=\"651.5,-276 755.5,-276 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"703.5\" y=\"-260.8\">(None, 71, 300)</text>\n",
       "</g>\n",
       "<!-- 140600850973752&#45;&gt;140600850973584 -->\n",
       "<g class=\"edge\" id=\"edge2\"><title>140600850973752-&gt;140600850973584</title>\n",
       "<path d=\"M569,-336.593C569,-328.118 569,-318.297 569,-309.104\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"572.5,-309.096 569,-299.096 565.5,-309.096 572.5,-309.096\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 140600850973416 -->\n",
       "<g class=\"node\" id=\"node5\"><title>140600850973416</title>\n",
       "<polygon fill=\"none\" points=\"62,-169 62,-215 364,-215 364,-169 62,-169\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"133.5\" y=\"-188.3\">left_lstm_layer: LSTM</text>\n",
       "<polyline fill=\"none\" points=\"205,-169 205,-215 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"232.5\" y=\"-199.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"205,-192 260,-192 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"232.5\" y=\"-176.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"260,-169 260,-215 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"312\" y=\"-199.8\">(None, 66, 300)</text>\n",
       "<polyline fill=\"none\" points=\"260,-192 364,-192 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"312\" y=\"-176.8\">(None, 300)</text>\n",
       "</g>\n",
       "<!-- 140600850973528&#45;&gt;140600850973416 -->\n",
       "<g class=\"edge\" id=\"edge3\"><title>140600850973528-&gt;140600850973416</title>\n",
       "<path d=\"M190.471,-252.593C193.744,-243.935 197.548,-233.874 201.087,-224.512\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"204.384,-225.687 204.647,-215.096 197.837,-223.212 204.384,-225.687\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 140600850972800 -->\n",
       "<g class=\"node\" id=\"node6\"><title>140600850972800</title>\n",
       "<polygon fill=\"none\" points=\"398,-169 398,-215 708,-215 708,-169 398,-169\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"473.5\" y=\"-188.3\">right_lstm_layer: LSTM</text>\n",
       "<polyline fill=\"none\" points=\"549,-169 549,-215 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"576.5\" y=\"-199.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"549,-192 604,-192 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"576.5\" y=\"-176.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"604,-169 604,-215 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"656\" y=\"-199.8\">(None, 71, 300)</text>\n",
       "<polyline fill=\"none\" points=\"604,-192 708,-192 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"656\" y=\"-176.8\">(None, 300)</text>\n",
       "</g>\n",
       "<!-- 140600850973584&#45;&gt;140600850972800 -->\n",
       "<g class=\"edge\" id=\"edge4\"><title>140600850973584-&gt;140600850972800</title>\n",
       "<path d=\"M564.628,-252.593C562.974,-244.118 561.058,-234.297 559.264,-225.104\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"562.662,-224.24 557.311,-215.096 555.791,-225.581 562.662,-224.24\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 140600842951648 -->\n",
       "<g class=\"node\" id=\"node7\"><title>140600842951648</title>\n",
       "<polygon fill=\"none\" points=\"166.5,-85 166.5,-131 599.5,-131 599.5,-85 166.5,-85\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"273\" y=\"-104.3\">left_right_lstm_merge: Concatenate</text>\n",
       "<polyline fill=\"none\" points=\"379.5,-85 379.5,-131 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"407\" y=\"-115.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"379.5,-108 434.5,-108 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"407\" y=\"-92.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"434.5,-85 434.5,-131 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"517\" y=\"-115.8\">[(None, 300), (None, 300)]</text>\n",
       "<polyline fill=\"none\" points=\"434.5,-108 599.5,-108 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"517\" y=\"-92.8\">(None, 600)</text>\n",
       "</g>\n",
       "<!-- 140600850973416&#45;&gt;140600842951648 -->\n",
       "<g class=\"edge\" id=\"edge5\"><title>140600850973416-&gt;140600842951648</title>\n",
       "<path d=\"M259.004,-168.81C280.351,-158.513 305.888,-146.195 328.223,-135.422\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"329.767,-138.563 337.254,-131.066 326.726,-132.258 329.767,-138.563\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 140600850972800&#45;&gt;140600842951648 -->\n",
       "<g class=\"edge\" id=\"edge6\"><title>140600850972800-&gt;140600842951648</title>\n",
       "<path d=\"M506.996,-168.81C485.649,-158.513 460.112,-146.195 437.777,-135.422\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"439.274,-132.258 428.746,-131.066 436.233,-138.563 439.274,-132.258\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 140600842951872 -->\n",
       "<g class=\"node\" id=\"node8\"><title>140600842951872</title>\n",
       "<polygon fill=\"none\" points=\"268,-1 268,-47 498,-47 498,-1 268,-1\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"314\" y=\"-20.3\">output: Dense</text>\n",
       "<polyline fill=\"none\" points=\"360,-1 360,-47 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"387.5\" y=\"-31.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"360,-24 415,-24 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"387.5\" y=\"-8.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"415,-1 415,-47 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"456.5\" y=\"-31.8\">(None, 600)</text>\n",
       "<polyline fill=\"none\" points=\"415,-24 498,-24 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"456.5\" y=\"-8.8\">(None, 3)</text>\n",
       "</g>\n",
       "<!-- 140600842951648&#45;&gt;140600842951872 -->\n",
       "<g class=\"edge\" id=\"edge7\"><title>140600842951648-&gt;140600842951872</title>\n",
       "<path d=\"M383,-84.5931C383,-76.1177 383,-66.2974 383,-57.104\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"386.5,-57.0958 383,-47.0959 379.5,-57.0959 386.5,-57.0958\" stroke=\"black\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from bella.models.tdlstm import TDLSTM\n",
    "from IPython.display import SVG\n",
    "from keras.utils.vis_utils import model_to_dot\n",
    "\n",
    "tdlstm_model_path = Path(model_zoo_folder, \n",
    "                         f'{TDLSTM.name()} {dataset_name}')\n",
    "tdlstm = TDLSTM.load(tdlstm_model_path)\n",
    "\n",
    "sentiment_mapper = {0: -1, 1: 0, 2: 1}\n",
    "\n",
    "tdlstm_preds = tdlstm.predict(test_data.data_dict())\n",
    "tdlstm_preds = [sentiment_mapper[pred] for pred in tdlstm_preds]\n",
    "# Evaluates the results\n",
    "tdlstm_f1 = f1_score(y_test, tdlstm_preds, average='macro') * 100\n",
    "tdlstm_acc = accuracy_score(y_test, tdlstm_preds) * 100\n",
    "print(f'TDLSTM scores:\\nMacro F1 score: {tdlstm_f1}\\n'\n",
    "      f'Accuracy: {tdlstm_acc}')\n",
    "\n",
    "SVG(model_to_dot(tdlstm.model, show_shapes=True, \n",
    "                 show_layer_names=True).create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TCLSTM scores:\n",
      "Macro F1 score: 55.375232710170344\n",
      "Accuracy: 72.41071428571428\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"476pt\" viewBox=\"0.00 0.00 1431.00 476.00\" width=\"1431pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 472)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"white\" points=\"-4,4 -4,-472 1427,-472 1427,4 -4,4\" stroke=\"none\"/>\n",
       "<!-- 140601819261416 -->\n",
       "<g class=\"node\" id=\"node1\"><title>140601819261416</title>\n",
       "<polygon fill=\"none\" points=\"34.5,-421 34.5,-467 329.5,-467 329.5,-421 34.5,-421\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"116.5\" y=\"-440.3\">left_text_input: InputLayer</text>\n",
       "<polyline fill=\"none\" points=\"198.5,-421 198.5,-467 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"226\" y=\"-451.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"198.5,-444 253.5,-444 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"226\" y=\"-428.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"253.5,-421 253.5,-467 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"291.5\" y=\"-451.8\">(None, 66)</text>\n",
       "<polyline fill=\"none\" points=\"253.5,-444 329.5,-444 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"291.5\" y=\"-428.8\">(None, 66)</text>\n",
       "</g>\n",
       "<!-- 140601819261136 -->\n",
       "<g class=\"node\" id=\"node3\"><title>140601819261136</title>\n",
       "<polygon fill=\"none\" points=\"-0.5,-337 -0.5,-383 364.5,-383 364.5,-337 -0.5,-337\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"102.5\" y=\"-356.3\">left_embedding_layer: Embedding</text>\n",
       "<polyline fill=\"none\" points=\"205.5,-337 205.5,-383 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"233\" y=\"-367.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"205.5,-360 260.5,-360 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"233\" y=\"-344.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"260.5,-337 260.5,-383 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"312.5\" y=\"-367.8\">(None, 66)</text>\n",
       "<polyline fill=\"none\" points=\"260.5,-360 364.5,-360 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"312.5\" y=\"-344.8\">(None, 66, 300)</text>\n",
       "</g>\n",
       "<!-- 140601819261416&#45;&gt;140601819261136 -->\n",
       "<g class=\"edge\" id=\"edge1\"><title>140601819261416-&gt;140601819261136</title>\n",
       "<path d=\"M182,-420.593C182,-412.118 182,-402.297 182,-393.104\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"185.5,-393.096 182,-383.096 178.5,-393.096 185.5,-393.096\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 140601819261304 -->\n",
       "<g class=\"node\" id=\"node2\"><title>140601819261304</title>\n",
       "<polygon fill=\"none\" points=\"761,-421 761,-467 1063,-467 1063,-421 761,-421\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"846.5\" y=\"-440.3\">right_text_input: InputLayer</text>\n",
       "<polyline fill=\"none\" points=\"932,-421 932,-467 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"959.5\" y=\"-451.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"932,-444 987,-444 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"959.5\" y=\"-428.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"987,-421 987,-467 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1025\" y=\"-451.8\">(None, 71)</text>\n",
       "<polyline fill=\"none\" points=\"987,-444 1063,-444 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1025\" y=\"-428.8\">(None, 71)</text>\n",
       "</g>\n",
       "<!-- 140601819261024 -->\n",
       "<g class=\"node\" id=\"node5\"><title>140601819261024</title>\n",
       "<polygon fill=\"none\" points=\"725.5,-337 725.5,-383 1098.5,-383 1098.5,-337 725.5,-337\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"832.5\" y=\"-356.3\">right_embedding_layer: Embedding</text>\n",
       "<polyline fill=\"none\" points=\"939.5,-337 939.5,-383 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"967\" y=\"-367.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"939.5,-360 994.5,-360 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"967\" y=\"-344.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"994.5,-337 994.5,-383 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1046.5\" y=\"-367.8\">(None, 71)</text>\n",
       "<polyline fill=\"none\" points=\"994.5,-360 1098.5,-360 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1046.5\" y=\"-344.8\">(None, 71, 300)</text>\n",
       "</g>\n",
       "<!-- 140601819261304&#45;&gt;140601819261024 -->\n",
       "<g class=\"edge\" id=\"edge2\"><title>140601819261304-&gt;140601819261024</title>\n",
       "<path d=\"M912,-420.593C912,-412.118 912,-402.297 912,-393.104\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"915.5,-393.096 912,-383.096 908.5,-393.096 915.5,-393.096\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 140601819260968 -->\n",
       "<g class=\"node\" id=\"node7\"><title>140601819260968</title>\n",
       "<polygon fill=\"none\" points=\"288.5,-253 288.5,-299 723.5,-299 723.5,-253 288.5,-253\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"375\" y=\"-272.3\">left_text_target: Concatenate</text>\n",
       "<polyline fill=\"none\" points=\"461.5,-253 461.5,-299 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"489\" y=\"-283.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"461.5,-276 516.5,-276 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"489\" y=\"-260.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"516.5,-253 516.5,-299 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"620\" y=\"-283.8\">[(None, 66, 300), (None, 66, 300)]</text>\n",
       "<polyline fill=\"none\" points=\"516.5,-276 723.5,-276 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"620\" y=\"-260.8\">(None, 66, 600)</text>\n",
       "</g>\n",
       "<!-- 140601819261136&#45;&gt;140601819260968 -->\n",
       "<g class=\"edge\" id=\"edge3\"><title>140601819261136-&gt;140601819260968</title>\n",
       "<path d=\"M269.252,-336.918C312.379,-326.003 364.559,-312.797 408.944,-301.564\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"410.037,-304.897 418.873,-299.051 408.32,-298.111 410.037,-304.897\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 140601819261192 -->\n",
       "<g class=\"node\" id=\"node4\"><title>140601819261192</title>\n",
       "<polygon fill=\"none\" points=\"382.5,-337 382.5,-383 681.5,-383 681.5,-337 382.5,-337\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"452.5\" y=\"-356.3\">left_target: InputLayer</text>\n",
       "<polyline fill=\"none\" points=\"522.5,-337 522.5,-383 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"550\" y=\"-367.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"522.5,-360 577.5,-360 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"550\" y=\"-344.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"577.5,-337 577.5,-383 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"629.5\" y=\"-367.8\">(None, 66, 300)</text>\n",
       "<polyline fill=\"none\" points=\"577.5,-360 681.5,-360 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"629.5\" y=\"-344.8\">(None, 66, 300)</text>\n",
       "</g>\n",
       "<!-- 140601819261192&#45;&gt;140601819260968 -->\n",
       "<g class=\"edge\" id=\"edge4\"><title>140601819261192-&gt;140601819260968</title>\n",
       "<path d=\"M524.895,-336.593C522.179,-328.027 519.027,-318.086 516.085,-308.808\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"519.365,-307.57 513.006,-299.096 512.692,-309.686 519.365,-307.57\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 140601819259400 -->\n",
       "<g class=\"node\" id=\"node8\"><title>140601819259400</title>\n",
       "<polygon fill=\"none\" points=\"741.5,-253 741.5,-299 1184.5,-299 1184.5,-253 741.5,-253\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"832\" y=\"-272.3\">right_text_target: Concatenate</text>\n",
       "<polyline fill=\"none\" points=\"922.5,-253 922.5,-299 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"950\" y=\"-283.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"922.5,-276 977.5,-276 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"950\" y=\"-260.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"977.5,-253 977.5,-299 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1081\" y=\"-283.8\">[(None, 71, 300), (None, 71, 300)]</text>\n",
       "<polyline fill=\"none\" points=\"977.5,-276 1184.5,-276 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1081\" y=\"-260.8\">(None, 71, 600)</text>\n",
       "</g>\n",
       "<!-- 140601819261024&#45;&gt;140601819259400 -->\n",
       "<g class=\"edge\" id=\"edge5\"><title>140601819261024-&gt;140601819259400</title>\n",
       "<path d=\"M925.936,-336.593C931.491,-327.662 937.974,-317.238 943.952,-307.627\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"946.948,-309.436 949.257,-299.096 941.004,-305.739 946.948,-309.436\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 140601819259848 -->\n",
       "<g class=\"node\" id=\"node6\"><title>140601819259848</title>\n",
       "<polygon fill=\"none\" points=\"1117,-337 1117,-383 1423,-383 1423,-337 1117,-337\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1190.5\" y=\"-356.3\">right_target: InputLayer</text>\n",
       "<polyline fill=\"none\" points=\"1264,-337 1264,-383 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1291.5\" y=\"-367.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"1264,-360 1319,-360 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1291.5\" y=\"-344.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"1319,-337 1319,-383 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1371\" y=\"-367.8\">(None, 71, 300)</text>\n",
       "<polyline fill=\"none\" points=\"1319,-360 1423,-360 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1371\" y=\"-344.8\">(None, 71, 300)</text>\n",
       "</g>\n",
       "<!-- 140601819259848&#45;&gt;140601819259400 -->\n",
       "<g class=\"edge\" id=\"edge6\"><title>140601819259848-&gt;140601819259400</title>\n",
       "<path d=\"M1187.33,-336.918C1146.55,-326.026 1097.23,-312.852 1055.23,-301.634\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"1056.12,-298.25 1045.56,-299.051 1054.31,-305.013 1056.12,-298.25\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 140601819260912 -->\n",
       "<g class=\"node\" id=\"node9\"><title>140601819260912</title>\n",
       "<polygon fill=\"none\" points=\"421,-169 421,-215 723,-215 723,-169 421,-169\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"492.5\" y=\"-188.3\">left_lstm_layer: LSTM</text>\n",
       "<polyline fill=\"none\" points=\"564,-169 564,-215 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"591.5\" y=\"-199.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"564,-192 619,-192 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"591.5\" y=\"-176.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"619,-169 619,-215 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"671\" y=\"-199.8\">(None, 66, 600)</text>\n",
       "<polyline fill=\"none\" points=\"619,-192 723,-192 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"671\" y=\"-176.8\">(None, 600)</text>\n",
       "</g>\n",
       "<!-- 140601819260968&#45;&gt;140601819260912 -->\n",
       "<g class=\"edge\" id=\"edge7\"><title>140601819260968-&gt;140601819260912</title>\n",
       "<path d=\"M524.035,-252.593C531.37,-243.48 539.957,-232.811 547.822,-223.04\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"550.672,-225.081 554.216,-215.096 545.219,-220.691 550.672,-225.081\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 140601819260856 -->\n",
       "<g class=\"node\" id=\"node10\"><title>140601819260856</title>\n",
       "<polygon fill=\"none\" points=\"774,-169 774,-215 1084,-215 1084,-169 774,-169\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"849.5\" y=\"-188.3\">right_lstm_layer: LSTM</text>\n",
       "<polyline fill=\"none\" points=\"925,-169 925,-215 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"952.5\" y=\"-199.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"925,-192 980,-192 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"952.5\" y=\"-176.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"980,-169 980,-215 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1032\" y=\"-199.8\">(None, 71, 600)</text>\n",
       "<polyline fill=\"none\" points=\"980,-192 1084,-192 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"1032\" y=\"-176.8\">(None, 600)</text>\n",
       "</g>\n",
       "<!-- 140601819259400&#45;&gt;140601819260856 -->\n",
       "<g class=\"edge\" id=\"edge8\"><title>140601819259400-&gt;140601819260856</title>\n",
       "<path d=\"M953.709,-252.593C950.12,-243.935 945.948,-233.874 942.066,-224.512\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"945.225,-222.993 938.162,-215.096 938.759,-225.674 945.225,-222.993\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 140601819259960 -->\n",
       "<g class=\"node\" id=\"node11\"><title>140601819259960</title>\n",
       "<polygon fill=\"none\" points=\"533.5,-85 533.5,-131 966.5,-131 966.5,-85 533.5,-85\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"640\" y=\"-104.3\">left_right_lstm_merge: Concatenate</text>\n",
       "<polyline fill=\"none\" points=\"746.5,-85 746.5,-131 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"774\" y=\"-115.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"746.5,-108 801.5,-108 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"774\" y=\"-92.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"801.5,-85 801.5,-131 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"884\" y=\"-115.8\">[(None, 600), (None, 600)]</text>\n",
       "<polyline fill=\"none\" points=\"801.5,-108 966.5,-108 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"884\" y=\"-92.8\">(None, 1200)</text>\n",
       "</g>\n",
       "<!-- 140601819260912&#45;&gt;140601819259960 -->\n",
       "<g class=\"edge\" id=\"edge9\"><title>140601819260912-&gt;140601819259960</title>\n",
       "<path d=\"M620.169,-168.81C642.62,-158.467 669.497,-146.086 692.957,-135.278\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"694.483,-138.429 702.101,-131.066 691.554,-132.071 694.483,-138.429\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 140601819260856&#45;&gt;140601819259960 -->\n",
       "<g class=\"edge\" id=\"edge10\"><title>140601819260856-&gt;140601819259960</title>\n",
       "<path d=\"M880.56,-168.81C857.983,-158.467 830.955,-146.086 807.363,-135.278\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"808.717,-132.049 798.168,-131.066 805.802,-138.413 808.717,-132.049\" stroke=\"black\"/>\n",
       "</g>\n",
       "<!-- 140601819259064 -->\n",
       "<g class=\"node\" id=\"node12\"><title>140601819259064</title>\n",
       "<polygon fill=\"none\" points=\"632,-1 632,-47 868,-47 868,-1 632,-1\" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"678\" y=\"-20.3\">output: Dense</text>\n",
       "<polyline fill=\"none\" points=\"724,-1 724,-47 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"751.5\" y=\"-31.8\">input:</text>\n",
       "<polyline fill=\"none\" points=\"724,-24 779,-24 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"751.5\" y=\"-8.8\">output:</text>\n",
       "<polyline fill=\"none\" points=\"779,-1 779,-47 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"823.5\" y=\"-31.8\">(None, 1200)</text>\n",
       "<polyline fill=\"none\" points=\"779,-24 868,-24 \" stroke=\"black\"/>\n",
       "<text font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"823.5\" y=\"-8.8\">(None, 3)</text>\n",
       "</g>\n",
       "<!-- 140601819259960&#45;&gt;140601819259064 -->\n",
       "<g class=\"edge\" id=\"edge11\"><title>140601819259960-&gt;140601819259064</title>\n",
       "<path d=\"M750,-84.5931C750,-76.1177 750,-66.2974 750,-57.104\" fill=\"none\" stroke=\"black\"/>\n",
       "<polygon fill=\"black\" points=\"753.5,-57.0958 750,-47.0959 746.5,-57.0959 753.5,-57.0958\" stroke=\"black\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from bella.models.tdlstm import TCLSTM\n",
    "from IPython.display import SVG\n",
    "from keras.utils.vis_utils import model_to_dot\n",
    "tclstm_model_path = Path(model_zoo_folder, \n",
    "                         f'{TCLSTM.name()} {dataset_name}')\n",
    "tclstm = TCLSTM.load(tclstm_model_path)\n",
    "\n",
    "sentiment_mapper = {0: -1, 1: 0, 2: 1}\n",
    "\n",
    "tclstm_preds = tclstm.predict(test_data.data_dict())\n",
    "tclstm_preds = [sentiment_mapper[pred] for pred in tclstm_preds]\n",
    "# Evaluates the results\n",
    "tclstm_f1 = f1_score(y_test, tclstm_preds, average='macro') * 100\n",
    "tclstm_acc = accuracy_score(y_test, tclstm_preds) * 100\n",
    "print(f'TCLSTM scores:\\nMacro F1 score: {tclstm_f1}\\n'\n",
    "      f'Accuracy: {tclstm_acc}')\n",
    "SVG(model_to_dot(tclstm.model, show_shapes=True, \n",
    "                 show_layer_names=True).create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}