HMM_UD.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring the UD tagset with Hidden Markov Model\n",
    "---\n",
    "### Introduction\n",
    "\n",
    "In this notebook, we will use the [Pomegranate](http://pomegranate.readthedocs.io/en/latest/index.html) library to build a simple Hidden Markov Model for part-of-speech tagging.\n",
    " \n",
    "The goal here is breadth rather than depth: we want to cover as many languages in the UD tagset as possible, therefore we did not implement additional features like:\n",
    "- Laplace Smoothing [Wiki Link](https://en.wikipedia.org/wiki/Additive_smoothing) \n",
    "    \n",
    "- Backoff Smoothing [Speech & Language Processing Ch. 4,9,10](https://web.stanford.edu/~jurafsky/slp3/) \n",
    "\n",
    "- Extending to Trigrams [Trigram Paper](http://www.coli.uni-saarland.de/~thorsten/publications/Brants-ANLP00.pdf)\n",
    "\n",
    "\n",
    "આ પોથીમાં આપણે વિવિધ ભાષાઓમાં શબ્દ ભેદ (પાર્ટ્સ ઓફ સ્પીચ) ઉકેલવાનું કામ હિડેન માર્કોવ મોડેલ (HMM) વડે કરીશું. \n",
    "\n",
    "અહીં દાડમ લાઇબ્રેરી [Pomegranate](http://pomegranate.readthedocs.io/en/latest/index.html) વાપરવામાં આવી છે.\n",
    "\n",
    "આપણું લક્ષ્ય ઊંડાણ ને બદલે વિસ્તારનો છે, એટલા માટે નીચે આપેલ ગુણવિશેષનો સમાવેશ નથી: \n",
    "- લાપ્લેસ નિયમિતકારણ [વિકિપીડિયા](https://en.wikipedia.org/wiki/Additive_smoothing) \n",
    "    \n",
    "- બેકઓફ નિયમિતકરણ [ચોંપડી પાઠ 4,9,10](https://web.stanford.edu/~jurafsky/slp3/) \n",
    "\n",
    "- ટ્રાઇગ્રામ [સંશોધન પેપર](http://www.coli.uni-saarland.de/~thorsten/publications/Brants-ANLP00.pdf)\n",
    "\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing and imports\n",
    "We scan the entire UD folder to read in all the names of the respective language subdirectories, and prune out datasets that don't have train sets. Lack of a dev set is tolerated, as dev sets are fused to the training set, given the lack of iterative training in our HMM implementation. \n",
    "\n",
    "We need the following libraries installed:\n",
    "1. Pomegranate\n",
    "2. Numpy\n",
    "3. Collections\n",
    "4. pyconll\n",
    "\n",
    "In addition, helper functions are found in data_prep.py and hmm_utils.py. Make sure you have these files in the same directory as this notebook!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Jupyter \"magic methods\" -- only need to be run once per kernel restart\n",
    "%load_ext autoreload\n",
    "%autoreload 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "from pomegranate import HiddenMarkovModel, DiscreteDistribution, State\n",
    "import numpy as np\n",
    "from data_prep import *\n",
    "from hmm_utils import *\n",
    "import os\n",
    "import fnmatch\n",
    "import re\n",
    "import pandas as pd\n",
    "UD_PATH = 'C:/Users/DELL/Downloads/ud-treebanks-v2.6/ud-treebanks-v2.6/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#store all languages with a train conllu file as tuples language_dirs. \n",
    "#First element of the tuple is the directory name, second element is the prefix before each conllu file\n",
    "language_dirs = []\n",
    "subdirs = os.listdir(UD_PATH)\n",
    "for subdir in subdirs:\n",
    "    for file in os.listdir(UD_PATH+subdir):\n",
    "        if fnmatch.fnmatch(file, '*train.conllu'):\n",
    "            file_prefix = re.compile('^.*(?=(train\\.conllu))').search(file).group()\n",
    "            language_dirs.append((subdir+'/', file_prefix))\n",
    "            break"
   ]
  },
  {
   "attachments": {
    "pos_hmm.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWYAAACNCAMAAACzDCDRAAAAyVBMVEX///+/v78AAAD8/Pz5+fn39/fCwsLz8/PFxcXu7u6wsLCNjY3s7OzHx8fd3d3o6OjT09Pa2tp4eHioqKjNzc24uLjj4+Ofn5+1tbWVlZU+Pj6Hh4eOjo6UlJTQ0ND/AABsbGxaWlp9fX1gYGAcHBxHR0dRUVEvLy9cXFwqKio4ODgWFhZpaWkNDQ0jIyMaGhr/8/P/4uL/0dH/f3//qan/lJT/Nzf/oKD/QED/Vlb/aWn/Kyv/jIz/tbX/3d3/yMj/eXn/Gxv/U1PQ5kBaAAAgAElEQVR4nO19CZujyLEtBDsSIPZNUiMQIKGlPLav77Pv9fj5/f8f9SJAVYUQi9TVqhpPT3wz1SoEUZknFpIk8iTD/CF/yB/yh/xGhOeuf+UnTn9iU37HsozmV78nyejpSrR6YmN+r8LlS/qHN96OWFZ93Bi4gGHi6tmN+v1Juqn/WRVvRxqY3fPwNbv05hDnuiz+13u6Gen0jxE5U40Rcu39F3VDycyrFkZlaNGw1X+7opgcw+sCfjLAowMCxLp5+bKGmcszzxy63ICbTnt5Duf8Fn4SuZAcMBkz96batYDW3zxG9HOdKypI7oZxQZm6/DcmdvKynL/AGXEOSzqgxxAv/cu3BLMRgrUczsEn//aYCTLDq4rNcCqrB4y8MG3M+Zq9YNaikoBv6C7+ITvgGc9V8CiKF0j2GpujqArj2BgKNsxt9HzDXmhrZhs7KscsWMYqGc3xMlAXNs9o/RHzWxSPSZOE1yFgmE2dIRgR3kcbtTd7o75jFbfHbHRxFgCkl20FOXM4lBAxFmRbZp+JsM0U/LSCHR6M4VyChJeEUEYwN2AL3hG/SfD0sgR+AfkJfA62BaRMnjN5wkAcQJmoGHp41n+ORCXCCjbDlI3LXnBzZeYCsw/1Abnfd0K4PZa8cMwSbD5Gi0HIQUFeeDjiN+Aze9QJsQcqKpY2ILB12jmCMYc1C7GkYrawDsz2wMigRxtGh4UDIRNFzGmG7eTRI2CJyUq+Siu/deEB4VNAxA8NzFVY/7OiO1ANc9Z4+eoKZi+5fPB7YI7QUkWOCcVi5qAhTkyeMSqArsNaQYtKwIbowyEI+DcbOx1SzMb6DD/vDgj6ngzvgg4uXsfZaK7tUQHNI8gNj5oMapY/BZDniAP4jBHQj8abJagHAV49gK5hRr+jA+3Mwaf5K7rLW5gJPP4U45Uz9FJ0T0XAwEfNPgsC+aAD8xh44eU4R8CKor7EZxJgooxh8j2mKV8HDZOEBw53OBDqJrAaGCqgWTmW7tX7HNZPhOVHi4/Ow5xn+GNDUY39mTOxdPExgplw93XsnvB+keRJr+gmmxuVC4yN2uMK2O+BSQ4Y+GsLE7RinVH/QQ6BX8MeDhKLaR/QHowGUAHLYTbAxF5hlrEBSjA5OB0OGXPAkzeYx5nkRFEiwsHB3H18MjI/VAoI1tWOPvl7+mlAHmGitutONN5cFohEgE6nLEyUBTn6/BVmuB1pzF2BkVi0Cs/qnsOsF4zicgLL8oyDowqHZUwMGI/FrOSJDMeSOh/mrIHn02eTxbP0hcHirXHOSo7HiJKGJy8cul7DhOyIaN622X/zwoHrL5sHgXlzS5FXFIyZTZ9rmNehSB8RTnO2RJk5zDvM5Pw/QDY945VR0TBr/weJ3nq8iFv3FKif1pqnwFr2i6vrlAvM+eyHNCNVH71g+UP+7meJ2r6DZdbrkBkHCDTgeIdZAOfqEWXRXJc86oU/qRzBbv1mV6++Hc5okrM1Q+fH0vt5nBqnocYoZfviP6RX/vztT1/dhJ9Afvn231/dhJ9A/vTtL1/dhJ9B/s/fv7oFP4P843+46ZP+kA/K//5x+/sE+ee3X7+6Cf+BwnEczwsoPI8fp8//17d/9SlpdNRKvrshH9fBoI5HOvNZUndNkhRlrqBIAj/Vtp6h3K2OxzuIGDc6GrmjIX1K6oZ8TMcThPxHWqhxlkf5xvJZ3TDmE43707f/7SrBvunu7FhEUXFcBQvPmCuPdpB80BDDhHRksbr2PANRelyHovnWDjuTpbZW6/gtAI0N81Z7gCL1fT/OXuB0ZE3PGO3f3zpDOdSh2DlAbq18f5aUABvb1I2HgCZjsxuAKln6/sqKACJ/rXvzR0AiY6+tE2yPse+HaQFQxs6DOp4iHM95Fux9/e3IXC0gV019pG1//1tXibQCmL1PwEliAmd/QUDf2z9MF8EWLO1tCphzYoDYMb1HdHBaDjv3/S2NGW4hEc2vBho750PUfVXjHSGTh/v3l+uhHLoQC4fuNKQ0gyhYjBmro8OMIOxMs3M2HPy1adyJEccbCVjdog+5gpljjkfnk4XjlQ29+L+RxfakIka9Tfvrt1+udfAx9E0ZGwWsnPswQjdU4dhTZMCn6OALT7kHZ45bwF7v+cKGQlw8EBQ/Wjjeg0jq/84ijObSbdP+9e3P1zqkYug1vA+WvPCkyaLP2lIDs/IO5Oz6How4tFTc/5VRvqjOgNM8XwjlbPDbEFaaedO0X3/99terA7yQbwdrXTRI5IUxiTOHKC+GvvROuetM+zPHBTA4dc3lJ8T5a/wZM8ZpGGXyRf8GZ+7bt/+6Oonnj9uRF5cOWCziPN49jvPHylIUKBDnidzDcc5QPNRSnNU7bPUM4fmsGv2zFqiarly54q/fvv3f9u8cF94WGrZFhRV7GxNXwvEajJaymWC563FbUWCOvlIUto2txs55inC8OlUVWdZNa3fvv/7916tRBm+OI8QwycGWO7bqCK8cJl66qhCypjFmK47bTRQQ6ZC6i1EdzxF+PpzLLqJDzC6uXPGX6xM4Pk+m/swpCzq26ujg4sOEDmZTjduK49nxoEJZgS2ao/Z+hnD8cqS0+yLJWdVGIo3jxekyYRV8TBsjOgwQp3SgKwZjaYOTqoFBxrvwp6M6au+nCD+fCneG6ofqSBv6muOiyd4xzDkb6x6OMqJpHcfSZofdmZx52k1tsvcnuzNl5jtO20W2NggRh5m573GgIzhicYe7x89Pd9SxLGCmrgczK8dtkmkdPFiq88nZmeeLOxyRYSF0ByOe41d3OCKFxHD3KO/coYPZZ5idh+yNkTk47G5JUvkjIfEUwYzYWSLTu4BPajygXwcn5OEdOpiq8MUhiDBnFN0jfeel+9BdDNib491OzuB6B8guhMGQjicJjlWvb1469HpmXtjyTdYQmsJEdKJrU3EH6Ht0j/d9WaO57/FC1DHVrtczWYh77O3UB25N5fcUmdZhZTteV8fk/fcDwgl+Z5xh9sOclnTf6PpGUq8W49edylVh2zvwCGCJibULs32iOSs0lXZ9POut7fbA6rlLmOBL1JmiM/Duh5k/Z754o8MdfXr8mHBCxwFMACh7XDF8weR8c1wBKNYMx16nVX6PSnpc0UFPXM9vwngPVcBwOlwvPDuijp7xvASJr3k3AW/ByecYpbyeZfRRh9U9FSXaIMw3GbCC/bOA5qXkuh1zC84+9qJbaqyessyKZx1ZVdiRXLOvI4JbnSA2mHlXhw7HTZJ2law2qGOrOp3niqACa81INw8bsCmONw1ZWqgDQu8gX53rbGDjYnu6Dcn2PTpWGao4qU9J2TcwY1TS06oPHadTTyUMSlF21J4xi6ygu9YAVY/o6BaRZ2AyIkDUST9jOqJu4rHrpJF1AoXJhlWMTFV+QBDm5PoI3QJ5dK8OzDYcraWvdiQgb049ddvRugVJAVXvPNahNyexb3d0uNTpbG12n5IzzDsvM76bXWFzTMOujiAmkFnl1LmP1blZPHVh3pVZurrRkaCOzT0DwseFV+LOVEs90uD8Uwfm8DDryas6gqwwgtsZ8hLMPIvee91rB9K+vIogm4ywuHW5BSNIdifJS3Dsu0nsIJIZzig7yZVg5l7YruqoWOGQp6vjAMVzQCaY/Y4nNgO6m6RhlbOegVRBb9w4yemM3xBmge6mHQuqCPPtqHBJHsRJXZvUMDNe1Rkr6pCEt6NC7UwXC0r3UYtgnsXzTqDw283qduDs55Nr7L9beIXtjL0MgLN0C3OUkzd3WibUzSeIrnMih0MNk4Xu6uz4HPu3gzGv0THvEmrgUMP2Daa6TvAuWARznw4Kzc642cZoo7xWXjXcgGR1q+OO6YLvFk4xu7eNILGEG5glyHoc4KJDMroQra1sAefYuh75lnnc81jQiDBPO75vxJm2L4JObk7P6eCjJK+onQE8vzr6hs7C9QCETNXzEPBEQYiKPh4FsXNwtGXCfNYdaiBKlmUlV2HoQTbreTxphJ8HPZNrXtYxIHdGUw1NYpHP9AX+vPOuPCuHTfUc4QTjJj/0SVH2pdWL8HP5nqXlYRPvAzAr5uGORT4OHPvuEY2Qz9yxXlWCzXBUPUl4Y3HHQ6ZOjjg4u8YrXt73sHUt3CkfcSJM8OltSNzI5myNTB0JxuT7Nobui8lgAnyWcIqedsfIt3LcWmMToZKnTr4bot6NmIoTPGd6tekCNhhVwxOhiln1U8m0RAA0N6t/ZmquIXJ651fasoZd3DNB9yb83Mynnp4kyEdyBunQrcn0lZ+t5di0vuTZky8YYkhiFXPG576lQojC7vC9e8q5sijMBluG3WOnUk92SigjDr+kQnufJ1KPD1k6OiMvGOZmIvU4VPL66S+pqHvr4jz6RzO0v63pI69csXvxuBv5sEt9yjuDMAtz0x631ZoKetWxV668gqO30bQhnUoMiM9/5UoJTTuPFTfE5EPuaC0L2SobS88BFIjQeAGB5C3i7iC+LR5UaO7xAoLaVuHg94xQHZLY/4ICAhoGLdRDMfh3Y9hMIVR3T85Pg7kngMiaTZfD6FoyXERgwh4RmihlIXuvbh4/30Sq4JjiUGWqzOwZguMxR91u+zESdojy8qa4qytkK7YYwiiGHBGaLO4iWyVDvuhCmWBQrSfCHW3lLCHpP2dxOCDK6pcUd1Gpou6oee94QzxBZi1teQIhmlH11mwCVs+gX48wY9R+OFWqiLYSYyh6DC4d0VJpGEyXKpKt/NOh52mQW5GlwtplvqQkVJjrWmDBvjtudQpMh+lKFSdRJlshzrMT+B1H8Sw4Z9bMZ+8ocEZbLVi/hLST5WmpxQYtFdxTeIu3Y9newK47qaluoUjiGuWpzuh9hezitQeZjz9FcgL6s+tvAOL165Qmr/t72GfYObWnvLlHB+K8YNUEIBFfW8kZGCSHjZWu7BrlSSWEs6imCEjwOrfNKewRyJWXlDHu8EOMCVMLlhGU/tvctuSkAFGC1nany5uFIuuB2eiMerVy6mmjp2kC+aK6Om6xR8c0jpPigO1CkFe2e2d9e42zrPpWBVDhlXGy2eOnnZXOyFJ3LoqoMfLTCGCP/htbWQlwLhIL3ZC9c0EDBafj2nEB8IL+G6cZajvQ27/QvqPgXzj1rQ55hdl5n7zpnXObapqC/XNtHwGOtnCq8k1ipdiuQL5/tQYvYf9Y1V9ZmwgRLiPsWhovfZVd37vEp9aBBvfDNMsR4X1VZLUOtPb9S3wEMnhg++kxR5ufq3yXpOkstF3tjiU+WdJ7+AJza8TJdeZX72oa9s9baK7q+2EG+SyeLUPsm3w/QEzTPxOBthElOJOOlW8H7GOLzcjgDosghcsTxHE8Qx0qS+t77l5dSEGhr2X0mjAsIaHX1qGvuqhjqDML3yCqVoZG5wRn0HC34hf2pZCzhlnwYeW/TaLfVdJ22z8E2nRk1rVgp9pq4IrOQifrP7LsUUKg1xrr+rBHHaqL+OjeQ+tLSYehL1CHegIbdQSstn50RR8ZS184outGMKP3qazskI4Ba6Nr7RZlRdO5fv2wblCNiA08w58uEwA1zItjudbe7s/695ESE9CGZ5orSBbOemHSstDHFwIjSJ5u4jD3Vcd3LCYWah3OATSHlDQLgR9SwtVeo5t6DvZ6UgfHxPQakca0WTMHdl4RfaHECK/Ptk3S2LTzsTBdkD3QNl6QFMmHVDGM+fcua+cRaWUNVa3jYXwuOmhhvHEA3XhtyOM6ms4UwN6joyDKeYL5MnOO/ywS9G71SAsblQvMfDNXK1T1eIz7/nIwIkSgdXUfJGngqBQPVXxIh/QCyodIGujCDTj8kI73pR01Wa9AP6oG5uOOyed4jzshwhJhW8N8SRNCAy/Xu2L1bhlcvviADBSWPiTC9geQd26GS55/eecCWdBfql/YFkl9IC3WKVO6Mfq3rtIgrobZrgtSdLVRyX+M9fI3BPPH5x6GYW6zVBB1tFCXRl5esPt7bH/W8I2eyXlrmEPAVMIw++bJ5L6i9UH5SWBus1TsIPFPddWu2JT2uHR7s5rPdXVfDbMLp5x7ZT1lgrvWFwzKzwHzP9osFSDq2qVUskm4Uj2Grp9IvBrNZqRhNkPrJs9HH+OO/SlgvmKp8FoTFnL3j6oZDvjacxpBQQeYvsKSR+RngPmapeIKMfV0XXSyzC3vCuawSnW6HU5uuDIuPwHMHZaKMG+/cJkfr3AWAhrFzd93/OGDOT04frRlv3+YuywVXyK/e5hvCce+Qn7vMHP/848Pq/0B8nuH+f/9re/ET5ffOcx/+fdvgzv29w3zf3/7pf/Mz5bfNcz/vB7KfZXoGsI8o31uvl8kFZ+rImb+oXlCW+C2wHD2R0oqXIPZ0aTx+1uOX7/98yON+nFyLmI4fnD/IOscQrX66ARWeIb+Rdd3iwdxAWG5e/39n3/uEI4x5mKxaL1qUltx7CWTj3lK0qrPDOpr9aPiJZJjTcWhWC/2/FDvmPmPWDBKS6Bgkv5oXJK6IW9Y/P1bZyjn0detwM1abXanS+OvmJXq/d3wuV0IgVtVk/N3Ef3pDybFlHR8cKtDl3R8cNOZGsbk9bc//ftbZ5TR4LRQPI3hNXqnKjEG7QhmanOGdp01NIzI+Zpr3rMa+GHOcDKncXiGTvRUnCYR4cXc9Biw9TX+RZ3JcsY0jA04Hp03FNLih525Xl348d3hzgDnjxa70Zr5t9j/9W9d6vQZeHOeO1VbyPYA0py2EixBSOEcMVWOMO7BYmL8Wc99poARxqwhAinBYz5TnM+wx9v9ARGbQ3lGrzgfmW3MgD+Dba7CnBmGoYTTh18OpR925tqdP7yDktE29+14mdgPHAV2TAG6Cx4L0rFkJA7HAApCZYKKduJ29AXZqgDDAc+G1KTtS6OIwZymAl9ECP3ChZCBhAdfANYDmcO26+CY/Ts+0qtJEe0kCB+go8cr59j0j3Lrc9UL/9FXvwiTPvjKlXndyFGnnUlDQO+lWJbRSfaCB4v4hOFwoB0e660lOVhhlpGSA60kJcIcBVzGBwN/BsDjtQzMPDA1mLsgLWgAAGrSs0k7V1P+S0pkfIwXH3UIidxQ2n9fAUGtQ7LDu0n++3WgknUmDXfGqMnhEEMJoSpy2ieW4cuCXsG6LPDWmRFOFn23oacA2oYzOzQb9R7xNxtdm4k2OjpseWLyHV7lqCD4wMRnxqaX4IfNbSkH+Y/hxgURZ5xzy3emufV7OsdzirbKqhPAS3QM5e/j1ueEhZ/kmPFOVTYjkv/vKIfhed228jOt2C5i1aw709Xhwi5JhE3JUFLATAEhe0ggyIoCpHSLUG1OtN/xvPF6TGIb0GijXtqXFArM7FtMNgJUexxV4cgKI9iq9y6tMvTywsOBzu6mYbygHQGqYxz6YWzlxGm/8B6ga286Z6Yn2Gfpyg9nVnGA/cx5cPMADG8PR8yHjbUM/VW6O8PJEh/l1uc4RY2I439GnTmi52SBfrtRgOn7vs2oGn5gON8T8D/VXzC6byuMi44u+wHHLGz8jsaE8Xntm4zi00tD18evRU32DSrA04MFb3s2Po4FLKM6jL1meNumO1SHi6Whoy9CVtNkEUXWRP8Ih9DU763DrHvnHaGaBW86ZBXv9KmjP1KqyCkzOKXqmw5NjdFZ2LqK7m4dvH2CxBbfdLirnDYK+Bi3fpQ8eIHZvYVz/NyCTaCJ7LuIMmtBFdxdzolh6kNkd3TIVMF/N7c+U/Pzh7J8pUTzK4jXpnHnYgaOW0SQih0dagGJ9kht6o3kj9ZydWt4OU7fn2yN7YoW5BAOceszXR3SBmZXINcis0ew7ufWj8ESb3SIctzw4t/FrU/L41z5Rofmw9n9Um59ToNCvulc3ba0wWh6LQLnbfe3nauNFUJxz9ITKifdgH9r7drgh21wF187xy0h7tUhsjn4X8itzzmQ9AJE/VtCqo1zUzdK5i/RrRs2IquI8/qOJT5CBsGADpEtz8SLP82tvxqwFHVm93U4E1H6cahh2LQVOscdC9aiagAgwkiF4z3c+tYgyoTzuZpeSkX7RPhDLoOd2YD9RQvWEKF8uGHYNAvdY2r5JY4k2WGYWdknQvPxmCCE7BEdYgBZl3z+VsliIGO8Kqmq4Gu49THKRtpFGEXVOF97ndz9EYTQVgnYsqmMzZVgUKVjCLFyCLMJW3F8VYzqQFslX7GYmON1WI0iRE2boP3npHIk7TRKzsXE0njuWE7o0DZnWxxdGs/Z+JQ8Kjhm+QKeB4TZKsdSRt29iW0eKNwnVLBi2DBtD5M0mBMBUdvbGo0rTtlaU50Rz5uv4Nb3JnvHsi52bz3Crc/vJ3vHyvtihDiezB1NODPaOzuPuSIx2E/2RV6OMas/STjeP00ihNFajZLwOKBOdy8e7R5vTOUuFNEmbuIRbv1oKneREjh+OgkPL0SDQ+ZWy3wYJswiFvBqundsQMTxQ90jwvZJlLEh+80IhRxG5thQ5SLarvz07HxXzkAZ212Bk6J02lSsVo5x6wtWfoep5OQ8xq0/fYtg67vEZxOk0QYNrVszAV4/ynWRl6NimJgM471lKrq+XwdBNEj3J1St9P52fVeHuOrl1n/tTNo21ZuOrhIVLPtzswYn+C/vLXNtTLG2jQ9tdmdYJGelf8teehFMza1T8XpWRR2BHXQgWvZy6190XJmqaUMgYoM6OgiiobsEJxWtBIh9EF871JFt9unklfH7/R0z8BlvMhBoJ8pxbSeQrS0GWr8ncsJVWgVQ8f+VU0B8rQPVx/aQJwpmK63KJV4fwdGJYSNfe7QLic8OUrGW8buOI15vwd6xsVPXYSFi9vpcKlZeStpPTS/gzgDvdrDFg20nkOPT8pZYuBFOsE/vvdByCAOARNtDIJNXv/fOrmmye4OVE9atwYpsQeIARM4GQi3w3RZGImQ9xMKXzhiH98EKGjVCU4MYY1vQt1vhqeX5GPHgEwRh3rzDrBXgH7fbxKeppBRaQY8wD1KgIsyHFpgpWCFUFQuVFgOc36O+hvl2I4xGh4RjwvaZkQsRsHtwQ6qJaalvYO6FSGjDTGElniOwM8xG2hGWbU+IPvkeyCtWy5vR8laUFbmFz6PYuzbM6Wkw4DnJhiufLZJ9CrR/K/pRa4a1gbk/4Dmp7c2seNrOYAW0WXK5cw7H1jiGiKQHYd4ur8MKlhCXIIr2oQ2zHOXLz4Z51p5ICCA6La1thaMPfJpow5ycR2C+ys3ieVtmPmBgiL6LA8F3mP0xmNu5mZU3kO9Z1JHW9B6tm6MLx2GY51HaMlWM17NldCo0bb9qwyzui0+H2W9DhAkVVAzTQu7CXFTDSQM9sXWqlgHELgDma81+Ob/nRDnt5da/6DCugMA7xGa9xcaI9J7s/caGEdHHrd+IoOyyVmiqAIf1DmCmxZHY1u6edp+eNMR2sMoJJjSEKJbJGVrYiediqQ6Svnvt52Qc/YMt76HQ0H0z7T3etU1JTOL9t0BpXrRTg0obxe9wjCAe/UVevX2DxrfCIVZgCs2WDvEMBe04r+ItB+A99b+a6jMHdJJ+5UVqvBLFWeyy1zDT24+VOzigM4qWF7FuPEOsY1sUt1Dk72Esn4qRAd083rYfLZZxINox3ryibQrvBtCIFn3gNoowu1fPWj62IYixe4Gqwns7RIqqoUHhcwSDNbuaB39/hvOTq3i3hlsmzMOTeKNDZN0E5c2I6ETHwYhg+Dnbjp43HaybRVZ7+FAMRgTDKV7bZ5rrLw+UyXvm16oqHmawf44QT3r/RLjYij95H/XtpXERXln3T4zIcqvgQs5exrn1y6R3UgN1vLdpSZTmQ1Oy6DPHoTms93ZgZO5mg1H1JOHn5taamrPBJHsc7h0l5930XLFL/NSDToQ6ltPzPlpZjVGaC4Y7PUWnHQ/WF3Dre6up9zoYZmU6sg8rJmd2cp4PnxCSkX1YUccCpuwtriAb3o+jprstpub50JmLz96Po+neeTPeNHkGx7HeUfey/cT7RHuCbB11zEbKBxo51OzzY9z6LoTjc7JafrBGtlJ4lmD37PGmof3zdGRWn2n2QejPrG9K9nvq3Si3/jqPRiHSdmRucWSTB2FuWuCO2UquudW/hFt/YY2+Y5LPW2s25sx1hvdhNYIRIjTVO7wJumNlOayGj3VWOOLMjb3LMVthUDX8/J9eeEQk2flwuIpijj40YX+yVTpcU0VVGps0HN3wvSbJ9kcqNbQVlSvbo7UsRJLt0jPsgMgqlFZsTxX3PEOIV5qNTmp/20Q2qncwGEWosdURwn6MRPmIfrhUtfFqHyKfnw3eBtGXoyS2J5jjyVZ4Fxgo5pPtmp+fHdlb5HlCWdHNYdXXP83enrIL2fqojoYX3+orK5VReUE7GJjj9ZykQ5tBwfYYHB+6CWV/shCPSNHt017t6YyIlqr5+b9kB4OGF99NoAi6xckym6D5a9L/qeq+Cy/+3u7qwGd3OBA//zRzfMOLv8Uk39Uh++cLP/9U+VuzUUAEaddYoqbmmHXieheEL9jBoG6a57DLMxxV7S3cRFFj00PdubvqinnC2Ue/tTW5pUNcleiG6QpRnvYh4muX1R1UofZuLVHWUG15xHi4B6Gaizyw4BCzV51RM7J23DDHPwDOD5Rm/wHV2kKV2q6oUSfVWdFQ2mPn7qneJpxNOYgr2Fp+wKIOmVXDjNYMNfz890Rqw4sf5gBZqLIy6Qj85FBz/Ie2eBc/f71RAGvjXy5WKlt3xrXTEg7FheP/a/aJqJuGOCNGfkq7K5+rKNqfAMpNTYsv3ruOgJfmJnHab14AtqijPLzS4t/Pz8/VvPH2ipb2HkrUscV/czTUyg/u5OdvON9Z1U8qgNM+iipatBZlj3D8P0u4N077ONkVeV7siFp/5ausc/9Sqrp/WmD7MysjHZvjhRZfu3/FR80bTyT/y1pHviFq/VloByIxtt+pgwKrJvlPjxvUUWS0iXqIHvMQhf0zhKs57Qg2OLoAAADQSURBVNdiQMT4q9VyFYZvtPh3p7LaWETQT6T2qGMVXmjxH+Hnb0j+2YuOZa2DOP6nNx+46ozXEPT7Fx2+6iLIj61yfIrUVOv6gsj1AxSXlZ1HafHJGSWjIdd3AzVwMc+vRyjtB3XMSYd4aYioOUTZ/tBa2dpY7c5otE/Aw+ttnyLEkj4nUnuiTDF17zto8WtOe+VNByp5bInrRQd/o+PxhqDXzD1U8gEdzxKuIbVX5kSK/30r4+tlmAi1Uq9ql2jx//c2RGk15Ht1fLQzP0T+P4MuhqVGr53xAAAAAElFTkSuQmCC"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HMM Tagger\n",
    "\n",
    "![pos_hmm.png](attachment:pos_hmm.png)\n",
    "<center>(Image courtesy: Georgetown University ENLP Lectures)</center>\n",
    "\n",
    "As shown in the figure above, each word $w_i$ has a corresponding tag $t_i$. The assumption is that the current tag determines directly the next tag, and once we know the tag we predict the word $w_i$ that could have occurred in the spot corresponding to the tag $t_i$. This arrow-based representation of a probability model is called a *graphical representation*. In the function `HMM_tag()` given below, we search for the 'train', 'dev' and 'test' conll files present in a given subdirectory `subdir` that have the file prefix `file_prefix`. Tuples of `subdir` and `file_prefix` are stored in the list variable *language_dirs*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def HMM_tag(UD_PATH, subdir, file_prefix):\n",
    "    #prepare train and test data, refer to data_prep.py for more details on \"create_dataset\"\n",
    "    tag_list, word_list, X_train, Y_train, X_test, Y_test = create_dataset(UD_PATH, subdir, file_prefix)\n",
    "    #create vocabulary by selecting the unique elements in word_list\n",
    "    vocab = list(np.unique([x for x in word_list if x != None]))\n",
    "    #P(word|tag)=C(word, tag)/C(tag) = emission_counts[tag][word]/unigram_counts[tag]\n",
    "    emission_counts = pair_counts(tag_list, word_list)\n",
    "    unigram_counts = Counter(tag_list)\n",
    "    #P(t_{i+1}|t_i)=C(t_{i},t_{i+1})/C(t_{i}) = bigram_counts[(t_{i},t_{i+1})]/unigram_counts[t_{i}]\n",
    "    bigram_counts = Counter([(tag_list[i],tag_list[i+1]) for i in range(len(word_list)-1)])\n",
    "    #calculate counts for starting/ending with a certain tag\n",
    "    start_counts = Counter([sentence[0] for sentence in Y_train])\n",
    "    end_counts = Counter([sentence[-1] for sentence in Y_train])\n",
    "    #Initialize a Pomrgranate HMM model\n",
    "    basic_model = HiddenMarkovModel(name=\"hmm-tagger\")\n",
    "    #add HMM states corresponding to each tag\n",
    "    cnt = 0\n",
    "    states = []\n",
    "    tags = list(unigram_counts.keys()) #list of all tags\n",
    "    for tag in tags:\n",
    "        emissions = DiscreteDistribution({word: emission_counts[tag][word]/unigram_counts[tag] for word in emission_counts[tag].keys()})\n",
    "        states.append(State(emissions, name=tag))\n",
    "        basic_model.add_states(states[cnt])\n",
    "        cnt += 1\n",
    "    #add edge from model.start to each tag, each tag to model.end\n",
    "    cnt = 0\n",
    "    for tag in tags:\n",
    "        basic_model.add_transition(basic_model.start, states[cnt], start_counts[tag]/unigram_counts[tag])\n",
    "        basic_model.add_transition(states[cnt], basic_model.end, end_counts[tag]/unigram_counts[tag])\n",
    "        cnt += 1\n",
    "    #adding transitions between various states in the model\n",
    "    for i in range(len(tags)):\n",
    "        for j in range(len(tags)):\n",
    "            basic_model.add_transition(states[i], states[j], bigram_counts[(tags[i],tags[j])]/unigram_counts[tags[i]])\n",
    "    #model checking\n",
    "    basic_model.bake()\n",
    "    #calculate train and test accuracy\n",
    "    train_acc = accuracy(X_train, Y_train, basic_model, vocab)\n",
    "    test_acc = accuracy(X_test, Y_test, basic_model, vocab)\n",
    "    return train_acc, test_acc, len(X_train), len(X_test), len(word_list), len(vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TrainAcc           0.980505\n",
      "TestAcc            0.923100\n",
      "TrainLength     1316.000000\n",
      "TestLength       425.000000\n",
      "NumWords       39211.000000\n",
      "NumVocab        4423.000000\n",
      "Name: UD_Afrikaans-AfriBooms/, dtype: float64\n",
      "TrainAcc            0.931551\n",
      "TestAcc             0.816928\n",
      "TrainLength     11477.000000\n",
      "TestLength       1306.000000\n",
      "NumWords       182030.000000\n",
      "NumVocab        12100.000000\n",
      "Name: UD_Ancient_Greek-Perseus/, dtype: float64\n",
      "TrainAcc            0.964709\n",
      "TestAcc             0.949527\n",
      "TrainLength     15015.000000\n",
      "TestLength       1047.000000\n",
      "NumWords       200685.000000\n",
      "NumVocab         9137.000000\n",
      "Name: UD_Ancient_Greek-PROIEL/, dtype: float64\n",
      "TrainAcc            0.343521\n",
      "TestAcc             0.338337\n",
      "TrainLength     15790.000000\n",
      "TestLength       1963.000000\n",
      "NumWords       757062.000000\n",
      "NumVocab         4613.000000\n",
      "Name: UD_Arabic-NYUAD/, dtype: float64\n",
      "TrainAcc            0.871067\n",
      "TestAcc             0.847317\n",
      "TrainLength      6076.000000\n",
      "TestLength        680.000000\n",
      "NumWords       288661.000000\n",
      "NumVocab        14520.000000\n",
      "Name: UD_Arabic-PADT/, dtype: float64\n",
      "TrainAcc           0.95543\n",
      "TestAcc            0.85853\n",
      "TrainLength     1976.00000\n",
      "TestLength       278.00000\n",
      "NumWords       47743.00000\n",
      "NumVocab        6630.00000\n",
      "Name: UD_Armenian-ArmTDP/, dtype: float64\n",
      "TrainAcc           0.966013\n",
      "TestAcc            0.926561\n",
      "TrainLength     5397.000000\n",
      "TestLength      1799.000000\n",
      "NumWords       97069.000000\n",
      "NumVocab        9929.000000\n",
      "Name: UD_Basque-BDT/, dtype: float64\n",
      "TrainAcc          0.986898\n",
      "TestAcc           0.828519\n",
      "TrainLength     320.000000\n",
      "TestLength      253.000000\n",
      "NumWords       7890.000000\n",
      "NumVocab       2393.000000\n",
      "Name: UD_Belarusian-HSE/, dtype: float64\n",
      "TrainAcc            0.986843\n",
      "TestAcc             0.958853\n",
      "TrainLength      8908.000000\n",
      "TestLength       1116.000000\n",
      "NumWords       140425.000000\n",
      "NumVocab        14168.000000\n",
      "Name: UD_Bulgarian-BTB/, dtype: float64\n",
      "Dev set not found, no worries! Train set must be good enough.\n",
      "TrainAcc         1.000000\n",
      "TestAcc          0.395235\n",
      "TrainLength     19.000000\n",
      "TestLength     908.000000\n",
      "NumWords       153.000000\n",
      "NumVocab        99.000000\n",
      "Name: UD_Buryat-BDT/, dtype: float64\n",
      "TrainAcc            0.975822\n",
      "TestAcc             0.951876\n",
      "TrainLength     13124.000000\n",
      "TestLength       1846.000000\n",
      "NumWords       475133.000000\n",
      "NumVocab        23372.000000\n",
      "Name: UD_Catalan-AnCora/, dtype: float64\n",
      "TrainAcc            0.941908\n",
      "TestAcc             0.844322\n",
      "TrainLength      3998.000000\n",
      "TestLength        500.000000\n",
      "NumWords       111279.000000\n",
      "NumVocab        18992.000000\n",
      "Name: UD_Chinese-GSD/, dtype: float64\n",
      "TrainAcc            0.941685\n",
      "TestAcc             0.844905\n",
      "TrainLength      3998.000000\n",
      "TestLength        500.000000\n",
      "NumWords       111279.000000\n",
      "NumVocab        18902.000000\n",
      "Name: UD_Chinese-GSDSimp/, dtype: float64\n",
      "TrainAcc            0.889182\n",
      "TestAcc             0.879095\n",
      "TrainLength     22521.000000\n",
      "TestLength       2073.000000\n",
      "NumWords       120152.000000\n",
      "NumVocab         3785.000000\n",
      "Name: UD_Classical_Chinese-Kyoto/, dtype: float64\n",
      "TrainAcc           0.749342\n",
      "TestAcc            0.726227\n",
      "TrainLength      885.000000\n",
      "TestLength       403.000000\n",
      "NumWords       41921.000000\n",
      "NumVocab        1931.000000\n",
      "Name: UD_Coptic-Scriptorium/, dtype: float64\n",
      "TrainAcc            0.974191\n",
      "TestAcc             0.927947\n",
      "TrainLength      6915.000000\n",
      "TestLength       1136.000000\n",
      "NumWords       175149.000000\n",
      "NumVocab        17584.000000\n",
      "Name: UD_Croatian-SET/, dtype: float64\n",
      "TrainAcc            0.988501\n",
      "TestAcc             0.962960\n",
      "TrainLength     23479.000000\n",
      "TestLength        628.000000\n",
      "NumWords       485648.000000\n",
      "NumVocab        28327.000000\n",
      "Name: UD_Czech-CAC/, dtype: float64\n",
      "TrainAcc           0.987088\n",
      "TestAcc            0.830829\n",
      "TrainLength      861.000000\n",
      "TestLength       136.000000\n",
      "NumWords       31785.000000\n",
      "NumVocab        2455.000000\n",
      "Name: UD_Czech-CLTT/, dtype: float64\n",
      "TrainAcc            0.976622\n",
      "TestAcc             0.951057\n",
      "TrainLength     10161.000000\n",
      "TestLength       1291.000000\n",
      "NumWords       151200.000000\n",
      "NumVocab        13072.000000\n",
      "Name: UD_Czech-FicTree/, dtype: float64\n",
      "TrainAcc       9.867107e-01\n",
      "TestAcc        9.688990e-01\n",
      "TrainLength    6.849600e+04\n",
      "TestLength     1.014800e+04\n",
      "NumWords       1.337276e+06\n",
      "NumVocab       5.603400e+04\n",
      "Name: UD_Czech-PDT/, dtype: float64\n",
      "TrainAcc           0.966960\n",
      "TestAcc            0.911903\n",
      "TrainLength     4384.000000\n",
      "TestLength       565.000000\n",
      "NumWords       90710.000000\n",
      "NumVocab       12488.000000\n",
      "Name: UD_Danish-DDT/, dtype: float64\n",
      "TrainAcc            0.967609\n",
      "TestAcc             0.896770\n",
      "TrainLength     12265.000000\n",
      "TestLength        596.000000\n",
      "NumWords       197548.000000\n",
      "NumVocab        22263.000000\n",
      "Name: UD_Dutch-Alpino/, dtype: float64\n",
      "TrainAcc           0.97413\n",
      "TestAcc            0.90493\n",
      "TrainLength     5788.00000\n",
      "TestLength       875.00000\n",
      "NumWords       86582.00000\n",
      "NumVocab       12367.00000\n",
      "Name: UD_Dutch-LassySmall/, dtype: float64\n",
      "TrainAcc           0.197006\n",
      "TestAcc            0.197790\n",
      "TrainLength     4125.000000\n",
      "TestLength       500.000000\n",
      "NumWords       88090.000000\n",
      "NumVocab           1.000000\n",
      "Name: UD_English-ESL/, dtype: float64\n",
      "TrainAcc            0.950479\n",
      "TestAcc             0.899749\n",
      "TrainLength     12544.000000\n",
      "TestLength       2077.000000\n",
      "NumWords       229759.000000\n",
      "NumVocab        16150.000000\n",
      "Name: UD_English-EWT/, dtype: float64\n",
      "TrainAcc           0.956525\n",
      "TestAcc            0.873210\n",
      "TrainLength     4288.000000\n",
      "TestLength       890.000000\n",
      "NumWords       97450.000000\n",
      "NumVocab       10707.000000\n",
      "Name: UD_English-GUM/, dtype: float64\n",
      "TrainAcc           0.172274\n",
      "TestAcc            0.161834\n",
      "TrainLength      588.000000\n",
      "TestLength       158.000000\n",
      "NumWords       13320.000000\n",
      "NumVocab           1.000000\n",
      "Name: UD_English-GUMReddit/, dtype: float64\n",
      "TrainAcc           0.958644\n",
      "TestAcc            0.914795\n",
      "TrainLength     3177.000000\n",
      "TestLength      1035.000000\n",
      "NumWords       76542.000000\n",
      "NumVocab        7003.000000\n",
      "Name: UD_English-LinES/, dtype: float64\n",
      "TrainAcc           0.965569\n",
      "TestAcc            0.902110\n",
      "TrainLength     1782.000000\n",
      "TestLength       153.000000\n",
      "NumWords       46254.000000\n",
      "NumVocab        5565.000000\n",
      "Name: UD_English-ParTUT/, dtype: float64\n",
      "TrainAcc            0.974473\n",
      "TestAcc             0.916591\n",
      "TrainLength     24634.000000\n",
      "TestLength       3214.000000\n",
      "NumWords       389639.000000\n",
      "NumVocab        39803.000000\n",
      "Name: UD_Estonian-EDT/, dtype: float64\n",
      "TrainAcc           0.965879\n",
      "TestAcc            0.813317\n",
      "TrainLength     1565.000000\n",
      "TestLength       912.000000\n",
      "NumWords       29993.000000\n",
      "NumVocab        5189.000000\n",
      "Name: UD_Estonian-EWT/, dtype: float64\n",
      "TrainAcc            0.954717\n",
      "TestAcc             0.896634\n",
      "TrainLength     14982.000000\n",
      "TestLength       1867.000000\n",
      "NumWords       143599.000000\n",
      "NumVocab        20135.000000\n",
      "Name: UD_Finnish-FTB/, dtype: float64\n",
      "TrainAcc            0.984648\n",
      "TestAcc             0.922371\n",
      "TrainLength     12218.000000\n",
      "TestLength       1555.000000\n",
      "NumWords       181571.000000\n",
      "NumVocab        24659.000000\n",
      "Name: UD_Finnish-TDT/, dtype: float64\n",
      "TrainAcc            0.241487\n",
      "TestAcc             0.237774\n",
      "TrainLength     14760.000000\n",
      "TestLength       2541.000000\n",
      "NumWords       511033.000000\n",
      "NumVocab         1466.000000\n",
      "Name: UD_French-FTB/, dtype: float64\n",
      "TrainAcc            0.957992\n",
      "TestAcc             0.927559\n",
      "TrainLength     14450.000000\n",
      "TestLength        416.000000\n",
      "NumWords       401123.000000\n",
      "NumVocab        33940.000000\n",
      "Name: UD_French-GSD/, dtype: float64\n",
      "TrainAcc           0.949597\n",
      "TestAcc            0.910475\n",
      "TrainLength      804.000000\n",
      "TestLength       110.000000\n",
      "NumWords       26840.000000\n",
      "NumVocab        2932.000000\n",
      "Name: UD_French-ParTUT/, dtype: float64\n",
      "TrainAcc           0.960428\n",
      "TestAcc            0.927496\n",
      "TrainLength     2232.000000\n",
      "TestLength       456.000000\n",
      "NumWords       62161.000000\n",
      "NumVocab        6308.000000\n",
      "Name: UD_French-Sequoia/, dtype: float64\n",
      "TrainAcc           0.954188\n",
      "TestAcc            0.890780\n",
      "TrainLength     1157.000000\n",
      "TestLength       726.000000\n",
      "NumWords       25743.000000\n",
      "NumVocab        2708.000000\n",
      "Name: UD_French-Spoken/, dtype: float64\n",
      "TrainAcc            0.900723\n",
      "TestAcc             0.877956\n",
      "TrainLength      2273.000000\n",
      "TestLength        861.000000\n",
      "NumWords       119197.000000\n",
      "NumVocab         9233.000000\n",
      "Name: UD_Galician-CTG/, dtype: float64\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dev set not found, no worries! Train set must be good enough.\n",
      "TrainAcc           0.893159\n",
      "TestAcc            0.828807\n",
      "TrainLength      600.000000\n",
      "TestLength       400.000000\n",
      "NumWords       16707.000000\n",
      "NumVocab        3018.000000\n",
      "Name: UD_Galician-TreeGal/, dtype: float64\n",
      "TrainAcc            0.942002\n",
      "TestAcc             0.886299\n",
      "TrainLength     13815.000000\n",
      "TestLength        977.000000\n",
      "NumWords       281064.000000\n",
      "NumVocab        41665.000000\n",
      "Name: UD_German-GSD/, dtype: float64\n",
      "TrainAcc           0.963948\n",
      "TestAcc            0.950677\n",
      "TrainLength     3388.000000\n",
      "TestLength      1029.000000\n",
      "NumWords       45138.000000\n",
      "NumVocab        3070.000000\n",
      "Name: UD_Gothic-PROIEL/, dtype: float64\n",
      "TrainAcc           0.965071\n",
      "TestAcc            0.923640\n",
      "TrainLength     1663.000000\n",
      "TestLength       456.000000\n",
      "NumWords       54187.000000\n",
      "NumVocab        5741.000000\n",
      "Name: UD_Greek-GDT/, dtype: float64\n",
      "TrainAcc            0.786980\n",
      "TestAcc             0.739975\n",
      "TrainLength      5242.000000\n",
      "TestLength        491.000000\n",
      "NumWords       183315.000000\n",
      "NumVocab         9891.000000\n",
      "Name: UD_Hebrew-HTB/, dtype: float64\n",
      "TrainAcc            0.952661\n",
      "TestAcc             0.924979\n",
      "TrainLength     13305.000000\n",
      "TestLength       1684.000000\n",
      "NumWords       316274.000000\n",
      "NumVocab        14633.000000\n",
      "Name: UD_Hindi-HDTB/, dtype: float64\n",
      "TrainAcc           0.206697\n",
      "TestAcc            0.225493\n",
      "TrainLength     1449.000000\n",
      "TestLength       225.000000\n",
      "NumWords       23614.000000\n",
      "NumVocab           1.000000\n",
      "Name: UD_Hindi_English-HIENCS/, dtype: float64\n",
      "TrainAcc           0.970421\n",
      "TestAcc            0.830015\n",
      "TrainLength      911.000000\n",
      "TestLength       449.000000\n",
      "NumWords       31584.000000\n",
      "NumVocab        7224.000000\n",
      "Name: UD_Hungarian-Szeged/, dtype: float64\n",
      "TrainAcc            0.935596\n",
      "TestAcc             0.861290\n",
      "TrainLength      4478.000000\n",
      "TestLength        557.000000\n",
      "NumWords       110143.000000\n",
      "NumVocab        17748.000000\n",
      "Name: UD_Indonesian-GSD/, dtype: float64\n",
      "TrainAcc           0.952738\n",
      "TestAcc            0.898210\n",
      "TrainLength     2020.000000\n",
      "TestLength       454.000000\n",
      "NumWords       54636.000000\n",
      "NumVocab        6639.000000\n",
      "Name: UD_Irish-IDT/, dtype: float64\n",
      "TrainAcc            0.913541\n",
      "TestAcc             0.883708\n",
      "TrainLength     13122.000000\n",
      "TestLength        482.000000\n",
      "NumWords       307078.000000\n",
      "NumVocab        19161.000000\n",
      "Name: UD_Italian-ISDT/, dtype: float64\n",
      "TrainAcc           0.917050\n",
      "TestAcc            0.880122\n",
      "TrainLength     1782.000000\n",
      "TestLength       153.000000\n",
      "NumWords       55572.000000\n",
      "NumVocab        5602.000000\n",
      "Name: UD_Italian-ParTUT/, dtype: float64\n",
      "TrainAcc            0.935241\n",
      "TestAcc             0.876466\n",
      "TrainLength      5369.000000\n",
      "TestLength        674.000000\n",
      "NumWords       116371.000000\n",
      "NumVocab        13524.000000\n",
      "Name: UD_Italian-PoSTWITA/, dtype: float64\n",
      "TrainAcc           0.937414\n",
      "TestAcc            0.855982\n",
      "TrainLength     1139.000000\n",
      "TestLength       142.000000\n",
      "NumWords       27742.000000\n",
      "NumVocab        4699.000000\n",
      "Name: UD_Italian-TWITTIRO/, dtype: float64\n",
      "TrainAcc            0.909450\n",
      "TestAcc             0.883616\n",
      "TrainLength      8278.000000\n",
      "TestLength       1067.000000\n",
      "NumWords       272114.000000\n",
      "NumVocab        15537.000000\n",
      "Name: UD_Italian-VIT/, dtype: float64\n",
      "TrainAcc       2.645903e-01\n",
      "TestAcc        2.483601e-01\n",
      "TrainLength    4.080200e+04\n",
      "TestLength     7.881000e+03\n",
      "NumWords       1.104528e+06\n",
      "NumVocab       1.000000e+00\n",
      "Name: UD_Japanese-BCCWJ/, dtype: float64\n",
      "TrainAcc            0.971801\n",
      "TestAcc             0.928265\n",
      "TrainLength      7028.000000\n",
      "TestLength        543.000000\n",
      "NumWords       179571.000000\n",
      "NumVocab        19898.000000\n",
      "Name: UD_Japanese-GSD/, dtype: float64\n",
      "Dev set not found, no worries! Train set must be good enough.\n",
      "TrainAcc          0.950640\n",
      "TestAcc           0.454151\n",
      "TrainLength      31.000000\n",
      "TestLength     1047.000000\n",
      "NumWords        547.000000\n",
      "NumVocab        253.000000\n",
      "Name: UD_Kazakh-KTB/, dtype: float64\n",
      "TrainAcc           0.976223\n",
      "TestAcc            0.806543\n",
      "TrainLength     4401.000000\n",
      "TestLength       989.000000\n",
      "NumWords       68645.000000\n",
      "NumVocab       32388.000000\n",
      "Name: UD_Korean-GSD/, dtype: float64\n",
      "TrainAcc            0.975939\n",
      "TestAcc             0.800536\n",
      "TrainLength     23011.000000\n",
      "TestLength       2287.000000\n",
      "NumWords       321724.000000\n",
      "NumVocab        92850.000000\n",
      "Name: UD_Korean-Kaist/, dtype: float64\n",
      "Dev set not found, no worries! Train set must be good enough.\n",
      "TrainAcc         0.987603\n",
      "TestAcc          0.348959\n",
      "TrainLength     20.000000\n",
      "TestLength     734.000000\n",
      "NumWords       242.000000\n",
      "NumVocab       141.000000\n",
      "Name: UD_Kurmanji-MG/, dtype: float64\n",
      "TrainAcc            0.978361\n",
      "TestAcc             0.969037\n",
      "TrainLength     16810.000000\n",
      "TestLength       2101.000000\n",
      "NumWords       323193.000000\n",
      "NumVocab         4546.000000\n",
      "Name: UD_Latin-ITTB/, dtype: float64\n",
      "TrainAcc            0.983055\n",
      "TestAcc             0.972881\n",
      "TrainLength      7290.000000\n",
      "TestLength        884.000000\n",
      "NumWords       218332.000000\n",
      "NumVocab         3370.000000\n",
      "Name: UD_Latin-LLCT/, dtype: float64\n",
      "Dev set not found, no worries! Train set must be good enough.\n",
      "TrainAcc           0.968929\n",
      "TestAcc            0.848731\n",
      "TrainLength     1334.000000\n",
      "TestLength       939.000000\n",
      "NumWords       18184.000000\n",
      "NumVocab        3252.000000\n",
      "Name: UD_Latin-Perseus/, dtype: float64\n",
      "TrainAcc            0.964830\n",
      "TestAcc             0.949968\n",
      "TrainLength     15918.000000\n",
      "TestLength       1260.000000\n",
      "NumWords       186072.000000\n",
      "NumVocab         8287.000000\n",
      "Name: UD_Latin-PROIEL/, dtype: float64\n",
      "TrainAcc            0.971609\n",
      "TestAcc             0.920778\n",
      "TrainLength     10157.000000\n",
      "TestLength       1823.000000\n",
      "NumWords       194003.000000\n",
      "NumVocab        18389.000000\n",
      "Name: UD_Latvian-LVTB/, dtype: float64\n",
      "TrainAcc           0.982934\n",
      "TestAcc            0.875438\n",
      "TrainLength     2342.000000\n",
      "TestLength       684.000000\n",
      "NumWords       59205.000000\n",
      "NumVocab        7321.000000\n",
      "Name: UD_Lithuanian-ALKSNIS/, dtype: float64\n",
      "TrainAcc          0.969088\n",
      "TestAcc           0.695283\n",
      "TrainLength     154.000000\n",
      "TestLength       55.000000\n",
      "NumWords       4296.000000\n",
      "NumVocab       1335.000000\n",
      "Name: UD_Lithuanian-HSE/, dtype: float64\n",
      "Dev set not found, no worries! Train set must be good enough.\n",
      "TrainAcc         0.993056\n",
      "TestAcc          0.380376\n",
      "TrainLength     19.000000\n",
      "TestLength     106.000000\n",
      "NumWords       144.000000\n",
      "NumVocab        88.000000\n",
      "Name: UD_Livvi-KKPP/, dtype: float64\n",
      "TrainAcc           0.060273\n",
      "TestAcc            0.047142\n",
      "TrainLength     1124.000000\n",
      "TestLength       518.000000\n",
      "NumWords       33089.000000\n",
      "NumVocab           0.000000\n",
      "Name: UD_Maltese-MUDT/, dtype: float64\n",
      "TrainAcc          0.882966\n",
      "TestAcc           0.779018\n",
      "TrainLength     374.000000\n",
      "TestLength       47.000000\n",
      "NumWords       3732.000000\n",
      "NumVocab        659.000000\n",
      "Name: UD_Marathi-UFAL/, dtype: float64\n",
      "TrainAcc            0.970806\n",
      "TestAcc             0.941204\n",
      "TrainLength      6002.000000\n",
      "TestLength        804.000000\n",
      "NumWords       114025.000000\n",
      "NumVocab         3753.000000\n",
      "Name: UD_Naija-NSC/, dtype: float64\n",
      "Dev set not found, no worries! Train set must be good enough.\n",
      "TrainAcc           0.966380\n",
      "TestAcc            0.844256\n",
      "TrainLength     2257.000000\n",
      "TestLength       865.000000\n",
      "NumWords       16835.000000\n",
      "NumVocab        3017.000000\n",
      "Name: UD_North_Sami-Giella/, dtype: float64\n",
      "TrainAcc            0.964829\n",
      "TestAcc             0.922345\n",
      "TrainLength     15697.000000\n",
      "TestLength       1939.000000\n",
      "NumWords       280255.000000\n",
      "NumVocab        22139.000000\n",
      "Name: UD_Norwegian-Bokmaal/, dtype: float64\n",
      "TrainAcc            0.954453\n",
      "TestAcc             0.905300\n",
      "TrainLength     14175.000000\n",
      "TestLength       1511.000000\n",
      "NumWords       276580.000000\n",
      "NumVocab        21899.000000\n",
      "Name: UD_Norwegian-Nynorsk/, dtype: float64\n",
      "TrainAcc           0.952037\n",
      "TestAcc            0.906773\n",
      "TrainLength     3413.000000\n",
      "TestLength       957.000000\n",
      "NumWords       45370.000000\n",
      "NumVocab        2896.000000\n",
      "Name: UD_Norwegian-NynorskLIA/, dtype: float64\n",
      "TrainAcc           0.954624\n",
      "TestAcc            0.934702\n",
      "TrainLength     4125.000000\n",
      "TestLength      1141.000000\n",
      "NumWords       47532.000000\n",
      "NumVocab        2787.000000\n",
      "Name: UD_Old_Church_Slavonic-PROIEL/, dtype: float64\n",
      "TrainAcc            0.134526\n",
      "TestAcc             0.144371\n",
      "TrainLength     13910.000000\n",
      "TestLength       1927.000000\n",
      "NumWords       153382.000000\n",
      "NumVocab            0.000000\n",
      "Name: UD_Old_French-SRCMF/, dtype: float64\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dev set not found, no worries! Train set must be good enough.\n",
      "TrainAcc           0.989858\n",
      "TestAcc            0.775751\n",
      "TrainLength      320.000000\n",
      "TestLength       637.000000\n",
      "NumWords       10156.000000\n",
      "NumVocab        1944.000000\n",
      "Name: UD_Old_Russian-RNC/, dtype: float64\n",
      "TrainAcc            0.958899\n",
      "TestAcc             0.916271\n",
      "TrainLength     13337.000000\n",
      "TestLength       1756.000000\n",
      "NumWords       134421.000000\n",
      "NumVocab         8399.000000\n",
      "Name: UD_Old_Russian-TOROT/, dtype: float64\n",
      "TrainAcc            0.925901\n",
      "TestAcc             0.927180\n",
      "TrainLength      4799.000000\n",
      "TestLength        600.000000\n",
      "NumWords       138089.000000\n",
      "NumVocab         6955.000000\n",
      "Name: UD_Persian-Seraji/, dtype: float64\n",
      "TrainAcc            0.979829\n",
      "TestAcc             0.929301\n",
      "TrainLength     13775.000000\n",
      "TestLength       1727.000000\n",
      "NumWords       117855.000000\n",
      "NumVocab        14933.000000\n",
      "Name: UD_Polish-LFG/, dtype: float64\n",
      "TrainAcc            0.951613\n",
      "TestAcc             0.913337\n",
      "TrainLength     17723.000000\n",
      "TestLength       2215.000000\n",
      "NumWords       318664.000000\n",
      "NumVocab        26736.000000\n",
      "Name: UD_Polish-PDB/, dtype: float64\n",
      "TrainAcc            0.900777\n",
      "TestAcc             0.866253\n",
      "TrainLength      8329.000000\n",
      "TestLength        476.000000\n",
      "NumWords       233746.000000\n",
      "NumVocab        18837.000000\n",
      "Name: UD_Portuguese-Bosque/, dtype: float64\n",
      "TrainAcc            0.590644\n",
      "TestAcc             0.581913\n",
      "TrainLength      9665.000000\n",
      "TestLength       1204.000000\n",
      "NumWords       308074.000000\n",
      "NumVocab         9469.000000\n",
      "Name: UD_Portuguese-GSD/, dtype: float64\n",
      "TrainAcc            0.962092\n",
      "TestAcc             0.950348\n",
      "TrainLength     19300.000000\n",
      "TestLength       1052.000000\n",
      "NumWords       428973.000000\n",
      "NumVocab        10337.000000\n",
      "Name: UD_Romanian-Nonstandard/, dtype: float64\n",
      "TrainAcc            0.982035\n",
      "TestAcc             0.951299\n",
      "TrainLength      8044.000000\n",
      "TestLength        729.000000\n",
      "NumWords       202187.000000\n",
      "NumVocab        16784.000000\n",
      "Name: UD_Romanian-RRT/, dtype: float64\n",
      "TrainAcc           0.986468\n",
      "TestAcc            0.893281\n",
      "TrainLength     3851.000000\n",
      "TestLength       601.000000\n",
      "NumWords       86615.000000\n",
      "NumVocab       17399.000000\n",
      "Name: UD_Russian-GSD/, dtype: float64\n",
      "TrainAcc            0.983468\n",
      "TestAcc             0.961599\n",
      "TrainLength     48815.000000\n",
      "TestLength       6491.000000\n",
      "NumWords       990218.000000\n",
      "NumVocab        42614.000000\n",
      "Name: UD_Russian-SynTagRus/, dtype: float64\n",
      "TrainAcc           0.978171\n",
      "TestAcc            0.893661\n",
      "TrainLength     3139.000000\n",
      "TestLength       881.000000\n",
      "NumWords       53651.000000\n",
      "NumVocab       11076.000000\n",
      "Name: UD_Russian-Taiga/, dtype: float64\n",
      "Dev set not found, no worries! Train set must be good enough.\n",
      "TrainAcc           0.970421\n",
      "TestAcc            0.887097\n",
      "TrainLength     2524.000000\n",
      "TestLength      1473.000000\n",
      "NumWords       17445.000000\n",
      "NumVocab        2717.000000\n",
      "Name: UD_Sanskrit-Vedic/, dtype: float64\n",
      "TrainAcc           0.915484\n",
      "TestAcc            0.848204\n",
      "TrainLength     1016.000000\n",
      "TestLength       538.000000\n",
      "NumWords       32983.000000\n",
      "NumVocab        3918.000000\n",
      "Name: UD_Scottish_Gaelic-ARCOSG/, dtype: float64\n",
      "TrainAcc           0.977549\n",
      "TestAcc            0.933193\n",
      "TrainLength     3329.000000\n",
      "TestLength       520.000000\n",
      "NumWords       86252.000000\n",
      "NumVocab        9069.000000\n",
      "Name: UD_Serbian-SET/, dtype: float64\n",
      "TrainAcc           0.978512\n",
      "TestAcc            0.851150\n",
      "TrainLength     8484.000000\n",
      "TestLength      1061.000000\n",
      "NumWords       93090.000000\n",
      "NumVocab       12716.000000\n",
      "Name: UD_Slovak-SNK/, dtype: float64\n",
      "TrainAcc            0.987739\n",
      "TestAcc             0.935498\n",
      "TrainLength      6479.000000\n",
      "TestLength        788.000000\n",
      "NumWords       126593.000000\n",
      "NumVocab        15918.000000\n",
      "Name: UD_Slovenian-SSJ/, dtype: float64\n",
      "Dev set not found, no worries! Train set must be good enough.\n",
      "TrainAcc           0.958609\n",
      "TestAcc            0.874688\n",
      "TrainLength     2078.000000\n",
      "TestLength      1110.000000\n",
      "NumWords       19473.000000\n",
      "NumVocab        2999.000000\n",
      "Name: UD_Slovenian-SST/, dtype: float64\n",
      "TrainAcc            0.976762\n",
      "TestAcc             0.953372\n",
      "TrainLength     14306.000000\n",
      "TestLength       1721.000000\n",
      "NumWords       498655.000000\n",
      "NumVocab        26182.000000\n",
      "Name: UD_Spanish-AnCora/, dtype: float64\n",
      "TrainAcc            0.938964\n",
      "TestAcc             0.910573\n",
      "TrainLength     14188.000000\n",
      "TestLength        426.000000\n",
      "NumWords       427539.000000\n",
      "NumVocab        35328.000000\n",
      "Name: UD_Spanish-GSD/, dtype: float64\n",
      "TrainAcc           0.966949\n",
      "TestAcc            0.917500\n",
      "TrainLength     3177.000000\n",
      "TestLength      1035.000000\n",
      "NumWords       73966.000000\n",
      "NumVocab        8920.000000\n",
      "Name: UD_Swedish-LinES/, dtype: float64\n",
      "TrainAcc           0.963445\n",
      "TestAcc            0.917689\n",
      "TrainLength     4304.000000\n",
      "TestLength      1219.000000\n",
      "NumWords       76472.000000\n",
      "NumVocab        8910.000000\n",
      "Name: UD_Swedish-Talbanken/, dtype: float64\n",
      "TrainAcc          0.408257\n",
      "TestAcc           0.425532\n",
      "TrainLength      88.000000\n",
      "TestLength       34.000000\n",
      "NumWords       1328.000000\n",
      "NumVocab          0.000000\n",
      "Name: UD_Swedish_Sign_Language-SSLC/, dtype: float64\n",
      "TrainAcc          0.860160\n",
      "TestAcc           0.702703\n",
      "TrainLength     401.000000\n",
      "TestLength      120.000000\n",
      "NumWords       8233.000000\n",
      "NumVocab       1783.000000\n",
      "Name: UD_Tamil-TTB/, dtype: float64\n",
      "TrainAcc          0.467859\n",
      "TestAcc           0.463245\n",
      "TrainLength    1052.000000\n",
      "TestLength      146.000000\n",
      "NumWords       5744.000000\n",
      "NumVocab          0.000000\n",
      "Name: UD_Telugu-MTG/, dtype: float64\n",
      "TrainAcc           0.945392\n",
      "TestAcc            0.910767\n",
      "TrainLength     3665.000000\n",
      "TestLength       983.000000\n",
      "NumWords       49043.000000\n",
      "NumVocab        5526.000000\n",
      "Name: UD_Turkish-IMST/, dtype: float64\n",
      "TrainAcc            0.979028\n",
      "TestAcc             0.903137\n",
      "TrainLength      5497.000000\n",
      "TestLength        892.000000\n",
      "NumWords       105225.000000\n",
      "NumVocab        16597.000000\n",
      "Name: UD_Ukrainian-IU/, dtype: float64\n",
      "Dev set not found, no worries! Train set must be good enough.\n",
      "TrainAcc         1.000000\n",
      "TestAcc          0.452776\n",
      "TrainLength     23.000000\n",
      "TestLength     623.000000\n",
      "NumWords       460.000000\n",
      "NumVocab       247.000000\n",
      "Name: UD_Upper_Sorbian-UFAL/, dtype: float64\n",
      "TrainAcc            0.933719\n",
      "TestAcc             0.904566\n",
      "TrainLength      4044.000000\n",
      "TestLength        535.000000\n",
      "NumWords       123271.000000\n",
      "NumVocab         8979.000000\n",
      "Name: UD_Urdu-UDTB/, dtype: float64\n",
      "TrainAcc           0.866221\n",
      "TestAcc            0.823814\n",
      "TrainLength     1657.000000\n",
      "TestLength       900.000000\n",
      "NumWords       29906.000000\n",
      "NumVocab        2540.000000\n",
      "Name: UD_Uyghur-UDT/, dtype: float64\n",
      "TrainAcc           0.946018\n",
      "TestAcc            0.862819\n",
      "TrainLength     1401.000000\n",
      "TestLength       800.000000\n",
      "NumWords       31799.000000\n",
      "NumVocab        4708.000000\n",
      "Name: UD_Vietnamese-VTB/, dtype: float64\n",
      "Dev set not found, no worries! Train set must be good enough.\n",
      "TrainAcc           0.910186\n",
      "TestAcc            0.825016\n",
      "TrainLength      614.000000\n",
      "TestLength       953.000000\n",
      "NumWords       13450.000000\n",
      "NumVocab        2416.000000\n",
      "Name: UD_Welsh-CCG/, dtype: float64\n",
      "TrainAcc           0.912050\n",
      "TestAcc            0.826176\n",
      "TrainLength     1189.000000\n",
      "TestLength       470.000000\n",
      "NumWords       34821.000000\n",
      "NumVocab        3642.000000\n",
      "Name: UD_Wolof-WTB/, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# Iterate through all datasets (~100) and perform HMM tagging, calculate test/training accuracy.\n",
    "\n",
    "#German-HDT takes too much time and chokes up RAM, so get rid of it; there are other German datasets to work with\n",
    "language_dirs.remove(('UD_German-HDT/', 'de_hdt-ud-'))\n",
    "\n",
    "#initialize results DataFrame\n",
    "results = pd.DataFrame(columns=['TrainAcc', 'TestAcc', 'TrainLength', 'TestLength', 'NumWords', 'NumVocab'])\n",
    "\n",
    "#iterate through each dataset and calculating train and test accuracy for HMM based tagging: time-consuming\n",
    "for subdir, file_prefix in language_dirs:\n",
    "    results.loc[subdir] = HMM_tag(UD_PATH, subdir, file_prefix)\n",
    "    print(results.loc[subdir])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results and analysis\n",
    "Of the 100 language datasets we analyzed, we focus on a chosen few (~20) and display the training and test accuracy as a bar chart. We also plot the sizes of the training data in terms of the number of words and the size of the vocabulary to get a sense of which datasets are resource-rich and which aren't. Surprisingly, the performance of HMM is good even for low resources, when seen by the standards of routine NLP tasks like question answering and machine translation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(110, 7)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Convert columns such as NumVocab from decimal into integer (int64) format \n",
    "results = results.astype({'TrainLength':'int64', 'TestLength':'int64', 'NumWords':'int64', 'NumVocab':'int64'})\n",
    "#store dataset names as a separate column too\n",
    "results['index'] = results.index\n",
    "#Take a look at the first few rows of the results dataframe\n",
    "results.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_datasets = ['UD_Czech-PDT/', 'UD_Russian-SynTagRus/', 'UD_Spanish-AnCora/', \n",
    "           'UD_Catalan-AnCora/', 'UD_Romanian-Nonstandard/', 'UD_French-GSD/', 'UD_Estonian-EDT/', \n",
    "           'UD_Latin-ITTB/', 'UD_Korean-Kaist/', 'UD_Polish-PDB/', 'UD_Hindi-HDTB/', 'UD_Italian-ISDT/', \n",
    "           'UD_Arabic-PADT/', 'UD_German-GSD/', 'UD_Norwegian-Bokmaal/', 'UD_Portuguese-Bosque/',\n",
    "           'UD_English-EWT/', 'UD_Dutch-Alpino/', 'UD_Latvian-LVTB/', 'UD_Hebrew-HTB/', 'UD_Finnish-TDT/',\n",
    "           'UD_Japanese-GSD/', 'UD_Croatian-SET/', 'UD_Bulgarian-BTB/', 'UD_Persian-Seraji/', 'UD_Slovenian-SSJ/',\n",
    "           'UD_Urdu-UDTB/', 'UD_Galician-CTG/', 'UD_Naija-NSC/', 'UD_Chinese-GSD/', 'UD_Indonesian-GSD/', \n",
    "           'UD_Ukrainian-IU/', 'UD_Basque-BDT/', 'UD_Slovak-SNK/', 'UD_Danish-DDT/',\n",
    "           'UD_Serbian-SET/', 'UD_Swedish-Talbanken/', 'UD_Lithuanian-ALKSNIS/', \n",
    "           'UD_Irish-IDT/', 'UD_Greek-GDT/', 'UD_Turkish-IMST/', 'UD_Armenian-ArmTDP/', 'UD_Gothic-PROIEL/',\n",
    "           'UD_Coptic-Scriptorium/', 'UD_Afrikaans-AfriBooms/', 'UD_Wolof-WTB/',\n",
    "           'UD_Scottish_Gaelic-ARCOSG/', 'UD_Vietnamese-VTB/', 'UD_Hungarian-Szeged/',\n",
    "           'UD_Uyghur-UDT/', 'UD_Sanskrit-Vedic/', 'UD_North_Sami-Giella/', \n",
    "           'UD_Welsh-CCG/', 'UD_Tamil-TTB/', 'UD_Belarusian-HSE/',\n",
    "           'UD_Marathi-UFAL/', 'UD_Kazakh-KTB/', 'UD_Upper_Sorbian-UFAL/',\n",
    "           'UD_Kurmanji-MG/', 'UD_Buryat-BDT/', 'UD_Livvi-KKPP/'\n",
    "          ]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "#cherry pick rows to plot histogram; I focused on languages with the most amount of training data, \n",
    "#measured in terms of number of words\n",
    "results_shrink = results.loc[plot_datasets]\n",
    "#use regex to keep extract language names from the subdirectories, viz. 'UD_Kurmanji-MG/' --> 'Kurmanji'\n",
    "plot_languages = [re.compile('(?<=\\_)(.*)(?=\\-)').search(x).group() for x in plot_datasets]\n",
    "#set index of rows to plot_languages values, for easy plotting\n",
    "results_shrink.index = plot_languages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualizing this data is a mundane yet important task. I plot here the bar charts of train and test accuracies for the various languages in the `plot_datasets` list. As you can see, there are many European languages among this data. A nice thing to make bar plots less cluttered, I am dividing the languages into 2 sets: European and non-European."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "Europe = ['Czech', 'Russian', 'Spanish', 'Catalan', 'Romanian', 'French', 'Estonian', 'Latin', 'Polish', 'Italian', 'German', 'Norwegian', 'Portuguese', 'English', 'Dutch', 'Latvian', 'Finnish', 'Croatian', 'Bulgarian', 'Slovenian', 'Galician', 'Chinese', 'Basque', 'Slovak', 'Danish', 'Serbian', 'Swedish', 'Lithuanian', 'Irish', 'Greek', 'Gothic', 'Scottish_Gaelic', 'Hungarian', 'North_Sami', 'Welsh', 'Upper_Sorbian', 'Belarusian', 'Livvi']\n",
    "results_shrink_Europe = results_shrink.loc[Europe]\n",
    "results_shrink_NonEurope = results_shrink.drop(index=Europe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "name": "Training Accuracy",
         "type": "bar",
         "x": [
          "Czech",
          "Russian",
          "Spanish",
          "Catalan",
          "Romanian",
          "French",
          "Estonian",
          "Latin",
          "Polish",
          "Italian",
          "German",
          "Norwegian",
          "Portuguese",
          "English",
          "Dutch",
          "Latvian",
          "Finnish",
          "Croatian",
          "Bulgarian",
          "Slovenian",
          "Galician",
          "Chinese",
          "Basque",
          "Slovak",
          "Danish",
          "Serbian",
          "Swedish",
          "Lithuanian",
          "Irish",
          "Greek",
          "Gothic",
          "Scottish_Gaelic",
          "Hungarian",
          "North_Sami",
          "Welsh",
          "Belarusian",
          "Upper_Sorbian",
          "Livvi"
         ],
         "y": [
          0.9867107179711486,
          0.9834681726754413,
          0.9767615956679024,
          0.9758223483789576,
          0.9620922135291666,
          0.9579921182049705,
          0.9744726246003195,
          0.9783610000238653,
          0.9516130169090128,
          0.9135414579137044,
          0.942002153173323,
          0.96482941569597,
          0.9007768965548284,
          0.9504791785789337,
          0.9676089200464496,
          0.9716085323236473,
          0.9846475240609933,
          0.9741909321612642,
          0.9868434808482577,
          0.9877385648789849,
          0.9007231250072082,
          0.941907619935926,
          0.9660132330579871,
          0.9785123557054469,
          0.9669598318136017,
          0.9775489602261256,
          0.963445340367495,
          0.9829342949389912,
          0.9527381005686653,
          0.9650713299585826,
          0.9639482773385094,
          0.9154843763599965,
          0.9704209328782708,
          0.9663795663795663,
          0.9101858736059479,
          0.986897519887693,
          0.9909649776453056,
          0.9737903225806451
         ]
        },
        {
         "name": "Test Accuracy",
         "type": "bar",
         "x": [
          "Czech",
          "Russian",
          "Spanish",
          "Catalan",
          "Romanian",
          "French",
          "Estonian",
          "Latin",
          "Polish",
          "Italian",
          "German",
          "Norwegian",
          "Portuguese",
          "English",
          "Dutch",
          "Latvian",
          "Finnish",
          "Croatian",
          "Bulgarian",
          "Slovenian",
          "Galician",
          "Chinese",
          "Basque",
          "Slovak",
          "Danish",
          "Serbian",
          "Swedish",
          "Lithuanian",
          "Irish",
          "Greek",
          "Gothic",
          "Scottish_Gaelic",
          "Hungarian",
          "North_Sami",
          "Welsh",
          "Belarusian",
          "Upper_Sorbian",
          "Livvi"
         ],
         "y": [
          0.9688989732306564,
          0.9615990061519872,
          0.9533720952254692,
          0.9518761742247962,
          0.9503478509482512,
          0.9275587492717032,
          0.9165911151405258,
          0.9690369278198512,
          0.9133374671509139,
          0.8837084192593921,
          0.8862985928929168,
          0.9223453247013281,
          0.8662525123332724,
          0.8997489739809539,
          0.8967701076630779,
          0.9207778992198394,
          0.9223705386727256,
          0.9279472382522671,
          0.9588527092342916,
          0.9354976202315834,
          0.8779561593084285,
          0.8443223443223443,
          0.9265610896857307,
          0.8511503067484663,
          0.9119026239648808,
          0.9331932405218457,
          0.9176886098302757,
          0.8754379494744606,
          0.8982095162726283,
          0.9236403589086248,
          0.9506766032555403,
          0.8482035928143713,
          0.8300153139356815,
          0.8442557442557442,
          0.8250160884572632,
          0.8285188592456302,
          0.8586956521739131,
          0.6041666666666666
         ]
        }
       ],
       "layout": {
        "barmode": "group",
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "European Languages"
        },
        "yaxis": {
         "dtick": 0.1,
         "tick0": 0,
         "tickmode": "linear"
        }
       }
      },
      "text/html": [
       "<div>\n",
       "        \n",
       "        \n",
       "            <div id=\"7bf4e436-59ec-4bbe-85e0-86efda1512a9\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>\n",
       "            <script type=\"text/javascript\">\n",
       "                require([\"plotly\"], function(Plotly) {\n",
       "                    window.PLOTLYENV=window.PLOTLYENV || {};\n",
       "                    \n",
       "                if (document.getElementById(\"7bf4e436-59ec-4bbe-85e0-86efda1512a9\")) {\n",
       "                    Plotly.newPlot(\n",
       "                        '7bf4e436-59ec-4bbe-85e0-86efda1512a9',\n",
       "                        [{\"name\": \"Training Accuracy\", \"type\": \"bar\", \"x\": [\"Czech\", \"Russian\", \"Spanish\", \"Catalan\", \"Romanian\", \"French\", \"Estonian\", \"Latin\", \"Polish\", \"Italian\", \"German\", \"Norwegian\", \"Portuguese\", \"English\", \"Dutch\", \"Latvian\", \"Finnish\", \"Croatian\", \"Bulgarian\", \"Slovenian\", \"Galician\", \"Chinese\", \"Basque\", \"Slovak\", \"Danish\", \"Serbian\", \"Swedish\", \"Lithuanian\", \"Irish\", \"Greek\", \"Gothic\", \"Scottish_Gaelic\", \"Hungarian\", \"North_Sami\", \"Welsh\", \"Belarusian\", \"Upper_Sorbian\", \"Livvi\"], \"y\": [0.9867107179711486, 0.9834681726754413, 0.9767615956679024, 0.9758223483789576, 0.9620922135291666, 0.9579921182049705, 0.9744726246003195, 0.9783610000238653, 0.9516130169090128, 0.9135414579137044, 0.942002153173323, 0.96482941569597, 0.9007768965548284, 0.9504791785789337, 0.9676089200464496, 0.9716085323236473, 0.9846475240609933, 0.9741909321612642, 0.9868434808482577, 0.9877385648789849, 0.9007231250072082, 0.941907619935926, 0.9660132330579871, 0.9785123557054469, 0.9669598318136017, 0.9775489602261256, 0.963445340367495, 0.9829342949389912, 0.9527381005686653, 0.9650713299585826, 0.9639482773385094, 0.9154843763599965, 0.9704209328782708, 0.9663795663795663, 0.9101858736059479, 0.986897519887693, 0.9909649776453056, 0.9737903225806451]}, {\"name\": \"Test Accuracy\", \"type\": \"bar\", \"x\": [\"Czech\", \"Russian\", \"Spanish\", \"Catalan\", \"Romanian\", \"French\", \"Estonian\", \"Latin\", \"Polish\", \"Italian\", \"German\", \"Norwegian\", \"Portuguese\", \"English\", \"Dutch\", \"Latvian\", \"Finnish\", \"Croatian\", \"Bulgarian\", \"Slovenian\", \"Galician\", \"Chinese\", \"Basque\", \"Slovak\", \"Danish\", \"Serbian\", \"Swedish\", \"Lithuanian\", \"Irish\", \"Greek\", \"Gothic\", \"Scottish_Gaelic\", \"Hungarian\", \"North_Sami\", \"Welsh\", \"Belarusian\", \"Upper_Sorbian\", \"Livvi\"], \"y\": [0.9688989732306564, 0.9615990061519872, 0.9533720952254692, 0.9518761742247962, 0.9503478509482512, 0.9275587492717032, 0.9165911151405258, 0.9690369278198512, 0.9133374671509139, 0.8837084192593921, 0.8862985928929168, 0.9223453247013281, 0.8662525123332724, 0.8997489739809539, 0.8967701076630779, 0.9207778992198394, 0.9223705386727256, 0.9279472382522671, 0.9588527092342916, 0.9354976202315834, 0.8779561593084285, 0.8443223443223443, 0.9265610896857307, 0.8511503067484663, 0.9119026239648808, 0.9331932405218457, 0.9176886098302757, 0.8754379494744606, 0.8982095162726283, 0.9236403589086248, 0.9506766032555403, 0.8482035928143713, 0.8300153139356815, 0.8442557442557442, 0.8250160884572632, 0.8285188592456302, 0.8586956521739131, 0.6041666666666666]}],\n",
       "                        {\"barmode\": \"group\", \"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"title\": {\"text\": \"European Languages\"}, \"yaxis\": {\"dtick\": 0.1, \"tick0\": 0, \"tickmode\": \"linear\"}},\n",
       "                        {\"responsive\": true}\n",
       "                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('7bf4e436-59ec-4bbe-85e0-86efda1512a9');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })\n",
       "                };\n",
       "                });\n",
       "            </script>\n",
       "        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "name": "Training Accuracy",
         "type": "bar",
         "x": [
          "Korean",
          "Hindi",
          "Arabic",
          "Hebrew",
          "Japanese",
          "Persian",
          "Urdu",
          "Naija",
          "Indonesian",
          "Ukrainian",
          "Turkish",
          "Armenian",
          "Coptic",
          "Afrikaans",
          "Wolof",
          "Vietnamese",
          "Uyghur",
          "Sanskrit",
          "Tamil",
          "Marathi",
          "Kazakh",
          "Kurmanji",
          "Buryat"
         ],
         "y": [
          0.9759393373113989,
          0.9526608063351346,
          0.8710673903388971,
          0.7869803625288483,
          0.9718010553711707,
          0.9259013764095514,
          0.9337190903738593,
          0.9708064484303163,
          0.9355956175502967,
          0.9790283759133935,
          0.9453921391519781,
          0.95543024325791,
          0.7493423660408634,
          0.9805049253819383,
          0.9120503745164211,
          0.946017829877358,
          0.8662205622989937,
          0.9704213241616509,
          0.8601602330662782,
          0.8829656862745098,
          0.9394596726483928,
          0.9692765113974232,
          0.9668062200956937
         ]
        },
        {
         "name": "Test Accuracy",
         "type": "bar",
         "x": [
          "Korean",
          "Hindi",
          "Arabic",
          "Hebrew",
          "Japanese",
          "Persian",
          "Urdu",
          "Naija",
          "Indonesian",
          "Ukrainian",
          "Turkish",
          "Armenian",
          "Coptic",
          "Afrikaans",
          "Wolof",
          "Vietnamese",
          "Uyghur",
          "Sanskrit",
          "Tamil",
          "Marathi",
          "Kazakh",
          "Kurmanji",
          "Buryat"
         ],
         "y": [
          0.800535852781499,
          0.9249788314987298,
          0.8473173160712063,
          0.7399748959503204,
          0.9282645388982661,
          0.9271802505892569,
          0.9045657166013779,
          0.9412043248362403,
          0.8612903225806452,
          0.9031373921156987,
          0.9107665301345816,
          0.8585300837776085,
          0.7262267092272289,
          0.9230998509687034,
          0.8261762509335325,
          0.862818904224174,
          0.8238141335914811,
          0.8870967741935484,
          0.7027027027027027,
          0.7790178571428571,
          0.7074954296160878,
          0.8471074380165289,
          0.8104575163398693
         ]
        }
       ],
       "layout": {
        "barmode": "group",
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Non-European Languages"
        },
        "yaxis": {
         "dtick": 0.1,
         "tick0": 0,
         "tickmode": "linear"
        }
       }
      },
      "text/html": [
       "<div>\n",
       "        \n",
       "        \n",
       "            <div id=\"9b97b4ff-c98d-45fb-9f67-65a0ccd97adf\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>\n",
       "            <script type=\"text/javascript\">\n",
       "                require([\"plotly\"], function(Plotly) {\n",
       "                    window.PLOTLYENV=window.PLOTLYENV || {};\n",
       "                    \n",
       "                if (document.getElementById(\"9b97b4ff-c98d-45fb-9f67-65a0ccd97adf\")) {\n",
       "                    Plotly.newPlot(\n",
       "                        '9b97b4ff-c98d-45fb-9f67-65a0ccd97adf',\n",
       "                        [{\"name\": \"Training Accuracy\", \"type\": \"bar\", \"x\": [\"Korean\", \"Hindi\", \"Arabic\", \"Hebrew\", \"Japanese\", \"Persian\", \"Urdu\", \"Naija\", \"Indonesian\", \"Ukrainian\", \"Turkish\", \"Armenian\", \"Coptic\", \"Afrikaans\", \"Wolof\", \"Vietnamese\", \"Uyghur\", \"Sanskrit\", \"Tamil\", \"Marathi\", \"Kazakh\", \"Kurmanji\", \"Buryat\"], \"y\": [0.9759393373113989, 0.9526608063351346, 0.8710673903388971, 0.7869803625288483, 0.9718010553711707, 0.9259013764095514, 0.9337190903738593, 0.9708064484303163, 0.9355956175502967, 0.9790283759133935, 0.9453921391519781, 0.95543024325791, 0.7493423660408634, 0.9805049253819383, 0.9120503745164211, 0.946017829877358, 0.8662205622989937, 0.9704213241616509, 0.8601602330662782, 0.8829656862745098, 0.9394596726483928, 0.9692765113974232, 0.9668062200956937]}, {\"name\": \"Test Accuracy\", \"type\": \"bar\", \"x\": [\"Korean\", \"Hindi\", \"Arabic\", \"Hebrew\", \"Japanese\", \"Persian\", \"Urdu\", \"Naija\", \"Indonesian\", \"Ukrainian\", \"Turkish\", \"Armenian\", \"Coptic\", \"Afrikaans\", \"Wolof\", \"Vietnamese\", \"Uyghur\", \"Sanskrit\", \"Tamil\", \"Marathi\", \"Kazakh\", \"Kurmanji\", \"Buryat\"], \"y\": [0.800535852781499, 0.9249788314987298, 0.8473173160712063, 0.7399748959503204, 0.9282645388982661, 0.9271802505892569, 0.9045657166013779, 0.9412043248362403, 0.8612903225806452, 0.9031373921156987, 0.9107665301345816, 0.8585300837776085, 0.7262267092272289, 0.9230998509687034, 0.8261762509335325, 0.862818904224174, 0.8238141335914811, 0.8870967741935484, 0.7027027027027027, 0.7790178571428571, 0.7074954296160878, 0.8471074380165289, 0.8104575163398693]}],\n",
       "                        {\"barmode\": \"group\", \"template\": {\"data\": {\"bar\": [{\"error_x\": {\"color\": \"#2a3f5f\"}, \"error_y\": {\"color\": \"#2a3f5f\"}, \"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"bar\"}], \"barpolar\": [{\"marker\": {\"line\": {\"color\": \"#E5ECF6\", \"width\": 0.5}}, \"type\": \"barpolar\"}], \"carpet\": [{\"aaxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"baxis\": {\"endlinecolor\": \"#2a3f5f\", \"gridcolor\": \"white\", \"linecolor\": \"white\", \"minorgridcolor\": \"white\", \"startlinecolor\": \"#2a3f5f\"}, \"type\": \"carpet\"}], \"choropleth\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"choropleth\"}], \"contour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"contour\"}], \"contourcarpet\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"contourcarpet\"}], \"heatmap\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmap\"}], \"heatmapgl\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"heatmapgl\"}], \"histogram\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"histogram\"}], \"histogram2d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2d\"}], \"histogram2dcontour\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"histogram2dcontour\"}], \"mesh3d\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"type\": \"mesh3d\"}], \"parcoords\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"parcoords\"}], \"pie\": [{\"automargin\": true, \"type\": \"pie\"}], \"scatter\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter\"}], \"scatter3d\": [{\"line\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatter3d\"}], \"scattercarpet\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattercarpet\"}], \"scattergeo\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergeo\"}], \"scattergl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattergl\"}], \"scattermapbox\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scattermapbox\"}], \"scatterpolar\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolar\"}], \"scatterpolargl\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterpolargl\"}], \"scatterternary\": [{\"marker\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"type\": \"scatterternary\"}], \"surface\": [{\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}, \"colorscale\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"type\": \"surface\"}], \"table\": [{\"cells\": {\"fill\": {\"color\": \"#EBF0F8\"}, \"line\": {\"color\": \"white\"}}, \"header\": {\"fill\": {\"color\": \"#C8D4E3\"}, \"line\": {\"color\": \"white\"}}, \"type\": \"table\"}]}, \"layout\": {\"annotationdefaults\": {\"arrowcolor\": \"#2a3f5f\", \"arrowhead\": 0, \"arrowwidth\": 1}, \"coloraxis\": {\"colorbar\": {\"outlinewidth\": 0, \"ticks\": \"\"}}, \"colorscale\": {\"diverging\": [[0, \"#8e0152\"], [0.1, \"#c51b7d\"], [0.2, \"#de77ae\"], [0.3, \"#f1b6da\"], [0.4, \"#fde0ef\"], [0.5, \"#f7f7f7\"], [0.6, \"#e6f5d0\"], [0.7, \"#b8e186\"], [0.8, \"#7fbc41\"], [0.9, \"#4d9221\"], [1, \"#276419\"]], \"sequential\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]], \"sequentialminus\": [[0.0, \"#0d0887\"], [0.1111111111111111, \"#46039f\"], [0.2222222222222222, \"#7201a8\"], [0.3333333333333333, \"#9c179e\"], [0.4444444444444444, \"#bd3786\"], [0.5555555555555556, \"#d8576b\"], [0.6666666666666666, \"#ed7953\"], [0.7777777777777778, \"#fb9f3a\"], [0.8888888888888888, \"#fdca26\"], [1.0, \"#f0f921\"]]}, \"colorway\": [\"#636efa\", \"#EF553B\", \"#00cc96\", \"#ab63fa\", \"#FFA15A\", \"#19d3f3\", \"#FF6692\", \"#B6E880\", \"#FF97FF\", \"#FECB52\"], \"font\": {\"color\": \"#2a3f5f\"}, \"geo\": {\"bgcolor\": \"white\", \"lakecolor\": \"white\", \"landcolor\": \"#E5ECF6\", \"showlakes\": true, \"showland\": true, \"subunitcolor\": \"white\"}, \"hoverlabel\": {\"align\": \"left\"}, \"hovermode\": \"closest\", \"mapbox\": {\"style\": \"light\"}, \"paper_bgcolor\": \"white\", \"plot_bgcolor\": \"#E5ECF6\", \"polar\": {\"angularaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"radialaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"scene\": {\"xaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"yaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}, \"zaxis\": {\"backgroundcolor\": \"#E5ECF6\", \"gridcolor\": \"white\", \"gridwidth\": 2, \"linecolor\": \"white\", \"showbackground\": true, \"ticks\": \"\", \"zerolinecolor\": \"white\"}}, \"shapedefaults\": {\"line\": {\"color\": \"#2a3f5f\"}}, \"ternary\": {\"aaxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"baxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}, \"bgcolor\": \"#E5ECF6\", \"caxis\": {\"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\"}}, \"title\": {\"x\": 0.05}, \"xaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}, \"yaxis\": {\"automargin\": true, \"gridcolor\": \"white\", \"linecolor\": \"white\", \"ticks\": \"\", \"title\": {\"standoff\": 15}, \"zerolinecolor\": \"white\", \"zerolinewidth\": 2}}}, \"title\": {\"text\": \"Non-European Languages\"}, \"yaxis\": {\"dtick\": 0.1, \"tick0\": 0, \"tickmode\": \"linear\"}},\n",
       "                        {\"responsive\": true}\n",
       "                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('9b97b4ff-c98d-45fb-9f67-65a0ccd97adf');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })\n",
       "                };\n",
       "                });\n",
       "            </script>\n",
       "        </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import plotly.graph_objects as go\n",
    "\n",
    "fig1 = go.Figure(data=[\n",
    "    go.Bar(name='Training Accuracy', x=results_shrink_Europe.index, y=results_shrink_Europe['TrainAcc']),\n",
    "    go.Bar(name='Test Accuracy', x=results_shrink_Europe.index, y=results_shrink_Europe['TestAcc'])\n",
    "])\n",
    "# Change the bar mode\n",
    "fig1.update_layout(barmode='group', yaxis = dict(tickmode = 'linear',tick0 = 0,dtick = 0.1), title='European Languages')\n",
    "fig1.show()\n",
    "\n",
    "fig2 = go.Figure(data=[\n",
    "    go.Bar(name='Training Accuracy', x=results_shrink_NonEurope.index, y=results_shrink_NonEurope['TrainAcc']),\n",
    "    go.Bar(name='Test Accuracy', x=results_shrink_NonEurope.index, y=results_shrink_NonEurope['TestAcc'])\n",
    "])\n",
    "# Change the bar mode\n",
    "fig2.update_layout(barmode='group', yaxis = dict(tickmode = 'linear',tick0 = 0,dtick = 0.1), title='Non-European Languages')\n",
    "fig2.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusions:\n",
    "There are some languages that don't do that well, for example, Livvi. The languages have been roughly sorted according to the total number of words (allowing repetitions) encountered during HMM training. This is very likely due to lack of training data. Livvi (KKPP), figuring at the bottom of the list, has only 106 training sentences, whereas Czech (PDT), appearing at the top of the European list, has 68496 training sentences. Assuming that each sentence in Livvi packs roughly as much info as a sentence in Czech, one should expect the Czech HMM to do better. The interesting point, however, is that the treshold to hit a 85%+ test accuracy isn't that high. \n",
    "\n",
    "On the other hand, one shouldn't read too much into the finer differences in the results shown above, as the UD data has deficiencies such as incomplete annotation. In some cases such as Upper Sorbian, the test data file had 30 times as many sentences as the training data file (dev data didn't exist), which is outright absurd! In most of such cases, I swapped the train and test data to make things smoother. Here are the languages where this was done:\n",
    "\n",
    "1. Upper Sorbian\n",
    "2. Livvi\n",
    "3. Buryat\n",
    "4. Kurmanji\n",
    "5. Kazakh\n",
    "\n",
    "Some datasets were discarded (e.g., Telugu) because of missing lemmatizations, etc.\n",
    "\n",
    "#### As a general rule of thumb, investigate the dataset when you find really low accuracies!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}