Ejercicio_4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ejercicio 4\n",
    "\n",
    "Usaremos el script eval.py con los modelos ya entrenados guardados en la carpeta trainedModels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run tagging/scripts/eval.py --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls trainedModels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation time 3.951965093612671\n",
      "\n",
      "Accuracy: 91.69% / 95.01% / 61.68% (total / known / unk)\n",
      "\n",
      "g \\ m\tsp000\tnc0s000\tda0000\taq0000\tfc\tnc0p000\trg\tnp00000\tfp\tcc\n",
      "sp000\t14.26\t0.01\t-\t0.05\t-\t-\t0.01\t0.00\t-\t-\t\n",
      "nc0s000\t0.00\t11.89\t0.01\t0.52\t-\t0.02\t0.02\t0.09\t-\t-\t\n",
      "da0000\t-\t0.11\t9.50\t0.01\t-\t0.01\t-\t0.01\t-\t-\t\n",
      "aq0000\t0.00\t0.51\t-\t6.43\t-\t0.17\t0.01\t0.06\t-\t-\t\n",
      "fc\t-\t-\t-\t-\t5.85\t-\t-\t-\t-\t-\t\n",
      "nc0p000\t-\t0.35\t-\t0.46\t-\t4.61\t-\t0.04\t-\t-\t\n",
      "rg\t0.02\t0.04\t0.00\t0.39\t-\t0.00\t3.11\t0.03\t-\t0.02\t\n",
      "np00000\t-\t0.22\t-\t0.08\t-\t0.01\t-\t3.24\t-\t0.00\t\n",
      "fp\t-\t-\t-\t-\t-\t-\t-\t-\t3.55\t-\t\n",
      "cc\t0.00\t0.00\t-\t0.01\t-\t0.00\t0.05\t0.00\t-\t3.34\t\n"
     ]
    }
   ],
   "source": [
    "%run tagging/scripts/eval.py -c ancora-3.0.1es -i trainedModels/lrModel -m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive bayes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation time 207.7183539867401\n",
      "\n",
      "Accuracy: 84.28% / 88.07% / 49.99% (total / known / unk)\n",
      "\n",
      "g \\ m\tsp000\tnc0s000\tda0000\taq0000\tfc\tnc0p000\trg\tnp00000\tfp\tcc\n",
      "sp000\t14.31\t-\t0.01\t0.00\t0.00\t0.00\t0.00\t0.00\t-\t-\t\n",
      "nc0s000\t0.05\t12.05\t0.07\t0.09\t0.02\t0.01\t0.00\t0.29\t-\t0.00\t\n",
      "da0000\t0.00\t0.14\t9.52\t0.00\t0.00\t0.00\t-\t0.01\t-\t-\t\n",
      "aq0000\t0.22\t0.88\t0.19\t5.11\t0.14\t0.23\t0.01\t0.40\t0.00\t0.01\t\n",
      "fc\t0.00\t-\t-\t-\t5.85\t-\t-\t0.00\t-\t-\t\n",
      "nc0p000\t0.06\t0.28\t0.17\t0.07\t0.02\t4.64\t0.00\t0.26\t-\t0.00\t\n",
      "rg\t0.20\t0.17\t0.08\t0.14\t0.04\t0.01\t2.77\t0.23\t-\t0.02\t\n",
      "np00000\t0.03\t0.27\t0.05\t0.00\t0.00\t0.01\t0.00\t3.21\t-\t-\t\n",
      "fp\t0.00\t-\t-\t-\t-\t-\t-\t-\t3.55\t-\t\n",
      "cc\t0.01\t0.00\t0.00\t-\t-\t0.00\t0.05\t0.05\t-\t3.29\t\n"
     ]
    }
   ],
   "source": [
    "%run tagging/scripts/eval.py -c ancora-3.0.1es -i trainedModels/mnbModel -m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vector machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation time 4.85071587562561\n",
      "\n",
      "Accuracy: 94.11% / 97.57% / 62.76% (total / known / unk)\n",
      "\n",
      "g \\ m\tsp000\tnc0s000\tda0000\taq0000\tfc\tnc0p000\trg\tnp00000\tfp\tcc\n",
      "sp000\t14.30\t0.00\t-\t0.03\t-\t-\t0.00\t-\t-\t-\t\n",
      "nc0s000\t0.00\t12.09\t0.01\t0.33\t-\t0.02\t0.02\t0.08\t-\t0.00\t\n",
      "da0000\t-\t0.08\t9.53\t-\t-\t0.00\t-\t0.00\t-\t-\t\n",
      "aq0000\t0.00\t0.34\t-\t6.61\t-\t0.15\t0.01\t0.05\t-\t-\t\n",
      "fc\t-\t-\t-\t-\t5.85\t-\t-\t-\t-\t-\t\n",
      "nc0p000\t-\t0.25\t-\t0.31\t-\t4.90\t-\t0.04\t-\t-\t\n",
      "rg\t0.02\t0.02\t0.00\t0.21\t-\t0.01\t3.35\t0.01\t-\t0.02\t\n",
      "np00000\t0.00\t0.21\t-\t0.08\t-\t0.01\t-\t3.26\t-\t0.00\t\n",
      "fp\t-\t-\t-\t-\t-\t-\t-\t-\t3.55\t-\t\n",
      "cc\t0.00\t-\t-\t0.01\t-\t-\t0.05\t0.00\t-\t3.34\t\n"
     ]
    }
   ],
   "source": [
    "%run tagging/scripts/eval.py -c ancora-3.0.1es -i trainedModels/svmModel -m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Análisis de Errores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "El script de evaluación eval.py calcula variables útiles para hacer análisis de errores:\n",
    "\n",
    "- **error_count:** matriz de confusión completa.\n",
    "- **error_sents:** matriz de índices de las oraciones en las que ocurre cada error.\n",
    "\n",
    "Evaluamos el tagger con mejor performance (SVM) para hacer el análisis de errores."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creamos un Pandas DF con las combinaciones de tags, el error count, el % y los indices de las oraciones donde ocurre ese error.\n",
    "\n",
    "Imprimimos el las primeras filas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>correct</th>\n",
       "      <th>predicted</th>\n",
       "      <th>n_errors</th>\n",
       "      <th>total</th>\n",
       "      <th>ind_error_sents</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>241</th>\n",
       "      <td>aq0000</td>\n",
       "      <td>nc0s000</td>\n",
       "      <td>319</td>\n",
       "      <td>0.34</td>\n",
       "      <td>[32, 50, 70, 77, 112, 122, 127, 129, 135, 164,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>nc0s000</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>312</td>\n",
       "      <td>0.33</td>\n",
       "      <td>[23, 26, 55, 60, 105, 106, 122, 132, 134, 137,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>723</th>\n",
       "      <td>nc0p000</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>290</td>\n",
       "      <td>0.31</td>\n",
       "      <td>[141, 148, 151, 180, 231, 236, 241, 261, 266, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>721</th>\n",
       "      <td>nc0p000</td>\n",
       "      <td>nc0s000</td>\n",
       "      <td>241</td>\n",
       "      <td>0.25</td>\n",
       "      <td>[11, 44, 50, 82, 85, 113, 192, 252, 261, 265, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1291</th>\n",
       "      <td>cs</td>\n",
       "      <td>pr000000</td>\n",
       "      <td>225</td>\n",
       "      <td>0.24</td>\n",
       "      <td>[2, 14, 32, 72, 106, 148, 151, 175, 185, 193, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>321</th>\n",
       "      <td>np00000</td>\n",
       "      <td>nc0s000</td>\n",
       "      <td>202</td>\n",
       "      <td>0.21</td>\n",
       "      <td>[6, 9, 12, 29, 36, 47, 49, 51, 55, 59, 61, 68,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1683</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>196</td>\n",
       "      <td>0.21</td>\n",
       "      <td>[31, 43, 135, 154, 177, 212, 221, 286, 326, 44...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248</th>\n",
       "      <td>aq0000</td>\n",
       "      <td>nc0p000</td>\n",
       "      <td>141</td>\n",
       "      <td>0.15</td>\n",
       "      <td>[42, 71, 103, 106, 126, 216, 223, 259, 342, 37...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3134</th>\n",
       "      <td>vmii000</td>\n",
       "      <td>vmip000</td>\n",
       "      <td>124</td>\n",
       "      <td>0.13</td>\n",
       "      <td>[359, 415, 473, 525, 689, 797, 798, 850, 855, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2403</th>\n",
       "      <td>vmg0000</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>112</td>\n",
       "      <td>0.12</td>\n",
       "      <td>[69, 111, 149, 226, 336, 372, 375, 420, 486, 5...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      correct predicted  n_errors  total  \\\n",
       "241    aq0000   nc0s000       319   0.34   \n",
       "82    nc0s000    aq0000       312   0.33   \n",
       "723   nc0p000    aq0000       290   0.31   \n",
       "721   nc0p000   nc0s000       241   0.25   \n",
       "1291       cs  pr000000       225   0.24   \n",
       "321   np00000   nc0s000       202   0.21   \n",
       "1683       rg    aq0000       196   0.21   \n",
       "248    aq0000   nc0p000       141   0.15   \n",
       "3134  vmii000   vmip000       124   0.13   \n",
       "2403  vmg0000    aq0000       112   0.12   \n",
       "\n",
       "                                        ind_error_sents  \n",
       "241   [32, 50, 70, 77, 112, 122, 127, 129, 135, 164,...  \n",
       "82    [23, 26, 55, 60, 105, 106, 122, 132, 134, 137,...  \n",
       "723   [141, 148, 151, 180, 231, 236, 241, 261, 266, ...  \n",
       "721   [11, 44, 50, 82, 85, 113, 192, 252, 261, 265, ...  \n",
       "1291  [2, 14, 32, 72, 106, 148, 151, 175, 185, 193, ...  \n",
       "321   [6, 9, 12, 29, 36, 47, 49, 51, 55, 59, 61, 68,...  \n",
       "1683  [31, 43, 135, 154, 177, 212, 221, 286, 326, 44...  \n",
       "248   [42, 71, 103, 106, 126, 216, 223, 259, 342, 37...  \n",
       "3134  [359, 415, 473, 525, 689, 797, 798, 850, 855, ...  \n",
       "2403  [69, 111, 149, 226, 336, 372, 375, 420, 486, 5...  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "columns = []\n",
    "\n",
    "tagsVocab = error_count.keys()\n",
    "\n",
    "for tag in tagsVocab:\n",
    "    firstTag = tag\n",
    "    for tag in tagsVocab:\n",
    "        if tag == firstTag:\n",
    "            continue            \n",
    "        columns.append([firstTag, tag])\n",
    "errors = pd.DataFrame(columns=['correct', 'predicted', 'n_errors', 'total', 'ind_error_sents'])\n",
    "errors['correct'] = [col[0] for col in columns]\n",
    "errors['predicted'] = [col[1] for col in columns]\n",
    "n_errors = []\n",
    "for tag in tagsVocab:\n",
    "    firstTag = tag\n",
    "    for tag in tagsVocab:\n",
    "        if tag == firstTag:\n",
    "            continue\n",
    "        n_errors.append(error_count[firstTag][tag])  \n",
    "errors['n_errors'] = n_errors\n",
    "ind_error_sents = []\n",
    "for tag in tagsVocab:\n",
    "    firstTag = tag\n",
    "    for tag in tagsVocab:\n",
    "        if tag == firstTag:\n",
    "            continue\n",
    "        ind_error_sents.append(list(sorted(error_sents[firstTag][tag]))) \n",
    "\n",
    "errors['ind_error_sents'] = ind_error_sents\n",
    "errors['total'] = round((errors['n_errors'] / total) * 100, 2)\n",
    "errors.sort_values(by=['n_errors'], ascending = False, inplace=True)\n",
    "errors.iloc[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recordamos los significados de los tag más frecuentes\n",
    "\n",
    "| Tag    | Meaning  |\n",
    "|------- | --------                    |\n",
    "|sp000   |  Preposición\n",
    "|nc0s000 |  Sustantivo común singular\n",
    "|da0000  |  Artículo\n",
    "|aq0000  |  Adjetivo Descriptivo\n",
    "|fc      |  Coma\n",
    "|np00000 |  Nombre Propio (sustantivo)\n",
    "|nc0p000 |  Sustantivo común plural\n",
    "|fp      |  Punto\n",
    "|rg      |  Adverbio\n",
    "|cc      |  Conjunción\n",
    "|cs      |  Conjunción subordinante (que, porque, como)\n",
    "|pr      |  Pronombre relativo (como, donde, cuando)\n",
    "\n",
    "\n",
    "Vemos que los errores más frecuentes se dan cuando:\n",
    "\n",
    " - Un adjetivo se etiqueta como un sustantivo común singular\n",
    " - Un sustantivo común singular se etiqueta como un adjetivo\n",
    " - Un sustantivo común plural se etiqueta como un adjetivo\n",
    " - Un sustantivo común plural se etiqueta como singular\n",
    " - Una conjunción subordinante se etiqueta como  un pronombre relativo\n",
    " - Un adverbio se etiqueta como un adjetivo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Caso más sencillo\n",
    "\n",
    "El caso más sencillo de arreglar con features es el de **sustantivo plural como plural**.\n",
    "Veamos una oración."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Este', 'dd0000'),\n",
       " ('documento', 'nc0s000'),\n",
       " (',', 'fc'),\n",
       " ('que', 'pr000000'),\n",
       " ('ha', 'vaip000'),\n",
       " ('tomado', 'vmp0000'),\n",
       " ('como', 'sp000'),\n",
       " ('referencia', 'nc0s000'),\n",
       " ('la', 'da0000'),\n",
       " ('Comunidad_de_Madrid', 'np00000')]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent = sents[11]\n",
    "words, tags = zip(*sent)  # separar las palabras de los tags\n",
    "sent[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>correct</th>\n",
       "      <th>predicted</th>\n",
       "      <th>word</th>\n",
       "      <th>is_correct</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>nc0p000</td>\n",
       "      <td>nc0p000</td>\n",
       "      <td>retornos</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>nc0p000</td>\n",
       "      <td>nc0p000</td>\n",
       "      <td>inversiones</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>nc0p000</td>\n",
       "      <td>nc0s000</td>\n",
       "      <td>innovaciones</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>nc0p000</td>\n",
       "      <td>nc0p000</td>\n",
       "      <td>beneficios</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    correct predicted          word  is_correct\n",
       "15  nc0p000   nc0p000      retornos        True\n",
       "19  nc0p000   nc0p000   inversiones        True\n",
       "40  nc0p000   nc0s000  innovaciones       False\n",
       "44  nc0p000   nc0p000    beneficios        True"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(columns = ['correct', 'predicted', 'word'])\n",
    "df['correct'], df['predicted'], df['word'] = tags,list(model.tag(words)), words\n",
    "df['is_correct'] = df['correct'] == df['predicted']\n",
    "df.loc[df['correct'] == 'nc0p000']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vemos que taggeó mal la palabra 'innovaciones'. Chequeamos si es una palabra desconocida."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.unknown('innovaciones')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Al ser una palabra desconocida solo se fija en las features que habíamos propuesto al principio: \n",
    "\n",
    "    - la palabra actual en minúsculas.\n",
    "    - si la palabra actual empieza en mayúsculas.\n",
    "    - si la palabra actual está en mayúsculas.\n",
    "    - si la palabra actual es un número.\n",
    "    - mismos features para la palabra anterior y para la siguiente.\n",
    "\n",
    "\n",
    "Podemos agregar como feature si la palabra termina en 's' para aprovechar el caso de los plurales.\n",
    "\n",
    "Modificamos creamos classfierV2.py donde agregamos como feature si la palabra termina con 's'.\n",
    "\n",
    "Entrenamos nuevamente (el resultado se guardó en trainedModels/svmModelv2) y evaluamos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training classifier with: svm  model\n",
      "Extracting features V2\n",
      "\n",
      "Executing pipeline\n",
      "Training time 144.99225401878357\n",
      "Saving...\n",
      "All done!\n"
     ]
    }
   ],
   "source": [
    "%run tagging/scripts/train.py -c ancora-3.0.1es -m classifier -v 2  -t svm -o trainedModels/svmModelv2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation time 3.446685314178467\n",
      "\n",
      "Accuracy: 94.46% / 97.57% / 66.31% (total / known / unk)\n"
     ]
    }
   ],
   "source": [
    "%run tagging/scripts/eval.py -c ancora-3.0.1es -i trainedModels/svmModelv2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Como vemos aumentó un poco la accuracy en palabras no vistas (62.76% vs 66.31%)\n",
    "\n",
    "Chequeamos si la categoría que estábamos revisando mejoró (valor previo 0.25).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.006331919204710948\n"
     ]
    }
   ],
   "source": [
    "print((error_count['nc0p000']['nc0s000'] / total) * 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mejoró notablemente con la adición de esa feature."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adverbios como adjetivos\n",
    "\n",
    "Otro error que podría resolverse es el de **adverbios etiquetados como adjetivos**.\n",
    "Hacemos una tabla con todos los casos donde esto ocurre y vemos si hay alguna regularidad en la palabra mal etiquetada que se pueda utilizar como feature (imprimimos los primeros 20)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>correct</th>\n",
       "      <th>predicted</th>\n",
       "      <th>word</th>\n",
       "      <th>is_correct</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>un_poco_menos_de</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>cerca_del</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>judicialmente</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>134</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>por_igual</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>170</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>detalladamente</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>rg</td>\n",
       "      <td>pi000000</td>\n",
       "      <td>nada</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>217</th>\n",
       "      <td>rg</td>\n",
       "      <td>nc0p000</td>\n",
       "      <td>sin_lugar_a_dudas</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>244</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>medio</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>288</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>al_alcance_de_la_mano</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>319</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>penosamente</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>332</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>más_de</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>356</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>Más_de</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>484</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>cuando_mucho</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>537</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>sencillamente</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>548</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>seguro</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>580</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>a_la_ligera</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>608</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>de_tarde_en_tarde</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>631</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>desordenadamente</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>649</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>calladamente</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>668</th>\n",
       "      <td>rg</td>\n",
       "      <td>aq0000</td>\n",
       "      <td>de_tanto_en_tanto</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    correct predicted                   word  is_correct\n",
       "23       rg    aq0000       un_poco_menos_de       False\n",
       "34       rg    aq0000              cerca_del       False\n",
       "94       rg    aq0000          judicialmente       False\n",
       "134      rg    aq0000              por_igual       False\n",
       "170      rg    aq0000         detalladamente       False\n",
       "195      rg  pi000000                   nada       False\n",
       "217      rg   nc0p000      sin_lugar_a_dudas       False\n",
       "244      rg    aq0000                  medio       False\n",
       "288      rg    aq0000  al_alcance_de_la_mano       False\n",
       "319      rg    aq0000            penosamente       False\n",
       "332      rg    aq0000                 más_de       False\n",
       "356      rg    aq0000                 Más_de       False\n",
       "484      rg    aq0000           cuando_mucho       False\n",
       "537      rg    aq0000          sencillamente       False\n",
       "548      rg    aq0000                 seguro       False\n",
       "580      rg    aq0000            a_la_ligera       False\n",
       "608      rg    aq0000      de_tarde_en_tarde       False\n",
       "631      rg    aq0000       desordenadamente       False\n",
       "649      rg    aq0000           calladamente       False\n",
       "668      rg    aq0000      de_tanto_en_tanto       False"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "error_sentences = errors.loc[(errors['correct'] == 'rg') & (errors['predicted'] == 'aq0000'), 'ind_error_sents']\n",
    "error_sentences = [list(error_sentences)[0][i] for i in range(len(list(error_sentences)[0]))]\n",
    "sent_w_errors = []\n",
    "\n",
    "for i in error_sentences:\n",
    "    sent_w_errors.extend(sents[i])\n",
    "    \n",
    "words, tags = zip(*sent_w_errors)  # separar las palabras de los tags\n",
    "df = pd.DataFrame(columns = ['correct', 'predicted', 'word'])\n",
    "df['correct'], df['predicted'], df['word'] = tags,list(model.tag(words)), words\n",
    "df['is_correct'] = df['correct'] == df['predicted']\n",
    "df.loc[(df['correct'] == 'rg') & (df['is_correct']==False)][:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "True\n",
      "True\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "print(model.unknown('judicialmente'))\n",
    "print(model.unknown('detalladamente'))\n",
    "print(model.unknown('penosamente'))\n",
    "print(model.unknown('sencillamente'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pareciera que la terminación '-mente', muy común en los adverbios, podría servir en este problema. Lo agregamos como features en classifierV3.py\n",
    "\n",
    "Evaluamos nuevamente con classifierV3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training classifier with: svm  model\n",
      "Extracting features V3\n",
      "\n",
      "Executing pipeline\n",
      "Training time 167.7551519870758\n",
      "Saving...\n",
      "All done!\n"
     ]
    }
   ],
   "source": [
    "%run tagging/scripts/train.py -c ancora-3.0.1es -m classifier -v 3  -t svm -o trainedModels/svmModelv3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation time 4.028918981552124\n",
      "\n",
      "Accuracy: 94.57% / 97.56% / 67.42% (total / known / unk)\n"
     ]
    }
   ],
   "source": [
    "%run tagging/scripts/eval.py -c ancora-3.0.1es -i trainedModels/svmModelv3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.09708942780556787\n"
     ]
    }
   ],
   "source": [
    "print((error_count['rg']['aq0000'] / total) * 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vemos que la accuracy en palabras desconocidas subió de 66.31 a 67.42. Mientras que el error en adverbios etiquetados como adjetivos bajó de 0.21 a 0.09."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Errores restantes\n",
    "\n",
    "Los errores restantes son:\n",
    "\n",
    " - Un adjetivo se etiqueta como un sustantivo común singular y viceversa\n",
    " - Una conjunción subordinante se etiqueta como  un pronombre relativo\n",
    " \n",
    "Veamos si esos casos se dan en palabras conocidas o desconocidas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}