Intent Classification with Small Transformers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intent Classification with Small Transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In recent years, the transformer has been the go-to model in Natural Language Processing for almost all NLP tasks. Pretrained transformer models are easily available, but they are often large and fairly slow, which can make them hard to deploy. That's why a variety of smaller transformer models have emerged, which aim to compete with their bigger rivals, but at a lower computational cost. In this notebook we'll explore some of these smaller transformers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intent classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll evaluate a range of pretrained transformer models on intent classification. Intent classification is a crucial task in the development of chatbots, where the computer needs to figure out how to respond to the input of a user. An interesting example dataset is `banking77`, which is available in the `datasets` library. It contains 13,083 customer service queries from the banking domain: 10,003 for training and 3080 for testing. Since there are 77 intents, it's a fairly challenging dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Couldn't find file locally at banking77/banking77.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.4.1/datasets/banking77/banking77.py.\n",
      "The file was picked from the master branch on github instead at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/banking77/banking77.py.\n",
      "Using custom data configuration default\n",
      "Reusing dataset banking77 (/home/yves/.cache/huggingface/datasets/banking77/default/1.1.0/17ffc2ed47c2ed928bee64127ff1dbc97204cb974c2f980becae7c864007aed9)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['text', 'label'],\n",
       "        num_rows: 10003\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['text', 'label'],\n",
       "        num_rows: 3080\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "dataset = load_dataset('banking77')\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The intents cover a wide variety of banking topics, from `pin_blocked` to `declined_transfer` and `terminate_account`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': Value(dtype='string', id=None),\n",
       " 'label': ClassLabel(num_classes=77, names=['activate_my_card', 'age_limit', 'apple_pay_or_google_pay', 'atm_support', 'automatic_top_up', 'balance_not_updated_after_bank_transfer', 'balance_not_updated_after_cheque_or_cash_deposit', 'beneficiary_not_allowed', 'cancel_transfer', 'card_about_to_expire', 'card_acceptance', 'card_arrival', 'card_delivery_estimate', 'card_linking', 'card_not_working', 'card_payment_fee_charged', 'card_payment_not_recognised', 'card_payment_wrong_exchange_rate', 'card_swallowed', 'cash_withdrawal_charge', 'cash_withdrawal_not_recognised', 'change_pin', 'compromised_card', 'contactless_not_working', 'country_support', 'declined_card_payment', 'declined_cash_withdrawal', 'declined_transfer', 'direct_debit_payment_not_recognised', 'disposable_card_limits', 'edit_personal_details', 'exchange_charge', 'exchange_rate', 'exchange_via_app', 'extra_charge_on_statement', 'failed_transfer', 'fiat_currency_support', 'get_disposable_virtual_card', 'get_physical_card', 'getting_spare_card', 'getting_virtual_card', 'lost_or_stolen_card', 'lost_or_stolen_phone', 'order_physical_card', 'passcode_forgotten', 'pending_card_payment', 'pending_cash_withdrawal', 'pending_top_up', 'pending_transfer', 'pin_blocked', 'receiving_money', 'Refund_not_showing_up', 'request_refund', 'reverted_card_payment?', 'supported_cards_and_currencies', 'terminate_account', 'top_up_by_bank_transfer_charge', 'top_up_by_card_charge', 'top_up_by_cash_or_cheque', 'top_up_failed', 'top_up_limits', 'top_up_reverted', 'topping_up_by_card', 'transaction_charged_twice', 'transfer_fee_charged', 'transfer_into_account', 'transfer_not_received_by_recipient', 'transfer_timing', 'unable_to_verify_identity', 'verify_my_identity', 'verify_source_of_funds', 'verify_top_up', 'virtual_card_not_working', 'visa_or_mastercard', 'why_verify_identity', 'wrong_amount_of_cash_received', 'wrong_exchange_rate_for_cash_withdrawal'], names_file=None, id=None)}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset[\"train\"].features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The user inputs themselves are fairly short: they're questions like \"What can I do if my card still hasn't arrived after 2 weeks?\" or \"How do I know if I will get my card, or if it is lost?\" Every item has one label, which makes this a straightforward single-label classification task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I am still waiting on my card?\n",
      "What can I do if my card still hasn't arrived after 2 weeks?\n",
      "I have been waiting over a week. Is the card still coming?\n",
      "Can I track my card while it is in the process of delivery?\n",
      "How do I know if I will get my card, or if it is lost?\n"
     ]
    }
   ],
   "source": [
    "for item in dataset[\"train\"][\"text\"][:5]:\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the distribution of the labels. Many text classification problems are hard, not only because of the number of labels, but also because of their skewed distribution. Fortunately, the situation in the `banking77` dataset is not too bad: there's a handful of infrequent labels, but overall, the skew in the distribution is not very pronounced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAFlCAYAAAApo6aBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAVNElEQVR4nO3df6yeZ3kf8O9VsoLWlgDNaYgImUOUElOsGsmCaiWIJGwLJC2l2ggpYoGxGUvJuomgYbofnCFV87amaFNTogApAUEa2owWJZSCYjTCNlYcmmKCE5VQMxzlhyEQqlJoA9f+8GtzXuPY5rz38XvO8ecjHfm+7+d93udS/vrmup/3eaq7AwDA7H5k3gUAAKwXghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAIKfMu4AkOe2003rDhg3zLgMA4JjuvPPOr3b3wpGOrYpgtWHDhuzatWveZQAAHFNVffnxjtkKBAAYRLACABhEsAIAGESwAgAYRLACABhEsAIAGESwAgAYRLACABhEsAIAGESwAgAYRLACABhEsAIAGESwAgAY5JR5FwA/rA3bbzs03rvjkjlWAgDTdKwAAAYRrAAABhGsAAAGEawAAAYRrAAABhGsAAAGEawAAAbxHCvWtsVTD5s/Op86ACA6VgAAw+hYsa5sunHT1Hz3FbvnVAkAJyMdKwCAQXSsWNf2nLfx0HjjPXvmWAkAJwMdKwCAQQQrAIBBBCsAgEEEKwCAQY5583pV3ZDk0iQPd/dzJ2s3J3n25CNPSfKN7t5cVRuS7Ely7+TYp7t72+iiYTmu3bZzan7ldRfOqRIA1qvj+VXge5L8VpL3Hlzo7ssOjqvqmiRLH3d9X3dvHlQfAMCaccxg1d2fnHSifkBVVZJXJvG//gDASW/W51idn+Sh7v7zJWtnV9WfJvlmkn/X3Xcc6cSq2ppka5KcddZZM5YBP7xrLrt0an71zbfOqRIA1otZb16/PMlNS+YPJDmru5+X5I1JPlBVTz7Sid19fXdv6e4tCwsLM5YBADB/yw5WVXVKkl9OcvPBte7+Tnd/bTK+M8l9SX561iIBANaCWbYCX5Lknu7ed3ChqhaSPNLd362qZyU5N8mXZqwRToh927+/a33mjvPnWAkAa9UxO1ZVdVOS/5Pk2VW1r6pePzn0qkxvAybJi5J8rqruSvL7SbZ19yMD6wUAWLWO51eBlz/O+muPsHZLkltmLwvma3Fx8ahzADgST14HABhk1sctwEnh9p3nHBq/uqabsg9esPkEVwPAaqVjBQAwiGAFADCIrUCY0Ybtt03N9+64ZE6VADBvOlYAAIPoWMFoi6cuGT86vzoAOOEEK1hBm27cNDXffcXuOVUCwIlgKxAAYBAdKziB9py3cWq+8Z49c6oEgJWgYwUAMIhgBQAwiGAFADCIYAUAMIib12GOrt2289D4yusunGMlAIygYwUAMIiOFawS11x26dT86ptvnVMlACyXYAWr1L7tdxwan7nj/DlWAsDxshUIADCIjhWsAYuLi0edA7A66FgBAAyiYwVr0O07z5mav7puOTR+8ILNJ7gaAA7SsQIAGESwAgAYxFYgrDMbtt82Nd+745I5VQJw8tGxAgAYRMcK1rvFUw+bPzqfOgBOAjpWAACDCFYAAIMIVgAAgwhWAACDuHkdTjKbbtx0aPzB//TY1LGN9+w50eUArCvH7FhV1Q1V9XBVfX7J2mJV3V9Vd03+Xrbk2Fuq6otVdW9V/aOVKhwAYLU5nq3A9yS5+Ajrb+/uzZO/jyRJVT0nyauS/MzknN+uqieMKhYAYDU7ZrDq7k8meeQ4v+/lSX63u7/T3X+R5ItJnj9DfQAAa8YsN69fVVWfm2wVPnWy9owkX1nymX2TNQCAdW+5weodSc5JsjnJA0mu+WG/oKq2VtWuqtq1f//+ZZYBALB6LOtXgd390MFxVb0zya2T6f1Jnrnko2dO1o70HdcnuT5JtmzZ0supAxjr2m07D42vvO7COVYCsDYtq2NVVWcsmb4iycFfDH44yauq6olVdXaSc5P8yWwlAgCsDcfsWFXVTUlenOS0qtqX5K1JXlxVm5N0kr1J3pAk3X13VX0wyReSPJbkyu7+7opUDqyoay67dGp+9c23Ps4nATjomMGquy8/wvK7j/L5X0/y67MUBQCwFnmlDQDAIF5pAxyXfdvvmJqfueP8OVUCsHrpWAEADKJjBSzL4uLiEccAJzMdKwCAQQQrAIBBbAUCM7t95zlT84suvG9OlQDMl44VAMAgghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAIB63AAz39E/cNTV/8ILNc6kD4ETTsQIAGESwAgAYxFYgsOI2bL/t0HjvjkvmWAnAytKxAgAYRLACABhEsAIAGMQ9VsCJtXjqYfNH51MHwArQsQIAGESwAgAYxFYgMFebbtx0aLz7it1zrARgdjpWAACDCFYAAIMIVgAAgwhWAACDCFYAAIMIVgAAg3jcArBq7Dlv49R854uvnZp/++u/eWh82dlvnjp25o7zV64wgOMkWAHrwuLi4lHnACeCrUAAgEGO2bGqqhuSXJrk4e5+7mTtvyb5hSR/k+S+JK/r7m9U1YYke5LcOzn90929bSUKBzia23eec2h80YX3zbES4GRyPFuB70nyW0neu2Tt40ne0t2PVdV/TvKWJAdveLivuzePLBJgFk//xF1T8yf98f2Hxnuf9CvTH1589ARUBKxXx9wK7O5PJnnksLWPdfdjk+mnk5y5ArUBAKwpI+6x+mdJ/mjJ/Oyq+tOq+p9V9bg/06mqrVW1q6p27d+/f0AZAADzNVOwqqp/m+SxJO+fLD2Q5Kzufl6SNyb5QFU9+Ujndvf13b2lu7csLCzMUgYAwKqw7GBVVa/NgZvaX93dnSTd/Z3u/tpkfGcO3Nj+0wPqBABY9ZYVrKrq4iT/Jskvdve3lqwvVNUTJuNnJTk3yZdGFAoAsNodz+MWbkry4iSnVdW+JG/NgV8BPjHJx6sq+f5jFV6U5G1V9bdJvpdkW3c/csQvBgBYZ44ZrLr78iMsv/txPntLkltmLQpgXjbduGlqvvuK3XOqBFiLPHkdAGAQ7woEOIqlL4Y+2kuhk+kXQ7/rSbdPHfPuQjg5CFYAJ4BX7MDJwVYgAMAgghUAwCCCFQDAIO6xAjjBnv6Ju6bmD16weS51AOPpWAEADCJYAQAMIlgBAAwiWAEADOLmdYA527D9tqn53h2XzKkSYFaCFcBqs3jqoeGms8+aOuSl0LC62QoEABhExwpgDTnaS6GvvO7CE10OcBgdKwCAQXSsANaJay67dGp+2dlvPjR+15Nunzq2uLh4IkqCk45gBXASun3nOVPziy68b06VwPpiKxAAYBAdKwCmXgztpdCwfDpWAACDCFYAAIPYCgRgilfswPLpWAEADCJYAQAMIlgBAAziHisAjm7x1CXjR+dXB6wBghUAx23TjZum5ruv2D2nSmB1shUIADCIjhUAy7bnvI1T84337JlTJbA66FgBAAyiYwXAMNdu23lofOV1F86xEpiP4wpWVXVDkkuTPNzdz52sPS3JzUk2JNmb5JXd/fWqqiT/LcnLknwryWu7+7PjSwdgNbvmskun5lfffOucKoET53i3At+T5OLD1rYnub27z01y+2SeJC9Ncu7kb2uSd8xeJgDA6ndcwaq7P5nkkcOWX57kxsn4xiS/tGT9vX3Ap5M8parOGFArAMCqNsvN66d39wOT8YNJTp+Mn5HkK0s+t2+yBgCwrg35VWB3d5L+Yc6pqq1Vtauqdu3fv39EGQAAczVLsHro4Bbf5N+HJ+v3J3nmks+dOVmb0t3Xd/eW7t6ysLAwQxkAAKvDLMHqw0mumIyvSPKHS9b/aR3wc0keXbJlCACwbh3v4xZuSvLiJKdV1b4kb02yI8kHq+r1Sb6c5JWTj38kBx618MUceNzC6wbXDMAatG/7HVPzM3ecP6dKYOUcV7Dq7ssf59BFR/hsJ7lylqIAANYiT14HYC4WFxcPjc9/0fumjl104X0nuBoYw7sCAQAGEawAAAaxFQjAqvP0T9x1aPzgBZvnVgf8sHSsAAAGEawAAAYRrAAABhGsAAAGEawAAAYRrAAABhGsAAAGEawAAAYRrAAABhGsAAAGEawAAAYRrAAABvESZgBWtQ3bb5ua791xyZwqgWPTsQIAGETHCoC1ZfHUw+aPzqcOOAIdKwCAQQQrAIBBBCsAgEEEKwCAQQQrAIBBBCsAgEE8bgGANW3TjZsOjXdfsXuOlYCOFQDAMIIVAMAgghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAIMt+jlVVPTvJzUuWnpXkPyR5SpJ/kWT/ZP3Xuvsjy70OAMBasexg1d33JtmcJFX1hCT3J/lQktcleXt3/8aIAgEA1opRW4EXJbmvu7886PsAANacUcHqVUluWjK/qqo+V1U3VNVTj3RCVW2tql1VtWv//v1H+ggAwJoyc7Cqqh9N8otJfm+y9I4k5+TANuEDSa450nndfX13b+nuLQsLC7OWAQAwdyM6Vi9N8tnufihJuvuh7v5ud38vyTuTPH/ANQAAVr0RweryLNkGrKozlhx7RZLPD7gGAMCqt+xfBSZJVf1Ykn+Q5A1Llv9LVW1O0kn2HnYMAGDdmilYdfdfJfnJw9ZeM1NFAABrlCevAwAMIlgBAAwiWAEADCJYAQAMIlgBAAwiWAEADCJYAQAMIlgBAAwiWAEADCJYAQAMIlgBAAwiWAEADCJYAQAMIlgBAAwiWAEADCJYAQAMIlgBAAxyyrwLAIBR9py3cWq+8Z49c6qEk5WOFQDAIIIVAMAgghUAwCDusQJg3bp2286p+ZXXXTinSjhZ6FgBAAwiWAEADCJYAQAMIlgBAAwiWAEADCJYAQAMIlgBAAwiWAEADCJYAQAMIlgBAAwy8yttqmpvkr9M8t0kj3X3lqp6WpKbk2xIsjfJK7v767NeCwBgNRvVsbqguzd395bJfHuS27v73CS3T+YAAOvaSm0FvjzJjZPxjUl+aYWuAwCwaowIVp3kY1V1Z1Vtnayd3t0PTMYPJjl9wHUAAFa1me+xSvLC7r6/qn4qycer6p6lB7u7q6oPP2kSwrYmyVlnnTWgDACA+Zq5Y9Xd90/+fTjJh5I8P8lDVXVGkkz+ffgI513f3Vu6e8vCwsKsZQAAzN1MwaqqfqyqfuLgOMk/TPL5JB9OcsXkY1ck+cNZrgMAsBbMuhV4epIPVdXB7/pAd3+0qj6T5INV9fokX07yyhmvAwCw6s0UrLr7S0l+9gjrX0ty0SzfDQCw1njyOgDAIIIVAMAgghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAIIIVAMAgghUAwCCCFQDAILO+KxAA1oxrLrv00Pjqm2+dYyWsVzpWAACDCFYAAIMIVgAAgwhWAACDCFYAAIMIVgAAgwhWAACDCFYAAIMIVgAAg3jyOgAnpX3b75ian7nj/DlVwnqiYwUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADCIYAUAMIhgBQAwiGAFADDIsoNVVT2zqj5RVV+oqrur6l9N1her6v6qumvy97Jx5QIArF6nzHDuY0mu7u7PVtVPJLmzqj4+Ofb27v6N2csDAFg7lh2suvuBJA9Mxn9ZVXuSPGNUYQAAa82Qe6yqakOS5yX5v5Olq6rqc1V1Q1U99XHO2VpVu6pq1/79+0eUAQAwVzMHq6r68SS3JPnX3f3NJO9Ick6SzTnQ0brmSOd19/XdvaW7tywsLMxaBgDA3M0UrKrq7+RAqHp/d/+PJOnuh7r7u939vSTvTPL82csEAFj9ZvlVYCV5d5I93f2bS9bPWPKxVyT5/PLLAwBYO2b5VeDPJ3lNkt1Vdddk7deSXF5Vm5N0kr1J3jDDNQAA1oxZfhX4qSR1hEMfWX45AABrlyevAwAMIlgBAAwiWAEADCJYAQAMIlgBAAwiWAEADCJYAQAMIlgBAAwiWAEADCJYAQAMIlgBAAwiWAEADCJYAQAMIlgBAAwiWAEADCJYAQAMIlgBAAwiWAEADCJYAQAMIlgBAAwiWAEADCJYAQAMIlgBAAwiWAEADCJYAQAMIlgBAAwiWAEADCJYAQAMIlgBAAwiWAEADCJYAQAMIlgBAAyyYsGqqi6uqnur6otVtX2lrgMAsFqsSLCqqickuTbJS5M8J8nlVfWclbgWAMBqsVIdq+cn+WJ3f6m7/ybJ7yZ5+QpdCwBgVVipYPWMJF9ZMt83WQMAWLequ8d/adU/TnJxd//zyfw1SV7Q3Vct+czWJFsn02cnuXd4IcDJ7rQkX513EcC68/e6e+FIB05ZoQven+SZS+ZnTtYO6e7rk1y/QtcHSFXt6u4t864DOHms1FbgZ5KcW1VnV9WPJnlVkg+v0LUAAFaFFelYdfdjVXVVkj9O8oQkN3T33StxLQCA1WJF7rECWA2qauvktgOAE0KwAgAYxCttAAAGEayAuamqX62qPVX1/nnXAjCCrUBgbqrqniQv6e59S9ZO6e7H5lgWwLLpWAFzUVXXJXlWkj+qqker6n1V9b+SvK+qFqrqlqr6zOTv5yfn/GRVfayq7q6qd1XVl6vqtKraUFWfX/Ldb6qqxcn4nKr6aFXdWVV3VNV5k/X3VNV/r6r/XVVfmjzY+OD5b66q3VX1Z1W1Y/Idn11y/Nylc4CDVuoBoQBH1d3bquriJBckuSrJLyR5YXf/dVV9IMnbu/tTVXVWDjy6ZWOStyb5VHe/raouSfL647jU9Um2dfefV9ULkvx2kgsnx85I8sIk5+XAs/Z+v6pemgPvNn1Bd3+rqp7W3Y9Mwt/m7r4ryeuS/M6Y/xLAeiJYAavFh7v7ryfjlyR5TlUdPPbkqvrxJC9K8stJ0t23VdXXj/aFk3P+fpLfW/JdT1zykT/o7u8l+UJVnb7k2r/T3d+aXOeRyfq7kryuqt6Y5LIceNk8wBTBClgt/mrJ+EeS/Fx3f3vpB5aEo8M9lulbG5605Hu+0d2bH+e87yz9+mPUd0sOdMx2Jrmzu792jM8DJyH3WAGr0ceS/MuDk6raPBl+MsmvTNZemuSpk/WHkvzU5B6sJya5NEm6+5tJ/qKq/snknKqqnz3GtT+eA52pvzs552mT7/p2DmxJviO2AYHHIVgBq9GvJtlSVZ+rqi8k2TZZ/49JXlRVd+fAluD/S5Lu/tskb0vyJzkQjO5Z8l2vTvL6qvqzJHfnwP1Tj6u7P5oD91vtqqq7krxpyeH3J/leDgQ/gB/gcQvAmlVVe5Ns6e6vnqDrvSnJqd3970/E9YC1xz1WAMehqj6U5Jx8/xeFAD9AxwoAYBD3WAEADCJYAQAMIlgBAAwiWAEADCJYAQAMIlgBAAzy/wHzElsxRz6f2QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from collections import Counter\n",
    "\n",
    "train_texts = [item[\"text\"] for item in dataset[\"train\"]]\n",
    "train_labels = [item[\"label\"] for item in dataset[\"train\"]]\n",
    "\n",
    "test_texts = [item[\"text\"] for item in dataset[\"test\"]]\n",
    "test_labels = [item[\"label\"] for item in dataset[\"test\"]]\n",
    "\n",
    "label_counter = Counter(train_labels)\n",
    "label_names = dataset[\"train\"].features[\"label\"].names\n",
    "label_frequencies = {label_names[label]: [label_counter[label]] for label in label_counter}\n",
    "\n",
    "df = pd.DataFrame.from_dict(label_frequencies, orient=\"index\", columns=[\"frequency\"])\n",
    "df = df.sort_values(\"frequency\", ascending=False)\n",
    "\n",
    "plt.rcParams['figure.figsize'] = (10,6)\n",
    "ax = df.transpose().plot(kind=\"bar\", rot=0)\n",
    "ax.get_legend().remove()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we start training, we need to take a few preparatory steps. First of all, we set aside 10% of the training data as our development (or validation) set, which we'll use to evaluate the performance of the models during training. We keep the test set intact."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train: 9002\n",
      "Dev: 1001\n",
      "Test: 3080\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train_texts, dev_texts, train_labels, dev_labels = train_test_split(train_texts, \n",
    "                                                                    train_labels, \n",
    "                                                                    test_size=0.1, \n",
    "                                                                    shuffle=True, \n",
    "                                                                    random_state=1)\n",
    "\n",
    "print(\"Train:\", len(train_texts))\n",
    "print(\"Dev:\", len(dev_texts))\n",
    "print(\"Test:\", len(test_texts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we define a dataset class to wrap the data in. When the model trainer gets an item from this class, it returns all information in the encoding (the token ids, the mask ids, etc.), together with the label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "class ClassificationDataset(torch.utils.data.Dataset):\n",
    "    def __init__(self, encodings, labels):\n",
    "        self.encodings = encodings\n",
    "        self.labels = labels\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}\n",
    "        item['label'] = self.labels[idx]\n",
    "        return item\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we define the metrics that we'll evaluate the models with. As this is a single-label classification task, we'll simply compute the overall accuracy of the model output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "def compute_metrics(pred):\n",
    "    labels = pred.label_ids\n",
    "    preds = pred.predictions.argmax(-1)\n",
    "    acc = accuracy_score(labels, preds)\n",
    "    return {\n",
    "        'accuracy': acc\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Smaller transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We evaluate two types of small transformer models: ALBERT, and a family of smaller BERT models.\n",
    "    \n",
    "ALBERT was introduced in the paper [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://arxiv.org/abs/1909.11942). It uses various insights to shrink the footprint of a BERT model:\n",
    "\n",
    "- In contrast to models like BERT and XLNet, it decouples the size of the token embeddings from the size of the hidden layer. This allows it to have a much smaller embedding layer, and therefore, a more efficient model.\n",
    "- It further reduces BERT's memory requirements by sharing all parameters across layers. \n",
    "\n",
    "Finally, in pretraining, the original next-sentence prediction task is replaced by a sentence order prediction task, where the model has to decide whether two sentences are presented in their original order, or whether the order has been swapped. This obviously has no effect on the size or speed of the model, but is a more challenging pretraining task that forces the model to focus more on sentence coherence and not just sentence topic.\n",
    "\n",
    "While the original BERT has 108M parameters, ALBERT-base has just 12M and ALBERT-large has 18M."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another type of smaller BERT models was presented in the paper [Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://arxiv.org/abs/1908.08962). The researchers combine pretraining with model distillation to train smaller models:\n",
    "\n",
    "- In a first step, a compact model is pretrained using the well-known masked language modeling objective.\n",
    "- In a second step, these small student models are further trained on the label probabilities (the so-called soft labels) that are produced by a larger teacher model.\n",
    "\n",
    "We'll test models with four different sizes: BERT-tiny (4.4M parameters), BERT-mini (11.3M parameters), BERT-small (29.1M parameters), and BERT-medium (41.7M parameters). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the `transformer` library to train and evaluate all models. We train every model for 3 epochs with a batch size of 16, and evaluate every 50 steps. At the end of every training cycle, we evaluate the best checkpoint on the test data and remember the accuracy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** prajjwal1/bert-tiny ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at prajjwal1/bert-tiny were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.predictions.decoder.bias', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at prajjwal1/bert-tiny and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n",
      "Trainer is attempting to log a value of \"{0: 'LABEL_0', 1: 'LABEL_1', 2: 'LABEL_2', 3: 'LABEL_3', 4: 'LABEL_4', 5: 'LABEL_5', 6: 'LABEL_6', 7: 'LABEL_7', 8: 'LABEL_8', 9: 'LABEL_9', 10: 'LABEL_10', 11: 'LABEL_11', 12: 'LABEL_12', 13: 'LABEL_13', 14: 'LABEL_14', 15: 'LABEL_15', 16: 'LABEL_16', 17: 'LABEL_17', 18: 'LABEL_18', 19: 'LABEL_19', 20: 'LABEL_20', 21: 'LABEL_21', 22: 'LABEL_22', 23: 'LABEL_23', 24: 'LABEL_24', 25: 'LABEL_25', 26: 'LABEL_26', 27: 'LABEL_27', 28: 'LABEL_28', 29: 'LABEL_29', 30: 'LABEL_30', 31: 'LABEL_31', 32: 'LABEL_32', 33: 'LABEL_33', 34: 'LABEL_34', 35: 'LABEL_35', 36: 'LABEL_36', 37: 'LABEL_37', 38: 'LABEL_38', 39: 'LABEL_39', 40: 'LABEL_40', 41: 'LABEL_41', 42: 'LABEL_42', 43: 'LABEL_43', 44: 'LABEL_44', 45: 'LABEL_45', 46: 'LABEL_46', 47: 'LABEL_47', 48: 'LABEL_48', 49: 'LABEL_49', 50: 'LABEL_50', 51: 'LABEL_51', 52: 'LABEL_52', 53: 'LABEL_53', 54: 'LABEL_54', 55: 'LABEL_55', 56: 'LABEL_56', 57: 'LABEL_57', 58: 'LABEL_58', 59: 'LABEL_59', 60: 'LABEL_60', 61: 'LABEL_61', 62: 'LABEL_62', 63: 'LABEL_63', 64: 'LABEL_64', 65: 'LABEL_65', 66: 'LABEL_66', 67: 'LABEL_67', 68: 'LABEL_68', 69: 'LABEL_69', 70: 'LABEL_70', 71: 'LABEL_71', 72: 'LABEL_72', 73: 'LABEL_73', 74: 'LABEL_74', 75: 'LABEL_75', 76: 'LABEL_76'}\" for key \"id2label\" as a parameter. MLflow's log_param() only accepts values no longer than 250 characters so we dropped this attribute.\n",
      "Trainer is attempting to log a value of \"{'LABEL_0': 0, 'LABEL_1': 1, 'LABEL_2': 2, 'LABEL_3': 3, 'LABEL_4': 4, 'LABEL_5': 5, 'LABEL_6': 6, 'LABEL_7': 7, 'LABEL_8': 8, 'LABEL_9': 9, 'LABEL_10': 10, 'LABEL_11': 11, 'LABEL_12': 12, 'LABEL_13': 13, 'LABEL_14': 14, 'LABEL_15': 15, 'LABEL_16': 16, 'LABEL_17': 17, 'LABEL_18': 18, 'LABEL_19': 19, 'LABEL_20': 20, 'LABEL_21': 21, 'LABEL_22': 22, 'LABEL_23': 23, 'LABEL_24': 24, 'LABEL_25': 25, 'LABEL_26': 26, 'LABEL_27': 27, 'LABEL_28': 28, 'LABEL_29': 29, 'LABEL_30': 30, 'LABEL_31': 31, 'LABEL_32': 32, 'LABEL_33': 33, 'LABEL_34': 34, 'LABEL_35': 35, 'LABEL_36': 36, 'LABEL_37': 37, 'LABEL_38': 38, 'LABEL_39': 39, 'LABEL_40': 40, 'LABEL_41': 41, 'LABEL_42': 42, 'LABEL_43': 43, 'LABEL_44': 44, 'LABEL_45': 45, 'LABEL_46': 46, 'LABEL_47': 47, 'LABEL_48': 48, 'LABEL_49': 49, 'LABEL_50': 50, 'LABEL_51': 51, 'LABEL_52': 52, 'LABEL_53': 53, 'LABEL_54': 54, 'LABEL_55': 55, 'LABEL_56': 56, 'LABEL_57': 57, 'LABEL_58': 58, 'LABEL_59': 59, 'LABEL_60': 60, 'LABEL_61': 61, 'LABEL_62': 62, 'LABEL_63': 63, 'LABEL_64': 64, 'LABEL_65': 65, 'LABEL_66': 66, 'LABEL_67': 67, 'LABEL_68': 68, 'LABEL_69': 69, 'LABEL_70': 70, 'LABEL_71': 71, 'LABEL_72': 72, 'LABEL_73': 73, 'LABEL_74': 74, 'LABEL_75': 75, 'LABEL_76': 76}\" for key \"label2id\" as a parameter. MLflow's log_param() only accepts values no longer than 250 characters so we dropped this attribute.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='1689' max='1689' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [1689/1689 00:25, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Runtime</th>\n",
       "      <th>Samples Per Second</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.356461</td>\n",
       "      <td>0.008991</td>\n",
       "      <td>0.065900</td>\n",
       "      <td>15195.764000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>100</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.353720</td>\n",
       "      <td>0.008991</td>\n",
       "      <td>0.065500</td>\n",
       "      <td>15283.272000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>150</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.348154</td>\n",
       "      <td>0.008991</td>\n",
       "      <td>0.065500</td>\n",
       "      <td>15283.606000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>200</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.341264</td>\n",
       "      <td>0.008991</td>\n",
       "      <td>0.064500</td>\n",
       "      <td>15514.941000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>250</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.333517</td>\n",
       "      <td>0.010989</td>\n",
       "      <td>0.065500</td>\n",
       "      <td>15290.230000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>300</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.324213</td>\n",
       "      <td>0.011988</td>\n",
       "      <td>0.066200</td>\n",
       "      <td>15131.741000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>350</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.313622</td>\n",
       "      <td>0.032967</td>\n",
       "      <td>0.065100</td>\n",
       "      <td>15380.241000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>400</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.296056</td>\n",
       "      <td>0.043956</td>\n",
       "      <td>0.065100</td>\n",
       "      <td>15378.382000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>450</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.269078</td>\n",
       "      <td>0.062937</td>\n",
       "      <td>0.064600</td>\n",
       "      <td>15485.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>500</td>\n",
       "      <td>4.340500</td>\n",
       "      <td>4.240679</td>\n",
       "      <td>0.089910</td>\n",
       "      <td>0.064900</td>\n",
       "      <td>15433.386000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>550</td>\n",
       "      <td>4.340500</td>\n",
       "      <td>4.194570</td>\n",
       "      <td>0.112887</td>\n",
       "      <td>0.066900</td>\n",
       "      <td>14955.219000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>600</td>\n",
       "      <td>4.340500</td>\n",
       "      <td>4.132773</td>\n",
       "      <td>0.179820</td>\n",
       "      <td>0.065300</td>\n",
       "      <td>15331.492000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>650</td>\n",
       "      <td>4.340500</td>\n",
       "      <td>4.076171</td>\n",
       "      <td>0.213786</td>\n",
       "      <td>0.065300</td>\n",
       "      <td>15318.123000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>700</td>\n",
       "      <td>4.340500</td>\n",
       "      <td>4.025240</td>\n",
       "      <td>0.244755</td>\n",
       "      <td>0.067900</td>\n",
       "      <td>14745.284000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>750</td>\n",
       "      <td>4.340500</td>\n",
       "      <td>3.978513</td>\n",
       "      <td>0.306693</td>\n",
       "      <td>0.065400</td>\n",
       "      <td>15314.938000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>800</td>\n",
       "      <td>4.340500</td>\n",
       "      <td>3.933975</td>\n",
       "      <td>0.323676</td>\n",
       "      <td>0.065900</td>\n",
       "      <td>15181.697000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>850</td>\n",
       "      <td>4.340500</td>\n",
       "      <td>3.886502</td>\n",
       "      <td>0.351648</td>\n",
       "      <td>0.067500</td>\n",
       "      <td>14824.211000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>900</td>\n",
       "      <td>4.340500</td>\n",
       "      <td>3.846971</td>\n",
       "      <td>0.335664</td>\n",
       "      <td>0.067600</td>\n",
       "      <td>14801.686000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>950</td>\n",
       "      <td>4.340500</td>\n",
       "      <td>3.806360</td>\n",
       "      <td>0.350649</td>\n",
       "      <td>0.065400</td>\n",
       "      <td>15314.267000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1000</td>\n",
       "      <td>4.040900</td>\n",
       "      <td>3.768915</td>\n",
       "      <td>0.358641</td>\n",
       "      <td>0.065800</td>\n",
       "      <td>15204.679000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1050</td>\n",
       "      <td>4.040900</td>\n",
       "      <td>3.733139</td>\n",
       "      <td>0.358641</td>\n",
       "      <td>0.066300</td>\n",
       "      <td>15093.553000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1100</td>\n",
       "      <td>4.040900</td>\n",
       "      <td>3.703229</td>\n",
       "      <td>0.380619</td>\n",
       "      <td>0.065800</td>\n",
       "      <td>15223.368000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1150</td>\n",
       "      <td>4.040900</td>\n",
       "      <td>3.672989</td>\n",
       "      <td>0.379620</td>\n",
       "      <td>0.065400</td>\n",
       "      <td>15298.197000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1200</td>\n",
       "      <td>4.040900</td>\n",
       "      <td>3.645903</td>\n",
       "      <td>0.379620</td>\n",
       "      <td>0.065100</td>\n",
       "      <td>15379.734000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1250</td>\n",
       "      <td>4.040900</td>\n",
       "      <td>3.622068</td>\n",
       "      <td>0.382617</td>\n",
       "      <td>0.066000</td>\n",
       "      <td>15158.949000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1300</td>\n",
       "      <td>4.040900</td>\n",
       "      <td>3.602915</td>\n",
       "      <td>0.389610</td>\n",
       "      <td>0.065800</td>\n",
       "      <td>15222.264000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1350</td>\n",
       "      <td>4.040900</td>\n",
       "      <td>3.587602</td>\n",
       "      <td>0.384615</td>\n",
       "      <td>0.067600</td>\n",
       "      <td>14803.826000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1400</td>\n",
       "      <td>4.040900</td>\n",
       "      <td>3.568307</td>\n",
       "      <td>0.391608</td>\n",
       "      <td>0.068600</td>\n",
       "      <td>14597.024000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1450</td>\n",
       "      <td>4.040900</td>\n",
       "      <td>3.553015</td>\n",
       "      <td>0.398601</td>\n",
       "      <td>0.068400</td>\n",
       "      <td>14633.809000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1500</td>\n",
       "      <td>3.701200</td>\n",
       "      <td>3.542023</td>\n",
       "      <td>0.395604</td>\n",
       "      <td>0.066200</td>\n",
       "      <td>15121.659000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1550</td>\n",
       "      <td>3.701200</td>\n",
       "      <td>3.533484</td>\n",
       "      <td>0.397602</td>\n",
       "      <td>0.066000</td>\n",
       "      <td>15172.096000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1600</td>\n",
       "      <td>3.701200</td>\n",
       "      <td>3.527247</td>\n",
       "      <td>0.405594</td>\n",
       "      <td>0.066300</td>\n",
       "      <td>15100.556000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1650</td>\n",
       "      <td>3.701200</td>\n",
       "      <td>3.523768</td>\n",
       "      <td>0.406593</td>\n",
       "      <td>0.064400</td>\n",
       "      <td>15546.942000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='49' max='49' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [49/49 00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** prajjwal1/bert-mini ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at prajjwal1/bert-mini were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.predictions.decoder.bias', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at prajjwal1/bert-mini and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n",
      "Trainer is attempting to log a value of \"{0: 'LABEL_0', 1: 'LABEL_1', 2: 'LABEL_2', 3: 'LABEL_3', 4: 'LABEL_4', 5: 'LABEL_5', 6: 'LABEL_6', 7: 'LABEL_7', 8: 'LABEL_8', 9: 'LABEL_9', 10: 'LABEL_10', 11: 'LABEL_11', 12: 'LABEL_12', 13: 'LABEL_13', 14: 'LABEL_14', 15: 'LABEL_15', 16: 'LABEL_16', 17: 'LABEL_17', 18: 'LABEL_18', 19: 'LABEL_19', 20: 'LABEL_20', 21: 'LABEL_21', 22: 'LABEL_22', 23: 'LABEL_23', 24: 'LABEL_24', 25: 'LABEL_25', 26: 'LABEL_26', 27: 'LABEL_27', 28: 'LABEL_28', 29: 'LABEL_29', 30: 'LABEL_30', 31: 'LABEL_31', 32: 'LABEL_32', 33: 'LABEL_33', 34: 'LABEL_34', 35: 'LABEL_35', 36: 'LABEL_36', 37: 'LABEL_37', 38: 'LABEL_38', 39: 'LABEL_39', 40: 'LABEL_40', 41: 'LABEL_41', 42: 'LABEL_42', 43: 'LABEL_43', 44: 'LABEL_44', 45: 'LABEL_45', 46: 'LABEL_46', 47: 'LABEL_47', 48: 'LABEL_48', 49: 'LABEL_49', 50: 'LABEL_50', 51: 'LABEL_51', 52: 'LABEL_52', 53: 'LABEL_53', 54: 'LABEL_54', 55: 'LABEL_55', 56: 'LABEL_56', 57: 'LABEL_57', 58: 'LABEL_58', 59: 'LABEL_59', 60: 'LABEL_60', 61: 'LABEL_61', 62: 'LABEL_62', 63: 'LABEL_63', 64: 'LABEL_64', 65: 'LABEL_65', 66: 'LABEL_66', 67: 'LABEL_67', 68: 'LABEL_68', 69: 'LABEL_69', 70: 'LABEL_70', 71: 'LABEL_71', 72: 'LABEL_72', 73: 'LABEL_73', 74: 'LABEL_74', 75: 'LABEL_75', 76: 'LABEL_76'}\" for key \"id2label\" as a parameter. MLflow's log_param() only accepts values no longer than 250 characters so we dropped this attribute.\n",
      "Trainer is attempting to log a value of \"{'LABEL_0': 0, 'LABEL_1': 1, 'LABEL_2': 2, 'LABEL_3': 3, 'LABEL_4': 4, 'LABEL_5': 5, 'LABEL_6': 6, 'LABEL_7': 7, 'LABEL_8': 8, 'LABEL_9': 9, 'LABEL_10': 10, 'LABEL_11': 11, 'LABEL_12': 12, 'LABEL_13': 13, 'LABEL_14': 14, 'LABEL_15': 15, 'LABEL_16': 16, 'LABEL_17': 17, 'LABEL_18': 18, 'LABEL_19': 19, 'LABEL_20': 20, 'LABEL_21': 21, 'LABEL_22': 22, 'LABEL_23': 23, 'LABEL_24': 24, 'LABEL_25': 25, 'LABEL_26': 26, 'LABEL_27': 27, 'LABEL_28': 28, 'LABEL_29': 29, 'LABEL_30': 30, 'LABEL_31': 31, 'LABEL_32': 32, 'LABEL_33': 33, 'LABEL_34': 34, 'LABEL_35': 35, 'LABEL_36': 36, 'LABEL_37': 37, 'LABEL_38': 38, 'LABEL_39': 39, 'LABEL_40': 40, 'LABEL_41': 41, 'LABEL_42': 42, 'LABEL_43': 43, 'LABEL_44': 44, 'LABEL_45': 45, 'LABEL_46': 46, 'LABEL_47': 47, 'LABEL_48': 48, 'LABEL_49': 49, 'LABEL_50': 50, 'LABEL_51': 51, 'LABEL_52': 52, 'LABEL_53': 53, 'LABEL_54': 54, 'LABEL_55': 55, 'LABEL_56': 56, 'LABEL_57': 57, 'LABEL_58': 58, 'LABEL_59': 59, 'LABEL_60': 60, 'LABEL_61': 61, 'LABEL_62': 62, 'LABEL_63': 63, 'LABEL_64': 64, 'LABEL_65': 65, 'LABEL_66': 66, 'LABEL_67': 67, 'LABEL_68': 68, 'LABEL_69': 69, 'LABEL_70': 70, 'LABEL_71': 71, 'LABEL_72': 72, 'LABEL_73': 73, 'LABEL_74': 74, 'LABEL_75': 75, 'LABEL_76': 76}\" for key \"label2id\" as a parameter. MLflow's log_param() only accepts values no longer than 250 characters so we dropped this attribute.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='1689' max='1689' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [1689/1689 00:51, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Runtime</th>\n",
       "      <th>Samples Per Second</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.375497</td>\n",
       "      <td>0.004995</td>\n",
       "      <td>0.138200</td>\n",
       "      <td>7243.961000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>100</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.363358</td>\n",
       "      <td>0.005994</td>\n",
       "      <td>0.138600</td>\n",
       "      <td>7220.825000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>150</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.340960</td>\n",
       "      <td>0.017982</td>\n",
       "      <td>0.139000</td>\n",
       "      <td>7199.355000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>200</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.316107</td>\n",
       "      <td>0.032967</td>\n",
       "      <td>0.138400</td>\n",
       "      <td>7230.662000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>250</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.277944</td>\n",
       "      <td>0.038961</td>\n",
       "      <td>0.138500</td>\n",
       "      <td>7228.658000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>300</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.215641</td>\n",
       "      <td>0.060939</td>\n",
       "      <td>0.138600</td>\n",
       "      <td>7222.341000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>350</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.135591</td>\n",
       "      <td>0.123876</td>\n",
       "      <td>0.139000</td>\n",
       "      <td>7203.011000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>400</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.013452</td>\n",
       "      <td>0.223776</td>\n",
       "      <td>0.138700</td>\n",
       "      <td>7216.779000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>450</td>\n",
       "      <td>No log</td>\n",
       "      <td>3.882334</td>\n",
       "      <td>0.304695</td>\n",
       "      <td>0.138900</td>\n",
       "      <td>7204.123000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>500</td>\n",
       "      <td>4.221700</td>\n",
       "      <td>3.747517</td>\n",
       "      <td>0.382617</td>\n",
       "      <td>0.139300</td>\n",
       "      <td>7187.917000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>550</td>\n",
       "      <td>4.221700</td>\n",
       "      <td>3.602087</td>\n",
       "      <td>0.390609</td>\n",
       "      <td>0.139500</td>\n",
       "      <td>7176.847000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>600</td>\n",
       "      <td>4.221700</td>\n",
       "      <td>3.462773</td>\n",
       "      <td>0.441558</td>\n",
       "      <td>0.139300</td>\n",
       "      <td>7185.850000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>650</td>\n",
       "      <td>4.221700</td>\n",
       "      <td>3.337019</td>\n",
       "      <td>0.462537</td>\n",
       "      <td>0.138700</td>\n",
       "      <td>7219.323000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>700</td>\n",
       "      <td>4.221700</td>\n",
       "      <td>3.204512</td>\n",
       "      <td>0.522478</td>\n",
       "      <td>0.139100</td>\n",
       "      <td>7194.112000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>750</td>\n",
       "      <td>4.221700</td>\n",
       "      <td>3.106938</td>\n",
       "      <td>0.536464</td>\n",
       "      <td>0.139100</td>\n",
       "      <td>7194.938000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>800</td>\n",
       "      <td>4.221700</td>\n",
       "      <td>3.004632</td>\n",
       "      <td>0.561439</td>\n",
       "      <td>0.139900</td>\n",
       "      <td>7156.003000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>850</td>\n",
       "      <td>4.221700</td>\n",
       "      <td>2.907292</td>\n",
       "      <td>0.546454</td>\n",
       "      <td>0.139500</td>\n",
       "      <td>7173.364000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>900</td>\n",
       "      <td>4.221700</td>\n",
       "      <td>2.831156</td>\n",
       "      <td>0.580420</td>\n",
       "      <td>0.140000</td>\n",
       "      <td>7151.834000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>950</td>\n",
       "      <td>4.221700</td>\n",
       "      <td>2.750956</td>\n",
       "      <td>0.586414</td>\n",
       "      <td>0.139900</td>\n",
       "      <td>7154.137000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1000</td>\n",
       "      <td>3.221300</td>\n",
       "      <td>2.676676</td>\n",
       "      <td>0.580420</td>\n",
       "      <td>0.140200</td>\n",
       "      <td>7142.174000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1050</td>\n",
       "      <td>3.221300</td>\n",
       "      <td>2.611545</td>\n",
       "      <td>0.623377</td>\n",
       "      <td>0.139400</td>\n",
       "      <td>7179.227000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1100</td>\n",
       "      <td>3.221300</td>\n",
       "      <td>2.555033</td>\n",
       "      <td>0.626374</td>\n",
       "      <td>0.140600</td>\n",
       "      <td>7119.816000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1150</td>\n",
       "      <td>3.221300</td>\n",
       "      <td>2.499551</td>\n",
       "      <td>0.641359</td>\n",
       "      <td>0.139700</td>\n",
       "      <td>7167.143000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1200</td>\n",
       "      <td>3.221300</td>\n",
       "      <td>2.452430</td>\n",
       "      <td>0.630370</td>\n",
       "      <td>0.139900</td>\n",
       "      <td>7157.259000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1250</td>\n",
       "      <td>3.221300</td>\n",
       "      <td>2.408860</td>\n",
       "      <td>0.630370</td>\n",
       "      <td>0.140900</td>\n",
       "      <td>7106.090000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1300</td>\n",
       "      <td>3.221300</td>\n",
       "      <td>2.369654</td>\n",
       "      <td>0.655345</td>\n",
       "      <td>0.139600</td>\n",
       "      <td>7167.963000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1350</td>\n",
       "      <td>3.221300</td>\n",
       "      <td>2.338646</td>\n",
       "      <td>0.663337</td>\n",
       "      <td>0.139300</td>\n",
       "      <td>7183.895000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1400</td>\n",
       "      <td>3.221300</td>\n",
       "      <td>2.307493</td>\n",
       "      <td>0.664336</td>\n",
       "      <td>0.139600</td>\n",
       "      <td>7169.261000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1450</td>\n",
       "      <td>3.221300</td>\n",
       "      <td>2.283419</td>\n",
       "      <td>0.673327</td>\n",
       "      <td>0.140200</td>\n",
       "      <td>7142.077000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1500</td>\n",
       "      <td>2.504900</td>\n",
       "      <td>2.266297</td>\n",
       "      <td>0.676324</td>\n",
       "      <td>0.139400</td>\n",
       "      <td>7182.519000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1550</td>\n",
       "      <td>2.504900</td>\n",
       "      <td>2.251529</td>\n",
       "      <td>0.682318</td>\n",
       "      <td>0.139800</td>\n",
       "      <td>7161.129000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1600</td>\n",
       "      <td>2.504900</td>\n",
       "      <td>2.241883</td>\n",
       "      <td>0.680320</td>\n",
       "      <td>0.140400</td>\n",
       "      <td>7128.229000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1650</td>\n",
       "      <td>2.504900</td>\n",
       "      <td>2.237252</td>\n",
       "      <td>0.679321</td>\n",
       "      <td>0.140300</td>\n",
       "      <td>7136.771000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='49' max='49' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [49/49 00:00]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** prajjwal1/bert-small ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at prajjwal1/bert-small were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.predictions.decoder.bias', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at prajjwal1/bert-small and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n",
      "Trainer is attempting to log a value of \"{0: 'LABEL_0', 1: 'LABEL_1', 2: 'LABEL_2', 3: 'LABEL_3', 4: 'LABEL_4', 5: 'LABEL_5', 6: 'LABEL_6', 7: 'LABEL_7', 8: 'LABEL_8', 9: 'LABEL_9', 10: 'LABEL_10', 11: 'LABEL_11', 12: 'LABEL_12', 13: 'LABEL_13', 14: 'LABEL_14', 15: 'LABEL_15', 16: 'LABEL_16', 17: 'LABEL_17', 18: 'LABEL_18', 19: 'LABEL_19', 20: 'LABEL_20', 21: 'LABEL_21', 22: 'LABEL_22', 23: 'LABEL_23', 24: 'LABEL_24', 25: 'LABEL_25', 26: 'LABEL_26', 27: 'LABEL_27', 28: 'LABEL_28', 29: 'LABEL_29', 30: 'LABEL_30', 31: 'LABEL_31', 32: 'LABEL_32', 33: 'LABEL_33', 34: 'LABEL_34', 35: 'LABEL_35', 36: 'LABEL_36', 37: 'LABEL_37', 38: 'LABEL_38', 39: 'LABEL_39', 40: 'LABEL_40', 41: 'LABEL_41', 42: 'LABEL_42', 43: 'LABEL_43', 44: 'LABEL_44', 45: 'LABEL_45', 46: 'LABEL_46', 47: 'LABEL_47', 48: 'LABEL_48', 49: 'LABEL_49', 50: 'LABEL_50', 51: 'LABEL_51', 52: 'LABEL_52', 53: 'LABEL_53', 54: 'LABEL_54', 55: 'LABEL_55', 56: 'LABEL_56', 57: 'LABEL_57', 58: 'LABEL_58', 59: 'LABEL_59', 60: 'LABEL_60', 61: 'LABEL_61', 62: 'LABEL_62', 63: 'LABEL_63', 64: 'LABEL_64', 65: 'LABEL_65', 66: 'LABEL_66', 67: 'LABEL_67', 68: 'LABEL_68', 69: 'LABEL_69', 70: 'LABEL_70', 71: 'LABEL_71', 72: 'LABEL_72', 73: 'LABEL_73', 74: 'LABEL_74', 75: 'LABEL_75', 76: 'LABEL_76'}\" for key \"id2label\" as a parameter. MLflow's log_param() only accepts values no longer than 250 characters so we dropped this attribute.\n",
      "Trainer is attempting to log a value of \"{'LABEL_0': 0, 'LABEL_1': 1, 'LABEL_2': 2, 'LABEL_3': 3, 'LABEL_4': 4, 'LABEL_5': 5, 'LABEL_6': 6, 'LABEL_7': 7, 'LABEL_8': 8, 'LABEL_9': 9, 'LABEL_10': 10, 'LABEL_11': 11, 'LABEL_12': 12, 'LABEL_13': 13, 'LABEL_14': 14, 'LABEL_15': 15, 'LABEL_16': 16, 'LABEL_17': 17, 'LABEL_18': 18, 'LABEL_19': 19, 'LABEL_20': 20, 'LABEL_21': 21, 'LABEL_22': 22, 'LABEL_23': 23, 'LABEL_24': 24, 'LABEL_25': 25, 'LABEL_26': 26, 'LABEL_27': 27, 'LABEL_28': 28, 'LABEL_29': 29, 'LABEL_30': 30, 'LABEL_31': 31, 'LABEL_32': 32, 'LABEL_33': 33, 'LABEL_34': 34, 'LABEL_35': 35, 'LABEL_36': 36, 'LABEL_37': 37, 'LABEL_38': 38, 'LABEL_39': 39, 'LABEL_40': 40, 'LABEL_41': 41, 'LABEL_42': 42, 'LABEL_43': 43, 'LABEL_44': 44, 'LABEL_45': 45, 'LABEL_46': 46, 'LABEL_47': 47, 'LABEL_48': 48, 'LABEL_49': 49, 'LABEL_50': 50, 'LABEL_51': 51, 'LABEL_52': 52, 'LABEL_53': 53, 'LABEL_54': 54, 'LABEL_55': 55, 'LABEL_56': 56, 'LABEL_57': 57, 'LABEL_58': 58, 'LABEL_59': 59, 'LABEL_60': 60, 'LABEL_61': 61, 'LABEL_62': 62, 'LABEL_63': 63, 'LABEL_64': 64, 'LABEL_65': 65, 'LABEL_66': 66, 'LABEL_67': 67, 'LABEL_68': 68, 'LABEL_69': 69, 'LABEL_70': 70, 'LABEL_71': 71, 'LABEL_72': 72, 'LABEL_73': 73, 'LABEL_74': 74, 'LABEL_75': 75, 'LABEL_76': 76}\" for key \"label2id\" as a parameter. MLflow's log_param() only accepts values no longer than 250 characters so we dropped this attribute.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='1689' max='1689' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [1689/1689 01:40, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Runtime</th>\n",
       "      <th>Samples Per Second</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.369976</td>\n",
       "      <td>0.011988</td>\n",
       "      <td>0.381300</td>\n",
       "      <td>2625.318000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>100</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.329698</td>\n",
       "      <td>0.014985</td>\n",
       "      <td>0.380700</td>\n",
       "      <td>2629.636000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>150</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.270380</td>\n",
       "      <td>0.034965</td>\n",
       "      <td>0.379800</td>\n",
       "      <td>2635.807000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>200</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.176816</td>\n",
       "      <td>0.080919</td>\n",
       "      <td>0.380100</td>\n",
       "      <td>2633.286000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>250</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.009062</td>\n",
       "      <td>0.184815</td>\n",
       "      <td>0.379200</td>\n",
       "      <td>2639.936000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>300</td>\n",
       "      <td>No log</td>\n",
       "      <td>3.775931</td>\n",
       "      <td>0.272727</td>\n",
       "      <td>0.379100</td>\n",
       "      <td>2640.235000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>350</td>\n",
       "      <td>No log</td>\n",
       "      <td>3.519529</td>\n",
       "      <td>0.351648</td>\n",
       "      <td>0.381300</td>\n",
       "      <td>2625.066000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>400</td>\n",
       "      <td>No log</td>\n",
       "      <td>3.236091</td>\n",
       "      <td>0.403596</td>\n",
       "      <td>0.379800</td>\n",
       "      <td>2635.468000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>450</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.936636</td>\n",
       "      <td>0.491508</td>\n",
       "      <td>0.380600</td>\n",
       "      <td>2629.989000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>500</td>\n",
       "      <td>3.897200</td>\n",
       "      <td>2.664321</td>\n",
       "      <td>0.578422</td>\n",
       "      <td>0.382800</td>\n",
       "      <td>2614.888000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>550</td>\n",
       "      <td>3.897200</td>\n",
       "      <td>2.414597</td>\n",
       "      <td>0.567433</td>\n",
       "      <td>0.381500</td>\n",
       "      <td>2623.696000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>600</td>\n",
       "      <td>3.897200</td>\n",
       "      <td>2.162044</td>\n",
       "      <td>0.634366</td>\n",
       "      <td>0.381100</td>\n",
       "      <td>2626.877000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>650</td>\n",
       "      <td>3.897200</td>\n",
       "      <td>1.929211</td>\n",
       "      <td>0.690310</td>\n",
       "      <td>0.380600</td>\n",
       "      <td>2629.869000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>700</td>\n",
       "      <td>3.897200</td>\n",
       "      <td>1.772261</td>\n",
       "      <td>0.689311</td>\n",
       "      <td>0.379700</td>\n",
       "      <td>2636.430000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>750</td>\n",
       "      <td>3.897200</td>\n",
       "      <td>1.603802</td>\n",
       "      <td>0.743257</td>\n",
       "      <td>0.379600</td>\n",
       "      <td>2636.979000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>800</td>\n",
       "      <td>3.897200</td>\n",
       "      <td>1.485672</td>\n",
       "      <td>0.765235</td>\n",
       "      <td>0.380300</td>\n",
       "      <td>2631.903000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>850</td>\n",
       "      <td>3.897200</td>\n",
       "      <td>1.377897</td>\n",
       "      <td>0.774226</td>\n",
       "      <td>0.381000</td>\n",
       "      <td>2627.117000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>900</td>\n",
       "      <td>3.897200</td>\n",
       "      <td>1.271412</td>\n",
       "      <td>0.786214</td>\n",
       "      <td>0.380400</td>\n",
       "      <td>2631.326000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>950</td>\n",
       "      <td>3.897200</td>\n",
       "      <td>1.198525</td>\n",
       "      <td>0.792208</td>\n",
       "      <td>0.381600</td>\n",
       "      <td>2622.894000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1000</td>\n",
       "      <td>1.840600</td>\n",
       "      <td>1.110979</td>\n",
       "      <td>0.806194</td>\n",
       "      <td>0.379900</td>\n",
       "      <td>2634.782000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1050</td>\n",
       "      <td>1.840600</td>\n",
       "      <td>1.038684</td>\n",
       "      <td>0.826174</td>\n",
       "      <td>0.377900</td>\n",
       "      <td>2648.990000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1100</td>\n",
       "      <td>1.840600</td>\n",
       "      <td>1.017144</td>\n",
       "      <td>0.832168</td>\n",
       "      <td>0.379800</td>\n",
       "      <td>2635.883000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1150</td>\n",
       "      <td>1.840600</td>\n",
       "      <td>0.942438</td>\n",
       "      <td>0.839161</td>\n",
       "      <td>0.382100</td>\n",
       "      <td>2619.420000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1200</td>\n",
       "      <td>1.840600</td>\n",
       "      <td>0.900426</td>\n",
       "      <td>0.847153</td>\n",
       "      <td>0.380600</td>\n",
       "      <td>2630.048000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1250</td>\n",
       "      <td>1.840600</td>\n",
       "      <td>0.881030</td>\n",
       "      <td>0.854146</td>\n",
       "      <td>0.380100</td>\n",
       "      <td>2633.177000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1300</td>\n",
       "      <td>1.840600</td>\n",
       "      <td>0.841051</td>\n",
       "      <td>0.854146</td>\n",
       "      <td>0.382800</td>\n",
       "      <td>2615.046000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1350</td>\n",
       "      <td>1.840600</td>\n",
       "      <td>0.824189</td>\n",
       "      <td>0.854146</td>\n",
       "      <td>0.380800</td>\n",
       "      <td>2628.683000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1400</td>\n",
       "      <td>1.840600</td>\n",
       "      <td>0.798897</td>\n",
       "      <td>0.853147</td>\n",
       "      <td>0.379400</td>\n",
       "      <td>2638.628000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1450</td>\n",
       "      <td>1.840600</td>\n",
       "      <td>0.780440</td>\n",
       "      <td>0.860140</td>\n",
       "      <td>0.379700</td>\n",
       "      <td>2636.276000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1500</td>\n",
       "      <td>0.961400</td>\n",
       "      <td>0.765139</td>\n",
       "      <td>0.864136</td>\n",
       "      <td>0.380400</td>\n",
       "      <td>2631.514000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1550</td>\n",
       "      <td>0.961400</td>\n",
       "      <td>0.757299</td>\n",
       "      <td>0.866134</td>\n",
       "      <td>0.379900</td>\n",
       "      <td>2635.012000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1600</td>\n",
       "      <td>0.961400</td>\n",
       "      <td>0.746712</td>\n",
       "      <td>0.867133</td>\n",
       "      <td>0.380100</td>\n",
       "      <td>2633.509000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1650</td>\n",
       "      <td>0.961400</td>\n",
       "      <td>0.742536</td>\n",
       "      <td>0.868132</td>\n",
       "      <td>0.380300</td>\n",
       "      <td>2632.414000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='49' max='49' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [49/49 00:01]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** prajjwal1/bert-medium ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at prajjwal1/bert-medium were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.predictions.decoder.bias', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at prajjwal1/bert-medium and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n",
      "Trainer is attempting to log a value of \"{0: 'LABEL_0', 1: 'LABEL_1', 2: 'LABEL_2', 3: 'LABEL_3', 4: 'LABEL_4', 5: 'LABEL_5', 6: 'LABEL_6', 7: 'LABEL_7', 8: 'LABEL_8', 9: 'LABEL_9', 10: 'LABEL_10', 11: 'LABEL_11', 12: 'LABEL_12', 13: 'LABEL_13', 14: 'LABEL_14', 15: 'LABEL_15', 16: 'LABEL_16', 17: 'LABEL_17', 18: 'LABEL_18', 19: 'LABEL_19', 20: 'LABEL_20', 21: 'LABEL_21', 22: 'LABEL_22', 23: 'LABEL_23', 24: 'LABEL_24', 25: 'LABEL_25', 26: 'LABEL_26', 27: 'LABEL_27', 28: 'LABEL_28', 29: 'LABEL_29', 30: 'LABEL_30', 31: 'LABEL_31', 32: 'LABEL_32', 33: 'LABEL_33', 34: 'LABEL_34', 35: 'LABEL_35', 36: 'LABEL_36', 37: 'LABEL_37', 38: 'LABEL_38', 39: 'LABEL_39', 40: 'LABEL_40', 41: 'LABEL_41', 42: 'LABEL_42', 43: 'LABEL_43', 44: 'LABEL_44', 45: 'LABEL_45', 46: 'LABEL_46', 47: 'LABEL_47', 48: 'LABEL_48', 49: 'LABEL_49', 50: 'LABEL_50', 51: 'LABEL_51', 52: 'LABEL_52', 53: 'LABEL_53', 54: 'LABEL_54', 55: 'LABEL_55', 56: 'LABEL_56', 57: 'LABEL_57', 58: 'LABEL_58', 59: 'LABEL_59', 60: 'LABEL_60', 61: 'LABEL_61', 62: 'LABEL_62', 63: 'LABEL_63', 64: 'LABEL_64', 65: 'LABEL_65', 66: 'LABEL_66', 67: 'LABEL_67', 68: 'LABEL_68', 69: 'LABEL_69', 70: 'LABEL_70', 71: 'LABEL_71', 72: 'LABEL_72', 73: 'LABEL_73', 74: 'LABEL_74', 75: 'LABEL_75', 76: 'LABEL_76'}\" for key \"id2label\" as a parameter. MLflow's log_param() only accepts values no longer than 250 characters so we dropped this attribute.\n",
      "Trainer is attempting to log a value of \"{'LABEL_0': 0, 'LABEL_1': 1, 'LABEL_2': 2, 'LABEL_3': 3, 'LABEL_4': 4, 'LABEL_5': 5, 'LABEL_6': 6, 'LABEL_7': 7, 'LABEL_8': 8, 'LABEL_9': 9, 'LABEL_10': 10, 'LABEL_11': 11, 'LABEL_12': 12, 'LABEL_13': 13, 'LABEL_14': 14, 'LABEL_15': 15, 'LABEL_16': 16, 'LABEL_17': 17, 'LABEL_18': 18, 'LABEL_19': 19, 'LABEL_20': 20, 'LABEL_21': 21, 'LABEL_22': 22, 'LABEL_23': 23, 'LABEL_24': 24, 'LABEL_25': 25, 'LABEL_26': 26, 'LABEL_27': 27, 'LABEL_28': 28, 'LABEL_29': 29, 'LABEL_30': 30, 'LABEL_31': 31, 'LABEL_32': 32, 'LABEL_33': 33, 'LABEL_34': 34, 'LABEL_35': 35, 'LABEL_36': 36, 'LABEL_37': 37, 'LABEL_38': 38, 'LABEL_39': 39, 'LABEL_40': 40, 'LABEL_41': 41, 'LABEL_42': 42, 'LABEL_43': 43, 'LABEL_44': 44, 'LABEL_45': 45, 'LABEL_46': 46, 'LABEL_47': 47, 'LABEL_48': 48, 'LABEL_49': 49, 'LABEL_50': 50, 'LABEL_51': 51, 'LABEL_52': 52, 'LABEL_53': 53, 'LABEL_54': 54, 'LABEL_55': 55, 'LABEL_56': 56, 'LABEL_57': 57, 'LABEL_58': 58, 'LABEL_59': 59, 'LABEL_60': 60, 'LABEL_61': 61, 'LABEL_62': 62, 'LABEL_63': 63, 'LABEL_64': 64, 'LABEL_65': 65, 'LABEL_66': 66, 'LABEL_67': 67, 'LABEL_68': 68, 'LABEL_69': 69, 'LABEL_70': 70, 'LABEL_71': 71, 'LABEL_72': 72, 'LABEL_73': 73, 'LABEL_74': 74, 'LABEL_75': 75, 'LABEL_76': 76}\" for key \"label2id\" as a parameter. MLflow's log_param() only accepts values no longer than 250 characters so we dropped this attribute.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='1689' max='1689' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [1689/1689 02:59, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Runtime</th>\n",
       "      <th>Samples Per Second</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.354683</td>\n",
       "      <td>0.011988</td>\n",
       "      <td>0.751100</td>\n",
       "      <td>1332.642000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>100</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.303425</td>\n",
       "      <td>0.022977</td>\n",
       "      <td>0.752100</td>\n",
       "      <td>1330.895000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>150</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.216712</td>\n",
       "      <td>0.060939</td>\n",
       "      <td>0.751700</td>\n",
       "      <td>1331.691000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>200</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.050876</td>\n",
       "      <td>0.136863</td>\n",
       "      <td>0.753500</td>\n",
       "      <td>1328.548000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>250</td>\n",
       "      <td>No log</td>\n",
       "      <td>3.809948</td>\n",
       "      <td>0.279720</td>\n",
       "      <td>0.752200</td>\n",
       "      <td>1330.726000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>300</td>\n",
       "      <td>No log</td>\n",
       "      <td>3.496346</td>\n",
       "      <td>0.448551</td>\n",
       "      <td>0.750200</td>\n",
       "      <td>1334.291000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>350</td>\n",
       "      <td>No log</td>\n",
       "      <td>3.193066</td>\n",
       "      <td>0.435564</td>\n",
       "      <td>0.752000</td>\n",
       "      <td>1331.061000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>400</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.877696</td>\n",
       "      <td>0.562438</td>\n",
       "      <td>0.751200</td>\n",
       "      <td>1332.489000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>450</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.577512</td>\n",
       "      <td>0.632368</td>\n",
       "      <td>0.751100</td>\n",
       "      <td>1332.728000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>500</td>\n",
       "      <td>3.709300</td>\n",
       "      <td>2.288523</td>\n",
       "      <td>0.629371</td>\n",
       "      <td>0.752600</td>\n",
       "      <td>1330.052000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>550</td>\n",
       "      <td>3.709300</td>\n",
       "      <td>2.067647</td>\n",
       "      <td>0.682318</td>\n",
       "      <td>0.754300</td>\n",
       "      <td>1327.131000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>600</td>\n",
       "      <td>3.709300</td>\n",
       "      <td>1.785550</td>\n",
       "      <td>0.731269</td>\n",
       "      <td>0.755300</td>\n",
       "      <td>1325.279000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>650</td>\n",
       "      <td>3.709300</td>\n",
       "      <td>1.607896</td>\n",
       "      <td>0.732268</td>\n",
       "      <td>0.754600</td>\n",
       "      <td>1326.570000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>700</td>\n",
       "      <td>3.709300</td>\n",
       "      <td>1.432279</td>\n",
       "      <td>0.763237</td>\n",
       "      <td>0.755400</td>\n",
       "      <td>1325.088000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>750</td>\n",
       "      <td>3.709300</td>\n",
       "      <td>1.287553</td>\n",
       "      <td>0.784216</td>\n",
       "      <td>0.754100</td>\n",
       "      <td>1327.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>800</td>\n",
       "      <td>3.709300</td>\n",
       "      <td>1.167925</td>\n",
       "      <td>0.835165</td>\n",
       "      <td>0.753900</td>\n",
       "      <td>1327.698000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>850</td>\n",
       "      <td>3.709300</td>\n",
       "      <td>1.089155</td>\n",
       "      <td>0.840160</td>\n",
       "      <td>0.755000</td>\n",
       "      <td>1325.904000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>900</td>\n",
       "      <td>3.709300</td>\n",
       "      <td>1.011522</td>\n",
       "      <td>0.843157</td>\n",
       "      <td>0.754200</td>\n",
       "      <td>1327.247000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>950</td>\n",
       "      <td>3.709300</td>\n",
       "      <td>0.930479</td>\n",
       "      <td>0.845155</td>\n",
       "      <td>0.754300</td>\n",
       "      <td>1327.119000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1000</td>\n",
       "      <td>1.498800</td>\n",
       "      <td>0.870180</td>\n",
       "      <td>0.863137</td>\n",
       "      <td>0.754700</td>\n",
       "      <td>1326.347000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1050</td>\n",
       "      <td>1.498800</td>\n",
       "      <td>0.797732</td>\n",
       "      <td>0.872128</td>\n",
       "      <td>0.753500</td>\n",
       "      <td>1328.552000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1100</td>\n",
       "      <td>1.498800</td>\n",
       "      <td>0.760763</td>\n",
       "      <td>0.877123</td>\n",
       "      <td>0.753300</td>\n",
       "      <td>1328.748000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1150</td>\n",
       "      <td>1.498800</td>\n",
       "      <td>0.720300</td>\n",
       "      <td>0.874126</td>\n",
       "      <td>0.754500</td>\n",
       "      <td>1326.636000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1200</td>\n",
       "      <td>1.498800</td>\n",
       "      <td>0.679203</td>\n",
       "      <td>0.894106</td>\n",
       "      <td>0.754200</td>\n",
       "      <td>1327.188000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1250</td>\n",
       "      <td>1.498800</td>\n",
       "      <td>0.659802</td>\n",
       "      <td>0.887113</td>\n",
       "      <td>0.754500</td>\n",
       "      <td>1326.684000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1300</td>\n",
       "      <td>1.498800</td>\n",
       "      <td>0.634539</td>\n",
       "      <td>0.881119</td>\n",
       "      <td>0.753300</td>\n",
       "      <td>1328.801000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1350</td>\n",
       "      <td>1.498800</td>\n",
       "      <td>0.620313</td>\n",
       "      <td>0.886114</td>\n",
       "      <td>0.755100</td>\n",
       "      <td>1325.730000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1400</td>\n",
       "      <td>1.498800</td>\n",
       "      <td>0.602780</td>\n",
       "      <td>0.892108</td>\n",
       "      <td>0.754500</td>\n",
       "      <td>1326.704000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1450</td>\n",
       "      <td>1.498800</td>\n",
       "      <td>0.585052</td>\n",
       "      <td>0.890110</td>\n",
       "      <td>0.753900</td>\n",
       "      <td>1327.777000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1500</td>\n",
       "      <td>0.686200</td>\n",
       "      <td>0.574158</td>\n",
       "      <td>0.892108</td>\n",
       "      <td>0.754700</td>\n",
       "      <td>1326.346000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1550</td>\n",
       "      <td>0.686200</td>\n",
       "      <td>0.565199</td>\n",
       "      <td>0.895105</td>\n",
       "      <td>0.754700</td>\n",
       "      <td>1326.308000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1600</td>\n",
       "      <td>0.686200</td>\n",
       "      <td>0.558353</td>\n",
       "      <td>0.900100</td>\n",
       "      <td>0.754500</td>\n",
       "      <td>1326.767000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1650</td>\n",
       "      <td>0.686200</td>\n",
       "      <td>0.551075</td>\n",
       "      <td>0.901099</td>\n",
       "      <td>0.754900</td>\n",
       "      <td>1325.966000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='49' max='49' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [49/49 00:02]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** albert-base-v2 ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at albert-base-v2 were not used when initializing AlbertForSequenceClassification: ['predictions.bias', 'predictions.LayerNorm.weight', 'predictions.LayerNorm.bias', 'predictions.dense.weight', 'predictions.dense.bias', 'predictions.decoder.weight', 'predictions.decoder.bias']\n",
      "- This IS expected if you are initializing AlbertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing AlbertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of AlbertForSequenceClassification were not initialized from the model checkpoint at albert-base-v2 and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Trainer is attempting to log a value of \"{0: 'LABEL_0', 1: 'LABEL_1', 2: 'LABEL_2', 3: 'LABEL_3', 4: 'LABEL_4', 5: 'LABEL_5', 6: 'LABEL_6', 7: 'LABEL_7', 8: 'LABEL_8', 9: 'LABEL_9', 10: 'LABEL_10', 11: 'LABEL_11', 12: 'LABEL_12', 13: 'LABEL_13', 14: 'LABEL_14', 15: 'LABEL_15', 16: 'LABEL_16', 17: 'LABEL_17', 18: 'LABEL_18', 19: 'LABEL_19', 20: 'LABEL_20', 21: 'LABEL_21', 22: 'LABEL_22', 23: 'LABEL_23', 24: 'LABEL_24', 25: 'LABEL_25', 26: 'LABEL_26', 27: 'LABEL_27', 28: 'LABEL_28', 29: 'LABEL_29', 30: 'LABEL_30', 31: 'LABEL_31', 32: 'LABEL_32', 33: 'LABEL_33', 34: 'LABEL_34', 35: 'LABEL_35', 36: 'LABEL_36', 37: 'LABEL_37', 38: 'LABEL_38', 39: 'LABEL_39', 40: 'LABEL_40', 41: 'LABEL_41', 42: 'LABEL_42', 43: 'LABEL_43', 44: 'LABEL_44', 45: 'LABEL_45', 46: 'LABEL_46', 47: 'LABEL_47', 48: 'LABEL_48', 49: 'LABEL_49', 50: 'LABEL_50', 51: 'LABEL_51', 52: 'LABEL_52', 53: 'LABEL_53', 54: 'LABEL_54', 55: 'LABEL_55', 56: 'LABEL_56', 57: 'LABEL_57', 58: 'LABEL_58', 59: 'LABEL_59', 60: 'LABEL_60', 61: 'LABEL_61', 62: 'LABEL_62', 63: 'LABEL_63', 64: 'LABEL_64', 65: 'LABEL_65', 66: 'LABEL_66', 67: 'LABEL_67', 68: 'LABEL_68', 69: 'LABEL_69', 70: 'LABEL_70', 71: 'LABEL_71', 72: 'LABEL_72', 73: 'LABEL_73', 74: 'LABEL_74', 75: 'LABEL_75', 76: 'LABEL_76'}\" for key \"id2label\" as a parameter. MLflow's log_param() only accepts values no longer than 250 characters so we dropped this attribute.\n",
      "Trainer is attempting to log a value of \"{'LABEL_0': 0, 'LABEL_1': 1, 'LABEL_2': 2, 'LABEL_3': 3, 'LABEL_4': 4, 'LABEL_5': 5, 'LABEL_6': 6, 'LABEL_7': 7, 'LABEL_8': 8, 'LABEL_9': 9, 'LABEL_10': 10, 'LABEL_11': 11, 'LABEL_12': 12, 'LABEL_13': 13, 'LABEL_14': 14, 'LABEL_15': 15, 'LABEL_16': 16, 'LABEL_17': 17, 'LABEL_18': 18, 'LABEL_19': 19, 'LABEL_20': 20, 'LABEL_21': 21, 'LABEL_22': 22, 'LABEL_23': 23, 'LABEL_24': 24, 'LABEL_25': 25, 'LABEL_26': 26, 'LABEL_27': 27, 'LABEL_28': 28, 'LABEL_29': 29, 'LABEL_30': 30, 'LABEL_31': 31, 'LABEL_32': 32, 'LABEL_33': 33, 'LABEL_34': 34, 'LABEL_35': 35, 'LABEL_36': 36, 'LABEL_37': 37, 'LABEL_38': 38, 'LABEL_39': 39, 'LABEL_40': 40, 'LABEL_41': 41, 'LABEL_42': 42, 'LABEL_43': 43, 'LABEL_44': 44, 'LABEL_45': 45, 'LABEL_46': 46, 'LABEL_47': 47, 'LABEL_48': 48, 'LABEL_49': 49, 'LABEL_50': 50, 'LABEL_51': 51, 'LABEL_52': 52, 'LABEL_53': 53, 'LABEL_54': 54, 'LABEL_55': 55, 'LABEL_56': 56, 'LABEL_57': 57, 'LABEL_58': 58, 'LABEL_59': 59, 'LABEL_60': 60, 'LABEL_61': 61, 'LABEL_62': 62, 'LABEL_63': 63, 'LABEL_64': 64, 'LABEL_65': 65, 'LABEL_66': 66, 'LABEL_67': 67, 'LABEL_68': 68, 'LABEL_69': 69, 'LABEL_70': 70, 'LABEL_71': 71, 'LABEL_72': 72, 'LABEL_73': 73, 'LABEL_74': 74, 'LABEL_75': 75, 'LABEL_76': 76}\" for key \"label2id\" as a parameter. MLflow's log_param() only accepts values no longer than 250 characters so we dropped this attribute.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='1689' max='1689' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [1689/1689 06:40, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Runtime</th>\n",
       "      <th>Samples Per Second</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.359170</td>\n",
       "      <td>0.012987</td>\n",
       "      <td>2.666800</td>\n",
       "      <td>375.360000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>100</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.305244</td>\n",
       "      <td>0.021978</td>\n",
       "      <td>2.669800</td>\n",
       "      <td>374.940000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>150</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.244161</td>\n",
       "      <td>0.038961</td>\n",
       "      <td>2.670000</td>\n",
       "      <td>374.911000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>200</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.155644</td>\n",
       "      <td>0.055944</td>\n",
       "      <td>2.673300</td>\n",
       "      <td>374.445000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>250</td>\n",
       "      <td>No log</td>\n",
       "      <td>3.990192</td>\n",
       "      <td>0.117882</td>\n",
       "      <td>2.673400</td>\n",
       "      <td>374.431000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>300</td>\n",
       "      <td>No log</td>\n",
       "      <td>3.659245</td>\n",
       "      <td>0.192807</td>\n",
       "      <td>2.672500</td>\n",
       "      <td>374.559000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>350</td>\n",
       "      <td>No log</td>\n",
       "      <td>3.335208</td>\n",
       "      <td>0.267732</td>\n",
       "      <td>2.673400</td>\n",
       "      <td>374.423000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>400</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.964495</td>\n",
       "      <td>0.349650</td>\n",
       "      <td>2.674200</td>\n",
       "      <td>374.317000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>450</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.594313</td>\n",
       "      <td>0.441558</td>\n",
       "      <td>2.675700</td>\n",
       "      <td>374.108000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>500</td>\n",
       "      <td>3.684500</td>\n",
       "      <td>2.198615</td>\n",
       "      <td>0.540460</td>\n",
       "      <td>2.673500</td>\n",
       "      <td>374.412000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>550</td>\n",
       "      <td>3.684500</td>\n",
       "      <td>1.826353</td>\n",
       "      <td>0.628372</td>\n",
       "      <td>2.671600</td>\n",
       "      <td>374.679000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>600</td>\n",
       "      <td>3.684500</td>\n",
       "      <td>1.556708</td>\n",
       "      <td>0.680320</td>\n",
       "      <td>2.673900</td>\n",
       "      <td>374.359000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>650</td>\n",
       "      <td>3.684500</td>\n",
       "      <td>1.297379</td>\n",
       "      <td>0.745255</td>\n",
       "      <td>2.675100</td>\n",
       "      <td>374.189000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>700</td>\n",
       "      <td>3.684500</td>\n",
       "      <td>1.190744</td>\n",
       "      <td>0.771229</td>\n",
       "      <td>2.673600</td>\n",
       "      <td>374.395000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>750</td>\n",
       "      <td>3.684500</td>\n",
       "      <td>0.983538</td>\n",
       "      <td>0.800200</td>\n",
       "      <td>2.672600</td>\n",
       "      <td>374.543000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>800</td>\n",
       "      <td>3.684500</td>\n",
       "      <td>0.895457</td>\n",
       "      <td>0.817183</td>\n",
       "      <td>2.672200</td>\n",
       "      <td>374.598000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>850</td>\n",
       "      <td>3.684500</td>\n",
       "      <td>0.869834</td>\n",
       "      <td>0.812188</td>\n",
       "      <td>2.673900</td>\n",
       "      <td>374.356000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>900</td>\n",
       "      <td>3.684500</td>\n",
       "      <td>0.799584</td>\n",
       "      <td>0.832168</td>\n",
       "      <td>2.675000</td>\n",
       "      <td>374.211000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>950</td>\n",
       "      <td>3.684500</td>\n",
       "      <td>0.705545</td>\n",
       "      <td>0.849151</td>\n",
       "      <td>2.677600</td>\n",
       "      <td>373.844000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1000</td>\n",
       "      <td>1.142500</td>\n",
       "      <td>0.632248</td>\n",
       "      <td>0.860140</td>\n",
       "      <td>2.673900</td>\n",
       "      <td>374.358000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1050</td>\n",
       "      <td>1.142500</td>\n",
       "      <td>0.581242</td>\n",
       "      <td>0.869131</td>\n",
       "      <td>2.673100</td>\n",
       "      <td>374.469000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1100</td>\n",
       "      <td>1.142500</td>\n",
       "      <td>0.575546</td>\n",
       "      <td>0.869131</td>\n",
       "      <td>2.669400</td>\n",
       "      <td>374.989000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1150</td>\n",
       "      <td>1.142500</td>\n",
       "      <td>0.548157</td>\n",
       "      <td>0.880120</td>\n",
       "      <td>2.673900</td>\n",
       "      <td>374.363000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1200</td>\n",
       "      <td>1.142500</td>\n",
       "      <td>0.500336</td>\n",
       "      <td>0.883117</td>\n",
       "      <td>2.673600</td>\n",
       "      <td>374.405000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1250</td>\n",
       "      <td>1.142500</td>\n",
       "      <td>0.510000</td>\n",
       "      <td>0.880120</td>\n",
       "      <td>2.677100</td>\n",
       "      <td>373.912000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1300</td>\n",
       "      <td>1.142500</td>\n",
       "      <td>0.466405</td>\n",
       "      <td>0.884116</td>\n",
       "      <td>2.673700</td>\n",
       "      <td>374.383000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1350</td>\n",
       "      <td>1.142500</td>\n",
       "      <td>0.467148</td>\n",
       "      <td>0.886114</td>\n",
       "      <td>2.673600</td>\n",
       "      <td>374.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1400</td>\n",
       "      <td>1.142500</td>\n",
       "      <td>0.448761</td>\n",
       "      <td>0.890110</td>\n",
       "      <td>2.674300</td>\n",
       "      <td>374.302000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1450</td>\n",
       "      <td>1.142500</td>\n",
       "      <td>0.416922</td>\n",
       "      <td>0.904096</td>\n",
       "      <td>2.673600</td>\n",
       "      <td>374.405000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1500</td>\n",
       "      <td>0.415000</td>\n",
       "      <td>0.419380</td>\n",
       "      <td>0.895105</td>\n",
       "      <td>2.672600</td>\n",
       "      <td>374.539000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1550</td>\n",
       "      <td>0.415000</td>\n",
       "      <td>0.400754</td>\n",
       "      <td>0.898102</td>\n",
       "      <td>2.675200</td>\n",
       "      <td>374.183000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1600</td>\n",
       "      <td>0.415000</td>\n",
       "      <td>0.392787</td>\n",
       "      <td>0.896104</td>\n",
       "      <td>2.675500</td>\n",
       "      <td>374.139000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1650</td>\n",
       "      <td>0.415000</td>\n",
       "      <td>0.384444</td>\n",
       "      <td>0.895105</td>\n",
       "      <td>2.674700</td>\n",
       "      <td>374.250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='49' max='49' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [49/49 00:09]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** albert-large-v2 ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at albert-large-v2 were not used when initializing AlbertForSequenceClassification: ['predictions.bias', 'predictions.LayerNorm.weight', 'predictions.LayerNorm.bias', 'predictions.dense.weight', 'predictions.dense.bias', 'predictions.decoder.weight', 'predictions.decoder.bias']\n",
      "- This IS expected if you are initializing AlbertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing AlbertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of AlbertForSequenceClassification were not initialized from the model checkpoint at albert-large-v2 and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Trainer is attempting to log a value of \"{0: 'LABEL_0', 1: 'LABEL_1', 2: 'LABEL_2', 3: 'LABEL_3', 4: 'LABEL_4', 5: 'LABEL_5', 6: 'LABEL_6', 7: 'LABEL_7', 8: 'LABEL_8', 9: 'LABEL_9', 10: 'LABEL_10', 11: 'LABEL_11', 12: 'LABEL_12', 13: 'LABEL_13', 14: 'LABEL_14', 15: 'LABEL_15', 16: 'LABEL_16', 17: 'LABEL_17', 18: 'LABEL_18', 19: 'LABEL_19', 20: 'LABEL_20', 21: 'LABEL_21', 22: 'LABEL_22', 23: 'LABEL_23', 24: 'LABEL_24', 25: 'LABEL_25', 26: 'LABEL_26', 27: 'LABEL_27', 28: 'LABEL_28', 29: 'LABEL_29', 30: 'LABEL_30', 31: 'LABEL_31', 32: 'LABEL_32', 33: 'LABEL_33', 34: 'LABEL_34', 35: 'LABEL_35', 36: 'LABEL_36', 37: 'LABEL_37', 38: 'LABEL_38', 39: 'LABEL_39', 40: 'LABEL_40', 41: 'LABEL_41', 42: 'LABEL_42', 43: 'LABEL_43', 44: 'LABEL_44', 45: 'LABEL_45', 46: 'LABEL_46', 47: 'LABEL_47', 48: 'LABEL_48', 49: 'LABEL_49', 50: 'LABEL_50', 51: 'LABEL_51', 52: 'LABEL_52', 53: 'LABEL_53', 54: 'LABEL_54', 55: 'LABEL_55', 56: 'LABEL_56', 57: 'LABEL_57', 58: 'LABEL_58', 59: 'LABEL_59', 60: 'LABEL_60', 61: 'LABEL_61', 62: 'LABEL_62', 63: 'LABEL_63', 64: 'LABEL_64', 65: 'LABEL_65', 66: 'LABEL_66', 67: 'LABEL_67', 68: 'LABEL_68', 69: 'LABEL_69', 70: 'LABEL_70', 71: 'LABEL_71', 72: 'LABEL_72', 73: 'LABEL_73', 74: 'LABEL_74', 75: 'LABEL_75', 76: 'LABEL_76'}\" for key \"id2label\" as a parameter. MLflow's log_param() only accepts values no longer than 250 characters so we dropped this attribute.\n",
      "Trainer is attempting to log a value of \"{'LABEL_0': 0, 'LABEL_1': 1, 'LABEL_2': 2, 'LABEL_3': 3, 'LABEL_4': 4, 'LABEL_5': 5, 'LABEL_6': 6, 'LABEL_7': 7, 'LABEL_8': 8, 'LABEL_9': 9, 'LABEL_10': 10, 'LABEL_11': 11, 'LABEL_12': 12, 'LABEL_13': 13, 'LABEL_14': 14, 'LABEL_15': 15, 'LABEL_16': 16, 'LABEL_17': 17, 'LABEL_18': 18, 'LABEL_19': 19, 'LABEL_20': 20, 'LABEL_21': 21, 'LABEL_22': 22, 'LABEL_23': 23, 'LABEL_24': 24, 'LABEL_25': 25, 'LABEL_26': 26, 'LABEL_27': 27, 'LABEL_28': 28, 'LABEL_29': 29, 'LABEL_30': 30, 'LABEL_31': 31, 'LABEL_32': 32, 'LABEL_33': 33, 'LABEL_34': 34, 'LABEL_35': 35, 'LABEL_36': 36, 'LABEL_37': 37, 'LABEL_38': 38, 'LABEL_39': 39, 'LABEL_40': 40, 'LABEL_41': 41, 'LABEL_42': 42, 'LABEL_43': 43, 'LABEL_44': 44, 'LABEL_45': 45, 'LABEL_46': 46, 'LABEL_47': 47, 'LABEL_48': 48, 'LABEL_49': 49, 'LABEL_50': 50, 'LABEL_51': 51, 'LABEL_52': 52, 'LABEL_53': 53, 'LABEL_54': 54, 'LABEL_55': 55, 'LABEL_56': 56, 'LABEL_57': 57, 'LABEL_58': 58, 'LABEL_59': 59, 'LABEL_60': 60, 'LABEL_61': 61, 'LABEL_62': 62, 'LABEL_63': 63, 'LABEL_64': 64, 'LABEL_65': 65, 'LABEL_66': 66, 'LABEL_67': 67, 'LABEL_68': 68, 'LABEL_69': 69, 'LABEL_70': 70, 'LABEL_71': 71, 'LABEL_72': 72, 'LABEL_73': 73, 'LABEL_74': 74, 'LABEL_75': 75, 'LABEL_76': 76}\" for key \"label2id\" as a parameter. MLflow's log_param() only accepts values no longer than 250 characters so we dropped this attribute.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='1689' max='1689' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [1689/1689 20:23, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Runtime</th>\n",
       "      <th>Samples Per Second</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.368546</td>\n",
       "      <td>0.013986</td>\n",
       "      <td>8.373100</td>\n",
       "      <td>119.550000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>100</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.306966</td>\n",
       "      <td>0.039960</td>\n",
       "      <td>8.377100</td>\n",
       "      <td>119.492000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>150</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.267850</td>\n",
       "      <td>0.036963</td>\n",
       "      <td>8.377800</td>\n",
       "      <td>119.482000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>200</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.124138</td>\n",
       "      <td>0.026973</td>\n",
       "      <td>8.374300</td>\n",
       "      <td>119.532000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>250</td>\n",
       "      <td>No log</td>\n",
       "      <td>3.876354</td>\n",
       "      <td>0.105894</td>\n",
       "      <td>8.377200</td>\n",
       "      <td>119.492000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>300</td>\n",
       "      <td>No log</td>\n",
       "      <td>3.503984</td>\n",
       "      <td>0.214785</td>\n",
       "      <td>8.377100</td>\n",
       "      <td>119.492000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>350</td>\n",
       "      <td>No log</td>\n",
       "      <td>3.135200</td>\n",
       "      <td>0.256743</td>\n",
       "      <td>8.374300</td>\n",
       "      <td>119.533000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>400</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.851930</td>\n",
       "      <td>0.313686</td>\n",
       "      <td>8.377700</td>\n",
       "      <td>119.484000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>450</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.501250</td>\n",
       "      <td>0.435564</td>\n",
       "      <td>8.371400</td>\n",
       "      <td>119.574000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>500</td>\n",
       "      <td>3.621500</td>\n",
       "      <td>2.241581</td>\n",
       "      <td>0.451548</td>\n",
       "      <td>8.370400</td>\n",
       "      <td>119.588000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>550</td>\n",
       "      <td>3.621500</td>\n",
       "      <td>2.010210</td>\n",
       "      <td>0.486513</td>\n",
       "      <td>8.372900</td>\n",
       "      <td>119.552000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>600</td>\n",
       "      <td>3.621500</td>\n",
       "      <td>1.775531</td>\n",
       "      <td>0.544456</td>\n",
       "      <td>8.371200</td>\n",
       "      <td>119.577000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>650</td>\n",
       "      <td>3.621500</td>\n",
       "      <td>1.603582</td>\n",
       "      <td>0.612388</td>\n",
       "      <td>8.369200</td>\n",
       "      <td>119.605000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>700</td>\n",
       "      <td>3.621500</td>\n",
       "      <td>1.376550</td>\n",
       "      <td>0.661339</td>\n",
       "      <td>8.373400</td>\n",
       "      <td>119.545000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>750</td>\n",
       "      <td>3.621500</td>\n",
       "      <td>1.207317</td>\n",
       "      <td>0.732268</td>\n",
       "      <td>8.373300</td>\n",
       "      <td>119.546000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>800</td>\n",
       "      <td>3.621500</td>\n",
       "      <td>1.127614</td>\n",
       "      <td>0.749251</td>\n",
       "      <td>8.368300</td>\n",
       "      <td>119.618000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>850</td>\n",
       "      <td>3.621500</td>\n",
       "      <td>1.032580</td>\n",
       "      <td>0.765235</td>\n",
       "      <td>8.372200</td>\n",
       "      <td>119.563000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>900</td>\n",
       "      <td>3.621500</td>\n",
       "      <td>0.925415</td>\n",
       "      <td>0.785215</td>\n",
       "      <td>8.372800</td>\n",
       "      <td>119.554000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>950</td>\n",
       "      <td>3.621500</td>\n",
       "      <td>0.861147</td>\n",
       "      <td>0.802198</td>\n",
       "      <td>8.370300</td>\n",
       "      <td>119.589000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1000</td>\n",
       "      <td>1.344200</td>\n",
       "      <td>0.704539</td>\n",
       "      <td>0.843157</td>\n",
       "      <td>8.376500</td>\n",
       "      <td>119.501000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1050</td>\n",
       "      <td>1.344200</td>\n",
       "      <td>0.695871</td>\n",
       "      <td>0.840160</td>\n",
       "      <td>8.376100</td>\n",
       "      <td>119.507000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1100</td>\n",
       "      <td>1.344200</td>\n",
       "      <td>0.648076</td>\n",
       "      <td>0.850150</td>\n",
       "      <td>8.374000</td>\n",
       "      <td>119.536000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1150</td>\n",
       "      <td>1.344200</td>\n",
       "      <td>0.604716</td>\n",
       "      <td>0.842158</td>\n",
       "      <td>8.373400</td>\n",
       "      <td>119.545000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1200</td>\n",
       "      <td>1.344200</td>\n",
       "      <td>0.538813</td>\n",
       "      <td>0.882118</td>\n",
       "      <td>8.377000</td>\n",
       "      <td>119.494000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1250</td>\n",
       "      <td>1.344200</td>\n",
       "      <td>0.517043</td>\n",
       "      <td>0.876124</td>\n",
       "      <td>8.377200</td>\n",
       "      <td>119.491000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1300</td>\n",
       "      <td>1.344200</td>\n",
       "      <td>0.472383</td>\n",
       "      <td>0.878122</td>\n",
       "      <td>8.372900</td>\n",
       "      <td>119.552000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1350</td>\n",
       "      <td>1.344200</td>\n",
       "      <td>0.465411</td>\n",
       "      <td>0.893107</td>\n",
       "      <td>8.371900</td>\n",
       "      <td>119.567000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1400</td>\n",
       "      <td>1.344200</td>\n",
       "      <td>0.449181</td>\n",
       "      <td>0.894106</td>\n",
       "      <td>8.372500</td>\n",
       "      <td>119.558000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1450</td>\n",
       "      <td>1.344200</td>\n",
       "      <td>0.423572</td>\n",
       "      <td>0.896104</td>\n",
       "      <td>8.370400</td>\n",
       "      <td>119.588000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1500</td>\n",
       "      <td>0.494600</td>\n",
       "      <td>0.415764</td>\n",
       "      <td>0.903097</td>\n",
       "      <td>8.374500</td>\n",
       "      <td>119.530000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1550</td>\n",
       "      <td>0.494600</td>\n",
       "      <td>0.416401</td>\n",
       "      <td>0.900100</td>\n",
       "      <td>8.374100</td>\n",
       "      <td>119.535000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1600</td>\n",
       "      <td>0.494600</td>\n",
       "      <td>0.386878</td>\n",
       "      <td>0.907093</td>\n",
       "      <td>8.370600</td>\n",
       "      <td>119.586000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1650</td>\n",
       "      <td>0.494600</td>\n",
       "      <td>0.376689</td>\n",
       "      <td>0.906094</td>\n",
       "      <td>8.373300</td>\n",
       "      <td>119.547000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='49' max='49' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [49/49 00:30]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** bert-base-uncased ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
      "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "Trainer is attempting to log a value of \"{0: 'LABEL_0', 1: 'LABEL_1', 2: 'LABEL_2', 3: 'LABEL_3', 4: 'LABEL_4', 5: 'LABEL_5', 6: 'LABEL_6', 7: 'LABEL_7', 8: 'LABEL_8', 9: 'LABEL_9', 10: 'LABEL_10', 11: 'LABEL_11', 12: 'LABEL_12', 13: 'LABEL_13', 14: 'LABEL_14', 15: 'LABEL_15', 16: 'LABEL_16', 17: 'LABEL_17', 18: 'LABEL_18', 19: 'LABEL_19', 20: 'LABEL_20', 21: 'LABEL_21', 22: 'LABEL_22', 23: 'LABEL_23', 24: 'LABEL_24', 25: 'LABEL_25', 26: 'LABEL_26', 27: 'LABEL_27', 28: 'LABEL_28', 29: 'LABEL_29', 30: 'LABEL_30', 31: 'LABEL_31', 32: 'LABEL_32', 33: 'LABEL_33', 34: 'LABEL_34', 35: 'LABEL_35', 36: 'LABEL_36', 37: 'LABEL_37', 38: 'LABEL_38', 39: 'LABEL_39', 40: 'LABEL_40', 41: 'LABEL_41', 42: 'LABEL_42', 43: 'LABEL_43', 44: 'LABEL_44', 45: 'LABEL_45', 46: 'LABEL_46', 47: 'LABEL_47', 48: 'LABEL_48', 49: 'LABEL_49', 50: 'LABEL_50', 51: 'LABEL_51', 52: 'LABEL_52', 53: 'LABEL_53', 54: 'LABEL_54', 55: 'LABEL_55', 56: 'LABEL_56', 57: 'LABEL_57', 58: 'LABEL_58', 59: 'LABEL_59', 60: 'LABEL_60', 61: 'LABEL_61', 62: 'LABEL_62', 63: 'LABEL_63', 64: 'LABEL_64', 65: 'LABEL_65', 66: 'LABEL_66', 67: 'LABEL_67', 68: 'LABEL_68', 69: 'LABEL_69', 70: 'LABEL_70', 71: 'LABEL_71', 72: 'LABEL_72', 73: 'LABEL_73', 74: 'LABEL_74', 75: 'LABEL_75', 76: 'LABEL_76'}\" for key \"id2label\" as a parameter. MLflow's log_param() only accepts values no longer than 250 characters so we dropped this attribute.\n",
      "Trainer is attempting to log a value of \"{'LABEL_0': 0, 'LABEL_1': 1, 'LABEL_2': 2, 'LABEL_3': 3, 'LABEL_4': 4, 'LABEL_5': 5, 'LABEL_6': 6, 'LABEL_7': 7, 'LABEL_8': 8, 'LABEL_9': 9, 'LABEL_10': 10, 'LABEL_11': 11, 'LABEL_12': 12, 'LABEL_13': 13, 'LABEL_14': 14, 'LABEL_15': 15, 'LABEL_16': 16, 'LABEL_17': 17, 'LABEL_18': 18, 'LABEL_19': 19, 'LABEL_20': 20, 'LABEL_21': 21, 'LABEL_22': 22, 'LABEL_23': 23, 'LABEL_24': 24, 'LABEL_25': 25, 'LABEL_26': 26, 'LABEL_27': 27, 'LABEL_28': 28, 'LABEL_29': 29, 'LABEL_30': 30, 'LABEL_31': 31, 'LABEL_32': 32, 'LABEL_33': 33, 'LABEL_34': 34, 'LABEL_35': 35, 'LABEL_36': 36, 'LABEL_37': 37, 'LABEL_38': 38, 'LABEL_39': 39, 'LABEL_40': 40, 'LABEL_41': 41, 'LABEL_42': 42, 'LABEL_43': 43, 'LABEL_44': 44, 'LABEL_45': 45, 'LABEL_46': 46, 'LABEL_47': 47, 'LABEL_48': 48, 'LABEL_49': 49, 'LABEL_50': 50, 'LABEL_51': 51, 'LABEL_52': 52, 'LABEL_53': 53, 'LABEL_54': 54, 'LABEL_55': 55, 'LABEL_56': 56, 'LABEL_57': 57, 'LABEL_58': 58, 'LABEL_59': 59, 'LABEL_60': 60, 'LABEL_61': 61, 'LABEL_62': 62, 'LABEL_63': 63, 'LABEL_64': 64, 'LABEL_65': 65, 'LABEL_66': 66, 'LABEL_67': 67, 'LABEL_68': 68, 'LABEL_69': 69, 'LABEL_70': 70, 'LABEL_71': 71, 'LABEL_72': 72, 'LABEL_73': 73, 'LABEL_74': 74, 'LABEL_75': 75, 'LABEL_76': 76}\" for key \"label2id\" as a parameter. MLflow's log_param() only accepts values no longer than 250 characters so we dropped this attribute.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='1689' max='1689' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [1689/1689 07:42, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Runtime</th>\n",
       "      <th>Samples Per Second</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.396853</td>\n",
       "      <td>0.014985</td>\n",
       "      <td>2.295000</td>\n",
       "      <td>436.175000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>100</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.372972</td>\n",
       "      <td>0.019980</td>\n",
       "      <td>2.293300</td>\n",
       "      <td>436.483000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>150</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.224295</td>\n",
       "      <td>0.039960</td>\n",
       "      <td>2.292200</td>\n",
       "      <td>436.701000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>200</td>\n",
       "      <td>No log</td>\n",
       "      <td>4.035959</td>\n",
       "      <td>0.100899</td>\n",
       "      <td>2.292800</td>\n",
       "      <td>436.591000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>250</td>\n",
       "      <td>No log</td>\n",
       "      <td>3.747108</td>\n",
       "      <td>0.222777</td>\n",
       "      <td>2.290100</td>\n",
       "      <td>437.102000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>300</td>\n",
       "      <td>No log</td>\n",
       "      <td>3.381693</td>\n",
       "      <td>0.370629</td>\n",
       "      <td>2.290800</td>\n",
       "      <td>436.974000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>350</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.986459</td>\n",
       "      <td>0.455544</td>\n",
       "      <td>2.289600</td>\n",
       "      <td>437.197000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>400</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.600285</td>\n",
       "      <td>0.571429</td>\n",
       "      <td>2.291300</td>\n",
       "      <td>436.861000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>450</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.234993</td>\n",
       "      <td>0.627373</td>\n",
       "      <td>2.291100</td>\n",
       "      <td>436.902000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>500</td>\n",
       "      <td>3.563900</td>\n",
       "      <td>1.875070</td>\n",
       "      <td>0.705295</td>\n",
       "      <td>2.289100</td>\n",
       "      <td>437.295000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>550</td>\n",
       "      <td>3.563900</td>\n",
       "      <td>1.561354</td>\n",
       "      <td>0.758242</td>\n",
       "      <td>2.290700</td>\n",
       "      <td>436.992000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>600</td>\n",
       "      <td>3.563900</td>\n",
       "      <td>1.322154</td>\n",
       "      <td>0.767233</td>\n",
       "      <td>2.292000</td>\n",
       "      <td>436.731000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>650</td>\n",
       "      <td>3.563900</td>\n",
       "      <td>1.066795</td>\n",
       "      <td>0.808192</td>\n",
       "      <td>2.291200</td>\n",
       "      <td>436.892000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>700</td>\n",
       "      <td>3.563900</td>\n",
       "      <td>0.944397</td>\n",
       "      <td>0.826174</td>\n",
       "      <td>2.291500</td>\n",
       "      <td>436.831000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>750</td>\n",
       "      <td>3.563900</td>\n",
       "      <td>0.799313</td>\n",
       "      <td>0.847153</td>\n",
       "      <td>2.290900</td>\n",
       "      <td>436.941000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>800</td>\n",
       "      <td>3.563900</td>\n",
       "      <td>0.714532</td>\n",
       "      <td>0.871129</td>\n",
       "      <td>2.292100</td>\n",
       "      <td>436.724000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>850</td>\n",
       "      <td>3.563900</td>\n",
       "      <td>0.685885</td>\n",
       "      <td>0.870130</td>\n",
       "      <td>2.292500</td>\n",
       "      <td>436.644000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>900</td>\n",
       "      <td>3.563900</td>\n",
       "      <td>0.610617</td>\n",
       "      <td>0.869131</td>\n",
       "      <td>2.290200</td>\n",
       "      <td>437.074000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>950</td>\n",
       "      <td>3.563900</td>\n",
       "      <td>0.564652</td>\n",
       "      <td>0.877123</td>\n",
       "      <td>2.292500</td>\n",
       "      <td>436.635000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1000</td>\n",
       "      <td>1.000100</td>\n",
       "      <td>0.494191</td>\n",
       "      <td>0.897103</td>\n",
       "      <td>2.290600</td>\n",
       "      <td>437.003000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1050</td>\n",
       "      <td>1.000100</td>\n",
       "      <td>0.448275</td>\n",
       "      <td>0.907093</td>\n",
       "      <td>2.292100</td>\n",
       "      <td>436.714000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1100</td>\n",
       "      <td>1.000100</td>\n",
       "      <td>0.447728</td>\n",
       "      <td>0.903097</td>\n",
       "      <td>2.294000</td>\n",
       "      <td>436.352000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1150</td>\n",
       "      <td>1.000100</td>\n",
       "      <td>0.413032</td>\n",
       "      <td>0.912088</td>\n",
       "      <td>2.292800</td>\n",
       "      <td>436.578000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1200</td>\n",
       "      <td>1.000100</td>\n",
       "      <td>0.371782</td>\n",
       "      <td>0.929071</td>\n",
       "      <td>2.294600</td>\n",
       "      <td>436.235000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1250</td>\n",
       "      <td>1.000100</td>\n",
       "      <td>0.373724</td>\n",
       "      <td>0.916084</td>\n",
       "      <td>2.294100</td>\n",
       "      <td>436.335000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1300</td>\n",
       "      <td>1.000100</td>\n",
       "      <td>0.367473</td>\n",
       "      <td>0.915085</td>\n",
       "      <td>2.292400</td>\n",
       "      <td>436.659000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1350</td>\n",
       "      <td>1.000100</td>\n",
       "      <td>0.361103</td>\n",
       "      <td>0.920080</td>\n",
       "      <td>2.295400</td>\n",
       "      <td>436.092000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1400</td>\n",
       "      <td>1.000100</td>\n",
       "      <td>0.342265</td>\n",
       "      <td>0.924076</td>\n",
       "      <td>2.294200</td>\n",
       "      <td>436.318000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1450</td>\n",
       "      <td>1.000100</td>\n",
       "      <td>0.338041</td>\n",
       "      <td>0.930070</td>\n",
       "      <td>2.294200</td>\n",
       "      <td>436.311000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1500</td>\n",
       "      <td>0.329800</td>\n",
       "      <td>0.329306</td>\n",
       "      <td>0.923077</td>\n",
       "      <td>2.295500</td>\n",
       "      <td>436.066000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1550</td>\n",
       "      <td>0.329800</td>\n",
       "      <td>0.326697</td>\n",
       "      <td>0.926074</td>\n",
       "      <td>2.295700</td>\n",
       "      <td>436.031000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1600</td>\n",
       "      <td>0.329800</td>\n",
       "      <td>0.321414</td>\n",
       "      <td>0.926074</td>\n",
       "      <td>2.294700</td>\n",
       "      <td>436.224000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1650</td>\n",
       "      <td>0.329800</td>\n",
       "      <td>0.316192</td>\n",
       "      <td>0.925075</td>\n",
       "      <td>2.295800</td>\n",
       "      <td>436.009000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='49' max='49' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [49/49 00:08]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from transformers import AutoTokenizer, Trainer, TrainingArguments, AutoModelForSequenceClassification\n",
    "\n",
    "model_ids = [\"prajjwal1/bert-tiny\", \"prajjwal1/bert-mini\", \n",
    "             \"prajjwal1/bert-small\", \"prajjwal1/bert-medium\",\n",
    "             \"albert-base-v2\", \"albert-large-v2\", \"bert-base-uncased\"]\n",
    "\n",
    "accuracies = []\n",
    "for model_id in model_ids:\n",
    "    \n",
    "    print(f\"*** {model_id} ***\")\n",
    "\n",
    "    tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
    "    model = AutoModelForSequenceClassification.from_pretrained(model_id, num_labels=len(label_names))\n",
    "\n",
    "    train_texts_encoded = tokenizer(train_texts, padding=True, truncation=True, return_tensors=\"pt\")\n",
    "    dev_texts_encoded = tokenizer(dev_texts, padding=True, truncation=True, return_tensors=\"pt\")\n",
    "    test_texts_encoded = tokenizer(test_texts, padding=True, truncation=True, return_tensors=\"pt\")\n",
    "    \n",
    "    train_dataset = ClassificationDataset(train_texts_encoded, train_labels)\n",
    "    dev_dataset = ClassificationDataset(dev_texts_encoded, dev_labels)\n",
    "    test_dataset = ClassificationDataset(test_texts_encoded, test_labels)\n",
    "    \n",
    "    training_args = TrainingArguments(\n",
    "        output_dir='./results',\n",
    "        num_train_epochs=3,\n",
    "        per_device_train_batch_size=16,\n",
    "        per_device_eval_batch_size=64,\n",
    "        warmup_steps=int(len(train_dataset)/16),\n",
    "        weight_decay=0.01,\n",
    "        logging_dir='./logs',\n",
    "        evaluation_strategy=\"steps\",\n",
    "        eval_steps=50,\n",
    "        save_steps=50,\n",
    "        save_total_limit=10,\n",
    "        load_best_model_at_end=True,\n",
    "        no_cuda=False\n",
    "    )\n",
    "\n",
    "    trainer = Trainer(\n",
    "        model=model,\n",
    "        args=training_args,\n",
    "        compute_metrics=compute_metrics,\n",
    "        train_dataset=train_dataset,\n",
    "        eval_dataset=dev_dataset,\n",
    "    )\n",
    "\n",
    "    trainer.train()\n",
    "    test_results = trainer.evaluate(test_dataset)\n",
    "    \n",
    "    accuracies.append(test_results[\"eval_accuracy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the results. It's no surprise that BERT-base-uncased emerges as the best model: larger models have some clear advantages to their smaller competitors. At the other end of the spectrum, BERT-tiny and BERT-mini are definitely too small for this task. Still, the four models in the middle prove fairly strong competitors to BERT. The accuracy of BERT-small is around 6% lower than that of BERT, but BERT-medium is less than 3% behind. ALBERT is even closer, with ALBERT-base landing at less than 1.5% below the accuracy of the much larger BERT. In environments where there is not sufficient memory to run a full BERT, ALBERT-base may prove a very effictive solution indeed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                          model\n",
      "prajjwal1/bert-tiny    0.349351\n",
      "prajjwal1/bert-mini    0.631169\n",
      "prajjwal1/bert-small   0.861364\n",
      "prajjwal1/bert-medium  0.892857\n",
      "albert-base-v2         0.907468\n",
      "albert-large-v2        0.904545\n",
      "bert-base-uncased      0.920779\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f461444af40>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAFlCAYAAADPim3FAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAA4BklEQVR4nO3de3RV9Z3//9cnXOVWLGIXGJwEa8Ph5J5ARASCYAkDRA0gKJkaaMGQAVtoKUwFRkErDkx1YEIpFmVsoSNEAlguKkIKNZFc5KAJIfUWIX5dijTyI8RALvv3R2BPAichZAdOoM/HWq519t6f/fm8z9HVvtZnf7I/xrIsAQAAoHn8fF0AAADA9YwwBQAA4ABhCgAAwAHCFAAAgAOEKQAAAAcIUwAAAA609dXAt9xyixUQEOCr4QEAAJosLy/va8uyenq75rMwFRAQoNzcXF8NDwAA0GTGmM8ausZjPgAAAAcIUwAAAA4QpgAAABwgTAEAADhAmAIAAHCAMAUAAOAAYQoAAMABwhQAAIADhCkAAAAHCFMAAAAOEKYAAAAcIEwBAAA4QJgCAABwoK2vCwAA3LhKFhzwdQmN8l82xNcl4AbAzBQAAIADhCkAAAAHCFMAAAAOEKYAAAAcYAE6ADQiNXmvr0to1L+uudfXJQD/8AhTAHAd+89JY31dQqMmBc73dQnAVcdjPgAAAAcIUwAAAA4QpgAAABxgzRQAnyrs5/J1CY2LTfV1BQBaOWamAAAAHCBMAQAAOECYAgAAcIAwBQAA4ABhCgAAwAHCFAAAgAOEKQAAAAcIUwAAAA4QpgAAABzgDejADS7kf0J8XUKjNvm6AABwiJkpAAAABwhTAAAADhCmAAAAHCBMAQAAOECYAgAAcIAwBQAA4ABhCgAAwAHCFAAAgAOEKQAAAAcIUwAAAA4QpgAAABwgTAEAADhAmAIAAHCAMAUAAOAAYQoAAMABwhQAAIADTQpTxpg4Y0yRMeYjY8wCL9dvN8bsM8YcMsa8b4z555YvFQAAoPW5bJgyxrSRlCpptKT+kh42xvS/qNlCSZssy4qQNFnS6pYuFAAAoDVqyszUQEkfWZb1iWVZ5yT9r6T7L2pjSep2/vN3JP2/lisRAACg9WrbhDa3STpe57hEUsxFbZ6U9KYxZrakzpJGtkh1AAAArVxLLUB/WNJ6y7L8Jf2zpD8YYy7p2xgzwxiTa4zJPXHiRAsNDQAA4DtNCVOfS+pT59j//Lm6fixpkyRZlpUlqaOkWy7uyLKstZZlRVuWFd2zZ8/mVQwAANCKNCVM5Ui60xgTaIxpr9oF5tsvanNM0ghJMsa4VBummHoCAAA3vMuGKcuyqiTNkvSGpELV/tVegTFmiTEm/nyzn0uabow5LOlPkpIsy7KuVtEAAACtRVMWoMuyrJ2Sdl50bnGdz0ckDW7Z0gAAAFo/3oAOAADgAGEKAADAAcIUAACAA4QpAAAABwhTAAAADhCmAAAAHCBMAQAAOECYAgAAcIAwBQAA4ABhCgAAwAHCFAAAgAOEKQAAAAcIUwAAAA4QpgAAABwgTAEAADhAmAIAAHCAMAUAAOAAYQoAAMABwhQAAIADhCkAAAAHCFMAAAAOEKYAAAAcIEwBAAA4QJgCAABwgDAFAADgAGEKAADAAcIUAACAA4QpAAAABwhTAAAADhCmAAAAHCBMAQAAOECYAgAAcIAwBQAA4ABhCgAAwAHCFAAAgAOEKQAAAAcIUwAAAA4QpgAAABwgTAEAADhAmAIAAHCAMAUAAOAAYQoAAMABwhQAAIADhCkAAAAHCFMAAAAOEKYAAAAcIEwBAAA4QJgCAABwgDAFAADgAGEKAADAAcIUAACAA4QpAAAAB9r6ugDguvfkd3xdQeMCb/d1BQBwQ2NmCgAAwAHCFAAAgAOEKQAAAAcIUwAAAA4QpgAAABwgTAEAADjQpDBljIkzxhQZYz4yxixooM1DxpgjxpgCY8zGli0TAACgdbrse6aMMW0kpUq6T1KJpBxjzHbLso7UaXOnpH+TNNiyrFJjzK1Xq2AAAIDWpCkzUwMlfWRZ1ieWZZ2T9L+S7r+ozXRJqZZllUqSZVlftWyZAAAArVNTwtRtko7XOS45f66uH0j6gTHmHWPMu8aYOG8dGWNmGGNyjTG5J06caF7FAAAArUhLLUBvK+lOSbGSHpb0ojGm+8WNLMtaa1lWtGVZ0T179myhoQEAAHynKWHqc0l96hz7nz9XV4mk7ZZlVVqW9amkv6k2XAEAANzQmhKmciTdaYwJNMa0lzRZ0vaL2mxV7ayUjDG3qPax3yctVyYAAEDrdNkwZVlWlaRZkt6QVChpk2VZBcaYJcaY+PPN3pB00hhzRNI+SfMsyzp5tYoGAABoLS77agRJsixrp6SdF51bXOezJWnu+X8AAAD+YfAGdAAAAAcIUwAAAA4QpgAAABwgTAEAADhAmAIAAHCAMAUAAOAAYQoAAMABwhQAAIADhCkAAAAHCFMAAAAOEKYAAAAcIEwBAAA4QJgCAABwgDAFAADgAGEKAADAAcIUAACAA4QpAAAABwhTAAAADhCmAAAAHCBMAQAAOECYAgAAcIAwBQAA4ABhCgAAwAHCFAAAgAOEKQAAAAcIUwAAAA4QpgAAABwgTAEAADhAmAIAAHCAMAUAAOAAYQoAAMABwhQAAIADhCkAAAAHCFMAAAAOEKYAAAAcIEwBAAA4QJgCAABwgDAFAADgAGEKAADAAcIUAACAA4QpAAAABwhTAAAADhCmAAAAHCBMAQAAOECYAgAAcIAwBQAA4ABhCgAAwAHCFAAAgAOEKQAAAAcIUwAAAA4QpgAAABwgTAEAADhAmAIAAHCAMAUAAOAAYQoAAMABwhQAAIADhCkAAAAHCFMAAAAONClMGWPijDFFxpiPjDELGmk33hhjGWOiW65EAACA1uuyYcoY00ZSqqTRkvpLetgY099Lu66SfirpYEsXCQAA0Fo1ZWZqoKSPLMv6xLKsc5L+V9L9XtotlfScpIoWrA8AAKBVa0qYuk3S8TrHJefP2YwxkZL6WJa1owVrAwAAaPUcL0A3xvhJ+o2knzeh7QxjTK4xJvfEiRNOhwYAAPC5poSpzyX1qXPsf/7cBV0lBUvKMMYUS7pL0nZvi9Aty1prWVa0ZVnRPXv2bH7VAAAArURTwlSOpDuNMYHGmPaSJkvafuGiZVmnLMu6xbKsAMuyAiS9Kynesqzcq1IxAABAK3LZMGVZVpWkWZLekFQoaZNlWQXGmCXGmPirXSAAAEBr1rYpjSzL2ilp50XnFjfQNtZ5WQAAANcH3oAOAADgAGEKAADAAcIUAACAA4QpAAAABwhTAAAADhCmAAAAHCBMAQAAOECYAgAAcIAwBQAA4ABhCgAAwAHCFAAAgAOEKQAAAAeatNEx4GsBC3b4uoQGFXf0dQUAAF9iZgoAAMABwhQAAIADhCkAAAAHCFMAAAAOsAAdAHDdqukglcV0UnX35s0NnC4sbOGKcL3r2LGj/P391a5duybfQ5gCAFy3ymI66bt33KqbO31Hxpgrvr+9f9erUBWuV5Zl6eTJkyopKVFgYGCT7+MxHwDgulXd3a/ZQQq4mDFGPXr0UEVFxRXdR5gCAFzXCFJoSc3574kwBQCAD61Zs0avvPKKJGnx4sXas2ePo/6efPJJrVixQpK0efNmud1u+fn5KTc395K2UVFROnv2rLp06eJozIyMDGVmZjZ4/de//nW947vvvtvReK0Na6YAADeMH/z3/hbtr3jZmBbpp7q6Wm3atPF6LTk52f68ZMmSFhnvguDgYG3ZskWPPfbYJdc+/fRT3XbbberQoYOjMaqqqpSRkaEuXbo0GJJ+/etf61e/+pV93Fjwuh4xMwUAgAPFxcXq16+fpkyZIpfLpQkTJqi8vFwBAQGaP3++IiMjtXnzZr344osaMGCAwsLCNH78eJWXl0uqP5OUlJSktLQ05eTkKCEhQZK0bds23XTTTTp37pwqKirUt29fSWqwv7pcLpeCgoK81r17927FxcXZx3PmzJHb7daIESN04sQJSdLHH3+suLg4RUVFaciQITp69KhdZ3JysmJiYvTQQw9pzZo1ev755xUeHq4DBw7UG2fBggX69ttvFR4erilTpkiSPROWkZGh2NhYTZgwwf4NLcvS3r179cADD9h9vPXWW3rwwQev7F/MNUSYAgDAoaKiIqWkpKiwsFDdunXT6tWrJUk9evTQe++9p8mTJyshIUE5OTk6fPiwXC6X1q1b12B/ERER8ng8kqQDBw4oODhYOTk5OnjwoGJiYiTpivrzpm6YOnPmjKKjo1VQUKBhw4bpqaeekiTNmDFDq1atUl5enlasWKGUlBT7/pKSEmVmZmrLli1KTk7WnDlz5PF4NGTIkHrjLFu2TDfddJM8Ho82bNhwSR2HDh3SCy+8oCNHjuiTTz7RO++8o+HDh+vo0aN2qHv55Zc1bdq0K/p+1xKP+QAAcKhPnz4aPHiwJCkxMVErV66UJE2aNMluk5+fr4ULF+qbb75RWVmZRo0a1WB/bdu21R133KHCwkJlZ2dr7ty52r9/v6qrq+2wciX9XezcuXMqKSmxZ7n8/PzsWhMTE5WQkKCysjJlZmZq4sSJ9n1nz561P0+cOLHBR5dXYuDAgfL395ckhYeHq7i4WPfcc4/+5V/+RX/84x81depUZWVl2evKWiPCFAAADl38F2AXjjt37myfS0pK0tatWxUWFqb169crIyOj0T6HDh2qXbt2qV27dho5cqSSkpJUXV2t5cuXN6u/ug4cOKB77rmn0e9TU1Oj7t272zNkF6v73eqqrq5WVFSUJCk+Pv6y68Dqrtlq06aNqqqqJElTp07VuHHj1LFjR02cOFFt27beyMJjPgAAHDp27JiysrIkSRs3bvQaVE6fPq1evXqpsrLS6+Ouiw0ZMkQvvPCCBg0apJ49e+rkyZMqKipScHBws/qra/fu3Ro9erR9XFNTo7S0tHr1d+vWTYGBgdq8ebOk2hdaHj582Gt/Xbt21enTpyXVBiKPxyOPx2MHqXbt2qmysvKKauzdu7d69+6tp59+WlOnTr2ie681whQAAA4FBQUpNTVVLpdLpaWlmjlz5iVtli5dqpiYGA0ePFj9+vWrd63uzNaFzzExMfryyy81dOhQSVJoaKhCQkLs6431d0F6err8/f2VlZWlMWPG2I8CMzIyNGzYMLtd586dlZ2dreDgYO3du1eLFy+WJG3YsEHr1q1TWFiY3G63tm3b5nWccePGKT093esCdKl27VVoaKi9AL2ppkyZoj59+sjlcl3RfdeasSzLJwNHR0db3t55AXgTsGCHr0toUHHHR3xdQqNCAm/3dQmN2vRsla9LaNTe2FRfl9CoitLf+LqERk0KnH9V+y+N76Kg27/f7PtbYjuZ4uJijR07Vvn5+c26f/bs2YqMjLQfa82dO1fDhw93XFdDSkpKNH36dO3ateuqjdFSZs2apYiICP34xz++puMWFhZeEuCMMXmWZUV7a8/MFAAAPrJo0SIdPHhQ8fHxmjZtmsrLyxtdy9QS/P39r4sgFRUVpffff1+JiYm+LuWyWu9qLgAArgMBAQHNnpVaunSpli5dKkl66aWXWrKs615eXp6vS2gyZqYAAAAcIEwBAAA4QJgCAABwgDAFAADgAGEKAAAfWrNmjb1VyuLFi7Vnzx5H/dXdOHnz5s1yu93y8/OTt9cRRUVF6ezZs/bGw82VkZGhzMxMR33k5ubq8ccfv2y7u+++29E4VwN/zQcAuGG0/71/y3b45KkW6aa6urrBfeySk5Ptz5fbeuVKBQcHa8uWLXrssccuufbpp5/qtttuq7edS3NUVVUpIyNDXbp0cRR0oqOjFR3t9TVO9TgNbVcDM1MAADhQXFysfv36acqUKXK5XJowYYLKy8sVEBCg+fPnKzIyUps3b9aLL76oAQMGKCwsTOPHj1d5ebmk+jNJSUlJSktLU05OjhISEiRJ27Zt00033aRz586poqLC3py4of7qcrlcCgoK8lr37t27FRcXZx/PmTNHbrdbI0aM0IkTJyRJH3/8seLi4hQVFaUhQ4bo6NGjdp3JycmKiYnRQw89pDVr1uj5559v8A3oXbp00bx58+R2uzVy5EhlZ2crNjZWffv21fbt2yXVzm6NHTvW/k2mTZtmt7mwcfSFvlobwhQAAA4VFRUpJSVFhYWF6tatm1avXi1J6tGjh9577z1NnjxZCQkJysnJ0eHDh+VyubRu3boG+4uIiLA3GD5w4ICCg4OVk5OjgwcPKiYmRpKuqD9v6oapM2fOKDo6WgUFBRo2bJieeuopSbXbwKxatUp5eXlasWKFUlJS7PtLSkqUmZmpLVu2KDk5WXPmzJHH49GQIUMuGevMmTO69957VVBQoK5du2rhwoV66623lJ6ebm9dc7GjR4/qjTfeUHZ2tp566qkr3tvvWuIxHwAADvXp00eDBw+WJCUmJtozKZMmTbLb5Ofna+HChfrmm29UVlZm75PnTdu2bXXHHXeosLBQ2dnZmjt3rvbv36/q6mo7rFxJfxc7d+6cSkpK7FkuPz8/u9bExEQlJCSorKxMmZmZmjhxon3f2bNn7c8TJ05s8NHlxdq3b28Ht5CQEHXo0EHt2rVTSEiIiouLvd4zZswYdejQQR06dNCtt96qL7/8Uv7+LfwYt4UQpgAAcKjuRsV1jzt37myfS0pK0tatWxUWFqb169crIyOj0T6HDh2qXbt2qV27dho5cqSSkpJUXV2t5cuXN6u/ug4cONDotjXGGNXU1Kh79+72DNnF6n63uqqrqxUVFSVJio+P15IlS9SuXTv7N/Hz87PXafn5+amqyvv+nHXXcrVp06bBdq0Bj/kAAHDo2LFjysrKkiRt3LjRa1A5ffq0evXqpcrKSm3YsOGyfQ4ZMkQvvPCCBg0apJ49e+rkyZMqKipScHBws/qra/fu3Ro9erR9XFNTo7S0tHr1d+vWTYGBgdq8ebMkybIsHT582Gt/Xbt21enTpyXVBh+PxyOPx9PiC+pbK8IUAAAOBQUFKTU1VS6XS6WlpZo5c+YlbZYuXaqYmBgNHjxY/fr1q3et7szWhc8xMTH68ssvNXToUElSaGioQkJC7OuN9XdBenq6/P39lZWVpTFjxtiPAjMyMjRs2DC7XefOnZWdna3g4GDt3bvXXse0YcMGrVu3TmFhYXK73dq2bZvXccaNG6f09PQGF6Df6IxlWT4ZODo62vL2zgvAm4AFO3xdQoOKOz7i6xIaFRJ4u69LaNSmZ1vv1L0k7Y1N9XUJjaoo/Y2vS2jUpMD5V7X/0vguCrr9+82+v71/V8c1FBcXa+zYsc3e7Hj27NmKjIzU1KlTNW7cOM2dO1fDhw93XFdDSkpKNH36dO3ateuqjXG9KywslMvlqnfOGJNnWZbXdzcwMwUAgI8sWrRIBw8eVHx8vKZNm6by8vJG1zK1BH9/f4JUC2MBOgAADgQEBDR7Vmrp0qVaunSpJOmll15qybJwDTEzBQAA4ABhCgAAwAHCFAAAgAOEKQAAAAcIUwAA+NCaNWv0yiuvSJIWL16sPXv2OOqv7sbJmzdvltvtlp+fn7y9jigqKkpnz551vHlwRkaGMjMzHfVxpQICAvT1119L8v3mx/w1HwDghhH19t0t2t8Hj37QIv1UV1c3uI9dcnKy/bml3xgeHBysLVu26LHHHrvk2qeffqrbbrut3rYtzVFVVaWMjAx16dJFd9/dsr//9YKZKQAAHCguLla/fv00ZcoUuVwuTZgwQeXl5QoICND8+fMVGRmpzZs368UXX9SAAQMUFham8ePHq7y8XFL9maSkpCSlpaUpJydHCQkJkqRt27bppptu0rlz51RRUWFvTtxQf3W5XC4FBQV5rXv37t325sOSNGfOHLndbo0YMUInTpyQJH388ceKi4tTVFSUhgwZoqNHj9p1JicnKyYmRg899JDWrFmj559/3usb0Kurq5WUlKTg4GCFhITo+eeflyTFxsZqzpw5io6Olsvlsr/znXfeqYULF9r3P/DAA4qKipLb7dbatWuv/F/QNUCYAgDAoaKiIqWkpKiwsFDdunXT6tWrJUk9evTQe++9p8mTJyshIUE5OTk6fPiwXC6X1q1b12B/ERER9gbDBw4cUHBwsHJycnTw4EHFxMRI0hX1503dMHXmzBlFR0eroKBAw4YN01NPPSVJmjFjhlatWqW8vDytWLFCKSkp9v0lJSXKzMzUli1blJycrDlz5sjj8WjIkCH1xvF4PPr888+Vn5+vDz74QFOnTrWvtW/fXrm5uUpOTtb999+v1NRU5efna/369Tp58qSk2vdv5eXlKTc3VytXrrTPtyY85gMAwKE+ffpo8ODBkqTExEStXLlSkjRp0iS7TX5+vhYuXKhvvvlGZWVl9j553rRt21Z33HGHCgsLlZ2drblz52r//v2qrq62w8qV9Hexc+fOqaSkxJ7l8vPzs2tNTExUQkKCysrKlJmZqYkTJ9r3nT171v48ceLEBh9d1tW3b1998sknmj17tsaMGaMf/vCH9rX4+HhJUkhIiNxut3r16mXfc/z4cfXo0UMrV65Uenq6JOn48eP68MMP1aNHjyZ/12uBMAUAgEN1Nyque9y5c2f7XFJSkrZu3aqwsDCtX79eGRkZjfY5dOhQ7dq1S+3atdPIkSOVlJSk6upqLV++vFn91XXgwIFGt60xxqimpkbdu3e3Z8guVve71VVdXa2oqChJtWFpyZIlOnz4sN544w2tWbNGmzZtst/2fmG9lp+fX721W35+fvZarD179igrK0udOnVSbGysKioqmvw9rxUe8wEA4NCxY8eUlZUlSdq4caPXoHL69Gn16tVLlZWV2rBhw2X7HDJkiF544QUNGjRIPXv21MmTJ1VUVKTg4OBm9VfX7t27NXr0aPu4pqZGaWlp9erv1q2bAgMDtXnzZkmSZVk6fPiw1/66du2q06dPS5LatGkjj8cjj8ejJUuW6Ouvv1ZNTY3Gjx+vp59+Wu+9916T6zx16pRuvvlmderUSUePHtW77757Rd/zWiFMAQDgUFBQkFJTU+VyuVRaWqqZM2de0mbp0qWKiYnR4MGD1a9fv3rX6s5sXfgcExOjL7/8UkOHDpUkhYaGKiQkxL7eWH8XpKeny9/fX1lZWRozZoz9KDAjI0PDhg2z23Xu3FnZ2dkKDg7W3r17tXjxYknShg0btG7dOoWFhcntdmvbtm1exxk3bpzS09O9LkD//PPPFRsbq/DwcCUmJurZZ59t+Ie8SFxcnKqqquRyubRgwQLdddddTb73WjKWZV2+kTFxkv5LUhtJv7csa9lF1+dK+omkKkknJE2zLOuzxvqMjo62vL3zAvAmYMEOX5fQoOKOj/i6hEaFBN7u6xIatenZKl+X0Ki9sam+LqFRFaW/8XUJjZoUOP+q9l8a30VBt3+/2fe39+/quIbi4mKNHTu22Zsdz549W5GRkZo6darGjRunuXPnavjw4Y7rakhJSYmmT5+uXbt2XbUxrneFhYVyuVz1zhlj8izLivbW/rIzU8aYNpJSJY2W1F/Sw8aY/hc1OyQp2rKsUElpkv6jGbUDAPAPZdGiRTp48KDi4+M1bdo0lZeXN7qWqSX4+/sTpFpYUx7zDZT0kWVZn1iWdU7S/0q6v24Dy7L2WZZ14QUX70ryb9kyAQBonQICApo9K7V06VJlZ2erR48eeumll/T222+rXbt2LVwhrramhKnbJB2vc1xy/lxDfizJa+Q1xswwxuQaY3IvvBAMAADgetaiC9CNMYmSoiUt93bdsqy1lmVFW5YV3bNnz5YcGgAAwCea8p6pzyX1qXPsf/5cPcaYkZKekDTMsqyzF18HAAC4ETVlZipH0p3GmEBjTHtJkyVtr9vAGBMh6XeS4i3L+qrlywQAAGidLhumLMuqkjRL0huSCiVtsiyrwBizxBgTf77ZckldJG02xniMMdsb6A4AANSxZs0avfLKK5KkxYsXa8+ePY76q7tx8ubNm+V2u+Xn5ydvryOKiorS2bNn1aVLF0djZmRkKDMz01EfTXVhM2hJ+slPfqIjR45ck3Eb06TtZCzL2ilp50XnFtf5PLKF6wIA4Ip9PHJgi/bnOlrYIv1UV1c3uI9dcnKy/XnJkiUtMt4FwcHB2rJlix577LFLrn366ae67bbb6m3j0hwXtn3p0qWL7r77bkd9Xanf//7313S8hvAGdAAAHCguLla/fv00ZcoUuVwuTZgwQeXl5QoICND8+fMVGRmpzZs368UXX9SAAQMUFham8ePHq7y89o1CdWeSLsy65OTkKCEhQZK0bds23XTTTTp37pwqKirszYkb6q8ul8uloKAgr3Xv3r1bcXFx9vGcOXPkdrs1YsQIXfiL+48//lhxcXGKiorSkCFDdPToUbvO5ORkxcTE6KGHHtKaNWv0/PPPe30DuiR16dJF8+bNk9vt1siRI5Wdna3Y2Fj17dtX27fXPsyqrq7WvHnzNGDAAIWGhup3v/udpNptbGbNmqWgoCCNHDlSX331f6uJYmNj7Rm3urNraWlpSkpKsmudOXOm7rrrLvXt21cZGRmaNm2aXC6X3cYpwhQAAA4VFRUpJSVFhYWF6tatm1avXi1J6tGjh9577z1NnjxZCQkJysnJ0eHDh+VyubRu3boG+4uIiLA3GD5w4ICCg4OVk5OjgwcPKiYmRpKuqD9v6oapM2fOKDo6WgUFBRo2bJieeuopSdKMGTO0atUq5eXlacWKFUpJSbHvLykpUWZmprZs2aLk5GTNmTNHHo9HQ4YMuWSsM2fO6N5771VBQYG6du2qhQsX6q233lJ6erq9dc26dev0ne98Rzk5OcrJydGLL76oTz/9VOnp6SoqKtKRI0f0yiuvNOtxYmlpqbKysvT8888rPj5ec+bMUUFBgT744IMGN3K+Ek16zAcAABrWp08fDR48WJKUmJiolStXSpImTZpkt8nPz9fChQv1zTffqKyszN4nz5u2bdvqjjvuUGFhobKzszV37lzt379f1dXVdli5kv4udu7cOZWUlNizXH5+fnatiYmJSkhIUFlZmTIzMzVx4kT7vrNn/++P9SdOnNjgo8uLtW/f3g5uISEh6tChg9q1a6eQkBAVFxdLkt588029//779nqoU6dO6cMPP9T+/fv18MMPq02bNurdu7fuvffeJn/PC8aNGydjjEJCQvS9731PISEhkiS3263i4mKFh4dfcZ91EaYAAHCo7kbFdY87d+5sn0tKStLWrVsVFham9evXKyMjo9E+hw4dql27dqldu3YaOXKkkpKSVF1dreXLlzerv7oOHDjQ6LY1xhjV1NSoe/fuDc7c1P1udVVXVysqKkqSFB8fryVLlqhdu3b2b+Ln52ev0/Lz81NVVe3+nJZladWqVZeEwp076y3ZbrTmCyoqKupdqzte3TVidcd3gsd8AAA4dOzYMWVlZUmSNm7c6DWonD59Wr169VJlZaU2bNhw2T6HDBmiF154QYMGDVLPnj118uRJFRUVKTg4uFn91bV7926NHj3aPq6pqbFnhC7U361bNwUGBmrz5s2SasPO4cOHvfbXtWtXnT59WpLUpk0beTweeTyeK1pQP2rUKP32t79VZWWlJOlvf/ubzpw5o6FDh+rVV19VdXW1vvjiC+3bt8/r/d/73vdUWFiompoapaenN3nclkCYAgDAoaCgIKWmpsrlcqm0tFQzZ868pM3SpUsVExOjwYMHq1+/fvWu1Z1VufA5JiZGX375pYYOHSpJCg0NVUhIiH29sf4uSE9Pl7+/v7KysjRmzBh71icjI0PDhg2z23Xu3FnZ2dkKDg7W3r177XVMGzZs0Lp16xQWFia3261t27Z5HWfcuHFKT09vcAF6U/zkJz9R//79FRkZqeDgYD322GOqqqrSgw8+qDvvvFP9+/fXj370Iw0aNMjr/cuWLdPYsWN19913q1evXs2qobmMZVnXdMALoqOjLW/vvAC8CViww9clNKi44yO+LqFRIYG3+7qERm161vkU+9W0NzbV1yU0qqL0N74uoVGTAudf1f5L47so6PbvN/v+9v5dHddQXFyssWPHNnuz49mzZysyMlJTp07VuHHjNHfuXA0fPtxxXQ0pKSnR9OnTtWuX1210IamwsFAul6veOWNMnmVZ0d7aMzMFAICPLFq0SAcPHlR8fLymTZum8vLyRtcytQR/f3+CVAtjAToAAA4EBAQ0e1Zq6dKlWrp0qSTppZdeasmycA0xMwUAAOAAYQoAAMABwhQAAIADhCkAAAAHCFMAAFwFAQEB+vrrr1VcXGy/aLO5tm7dqiNHjni91hL9O3X8+HENHz5c/fv3l9vt1n/913/5tJ5rjb/mAwDcMF58OqdF+/vXNVe+D1xLq6qq0tatWzV27Fj179/f1+V41bZtW/3nf/6nIiMjdfr0aUVFRem+++5rtfW2NGamAABw6IEHHlBUVJTcbrfWrl17yfWqqipNmTJFLpdLEyZMUHl5uSQpLy9Pw4YNU1RUlEaNGqUvvvhCkhQbG6uf/exnio6O1nPPPaft27dr3rx5Cg8P18cff9zk/pcsWaIBAwYoODhYM2bM0IUXda9cuVL9+/dXaGioJk+eLEk6c+aMpk2bpoEDByoiIsLr287XrFmjefPm2cfr16/XrFmz1KtXL0VGRkqq3VrG5XLp888/d/KTXlcIUwAAOPTSSy8pLy9Pubm5WrlypU6ePFnvelFRkVJSUlRYWKhu3bpp9erVqqys1OzZs5WWlqa8vDxNmzZNTzzxhH3PuXPnlJubqyeeeELx8fFavny5PB6P7rjjjkvG99a/JM2aNUs5OTnKz8/Xt99+qz//+c+SardeOXTokN5//32tWbNGkvTMM8/o3nvvVXZ2tvbt26d58+bpzJkz9cYZP358vX3vXn31VTuMXVBcXKxDhw4pJibGwS96fSFMAQDg0MqVKxUWFqa77rpLx48f14cffljvep8+fTR48GBJUmJiov7617+qqKhI+fn5uu+++xQeHq6nn35aJSUl9j2TJk1q8vje+pekffv2KSYmRiEhIdq7d68KCgok1e7zN2XKFP3xj39U27a1K37efPNNLVu2TOHh4YqNjVVFRYWOHTtWb5yePXuqb9++evfdd3Xy5EkdPXrUHleSysrKNH78eL3wwgvq1q1bk+u/3rFmCgAABzIyMrRnzx5lZWWpU6dOdhCpq+5GxheOLcuS2+1WVlaW1347d+7s9fzx48c1btw4SVJycrLi4uK89l9RUaGUlBTl5uaqT58+evLJJ+26duzYof379+v111/XM888ow8++ECWZem1115TUFBQvb6mTp2qQ4cOqXfv3tq5c6cmT56sTZs2qV+/fnrwwQftsSsrKzV+/HhNmTJFCQkJTfz1bgzMTAEA4MCpU6d08803q1OnTjp69KjefffdS9ocO3bMDk0bN27UPffco6CgIJ04ccI+X1lZac8cXaxr1646ffq0pNpZKI/HI4/Ho+Tk5Ab7vxCcbrnlFpWVlSktLU2SVFNTY//13XPPPadTp06prKxMo0aN0qpVq+x1VYcOHZIkvfzyy/J4PNq5c6ck6cEHH9S2bdv0pz/9yX7EZ1mWfvzjH8vlcmnu3LkOf9HrD2EKAAAH4uLiVFVVJZfLpQULFuiuu+66pE1QUJBSU1PlcrlUWlqqmTNnqn379kpLS9P8+fMVFham8PBwZWZmeh1j8uTJWr58uSIiIrwuQPfWf/fu3TV9+nQFBwdr1KhRGjBggCSpurpaiYmJCgkJUUREhB5//HF1795dixYtUmVlpUJDQ+V2u7Vo0SKvtdx8881yuVz67LPPNHDgQEnSO++8oz/84Q/au3evwsPDFR4eboevfwTmQgK91qKjo63c3FyfjI3rT8CCHb4uoUHFHR/xdQmNCgm83dclNGrTs1W+LqFRe2NTfV1CoypKf+PrEho1KXD+Ve2/NL6Lgm7/frPvb+/ftQWrwY2isLBQLper3jljTJ5lWdHe2jMzBQAA4ABhCgAAwAHCFAAAgAOEKQAAAAcIUwAAAA4QpgAAABwgTAEAcBUEBATo66+/VnFxsYKDgx31tXXrVh05cqTB6126dHHUf0t56623FBUVpZCQEEVFRWnv3r2+LumaYDsZAMANY9XPH27R/n7+6p9btL/mqKqq0tatWzV27Fj179/fcX+WZcmyLPn5tfx8yi233KLXX39dvXv3Vn5+vkaNGqXPP/+8xcdpbZiZAgDAoQceeEBRUVFyu91au3btJderqqo0ZcoUuVwuTZgwQeXl5ZKkvLw8DRs2TFFRURo1apS++OILSVJsbKx+9rOfKTo6Ws8995y2b9+uefPmKTw83Osb0C8oKyvTiBEjFBkZqZCQEG3btk2SVFxcrKCgIP3oRz9ScHCwjh8/rqVLlyooKEj33HOPHn74Ya1YsUKS9PHHHysuLk5RUVEaMmSIjh49esk4kydP1o4d//cy5aSkJKWlpSkiIkK9e/eWJLndbn377bc6e/ZsM3/V6wdhCgAAh1566SXl5eUpNzdXK1eu1MmTJ+tdLyoqUkpKigoLC9WtWzetXr1alZWVmj17ttLS0pSXl6dp06bpiSeesO85d+6ccnNz9cQTTyg+Pl7Lly+Xx+PRHXfc0WAdHTt2VHp6ut577z3t27dPP//5z+299j788EOlpKSooKBAX331lV577TUdPnxYu3btUt0dSWbMmKFVq1YpLy9PK1asUEpKyiXjTJo0SZs2bbLrfPvttzVmzJh6bV577TVFRkaqQ4cOV/6DXmd4zAcAgEMrV65Uenq6JOn48eP68MMP613v06ePBg8eLElKTEzUypUrFRcXp/z8fN13332SavfM69Wrl33PpEmTrrgOy7L0q1/9Svv375efn58+//xzffnll5Kkf/qnf7L3DXznnXd0//33q2PHjurYsaPGjRsnqXZmKzMzUxMnTrT79DazNHr0aP30pz/V2bNntXv3bg0dOlQ33XSTfb2goEDz58/Xm2++ecXf4XpEmAIAwIGMjAzt2bNHWVlZ6tSpk2JjY1VRUVGvjTHmkmPLsuR2u5WVleW1386dO3s9f/z4cTv8JCcnKzk52b62YcMGnThxQnl5eWrXrp0CAgLsWhrqr66amhp1795dHo+n3vnq6mpFRUVJkuLj47VkyRLFxsbqjTfe0KuvvqrJkyfbbUtKSvTggw/qlVdeaXQW7UbCYz4AABw4deqUbr75ZnXq1ElHjx7Vu+++e0mbY8eO2aFp48aNuueeexQUFKQTJ07Y5ysrK1VQUOB1jK5du+r06dOSame5PB6PPB5PvSB1oZZbb71V7dq10759+/TZZ5957W/w4MF6/fXXVVFRobKyMv35z7UL7bt166bAwEBt3rxZUu1M1+HDh9WmTRt7zCVLlkiqnTl7+eWXdeDAAcXFxUmSvvnmG40ZM0bLli2zZ+L+ERCmAABwIC4uTlVVVXK5XFqwYIH9KK2uoKAgpaamyuVyqbS0VDNnzlT79u2Vlpam+fPnKywsTOHh4crMzPQ6xuTJk7V8+XJFREQ0ugB9ypQpys3NVUhIiF555RX169fPa7sBAwYoPj5eoaGhGj16tEJCQvSd73xHUu3s1rp16xQWFia3220vYr/YD3/4Q/3lL3/RyJEj1b59e0nSf//3f+ujjz7SkiVLFB4ervDwcH311VeN/n43AnNhYdq1Fh0dbdVd8AY0JmDBjss38pHijo/4uoRGhQTe7usSGrXp2Spfl9CovbGpvi6hURWlv/F1CY2aFDj/qvZfGt9FQbd/v9n3t/fv2oLVXF/KysrUpUsXlZeXa+jQoVq7dq0iIyN9XVarUFhYKJfLVe+cMSbPsqxob+1ZMwUAwD+gGTNm6MiRI6qoqNCjjz5KkHKAMAUAwD+gjRs3+rqEGwZrpgAAABwgTAEAADhAmAIAAHCAMAUAAOAAYQoAAAeKi4sVHBzc7Pu3bt2qI0eOXJW+byTr16/XrFmzfF2GV/w1HwDghvHVf3tatD//ZUNatL+LVVVVaevWrRo7dqz69+9/VcfC1cPMFAAADlVVVWnKlClyuVyaMGGCysvLlZeXp2HDhikqKkqjRo3SF198IUmKjY3Vz372M0VHR+u5557T9u3bNW/ePIWHh3t9u7m3viVpyZIlGjBggIKDgzVjxgxdeAn3ypUr1b9/f4WGhtp75p05c0bTpk3TwIEDFRER0eBbzWNjY3Xhhdpff/21AgICJNXOCiUkJCguLk533nmnfvnLX9r37N69W5GRkQoLC9OIESMkSdnZ2Ro0aJAiIiJ09913q6ioSFLtBsgDBw5UeHi4QkND7Q2h//jHP9rnH3vsMVVXV0uSXn75Zf3gBz/QwIED9c477zT/X9BVRpgCAMChoqIipaSkqLCwUN26dVNqaqpmz56ttLQ05eXladq0aXriiSfs9ufOnVNubq6eeOIJxcfHa/ny5fJ4PF43Br6479WrV0uSZs2apZycHOXn5+vbb7+199dbtmyZDh06pPfff19r1qyRJD3zzDO69957lZ2drX379mnevHk6c+bMFX1Hj8ejV199VR988IFeffVVHT9+XCdOnND06dP12muv6fDhw/aefv369dOBAwd06NAhLVmyRL/61a8kSWvWrNFPf/pTeTwe5ebmyt/fX4WFhXr11Vf1zjvvyOPxqE2bNtqwYYO++OIL/fu//7veeecd/fWvf23wUWhrwGM+AAAc6tOnj72xb2Jion79618rPz9f9913nySpurpavXr1sttPmjSp2X2vXLlSv/jFL7Rv3z79x3/8h8rLy/X3v/9dbrdb48aNU2hoqKZMmaIHHnhADzzwgCTpzTff1Pbt27VixQpJUkVFhY4dO3bJlimNGTFihL1/X//+/fXZZ5+ptLRUQ4cOVWBgoCTpu9/9rqTaDZcfffRRffjhhzLGqLKyUpI0aNAgPfPMMyopKVFCQoLuvPNOvf3228rLy9OAAQMkSd9++61uvfVWHTx4ULGxserZs6f9m/3tb39rcr3XEmEKAACHjDH1jrt27Sq3262srCyv7Tt37uz1/PHjxzVu3DhJUnJysuLi4i7p2xijiooKpaSkKDc3V3369NGTTz6piooKSdKOHTu0f/9+vf7663rmmWf0wQcfyLIsvfbaawoKCqrX19SpU3Xo0CH17t1bO3fuVNu2bVVTUyNJdn8XdOjQwf7cpk0bVVU1vK/mokWLNHz4cKWnp6u4uFixsbGSpEceeUQxMTHasWOH/vmf/1m/+93vZFmWHn30UT377LP1+ti6dWuD/bc2POYDAMChY8eO2cFp48aNuuuuu3TixAn7XGVlpQoKCrze27VrV50+fVpS7SyUx+ORx+NRcnKy177vueceO+jccsstKisrU1pamiSppqZGx48f1/Dhw/Xcc8/p1KlTKisr06hRo7Rq1Sp7XdWhQ4ck1a5J8ng82rlzpyQpICBAeXl5kmT32Zi77rpL+/fv16effipJ+vvf/y6pdmbqtttuk1S73uqCTz75RH379tXjjz+u+++/X++//75GjBihtLQ0ffXVV3Yfn332mWJiYvSXv/xFJ0+eVGVlpf0IsTUiTAEA4FBQUJBSU1PlcrlUWlpqr5eaP3++wsLCFB4erszMTK/3Tp48WcuXL1dERITXBegX9z1z5kx1795d06dPV3BwsEaNGmU/IquurlZiYqJCQkIUERGhxx9/XN27d9eiRYtUWVmp0NBQud1uLVq0yGstv/jFL/Tb3/5WERER+vrrry/7vXv27Km1a9cqISFBYWFh9uPLX/7yl/q3f/s3RURE1JvB2rRpk4KDgxUeHq78/Hz96Ec/Uv/+/fX000/rhz/8oUJDQ3Xffffpiy++UK9evfTkk09q0KBBGjx48BU9krzWzIWUeq1FR0dbF/5iALicgAU7fF1Cg4o7PuLrEhoVEni7r0to1KZnG35U0BrsjU31dQmNqij9ja9LaNSkwPlXtf/S+C4Kuv37zb6/vX/XFqwGN4rCwsJLwpsxJs+yrGhv7ZmZAgAAcIAwBQAA4ABhCgAAwAHCFADguuartb+4MTXnvyfCFADgutXmmxqVlp8iUKFFWJalkydPqmPHjld0Hy/tBABct7ocLNff9ZVOdL/8n/F70/b0lf2fJm58HTt2lL+//xXd06QwZYyJk/RfktpI+r1lWcsuut5B0iuSoiSdlDTJsqziK6oEAIAr5HdW6ra/vNn3+y+LaMFq8I/qso/5jDFtJKVKGi2pv6SHjTH9L2r2Y0mllmV9X9Lzkp5r6UIBAABao6asmRoo6SPLsj6xLOucpP+VdP9Fbe6X9D/nP6dJGmEu3kwIAADgBtSUMHWbpON1jkvOn/PaxrKsKkmnJPVoiQIBAABas2u6AN0YM0PSjPOHZcaYoms5PnA1tP4p2HxfF9Coi9cMtIBbJDVvNbI3RSNarKt/RL9Q690KShKLUnAl/qmhC00JU59L6lPn2P/8OW9tSowxbSV9R7UL0euxLGutpLVNGBMAmsUYk9vQ/lkAcDU05TFfjqQ7jTGBxpj2kiZL2n5Rm+2SHj3/eYKkvRYv/QAAAP8ALjszZVlWlTFmlqQ3VPtqhJcsyyowxiyRlGtZ1nZJ6yT9wRjzkaS/qzZwAQAA3PAME0gAbiTGmBnnlxQAwDVBmAIAAHCAvfkAAAAcIEwBuKEZY4qNMbc4bQMADSFMAQAAOECYAtDqGGMCjDFHjTHrjTF/M8ZsMMaMNMa8Y4z50Bgz0BjzXWPMVmPM+8aYd40xoefv7WGMedMYU2CM+b3qvFfVGJNojMk2xniMMb87v/coADhCmALQWn1f0n9K6nf+n0ck3SPpF5J+JekpSYcsywo9f/zK+fv+XdJfLctyS0qXdLskGWNckiZJGmxZVrikaklTrtWXAXDjuqbbyQDAFfjUsqwPJMkYUyDpbcuyLGPMB5ICVLu1w3hJsixr7/kZqW6ShkpKOH9+hzGm9Hx/IyRFSco5vw/7TZK+uobfB8ANijAFoLU6W+dzTZ3jGtX+b1flFfZnJP2PZVn/1gK1AYCNx3wArlcHdP4xnTEmVtLXlmX9f5L2q/aRoIwxoyXdfL7925ImGGNuPX/tu8aYBjcuBYCmYmYKwPXqSUkvGWPel1Su/9sf9ClJfzr/aDBT0jFJsizriDFmoaQ3jTF+qp3Z+ldJn13rwgHcWHgDOgAAgAM85gMAAHCAMAUAAOAAYQoAAMABwhQAAIADhCkAAAAHCFMAAAAOEKYAAAAcIEwBAAA48P8DQyP9H2RPolgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "df = pd.DataFrame({\"model\": accuracies}, index=model_ids)\n",
    "\n",
    "print(df)\n",
    "plt.rcParams['figure.figsize'] = (10,6)\n",
    "ax = df.transpose().plot(kind=\"bar\", rot=0)\n",
    "ax.legend(loc=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The huge transformer models that have become so popular in NLP, are often hard to deploy. Sometimes they're too slow to process enormous amounts of data cost-effectively, sometimes they need more memory than is available. In this notebook, we've explored some more efficient transformer models. An evaluation on intent classification shows that they can often hold their own against their larger competitors. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "notebooks",
   "language": "python",
   "name": "notebooks"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}