BertSum_for_Thai_text.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "P1nv6m0Ly-RV"
   },
   "source": [
    "This model shows how to train BertSum model and evaluate the model on ThaiSum dataset. More details on README.md."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uazkRpnAn8Rj"
   },
   "source": [
    "# Install requirements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "QRgsxxsx-yzV"
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "!git clone https://github.com/nakhunchumpolsathien/ThaiSum.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "WgL1f3p_ndIg",
    "outputId": "1babc407-f109-4e61-fa0e-7b5fbba8b920"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-13 14:50:01,979 [MainThread  ] [INFO ]  Set ROUGE home directory to /content/ThaiSum/BertSum/ROUGE-1.5.5.\n",
      "Hit:1 https://cloud.r-project.org/bin/linux/ubuntu bionic-cran40/ InRelease\n",
      "Ign:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease\n",
      "Hit:3 http://ppa.launchpad.net/c2d4u.team/c2d4u4.0+/ubuntu bionic InRelease\n",
      "Hit:4 http://security.ubuntu.com/ubuntu bionic-security InRelease\n",
      "Ign:5 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease\n",
      "Hit:6 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release\n",
      "Hit:7 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release\n",
      "Hit:8 http://archive.ubuntu.com/ubuntu bionic InRelease\n",
      "Hit:9 http://archive.ubuntu.com/ubuntu bionic-updates InRelease\n",
      "Hit:10 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease\n",
      "Hit:11 http://archive.ubuntu.com/ubuntu bionic-backports InRelease\n",
      "Reading package lists... Done\n",
      "Building dependency tree       \n",
      "Reading state information... Done\n",
      "40 packages can be upgraded. Run 'apt list --upgradable' to see them.\n",
      "Reading package lists...\n",
      "Building dependency tree...\n",
      "Reading state information...\n",
      "libxml-parser-perl is already the newest version (2.44-2build3).\n",
      "0 upgraded, 0 newly installed, 0 to remove and 40 not upgraded.\n",
      "/content/ThaiSum/BertSum/ROUGE-1.5.5/data\n",
      "/content/ThaiSum/BertSum/src\n"
     ]
    }
   ],
   "source": [
    "!pip install -q pytorch_transformers\n",
    "!pip install -q tensorboardX\n",
    "!pip install -q pyrouge\n",
    "!pyrouge_set_rouge_path \"/content/ThaiSum/BertSum/ROUGE-1.5.5\"\n",
    "!apt update\n",
    "!apt install -q libxml-parser-perl\n",
    "%cd \"/content/ThaiSum/BertSum/ROUGE-1.5.5/data\"\n",
    "!perl WordNet-2.0-Exceptions/buildExeptionDB.pl ./WordNet-2.0-Exceptions ./smart_common_words.txt ./WordNet-2.0.exc.db\n",
    "%cd '/content/ThaiSum/BertSum/src'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nX9CPmSMoqfP"
   },
   "source": [
    "# Preprocessing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MOq344MRrvBg"
   },
   "source": [
    "\n",
    "## Step 5. Format to PyTorch Files\n",
    "```\n",
    "!python preprocess.py -mode format_to_bert -raw_path \"/content/BertSum/simple_js/thaisum\" -save_path \"/content/BertSum/bert_data/thaisum\"  -lower -n_cpus 1 -log_file ../logs/preprocess.log\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DWNUZayOoutP",
    "outputId": "136328ff-5ba6-421a-c83d-867233864a79"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('train', '/content/ThaiSum/BertSum/simple_js/thaisum.train.0.json', Namespace(dataset='', log_file='/content/ThaiSum/BertSum/logs/thaisum_preprocess.txt', lower=True, map_path='../../data/', max_src_nsents=100, max_src_ntokens_per_sent=200, max_tgt_ntokens=500, min_src_nsents=3, min_src_ntokens_per_sent=5, min_tgt_ntokens=5, mode='format_to_bert', n_cpus=1, pretrained_model='bert', raw_path='/content/ThaiSum/BertSum/simple_js', save_path='/content/ThaiSum/BertSum/bert_data', select_mode='greedy', shard_size=2000, use_bert_basic_tokenizer=False), '/content/ThaiSum/BertSum/bert_data/thaisum.train.0.bert.pt')]\n",
      "[2020-11-13 14:51:14,210 INFO] loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-vocab.txt from cache at /root/.cache/torch/pytorch_transformers/96435fa287fbf7e469185f1062386e05a075cadbf6838b74da22bf64b080bc32.99bcd55fc66f4f3360bc49ba472b940b8dcf223ea6a345deb969d607ca900729\n",
      "[2020-11-13 14:51:14,362 INFO] Processing /content/ThaiSum/BertSum/simple_js/thaisum.train.0.json\n",
      "[2020-11-13 14:51:14,382 INFO] Processed instances 2\n",
      "[2020-11-13 14:51:14,382 INFO] Saving to /content/ThaiSum/BertSum/bert_data/thaisum.train.0.bert.pt\n",
      "[]\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "# Step 5. Format to PyTorch Files\n",
    "!python preprocess.py -mode format_to_bert -raw_path \"/content/ThaiSum/BertSum/simple_js\" -save_path \"/content/ThaiSum/BertSum/bert_data\"  -lower -n_cpus 1 -log_file \"/content/ThaiSum/BertSum/logs/thaisum_preprocess.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JTXP21OZu_us"
   },
   "source": [
    "# Train Model\n",
    "You can train BertSum model on Google Colab with one GPU, especially BertSumExt,  just try to adjust the parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_TqkdYPXvEY0"
   },
   "source": [
    "## Train BertSumExt\n",
    "\n",
    "```\n",
    "python train.py -task ext -mode train -bert_data_path BERT_DATA_PATH -ext_dropout 0.1 -model_path MODEL_PATH -lr 2e-3 -visible_gpus 0,1,2 -report_every 50 -save_checkpoint_steps 1000 -batch_size 3000 -train_steps 50000 -accum_count 2 -log_file ../logs/ext_bert_cnndm -use_interval true -warmup_steps 10000 -max_pos 512\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "iaNS_jfdvHh7",
    "outputId": "8cd99d88-4ed4-4175-b9a1-5e5b527ee447"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2020-11-13 14:51:57,987 INFO] Device ID 0\n",
      "[2020-11-13 14:51:57,987 INFO] Device cuda\n",
      "[2020-11-13 14:51:58,119 INFO] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-config.json from cache at ../temp/45629519f3117b89d89fd9c740073d8e4c1f0a70f9842476185100a8afe715d1.65df3cef028a0c91a7b059e4c404a975ebe6843c71267b67019c0e9cfa8a88f0\n",
      "[2020-11-13 14:51:58,120 INFO] Model config {\n",
      "  \"architectures\": [\n",
      "    \"BertForMaskedLM\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"directionality\": \"bidi\",\n",
      "  \"finetuning_task\": null,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"model_type\": \"bert\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"num_labels\": 2,\n",
      "  \"output_attentions\": false,\n",
      "  \"output_hidden_states\": false,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"pooler_fc_size\": 768,\n",
      "  \"pooler_num_attention_heads\": 12,\n",
      "  \"pooler_num_fc_layers\": 3,\n",
      "  \"pooler_size_per_head\": 128,\n",
      "  \"pooler_type\": \"first_token_transform\",\n",
      "  \"pruned_heads\": {},\n",
      "  \"torchscript\": false,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"vocab_size\": 119547\n",
      "}\n",
      "\n",
      "[2020-11-13 14:51:58,204 INFO] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-pytorch_model.bin from cache at ../temp/5b5b80054cd2c95a946a8e0ce0b93f56326dff9fbda6a6c3e02de3c91c918342.7131dcb754361639a7d5526985f880879c9bfd144b65a0bf50590bddb7de9059\n",
      "[2020-11-13 14:52:33,407 INFO] ExtSummarizer(\n",
      "  (bert): Bert(\n",
      "    (model): BertModel(\n",
      "      (embeddings): BertEmbeddings(\n",
      "        (word_embeddings): Embedding(119547, 768, padding_idx=0)\n",
      "        (position_embeddings): Embedding(512, 768)\n",
      "        (token_type_embeddings): Embedding(2, 768)\n",
      "        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "        (dropout): Dropout(p=0.1, inplace=False)\n",
      "      )\n",
      "      (encoder): BertEncoder(\n",
      "        (layer): ModuleList(\n",
      "          (0): BertLayer(\n",
      "            (attention): BertAttention(\n",
      "              (self): BertSelfAttention(\n",
      "                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "              (output): BertSelfOutput(\n",
      "                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (intermediate): BertIntermediate(\n",
      "              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "            )\n",
      "            (output): BertOutput(\n",
      "              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "            )\n",
      "          )\n",
      "          (1): BertLayer(\n",
      "            (attention): BertAttention(\n",
      "              (self): BertSelfAttention(\n",
      "                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "              (output): BertSelfOutput(\n",
      "                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (intermediate): BertIntermediate(\n",
      "              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "            )\n",
      "            (output): BertOutput(\n",
      "              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "            )\n",
      "          )\n",
      "          (2): BertLayer(\n",
      "            (attention): BertAttention(\n",
      "              (self): BertSelfAttention(\n",
      "                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "              (output): BertSelfOutput(\n",
      "                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (intermediate): BertIntermediate(\n",
      "              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "            )\n",
      "            (output): BertOutput(\n",
      "              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "            )\n",
      "          )\n",
      "          (3): BertLayer(\n",
      "            (attention): BertAttention(\n",
      "              (self): BertSelfAttention(\n",
      "                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "              (output): BertSelfOutput(\n",
      "                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (intermediate): BertIntermediate(\n",
      "              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "            )\n",
      "            (output): BertOutput(\n",
      "              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "            )\n",
      "          )\n",
      "          (4): BertLayer(\n",
      "            (attention): BertAttention(\n",
      "              (self): BertSelfAttention(\n",
      "                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "              (output): BertSelfOutput(\n",
      "                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (intermediate): BertIntermediate(\n",
      "              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "            )\n",
      "            (output): BertOutput(\n",
      "              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "            )\n",
      "          )\n",
      "          (5): BertLayer(\n",
      "            (attention): BertAttention(\n",
      "              (self): BertSelfAttention(\n",
      "                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "              (output): BertSelfOutput(\n",
      "                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (intermediate): BertIntermediate(\n",
      "              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "            )\n",
      "            (output): BertOutput(\n",
      "              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "            )\n",
      "          )\n",
      "          (6): BertLayer(\n",
      "            (attention): BertAttention(\n",
      "              (self): BertSelfAttention(\n",
      "                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "              (output): BertSelfOutput(\n",
      "                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (intermediate): BertIntermediate(\n",
      "              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "            )\n",
      "            (output): BertOutput(\n",
      "              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "            )\n",
      "          )\n",
      "          (7): BertLayer(\n",
      "            (attention): BertAttention(\n",
      "              (self): BertSelfAttention(\n",
      "                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "              (output): BertSelfOutput(\n",
      "                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (intermediate): BertIntermediate(\n",
      "              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "            )\n",
      "            (output): BertOutput(\n",
      "              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "            )\n",
      "          )\n",
      "          (8): BertLayer(\n",
      "            (attention): BertAttention(\n",
      "              (self): BertSelfAttention(\n",
      "                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "              (output): BertSelfOutput(\n",
      "                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (intermediate): BertIntermediate(\n",
      "              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "            )\n",
      "            (output): BertOutput(\n",
      "              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "            )\n",
      "          )\n",
      "          (9): BertLayer(\n",
      "            (attention): BertAttention(\n",
      "              (self): BertSelfAttention(\n",
      "                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "              (output): BertSelfOutput(\n",
      "                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (intermediate): BertIntermediate(\n",
      "              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "            )\n",
      "            (output): BertOutput(\n",
      "              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "            )\n",
      "          )\n",
      "          (10): BertLayer(\n",
      "            (attention): BertAttention(\n",
      "              (self): BertSelfAttention(\n",
      "                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "              (output): BertSelfOutput(\n",
      "                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (intermediate): BertIntermediate(\n",
      "              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "            )\n",
      "            (output): BertOutput(\n",
      "              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "            )\n",
      "          )\n",
      "          (11): BertLayer(\n",
      "            (attention): BertAttention(\n",
      "              (self): BertSelfAttention(\n",
      "                (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "              (output): BertSelfOutput(\n",
      "                (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "                (dropout): Dropout(p=0.1, inplace=False)\n",
      "              )\n",
      "            )\n",
      "            (intermediate): BertIntermediate(\n",
      "              (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "            )\n",
      "            (output): BertOutput(\n",
      "              (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "            )\n",
      "          )\n",
      "        )\n",
      "      )\n",
      "      (pooler): BertPooler(\n",
      "        (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (activation): Tanh()\n",
      "      )\n",
      "    )\n",
      "  )\n",
      "  (ext_layer): ExtTransformerEncoder(\n",
      "    (pos_emb): PositionalEncoding(\n",
      "      (dropout): Dropout(p=0.1, inplace=False)\n",
      "    )\n",
      "    (transformer_inter): ModuleList(\n",
      "      (0): TransformerEncoderLayer(\n",
      "        (self_attn): MultiHeadedAttention(\n",
      "          (linear_keys): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (linear_values): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (linear_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (softmax): Softmax(dim=-1)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "          (final_linear): Linear(in_features=768, out_features=768, bias=True)\n",
      "        )\n",
      "        (feed_forward): PositionwiseFeedForward(\n",
      "          (w_1): Linear(in_features=768, out_features=2048, bias=True)\n",
      "          (w_2): Linear(in_features=2048, out_features=768, bias=True)\n",
      "          (layer_norm): LayerNorm((768,), eps=1e-06, elementwise_affine=True)\n",
      "          (dropout_1): Dropout(p=0.1, inplace=False)\n",
      "          (dropout_2): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (layer_norm): LayerNorm((768,), eps=1e-06, elementwise_affine=True)\n",
      "        (dropout): Dropout(p=0.1, inplace=False)\n",
      "      )\n",
      "      (1): TransformerEncoderLayer(\n",
      "        (self_attn): MultiHeadedAttention(\n",
      "          (linear_keys): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (linear_values): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (linear_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "          (softmax): Softmax(dim=-1)\n",
      "          (dropout): Dropout(p=0.1, inplace=False)\n",
      "          (final_linear): Linear(in_features=768, out_features=768, bias=True)\n",
      "        )\n",
      "        (feed_forward): PositionwiseFeedForward(\n",
      "          (w_1): Linear(in_features=768, out_features=2048, bias=True)\n",
      "          (w_2): Linear(in_features=2048, out_features=768, bias=True)\n",
      "          (layer_norm): LayerNorm((768,), eps=1e-06, elementwise_affine=True)\n",
      "          (dropout_1): Dropout(p=0.1, inplace=False)\n",
      "          (dropout_2): Dropout(p=0.1, inplace=False)\n",
      "        )\n",
      "        (layer_norm): LayerNorm((768,), eps=1e-06, elementwise_affine=True)\n",
      "        (dropout): Dropout(p=0.1, inplace=False)\n",
      "      )\n",
      "    )\n",
      "    (dropout): Dropout(p=0.1, inplace=False)\n",
      "    (layer_norm): LayerNorm((768,), eps=1e-06, elementwise_affine=True)\n",
      "    (wo): Linear(in_features=768, out_features=1, bias=True)\n",
      "    (sigmoid): Sigmoid()\n",
      "  )\n",
      ")\n",
      "gpu_rank 0\n",
      "[2020-11-13 14:52:33,492 INFO] * number of parameters: 188883713\n",
      "[2020-11-13 14:52:33,492 INFO] Start training...\n",
      "[2020-11-13 14:52:33,496 INFO] Loading train dataset from /content/ThaiSum/BertSum/bert_data/thaisum.train.0.bert.pt, number of examples: 2\n",
      "[2020-11-13 14:52:34,101 INFO] Loading train dataset from /content/ThaiSum/BertSum/bert_data/thaisum.train.1.bert.pt, number of examples: 2000\n",
      "[2020-11-13 14:53:09,965 INFO] Step 50/50000; xent: 3.67; lr: 0.0000001;  27 docs/s;     36 sec\n",
      "Traceback (most recent call last):\n",
      "  File \"train.py\", line 146, in <module>\n",
      "    train_ext(args, device_id)\n",
      "  File \"/content/ThaiSum/BertSum/src/train_extractive.py\", line 203, in train_ext\n",
      "    train_single_ext(args, device_id)\n",
      "  File \"/content/ThaiSum/BertSum/src/train_extractive.py\", line 245, in train_single_ext\n",
      "    trainer.train(train_iter_fct, args.train_steps)\n",
      "  File \"/content/ThaiSum/BertSum/src/models/trainer_ext.py\", line 152, in train\n",
      "    report_stats)\n",
      "  File \"/content/ThaiSum/BertSum/src/models/trainer_ext.py\", line 316, in _gradient_accumulation\n",
      "    (loss / loss.numel()).backward()\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/torch/tensor.py\", line 221, in backward\n",
      "    torch.autograd.backward(self, gradient, retain_graph, create_graph)\n",
      "  File \"/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py\", line 132, in backward\n",
      "    allow_unreachable=True)  # allow_unreachable flag\n",
      "KeyboardInterrupt\n"
     ]
    }
   ],
   "source": [
    "!python train.py -task ext -mode train -bert_data_path \"/content/ThaiSum/BertSum/bert_data/thaisum\" -ext_dropout 0.1 -model_path \"/content/ARedSum/model_checkpoint/BertSumExt\" -lr 2e-3 -visible_gpus 0 -report_every 50 -save_checkpoint_steps 1000 -batch_size 3000 -train_steps 50000 -accum_count 2 -log_file ../logs/train_bertsumext_thaisum -use_interval true -warmup_steps 10000 -max_pos 512"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kq5DR8DpvHti"
   },
   "source": [
    "## Train BertSumAbs\n",
    "\n",
    "```\n",
    "python train.py  -task abs -mode train -bert_data_path BERT_DATA_PATH -dec_dropout 0.2  -model_path MODEL_PATH -sep_optim true -lr_bert 0.002 -lr_dec 0.2 -save_checkpoint_steps 2000 -batch_size 140 -train_steps 200000 -report_every 50 -accum_count 5 -use_bert_emb true -use_interval true -warmup_steps_bert 20000 -warmup_steps_dec 10000 -max_pos 512 -visible_gpus 0,1,2,3  -log_file ../logs/abs_bert_cnndm\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FSyboLM4vIJV"
   },
   "source": [
    "## Train BertSumExtAbs\n",
    "```\n",
    "python train.py  -task abs -mode train -bert_data_path BERT_DATA_PATH -dec_dropout 0.2  -model_path MODEL_PATH -sep_optim true -lr_bert 0.002 -lr_dec 0.2 -save_checkpoint_steps 2000 -batch_size 140 -train_steps 200000 -report_every 50 -accum_count 5 -use_bert_emb true -use_interval true -warmup_steps_bert 20000 -warmup_steps_dec 10000 -max_pos 512 -visible_gpus 0,1,2,3 -log_file ../logs/abs_bert_cnndm  -load_from_extractive EXT_CKPT   \n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n5MEj23F1llg"
   },
   "source": [
    "# Evaluate the model \n",
    "You can download trained BertSum models from https://bit.ly/bertsum."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ed6Ub23V5IiC"
   },
   "source": [
    "## Evaluate with ROUGE\n",
    "[Original ROUGE Script](https://github.com/bheinzerling/pyrouge)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mUImanS81rVb",
    "outputId": "78ef7e20-7881-406d-d69e-7c9ceaf61736"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2020-11-13 14:54:03,428 INFO] Loading checkpoint from /content/ThaiSum/BertSum/model_checkpoint/BertsumExt_thaisum.pt\n",
      "Namespace(accum_count=1, alpha=0.95, batch_size=3000, beam_size=5, bert_data_path='/content/ThaiSum/BertSum/bert_data/thaisum', beta1=0.9, beta2=0.999, block_trigram=True, dec_dropout=0.2, dec_ff_size=2048, dec_heads=8, dec_hidden_size=768, dec_layers=6, enc_dropout=0.2, enc_ff_size=512, enc_hidden_size=512, enc_layers=6, encoder='bert', ext_dropout=0.2, ext_ff_size=2048, ext_heads=8, ext_hidden_size=768, ext_layers=2, finetune_bert=True, generator_shard_size=32, gpu_ranks=[0], label_smoothing=0.1, large=False, load_from_extractive='', log_file='../logs/eval_bertsumext', lr=1, lr_bert=0.002, lr_dec=0.002, max_grad_norm=0, max_length=200, max_pos=512, max_tgt_len=140, min_length=50, mode='test', model_path='/content/ThaiSum/BertSum/model_checkpoint/', optim='adam', param_init=0, param_init_glorot=True, recall_eval=False, report_every=1, report_rouge=True, result_path='../results/bertsumext', save_checkpoint_steps=5, seed=666, sep_optim=True, share_emb=False, task='ext', temp_dir='../temp', test_all=False, test_batch_size=500, test_from='/content/ThaiSum/BertSum/model_checkpoint/BertsumExt_thaisum.pt', test_start_from=-1, train_from='', train_steps=1000, use_bert_emb=False, use_interval=True, visible_gpus='0', warmup_steps=8000, warmup_steps_bert=8000, warmup_steps_dec=8000, world_size=1)\n",
      "[2020-11-13 14:54:31,206 INFO] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-config.json from cache at ../temp/45629519f3117b89d89fd9c740073d8e4c1f0a70f9842476185100a8afe715d1.65df3cef028a0c91a7b059e4c404a975ebe6843c71267b67019c0e9cfa8a88f0\n",
      "[2020-11-13 14:54:31,206 INFO] Model config {\n",
      "  \"architectures\": [\n",
      "    \"BertForMaskedLM\"\n",
      "  ],\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"directionality\": \"bidi\",\n",
      "  \"finetuning_task\": null,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"model_type\": \"bert\",\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"num_labels\": 2,\n",
      "  \"output_attentions\": false,\n",
      "  \"output_hidden_states\": false,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"pooler_fc_size\": 768,\n",
      "  \"pooler_num_attention_heads\": 12,\n",
      "  \"pooler_num_fc_layers\": 3,\n",
      "  \"pooler_size_per_head\": 128,\n",
      "  \"pooler_type\": \"first_token_transform\",\n",
      "  \"pruned_heads\": {},\n",
      "  \"torchscript\": false,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"vocab_size\": 119547\n",
      "}\n",
      "\n",
      "[2020-11-13 14:54:31,286 INFO] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-pytorch_model.bin from cache at ../temp/5b5b80054cd2c95a946a8e0ce0b93f56326dff9fbda6a6c3e02de3c91c918342.7131dcb754361639a7d5526985f880879c9bfd144b65a0bf50590bddb7de9059\n",
      "[2020-11-13 14:54:41,293 INFO] Loading test dataset from /content/ThaiSum/BertSum/bert_data/thaisum.test.0.bert.pt, number of examples: 2\n",
      "gpu_rank 0\n",
      "[2020-11-13 14:54:41,297 INFO] * number of parameters: 188883713\n",
      "2\n",
      "2\n",
      "2020-11-13 14:54:41,425 [MainThread  ] [INFO ]  Writing summaries.\n",
      "[2020-11-13 14:54:41,425 INFO] Writing summaries.\n",
      "2020-11-13 14:54:41,426 [MainThread  ] [INFO ]  Processing summaries. Saving system files to ../temp/tmpmwdmu4xf/system and model files to ../temp/tmpmwdmu4xf/model.\n",
      "[2020-11-13 14:54:41,426 INFO] Processing summaries. Saving system files to ../temp/tmpmwdmu4xf/system and model files to ../temp/tmpmwdmu4xf/model.\n",
      "2020-11-13 14:54:41,426 [MainThread  ] [INFO ]  Processing files in ../temp/rouge-tmp-2020-11-13-14-54-41/candidate/.\n",
      "[2020-11-13 14:54:41,426 INFO] Processing files in ../temp/rouge-tmp-2020-11-13-14-54-41/candidate/.\n",
      "2020-11-13 14:54:41,427 [MainThread  ] [INFO ]  Saved processed files to ../temp/tmpmwdmu4xf/system.\n",
      "[2020-11-13 14:54:41,427 INFO] Saved processed files to ../temp/tmpmwdmu4xf/system.\n",
      "2020-11-13 14:54:41,427 [MainThread  ] [INFO ]  Processing files in ../temp/rouge-tmp-2020-11-13-14-54-41/reference/.\n",
      "[2020-11-13 14:54:41,427 INFO] Processing files in ../temp/rouge-tmp-2020-11-13-14-54-41/reference/.\n",
      "2020-11-13 14:54:41,428 [MainThread  ] [INFO ]  Saved processed files to ../temp/tmpmwdmu4xf/model.\n",
      "[2020-11-13 14:54:41,428 INFO] Saved processed files to ../temp/tmpmwdmu4xf/model.\n",
      "2020-11-13 14:54:41,428 [MainThread  ] [INFO ]  Written ROUGE configuration to ../temp/tmpzg9_46o7/rouge_conf.xml\n",
      "[2020-11-13 14:54:41,428 INFO] Written ROUGE configuration to ../temp/tmpzg9_46o7/rouge_conf.xml\n",
      "2020-11-13 14:54:41,428 [MainThread  ] [INFO ]  Running ROUGE with command perl /content/ThaiSum/BertSum/ROUGE-1.5.5/ROUGE-1.5.5.pl -e /content/ThaiSum/BertSum/ROUGE-1.5.5/data -c 95 -m -r 1000 -n 2 -a ../temp/tmpzg9_46o7/rouge_conf.xml\n",
      "[2020-11-13 14:54:41,428 INFO] Running ROUGE with command perl /content/ThaiSum/BertSum/ROUGE-1.5.5/ROUGE-1.5.5.pl -e /content/ThaiSum/BertSum/ROUGE-1.5.5/data -c 95 -m -r 1000 -n 2 -a ../temp/tmpzg9_46o7/rouge_conf.xml\n",
      "../temp/tmpzg9_46o7/rouge_conf.xml---------------------------------------------\n",
      "1 ROUGE-1 Average_R: 0.70000 (95%-conf.int. 0.40000 - 1.00000)\n",
      "1 ROUGE-1 Average_P: 1.00000 (95%-conf.int. 1.00000 - 1.00000)\n",
      "1 ROUGE-1 Average_F: 0.78571 (95%-conf.int. 0.57143 - 1.00000)\n",
      "---------------------------------------------\n",
      "1 ROUGE-2 Average_R: 0.12500 (95%-conf.int. 0.00000 - 0.25000)\n",
      "1 ROUGE-2 Average_P: 0.50000 (95%-conf.int. 0.00000 - 1.00000)\n",
      "1 ROUGE-2 Average_F: 0.20000 (95%-conf.int. 0.00000 - 0.40000)\n",
      "---------------------------------------------\n",
      "1 ROUGE-L Average_R: 0.70000 (95%-conf.int. 0.40000 - 1.00000)\n",
      "1 ROUGE-L Average_P: 1.00000 (95%-conf.int. 1.00000 - 1.00000)\n",
      "1 ROUGE-L Average_F: 0.78571 (95%-conf.int. 0.57143 - 1.00000)\n",
      "\n",
      "[2020-11-13 14:54:41,786 INFO] Rouges at step 0 \n",
      ">> ROUGE-F(1/2/3/l): 78.57/20.00/78.57\n",
      "ROUGE-R(1/2/3/l): 70.00/12.50/70.00\n",
      "\n",
      "[2020-11-13 14:54:41,786 INFO] Validation xent: 1.8466 at step 0\n"
     ]
    }
   ],
   "source": [
    "!python train.py -task ext -mode test -batch_size 3000 -test_batch_size 500 -bert_data_path \"/content/ThaiSum/BertSum/bert_data/thaisum\" -log_file ../logs/eval_bertsumext -model_path \"/content/ThaiSum/BertSum/model_checkpoint/\" -sep_optim true -use_interval true -visible_gpus 0 -max_pos 512 -max_length 200 -alpha 0.95 -min_length 50 -result_path ../results/bertsumext -test_from \"/content/ThaiSum/BertSum/model_checkpoint/BertsumExt_thaisum.pt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_g4jCjwy5V3q"
   },
   "source": [
    "## Evaluate with BertScore "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Yy1v51Cy5h2m",
    "outputId": "2f0f95c2-8068-4b08-ff77-dab262f1402a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[K     |████████████████████████████████| 61kB 3.3MB/s \n",
      "\u001B[K     |████████████████████████████████| 1.3MB 6.1MB/s \n",
      "\u001B[K     |████████████████████████████████| 1.1MB 11.0MB/s \n",
      "\u001B[K     |████████████████████████████████| 2.9MB 21.3MB/s \n",
      "\u001B[?25h"
     ]
    }
   ],
   "source": [
    "!pip install -q bert_score\n",
    "import bert_score\n",
    "from bert_score import score\n",
    "import logging\n",
    "import transformers\n",
    "transformers.tokenization_utils.logger.setLevel(logging.ERROR)\n",
    "transformers.configuration_utils.logger.setLevel(logging.ERROR)\n",
    "transformers.modeling_utils.logger.setLevel(logging.ERROR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "hO2psf1m6s4i"
   },
   "outputs": [],
   "source": [
    "with open(\"/content/ThaiSum/BertSum/results/bertsumext_step0.candidate\") as f: # Output Summary \n",
    "    cands = [line.strip() for line in f]\n",
    "\n",
    "with open(\"/content/ThaiSum/BertSum/results/bertsumext_step0.gold\") as f:  # Reference Summary\n",
    "    refs = [line.strip() for line in f]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 269,
     "referenced_widgets": [
      "13409a6fb2d84d54a531f03020b735f3",
      "223d2f754f634450bd2e210a473f0d08",
      "4abbf3d29fed423e85e9f8d42b560891",
      "044e304672c94032b9e500967e593099",
      "e9ee8e428d034d7fb4898e45ce302e9b",
      "2819aa0a98994bf8bc29a64f6fa98e83",
      "556e2a3576e04c08a47ab1c12347b964",
      "cc04e29e99b14e389d62a752e759866e",
      "852086da86634f91a4f0e90cbf4f814b",
      "5517fb0fbfbd48ec810da182f3d83607",
      "f1cc15688bb34ed5bd57602a5ecc9836",
      "8bf5991bf91c49049009850b8d3df1e1",
      "dce02191221d4b58a38dd67de5391561",
      "0ecc2ef080694370afcff48a7d011e89",
      "2f87e721ea5248209962d4b6bb2445fc",
      "023ed75d6a7a4e10959c9dbcbf8b84b4",
      "f0398a91b9be4595924bc3f1b4fc6aa5",
      "648267140de34a208c728ac1fd8cb7a4",
      "0a8ab4d33e1646cabbc41a0d055d1109",
      "83f274c504c243a994410d64319f9df6",
      "f0bd76b11fc44d25873a24477f1991c2",
      "7ff4756d27d54aaf9c9bbca27954d3f2",
      "afeea3a465164b158ebdfdbfbab54fc3",
      "99d3b5b8e5c94ee9ab49333b794bdcb8"
     ]
    },
    "id": "N5vR045764ee",
    "outputId": "b98dd2e5-26f1-414b-ab91-d7682f1dbac5"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "13409a6fb2d84d54a531f03020b735f3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=625.0, style=ProgressStyle(description_…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "852086da86634f91a4f0e90cbf4f814b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=995526.0, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f0398a91b9be4595924bc3f1b4fc6aa5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=714314041.0, style=ProgressStyle(descri…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "System level F1 score: 92.794\n",
      "System level P score: 94.092\n",
      "System level R score: 91.614\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py:1423: FutureWarning: The `max_len` attribute has been deprecated and will be removed in a future version, use `model_max_length` instead.\n",
      "  FutureWarning,\n"
     ]
    }
   ],
   "source": [
    "P, R, F1 = score(cands, refs, lang='th', verbose=False)\n",
    "\n",
    "print(f\"System level F1 score: {F1.mean()*100:.3f}\") ##  *100 to make it simpler to read similar to ROUGE.\n",
    "print(f\"System level P score: {P.mean()*100:.3f}\")\n",
    "print(f\"System level R score: {R.mean()*100:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sZRTDDsL5iU4"
   },
   "source": [
    "## Show Output Summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "oXnKOhgr5o8d",
    "outputId": "aa1a9d98-cdcf-4c80-906c-d52f8fb9c8ed"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "โควิด - 19 พ่น พิษ การ บิน ไทย ร่อน ไม่ ตรง รันเวย์ ทำ เที่ยวบิน และ รายได้ เดือน<q>มี .ค. - พ.ค. ลดลง 20 % แต่ ยัง ถูกหวย ขาย ที่นั่ง จอง ลอง เลก เพิ่ม รายได้ อย่าง น่าพิศวง\n",
      "\n",
      "เปิดเผย ว่า จาก สถานการณ์ การ แพร่ระบาด ของ เชื้อไวรัส โควิด - 19 ที่ ยัง ไม่ ปรากฏ สัญญาณ ว่า จะ มีแนวโน้ม สิ้นสุด<q>ส่งผล ให้ ธุรกิจ ส่ง ออกดอก กล้วยไม้ รวมถึง กลุ่ม เกษตรกร ผู้ผลิต ได้รับ ผลกระทบ อย่าง หนัก\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open(\"/content/ThaiSum/BertSum/results/bertsumext_step0.candidate\") as f:\n",
    "    for line in f:\n",
    "        print(line)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "BertSum_for_Thai_text.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "023ed75d6a7a4e10959c9dbcbf8b84b4": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "044e304672c94032b9e500967e593099": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_cc04e29e99b14e389d62a752e759866e",
      "placeholder": "​",
      "style": "IPY_MODEL_556e2a3576e04c08a47ab1c12347b964",
      "value": " 625/625 [00:00&lt;00:00, 1.02kB/s]"
     }
    },
    "0a8ab4d33e1646cabbc41a0d055d1109": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7ff4756d27d54aaf9c9bbca27954d3f2",
      "max": 714314041,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_f0bd76b11fc44d25873a24477f1991c2",
      "value": 714314041
     }
    },
    "0ecc2ef080694370afcff48a7d011e89": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "13409a6fb2d84d54a531f03020b735f3": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_4abbf3d29fed423e85e9f8d42b560891",
       "IPY_MODEL_044e304672c94032b9e500967e593099"
      ],
      "layout": "IPY_MODEL_223d2f754f634450bd2e210a473f0d08"
     }
    },
    "223d2f754f634450bd2e210a473f0d08": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2819aa0a98994bf8bc29a64f6fa98e83": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2f87e721ea5248209962d4b6bb2445fc": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4abbf3d29fed423e85e9f8d42b560891": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2819aa0a98994bf8bc29a64f6fa98e83",
      "max": 625,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_e9ee8e428d034d7fb4898e45ce302e9b",
      "value": 625
     }
    },
    "5517fb0fbfbd48ec810da182f3d83607": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "556e2a3576e04c08a47ab1c12347b964": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "648267140de34a208c728ac1fd8cb7a4": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7ff4756d27d54aaf9c9bbca27954d3f2": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "83f274c504c243a994410d64319f9df6": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_99d3b5b8e5c94ee9ab49333b794bdcb8",
      "placeholder": "​",
      "style": "IPY_MODEL_afeea3a465164b158ebdfdbfbab54fc3",
      "value": " 714M/714M [00:16&lt;00:00, 42.4MB/s]"
     }
    },
    "852086da86634f91a4f0e90cbf4f814b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_f1cc15688bb34ed5bd57602a5ecc9836",
       "IPY_MODEL_8bf5991bf91c49049009850b8d3df1e1"
      ],
      "layout": "IPY_MODEL_5517fb0fbfbd48ec810da182f3d83607"
     }
    },
    "8bf5991bf91c49049009850b8d3df1e1": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_023ed75d6a7a4e10959c9dbcbf8b84b4",
      "placeholder": "​",
      "style": "IPY_MODEL_2f87e721ea5248209962d4b6bb2445fc",
      "value": " 996k/996k [00:00&lt;00:00, 3.59MB/s]"
     }
    },
    "99d3b5b8e5c94ee9ab49333b794bdcb8": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "afeea3a465164b158ebdfdbfbab54fc3": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "cc04e29e99b14e389d62a752e759866e": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "dce02191221d4b58a38dd67de5391561": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "e9ee8e428d034d7fb4898e45ce302e9b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "f0398a91b9be4595924bc3f1b4fc6aa5": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_0a8ab4d33e1646cabbc41a0d055d1109",
       "IPY_MODEL_83f274c504c243a994410d64319f9df6"
      ],
      "layout": "IPY_MODEL_648267140de34a208c728ac1fd8cb7a4"
     }
    },
    "f0bd76b11fc44d25873a24477f1991c2": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "f1cc15688bb34ed5bd57602a5ecc9836": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0ecc2ef080694370afcff48a7d011e89",
      "max": 995526,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_dce02191221d4b58a38dd67de5391561",
      "value": 995526
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}