notebooks/joey_v1_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/joeynmt/joeynmt/blob/main/joey_v1_demo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SjX7C7AHX-8F"
   },
   "source": [
    "# Joey NMT v1 Demo\n",
    "\n",
    "In this notebook, we'll train a Transformer model for translating between simple sentences in Esperanto (*epo*) and English (*eng*). You'll have the option to choose your own languages as well.\n",
    "\n",
    "> **Note:** This notebook was initially created for Joey NMT v1.x.  \n",
    "\n",
    "**Important:** Before you start, set runtime type to GPU.\n",
    "\n",
    "Author: Julia Kreutzer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QYRtQZFPYzES"
   },
   "source": [
    "## Installation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y4VofGiF2MMr"
   },
   "source": [
    "Install JoeyNMT v1.5 via [pip](https://pypi.org/project/joeynmt/).\n",
    "\n",
    "It works on PyTorch version 1.10.1. If that's not automatically installed, run the following command to install it.\n",
    "\n",
    "\n",
    "\n",
    "```\n",
    "!pip install torch==1.10.1+cu102 torchtext==0.11.1 -f https://download.pytorch.org/whl/torch_stable.html\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "a4u741jiY2-O",
    "outputId": "b8600721-7da4-48f6-e5ba-488bf3b7899d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[K     |████████████████████████████████| 116 kB 30.2 MB/s \n",
      "\u001b[K     |████████████████████████████████| 505 kB 54.9 MB/s \n",
      "\u001b[K     |████████████████████████████████| 103 kB 49.9 MB/s \n",
      "\u001b[K     |████████████████████████████████| 262 kB 72.9 MB/s \n",
      "\u001b[K     |████████████████████████████████| 843 kB 61.5 MB/s \n",
      "\u001b[K     |████████████████████████████████| 57 kB 6.0 MB/s \n",
      "\u001b[?25h  Building wheel for joeynmt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for wrapt (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
     ]
    }
   ],
   "source": [
    "!pip install -q git+https://github.com/joeynmt/joeynmt.git@1.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HrcFeSdabKHw"
   },
   "source": [
    "# Data Preparation\n",
    "\n",
    "We'll use English - Esperanto translations from the Tatoeba challenge. \n",
    "\n",
    "If you want to use a different language pair, you'll need to replace all instances of `eng` and `epo` language identifiers to your languages of choice. You can find all available languages [here](https://github.com/Helsinki-NLP/Tatoeba-Challenge/blob/master/Data.md). Note that if the training data size is larger, data preparation and training might take much longer than the example with only 400k sentence pairs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dF1998yPmtbS"
   },
   "source": [
    "## Download"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Z6cux99eZ-gW",
    "outputId": "4b02d258-7d3e-4935-e155-d69af32f1f1d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2022-09-04 07:56:20--  https://object.pouta.csc.fi/Tatoeba-Challenge/eng-epo.tar\n",
      "Resolving object.pouta.csc.fi (object.pouta.csc.fi)... 86.50.254.18, 86.50.254.19\n",
      "Connecting to object.pouta.csc.fi (object.pouta.csc.fi)|86.50.254.18|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 35747840 (34M) [application/x-tar]\n",
      "Saving to: ‘eng-epo.tar’\n",
      "\n",
      "eng-epo.tar         100%[===================>]  34.09M  8.18MB/s    in 4.2s    \n",
      "\n",
      "2022-09-04 07:56:27 (8.18 MB/s) - ‘eng-epo.tar’ saved [35747840/35747840]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://object.pouta.csc.fi/Tatoeba-Challenge/eng-epo.tar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "R_kVvMcBbgLg",
    "outputId": "b57db918-55c5-4edd-c908-bf14717c2e0b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data/eng-epo/\n",
      "data/eng-epo/train.src.gz\n",
      "data/eng-epo/dev.trg\n",
      "data/eng-epo/train.id.gz\n",
      "data/eng-epo/test.trg\n",
      "data/eng-epo/test.id\n",
      "data/eng-epo/dev.src\n",
      "data/eng-epo/dev.id\n",
      "data/eng-epo/test.src\n",
      "data/eng-epo/train.trg.gz\n"
     ]
    }
   ],
   "source": [
    "!tar -xvf  'eng-epo.tar'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "WeIGVD2rcfFD"
   },
   "outputs": [],
   "source": [
    "!gunzip 'data/eng-epo/train.src.gz'\n",
    "!gunzip 'data/eng-epo/train.trg.gz'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LBVyCK9aiPcE"
   },
   "source": [
    "We'll only use a subset of dev and test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "yHm_0HlIeshK"
   },
   "outputs": [],
   "source": [
    "!mv 'data/eng-epo/train.src' 'data/eng-epo/train.eng'\n",
    "!head -n 1000 'data/eng-epo/dev.src' > 'data/eng-epo/dev.eng'\n",
    "!head -n 1000 'data/eng-epo/test.src' > 'data/eng-epo/test.eng'\n",
    "!mv 'data/eng-epo/train.trg' 'data/eng-epo/train.epo'\n",
    "!head -n 1000 'data/eng-epo/dev.trg' > 'data/eng-epo/dev.epo'\n",
    "!head -n 1000 'data/eng-epo/test.trg' > 'data/eng-epo/test.epo'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FoveftbNCyv8"
   },
   "source": [
    "The data is sentence-aligned, that means that source and target file contain one sentence per line which correspond to each other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FLNgutcOcHK2",
    "outputId": "c61d5bc4-be58-4113-b7d6-2a0f4b14a699"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I made you something.\n",
      "Twice two is four.\n",
      "That hurts! Stop it!\n",
      "It is too much for me. I need to slow down.\n",
      "I never want to see him again.\n",
      "At what hour was she born?\n",
      "The traffic jam lasted one hour.\n",
      "After dinner, we played cards till eleven.\n",
      "I doubt if he is a lawyer.\n",
      "Who's on duty today?\n"
     ]
    }
   ],
   "source": [
    "! head data/eng-epo/dev.eng"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "BrP5Ujk0cXxA",
    "outputId": "0ded9c4e-a2ef-4b34-f1d0-046fc022f497"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mi faris ion por vi.\n",
      "Du oble du faras kvar.\n",
      "Tio suferigas min! Ĉesu!\n",
      "Estas tro multe por mi. Mi devas malrapidiĝi.\n",
      "Mi volas neniam plu vidi lin.\n",
      "Je kioma horo ŝi naskiĝis?\n",
      "La trafikmalfluo daŭris unu horon.\n",
      "Post la vespermanĝo ni kartludis ĝis la dudek tria.\n",
      "Mi dubas ĉu li estas advokato.\n",
      "Kiu deĵoras hodiaŭ?\n"
     ]
    }
   ],
   "source": [
    "! head data/eng-epo/dev.epo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ydKB8OS1c0r1",
    "outputId": "9be84d9b-c6db-4dda-e344-0197842a0f00"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    1000 data/eng-epo/dev.eng\n",
      "    1000 data/eng-epo/dev.epo\n",
      "  235834 data/eng-epo/dev.id\n",
      "  235834 data/eng-epo/dev.src\n",
      "  235834 data/eng-epo/dev.trg\n",
      "    1000 data/eng-epo/test.eng\n",
      "    1000 data/eng-epo/test.epo\n",
      "   10000 data/eng-epo/test.id\n",
      "   10000 data/eng-epo/test.src\n",
      "   10000 data/eng-epo/test.trg\n",
      "  402180 data/eng-epo/train.eng\n",
      "  402180 data/eng-epo/train.epo\n",
      "    1389 data/eng-epo/train.id.gz\n",
      " 1547251 total\n"
     ]
    }
   ],
   "source": [
    "! wc -l data/eng-epo/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WZg4k1rem1jI"
   },
   "source": [
    "## Subword model training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vJl9sQq22c5Z"
   },
   "source": [
    "We will use the `subword_nmt` library to split words into subwords (BPE) according to their frequency in the training corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "tGDmK7rnqc6r"
   },
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "2wNsekzFd1BD"
   },
   "outputs": [],
   "source": [
    "src_lang = 'epo'\n",
    "trg_lang = 'eng'\n",
    "bpe_size = 4000\n",
    "datadir = '/content/data/eng-epo/'\n",
    "name = f'{src_lang}_{trg_lang}_bpe{bpe_size}'\n",
    "\n",
    "\n",
    "train_src_file = os.path.join(datadir, f'train.{src_lang}')\n",
    "train_trg_file = os.path.join(datadir, f'train.{trg_lang}')\n",
    "train_joint_file = os.path.join(datadir, f'train.{src_lang}-{trg_lang}')\n",
    "dev_src_file = os.path.join(datadir, f'dev.{src_lang}')\n",
    "dev_trg_file = os.path.join(datadir, f'dev.{trg_lang}')\n",
    "test_src_file = os.path.join(datadir, f'test.{src_lang}')\n",
    "test_trg_file = os.path.join(datadir, f'test.{trg_lang}')\n",
    "src_files = {'train': train_src_file, 'dev': dev_src_file, 'test': test_src_file}\n",
    "trg_files = {'train': train_trg_file, 'dev': dev_trg_file, 'test': test_trg_file}\n",
    "\n",
    "\n",
    "vocab_src_file = os.path.join(datadir, f'vocab.{bpe_size}.{src_lang}')\n",
    "vocab_trg_file = os.path.join(datadir, f'vocab.{bpe_size}.{trg_lang}')\n",
    "bpe_file = os.path.join(datadir, f'bpe.codes.{bpe_size}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oJPV_l_G2ny1"
   },
   "source": [
    "Train a BPE model with 4000 symbols for both languages jointly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "eVYh30Mjm3zu",
    "outputId": "79cc6338-93ff-4023-e861-d0519ecb5d6f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100% 4000/4000 [00:24<00:00, 161.94it/s]\n"
     ]
    }
   ],
   "source": [
    "! cat $train_src_file $train_trg_file > $train_joint_file\n",
    "\n",
    "! subword-nmt learn-bpe \\\n",
    "  --input $train_joint_file \\\n",
    "  -s $bpe_size \\\n",
    "  -o $bpe_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QEQjXFqv2u-3"
   },
   "source": [
    "This file contains the merges of character sequences that subwords are made of."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ZtUq1rg1sA8H",
    "outputId": "dc6f3a8b-fbfe-4b69-9069-7aa8dc254c4a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#version: 0.2\n",
      "t h\n",
      "a n\n",
      "o n\n",
      "e r\n",
      "i n\n",
      "e n\n",
      "s t\n",
      "l a</w>\n",
      "o r\n"
     ]
    }
   ],
   "source": [
    "! head $bpe_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tWdUshWa2z3V"
   },
   "source": [
    "We apply the learned BPE merges to training, development and test data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "Qgdj1d3wpA1c"
   },
   "outputs": [],
   "source": [
    "src_bpe_files = {}\n",
    "trg_bpe_files = {}\n",
    "for split in ['train', 'dev', 'test']:\n",
    "  src_input_file = src_files[split]\n",
    "  trg_input_file = trg_files[split]\n",
    "  src_output_file = src_input_file.replace(split, f'{split}.{bpe_size}.bpe')\n",
    "  trg_output_file = trg_input_file.replace(split, f'{split}.{bpe_size}.bpe')\n",
    "  src_bpe_files[split] = src_output_file\n",
    "  trg_bpe_files[split] = trg_output_file\n",
    "\n",
    "  ! subword-nmt apply-bpe \\\n",
    "    -c $bpe_file \\\n",
    "    < $src_input_file > $src_output_file\n",
    "\n",
    "  ! subword-nmt apply-bpe \\\n",
    "    -c $bpe_file \\\n",
    "    < $trg_input_file > $trg_output_file\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2KrKE20C27z3"
   },
   "source": [
    "The subword-split data contains `@@ ` to indicate where words were split into subwords."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SZvrbM5Qx418",
    "outputId": "7d58f24d-d7a0-4a8c-fbf8-d365acd470d2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I made you some@@ thing.\n",
      "T@@ w@@ ice two is f@@ our@@ .\n",
      "That h@@ ur@@ t@@ s! Stop it@@ !\n",
      "It is too much for me. I need to s@@ low dow@@ n.\n",
      "I never want to see him again@@ .\n",
      "A@@ t what h@@ our was she bor@@ n@@ ?\n",
      "The traf@@ fi@@ c jam last@@ ed one h@@ our@@ .\n",
      "Af@@ ter d@@ in@@ n@@ er, we played c@@ ards til@@ l el@@ ev@@ en.\n",
      "I d@@ ou@@ b@@ t if he is a law@@ y@@ er.\n",
      "Wh@@ o@@ 's on d@@ ut@@ y t@@ od@@ ay@@ ?\n"
     ]
    }
   ],
   "source": [
    "! head data/eng-epo/dev.4000.bpe.eng"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "g3uCvTzuyEqo",
    "outputId": "e4c5caa3-8509-40b7-efd9-9b697adee46a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mi faris ion por vi.\n",
      "D@@ u o@@ ble du faras kvar@@ .\n",
      "Tio sufer@@ igas min@@ ! Ĉes@@ u!\n",
      "Estas tro multe por mi. Mi devas mal@@ rapi@@ di@@ ĝ@@ i.\n",
      "Mi volas neniam plu vidi lin.\n",
      "J@@ e ki@@ om@@ a h@@ oro ŝi n@@ aski@@ ĝ@@ is?\n",
      "La tra@@ fik@@ mal@@ fl@@ uo daŭ@@ ris unu hor@@ on.\n",
      "Post la vesper@@ man@@ ĝo ni kart@@ lud@@ is ĝis la dudek tri@@ a.\n",
      "Mi du@@ b@@ as ĉu li estas ad@@ vok@@ at@@ o.\n",
      "Kiu de@@ ĵ@@ or@@ as hodi@@ aŭ@@ ?\n"
     ]
    }
   ],
   "source": [
    "! head data/eng-epo/dev.4000.bpe.epo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7jS8RHLZyKKf"
   },
   "source": [
    "## Prepare the vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ndkGOp7F3LWY"
   },
   "source": [
    "From the pre-processed training data, we extract the final vocabulary for the translation model. It should contain all subwords needed for representing the source and target training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "AGq8KMqjySXm",
    "outputId": "09a3d51b-c143-4ae5-8364-9f77bb8cc296"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2022-09-04 07:57:40--  https://raw.githubusercontent.com/joeynmt/joeynmt/1.5/scripts/build_vocab.py\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2034 (2.0K) [text/plain]\n",
      "Saving to: ‘build_vocab.py’\n",
      "\n",
      "build_vocab.py      100%[===================>]   1.99K  --.-KB/s    in 0s      \n",
      "\n",
      "2022-09-04 07:57:40 (35.6 MB/s) - ‘build_vocab.py’ saved [2034/2034]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "! wget https://raw.githubusercontent.com/joeynmt/joeynmt/1.5/scripts/build_vocab.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "k1_iQEQEyJyS"
   },
   "outputs": [],
   "source": [
    "vocab_src_file = src_bpe_files['train']\n",
    "vocab_trg_file = trg_bpe_files['train']\n",
    "bpe_vocab_file = os.path.join(datadir, f'joint.{bpe_size}bpe.vocab')\n",
    "\n",
    "! python build_vocab.py  \\\n",
    "  $vocab_src_file $vocab_trg_file \\\n",
    "  --output_path $bpe_vocab_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XSuYid3JdECc"
   },
   "source": [
    "# Model configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "45uGP83v3Y24"
   },
   "source": [
    "Joey NMT reads model and training hyperparameters from a configuration file. We're generating this now to configure paths in the appropriate places. \n",
    "\n",
    "The configuration below builds a small Transformer model with shared embeddings between source and target language on the base of the subword vocabularies created above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "EirEmJmkc7sx"
   },
   "outputs": [],
   "source": [
    "# Create the config\n",
    "config = \"\"\"\n",
    "name: \"{name}_transformer\"\n",
    "\n",
    "data:\n",
    "    src: \"{source_language}\"\n",
    "    trg: \"{target_language}\"\n",
    "    train: \"{datadir}/train.{bpe_size}.bpe\"\n",
    "    dev:   \"{datadir}/dev.{bpe_size}.bpe\"\n",
    "    test:  \"{datadir}/test.{bpe_size}.bpe\"\n",
    "    level: \"bpe\"\n",
    "    lowercase: False                \n",
    "    max_sent_length: 30             # Extend to longer sentences.\n",
    "    src_vocab: \"{vocab_src_file}\"\n",
    "    trg_vocab: \"{vocab_trg_file}\"\n",
    "\n",
    "testing:\n",
    "    beam_size: 5\n",
    "    alpha: 1.0\n",
    "    sacrebleu:                      # sacrebleu options\n",
    "        remove_whitespace: True     # `remove_whitespace` option in sacrebleu.corpus_chrf() function (defalut: True)\n",
    "        tokenize: \"intl\"            # `tokenize` option in sacrebleu.corpus_bleu() function (options include: \"none\" (use for already tokenized test data), \"13a\" (default minimal tokenizer), \"intl\" which mostly does punctuation and unicode, etc) \n",
    "\n",
    "training:\n",
    "    #load_model: \"models/{name}_transformer/1.ckpt\" # if uncommented, load a pre-trained model from this checkpoint\n",
    "    random_seed: 42\n",
    "    optimizer: \"adam\"\n",
    "    normalization: \"tokens\"\n",
    "    adam_betas: [0.9, 0.999] \n",
    "    scheduling: \"plateau\"           # Alternative: try switching from plateau to Noam scheduling\n",
    "    patience: 5                     # For plateau: decrease learning rate by decrease_factor if validation score has not improved for this many validation rounds.\n",
    "    learning_rate_factor: 0.5       # factor for Noam scheduler (used with Transformer)\n",
    "    learning_rate_warmup: 1000      # warmup steps for Noam scheduler (used with Transformer)\n",
    "    decrease_factor: 0.7\n",
    "    loss: \"crossentropy\"\n",
    "    learning_rate: 0.0003\n",
    "    learning_rate_min: 0.00000001\n",
    "    weight_decay: 0.0\n",
    "    label_smoothing: 0.1\n",
    "    batch_size: 4096\n",
    "    batch_type: \"token\"\n",
    "    eval_batch_size: 3600\n",
    "    eval_batch_type: \"token\"\n",
    "    batch_multiplier: 1\n",
    "    early_stopping_metric: \"ppl\"\n",
    "    epochs: 30                     # Decrease for when playing around and checking of working. Around 30 is sufficient to check if its working at all\n",
    "    validation_freq: 2000          # Set to at least once per epoch.\n",
    "    logging_freq: 200\n",
    "    eval_metric: \"bleu\"\n",
    "    model_dir: \"models/{name}_transformer\"\n",
    "    overwrite: False               # Set to True if you want to overwrite possibly existing models. \n",
    "    shuffle: True\n",
    "    use_cuda: True\n",
    "    max_output_length: 100\n",
    "    print_valid_sents: [0, 1, 2, 3]\n",
    "    keep_best_ckpts: 3\n",
    "\n",
    "model:\n",
    "    initializer: \"xavier\"\n",
    "    bias_initializer: \"zeros\"\n",
    "    init_gain: 1.0\n",
    "    embed_initializer: \"xavier\"\n",
    "    embed_init_gain: 1.0\n",
    "    tied_embeddings: True       # Requires joint vocabulary.\n",
    "    tied_softmax: True\n",
    "    encoder:\n",
    "        type: \"transformer\"\n",
    "        num_layers: 6\n",
    "        num_heads: 4             # Increase to 8 for larger data.\n",
    "        embeddings:\n",
    "            embedding_dim: 256   # Increase to 512 for larger data.\n",
    "            scale: True\n",
    "            dropout: 0.2\n",
    "        # typically ff_size = 4 x hidden_size\n",
    "        hidden_size: 256         # Increase to 512 for larger data.\n",
    "        ff_size: 1024            # Increase to 2048 for larger data.\n",
    "        dropout: 0.3\n",
    "    decoder:\n",
    "        type: \"transformer\"\n",
    "        num_layers: 6\n",
    "        num_heads: 4              # Increase to 8 for larger data.\n",
    "        embeddings:\n",
    "            embedding_dim: 256    # Increase to 512 for larger data.\n",
    "            scale: True\n",
    "            dropout: 0.2\n",
    "        # typically ff_size = 4 x hidden_size\n",
    "        hidden_size: 256         # TODO: Increase to 512 for larger data.\n",
    "        ff_size: 1024            # TODO: Increase to 2048 for larger data.\n",
    "        dropout: 0.3\n",
    "\"\"\".format(name=name, source_language=src_lang, target_language=trg_lang,\n",
    "           datadir=datadir, vocab_src_file=bpe_vocab_file, \n",
    "           vocab_trg_file=bpe_vocab_file, bpe_size=bpe_size)\n",
    "with open(\"transformer_{name}.yaml\".format(name=name),'w') as f:\n",
    "    f.write(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pIOosBx1fDIQ"
   },
   "source": [
    "# Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "D20-6ecg4PvC"
   },
   "source": [
    "This will take a while. The log reports the training process, look out for the prints of example translations and the BLEU evaluation scores to get an impression of the current quality. \n",
    "\n",
    "The log is also stored in the model directory within this runtime (inspect files in the menu on the left). There you can also find a summary report of all validations. We'll also use TensorBoard to visualize the training progress on the go. This requires enabling Cookies in the browser.\n",
    "\n",
    "After 12h at the latest, Colab will disconnect, so to make sure you're progress is not lost, download the checkpoints from the model directory from time to time. You'll later be able to reload them if model hyperparameters match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "xGzLfqzQEqq9"
   },
   "outputs": [],
   "source": [
    "# Load the TensorBoard notebook extension. It will be empty at first.\n",
    "%load_ext tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9QYsZ513DuFi"
   },
   "outputs": [],
   "source": [
    "%tensorboard --logdir models/epo_eng_bpe4000_transformer/tensorboard "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FF9do6ohedY6",
    "outputId": "4b005797-6118-432a-8abe-9f973ff89d53"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-09-04 08:04:38,110 - INFO - root - Hello! This is Joey-NMT (version 1.5).\n",
      "2022-09-04 08:04:38,123 - INFO - joeynmt.data - Loading training data...\n",
      "2022-09-04 08:04:43,183 - INFO - joeynmt.data - Building vocabulary...\n",
      "2022-09-04 08:04:43,497 - INFO - joeynmt.data - Loading dev data...\n",
      "2022-09-04 08:04:43,522 - INFO - joeynmt.data - Loading test data...\n",
      "2022-09-04 08:04:44,235 - INFO - joeynmt.data - Data loaded.\n",
      "2022-09-04 08:04:44,235 - INFO - joeynmt.model - Building an encoder-decoder model...\n",
      "2022-09-04 08:04:44,434 - INFO - joeynmt.model - Enc-dec model built.\n",
      "2022-09-04 08:04:48,423 - INFO - joeynmt.training - Total params: 12223488\n",
      "2022-09-04 08:04:52,040 - INFO - joeynmt.helpers -                           cfg.name : epo_eng_bpe4000_transformer\n",
      "2022-09-04 08:04:52,040 - INFO - joeynmt.helpers -                       cfg.data.src : epo\n",
      "2022-09-04 08:04:52,040 - INFO - joeynmt.helpers -                       cfg.data.trg : eng\n",
      "2022-09-04 08:04:52,040 - INFO - joeynmt.helpers -                     cfg.data.train : /content/data/eng-epo//train.4000.bpe\n",
      "2022-09-04 08:04:52,040 - INFO - joeynmt.helpers -                       cfg.data.dev : /content/data/eng-epo//dev.4000.bpe\n",
      "2022-09-04 08:04:52,040 - INFO - joeynmt.helpers -                      cfg.data.test : /content/data/eng-epo//test.4000.bpe\n",
      "2022-09-04 08:04:52,040 - INFO - joeynmt.helpers -                     cfg.data.level : bpe\n",
      "2022-09-04 08:04:52,040 - INFO - joeynmt.helpers -                 cfg.data.lowercase : False\n",
      "2022-09-04 08:04:52,040 - INFO - joeynmt.helpers -           cfg.data.max_sent_length : 30\n",
      "2022-09-04 08:04:52,040 - INFO - joeynmt.helpers -                 cfg.data.src_vocab : /content/data/eng-epo/joint.4000bpe.vocab\n",
      "2022-09-04 08:04:52,040 - INFO - joeynmt.helpers -                 cfg.data.trg_vocab : /content/data/eng-epo/joint.4000bpe.vocab\n",
      "2022-09-04 08:04:52,040 - INFO - joeynmt.helpers -              cfg.testing.beam_size : 5\n",
      "2022-09-04 08:04:52,040 - INFO - joeynmt.helpers -                  cfg.testing.alpha : 1.0\n",
      "2022-09-04 08:04:52,040 - INFO - joeynmt.helpers - cfg.testing.sacrebleu.remove_whitespace : True\n",
      "2022-09-04 08:04:52,041 - INFO - joeynmt.helpers -     cfg.testing.sacrebleu.tokenize : intl\n",
      "2022-09-04 08:04:52,041 - INFO - joeynmt.helpers -           cfg.training.random_seed : 42\n",
      "2022-09-04 08:04:52,041 - INFO - joeynmt.helpers -             cfg.training.optimizer : adam\n",
      "2022-09-04 08:04:52,041 - INFO - joeynmt.helpers -         cfg.training.normalization : tokens\n",
      "2022-09-04 08:04:52,041 - INFO - joeynmt.helpers -            cfg.training.adam_betas : [0.9, 0.999]\n",
      "2022-09-04 08:04:52,041 - INFO - joeynmt.helpers -            cfg.training.scheduling : plateau\n",
      "2022-09-04 08:04:52,041 - INFO - joeynmt.helpers -              cfg.training.patience : 5\n",
      "2022-09-04 08:04:52,041 - INFO - joeynmt.helpers -  cfg.training.learning_rate_factor : 0.5\n",
      "2022-09-04 08:04:52,041 - INFO - joeynmt.helpers -  cfg.training.learning_rate_warmup : 1000\n",
      "2022-09-04 08:04:52,041 - INFO - joeynmt.helpers -       cfg.training.decrease_factor : 0.7\n",
      "2022-09-04 08:04:52,041 - INFO - joeynmt.helpers -                  cfg.training.loss : crossentropy\n",
      "2022-09-04 08:04:52,041 - INFO - joeynmt.helpers -         cfg.training.learning_rate : 0.0003\n",
      "2022-09-04 08:04:52,041 - INFO - joeynmt.helpers -     cfg.training.learning_rate_min : 1e-08\n",
      "2022-09-04 08:04:52,041 - INFO - joeynmt.helpers -          cfg.training.weight_decay : 0.0\n",
      "2022-09-04 08:04:52,041 - INFO - joeynmt.helpers -       cfg.training.label_smoothing : 0.1\n",
      "2022-09-04 08:04:52,041 - INFO - joeynmt.helpers -            cfg.training.batch_size : 4096\n",
      "2022-09-04 08:04:52,041 - INFO - joeynmt.helpers -            cfg.training.batch_type : token\n",
      "2022-09-04 08:04:52,041 - INFO - joeynmt.helpers -       cfg.training.eval_batch_size : 3600\n",
      "2022-09-04 08:04:52,042 - INFO - joeynmt.helpers -       cfg.training.eval_batch_type : token\n",
      "2022-09-04 08:04:52,042 - INFO - joeynmt.helpers -      cfg.training.batch_multiplier : 1\n",
      "2022-09-04 08:04:52,042 - INFO - joeynmt.helpers - cfg.training.early_stopping_metric : ppl\n",
      "2022-09-04 08:04:52,042 - INFO - joeynmt.helpers -                cfg.training.epochs : 30\n",
      "2022-09-04 08:04:52,042 - INFO - joeynmt.helpers -       cfg.training.validation_freq : 2000\n",
      "2022-09-04 08:04:52,042 - INFO - joeynmt.helpers -          cfg.training.logging_freq : 200\n",
      "2022-09-04 08:04:52,042 - INFO - joeynmt.helpers -           cfg.training.eval_metric : bleu\n",
      "2022-09-04 08:04:52,042 - INFO - joeynmt.helpers -             cfg.training.model_dir : models/epo_eng_bpe4000_transformer\n",
      "2022-09-04 08:04:52,042 - INFO - joeynmt.helpers -             cfg.training.overwrite : False\n",
      "2022-09-04 08:04:52,042 - INFO - joeynmt.helpers -               cfg.training.shuffle : True\n",
      "2022-09-04 08:04:52,042 - INFO - joeynmt.helpers -              cfg.training.use_cuda : True\n",
      "2022-09-04 08:04:52,042 - INFO - joeynmt.helpers -     cfg.training.max_output_length : 100\n",
      "2022-09-04 08:04:52,042 - INFO - joeynmt.helpers -     cfg.training.print_valid_sents : [0, 1, 2, 3]\n",
      "2022-09-04 08:04:52,042 - INFO - joeynmt.helpers -       cfg.training.keep_best_ckpts : 3\n",
      "2022-09-04 08:04:52,042 - INFO - joeynmt.helpers -              cfg.model.initializer : xavier\n",
      "2022-09-04 08:04:52,042 - INFO - joeynmt.helpers -         cfg.model.bias_initializer : zeros\n",
      "2022-09-04 08:04:52,042 - INFO - joeynmt.helpers -                cfg.model.init_gain : 1.0\n",
      "2022-09-04 08:04:52,042 - INFO - joeynmt.helpers -        cfg.model.embed_initializer : xavier\n",
      "2022-09-04 08:04:52,043 - INFO - joeynmt.helpers -          cfg.model.embed_init_gain : 1.0\n",
      "2022-09-04 08:04:52,043 - INFO - joeynmt.helpers -          cfg.model.tied_embeddings : True\n",
      "2022-09-04 08:04:52,043 - INFO - joeynmt.helpers -             cfg.model.tied_softmax : True\n",
      "2022-09-04 08:04:52,043 - INFO - joeynmt.helpers -             cfg.model.encoder.type : transformer\n",
      "2022-09-04 08:04:52,043 - INFO - joeynmt.helpers -       cfg.model.encoder.num_layers : 6\n",
      "2022-09-04 08:04:52,043 - INFO - joeynmt.helpers -        cfg.model.encoder.num_heads : 4\n",
      "2022-09-04 08:04:52,043 - INFO - joeynmt.helpers - cfg.model.encoder.embeddings.embedding_dim : 256\n",
      "2022-09-04 08:04:52,043 - INFO - joeynmt.helpers - cfg.model.encoder.embeddings.scale : True\n",
      "2022-09-04 08:04:52,043 - INFO - joeynmt.helpers - cfg.model.encoder.embeddings.dropout : 0.2\n",
      "2022-09-04 08:04:52,043 - INFO - joeynmt.helpers -      cfg.model.encoder.hidden_size : 256\n",
      "2022-09-04 08:04:52,043 - INFO - joeynmt.helpers -          cfg.model.encoder.ff_size : 1024\n",
      "2022-09-04 08:04:52,043 - INFO - joeynmt.helpers -          cfg.model.encoder.dropout : 0.3\n",
      "2022-09-04 08:04:52,043 - INFO - joeynmt.helpers -             cfg.model.decoder.type : transformer\n",
      "2022-09-04 08:04:52,043 - INFO - joeynmt.helpers -       cfg.model.decoder.num_layers : 6\n",
      "2022-09-04 08:04:52,043 - INFO - joeynmt.helpers -        cfg.model.decoder.num_heads : 4\n",
      "2022-09-04 08:04:52,043 - INFO - joeynmt.helpers - cfg.model.decoder.embeddings.embedding_dim : 256\n",
      "2022-09-04 08:04:52,043 - INFO - joeynmt.helpers - cfg.model.decoder.embeddings.scale : True\n",
      "2022-09-04 08:04:52,043 - INFO - joeynmt.helpers - cfg.model.decoder.embeddings.dropout : 0.2\n",
      "2022-09-04 08:04:52,044 - INFO - joeynmt.helpers -      cfg.model.decoder.hidden_size : 256\n",
      "2022-09-04 08:04:52,044 - INFO - joeynmt.helpers -          cfg.model.decoder.ff_size : 1024\n",
      "2022-09-04 08:04:52,044 - INFO - joeynmt.helpers -          cfg.model.decoder.dropout : 0.3\n",
      "2022-09-04 08:04:52,044 - INFO - joeynmt.helpers - Data set sizes: \n",
      "\ttrain 347669,\n",
      "\tvalid 1000,\n",
      "\ttest 1000\n",
      "2022-09-04 08:04:52,044 - INFO - joeynmt.helpers - First training example:\n",
      "\t[SRC] At@@ ing@@ ebla tabela ap@@ ud@@ skribo\n",
      "\t[TRG] Acces@@ sible T@@ able C@@ ap@@ tion\n",
      "2022-09-04 08:04:52,044 - INFO - joeynmt.helpers - First 10 words (src): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) la (5) the (6) of (7) de (8) and (9) kaj\n",
      "2022-09-04 08:04:52,044 - INFO - joeynmt.helpers - First 10 words (trg): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) la (5) the (6) of (7) de (8) and (9) kaj\n",
      "2022-09-04 08:04:52,044 - INFO - joeynmt.helpers - Number of Src words (types): 4544\n",
      "2022-09-04 08:04:52,044 - INFO - joeynmt.helpers - Number of Trg words (types): 4544\n",
      "2022-09-04 08:04:52,044 - INFO - joeynmt.training - Model(\n",
      "\tencoder=TransformerEncoder(num_layers=6, num_heads=4),\n",
      "\tdecoder=TransformerDecoder(num_layers=6, num_heads=4),\n",
      "\tsrc_embed=Embeddings(embedding_dim=256, vocab_size=4544),\n",
      "\ttrg_embed=Embeddings(embedding_dim=256, vocab_size=4544))\n",
      "2022-09-04 08:04:52,047 - INFO - joeynmt.training - Train stats:\n",
      "\tdevice: cuda\n",
      "\tn_gpu: 1\n",
      "\t16-bits training: False\n",
      "\tgradient accumulation: 1\n",
      "\tbatch size per device: 4096\n",
      "\ttotal batch size (w. parallel & accumulation): 4096\n",
      "2022-09-04 08:04:52,047 - INFO - joeynmt.training - EPOCH 1\n",
      "2022-09-04 08:05:22,969 - INFO - joeynmt.training - Epoch   1, Step:      200, Batch Loss:     5.399559, Tokens per Sec:     7903, Lr: 0.000300\n",
      "2022-09-04 08:05:54,165 - INFO - joeynmt.training - Epoch   1, Step:      400, Batch Loss:     5.058511, Tokens per Sec:     7874, Lr: 0.000300\n",
      "2022-09-04 08:06:26,544 - INFO - joeynmt.training - Epoch   1, Step:      600, Batch Loss:     4.696565, Tokens per Sec:     7535, Lr: 0.000300\n",
      "2022-09-04 08:06:59,004 - INFO - joeynmt.training - Epoch   1, Step:      800, Batch Loss:     4.576544, Tokens per Sec:     7600, Lr: 0.000300\n",
      "2022-09-04 08:07:31,194 - INFO - joeynmt.training - Epoch   1, Step:     1000, Batch Loss:     4.254906, Tokens per Sec:     7643, Lr: 0.000300\n",
      "2022-09-04 08:08:03,877 - INFO - joeynmt.training - Epoch   1, Step:     1200, Batch Loss:     4.274159, Tokens per Sec:     7549, Lr: 0.000300\n",
      "2022-09-04 08:08:36,266 - INFO - joeynmt.training - Epoch   1, Step:     1400, Batch Loss:     4.017766, Tokens per Sec:     7583, Lr: 0.000300\n",
      "2022-09-04 08:09:08,620 - INFO - joeynmt.training - Epoch   1, Step:     1600, Batch Loss:     4.085143, Tokens per Sec:     7572, Lr: 0.000300\n",
      "2022-09-04 08:09:41,270 - INFO - joeynmt.training - Epoch   1, Step:     1800, Batch Loss:     3.965685, Tokens per Sec:     7548, Lr: 0.000300\n",
      "2022-09-04 08:10:13,622 - INFO - joeynmt.training - Epoch   1, Step:     2000, Batch Loss:     3.912527, Tokens per Sec:     7545, Lr: 0.000300\n",
      "2022-09-04 08:10:26,676 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2022-09-04 08:10:27,069 - INFO - joeynmt.training - Example #0\n",
      "2022-09-04 08:10:27,069 - INFO - joeynmt.training - \tSource:     Mi faris ion por vi.\n",
      "2022-09-04 08:10:27,069 - INFO - joeynmt.training - \tReference:  I made you something.\n",
      "2022-09-04 08:10:27,070 - INFO - joeynmt.training - \tHypothesis: I have to see you.\n",
      "2022-09-04 08:10:27,070 - INFO - joeynmt.training - Example #1\n",
      "2022-09-04 08:10:27,070 - INFO - joeynmt.training - \tSource:     Du oble du faras kvar.\n",
      "2022-09-04 08:10:27,070 - INFO - joeynmt.training - \tReference:  Twice two is four.\n",
      "2022-09-04 08:10:27,070 - INFO - joeynmt.training - \tHypothesis: - What is a man of the way.\n",
      "2022-09-04 08:10:27,070 - INFO - joeynmt.training - Example #2\n",
      "2022-09-04 08:10:27,070 - INFO - joeynmt.training - \tSource:     Tio suferigas min! Ĉesu!\n",
      "2022-09-04 08:10:27,070 - INFO - joeynmt.training - \tReference:  That hurts! Stop it!\n",
      "2022-09-04 08:10:27,070 - INFO - joeynmt.training - \tHypothesis: It's a cance.\n",
      "2022-09-04 08:10:27,070 - INFO - joeynmt.training - Example #3\n",
      "2022-09-04 08:10:27,070 - INFO - joeynmt.training - \tSource:     Estas tro multe por mi. Mi devas malrapidiĝi.\n",
      "2022-09-04 08:10:27,070 - INFO - joeynmt.training - \tReference:  It is too much for me. I need to slow down.\n",
      "2022-09-04 08:10:27,070 - INFO - joeynmt.training - \tHypothesis: It's not not a thing I have to be a bir.\n",
      "2022-09-04 08:10:27,070 - INFO - joeynmt.training - Validation result (greedy) at epoch   1, step     2000: bleu:   2.60, loss: 46577.3711, ppl:  48.5886, duration: 13.4482s\n",
      "2022-09-04 08:10:59,456 - INFO - joeynmt.training - Epoch   1, Step:     2200, Batch Loss:     3.577899, Tokens per Sec:     7585, Lr: 0.000300\n",
      "2022-09-04 08:11:31,775 - INFO - joeynmt.training - Epoch   1, Step:     2400, Batch Loss:     3.644752, Tokens per Sec:     7586, Lr: 0.000300\n",
      "2022-09-04 08:12:04,306 - INFO - joeynmt.training - Epoch   1, Step:     2600, Batch Loss:     3.372285, Tokens per Sec:     7561, Lr: 0.000300\n",
      "2022-09-04 08:12:11,196 - INFO - joeynmt.training - Epoch   1: total training loss 11337.92\n",
      "2022-09-04 08:12:11,197 - INFO - joeynmt.training - EPOCH 2\n",
      "2022-09-04 08:12:36,920 - INFO - joeynmt.training - Epoch   2, Step:     2800, Batch Loss:     3.520112, Tokens per Sec:     7478, Lr: 0.000300\n",
      "2022-09-04 08:13:09,270 - INFO - joeynmt.training - Epoch   2, Step:     3000, Batch Loss:     3.286849, Tokens per Sec:     7617, Lr: 0.000300\n",
      "2022-09-04 08:13:41,711 - INFO - joeynmt.training - Epoch   2, Step:     3200, Batch Loss:     3.484707, Tokens per Sec:     7561, Lr: 0.000300\n",
      "2022-09-04 08:14:13,977 - INFO - joeynmt.training - Epoch   2, Step:     3400, Batch Loss:     3.251159, Tokens per Sec:     7556, Lr: 0.000300\n",
      "2022-09-04 08:14:46,444 - INFO - joeynmt.training - Epoch   2, Step:     3600, Batch Loss:     3.192864, Tokens per Sec:     7541, Lr: 0.000300\n",
      "2022-09-04 08:15:18,639 - INFO - joeynmt.training - Epoch   2, Step:     3800, Batch Loss:     3.129761, Tokens per Sec:     7623, Lr: 0.000300\n",
      "2022-09-04 08:15:50,943 - INFO - joeynmt.training - Epoch   2, Step:     4000, Batch Loss:     3.189088, Tokens per Sec:     7611, Lr: 0.000300\n",
      "2022-09-04 08:15:58,926 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2022-09-04 08:15:59,361 - INFO - joeynmt.training - Example #0\n",
      "2022-09-04 08:15:59,361 - INFO - joeynmt.training - \tSource:     Mi faris ion por vi.\n",
      "2022-09-04 08:15:59,361 - INFO - joeynmt.training - \tReference:  I made you something.\n",
      "2022-09-04 08:15:59,361 - INFO - joeynmt.training - \tHypothesis: I have to see you.\n",
      "2022-09-04 08:15:59,361 - INFO - joeynmt.training - Example #1\n",
      "2022-09-04 08:15:59,362 - INFO - joeynmt.training - \tSource:     Du oble du faras kvar.\n",
      "2022-09-04 08:15:59,362 - INFO - joeynmt.training - \tReference:  Twice two is four.\n",
      "2022-09-04 08:15:59,362 - INFO - joeynmt.training - \tHypothesis: Did you have a two of two of here.\n",
      "2022-09-04 08:15:59,362 - INFO - joeynmt.training - Example #2\n",
      "2022-09-04 08:15:59,362 - INFO - joeynmt.training - \tSource:     Tio suferigas min! Ĉesu!\n",
      "2022-09-04 08:15:59,362 - INFO - joeynmt.training - \tReference:  That hurts! Stop it!\n",
      "2022-09-04 08:15:59,362 - INFO - joeynmt.training - \tHypothesis: This is a little mind!\n",
      "2022-09-04 08:15:59,362 - INFO - joeynmt.training - Example #3\n",
      "2022-09-04 08:15:59,362 - INFO - joeynmt.training - \tSource:     Estas tro multe por mi. Mi devas malrapidiĝi.\n",
      "2022-09-04 08:15:59,362 - INFO - joeynmt.training - \tReference:  It is too much for me. I need to slow down.\n",
      "2022-09-04 08:15:59,362 - INFO - joeynmt.training - \tHypothesis: It's too too too long to be a little little little go.\n",
      "2022-09-04 08:15:59,362 - INFO - joeynmt.training - Validation result (greedy) at epoch   2, step     4000: bleu:   5.70, loss: 38789.8320, ppl:  25.3836, duration: 8.4191s\n",
      "2022-09-04 08:16:31,911 - INFO - joeynmt.training - Epoch   2, Step:     4200, Batch Loss:     3.090279, Tokens per Sec:     7560, Lr: 0.000300\n",
      "2022-09-04 08:17:04,360 - INFO - joeynmt.training - Epoch   2, Step:     4400, Batch Loss:     3.015362, Tokens per Sec:     7587, Lr: 0.000300\n",
      "2022-09-04 08:17:37,115 - INFO - joeynmt.training - Epoch   2, Step:     4600, Batch Loss:     2.896784, Tokens per Sec:     7535, Lr: 0.000300\n",
      "Traceback (most recent call last):\n",
      "  File \"/usr/lib/python3.7/runpy.py\", line 193, in _run_module_as_main\n",
      "    \"__main__\", mod_spec)\n",
      "  File \"/usr/lib/python3.7/runpy.py\", line 85, in _run_code\n",
      "    exec(code, run_globals)\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/__main__.py\", line 48, in <module>\n",
      "    main()\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/__main__.py\", line 35, in main\n",
      "    train(cfg_file=args.config_path, skip_test=args.skip_test)\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/training.py\", line 846, in train\n",
      "    trainer.train_and_validate(train_data=train_data, valid_data=dev_data)\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/training.py\", line 447, in train_and_validate\n",
      "    batch_loss += self._train_step(batch)\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/training.py\", line 569, in _train_step\n",
      "    norm_batch_loss.backward()\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/torch/_tensor.py\", line 307, in backward\n",
      "    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py\", line 156, in backward\n",
      "    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag\n",
      "KeyboardInterrupt\n"
     ]
    }
   ],
   "source": [
    "!python -m joeynmt train transformer_epo_eng_bpe4000.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pgP9fnZsjO7K"
   },
   "source": [
    "## Continue training after interruption"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Kj92bqnf8aOe"
   },
   "source": [
    "To continue after an interruption, the configuration needs to be modified in 2 places: \n",
    "1. `load_model` to point to the checkpoint to load.\n",
    "2. `model_dir` to create a new directory.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "qS7ixekMfG5F"
   },
   "outputs": [],
   "source": [
    "ckpt_number = 'latest' #2000\n",
    "reload_config = config.replace(\n",
    "    f'#load_model: \"models/{name}_transformer/1.ckpt\"', f'load_model: \"models/{name}_transformer/{ckpt_number}.ckpt\"').replace(\n",
    "        f'model_dir: \"models/{name}_transformer\"', f'model_dir: \"models/{name}_transformer_continued\"')\n",
    "with open(\"transformer_{name}_reload.yaml\".format(name=name),'w') as f:\n",
    "    f.write(reload_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "05nDex2g9URU"
   },
   "source": [
    "Joey NMT then picks up training from there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "iU63igoskRXJ",
    "outputId": "b582611c-1173-45a8-c730-f616dd7e1b1f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-09-04 08:18:24,268 - INFO - root - Hello! This is Joey-NMT (version 1.5).\n",
      "2022-09-04 08:18:24,304 - INFO - joeynmt.data - Loading training data...\n",
      "2022-09-04 08:18:29,488 - INFO - joeynmt.data - Building vocabulary...\n",
      "2022-09-04 08:18:29,811 - INFO - joeynmt.data - Loading dev data...\n",
      "2022-09-04 08:18:29,835 - INFO - joeynmt.data - Loading test data...\n",
      "2022-09-04 08:18:30,594 - INFO - joeynmt.data - Data loaded.\n",
      "2022-09-04 08:18:30,594 - INFO - joeynmt.model - Building an encoder-decoder model...\n",
      "2022-09-04 08:18:30,804 - INFO - joeynmt.model - Enc-dec model built.\n",
      "2022-09-04 08:18:32,550 - INFO - joeynmt.training - Total params: 12223488\n",
      "2022-09-04 08:18:36,095 - INFO - joeynmt.training - Loading model from models/epo_eng_bpe4000_transformer/latest.ckpt\n",
      "2022-09-04 08:18:36,327 - INFO - joeynmt.helpers -                           cfg.name : epo_eng_bpe4000_transformer\n",
      "2022-09-04 08:18:36,328 - INFO - joeynmt.helpers -                       cfg.data.src : epo\n",
      "2022-09-04 08:18:36,328 - INFO - joeynmt.helpers -                       cfg.data.trg : eng\n",
      "2022-09-04 08:18:36,328 - INFO - joeynmt.helpers -                     cfg.data.train : /content/data/eng-epo//train.4000.bpe\n",
      "2022-09-04 08:18:36,328 - INFO - joeynmt.helpers -                       cfg.data.dev : /content/data/eng-epo//dev.4000.bpe\n",
      "2022-09-04 08:18:36,328 - INFO - joeynmt.helpers -                      cfg.data.test : /content/data/eng-epo//test.4000.bpe\n",
      "2022-09-04 08:18:36,328 - INFO - joeynmt.helpers -                     cfg.data.level : bpe\n",
      "2022-09-04 08:18:36,328 - INFO - joeynmt.helpers -                 cfg.data.lowercase : False\n",
      "2022-09-04 08:18:36,328 - INFO - joeynmt.helpers -           cfg.data.max_sent_length : 30\n",
      "2022-09-04 08:18:36,328 - INFO - joeynmt.helpers -                 cfg.data.src_vocab : /content/data/eng-epo/joint.4000bpe.vocab\n",
      "2022-09-04 08:18:36,328 - INFO - joeynmt.helpers -                 cfg.data.trg_vocab : /content/data/eng-epo/joint.4000bpe.vocab\n",
      "2022-09-04 08:18:36,328 - INFO - joeynmt.helpers -              cfg.testing.beam_size : 5\n",
      "2022-09-04 08:18:36,328 - INFO - joeynmt.helpers -                  cfg.testing.alpha : 1.0\n",
      "2022-09-04 08:18:36,328 - INFO - joeynmt.helpers - cfg.testing.sacrebleu.remove_whitespace : True\n",
      "2022-09-04 08:18:36,328 - INFO - joeynmt.helpers -     cfg.testing.sacrebleu.tokenize : intl\n",
      "2022-09-04 08:18:36,328 - INFO - joeynmt.helpers -            cfg.training.load_model : models/epo_eng_bpe4000_transformer/latest.ckpt\n",
      "2022-09-04 08:18:36,328 - INFO - joeynmt.helpers -           cfg.training.random_seed : 42\n",
      "2022-09-04 08:18:36,328 - INFO - joeynmt.helpers -             cfg.training.optimizer : adam\n",
      "2022-09-04 08:18:36,329 - INFO - joeynmt.helpers -         cfg.training.normalization : tokens\n",
      "2022-09-04 08:18:36,329 - INFO - joeynmt.helpers -            cfg.training.adam_betas : [0.9, 0.999]\n",
      "2022-09-04 08:18:36,329 - INFO - joeynmt.helpers -            cfg.training.scheduling : plateau\n",
      "2022-09-04 08:18:36,329 - INFO - joeynmt.helpers -              cfg.training.patience : 5\n",
      "2022-09-04 08:18:36,329 - INFO - joeynmt.helpers -  cfg.training.learning_rate_factor : 0.5\n",
      "2022-09-04 08:18:36,329 - INFO - joeynmt.helpers -  cfg.training.learning_rate_warmup : 1000\n",
      "2022-09-04 08:18:36,329 - INFO - joeynmt.helpers -       cfg.training.decrease_factor : 0.7\n",
      "2022-09-04 08:18:36,329 - INFO - joeynmt.helpers -                  cfg.training.loss : crossentropy\n",
      "2022-09-04 08:18:36,329 - INFO - joeynmt.helpers -         cfg.training.learning_rate : 0.0003\n",
      "2022-09-04 08:18:36,329 - INFO - joeynmt.helpers -     cfg.training.learning_rate_min : 1e-08\n",
      "2022-09-04 08:18:36,329 - INFO - joeynmt.helpers -          cfg.training.weight_decay : 0.0\n",
      "2022-09-04 08:18:36,329 - INFO - joeynmt.helpers -       cfg.training.label_smoothing : 0.1\n",
      "2022-09-04 08:18:36,329 - INFO - joeynmt.helpers -            cfg.training.batch_size : 4096\n",
      "2022-09-04 08:18:36,329 - INFO - joeynmt.helpers -            cfg.training.batch_type : token\n",
      "2022-09-04 08:18:36,329 - INFO - joeynmt.helpers -       cfg.training.eval_batch_size : 3600\n",
      "2022-09-04 08:18:36,329 - INFO - joeynmt.helpers -       cfg.training.eval_batch_type : token\n",
      "2022-09-04 08:18:36,329 - INFO - joeynmt.helpers -      cfg.training.batch_multiplier : 1\n",
      "2022-09-04 08:18:36,329 - INFO - joeynmt.helpers - cfg.training.early_stopping_metric : ppl\n",
      "2022-09-04 08:18:36,330 - INFO - joeynmt.helpers -                cfg.training.epochs : 30\n",
      "2022-09-04 08:18:36,330 - INFO - joeynmt.helpers -       cfg.training.validation_freq : 2000\n",
      "2022-09-04 08:18:36,330 - INFO - joeynmt.helpers -          cfg.training.logging_freq : 200\n",
      "2022-09-04 08:18:36,330 - INFO - joeynmt.helpers -           cfg.training.eval_metric : bleu\n",
      "2022-09-04 08:18:36,330 - INFO - joeynmt.helpers -             cfg.training.model_dir : models/epo_eng_bpe4000_transformer_continued\n",
      "2022-09-04 08:18:36,330 - INFO - joeynmt.helpers -             cfg.training.overwrite : False\n",
      "2022-09-04 08:18:36,330 - INFO - joeynmt.helpers -               cfg.training.shuffle : True\n",
      "2022-09-04 08:18:36,330 - INFO - joeynmt.helpers -              cfg.training.use_cuda : True\n",
      "2022-09-04 08:18:36,330 - INFO - joeynmt.helpers -     cfg.training.max_output_length : 100\n",
      "2022-09-04 08:18:36,330 - INFO - joeynmt.helpers -     cfg.training.print_valid_sents : [0, 1, 2, 3]\n",
      "2022-09-04 08:18:36,330 - INFO - joeynmt.helpers -       cfg.training.keep_best_ckpts : 3\n",
      "2022-09-04 08:18:36,330 - INFO - joeynmt.helpers -              cfg.model.initializer : xavier\n",
      "2022-09-04 08:18:36,330 - INFO - joeynmt.helpers -         cfg.model.bias_initializer : zeros\n",
      "2022-09-04 08:18:36,330 - INFO - joeynmt.helpers -                cfg.model.init_gain : 1.0\n",
      "2022-09-04 08:18:36,330 - INFO - joeynmt.helpers -        cfg.model.embed_initializer : xavier\n",
      "2022-09-04 08:18:36,330 - INFO - joeynmt.helpers -          cfg.model.embed_init_gain : 1.0\n",
      "2022-09-04 08:18:36,331 - INFO - joeynmt.helpers -          cfg.model.tied_embeddings : True\n",
      "2022-09-04 08:18:36,331 - INFO - joeynmt.helpers -             cfg.model.tied_softmax : True\n",
      "2022-09-04 08:18:36,331 - INFO - joeynmt.helpers -             cfg.model.encoder.type : transformer\n",
      "2022-09-04 08:18:36,331 - INFO - joeynmt.helpers -       cfg.model.encoder.num_layers : 6\n",
      "2022-09-04 08:18:36,331 - INFO - joeynmt.helpers -        cfg.model.encoder.num_heads : 4\n",
      "2022-09-04 08:18:36,331 - INFO - joeynmt.helpers - cfg.model.encoder.embeddings.embedding_dim : 256\n",
      "2022-09-04 08:18:36,331 - INFO - joeynmt.helpers - cfg.model.encoder.embeddings.scale : True\n",
      "2022-09-04 08:18:36,331 - INFO - joeynmt.helpers - cfg.model.encoder.embeddings.dropout : 0.2\n",
      "2022-09-04 08:18:36,331 - INFO - joeynmt.helpers -      cfg.model.encoder.hidden_size : 256\n",
      "2022-09-04 08:18:36,331 - INFO - joeynmt.helpers -          cfg.model.encoder.ff_size : 1024\n",
      "2022-09-04 08:18:36,331 - INFO - joeynmt.helpers -          cfg.model.encoder.dropout : 0.3\n",
      "2022-09-04 08:18:36,331 - INFO - joeynmt.helpers -             cfg.model.decoder.type : transformer\n",
      "2022-09-04 08:18:36,331 - INFO - joeynmt.helpers -       cfg.model.decoder.num_layers : 6\n",
      "2022-09-04 08:18:36,331 - INFO - joeynmt.helpers -        cfg.model.decoder.num_heads : 4\n",
      "2022-09-04 08:18:36,331 - INFO - joeynmt.helpers - cfg.model.decoder.embeddings.embedding_dim : 256\n",
      "2022-09-04 08:18:36,331 - INFO - joeynmt.helpers - cfg.model.decoder.embeddings.scale : True\n",
      "2022-09-04 08:18:36,332 - INFO - joeynmt.helpers - cfg.model.decoder.embeddings.dropout : 0.2\n",
      "2022-09-04 08:18:36,332 - INFO - joeynmt.helpers -      cfg.model.decoder.hidden_size : 256\n",
      "2022-09-04 08:18:36,332 - INFO - joeynmt.helpers -          cfg.model.decoder.ff_size : 1024\n",
      "2022-09-04 08:18:36,332 - INFO - joeynmt.helpers -          cfg.model.decoder.dropout : 0.3\n",
      "2022-09-04 08:18:36,332 - INFO - joeynmt.helpers - Data set sizes: \n",
      "\ttrain 347669,\n",
      "\tvalid 1000,\n",
      "\ttest 1000\n",
      "2022-09-04 08:18:36,332 - INFO - joeynmt.helpers - First training example:\n",
      "\t[SRC] At@@ ing@@ ebla tabela ap@@ ud@@ skribo\n",
      "\t[TRG] Acces@@ sible T@@ able C@@ ap@@ tion\n",
      "2022-09-04 08:18:36,332 - INFO - joeynmt.helpers - First 10 words (src): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) la (5) the (6) of (7) de (8) and (9) kaj\n",
      "2022-09-04 08:18:36,332 - INFO - joeynmt.helpers - First 10 words (trg): (0) <unk> (1) <pad> (2) <s> (3) </s> (4) la (5) the (6) of (7) de (8) and (9) kaj\n",
      "2022-09-04 08:18:36,332 - INFO - joeynmt.helpers - Number of Src words (types): 4544\n",
      "2022-09-04 08:18:36,332 - INFO - joeynmt.helpers - Number of Trg words (types): 4544\n",
      "2022-09-04 08:18:36,332 - INFO - joeynmt.training - Model(\n",
      "\tencoder=TransformerEncoder(num_layers=6, num_heads=4),\n",
      "\tdecoder=TransformerDecoder(num_layers=6, num_heads=4),\n",
      "\tsrc_embed=Embeddings(embedding_dim=256, vocab_size=4544),\n",
      "\ttrg_embed=Embeddings(embedding_dim=256, vocab_size=4544))\n",
      "2022-09-04 08:18:36,335 - INFO - joeynmt.training - Train stats:\n",
      "\tdevice: cuda\n",
      "\tn_gpu: 1\n",
      "\t16-bits training: False\n",
      "\tgradient accumulation: 1\n",
      "\tbatch size per device: 4096\n",
      "\ttotal batch size (w. parallel & accumulation): 4096\n",
      "2022-09-04 08:18:36,335 - INFO - joeynmt.training - EPOCH 1\n",
      "2022-09-04 08:19:09,376 - INFO - joeynmt.training - Epoch   1, Step:     4200, Batch Loss:     3.112499, Tokens per Sec:     7448, Lr: 0.000300\n",
      "2022-09-04 08:19:42,005 - INFO - joeynmt.training - Epoch   1, Step:     4400, Batch Loss:     3.039347, Tokens per Sec:     7545, Lr: 0.000300\n",
      "2022-09-04 08:20:14,249 - INFO - joeynmt.training - Epoch   1, Step:     4600, Batch Loss:     2.896453, Tokens per Sec:     7655, Lr: 0.000300\n",
      "2022-09-04 08:20:46,442 - INFO - joeynmt.training - Epoch   1, Step:     4800, Batch Loss:     3.001946, Tokens per Sec:     7583, Lr: 0.000300\n",
      "2022-09-04 08:21:18,915 - INFO - joeynmt.training - Epoch   1, Step:     5000, Batch Loss:     2.685496, Tokens per Sec:     7548, Lr: 0.000300\n",
      "2022-09-04 08:21:51,347 - INFO - joeynmt.training - Epoch   1, Step:     5200, Batch Loss:     2.657134, Tokens per Sec:     7541, Lr: 0.000300\n",
      "2022-09-04 08:22:05,375 - INFO - joeynmt.training - Epoch   1: total training loss 3812.93\n",
      "2022-09-04 08:22:05,375 - INFO - joeynmt.training - EPOCH 2\n",
      "2022-09-04 08:22:23,997 - INFO - joeynmt.training - Epoch   2, Step:     5400, Batch Loss:     2.880748, Tokens per Sec:     7512, Lr: 0.000300\n",
      "2022-09-04 08:22:56,540 - INFO - joeynmt.training - Epoch   2, Step:     5600, Batch Loss:     3.053030, Tokens per Sec:     7521, Lr: 0.000300\n",
      "2022-09-04 08:23:28,866 - INFO - joeynmt.training - Epoch   2, Step:     5800, Batch Loss:     2.653758, Tokens per Sec:     7592, Lr: 0.000300\n",
      "2022-09-04 08:24:01,147 - INFO - joeynmt.training - Epoch   2, Step:     6000, Batch Loss:     2.891744, Tokens per Sec:     7587, Lr: 0.000300\n",
      "2022-09-04 08:24:12,479 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2022-09-04 08:24:12,838 - INFO - joeynmt.training - Example #0\n",
      "2022-09-04 08:24:12,838 - INFO - joeynmt.training - \tSource:     Mi faris ion por vi.\n",
      "2022-09-04 08:24:12,838 - INFO - joeynmt.training - \tReference:  I made you something.\n",
      "2022-09-04 08:24:12,838 - INFO - joeynmt.training - \tHypothesis: I have to do you.\n",
      "2022-09-04 08:24:12,838 - INFO - joeynmt.training - Example #1\n",
      "2022-09-04 08:24:12,839 - INFO - joeynmt.training - \tSource:     Du oble du faras kvar.\n",
      "2022-09-04 08:24:12,839 - INFO - joeynmt.training - \tReference:  Twice two is four.\n",
      "2022-09-04 08:24:12,839 - INFO - joeynmt.training - \tHypothesis: Take two two of two four.\n",
      "2022-09-04 08:24:12,839 - INFO - joeynmt.training - Example #2\n",
      "2022-09-04 08:24:12,839 - INFO - joeynmt.training - \tSource:     Tio suferigas min! Ĉesu!\n",
      "2022-09-04 08:24:12,839 - INFO - joeynmt.training - \tReference:  That hurts! Stop it!\n",
      "2022-09-04 08:24:12,839 - INFO - joeynmt.training - \tHypothesis: This is a minute.\n",
      "2022-09-04 08:24:12,839 - INFO - joeynmt.training - Example #3\n",
      "2022-09-04 08:24:12,839 - INFO - joeynmt.training - \tSource:     Estas tro multe por mi. Mi devas malrapidiĝi.\n",
      "2022-09-04 08:24:12,839 - INFO - joeynmt.training - \tReference:  It is too much for me. I need to slow down.\n",
      "2022-09-04 08:24:12,839 - INFO - joeynmt.training - \tHypothesis: It's too much for me.\n",
      "2022-09-04 08:24:12,839 - INFO - joeynmt.training - Validation result (greedy) at epoch   2, step     6000: bleu:   7.51, loss: 34536.4961, ppl:  17.8050, duration: 11.6917s\n",
      "2022-09-04 08:24:45,236 - INFO - joeynmt.training - Epoch   2, Step:     6200, Batch Loss:     2.696601, Tokens per Sec:     7567, Lr: 0.000300\n",
      "2022-09-04 08:25:17,529 - INFO - joeynmt.training - Epoch   2, Step:     6400, Batch Loss:     2.728019, Tokens per Sec:     7608, Lr: 0.000300\n",
      "2022-09-04 08:25:49,942 - INFO - joeynmt.training - Epoch   2, Step:     6600, Batch Loss:     2.699455, Tokens per Sec:     7598, Lr: 0.000300\n",
      "2022-09-04 08:26:22,180 - INFO - joeynmt.training - Epoch   2, Step:     6800, Batch Loss:     2.787544, Tokens per Sec:     7604, Lr: 0.000300\n",
      "2022-09-04 08:26:54,441 - INFO - joeynmt.training - Epoch   2, Step:     7000, Batch Loss:     2.431699, Tokens per Sec:     7606, Lr: 0.000300\n",
      "2022-09-04 08:27:26,761 - INFO - joeynmt.training - Epoch   2, Step:     7200, Batch Loss:     2.490031, Tokens per Sec:     7599, Lr: 0.000300\n",
      "2022-09-04 08:27:59,038 - INFO - joeynmt.training - Epoch   2, Step:     7400, Batch Loss:     2.440439, Tokens per Sec:     7582, Lr: 0.000300\n",
      "2022-09-04 08:28:31,472 - INFO - joeynmt.training - Epoch   2, Step:     7600, Batch Loss:     2.552195, Tokens per Sec:     7523, Lr: 0.000300\n",
      "2022-09-04 08:29:03,804 - INFO - joeynmt.training - Epoch   2, Step:     7800, Batch Loss:     2.535806, Tokens per Sec:     7675, Lr: 0.000300\n",
      "2022-09-04 08:29:24,729 - INFO - joeynmt.training - Epoch   2: total training loss 7022.05\n",
      "2022-09-04 08:29:24,729 - INFO - joeynmt.training - EPOCH 3\n",
      "2022-09-04 08:29:36,450 - INFO - joeynmt.training - Epoch   3, Step:     8000, Batch Loss:     2.426188, Tokens per Sec:     7349, Lr: 0.000300\n",
      "2022-09-04 08:29:52,052 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2022-09-04 08:29:52,581 - INFO - joeynmt.training - Example #0\n",
      "2022-09-04 08:29:52,582 - INFO - joeynmt.training - \tSource:     Mi faris ion por vi.\n",
      "2022-09-04 08:29:52,582 - INFO - joeynmt.training - \tReference:  I made you something.\n",
      "2022-09-04 08:29:52,582 - INFO - joeynmt.training - \tHypothesis: I made it for you.\n",
      "2022-09-04 08:29:52,582 - INFO - joeynmt.training - Example #1\n",
      "2022-09-04 08:29:52,582 - INFO - joeynmt.training - \tSource:     Du oble du faras kvar.\n",
      "2022-09-04 08:29:52,582 - INFO - joeynmt.training - \tReference:  Twice two is four.\n",
      "2022-09-04 08:29:52,582 - INFO - joeynmt.training - \tHypothesis: Did you have two four.\n",
      "2022-09-04 08:29:52,582 - INFO - joeynmt.training - Example #2\n",
      "2022-09-04 08:29:52,583 - INFO - joeynmt.training - \tSource:     Tio suferigas min! Ĉesu!\n",
      "2022-09-04 08:29:52,583 - INFO - joeynmt.training - \tReference:  That hurts! Stop it!\n",
      "2022-09-04 08:29:52,583 - INFO - joeynmt.training - \tHypothesis: This is a few mind!\n",
      "2022-09-04 08:29:52,583 - INFO - joeynmt.training - Example #3\n",
      "2022-09-04 08:29:52,583 - INFO - joeynmt.training - \tSource:     Estas tro multe por mi. Mi devas malrapidiĝi.\n",
      "2022-09-04 08:29:52,583 - INFO - joeynmt.training - \tReference:  It is too much for me. I need to slow down.\n",
      "2022-09-04 08:29:52,583 - INFO - joeynmt.training - \tHypothesis: It's too much for me.\n",
      "2022-09-04 08:29:52,583 - INFO - joeynmt.training - Validation result (greedy) at epoch   3, step     8000: bleu:  10.04, loss: 31482.8164, ppl:  13.8029, duration: 16.1327s\n",
      "2022-09-04 08:30:24,749 - INFO - joeynmt.training - Epoch   3, Step:     8200, Batch Loss:     2.521933, Tokens per Sec:     7590, Lr: 0.000300\n",
      "2022-09-04 08:30:56,854 - INFO - joeynmt.training - Epoch   3, Step:     8400, Batch Loss:     2.464072, Tokens per Sec:     7645, Lr: 0.000300\n",
      "2022-09-04 08:31:29,099 - INFO - joeynmt.training - Epoch   3, Step:     8600, Batch Loss:     2.389722, Tokens per Sec:     7608, Lr: 0.000300\n",
      "2022-09-04 08:32:01,381 - INFO - joeynmt.training - Epoch   3, Step:     8800, Batch Loss:     2.660061, Tokens per Sec:     7577, Lr: 0.000300\n",
      "2022-09-04 08:32:33,610 - INFO - joeynmt.training - Epoch   3, Step:     9000, Batch Loss:     2.703629, Tokens per Sec:     7582, Lr: 0.000300\n",
      "2022-09-04 08:33:05,943 - INFO - joeynmt.training - Epoch   3, Step:     9200, Batch Loss:     2.591011, Tokens per Sec:     7684, Lr: 0.000300\n",
      "2022-09-04 08:33:38,154 - INFO - joeynmt.training - Epoch   3, Step:     9400, Batch Loss:     2.256877, Tokens per Sec:     7628, Lr: 0.000300\n",
      "2022-09-04 08:34:10,542 - INFO - joeynmt.training - Epoch   3, Step:     9600, Batch Loss:     2.451584, Tokens per Sec:     7596, Lr: 0.000300\n",
      "2022-09-04 08:34:42,724 - INFO - joeynmt.training - Epoch   3, Step:     9800, Batch Loss:     2.329672, Tokens per Sec:     7631, Lr: 0.000300\n",
      "2022-09-04 08:35:15,075 - INFO - joeynmt.training - Epoch   3, Step:    10000, Batch Loss:     2.370082, Tokens per Sec:     7550, Lr: 0.000300\n",
      "2022-09-04 08:35:28,874 - INFO - joeynmt.training - Hooray! New best validation result [ppl]!\n",
      "2022-09-04 08:35:29,284 - INFO - joeynmt.training - Example #0\n",
      "2022-09-04 08:35:29,285 - INFO - joeynmt.training - \tSource:     Mi faris ion por vi.\n",
      "2022-09-04 08:35:29,285 - INFO - joeynmt.training - \tReference:  I made you something.\n",
      "2022-09-04 08:35:29,285 - INFO - joeynmt.training - \tHypothesis: I have done for you.\n",
      "2022-09-04 08:35:29,285 - INFO - joeynmt.training - Example #1\n",
      "2022-09-04 08:35:29,285 - INFO - joeynmt.training - \tSource:     Du oble du faras kvar.\n",
      "2022-09-04 08:35:29,285 - INFO - joeynmt.training - \tReference:  Twice two is four.\n",
      "2022-09-04 08:35:29,285 - INFO - joeynmt.training - \tHypothesis: Tell us two things are four of four.\n",
      "2022-09-04 08:35:29,285 - INFO - joeynmt.training - Example #2\n",
      "2022-09-04 08:35:29,285 - INFO - joeynmt.training - \tSource:     Tio suferigas min! Ĉesu!\n",
      "2022-09-04 08:35:29,285 - INFO - joeynmt.training - \tReference:  That hurts! Stop it!\n",
      "2022-09-04 08:35:29,285 - INFO - joeynmt.training - \tHypothesis: This conflicts to me!\n",
      "2022-09-04 08:35:29,285 - INFO - joeynmt.training - Example #3\n",
      "2022-09-04 08:35:29,286 - INFO - joeynmt.training - \tSource:     Estas tro multe por mi. Mi devas malrapidiĝi.\n",
      "2022-09-04 08:35:29,286 - INFO - joeynmt.training - \tReference:  It is too much for me. I need to slow down.\n",
      "2022-09-04 08:35:29,286 - INFO - joeynmt.training - \tHypothesis: It's too much for me. I must have a few to be a few of me.\n",
      "2022-09-04 08:35:29,286 - INFO - joeynmt.training - Validation result (greedy) at epoch   3, step    10000: bleu:  12.91, loss: 29430.0977, ppl:  11.6317, duration: 14.2101s\n",
      "2022-09-04 08:36:01,619 - INFO - joeynmt.training - Epoch   3, Step:    10200, Batch Loss:     2.237199, Tokens per Sec:     7601, Lr: 0.000300\n",
      "Traceback (most recent call last):\n",
      "  File \"/usr/lib/python3.7/runpy.py\", line 193, in _run_module_as_main\n",
      "    \"__main__\", mod_spec)\n",
      "  File \"/usr/lib/python3.7/runpy.py\", line 85, in _run_code\n",
      "    exec(code, run_globals)\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/__main__.py\", line 48, in <module>\n",
      "    main()\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/__main__.py\", line 35, in main\n",
      "    train(cfg_file=args.config_path, skip_test=args.skip_test)\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/training.py\", line 846, in train\n",
      "    trainer.train_and_validate(train_data=train_data, valid_data=dev_data)\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/joeynmt/training.py\", line 441, in train_and_validate\n",
      "    for i, batch in enumerate(iter(self.train_iter)):\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/torchtext/legacy/data/iterator.py\", line 160, in __iter__\n",
      "    yield Batch(minibatch, self.dataset, self.device)\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/torchtext/legacy/data/batch.py\", line 34, in __init__\n",
      "    setattr(self, name, field.process(batch, device=device))\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/torchtext/legacy/data/field.py\", line 231, in process\n",
      "    tensor = self.numericalize(padded, device=device)\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/torchtext/legacy/data/field.py\", line 330, in numericalize\n",
      "    arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/torchtext/legacy/data/field.py\", line 330, in <listcomp>\n",
      "    arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/torchtext/legacy/data/field.py\", line 330, in <listcomp>\n",
      "    arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]\n",
      "KeyboardInterrupt\n",
      "Error in atexit._run_exitfuncs:\n",
      "Traceback (most recent call last):\n",
      "  File \"/usr/local/lib/python3.7/dist-packages/matplotlib/_pylab_helpers.py\", line 76, in destroy_all\n",
      "    gc.collect(1)\n",
      "KeyboardInterrupt\n"
     ]
    }
   ],
   "source": [
    "!python -m joeynmt train transformer_epo_eng_bpe4000_reload.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "99Y-03KklJd3"
   },
   "source": [
    "## Let's Translate!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vhR4-RFF5_dJ"
   },
   "source": [
    "The `test` mode can be used to translate (and evaluate on) the test set specified in the configuration. We usually do this only once after we've tuned hyperparameters on the dev set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pu59HTo_lLJG",
    "outputId": "21ff8023-f60b-451d-e103-125a620c6fac"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-09-04 08:36:47,258 - INFO - root - Hello! This is Joey-NMT (version 1.5).\n",
      "2022-09-04 08:36:47,259 - INFO - joeynmt.data - Building vocabulary...\n",
      "2022-09-04 08:36:47,869 - INFO - joeynmt.data - Loading dev data...\n",
      "2022-09-04 08:36:47,881 - INFO - joeynmt.data - Loading test data...\n",
      "2022-09-04 08:36:47,890 - INFO - joeynmt.data - Data loaded.\n",
      "2022-09-04 08:36:47,943 - INFO - joeynmt.prediction - Process device: cuda, n_gpu: 1, batch_size per device: 3600\n",
      "2022-09-04 08:36:47,944 - INFO - joeynmt.prediction - Loading model from models/epo_eng_bpe4000_transformer_continued/10000.ckpt\n",
      "2022-09-04 08:36:51,676 - INFO - joeynmt.model - Building an encoder-decoder model...\n",
      "2022-09-04 08:36:51,880 - INFO - joeynmt.model - Enc-dec model built.\n",
      "2022-09-04 08:36:51,950 - INFO - joeynmt.prediction - Decoding on dev set (/content/data/eng-epo//dev.4000.bpe.eng)...\n",
      "2022-09-04 08:37:05,451 - INFO - joeynmt.prediction -  dev bleu[intl]:  15.11 [Beam search decoding with beam size = 5 and alpha = {beam_alpha}]\n",
      "2022-09-04 08:37:05,452 - INFO - joeynmt.prediction - Decoding on test set (/content/data/eng-epo//test.4000.bpe.eng)...\n",
      "2022-09-04 08:37:20,937 - INFO - joeynmt.prediction - test bleu[intl]:  14.86 [Beam search decoding with beam size = 5 and alpha = {beam_alpha}]\n"
     ]
    }
   ],
   "source": [
    "!python -m joeynmt test models/epo_eng_bpe4000_transformer_continued/config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8RgkT7mp6T2l"
   },
   "source": [
    "The `translate` mode is more interactive and takes prompts to translate interactively. Warning: it requires applying the same pre-processing steps to the new input as you've applied before model training (i.e. splitting into subwords)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "YpBTLaoM9g58"
   },
   "outputs": [],
   "source": [
    "from subword_nmt import apply_bpe\n",
    "\n",
    "with open(bpe_file, \"r\") as merge_file:\n",
    "  bpe = apply_bpe.BPE(codes=merge_file)\n",
    "\n",
    "preprocess = lambda x: bpe.process_line(x.strip())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "4clB81W46oTB"
   },
   "outputs": [],
   "source": [
    "my_sentence = 'Esperanto, origine la Lingvo Internacia, estas la plej disvastiĝinta internacia planlingvo.'   # From https://eo.wikipedia.org/wiki/Esperanto"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 53
    },
    "id": "OUkf54ip-hVw",
    "outputId": "23910371-e9f4-4bc9-91c4-c97f1fb62b30"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'E@@ sper@@ ant@@ o, or@@ ig@@ ine la L@@ ingv@@ o In@@ tern@@ aci@@ a, estas la plej dis@@ v@@ ast@@ iĝ@@ inta intern@@ ac@@ ia plan@@ lingv@@ o.'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocess(my_sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-jHqeqiSctGU"
   },
   "source": [
    "Copy the above pre-processed sentence into the field when prompted below. Stop the cell to leave the interactive mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YtS7xyEmmD90",
    "outputId": "2725c6e8-bb7c-4839-a935-d859d88fb7ba"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-09-04 08:37:42,784 - INFO - root - Hello! This is Joey-NMT (version 1.5).\n",
      "2022-09-04 08:37:43,138 - INFO - joeynmt.prediction - Loading model from models/epo_eng_bpe4000_transformer_continued/10000.ckpt\n",
      "2022-09-04 08:37:46,836 - INFO - joeynmt.model - Building an encoder-decoder model...\n",
      "2022-09-04 08:37:47,048 - INFO - joeynmt.model - Enc-dec model built.\n",
      "\n",
      "Please enter a source sentence (pre-processed): \n",
      "E@@ sper@@ ant@@ o, or@@ ig@@ ine la L@@ ingv@@ o In@@ tern@@ aci@@ a, estas la plej dis@@ v@@ ast@@ iĝ@@ inta intern@@ ac@@ ia plan@@ lingv@@ o.\n",
      "JoeyNMT: Hypotheses ranked by score\n",
      "JoeyNMT #1: Esperantant, the Language Internal Internation is the most increated language.\n",
      "\n",
      "Please enter a source sentence (pre-processed): \n",
      "\n",
      "Bye.\n",
      "^C\n"
     ]
    }
   ],
   "source": [
    "!python -m joeynmt translate models/epo_eng_bpe4000_transformer_continued/config.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cX0Pk16Ac4fI"
   },
   "source": [
    "You can also get the n-best hypotheses (up to the size of the beam, in our example 5), not only the highest scoring one. The better your model gets, the more interesting should the alternatives be.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fT_SslRec__L",
    "outputId": "cfcd128a-bdc9-48ce-f794-31f347aea3a9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-09-04 08:38:10,660 - INFO - root - Hello! This is Joey-NMT (version 1.5).\n",
      "2022-09-04 08:38:11,018 - INFO - joeynmt.prediction - Loading model from models/epo_eng_bpe4000_transformer_continued/10000.ckpt\n",
      "2022-09-04 08:38:14,730 - INFO - joeynmt.model - Building an encoder-decoder model...\n",
      "2022-09-04 08:38:14,936 - INFO - joeynmt.model - Enc-dec model built.\n",
      "\n",
      "Please enter a source sentence (pre-processed): \n",
      "E@@ sper@@ ant@@ o, or@@ ig@@ ine la L@@ ingv@@ o In@@ tern@@ aci@@ a, estas la plej dis@@ v@@ ast@@ iĝ@@ inta intern@@ ac@@ ia plan@@ lingv@@ o.\n",
      "JoeyNMT: Hypotheses ranked by score\n",
      "JoeyNMT #1: Esperantant, the Language Internal Internation is the most increated language.\n",
      "JoeyNMT #2: Esperantant, the Language Internal Internation is the most created language.\n",
      "JoeyNMT #3: Esperantant, the Language Internal Internation is the most most increated language.\n",
      "\n",
      "Please enter a source sentence (pre-processed): \n",
      "\n",
      "Bye.\n",
      "^C\n"
     ]
    }
   ],
   "source": [
    "!python -m joeynmt translate models/epo_eng_bpe4000_transformer_continued/config.yaml -n 3"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Kopie von joey-demo.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}