beatles-full-second-iter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Beatles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import random\n",
    "import shlex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.text import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import PosixPath"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs=48"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/random-train'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/valid-lyrics'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train-lyrics3'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train-lyrics2'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/export.pkl'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/data_lm_lyrics.pkl'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/data.pkl'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train-lyrics'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/models'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/lyrics.pkl'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/.ipynb_checkpoints'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/music.pkl'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/data_lm.pkl'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train-lyrics-one-file')]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles')\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm_lyrics = (TextList.from_folder(path)\n",
    "           #Inputs: all the text files in path\n",
    "            .filter_by_folder(include=['train-lyrics3']) \n",
    "           #We may have other temp folders that contain text files so we only keep what's in train\n",
    "            .split_by_rand_pct(0.2)\n",
    "#             .split_by_folder(\"train-lyrics\", \"valid-lyrics\")\n",
    "            #.split_none()\n",
    "           #We randomly split and keep 10% (10,000 reviews) for validation\n",
    "            .label_for_lm()           \n",
    "           #We want to do a language model so we label accordingly\n",
    "            .databunch(bs=bs))\n",
    "data_lm_lyrics.save('data_lm_lyrics.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm_lyrics = load_data(path, 'data_lm_lyrics.pkl', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>idx</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>do n't let me down xxmaj do n't let me down xxmaj do n't let me down i 'm in love for the first time xxmaj do n't you know it 's going to last xxmaj it 's a love that xxunk forever xxmaj it 's a love that has no past xxmaj do n't let me down xxmaj do n't let me down xxmaj do n't let me down</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>xxunk died in the xxunk and was xxunk along with her name . xxmaj nobody came . xxmaj father xxunk , xxunk the dirt from his hands as he xxunk from the xxunk . xxmaj no - one was xxunk . xxmaj all the lonely people , where do they all come from ? xxmaj all the lonely people , where do they all belong ? xxmaj oh yeah alright</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>going down xxmaj and the eyes in his head see the world spinning round . xxmaj well on the way , head in a cloud , xxmaj the man of a thousand voices talking xxunk xxunk . xxmaj but nobody ever xxunk him xxmaj or the sound he xxunk to make xxmaj and he never seems to notice . xxmaj but the fool on the hill sees the sun going</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>. xxmaj go home xxmaj get back , get back . xxmaj back to where you once belonged xxmaj get back , get back . xxmaj back to where you once belonged . xxmaj get back xxmaj xxunk . xxmaj sweet xxmaj loretta xxmaj xxunk thought she was a woman xxmaj but she was another man xxmaj all the girls around her say she 's got it coming xxmaj but</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>mind , xxmaj shine , the weather 's fine . i can show you that when it starts to rain , xxmaj everything 's the same , i can show you , i can show you . xxmaj rain , i do n't mind , xxmaj shine , the weather 's fine . xxmaj can you hear me that when it xxunk and shines , xxmaj it 's just a</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_lm_lyrics.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then put this in a learner object very easily with a model loaded with the pretrained weights. They'll be downloaded the first time you'll execute the following line and stored in `~/.fastai/models/` (or elsewhere if you specified different paths in your config file)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lyrics = language_model_learner(data_lm_lyrics, AWD_LSTM, drop_mult=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_lyrics.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEKCAYAAADTgGjXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4VOXd//H3N5M9EAhZWBIgEFAEVIQIIu5al+qDe11K6/poba221trHtk9/VtvaahfbWqvW1rYute4VfOourmxBNlllCbskBCQhkP3+/TGDRpqNyZw5M5PP67rmYuace2a+N5PkM/dZ7mPOOURERMKR5HcBIiISvxQiIiISNoWIiIiETSEiIiJhU4iIiEjYFCIiIhI2hYiIiIRNISIiImFTiIiISNiS/S7gQOXl5bni4mK/yxARiSvz58/f7pzLj/Trxl2IFBcXU1ZW5ncZIiJxxczWe/G62pwlIiJhU4iIiEjYFCIiIhI2hYiIiIRNISIiImFTiIiISNg8DxEzC5jZAjOb0c76L5nZMjNbamaPe12PiIhETjTOE7kRWA5k77/CzEYCtwJTnHM7zawgCvXErXXba3ljRQXFuZlMLsklMzXuTvMRkQTj6V8hMysCzgR+CtzURpP/Bv7gnNsJ4Jyr8LKeeFRRXceMxVt5fuFmFm/a9eny1OQkJg3rxwkHF1DYN52UQBLJgSRSAsYhA7LJyUr1sWoR6Sm8/ip7D3AL0Lud9QcBmNl7QAC4zTn3ksc1xaQdtQ28uGQrb6+qpKKmnqrd9VTtbmBvYzMAYwZl84MvHsLpYwewvmoPM1dW8ObKCu6Ysew/Xis1OYmphw/issnFHFrUJ9pdEZEexLMQMbOzgArn3HwzO6GD9x8JnAAUAe+Y2Vjn3Cf7vdY1wDUAQ4YM8arkqKtrbOalDz/mXws3885H22lqcQzNzWRIv0yG52WRm5VKQXYaJ40qYETBZzk8uF8mx4zM44dnjWbrrr3srG2kqaWFxuYW9ja08NLSrTz7wWaenr+J8UP68rXjS/jC6P6YmY+9FZFEZM45b17Y7E7gK0ATkE5wn8izzrlprdrcD8x2zv019Ph14H+cc/Pae93S0lIX73Nn1TU28895G7lv5mq2VddT2DeD/zp8EFMPH8QhA3tH5I99dV0jT5dt4u+zyimv2sOJB+fz46ljGZKb2f0OiEjcMbP5zrnSiL+uVyHyuTcJjkRuds6dtd/y04FLnHOXmVkesAAY55yrau+1YjVEnHNc9bcylm+tprS4HxOLc5g4LJehuZnUN7ZQ19TM3oZm3lm9nfveXM3WXXVMLO7HDSeP5OiSXJKSvBklNDW38Nf3y/nNq6toanFcf+IIrjl+OGnJAU/eT0Rik1chEvXDe8zsdqDMOfcC8DJwqpktA5qB73YUILHs+YWbeWNFBZOG9WPuuiqmL9rSbtvSoTn88sLDObok1/NNTMmBJK4+djhnHTaIO2Ys41evruKZDzbxwzNHc/IhBdrEJSLdEpWRSCTF4khk195GTv7VWxTmZPDcdUdjBht37GXOuioqaupJTwmQkRIgPSWJwf0yKR2a49sf77dXVfLj6UtZU1nLsSPz+N+zRnNQ//aOexCRRBHXm7MiKRZD5LYXlvK3WeW88I1j4uJoqMbmFh6ZtZ57XltFbUMzl04cwjdPGkFBdrrfpXmmoqaOypp66hpbqG9spq6pmbTkAP2yUsnNSiUnK5UkM2rqGqne28SuvY30zUxhcD/tQ5LEkDCbsxLN0i27+PuscqZNGhoXAQKQEkjiymOGcc4Rhfz61ZU8PncDT5Zt5KuTh/K140vI7ZXmd4nd5pxj6ZZqXl22jddXbOPDzdWdPscM9v9OdcyIPL4yeSgnjyogOaBZgkT2p5FIN7S0OC64/33WV+3hje+cQJ/MFL9LCkv59lp+9/pHPL9wM+kpAS4/upirjx1Ovzg7YbGxuYU5a3fwyrKPeWXpNj6ursMMJgzJ4eRD+jMsL4v0lCTSUwKkpwSoa2xmR20DVbUN7NjdQHNLC9kZKfQJ3VZtq+GxORvYuquOQX3SOW98EQcP6M3w/CyG5WVpxgCJK9qcFeJXiDjnKK/aw669jRjBb62z1lRx579XcPcFh3Fh6eCo1xRpqyt2c89rq3hxyVYyUgJMO2ooVx87jILeXdvMtWtvIzMWb2HGoq00NLeQm5VKbq808nqlMqKgF0cW92NQ34xu17lgw06emr+JhqaWT5ftaWjivdVV7NrbSHpKEseNzOfUMQM48eD8bo2smppbeG15BY/MLuf9NVWfG6kU52ZyzXElfKm06D9GKXWNzcxeW0VRTiYl+Vk6gEF8pxAJiWaIOOdYua2G/1u8lReXbGVNZe1/1jM0hyevnezZIbp++GhbDffNXMO/Fm4mOZDEueMKGdwvg4zUZDJTgwcJJAeMgBmBJKOhuYWXPvyYV5Zto6GphZEFvSjITqNqdwPbdzewo7aeltCPWWHfDCYO68fpYwdw6gGeALm6Yje/fHklLy39mF5pyfTJ+Gzkl5QEE4tzOW1Mf44dmU9GauQPYd7b0Ex5VS3rtteytnI3r6+oYMGGTxiel8XNpx3MGWMHsOLjGp6Yu4HnFmymuq4JgJzMFCYMzWHC0H6MGtib4XlZFPbNOODNY5U19QDk947/zY0SfQqREK9CZN829GVbq1lbGfwjsXJbDeur9pBkMGlYLl88dACFORk499m288kluWSlJeZmjfLttdz/1hqeW7CZ+lbf+tvSNzOFc8YVcv74IsYWZn8uHJqaW1jxcQ1z1+1gXnnwtn13A0cW5/DDM0dz+OC+bb5mXWMzG3bsoTw08eRT8zeRnpzENceVcPWxw3z/f3fO8dryCu5+eQWrtu0mv3calTX1pCYnccbYAZwzrpCKmjrKyncyf/1O1m7/7EtISsAY0i+TAX3SyclMpV9WKjmZqWSmBggkGUlmJAeMiup6lm7ZxdIt1VTU1BNIMs4+fBBfP7Hkc7MYiHRGIRLiRYg457jz3yt48O21QPAXfGhuFsPzsjj2oHxOHzOgR3/7c87R0NxCXUMLexqb2NvQTHOLo9k5mpqDPz8j+/fq8gmMTc0tPFm2iV+/upLtuxs494hCzj2ikPU79rB6Ww2rK3eztrKWrbvqPn1OSsD48qShXH/SCPJibMd/c4vj+QWbeXHJVqaMyOO8IwrbnABzZ20Dayp3s3b7Z6OZypp6du5pZOeeBj7Z0/gfzwkkGSPyezFmUDajB2WzdVcdj8/ZQF1TM6eNHsD1J41gbGF8HNAh/lKIhEQ6RJpbHD94bglPzNvItKOGcPUxwynKOfBNDXLgauoa+ePMNTz07rpP929kpQYYUdCLkvxeFOdlMTQ3k+LcLIblZ5GdHp8HLnRVU3MLDc0twYAO3bLSkklP+Xw476ht4OH31vHX98uprW/iltNHce1xw7XfRTqkEAkJN0TqGptJCSQRaLXvoqGphZueXMiMxVu5/sQRfOfUg/SL6IMtn+xlbWUtJQVZDMhO12fQRdV1jdz6zBJeXLKVM8YO4O4LD6dXgm5ale5TiISEGyIPv7eOO/+9gmG5WZQUZFGS34uFGz/hnY+28/0vjuKa40o8qFbEW845HnpnHT9/aQXFuZk88JUJ2lcibVKIhIQbIvPX7+CVpdtYU7mbNZW1rK8K7uT86bmHcsnExJleXnqm99ds55uPL6CusZkHv1rKlBF5fpckMUYhEhKpfSL1Tc3UNbZ87jBRkXi2dddeLv/LPNZtr+X3lx7BaWMG+F2SxBCvQqTH7j1OSw4oQCShDOyTwT+vPYoxhdlc9+h8nizb6HdJ0gP02BARSUR9M1N59KpJTBmRxy1PL+ahd9b6XZIkOIWISILJSkvmoctKOWPsAH7y4nLumLGM5pb42mwt8UMhIpKA0pID3HvpeC4/upg/v7uOax+ZT219k99lSQJSiIgkqECScdvUMfx46hjeWLGNC++fxdZde/0uSxKMQkQkwV12dDF/vuxI1lfVcs4f3uPDzbv8LkkSiEJEpAc4cVQBT193NAEzvvTALN5Ysc3vkiRBKEREeohDBmbz3DemMDw/i6v/VsbfZ5X7XZIkAIWISA/SPzudJ6+dzEmjCvjRv5bqyC3pNoWISA+TmZrMA18p/fTIrcv+MpeK6rrOnyjSBoWISA+078itn593KGXrd3DGb9/hzZUVfpclcUghItKDXTxxCNOvP4b83mlc8fA87pixjPqmZr/LkjiiEBHp4Ub2783z35jy6eatix6YzTZt3pIuUoiICOkpAW6bOob7p41n1bYapt77Los2fuJ3WRIHFCIi8qnTxw7k2a8fTUogiQsfmMVzCzb5XZLEOIWIiHzOqAHZvHD9MYwf0pdv/3MRd720gni77pBEj0JERP5Dv6xUHrlqEpdMHMJ9M9fw/eeW6HwSaVOy3wWISGxKCSTxs3PHktcrld+/sZrqvU38+qLDSUsO+F2axBCFiIi0y8z4zqkH0ycjhZ+8uJzqukbunzaBrDT96ZAgbc4SkU5dfexw7r7gMN5bvZ0vPzSHnbUNfpckMUIhIiJdcmHpYP44bQLLtlZz4QOz2PKJrk0iChEROQCnjRnA36+cyLZddZz/x/dZXVHjd0niM89DxMwCZrbAzGa0se5yM6s0s4Wh29Ve1yMi3XPU8FyeuPYoGpsdF9w/iwUbdvpdkvgoGiORG4HlHaz/p3NuXOj2UBTqEZFuGjOoD89cN5ns9BS+8ue5LN2iqyX2VJ6GiJkVAWcCCgeRBDM0N4t/XnsUvdOTueLheWzWPpIeyeuRyD3ALUBLB23ON7PFZva0mQ32uB4RiaCBfTJ4+Ioj2dvQzOV/mcuuPY1+lyRR5lmImNlZQIVzbn4HzaYDxc65w4DXgL+181rXmFmZmZVVVlZ6UK2IhGvUgGwe+MoEyqtqueaRMk0l38N4ORKZAkw1s3LgCeAkM3u0dQPnXJVzrj708E/AhLZeyDn3oHOu1DlXmp+f72HJIhKOo0fk8csLD2fOuh3c/NRiWjRFSo/hWYg45251zhU554qBi4E3nHPTWrcxs4GtHk6l4x3wIhLDzh5XyPdOH8X0RVv48fSlmrSxh4j63AVmdjtQ5px7AbjBzKYCTcAO4PJo1yMikfO144ezo7aeP72zjj6Zqdz0hYP8Lkk8FpUQcc7NBGaG7v+o1fJbgVujUYOIeM/M+P4XD6F6bxO/e/0j+mSkcNUxw/wuSzykWdREJKLMjJ+ddyjVdY3cMWMZfTJSuGBCkd9liUc07YmIRFwgybjn4nEcMyKP7z2zmPnrd/hdknhEISIinkhLDnD/VyYwqG863/7nImrrm/wuSTygEBERz/RKS+bXXxrHxp17+MmLy/wuRzygEBERTx1Z3I+vHV/CP+Zu5LVl2/wuRyJMISIinvv2KQdxyMBs/ufZxVTtru/8CRI3FCIi4rnU5CTuuWgc1XubuPXZJToRMYEoREQkKg4e0JtbTj+YV5ZtY/rirX6XIxGiEBGRqLliyjDGFmbzsxeX62itBKEQEZGoCSQZP546lo+r67j3zdV+lyMRoBARkaiaMDSH88cX8dA7a1lbudvvcqSbFCIiEnXfO+Ng0pID3D5jmXayxzmFiIhEXUHvdL51ykhmrqzk9eUVfpcj3aAQERFfXHZ0MSMKenH7jGXUNepqiPFKISIivkgJJHHbf41hw449PD5ng9/lSJgUIiLim2NG5jFhaA4Pv7+OZl1SNy4pRETEV1cdM4yNO/by2nLNqxWPFCIi4qtTR/ensG8Gf353nd+lSBgUIiLiq+RAEpcfXczcdTv4cPMuv8uRA6QQERHfXTRxMFmpAf6i0UjcUYiIiO+y01O4sHQw0xdvoaK6zu9y5AAoREQkJlwxpZimFscjs9f7XYocAIWIiMSEoblZnHJIfx6bs0EnH8YRhYiIxIwrpwxjR20Dz36w2e9SpIsUIiISM44a3o/Di/pw38zVNDa3+F2OdIFCRERihplx4ykj2bRzL89+sMnvcqQLFCIiElNOPLiAw4r6cO+bGo3EA4WIiMQUM+PGk0eyccdentO+kZinEBGRmHPSqAIOLezD79/8SKORGKcQEZGYY2Z865TQaGSBRiOxTCEiIjFp32jk3je0bySWKUREJCbt2zeyYccejUZimEJERGLWyYcUMGZQNvfPXKOLVsUohYiIxCwz4+snjGDt9lpeXvqx3+VIGxQiIhLTTh87gGF5Wdw3czXOaTQSazwPETMLmNkCM5vRQZsLzMyZWanX9YhIfAkkGdceN5wPN1fzzkfb/S5H9hONkciNwPL2VppZb+AGYE4UahGROHTu+EL6Z6fxx5lr/C5F9uNpiJhZEXAm8FAHze4A7gJ0JRoRaVNacoD/PnY4s9ZWsWDDTr/LkVa8HoncA9wCtHmQt5kdAQx2zrW7qSvU7hozKzOzssrKSg/KFJFYd8nEIfTJSOE+jUZiimchYmZnARXOufntrE8CfgN8p7PXcs496Jwrdc6V5ufnR7hSEYkHWWnJXHZ0Ma8u28aqbTV+lyMhXQoRMysxs7TQ/RPM7AYz69vJ06YAU82sHHgCOMnMHm21vjcwFpgZanMU8IJ2rotIe644upiMlAD3azQSM7o6EnkGaDazEcCfgWHA4x09wTl3q3OuyDlXDFwMvOGcm9Zq/S7nXJ5zrjjUZjYw1TlXFkY/RKQHyMlK5cuThvCvRVvYULXH73KErodIi3OuCTgXuMc5921gYDhvaGa3m9nUcJ4rInLNccMJJBl/fGu136UIXQ+RRjO7BLgM2LcTPKWrb+Kcm+mcOyt0/0fOuRfaaHOCRiEi0pmC7HQuPnIwT8/fxOZP9vpdTo/X1RC5ApgM/NQ5t87MhgGPdvIcERFPXHt8CQAPvqV9I37rUog455Y5525wzv3DzHKA3s65n3tcm4hImwr7ZnD++CL+MW8jFdU6xcxPXT06a6aZZZtZP2AR8LCZ/drb0kRE2nfdCSU0tzj+9M5av0vp0bq6OauPc64aOA942Dk3ATjFu7JERDo2NDeLsw8fxKOzN1C1u97vcnqsroZIspkNBL7EZzvWRUR89fUTR1DX1Myf313ndyk9VldD5HbgZWCNc26emQ0HPvKuLBGRzo0o6MUZYwfw2JwN1DU2+11Oj9TVHetPOecOc85dF3q81jl3vreliYh07tKJQ9m1t1EXrfJJV3esF5nZc2ZWYWbbzOyZ0Ay9IiK+Orokl6KcDJ4s2+h3KT1SVzdnPQy8AAwCCoHpoWUiIr5KSjK+VDqY91ZXaSoUH3Q1RPKdcw8755pCt78Cmk5XRGLCBROKSDJ4ar5GI9HW1RDZbmbTQpe6DZjZNKDKy8JERLpqUN8Mjjson6fKNtHcouuwR1NXQ+RKgof3fgxsBS4gOBWKiEhMuKh0MB9X1/H2Kl24Lpq6enTWBufcVOdcvnOuwDl3DsETD0VEYsLJh/QnNyuVJ+Zt8LuUHqU7Vza8KWJViIh0U2pyEueNL+T15RVU1ugM9mjpTohYxKoQEYmAi44cTFOL49kPNvldSo/RnRDR3isRiSkjCnozYWgO/yzbiHP6ExUNHYaImdWYWXUbtxqC54yIiMSUi48czNrKWuas2+F3KT1ChyHinOvtnMtu49bbOZccrSJFRLrqrMMG0Ts9mcfnaAd7NHRnc5aISMzJSA1w/vgi/v3hVk0RHwUKERFJOJdOGkJjs+Pp+drB7jWFiIgknIP69+bI4hz+MXcDLTqD3VMKERFJSJdOGkJ51R5mrdUMTV5SiIhIQjpj7ED6Zqbw2Jz1fpeS0BQiIpKQ0lMCXDC+iFeWbqOips7vchKWQkREEtYlk4bQ1OJ4qkw72L2iEBGRhFWS34vJw3P5x9wNmiLeIwoREUloXz5qCJt27uWtVRV+l5KQFCIiktBOGzOAgt5p/H2WdrB7QSEiIgktJZDEpZOG8NaqSsq31/pdTsJRiIhIwrt04hACZjw6W6ORSFOIiEjCK8hO57SxA3iybCN7G5r9LiehKEREpEe4bHIx1XVN/GvhZr9LSSgKERHpEY4szmHUgN78fdZ6XbAqghQiItIjmBlfnVzMsq3VzF+/0+9yEobnIWJmATNbYGYz2lj3NTNbYmYLzexdMxvtdT0i0nOdc0TwglU63DdyojESuRFY3s66x51zhzrnxgF3Ab+OQj0i0kNlpiZzwYTgBas0n1ZkeBoiZlYEnAk81NZ651x1q4dZgDZUioinvjq5mKYWx2OzdfncSPB6JHIPcAvQ0l4DM/uGma0hOBK5oZ0215hZmZmVVVZWelOpiPQIw/KyOPHgAh6bs576Jh3u212ehYiZnQVUOOfmd9TOOfcH51wJ8D3gh+20edA5V+qcK83Pz/egWhHpSa6YUsz23Q1MX7TV71LinpcjkSnAVDMrB54ATjKzRzto/wRwjof1iIgAcMyIPEYW9OLh99bpcN9u8ixEnHO3OueKnHPFwMXAG865aa3bmNnIVg/PBD7yqh4RkX3MjMunFLN0SzXzynW4b3dE/TwRM7vdzKaGHl5vZkvNbCFwE3BZtOsRkZ7pvCOK6JORwsPvrfO7lLiWHI03cc7NBGaG7v+o1fIbo/H+IiL7y0gNcMnEITz49ho27dxDUU6m3yXFJZ2xLiI91lcnD8XMeEQnH4ZNISIiPdagvhmcPmYA/5i7gT0NTX6XE5cUIiLSo115zDCq65q4/621fpcSlxQiItKjTRiaw9njBvHHmatZXVHjdzlxRyEiIj3e/541mszUZL7/7Ie0tOi8kQOhEBGRHi+vVxrf/+Io5pbv4MmyjX6XE1cUIiIiwJdKBzNxWD9+9n/Lqayp97ucuKEQEREheBb7z849lLrGFu6YsczvcuKGQkREJGREQS+uO6GEFxZt4d2PtvtdTlxQiIiItPL1E0so7JvBr15dqckZu0AhIiLSSlpygOtOKGHBhk94b3WV3+XEPIWIiMh+LiwtYkB2Or99fZVGI51QiIiI7GffaGRe+U5mr93hdzkxTSEiItKGi44cTEHvNH73ui5z1BGFiIhIG9JTAlx7fAmz1lYxd51GI+1RiIiItOPSiUPI65Wq0UgHFCIiIu3ISA1wzXHDeXf1duav12ikLQoREZEOTDtqKLlZqfzyZR2p1RaFiIhIBzJTk7n+pBHMWlvFu6t1Fvv+FCIiIp24dNIQCvtmcNdLKzVV/H4UIiIinUhLDnDTFw5iyeZd/PvDj/0uJ6YoREREuuCcIwo5qH8vfvXKSpqaW/wuJ2YoREREuiCQZHz3tFGs3V7L0/M3+V1OzFCIiIh00SmHFDB+SF/uee0j6hqb/S4nJihERES6yMz43umj+Li6jr+8t87vcmKCQkRE5ABMGp7LqaP789vXPmJ1xW6/y/GdQkRE5AD95NyxZKQG+M5Ti3r8TnaFiIjIASronc4dZ49l0cZPeODttX6X4yuFiIhIGP7r8EGcedhA7nltFcu3Vvtdjm8UIiIiYbrj7LH0yUjhO08uoqGpZ27WUoiIiISpX1Yqd553GMu2VvP7N3rmdPEKERGRbvjC6P6cP76IP7y5ukdevEohIiLSTT8+ewyD+2XyrScWsGtPo9/lRJVCRESkm3qlJfPbi4+goqae7z+/pEddd8TzEDGzgJktMLMZbay7ycyWmdliM3vdzIZ6XY+IiBfGDe7LTacexIuLt/JUD5pbKxojkRuB5e2sWwCUOucOA54G7opCPSIinrj2uBImD8/ltheWsrayZ5zN7mmImFkRcCbwUFvrnXNvOuf2hB7OBoq8rEdExEuBJOM3F40jNTmJG55Y0CMO+/V6JHIPcAvQlf/Jq4B/t7XCzK4xszIzK6usrIxkfSIiETWgTzp3nX8YH26u5u6XV/hdjuc8CxEzOwuocM7N70LbaUApcHdb651zDzrnSp1zpfn5+RGuVEQksk4dM4BpRw3hT++s461Vif3F18uRyBRgqpmVA08AJ5nZo/s3MrNTgB8AU51z9R7WIyISNT88czQH9e/Fd55cSGVN4v5p8yxEnHO3OueKnHPFwMXAG865aa3bmNkRwAMEA6TCq1pERKItPSXA7y8ZT01dEzc/tYiWlsQ87Dfq54mY2e1mNjX08G6gF/CUmS00sxeiXY+IiFcOHtCbH541mrdWVSbsRaySo/EmzrmZwMzQ/R+1Wn5KNN5fRMQv0yYN4e1Vldz18krOOHQghX0z/C4ponTGuoiIh8yM26aOAeA3r67yuZrIU4iIiHissG8GXz1qKM9+sIlV22r8LieiFCIiIlHwjRNHkJWazF0vrfS7lIhSiIiIREFOVirXHj+c15Zvo6w8caaMV4iIiETJlccMI793Gr94aUXCzPSrEBERiZLM1GRuPHkk88p38saKxDg1TiEiIhJFFx05mGF5Wdz10kqamuN/gkaFiIhIFKUEkrjltINZua2Gv75f7nc53aYQERGJstPHDuCUQwr41Sur2LhjT+dPiGEKERGRKDMzbj97LEkG338uvi+nqxAREfHBoL4ZfO+MUbzz0XaeX7jZ73LCphAREfHJtElDGT+kL7dPX0bV7vicLl4hIiLik6Qk4xfnH8bu+ibumLHM73LCohAREfHRyP69+foJI3h+4RYenb3e73IOWFSmghcRkfZdf9IIlm7Zxf/+60Oy0gKce0SR3yV1mUYiIiI+Swkkce+l45k8PJebn1rMK0s/9rukLlOIiIjEgPSUAH/6aimHFfXh+scX8O5H2/0uqUu0OUtEJEZkpSXz18snctGDs7jyb/Mo6ptBs3O0OEdLC9x82kExt6lLISIiEkP6ZKbwyFWT+PWrK6mpayKQZATMMDP6Z6f7Xd5/UIiIiMSY/N5p3HneYX6X0SXaJyIiImFTiIiISNgUIiIiEjaFiIiIhE0hIiIiYVOIiIhI2BQiIiISNoWIiIiEzeLtsoxmVgnsP19yH2BXJ8s6erzvfutleUC4k9e0Vc+BtDnQ/nR23+u+dNQulvrSUZ1daaOfM302XRWLn02Wcy6/08oPlHMu7m/Ag50t6+jxvvv7LSuLZD0H0uZA+9PZfa/70lG7WOpLdz8b/Zzps+mpn01Ht0TZnDW9C8s6ejy9nTY8GsOfAAAHlklEQVSRrOdA2hxof7pyP1xdfY322sVSX7r6OpHqy/6P9XPWMX02Ha/z87NpV9xtzooWMytzzpX6XUckqC+xK5H6k0h9gcTqj5d9SZSRiBce9LuACFJfYlci9SeR+gKJ1R/P+qKRiIiIhE0jERERCVvCh4iZ/cXMKszswzCeO8HMlpjZajP7nZlZq3XfNLOVZrbUzO6KbNUd1hTx/pjZbWa22cwWhm5fjHzlbdbjyWcTWn+zmTkzy4tcxZ3W5MVnc4eZLQ59Lq+Y2aDIV95mPV705W4zWxHqz3Nm1jfylbdZjxd9uTD0u99iZp7vN+lOH9p5vcvM7KPQ7bJWyzv8vWqTV4d9xcoNOA4YD3wYxnPnApMBA/4NnBFafiLwGpAWelwQ5/25Dbg5ET6b0LrBwMsEzyfKi+f+ANmt2twA3B/HfTkVSA7d/wXwizjuyyHAwcBMoDRW+xCqr3i/Zf2AtaF/c0L3czrqb0e3hB+JOOfeBna0XmZmJWb2kpnNN7N3zGzU/s8zs4EEf4FnueD/7t+Bc0KrrwN+7pyrD71Hhbe9+IxH/fGFh335DXALENUdfl70xzlX3appFlHqk0d9ecU51xRqOhuIysXCPerLcufcymjUH3q/sPrQjtOAV51zO5xzO4FXgdPD/RuR8CHSjgeBbzrnJgA3A/e10aYQ2NTq8abQMoCDgGPNbI6ZvWVmR3pabee62x+A60ObGf5iZjneldqpbvXFzKYCm51zi7wutIu6/dmY2U/NbCPwZeBHHtbamUj8nO1zJcFvun6JZF/80pU+tKUQ2Njq8b5+hdXfHneNdTPrBRwNPNVqc19aW03bWLbvW2AywWHgUcCRwJNmNjyU3lEVof78Ebgj9PgO4FcEf8mjqrt9MbNM4AcEN5v4LkKfDc65HwA/MLNbgeuB/xfhUjsVqb6EXusHQBPwWCRr7KpI9sUvHfXBzK4AbgwtGwH8n5k1AOucc+fSfr/C6m+PCxGCo69PnHPjWi80swAwP/TwBYJ/WFsPt4uALaH7m4BnQ6Ex18xaCM5NU+ll4e3odn+cc9taPe9PwAwvC+5Ad/tSAgwDFoV+sYqAD8xsonPuY49rb0skftZaexx4ER9ChAj1JbQT9yzgZD++dIVE+nPxQ5t9AHDOPQw8DGBmM4HLnXPlrZpsAk5o9biI4L6TTYTTX693CMXCDSim1Q4p4H3gwtB9Aw5v53nzCI429u1k+mJo+deA20P3DyI4NLQ47s/AVm2+DTwRr33Zr005Udyx7tFnM7JVm28CT8dxX04HlgH50fxMvPw5I0o71sPtA+3vWF9HcGtKTuh+v670t826ov1h+vDD8w9gK9BIMGmvIvht9SVgUeiH+kftPLcU+BBYA9zLZydnpgKPhtZ9AJwU5/15BFgCLCb4DWxgvPZlvzblRPfoLC8+m2dCyxcTnAepMI77sprgF66FoVu0jjTzoi/nhl6rHtgGvByLfaCNEAktvzL0eawGruisvx3ddMa6iIiEracenSUiIhGgEBERkbApREREJGwKERERCZtCREREwqYQkbhnZruj/H4PmdnoCL1WswVn6P3QzKZ3NrOtmfU1s69H4r1FIkGH+ErcM7PdzrleEXy9ZPfZRIGeal27mf0NWOWc+2kH7YuBGc65sdGoT6QzGolIQjKzfDN7xszmhW5TQssnmtn7ZrYg9O/BoeWXm9lTZjYdeMXMTjCzmWb2tAWvgfHYvmsrhJaXhu7vDk2QuMjMZptZ/9DyktDjeWZ2exdHS7P4bCLJXmb2upl9YMHrO5wdavNzoCQ0erk71Pa7ofdZbGY/juB/o0inFCKSqH4L/MY5dyRwPvBQaPkK4Djn3BEEZ8T9WavnTAYuc86dFHp8BPAtYDQwHJjSxvtkAbOdc4cDbwP/3er9fxt6/07nHwrN23QywRkDAOqAc51z4wlev+ZXoRD7H2CNc26cc+67ZnYqMBKYCIwDJpjZcZ29n0ik9MQJGKVnOAUY3WqG02wz6w30Af5mZiMJzlCa0uo5rzrnWl+zYa5zbhOAmS0kOHfRu/u9TwOfTVg5H/hC6P5kPrsWw+PAL9upM6PVa88neG0HCM5d9LNQILQQHKH0b+P5p4ZuC0KPexEMlbfbeT+RiFKISKJKAiY75/a2XmhmvwfedM6dG9q/MLPV6tr9XqO+1f1m2v59aXSf7Vhsr01H9jrnxplZH4Jh9A3gdwSvHZIPTHDONZpZOZDexvMNuNM598ABvq9IRGhzliSqVwheewMAM9s3ZXYfYHPo/uUevv9sgpvRAC7urLFzbhfBy9/ebGYpBOusCAXIicDQUNMaoHerp74MXBm6vgRmVmhmBRHqg0inFCKSCDLNbFOr200E/yCXhnY2LyM4fT/AXcCdZvYeEPCwpm8BN5nZXGAgsKuzJzjnFhCckfVighdsKjWzMoKjkhWhNlXAe6FDgu92zr1CcHPZLDNbAjzN50NGxFM6xFfEA6GrLO51zjkzuxi4xDl3dmfPE4k32ici4o0JwL2hI6o+wYfLDYtEg0YiIiISNu0TERGRsClEREQkbAoREREJm0JERETCphAREZGwKURERCRs/x98S5IqB2+hpwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lyrics.recorder.plot(skip_end=15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:02 <p><table style='width:375px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "    <th>time</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>0</th>\n",
       "    <th>3.928599</th>\n",
       "    <th>3.641764</th>\n",
       "    <th>0.263244</th>\n",
       "    <th>00:02</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lyrics.fit_one_cycle(1, 1e-01, moms=(0.6,0.5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([array([  2,   5,  34,  59, ..., 108,  57,  89,  11]), array([  2,  12,  95, 247, ...,  39,   9,  39,  11])],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_lm_lyrics.valid_ds.items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lyrics.save('fit_head_lyrics')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lyrics.load('fit_head_lyrics');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To complete the fine-tuning, we can then unfeeze and launch a new training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lyrics.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:21 <p><table style='width:375px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "    <th>time</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>0</th>\n",
       "    <th>2.978756</th>\n",
       "    <th>3.615579</th>\n",
       "    <th>0.271577</th>\n",
       "    <th>00:03</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>2.750927</th>\n",
       "    <th>3.790543</th>\n",
       "    <th>0.262128</th>\n",
       "    <th>00:03</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>2.366141</th>\n",
       "    <th>3.809052</th>\n",
       "    <th>0.262351</th>\n",
       "    <th>00:03</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>3</th>\n",
       "    <th>1.943161</th>\n",
       "    <th>3.971256</th>\n",
       "    <th>0.266964</th>\n",
       "    <th>00:03</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>4</th>\n",
       "    <th>1.544134</th>\n",
       "    <th>4.060877</th>\n",
       "    <th>0.265551</th>\n",
       "    <th>00:03</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>5</th>\n",
       "    <th>1.219733</th>\n",
       "    <th>4.155464</th>\n",
       "    <th>0.263095</th>\n",
       "    <th>00:03</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lyrics.fit_one_cycle(6, 1e-2, moms=(0.8,0.7)) # moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lyrics.save('fine_tuned_lyrics')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How good is our model? Well let's try to see what it predicts after a few given words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lyrics.load('fine_tuned_lyrics');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEXT = 'love'# '[\"I\"'#'[\"I\", \n",
    "N_WORDS = 100\n",
    "N_SENTENCES = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "bridge_words = [\"Love\", \"Can\\'t\", \"Day\", \"Eight\", \"ah\", \"When\", \"Help\", \"Nah\", \"I\\'m\", \"That\", \"And\", \"She\", \"Because\", \"Yeah\", \"you\\'re\", \"I\", \"Life\"]\n",
    "verse_words = [\"There\\'s\", \"Something\", \"Nothing\", \"Can\\'t\", \"Ooh\", \"Saw\", \"Here\", \"He\", \"Got\", \"She\", \"Tried\", \"Love\", \"Eleanor\", \"Father\", \"Jo\", \"Sweet\", \"It\\'s\", \"you\", \"you\", \"I\", \"When\", \"Now\", \"Hey\", \"baby\\'s\", \"baby\", \"Oh\", \"Yeah\", \"Lady\", \"When\", \"And\", \"For\", \"I\", \"Love\", \"Dear\", \"It\\'s\", \"If\", \"In\", \"On\", \"Behind\", \"You\", \"I\", \"She\", \"Try\", \"Think\", \"In\", \"And\", \"So\", \"yesterday\", \"As\", \"Suddenly\", \"yesterday\"]\n",
    "chorus_words = [\"all\", \"We\", \"Why\", \"Penny\", \"We\", \"She\", \"I\", \"She\\'s\", \"Don\\'t\", \"We\",\"Paperback\", \"She\", \"Let\", \"Someone\", \"Can\\'t\", \"hold\", \"Come\", \"All\", \"She\", \"Get\", \"Oh\", \"When\", \"Hello\", \"You\", \"I\", \"Help\", \"And\", \"So\", \"I\\'m\", \"She\\'s\", \"I\", \"Tuesday\", \"See\", \"Friday\"]\n",
    "bridge_words = [x.lower() for x in bridge_words]       \n",
    "verse_words = [x.lower() for x in verse_words]\n",
    "chorus_words = [x.lower() for x in chorus_words]\n",
    "import random\n",
    "\n",
    "VERSE = random.choice(verse_words)\n",
    "VERSE2 = random.choice(verse_words)\n",
    "VERSE3 = random.choice(verse_words)\n",
    "CHORUS = random.choice(chorus_words)\n",
    "BRIDGE = random.choice(bridge_words)\n",
    "V_WORDS = 50\n",
    "C_WORDS = 50\n",
    "B_WORDS = 50\n",
    "N_SENTENCES = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dear The Way out ( the way that you shout ) But when i get home with you , i 'm thinking of it You know i 'm gon na be my baby Everybody 's got a baby , yeah , Well you know my\n",
      "friday there is a thing to hide a fool and about the thrill of love . Each one shows you , one , Make you love me for loving me . You know you may be true , But you may be glad . All you\n",
      "dear Though the sun was in the sky , the sky was blue , his sky was blue , City sky was blue , yellow , green , and blue , green , It 's a rich and beautiful thing . You know , here i 'm\n",
      "friday there is a thing to hide a fool and about the thrill of love . Each one shows you , one , Make you love me for loving me . You know you may be true , But you may be glad . All you\n",
      "because a car window shop made a thing to Buy , As they could It 's a feeling i said , i know you 're gon na have a real old boys , If you 're feeling sorry , i ca n't see you again .\n",
      "friday there is a thing to hide a fool and about the thrill of love . Each one shows you , one , Make you love me for loving me . You know you may be true , But you may be glad . All you\n"
     ]
    }
   ],
   "source": [
    "#print(\"\\n\".join(learn.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))\n",
    "chorus = \"\\n\".join(learn_lyrics.predict(CHORUS, C_WORDS, temperature=0.75) for _ in range(N_SENTENCES))\n",
    "print(\"\\n\".join(learn_lyrics.predict(VERSE, V_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))\n",
    "print(chorus)\n",
    "print(\"\\n\".join(learn_lyrics.predict(VERSE2, V_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))\n",
    "print(chorus)\n",
    "print(\"\\n\".join(learn_lyrics.predict(BRIDGE, B_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))\n",
    "print(chorus)\n",
    "#print(\"\\n\".join(learn.predict(VERSE3, V_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to save not only the model, but also its encoder, the part that's responsible for creating and updating the hidden state. For the next part, we don't care about the part that tries to guess the next word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lyrics.save_encoder('fine_tuned_enc_lyrics')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lyrics.export()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that language models can use a lot of GPU, so you may need to decrease batchsize here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs=48"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/random-train'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/valid-lyrics'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train-lyrics2'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/export.pkl'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/data_lm_lyrics.pkl'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/data.pkl'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train-lyrics'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/models'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/lyrics.pkl'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/.ipynb_checkpoints'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/music.pkl'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/data_lm.pkl'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train-lyrics-one-file')]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles')\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/help.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/eleanor-rigby.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/something.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/day-tripper.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/ticket-to-ride.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/hello-goodbye.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/get-back.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/she-loves-you.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/lady-madonna.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/yesterday.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/eight-days-a-week.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/love-me-do.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/a-hard-days-night.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/i-feel-fine.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/.ipynb_checkpoints'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/paperback-writer.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/cant-buy-me-love.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/penny-lane.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/we-can-work-it-out.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/yellow-submarine.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/hey-jude.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/all-you-need-is-love.txt'),\n",
       " PosixPath('/home/ubuntu/course-v3/nbs/dl1/data/beatles/train/I-want-to-hold-your-hand.txt')]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(path/'train').ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = (TextList.from_folder(path)\n",
    "           #Inputs: all the text files in path\n",
    "            .filter_by_folder(include=['train']) \n",
    "           #We may have other temp folders that contain text files so we only keep what's in train and test\n",
    "            .split_by_rand_pct(0.1)\n",
    "           #We randomly split and keep 10% (10,000 reviews) for validation\n",
    "            .label_for_lm()           \n",
    "           #We want to do a language model so we label accordingly\n",
    "            .databunch(bs=bs))\n",
    "data_lm.save('data_lm.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm = load_data(path, 'data_lm.pkl', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>idx</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>, 0.5 ] , [ \" iimin \" , \" c \" , 7 , 0.5 ] , [ \" iimin / xxmaj imaj \" , \" c \" , 7 , 1 ] , [ \" iimin / xxmaj imaj \" , \" w \" , 5 , 1 ] , \\n  [ \" bviimaj \" , \" w \" , 10 , 2 ] , [ \"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>\" , 4 , 0.5 ] , [ \" bviimaj \" , \" c \" , 2 , 1.5 ] , \\n  [ \" bviimaj \" , \" c \" , 2 , 2 ] , [ \" bviimaj / vimin \" , \" c \" , 2 , 2 ] , \\n  [ \" xxmaj vmaj7 \" , \" w \" , 7 , 1 ] ,</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>[ \" bviimaj \" , \" c \" , 2 , 1 ] , [ \" bviimaj \" , \" w \" , 5 , 0.5 ] , \\n  [ \" bviimaj \" , \" c \" , 5 , 2 ] , [ \" bviimaj \" , \" c \" , 4 , 0.5 ] , [ \" bviimaj \" , \" c \" , 2 , 1.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>\" , 7 , 0.5 ] , \\n  [ \" bviimaj \" , \" c \" , 7 , 1 ] , [ \" bviimaj \" , \" w \" , 5 , 1 ] , [ \" bviimaj \" , \" w \" , 4 , 0.5 ] , [ \" bviimaj \" , \" c \" , 2 , 1 ] , [ \" bviimaj \" ,</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>] , \\n  [ \" vimin6 \" , \" c \" , 0 , 0.5 ] , [ \" vimin6 \" , \" c \" , -3 , 1.5 ] , [ \" vimin6 \" , \" n \" , null , 1 ] , [ \" vimin6 \" , \" w \" , -3 , 0.5 ] , [ \" vimin6 \" , \" w \" , 9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_lm.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then put this in a learner object very easily with a model loaded with the pretrained weights. They'll be downloaded the first time you'll execute the following line and stored in `~/.fastai/models/` (or elsewhere if you specified different paths in your config file)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = language_model_learner(data_lm, AWD_LSTM, drop_mult=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4XGXd//H3N/u+tE33JU03CoW2NJRCWQuyqSCKCIoioPx4REBFHxcuN3hcEAXEBa0g6iPgAojAIzsUKEtLC903WrqXNmnTpEmzJ9/fHzOFENImbebkzCSf13XN1Zkz98x8754kn7nPch9zd0RERA5VUtgFiIhIYlOQiIhItyhIRESkWxQkIiLSLQoSERHpFgWJiIh0i4JERES6RUEiIiLdoiAREZFuSQm7gIM1YMAALy4uDrsMEZGEsnDhwp3uXhTEeydckBQXF7NgwYKwyxARSShmtjGo9w5s05aZZZjZfDNbbGbLzeyH+2l3oZmtiLa5L6h6REQkGEGOSBqAWe5eY2apwFwze9zdX9vXwMzGAd8GZrr7bjMbGGA9IiISgMCCxCPTCtdEH6ZGb+2nGv4i8Bt33x19TVlQ9YiISDACPWrLzJLNbBFQBjzt7vPaNRkPjDezl83sNTM7az/vc6WZLTCzBeXl5UGWLCIiBynQIHH3FnefAgwHppvZpHZNUoBxwCnAxcBdZlbQwfvMdvdSdy8tKgrkoAMRETlEPXIeibtXAnOA9iOOLcC/3b3J3dcDq4kEi4iIJIggj9oq2je6MLNM4HRgVbtmDwOnRtsMILKp6+2gahIRkdgL8qitIcCfzSyZSGD9w90fM7MbgQXu/gjwJHCGma0AWoBvuPuuAGuSbqqqa2Lr7jp27W1gd20TlbWNVNY2YUBmWjLpqclkpCQxJD+TCYNzKcpND7tkEQmYJdo120tLS10nJAavuaWVFe/sYf76ChZu3M3GXbVs2V3Lnvrmg3qf/tlpTBicy7CCTLLSkslISyYzNZmh+ZmcPKGIQXkZAfVARNoys4XuXhrEeyfcme2Ham1ZDY8t2cawgkyGF2YxvDCTwfkZpCb37unGWlqdLbtrqdjb+O6tsraJ6oZmquubqK5vpq6xhVZ33MFxahqaWbSpkr2NLQCM6p/FmKIcSosLGVGYxbDCTAbkpFOYlUpBVhr5mamYQV1TC/VNLdQ3trJldy0rt1ezevseVm2v5qW3dlLX1EJdUwuNza3v1nfksHxmHTaQGSX96ZedRl5mCnkZqWSlJWNmYf23ichB6DMjkkcXb+Oa+99837LkJGNkvyzGFGVTUpTD6AHZZKYm40T+qAKUFOVwxNC8hAyczRW1fOneN1i6teoDz5lBTloKuRkpZKYlk2SGGRhGWkoSk0fkM310f44d3S/mo4aWVuetsmqeXVnGc6vKeGPTbtr/GKYlJzG8MJMR/bIY1T+LIfmZtLrT0NxKQ3MLLS1OSVEORw7LZ/zgHNJTkmNao0hvE+SIpM8ECUBDcwvvVNaztbKOLbtr2VxRx/qde1lXXsPbO/e+75tyW5mpyUwdWUBpcT8mDMplUF46g/IyGJSXQVrKewGz7/8yHr5JP7tyB1/9+yIcuP5D4xnVP5vC7DT6Z6eRn5VKTloKSUnh1wmwq6aBFe/sYU9dM3vqm9hT10TF3kY2765lU0UtG3fVUt1mk1p6ShJmUN8UWV+pycaEwblMGVHA0SMLmTqykOL+WXGxHkTihYKkjaD2kbS0Ou9U1dHU4uz789Pizqp3qnl9QwULNlawYtseWtt/c05JorXVaXWn1SN/1PIz0yjMSqUwK42hBRkcP2YAM8cNYFhBZszrbq+5pZVbn17Db+es44ihedz5mWmM7J8V+OcGrbaxmeQkIy05CTPD3dlcUcfSrVUs2VrJ0i1VLNlSRU1DJHAKs1IZ2S+L/KzIuijITCUtJYmWVqLryklLTno3XAuz0xiQk86gvHQG5r7/C4JIb6AgaSPMne01Dc1srqhl+556yvbUs72qgbqmFpIsspksyYzGllYqaxvZvbeJ3bWNrCvfy86aBgBKBmRz1PB86ppa3v323djcysQheUwZUcDUkQUcPjTv3c007k5zq1Oxt5Ede+rZsaeBsup69jY009DUSn1zCw1NrVTWNVFe3cDOmga2V9Wza28jF08fwfc/egQZqX1nk09Lq7O2rIY3Nu1m0aZKtu+pjxxVVtfE7r2NNLc6SWbvrq/6plbqmlo6fK/+2WkM75fFMaMKObakP8cUF1KQldbDPRKJHQVJG4l21Ja7s2ZHDXPX7mTuW+Ws2VFDTnrKuzuVzYzl26p4p6oegCSDJDNa3D+w36C9JIP0lGQKslIZkJNOUW46RTnpnDh+AB85amgP9C7x1TW2UFHbSEVNIztrGtixp57t0dBeV1bDoi2VNDa3YgbjBuYwdmBkX9roAZF/xw7MIT8zNexuiHRKQdJGogVJV22vqmfR5t2seKealtbW6DdnIznJ6JedxqC8DAbmpjMwL53cjFTSU5IS8gCARFPf1MLizZXMX1/Bos2VrN+5l00VtTS32cY5OC+DcYNymDAolxkl/TluTH+y0/vMAZGSIBQkbfTWIJHE0dTSypbddbxdXsOaHTW8taOaNWXVvLWjhobmVlKTjdJR/ThpfBFHDstndFE2Q/Iy3j24Yd/+uM0VdaSnJjF2YA55GRrVSLB0HolIHElNTopu3srmtImD3l3e2NzKgo0VvLCmnBdWl3PzE+/NCJSekkRx/2zqm1vYurvufSMagCH5GYwdmMPhQ/KYNqqQaaMK6Z+jWQEkMWhEIhKQ8uoG1pbVsH7nXtbvrGH9zloy05IZUZjJyH5ZjOiXRV1jC2vKqlm7o4Y1ZdWs3l5NU0vkd7JkQDYzxvTno0cN5djR/eLmcG1JTNq01YaCRHqz+qYWlm6tYsGG3SzcWMEr63ZR29jC4LwMzp0ylPOnDmPikLywy5QEpCBpQ0EifUltYzPPrCzjkUVbmbO6nOZW5+TxRVx72limjeoXdnmSQBQkbShIpK/avbeR+1/fxF0vradibyPHj+nPNbPGcdyY/mGXJglAQdKGgkT6utrGZu6bt4nfv/g25dUNHDu6H185fbwCRQ5IQdKGgkQkor6phb/N38Rv56yjrLqBGSWRQJlRokCRD1KQtKEgEXm/+qYW7o8GSnl1A0ePLODKk8bwocMHkawjvSRKQdKGgkSkY/VNLfxjwWbuemk9mypqGT0gmytOGM2FpSM0CaUoSNpSkIgcWEur88Sy7cx+cR2Lt1RxxNA8bvvUFMYPyg27NAlRkEGirykivUxykvHho4bw8NUz+f1np7G9qp6P/Gouf3jxbVrbXwdBJAYUJCK9lJlx5hGDefKrJ3HK+CJ+9J+VXPyH19hcURt2adLLKEhEerkBOen8/rPTuOWCo1i+bQ8fuu0FZr+4juaWjq8IKnKwFCQifYCZ8cnSETz9tZM4YWwRP/7PKs77zcss21oVdmnSCyhIRPqQIfmZ/OFz07jzM0dTVt3Aub+eyy+eWk2L9p1INyhIRPoYM+PsI4fwzNdO5vypw/nVc2v5/D3zqdjbGHZpkqACCxIzyzCz+Wa22MyWm9kPD9D2AjNzMwvk0DQR+aD8zFR+ceFkfvrxI5m3voKP3PESizZXhl2WJKAgRyQNwCx3nwxMAc4ysxntG5lZLnAtMC/AWkRkPy6aPpIHrzqepCTjwt+9yv3zN4VdkiSYwILEI2qiD1Ojt442xN4E/AyoD6oWETmwI4fn89g1JzBjTH++/dBSbnx0hfabSJcFuo/EzJLNbBFQBjzt7vPaPT8VGOHujwVZh4h0riArjT9eWsplM4v548vr+eJfFlBd3xR2WZIAAg0Sd29x9ynAcGC6mU3a95yZJQG3Add39j5mdqWZLTCzBeXl5cEVLNLHpSQn8f2PHsH/fGwSL6wp54I7X9UJjNKpHjlqy90rgTnAWW0W5wKTgDlmtgGYATzS0Q53d5/t7qXuXlpUVNQDFYv0bZfMGMWfL5vOtqo6PnHnK6zfuTfskiSOBXnUVpGZFUTvZwKnA6v2Pe/uVe4+wN2L3b0YeA041901I6NIHDhh3AAeuOp4mludi2e/xsZdChPpWJAjkiHA82a2BHidyD6Sx8zsRjM7N8DPFZEYmTA4l3u/cCz1zS1cPFvzdEnHNI28iHRq2dYqPnPXPHIzUvjblTMYXpgVdklykDSNvIiEatKwfP56xbHsqWviM3fNY1dNQ9glSRxRkIhIlxw5PJ8/XT6d7VX1XPm/C6lvagm7JIkTChIR6bKjRxZy26emsHDjbv77gSUk2qZxCYaCREQOyjlHDuGbZx3GI4u3cdvTa8IuR+JAStgFiEjiuerkEjbs3Msdz61lZP9sLpg2POySJEQakYjIQTMz/uf8Scwc259vP7SEhRsrwi5JQqQgEZFDkpqcxG8/PY2hBZlc9dc32F6leVf7KgWJiByy/KxU/vC5UvY2NHPVXxfS0KwjufoiBYmIdMv4QbnceuFkFm2u5LsPL9ORXH2QgkREuu2sSUO4ZtZY/rFgC399bWPY5UgPU5CISEx89fTxnHbYQH746AoWbtwddjnSgxQkIhITSUnGrZ+awuD8DK69/02q6nRRrL5CQSIiMZOfmcqvLp7Kjj31fOtBnfneVyhIRCSmpo4s5BtnTuDxZdu5b/6msMuRHqAgEZGY++KJJZw0vogbH13Bqu17wi5HAqYgEZGYS0oybr1wMnmZqXz5vjepa9T5Jb2ZgkREAjEgJ53bPzWFtWU13PaMJnfszRQkIhKYmWMHcPH0Edw9dz3Lt1WFXY4EREEiIoH61lkTKcxK49sPLaWlVUdx9UYKEhEJVH5WKt//6OEs2VLFX17dEHY5EgAFiYgE7iNHDeGUCUX8/MnVbKusC7sciTEFiYgEzsy46bxJtDp879/LdaJiL6MgEZEeMaJfFl/90DieWbmDZ1aWhV2OxJCCRER6zOUzR1NSlM0tT67SjvdeJLAgMbMMM5tvZovNbLmZ/bCDNl8zsxVmtsTMnjWzUUHVIyLhS0lO4voPTWDNjhoeWbw17HIkRoIckTQAs9x9MjAFOMvMZrRr8yZQ6u5HAQ8APwuwHhGJA2dPGswRQ/O47em3aGxuDbsciYHAgsQjaqIPU6M3b9fmeXevjT58DRgeVD0iEh+SkoyvnzmBTRW1/GPB5rDLkRgIdB+JmSWb2SKgDHja3ecdoPkVwONB1iMi8eGU8UUcU1zIHc++RX2T5uFKdIEGibu3uPsUIiON6WY2qaN2ZnYJUArcsp/nrzSzBWa2oLy8PLiCRaRHmBnfOPMwyqobdJJiL9AjR225eyUwBzir/XNmdjpwA3Cuuzfs5/Wz3b3U3UuLiooCrVVEesb00f04eXwRv52zjup6XU0xkQV51FaRmRVE72cCpwOr2rWZCvyeSIjowHKRPubrZ0ygsraJu15aH3Yp0g1BjkiGAM+b2RLgdSL7SB4zsxvN7Nxom1uAHOCfZrbIzB4JsB4RiTNHDs/nrCMG88e566mq1agkUaUE9cbuvgSY2sHy77W5f3pQny8iieG608fxxPLt3DX3ba4/Y0LY5cgh0JntIhKqiUPy+PCRQ7jn5Q3s3tsYdjlyCBQkIhK6604fx97GZv7w0tthlyKHQEEiIqEbPyiXjxw1lD+9soEKjUoSjoJEROLCdaeNpa6phd+/uC7sUuQgKUhEJC6MHZjLeZOH8pdXNrKzpsNTyiROKUhEJG5ce9o4GppbmP2i9pUkEgWJiMSNkqIczpsyjL++tlH7ShKIgkRE4sqXThlDXVML97yss90ThYJEROLKuEG5nD1pMH96eQNVdTrbPREoSEQk7lx96liqG5r531c3hF2KdIGCRETizhFD8zntsIHcPXc9exuawy5HOqEgEZG4dPWsseyubeK+eZvCLkU6oSARkbh09MhCThg7gNkvva2rKMY5BYmIxK0vzxpLeXUDf39d13aPZwoSEYlbx47uR+moQn7/wjoam1vDLkf2Q0EiInHLzLh61li2VdXz70Vbwy5H9kNBIiJx7ZTxRRw+JI87X1hHS6uHXY50QEEiInHNzPjSqWN4u3wvTy7fHnY50gEFiYjEvbMnDWH0gGx+8/xa3DUqiTcKEhGJe8lJxn+dPIbl2/bw4ls7wy5H2lGQiEhC+NjUYQzJz+A3z68NuxRpR0EiIgkhLSWJK08qYf76Cl7fUBF2OdKGgkREEsZFx4ykX3aaRiVxpktBYmZjzCw9ev8UM7vWzAqCLU1E5P0y05K54oTRzFldzpItlWGXI1FdHZE8CLSY2VjgbmA0cN+BXmBmGWY238wWm9lyM/thB23SzezvZrbWzOaZWfFB1i8ifcznjhtFfmYqdzz7VtilSFRXg6TV3ZuB84Hb3f2rwJBOXtMAzHL3ycAU4Cwzm9GuzRXAbncfC9wG3Nz10kWkL8rNSOULJ4zmmZVlLNtaFXY5QteDpMnMLgYuBR6LLks90As8oqZN21Sg/QHg5wF/jt5/ADjNzKyLNYlIH3XpzGLyMlI0KokTXQ2Sy4DjgB+5+3ozGw38tbMXmVmymS0CyoCn3X1euybDgM0A0RFPFdC/q8WLSN+Ul5HK5SeM5qkVO1ixbU/Y5fR5XQoSd1/h7te6+/1mVgjkuvtPu/C6FnefAgwHppvZpHZNOhp9fOC0VTO70swWmNmC8vLyrpQsIr3cZcePJjc9hV89p1FJ2Lp61NYcM8szs37AYuAeM7u1qx/i7pXAHOCsdk9tAUZEPyMFyAc+cIC4u89291J3Ly0qKurqx4pIL5aflcplM4t5fNl2Vm+vDrucPq2rm7by3X0P8HHgHnefBpx+oBeYWdG+Q4TNLDPaflW7Zo8Q2e8CcAHwnGsiHRHpostPGE1Oegp3aFQSqq4GSYqZDQEu5L2d7Z0ZAjxvZkuA14nsI3nMzG40s3Ojbe4G+pvZWuBrwLcOonYR6eMKstK49PhR/GfpO6wt06gkLCldbHcj8CTwsru/bmYlwAG/Arj7EmBqB8u/1+Z+PfDJrpcrIvJ+V5xQwj0vb+DXz63l9os+8CdHekBXd7b/092Pcvf/ij5+290/EWxpIiKd65edxmdnjOKRxdtYv3Nv2OX0SV3d2T7czP5lZmVmtsPMHjSz4UEXJyLSFV84sYS0lCTNwRWSru4juYfIjvGhRM79eDS6TEQkdEW56Xx6+ij+9eZWNu2qDbucPqerQVLk7ve4e3P09idAx+GKSNz4fyeXkJxk3PmCRiU9ratBstPMLomeqZ5sZpcAu4IsTETkYAzKy+CiY0bwwMItbK2sC7ucPqWrQXI5kUN/twPvEDnn47KgihIRORRXnTwGgN/NWRdyJX1LV4/a2uTu57p7kbsPdPePETk5UUQkbgwtyOSCaSP4++ub2aZRSY/pzhUSvxazKkREYuTLs8YC8KvntK+kp3QnSDTdu4jEnWEFmXz62JH8c8FmNu7SeSU9oTtBojmxRCQufenUMaQkG7c/ozm4esIBg8TMqs1sTwe3aiLnlIiIxJ2BuRlcenwxDy/aypodmoMraAcMEnfPdfe8Dm657t7VebpERHrcVSeNIScthVufWhN2Kb1edzZtiYjErcLsNK44cTRPLN/O0i26tnuQFCQi0mtdccJoCrNS+flTq8MupVdTkIhIr5WbkcpVJ4/hhTXlLNjwgYuvSowoSESkV/vcccUMyEnTEVwBUpCISK+WmZbMVSePYe7anbyuUUkgFCQi0ut95thRDMhJ45calQRCQSIivZ5GJcFSkIhIn7BvVHL7MzqvJNYUJCLSJ+wblby8dhfz12tUEksKEhHpM97dV/KsRiWxpCARkT5Do5JgKEhEpE/ZNyq541kdwRUrChIR6VMy05K58qQS5q7dycKNGpXEQmBBYmYjzOx5M1tpZsvN7LoO2uSb2aNmtjjaRteBF5HAXTJjFP2y0/jls7qKYiwEOSJpBq5394nADOBqMzu8XZurgRXuPhk4BfiFmaUFWJOICFlpKXzxxBJeXFPOm5t2h11OwgssSNz9HXd/I3q/GlgJDGvfDMg1MwNygAoiASQiEqjPHTeKwqxU7SuJgR7ZR2JmxcBUYF67p34NTAS2AUuB69y9tYPXX2lmC8xsQXl5ecDVikhfkJ2ewhdOLOH51eUs3lwZdjkJLfAgMbMc4EHgK+6+p93TZwKLiFy2dwrwazPLa/8e7j7b3UvdvbSoqCjokkWkj7j0+GIKNCrptkCDxMxSiYTIve7+UAdNLgMe8oi1wHrgsCBrEhHZJyc9hStmjubZVWUs2aJRyaEK8qgtA+4GVrr7rftptgk4Ldp+EDABeDuomkRE2rt0ZjH9s9P47sPLaGn1sMtJSEGOSGYCnwVmmdmi6O0cM7vKzK6KtrkJON7MlgLPAt90950B1iQi8j55Gal8/9wjWLylij+9siHschJSSlBv7O5zAeukzTbgjKBqEBHpio8eNYSH39zKz59czRmHD2JEv6ywS0ooOrNdRPo8M+Omj00iyeCGh5fhrk1cB0NBIiICDCvI5L/POowX15Tz8KKtYZeTUBQkIiJRl8wYxdEjC7jx0RXsqmkIu5yEoSAREYlKTjJ++omjqGlo5pYnV4ddTsJQkIiItDF+UC6XzBjFPxZsZm1ZTdjlJAQFiYhIO18+dSxZaSnc8uSqsEtJCAoSEZF2+uek8/9OKuHJ5TtYuFGzA3dGQSIi0oErThzNgJx0bn58lQ4H7oSCRESkA1lpKVx3+jjmb6jg+dVlYZcT1xQkIiL7cdExIyjun8XNj6/WPFwHoCAREdmP1OQkvnHmYazeUc1Db2wJu5y4pSARETmAc44czOTh+dz69BrqGlvCLicuKUhERA7AzPjOORN5p6qeu+fqKhcdUZCIiHTi2JL+nHH4IO6cs47yak2d0p6CRESkC7519mE0NLdy+zNrwi4l7ihIRES6oKQoh0tmjOL++Zt4a0d12OXEFQWJiEgXXXvaOLLTUvjJ45o6pS0FiYhIF/XLTuPqWWN5blUZL6/VVcH3UZCIiByEzx9fzLCCTG56bAXNLa1hlxMXFCQiIgchIzWZGz48kVXbq7lv/qawy4kLChIRkYN09qTBzBzbn58/uVpXUkRBIiJy0MyMH3z0CGobW/j5U7qSooJEROQQjBuUy6XHF/O31zezZEtl2OWESkEiInKIrjt9HP2z0/n+I8tp7cOzAwcWJGY2wsyeN7OVZrbczK7bT7tTzGxRtM0LQdUjIhJreRmpfOvsw3hzUyUP9uHZgYMckTQD17v7RGAGcLWZHd62gZkVAL8FznX3I4BPBliPiEjMfXzqMI4eWcDNT6yiqq4p7HJCEViQuPs77v5G9H41sBIY1q7Zp4GH3H1TtJ0uQyYiCSUpybjxvElU7G3ktqf75jxcPbKPxMyKganAvHZPjQcKzWyOmS00s8/1RD0iIrE0aVg+l8wYxV9e3cDybVVhl9PjAg8SM8sBHgS+4u572j2dAkwDPgycCXzXzMZ38B5XmtkCM1tQXl4edMkiIgft+jMmUJiVxncfXtbndrwHGiRmlkokRO5194c6aLIFeMLd97r7TuBFYHL7Ru4+291L3b20qKgoyJJFRA5JfmYq3z5nIm9squSBPrbjPcijtgy4G1jp7rfup9m/gRPNLMXMsoBjiexLERFJOB+fOozSUYX89PFVVNX2nR3vQY5IZgKfBWZFD+9dZGbnmNlVZnYVgLuvBJ4AlgDzgbvcfVmANYmIBGbfjvfK2kZuearvTDWfEtQbu/tcwLrQ7hbglqDqEBHpSYcPzeNzxxXz51c3cMG0EUwZURB2SYHTme0iIjF2/RnjGZibznceWtonpppXkIiIxFhuRirf/+gRrHhnD396ZUPY5QROQSIiEoCzJw3m1AlF3Pr0GrZV1oVdTqAUJCIiATCL7HhvdecHjywPu5xAKUhERAIyol8W1502nqdW7OCp5dvDLicwChIRkQB94cTRTBiUyw8eWc7ehuawywmEgkREJECpyUn8+OOT2FZVz+3P9M5JHRUkIiIBmzaqHxdPH8EfX97Aim3tpxxMfAoSEZEe8M2zDqMgM5UbHl7a6yZ1VJCIiPSAgqw0bvjwRN7cVMn9r28Ku5yYUpCIiPSQ86cO47iS/tz8+CrKqxvCLidmFCQiIj3EzPif8ydR39TKj//TeyY6V5CIiPSgMUU5XHVyCf96cysvrOkdF+pTkIiI9LAvnTqWsQNz+PaDS6iuT/zrlihIRER6WEZqMj+74Ci276nnx/9J/OuWKEhEREJw9MhCvnhiCffP38Tct3aGXU63KEhERELy1Q+Np6Qom28m+CYuBYmISEgyUpO55YLJbKuq4yePJ+4mLgWJiEiIpo0q5AsnjOa+eZt46a3EPIpLQSIiErLrz5jAmKJsvvHPJVTVJt4mLgWJiEjIMlKTue1TU9hZ08D3HlkWdjkHTUEiIhIHjhpewDWzxvHvRdt4bMm2sMs5KAoSEZE4cfWpY5g8ooAb/rWMHXvqwy6nyxQkIiJxIiU5idsunExDcwvfeGAJ7okx3XxgQWJmI8zseTNbaWbLzey6A7Q9xsxazOyCoOoREUkEJUU5fOeciby4ppx75yXGdPNBjkiagevdfSIwA7jazA5v38jMkoGbgScDrEVEJGF8dsYoThg7gJ/8ZyWbK2rDLqdTgQWJu7/j7m9E71cDK4FhHTS9BngQKAuqFhGRRGJm/PQTR2JmfOuh+N/E1SP7SMysGJgKzGu3fBhwPvC7nqhDRCRRDC/M4jvnTOTltbu4b358b+IKPEjMLIfIiOMr7t7+qve3A99095ZO3uNKM1tgZgvKyxPzzE8RkYN18fQRnDB2AD/+v/jexBVokJhZKpEQudfdH+qgSSnwNzPbAFwA/NbMPta+kbvPdvdSdy8tKioKsmQRkbiRKJu4gjxqy4C7gZXufmtHbdx9tLsXu3sx8ADwJXd/OKiaREQSTSJs4koJ8L1nAp8FlprZouiy7wAjAdxd+0VERLrg4ukjePXtXfTLSgu7lA4FFiTuPhewg2j/+aBqERFJZGbGry6eGnYZ+6Uz20VEpFsUJCIi0i0KEhFOPzcnAAAITElEQVQR6RYFiYiIdIuCREREukVBIiIi3aIgERGRblGQiIhIt1i8zt2yP2ZWDmxstzgfqOpk2YEe77vfdtkAYOchltlRPQfT5mD709n97vSls1o7a9Ob1k1X+tJ+WZDrRj9nB16eqD9n+3uuu+sm292DmazQ3RP+BszubNmBHu+7327ZgljWczBtDrY/nd3vTl+625/etG660peeXDf6OeudP2fxuG46u/WWTVuPdmHZgR4/up82saznYNocbH+6cr87utOf3rRuutKX9suCXDf6OTvw8kT9Odvfc2GumwNKuE1bPcXMFrh7adh1xEJv6gv0rv6oL/GrN/Un6L70lhFJEGaHXUAM9aa+QO/qj/oSv3pTfwLti0YkIiLSLRqRiIhIt/T6IDGzP5pZmZktO4TXTjOzpWa21szuiF71cd9z15jZajNbbmY/i23VB6wp5v0xsx+Y2VYzWxS9nRP7yjusJ5B1E33+62bmZjYgdhV3WlMQ6+YmM1sSXS9PmdnQ2FfeYT1B9OUWM1sV7c+/zKwg9pXvt6Yg+vPJ6O9/q5kFvi+lO33Yz/tdamZvRW+Xtll+wN+tDgV5SFg83ICTgKOBZYfw2vnAcUQu0PU4cHZ0+anAM0B69PHABO/PD4Cv94Z1E31uBPAkkfONBiRyf4C8Nm2uBX6XwH05A0iJ3r8ZuDnB181EYAIwByiN1z5E6ytut6wf8Hb038Lo/cID9fdAt14/InH3F4GKtsvMbIyZPWFmC83sJTM7rP3rzGwIkV/iVz3yv/sX4GPRp/8L+Km7N0Q/oyzYXrwnoP6EIsC+3Ab8N9CjOwCD6I+772nTNJse6lNAfXnK3ZujTV8Dhgfbi/cE1J+V7r66J+qPft4h9WE/zgSedvcKd98NPA2cdah/J3p9kOzHbOAad58GfB34bQdthgFb2jzeEl0GMB440czmmdkLZnZMoNV2rrv9AfhydJPDH82sMLhSO9WtvpjZucBWd18cdKFd1O11Y2Y/MrPNwGeA7wVYa2di8XO2z+VEvu2GKZb9CUtX+tCRYcDmNo/39euQ+hvYNdvjlZnlAMcD/2yz6S+9o6YdLNv3bTCFyHBwBnAM8A8zK4kmeI+KUX/uBG6KPr4J+AWRX/Qe1d2+mFkWcAORTSihi9G6wd1vAG4ws28DXwa+H+NSOxWrvkTf6wagGbg3ljUejFj2JywH6oOZXQZcF102FviPmTUC6939fPbfr0Pqb58LEiKjsEp3n9J2oZklAwujDx8h8se17dB7OLAten8L8FA0OOabWSuRuWzKgyx8P7rdH3ff0eZ1fwAeC7LgA+huX8YAo4HF0V+s4cAbZjbd3bcHXHtHYvGz1tZ9wP8RQpAQo75Ed+p+BDgtjC9ebcR63YShwz4AuPs9wD0AZjYH+Ly7b2jTZAtwSpvHw4nsS9nCofQ36B1E8XADimmzgwp4Bfhk9L4Bk/fzuteJjDr27XQ6J7r8KuDG6P3xRIaIlsD9GdKmzVeBvyVqX9q12UAP7mwPaN2Ma9PmGuCBBO7LWcAKoKgn10nQP2v00M72Q+0D+9/Zvp7IlpXC6P1+Xelvh3WFsUJ7+IfnfuAdoIlI2l5B5FvrE8Di6A/29/bz2lJgGbAO+DXvncCZBvw1+twbwKwE78//AkuBJUS+hQ1J1L60a7OBnj1qK4h182B0+RIi8yYNS+C+rCXypWtR9NYjR6AF2J/zo+/VAOwAnozHPtBBkESXXx5dJ2uByzrr74FuOrNdRES6pa8etSUiIjGiIBERkW5RkIiISLcoSEREpFsUJCIi0i0KEukVzKymhz/vLjM7PEbv1WKR2X2Xmdmjnc2Ka2YFZvalWHy2SCzo8F/pFcysxt1zYvh+Kf7eBIOBalu7mf0ZWOPuPzpA+2LgMXef1BP1iXRGIxLptcysyMweNLPXo7eZ0eXTzewVM3sz+u+E6PLPm9k/zexR4CkzO8XM5pjZAxa5jsa9+67NEF1eGr1fE51YcbGZvWZmg6LLx0Qfv25mN3Zx1PQq701AmWNmz5rZGxa5PsR50TY/BcZERzG3RNt+I/o5S8zshzH8bxTplIJEerNfAre5+zHAJ4C7ostXASe5+1Qis+n+uM1rjgMudfdZ0cdTga8AhwMlwMwOPicbeM3dJwMvAl9s8/m/jH5+p/MVRed5Oo3I7AIA9cD57n40kWvg/CIaZN8C1rn7FHf/hpmdAYwDpgNTgGlmdlJnnycSK31x0kbpO04HDm8zM2qemeUC+cCfzWwckZlNU9u85ml3b3vNh/nuvgXAzBYRmetobrvPaeS9iS4XAh+K3j+O967lcB/w8/3UmdnmvRcSuTYEROY6+nE0FFqJjFQGdfD6M6K3N6OPc4gEy4v7+TyRmFKQSG+WBBzn7nVtF5rZr4Dn3f386P6GOW2e3tvuPRra3G+h49+ZJn9vZ+P+2hxInbtPMbN8IoF0NXAHkeuPFAHT3L3JzDYAGR283oCfuPvvD/JzRWJCm7akN3uKyPU7ADCzfdNt5wNbo/c/H+Dnv0ZkkxrARZ01dvcqIpfT/bqZpRKpsywaIqcCo6JNq4HcNi99Erg8en0KzGyYmQ2MUR9EOqUgkd4iy8y2tLl9jcgf5dLoDugVRKb/B/gZ8BMzexlIDrCmrwBfM7P5wBCgqrMXuPubRGZyvYjIhZ9KzWwBkdHJqmibXcDL0cOFb3H3p4hsOnvVzJYCD/D+oBEJlA7/FQlI9IqNde7uZnYRcLG7n9fZ60QSjfaRiARnGvDr6JFWlYRw+WKRnqARiYiIdIv2kYiISLcoSEREpFsUJCIi0i0KEhER6RYFiYiIdIuCREREuuX/A4H7QvY+m1dEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot(skip_end=15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:03 <p><table style='width:375px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "    <th>time</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>0</th>\n",
       "    <th>1.925745</th>\n",
       "    <th>0.881683</th>\n",
       "    <th>0.794643</th>\n",
       "    <th>00:03</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 1e-02, moms=(0.5,0.5)) # moms=(0.4,0.3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('fit_head')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('fit_head');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To complete the fine-tuning, we can then unfeeze and launch a new training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:15 <p><table style='width:375px; margin-bottom:10px'>\n",
       "  <tr>\n",
       "    <th>epoch</th>\n",
       "    <th>train_loss</th>\n",
       "    <th>valid_loss</th>\n",
       "    <th>accuracy</th>\n",
       "    <th>time</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>0</th>\n",
       "    <th>0.865443</th>\n",
       "    <th>0.568694</th>\n",
       "    <th>0.838542</th>\n",
       "    <th>00:05</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>1</th>\n",
       "    <th>0.650101</th>\n",
       "    <th>0.504369</th>\n",
       "    <th>0.847173</th>\n",
       "    <th>00:04</th>\n",
       "  </tr>\n",
       "  <tr>\n",
       "    <th>2</th>\n",
       "    <th>0.562353</th>\n",
       "    <th>0.538914</th>\n",
       "    <th>0.841518</th>\n",
       "    <th>00:04</th>\n",
       "  </tr>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(3, 1e-2, moms=(0.7,0.6)) # moms=(0.4,0.3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('fine_tuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How good is our model? Well let's try to see what it predicts after a few given words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('fine_tuned');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save_encoder('fine_tuned_enc_music')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.export()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "N_SENTENCES = 1\n",
    "V = 'xxverse'\n",
    "C = 'xxchorus'\n",
    "B = 'xxbridge'\n",
    "chars_in_group = 10\n",
    "V_WORDS = 32 * chars_in_group\n",
    "C_WORDS = 23 * chars_in_group\n",
    "B_WORDS = 15 * chars_in_group"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "bridge_words = [\"Love\", \"Can\\'t\", \"Day\", \"Eight\", \"ah\", \"When\", \"Help\", \"Nah\", \"I\\'m\", \"That\", \"And\", \"She\", \"Because\", \"Yeah\", \"you\\'re\", \"I\", \"Life\"]\n",
    "verse_words = [\"There\\'s\", \"Something\", \"Nothing\", \"Can\\'t\", \"Ooh\", \"Saw\", \"Here\", \"He\", \"Got\", \"She\", \"Tried\", \"Love\", \"Eleanor\", \"Father\", \"Jo\", \"Sweet\", \"It\\'s\", \"you\", \"you\", \"I\", \"When\", \"Now\", \"Hey\", \"baby\\'s\", \"baby\", \"Oh\", \"Yeah\", \"Lady\", \"When\", \"And\", \"For\", \"I\", \"Love\", \"Dear\", \"It\\'s\", \"If\", \"In\", \"On\", \"Behind\", \"You\", \"I\", \"She\", \"Try\", \"Think\", \"In\", \"And\", \"So\", \"yesterday\", \"As\", \"Suddenly\", \"yesterday\"]\n",
    "chorus_words = [\"all\", \"We\", \"Why\", \"Penny\", \"We\", \"She\", \"I\", \"She\\'s\", \"Don\\'t\", \"We\",\"Paperback\", \"She\", \"Let\", \"Someone\", \"Can\\'t\", \"hold\", \"Come\", \"All\", \"She\", \"Get\", \"Oh\", \"When\", \"Hello\", \"You\", \"I\", \"Help\", \"And\", \"So\", \"I\\'m\", \"She\\'s\", \"I\", \"Tuesday\", \"See\", \"Friday\"]\n",
    "bridge_words = [x.lower() for x in bridge_words]       \n",
    "verse_words = [x.lower() for x in verse_words]\n",
    "chorus_words = [x.lower() for x in chorus_words]\n",
    "\n",
    "\n",
    "VERSE_start = random.choice(verse_words)\n",
    "VERSE2_start = random.choice(verse_words)\n",
    "VERSE3_start = random.choice(verse_words)\n",
    "CHORUS_start = random.choice(chorus_words)\n",
    "BRIDGE_start = random.choice(bridge_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print(\"\\n\".join(learn.predict(TEXT, N_WORDS, temperature=0.75) for _ in range(N_SENTENCES)))\n",
    "chorus_lyrics = \"\".join(learn_lyrics.predict(CHORUS_start, C_WORDS, temperature=0.75))\n",
    "verse_lyrics = \"\".join(learn_lyrics.predict(VERSE_start, V_WORDS, temperature=0.75))\n",
    "verse2_lyrics = \"\".join(learn_lyrics.predict(VERSE2_start, V_WORDS, temperature=0.75))\n",
    "\n",
    "bridge_lyrics = \"\".join(learn_lyrics.predict(BRIDGE_start, B_WORDS, temperature=0.75))\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def remove_stray_quote_marks(string):\n",
    "    new_string = string.replace(\"'\", '')\n",
    "    return new_string.replace('\"', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleaned_chorus_lyrics = remove_stray_quote_marks(chorus_lyrics)\n",
    "cleaned_verse_lyrics = remove_stray_quote_marks(verse_lyrics)\n",
    "cleaned_verse2_lyrics = remove_stray_quote_marks(verse2_lyrics)\n",
    "cleaned_bridge_lyrics = remove_stray_quote_marks(bridge_lyrics)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "chorus_lyrics_split = shlex.split(cleaned_chorus_lyrics)\n",
    "cls = cleaned_chorus_lyrics.split()\n",
    "verse_lyrics_split = shlex.split(cleaned_verse_lyrics)\n",
    "verse2_lyrics_split = shlex.split(cleaned_verse2_lyrics)\n",
    "bridge_lyrics_split = shlex.split(cleaned_bridge_lyrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['she', ',', 'The', 'key', 'for', 'a', 'woman', 'that', 'Come', 'on', 'is', 'shake', 'it', 'Come', 'on', 'and', 'come', 'on', ',', 'come', 'on', ',', 'come', 'on', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'take', 'it', 'easy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'take', 'it', 'easy', ',', 'come', 'on', ',', 'come', 'on', ',', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'take', 'it', 'easy', ',', 'it', 's', 'such', 'a', 'joy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'take', 'it', 'easy', 'Come', 'on', 'take', 'it', 'easy', '.', 'They', 'sit', 'at', 'you', 'and', 'fly', ',', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'take', 'it', 'easy', ',', 'take', 'it', 'easy', 'Come', 'on', 'take', 'it', 'easy', 'Come', 'on', 'take', 'it', 'easy', ',', 'take', 'it', 'easy', 'Come', 'on', 'take', 'it', 'easy', '.', 'Come', 'on', 'come', 'on', 'come', 'on', 'come', 'on', ',', 'come', 'on', 'and', 'come', 'on', '.', 'Come', 'on', 'take', 'it', 'easy', ',', 'take', 'it', 'easy', 'Come', 'on', 'take', 'it', 'easy', '.', 'Come', 'on', 'little', 'baby', 'go', 'go', 'go', '!', 'Go', 'go', 'go', 'little']\n"
     ]
    }
   ],
   "source": [
    "print(chorus_lyrics_split)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['you', 'think', 'in', 'of', 'his', 'life', 'Evrything', 'has', 'been', 'like', 'having', 'been', 'realised', ',', 'For', 'at', 'nine', 'o', 'and', 'i', 'my', 'mind', 'is', 'in', 'light', 'Evrything', 'has', 'been', 'set', 'upon', 'my', 'mind', '.', 'Evrything', 'has', 'got', 'to', 'bring', 'down', 'though', 'my', 'evrything', 'has', 'been', 'right', 'I', 'm', 'so', 'sorry', 'i', 'know', 'that', 'i', 'm', 'lying', 'there', ',', 'i', 'm', 'there', 'Tell', 'me', 'why', 'i', 'm', 'far', 'apart', 'from', 'me', '.', 'Do', 'nt', 'let', 'me', 'know', 'why', 'you', 'cried', ',', 'When', 'i', 'saw', 'you', 'standing', 'there', ',', 'i', 'm', 'gon', 'na', 'walk', 'right', 'into', 'my', 'head', '.', 'Crying', ',', 'crying', ',', 'waiting', ',', 'hoping', ',', 'hoping', ',', 'hoping', ',', 'you', 'll', 'understand', ',', 'Please', 'll', 'be', 'all', 'right', ',', 'crying', ',', 'crying', ',', 'waiting', ',', 'hoping', ',', 'hoping', ',', 'you', 'll', 'meet', ',', 'i', 've', 'been', 'here', ',', 'i', 'll', 'be', 'here', '.', 'My', 'mind', ',', 'i', 'm', 'gon', 'na', 'sit', 'right', 'down', 'and', 'cry', 'over', 'you', '.', 'You', 've', 'got', 'me', 'too', 'crying', '.', 'i', 'm', 'gon', 'na', 'laugh', ',', 'i', 'm', 'gon', 'na', 'sit', 'right', 'down', 'and', 'cry', 'over', 'you', '.', 'Well', 'i', 'm', 'so', 'crying', ',', 'crying', ',', 'crying', ',', 'crying', ',', 'waiting', ',', 'hoping', ',', 'you', 'll', 'come', 'back', ',', 'i', 'll', 'make', 'you', 'look', 'a', '-', 'there', '.', 'Once', ',', 'i', 'll', 'be', 'there', ',', 'i', 've', 'got', 'the', 'hope', ',', 'i', 'll', 'begin', 'the', 'sea', 'in', 'an', 'octopus', 's', 'garden', '.', 'Look', 'at', 'you', 'all', 'the', 'time', '.', 'It', 's', 'time', 'for', 'me', 'to', 'feel', 'so', 'fine', '.', 'You', 're', 'so', 'happy', 'when', 'you', 'see', 'me', '.', 'Ooh', 'yeah', 'Yeah', 'i', 'm', 'gon', 'na', 'see', 'you', 'again', '.', 'Well', ',', 'i', 'told', 'you', 'all', 'night', 'long', '.', 'Well', 'in', 'my', 'mind', 'i', 'm', 'gon', 'na', 'walk', 'down', 'Street', 'far', 'away', ',', 'For', 'me']\n",
      "['she', ',', 'The', 'key', 'for', 'a', 'woman', 'that', 'Come', 'on', 'is', 'shake', 'it', 'Come', 'on', 'and', 'come', 'on', ',', 'come', 'on', ',', 'come', 'on', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'take', 'it', 'easy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'take', 'it', 'easy', ',', 'come', 'on', ',', 'come', 'on', ',', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'take', 'it', 'easy', ',', 'it', 's', 'such', 'a', 'joy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'take', 'it', 'easy', 'Come', 'on', 'take', 'it', 'easy', '.', 'They', 'sit', 'at', 'you', 'and', 'fly', ',', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'take', 'it', 'easy', ',', 'take', 'it', 'easy', 'Come', 'on', 'take', 'it', 'easy', 'Come', 'on', 'take', 'it', 'easy', ',', 'take', 'it', 'easy', 'Come', 'on', 'take', 'it', 'easy', '.', 'Come', 'on', 'come', 'on', 'come', 'on', 'come', 'on', ',', 'come', 'on', 'and', 'come', 'on', '.', 'Come', 'on', 'take', 'it', 'easy', ',', 'take', 'it', 'easy', 'Come', 'on', 'take', 'it', 'easy', '.', 'Come', 'on', 'little', 'baby', 'go', 'go', 'go', '!', 'Go', 'go', 'go', 'little']\n",
      "['dear', 'The', 'Sea', 'Ticket', 's', 'In', 'the', 'English', 'of', 'the', 'USSR', '.', 'The', 'rain', 'side', 'makes', 'all', 'the', 'lonely', 'people', '.', 'Crying', ',', 'crying', ',', 'waiting', ',', 'waiting', ',', 'hoping', ',', 'you', 'll', 'come', 'back', '.', 'All', 'the', 'lonely', 'people', '.', 'You', 'know', 'i', 'm', 'gon', 'na', 'walk', 'and', 'walk', 'over', ',', 'i', 'm', 'gon', 'na', 'sit', 'right', 'down', 'and', 'walk', 'down', 'and', 'walk', 'over', ',', 'But', 'i', 'm', 'by', 'you', 'i', 'm', 'gon', 'na', 'walk', 'right', 'right', 'over', 'and', 'i', 'ca', 'nt', 'break', 'past', 'the', 'trees', '.', 'Look', 'up', 'the', 'sun', ',', 'i', 'm', 'the', 'sun', '.', 'And', 'if', 'the', 'sun', 'comes', 'up', ',', 'you', 've', 'got', 'to', 'drive', 'it', 'Come', 'on', ',', 'come', 'on', ',', 'come', 'on', ',', 'take', 'it', 'easy', '.', 'Wo', 'nt', 'you', 'let', 'me', 'down', 'If', 'you', 'want', 'me', 'to', 'do', 'Well', ',', 'i', 'told', 'you', 'so', 'many', 'promising', 'lies', '.', 'i', 'told', 'you', 'about', 'the', 'love', 'that', 'we', 'once', 'had', '.', 'So', 'if', 'you', 'ca', 'nt', 'be', 'around', ',', 'you', 're', 'going', 'to', 'lose', 'that', 'girl', '.', 'You', 'know', 'it', 's', 'a', 'shame', 'My', 'Bonnie', ',', 'i', 'never', 'ever', 'knew', 'how', 'much', 'i', 'knew', 'you', '.', 'When', 'i', 'saw', 'you', 'standing', 'there', 'There', 'was', 'a', 'way', 'to', 'get', 'you', 'back', '.', 'If', 'you', 'ever', 'even', 'try', 'i', 'm', 'gon', 'na', 'walk', 'right', 'down', 'the', 'street', '.', 'You', 'say', 'you', 're', 'the', 'kind', 'of', 'man', '.', 'Do', 'nt', 'you', 'know', 'why', 'you', 'cried', ',', 'You', 'know', 'you', 're', 'gon', 'na', 'find', 'my', 'baby', ',', 'do', 'nt', 'you', 'know', 'you', 're', 'looking', 'like', 'you', '?', 'You', 'were', 'right', 'now', ',', 'you', 'know', ',', 'you', 'know', ',', 'you', 'know', 'i', 'know', 'you', 'know', 'that', 'you', 'know', 'that', 'you', 'love', 'me', 'so', 'fine', '.', 'You', 'know', 'i', 'know', 'it', 's']\n",
      "['she', ',', 'The', 'key', 'for', 'a', 'woman', 'that', 'Come', 'on', 'is', 'shake', 'it', 'Come', 'on', 'and', 'come', 'on', ',', 'come', 'on', ',', 'come', 'on', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'take', 'it', 'easy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'take', 'it', 'easy', ',', 'come', 'on', ',', 'come', 'on', ',', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'take', 'it', 'easy', ',', 'it', 's', 'such', 'a', 'joy', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'take', 'it', 'easy', 'Come', 'on', 'take', 'it', 'easy', '.', 'They', 'sit', 'at', 'you', 'and', 'fly', ',', 'Come', 'on', 'is', 'such', 'a', 'joy', 'Come', 'on', 'take', 'it', 'easy', ',', 'take', 'it', 'easy', 'Come', 'on', 'take', 'it', 'easy', 'Come', 'on', 'take', 'it', 'easy', ',', 'take', 'it', 'easy', 'Come', 'on', 'take', 'it', 'easy', '.', 'Come', 'on', 'come', 'on', 'come', 'on', 'come', 'on', ',', 'come', 'on', 'and', 'come', 'on', '.', 'Come', 'on', 'take', 'it', 'easy', ',', 'take', 'it', 'easy', 'Come', 'on', 'take', 'it', 'easy', '.', 'Come', 'on', 'little', 'baby', 'go', 'go', 'go', '!', 'Go', 'go', 'go', 'little']\n",
      "['life', 'from', 'the', 'Cold', 'War', 'of', 'the', 'End', 'of', 'the', 'world', 'is', 'not', 'as', 'hard', 'as', 'the', 'rest', 'of', 'World', 'Sky', '.', 'The', 'world', 'is', 'broken', 'and', 'the', 'sun', 'is', 'blue', '.', 'All', 'the', 'world', 'is', 'dead', ',', 'all', 'together', '.', 'All', 'people', 'need', 'loving', ',', 'some', 'people', ',', 'where', 'they', 'are', '.', 'All', 'the', 'lonely', 'people', ',', 'where', 'you', 'are', '.', 'All', 'you', 'need', 'is', 'love', ',', 'all', 'you', 'need', 'is', 'love', ',', 'All', 'you', 've', 'got', 'ta', 'do', 'is', 'love', 'me', 'all', 'alone', ',', 'You', 'know', 'i', 'love', 'you', ',', 'You', 'know', 'you', 'know', 'i', 'love', 'you', '.', 'Please', 'do', 'nt', 'you', 'know', 'that', 'you', 'know', 'what', 'to', 'do', '.', 'You', 'know', 'the', 'reason', 'why', ',', 'In', 'my', 'life', 'i', 'love', 'you', ',', 'and', 'you', 'know', 'my', 'love', 'is', 'so', 'hard', 'to', 'know']\n"
     ]
    }
   ],
   "source": [
    "print(verse_lyrics_split)\n",
    "print(chorus_lyrics_split)\n",
    "print(verse2_lyrics_split)\n",
    "print(chorus_lyrics_split)\n",
    "print(bridge_lyrics_split)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "keys = ['a', 'b', 'c', 'd', 'e', 'f', 'g']\n",
    "key_weights = [4 / 22, 1 / 22, 3 / 22, 2 / 22, 1 / 22, 2 / 22, 9 / 22 ]\n",
    "key = random.choices(\n",
    "         population=keys,\n",
    "         weights=key_weights,\n",
    "         k=1)\n",
    "key = key[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "tempos = [138, 98, 120, 102, 125, 65, 72, 112, 150, 165, 108, 138, 70, 78, 94, 99, 123, 138, 138, 137, 171, 100]\n",
    "tempo = random.choice(tempos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_line(line):\n",
    "    start = line.find('[')\n",
    "    clean_start = line[start:]\n",
    "    clean_end = ']'.join(clean_start.split(']')[:-1]) + ']'\n",
    "    return clean_end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_obj = {\n",
    "    'tempo': tempo,\n",
    "    'key': key,\n",
    "    'verse_one': [],\n",
    "    'verse_two': [],\n",
    "    'verse_three': [],\n",
    "    'chorus': [],\n",
    "    'bridge': []\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_chords_readable(chord):\n",
    "    lower_chord = chord.lower().strip()\n",
    "    chord_list = {\n",
    "        'isus': 'Isus',\n",
    "        'i6': 'I6',\n",
    "        'imaj': 'I',\n",
    "        'imaj7': 'I7',\n",
    "        'imaj / viidim': 'I/vii',\n",
    "        'imin': 'i',\n",
    "        'iimaj': 'II',\n",
    "        'iimaj7': 'II7',\n",
    "        'iimin': 'ii',\n",
    "        'iimin7': 'ii7',\n",
    "        'biiimaj': 'bIII',\n",
    "        'iiimin': 'iii',\n",
    "        'iiimin7': 'iii7',\n",
    "        'iiimaj': 'III',\n",
    "        'iiimaj7': 'III7',\n",
    "        'ivmaj': 'IV',\n",
    "        'ivmaj7': 'IV7',\n",
    "        'ivmaj / vmaj': 'IV/V',\n",
    "        'ivmin': 'iv',\n",
    "        'vmaj': 'V',\n",
    "        'vmaj7': 'V7',\n",
    "        'bvimaj': 'bVI',\n",
    "        'vimin': 'vi',\n",
    "        'vimin7': 'vi7',\n",
    "        'vimin / vmaj': 'vi/V',\n",
    "        'bviimaj': 'VII',\n",
    "        'xxbridge': 'I',\n",
    "        'w': 'IV',\n",
    "        'n': 'V',\n",
    "        'c': 'iv'\n",
    "    }\n",
    "    return chord_list[lower_chord]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "def simplify_chords(piece_str):\n",
    "    piece_list = json.loads(piece_str)\n",
    "    mod_piece_list = []\n",
    "    for i, piece in enumerate(piece_list):\n",
    "        chord = make_chords_readable(piece[0])\n",
    "        new_piece = []\n",
    "        new_piece.append(chord)\n",
    "        new_piece.extend([piece[1], piece[2], piece[3]])\n",
    "        mod_piece_list.append(new_piece)\n",
    "    return mod_piece_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: determine how many bars are in most choruses, verses etc \n",
    "# and then make sure the total length of all song parts is divisible by 4\n",
    "#TODO: remove excess white space between everything\n",
    "verse_str = clean_line(\"\".join(learn.predict(V, V_WORDS, temperature=0.75)))\n",
    "chorus_str = clean_line(\"\".join(learn.predict(C, C_WORDS, temperature=0.75)))\n",
    "bridge_str = clean_line(\"\".join(learn.predict(B, B_WORDS, temperature=0.75)))\n",
    "#print(verse)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "verse = simplify_chords('['+ verse_str + ']')\n",
    "chorus = simplify_chords('['+ chorus_str + ']')\n",
    "bridge = simplify_chords('['+ bridge_str + ']')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(verse)\n",
    "print(chorus)\n",
    "print(bridge)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "## THIS STEP IS WHERE I PUT LYRICS INTO\n",
    "#print(verse)\n",
    "# FIX changing original thing in place\n",
    "def replace_w_with_word(part_of_song, part_of_song_lyrics):\n",
    "    new_part_of_song = part_of_song\n",
    "    new_part_of_song_lyrics = part_of_song_lyrics\n",
    "    for i, grouping in enumerate(new_part_of_song):\n",
    "        if grouping[1] == ' w ':\n",
    "            grouping[1] = new_part_of_song_lyrics[i]\n",
    "    print(new_part_of_song)\n",
    "    return new_part_of_song"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(chorus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['I', ' n ', None, 2], ['I', 'think', -5, 0.5], ['I', 'in', 2, 1], ['I', 'of', 2, 0.5], ['I', 'his', 0, 0.5], ['I', 'life', -5, 0.33], ['I', 'Evrything', 0, 0.5], ['V', 'has', 0, 1.5], ['V', 'been', None, 2], ['V', 'like', 4, 0.5], ['V', 'having', 0, 1], ['V', 'been', 2, 0.5], ['V', 'realised', 2, 0.25], ['I', ' c ', 0, 0.5], ['I', 'For', 0, 1], ['I', 'at', 0, 0.5], ['IV', 'nine', 4, 0.25], ['IV', ' c ', 2, 0.5], ['IV', 'and', 2, 0.5], ['IV', 'i', 0, 0.5], ['IV', ' c ', 0, 0.5]]\n",
      "[['I', ' n ', None, 2], ['I', 'think', -5, 0.5], ['I', 'in', 2, 1], ['I', 'of', 2, 0.5], ['I', 'his', 0, 0.5], ['I', 'life', -5, 0.33], ['I', 'Evrything', 0, 0.5], ['V', 'has', 0, 1.5], ['V', 'been', None, 2], ['V', 'like', 4, 0.5], ['V', 'having', 0, 1], ['V', 'been', 2, 0.5], ['V', 'realised', 2, 0.25], ['I', ' c ', 0, 0.5], ['I', 'For', 0, 1], ['I', 'at', 0, 0.5], ['IV', 'nine', 4, 0.25], ['IV', ' c ', 2, 0.5], ['IV', 'and', 2, 0.5], ['IV', 'i', 0, 0.5], ['IV', ' c ', 0, 0.5]]\n",
      "[['I', 'she', 4, 0.5], ['I', ',', 5, 0.5], ['I', ' n ', None, 2.5], ['I', ' c ', 4, 0.5], ['I', 'for', 5, 0.5], ['I', 'a', 7, 0.5], ['I', ' c ', 4, 0.5], ['I', 'that', 4, 0.5], ['I', 'Come', 5, 0.5], ['I', 'on', 7, 1], ['I', 'is', 16, 0.5], ['I', 'shake', 9, 0.5], ['I', 'it', 4, 0.5], ['I', ' c ', 4, 0.5], ['I', 'on', 5, 0.5]]\n",
      "[['I', 'life', 0, 1], ['I', 'from', 0, 0.5], ['I', 'the', 0, 0.5], ['I', 'Cold', 0, 1], ['I', 'War', 0, 1], ['I', 'of', 0, 0.5], ['I', 'the', 4, 0.5], ['I', 'End', 0, 1], ['I', 'of', 0, 1]]\n"
     ]
    }
   ],
   "source": [
    "verse_1_complete = replace_w_with_word(verse, verse_lyrics_split)\n",
    "verse_2_complete = replace_w_with_word(verse, verse2_lyrics_split)\n",
    "chorus_complete = replace_w_with_word(chorus, chorus_lyrics_split)\n",
    "bridge_complete = replace_w_with_word(bridge, bridge_lyrics_split)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(verse_1_complete)\n",
    "print(verse_2_complete)\n",
    "print(chorus_complete)\n",
    "print(bridge_complete)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_obj['verse_one'] = verse_1_complete\n",
    "response_obj['verse_two'] = verse_2_complete\n",
    "response_obj['chorus'] = chorus_complete\n",
    "response_obj['bridge'] = bridge_complete\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"tempo\": 138, \"key\": \"d\", \"verse_one\": [[\"I\", \" n \", null, 2], [\"I\", \"think\", -5, 0.5], [\"I\", \"in\", 2, 1], [\"I\", \"of\", 2, 0.5], [\"I\", \"his\", 0, 0.5], [\"I\", \"life\", -5, 0.33], [\"I\", \"Evrything\", 0, 0.5], [\"V\", \"has\", 0, 1.5], [\"V\", \"been\", null, 2], [\"V\", \"like\", 4, 0.5], [\"V\", \"having\", 0, 1], [\"V\", \"been\", 2, 0.5], [\"V\", \"realised\", 2, 0.25], [\"I\", \" c \", 0, 0.5], [\"I\", \"For\", 0, 1], [\"I\", \"at\", 0, 0.5], [\"IV\", \"nine\", 4, 0.25], [\"IV\", \" c \", 2, 0.5], [\"IV\", \"and\", 2, 0.5], [\"IV\", \"i\", 0, 0.5], [\"IV\", \" c \", 0, 0.5]], \"verse_two\": [[\"I\", \" n \", null, 2], [\"I\", \"think\", -5, 0.5], [\"I\", \"in\", 2, 1], [\"I\", \"of\", 2, 0.5], [\"I\", \"his\", 0, 0.5], [\"I\", \"life\", -5, 0.33], [\"I\", \"Evrything\", 0, 0.5], [\"V\", \"has\", 0, 1.5], [\"V\", \"been\", null, 2], [\"V\", \"like\", 4, 0.5], [\"V\", \"having\", 0, 1], [\"V\", \"been\", 2, 0.5], [\"V\", \"realised\", 2, 0.25], [\"I\", \" c \", 0, 0.5], [\"I\", \"For\", 0, 1], [\"I\", \"at\", 0, 0.5], [\"IV\", \"nine\", 4, 0.25], [\"IV\", \" c \", 2, 0.5], [\"IV\", \"and\", 2, 0.5], [\"IV\", \"i\", 0, 0.5], [\"IV\", \" c \", 0, 0.5]], \"verse_three\": [], \"chorus\": [[\"I\", \"she\", 4, 0.5], [\"I\", \",\", 5, 0.5], [\"I\", \" n \", null, 2.5], [\"I\", \" c \", 4, 0.5], [\"I\", \"for\", 5, 0.5], [\"I\", \"a\", 7, 0.5], [\"I\", \" c \", 4, 0.5], [\"I\", \"that\", 4, 0.5], [\"I\", \"Come\", 5, 0.5], [\"I\", \"on\", 7, 1], [\"I\", \"is\", 16, 0.5], [\"I\", \"shake\", 9, 0.5], [\"I\", \"it\", 4, 0.5], [\"I\", \" c \", 4, 0.5], [\"I\", \"on\", 5, 0.5]], \"bridge\": [[\"I\", \"life\", 0, 1], [\"I\", \"from\", 0, 0.5], [\"I\", \"the\", 0, 0.5], [\"I\", \"Cold\", 0, 1], [\"I\", \"War\", 0, 1], [\"I\", \"of\", 0, 0.5], [\"I\", \"the\", 4, 0.5], [\"I\", \"End\", 0, 1], [\"I\", \"of\", 0, 1]]}\n"
     ]
    }
   ],
   "source": [
    "j= json.dumps(response_obj)\n",
    "print(j)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: add in lyrics\n",
    "# TODO: create server, first in flask, then in fastapi?\n",
    "# add functionality to change key, tempo, and save on front end\n",
    "# create front end and make it able to play twice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to save not only the model, but also its encoder, the part that's responsible for creating and updating the hidden state. For the next part, we don't care about the part that tries to guess the next word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pytorch_fastai",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}