t81_558_class_12_4_atari.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "klGNgWREsvQv"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jeffheaton/app_deep_learning/blob/main/t81_558_class_12_4_atari.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yzzbqc-JS2z9"
      },
      "source": [
        "# T81-558: Applications of Deep Neural Networks\n",
        "**Module 12: Reinforcement Learning**\n",
        "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
        "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pCrXGd_CS6eB"
      },
      "source": [
        "# Module 12 Video Material\n",
        "\n",
        "* Part 12.1: Introduction to Introduction to Gymnasium [[Video]](https://www.youtube.com/watch?v=FvuyrpzvwdI&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_12_1_reinforcement.ipynb)\n",
        "* Part 12.2: Introduction to Q-Learning [[Video]](https://www.youtube.com/watch?v=VKuqvbG_KAw&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_12_2_qlearningreinforcement.ipynb)\n",
        "* Part 12.3: Stable Baselines Q-Learning [[Video]](https://www.youtube.com/watch?v=kl7zsCjULN0&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_12_3_pytorch_reinforce.ipynb)\n",
        "* **Part 12.4: Atari Games with Stable Baselines Neural Networks** [[Video]](https://www.youtube.com/watch?v=maLA1_d4pzQ&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_12_4_atari.ipynb)\n",
        "* Part 12.5: Future of Reinforcement Learning [[Video]](https://www.youtube.com/watch?v=-euo5pTjP8E&list=PLjy4p-07OYzuy_lHcRW8lPTLPTTOmUpmi) [[Notebook]](t81_558_class_12_5_rl_future.ipynb)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pmDI-h7cI0tI"
      },
      "source": [
        "# Google CoLab Instructions\n",
        "\n",
        "The following code ensures that Google CoLab is running the correct version of TensorFlow, and has the necessary Python libraries installed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9KQhYThvTCQC",
        "outputId": "9673a6c4-0366-4d70-9645-f6c25c84b664"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Note: using Google CoLab\n",
            "Collecting stable-baselines3[extra]\n",
            "  Downloading stable_baselines3-2.1.0-py3-none-any.whl (178 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m178.7/178.7 kB\u001b[0m \u001b[31m5.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting gymnasium\n",
            "  Downloading gymnasium-0.29.1-py3-none-any.whl (953 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m953.9/953.9 kB\u001b[0m \u001b[31m18.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.20 in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (1.23.5)\n",
            "Requirement already satisfied: torch>=1.13 in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (2.1.0+cu118)\n",
            "Requirement already satisfied: cloudpickle in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (2.2.1)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (1.5.3)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (3.7.1)\n",
            "Requirement already satisfied: opencv-python in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (4.8.0.76)\n",
            "Requirement already satisfied: pygame in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (2.5.2)\n",
            "Requirement already satisfied: tensorboard>=2.9.1 in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (2.14.1)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (5.9.5)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (4.66.1)\n",
            "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (13.6.0)\n",
            "Collecting shimmy[atari]~=1.1.0 (from stable-baselines3[extra])\n",
            "  Downloading Shimmy-1.1.0-py3-none-any.whl (37 kB)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.10/dist-packages (from stable-baselines3[extra]) (9.4.0)\n",
            "Collecting autorom[accept-rom-license]~=0.6.1 (from stable-baselines3[extra])\n",
            "  Downloading AutoROM-0.6.1-py3-none-any.whl (9.4 kB)\n",
            "Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.10/dist-packages (from gymnasium) (4.5.0)\n",
            "Collecting farama-notifications>=0.0.1 (from gymnasium)\n",
            "  Downloading Farama_Notifications-0.0.4-py3-none-any.whl (2.5 kB)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from autorom[accept-rom-license]~=0.6.1->stable-baselines3[extra]) (8.1.7)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from autorom[accept-rom-license]~=0.6.1->stable-baselines3[extra]) (2.31.0)\n",
            "Collecting AutoROM.accept-rom-license (from autorom[accept-rom-license]~=0.6.1->stable-baselines3[extra])\n",
            "  Downloading AutoROM.accept-rom-license-0.6.1.tar.gz (434 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m434.7/434.7 kB\u001b[0m \u001b[31m21.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting ale-py~=0.8.1 (from shimmy[atari]~=1.1.0->stable-baselines3[extra])\n",
            "  Downloading ale_py-0.8.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m28.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (1.4.0)\n",
            "Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (1.59.2)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (2.17.3)\n",
            "Requirement already satisfied: google-auth-oauthlib<1.1,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (1.0.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (3.5.1)\n",
            "Requirement already satisfied: protobuf>=3.19.6 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (3.20.3)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (67.7.2)\n",
            "Requirement already satisfied: six>1.9 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (1.16.0)\n",
            "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (0.7.2)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.9.1->stable-baselines3[extra]) (3.0.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=1.13->stable-baselines3[extra]) (3.13.1)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.13->stable-baselines3[extra]) (1.12)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.13->stable-baselines3[extra]) (3.2.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13->stable-baselines3[extra]) (3.1.2)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch>=1.13->stable-baselines3[extra]) (2023.6.0)\n",
            "Requirement already satisfied: triton==2.1.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13->stable-baselines3[extra]) (2.1.0)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3[extra]) (1.2.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3[extra]) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3[extra]) (4.44.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3[extra]) (1.4.5)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3[extra]) (23.2)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3[extra]) (3.1.1)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib->stable-baselines3[extra]) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->stable-baselines3[extra]) (2023.3.post1)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->stable-baselines3[extra]) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->stable-baselines3[extra]) (2.16.1)\n",
            "Requirement already satisfied: importlib-resources in /usr/local/lib/python3.10/dist-packages (from ale-py~=0.8.1->shimmy[atari]~=1.1.0->stable-baselines3[extra]) (6.1.0)\n",
            "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.9.1->stable-baselines3[extra]) (5.3.2)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.9.1->stable-baselines3[extra]) (0.3.0)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.9.1->stable-baselines3[extra]) (4.9)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard>=2.9.1->stable-baselines3[extra]) (1.3.1)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich->stable-baselines3[extra]) (0.1.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->autorom[accept-rom-license]~=0.6.1->stable-baselines3[extra]) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->autorom[accept-rom-license]~=0.6.1->stable-baselines3[extra]) (3.4)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->autorom[accept-rom-license]~=0.6.1->stable-baselines3[extra]) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->autorom[accept-rom-license]~=0.6.1->stable-baselines3[extra]) (2023.7.22)\n",
            "Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard>=2.9.1->stable-baselines3[extra]) (2.1.3)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.13->stable-baselines3[extra]) (1.3.0)\n",
            "Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard>=2.9.1->stable-baselines3[extra]) (0.5.0)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<1.1,>=0.5->tensorboard>=2.9.1->stable-baselines3[extra]) (3.2.2)\n",
            "Building wheels for collected packages: AutoROM.accept-rom-license\n",
            "  Building wheel for AutoROM.accept-rom-license (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for AutoROM.accept-rom-license: filename=AutoROM.accept_rom_license-0.6.1-py3-none-any.whl size=446660 sha256=4d531e9e9ac9e6348ced37ba173e62882d93ea6b7ba0160103d147551be2167d\n",
            "  Stored in directory: /root/.cache/pip/wheels/6b/1b/ef/a43ff1a2f1736d5711faa1ba4c1f61be1131b8899e6a057811\n",
            "Successfully built AutoROM.accept-rom-license\n",
            "Installing collected packages: farama-notifications, gymnasium, ale-py, shimmy, AutoROM.accept-rom-license, autorom, stable-baselines3\n",
            "Successfully installed AutoROM.accept-rom-license-0.6.1 ale-py-0.8.1 autorom-0.6.1 farama-notifications-0.0.4 gymnasium-0.29.1 shimmy-1.1.0 stable-baselines3-2.1.0\n",
            "Requirement already satisfied: gymnasium[accept-rom-license,atari] in /usr/local/lib/python3.10/dist-packages (0.29.1)\n",
            "Requirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.10/dist-packages (from gymnasium[accept-rom-license,atari]) (1.23.5)\n",
            "Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from gymnasium[accept-rom-license,atari]) (2.2.1)\n",
            "Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.10/dist-packages (from gymnasium[accept-rom-license,atari]) (4.5.0)\n",
            "Requirement already satisfied: farama-notifications>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from gymnasium[accept-rom-license,atari]) (0.0.4)\n",
            "Collecting shimmy[atari]<1.0,>=0.1.0 (from gymnasium[accept-rom-license,atari])\n",
            "  Downloading Shimmy-0.2.1-py3-none-any.whl (25 kB)\n",
            "Collecting autorom[accept-rom-license]~=0.4.2 (from gymnasium[accept-rom-license,atari])\n",
            "  Downloading AutoROM-0.4.2-py3-none-any.whl (16 kB)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from autorom[accept-rom-license]~=0.4.2->gymnasium[accept-rom-license,atari]) (8.1.7)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from autorom[accept-rom-license]~=0.4.2->gymnasium[accept-rom-license,atari]) (2.31.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from autorom[accept-rom-license]~=0.4.2->gymnasium[accept-rom-license,atari]) (4.66.1)\n",
            "Requirement already satisfied: AutoROM.accept-rom-license in /usr/local/lib/python3.10/dist-packages (from autorom[accept-rom-license]~=0.4.2->gymnasium[accept-rom-license,atari]) (0.6.1)\n",
            "Requirement already satisfied: ale-py~=0.8.1 in /usr/local/lib/python3.10/dist-packages (from shimmy[atari]<1.0,>=0.1.0->gymnasium[accept-rom-license,atari]) (0.8.1)\n",
            "Requirement already satisfied: importlib-resources in /usr/local/lib/python3.10/dist-packages (from ale-py~=0.8.1->shimmy[atari]<1.0,>=0.1.0->gymnasium[accept-rom-license,atari]) (6.1.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->autorom[accept-rom-license]~=0.4.2->gymnasium[accept-rom-license,atari]) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->autorom[accept-rom-license]~=0.4.2->gymnasium[accept-rom-license,atari]) (3.4)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->autorom[accept-rom-license]~=0.4.2->gymnasium[accept-rom-license,atari]) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->autorom[accept-rom-license]~=0.4.2->gymnasium[accept-rom-license,atari]) (2023.7.22)\n",
            "Installing collected packages: shimmy, autorom\n",
            "  Attempting uninstall: shimmy\n",
            "    Found existing installation: Shimmy 1.1.0\n",
            "    Uninstalling Shimmy-1.1.0:\n",
            "      Successfully uninstalled Shimmy-1.1.0\n",
            "  Attempting uninstall: autorom\n",
            "    Found existing installation: AutoROM 0.6.1\n",
            "    Uninstalling AutoROM-0.6.1:\n",
            "      Successfully uninstalled AutoROM-0.6.1\n",
            "Successfully installed autorom-0.4.2 shimmy-0.2.1\n",
            "Collecting pyvirtualdisplay\n",
            "  Downloading PyVirtualDisplay-3.0-py3-none-any.whl (15 kB)\n",
            "Installing collected packages: pyvirtualdisplay\n",
            "Successfully installed pyvirtualdisplay-3.0\n",
            "Reading package lists... Done\n",
            "Building dependency tree... Done\n",
            "Reading state information... Done\n",
            "E: Unable to locate package python-opengl\n",
            "Reading package lists... Done\n",
            "Building dependency tree... Done\n",
            "Reading state information... Done\n",
            "The following additional packages will be installed:\n",
            "  libfontenc1 libxfont2 libxkbfile1 x11-xkb-utils xfonts-base xfonts-encodings\n",
            "  xfonts-utils xserver-common\n",
            "The following NEW packages will be installed:\n",
            "  libfontenc1 libxfont2 libxkbfile1 x11-xkb-utils xfonts-base xfonts-encodings\n",
            "  xfonts-utils xserver-common xvfb\n",
            "0 upgraded, 9 newly installed, 0 to remove and 19 not upgraded.\n",
            "Need to get 7,814 kB of archives.\n",
            "After this operation, 11.9 MB of additional disk space will be used.\n",
            "Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfontenc1 amd64 1:1.1.4-1build3 [14.7 kB]\n",
            "Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxfont2 amd64 1:2.0.5-1build1 [94.5 kB]\n",
            "Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxkbfile1 amd64 1:1.1.0-1build3 [71.8 kB]\n",
            "Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 x11-xkb-utils amd64 7.7+5build4 [172 kB]\n",
            "Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 xfonts-encodings all 1:1.0.5-0ubuntu2 [578 kB]\n",
            "Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 xfonts-utils amd64 1:7.7+6build2 [94.6 kB]\n",
            "Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 xfonts-base all 1:1.0.5 [5,896 kB]\n",
            "Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 xserver-common all 2:21.1.4-2ubuntu1.7~22.04.2 [28.1 kB]\n",
            "Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 xvfb amd64 2:21.1.4-2ubuntu1.7~22.04.2 [864 kB]\n",
            "Fetched 7,814 kB in 2s (4,066 kB/s)\n",
            "debconf: unable to initialize frontend: Dialog\n",
            "debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78, <> line 9.)\n",
            "debconf: falling back to frontend: Readline\n",
            "debconf: unable to initialize frontend: Readline\n",
            "debconf: (This frontend requires a controlling tty.)\n",
            "debconf: falling back to frontend: Teletype\n",
            "dpkg-preconfigure: unable to re-open stdin: \n",
            "Selecting previously unselected package libfontenc1:amd64.\n",
            "(Reading database ... 120874 files and directories currently installed.)\n",
            "Preparing to unpack .../0-libfontenc1_1%3a1.1.4-1build3_amd64.deb ...\n",
            "Unpacking libfontenc1:amd64 (1:1.1.4-1build3) ...\n",
            "Selecting previously unselected package libxfont2:amd64.\n",
            "Preparing to unpack .../1-libxfont2_1%3a2.0.5-1build1_amd64.deb ...\n",
            "Unpacking libxfont2:amd64 (1:2.0.5-1build1) ...\n",
            "Selecting previously unselected package libxkbfile1:amd64.\n",
            "Preparing to unpack .../2-libxkbfile1_1%3a1.1.0-1build3_amd64.deb ...\n",
            "Unpacking libxkbfile1:amd64 (1:1.1.0-1build3) ...\n",
            "Selecting previously unselected package x11-xkb-utils.\n",
            "Preparing to unpack .../3-x11-xkb-utils_7.7+5build4_amd64.deb ...\n",
            "Unpacking x11-xkb-utils (7.7+5build4) ...\n",
            "Selecting previously unselected package xfonts-encodings.\n",
            "Preparing to unpack .../4-xfonts-encodings_1%3a1.0.5-0ubuntu2_all.deb ...\n",
            "Unpacking xfonts-encodings (1:1.0.5-0ubuntu2) ...\n",
            "Selecting previously unselected package xfonts-utils.\n",
            "Preparing to unpack .../5-xfonts-utils_1%3a7.7+6build2_amd64.deb ...\n",
            "Unpacking xfonts-utils (1:7.7+6build2) ...\n",
            "Selecting previously unselected package xfonts-base.\n",
            "Preparing to unpack .../6-xfonts-base_1%3a1.0.5_all.deb ...\n",
            "Unpacking xfonts-base (1:1.0.5) ...\n",
            "Selecting previously unselected package xserver-common.\n",
            "Preparing to unpack .../7-xserver-common_2%3a21.1.4-2ubuntu1.7~22.04.2_all.deb ...\n",
            "Unpacking xserver-common (2:21.1.4-2ubuntu1.7~22.04.2) ...\n",
            "Selecting previously unselected package xvfb.\n",
            "Preparing to unpack .../8-xvfb_2%3a21.1.4-2ubuntu1.7~22.04.2_amd64.deb ...\n",
            "Unpacking xvfb (2:21.1.4-2ubuntu1.7~22.04.2) ...\n",
            "Setting up libfontenc1:amd64 (1:1.1.4-1build3) ...\n",
            "Setting up xfonts-encodings (1:1.0.5-0ubuntu2) ...\n",
            "Setting up libxkbfile1:amd64 (1:1.1.0-1build3) ...\n",
            "Setting up libxfont2:amd64 (1:2.0.5-1build1) ...\n",
            "Setting up x11-xkb-utils (7.7+5build4) ...\n",
            "Setting up xfonts-utils (1:7.7+6build2) ...\n",
            "Setting up xfonts-base (1:1.0.5) ...\n",
            "Setting up xserver-common (2:21.1.4-2ubuntu1.7~22.04.2) ...\n",
            "Setting up xvfb (2:21.1.4-2ubuntu1.7~22.04.2) ...\n",
            "Processing triggers for man-db (2.10.2-1) ...\n",
            "Processing triggers for fontconfig (2.13.1-4.2ubuntu5) ...\n",
            "Processing triggers for libc-bin (2.35-0ubuntu3.1) ...\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_0.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbb.so.12 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc_proxy.so.2 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_5.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc.so.2 is not a symbolic link\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# HIDE OUTPUT\n",
        "try:\n",
        "    from google.colab import drive\n",
        "    COLAB = True\n",
        "    print(\"Note: using Google CoLab\")\n",
        "except:\n",
        "    print(\"Note: not using Google CoLab\")\n",
        "    COLAB = False\n",
        "\n",
        "if COLAB:\n",
        "  !pip install stable-baselines3[extra] gymnasium\n",
        "  !pip install gymnasium[accept-rom-license,atari]\n",
        "  !pip install pyvirtualdisplay\n",
        "  !sudo apt-get install -y python-opengl ffmpeg\n",
        "  !sudo apt-get install -y xvfb"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lsaQlK8fFQqH"
      },
      "source": [
        "# Part 12.4: Atari Games with Stable Baselines Neural Networks\n",
        "\n",
        "The Atari 2600 is a home video game console from Atari, Inc., Released on September 11, 1977. Most credit the Atari with popularizing microprocessor-based hardware and games stored on ROM cartridges instead of dedicated hardware with games built into the unit. Atari bundled their console with two joystick controllers, a conjoined pair of paddle controllers, and a game cartridge: initially [Combat](https://en.wikipedia.org/wiki/Combat_(Atari_2600)), and later [Pac-Man](https://en.wikipedia.org/wiki/Pac-Man_(Atari_2600)).\n",
        "\n",
        "Atari emulators are popular and allow gamers to play many old Atari video games on modern computers. These emulators are even available as JavaScript.\n",
        "\n",
        "* [Virtual Atari](http://www.virtualatari.org/listP.html)\n",
        "\n",
        "Atari games have become popular benchmarks for AI systems, particularly reinforcement learning. OpenAI Gym internally uses the [Stella Atari Emulator](https://stella-emu.github.io/). You can see the Atari 2600 in Figure 12.ATARI.\n",
        "\n",
        "**Figure 12.ATARI: The Atari 2600**\n",
        "![Atari 2600 Console](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/atari-1.png \"Atari 2600 Console\")\n",
        "\n",
        "## Actual Atari 2600 Specs\n",
        "\n",
        "* CPU: 1.19 MHz MOS Technology 6507\n",
        "* Audio + Video processor: Television Interface Adapter (TIA)\n",
        "* Playfield resolution: 40 x 192 pixels (NTSC). It uses a 20-pixel register that is mirrored or copied, left side to right side, to achieve the width of 40 pixels.\n",
        "* Player sprites: 8 x 192 pixels (NTSC). Player, ball, and missile sprites use pixels 1/4 the width of playfield pixels (unless stretched).\n",
        "* Ball and missile sprites: 1 x 192 pixels (NTSC).\n",
        "* Maximum resolution: 160 x 192 pixels (NTSC). Max resolution is achievable only with programming tricks that combine sprite pixels with playfield pixels.\n",
        "* 128 colors (NTSC). 128 possible on screen. Max of 4 per line: background, playfield, player0 sprite, and player1 sprite. Palette switching between lines is common. Palette switching mid-line is possible but not common due to resource limitations.\n",
        "* 2 channels of 1-bit monaural sound with 4-bit volume control.\n",
        "\n",
        "## Gymnasium Atari Breakout\n",
        "\n",
        "You can use OpenAI Gym with Windows; however, it requires a special [installation procedure](https://towardsdatascience.com/how-to-install-openai-gym-in-a-windows-environment-338969e24d30).\n",
        "\n",
        "This chapter demonstrates playing [Atari Breakout](https://en.wikipedia.org/wiki/Breakout_(video_game)). Atari Breakout is a classic arcade game that was released by Atari, Inc. in 1976. In the game, the player controls a paddle at the bottom of the screen, using it to bounce a ball against a wall of bricks at the top. The objective is to destroy all the bricks by hitting them with the ball, which the player deflects with the paddle. As the player progresses, the ball moves increasingly faster, and some bricks may require multiple hits to break. The player loses a turn when the ball misses the paddle and hits the bottom of the screen. The simplicity of Breakout's gameplay, combined with its increasing difficulty as the game progresses, has made it a quintessential example of the easy-to-learn-yet-hard-to-master design ethos that characterized many early video games.\n",
        "\n",
        "In the context of artificial intelligence research and particularly within reinforcement learning, Atari Breakout has been adapted as an environment within OpenAI's Gym toolkit, a collection of environments that provide a standardized interface for algorithm development and benchmarking. Stable Baselines is a set of high-quality implementations of reinforcement learning algorithms, which offers a simple way to train and evaluate agents on various tasks, including playing Atari games like Breakout. The adaptation of Breakout to the Gym environment, often referred to as 'Breakout-v0' or 'BreakoutDeterministic-v4' in the Gym library, abstracts the game's mechanics into observations, actions, and rewards, which an AI agent can interact with. In this setup, the agent observes the game state (typically the pixel data from the screen), selects actions (like moving the paddle left or right), and receives rewards (such as the score for breaking bricks). This allows researchers and enthusiasts to apply and test reinforcement learning algorithms using Stable Baselines to develop AI agents that can learn to play Breakout at a superhuman level, offering a playground to advance the field of machine learning.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hBc9lj9VWWtZ"
      },
      "source": [
        "## Training the Agent\n",
        "\n",
        "We are now ready to train the DQN. Depending on how many episodes you wish to run through, this process can take many hours. This code will update both the loss and average return as training occurs. As training becomes more successful, the average return should increase. The losses reported reflecting the average loss for individual training batches."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Aq8VWUBFz5_t",
        "outputId": "0ee50424-66a5-4975-b65d-13efe4b5d0ad"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/tensorflow/python/framework/dtypes.py:35: DeprecationWarning: ml_dtypes.float8_e4m3b11 is deprecated. Use ml_dtypes.float8_e4m3b11fnuz\n",
            "  from tensorflow.tsl.python.lib.core import pywrap_ml_dtypes\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Using cuda device\n",
            "Wrapping the env in a VecTransposeImage.\n",
            "Logging to ./atari_ppo_tensorboard/PPO_1\n",
            "---------------------------------\n",
            "| rollout/           |          |\n",
            "|    ep_len_mean     | 774      |\n",
            "|    ep_rew_mean     | 1.52     |\n",
            "| time/              |          |\n",
            "|    fps             | 336      |\n",
            "|    iterations      | 1        |\n",
            "|    time_elapsed    | 24       |\n",
            "|    total_timesteps | 8192     |\n",
            "---------------------------------\n",
            "------------------------------------------\n",
            "| rollout/                |              |\n",
            "|    ep_len_mean          | 745          |\n",
            "|    ep_rew_mean          | 1.4          |\n",
            "| time/                   |              |\n",
            "|    fps                  | 318          |\n",
            "|    iterations           | 2            |\n",
            "|    time_elapsed         | 51           |\n",
            "|    total_timesteps      | 16384        |\n",
            "| train/                  |              |\n",
            "|    approx_kl            | 0.0089585865 |\n",
            "|    clip_fraction        | 0.0609       |\n",
            "|    clip_range           | 0.2          |\n",
            "|    entropy_loss         | -1.38        |\n",
            "|    explained_variance   | -0.0165      |\n",
            "|    learning_rate        | 0.0003       |\n",
            "|    loss                 | -0.0132      |\n",
            "|    n_updates            | 10           |\n",
            "|    policy_gradient_loss | -0.00591     |\n",
            "|    value_loss           | 0.0196       |\n",
            "------------------------------------------\n",
            "----------------------------------------\n",
            "| rollout/                |            |\n",
            "|    ep_len_mean          | 745        |\n",
            "|    ep_rew_mean          | 1.46       |\n",
            "| time/                   |            |\n",
            "|    fps                  | 315        |\n",
            "|    iterations           | 3          |\n",
            "|    time_elapsed         | 77         |\n",
            "|    total_timesteps      | 24576      |\n",
            "| train/                  |            |\n",
            "|    approx_kl            | 0.01825708 |\n",
            "|    clip_fraction        | 0.168      |\n",
            "|    clip_range           | 0.2        |\n",
            "|    entropy_loss         | -1.36      |\n",
            "|    explained_variance   | 0.594      |\n",
            "|    learning_rate        | 0.0003     |\n",
            "|    loss                 | -0.062     |\n",
            "|    n_updates            | 20         |\n",
            "|    policy_gradient_loss | -0.0327    |\n",
            "|    value_loss           | 0.0221     |\n",
            "----------------------------------------\n",
            "-----------------------------------------\n",
            "| rollout/                |             |\n",
            "|    ep_len_mean          | 821         |\n",
            "|    ep_rew_mean          | 2.04        |\n",
            "| time/                   |             |\n",
            "|    fps                  | 315         |\n",
            "|    iterations           | 4           |\n",
            "|    time_elapsed         | 103         |\n",
            "|    total_timesteps      | 32768       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.021493958 |\n",
            "|    clip_fraction        | 0.242       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -1.34       |\n",
            "|    explained_variance   | 0.74        |\n",
            "|    learning_rate        | 0.0003      |\n",
            "|    loss                 | -0.0815     |\n",
            "|    n_updates            | 30          |\n",
            "|    policy_gradient_loss | -0.0474     |\n",
            "|    value_loss           | 0.0257      |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| rollout/                |             |\n",
            "|    ep_len_mean          | 886         |\n",
            "|    ep_rew_mean          | 2.42        |\n",
            "| time/                   |             |\n",
            "|    fps                  | 315         |\n",
            "|    iterations           | 5           |\n",
            "|    time_elapsed         | 129         |\n",
            "|    total_timesteps      | 40960       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.028932216 |\n",
            "|    clip_fraction        | 0.293       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -1.3        |\n",
            "|    explained_variance   | 0.755       |\n",
            "|    learning_rate        | 0.0003      |\n",
            "|    loss                 | -0.0925     |\n",
            "|    n_updates            | 40          |\n",
            "|    policy_gradient_loss | -0.058      |\n",
            "|    value_loss           | 0.0294      |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| rollout/                |             |\n",
            "|    ep_len_mean          | 1e+03       |\n",
            "|    ep_rew_mean          | 3.25        |\n",
            "| time/                   |             |\n",
            "|    fps                  | 315         |\n",
            "|    iterations           | 6           |\n",
            "|    time_elapsed         | 155         |\n",
            "|    total_timesteps      | 49152       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.033442706 |\n",
            "|    clip_fraction        | 0.333       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -1.27       |\n",
            "|    explained_variance   | 0.832       |\n",
            "|    learning_rate        | 0.0003      |\n",
            "|    loss                 | -0.0898     |\n",
            "|    n_updates            | 50          |\n",
            "|    policy_gradient_loss | -0.0599     |\n",
            "|    value_loss           | 0.0188      |\n",
            "-----------------------------------------\n",
            "----------------------------------------\n",
            "| rollout/                |            |\n",
            "|    ep_len_mean          | 1.1e+03    |\n",
            "|    ep_rew_mean          | 3.87       |\n",
            "| time/                   |            |\n",
            "|    fps                  | 316        |\n",
            "|    iterations           | 7          |\n",
            "|    time_elapsed         | 181        |\n",
            "|    total_timesteps      | 57344      |\n",
            "| train/                  |            |\n",
            "|    approx_kl            | 0.04295888 |\n",
            "|    clip_fraction        | 0.378      |\n",
            "|    clip_range           | 0.2        |\n",
            "|    entropy_loss         | -1.22      |\n",
            "|    explained_variance   | 0.793      |\n",
            "|    learning_rate        | 0.0003     |\n",
            "|    loss                 | -0.0944    |\n",
            "|    n_updates            | 60         |\n",
            "|    policy_gradient_loss | -0.0671    |\n",
            "|    value_loss           | 0.023      |\n",
            "----------------------------------------\n",
            "---------------------------------------\n",
            "| rollout/                |           |\n",
            "|    ep_len_mean          | 1.19e+03  |\n",
            "|    ep_rew_mean          | 4.43      |\n",
            "| time/                   |           |\n",
            "|    fps                  | 316       |\n",
            "|    iterations           | 8         |\n",
            "|    time_elapsed         | 207       |\n",
            "|    total_timesteps      | 65536     |\n",
            "| train/                  |           |\n",
            "|    approx_kl            | 0.0499343 |\n",
            "|    clip_fraction        | 0.396     |\n",
            "|    clip_range           | 0.2       |\n",
            "|    entropy_loss         | -1.18     |\n",
            "|    explained_variance   | 0.819     |\n",
            "|    learning_rate        | 0.0003    |\n",
            "|    loss                 | -0.123    |\n",
            "|    n_updates            | 70        |\n",
            "|    policy_gradient_loss | -0.0707   |\n",
            "|    value_loss           | 0.0197    |\n",
            "---------------------------------------\n",
            "----------------------------------------\n",
            "| rollout/                |            |\n",
            "|    ep_len_mean          | 1.25e+03   |\n",
            "|    ep_rew_mean          | 4.89       |\n",
            "| time/                   |            |\n",
            "|    fps                  | 315        |\n",
            "|    iterations           | 9          |\n",
            "|    time_elapsed         | 233        |\n",
            "|    total_timesteps      | 73728      |\n",
            "| train/                  |            |\n",
            "|    approx_kl            | 0.05586449 |\n",
            "|    clip_fraction        | 0.425      |\n",
            "|    clip_range           | 0.2        |\n",
            "|    entropy_loss         | -1.14      |\n",
            "|    explained_variance   | 0.773      |\n",
            "|    learning_rate        | 0.0003     |\n",
            "|    loss                 | -0.104     |\n",
            "|    n_updates            | 80         |\n",
            "|    policy_gradient_loss | -0.0739    |\n",
            "|    value_loss           | 0.0238     |\n",
            "----------------------------------------\n",
            "----------------------------------------\n",
            "| rollout/                |            |\n",
            "|    ep_len_mean          | 1.36e+03   |\n",
            "|    ep_rew_mean          | 5.82       |\n",
            "| time/                   |            |\n",
            "|    fps                  | 315        |\n",
            "|    iterations           | 10         |\n",
            "|    time_elapsed         | 259        |\n",
            "|    total_timesteps      | 81920      |\n",
            "| train/                  |            |\n",
            "|    approx_kl            | 0.06363768 |\n",
            "|    clip_fraction        | 0.439      |\n",
            "|    clip_range           | 0.2        |\n",
            "|    entropy_loss         | -1.12      |\n",
            "|    explained_variance   | 0.773      |\n",
            "|    learning_rate        | 0.0003     |\n",
            "|    loss                 | -0.127     |\n",
            "|    n_updates            | 90         |\n",
            "|    policy_gradient_loss | -0.0777    |\n",
            "|    value_loss           | 0.0227     |\n",
            "----------------------------------------\n",
            "----------------------------------------\n",
            "| rollout/                |            |\n",
            "|    ep_len_mean          | 1.46e+03   |\n",
            "|    ep_rew_mean          | 6.55       |\n",
            "| time/                   |            |\n",
            "|    fps                  | 315        |\n",
            "|    iterations           | 11         |\n",
            "|    time_elapsed         | 285        |\n",
            "|    total_timesteps      | 90112      |\n",
            "| train/                  |            |\n",
            "|    approx_kl            | 0.07362982 |\n",
            "|    clip_fraction        | 0.467      |\n",
            "|    clip_range           | 0.2        |\n",
            "|    entropy_loss         | -1.09      |\n",
            "|    explained_variance   | 0.731      |\n",
            "|    learning_rate        | 0.0003     |\n",
            "|    loss                 | -0.117     |\n",
            "|    n_updates            | 100        |\n",
            "|    policy_gradient_loss | -0.0816    |\n",
            "|    value_loss           | 0.0216     |\n",
            "----------------------------------------\n",
            "----------------------------------------\n",
            "| rollout/                |            |\n",
            "|    ep_len_mean          | 1.51e+03   |\n",
            "|    ep_rew_mean          | 7.21       |\n",
            "| time/                   |            |\n",
            "|    fps                  | 315        |\n",
            "|    iterations           | 12         |\n",
            "|    time_elapsed         | 311        |\n",
            "|    total_timesteps      | 98304      |\n",
            "| train/                  |            |\n",
            "|    approx_kl            | 0.08955769 |\n",
            "|    clip_fraction        | 0.47       |\n",
            "|    clip_range           | 0.2        |\n",
            "|    entropy_loss         | -1.03      |\n",
            "|    explained_variance   | 0.698      |\n",
            "|    learning_rate        | 0.0003     |\n",
            "|    loss                 | -0.121     |\n",
            "|    n_updates            | 110        |\n",
            "|    policy_gradient_loss | -0.0839    |\n",
            "|    value_loss           | 0.0282     |\n",
            "----------------------------------------\n",
            "----------------------------------------\n",
            "| rollout/                |            |\n",
            "|    ep_len_mean          | 1.57e+03   |\n",
            "|    ep_rew_mean          | 7.84       |\n",
            "| time/                   |            |\n",
            "|    fps                  | 316        |\n",
            "|    iterations           | 13         |\n",
            "|    time_elapsed         | 337        |\n",
            "|    total_timesteps      | 106496     |\n",
            "| train/                  |            |\n",
            "|    approx_kl            | 0.09902543 |\n",
            "|    clip_fraction        | 0.476      |\n",
            "|    clip_range           | 0.2        |\n",
            "|    entropy_loss         | -0.977     |\n",
            "|    explained_variance   | 0.719      |\n",
            "|    learning_rate        | 0.0003     |\n",
            "|    loss                 | -0.101     |\n",
            "|    n_updates            | 120        |\n",
            "|    policy_gradient_loss | -0.0813    |\n",
            "|    value_loss           | 0.0268     |\n",
            "----------------------------------------\n",
            "Mean reward: 12.2 +/- 6.046486583132389\n"
          ]
        }
      ],
      "source": [
        "import gymnasium as gym\n",
        "from stable_baselines3 import PPO\n",
        "from stable_baselines3.common.evaluation import evaluate_policy\n",
        "from stable_baselines3.common.env_util import make_atari_env\n",
        "from stable_baselines3.common.vec_env import VecFrameStack\n",
        "\n",
        "# Set this constant to either 'Breakout' or 'Atlantis' to choose the game\n",
        "GAME_NAME = 'Breakout'  # Or 'Atlantis'\n",
        "\n",
        "# Create the game environment, note that we wrap it with VecFrameStack for preprocessing\n",
        "env_id = f\"{GAME_NAME}NoFrameskip-v4\"\n",
        "env = make_atari_env(env_id, n_envs=4, seed=0)\n",
        "env = VecFrameStack(env, n_stack=4)\n",
        "\n",
        "# Initialize the agent, here we use Proximal Policy Optimization (PPO)\n",
        "model = PPO('CnnPolicy', env, verbose=1, tensorboard_log=\"./atari_ppo_tensorboard/\")\n",
        "\n",
        "# Train the agent\n",
        "TIMESTEPS = 1e5\n",
        "model.learn(total_timesteps=TIMESTEPS)\n",
        "\n",
        "# Save the model\n",
        "model.save(f\"{GAME_NAME}_ppo_model\")\n",
        "\n",
        "# Evaluate the trained agent\n",
        "mean_reward, std_reward = evaluate_policy(model, env, n_eval_episodes=10)\n",
        "\n",
        "print(f\"Mean reward: {mean_reward} +/- {std_reward}\")\n",
        "\n",
        "# Don't forget to close the environment when you are done\n",
        "env.close()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M7-XpPP99Cy7"
      },
      "source": [
        "## Videos\n",
        "\n",
        "Perhaps the most compelling way to view an Atari game's results is a video that allows us to see the agent play the game. We now have a trained model and observed its training progress on a graph. The following functions are defined to watch the agent play the game in the notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_Qpx5qvI3wD0",
        "outputId": "544f2f62-b04a-4a79-d504-cfcc6df7d521"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
            "  and should_run_async(code)\n",
            "/usr/local/lib/python3.10/dist-packages/gymnasium/utils/passive_env_checker.py:335: UserWarning: \u001b[33mWARN: No render fps was declared in the environment (env.metadata['render_fps'] is None or not defined), rendering may occur at inconsistent fps.\u001b[0m\n",
            "  logger.warn(\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Initial observation shape: (1, 84, 84, 4)\n",
            "Moviepy - Building video /content/videos/Breakout-agent-step-0-to-step-500.mp4.\n",
            "Moviepy - Writing video /content/videos/Breakout-agent-step-0-to-step-500.mp4\n",
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "                                                               "
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Moviepy - Done !\n",
            "Moviepy - video ready /content/videos/Breakout-agent-step-0-to-step-500.mp4\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\r"
          ]
        }
      ],
      "source": [
        "from stable_baselines3.common.vec_env import VecFrameStack\n",
        "from stable_baselines3.common.env_util import make_atari_env\n",
        "from stable_baselines3.common.vec_env import VecVideoRecorder\n",
        "from stable_baselines3 import PPO\n",
        "import os\n",
        "\n",
        "# Set the game name here\n",
        "GAME_NAME = 'Breakout'  # Can be 'Atlantis' as well\n",
        "\n",
        "# Load your previously trained model\n",
        "model_path = f\"{GAME_NAME}_ppo_model.zip\"\n",
        "model = PPO.load(model_path)\n",
        "\n",
        "# Create the Atari environment and apply the correct wrappers\n",
        "env_id = f\"{GAME_NAME}NoFrameskip-v4\"\n",
        "env = make_atari_env(env_id, n_envs=1, seed=0)\n",
        "env = VecFrameStack(env, n_stack=4)\n",
        "\n",
        "# Record the environment\n",
        "video_folder = '/content/videos'\n",
        "if not os.path.exists(video_folder):\n",
        "    os.makedirs(video_folder)\n",
        "\n",
        "env = VecVideoRecorder(env, video_folder,\n",
        "                       record_video_trigger=lambda step: step == 0,\n",
        "                       video_length=500,\n",
        "                       name_prefix=f\"{GAME_NAME}-agent\")\n",
        "\n",
        "# Reset the environment and observe the initial observation shape\n",
        "obs = env.reset()\n",
        "print(\"Initial observation shape:\", obs.shape)  # Should be (1, 4, 84, 84)\n",
        "\n",
        "# Run one episode\n",
        "done = False\n",
        "while not done:\n",
        "    action, _states = model.predict(obs, deterministic=True)\n",
        "    obs, rewards, done, info = env.step(action)\n",
        "    env.render()\n",
        "\n",
        "# Close the environment which should also save the video\n",
        "env.close()\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 557
        },
        "id": "YTTIWddpAawT",
        "outputId": "45e91a16-e603-472c-abe2-d00e2c0c8a4d"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.\n",
            "  and should_run_async(code)\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<video width=\"640\" height=\"480\" controls><source src=\"data:video/mp4;base64,AAAAIGZ0eXBpc29tAAACAGlzb21pc28yYXZjMW1wNDEAAAAIZnJlZQAAKd9tZGF0AAACrgYF//+q3EXpvebZSLeWLNgg2SPu73gyNjQgLSBjb3JlIDE1OSByMjk5MSAxNzcxYjU1IC0gSC4yNjQvTVBFRy00IEFWQyBjb2RlYyAtIENvcHlsZWZ0IDIwMDMtMjAxOSAtIGh0dHA6Ly93d3cudmlkZW9sYW4ub3JnL3gyNjQuaHRtbCAtIG9wdGlvbnM6IGNhYmFjPTEgcmVmPTMgZGVibG9jaz0xOjA6MCBhbmFseXNlPTB4MzoweDExMyBtZT1oZXggc3VibWU9NyBwc3k9MSBwc3lfcmQ9MS4wMDowLjAwIG1peGVkX3JlZj0xIG1lX3JhbmdlPTE2IGNocm9tYV9tZT0xIHRyZWxsaXM9MSA4eDhkY3Q9MSBjcW09MCBkZWFkem9uZT0yMSwxMSBmYXN0X3Bza2lwPTEgY2hyb21hX3FwX29mZnNldD0tMiB0aHJlYWRzPTMgbG9va2FoZWFkX3RocmVhZHM9MSBzbGljZWRfdGhyZWFkcz0wIG5yPTAgZGVjaW1hdGU9MSBpbnRlcmxhY2VkPTAgYmx1cmF5X2NvbXBhdD0wIGNvbnN0cmFpbmVkX2ludHJhPTAgYmZyYW1lcz0zIGJfcHlyYW1pZD0yIGJfYWRhcHQ9MSBiX2JpYXM9MCBkaXJlY3Q9MSB3ZWlnaHRiPTEgb3Blbl9nb3A9MCB3ZWlnaHRwPTIga2V5aW50PTI1MCBrZXlpbnRfbWluPTI1IHNjZW5lY3V0PTQwIGludHJhX3JlZnJlc2g9MCByY19sb29rYWhlYWQ9NDAgcmM9Y3JmIG1idHJlZT0xIGNyZj0yMy4wIHFjb21wPTAuNjAgcXBtaW49MCBxcG1heD02OSBxcHN0ZXA9NCBpcF9yYXRpbz0xLjQwIGFxPTE6MS4wMACAAAACPGWIhAAz//727L4FNhTIUGV5w7TCGgEJgSdzsyckV3S77Dm8Ag1mH56pG01iUfoqGJvSBlpGDUJHrm1XsxLEEWCpUTZmUUmjvvCYBgGoikrw2+ssYLKBxLBxL0+ZE3oioFJuahdgPCzVdK7oifhUyHum1y+H/n1IxfZqe5Q6a/qB80iOWzBXOZk5hpNEJ6YI8Htq5Ycx+fStwR4MJbjva8zgSaMI8mOGNDEN41M5DSR1b1O2wUMCbawvRX6v++m1dk/nJYumv6we4Umu74BJIWmf1jgrvXMDs/VbtW3rlCQ8k7kUaV1IAM+ZyEQZ+KdH9T98tnbbZiT+cqG4bGlod+t9aGPYUmL8Ao7oAAHJov0zPrj9pZxZh5UBRIJcYSgthDRdFLuh3VLah40X9ysi6Ms0zInxvVJGu1iSPjMSh+dfK+wRAFJzHgw45pyLSuH19jT6JxI8EWFJRy5+w6VOBDkc1EELzu97Qt2o/OVyxPIUf7yus1eR2kXh4o9zXoXIZxQeQzpPP2EsAybGcAlDP1x8fq7B1W2bIguyBDibdrKFxpnXjdEI0i7lmxBRO+LuDESCn6nIdkRbOwOvkAzblqztZAzUIzRbQ09PzejkaQ36IwYgpYmqnZUegkiCAqS8A4HIXALOJ+z2N9l0dD7d6MHs3kjcAWOej6FgY0oeRTUGGMACdYOezs66FrEwSFLGGUEv5GVJe5zuN7pAWZ8fmxjBIDq1WFvnFKNRgjLvFqHPbEGwcaZCFOmyE82pAAAARkGaIWxDP/6eEAuFQyACjTFqb5TaeFl/ST3MjyJoMt95xE1n4P53aXPxfikvhm32KeP/+5MNW4BPmimIa/OIAH//0NGGWlgAAAA0QZpDPCGTKYQz//6eEAr/ZDaimAH1P/gcVMAFzh4SSKNZUELQYcxL7A6WLpgqjvVD7DtJgQAAABYBnmJqQn8A1/lDhXF7U+U2UORuwyLgAAAAT0GaZEnhDyZTAhv//qeEAu3YV9yGyD8j/wOTxAC3Ln/NROvKhksic0VxgynzXqe9FZUdZ2pgz3bZPkJMRKEX3KJ6SZDrNJlmplx2FFNoKg8AAABaQZqFSeEPJlMCG//+p4QC7diiAFoelHwDZuiP/A5NAFF/egl2GxoHIkX8vtmzq8clofLd016nK6ur5rJBFV6He5nIpvOE3lHM+S6lconacbjZHeNUk2lVi2PhAAAAX0GaqUnhDyZTAhv//qeEAu3YogBaHpR8A18oUZgZ8SOIWFmjn2dQSBav3j7z1azDpi87P0BnKSdYphUIoBtX/ICPIhDnhh2cTmfUIZKGZlAPVmM9RHGcrf1iBIf2DclhAAAAPEGex0URPCv/AIM+QABCji2e+cn6fElxZIroIK4AQl0NUb2jImhAVLwucrEmgVAJFwmTjEk+kqJjg7A4QQAAABcBnuZ0Qn8ArEbWHCvu0wMnjgS13EddIwAAADkBnuhqQn8Aqsq/EY1mL7UsMQWswBD3x4hXDC6yLkHYbTnLBqkIASV2GM61umnmMo6YM/6s/WUIklwAAABYQZrtSahBaJlMCG///qeEAu3YogBaHpR8A18oUZgZ8SOIWzbH/6b1wuCvHmP4virPmcpKnAfhUFGCIHBx3zO3LguByiYICzelPfQb/4NLaZMYKEFd261yKwAAADhBnwtFESwr/wBoh18C8mTADjFUdiYqruuhvrNkJ8QdIlZoGs7DRtBhqfT+JFAMJfrABn2hIcqgrQAAADgBnyp0Qn8AhUKuBxQOACNVTxCuGI9dDsg2IBHxB69q0fD2DWHD8xH2Fp0ABkghuAQdda9oFU1fRAAAACwBnyxqQn8AhsRjCKcYFma7gBa3xwu8GgnY6hSk0x5jf9T3DvgwmZZrkiMYLwAAAExBmzFJqEFsmUwIb//+p4QC7diiAFoelHwDXyqu1P+ApiGf8vA23CMxWK3+wmvpyLU8gM2ghKBLDIZMyulU8we4ee37WDkhEW8vUSkxAAAAR0GfT0UVLCv/AXV0EQA3HYiQBXYI+aPgD0w3NtgtIJEMK9v2XMjDGgkT57wHJe+i51nI6IM6DZzIiXoIdWmkjkp96YgAHKzBAAAANgGfbnRCfwHjv7lOeV2UReQwtm6oKtAAcGnkOK7Q3e8qAIfGZT3ygwgIAE+vwlUAIb/4TQMu+gAAACEBn3BqQn8B2XWzXpysqDCNIR6ojUXkFHngp+vGpffkasIAAABRQZt1SahBbJlMCG///qeEAu3YogBaHpR8A18oUZgZ8SOIWhPWWixoxVGZ9JdppyLU8gM2ghKBLDIZMyulUc6iFUPD9vV8/jq9BqiXBqjSZb0hAAAAQEGfk0UVLCv/AXXkLHNE2o4OViI8bH+v1ADXlrZ75yfp8SXFkiuggrf/yPYhvDcJs7E+REdzSaosoUqcKCEvTC0AAAA9AZ+ydEJ/AdjX0EL+mpYtga19piNKvwARqqeIZgrXJZ6ZGkyZ97ny+4KWfY3G5AV8yamhhcaiVrUEV0e/mAAAADwBn7RqQn8B2XgJnnp9XFaDaW2RED2AHHtCEI82SLZVEqsOiSP4WLkRXUKzTkwrV5aiMmLo9TmXacNbN/0AAABTQZu5SahBbJlMCG///qeEAu3YogBwNSj4Br5QozAz4kcy2w53uX7bQK/3uvg7/yLU8gM2ghKBLDIZMyulU9A24efgvzzTI/+TM0gloQZHFWe2XvgAAAA8QZ/XRRUsK/8BdeSnyyDk/6LuFcc/j2rgAcWfJWdFVRWRUmYmmsBaT4+T5IgFD9zwM6GdMcqzXULXQGzBAAAAPAGf9nRCfwHY19BED+bW4GxUzAFWzv1iYK1yWemRpMmWLRHulzBupdl2CAVeXHb6Jrc9dETR9+eoSqPcbwAAAEMBn/hqQn8B2XgJnnygc+3BIjxgB0jqeIZi29q7Jum2KgLX9rO4Wq/dZwhmdKtboWOfQhN4Ik5LGBQZ6vFDZtXBEBE4AAAAVkGb+0moQWyZTBRMN//+p4QC7diiAHA1KPgGztEf+ByZ5KL+P91Ass8Z8kGUYg6sa+upPskEVXogh2XHHXX/ib+bQ6ARTecJxzZtELG2TmWfSIbErnuBAAAAPQGeGmpCfwHZBpCczb8+Hy0xEByuAI1VPEMwVrnBJBLSZPTKWClHjF1SJuNw4r5k1NR8M3Nrs9Lm6gJcsjgAAAJMZYiCAAQ//veBvzLLZD+qxl/aQhpDAAOnC++oUp5Pvb1jT4GOECC9SpnxEaL74nc4kyCAsTDB9VWkhZItS2rxwW3FTQkPKEtgewvT9T5EvNZ9G4B0wIHsVRjk3M5BthQV/EkPs3WWKTHwCQfE9fY3GRYscOxMKJjtNmf+cyofo/Ix0oD/v3y/Rk9H0YQ7mArslZr6xaFA2CUmK4mGhXePPto5DVv/IUYzLUE9eKo7gI9UClLGcEp2UnXorGgGZnAq1d51iwiOOwsPUGifIBP5LHa3d7GUivZP6Xu2yjZKq2b1gkWOFYbkSvGtq02VlCAI/xM5uoPX9vfJVKd+pRRCIPknTpO/95ivJGxkCaOK7YWBfNQ0pKliYAAEd+rs063TtayKJZHE0CVuYqiLOhrU9vZ5zxO/6C4oLIrKDNVUb2z2sd7A3pTuOYhoLWJIARcoABpMmbh0C+WOGtDbB1PbNcDh1oevlPOl+GSiwu/N8eHJV9Gs/x+qIqURorF9nNSLHynhmiUwgftYE67Cjgwt+ZePsATSv7TQxcargawLv+vrEmNwNkids5zA3QwvvoSG+2C96Vie1vAUjgTVZAjNyPd6dgSEl9DPL7gHUcAjVeabdoPsLrZiFsUws+Us3rPkUhQ+XOvyPgsKDzFcbpfbkF0iOLStYqOKnpVNt5hDMOb0bNHmwxXSLqsD2mq/DfF4M4a7VVM7k+OfYc4YVIj9hdthtRDWdAB0syta5ktUmReBLnpygK3RD30ZejloMVLS9pM2HnKLUqPFtBtBAAAAakGaI2xDf/4ocmaicOqIBNcA//hDrIEuqBos+40UvpGe+7keAP3Tuc8hF6pwoQUaYQQAeDb0dbK1hKp7r/EDaq/2v1K9UjEGE7OS0lbmBygvdyWLdyGqaL8LwHHl0AgGx1wRsALz/teBfY8AAABuQZ5BeIV/WAKl0fuW8ahPNzqJKE48yLi2cN+AHFnyVnRVUVkX9DPTSP2Dv43cY86x2lHziCfaimon9s3Lc1C7qcqiUrycKAvS4U2cE3Bm/gPCIKQXE09l8ypYuKb2fEloFX3R1UpywFp9znJrPIAAAAAuAZ5iakJ/X/Th+7t/wAt4zu9i67C+7KOmaXtPJbI/J/SvMhgOP4uA7DNePxji4AAAAGlBmmdJqEFomUwIb//+p4QBwahYe/VfmlOS4AQm2I+gBWpWokQVnL+AOZdSovTuDJwlHOD9EdDG4AHjQeBjECLSmLaQ9S+2w5+apVpJvN/lNPstpUvwDVRduqyc/mNaMXVNcgsJvhJ/5IEAAAAnQZ6FRREsK/8BUX+iNt8EAA5yMcz5nN0Ee0th1PqqAnuGBE94ATvsAAAAIgGepHRCfwGtajFJyADQOfMlx/m7V6ByTg+DQUdN8uzn8yEAAAAtAZ6makJ/AaQWJFFeABCnb6VCuUoHDGpXdZCmCmuveBXIa5mPuVO7kw1G9pFMAAAAQkGaqkmoQWyZTAhv//6nhAE1+CEzpzsALqtN9ojlGREPWwv1dB1mz2pvz7INKhdogVHfc21ltwtdiAUiIHPhvSEyZQAAADhBnshFFSwr/wD4Ad1h8QAcGtQ1P3m4c0w+1io4YDw9oMimeVM6hDVAXXu7O8kzZEOwZXnmIZ6GKQAAACYBnulqQn8A9g2XQA+DWvzY8OBtEPS8ZQmVzm8vhaeM+xcPIs9JiQAAADxBmuxJqEFsmUwUTDf//qeEAO16jKsIsWsikQAQ4UPaIwjcnzSAPzZp3fpnrFj4SWPixPGFt9id+xZX4gsAAAAUAZ8LakJ/APiDStBqdCmO773BK1QAAABBQZsQSeEKUmUwIb/+p4QAtfvClgC6pUqzIAizWlrU2QtSkkxIBg0gQQfgzFAYhJi6XHm/pCbOxAJ5miY3vFFCLcAAAAAwQZ8uRTRMK/8AkskVMbAETGlQ0x0aegYm+38Lphfww7w9oMvXPKmdQlSbN0C1g6vbAAAAHgGfTXRCfwC++eFxXgBDTvaUFVQLk5BjH8TWYBSrBAAAAB4Bn09qQn8AksRtKvACL8b3lwVVAuAwCgLgBk8X5dwAAAAuQZtUSahBaJlMCG///qeEAItyD4GhfABDZFyCmwhF1SoOBmKU/1M6yjX7e+F6YQAAACVBn3JFESwr/wBxWSVsvqXIAW79sXTQh8gmTi5CrOOAFVVByvThAAAAGQGfkXRCfwBukNrwAsZ3s7E/s4q23RExXkEAAAAjAZ+TakJ/AJLt19YAF9gqWgZuvo7/yaCXxxUOz4sohd3YgHQAAAAvQZuYSahBbJlMCG///qeEALlxBCACGyLkFNhCLqlQcDMUTwb7yzZ8kXCemA5ZIjAAAAAnQZ+2RRUsK/8AluvYpiAEY/bF1JN5PoURzgErC9DDLGzSpqu0TdNRAAAAJwGf1XRCfwDES/IA0k4LTpwA2BqWb9iDqSNJXIymWH7yQbY7VTOQaAAAACYBn9dqQn8AxF5Za+QXiCYAiOGpZv2IOpI0lcjKqm2+TIOLd9y3oAAAACxBm9pJqEFsmUwUTDf//qeEAOwXIQgAnbTXIKazlru6g4GbUU5idQRKiSBM4QAAABMBn/lqQn8A+GvO1ENuZ/LnvroQAAAAKEGb/knhClJlMCGf/p4QBrirA4AF1pe3p1WZ6x5/Ym+UI5tlpSs3vsEAAAAjQZ4cRTRMK/8BUbrjVaQKACGouJbKsz7DHT5UUKKu+Mjh9OEAAAAUAZ47dEJ/AU/o0owaDOIA1E3RTtAAAAAgAZ49akJ/Aa7DPdIugwAgHQ1Gjpq+AtWw7ucFXu7+7iEAAAAzQZo/SahBaJlMCG///qeEAcGpHjgAo8Cq2jRIcEo7Ez9BEu4Mj+32nrMXp0jqdRDkzJ8UAAAAdkGaQknhClJlMCG//gIwjIBCM/Q/zGEW61CcE9RcWQd0yRz5SVD1Mq1IxVkK6spErWx0mPhfhZCFXg1hDix+zuWSTDoLLfB15W4/kTVkZ90H6iGGqKnSlG2/JKxlwq4JcAOczORL/BDFKegjjKCI6Z4vzCoxOB0AAAAzQZ5gRTRMK/9XpcD8II6YW9B0AGzdi6kowe2aBq8bq5hCNaT0RGtaNFtdn2re8zadXOKhAAAAIwGegWpCf1/04fhi3VgBMhS2+B9X8x+sJ2/Qa4rIKvpC5+EwAAAAOEGahEmoQWiZTBTw3/6nhALeT6f0KABL3ITr+8MHDmRTGrhuWKX8LBnbZoXJs9lQQ+vIpHw58avaAAAAMwGeo2pCf013+IrVon9fGACKRhq05KMlBVY94YVH4fvI6r2FZRtTe11/g54LJFRhwo9b+QAAADhBmqdJ4QpSZTAhv/6nhAK0y2l0p3quffA4v8AIPIuQU2EIuqVBwMxSn+pnWUZL98AFjyvZQYThVAAAADJBnsVFNEwr/0bxgD8DH0ZVoaAEY/bF1JN5PoURzgEKhG6GWNmlTtFZFcGrkHBZY9kg8QAAAC4BnuZqQn9NEGm93I2FTgs6yAGwNSzfsQdSRpK5GUyw/eSDbmJquQUH++itr2CAAAAAfUGa60moQWiZTAhn//6eEASX4sTBuv0kiOwAffH22eYQi6d/jCzISunoKSRz1//tRvuEgFkyUdiJWn7EiS9qdqWmB47208TMe1nbR3OEzJELwXZ7DDCY9KtMn6f+monjbgOU74Bh3wJkm1CSa0w5mKeNftwa4NJBFvArC0a/AAAAVEGfCUURLCv/RyBwOZNBLuLu/pegBGP2xdSTeT7K84u5sfumYRcMfInnvXM67b4GbKSQOTC3evvSNM4k30wO9Vg85VVk0szb/NsKi0FUfO1NWP0JEQAAAB0Bnyh0Qn9MLimwus1kGhT9OWAx1/tOZK4q0TTXgQAAAB0BnypqQn9NEGmmunoGNpRulosLTTg4oe6Z1g2dYQAAADdBmyxJqEFsmUwIZ//+nhACs+78bfTCVaUACdSvtxVYifly045giVUuJcxwNodZq68u+ruC+5fQAAAAhkGbTknhClJlMFFSwz/+nhACng89cDyAoxxwAlffbiqxE/LlpxzBDoedeR9o5SGS14XP6JecNVU7uYZOjESphmtL4W97t9U2rt3Ck+/gWaHz3ar7A/NGr4IA7KOeZhIcd9oElnoo6e7LRdb1mE6b13NdbmRv8tvzdvxHdMzvxVvGbIW3mwUMAAAALAGfbWpCfwC1umzABYcmpYobOT6FEbUxGqfyoOvDY/po69EwSqvczIzYQFSBAAAANkGbcEnhDomUwUTDP/6eEAH9+DUk9aAG5S9x513h9PPX/vBKdEjCDl8VG7605BejtEvxtpk8wAAAACoBn49qQn8AisozgA4ALAS4Rg9rGg68MKj7j15ckXAa7Ql7IKtxVa6wL/0AAABIQZuRSeEPJlMCGf/+nhAB8vY8f2wwf+Cc7VAAnbRmGzzCEXSISlm6yDwu24lEm8i3Erc66E9xNj6WiiY72XOaqYvB29sgb33hAAAAPEGbsknhDyZTAhv//qeEAGT178GKvuAFiy8XFwWlmGXJoXDNlmsAzsMPzcZUi1Ds5b143PscBmsuniVmwQAAAE1Bm9RJ4Q8mUwURPDf//qeEAGdO/6y8ABSRsTgInW/20lwso7pbX78q3KRsXmLRIBOT/s8anakp9JhHR0fuxkc4hrsXwgJtoMCdkCvQwQAAACMBn/NqQn8AbB3/b8BxVsEOAVsloAaypDCHzIGF0yqfRzKO8AAAAFhBm/hJ4Q8mUwIb//6nhACGl1RABDh9GypjaahR1+t+U4kfUfrsYhjfpyBBBjE4hAiKeR+4blRXjYckXzY2qSDAdQXWVQ7C9UObb0jLi28DPcWscKnE2peAAAAAKUGeFkURPCv/AG6dDMo6F8AFqyoDc/lia0Tt0WTI6NNggDUmfgJ0CUr1AAAAJwGeNXRCfwCK+/Dlq9AC3jkx49lYNyNGiPaBxlWMjTTLCLpgVKCjIAAAACcBnjdqQn8Aju3X1gAX4lVWC3qeMvl/5N+9ftGjEOoEyyNj3ObApPUAAAAyQZo8SahBaJlMCG///qeEALTvYFACED6NlTG01Cjr9b8pxJAjWJZRGN+rWXTezE5iItEAAAAqQZ5aRREsK/8AkSqUH4/sIANuwrm82ECBVJlNMfVAqZrTYIA1WZuLiDesAAAAJAGeeXRCfwCO2pSMAEZHJdQ8Yv15t9J4D/Wkt60sygdBV7tn4QAAACsBnntqQn8Avz6nwByilVYHNJ8dCN9klnxHdBYkswo4AdLuZgmV6EwTNwuBAAAAK0GaYEmoQWyZTAhv//6nhADy68Szb9bAARJENoeaoPmNOGWqMQnDkFEmDPwAAAArQZ6eRRUsK/8AyL+3vEyONgAWB89N5sIECqTKaY+qBWATNNwBqUoUzGZqEQAAACUBnr10Qn8A+Gz0U0gBGRyXUPGL9ebfSeA/1pL2eNzMy0PFCsXAAAAAKwGev2pCfwD+Qq5IjAHKKVVgc0nx0KiGsFnxHdBYkswoRoCuFfgN0aEMtDEAAAAwQZqkSahBbJlMCG///qeEATQuqIAISlhZUxtNXEdfrflOKbNUztTppt+v5Qo3au3RAAAAIkGewkUVLCv/APgy0YH7sABeuae0FY7f4JiHMwPIW5PqIqAAAAAQAZ7hdEJ/AP4imwMFdvPlwAAAABMBnuNqQn8BRkYQh+fG6sTFvxN1AAAAMUGa5kmoQWyZTBRMN//+p4QBNFBLfmEAE5IIzPQyF01CiwtkUpKKopOFLKIyA6G4SEAAAAAnAZ8FakJ/AUZ3JIARkcl1Dxi/Xm30ngP9aS9njY1vVnaVglOSNMNBAAAAS0GbCknhClJlMCG//qeEAY3vpcAGyfYWVMbTUKOv1vynEj/uMLGIY323g0Q5/cJ6tyVnU6pSKEzkhVLijwc6+5333z5tEywlk29qwAAAADNBnyhFNEwr/wE23ioQA1ySubzYQIFUmU0x9UCsjnLwQBqqHYMHrBa5CSqijZIjK1P9oYEAAAAeAZ9HdEJ/AZtegAOgUqrA5pPjoDrSmbP1ikm62PcdAAAAKAGfSWpCfwGQw6kYAIyOS6h4xfrzb6TwH+tJb1pZlCbiS2w9zJFYQocAAABvQZtNSahBaJlMCG///fyaQCPj4BBIf/hIwwcfGGZQRZSg46jBGiPBHHUk0MtuCqAWShTpGHuwAAMTSUjmD57TCtUp+hW4ikadPfNK6s8m4jh3aFeyy9DRegpqQVxemCitK29FEBBDoxPGl8CPueUJAAAAO0Gfa0URLCv/V6XA9+7P0ACH5w0MYBp6y7IsgOdKT6Oo0ulO5pcy+d3r+Iu83vFoJn+vAprOyA9E46TGAAAAMQGfjGpCf18Xw9+IFAAful3mg2afZkY1DVJ570IaRE89PWCyTcF3+9R0Ewx0nZMMvHAAAAA7QZuPSahBbJlMFEw3//6nhAGC9BOygQAISlhcFUNt3EddIxBQqbNUzpZQg2eL4Mr/5WimNKQicbc7UlEAAAAtAZ+uakJ/AYYT8mkaAImOS6MvVbLzb7YgZ9UJb1pWtn8YcmO5NhVlyYADIxSAAAAALkGbs0nhClJlMCG//qeEAXJG0x2UCXABq40bgqhtu4jrpGIKFTZ8xVLKEGzxISQAAAA0QZ/RRTRMK/8BJpI8XX3tABwc0VCmyXeqpKtQv9YwI1psDyTDmIBHLdDnpvsR+MNp0yofgQAAAC8Bn/B0Qn8BfEXk9YA5RSqs6GyOFhUhj8LPTpBvw6JhQjQFgsmAiSn1N9pYhKipkQAAACgBn/JqQn8BHZskgBGRyXRl6rZebfbEDPqhL2eNiuiWeGSLR0iloqQHAAAAPkGb90moQWiZTAhv//6nhAEF+Rs0V41kAISlhcFUNt3EddIxBQqbNUzpZQg2d7XvaMJYQeHZ0O+403jc7dJqAAAAMkGeFUURLCv/AOJXxfUKVL8XxADbVCvCmyXeqpKtQv9YwcEzTbyS9FYi8qA8g9kGgu0LAAAALQGeNHRCfwEVbs+AOUUqrOhsjhYSRXRc9OkG/DomFHADptCYA4LiSvV7C8quoQAAACkBnjZqQn8A1gtCS3BoAiY5Loy9VsvNvtiBn1QlvWla2fvlw2D7SnE6bgAAADVBmjlJqEFsmUwUTDf//qeEAMj7KkkUw+nvn9EwAKjwp7azQFoMKGSwtWHDfkjRoBhW+aZUtAAAACwBnlhqQn8BJhe6JZYlr8EAEPzTKf5GWIXruMcTWm0oY1pMmhz1Ekp2qSa/QQAAAEdBmlxJ4QpSZTAhv/6nhACbcg+ElxgAC6rku6iWUtq7sVoquAZHfQXv2sneFSUuiZOdDEYIZPqqXAP1AA3LIySYfW+T8TNjhgAAADJBnnpFNEwr/wDiRwDi2AuF6O1d9Lf1HAAdkXAPRAABJStU9o3xxQqei2DBwthT1PaFpwAAABUBnptqQn8Ao6MQ4W/sFbZu/STy0IEAAAA1QZqfSahBaJlMCG///qeEAJt8jZvnsDgBOJFyG5YpeuoM7sadL/SRVZGbo9ZOoirRi8kJiIAAAAAxQZ69RREsK/8A4le7LqBN2eMYR+aLb6ABs3YvZHMHUoZToRlMqzpp/PW37zVdzJw+wQAAADIBnt5qQn8Ao7OuxtLOJaXoARgVv1iYHijGcTGY9aVhYoYz5nvks0jQjeOT0MrUusoqHAAAADhBmsFJqEFsmUwUTDf//qeEAHn9lSm66utPDht7e4ANSq5DcsUv4WDO7T5FevUwcsZO/k70AgIDCQAAACQBnuBqQn8BJhe6JY2qzCAKDaEIR56mBWai8hE3veauauRiAEcAAAAqQZrjSeEKUmUwUsN//qeEAGReQDqAEyvJ7gOOBmgYwBsl8bUgLUXLJ5fAAAAAKwGfAmpCfwBpmiUOCBOkvABdBV2hh9jYfcLW7lpdkh9GFPVezK1WWSiyfwgAAAApQZsHSeEOiZTAhn/+nhACi5XxwAmUdfY/jgZvDNKAiJ1o4J/+kfDIT78AAAAmQZ8lRRU8K/8AhuwRPRABtNX0UNmPD5WwfKqi/LSTd8G+HZLOwNgAAAAiAZ9EdEJ/AIb8NorzXgTuAIdqP1zp8UsAB0tI1FbA/BQWWQAAACQBn0ZqQn8AsVwSvpoAg4b+KUcEtYfurJ46cGyB3HkuxyD0TCAAAAA0QZtISahBaJlMCG///qeEAKeEOO8AIPeT76o4GYTxu9EiN+zX8MlxAnpIomUUmQYR6iMeTQAAAC5Bm2tJ4QpSZTAhv/6nhAEcLqiACG3k9wOOBmKj83qFblHPhPUYS4fLZGKFokLHAAAAJ0GfiUU0TCv/AOeyzZLRYAau99FDZjxADm8yqovHswYI3to9fFPFfQAAACMBn6pqQn8BLdxqtPzIAQ0N/FKOCWqzlSRlDku2FdxRjxkvwQAAAHdBm69JqEFomUwIb//+p4QCYBP6w5xa6oa6URunZO8Wg/CMVvIARB7kZojCbi9aXSRkukDEV4Yh+IO7RKHnfYD2sl429fToSgKDu+Sxtv0k51/kaYIVN/NMUyYiJLMMlfNy5zXxOPm/OTCqF2UWosVb7gAJ4lolcAAAACdBn81FESwr/wGJdOJ6IANpq+ihsx4fK2D5VUWO4n7EIJqGKCTOPKgAAAAhAZ/sdEJ/AYaZQfU0AQ7UfrnT4pYADpaRqYfQ3+9+W+7/AAAAIwGf7mpCfwHvePThNAEHDfxSjglrD91ZPHSMaGweCvF5zZ5IAAAAYkGb8UmoQWyZTBRMN//+FjlirHAEe/4f/sM4x73sQN8BWF2Ubxr0DAR43zXpVIw3c4gHU0tdXC2AQ+LhzT0u1HiG34vNGr7Q7VK6ABYJzNCgdxGfQEpDGx47Qz+4g8Y6STU4AAAAOAGeEGpCf18Xw9+MMCACcceATmdXMSQnf9D/nU3CLYr4ldL30qM0GaEJXPPNVq2STam+jWXeCcL9AAAAL0GaE0nhClJlMFLDf/6nhAGOKHn8ds7wAbVOR8McbsA33P0BLB0PBhAJx2xuF4gLAAAAIAGeMmpCfwGQEzogQAi+7fxSjgDCt0brZx3AzBdDbMEZAAAAikGaN0nhDomUwIb//qeEASRQfczdZnwAbVOR8Mcbr9ng67WUEFxsH5whdEnDqQ8yndQNm1UKS3ADcyzDf9DvzNaNyd0BCQqr0zQ4jVs5ioAjelHy6s4YEA55YrXxGF5a1hgqljAsdgLfSyIfLHHjym0qIcDypHA3pR1P0Wms0phCt1U3F5eydSoi6QAAAGlBnlVFFTwr/wDtAehREAHFSr6KGzBkATDGqyVdCxhW8xWb4tnG/Bkha3dWHYP0l9slyemh+53paDlpOhZpTy2K5a+CZoNwDNAECfyfKhIcB0xuquz+lS5D5IDuF1uGwIZWjLR2HlX/l+gAAAAnAZ50dEJ/ATVva4AOPJJ+KUbrNqVUtzvYTwPe3nyXGRs1DsAJZWtdAAAAYwGedmpCfwDti7sAEX3b+KUcAX064lbOEm/bFayuBmfFpgaCsgVudeaIqSZ3WdDlu6Qn7KcCDOTiGXSt9J+rJDUuautNL/IJ8/E6WM+rt7eYnVTTSG1meg7vANobqCG/OTlmcAAAAFZBmntJqEFomUwIZ//+nhACs+7+XTo69FgBZpgACX/xUwbrCswestvVHq3KEJrMlxvx1A+ilG95XG6BEWwFmluHSKSABDbi6gj4O6gp+WHKdF/OSTDVKgAAACRBnplFESwr/wCOySV5QgBuWS+8vHAF3NLmL6BOHSHGLh0jEHkAAAAhAZ64dEJ/ALom0mACMhCfrnT0YVujY4Vc5H0U96DFMmnoAAAAHAGeumpCfwCOzGzABF92/ilHAGFbo3WzkPShXiEAAAAxQZq+SahBbJlMCF///oywAby+5dsAHOHkV9ln7rWXa2ghPk21NGAPHp9/9rYPqOdigQAAACBBntxFFSwr/wBdLrUi988AHG13/FKN1m1sHyRoqwKYqAAAABYBnv1qQn8AdoYUUe8vU+bXGkiD3imxAAAAWEGa4EmoQWyZTBRMJ//917UoAJ0z//GLo1zrS2Rw9y6TOA+2EQF/B7vLAf/EGX1QjpzJhH9NAz0DtXamJ5EY/QK3xS/UgRTjzbMhd+71jHV5ghRDkFPzuf4AAAAWAZ8fakJ/P9BpqxR8MGZ4jpCGfviQQQAAChNtb292AAAAbG12aGQAAAAAAAAAAAAAAAAAAAPoAAAUcgABAAABAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAAJPXRyYWsAAABcdGtoZAAAAAMAAAAAAAAAAAAAAAEAAAAAAAAUcgAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAEAAAAAAoAAAANIAAAAAACRlZHRzAAAAHGVsc3QAAAAAAAAAAQAAFHIAAAQAAAEAAAAACLVtZGlhAAAAIG1kaGQAAAAAAAAAAAAAAAAAADwAAAE6AFXEAAAAAAAtaGRscgAAAAAAAAAAdmlkZQAAAAAAAAAAAAAAAFZpZGVvSGFuZGxlcgAAAAhgbWluZgAAABR2bWhkAAAAAQAAAAAAAAAAAAAAJGRpbmYAAAAcZHJlZgAAAAAAAAABAAAADHVybCAAAAABAAAIIHN0YmwAAACYc3RzZAAAAAAAAAABAAAAiGF2YzEAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAoADSAEgAAABIAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAY//8AAAAyYXZjQwFkAAz/4QAZZ2QADKzZQod+IhAAAAMAEAAAAwPA8UKZYAEABmjr48siwAAAABhzdHRzAAAAAAAAAAEAAACdAAACAAAAABhzdHNzAAAAAAAAAAIAAAABAAAAHQAABJhjdHRzAAAAAAAAAJEAAAACAAAEAAAAAAEAAAYAAAAAAQAAAgAAAAACAAAEAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAGAAAAAAEAAAIAAAAAAQAABAAAAAABAAAIAAAAAAIAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAIAAAAAAIAAAIAAAAAAQAABgAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAYAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAQAAAAAAQAACAAAAAACAAACAAAAAAEAAAYAAAAAAQAAAgAAAAABAAAIAAAAAAIAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAEAAAAAAEAAAYAAAAAAQAAAgAAAAABAAAGAAAAAAEAAAIAAAAAAgAABAAAAAABAAAGAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAKAAAAAAEAAAQAAAAAAQAAAAAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAGAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAIAAAAAAIAAAIAAAAAAQAABgAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAGAAAAAAEAAAIAAAAAAQAACAAAAAACAAACAAAAAAEAAAgAAAAAAgAAAgAAAAABAAAGAAAAAAEAAAIAAAAAAQAABgAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAABAAAAAABAAAIAAAAAAIAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAGAAAAAAEAAAIAAAAAAQAABgAAAAABAAACAAAAAAEAAAoAAAAAAQAABAAAAAABAAAAAAAAAAEAAAIAAAAAAQAACgAAAAABAAAEAAAAAAEAAAAAAAAAAQAAAgAAAAABAAAIAAAAAAIAAAIAAAAAAQAABgAAAAABAAACAAAAABxzdHNjAAAAAAAAAAEAAAABAAAAnQAAAAEAAAKIc3RzegAAAAAAAAAAAAAAnQAABPIAAABKAAAAOAAAABoAAABTAAAAXgAAAGMAAABAAAAAGwAAAD0AAABcAAAAPAAAADwAAAAwAAAAUAAAAEsAAAA6AAAAJQAAAFUAAABEAAAAQQAAAEAAAABXAAAAQAAAAEAAAABHAAAAWgAAAEEAAAJQAAAAbgAAAHIAAAAyAAAAbQAAACsAAAAmAAAAMQAAAEYAAAA8AAAAKgAAAEAAAAAYAAAARQAAADQAAAAiAAAAIgAAADIAAAApAAAAHQAAACcAAAAzAAAAKwAAACsAAAAqAAAAMAAAABcAAAAsAAAAJwAAABgAAAAkAAAANwAAAHoAAAA3AAAAJwAAADwAAAA3AAAAPAAAADYAAAAyAAAAgQAAAFgAAAAhAAAAIQAAADsAAACKAAAAMAAAADoAAAAuAAAATAAAAEAAAABRAAAAJwAAAFwAAAAtAAAAKwAAACsAAAA2AAAALgAAACgAAAAvAAAALwAAAC8AAAApAAAALwAAADQAAAAmAAAAFAAAABcAAAA1AAAAKwAAAE8AAAA3AAAAIgAAACwAAABzAAAAPwAAADUAAAA/AAAAMQAAADIAAAA4AAAAMwAAACwAAABCAAAANgAAADEAAAAtAAAAOQAAADAAAABLAAAANgAAABkAAAA5AAAANQAAADYAAAA8AAAAKAAAAC4AAAAvAAAALQAAACoAAAAmAAAAKAAAADgAAAAyAAAAKwAAACcAAAB7AAAAKwAAACUAAAAnAAAAZgAAADwAAAAzAAAAJAAAAI4AAABtAAAAKwAAAGcAAABaAAAAKAAAACUAAAAgAAAANQAAACQAAAAaAAAAXAAAABoAAAAUc3RjbwAAAAAAAAABAAAAMAAAAGJ1ZHRhAAAAWm1ldGEAAAAAAAAAIWhkbHIAAAAAAAAAAG1kaXJhcHBsAAAAAAAAAAAAAAAALWlsc3QAAAAlqXRvbwAAAB1kYXRhAAAAAQAAAABMYXZmNTguMjkuMTAw\" type=\"video/mp4\"></video>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from IPython.display import HTML\n",
        "from base64 import b64encode\n",
        "\n",
        "# Load the video and encode it\n",
        "video_path = '/content/videos/'  # Make sure this matches the path where the videos are saved\n",
        "video_files = [f for f in os.listdir(video_path) if f.endswith('.mp4')]\n",
        "\n",
        "if video_files:\n",
        "    video_filename = video_files[-1]  # if you expect multiple videos, modify this to select the correct one\n",
        "    full_video_filename = f\"{video_path}/{video_filename}\"\n",
        "    mp4 = open(full_video_filename, 'rb').read()\n",
        "    encoded = b64encode(mp4).decode('ascii')\n",
        "    html = HTML(data=f'<video width=\"640\" height=\"480\" controls><source src=\"data:video/mp4;base64,{encoded}\" type=\"video/mp4\"></video>')\n",
        "else:\n",
        "    html = HTML(data=\"Error: No video found\")\n",
        "\n",
        "html\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "RPENO0Ne74f0"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "anaconda-cloud": {},
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3.9 (torch)",
      "language": "python",
      "name": "pytorch"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.18"
    },
    "varInspector": {
      "cols": {
        "lenName": 16,
        "lenType": 16,
        "lenVar": 40
      },
      "kernels_config": {
        "python": {
          "delete_cmd_postfix": "",
          "delete_cmd_prefix": "del ",
          "library": "var_list.py",
          "varRefreshCmd": "print(var_dic_list())"
        },
        "r": {
          "delete_cmd_postfix": ") ",
          "delete_cmd_prefix": "rm(",
          "library": "var_list.r",
          "varRefreshCmd": "cat(var_dic_list()) "
        }
      },
      "types_to_exclude": [
        "module",
        "function",
        "builtin_function_or_method",
        "instance",
        "_Feature"
      ],
      "window_display": false
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}