stable_video_diffusion_img2vid.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QIuzds5LLPyC"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mkshing/notebooks/blob/main/stable_video_diffusion_img2vid.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
        "# Stable Video Diffusion (image-to-video) Demo\n",
        "This notebook is the demo for the new image-to-video model, Stable Video Diffusion, from [Stability AI](https://stability.ai/) **on Colab free plan**.\n",
        "\n",
        "This was made by [mkshing](https://twitter.com/mk1stats).\n",
        "\n",
        "Visit the following links for the details of Stable Video Diffusion.\n",
        "* Codebase: https://github.com/Stability-AI/generative-models\n",
        "* HF: https://huggingface.co/stabilityai/stable-video-diffusion-img2vid\n",
        " * LICENSE: [STABLE VIDEO DIFFUSION NON-COMMERCIAL COMMUNITY LICENSE AGREEMENT](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid/blob/main/LICENSE)\n",
        "* Paper: https://stability.ai/research/stable-video-diffusion-scaling-latent-video-diffusion-models-to-large-datasets\n",
        "\n",
        "![000000](https://user-images.githubusercontent.com/33302880/284800538-f856b437-aa1f-4675-ba40-03da3e953358.gif)\n",
        "\n",
        "## Updates\n",
        "### 2023.11.27\n",
        "* Add the other hyper-parameters (`fps_id`, `motion_bucket_id`, `cond_aug`)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aaimSFWfLPgb",
        "outputId": "e0aa6dba-4b2e-49a1-eb41-a8be0db9fb83",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Wed Dec  6 07:48:18 2023       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   67C    P8    10W /  70W |      0MiB / 15360MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n",
            "Cloning into 'generative-models'...\n",
            "remote: Enumerating objects: 860, done.\u001b[K\n",
            "remote: Counting objects: 100% (513/513), done.\u001b[K\n",
            "remote: Compressing objects: 100% (244/244), done.\u001b[K\n",
            "remote: Total 860 (delta 365), reused 308 (delta 265), pack-reused 347\u001b[K\n",
            "Receiving objects: 100% (860/860), 42.67 MiB | 16.21 MiB/s, done.\n",
            "Resolving deltas: 100% (436/436), done.\n",
            "--2023-12-06 07:48:22--  https://gist.githubusercontent.com/mkshing/4ad40699756d996ba6b3f7934e6ca532/raw/3f0094272c7a2bd3eb5f1a0db91bed582c9e8f01/requirements.txt\n",
            "Resolving gist.githubusercontent.com (gist.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to gist.githubusercontent.com (gist.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 746 [text/plain]\n",
            "Saving to: ‘requirements.txt’\n",
            "\n",
            "requirements.txt    100%[===================>]     746  --.-KB/s    in 0s      \n",
            "\n",
            "2023-12-06 07:48:22 (47.4 MB/s) - ‘requirements.txt’ saved [746/746]\n",
            "\n",
            "Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cu118\n",
            "Collecting clip@ git+https://github.com/openai/CLIP.git (from -r requirements.txt (line 4))\n",
            "  Cloning https://github.com/openai/CLIP.git to /tmp/pip-install-gqdqe0w3/clip_955086c087944fa180db1b96754c4230\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/openai/CLIP.git /tmp/pip-install-gqdqe0w3/clip_955086c087944fa180db1b96754c4230\n",
            "  Resolved https://github.com/openai/CLIP.git to commit a1d071733d7111c9c014f024669f959182114e33\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting black==23.7.0 (from -r requirements.txt (line 2))\n",
            "  Downloading black-23.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m6.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting chardet==5.1.0 (from -r requirements.txt (line 3))\n",
            "  Downloading chardet-5.1.0-py3-none-any.whl (199 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m199.1/199.1 kB\u001b[0m \u001b[31m23.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting einops>=0.6.1 (from -r requirements.txt (line 5))\n",
            "  Downloading einops-0.7.0-py3-none-any.whl (44 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m44.6/44.6 kB\u001b[0m \u001b[31m6.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting fairscale>=0.4.13 (from -r requirements.txt (line 6))\n",
            "  Downloading fairscale-0.4.13.tar.gz (266 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m266.3/266.3 kB\u001b[0m \u001b[31m27.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Installing backend dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting fire>=0.5.0 (from -r requirements.txt (line 7))\n",
            "  Downloading fire-0.5.0.tar.gz (88 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m88.3/88.3 kB\u001b[0m \u001b[31m13.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: fsspec>=2023.6.0 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 8)) (2023.6.0)\n",
            "Collecting invisible-watermark>=0.2.0 (from -r requirements.txt (line 9))\n",
            "  Downloading invisible_watermark-0.2.0-py3-none-any.whl (1.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m61.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting kornia==0.6.9 (from -r requirements.txt (line 10))\n",
            "  Downloading kornia-0.6.9-py2.py3-none-any.whl (569 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m569.1/569.1 kB\u001b[0m \u001b[31m43.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting matplotlib>=3.7.2 (from -r requirements.txt (line 11))\n",
            "  Downloading matplotlib-3.8.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m11.6/11.6 MB\u001b[0m \u001b[31m110.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: natsort>=8.4.0 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 12)) (8.4.0)\n",
            "Collecting ninja>=1.11.1 (from -r requirements.txt (line 13))\n",
            "  Using cached ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)\n",
            "Collecting omegaconf>=2.3.0 (from -r requirements.txt (line 15))\n",
            "  Downloading omegaconf-2.3.0-py3-none-any.whl (79 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m79.5/79.5 kB\u001b[0m \u001b[31m10.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting open-clip-torch>=2.20.0 (from -r requirements.txt (line 16))\n",
            "  Downloading open_clip_torch-2.23.0-py3-none-any.whl (1.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.5/1.5 MB\u001b[0m \u001b[31m83.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting opencv-python==4.6.0.66 (from -r requirements.txt (line 17))\n",
            "  Downloading opencv_python-4.6.0.66-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (60.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m60.9/60.9 MB\u001b[0m \u001b[31m10.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pandas>=2.0.3 (from -r requirements.txt (line 18))\n",
            "  Downloading pandas-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.3/12.3 MB\u001b[0m \u001b[31m111.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pillow>=9.5.0 (from -r requirements.txt (line 19))\n",
            "  Downloading Pillow-10.1.0-cp310-cp310-manylinux_2_28_x86_64.whl (3.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.6/3.6 MB\u001b[0m \u001b[31m82.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pudb>=2022.1.3 (from -r requirements.txt (line 20))\n",
            "  Downloading pudb-2023.1.tar.gz (224 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m224.3/224.3 kB\u001b[0m \u001b[31m24.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting pytorch-lightning==2.0.1 (from -r requirements.txt (line 21))\n",
            "  Downloading pytorch_lightning-2.0.1-py3-none-any.whl (716 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m716.4/716.4 kB\u001b[0m \u001b[31m60.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyyaml>=6.0.1 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 22)) (6.0.1)\n",
            "Requirement already satisfied: scipy>=1.10.1 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 23)) (1.11.4)\n",
            "Collecting streamlit>=0.73.1 (from -r requirements.txt (line 24))\n",
            "  Downloading streamlit-1.29.0-py2.py3-none-any.whl (8.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.4/8.4 MB\u001b[0m \u001b[31m128.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tensorboardx==2.6 (from -r requirements.txt (line 25))\n",
            "  Downloading tensorboardX-2.6-py2.py3-none-any.whl (114 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.5/114.5 kB\u001b[0m \u001b[31m17.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting timm>=0.9.2 (from -r requirements.txt (line 26))\n",
            "  Downloading timm-0.9.12-py3-none-any.whl (2.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.2/2.2 MB\u001b[0m \u001b[31m104.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tokenizers==0.12.1 (from -r requirements.txt (line 27))\n",
            "  Downloading tokenizers-0.12.1-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.6/6.6 MB\u001b[0m \u001b[31m108.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: torch>=2.0.1 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 28)) (2.1.0+cu118)\n",
            "Requirement already satisfied: torchaudio>=2.0.2 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 29)) (2.1.0+cu118)\n",
            "Collecting torchdata==0.6.1 (from -r requirements.txt (line 30))\n",
            "  Downloading https://download.pytorch.org/whl/torchdata-0.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.6/4.6 MB\u001b[0m \u001b[31m102.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting torchmetrics>=1.0.1 (from -r requirements.txt (line 31))\n",
            "  Downloading torchmetrics-1.2.1-py3-none-any.whl (806 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m806.1/806.1 kB\u001b[0m \u001b[31m65.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: torchvision>=0.15.2 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 32)) (0.16.0+cu118)\n",
            "Requirement already satisfied: tqdm>=4.65.0 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 33)) (4.66.1)\n",
            "Collecting transformers==4.19.1 (from -r requirements.txt (line 34))\n",
            "  Downloading transformers-4.19.1-py3-none-any.whl (4.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.2/4.2 MB\u001b[0m \u001b[31m116.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting triton==2.0.0 (from -r requirements.txt (line 35))\n",
            "  Downloading https://download.pytorch.org/whl/triton-2.0.0-1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (63.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m63.3/63.3 MB\u001b[0m \u001b[31m10.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting urllib3<1.27,>=1.25.4 (from -r requirements.txt (line 36))\n",
            "  Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m143.8/143.8 kB\u001b[0m \u001b[31m19.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting wandb>=0.15.6 (from -r requirements.txt (line 37))\n",
            "  Downloading wandb-0.16.1-py3-none-any.whl (2.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m87.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting webdataset>=0.2.33 (from -r requirements.txt (line 38))\n",
            "  Downloading webdataset-0.2.83-py3-none-any.whl (68 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m68.8/68.8 kB\u001b[0m \u001b[31m8.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: wheel>=0.41.0 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 39)) (0.42.0)\n",
            "Collecting xformers>=0.0.20 (from -r requirements.txt (line 40))\n",
            "  Downloading https://download.pytorch.org/whl/cu118/xformers-0.0.22.post7%2Bcu118-cp310-cp310-manylinux2014_x86_64.whl (211.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m211.5/211.5 MB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: click>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from black==23.7.0->-r requirements.txt (line 2)) (8.1.7)\n",
            "Collecting mypy-extensions>=0.4.3 (from black==23.7.0->-r requirements.txt (line 2))\n",
            "  Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)\n",
            "Requirement already satisfied: packaging>=22.0 in /usr/local/lib/python3.10/dist-packages (from black==23.7.0->-r requirements.txt (line 2)) (23.2)\n",
            "Collecting pathspec>=0.9.0 (from black==23.7.0->-r requirements.txt (line 2))\n",
            "  Downloading pathspec-0.11.2-py3-none-any.whl (29 kB)\n",
            "Requirement already satisfied: platformdirs>=2 in /usr/local/lib/python3.10/dist-packages (from black==23.7.0->-r requirements.txt (line 2)) (4.0.0)\n",
            "Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from black==23.7.0->-r requirements.txt (line 2)) (2.0.1)\n",
            "Requirement already satisfied: numpy>=1.21.2 in /usr/local/lib/python3.10/dist-packages (from opencv-python==4.6.0.66->-r requirements.txt (line 17)) (1.23.5)\n",
            "Requirement already satisfied: typing-extensions>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==2.0.1->-r requirements.txt (line 21)) (4.5.0)\n",
            "Collecting lightning-utilities>=0.7.0 (from pytorch-lightning==2.0.1->-r requirements.txt (line 21))\n",
            "  Downloading lightning_utilities-0.10.0-py3-none-any.whl (24 kB)\n",
            "Requirement already satisfied: protobuf<4,>=3.8.0 in /usr/local/lib/python3.10/dist-packages (from tensorboardx==2.6->-r requirements.txt (line 25)) (3.20.3)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from torchdata==0.6.1->-r requirements.txt (line 30)) (2.31.0)\n",
            "Collecting torch>=2.0.1 (from -r requirements.txt (line 28))\n",
            "  Downloading https://download.pytorch.org/whl/cu118/torch-2.0.1%2Bcu118-cp310-cp310-linux_x86_64.whl (2267.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.3/2.3 GB\u001b[0m \u001b[31m606.3 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.1->-r requirements.txt (line 28)) (3.13.1)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.1->-r requirements.txt (line 28)) (1.12)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.1->-r requirements.txt (line 28)) (3.2.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=2.0.1->-r requirements.txt (line 28)) (3.1.2)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.1.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.19.1->-r requirements.txt (line 34)) (0.19.4)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.19.1->-r requirements.txt (line 34)) (2023.6.3)\n",
            "Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->-r requirements.txt (line 35)) (3.27.7)\n",
            "Collecting lit (from triton==2.0.0->-r requirements.txt (line 35))\n",
            "  Downloading lit-17.0.6.tar.gz (153 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m153.0/153.0 kB\u001b[0m \u001b[31m22.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Installing backend dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting ftfy (from clip@ git+https://github.com/openai/CLIP.git->-r requirements.txt (line 4))\n",
            "  Downloading ftfy-6.1.3-py3-none-any.whl (53 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.4/53.4 kB\u001b[0m \u001b[31m6.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.10/dist-packages (from fire>=0.5.0->-r requirements.txt (line 7)) (1.16.0)\n",
            "Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from fire>=0.5.0->-r requirements.txt (line 7)) (2.3.0)\n",
            "Requirement already satisfied: PyWavelets>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from invisible-watermark>=0.2.0->-r requirements.txt (line 9)) (1.5.0)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.7.2->-r requirements.txt (line 11)) (1.2.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.7.2->-r requirements.txt (line 11)) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.7.2->-r requirements.txt (line 11)) (4.45.1)\n",
            "Requirement already satisfied: kiwisolver>=1.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.7.2->-r requirements.txt (line 11)) (1.4.5)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.7.2->-r requirements.txt (line 11)) (3.1.1)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib>=3.7.2->-r requirements.txt (line 11)) (2.8.2)\n",
            "Collecting antlr4-python3-runtime==4.9.* (from omegaconf>=2.3.0->-r requirements.txt (line 15))\n",
            "  Downloading antlr4-python3-runtime-4.9.3.tar.gz (117 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m117.0/117.0 kB\u001b[0m \u001b[31m16.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting sentencepiece (from open-clip-torch>=2.20.0->-r requirements.txt (line 16))\n",
            "  Downloading sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m70.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas>=2.0.3->-r requirements.txt (line 18)) (2023.3.post1)\n",
            "Collecting tzdata>=2022.1 (from pandas>=2.0.3->-r requirements.txt (line 18))\n",
            "  Downloading tzdata-2023.3-py2.py3-none-any.whl (341 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m341.8/341.8 kB\u001b[0m \u001b[31m38.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting urwid>=1.1.1 (from pudb>=2022.1.3->-r requirements.txt (line 20))\n",
            "  Downloading urwid-2.3.4-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (274 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m274.9/274.9 kB\u001b[0m \u001b[31m36.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pygments>=2.7.4 in /usr/local/lib/python3.10/dist-packages (from pudb>=2022.1.3->-r requirements.txt (line 20)) (2.16.1)\n",
            "Collecting jedi<1,>=0.18 (from pudb>=2022.1.3->-r requirements.txt (line 20))\n",
            "  Downloading jedi-0.19.1-py2.py3-none-any.whl (1.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m93.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting urwid_readline (from pudb>=2022.1.3->-r requirements.txt (line 20))\n",
            "  Downloading urwid_readline-0.13.tar.gz (7.9 kB)\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: altair<6,>=4.0 in /usr/local/lib/python3.10/dist-packages (from streamlit>=0.73.1->-r requirements.txt (line 24)) (4.2.2)\n",
            "Requirement already satisfied: blinker<2,>=1.0.0 in /usr/lib/python3/dist-packages (from streamlit>=0.73.1->-r requirements.txt (line 24)) (1.4)\n",
            "Requirement already satisfied: cachetools<6,>=4.0 in /usr/local/lib/python3.10/dist-packages (from streamlit>=0.73.1->-r requirements.txt (line 24)) (5.3.2)\n",
            "Requirement already satisfied: importlib-metadata<7,>=1.4 in /usr/local/lib/python3.10/dist-packages (from streamlit>=0.73.1->-r requirements.txt (line 24)) (6.8.0)\n",
            "Requirement already satisfied: pyarrow>=6.0 in /usr/local/lib/python3.10/dist-packages (from streamlit>=0.73.1->-r requirements.txt (line 24)) (9.0.0)\n",
            "Requirement already satisfied: rich<14,>=10.14.0 in /usr/local/lib/python3.10/dist-packages (from streamlit>=0.73.1->-r requirements.txt (line 24)) (13.7.0)\n",
            "Requirement already satisfied: tenacity<9,>=8.1.0 in /usr/local/lib/python3.10/dist-packages (from streamlit>=0.73.1->-r requirements.txt (line 24)) (8.2.3)\n",
            "Requirement already satisfied: toml<2,>=0.10.1 in /usr/local/lib/python3.10/dist-packages (from streamlit>=0.73.1->-r requirements.txt (line 24)) (0.10.2)\n",
            "Requirement already satisfied: tzlocal<6,>=1.1 in /usr/local/lib/python3.10/dist-packages (from streamlit>=0.73.1->-r requirements.txt (line 24)) (5.2)\n",
            "Collecting validators<1,>=0.2 (from streamlit>=0.73.1->-r requirements.txt (line 24))\n",
            "  Downloading validators-0.22.0-py3-none-any.whl (26 kB)\n",
            "Collecting gitpython!=3.1.19,<4,>=3.0.7 (from streamlit>=0.73.1->-r requirements.txt (line 24))\n",
            "  Downloading GitPython-3.1.40-py3-none-any.whl (190 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m190.6/190.6 kB\u001b[0m \u001b[31m23.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pydeck<1,>=0.8.0b4 (from streamlit>=0.73.1->-r requirements.txt (line 24))\n",
            "  Downloading pydeck-0.8.1b0-py2.py3-none-any.whl (4.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.8/4.8 MB\u001b[0m \u001b[31m81.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tornado<7,>=6.0.3 in /usr/local/lib/python3.10/dist-packages (from streamlit>=0.73.1->-r requirements.txt (line 24)) (6.3.2)\n",
            "Collecting watchdog>=2.1.5 (from streamlit>=0.73.1->-r requirements.txt (line 24))\n",
            "  Downloading watchdog-3.0.0-py3-none-manylinux2014_x86_64.whl (82 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m82.1/82.1 kB\u001b[0m \u001b[31m10.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: safetensors in /usr/local/lib/python3.10/dist-packages (from timm>=0.9.2->-r requirements.txt (line 26)) (0.4.1)\n",
            "INFO: pip is looking at multiple versions of torchaudio to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting torchaudio>=2.0.2 (from -r requirements.txt (line 29))\n",
            "  Downloading https://download.pytorch.org/whl/cu118/torchaudio-2.1.1%2Bcu118-cp310-cp310-linux_x86_64.whl (3.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.2/3.2 MB\u001b[0m \u001b[31m78.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading torchaudio-2.1.1-cp310-cp310-manylinux1_x86_64.whl (3.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.3/3.3 MB\u001b[0m \u001b[31m110.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading torchaudio-2.1.0-cp310-cp310-manylinux1_x86_64.whl (3.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.3/3.3 MB\u001b[0m \u001b[31m120.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading https://download.pytorch.org/whl/cu118/torchaudio-2.0.2%2Bcu118-cp310-cp310-linux_x86_64.whl (4.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.4/4.4 MB\u001b[0m \u001b[31m114.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hINFO: pip is looking at multiple versions of torchvision to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting torchvision>=0.15.2 (from -r requirements.txt (line 32))\n",
            "  Downloading https://download.pytorch.org/whl/cu118/torchvision-0.16.1%2Bcu118-cp310-cp310-linux_x86_64.whl (6.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.1/6.1 MB\u001b[0m \u001b[31m118.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading torchvision-0.16.1-cp310-cp310-manylinux1_x86_64.whl (6.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.8/6.8 MB\u001b[0m \u001b[31m126.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading torchvision-0.16.0-cp310-cp310-manylinux1_x86_64.whl (6.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.9/6.9 MB\u001b[0m \u001b[31m128.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading https://download.pytorch.org/whl/cu118/torchvision-0.15.2%2Bcu118-cp310-cp310-linux_x86_64.whl (6.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.1/6.1 MB\u001b[0m \u001b[31m124.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: psutil>=5.0.0 in /usr/local/lib/python3.10/dist-packages (from wandb>=0.15.6->-r requirements.txt (line 37)) (5.9.5)\n",
            "Collecting sentry-sdk>=1.0.0 (from wandb>=0.15.6->-r requirements.txt (line 37))\n",
            "  Downloading sentry_sdk-1.38.0-py2.py3-none-any.whl (252 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m252.8/252.8 kB\u001b[0m \u001b[31m34.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting docker-pycreds>=0.4.0 (from wandb>=0.15.6->-r requirements.txt (line 37))\n",
            "  Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)\n",
            "Collecting setproctitle (from wandb>=0.15.6->-r requirements.txt (line 37))\n",
            "  Downloading setproctitle-1.3.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30 kB)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from wandb>=0.15.6->-r requirements.txt (line 37)) (67.7.2)\n",
            "Requirement already satisfied: appdirs>=1.4.3 in /usr/local/lib/python3.10/dist-packages (from wandb>=0.15.6->-r requirements.txt (line 37)) (1.4.4)\n",
            "Collecting braceexpand (from webdataset>=0.2.33->-r requirements.txt (line 38))\n",
            "  Downloading braceexpand-0.1.7-py2.py3-none-any.whl (5.9 kB)\n",
            "INFO: pip is looking at multiple versions of xformers to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting xformers>=0.0.20 (from -r requirements.txt (line 40))\n",
            "  Downloading xformers-0.0.22.post7-cp310-cp310-manylinux2014_x86_64.whl (211.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m211.8/211.8 MB\u001b[0m \u001b[31m2.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading https://download.pytorch.org/whl/cu118/xformers-0.0.22.post4%2Bcu118-cp310-cp310-manylinux2014_x86_64.whl (211.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m211.4/211.4 MB\u001b[0m \u001b[31m6.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading xformers-0.0.22.post4-cp310-cp310-manylinux2014_x86_64.whl (211.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m211.8/211.8 MB\u001b[0m \u001b[31m2.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading https://download.pytorch.org/whl/cu118/xformers-0.0.22.post3%2Bcu118-cp310-cp310-manylinux2014_x86_64.whl (211.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m211.5/211.5 MB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading xformers-0.0.22-cp310-cp310-manylinux2014_x86_64.whl (211.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m211.6/211.6 MB\u001b[0m \u001b[31m6.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair<6,>=4.0->streamlit>=0.73.1->-r requirements.txt (line 24)) (0.4)\n",
            "Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair<6,>=4.0->streamlit>=0.73.1->-r requirements.txt (line 24)) (4.19.2)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair<6,>=4.0->streamlit>=0.73.1->-r requirements.txt (line 24)) (0.12.0)\n",
            "Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /usr/local/lib/python3.10/dist-packages (from fsspec>=2023.6.0->-r requirements.txt (line 8)) (3.9.1)\n",
            "Collecting gitdb<5,>=4.0.1 (from gitpython!=3.1.19,<4,>=3.0.7->streamlit>=0.73.1->-r requirements.txt (line 24))\n",
            "  Downloading gitdb-4.0.11-py3-none-any.whl (62 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 kB\u001b[0m \u001b[31m8.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata<7,>=1.4->streamlit>=0.73.1->-r requirements.txt (line 24)) (3.17.0)\n",
            "Requirement already satisfied: parso<0.9.0,>=0.8.3 in /usr/local/lib/python3.10/dist-packages (from jedi<1,>=0.18->pudb>=2022.1.3->-r requirements.txt (line 20)) (0.8.3)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=2.0.1->-r requirements.txt (line 28)) (2.1.3)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->torchdata==0.6.1->-r requirements.txt (line 30)) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->torchdata==0.6.1->-r requirements.txt (line 30)) (3.6)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->torchdata==0.6.1->-r requirements.txt (line 30)) (2023.11.17)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich<14,>=10.14.0->streamlit>=0.73.1->-r requirements.txt (line 24)) (3.0.0)\n",
            "Requirement already satisfied: wcwidth<0.3.0,>=0.2.12 in /usr/local/lib/python3.10/dist-packages (from ftfy->clip@ git+https://github.com/openai/CLIP.git->-r requirements.txt (line 4)) (0.2.12)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=2.0.1->-r requirements.txt (line 28)) (1.3.0)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec>=2023.6.0->-r requirements.txt (line 8)) (23.1.0)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec>=2023.6.0->-r requirements.txt (line 8)) (6.0.4)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec>=2023.6.0->-r requirements.txt (line 8)) (1.9.3)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec>=2023.6.0->-r requirements.txt (line 8)) (1.4.0)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec>=2023.6.0->-r requirements.txt (line 8)) (1.3.1)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec>=2023.6.0->-r requirements.txt (line 8)) (4.0.3)\n",
            "Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->gitpython!=3.1.19,<4,>=3.0.7->streamlit>=0.73.1->-r requirements.txt (line 24))\n",
            "  Downloading smmap-5.0.1-py3-none-any.whl (24 kB)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit>=0.73.1->-r requirements.txt (line 24)) (2023.11.2)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit>=0.73.1->-r requirements.txt (line 24)) (0.31.1)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit>=0.73.1->-r requirements.txt (line 24)) (0.13.2)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich<14,>=10.14.0->streamlit>=0.73.1->-r requirements.txt (line 24)) (0.1.2)\n",
            "Building wheels for collected packages: clip, fairscale, fire, antlr4-python3-runtime, pudb, lit, urwid_readline\n",
            "  Building wheel for clip (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for clip: filename=clip-1.0-py3-none-any.whl size=1369497 sha256=d145492ca89a14df5fb199d957d0f52d92ce6f787a4f2c1b0666fad21e75ab1d\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-kzqh9bu2/wheels/da/2b/4c/d6691fa9597aac8bb85d2ac13b112deb897d5b50f5ad9a37e4\n",
            "  Building wheel for fairscale (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for fairscale: filename=fairscale-0.4.13-py3-none-any.whl size=332104 sha256=904f245281fcfebbfc60be4281bc0480266494c04978289b29b583dc2b638de2\n",
            "  Stored in directory: /root/.cache/pip/wheels/78/a4/c0/fb0a7ef03cff161611c3fa40c6cf898f76e58ec421b88e8cb3\n",
            "  Building wheel for fire (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for fire: filename=fire-0.5.0-py2.py3-none-any.whl size=116934 sha256=2234f4f30b2192d1d2f6a81c8e97bf6679b0a65fc895ded6f8025889db7893cd\n",
            "  Stored in directory: /root/.cache/pip/wheels/90/d4/f7/9404e5db0116bd4d43e5666eaa3e70ab53723e1e3ea40c9a95\n",
            "  Building wheel for antlr4-python3-runtime (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for antlr4-python3-runtime: filename=antlr4_python3_runtime-4.9.3-py3-none-any.whl size=144554 sha256=42cda9a9d87ad3fce45d8062fa275079862230261243781e05b281eb8a6203d2\n",
            "  Stored in directory: /root/.cache/pip/wheels/12/93/dd/1f6a127edc45659556564c5730f6d4e300888f4bca2d4c5a88\n",
            "  Building wheel for pudb (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pudb: filename=pudb-2023.1-py3-none-any.whl size=86117 sha256=62181a63b85195d15bd0851128f5eb586dad836bc29d4ece58ddbd8c9e1b1a6e\n",
            "  Stored in directory: /root/.cache/pip/wheels/50/03/e9/97cb1ce91fb1601f85470b71f11fcd3c6617e81735ccd4460c\n",
            "  Building wheel for lit (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for lit: filename=lit-17.0.6-py3-none-any.whl size=93255 sha256=d10d4548467adc2e6acf3f24fc718402eba2cbf5bcd7cf2313a8a264fc6e018c\n",
            "  Stored in directory: /root/.cache/pip/wheels/30/dd/04/47d42976a6a86dc2ab66d7518621ae96f43452c8841d74758a\n",
            "  Building wheel for urwid_readline (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for urwid_readline: filename=urwid_readline-0.13-py3-none-any.whl size=7549 sha256=2942c8dd45fccaa4f7450582ba4c8956594504a796579818900d1da77047003e\n",
            "  Stored in directory: /root/.cache/pip/wheels/58/1d/d8/20c6d76afd5bd205f5f95f19640df9a4e88fc6f1a4c25bb693\n",
            "Successfully built clip fairscale fire antlr4-python3-runtime pudb lit urwid_readline\n",
            "Installing collected packages: tokenizers, sentencepiece, ninja, lit, braceexpand, antlr4-python3-runtime, webdataset, watchdog, validators, urwid, urllib3, tzdata, tensorboardx, smmap, setproctitle, pillow, pathspec, opencv-python, omegaconf, mypy-extensions, lightning-utilities, jedi, ftfy, fire, einops, docker-pycreds, chardet, urwid_readline, sentry-sdk, pydeck, pandas, matplotlib, gitdb, black, pudb, gitpython, wandb, transformers, streamlit, triton, torch, torchvision, torchmetrics, timm, xformers, torchdata, torchaudio, pytorch-lightning, open-clip-torch, kornia, invisible-watermark, fairscale, clip\n",
            "  Attempting uninstall: tokenizers\n",
            "    Found existing installation: tokenizers 0.15.0\n",
            "    Uninstalling tokenizers-0.15.0:\n",
            "      Successfully uninstalled tokenizers-0.15.0\n",
            "  Attempting uninstall: urllib3\n",
            "    Found existing installation: urllib3 2.0.7\n",
            "    Uninstalling urllib3-2.0.7:\n",
            "      Successfully uninstalled urllib3-2.0.7\n",
            "  Attempting uninstall: pillow\n",
            "    Found existing installation: Pillow 9.4.0\n",
            "    Uninstalling Pillow-9.4.0:\n",
            "      Successfully uninstalled Pillow-9.4.0\n",
            "  Attempting uninstall: opencv-python\n",
            "    Found existing installation: opencv-python 4.8.0.76\n",
            "    Uninstalling opencv-python-4.8.0.76:\n",
            "      Successfully uninstalled opencv-python-4.8.0.76\n",
            "  Attempting uninstall: chardet\n",
            "    Found existing installation: chardet 5.2.0\n",
            "    Uninstalling chardet-5.2.0:\n",
            "      Successfully uninstalled chardet-5.2.0\n",
            "  Attempting uninstall: pandas\n",
            "    Found existing installation: pandas 1.5.3\n",
            "    Uninstalling pandas-1.5.3:\n",
            "      Successfully uninstalled pandas-1.5.3\n",
            "  Attempting uninstall: matplotlib\n",
            "    Found existing installation: matplotlib 3.7.1\n",
            "    Uninstalling matplotlib-3.7.1:\n",
            "      Successfully uninstalled matplotlib-3.7.1\n",
            "  Attempting uninstall: transformers\n",
            "    Found existing installation: transformers 4.35.2\n",
            "    Uninstalling transformers-4.35.2:\n",
            "      Successfully uninstalled transformers-4.35.2\n",
            "  Attempting uninstall: triton\n",
            "    Found existing installation: triton 2.1.0\n",
            "    Uninstalling triton-2.1.0:\n",
            "      Successfully uninstalled triton-2.1.0\n",
            "  Attempting uninstall: torch\n",
            "    Found existing installation: torch 2.1.0+cu118\n",
            "    Uninstalling torch-2.1.0+cu118:\n",
            "      Successfully uninstalled torch-2.1.0+cu118\n",
            "  Attempting uninstall: torchvision\n",
            "    Found existing installation: torchvision 0.16.0+cu118\n",
            "    Uninstalling torchvision-0.16.0+cu118:\n",
            "      Successfully uninstalled torchvision-0.16.0+cu118\n",
            "  Attempting uninstall: torchdata\n",
            "    Found existing installation: torchdata 0.7.0\n",
            "    Uninstalling torchdata-0.7.0:\n",
            "      Successfully uninstalled torchdata-0.7.0\n",
            "  Attempting uninstall: torchaudio\n",
            "    Found existing installation: torchaudio 2.1.0+cu118\n",
            "    Uninstalling torchaudio-2.1.0+cu118:\n",
            "      Successfully uninstalled torchaudio-2.1.0+cu118\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "lida 0.0.10 requires fastapi, which is not installed.\n",
            "lida 0.0.10 requires kaleido, which is not installed.\n",
            "lida 0.0.10 requires python-multipart, which is not installed.\n",
            "lida 0.0.10 requires uvicorn, which is not installed.\n",
            "google-colab 1.0.0 requires pandas==1.5.3, but you have pandas 2.1.3 which is incompatible.\n",
            "imageio 2.31.6 requires pillow<10.1.0,>=8.3.2, but you have pillow 10.1.0 which is incompatible.\n",
            "torchtext 0.16.0 requires torch==2.1.0, but you have torch 2.0.1+cu118 which is incompatible.\n",
            "torchtext 0.16.0 requires torchdata==0.7.0, but you have torchdata 0.6.1 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed antlr4-python3-runtime-4.9.3 black-23.7.0 braceexpand-0.1.7 chardet-5.1.0 clip-1.0 docker-pycreds-0.4.0 einops-0.7.0 fairscale-0.4.13 fire-0.5.0 ftfy-6.1.3 gitdb-4.0.11 gitpython-3.1.40 invisible-watermark-0.2.0 jedi-0.19.1 kornia-0.6.9 lightning-utilities-0.10.0 lit-17.0.6 matplotlib-3.8.2 mypy-extensions-1.0.0 ninja-1.11.1.1 omegaconf-2.3.0 open-clip-torch-2.23.0 opencv-python-4.6.0.66 pandas-2.1.3 pathspec-0.11.2 pillow-10.1.0 pudb-2023.1 pydeck-0.8.1b0 pytorch-lightning-2.0.1 sentencepiece-0.1.99 sentry-sdk-1.38.0 setproctitle-1.3.3 smmap-5.0.1 streamlit-1.29.0 tensorboardx-2.6 timm-0.9.12 tokenizers-0.12.1 torch-2.0.1+cu118 torchaudio-2.0.2+cu118 torchdata-0.6.1 torchmetrics-1.2.1 torchvision-0.15.2+cu118 transformers-4.19.1 triton-2.0.0 tzdata-2023.3 urllib3-1.26.18 urwid-2.3.4 urwid_readline-0.13 validators-0.22.0 wandb-0.16.1 watchdog-3.0.0 webdataset-0.2.83 xformers-0.0.22\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "PIL",
                  "matplotlib",
                  "mpl_toolkits",
                  "pydevd_plugins"
                ]
              }
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Obtaining file:///content/generative-models\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Checking if build backend supports build_editable ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build editable ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing editable metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Building wheels for collected packages: sgm\n",
            "  Building editable for sgm (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sgm: filename=sgm-0.1.0-py3-none-any.whl size=26920 sha256=f4cbe3b08861e0e5d364ff40e982b09292dd09b75b35c3340fcf1f0536e8a2de\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-zitk5fbz/wheels/12/9b/27/03142f4dee9fa0a99f6c146eae81eb66e17b781145ecb05fa5\n",
            "Successfully built sgm\n",
            "Installing collected packages: sgm\n",
            "Successfully installed sgm-0.1.0\n",
            "Obtaining sdata from git+https://github.com/Stability-AI/datapipelines.git@main#egg=sdata\n",
            "  Cloning https://github.com/Stability-AI/datapipelines.git (to revision main) to ./src/sdata\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/Stability-AI/datapipelines.git /content/src/sdata\n",
            "  Resolved https://github.com/Stability-AI/datapipelines.git to commit 8bce77d147033b3a5285b6d45ee85f33866964fc\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Installing collected packages: sdata\n",
            "  Running setup.py develop for sdata\n",
            "Successfully installed sdata-0.0.1\n",
            "Collecting gradio\n",
            "  Downloading gradio-4.8.0-py3-none-any.whl (16.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m16.5/16.5 MB\u001b[0m \u001b[31m75.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting aiofiles<24.0,>=22.0 (from gradio)\n",
            "  Downloading aiofiles-23.2.1-py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: altair<6.0,>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (4.2.2)\n",
            "Collecting fastapi (from gradio)\n",
            "  Downloading fastapi-0.104.1-py3-none-any.whl (92 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m92.9/92.9 kB\u001b[0m \u001b[31m13.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting ffmpy (from gradio)\n",
            "  Downloading ffmpy-0.3.1.tar.gz (5.5 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting gradio-client==0.7.1 (from gradio)\n",
            "  Downloading gradio_client-0.7.1-py3-none-any.whl (302 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m302.9/302.9 kB\u001b[0m \u001b[31m37.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting httpx (from gradio)\n",
            "  Downloading httpx-0.25.2-py3-none-any.whl (74 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.0/75.0 kB\u001b[0m \u001b[31m11.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: huggingface-hub>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (0.19.4)\n",
            "Requirement already satisfied: importlib-resources<7.0,>=1.3 in /usr/local/lib/python3.10/dist-packages (from gradio) (6.1.1)\n",
            "Requirement already satisfied: jinja2<4.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (3.1.2)\n",
            "Requirement already satisfied: markupsafe~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (2.1.3)\n",
            "Requirement already satisfied: matplotlib~=3.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (3.8.2)\n",
            "Requirement already satisfied: numpy~=1.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (1.23.5)\n",
            "Collecting orjson~=3.0 (from gradio)\n",
            "  Downloading orjson-3.9.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (138 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m138.7/138.7 kB\u001b[0m \u001b[31m20.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from gradio) (23.2)\n",
            "Requirement already satisfied: pandas<3.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (2.1.3)\n",
            "Requirement already satisfied: pillow<11.0,>=8.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (10.1.0)\n",
            "Collecting pydantic>=2.0 (from gradio)\n",
            "  Downloading pydantic-2.5.2-py3-none-any.whl (381 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m381.9/381.9 kB\u001b[0m \u001b[31m38.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pydub (from gradio)\n",
            "  Downloading pydub-0.25.1-py2.py3-none-any.whl (32 kB)\n",
            "Collecting python-multipart (from gradio)\n",
            "  Downloading python_multipart-0.0.6-py3-none-any.whl (45 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.7/45.7 kB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pyyaml<7.0,>=5.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (6.0.1)\n",
            "Requirement already satisfied: requests~=2.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (2.31.0)\n",
            "Collecting semantic-version~=2.0 (from gradio)\n",
            "  Downloading semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)\n",
            "Collecting tomlkit==0.12.0 (from gradio)\n",
            "  Downloading tomlkit-0.12.0-py3-none-any.whl (37 kB)\n",
            "Requirement already satisfied: typer[all]<1.0,>=0.9 in /usr/local/lib/python3.10/dist-packages (from gradio) (0.9.0)\n",
            "Requirement already satisfied: typing-extensions~=4.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (4.5.0)\n",
            "Collecting uvicorn>=0.14.0 (from gradio)\n",
            "  Downloading uvicorn-0.24.0.post1-py3-none-any.whl (59 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m59.7/59.7 kB\u001b[0m \u001b[31m9.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from gradio-client==0.7.1->gradio) (2023.6.0)\n",
            "Collecting websockets<12.0,>=10.0 (from gradio-client==0.7.1->gradio)\n",
            "  Downloading websockets-11.0.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (129 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m129.9/129.9 kB\u001b[0m \u001b[31m18.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio) (0.4)\n",
            "Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio) (4.19.2)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair<6.0,>=4.2.0->gradio) (0.12.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.14.0->gradio) (3.13.1)\n",
            "Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.14.0->gradio) (4.66.1)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio) (1.2.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio) (4.45.1)\n",
            "Requirement already satisfied: kiwisolver>=1.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio) (1.4.5)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio) (3.1.1)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib~=3.0->gradio) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas<3.0,>=1.0->gradio) (2023.3.post1)\n",
            "Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas<3.0,>=1.0->gradio) (2023.3)\n",
            "Collecting annotated-types>=0.4.0 (from pydantic>=2.0->gradio)\n",
            "  Downloading annotated_types-0.6.0-py3-none-any.whl (12 kB)\n",
            "Collecting pydantic-core==2.14.5 (from pydantic>=2.0->gradio)\n",
            "  Downloading pydantic_core-2.14.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m79.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting typing-extensions~=4.0 (from gradio)\n",
            "  Downloading typing_extensions-4.8.0-py3-none-any.whl (31 kB)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio) (3.6)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio) (1.26.18)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests~=2.0->gradio) (2023.11.17)\n",
            "Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/lib/python3.10/dist-packages (from typer[all]<1.0,>=0.9->gradio) (8.1.7)\n",
            "Collecting colorama<0.5.0,>=0.4.3 (from typer[all]<1.0,>=0.9->gradio)\n",
            "  Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)\n",
            "Collecting shellingham<2.0.0,>=1.3.0 (from typer[all]<1.0,>=0.9->gradio)\n",
            "  Downloading shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB)\n",
            "Requirement already satisfied: rich<14.0.0,>=10.11.0 in /usr/local/lib/python3.10/dist-packages (from typer[all]<1.0,>=0.9->gradio) (13.7.0)\n",
            "Collecting h11>=0.8 (from uvicorn>=0.14.0->gradio)\n",
            "  Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m9.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: anyio<4.0.0,>=3.7.1 in /usr/local/lib/python3.10/dist-packages (from fastapi->gradio) (3.7.1)\n",
            "Collecting starlette<0.28.0,>=0.27.0 (from fastapi->gradio)\n",
            "  Downloading starlette-0.27.0-py3-none-any.whl (66 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m67.0/67.0 kB\u001b[0m \u001b[31m10.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting httpcore==1.* (from httpx->gradio)\n",
            "  Downloading httpcore-1.0.2-py3-none-any.whl (76 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m76.9/76.9 kB\u001b[0m \u001b[31m11.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx->gradio) (1.3.0)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<4.0.0,>=3.7.1->fastapi->gradio) (1.2.0)\n",
            "Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (23.1.0)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (2023.11.2)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (0.31.1)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (0.13.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib~=3.0->gradio) (1.16.0)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich<14.0.0,>=10.11.0->typer[all]<1.0,>=0.9->gradio) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich<14.0.0,>=10.11.0->typer[all]<1.0,>=0.9->gradio) (2.16.1)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich<14.0.0,>=10.11.0->typer[all]<1.0,>=0.9->gradio) (0.1.2)\n",
            "Building wheels for collected packages: ffmpy\n",
            "  Building wheel for ffmpy (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for ffmpy: filename=ffmpy-0.3.1-py3-none-any.whl size=5579 sha256=3b6df48c247f86ffafa2c7fee9437a0a1334c31709455fefd0a71d2f3bb2c6a6\n",
            "  Stored in directory: /root/.cache/pip/wheels/01/a6/d1/1c0828c304a4283b2c1639a09ad86f83d7c487ef34c6b4a1bf\n",
            "Successfully built ffmpy\n",
            "Installing collected packages: pydub, ffmpy, websockets, typing-extensions, tomlkit, shellingham, semantic-version, python-multipart, orjson, h11, colorama, annotated-types, aiofiles, uvicorn, starlette, pydantic-core, httpcore, pydantic, httpx, gradio-client, fastapi, gradio\n",
            "  Attempting uninstall: typing-extensions\n",
            "    Found existing installation: typing_extensions 4.5.0\n",
            "    Uninstalling typing_extensions-4.5.0:\n",
            "      Successfully uninstalled typing_extensions-4.5.0\n",
            "  Attempting uninstall: pydantic\n",
            "    Found existing installation: pydantic 1.10.13\n",
            "    Uninstalling pydantic-1.10.13:\n",
            "      Successfully uninstalled pydantic-1.10.13\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "lida 0.0.10 requires kaleido, which is not installed.\n",
            "llmx 0.0.15a0 requires cohere, which is not installed.\n",
            "llmx 0.0.15a0 requires openai, which is not installed.\n",
            "llmx 0.0.15a0 requires tiktoken, which is not installed.\n",
            "tensorflow-probability 0.22.0 requires typing-extensions<4.6.0, but you have typing-extensions 4.8.0 which is incompatible.\n",
            "torchtext 0.16.0 requires torch==2.1.0, but you have torch 2.0.1+cu118 which is incompatible.\n",
            "torchtext 0.16.0 requires torchdata==0.7.0, but you have torchdata 0.6.1 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed aiofiles-23.2.1 annotated-types-0.6.0 colorama-0.4.6 fastapi-0.104.1 ffmpy-0.3.1 gradio-4.8.0 gradio-client-0.7.1 h11-0.14.0 httpcore-1.0.2 httpx-0.25.2 orjson-3.9.10 pydantic-2.5.2 pydantic-core-2.14.5 pydub-0.25.1 python-multipart-0.0.6 semantic-version-2.10.0 shellingham-1.5.4 starlette-0.27.0 tomlkit-0.12.0 typing-extensions-4.8.0 uvicorn-0.24.0.post1 websockets-11.0.3\n"
          ]
        }
      ],
      "source": [
        "#@title Setup\n",
        "!nvidia-smi\n",
        "!git clone https://github.com/Stability-AI/generative-models.git\n",
        "# install required packages from pypi\n",
        "# !pip3 install -r generative-models/requirements/pt2.txt\n",
        "# manually install only necesarry packages for colab\n",
        "!wget https://gist.githubusercontent.com/mkshing/4ad40699756d996ba6b3f7934e6ca532/raw/3f0094272c7a2bd3eb5f1a0db91bed582c9e8f01/requirements.txt\n",
        "!pip3 install -r requirements.txt\n",
        "!pip3 install -e generative-models\n",
        "!pip3 install -e git+https://github.com/Stability-AI/datapipelines.git@main#egg=sdata\n",
        "!pip3 install gradio"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zWlfaXvPbR1L"
      },
      "outputs": [],
      "source": [
        "#@title Colab hack for SVD\n",
        "# !pip uninstall -y numpy\n",
        "# !pip install -U numpy\n",
        "!mkdir -p /content/scripts/util/detection\n",
        "!ln -s /content/generative-models/scripts/util/detection/p_head_v1.npz /content/scripts/util/detection/p_head_v1.npz\n",
        "!ln -s /content/generative-models/scripts/util/detection/w_head_v1.npz /content/scripts/util/detection/w_head_v1.npz"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "v8O2yR3BLHv6",
        "outputId": "46bb82e4-dc9c-40db-d32e-d19c5b858070",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "download from https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt/resolve/main/svd_xt.safetensors to checkpoints/svd_xt.safetensors\n"
          ]
        }
      ],
      "source": [
        "# @title Download weights\n",
        "import os\n",
        "import subprocess\n",
        "version = \"svd_xt\" #@param [\"svd\", \"svd_xt\"]\n",
        "TYPE2PATH = {\n",
        "    \"svd\": [\"https://huggingface.co/stabilityai/stable-video-diffusion-img2vid/resolve/main/svd.safetensors\", \"checkpoints/svd.safetensors\"],\n",
        "    \"svd_xt\": [\"https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt/resolve/main/svd_xt.safetensors\", \"checkpoints/svd_xt.safetensors\"]\n",
        "}\n",
        "download_from, download_to = TYPE2PATH[version]\n",
        "# @markdown This will take several minutes. <br>\n",
        "# @markdown **Reference:**\n",
        "# @markdown * `svd`: [stabilityai/stable-video-diffusion-img2vid](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid) for 14 frames generation\n",
        "# @markdown * `svd_xt`: [stabilityai/stable-video-diffusion-img2vid-xt](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt) for 25 frames generation\n",
        "os.makedirs(\"checkpoints\", exist_ok=True)\n",
        "if os.path.exists(download_to):\n",
        "  print(\"Already downloaded\")\n",
        "else:\n",
        "  print(f\"download from {download_from} to {download_to}\")\n",
        "  subprocess.call([\"wget\", download_from, \"-O\", download_to])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9AZDrh-SUDt2",
        "outputId": "636ce9a8-1051-4bf5-a1df-8d4d63a5f311",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 460,
          "referenced_widgets": [
            "b4ef0f5ccf904c55a3b69b823801d1ef",
            "8444376bd68d42b9ac50aeed0e2c3b9a",
            "383448ef52ec4847b16ed729f2a0caca",
            "11489fe4651849a8bc054fa67e86b0f0",
            "71ccf91615414560a532a9edf392c05e",
            "ddf92eeafd3f49e0a94a795cf5f4faa5",
            "022b63015f3c420caca6f83ea8f8a3df",
            "7dbc547e38bb41bcbd6037f16d12e596",
            "3d0f376421324e2f8d3f4d2f8330613f",
            "3b480295989349af9b3233b1ace69249",
            "6df0e3bbf7054c5bb41c93bc72aac4e7"
          ]
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "VideoTransformerBlock is using checkpointing\n",
            "VideoTransformerBlock is using checkpointing\n",
            "VideoTransformerBlock is using checkpointing\n",
            "VideoTransformerBlock is using checkpointing\n",
            "VideoTransformerBlock is using checkpointing\n",
            "VideoTransformerBlock is using checkpointing\n",
            "VideoTransformerBlock is using checkpointing\n",
            "VideoTransformerBlock is using checkpointing\n",
            "VideoTransformerBlock is using checkpointing\n",
            "VideoTransformerBlock is using checkpointing\n",
            "VideoTransformerBlock is using checkpointing\n",
            "VideoTransformerBlock is using checkpointing\n",
            "VideoTransformerBlock is using checkpointing\n",
            "VideoTransformerBlock is using checkpointing\n",
            "VideoTransformerBlock is using checkpointing\n",
            "VideoTransformerBlock is using checkpointing\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "open_clip_pytorch_model.bin:   0%|          | 0.00/3.94G [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "b4ef0f5ccf904c55a3b69b823801d1ef"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Initialized embedder #0: FrozenOpenCLIPImagePredictionEmbedder with 683800065 params. Trainable: False\n",
            "Initialized embedder #1: ConcatTimestepEmbedderND with 0 params. Trainable: False\n",
            "Initialized embedder #2: ConcatTimestepEmbedderND with 0 params. Trainable: False\n",
            "Initialized embedder #3: VideoPredictionEmbedderWithEncoder with 83653863 params. Trainable: False\n",
            "Initialized embedder #4: ConcatTimestepEmbedderND with 0 params. Trainable: False\n",
            "Restored from checkpoints/svd_xt.safetensors with 0 missing and 0 unexpected keys\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|███████████████████████████████████████| 890M/890M [01:29<00:00, 10.4MiB/s]\n"
          ]
        }
      ],
      "source": [
        "#@title Load Model\n",
        "import sys\n",
        "from omegaconf import OmegaConf\n",
        "\n",
        "import torch\n",
        "\n",
        "sys.path.append(\"generative-models\")\n",
        "from sgm.util import default, instantiate_from_config\n",
        "from scripts.util.detection.nsfw_and_watermark_dectection import DeepFloydDataFiltering\n",
        "\n",
        "def load_model(\n",
        "    config: str,\n",
        "    device: str,\n",
        "    num_frames: int,\n",
        "    num_steps: int,\n",
        "):\n",
        "    config = OmegaConf.load(config)\n",
        "    config.model.params.conditioner_config.params.emb_models[\n",
        "        0\n",
        "    ].params.open_clip_embedding_config.params.init_device = device\n",
        "    config.model.params.sampler_config.params.num_steps = num_steps\n",
        "    config.model.params.sampler_config.params.guider_config.params.num_frames = (\n",
        "        num_frames\n",
        "    )\n",
        "    with torch.device(device):\n",
        "        model = instantiate_from_config(config.model).to(device).eval().requires_grad_(False)\n",
        "\n",
        "    filter = DeepFloydDataFiltering(verbose=False, device=device)\n",
        "    return model, filter\n",
        "\n",
        "\n",
        "if version == \"svd\":\n",
        "    num_frames = 14\n",
        "    num_steps = 25\n",
        "    # output_folder = default(output_folder, \"outputs/simple_video_sample/svd/\")\n",
        "    model_config = \"generative-models/scripts/sampling/configs/svd.yaml\"\n",
        "elif version == \"svd_xt\":\n",
        "    num_frames = 25\n",
        "    num_steps = 30\n",
        "    # output_folder = default(output_folder, \"outputs/simple_video_sample/svd_xt/\")\n",
        "    model_config = \"generative-models/scripts/sampling/configs/svd_xt.yaml\"\n",
        "else:\n",
        "    raise ValueError(f\"Version {version} does not exist.\")\n",
        "\n",
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "model, filter = load_model(\n",
        "    model_config,\n",
        "    device,\n",
        "    num_frames,\n",
        "    num_steps,\n",
        ")\n",
        "# move models expect unet to cpu\n",
        "model.conditioner.cpu()\n",
        "model.first_stage_model.cpu()\n",
        "# change the dtype of unet\n",
        "model.model.to(dtype=torch.float16)\n",
        "torch.cuda.empty_cache()\n",
        "model = model.requires_grad_(False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x1-dnq0RT95O"
      },
      "outputs": [],
      "source": [
        "# @title Sampling function\n",
        "import math\n",
        "import os\n",
        "from glob import glob\n",
        "from pathlib import Path\n",
        "from typing import Optional\n",
        "\n",
        "import cv2\n",
        "import numpy as np\n",
        "import torch\n",
        "from einops import rearrange, repeat\n",
        "from fire import Fire\n",
        "\n",
        "from PIL import Image\n",
        "from torchvision.transforms import ToTensor\n",
        "from torchvision.transforms import functional as TF\n",
        "\n",
        "from sgm.inference.helpers import embed_watermark\n",
        "from sgm.util import default, instantiate_from_config\n",
        "\n",
        "\n",
        "def get_unique_embedder_keys_from_conditioner(conditioner):\n",
        "    return list(set([x.input_key for x in conditioner.embedders]))\n",
        "\n",
        "\n",
        "def get_batch(keys, value_dict, N, T, device, dtype=None):\n",
        "    batch = {}\n",
        "    batch_uc = {}\n",
        "\n",
        "    for key in keys:\n",
        "        if key == \"fps_id\":\n",
        "            batch[key] = (\n",
        "                torch.tensor([value_dict[\"fps_id\"]])\n",
        "                .to(device, dtype=dtype)\n",
        "                .repeat(int(math.prod(N)))\n",
        "            )\n",
        "        elif key == \"motion_bucket_id\":\n",
        "            batch[key] = (\n",
        "                torch.tensor([value_dict[\"motion_bucket_id\"]])\n",
        "                .to(device, dtype=dtype)\n",
        "                .repeat(int(math.prod(N)))\n",
        "            )\n",
        "        elif key == \"cond_aug\":\n",
        "            batch[key] = repeat(\n",
        "                torch.tensor([value_dict[\"cond_aug\"]]).to(device, dtype=dtype),\n",
        "                \"1 -> b\",\n",
        "                b=math.prod(N),\n",
        "            )\n",
        "        elif key == \"cond_frames\":\n",
        "            batch[key] = repeat(value_dict[\"cond_frames\"], \"1 ... -> b ...\", b=N[0])\n",
        "        elif key == \"cond_frames_without_noise\":\n",
        "            batch[key] = repeat(\n",
        "                value_dict[\"cond_frames_without_noise\"], \"1 ... -> b ...\", b=N[0]\n",
        "            )\n",
        "        else:\n",
        "            batch[key] = value_dict[key]\n",
        "\n",
        "    if T is not None:\n",
        "        batch[\"num_video_frames\"] = T\n",
        "\n",
        "    for key in batch.keys():\n",
        "        if key not in batch_uc and isinstance(batch[key], torch.Tensor):\n",
        "            batch_uc[key] = torch.clone(batch[key])\n",
        "    return batch, batch_uc\n",
        "\n",
        "\n",
        "\n",
        "def sample(\n",
        "    input_path: str = \"assets/test_image.png\",  # Can either be image file or folder with image files\n",
        "    resize_image: bool = False,\n",
        "    num_frames: Optional[int] = None,\n",
        "    num_steps: Optional[int] = None,\n",
        "    fps_id: int = 6,\n",
        "    motion_bucket_id: int = 127,\n",
        "    cond_aug: float = 0.02,\n",
        "    seed: int = 23,\n",
        "    decoding_t: int = 14,  # Number of frames decoded at a time! This eats most VRAM. Reduce if necessary.\n",
        "    device: str = \"cuda\",\n",
        "    output_folder: Optional[str] = \"/content/outputs\",\n",
        "    skip_filter: bool = False,\n",
        "):\n",
        "    \"\"\"\n",
        "    Simple script to generate a single sample conditioned on an image `input_path` or multiple images, one for each\n",
        "    image file in folder `input_path`. If you run out of VRAM, try decreasing `decoding_t`.\n",
        "    \"\"\"\n",
        "    torch.manual_seed(seed)\n",
        "\n",
        "    path = Path(input_path)\n",
        "    all_img_paths = []\n",
        "    if path.is_file():\n",
        "        if any([input_path.endswith(x) for x in [\"jpg\", \"jpeg\", \"png\"]]):\n",
        "            all_img_paths = [input_path]\n",
        "        else:\n",
        "            raise ValueError(\"Path is not valid image file.\")\n",
        "    elif path.is_dir():\n",
        "        all_img_paths = sorted(\n",
        "            [\n",
        "                f\n",
        "                for f in path.iterdir()\n",
        "                if f.is_file() and f.suffix.lower() in [\".jpg\", \".jpeg\", \".png\"]\n",
        "            ]\n",
        "        )\n",
        "        if len(all_img_paths) == 0:\n",
        "            raise ValueError(\"Folder does not contain any images.\")\n",
        "    else:\n",
        "        raise ValueError\n",
        "    all_out_paths = []\n",
        "    for input_img_path in all_img_paths:\n",
        "        with Image.open(input_img_path) as image:\n",
        "            if image.mode == \"RGBA\":\n",
        "                image = image.convert(\"RGB\")\n",
        "            if resize_image and image.size != (1024, 576):\n",
        "                print(f\"Resizing {image.size} to (1024, 576)\")\n",
        "                image = TF.resize(TF.resize(image, 1024), (576, 1024))\n",
        "            w, h = image.size\n",
        "\n",
        "            if h % 64 != 0 or w % 64 != 0:\n",
        "                width, height = map(lambda x: x - x % 64, (w, h))\n",
        "                image = image.resize((width, height))\n",
        "                print(\n",
        "                    f\"WARNING: Your image is of size {h}x{w} which is not divisible by 64. We are resizing to {height}x{width}!\"\n",
        "                )\n",
        "\n",
        "            image = ToTensor()(image)\n",
        "            image = image * 2.0 - 1.0\n",
        "\n",
        "        image = image.unsqueeze(0).to(device)\n",
        "        H, W = image.shape[2:]\n",
        "        assert image.shape[1] == 3\n",
        "        F = 8\n",
        "        C = 4\n",
        "        shape = (num_frames, C, H // F, W // F)\n",
        "        if (H, W) != (576, 1024):\n",
        "            print(\n",
        "                \"WARNING: The conditioning frame you provided is not 576x1024. This leads to suboptimal performance as model was only trained on 576x1024. Consider increasing `cond_aug`.\"\n",
        "            )\n",
        "        if motion_bucket_id > 255:\n",
        "            print(\n",
        "                \"WARNING: High motion bucket! This may lead to suboptimal performance.\"\n",
        "            )\n",
        "\n",
        "        if fps_id < 5:\n",
        "            print(\"WARNING: Small fps value! This may lead to suboptimal performance.\")\n",
        "\n",
        "        if fps_id > 30:\n",
        "            print(\"WARNING: Large fps value! This may lead to suboptimal performance.\")\n",
        "\n",
        "        value_dict = {}\n",
        "        value_dict[\"motion_bucket_id\"] = motion_bucket_id\n",
        "        value_dict[\"fps_id\"] = fps_id\n",
        "        value_dict[\"cond_aug\"] = cond_aug\n",
        "        value_dict[\"cond_frames_without_noise\"] = image\n",
        "        value_dict[\"cond_frames\"] = image + cond_aug * torch.randn_like(image)\n",
        "        value_dict[\"cond_aug\"] = cond_aug\n",
        "        # low vram mode\n",
        "        model.conditioner.cpu()\n",
        "        model.first_stage_model.cpu()\n",
        "        torch.cuda.empty_cache()\n",
        "        model.sampler.verbose = True\n",
        "\n",
        "        with torch.no_grad():\n",
        "            with torch.autocast(device):\n",
        "                model.conditioner.to(device)\n",
        "                batch, batch_uc = get_batch(\n",
        "                    get_unique_embedder_keys_from_conditioner(model.conditioner),\n",
        "                    value_dict,\n",
        "                    [1, num_frames],\n",
        "                    T=num_frames,\n",
        "                    device=device,\n",
        "                )\n",
        "                c, uc = model.conditioner.get_unconditional_conditioning(\n",
        "                    batch,\n",
        "                    batch_uc=batch_uc,\n",
        "                    force_uc_zero_embeddings=[\n",
        "                        \"cond_frames\",\n",
        "                        \"cond_frames_without_noise\",\n",
        "                    ],\n",
        "                )\n",
        "                model.conditioner.cpu()\n",
        "                torch.cuda.empty_cache()\n",
        "\n",
        "                # from here, dtype is fp16\n",
        "                for k in [\"crossattn\", \"concat\"]:\n",
        "                    uc[k] = repeat(uc[k], \"b ... -> b t ...\", t=num_frames)\n",
        "                    uc[k] = rearrange(uc[k], \"b t ... -> (b t) ...\", t=num_frames)\n",
        "                    c[k] = repeat(c[k], \"b ... -> b t ...\", t=num_frames)\n",
        "                    c[k] = rearrange(c[k], \"b t ... -> (b t) ...\", t=num_frames)\n",
        "                for k in uc.keys():\n",
        "                    uc[k] = uc[k].to(dtype=torch.float16)\n",
        "                    c[k] = c[k].to(dtype=torch.float16)\n",
        "\n",
        "                randn = torch.randn(shape, device=device, dtype=torch.float16)\n",
        "\n",
        "                additional_model_inputs = {}\n",
        "                additional_model_inputs[\"image_only_indicator\"] = torch.zeros(\n",
        "                    2, num_frames\n",
        "                ).to(device, )\n",
        "                additional_model_inputs[\"num_video_frames\"] = batch[\"num_video_frames\"]\n",
        "\n",
        "                for k in additional_model_inputs:\n",
        "                    if isinstance(additional_model_inputs[k], torch.Tensor):\n",
        "                        additional_model_inputs[k] = additional_model_inputs[k].to(dtype=torch.float16)\n",
        "\n",
        "                def denoiser(input, sigma, c):\n",
        "                    return model.denoiser(\n",
        "                        model.model, input, sigma, c, **additional_model_inputs\n",
        "                    )\n",
        "\n",
        "                samples_z = model.sampler(denoiser, randn, cond=c, uc=uc)\n",
        "                samples_z.to(dtype=model.first_stage_model.dtype)\n",
        "                ##\n",
        "\n",
        "                model.en_and_decode_n_samples_a_time = decoding_t\n",
        "                model.first_stage_model.to(device)\n",
        "                samples_x = model.decode_first_stage(samples_z)\n",
        "                samples = torch.clamp((samples_x + 1.0) / 2.0, min=0.0, max=1.0)\n",
        "                model.first_stage_model.cpu()\n",
        "                torch.cuda.empty_cache()\n",
        "\n",
        "                os.makedirs(output_folder, exist_ok=True)\n",
        "                base_count = len(glob(os.path.join(output_folder, \"*.mp4\")))\n",
        "                video_path = os.path.join(output_folder, f\"{base_count:06d}.mp4\")\n",
        "                writer = cv2.VideoWriter(\n",
        "                    video_path,\n",
        "                    cv2.VideoWriter_fourcc(*\"MP4V\"),\n",
        "                    fps_id + 1,\n",
        "                    (samples.shape[-1], samples.shape[-2]),\n",
        "                )\n",
        "\n",
        "                samples = embed_watermark(samples)\n",
        "                if not skip_filter:\n",
        "                    samples = filter(samples)\n",
        "                else:\n",
        "                    print(\"WARNING: You have disabled the NSFW/Watermark filter. Please do not expose unfiltered results in services or applications open to the public.\")\n",
        "                vid = (\n",
        "                    (rearrange(samples, \"t c h w -> t h w c\") * 255)\n",
        "                    .cpu()\n",
        "                    .numpy()\n",
        "                    .astype(np.uint8)\n",
        "                )\n",
        "                for frame in vid:\n",
        "                    frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)\n",
        "                    writer.write(frame)\n",
        "                writer.release()\n",
        "                all_out_paths.append(video_path)\n",
        "    return all_out_paths\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5MdVILPlMUDe",
        "outputId": "9d75b30d-06f5-4a30-df4b-d97283bf57bb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 737
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Colab notebook detected. This cell will run indefinitely so that you can see errors and logs. To turn off, set debug=False in launch().\n",
            "Running on public URL: https://b5d0f0de40d1444557.gradio.live\n",
            "\n",
            "This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<div><iframe src=\"https://b5d0f0de40d1444557.gradio.live\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "##############################  Sampling setting  ##############################\n",
            "Sampler: EulerEDMSampler\n",
            "Discretization: EDMDiscretization\n",
            "Guider: LinearPredictionGuider\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\rSampling with EulerEDMSampler for 31 steps:   0%|          | 0/31 [00:00<?, ?it/s]/usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:31: UserWarning: None of the inputs have requires_grad=True. Gradients will be None\n",
            "  warnings.warn(\"None of the inputs have requires_grad=True. Gradients will be None\")\n",
            "Sampling with EulerEDMSampler for 31 steps:  39%|███▊      | 12/31 [02:49<04:28, 14.13s/it]"
          ]
        }
      ],
      "source": [
        "# @title Do the Run!\n",
        "# @markdown Generation takes about 10 mins for `svd_xt` on T4 (Colab free plan). Please be patient...\n",
        "import gradio as gr\n",
        "import random\n",
        "\n",
        "\n",
        "def infer(input_path: str, resize_image: bool, n_frames: int, n_steps: int, seed: str, decoding_t: int, fps_id: int, motion_bucket_id: int, cond_aug: float, skip_filter: bool = False) -> str:\n",
        "  if seed == \"random\":\n",
        "    seed = random.randint(0, 2**32)\n",
        "  seed = int(seed)\n",
        "  output_paths = sample(\n",
        "    input_path=input_path,\n",
        "    resize_image=resize_image,\n",
        "    num_frames=n_frames,\n",
        "    num_steps=n_steps,\n",
        "    fps_id=fps_id,\n",
        "    motion_bucket_id=motion_bucket_id,\n",
        "    cond_aug=cond_aug,\n",
        "    seed=seed,\n",
        "    decoding_t=decoding_t,  # Number of frames decoded at a time! This eats most VRAM. Reduce if necessary.\n",
        "    device=device,\n",
        "    skip_filter=skip_filter,\n",
        "  )\n",
        "  return output_paths[0]\n",
        "\n",
        "\n",
        "with gr.Blocks() as demo:\n",
        "  with gr.Column():\n",
        "    image = gr.Image(label=\"input image\", type=\"filepath\")\n",
        "    resize_image = gr.Checkbox(label=\"resize to optimal size\", value=True)\n",
        "    btn = gr.Button(\"Run\")\n",
        "    with gr.Accordion(label=\"Advanced options\", open=False):\n",
        "      n_frames = gr.Number(precision=0, label=\"number of frames\", value=num_frames)\n",
        "      n_steps = gr.Number(precision=0, label=\"number of steps\", value=num_steps)\n",
        "      seed = gr.Text(value=\"random\", label=\"seed (integer or 'random')\",)\n",
        "      decoding_t = gr.Number(precision=0, label=\"number of frames decoded at a time\", value=2)\n",
        "      fps_id = gr.Number(precision=0, label=\"frames per second\", value=6)\n",
        "      motion_bucket_id = gr.Number(precision=0, value=127, label=\"motion bucket id\")\n",
        "      cond_aug = gr.Number(label=\"condition augmentation factor\", value=0.02)\n",
        "      skip_filter = gr.Checkbox(value=False, label=\"skip nsfw/watermark filter\")\n",
        "  with gr.Column():\n",
        "    video_out = gr.Video(label=\"generated video\")\n",
        "  examples = [\n",
        "      [\"https://user-images.githubusercontent.com/33302880/284758167-367a25d8-8d7b-42d3-8391-6d82813c7b0f.png\"]\n",
        "  ]\n",
        "  inputs = [image, resize_image, n_frames, n_steps, seed, decoding_t, fps_id, motion_bucket_id, cond_aug, skip_filter]\n",
        "  outputs = [video_out]\n",
        "  btn.click(infer, inputs=inputs, outputs=outputs)\n",
        "  gr.Examples(examples=examples, inputs=inputs, outputs=outputs, fn=infer)\n",
        "  demo.queue().launch(debug=True, share=True, show_error=True)"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "n3s1XTyEIej8"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "b4ef0f5ccf904c55a3b69b823801d1ef": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_8444376bd68d42b9ac50aeed0e2c3b9a",
              "IPY_MODEL_383448ef52ec4847b16ed729f2a0caca",
              "IPY_MODEL_11489fe4651849a8bc054fa67e86b0f0"
            ],
            "layout": "IPY_MODEL_71ccf91615414560a532a9edf392c05e"
          }
        },
        "8444376bd68d42b9ac50aeed0e2c3b9a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ddf92eeafd3f49e0a94a795cf5f4faa5",
            "placeholder": "​",
            "style": "IPY_MODEL_022b63015f3c420caca6f83ea8f8a3df",
            "value": "open_clip_pytorch_model.bin: 100%"
          }
        },
        "383448ef52ec4847b16ed729f2a0caca": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7dbc547e38bb41bcbd6037f16d12e596",
            "max": 3944692325,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3d0f376421324e2f8d3f4d2f8330613f",
            "value": 3944692325
          }
        },
        "11489fe4651849a8bc054fa67e86b0f0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3b480295989349af9b3233b1ace69249",
            "placeholder": "​",
            "style": "IPY_MODEL_6df0e3bbf7054c5bb41c93bc72aac4e7",
            "value": " 3.94G/3.94G [00:24&lt;00:00, 257MB/s]"
          }
        },
        "71ccf91615414560a532a9edf392c05e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ddf92eeafd3f49e0a94a795cf5f4faa5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "022b63015f3c420caca6f83ea8f8a3df": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7dbc547e38bb41bcbd6037f16d12e596": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3d0f376421324e2f8d3f4d2f8330613f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "3b480295989349af9b3233b1ace69249": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6df0e3bbf7054c5bb41c93bc72aac4e7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}