avatarify.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Avatarify",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMqm8Y/SbrWZMeNGn+BbnaN",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/alievk/avatarify/blob/master/avatarify.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pEl-Q4OpsLb9",
        "colab_type": "text"
      },
      "source": [
        "# Avatarify Colab Server\n",
        "\n",
        "This Colab notebook is for running Avatarify rendering server. It allows you to run Avatarify on your computer **without GPU** in this way:\n",
        "\n",
        "1. When this notebook is executed, it starts listening for incoming requests from your computer;\n",
        "1. You start the client on your computer and it connects to the notebook and starts sending requests;\n",
        "1. This notebooks receives the requests from your computer, renders avatar images and sends them back;\n",
        "\n",
        "To this end, all the heavy work is offloaded from your computer to this notebook so you don't need to have a beafy hardware on your PC anymore.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wRpMPl7VyeoD",
        "colab_type": "text"
      },
      "source": [
        "## Start the server\n",
        "Run the cells below (Shift+Enter) sequentially and pay attention to the hints and instructions included in this notebook.\n",
        "\n",
        "At the end you will get a command for running the client on your computer.\n",
        "\n",
        "## Start the client\n",
        "\n",
        "Make sure you have installed the latest version of Avatarify on your computer. Refer to the [README](https://github.com/alievk/avatarify#install) for the instructions.\n",
        "\n",
        "When it's ready execute this notebook and get the command for running the client on your computer.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8h0f5WQEjgbH",
        "colab_type": "text"
      },
      "source": [
        "### Technical details\n",
        "\n",
        "The client on your computer connects to the server via `ngrok` TCP tunnel or a reverse `ssh` tunnel.\n",
        "\n",
        "`ngrok`, while easy to use, can induce a considerable network lag ranging from dozens of milliseconds to a second. This can lead to a poor experience.\n",
        "\n",
        "A more stable connection could be established using a reverse `ssh` tunnel to a host with a public IP, like an AWS `t3.micro` (free) instance. This notebook provides a script for creating a tunnel, but launching an instance in a cloud is on your own (find the manual below)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ZI4EvKaNUhL",
        "colab_type": "text"
      },
      "source": [
        "# Install"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9fYm9X3X125H",
        "colab_type": "text"
      },
      "source": [
        "### Avatarify\n",
        "Follow the steps below to clone Avatarify and install the dependencies."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LC1q-hdat-JP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!cd /content\n",
        "!rm -rf *"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kE4_YSbsiX_O",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!git clone https://github.com/alievk/avatarify.git"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I8fTBhOEUM_c",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "cd avatarify"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FONInDgZUmcZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!git clone https://github.com/alievk/first-order-model.git fomm\n",
        "!pip install face-alignment==1.0.0 msgpack_numpy pyyaml==5.1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hPgXoqE_gAyD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!scripts/download_data.sh"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j1soT4zEEFzp",
        "colab_type": "text"
      },
      "source": [
        "### ngrok\n",
        "Follow the steps below to setup ngrok. You will also need to sign up on the ngrok site and get your authtoken (free).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bbptNwHL1s61",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Download ngrok\n",
        "!scripts/get_ngrok.sh"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4qk1FCeeaviZ",
        "colab_type": "text"
      },
      "source": [
        "# Run\n",
        "Start here if the runtime was restarted after installation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_f2iYcQVI2ss",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "cd /content/avatarify"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qxK_ZZjPz_Rr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#!git pull origin"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MA-h22jF6-ks",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from subprocess import Popen, PIPE\n",
        "import shlex\n",
        "import json\n",
        "import time\n",
        "\n",
        "\n",
        "def run_with_pipe(command):\n",
        "  commands = list(map(shlex.split,command.split(\"|\")))\n",
        "  ps = Popen(commands[0], stdout=PIPE, stderr=PIPE)\n",
        "  for command in commands[1:]:\n",
        "    ps = Popen(command, stdin=ps.stdout, stdout=PIPE, stderr=PIPE)\n",
        "  return ps.stdout.readlines()\n",
        "\n",
        "\n",
        "def get_tunnel_adresses():\n",
        "  info = run_with_pipe(\"curl http://localhost:4040/api/tunnels\")\n",
        "  assert info\n",
        "\n",
        "  info = json.loads(info[0])\n",
        "  for tunnel in info['tunnels']:\n",
        "    url = tunnel['public_url']\n",
        "    port = url.split(':')[-1]\n",
        "    local_port = tunnel['config']['addr'].split(':')[-1]\n",
        "    print(f'{url} -> {local_port} [{tunnel[\"name\"]}]')\n",
        "    if tunnel['name'] == 'input':\n",
        "      in_addr = url\n",
        "    elif tunnel['name'] == 'output':\n",
        "      out_addr = url\n",
        "    else:\n",
        "      print(f'unknown tunnel: {tunnel[\"name\"]}')\n",
        "\n",
        "  return in_addr, out_addr"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RfHa02CBWoNN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Input and output ports for communication\n",
        "local_in_port = 5557\n",
        "local_out_port = 5558"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BcULGnhGJJjC",
        "colab_type": "text"
      },
      "source": [
        "# Start the worker\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8PnArK75mRqx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# (Re)Start the worker\n",
        "with open('/tmp/run.txt', 'w') as f:\n",
        "  ps = Popen(\n",
        "      shlex.split(f'./run.sh --is-worker --in-port {local_in_port} --out-port {local_out_port} --no-vcam --no-conda'),\n",
        "      stdout=f, stderr=f)\n",
        "  time.sleep(3)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XfUqQxMtRSvc",
        "colab_type": "text"
      },
      "source": [
        "This command should print lines if the worker is successfully started"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W0eY8gkBqUJG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!ps aux | grep 'python3 afy/cam_fomm.py' | grep -v grep | tee /tmp/ps_run\n",
        "!if [[ $(cat /tmp/ps_run | wc -l) == \"0\" ]]; then echo \"Worker failed to start\"; cat /tmp/run.txt; else echo \"Worker started\"; fi"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tz9gpLD0IsCL",
        "colab_type": "text"
      },
      "source": [
        "# Open ngrok tunnel"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gyB7XIxL0XpD",
        "colab_type": "text"
      },
      "source": [
        "#### Get ngrok token\n",
        "Go to https://dashboard.ngrok.com/auth/your-authtoken (sign up if required), copy your authtoken and put it below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YDtPpi77AkQ1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Paste your authtoken here in quotes\n",
        "authtoken = \"1cBzFFwzSlaLhlRPXIHJiVLqtiQ_2cVsonJXe52B6DDyp8su7\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gASaDrsFXLXA",
        "colab_type": "text"
      },
      "source": [
        "Set your region\n",
        "\n",
        "Code | Region\n",
        "--- | ---\n",
        "us | United States\n",
        "eu | Europe\n",
        "ap | Asia/Pacific\n",
        "au | Australia\n",
        "sa | South America\n",
        "jp | Japan\n",
        "in | India"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r5e9VR9NYckJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Set your region here in quotes\n",
        "region = \"eu\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jZ5_PE_EHpCg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "config =\\\n",
        "f\"\"\"\n",
        "version: 2\n",
        "authtoken: {authtoken}\n",
        "region: {region}\n",
        "console_ui: False\n",
        "tunnels:\n",
        "  input:\n",
        "    addr: {local_in_port}\n",
        "    proto: tcp    \n",
        "  output:\n",
        "    addr: {local_out_port}\n",
        "    proto: tcp\n",
        "\"\"\"\n",
        "\n",
        "with open('ngrok.conf', 'w') as f:\n",
        "  f.write(config)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z49OEhAdDI7Y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# (Re)Open tunnel\n",
        "ps = Popen('./scripts/open_tunnel_ngrok.sh', stdout=PIPE, stderr=PIPE)\n",
        "time.sleep(3)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JAyPH2t2C64H",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Get tunnel addresses\n",
        "try:\n",
        "  in_addr, out_addr = get_tunnel_adresses()\n",
        "  print(\"Tunnel opened\")\n",
        "except Exception as e:\n",
        "  [print(l.decode(), end='') for l in ps.stdout.readlines()]\n",
        "  print(\"Something went wrong, reopen the tunnel\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dc6rg2HQAocK",
        "colab_type": "text"
      },
      "source": [
        "### [Optional] AWS proxy\n",
        "Alternatively you can create a ssh reverse tunnel to an AWS `t3.micro` instance (it's free). It has lower latency than ngrok.\n",
        "\n",
        "1. In your AWS console go to Services -> EC2 -> Instances -> Launch Instance;\n",
        "1. Choose `Ubuntu Server 18.04 LTS` AMI;\n",
        "1. Choose `t3.micro` instance type and press Review and launch;\n",
        "1. Confirm your key pair and press Launch instances;\n",
        "1. Go to the security group of this instance and edit inbound rules. Add TCP ports 5557 and 5558 and set Source to Anywhere. Press Save rules;\n",
        "1. ssh into the instance (you can find the command in the Instances if you click on the Connect button) and add this line in the end of `/etc/ssh/sshd_config`:\n",
        "```\n",
        "GatewayPorts yes\n",
        "```\n",
        "then restart `sshd`\n",
        "```\n",
        "sudo service sshd restart\n",
        "```\n",
        "1. Copy your `key_pair.pem` by dragging and dropping it into avatarify folder in this notebook;\n",
        "1. Use the command below to open the tunnel;\n",
        "1. Start client with a command (substitute `run_mac.sh` with `run_windows.bat` or `run.sh`)\n",
        "```\n",
        "./run_mac.sh --is-client --in-addr tcp://instace.compute.amazonaws.com:5557 --out-addr tcp://instance.compute.amazonaws.com:5558\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zdN5Qj2BCYsr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Open reverse ssh tunnel (uncomment line below)\n",
        "# !./scripts/open_tunnel_ssh.sh key_pair.pem ubuntu@instance.compute.amazonaws.com"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ccZ24BT4Jdis",
        "colab_type": "text"
      },
      "source": [
        "# Start the client\n",
        "When you run the cell below it will print a command. Run this command on your computer:\n",
        "\n",
        "1. Open a terminal (in Windows open `Anaconda Prompt`);\n",
        "2. Change working directory to the `avatarify` directory:</br>\n",
        "* Windows (change `C:\\path\\to\\avatarify` to your path)</br>\n",
        "`cd C:\\path\\to\\avatarify`</br></br>\n",
        "* Mac/Linux (change `/path/to/avatarify` to your path)</br>\n",
        "`cd /path/to/avatarify`\n",
        "3. Copy-paste to the terminal the command below and run;\n",
        "4. It can take some time to connect (usually up to 10 seconds). If the preview window doesn't appear in a minute or two, look for the errors above in this notebook and report in the [issues](https://github.com/alievk/avatarify/issues) or [Slack](https://join.slack.com/t/avatarify/shared_invite/zt-dyoqy8tc-~4U2ObQ6WoxuwSaWKKVOgg)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4gaqS0mZWF1V",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print('Copy-paste to the terminal the command below and run (press Enter)\\n')\n",
        "print('Mac:')\n",
        "print(f'./run_mac.sh --is-client --in-addr {in_addr} --out-addr {out_addr}')\n",
        "print('\\nWindows:')\n",
        "print(f'run_windows.bat --is-client --in-addr {in_addr} --out-addr {out_addr}')\n",
        "print('\\nLinux:')\n",
        "print(f'./run.sh --is-client --in-addr {in_addr} --out-addr {out_addr}')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M3h92xQ9KA-R",
        "colab_type": "text"
      },
      "source": [
        "# Logs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SvodbjapKBQi",
        "colab_type": "text"
      },
      "source": [
        "If something doesn't work as expected, please run the cells below and include the logs in your report."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0GeT7KxON0Ke",
        "colab_type": "code",
        "colab": {},
        "cellView": "form"
      },
      "source": [
        "#@title\n",
        "!cat ./var/log/cam_fomm.log | head -100"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P1FQcdzwqdce",
        "colab_type": "code",
        "colab": {},
        "cellView": "form"
      },
      "source": [
        "#@title\n",
        "!cat ./var/log/recv_worker.log | tail -100"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YThWBXCf_yzI",
        "colab_type": "code",
        "colab": {},
        "cellView": "form"
      },
      "source": [
        "#@title\n",
        "!cat ./var/log/predictor_worker.log | tail -100"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zhJzygCP_6p3",
        "colab_type": "code",
        "colab": {},
        "cellView": "form"
      },
      "source": [
        "#@title\n",
        "!cat ./var/log/send_worker.log | tail -100"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nrzNffhR_8HQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}