MRM_modeling.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "MRM_modeling.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/asu-trans-ai-lab/Integrated_modeling_GMNS/blob/main/MRM_modeling.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HUArQQu8ckkC"
      },
      "source": [
        "## Introduction\n",
        "- **GMNS**: The General Modeling Network Specification (GMNS) defines a common human and machine-readable format for sharing routable road network files. It is designed to be used in multi-modal static and dynamic transportation planning and operations models.\n",
        "\n",
        "- **AMS**: The objective of the AMS data hub is to define a prototype of operations and associated requirements that will allow for the effective integration of analysis modeling and simulation tools across various domains and scales. Easily exchange information and data at both the input and output levels.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7TN3JHepPEq2",
        "outputId": "6926a443-594f-4da4-e2aa-3b9a1dd339ae"
      },
      "source": [
        "!pip install osm2gmns\n"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: osm2gmns in /usr/local/lib/python3.7/dist-packages (0.6.4)\n",
            "Requirement already satisfied: shapely>=1.7 in /usr/local/lib/python3.7/dist-packages (from osm2gmns) (1.8.1.post1)\n",
            "Requirement already satisfied: osmium>=3.1.3 in /usr/local/lib/python3.7/dist-packages (from osm2gmns) (3.2.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from osmium>=3.1.3->osm2gmns) (2.23.0)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->osmium>=3.1.3->osm2gmns) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->osmium>=3.1.3->osm2gmns) (2021.10.8)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->osmium>=3.1.3->osm2gmns) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->osmium>=3.1.3->osm2gmns) (3.0.4)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_5iPhrNgdP6A"
      },
      "source": [
        "\"import packages\"\n",
        "import pandas as pd\n",
        "\n",
        "\n",
        "\n",
        "from google.colab import files\n",
        "import requests\n",
        "import io\n",
        "import os\n",
        "import time"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "55xMlltndZBA"
      },
      "source": [
        "#### Step 0. OSM data download\n",
        "\n",
        "https://osm2gmns.readthedocs.io/en/latest/index.html\n",
        "----\n",
        "| Step |    Description    |       Software       | Input Files | Output Files |\n",
        "|:----:|:-----------------:|:--------------------:|:-----------:|:------------:|\n",
        "| 0    | OSM data download | Open Street Map(OSM) | - -         | Map.osm      |"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i3_IjzP9dVu4"
      },
      "source": [
        "\"Step 0.1 define output map folders\"\n",
        "map_folder = 'map_folder'\n"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oEhLBvjhwXzZ"
      },
      "source": [
        "Option 1: Load file from local drive."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Akk5eoqFv2Rn"
      },
      "source": [
        "Option 2: load file from Github\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a-8G5Q5lv9bY"
      },
      "source": [
        "import pandas as pd\n",
        "import io\n",
        "import requests\n",
        "from time import time\n",
        "from google.colab import files\n",
        "url='https://raw.githubusercontent.com/asu-trans-ai-lab/Integrated_modeling_GMNS/main/examples/university_campus/Arizona_State_University/map.osm'\n",
        "s=requests.get(url).content\n"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pfZ39iSliHaz"
      },
      "source": [
        "#### Step 1. Convert OSM data to GMNS\n",
        "one can display the network in NeXTA or QGIS.\n",
        "\n",
        "---\n",
        "| Step |    Description    |       Software       | Input Files | Output Files |\n",
        "|:----:|:-----------------:|:--------------------:|:-----------:|:------------:|\n",
        "| 0    | OSM data download | Open Street Map(OSM) | - -         | map.osm      |"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U3p_8G9VgxKc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 443
        },
        "outputId": "ddc65f9d-5b4a-441b-b2ef-29e5c4e1cb59"
      },
      "source": [
        "    net = og.getNetFromFile(filename=os.path.join(map_folder, 'map.osm'),\n",
        "                            network_types=('auto','bike','walk'),\n",
        "                            default_lanes=True, default_speed=True, default_capacity=True)\n",
        "\n",
        "    og.consolidateComplexIntersections(net, auto_identify=True)\n",
        "    og.generateNodeActivityInfo(net)\n",
        "    og.buildMultiResolutionNets(net)\n",
        "\n",
        "    og.outputNetToCSV(net, output_folder=map_folder)\n"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "arguments used for network parsing:\n",
            "  filename: map_folder/map.osm\n",
            "  network_types: ('auto', 'bike', 'walk')\n",
            "  link_types: all\n",
            "  POI: False\n",
            "  POI_sampling_ratio: 1.0\n",
            "  strict_mode: True\n",
            "  offset: no\n",
            "  min_nodes: 1\n",
            "  combine: False\n",
            "  bbox: None\n",
            "  default_lanes: True\n",
            "  default_speed: True\n",
            "  default_capacity: True\n",
            "\n",
            "Building Network from OSM file\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "SystemExit",
          "evalue": "ignored",
          "traceback": [
            "An exception has occurred, use %tb to see the full traceback.\n",
            "\u001b[0;31mSystemExit\u001b[0m\u001b[0;31m:\u001b[0m ERROR: file map_folder/map.osm does not exist\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py:2890: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.\n",
            "  warn(\"To exit: use 'exit', 'quit', or Ctrl-D.\", stacklevel=1)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2oLdpnhsikA6"
      },
      "source": [
        "#### Step 2. Please check content folder on the left side.  Convert GTFS data to GMNS"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6t5irvEgit9P"
      },
      "source": [
        "#### Step 3. Expand macroscopic network data to micro, meso\n",
        "one can display the networks in NeXTA or QGIS.\n",
        "\n",
        "---\n",
        "| Step |                   Description                  | Software |     Input Files    |                   Output Files                   |\n",
        "|:----:|:----------------------------------------------:|:--------:|:------------------:|:------------------------------------------------:|\n",
        "| 3    | Expand macroscopic network data to micro, meso | net2cell | Node.csv, Link.csv | Meso and micro networks in node.csv and link.csv |"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zVYpmnDo1-Tz",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 404
        },
        "outputId": "d5802b91-243b-454b-ecb9-e461beac1660"
      },
      "source": [
        "import path4gmns as pg\n",
        "\n",
        "%cd /content/map_folder/\n",
        "\n",
        "load_demand = False\n",
        "network = pg.read_network(load_demand)\n",
        "\n",
        "print('\\nshortest path (node id) from node 1 to node 2 is '\n",
        "      +network.find_shortest_path(1, 2))\n",
        "print('\\nshortest path (link id) from node 1 to node 2 is '\n",
        "      +network.find_shortest_path(1, 2, 'link'))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/map_folder\n",
            "read node.csv\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "error",
          "ename": "ValueError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-34-c5fbb63470b3>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mload_demand\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mnetwork\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_network\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mload_demand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m print('\\nshortest path (node id) from node 1 to node 2 is '\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/path4gmns/util.py\u001b[0m in \u001b[0;36mread_network\u001b[0;34m(load_demand, input_dir)\u001b[0m\n\u001b[1;32m    454\u001b[0m                \u001b[0mnetwork\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minternal_node_seq_no_dict\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    455\u001b[0m                \u001b[0mnetwork\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexternal_node_id_dict\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 456\u001b[0;31m                network.zone_to_nodes_dict)\n\u001b[0m\u001b[1;32m    457\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    458\u001b[0m     read_links(input_dir,\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/path4gmns/util.py\u001b[0m in \u001b[0;36mread_nodes\u001b[0;34m(input_dir, nodes, id_to_no_dict, no_to_id_dict, zone_to_node_dict)\u001b[0m\n\u001b[1;32m     38\u001b[0m                 \u001b[0mzone_id\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     39\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 40\u001b[0;31m                 \u001b[0mzone_id\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mzone_id\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     41\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     42\u001b[0m             \u001b[0;31m# treat them as string\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mValueError\u001b[0m: invalid literal for int() with base 10: '1.0'"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u558xfb5gSH6"
      },
      "source": [
        "#### Step 5. Traffic signal for timing\n",
        "\n",
        "---\n",
        "| Step |        Description        |                 Software                |            Input Files           | Output Files |\n",
        "|:----:|:-------------------------:|:---------------------------------------:|:--------------------------------:|:------------:|\n",
        "| 5    | Traffic signal for timing | Signal2timing (In development), Sigma-X | Node.csv, Link.csv, Movement.csv | Timing.csv   |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s6yxc-KIiBRJ"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ME9WqvMDjZPw",
        "outputId": "51b8ded0-98e3-4ec5-fc1f-7e961e5e6f72"
      },
      "source": [
        "\n",
        "!zip -r /content/MRM.zip /content\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "updating: content/map_folder/ (stored 0%)\n",
            "updating: content/map_folder/accessibility.csv (deflated 81%)\n",
            "updating: content/map_folder/input_agent.csv (deflated 80%)\n",
            "updating: content/map_folder/poi_trip_rate.csv (deflated 77%)\n",
            "updating: content/map_folder/poi.csv (deflated 74%)\n",
            "updating: content/map_folder/log.txt (deflated 92%)\n",
            "updating: content/map_folder/demand.csv (deflated 74%)\n",
            "updating: content/map_folder/link.csv (deflated 81%)\n",
            "updating: content/map_folder/node.csv (deflated 76%)\n",
            "updating: content/map_folder/zone.csv (deflated 76%)\n",
            "  adding: content/ (stored 0%)\n",
            "  adding: content/.config/ (stored 0%)\n",
            "  adding: content/.config/active_config (stored 0%)\n",
            "  adding: content/.config/gce (stored 0%)\n",
            "  adding: content/.config/logs/ (stored 0%)\n",
            "  adding: content/.config/logs/2021.03.25/ (stored 0%)\n",
            "  adding: content/.config/logs/2021.03.25/13.38.09.835215.log (deflated 54%)\n",
            "  adding: content/.config/logs/2021.03.25/13.37.29.839683.log (deflated 91%)\n",
            "  adding: content/.config/logs/2021.03.25/13.38.04.266969.log (deflated 87%)\n",
            "  adding: content/.config/logs/2021.03.25/13.38.26.474054.log (deflated 54%)\n",
            "  adding: content/.config/logs/2021.03.25/13.37.48.611688.log (deflated 53%)\n",
            "  adding: content/.config/logs/2021.03.25/13.38.27.034051.log (deflated 53%)\n",
            "  adding: content/.config/.last_update_check.json (deflated 22%)\n",
            "  adding: content/.config/.last_opt_in_prompt.yaml (stored 0%)\n",
            "  adding: content/.config/config_sentinel (stored 0%)\n",
            "  adding: content/.config/.last_survey_prompt.yaml (stored 0%)\n",
            "  adding: content/.config/configurations/ (stored 0%)\n",
            "  adding: content/.config/configurations/config_default (deflated 15%)\n",
            "  adding: content/network.jpg (deflated 4%)\n",
            "  adding: content/map.osm (deflated 91%)\n",
            "  adding: content/map_folder.zip (stored 0%)\n",
            "  adding: content/connector.csv (deflated 88%)\n",
            "  adding: content/MRM_net_folder/ (stored 0%)\n",
            "  adding: content/MRM_net_folder/poi.csv (deflated 73%)\n",
            "  adding: content/MRM_net_folder/micronet/ (stored 0%)\n",
            "  adding: content/MRM_net_folder/micronet/link.csv (deflated 86%)\n",
            "  adding: content/MRM_net_folder/micronet/node.csv (deflated 83%)\n",
            "  adding: content/MRM_net_folder/macronet/ (stored 0%)\n",
            "  adding: content/MRM_net_folder/macronet/link.csv (deflated 78%)\n",
            "  adding: content/MRM_net_folder/macronet/node.csv (deflated 79%)\n",
            "  adding: content/MRM_net_folder/link.csv (deflated 81%)\n",
            "  adding: content/MRM_net_folder/mesonet/ (stored 0%)\n",
            "  adding: content/MRM_net_folder/mesonet/link.csv (deflated 81%)\n",
            "  adding: content/MRM_net_folder/mesonet/node.csv (deflated 79%)\n",
            "  adding: content/MRM_net_folder/node.csv (deflated 73%)\n",
            "  adding: content/map_folder/ (stored 0%)\n",
            "  adding: content/map_folder/accessibility.csv (deflated 81%)\n",
            "  adding: content/map_folder/input_agent.csv (deflated 80%)\n",
            "  adding: content/map_folder/poi_trip_rate.csv (deflated 77%)\n",
            "  adding: content/map_folder/poi.csv (deflated 74%)\n",
            "  adding: content/map_folder/log.txt (deflated 92%)\n",
            "  adding: content/map_folder/demand.csv (deflated 74%)\n",
            "  adding: content/map_folder/link.csv (deflated 81%)\n",
            "  adding: content/map_folder/node.csv (deflated 76%)\n",
            "  adding: content/map_folder/zone.csv (deflated 76%)\n",
            "  adding: content/sample_data/ (stored 0%)\n",
            "  adding: content/sample_data/anscombe.json (deflated 83%)\n",
            "  adding: content/sample_data/README.md (deflated 42%)\n",
            "  adding: content/sample_data/california_housing_train.csv (deflated 79%)\n",
            "  adding: content/sample_data/mnist_test.csv (deflated 88%)\n",
            "  adding: content/sample_data/mnist_train_small.csv (deflated 88%)\n",
            "  adding: content/sample_data/california_housing_test.csv (deflated 76%)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nKNJzwHCbbXe",
        "colab": {
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgZG8gewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwoKICAgICAgbGV0IHBlcmNlbnREb25lID0gZmlsZURhdGEuYnl0ZUxlbmd0aCA9PT0gMCA/CiAgICAgICAgICAxMDAgOgogICAgICAgICAgTWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCk7CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPSBgJHtwZXJjZW50RG9uZX0lIGRvbmVgOwoKICAgIH0gd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCk7CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK",
              "ok": true,
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "status": 200,
              "status_text": ""
            }
          },
          "base_uri": "https://localhost:8080/",
          "height": 416
        },
        "outputId": "1e8ac503-11f5-4cca-b8bd-a625ff2ff62f"
      },
      "source": [
        "from google.colab import files\n",
        "\n",
        "uploaded = files.upload()\n",
        "\n",
        "for fn in uploaded.keys():\n",
        "  print('User uploaded file \"{name}\" with length {length} bytes'.format(\n",
        "      name=fn, length=len(uploaded[fn])))"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-819ad8ec-8e36-41e5-aba5-055257d555a3\" name=\"files[]\" multiple disabled\n",
              "        style=\"border:none\" />\n",
              "     <output id=\"result-819ad8ec-8e36-41e5-aba5-055257d555a3\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script src=\"/nbextensions/google.colab/files.js\"></script> "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-3-292f82be1b7a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mgoogle\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcolab\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mfiles\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0muploaded\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfiles\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mfn\u001b[0m \u001b[0;32min\u001b[0m \u001b[0muploaded\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mkeys\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/google/colab/files.py\u001b[0m in \u001b[0;36mupload\u001b[0;34m()\u001b[0m\n\u001b[1;32m     61\u001b[0m   result = _output.eval_js(\n\u001b[1;32m     62\u001b[0m       'google.colab._files._uploadFiles(\"{input_id}\", \"{output_id}\")'.format(\n\u001b[0;32m---> 63\u001b[0;31m           input_id=input_id, output_id=output_id))\n\u001b[0m\u001b[1;32m     64\u001b[0m   \u001b[0mfiles\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_collections\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdefaultdict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_six\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbinary_type\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     65\u001b[0m   \u001b[0;31m# Mapping from original filename to filename as saved locally.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/google/colab/output/_js.py\u001b[0m in \u001b[0;36meval_js\u001b[0;34m(script, ignore_result, timeout_sec)\u001b[0m\n\u001b[1;32m     38\u001b[0m   \u001b[0;32mif\u001b[0m \u001b[0mignore_result\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     39\u001b[0m     \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 40\u001b[0;31m   \u001b[0;32mreturn\u001b[0m \u001b[0m_message\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_reply_from_input\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrequest_id\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimeout_sec\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     41\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     42\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/google/colab/_message.py\u001b[0m in \u001b[0;36mread_reply_from_input\u001b[0;34m(message_id, timeout_sec)\u001b[0m\n\u001b[1;32m     99\u001b[0m     \u001b[0mreply\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_read_next_input_message\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    100\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mreply\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0m_NOT_READY\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreply\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdict\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 101\u001b[0;31m       \u001b[0mtime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msleep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0.025\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    102\u001b[0m       \u001b[0;32mcontinue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    103\u001b[0m     if (reply.get('type') == 'colab_reply' and\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    }
  ]
}