districtr_convert_json_to_partition.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "districtr_convert_json_to_partition.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/lieuzhenghong/districtr-eda/blob/master/districtr_convert_json_to_partition.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6mp7eoyTBomo",
        "colab_type": "text"
      },
      "source": [
        "# Converting Districtr assignments (JSON) to GerryChain partitions\n",
        "\n",
        "## Motivation\n",
        "\n",
        "to be able to do graphical analysis on the plans drawn by Districtr akin to that we can do with Gerrychain.\n",
        "\n",
        "## MVP\n",
        "\n",
        "a Python script that takes in shapefile + district assignments (in JSON) and returns several metrics like contiguity and number of cut edges.\n",
        "\n",
        "## Rough steps\n",
        "\n",
        "For each state:\n",
        "1. Import the shapefile\n",
        "2. Using the shapefile, create the dual graph of a state\n",
        "\n",
        "For each proposed districting plan:\n",
        "3. Import the district assignments JSON\n",
        "4. Using the district assignments JSON, form the graph partition (possibly with\n",
        "   Gerrychain, NetworkX, something else)\n",
        "5. Using the graph partition, answer queries like number of cut edges and\n",
        "   contiguity. (edited) "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yvG2NUk5Tata",
        "colab_type": "text"
      },
      "source": [
        "## First attempt at the problem\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XCd4KbZkK1yh",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 677
        },
        "outputId": "4d1b62be-025d-42ca-d4d2-4618658ac1e0"
      },
      "source": [
        "!pip install geopandas\n",
        "!pip install networkx\n",
        "!pip install gerrychain"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: geopandas in /usr/local/lib/python3.6/dist-packages (0.8.0)\n",
            "Requirement already satisfied: pyproj>=2.2.0 in /usr/local/lib/python3.6/dist-packages (from geopandas) (2.6.1.post1)\n",
            "Requirement already satisfied: fiona in /usr/local/lib/python3.6/dist-packages (from geopandas) (1.8.13.post1)\n",
            "Requirement already satisfied: pandas>=0.23.0 in /usr/local/lib/python3.6/dist-packages (from geopandas) (1.0.5)\n",
            "Requirement already satisfied: shapely in /usr/local/lib/python3.6/dist-packages (from geopandas) (1.7.0)\n",
            "Requirement already satisfied: six>=1.7 in /usr/local/lib/python3.6/dist-packages (from fiona->geopandas) (1.12.0)\n",
            "Requirement already satisfied: click<8,>=4.0 in /usr/local/lib/python3.6/dist-packages (from fiona->geopandas) (7.1.2)\n",
            "Requirement already satisfied: click-plugins>=1.0 in /usr/local/lib/python3.6/dist-packages (from fiona->geopandas) (1.1.1)\n",
            "Requirement already satisfied: attrs>=17 in /usr/local/lib/python3.6/dist-packages (from fiona->geopandas) (19.3.0)\n",
            "Requirement already satisfied: munch in /usr/local/lib/python3.6/dist-packages (from fiona->geopandas) (2.5.0)\n",
            "Requirement already satisfied: cligj>=0.5 in /usr/local/lib/python3.6/dist-packages (from fiona->geopandas) (0.5.0)\n",
            "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.23.0->geopandas) (2.8.1)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.23.0->geopandas) (2018.9)\n",
            "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.23.0->geopandas) (1.18.5)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.6/dist-packages (2.4)\n",
            "Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx) (4.4.2)\n",
            "Requirement already satisfied: gerrychain in /usr/local/lib/python3.6/dist-packages (0.2.12)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.6/dist-packages (from gerrychain) (2.4)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from gerrychain) (1.4.1)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from gerrychain) (1.0.5)\n",
            "Requirement already satisfied: shapely in /usr/local/lib/python3.6/dist-packages (from gerrychain) (1.7.0)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from gerrychain) (3.2.2)\n",
            "Requirement already satisfied: geopandas in /usr/local/lib/python3.6/dist-packages (from gerrychain) (0.8.0)\n",
            "Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx->gerrychain) (4.4.2)\n",
            "Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from scipy->gerrychain) (1.18.5)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->gerrychain) (2018.9)\n",
            "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas->gerrychain) (2.8.1)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->gerrychain) (1.2.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->gerrychain) (0.10.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->gerrychain) (2.4.7)\n",
            "Requirement already satisfied: pyproj>=2.2.0 in /usr/local/lib/python3.6/dist-packages (from geopandas->gerrychain) (2.6.1.post1)\n",
            "Requirement already satisfied: fiona in /usr/local/lib/python3.6/dist-packages (from geopandas->gerrychain) (1.8.13.post1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.6.1->pandas->gerrychain) (1.12.0)\n",
            "Requirement already satisfied: cligj>=0.5 in /usr/local/lib/python3.6/dist-packages (from fiona->geopandas->gerrychain) (0.5.0)\n",
            "Requirement already satisfied: attrs>=17 in /usr/local/lib/python3.6/dist-packages (from fiona->geopandas->gerrychain) (19.3.0)\n",
            "Requirement already satisfied: click-plugins>=1.0 in /usr/local/lib/python3.6/dist-packages (from fiona->geopandas->gerrychain) (1.1.1)\n",
            "Requirement already satisfied: click<8,>=4.0 in /usr/local/lib/python3.6/dist-packages (from fiona->geopandas->gerrychain) (7.1.2)\n",
            "Requirement already satisfied: munch in /usr/local/lib/python3.6/dist-packages (from fiona->geopandas->gerrychain) (2.5.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pMsb4avX-qWN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import sys\n",
        "import json"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6IjBVRypBl0L",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aELXbNjsDWEd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import geopandas as gpd"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tQ_z612RDv8D",
        "colab_type": "text"
      },
      "source": [
        "Use geopandas to read in the shapefile"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ptjOaiuNEzFn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "STATE_SHAPEFILE_PATH = \"./sample_data/IA_counties.shp\"\n",
        "state_gdf = gpd.read_file(STATE_SHAPEFILE_PATH)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_NQv5suaDut-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(state_shapefile)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pBsrDJQVFXAT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import networkx as nx"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SpDVdNZlFY2C",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jOo3594BFcqg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "f, ax = plt.subplots(figsize=(10, 10))\n",
        "state_gdf.plot(ax=ax)\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jDNjX15SHOHl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import gerrychain"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XsWB7tVlJZiP",
        "colab_type": "text"
      },
      "source": [
        "We first import the Districtr district assignments."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wKiZYMd8KwUj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "JSON_PATH = './sample_data/incomplete-islands.json'\n",
        "with open(f'{JSON_PATH}', 'r') as f:\n",
        "  json_assignment = json.load(f)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NqQmztxZLez5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "assignment = {int(k):v for k,v in json_assignment['assignment'].items()}\n",
        "print(len(assignment))\n",
        "print(assignment)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rSR5qBgFMtlq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "state_gdf['assignment'] = state_gdf.GEOID10.map(dict(assignment))\n",
        "state_gdf['assignment'] = state_gdf['assignment'].fillna(-1)\n",
        "print(state_gdf)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YWOz-TJ9OPXi",
        "colab_type": "text"
      },
      "source": [
        "We have the Districtr assignments and the GeoDataFrame. We now create a Graph using the GeoDataFrame, and assign each node to a district."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0NyWASbkOXyi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# There's a bug with Gerrychain here. Issue 328.\n",
        "state_graph = gerrychain.Graph.from_geodataframe(state_gdf) # will worry about the CRS later.\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xmZwo3xr9H73",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Workaround for issue 328: add columns manually\n",
        "state_graph.add_data(state_gdf, columns=['GEOID10', 'assignment'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UhApZDSL6cLI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "## This bit visualises the dual graph. GerryChain has an inbuilt plot function,\n",
        "## but I wrote my own in NetworkX\n",
        "\n",
        "def visualise_partition(state_graph):\n",
        "  node_assignments = (nx.get_node_attributes(state_graph, 'assignment'))\n",
        "  districts = set(node_assignments.values())\n",
        "  print(districts)\n",
        "\n",
        "  # this is kind of slow because it loops through all the assignments many times\n",
        "  # but we'll worry about this later\n",
        "  district_assignments = [\n",
        "                          [k for k, v in node_assignments.items() if v==district]\n",
        "                          for district in districts\n",
        "                        ]\n",
        "\n",
        "  print(district_assignments)\n",
        "\n",
        "  pos = nx.spectral_layout(state_graph)\n",
        "\n",
        "  import pylab\n",
        "  NUM_COLORS = len(districts)\n",
        "\n",
        "  cm = pylab.get_cmap('gist_rainbow')\n",
        "  cmap = [[cm(1.*i/NUM_COLORS)] for i in range(0, NUM_COLORS)]\n",
        "\n",
        "  print(cmap)\n",
        "\n",
        "  for idx, assignment in enumerate(district_assignments):\n",
        "    nx.draw_networkx_nodes(state_graph, pos, \n",
        "                          nodelist=assignment,\n",
        "                          node_color=cmap[idx],\n",
        "                          node_size=100\n",
        "                          )\n",
        "\n",
        "  nx.draw_networkx_edges(state_graph, pos, width=1.0, alpha=0.8)\n",
        "  nx.draw_networkx_labels(state_graph, pos, font_size=10)\n",
        "\n",
        "  plt.axis('off')\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o3JQr9AWNvqI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "visualise_partition(state_graph)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "724T-E9dQN_m",
        "colab_type": "text"
      },
      "source": [
        "## Final cleaned up version\n",
        "\n",
        "GerryChain has a `from_districtr_file` functions which is literally one line of code. This simplifies the problem greatly.\n",
        "\n",
        "It also has a built-in partition visualisation function, so I didn't need to write the `visualise_partition` function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cHCd7Sj7NKTg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 124
        },
        "outputId": "261cf91a-8196-4f04-e5f7-799398dac711"
      },
      "source": [
        "import sys\n",
        "import json\n",
        "import geopandas as gpd\n",
        "import gerrychain\n",
        "import networkx as nx\n",
        "import matplotlib.pyplot as plt\n",
        "STATE_SHAPEFILE_PATH = \"./sample_data/IA_counties.shp\"\n",
        "JSON_PATH = './sample_data/incomplete-contiguous.json'\n",
        "\n",
        "# Possible problems with the CRS here.\n",
        "state_graph = gerrychain.Graph.from_file(STATE_SHAPEFILE_PATH)\n",
        "state_graph.to_json('./sample_data/output_data/iowa_dual_graph')\n",
        "\n",
        "# Form the partition with the JSON path\n",
        "partition = gerrychain.Partition.from_districtr_file(state_graph, JSON_PATH, \n",
        "                                                     updaters=None)\n",
        "\n",
        "# Visualise the partition\n",
        "#state_gdf = gpd.read_file(STATE_SHAPEFILE_PATH)\n",
        "#partition.plot(geometries=state_gdf)\n",
        "\n",
        "# Now check for cut edges and for contiguity\n",
        "print(partition['cut_edges'])\n",
        "print(gerrychain.constraints.contiguity.contiguous(partition))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{(44, 66), (68, 71), (24, 92), (33, 45), (24, 59), (85, 92), (4, 27), (36, 48), (2, 12), (44, 48), (75, 88), (29, 80), (8, 24), (53, 59), (14, 18), (13, 69), (28, 35), (9, 87), (46, 97), (62, 74), (88, 98), (75, 82), (76, 92), (53, 89), (7, 58), (67, 97), (16, 92), (53, 62), (46, 58), (35, 93), (25, 34), (4, 62), (75, 81), (13, 56), (12, 19), (46, 77), (19, 56), (42, 90)}\n",
            "True\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/gerrychain/graph/graph.py:152: UserWarning: Geometry is in a geographic CRS. Results from 'area' are likely incorrect. Use 'GeoSeries.to_crs()' to re-project geometries to a projected CRS before this operation.\n",
            "\n",
            "  areas = df.geometry.area.to_dict()\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i6_d3EXqYPiq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 503
        },
        "outputId": "8031e196-e717-44f0-9e84-a2374967977c"
      },
      "source": [
        "STATE_SHAPEFILE_PATH = \"./sample_data/tx_shp/TX_vtds.shp\"\n",
        "\n",
        "state_gdf = gpd.read_file(STATE_SHAPEFILE_PATH)\n",
        "print(state_gdf)\n",
        "state_gdf['geometry'] = state_gdf.buffer(0)\n",
        "print(state_gdf)\n",
        "\n",
        "state_graph = gerrychain.Graph.from_geodataframe(state_gdf)\n",
        "#state_graph = gerrychain.Graph.from_file(STATE_SHAPEFILE_PATH)\n",
        "state_graph.to_json('./sample_data/output_data/texas_dual_graph')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "     CNTYVTD   VTD  ...  PERIM                                           geometry\n",
            "0      10001  0001  ...     15  POLYGON ((1413960.808 1073012.816, 1413971.571...\n",
            "1      10002  0002  ...     95  POLYGON ((1420165.429 1066385.798, 1420251.968...\n",
            "2      10003  0003  ...     55  POLYGON ((1416275.023 1072178.732, 1416410.201...\n",
            "3      10004  0004  ...     92  POLYGON ((1435604.819 1074650.256, 1435674.876...\n",
            "4      10005  0005  ...     87  POLYGON ((1436888.342 1072498.734, 1436911.364...\n",
            "...      ...   ...  ...    ...                                                ...\n",
            "8936  990412  0412  ...     18  POLYGON ((1199509.662 997181.911, 1199489.562 ...\n",
            "8937  990413  0413  ...    145  POLYGON ((1208431.172 1051581.578, 1208442.156...\n",
            "8938  990414  0414  ...     64  POLYGON ((1224967.807 1051996.567, 1225381.276...\n",
            "8939  990415  0415  ...     54  POLYGON ((1208533.058 1050659.635, 1208533.028...\n",
            "8940  990416  0416  ...     59  POLYGON ((1201864.290 1014975.205, 1201927.399...\n",
            "\n",
            "[8941 rows x 38 columns]\n",
            "     CNTYVTD   VTD  ...  PERIM                                           geometry\n",
            "0      10001  0001  ...     15  POLYGON ((1413960.808 1073012.816, 1413971.571...\n",
            "1      10002  0002  ...     95  POLYGON ((1420165.429 1066385.798, 1420251.968...\n",
            "2      10003  0003  ...     55  POLYGON ((1416275.023 1072178.732, 1416410.201...\n",
            "3      10004  0004  ...     92  POLYGON ((1435604.819 1074650.256, 1435674.876...\n",
            "4      10005  0005  ...     87  POLYGON ((1436888.342 1072498.734, 1436911.364...\n",
            "...      ...   ...  ...    ...                                                ...\n",
            "8936  990412  0412  ...     18  POLYGON ((1199509.662 997181.911, 1199489.562 ...\n",
            "8937  990413  0413  ...    145  POLYGON ((1208431.172 1051581.578, 1208442.156...\n",
            "8938  990414  0414  ...     64  POLYGON ((1224967.807 1051996.567, 1225381.276...\n",
            "8939  990415  0415  ...     54  POLYGON ((1208533.058 1050659.635, 1208533.028...\n",
            "8940  990416  0416  ...     59  POLYGON ((1201864.290 1014975.205, 1201927.399...\n",
            "\n",
            "[8941 rows x 38 columns]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9vIWIaRam4dv",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "079b20a0-bde8-470b-c3cd-625d2c62df5d"
      },
      "source": [
        "gerrychain.Graph.from_json('./sample_data/output_data/texas_dual_graph')\n",
        "gerrychain.Graph.from_json('./sample_data/output_data/iowa_dual_graph')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<Graph [99 nodes, 222 edges]>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UeGMdVUoSCB8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}