notebooks/NC-population-density.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "edb7ec4e-4c4f-4e1e-b864-53d28d9060aa",
   "metadata": {},
   "source": [
    "# Neighborhood Council Population Density Calculation\n",
    "\n",
    "This is an initial exploration to build a population density measure for the NC's\n",
    "\n",
    "Steps in this notebook:\n",
    "\n",
    "1.  Setup\n",
    "2.  Create geodataframe/dataframe from cleaned data and [census](https://data.lacity.org/Community-Economic-Development/Census-Data-by-Neighborhood-Council/nwj3-ufba)\n",
    "3.  Examine the data\n",
    "4.  Compute the measure\n",
    "5.  Show measure as choropleth\n",
    "6.  So what (next steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26d1bc47-3b56-4780-8645-728a1a6c433a",
   "metadata": {},
   "source": [
    "# 1 - Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43418e3e-4d3a-4765-b242-1063f3266881",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#%run start.py\n",
    "\n",
    "import nc"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4da1933f-7f7c-41c7-b786-135e2e6094d3",
   "metadata": {},
   "source": [
    "# 2 - Get Data Files\n",
    "\n",
    "Using two data files:\n",
    "\n",
    "  - The Certified Neighborhoods cleaned up in NC-service-regions.ipynb\n",
    "  - [Census](https://data.lacity.org/Community-Economic-Development/Census-Data-by-Neighborhood-Council/nwj3-ufba) data from data.lacity.org\n",
    "  \n",
    "**Note** - This is 2010 Census data.  Probably need to look for 2020."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98eca07c-90f1-49dc-b195-4d7e28f76849",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#neighborhoods_gdf = gpd.read_file('../data/neighborhoods/geo_export_6da7921e-60c3-4768-ba2c-486e681d16fe.shp')\n",
    "neighborhoods_gdf = gpd.read_file('../data/neighborhoods/Neighborhood_Councils_(Certified)_cleaned.shp')\n",
    "\n",
    "neighborhoods_gdf.rename(columns={'NAME': 'name',\n",
    "                        'NC_ID': 'nc_id',\n",
    "                        'SERVICE_RE': 'service_region'},\n",
    "              inplace=True);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8bb7a28f-3ec1-4e2f-9f73-aabb61b54ca3",
   "metadata": {},
   "outputs": [],
   "source": [
    "neighborhood_census_df = pd.read_csv('../data/neighborhoods/Census_Data_by_Neighborhood_Council.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e90a651-2923-4109-8418-0292251c81e6",
   "metadata": {},
   "source": [
    "# 3 - Quick Looksie"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14627c48-3722-45d6-99d5-ccfd7152d68e",
   "metadata": {},
   "outputs": [],
   "source": [
    "info_gdf = Output(layout={'border': '1px solid black',\n",
    "                            'width': '50%'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02d66572-c6f5-4109-aae3-dc1443f5850d",
   "metadata": {},
   "outputs": [],
   "source": [
    "info_df = Output(layout={'border': '1px solid black',\n",
    "                            'width': '50%'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "151823ae-97de-423f-82e9-3221b7f7b3ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "with info_gdf:\n",
    "    display(HTML('<center><b>gdf info()</b></center>'))\n",
    "    display(neighborhoods_gdf.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6151852-3d00-43b0-a5c9-45cb26998446",
   "metadata": {},
   "outputs": [],
   "source": [
    "with info_df:\n",
    "    display(HTML('<center><b>df info()</b></center>'))\n",
    "    display(neighborhood_census_df.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3049c949-6f40-4c3e-8138-00cf8fccaece",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "HBox([info_gdf, info_df])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96ac15bd-33e6-4813-b27c-18962f5a90cf",
   "metadata": {},
   "source": [
    "Well, there's a discrepancy here.  The census data has 97 NC's and the certified dataset has 99 (I think the right number is 99).\n",
    "\n",
    "Not going to agonize over this at this stage but want to understand things.  Adjusting for what matches as this stage should be good enough for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca1d2a45-74fd-438e-8eaf-48c0e3f62b1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "names_set = set(neighborhoods_gdf.name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a52cf5cf-c676-436c-9903-d056499808a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "NC_Names_set = set(neighborhood_census_df.NC_Name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b67e778-8287-4b70-92ed-d50202c658ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "names_set.difference(NC_Names_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b00e46de-fbac-49a8-9c29-eb006bf885bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "in_both = names_set.intersection(NC_Names_set)\n",
    "in_both;  #delete the ; if you want to see names in both"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17961fb1-5b0d-4921-ba46-6fdfe9ca5d97",
   "metadata": {},
   "source": [
    "So this will build a new gdf where neighborhoods are in both.  neighborhoods_gdf is still the baseline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abdb1656-6791-4e37-952f-c0c96fae5647",
   "metadata": {},
   "outputs": [],
   "source": [
    "neighborhoods_in_both = neighborhoods_gdf.query(f\"name in @in_both\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11683d88-2d81-4ca2-bb21-8324c46ea2d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(neighborhoods_in_both)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "125bbe83-11ba-4c74-8b15-f292bf455628",
   "metadata": {},
   "outputs": [],
   "source": [
    "neighborhoods_in_both.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9533ea6-2f93-470c-a91c-dfdd21c0b307",
   "metadata": {},
   "outputs": [],
   "source": [
    "neighborhood_census_df.rename(columns={'NC_Name': 'name'}, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30cc3526-89e1-4e82-b7f8-4373d3d2a2e3",
   "metadata": {},
   "source": [
    "Use standard pd.merge to create the gdf for the choropleth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e5c4845-43d5-43f9-a0f6-644a0ab9785e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "neighborhood_merged = pd.merge(neighborhoods_in_both, neighborhood_census_df, how=\"left\", on=[\"name\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f86ec2b3-0b31-4062-9801-a41e4ca362c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(neighborhood_merged)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15216aee-58c2-4a11-a8ab-403dd112c9d1",
   "metadata": {},
   "source": [
    "# 4 - Compute the Measure\n",
    "\n",
    "Computation is simple.  Use the geometry of the NC to compute area in miles squared.\n",
    "\n",
    "For the density I'm simply using total population.  I suspect it would be interesting to examine some of the other ethnic measures?  Maybe a nice pull down to select?  Ah... for another day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "377b676e-c953-43be-b11d-e9df11c42264",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from pyproj import Geod\n",
    "\n",
    "geod = Geod(ellps=\"WGS84\")\n",
    "\n",
    "def square_miles(geo):\n",
    "    square_meters = abs(geod.geometry_area_perimeter(geo)[0])\n",
    "    return (square_meters * 10.764) / 27878000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afb0095d-4efc-4f5e-b42d-b02658d897f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "neighborhood_merged['sq_miles'] = neighborhood_merged.apply(lambda row: square_miles(row.geometry), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed44fba2-d344-47e1-ae5b-d0344456981e",
   "metadata": {},
   "outputs": [],
   "source": [
    "neighborhood_merged['density'] = neighborhood_merged.apply(lambda row: row['Total Population'] / row['sq_miles'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65c50231-8eff-433c-98b4-9ca0a5e8e442",
   "metadata": {},
   "source": [
    "Remember I like to look at one of the values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef2ed982-1813-4855-9945-5f07124e0f45",
   "metadata": {},
   "outputs": [],
   "source": [
    "neighborhood_merged.iloc[27]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89a9d914-101d-475e-b256-7673c4282d94",
   "metadata": {},
   "source": [
    "Some sanity checking on the data before we generate the display.\n",
    "\n",
    "In the real world we'll have to do some more work on this data!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2828b068-03d4-4b2c-bcb8-f615e77bc956",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "neighborhood_merged.density.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66fd6e29-f5b6-4c73-9d6e-e9020b0e0f92",
   "metadata": {},
   "outputs": [],
   "source": [
    "neighborhood_merged.density.min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7661e42f-52ba-4906-915c-9ade965ddcb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(neighborhood_merged)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14a9cb11-4482-4c16-80c4-2e850c906b74",
   "metadata": {},
   "source": [
    "# 5 - Display the Choropleth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b13a342-6cd0-4b76-a003-841e566d95fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "imagery = basemap_to_tiles(basemaps.Esri.WorldImagery)\n",
    "imagery.base = True\n",
    "osm = basemap_to_tiles(basemaps.OpenStreetMap.Mapnik)\n",
    "osm.base = True\n",
    "\n",
    "\n",
    "map_display = Map(center=(34.05, -118.25), zoom=12,\n",
    "                  layers=[imagery, osm],\n",
    "                  layout=Layout(height=\"900px\"),\n",
    "                  scroll_wheel_zoom=True)\n",
    "\n",
    "#map_display.add_control(LayersControl())\n",
    "#map_display += nc_layer\n",
    "map_display"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb18dee2-88b0-453a-9e7e-094ffcf822fc",
   "metadata": {},
   "source": [
    "refer to : https://www.youtube.com/watch?v=wjzAy_yLrdA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "763c8f08-5faa-4ae6-ab3d-d0f309950a25",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipyleaflet import Choropleth, Map\n",
    "from branca.colormap import linear\n",
    "a_geojson = json.loads(neighborhood_merged.to_json())\n",
    "\n",
    "population_density = dict(zip(neighborhood_merged['name'].tolist(), neighborhood_merged['density'].tolist()))\n",
    "for i in a_geojson['features']:\n",
    "    i['id'] = i['properties']['name']\n",
    "\n",
    "layer = Choropleth(\n",
    "                    geo_data=a_geojson,\n",
    "                    choro_data=population_density,\n",
    "                    colormap=linear.YlOrRd_09, #linear.Blues_05,\n",
    "                    style={'fillOpacity': 1.0, \"color\":\"black\"},)\n",
    "                    #key_on=\"name\")\n",
    "\n",
    "map_display.add_layer(layer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4d0e1d6-daf2-4a04-80fa-ab3d7246bca7",
   "metadata": {},
   "source": [
    "I need to revisit a tooltip type popup.  For now this will work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e8dc5f4-fcb7-4720-abbd-bb079830bca7",
   "metadata": {},
   "outputs": [],
   "source": [
    "geo_json = GeoJSON(\n",
    "    data=a_geojson,\n",
    "    style={\n",
    "        'opacity': 1, 'dashArray': '9', 'fillOpacity': 0.6, 'weight': 1\n",
    "    },\n",
    "    hover_style={\n",
    "        'color': 'white', 'dashArray': '0', 'fillOpacity': 0.5\n",
    "    },\n",
    "    name='NCs'\n",
    ")\n",
    "\n",
    "html = HTML('''Hover over a district''')\n",
    "html.layout.margin = '0px 20px 20px 20 px'\n",
    "control = WidgetControl(widget=html, position='bottomright')\n",
    "\n",
    "def update_html(feature, **kwargs):\n",
    "    html.value = '''<h3><b>NC: {}</b></h3>\n",
    "                    <h4>Area (sq miles): {}</h4>\n",
    "                    <h4>Total Pop: {}</h4>\n",
    "                    <h4>Density: {}</h4>'''.format(feature['properties']['name'],\n",
    "                                                           feature['properties']['sq_miles'],\n",
    "                                                           feature['properties']['Total Population'],\n",
    "                                                           feature['properties']['density'])\n",
    "    \n",
    "map_display.add_control(control)  # does += work for this?\n",
    "\n",
    "layer.on_hover(update_html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "602ca48f-f526-4884-b9b3-67ea5c68cfcd",
   "metadata": {},
   "source": [
    "Go back and look at the map."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1456dbbd-0a78-4974-8713-eac557c40936",
   "metadata": {},
   "source": [
    "# 6 - So What?\n",
    "\n",
    "I say this tounge in cheeck.  Things to think about:\n",
    "\n",
    "  1. Should we examine measures besides total population?\n",
    "  2. Does it make sense to extend the 311 data as we did with the service regions?\n",
    "  3. Do we just use this to select an NC then query 311 (or ...)?\n",
    "  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "066e728e-fa09-418b-9bf1-332fa5631a23",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}