Segmenting and Clustering Neighborhoods in Toronto

import numpy as np
import pandas as pd # library for data analsysis
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

import json # library to handle JSON files

!conda install -c conda-forge lxml --yes  #helps in scraping the web for data


import requests # library to handle requests
from pandas.io.json import json_normalize # tranform JSON file into a pandas dataframe

# Matplotlib and associated plotting modules
import matplotlib.cm as cm
import matplotlib.colors as colors

# import k-means from clustering stage
from sklearn.cluster import KMeans

!conda install -c conda-forge folium=0.5.0 --yes 
import folium # map rendering library

print('Libraries imported.')

Solving environment: done

## Package Plan ##

  environment location: /opt/conda/envs/Python36

  added / updated specs: 
    - lxml


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    certifi-2019.9.11          |           py36_0         147 KB  conda-forge
    ca-certificates-2019.9.11  |       hecc5488_0         144 KB  conda-forge
    lxml-4.4.1                 |   py36h7ec2d77_0         1.6 MB  conda-forge
    openssl-1.1.1d             |       h516909a_0         2.1 MB  conda-forge
    ------------------------------------------------------------
                                           Total:         3.9 MB

The following packages will be UPDATED:

    certifi:         2019.9.11-py36_0     --> 2019.9.11-py36_0     conda-forge
    lxml:            4.3.1-py36hefd8a0e_0 --> 4.4.1-py36h7ec2d77_0 conda-forge

The following packages will be DOWNGRADED:

    ca-certificates: 2019.10.16-0         --> 2019.9.11-hecc5488_0 conda-forge
    openssl:         1.1.1d-h7b6447c_3    --> 1.1.1d-h516909a_0    conda-forge


Downloading and Extracting Packages
certifi-2019.9.11    | 147 KB    | ##################################### | 100% 
ca-certificates-2019 | 144 KB    | ##################################### | 100% 
lxml-4.4.1           | 1.6 MB    | ##################################### | 100% 
openssl-1.1.1d       | 2.1 MB    | ##################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Solving environment: done

## Package Plan ##

  environment location: /opt/conda/envs/Python36

  added / updated specs: 
    - folium=0.5.0


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    altair-3.2.0               |           py36_0         770 KB  conda-forge
    branca-0.3.1               |             py_0          25 KB  conda-forge
    vincent-0.4.4              |             py_1          28 KB  conda-forge
    folium-0.5.0               |             py_0          45 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         868 KB

The following NEW packages will be INSTALLED:

    altair:  3.2.0-py36_0 conda-forge
    branca:  0.3.1-py_0   conda-forge
    folium:  0.5.0-py_0   conda-forge
    vincent: 0.4.4-py_1   conda-forge


Downloading and Extracting Packages
altair-3.2.0         | 770 KB    | ##################################### | 100% 
branca-0.3.1         | 25 KB     | ##################################### | 100% 
vincent-0.4.4        | 28 KB     | ##################################### | 100% 
folium-0.5.0         | 45 KB     | ##################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Libraries imported.

1. First part of the assignment

Extracting data from a website

We are going to download and clean the Toronto neighborhoods in cells below

df=pd.read_html("https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M")
df=df[0]  # Taking the dataframe with the required data
df=df[df.Borough!='Not assigned']    #ignoring the rows with the non-assigned rows
df.reset_index(inplace=True)
df.drop(columns='index', inplace=True)
df.Neighbourhood[df.Neighbourhood=='Not assigned']=df.Borough[df.Neighbourhood=='Not assigned']  #Naming the  non-assigned neighborhoods
                                                                                                #with the borough names

#Grouping neighborhoods with same postal codes and separting them with a comma
df_table=df.groupby(['Postcode','Borough'])['Neighbourhood'].apply(', '.join).reset_index()

df_table.columns=['PostalCode', 'Borough', 'Neighborhood']
df_table.head(20)

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	PostalCode	Borough	Neighborhood
0	M1B	Scarborough	Rouge, Malvern
1	M1C	Scarborough	Highland Creek, Rouge Hill, Port Union
2	M1E	Scarborough	Guildwood, Morningside, West Hill
3	M1G	Scarborough	Woburn
4	M1H	Scarborough	Cedarbrae
5	M1J	Scarborough	Scarborough Village
6	M1K	Scarborough	East Birchmount Park, Ionview, Kennedy Park
7	M1L	Scarborough	Clairlea, Golden Mile, Oakridge
8	M1M	Scarborough	Cliffcrest, Cliffside, Scarborough Village West
9	M1N	Scarborough	Birch Cliff, Cliffside West
10	M1P	Scarborough	Dorset Park, Scarborough Town Centre, Wexford ...
11	M1R	Scarborough	Maryvale, Wexford
12	M1S	Scarborough	Agincourt
13	M1T	Scarborough	Clarks Corners, Sullivan, Tam O'Shanter
14	M1V	Scarborough	Agincourt North, L'Amoreaux East, Milliken, St...
15	M1W	Scarborough	L'Amoreaux West
16	M1X	Scarborough	Upper Rouge
17	M2H	North York	Hillcrest Village
18	M2J	North York	Fairview, Henry Farm, Oriole
19	M2K	North York	Bayview Village

shape=df_table.shape
print("The number of rows in the above dataframe is",shape[0])

The number of rows in the above dataframe is 103

2. Second part of the assignment

Finding out the coordinates of the neighborhoods

We will download the coordinates values based on the Postal Codes and merge them with the above table in the cells below

coord=pd.read_csv("https://cocl.us/Geospatial_data")  #Extracting the data from the given website
coord.set_index('Postal Code', inplace=True)
coord.head()

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	Latitude	Longitude
Postal Code
M1B	43.806686	-79.194353
M1C	43.784535	-79.160497
M1E	43.763573	-79.188711
M1G	43.770992	-79.216917
M1H	43.773136	-79.239476

#Merging the coordinates data with the borough and neighborhood data

temp=coord.loc[df_table.PostalCode]
temp.reset_index(inplace=True)
temp.head()
df_table[['Latitude','Longitude']]=temp[['Latitude','Longitude']]
df_table.head(20)

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	PostalCode	Borough	Neighborhood	Latitude	Longitude
0	M1B	Scarborough	Rouge, Malvern	43.806686	-79.194353
1	M1C	Scarborough	Highland Creek, Rouge Hill, Port Union	43.784535	-79.160497
2	M1E	Scarborough	Guildwood, Morningside, West Hill	43.763573	-79.188711
3	M1G	Scarborough	Woburn	43.770992	-79.216917
4	M1H	Scarborough	Cedarbrae	43.773136	-79.239476
5	M1J	Scarborough	Scarborough Village	43.744734	-79.239476
6	M1K	Scarborough	East Birchmount Park, Ionview, Kennedy Park	43.727929	-79.262029
7	M1L	Scarborough	Clairlea, Golden Mile, Oakridge	43.711112	-79.284577
8	M1M	Scarborough	Cliffcrest, Cliffside, Scarborough Village West	43.716316	-79.239476
9	M1N	Scarborough	Birch Cliff, Cliffside West	43.692657	-79.264848
10	M1P	Scarborough	Dorset Park, Scarborough Town Centre, Wexford ...	43.757410	-79.273304
11	M1R	Scarborough	Maryvale, Wexford	43.750072	-79.295849
12	M1S	Scarborough	Agincourt	43.794200	-79.262029
13	M1T	Scarborough	Clarks Corners, Sullivan, Tam O'Shanter	43.781638	-79.304302
14	M1V	Scarborough	Agincourt North, L'Amoreaux East, Milliken, St...	43.815252	-79.284577
15	M1W	Scarborough	L'Amoreaux West	43.799525	-79.318389
16	M1X	Scarborough	Upper Rouge	43.836125	-79.205636
17	M2H	North York	Hillcrest Village	43.803762	-79.363452
18	M2J	North York	Fairview, Henry Farm, Oriole	43.778517	-79.346556
19	M2K	North York	Bayview Village	43.786947	-79.385975

3. Third part of the assignment

We will be exploring and clustering the neighborhoods whose boroughs' have the word "Toronto" in them

#Separating the above dataframe such that the borough's have the word "Toronto" in them

df_toronto=df_table[df_table['Borough'].str.find('Toronto')>0]
df_toronto.reset_index(inplace=True)
df_toronto=df_toronto.drop('index', axis=1)
df_toronto.head()

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	PostalCode	Borough	Neighborhood	Latitude	Longitude
0	M4E	East Toronto	The Beaches	43.676357	-79.293031
1	M4K	East Toronto	The Danforth West, Riverdale	43.679557	-79.352188
2	M4L	East Toronto	The Beaches West, India Bazaar	43.668999	-79.315572
3	M4M	East Toronto	Studio District	43.659526	-79.340923
4	M4N	Central Toronto	Lawrence Park	43.728020	-79.388790

We will display the above neighborhoods on a map in the cells below

#Obtaining the coordnitaes for centering the map

latitude=df_toronto['Latitude'].mean()
longitude=df_toronto['Longitude'].mean()
print('The central geographical coordinate for Boroughs which contain the  word toronto are {}, {}.'.format(latitude, longitude))

The central geographical coordinate for Boroughs which contain the  word toronto are 43.66726218421052, -79.38988323421053.

# create map of Toronto using latitude and longitude values
map_toronto = folium.Map(location=[latitude, longitude], zoom_start=12)

# add markers to map
for lat, lng, borough, neighborhood in zip(df_toronto['Latitude'], df_toronto['Longitude'], df_toronto['Borough'], df_toronto['Neighborhood']):
    label = borough+": "+neighborhood   #'{}, {}'.format(neighborhood, borough)
    label = folium.Popup(label, parse_html=True)
    folium.CircleMarker(
        [lat, lng],
        radius=5,
        popup=label,
        color='blue',
        fill=True,
        fill_color='#3186cc',
        fill_opacity=0.7,
        parse_html=False).add_to(map_toronto)  
    
map_toronto

CLIENT_ID = 'FROUNNKNE2NFNJ3DPMSVCFHY5322WYELMEGST2E22VVQCDCN' # your Foursquare ID
CLIENT_SECRET = 'Z30WRUBMICAZU4NBL4UOTJTKB3LWTDVTRAXJU3AFIX3ONRA0' # your Foursquare Secret
VERSION = '20180605' # Foursquare API version

print('Your credentails:')
print('CLIENT_SECRET:' + CLIENT_SECRET)
print('CLIENT_ID: ' + CLIENT_ID)

Your credentails:
CLIENT_SECRET:Z30WRUBMICAZU4NBL4UOTJTKB3LWTDVTRAXJU3AFIX3ONRA0
CLIENT_ID: FROUNNKNE2NFNJ3DPMSVCFHY5322WYELMEGST2E22VVQCDCN

We will explore the neighborhoods using the Foursquare API in the following cells

#creating a function to get the  near by venues of a given neighborhood using the Foursquare API

def getNearbyVenues(names, latitudes, longitudes, radius=500):
    
    venues_list=[]
    for name, lat, lng in zip(names, latitudes, longitudes):
        #print(name)
            
        # create the API request URL
        url = 'https://api.foursquare.com/v2/venues/explore?&client_id={}&client_secret={}&v={}&ll={},{}&radius={}&limit={}'.format(
            CLIENT_ID, 
            CLIENT_SECRET, 
            VERSION, 
            lat, 
            lng, 
            radius, 
            LIMIT)
            
        # make the GET request
        results = requests.get(url).json()["response"]['groups'][0]['items']
        
        # return only relevant information for each nearby venue
        venues_list.append([(
            name, 
            lat, 
            lng, 
            v['venue']['name'], 
            v['venue']['location']['lat'], 
            v['venue']['location']['lng'],  
            v['venue']['categories'][0]['name']) for v in results])

    nearby_venues = pd.DataFrame([item for venue_list in venues_list for item in venue_list])
    nearby_venues.columns = ['Neighborhood', 
                  'Neighborhood Latitude', 
                  'Neighborhood Longitude', 
                  'Venue', 
                  'Venue Latitude', 
                  'Venue Longitude', 
                  'Venue Category']
    
    return(nearby_venues)

#Obtaining venues within 500m of each neighborhood with a limit of 100 venues per neighborhood
LIMIT=100
toronto_venues = getNearbyVenues(names=df_toronto['Neighborhood'],
                                   latitudes=df_toronto['Latitude'],
                                   longitudes=df_toronto['Longitude']
                                  )
toronto_venues.head()

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	Neighborhood	Neighborhood Latitude	Neighborhood Longitude	Venue	Venue Latitude	Venue Longitude	Venue Category
0	The Beaches	43.676357	-79.293031	Glen Manor Ravine	43.676821	-79.293942	Trail
1	The Beaches	43.676357	-79.293031	The Big Carrot Natural Food Market	43.678879	-79.297734	Health Food Store
2	The Beaches	43.676357	-79.293031	Grover Pub and Grub	43.679181	-79.297215	Pub
3	The Beaches	43.676357	-79.293031	Upper Beaches	43.680563	-79.292869	Neighborhood
4	The Danforth West, Riverdale	43.679557	-79.352188	Pantheon	43.677621	-79.351434	Greek Restaurant

We will check how many venues were obtained per neighborhood below

toronto_venues.groupby('Neighborhood').count()

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	Neighborhood Latitude	Neighborhood Longitude	Venue	Venue Latitude	Venue Longitude	Venue Category
Neighborhood
Adelaide, King, Richmond	100	100	100	100	100	100
Berczy Park	57	57	57	57	57	57
Brockton, Exhibition Place, Parkdale Village	21	21	21	21	21	21
Business Reply Mail Processing Centre 969 Eastern	18	18	18	18	18	18
CN Tower, Bathurst Quay, Island airport, Harbourfront West, King and Spadina, Railway Lands, South Niagara	16	16	16	16	16	16
Cabbagetown, St. James Town	43	43	43	43	43	43
Central Bay Street	82	82	82	82	82	82
Chinatown, Grange Park, Kensington Market	96	96	96	96	96	96
Christie	17	17	17	17	17	17
Church and Wellesley	89	89	89	89	89	89
Commerce Court, Victoria Hotel	100	100	100	100	100	100
Davisville	38	38	38	38	38	38
Davisville North	7	7	7	7	7	7
Deer Park, Forest Hill SE, Rathnelly, South Hill, Summerhill West	16	16	16	16	16	16
Design Exchange, Toronto Dominion Centre	100	100	100	100	100	100
Dovercourt Village, Dufferin	18	18	18	18	18	18
First Canadian Place, Underground city	100	100	100	100	100	100
Forest Hill North, Forest Hill West	4	4	4	4	4	4
Harbord, University of Toronto	36	36	36	36	36	36
Harbourfront	48	48	48	48	48	48
Harbourfront East, Toronto Islands, Union Station	100	100	100	100	100	100
High Park, The Junction South	27	27	27	27	27	27
Lawrence Park	3	3	3	3	3	3
Little Portugal, Trinity	64	64	64	64	64	64
Moore Park, Summerhill East	4	4	4	4	4	4
North Toronto West	21	21	21	21	21	21
Parkdale, Roncesvalles	15	15	15	15	15	15
Rosedale	4	4	4	4	4	4
Roselawn	2	2	2	2	2	2
Runnymede, Swansea	34	34	34	34	34	34
Ryerson, Garden District	100	100	100	100	100	100
St. James Town	100	100	100	100	100	100
Stn A PO Boxes 25 The Esplanade	99	99	99	99	99	99
Studio District	38	38	38	38	38	38
The Annex, North Midtown, Yorkville	21	21	21	21	21	21
The Beaches	4	4	4	4	4	4
The Beaches West, India Bazaar	21	21	21	21	21	21
The Danforth West, Riverdale	42	42	42	42	42	42

#Checking how many unique venue categories were obtained
print('There are {} uniques categories.'.format(len(toronto_venues['Venue Category'].unique())))

There are 234 uniques categories.

We will analyse each neigborhood in the cells below

# one hot encoding
toronto_onehot = pd.get_dummies(toronto_venues[['Venue Category']], prefix="", prefix_sep="")

# add neighborhood column back to dataframe
toronto_onehot['Neighborhoods'] = toronto_venues['Neighborhood'] 

# move neighborhood column to the first column
fixed_columns = [toronto_onehot.columns[-1]] + list(toronto_onehot.columns[:-1])
toronto_onehot = toronto_onehot[fixed_columns]

toronto_onehot.head()

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	Neighborhoods	Greek Restaurant	Health Food Store	Neighborhood	Pub	Trail
0	The Beaches	0	0	0	0	1
1	The Beaches	0	1	0	0	0
2	The Beaches	0	0	0	1	0
3	The Beaches	0	0	1	0	0
4	The Danforth West, Riverdale	1	0	0	0	0

let's group rows by neighborhood and by taking the mean of the frequency of occurrence of each category

#Grouping the 
toronto_grouped = toronto_onehot.groupby('Neighborhoods').mean().reset_index()
toronto_grouped.head()

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	Neighborhoods	Airport	Airport Food Court	Airport Gate	Airport Lounge	Airport Service	Airport Terminal	American Restaurant	Art Gallery	Asian Restaurant	Auto Workshop	BBQ Joint	Bagel Shop	Bakery	Bar	Basketball Stadium	Beach	Beer Bar	Bistro	Boat or Ferry	Bookstore	Boutique	Brazilian Restaurant	Breakfast Spot	Brewery	Building	Burger Joint	Burrito Place	Butcher	Café	Caribbean Restaurant	Cheese Shop	Climbing Gym	Clothing Store	Cocktail Bar	Coffee Shop	Colombian Restaurant	Comfort Food Restaurant	Comic Shop	Concert Hall	Convenience Store	Cosmetics Shop	Creperie	Deli / Bodega	Department Store	Diner	Eastern European Restaurant	Electronics Store	Farmers Market	Fast Food Restaurant	Fish Market	Food Court	Fountain	French Restaurant	Furniture / Home Store	Garden	Garden Center	Gastropub	General Travel	Gluten-free Restaurant	Gourmet Shop	Greek Restaurant	Grocery Store	Gym	Gym / Fitness Center	Harbor / Marina	Hotel	Ice Cream Shop	Indian Restaurant	Intersection	Irish Pub	Italian Restaurant	Japanese Restaurant	Jazz Club	Juice Bar	Latin American Restaurant	Light Rail Station	Liquor Store	Lounge	Mediterranean Restaurant	Monument / Landmark	Museum	Neighborhood	New American Restaurant	Nightclub	Noodle House	Office	Opera House	Park	Performing Arts Venue	Pet Store	Pizza Place	Plane	Plaza	Poke Place	Ramen Restaurant	Record Shop	Recording Studio	Restaurant	Salad Place	Salon / Barbershop	Sculpture Garden	Seafood Restaurant	Shopping Mall	Skate Park	Smoke Shop	Spa	Speakeasy	Stadium	Steakhouse	Sushi Restaurant	Tailor Shop	Tea Room	Thai Restaurant	Theater	Vegetarian / Vegan Restaurant	Wine Bar
0	Adelaide, King, Richmond	0.0000	0.0000	0.0000	0.000	0.0000	0.0000	0.03	0.000000	0.03	0.000000	0.000000	0.000000	0.030000	0.040000	0.000000	0.000000	0.000000	0.000000	0.0000	0.01	0.0000	0.01	0.030000	0.000000	0.01	0.02	0.010000	0.000000	0.050000	0.000000	0.000000	0.000000	0.010000	0.000000	0.070000	0.01	0.000000	0.000000	0.020000	0.000000	0.03	0.000000	0.01	0.01	0.000000	0.000000	0.01	0.000000	0.000000	0.000000	0.01	0.000000	0.000000	0.000000	0.000000	0.000000	0.02	0.01	0.01	0.000000	0.010000	0.000000	0.020000	0.010000	0.0000	0.030000	0.01	0.01	0.000000	0.000000	0.010000	0.010000	0.010000	0.01	0.01	0.000000	0.000000	0.010000	0.01	0.01	0.000000	0.01	0.01	0.010000	0.01	0.01	0.01	0.000000	0.000000	0.000000	0.020000	0.0000	0.01	0.01	0.01	0.01	0.000000	0.030000	0.01	0.01	0.0000	0.010000	0.000000	0.000000	0.010000	0.000000	0.01	0.000000	0.040000	0.03	0.000000	0.000000	0.030000	0.01	0.020000	0.01
1	Berczy Park	0.0000	0.0000	0.0000	0.000	0.0000	0.0000	0.00	0.017544	0.00	0.000000	0.017544	0.017544	0.052632	0.000000	0.017544	0.017544	0.035088	0.017544	0.0000	0.00	0.0000	0.00	0.017544	0.000000	0.00	0.00	0.000000	0.017544	0.035088	0.000000	0.035088	0.000000	0.017544	0.035088	0.070175	0.00	0.017544	0.000000	0.017544	0.000000	0.00	0.017544	0.00	0.00	0.017544	0.017544	0.00	0.035088	0.000000	0.017544	0.00	0.017544	0.017544	0.000000	0.000000	0.000000	0.00	0.00	0.00	0.017544	0.017544	0.017544	0.000000	0.000000	0.0000	0.017544	0.00	0.00	0.000000	0.017544	0.017544	0.017544	0.017544	0.00	0.00	0.000000	0.017544	0.017544	0.00	0.00	0.017544	0.00	0.00	0.017544	0.00	0.00	0.00	0.017544	0.000000	0.000000	0.000000	0.0000	0.00	0.00	0.00	0.00	0.000000	0.017544	0.00	0.00	0.0000	0.035088	0.017544	0.000000	0.000000	0.000000	0.00	0.000000	0.035088	0.00	0.017544	0.017544	0.017544	0.00	0.017544	0.00
2	Brockton, Exhibition Place, Parkdale Village	0.0000	0.0000	0.0000	0.000	0.0000	0.0000	0.00	0.000000	0.00	0.000000	0.000000	0.000000	0.047619	0.047619	0.000000	0.000000	0.000000	0.000000	0.0000	0.00	0.0000	0.00	0.095238	0.000000	0.00	0.00	0.047619	0.000000	0.095238	0.047619	0.000000	0.047619	0.000000	0.000000	0.095238	0.00	0.000000	0.000000	0.000000	0.047619	0.00	0.000000	0.00	0.00	0.000000	0.000000	0.00	0.000000	0.000000	0.000000	0.00	0.000000	0.000000	0.047619	0.000000	0.000000	0.00	0.00	0.00	0.000000	0.000000	0.047619	0.047619	0.000000	0.0000	0.000000	0.00	0.00	0.047619	0.000000	0.047619	0.000000	0.000000	0.00	0.00	0.000000	0.000000	0.000000	0.00	0.00	0.000000	0.00	0.00	0.000000	0.00	0.00	0.00	0.000000	0.047619	0.047619	0.000000	0.0000	0.00	0.00	0.00	0.00	0.000000	0.047619	0.00	0.00	0.0000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00	0.047619	0.000000	0.00	0.000000	0.000000	0.000000	0.00	0.000000	0.00
3	Business Reply Mail Processing Centre 969 Eastern	0.0000	0.0000	0.0000	0.000	0.0000	0.0000	0.00	0.000000	0.00	0.055556	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.0000	0.00	0.0000	0.00	0.000000	0.055556	0.00	0.00	0.055556	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00	0.000000	0.055556	0.000000	0.000000	0.00	0.000000	0.00	0.00	0.000000	0.000000	0.00	0.055556	0.055556	0.000000	0.00	0.000000	0.000000	0.000000	0.055556	0.055556	0.00	0.00	0.00	0.000000	0.000000	0.000000	0.000000	0.055556	0.0000	0.000000	0.00	0.00	0.000000	0.000000	0.000000	0.000000	0.000000	0.00	0.00	0.111111	0.000000	0.000000	0.00	0.00	0.000000	0.00	0.00	0.000000	0.00	0.00	0.00	0.055556	0.000000	0.000000	0.055556	0.0000	0.00	0.00	0.00	0.00	0.055556	0.055556	0.00	0.00	0.0000	0.000000	0.000000	0.055556	0.055556	0.055556	0.00	0.000000	0.000000	0.00	0.000000	0.000000	0.000000	0.00	0.000000	0.00
4	CN Tower, Bathurst Quay, Island airport, Harbo...	0.0625	0.0625	0.0625	0.125	0.1875	0.0625	0.00	0.000000	0.00	0.000000	0.000000	0.000000	0.000000	0.062500	0.000000	0.000000	0.000000	0.000000	0.0625	0.00	0.0625	0.00	0.000000	0.000000	0.00	0.00	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.062500	0.00	0.000000	0.000000	0.000000	0.000000	0.00	0.000000	0.00	0.00	0.000000	0.000000	0.00	0.000000	0.000000	0.000000	0.00	0.000000	0.000000	0.000000	0.000000	0.000000	0.00	0.00	0.00	0.000000	0.000000	0.000000	0.000000	0.000000	0.0625	0.000000	0.00	0.00	0.000000	0.000000	0.000000	0.000000	0.000000	0.00	0.00	0.000000	0.000000	0.000000	0.00	0.00	0.000000	0.00	0.00	0.000000	0.00	0.00	0.00	0.000000	0.000000	0.000000	0.000000	0.0625	0.00	0.00	0.00	0.00	0.000000	0.000000	0.00	0.00	0.0625	0.000000	0.000000	0.000000	0.000000	0.000000	0.00	0.000000	0.000000	0.00	0.000000	0.000000	0.000000	0.00	0.000000	0.00

Let's get the neighborhoods with their top 5 most common venues

def return_most_common_venues(row, num_top_venues):
    row_categories = row.iloc[1:]
    row_categories_sorted = row_categories.sort_values(ascending=False)
    
    return row_categories_sorted.index.values[0:num_top_venues]

num_top_venues = 5

indicators = ['st', 'nd', 'rd']

# create columns according to number of top venues
columns = ['Neighborhoods']
for ind in np.arange(num_top_venues):
    try:
        columns.append('{}{} Most Common Venue'.format(ind+1, indicators[ind]))
    except:
        columns.append('{}th Most Common Venue'.format(ind+1))

# create a new dataframe
neighborhoods_venues_sorted = pd.DataFrame(columns=columns)
neighborhoods_venues_sorted['Neighborhoods'] = toronto_grouped['Neighborhoods']

for ind in np.arange(toronto_grouped.shape[0]):
    neighborhoods_venues_sorted.iloc[ind, 1:] = return_most_common_venues(toronto_grouped.iloc[ind, :], num_top_venues)

neighborhoods_venues_sorted.rename(columns={'Neighborhoods':'Neighborhood'},inplace=True)
neighborhoods_venues_sorted.head()

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	Neighborhood	1st Most Common Venue	2nd Most Common Venue	3rd Most Common Venue	4th Most Common Venue	5th Most Common Venue
0	Adelaide, King, Richmond	Coffee Shop	Café	Bar	Steakhouse	American Restaurant
1	Berczy Park	Coffee Shop	Bakery	Café	Beer Bar	Cheese Shop
2	Brockton, Exhibition Place, Parkdale Village	Coffee Shop	Café	Breakfast Spot	Grocery Store	Intersection
3	Business Reply Mail Processing Centre 969 Eastern	Light Rail Station	Pizza Place	Auto Workshop	Comic Shop	Recording Studio
4	CN Tower, Bathurst Quay, Island airport, Harbo...	Airport Service	Airport Lounge	Plane	Boutique	Bar

Clustering the neighborhoods using Kmeans

# set number of clusters
kclusters = 5

toronto_grouped_clustering = toronto_grouped.drop('Neighborhoods', 1)

# run k-means clustering
kmeans = KMeans(n_clusters=kclusters, random_state=0).fit(toronto_grouped_clustering)

# check cluster labels generated for each row in the dataframe
kmeans.labels_

array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0,
       4, 0, 1, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int32)

Creating a dataframe with cluster labels and the top 5 venues

# add clustering labels
neighborhoods_venues_sorted.insert(0, 'Cluster Labels', kmeans.labels_)

toronto_merged = df_toronto

# merge toronto_grouped with toronto_data to add latitude/longitude for each neighborhood

toronto_merged = toronto_merged.join(neighborhoods_venues_sorted.set_index('Neighborhood'), on='Neighborhood')

toronto_merged.head()

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	PostalCode	Borough	Neighborhood	Latitude	Longitude	Cluster Labels	1st Most Common Venue	2nd Most Common Venue	3rd Most Common Venue	4th Most Common Venue	5th Most Common Venue
0	M4E	East Toronto	The Beaches	43.676357	-79.293031	0	Neighborhood	Health Food Store	Pub	Trail	Event Space
1	M4K	East Toronto	The Danforth West, Riverdale	43.679557	-79.352188	0	Greek Restaurant	Coffee Shop	Ice Cream Shop	Italian Restaurant	Furniture / Home Store
2	M4L	East Toronto	The Beaches West, India Bazaar	43.668999	-79.315572	0	Park	Sandwich Place	Brewery	Steakhouse	Sushi Restaurant
3	M4M	East Toronto	Studio District	43.659526	-79.340923	0	Café	Coffee Shop	Bakery	Italian Restaurant	American Restaurant
4	M4N	Central Toronto	Lawrence Park	43.728020	-79.388790	4	Park	Swim School	Bus Line	Yoga Studio	Diner

We will display the neighborhoods along with their cluster labels and colours on a map below

# create map
map_clusters = folium.Map(location=[latitude, longitude], zoom_start=12)

# set color scheme for the clusters
x = np.arange(kclusters)
ys = [i + x + (i*x)**2 for i in range(kclusters)]
colors_array = cm.rainbow(np.linspace(0, 1, len(ys)))
rainbow = [colors.rgb2hex(i) for i in colors_array]

# add markers to the map
markers_colors = []
for lat, lon, bor, poi, cluster in zip(toronto_merged['Latitude'], toronto_merged['Longitude'],toronto_merged['Borough'], toronto_merged['Neighborhood'], toronto_merged['Cluster Labels']):
    label = folium.Popup(str(bor)+': ' +str(poi) + ' Cluster ' + str(cluster), parse_html=True)
    folium.CircleMarker(
        [lat, lon],
        radius=5,
        popup=label,
        color=rainbow[cluster-1],
        fill=True,
        fill_color=rainbow[cluster-1],
        fill_opacity=0.7).add_to(map_clusters)
       
map_clusters

Lets examine the clusters closely and name them if possible

Neighborhoods with Cluster Label: 0

toronto_merged.loc[toronto_merged['Cluster Labels'] == 0, toronto_merged.columns[[2] + list(range(5, toronto_merged.shape[1]))]]

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	Neighborhood	1st Most Common Venue	2nd Most Common Venue	3rd Most Common Venue	4th Most Common Venue	5th Most Common Venue
0	The Beaches	Neighborhood	Health Food Store	Pub	Trail	Event Space
1	The Danforth West, Riverdale	Greek Restaurant	Coffee Shop	Ice Cream Shop	Italian Restaurant	Furniture / Home Store
2	The Beaches West, India Bazaar	Park	Sandwich Place	Brewery	Steakhouse	Sushi Restaurant
3	Studio District	Café	Coffee Shop	Bakery	Italian Restaurant	American Restaurant
5	Davisville North	Gym	Breakfast Spot	Food & Drink Shop	Hotel	Clothing Store
6	North Toronto West	Clothing Store	Coffee Shop	Sporting Goods Shop	Gym / Fitness Center	Mexican Restaurant
7	Davisville	Sandwich Place	Gym	Dessert Shop	Pizza Place	Café
9	Deer Park, Forest Hill SE, Rathnelly, South Hi...	Pub	Coffee Shop	Pizza Place	Liquor Store	Sports Bar
11	Cabbagetown, St. James Town	Restaurant	Coffee Shop	Pub	Italian Restaurant	Park
12	Church and Wellesley	Coffee Shop	Japanese Restaurant	Sushi Restaurant	Restaurant	Gay Bar
13	Harbourfront	Coffee Shop	Pub	Park	Bakery	Café
14	Ryerson, Garden District	Clothing Store	Coffee Shop	Cosmetics Shop	Café	Fast Food Restaurant
15	St. James Town	Café	Coffee Shop	Hotel	Restaurant	Italian Restaurant
16	Berczy Park	Coffee Shop	Bakery	Café	Beer Bar	Cheese Shop
17	Central Bay Street	Coffee Shop	Italian Restaurant	Ice Cream Shop	Café	Burger Joint
18	Adelaide, King, Richmond	Coffee Shop	Café	Bar	Steakhouse	American Restaurant
19	Harbourfront East, Toronto Islands, Union Station	Coffee Shop	Aquarium	Hotel	Café	Italian Restaurant
20	Design Exchange, Toronto Dominion Centre	Coffee Shop	Café	Hotel	Restaurant	American Restaurant
21	Commerce Court, Victoria Hotel	Coffee Shop	Café	Hotel	Restaurant	American Restaurant
24	The Annex, North Midtown, Yorkville	Café	Sandwich Place	Coffee Shop	Pizza Place	BBQ Joint
25	Harbord, University of Toronto	Café	Restaurant	Bar	Italian Restaurant	Japanese Restaurant
26	Chinatown, Grange Park, Kensington Market	Café	Bar	Vietnamese Restaurant	Vegetarian / Vegan Restaurant	Chinese Restaurant
27	CN Tower, Bathurst Quay, Island airport, Harbo...	Airport Service	Airport Lounge	Plane	Boutique	Bar
28	Stn A PO Boxes 25 The Esplanade	Coffee Shop	Restaurant	Café	Seafood Restaurant	Hotel
29	First Canadian Place, Underground city	Coffee Shop	Café	Hotel	Restaurant	Steakhouse
30	Christie	Grocery Store	Café	Park	Nightclub	Convenience Store
31	Dovercourt Village, Dufferin	Pharmacy	Supermarket	Bakery	Pizza Place	Furniture / Home Store
32	Little Portugal, Trinity	Bar	Coffee Shop	Restaurant	Asian Restaurant	Vietnamese Restaurant
33	Brockton, Exhibition Place, Parkdale Village	Coffee Shop	Café	Breakfast Spot	Grocery Store	Intersection
34	High Park, The Junction South	Discount Store	Bar	Mexican Restaurant	Café	Thai Restaurant
35	Parkdale, Roncesvalles	Gift Shop	Coffee Shop	Bookstore	Bank	Italian Restaurant
36	Runnymede, Swansea	Coffee Shop	Café	Italian Restaurant	Sushi Restaurant	Bookstore
37	Business Reply Mail Processing Centre 969 Eastern	Light Rail Station	Pizza Place	Auto Workshop	Comic Shop	Recording Studio

Neighborhoods with Cluster Label: 1

toronto_merged.loc[toronto_merged['Cluster Labels'] == 1, toronto_merged.columns[[2] + list(range(5, toronto_merged.shape[1]))]]

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	Neighborhood	Cluster Labels	1st Most Common Venue	2nd Most Common Venue	3rd Most Common Venue	4th Most Common Venue	5th Most Common Venue
8	Moore Park, Summerhill East	1	Trail	Playground	Park	Tennis Court	Donut Shop
10	Rosedale	1	Park	Playground	Trail	Yoga Studio	Dessert Shop

Neighborhoods with Cluster Label: 2

toronto_merged.loc[toronto_merged['Cluster Labels'] == 2, toronto_merged.columns[[2] + list(range(5, toronto_merged.shape[1]))]]

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	Neighborhood	Cluster Labels	1st Most Common Venue	2nd Most Common Venue	3rd Most Common Venue	4th Most Common Venue	5th Most Common Venue
22	Roselawn	2	Garden	Home Service	Dim Sum Restaurant	Farmers Market	Falafel Restaurant

Neighborhoods with Cluster Label: 3

toronto_merged.loc[toronto_merged['Cluster Labels'] == 3, toronto_merged.columns[[2] + list(range(5, toronto_merged.shape[1]))]]

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	Neighborhood	Cluster Labels	1st Most Common Venue	2nd Most Common Venue	3rd Most Common Venue	4th Most Common Venue	5th Most Common Venue
23	Forest Hill North, Forest Hill West	3	Park	Trail	Jewelry Store	Sushi Restaurant	Yoga Studio

Neighborhoods with Cluster Label: 4

toronto_merged.loc[toronto_merged['Cluster Labels'] == 4, toronto_merged.columns[[2] + list(range(5, toronto_merged.shape[1]))]]

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	Neighborhood	Cluster Labels	1st Most Common Venue	2nd Most Common Venue	3rd Most Common Venue	4th Most Common Venue	5th Most Common Venue
4	Lawrence Park	4	Park	Swim School	Bus Line	Yoga Studio	Diner

Observations about the clusters

Neighborhoods in Cluster-0 has the most common venues related to food. Hence the cluster can be named as the Eatery cluster. Neighborhoods in other clusters are not sufficent enough to give a proper name to those clusters

------End of Assignment-----

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

capstone_assignment.md

capstone_assignment.md

Segmenting and Clustering Neighborhoods in Toronto

Table of Contents

1. First part of the assignment

Extracting data from a website

2. Second part of the assignment

Finding out the coordinates of the neighborhoods

3. Third part of the assignment

We will be exploring and clustering the neighborhoods whose boroughs' have the word "Toronto" in them

We will display the above neighborhoods on a map in the cells below

We will explore the neighborhoods using the Foursquare API in the following cells

We will check how many venues were obtained per neighborhood below

We will analyse each neigborhood in the cells below

let's group rows by neighborhood and by taking the mean of the frequency of occurrence of each category

Let's get the neighborhoods with their top 5 most common venues

Clustering the neighborhoods using Kmeans

Creating a dataframe with cluster labels and the top 5 venues

We will display the neighborhoods along with their cluster labels and colours on a map below

Lets examine the clusters closely and name them if possible

Neighborhoods with Cluster Label: 0

Neighborhoods with Cluster Label: 1

Neighborhoods with Cluster Label: 2

Neighborhoods with Cluster Label: 3

Neighborhoods with Cluster Label: 4

Observations about the clusters

------End of Assignment-----

Files

capstone_assignment.md

Latest commit

History

capstone_assignment.md

File metadata and controls

Segmenting and Clustering Neighborhoods in Toronto

Table of Contents

1. First part of the assignment

Extracting data from a website

2. Second part of the assignment

Finding out the coordinates of the neighborhoods

3. Third part of the assignment

We will be exploring and clustering the neighborhoods whose boroughs' have the word "Toronto" in them

We will display the above neighborhoods on a map in the cells below

We will explore the neighborhoods using the Foursquare API in the following cells

We will check how many venues were obtained per neighborhood below

We will analyse each neigborhood in the cells below

let's group rows by neighborhood and by taking the mean of the frequency of occurrence of each category

Let's get the neighborhoods with their top 5 most common venues

Clustering the neighborhoods using Kmeans

Creating a dataframe with cluster labels and the top 5 venues

We will display the neighborhoods along with their cluster labels and colours on a map below

Lets examine the clusters closely and name them if possible

Neighborhoods with Cluster Label: 0

Neighborhoods with Cluster Label: 1

Neighborhoods with Cluster Label: 2

Neighborhoods with Cluster Label: 3

Neighborhoods with Cluster Label: 4

Observations about the clusters

------End of Assignment-----