python4gis.ipynb

{"cells":[{"cell_type":"markdown","metadata":{},"source":["# Introduction to Python for Geospatial Professionals"]},{"cell_type":"markdown","metadata":{},"source":["If you already know about GIS, and you want to learn some basics of Python, you're in the right spot. In this workshop, we will get hands-on practice with\n","\n","* The fundamental patterns commonly used in Python\n","* Useful Python modules for working with geospatial data\n","\n","Throughout this workshop, you will execute code in the notebook cells. You can do that by clicking the Run button at the top of the interface, or the keyboard shortcuts (ctrl+enter or shift+enter). Some of the code is written for you already, and some of it you will write yourself. Either way, you will get the most out of the workshop if you engage interactively with the code, rather than passively running it:\n","\n","* Ask questions about what you see in the code\n","* Before running a code cell, make a prediction about what will happen\n","* Change the code and see what happens"]},{"cell_type":"markdown","metadata":{},"source":["## Lesson 1: Perform numeric calculations"]},{"cell_type":"markdown","metadata":{},"source":["A major reason to use Python is to manipulate data. Often that involves performing calculations on data - for example using the Calculate Field tool in ArcGIS. One way to think about Python is that it is a very fancy calculator."]},{"cell_type":"markdown","metadata":{},"source":["### 1.1: Perform arithmetic with operators"]},{"cell_type":"markdown","metadata":{},"source":["We can use Python to determine values from inputs by using operators: addition (`+`), subtraction (`-`), multiplication (`*`), division (`/`) and exponentiation (`**`).\n","\n","Run each of the cells below to see the arithmetic operators."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["1 + 2"]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["1 - 2"]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["1 * 2"]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["1 / 2"]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["1 ** 2"]},{"cell_type":"markdown","metadata":{},"source":["Python operators follow a typical order of operations where exponentiation is evaluated first from left to right, then multiplication and division from left to right, then addition and subtraction from left to right. \n","\n","Run the cell below to see order of operations in action"]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["1 / 4 * 2"]},{"cell_type":"markdown","metadata":{},"source":["If the default order of operations will produce the wrong response, we can use parantheses to force the correct order. Any operation inside parentheses will be evaluated before any other operation. \n","\n","Run the cell below to see how parentheses can force a particular order of operations."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["1 / (4 * 2)"]},{"cell_type":"markdown","metadata":{},"source":["#### Checkpoint 1.1: Operators"]},{"cell_type":"markdown","metadata":{},"source":["US federal law defines a foot as exactly 0.3048 meters. In the cell below, use this definition to determine the number of meters in 50 feet."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["A temperature sensor reported readings of 30.1°C, 32.0°C, and 31.8°C. In the cell below, calculate the average temperature reading."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["### 1.2: Refer to complex values with variables"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["Instead of using values directly, it can be helpful to give those values names and refer to them by name. This name is called a variable. We can assign values to variables with the assignment operator (`=`). If we write `variable = value`, we are saying \"this variable refers to this value\".\n","\n","\n","\n","Run the cell below to assign coordinate values to variables. Unlike the code above, simply assigning variable values will not output anything to the notebook."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["x1 = 86946.415578\n","y1 = 2719456.806167\n","\n","x2 = 86803.207582\n","y2 = 2719506.303920"]},{"cell_type":"markdown","metadata":{},"source":["Using variables gives us two advantages:\n","\n","* We only need to define their value once and then we can reuse them. If we need to change our definition, we only need to make one update and that update is automatically reflected everywhere the variable is used. That makes code easier to write.\n","* Well-named variables provide context for the value. `2719456.806167` is just a number. Assigning that number to a variable called `y1` helps us understand what the value means. That makes code easier to read.\n","\n","Because good variable names can make a big difference in the readability of your code, take care to come up with effective names."]},{"cell_type":"markdown","metadata":{},"source":["We can perform the same types of operations on variables as we can on the values the variables represent.\n","\n","Run the cell below to use the [2-dimensional Euclidean distance formula](https://en.wikipedia.org/wiki/Euclidean_distance#Two_dimensions) to calculate the distance between (`x1`, `y1`) and (`x2`, `y2`) and assign that value to a variable named `distance`."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["distance = (((x1 - x2) ** 2) + ((y1 - y2) ** 2)) ** 0.5"]},{"cell_type":"markdown","metadata":{},"source":["A very common pattern is to reuse a calculated value in additional calculations to make those calculations easier to understand. For example, if we wanted twice the distance between (`x1`, `y1`) and (`x2`, `y2`), we could have calculated that all on a single line. But it is easier to follow what is happening if we calculate the distance, then calculate twice the distance.\n","\n","Run the cell below to see this use of variables."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["distance * 2"]},{"cell_type":"markdown","metadata":{},"source":["#### Checkpoint 1.2: Variables"]},{"cell_type":"markdown","metadata":{},"source":["In the cell below, write code to calculate half the distance between (`x1`, `y1`) and (`x2`, `y2`)."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["In the cell below, define new values for `x1` and `y1`. Then, without writing any additional code, calculate half the distance from that new point to (`x2`, `y2`). Hint: in a notebook, cells can be run in any order."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["### Application 1: Calculate population density"]},{"cell_type":"markdown","metadata":{},"source":["Consider a county with a population of 2,487,401 and an area of 2,107 mi<sup>2</sup>. In the cell below, create two variables, called `pop` and `sq_mi` and assign them the appropriate value."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["In the cell below, use the `pop` and `sq_mi` variables to calculate the population density of the county in people per square mile."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["There are 2.589988 square kilometers in a square mile. In the cell below, recalculate the population density of the county in people per square kilometer."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["In the cell below, calculate the population density in people per square mile if the county experiences 12% population growth."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["## Lesson 2: Simplify complexity with abstraction"]},{"cell_type":"markdown","metadata":{},"source":["Computers are complicated. Too complicated for people to fully understand. Instead, we rely on an abstract representation of the system. We say things like \"these circuits are logic gates.\"\n","\n","Even that abstraction is pretty complicated. So we represent it as an even higher-order abstraction, like \"these logic gates calculate values.\" Computer systems are layers on layers of abstraction until we get to \"click Run, get result.\"\n","\n","Abstraction is incredibly useful, because it allows us to manipulate something complex without having to deal with that complexity directly. Even if we did know exactly what happens when we click Run, abstraction means that we don't need to think about it every time we click."]},{"cell_type":"markdown","metadata":{},"source":["### 2.1: Perform complex actions with functions"]},{"cell_type":"markdown","metadata":{},"source":["We saw in the previous lesson how variables could make it easier to understand complexity. Variables are one type of abstraction in Python. Functions are another.\n","\n","Functions define some action for Python to perform. Instead of writing code for the action every time we want to use it, we can just use the name of the function followed by a pair of parentheses. Usually, we need to supply some information to the function. These pieces of information are called arguments, and we write them inside the parentheses.\n","\n","Run the cell below to use the `print` function to write some text to the notebook's output. The quotation marks show that the value should be interpreted as text and not Python commands. This type of value is called a string."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["tobler = 'Everything is related to everything else, but near things are more related than distant things.'\n","print(tobler)"]},{"cell_type":"markdown","metadata":{},"source":["The code required to get the notebook to show the text of the `tobler` string is fairly complex. But we don't need to worry about exactly how the `print` function works. We just need to be able to use it."]},{"cell_type":"markdown","metadata":{},"source":["Some functions, like `print`, can stand alone. Other functions are always attached to some value. These types of functions are called methods.\n","\n","Run the cell below to replace all the spaces in the `tobler` string with underscores."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["tobler.replace(' ', '_')"]},{"cell_type":"markdown","metadata":{},"source":["Let's break down this code:\n","\n","1. The dot between `tobler` and `replace` shows that the `replace` method will operate on the `tobler` string. That's allowed because somebody wrote code for a `replace` method that all string values have.\n","1. We put parentheses after the method name, just like with other functions.\n","1. The `replace` method needs two arguments. The first is the part of the string that will be replaced. The second is what will replace it.\n","1. When we supply multiple arguments, they are separated by a comma.\n","1. Because this value is created by the last line in the cell, the notebook writes the output for us to see.\n","\n","Strings have many different methods we can use. [This reference from W3 Schools](https://www.w3schools.com/python/python_ref_string.asp) describes each of them. "]},{"cell_type":"markdown","metadata":{},"source":["#### Checkpoint 2.1: Functions"]},{"cell_type":"markdown","metadata":{},"source":["The `help` function can provide information about a Python object. In the cell below, pass the `print` function as an argument to the `help` function to find out more about what we can do with `print` function."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["String values have a method called `endswith`. The method takes one string argument and returns `True` if the string ends with that argument. In the cell below, use the `endswith` method with the argument `'things'` on the `tobler` string."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["### 2.2: Use abstractions imported from modules."]},{"cell_type":"markdown","metadata":{},"source":["Many functions and methods, like `print`, are built in to Python. But even more are available in separate blocks of code called modules. We can import these modules into our scripts to use their functionality. Using modules effectively is a key Python pattern. Instead of creating new solutions from scratch, it makes more sense to use existing code to solve a problem.\n","\n","Run the cell below to import the `time` module. This module has many abstractions for making it easier to work with time."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["import time"]},{"cell_type":"markdown","metadata":{},"source":["Now that the module has been imported into our script, we can use it. Run the cell below to output a text representation of the current local date and time."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["time.ctime()"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["Let's breakdown this code:\n","1. We invoke the `time` module. That's allowed because we previously imported the module.\n","1. We use `.` after `time` to indicate that what comes next is defined inside the `time` module.\n","1. We call the `ctime` function. That's allowed because the `time` module defines a function called `ctime`.\n","1. We do not supply any argument to the function, so it returns its default value of the current local date and time.\n","1. Because this value is created by the last line in the cell, the notebook writes the output for us to see."]},{"cell_type":"markdown","metadata":{},"source":["#### Checkpoint 2.2: Modules "]},{"cell_type":"markdown","metadata":{},"source":["The `time` module contains another function called `localtime` that provides the time in a structured format more easily readable by computers. In the cell below, run the `localtime` function. "]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["The `math` module contains a function called `sqrt` that takes a single number as an argument and returns the square root of that number. In the cell below, use the `math` module to find the square root of any number."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["### Application 2: Buffer road features with Shapely\n"]},{"cell_type":"markdown","metadata":{},"source":["Consider a scenario where we have some street centerlines for a neighborhood and we need to find a 20 meter buffer around those centerlines as part of a environmental impact analysis.\n","\n","For this scenario, we will use [Shapely](https://shapely.readthedocs.io/en/stable/manual.html). Shapely is useful for geospatial professionals to be familiar with because it provides a way to work with geometry using Python. Several other modules rely on Shapely.\n","\n","Unlike the `time` and `math` modules, most specialized modules for geospatial analysis, including `shapely`, are not installed automatically when you install Python. The modules used in this workshop will import sucessfully in the Python environments used by ArcGIS, but other environments may require separate installations of the modules. Because managing environments can be challenging, if you do not have access to an ArcGIS environment, you may want to consider [Anaconda](https://www.anaconda.com/), which is a way of simplifying environment management.\n","\n","In the cell below, import the `shapely` module."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["The `geom` variable in the cell below is a well known text (wkt) representation of the geometry of St. Paul's Brainerd Avenue and intersecting streets. The triple quotation marks let us write a longer string on multiple lines.\n","\n","Run the cell below to read the string into memory so it can be used later."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["geom = '''MULTILINESTRING ((598 307, 597 607), (0 42, 99 87), (0 0, 0 103), (498 340, 598 409),\n","(200 135, 208 140), (208 140, 301 204), (301 204, 399 271), (399 271, 498 340),\n","(208 140, 200 173, 199 248), (598 409, 704 410), (498 340, 504 420), (100 1, 99 231),\n","(301 4, 299 407), (500 6, 498 340), (401 5, 399 271), (99 87, 200 135), (200 2, 200 135))'''"]},{"cell_type":"markdown","metadata":{},"source":["Shapely is typically referred to as a package or library (rather than a module), because it is made up of a collection of smaller modules. One of the Shapely submodules is `shapely.wkt`, which is used for working with wkt representations of geometry like `geom` above. The `shapely.wkt` submodule contains a function called `loads` that accepts a wkt string as an argument and turns it into geometry.\n","\n","In the cell below, import `shapely.wkt` and use its `loads` function to create Shapely geometry from the `geom` variable. Hint: recall how dot notation makes functions in a module accessible."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["Displaying the output of a function can be helpful for troubleshooting - we can make sure the output looks correct. But often we want to use the output, instead of just looking at it.\n","\n","In the cell below, rewrite the code above to assign the created geometry to a variable called `streets`. When we assign output to a variable, the notebook will not attempt to represent that output. Hint: recall how we use the assignment operator (`=`) to assign values to variables."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["Shapely geometries have many different methods. One of them is called `buffer`. The `buffer` method takes one argument: a number that represents how far to buffer the geometry. The coordinates used to create the `streets` geometry were in meters, so the buffer value will represent meters as well. \n","\n","In the cell below, use the the `streets` geometry's `buffer` method to create the 20 meter buffer around the geometry and assign the output to a variable called `buffer20m`."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["Even when we assign values to variables, we can still get the notebook to represent that value by putting the variable name on the last line of the cell. \n","\n","Run the cell below to get a represention of the `buffer20m` geometry."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["buffer20m"]},{"cell_type":"markdown","metadata":{},"source":["The `shapely.wkt` module contains another function called `dumps` that takes a geometry as an argument and returns the wkt representation of that geometry. In the cell below, do the following:\n","\n","1. Generate the wkt representation of the `buffer20m` geometry using the `dumps` function.\n","1. Assign that wkt representation to a variable named `buffer20m_wkt`.\n","1. Display the wkt representation in the notebook."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["The [Shapely documentation](https://shapely.readthedocs.io/en/stable/manual.html#object.buffer) for the `buffer` method includes several options for controlling the buffer. Refer to that documentation to complete the following step.\n","\n","In the cell below, display a 50 meter buffer on the right side only of the `streets` geometry."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["## Lesson 3: Represent data with the right structure"]},{"cell_type":"markdown","metadata":{},"source":["In order to deal with information in a way that both humans and computers can understand, we need a model for that information. The typical way to handle different varieties of information is to model them as different data types. You probably have experience with data types already. When you add a field to a table in GIS software, you need to specify the data type for that field. Maybe the field should hold integers, or text, or dates. You're probably also familiar with the pain that comes from having mismatched data types, like trying to deal with numeric values that are actually stored as text.\n","\n","Python has data types too. The most basic types are pretty much the same as the ones we're used to. There are also some more complex types that we'll take a look at later. Just like with the data types in an attribute table, we can find ourselves in trouble if we use the wrong Python data type."]},{"cell_type":"markdown","metadata":{},"source":["### 3.1: Differentiate between the primitive datatypes"]},{"cell_type":"markdown","metadata":{},"source":["There are four fundamental datatypes in Python, called primitives. We have already worked with each of these four types.\n","\n","**Name**|**Label**|**Used for**\n","-|-|-\n","`int`|Integer|Integer numbers\n","`float`|Float|Any real number\n","`str`|String|Text characters\n","`bool`|Boolean|True / False\n","\n"]},{"cell_type":"markdown","metadata":{},"source":["The `type` function takes a value as an argument, and returns the datatype of that value. Run the cell below to check the type of the `tobler` variable."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["type(tobler)"]},{"cell_type":"markdown","metadata":{},"source":["Sometimes a value of one datatype can be represented as a different datatype. For example, the `int` value `7` might also plausibly be represented as the `float` value `7.0` or the `str` value `'7'`. In some cases, Python will be able to implicitly treat a value as if it was a different datatype. But most of the time, if we need a value to be represented as a different type, we must do it explicitly ourselves.\n","\n","Run the three cells below to convert a value betwen different datatypes."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["str(7)"]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["int('7')"]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["float(7)"]},{"cell_type":"markdown","metadata":{},"source":["Sometimes type conversion (also called casting) has unexpected effects, so be careful using it. Run the cell below to see what happens when converting the string value `'False'` to a Boolean."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["bool('False')"]},{"cell_type":"markdown","metadata":{},"source":["Sometimes type coversion doesn't work at all, because there is no way to convert the value to the desired type. Run the cell below to see what happens when converting the string value in the `tobler` variable to a float."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["float(tobler)"]},{"cell_type":"markdown","metadata":{},"source":["#### Checkpoint 3.1: Primitives"]},{"cell_type":"markdown","metadata":{},"source":["In the cell below, determine the data type of the `streets` variable."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["In the cell below, convert the `x1` value to an integer."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["### 3.2: Interact with an object using its attributes"]},{"cell_type":"markdown","metadata":{},"source":["In Python, everything is an object. Values are objects. Functions are objects. Modules are objects. Being an object just means having a particular set of defined attributes. Attributes come in two varieties: properties and methods.\n","\n","Properties are information about an object. For example, if a person was a Python object, they might have a `name` property because their name is a piece of information about them. \n","\n","Methods are things that an object can do. For example, a person obect might have a `breathe` method because breathing is a thing a person can do. We've already seen how a Shapely geometry object can use its `buffer` method to create a new geometry object representing a buffer around itself.\n","\n","Shapely `Polygon` objects like `buffer20m` have an `exterior` property that returns the geometry of their boundary. Run the cell below to access this `exterior` property. "]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["buffer20m.exterior"]},{"cell_type":"markdown","metadata":{},"source":["In order to access a particular attribute of an object, that attribute must be defined for that object. Run the cell below to see what happens when attempting to access the `exterior` property of a `MultiLineString` object like `streets`."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["streets.exterior"]},{"cell_type":"markdown","metadata":{},"source":["When we encounter an error, the error message can be hard to read. Instead of starting at the top of the message, start at the bottom. `AttributeError: 'MultiLineString' object has no attribute 'exterior'` means that we tried to access the `exterior` attribute of a `MutliLineString` object. That's a problem because `MultiLineString` objects don't have an `exterior` attribute.\n","\n","We can use the `help` function to find information about the attributes of an object. Note that many attributes start with one or more underscores. These attributes are generally used by Python behind the scenes and we would not use them directly. "]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["help(streets)"]},{"cell_type":"markdown","metadata":{},"source":["#### Checkpoint 3.2: Attributes"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["In the cells below, write code that allows you to access two different attributes of the `streets` object that were listed by the `help` function above. Hint: Remember that because methods are a type of function, you need to include parentheses after the method name to use the method."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["### Application 3: Prepare data with Pandas"]},{"cell_type":"markdown","metadata":{},"source":["Consider an analysis in Ramsey county to determine if a voting precint's rate of absentee voting in 2022 is correlated with precint's partisan lean in 2020. The absentee ballot information is made available by the county in csv format. The election result data is made available by the state in Excel spreadsheet (xlsx) format. To perform this analysis, we will need to join the two datasets together.\n","\n","For this scenario, we will use the Pandas module. Pandas is useful for working with tabular data, and comes with a large number of tools for transforming and cleaning datasets with Python. It is especially good in this case because it is capable of turning data from both csv and xlsx into the same type of object, called a dataframe. With both datasets modeled as the same type of Python object, it will be easier to join them together.\n","\n","Run the cell below to import the Pandas module. Notice that the import statement is different than what we have used before. When we import a module, we have the option of providing an alias. Pandas is traditionally given the alias `pd`."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["import pandas as pd"]},{"cell_type":"markdown","metadata":{},"source":["The data on absentee ballots are stored online in csv format at https://umn.maps.arcgis.com/sharing/rest/content/items/ff2833751fce47e3bc5d7048c6ddd705/data. The Pandas module has a `read_csv` function. This function takes a string representing the file's file path or URL as an argument. It then converts the data in that file to a dataframe object. \n","\n","Run the cell below to do the following:\n","\n","* Using a string to represent the file path of the csv, assign that string to a varaible called `absentee_csv`. \n","* Assign the output of the `read_csv` function in the pandas module to a variable called `absentee_df`\n","\n","Because we imported the pandas module with an alias, as far as our Python code is concerned, this module is named `pd`. "]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["absentee_csv = 'https://umn.maps.arcgis.com/sharing/rest/content/items/ff2833751fce47e3bc5d7048c6ddd705/data'\n","absentee_df = pd.read_csv(absentee_csv)"]},{"cell_type":"markdown","metadata":{},"source":["Pandas dataframe objects like `absentee_df` have a `head` method. This method displays the first few rows of the dataframe, which is very useful for checking the schema of the data.\n","\n","In the cell below, use the `head` method to display the first few rows of the `absentee_df` dataframe."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["The relevant value from this table is the Ballot ratio column, which indicates the rate of absentee ballots sent per registered voter in each precinct."]},{"cell_type":"markdown","metadata":{},"source":["We also need data on the partisan lean of each precinct, as represented by the DFL share of the vote in the last presidential election. The data are stored in xlsx format at the URL https://umn.maps.arcgis.com/sharing/rest/content/items/6d7e8a2a7d454506b625abb61d4af303/data.\n","\n","The Pandas module contains a `read_excel` function that operates very similarly to `read_csv`. One difference is that in addition to the file, the `read_excel` funtion takes a second argument specifying the worksheet in the file. In this case, use `0` for the second argument to indicate the first worksheet (counting in Python starts at `0`, not `1`)\n","\n","In the cell below, do the following:\n","* Using a string to represent the full file path of the spreadsheet, assign that string to a varaible called `partisan_lean_xlsx`. \n","* Assign the output of the `read_excel` function in the pandas module to a variable called `partisan_lean_df`\n","* Use the `head` method on the `partisan_lean_df` dataframe to display the first few rows of the table."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["In this dataset, the relevant value is from the DFLPERC column. This value represents the ratio of presidential votes that went to the Democratic Party nominee in the precint."]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["Now that each dataset is represented as a dataframe with a common field (PCTCODE), we can join them together. The Pandas module has a function called `merge` to join tables. The `merge` function takes four arguments:\n","\n","1. The left dataframe in the join. In this case use `absentee_df`\n","1. The right dataframe in the join. In this case use `partisan_lean_df`\n","1. A string representing how the tables should be joined. In this case, we want only records that appear on both tables, so use the string `'inner'`.\n","1. A string representing the common field name between the two dataframes on which to base the join. In this case use the string `'PCTCODE'`\n","\n","In the cell below, join the two dataframes, assign the ouput to a new dataframe called `election_data_df` and display the first few rows of the new dataframe."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["Dataframe objects like `election_data_df` have a `plot` method to create graphs of the data in the dataframe. The `plot` method takes three arguments that we care about for our analysis: \n","\n","* A string representing the name of the column to plot on the x-axis\n","* A string representing the name of the column to plot on the y-axis\n","* A string representing the name of the kind of plot to create. Refer to [this Pandas documentation](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.plot.html?highlight=kind#pandas.DataFrame.plot) to see the possible options.\n","\n","In the cell below, create a scatter plot showing the relationship between the `'DFLPERC'` column (x-axis) and the `'Ballot ratio'` column (y-axis)."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["## Lesson 4: Automate repetitive processes"]},{"cell_type":"markdown","metadata":{},"source":["A certain kind of laziness is required to be a good programmer. Only a true lazy person would work incredibly hard in order to get out of doing work. An industrious person, for example, would have been happy to sit over a light table all day cutting lines out of a scribe sheet for a map compliation. It took a lazy person to figure out how to get the computer to draw the map for them.\n","\n","These kinds of lazy people hate doing the same thing over and over. It's boring. But since computers never get bored, repetitive processes are a good target to automate with Python. Probably the single biggest reason to learn Python is to get the computer to do the repetitive stuff so that we can focus on the fun stuff."]},{"cell_type":"markdown","metadata":{},"source":["### 4.1: Represent multiple values using lists"]},{"cell_type":"markdown","metadata":{},"source":["Repetitive tasks usually don't involve doing literally the exact same thing every time. More typically, they involved running the same process on different inputs. For example, reprojecting 100 different feature classes from UTM to state plane. \n","\n","A key pattern for automating repetitive tasks is to represent multiple inputs as a single object, called a container. In Python, one common container is a list.\n","\n","A list is an ordered collection of objects enclosed with brackets (`[]`). Each object in the list is separated by a comma. \n","\n","Run the cell below to assign a list of EPSG spatial reference codes to the variable `wkids`."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["wkids = [4326, 3745, 6505, 3857]"]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["Sometimes we want to access an individual element in a list, instead of the entire list. Bracket notation is the tool we use to do that.\n","\n","Run the cell below to access the first element in the `wkids` list (remember that Python starts counting at `0`, not `1`)."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["wkids[0]"]},{"cell_type":"markdown","metadata":{},"source":["#### Checkpoint 4.1: Lists"]},{"cell_type":"markdown","metadata":{},"source":["In the cell below, use a list to represent the names of any three Minnesota counties. Assign the list to a variable named `mn_counties`."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["In the cell below, access the third element of the `mn_counties` list."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["### 4.2: Execute code multiple times with iteration"]},{"cell_type":"markdown","metadata":{},"source":["With a list, we can use execute the same code code repeatedly for every element in that list using a `for` loop.\n","\n","Run the code below to print out each EPSG code in the `wkid` list."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["for id in wkids:\n","    print(id)"]},{"cell_type":"markdown","metadata":{},"source":["Here's what this code means:\n","* Take the first element in the `wkids` list, which is `4326`. \n","* Assign that value to a new variable called `id`. \n","* The colon (`:`) at the end of the line signals that the following indented code will be executed once for every element in the `wkids` list. In this case, we print out the value of `id`.\n","* After executing the code for the first element in the `wkid`, Python checks to see if there are any other values in the list. There are.\n","* The next value, `3745` gets assigned to `id`, and the indented code gets executed again. \n","* This process iterates through the list until there are no more values left in `wkids`.\n","\n","In this case every iteration executes only a single line of code, but there could be more. The entire block of indented code gets executed for every iteration of the loop."]},{"cell_type":"markdown","metadata":{},"source":["#### Checkpoint 4.2: Iteration"]},{"cell_type":"markdown","metadata":{},"source":["The `streets` object we worked with earlier is a Shapely `MultiLineString` object, which means it is a single line geometry made up of multiple lines. `MultiLineString` objects have a `geoms` property that returns all these component lines as seperate objects in a list that we can iterate over using a for loop. Each of these component lines is a Shapely `LineString` object. \n","\n","In the cell below, use `print` to output each of these `LineString` obects."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"attachments":{},"cell_type":"markdown","metadata":{},"source":["A Shapely `LineString` object has a `length` property that indicates how long it is.\n","\n","In the cell below, print out the length of each street represented in `streets.geoms`."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["### Application 4: Access metadata for feature layers with the ArcGIS API for Python"]},{"cell_type":"markdown","metadata":{},"source":["Consider a situation where we need to list the creation date for a number of layers published to ArcGIS Online. We could log into our portal and check the date of each item individually, but that would take too long. Instead, we can use the ArcGIS API for Python to get the information programatically.\n","\n","Run the cell below to import the relevant part of the ArcGIS API for Python."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["from arcgis.gis import GIS"]},{"cell_type":"markdown","metadata":{},"source":["This form of importing is different from others that we have used before. Like Shapely, the ArcGIS API for Python is made up of multiple modules, one of which is the `gis` module. Instead of making the entire `gis` module accessible to our script, however, we are just going to import one thing from it, called `GIS`. This part of the `gis` module allows us to make connections to our portal.\n","\n","Run the cell below to use `GIS` to make a connection to arcgis.com and assign that connection to a variable called `gis`."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["gis = GIS()"]},{"cell_type":"markdown","metadata":{},"source":["Our `gis` variable represents an anonymous connection. We could have connected as a specific user by passing a portal URL and credentials as arguments, but an anonymous connection is good enough for our purposes. "]},{"cell_type":"markdown","metadata":{},"source":["One of the properties of the `gis` object is `content`. This property is the Content Manager for managing content in our portal. The `content` object has a method called `search` that finds content items in the portal that match the search criteria.\n","\n","Run the cell below to create a list of Feature Layer items that have a `'Minnesota'` tag and store that list in a variable called `items`. "]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":["items = gis.content.search('tags: Minnesota', 'Feature Layer')"]},{"cell_type":"markdown","metadata":{},"source":["When working in a notebook, we have access to the `display` function. The `display` function is similar to `print`, but it can show more than just text. The `display` function takes one argument, which is an object to display, then displays that object in the notebook cell's output.\n","\n","In the cell below, use the `display` function to display one of the elements in the `items` container. Hint: you will need bracket notation to access an individual element in the list."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["Accessing individual items one at a time is hardly any better than just going to the portal web page. In the cell below, use a loop to display every element in `items`."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["The `display` function provides a nice representation of the item, and a link to its item details page, but we just want to know when each item was created. Each of the objects in the `items` list has a `created` property that stores this information.\n","\n","In the cell below, `print` the value of the `created` property for each element in `items`."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["The `created` property stores the item's creation time in miliseconds since January 1, 1970. This is a good way for computers to keep track of time, but it's not very helpful for people. \n","\n","Earlier, we used the `ctime` function from the `time` module to return a human-readable timestamp. When used without an argument, `ctime` returns the current time. But if you provide the time in seconds since January 1, 1970 as an argument, `ctime` will return the corresponding timestamp.\n","\n","In the cell below, print out a human-readable timestamp for the creation date of each of the elements in `items`. Hint: You will need to convert the miliseconds returned by the `created` property into the seconds expected by the `ctime` function. You will probably want to use a variable to store that time in seconds."]},{"cell_type":"code","execution_count":null,"metadata":{"trusted":false},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{},"source":["## Summary"]},{"cell_type":"markdown","metadata":{},"source":["Here are the top 6 takeaways from this workshop:\n","\n","* Python can be used as a calculator. This is a good way to start.\n","* Use variables with meaningful names.\n","* Worry about using abstractions, not the low-level details of how they work.\n","* There is a good chance a module already exists to solve your problem. \n","* Datatypes matter.\n","* Doing the same thing over and over is boring. Use iteration to get Python to do it for you.\n","\n","These are the geospatial modules we used:\n","\n","Module|Good for working with|Documentation\n","-|-|-\n","`shapely`|feature geometry|https://shapely.readthedocs.io/en/stable/manual.html\n","`pandas`|tabular data|https://pandas.pydata.org/docs/\n","`arcgis`|items in an ArcGIS portal|https://developers.arcgis.com/python/"]},{"cell_type":"markdown","metadata":{},"source":["## On your own"]},{"cell_type":"markdown","metadata":{},"source":["To maintain and improve on what you've learned, continue practicing with Python after this workshop. You will receive a copy of all the course materials so you can keep practicing with the code in this notebook.\n","\n","To advance you knowledge beyond the concepts in this workshop, look into the following:\n","\n","* Container types other than list: tuple, set, and dictionary\n","* Conditional statements: `if`, `else`, and `elif`\n","* Other useful modules for geospatial work: `arcpy`, `numpy`, `rasterio`, `fiona`\n","* Reading and writing files with Python\n","\n","You may also want to consider these resources for additional practice\n","\n","* [30 Days of Python](https://blog.teclado.com/30-days-of-python/)\n","* [Tutorials from Real Python](https://realpython.com/)\n","* [Intro to Python tutorial from DataCamp](https://www.learnpython.org/)\n","* [Python Tutorial from the Python Foundation](https://docs.python.org/3.7/tutorial/)\n","* [Esri Python tutorials](https://www.esri.com/training/Bookmark/FKCJSFXXD)\n","\n","The best way to hone your Python skills, however, is to incorporate them into your regular workflows. Find simple tasks that can be automated with Python, and build from there.\n","\n","Map on!"]}],"metadata":{"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.6"},"vscode":{"interpreter":{"hash":"0f27b128af07d2070dd8528f7021fe0faf81a6ddca52ed93c000f6b2452d2b36"}}},"nbformat":4,"nbformat_minor":2}