This repository contains the materials for D-Lab’s Python Web APIs Workshop.
We recommend attending Python Fundamentals and Python Data Wrangling prior to this workshop.
Check D-Lab's Learning Pathways to figure out which of our workshops to take!
In this workshop, we cover how to extract data from the web with APIs using Python. APIs are often official services offered by companies and other entities, which allow you to directly query their servers in order to retrieve their data. Platforms like The New York Times offer APIs to retrieve data.
When APIs are not available or restricted, one can turn to web scraping. If you want to learn how to do web scraping in Python, attend the D-Lab Python Web Scraping Workshop.
Anaconda is a useful package management software that allows you to run Python and Jupyter notebooks easily. Installing Anaconda is the easiest way to make sure you have all the necessary software to run the materials for this workshop. If you would like to run Python on your own computer, complete the following steps prior to the workshop:
-
Download and install Anaconda (Python 3.9 distribution). Click the "Download" button.
-
Download the Python Web APIs workshop materials:
- Click the green "Code" button in the top right of the repository information.
- Click "Download Zip".
- Extract this file to a folder on your computer where you can easily access it (we recommend Desktop).
-
Optional: if you're familiar with
git, you can instead clone this repository by opening a terminal and entering the commandgit clone git@github.com:dlab-berkeley/Python-Web-APIs.git.
If you do not have Anaconda installed and the materials loaded on your workshop by the time it starts, we strongly recommend using the UC Berkeley Datahub to run the materials for these lessons. You can access the DataHub by clicking this button:
The DataHub downloads this repository, along with any necessary packages, and
allows you to run the materials in a Jupyter notebook that is stored on UC
Berkeley's servers. No installation is necessary from your end - you only need
an internet browser and a CalNet ID to log in. By using the DataHub, you can
save your work and come back to it at any time. When you want to return to your
saved work, just go straight to DataHub, sign
in, and you click on the Python-Web-APIs folder.
If you don't have a Berkeley CalNet ID, you can still run these lessons in the cloud, by clicking this button:
By using this button, however, you cannot save your work.
-
Open the Anaconda Navigator application. You should see the green snake logo appear on your screen. Note that this can take a few minutes to load up the first time.
-
Click the "Launch" button under "Jupyter Notebooks" and navigate through your file system to the
Python-Web-APIsfolder you downloaded above. -
Open the
lessonsfolder, and click01_api_lesson.mdto begin. -
Press Shift + Enter (or Ctrl + Enter) to run a cell.
-
By default, the necessary packages for this workshop should already be installed. You can install them within the Jupyter notebook by running the following line in its own cell:
!pip install -r requirements.txt
Note that all of the above steps can be run from the terminal, if you're familiar with how to interact with Anaconda in that fashion. However, using Anaconda Navigator is the easiest way to get started if this is your first time working with Anaconda.
D-Lab works with Berkeley faculty, research staff, and students to advance data-intensive social science and humanities research. Our goal at D-Lab is to provide practical training, staff support, resources, and space to enable you to use R for your own research applications. Our services cater to all skill levels and no programming, statistical, or computer science backgrounds are necessary. We offer these services in the form of workshops, one-to-one consulting, and working groups that cover a variety of research topics, digital tools, and programming languages.
Visit the D-Lab homepage to learn more about us. You can view our calendar for upcoming events, learn about how to utilize our consulting and data services, and check out upcoming workshops. Subscribe to our newsletter to stay up to date on D-Lab events, services, and opportunities.
D-Lab offers a variety of Python workshops, catered toward different levels of expertise.