Working with web archives

Current version: v1.2.0

We tend to think of a web archive as a site we go to when links are broken – a useful fallback, rather than a source of new research data. But web archives don't just store old web pages, they capture multiple versions of web resources over time. Using web archives we can observe change – we can ask historical questions. This collection of notebooks is intended to help historians, and other researchers, frame those questions by revealing what sort of data is available, how to get it, and what you can do with it.

Web Archives share systems and standards, making it much easier for researchers wanting to get their hands on useful data. These notebooks focus on four particular web archives: the UK Web Archive, the Australian Web Archive (National Library of Australia ), the New Zealand Web Archive (National Library of New Zealand), the UK Government Web Archive, and the Internet Archive. However, the tools and approaches here could be easily extended to other web archives.

Web archives are huge, and access is often limited for legal reasons. These notebooks focus on data that is readily accessible and able to be used without the need for special equipment. They use existing APIs to get data in manageable chunks. But many of the examples demonstrated can also be scaled up to build substantial datasets for analysis – you just have to be patient!

These notebooks are a starting point that I hope will encourage researchers to investigate the possibilities of web archives in more detail. They're intended to compliment the fabulous work being by projects such as Archives Unleashed to open web archives to new research uses.

The development of these notebooks was supported by the International Internet Preservation Consortium's Discretionary Funding Programme 2019-2020, with the participation of the British Library, the National Library of Australia, and the National Library of New Zealand. Thanks all!

See the web archives section of the GLAM Workbench for more information.

Notebook topics

Types of data

• Timegates, Timemaps, and Mementos – explore how the Memento protocol helps you get machine-readable data about web archive captures
• Exploring the Internet Archive's CDX API – some web archives provide indexes of the web pages they've archived through an API, this notebook looks in detail at the data provided by the Internet Archive's CDX API
• Comparing CDX APIs – this notebook documents differences between the Internet Archive's Wayback CDX API and the PyWb CDX API (used by AWA and UKWA)
• Timemaps vs CDX APIs – both Timemaps and CDX APIs can give us a list of captures from a particular web page, this notebook compares the results

Harvesting data and creating datasets

• Get the archived version of a page closest to a particular date – the Memento API enables us to get the archived version of a page closest to a particular date, the functions in this notebook smooth out these some variations across repositories
• Find all the archived versions of a web page – you can get all the captures of an archived page using either Timemaps or the CDX API, this notebook demonstrates both
• Harvesting collections of text from archived web pages – create a dataset from the text contents of a single page across time, or multiple pages
• Harvesting data about a domain using the IA CDX API – extract information about a whole domain using prefix and domain queries
• Find and explore Powerpoint presentations from a specific domain – a complete workflow from web archive to Powerpoints to PDFS to images and text, and explore it all in Datasette
• Exploring subdomains in the whole of gov.au - scale up your harvesting to assemble a complete set of subdomains over time, and visualise the results as a dendrogram

Exploring change over time

• Compare two versions of an archived web page – demonstrates a number of different ways to versions of an archive web page can be compared, from metadata to screenshots
• Observing change in a web page over time – getting and visualising information about all the captures of a single page over time
• Create and compare full page screenshots from archived web pages – generate full page screenshots of archived web pages, compare pages, captures, even repositories
• Using screenshots to visualise change in a page over time – create a time series of screenshots, one for each year, compiled into a single image
• Display changes in the text of an archived web page over time – work through, capture by capture, showing how the text contents of an archived web page has changed
• Find when a piece of text appears in an archived web page – look for the first or last occurance of text string in an archived web page, or just find every occurance and chart the frequency

Run these notebooks

There are a number of different ways to use these notebooks. Binder is quickest and easiest, but it doesn't save your data. I've listed the options below from easiest to most complicated (requiring more technical knowledge).

Using Binder

Click on the button above to launch the notebooks in this repository using the Binder service (it might take a little while to load). This is a free service, but note that sessions will close if you stop using the notebooks, and no data will be saved. Make sure you download any changed notebooks or harvested data that you want to save.

See the Using Binder section of the GLAM Workbench for more details.

Using Reclaim Cloud

Reclaim Cloud is a paid hosting service, aimed particularly at supported digital scholarship in hte humanities. Unlike Binder, the environments you create on Reclaim Cloud will save your data – even if you switch them off! To run this repository on Reclaim Cloud for the first time:

• Create a Reclaim Cloud account and log in.
• Click on the button above to start the installation process.
• A dialogue box will ask you to set a password, this is used to limit access to your Jupyter installation.
• Sit back and wait for the installation to complete!
• Once the installation is finished click on the 'Open in Browser' button of your newly created environment (note that you might need to wait a few minutes before everything is ready).

See the Using Reclaim Cloud section GLAM Workbench [for more details.

Using the Nectar Research Cloud

The Nectar Research Cloud (part of the Australian Research Data Commons) provides cloud computing services to researchers in Australian and New Zealand universities. Any university-affiliated researcher can log on to Nectar and receive up to 6 months of free cloud computing time. And if you need more, you can apply for a specific project allocation.

The GLAM Workbench is available in the Nectar Cloud as a pre-configured application. This means you can get it up and going without worrying about the technical infrastructure – just fill in a few details and you're away! To create an instance of this repository in the Nectar Cloud:

• Log in to the Nectar Dashboard using your university credentials.
• From the Dashboard choose Applications -> Browse Local.
• Enter 'GLAM' in the filter box and hit Enter, you should see the GLAM Workbench application.
• Click on the GLAM Workbench application's Quick Deploy button.
• Step through the various configuration options. Some options are only available if you have a dedicated project allocation.
• When asked to select a GLAM Workbench repository, choose 'Web archives' from the dropdown.
• Complete the configuration and deploy your GLAM Workbench instance.
• The url to access your instance will be displayed once it's ready. Click on the url!

See Using Nectar for more details.

Using Docker

You can use Docker to run a pre-built computing environment on your own computer. It will set up everything you need to run the notebooks in this repository. This is free, but requires more technical knowledge – you'll have to install Docker on your computer, and be able to use the command line.

• Install Docker Desktop.
• Create a new directory for this repository and open it from the command line.
• From the command line, run the following command:

  docker run -p 8888:8888 --name web-archives -v "$PWD":/home/jovyan/work quay.io/glamworkbench/web-archives repo2docker-entrypoint jupyter lab --ip 0.0.0.0 --NotebookApp.token='' --LabApp.default_url='/lab/tree/index.ipynb'
  

• It will take a while to download and configure the Docker image. Once it's ready you'll see a message saying that Jupyter Notebook is running.
• Point your web browser to http://127.0.0.1:8888

See the Using Docker section of the GLAM Workbench for more details.

Setting up on your own computer

If you know your way around the command line and are comfortable installing software, you might want to set up your own computer to run these notebooks.

Assuming you have recent versions of Python and Git installed, the steps might be something like:

• Create a virtual environment, eg: python -m venv web-archives
• Open the new directory" cd web-archives
• Activate the environment source bin/activate
• Clone the repository: git clone https://github.com/GLAM-Workbench/web-archives.git notebooks
• Open the new notebooks directory: cd notebooks
• Install the necessary Python packages: pip install -r requirements.txt
• Run Jupyter: jupyter lab

See the GLAM Workbench for more details.

Cite as

See the GLAM Workbench or Zenodo for up-to-date citation details.

---

This repository is part of the GLAM Workbench.
If you think this project is worthwhile, you might like to sponsor me on GitHub.