harvesting_domain_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Harvesting data about a domain using the IA CDX API\n",
    "\n",
    "<p class=\"alert alert-info\">New to Jupyter notebooks? Try <a href=\"getting-started/Using_Jupyter_notebooks.ipynb\"><b>Using Jupyter notebooks</b></a> for a quick introduction.</p>\n",
    "\n",
    "In this notebook we'll look at how we can get domain level data from a CDX API. There are two types of search you can use:\n",
    "\n",
    "* a 'prefix' query – searching for `nla.gov.au/*` returns captures from the `nla.gov.au` domain\n",
    "* a 'domain' query – searching for `*.nla.gov.au` returns captures from the `nla.gov.au` domain *and any subdomains*\n",
    "\n",
    "These searches can be combined with any of the other filters supported by the CDX API, such as `mimetype` and `statuscode`.\n",
    "\n",
    "As noted in [Comparing CDX APIs](comparing_cdx_apis.ipynb), support for domain level searching varies across systems. The AWA allows prefix queries, but not domain queries. The UKWA provides both in theory, but timeouts are common for large domains. Neither the AWA or UKWA supports pagination, so harvesting data from large domains can cause difficulties. For these reasons it seems sensible to focus on the IA CDX API, unless you're after data from a single, modestly-sized domain.\n",
    "\n",
    "Related notebooks:\n",
    "\n",
    "* [Exploring the Internet Archive's CDX API](exploring_cdx_api.ipynb)\n",
    "* [Comparing CDX APIs](comparing_cdx_apis.ipynb)\n",
    "* [Find all the archived versions of a web page](find_all_captures.ipynb) – shows how to use an 'exact' search with the CDX API\n",
    "* [Find and explore Powerpoint presentations from a specific domain](explore_presentations.ipynb) – example of finding particular types of files within a domain\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In most other notebooks using the CDX API we've harvested data into memory and then saved to disk later on. Because we're potentially harvesting much larger quantities of data, it's probably a good idea to reverse this and save harvested data to disk as we download it. We can also use `requests-cache` to save responses from the API and make it easy to restart a failed harvest. This is the same strategy used in the [Exploring subdomains in the gov.au domain](harvesting_gov_au_domains.ipynb) notebook where I harvest data about 189 million captures.\n",
    "\n",
    "### Usage\n",
    "\n",
    "#### Prefix query\n",
    "\n",
    "Either using a url wildcard:\n",
    "\n",
    "```\n",
    "harvest_cdx_query_to_file('[domain]/*', [optional parameters])\n",
    "```\n",
    "\n",
    "or the `matchType` parameter:\n",
    "\n",
    "```\n",
    "harvest_cdx_query_to_file('[domain]', matchType='prefix', [optional parameters])\n",
    "```\n",
    "\n",
    "#### Domain query\n",
    "\n",
    "Either using a url wildcard:\n",
    "\n",
    "```\n",
    "harvest_cdx_query_to_file('*.[domain]', [optional parameters])\n",
    "```\n",
    "\n",
    "or the `matchType` parameter:\n",
    "\n",
    "```\n",
    "harvest_cdx_query_to_file('[domain]', matchType='domain', [optional parameters])\n",
    "```\n",
    "\n",
    "### Output\n",
    "\n",
    "The results of each harvest are stored in a timestamped `.ndjson` file in a subdirectory of the `domains` directory. For example, a harvest from `nla.gov.au` is stored in `domains/nla-gov-au`. The file names combine the domain, the type of query (either 'prefix' or 'domain') and a timestamp. For example, a prefix query in `nla.gov.au` might generate a file named:\n",
    "\n",
    "```\n",
    "nla-gov-au-prefix-20200526113338.ndjson\n",
    "```\n",
    "\n",
    "Each harvest also creates a metadata file that has a similar name, but is in JSON format, for example:\n",
    "\n",
    "```\n",
    "nla-gov-au-prefix-20200526113338-metadata.json\n",
    "```\n",
    "\n",
    "The metadata file captures information about your harvest including:\n",
    "\n",
    "* `params` – the parameters used in your query (including any filters)\n",
    "* `timestamp` – date and time the harvest was started\n",
    "* `file` – path to the `ndjson` data file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import what we need"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from requests.adapters import HTTPAdapter\n",
    "from requests.packages.urllib3.util.retry import Retry\n",
    "from tqdm.auto import tqdm\n",
    "import pandas as pd\n",
    "import time\n",
    "from requests_cache import CachedSession\n",
    "import ndjson\n",
    "from pathlib import Path\n",
    "from slugify import slugify\n",
    "import arrow\n",
    "import json\n",
    "\n",
    "# By using a cached session, all responses will be saved in a local cache\n",
    "s = CachedSession()\n",
    "retries = Retry(total=10, backoff_factor=1, status_forcelist=[ 502, 503, 504 ])\n",
    "s.mount('https://', HTTPAdapter(max_retries=retries))\n",
    "s.mount('http://', HTTPAdapter(max_retries=retries))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define some functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_total_pages(params):\n",
    "    '''\n",
    "    Gets the total number of pages in a set of results.\n",
    "    '''\n",
    "    these_params = params.copy()\n",
    "    these_params['showNumPages'] = 'true'\n",
    "    response = s.get('http://web.archive.org/cdx/search/cdx', params=these_params, headers={'User-Agent': ''})\n",
    "    return int(response.text)\n",
    "\n",
    "def prepare_params(url, **kwargs):\n",
    "    '''\n",
    "    Prepare the parameters for a CDX API requests.\n",
    "    Adds all supplied keyword arguments as parameters (changing from_ to from).\n",
    "    Adds in a few necessary parameters.\n",
    "    '''\n",
    "    params = kwargs\n",
    "    params['url'] = url\n",
    "    params['output'] = 'json'\n",
    "    params['pageSize'] = 5\n",
    "    # CDX accepts a 'from' parameter, but this is a reserved word in Python\n",
    "    # Use 'from_' to pass the value to the function & here we'll change it back to 'from'.\n",
    "    if 'from_' in params:\n",
    "        params['from'] = params['from_']\n",
    "        del(params['from_'])\n",
    "    return params\n",
    "\n",
    "def convert_lists_to_dicts(results):\n",
    "    '''\n",
    "    Converts IA style timemap (a JSON array of arrays) to a list of dictionaries.\n",
    "    Renames keys to standardise IA with other Timemaps.\n",
    "    '''\n",
    "    if results:\n",
    "        keys = results[0]\n",
    "        results_as_dicts = [dict(zip(keys, v)) for v in results[1:]]\n",
    "    else:\n",
    "        results_as_dicts = results\n",
    "    for d in results_as_dicts:\n",
    "        d['status'] = d.pop('statuscode')\n",
    "        d['mime'] = d.pop('mimetype')\n",
    "        d['url'] = d.pop('original')\n",
    "    return results_as_dicts\n",
    "\n",
    "def check_query_type(url):\n",
    "    if url.startswith('*'):\n",
    "        query_type = 'domain'\n",
    "    elif url.endswith('*'):\n",
    "        query_type = 'prefix'\n",
    "    else:\n",
    "        query_type = ''\n",
    "    return query_type\n",
    "\n",
    "def get_cdx_data(params):\n",
    "    '''\n",
    "    Make a request to the CDX API using the supplied parameters.\n",
    "    Return results converted to a list of dicts.\n",
    "    '''\n",
    "    response = s.get('http://web.archive.org/cdx/search/cdx', params=params)\n",
    "    response.raise_for_status()\n",
    "    results = response.json()\n",
    "    try:\n",
    "        if not response.from_cache:\n",
    "            time.sleep(0.2)\n",
    "    except AttributeError:\n",
    "        # Not using cache\n",
    "        time.sleep(0.2)\n",
    "    return convert_lists_to_dicts(results)\n",
    "\n",
    "def save_metadata(output_dir, params, query_type, timestamp, file_path):\n",
    "    md_path = Path(output_dir, f'{slugify(params[\"url\"])}-{query_type}-{timestamp}-metadata.json')\n",
    "    md = {\n",
    "        'params': params,\n",
    "        'timestamp': timestamp,\n",
    "        'file': str(file_path)\n",
    "    }\n",
    "    with md_path.open('wt') as md_json:\n",
    "        json.dump(md, md_json)\n",
    "\n",
    "def harvest_cdx_query_to_file(url, **kwargs):\n",
    "    '''\n",
    "    Harvest capture data from a CDX query.\n",
    "    Save results to a NDJSON formatted file.\n",
    "    '''\n",
    "    params = prepare_params(url, **kwargs)\n",
    "    total_pages = get_total_pages(params)\n",
    "    output_dir = Path('domains', slugify(url))\n",
    "    output_dir.mkdir(parents=True, exist_ok=True)\n",
    "    # We'll use a timestamp to distinguish between versions\n",
    "    timestamp = arrow.now().format('YYYYMMDDHHmmss')\n",
    "    query_type = params['matchType'] if 'matchType' in params else check_query_type(url)\n",
    "    file_path = Path(output_dir, f'{slugify(url)}-{query_type}-{timestamp}.ndjson')\n",
    "    save_metadata(output_dir, params, query_type, timestamp, file_path)\n",
    "    page = 0\n",
    "    with tqdm(total=total_pages-page) as pbar1:\n",
    "        with tqdm() as pbar2:\n",
    "            while page < total_pages:\n",
    "                params['page'] = page\n",
    "                results = get_cdx_data(params)\n",
    "                with file_path.open('a') as f:\n",
    "                    writer = ndjson.writer(f, ensure_ascii=False)\n",
    "                    for result in results:\n",
    "                        writer.writerow(result)\n",
    "                page += 1\n",
    "                pbar1.update(1)\n",
    "                pbar2.update(len(results) - 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prefix query\n",
    "\n",
    "For a 'prefix' query either set the `matchType` parameter to `prefix` or use a url wildcard like `nla.gov.au/*`.\n",
    "\n",
    "Get all successful web page captures from the `nla.gov.au` domain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "harvest_cdx_query_to_file('nla.gov.au/*', filter=['statuscode:200', 'mimetype:text/html'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `collapse` to limit the harvest to remove (most) records with duplicate values for `urlkey`. This should give us a list of unique urls from the `nla.gov.au` domain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "harvest_cdx_query_to_file('nla.gov.au/*', filter=['statuscode:200', 'mimetype:text/html'], collapse='urlkey')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Domain query\n",
    "\n",
    "For a 'domain' query either set the `matchType` parameter to `domain` or use a url wildcard like `*.nla.gov.au`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "harvest_cdx_query_to_file('*.nla.gov.au', filter=['statuscode:200', 'mimetype:text/html'], collapse='urlkey')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should be able to load smaller files using the `ndjson` module. If you're working with large data files (millions of captures) you might not want to load them all into memory. Have a look at [Exploring subdomains in the gov.au domain](harvesting_gov_au_domains.ipynb) for some ways of processing the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Edit to point to your data_file, eg: 'domains/nla-gov-au/nla-gov-au-prefix-20200526123711.ndjson'\n",
    "data_file = '[Path to data file]'\n",
    "data_file = 'domains/nla-gov-au/nla-gov-au-prefix-20200526123711.ndjson'\n",
    "with open(data_file) as f:\n",
    "    capture_data = ndjson.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You could then convert the capture data to a Pandas dataframe for analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(capture_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "Created by [Tim Sherratt](https://timsherratt.org) for the [GLAM Workbench](https://glam-workbench.github.io).\n",
    "\n",
    "Work on this notebook was supported by the [IIPC Discretionary Funding Programme 2019-2020](http://netpreserve.org/projects/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}