screenshots_over_time_using_timemaps.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using screenshots to visualise change in a page over time\n",
    "\n",
    "**TIMEMAPS VERSION**\n",
    "\n",
    "![Screenshots showing changes to the ABC Australia home page over time](images/abc-net-au.png)\n",
    "\n",
    "<p class=\"alert alert-info\">New to Jupyter notebooks? Try <a href=\"getting-started/Using_Jupyter_notebooks.ipynb\"><b>Using Jupyter notebooks</b></a> for a quick introduction.</p>\n",
    "\n",
    "This notebook helps you visualise changes in a web page by generating full page screenshots for each year from the captures available in an archive. You can then combine the individual screenshots into a single composite image.\n",
    "\n",
    "See [TimeMap Visualization](http://tmvis.ws-dl.cs.odu.edu/) from the Web Science and Digital Libraries Research Group at Old Dominion University for another way of using web archive thumbnails to explore change over time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating all the screenshots can be quite slow, and sometimes the captures themselves are incomplete, so I've divided the work into a number of stages:\n",
    "\n",
    "1. Create the individual screenshots\n",
    "2. Review the screenshot results and generate more screenshots if needed\n",
    "3. Create the composite image\n",
    "\n",
    "More details are below.\n",
    "\n",
    "If you want to create an individual screenshot, or compare a series of selected screenshots, see the [Get full page screenshots from archived web pages](save_screenshot.ipynb) notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import what we need"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import pandas as pd\n",
    "\n",
    "from selenium import webdriver\n",
    "import selenium\n",
    "import PIL\n",
    "from PIL import Image, ImageDraw, ImageFont\n",
    "import io\n",
    "import base64\n",
    "import time\n",
    "import re\n",
    "import json\n",
    "import math\n",
    "from slugify import slugify\n",
    "from webdriverdownloader import GeckoDriverDownloader\n",
    "from pathlib import Path\n",
    "from urllib.parse import urlparse\n",
    "\n",
    "gdd = GeckoDriverDownloader()\n",
    "geckodriver = gdd.download_and_install(\"v0.26.0\")[1]\n",
    "\n",
    "# See https://github.com/ouseful-template-repos/binder-selenium-demoscraper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define some functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "TIMEGATES = {\n",
    "    'nla': 'https://web.archive.org.au/awa/',\n",
    "    'nlnz': 'https://ndhadeliver.natlib.govt.nz/webarchive/wayback/',\n",
    "    'bl': 'https://www.webarchive.org.uk/wayback/archive/',\n",
    "    'ia': 'https://web.archive.org/web/',\n",
    "    'is': 'http://wayback.vefsafn.is/wayback/'\n",
    "}\n",
    "\n",
    "wayback = ['ndhadeliver.natlib.govt.nz', 'web.archive.org', 'wayback.vefsafn.is']\n",
    "pywb = {'web.archive.org.au': 'replayFrame', 'webarchive.nla.gov.au': 'replayFrame', 'webarchive.org.uk': 'replay_iframe'}\n",
    "\n",
    "html_output = []\n",
    "\n",
    "def convert_lists_to_dicts(results):\n",
    "    '''\n",
    "    Converts IA style timemap (a JSON array of arrays) to a list of dictionaries.\n",
    "    Renames keys to standardise IA with other Timemaps.\n",
    "    '''\n",
    "    if results:\n",
    "        keys = results[0]\n",
    "        results_as_dicts = [dict(zip(keys, v)) for v in results[1:]]\n",
    "    else:\n",
    "        results_as_dicts = results\n",
    "    # Rename keys\n",
    "    for d in results_as_dicts:\n",
    "        d['status'] = d.pop('statuscode')\n",
    "        d['mime'] = d.pop('mimetype')\n",
    "        d['url'] = d.pop('original')\n",
    "    return results_as_dicts\n",
    "\n",
    "def get_capture_data_from_memento(url, request_type='head'):\n",
    "    '''\n",
    "    For OpenWayback systems this can get some extra cpature info to insert in Timemaps.\n",
    "    '''\n",
    "    if request_type == 'head':\n",
    "        response = requests.head(url)\n",
    "    else:\n",
    "        response = requests.get(url)\n",
    "    headers = response.headers\n",
    "    length = headers.get('x-archive-orig-content-length')\n",
    "    status = headers.get('x-archive-orig-status')\n",
    "    status = status.split(' ')[0] if status else None\n",
    "    mime = headers.get('x-archive-orig-content-type')\n",
    "    mime = mime.split(';')[0] if mime else None\n",
    "    return {'length': length, 'status': status, 'mime': mime}\n",
    "\n",
    "def convert_link_to_json(results, enrich_data=False):\n",
    "    '''\n",
    "    Converts link formatted Timemap to JSON.\n",
    "    '''\n",
    "    data = []\n",
    "    for line in results.splitlines():\n",
    "        parts = line.split('; ')\n",
    "        if len(parts) > 1:\n",
    "            link_type = re.search(r'rel=\"(original|self|timegate|first memento|last memento|memento)\"', parts[1]).group(1)\n",
    "            if link_type == 'memento':\n",
    "                link = parts[0].strip('<>')\n",
    "                timestamp, original = re.search(r'/(\\d{14})/(.*)$', link).groups()\n",
    "                capture = {'timestamp': timestamp, 'url': original}\n",
    "                if enrich_data:\n",
    "                    capture.update(get_capture_data_from_memento(link))\n",
    "                data.append(capture)\n",
    "    return data\n",
    "                \n",
    "def get_timemap_as_json(timegate, url):\n",
    "    '''\n",
    "    Get a Timemap then normalise results (if necessary) to return a list of dicts.\n",
    "    '''\n",
    "    tg_url = f'{TIMEGATES[timegate]}timemap/json/{url}/'\n",
    "    print(tg_url)\n",
    "    response = requests.get(tg_url)\n",
    "    response_type = response.headers['content-type'].split(';')[0]\n",
    "    # pywb style Timemap\n",
    "    if response_type == 'text/x-ndjson':\n",
    "        data = [json.loads(line) for line in response.text.splitlines()]\n",
    "    # IA Wayback stype Timemap\n",
    "    elif response_type == 'application/json':\n",
    "        data = convert_lists_to_dicts(response.json())\n",
    "    # Link style Timemap (OpenWayback)\n",
    "    elif response_type in ['application/link-format', 'text/html']:\n",
    "        data = convert_link_to_json(response.text)\n",
    "    return data\n",
    "\n",
    "def get_full_page_screenshot(url, save_width=200):\n",
    "    '''\n",
    "    Gets a full page screenshot of the supplied url.\n",
    "    By default resizes the screenshot to a maximum width of 200px.\n",
    "    Provide a 'save_width' value to change this.\n",
    "    \n",
    "    NOTE the webdriver sometimes fails for unknown reasons. Just try again.\n",
    "    '''\n",
    "    global html_output\n",
    "    domain = urlparse(url)[1].replace('www.', '')\n",
    "    # NZ and IA inject content into the page, so we use if_ to get the original page (with rewritten urls)\n",
    "    if domain in wayback and 'if_' not in url:\n",
    "        url = re.sub(r'/(\\d{14})/http', r'/\\1if_/http', url)\n",
    "    print(url)\n",
    "    date_str, site = re.search(r'/(\\d{14})(?:if_|mp_)*/https*://(.+/)', url).groups()\n",
    "    ss_dir = Path('screenshots', slugify(site))\n",
    "    ss_dir.mkdir(parents=True, exist_ok=True)\n",
    "    ss_file = Path(ss_dir, f'{slugify(site)}-{date_str}-{save_width}.png')\n",
    "    if not ss_file.exists():\n",
    "        options = webdriver.FirefoxOptions()\n",
    "        options.headless = True\n",
    "        driver = webdriver.Firefox(executable_path=geckodriver, options=options)\n",
    "        driver.implicitly_wait(15)\n",
    "        driver.get(url)\n",
    "        # Give some time for everything to load\n",
    "        time.sleep(30)\n",
    "        driver.maximize_window()\n",
    "        current_width = driver.get_window_size()['width']\n",
    "        # UK and AU use pywb in framed replay mode, so we need to switch to the framed content\n",
    "        if domain in pywb:\n",
    "            try:\n",
    "                driver.switch_to.frame(pywb[domain])\n",
    "            except selenium.common.exceptions.NoSuchFrameException:\n",
    "                # If we pass here we'll probably still get a ss, just not full page -- better than failing?\n",
    "                pass\n",
    "        ss = None\n",
    "        for tag in ['body', 'html', 'frameset']:\n",
    "            try:\n",
    "                elem = driver.find_element_by_tag_name(tag)\n",
    "                ss = elem.screenshot_as_base64\n",
    "                break\n",
    "            except (selenium.common.exceptions.NoSuchElementException, selenium.common.exceptions.WebDriverException):\n",
    "                pass\n",
    "        driver.quit()\n",
    "        if not ss:\n",
    "            # show_error(f'Couldn\\'t get a screenshot of {url} – sorry...')\n",
    "            print(f'Couldn\\'t get a screenshot of {url} – sorry...')\n",
    "        else:\n",
    "            img = Image.open(io.BytesIO(base64.b64decode(ss)))\n",
    "            ratio = save_width / img.width\n",
    "            (width, height) = (save_width, math.ceil(img.height * ratio))\n",
    "            resized_img = img.resize((width, height), PIL.Image.LANCZOS)\n",
    "            resized_img.save(ss_file)\n",
    "            return ss_file\n",
    "    else:\n",
    "        return ss_file\n",
    "               \n",
    "def get_screenshots(timegate, url, num=1):\n",
    "    '''\n",
    "    Generate up to the specified number of screenshots for each year.\n",
    "    Queries Timemap for snapshots of the given url from the specified repository,\n",
    "    then gets the first 'num' timestamps for each year.\n",
    "    '''\n",
    "    data = get_timemap_as_json(timegate, url)\n",
    "    df = pd.DataFrame(data)\n",
    "    # Convert the timestamp string into a datetime object\n",
    "    df['date'] = pd.to_datetime(df['timestamp'])\n",
    "    # Sort by date\n",
    "    df.sort_values(by=['date'], inplace=True)\n",
    "    # Only keep the first instance of each digest if it exists\n",
    "    # OpenWayback systems won't return digests\n",
    "    try:\n",
    "        df.drop_duplicates(subset=['digest'], inplace=True)\n",
    "    except KeyError:\n",
    "        pass\n",
    "    # Extract year from date\n",
    "    df['year'] = df['date'].dt.year\n",
    "    # Get the first 'num' instances from each year \n",
    "    # (you only need one, but if there are failures, you might want a backup)\n",
    "    df_years = df.groupby('year', as_index=False).head(num)\n",
    "    timestamps = df_years['timestamp'].to_list()\n",
    "    for timestamp in timestamps:\n",
    "        capture_url = f'{TIMEGATES[timegate]}{timestamp}/{url}'\n",
    "        capture_url = f'{capture_url}/' if not capture_url.endswith('/') else capture_url\n",
    "        print(f'Generating screenshot for: {capture_url}...')\n",
    "        get_full_page_screenshot(capture_url)\n",
    "        \n",
    "def make_composite(url):\n",
    "    '''\n",
    "    Combine single screenshots into a composite image.\n",
    "    Loops through images in a directory with the given (slugified) domain.\n",
    "    '''\n",
    "    max_height = 0\n",
    "    url_path = re.sub(r'^https*://', '', url)\n",
    "    pngs = sorted(Path('screenshots', slugify(url_path)).glob('*.png'))\n",
    "    for png in pngs:\n",
    "        img = Image.open(png)\n",
    "        if img.height > max_height:\n",
    "            max_height = img.height\n",
    "    height = max_height + 110\n",
    "    width = (len(pngs) * 200) + (len(pngs) * 10) + 10\n",
    "    comp = Image.new('RGB', (width, height), (90,90,90))\n",
    "    # Canvas to write in the dates\n",
    "    draw = ImageDraw.Draw(comp)\n",
    "    # Change this to suit your system\n",
    "    # font = ImageFont.truetype(\"/Library/Fonts/Microsoft/Gill Sans MT Bold.ttf\", 36)\n",
    "    # Something like this should work on Binder?\n",
    "    font = ImageFont.truetype(\"/usr/share/fonts/truetype/liberation/LiberationSansNarrow-Regular.ttf\", 36)\n",
    "    draw.text((10, height - 50), url, (255,255,255), font=font)\n",
    "    for i, png in enumerate(pngs):\n",
    "        year = re.search(r'-(\\d{4})\\d{10}.*?\\.png', png.name).group(1)\n",
    "        draw.text(((i * 210) + 10, 10), year, (255,255,255), font=font)\n",
    "        img = Image.open(png)\n",
    "        comp.paste(img, ((i * 210) + 10, 50))\n",
    "    comp.save(Path('screenshots', f'{slugify(url_path)}.png'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create screenshots\n",
    "\n",
    "Edit the cell below to change the `timegate` and `url` values as desired. Timegate values can be one of:\n",
    "\n",
    "* `bl` – UK Web Archive\n",
    "* `nla` – National Library of Australia\n",
    "* `nlnz` – National Library of New Zealand\n",
    "* `ia` – Internet Archive\n",
    "* `is` – Icelandic Web Archive\n",
    "\n",
    "The `num` parameter is the maximum number of screenshots to create for each year. Of course you only need one screenshot per year for the composite image, but getting two is allows you to select the best capture before generating the composite.\n",
    "\n",
    "Screenshots will be saved in a sub-directory of the [screenshots](screenshots) directory. The name of the sub-directory will a slugified version of the url. Screenshots are named with the `url`, `timestamp` and width (which is set at 200 in this notebook). Existing screenshots will not be overwritten, so if the process fails you can easily pick up where it left off by just running the cell again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://web.archive.org/web/timemap/json/https://www.nlm.nih.gov//\n",
      "Generating screenshot for: https://web.archive.org/web/19970607005150/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/19970607005150if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/19980205174512/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/19980205174512if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/19990117033527/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/19990117033527if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20000302064951/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20000302064951if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20010118205700/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20010118205700if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20020328190748/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20020328190748if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20030505132514/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20030505132514if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20040315132254/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20040315132254if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20050104085638/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20050104085638if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20060104051554/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20060104051554if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20070105123032/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20070105123032if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20080102180203/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20080102180203if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20090102210721/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20090102210721if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20100101021819/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20100101021819if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20110106123427/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20110106123427if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20120112144457/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20120112144457if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20130118154416/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20130118154416if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20140115182923/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20140115182923if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20150112214136/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20150112214136if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20160108030113/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20160108030113if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20170106220316/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20170106220316if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20180104174258/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20180104174258if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20190104000829/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20190104000829if_/https://www.nlm.nih.gov/\n",
      "Generating screenshot for: https://web.archive.org/web/20200108023740/https://www.nlm.nih.gov/...\n",
      "https://web.archive.org/web/20200108023740if_/https://www.nlm.nih.gov/\n"
     ]
    }
   ],
   "source": [
    "# Edit timegate and url values\n",
    "# TRY #1\n",
    "# get_screenshots(timegate='ia', url='https://coronavirus.jhu.edu/', num=1000)\n",
    "# TRY #2\n",
    "# get_screenshots(timegate='ia', url='https://coronavirus.jhu.edu/data/new-cases-50-states', num=1)\n",
    "# TRY #3\n",
    "get_screenshots(timegate='ia', url='https://www.nlm.nih.gov/', num=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Review Screenshots\n",
    "\n",
    "Have a look at the results in the [screenshots](screenshots) directory. If you've created more than one per year, choose the best one and delete the others. If you're not satisfied with a particular capture, you can browse the web interface of the archive until you find the capture you're after. Then just copy the url of the capture (it should have a timestamp in it) and feed it directly to the `get_full_page_screenshot()` function as indicated in the cell below. Once you're happy with your collection of screenshots, move on to the next step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use this to add individual captures to fill gaps, or improve screenshots\n",
    "# Just paste in a url from the archive's web interface\n",
    "# OPTIONAL get_full_page_screenshot('http://wayback.vefsafn.is/wayback/20200107111708/https://www.government.is/')\n",
    "# TRY #2\n",
    "# get_full_page_screenshot('https://web.archive.org/web/20200530080723/https://coronavirus.jhu.edu/data/new-cases-50-states/', save_width=1200)\n",
    "# get_full_page_screenshot('https://web.archive.org/web/20200613154650/https://coronavirus.jhu.edu/data/new-cases-50-states/', save_width=1200)\n",
    "# get_full_page_screenshot('https://web.archive.org/web/20200627224514/https://coronavirus.jhu.edu/data/new-cases-50-states/', save_width=1200)\n",
    "# get_full_page_screenshot('https://web.archive.org/web/20200711204459/https://coronavirus.jhu.edu/data/new-cases-50-states/', save_width=1200)\n",
    "# get_full_page_screenshot('https://web.archive.org/web/20200725072550/https://coronavirus.jhu.edu/data/new-cases-50-states/', save_width=1200)\n",
    "# get_full_page_screenshot('https://web.archive.org/web/20200801224932/https://coronavirus.jhu.edu/data/new-cases-50-states/', save_width=1200)\n",
    "# TRY #3\n",
    "# get_full_page_screenshot('https://web.archive.org/web/20190401120425/https://www.nlm.nih.gov/') # ERROR\n",
    "# get_full_page_screenshot('https://web.archive.org/web/20190501083035/https://www.nlm.nih.gov/') # OK\n",
    "# get_full_page_screenshot('https://web.archive.org/web/20190601120242/https://www.nlm.nih.gov/') # OK\n",
    "# get_full_page_screenshot('https://web.archive.org/web/20190701102433/https://www.nlm.nih.gov/') # OK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create composite\n",
    "\n",
    "Edit the cell below to include the url that you've generated the screenshots from. The script will grap all of the screenshots in the corrresponding directory and build the composite image. The composite will be saved in the `screenshots` directory using a slugified version of the url as a name.\n",
    "\n",
    "If you're running this on your own machine, you'll probably need to edit the function to set the font location where indicated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Edit url as required\n",
    "# make_composite('https://coronavirus.jhu.edu/')\n",
    "# TRY #2\n",
    "# make_composite('https://coronavirus.jhu.edu/data/new-cases-50-states')\n",
    "# TRY #3\n",
    "make_composite('https://www.nlm.nih.gov/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "Created by [Tim Sherratt](https://timsherratt.org) for the [GLAM Workbench](https://glam-workbench.github.io).\n",
    "\n",
    "Work on this notebook was supported by the [IIPC Discretionary Funding Programme 2019-2020](http://netpreserve.org/projects/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}