Your-first-API-request.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Your first Trove API request\n",
    "\n",
    "<p class=\"alert alert-info\">New to Jupyter notebooks? Try <a href=\"getting-started/Using_Jupyter_notebooks.ipynb\"><b>Using Jupyter notebooks</b></a> for a quick introduction.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we're going to learn how to send a request for information to the Trove API.\n",
    "\n",
    "API requests are just like normal urls. However, instead of sending us back a web page, they deliver data in a form that computers can understand. We can then use that data in our own programs and pipelines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to use the Python [Requests](http://docs.python-requests.org/en/master/) module to handle our API queries, so let's import it now with a few other things we'll need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "import os\n",
    "\n",
    "# We're going to use the Python Requests module to handle our API queries\n",
    "import requests\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "# We'll use this to display nice;y formatted JSON results\n",
    "from IPython.display import JSON\n",
    "\n",
    "load_dotenv()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting an API key\n",
    "Any requests you make to the Trove API need to be authenticated with a 'key'. For non-commercial projects, you just  fill out a simple form and your API key is generated instantly. Follow the instructions in the Trove Help to [obtain your own Trove API Key](http://help.nla.gov.au/trove/building-with-trove/api).\n",
    "\n",
    "Once you've created a key, you can access it at any time on the 'For developers' tab of your Trove user profile.\n",
    "\n",
    "Copy your API key now, and paste it in the cell below, between the quotes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Your API key is: gq29l1g1h75pimh4\n"
     ]
    }
   ],
   "source": [
    "# This creates a variable called 'api_key', paste your key between the quotes\n",
    "API_KEY = \"INSERT YOUR API KEY HERE\"\n",
    "\n",
    "# Leave these lines as they are.\n",
    "# Use an api key value from environment variables if it is available (useful for testing)\n",
    "if os.getenv(\"TROVE_API_KEY\"):\n",
    "    API_KEY = os.getenv(\"TROVE_API_KEY\")\n",
    "\n",
    "# This displays a message with your key\n",
    "print(\"Your API key is: {}\".format(API_KEY))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All search queries to the Trove API start with the same base url. We'll save it as a variable here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a variable called 'api_search_url' and give it a value\n",
    "api_search_url = \"https://api.trove.nla.gov.au/v3/result\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Trove API queries are constructed by adding parameters to the base url. All of the parameters are optional, except for:\n",
    "\n",
    "* `category` &ndash; which Trove category (or categories) do you want to search, use `all` for everything\n",
    "\n",
    "You'll often want to supply a search query:\n",
    "\n",
    "* `q` &ndash; 'q' for query, this is where search terms go, if you don't supply a `q` value you'll get everything\n",
    "\n",
    "You might also want to specify the format in which the results are delivered. The default is `xml`, but for most applications you'll probably find it easier to work with `json`.\n",
    "\n",
    "* `encoding` &ndash; the format of the results, this can be set to either `xml` or `json` (`xml` is the default.\n",
    "\n",
    "We'll meet some other parameters later, but for now let's create a Python dictionary to store our basic parameters. The `requests` library will take this dictionary, turn it into a string, and add it to the base url.\n",
    "\n",
    "For our first API request we're going to search Trove's digitised newspapers, so we'll assign the value 'newspaper' to the `category` parameter. Feel free to edit the `q` value to search for something that interests you.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This creates a dictionary called 'params' and sets values for the API's mandatory parameters\n",
    "params = {\n",
    "    \"q\": \"cyclone\",  # Search for this keyword -- feel free to change!\n",
    "    \"category\": \"newspaper\",  # Search in the newspaper category\n",
    "    \"encoding\": \"json\",\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You supply your API key using `headers`. These are extra, hidden parameters that describe your request to the server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add your API key to the request headers\n",
    "headers = {\"X-API-KEY\": API_KEY}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making a request"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, we're now now ready to make our first query!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "API url: https://api.trove.nla.gov.au/v3/result?q=cyclone&category=newspaper&encoding=json\n"
     ]
    }
   ],
   "source": [
    "# This sends our request to the Trove API and stores the result in a variable called 'response'\n",
    "response = requests.get(api_search_url, params=params, headers=headers)\n",
    "\n",
    "# This shows us the url that's sent to the API\n",
    "print(f\"API url: {response.url}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See how `requests` has taken our parameters and turned them into a string with '&' between each one? \n",
    "\n",
    "The url above is live &ndash; try clicking on it to see the raw results from Trove."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Look at the results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `response` variable contains all the data returned to us by the Trove API. Let's get it out in a usable form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the Trove API's JSON results and make them available as a Python variable called 'data'\n",
    "data = response.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/json": {
       "category": [
        {
         "code": "newspaper",
         "name": "Newspapers & Gazettes",
         "records": {
          "article": [
           {
            "category": "Article",
            "date": "1872-03-23",
            "heading": "CYCLONE.",
            "id": "111525457",
            "page": "3",
            "pageSequence": "3",
            "relevance": {
             "score": 237.76849365234375,
             "value": "very relevant"
            },
            "snippet": "A cyclone, as the derivation of the Word implies, is a storm which travels in a circle. Cyclones are mereiy whirlwirds on a gigantic scale, being enormous volumes of air driven",
            "title": {
             "id": "386",
             "title": "The Herald (Fremantle, WA : 1867 - 1886)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article111525457?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/111525457"
           },
           {
            "category": "Article",
            "date": "1954-03-11",
            "heading": "CYCLONE",
            "id": "147162532",
            "page": "18",
            "pageSequence": "18",
            "relevance": {
             "score": 237.76849365234375,
             "value": "very relevant"
            },
            "snippet": "The \"Man on the air\" said: \"The storm is bad; The wind is travelling fast.\" I can hear it howling]",
            "title": {
             "id": "711",
             "title": "Catholic Weekly (Sydney, NSW : 1942 - 1954)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article147162532?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/147162532"
           },
           {
            "category": "Article",
            "date": "1934-01-20",
            "heading": "Cyclone!",
            "id": "277285161",
            "page": "7",
            "pageSequence": "7",
            "relevance": {
             "score": 237.76849365234375,
             "value": "very relevant"
            },
            "snippet": "Algy: When I found myself in debt I went to the old man to raise the wind. Alfy: Did you?",
            "status": "coming soon",
            "title": {
             "id": "1191",
             "title": "The Sun News-Pictorial (Melbourne, Vic. : 1922 - 1954; 1956)"
            },
            "url": "https://api.trove.nla.gov.au/v3/newspaper/277285161"
           },
           {
            "category": "Article",
            "date": "1976-02-18",
            "heading": "Cyclone",
            "id": "110803384",
            "page": "16",
            "pageSequence": "16",
            "relevance": {
             "score": 237.6962890625,
             "value": "very relevant"
            },
            "snippet": "BRISBANE, TUESDAY. —The Brisbane Weather Bureau has dropped cyclone warnings for the North",
            "title": {
             "id": "11",
             "title": "The Canberra Times (ACT : 1926 - 1995)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article110803384?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/110803384"
           },
           {
            "category": "Article",
            "date": "1949-01-20",
            "heading": "CYCLONE",
            "id": "113436680",
            "page": "3",
            "pageSequence": "3",
            "relevance": {
             "score": 237.6962890625,
             "value": "very relevant"
            },
            "snippet": "Residents along the Bimbi road who experienced the bud storm on Saturday inform us that the disturbance in that particular area",
            "title": {
             "id": "474",
             "title": "The Grenfell Record and Lachlan District Advertiser (NSW : 1876 - 1951)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article113436680?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/113436680"
           },
           {
            "category": "Article",
            "date": "1906-11-28",
            "heading": "Cyclone.",
            "id": "189663643",
            "page": "4",
            "pageSequence": "4",
            "relevance": {
             "score": 237.6962890625,
             "value": "very relevant"
            },
            "snippet": "A destructive cyclone swept over Mogara, West Australia, on Saturday night. Dwellings were levelled in all directions, and the Miners Institute",
            "title": {
             "id": "983",
             "title": "The Armidale Chronicle (NSW : 1894 - 1929)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article189663643?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/189663643"
           },
           {
            "category": "Article",
            "date": "1892-02-26",
            "heading": "CYCLONE.",
            "id": "260601160",
            "page": "3",
            "pageSequence": "3",
            "relevance": {
             "score": 237.6962890625,
             "value": "very relevant"
            },
            "snippet": "Brisbane, February 25— A destructive cyclone is reported from Port Douglas, and a fire has taken place at Bowen.",
            "title": {
             "id": "1606",
             "title": "The Australian Advertiser (Albany, WA : 1888 - 1897)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article260601160?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/260601160"
           },
           {
            "category": "Article",
            "date": "1882-02-08",
            "heading": "Cyclone.",
            "id": "169527399",
            "page": "2",
            "pageSequence": "2",
            "relevance": {
             "score": 237.5968017578125,
             "value": "very relevant"
            },
            "snippet": "THE cyclone, mentioned in our telegraphic columns as having done so much damage at Cardwell, seems to have raged with terrific violence. The S.S. Gunga, Captain S. Clark,",
            "title": {
             "id": "835",
             "title": "Mackay Mercury and South Kennedy Advertiser (Qld. : 1867 - 1887)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article169527399?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/169527399"
           },
           {
            "category": "Article",
            "date": "1996-10-18",
            "heading": "Cyclone",
            "id": "261023732",
            "page": "4",
            "pageSequence": "4 S",
            "relevance": {
             "score": 237.5968017578125,
             "value": "very relevant"
            },
            "snippet": "One late afternoon on a warm peaceful day Out of the blue a tree began to sway. The other trees of the village were all swaying too, And before we knew it, the sky was not blue.",
            "title": {
             "id": "1685",
             "title": "The Australian Jewish News (Melbourne, Vic. : 1935 - 1999)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article261023732?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/261023732"
           },
           {
            "category": "Article",
            "date": "1977-01-12",
            "heading": "Cyclone",
            "id": "110834707",
            "page": "3",
            "pageSequence": "3",
            "relevance": {
             "score": 237.455322265625,
             "value": "very relevant"
            },
            "snippet": "PERTH, Tuesday (AAP).—Cyclone Irene has moved away from the Western Australian coast.",
            "title": {
             "id": "11",
             "title": "The Canberra Times (ACT : 1926 - 1995)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article110834707?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/110834707"
           },
           {
            "category": "Article",
            "date": "1893-09-01",
            "heading": "CYCLONE.",
            "id": "80030164",
            "page": "2",
            "pageSequence": "2",
            "relevance": {
             "score": 237.455322265625,
             "value": "very relevant"
            },
            "snippet": "A terrible cyclone, followed by a tidal were,has [?]stated the eastara cosat of America. Charleston is almost in rums, and Port Royal, in South",
            "title": {
             "id": "270",
             "title": "Zeehan and Dundas Herald (Tas. : 1890 - 1922)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article80030164?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/80030164"
           },
           {
            "category": "Article",
            "date": "1969-02-11",
            "heading": "CYCLONE",
            "id": "107077865",
            "page": "6",
            "pageSequence": "6",
            "relevance": {
             "score": 237.25604248046875,
             "value": "very relevant"
            },
            "snippet": "TANANARIVE, Monday (AAP-Reuter). —Three people died, eight are missing and more than",
            "title": {
             "id": "11",
             "title": "The Canberra Times (ACT : 1926 - 1995)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article107077865?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/107077865"
           },
           {
            "category": "Article",
            "date": "1904-03-23",
            "heading": "CYCLONE.",
            "id": "143904027",
            "page": "2",
            "pageSequence": "2",
            "relevance": {
             "score": 237.25604248046875,
             "value": "very relevant"
            },
            "snippet": "A cyclone swept over the town of Forster on Monday evening, and torrents of rain fell, doing much damage to the township, trees being uprooted,",
            "title": {
             "id": "639",
             "title": "Cootamundra Herald (NSW : 1877 - 1954)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article143904027?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/143904027"
           },
           {
            "category": "Article",
            "date": "1924-10-30",
            "heading": "Cyclone.",
            "id": "121648670",
            "page": "2",
            "pageSequence": "2",
            "relevance": {
             "score": 237.1483154296875,
             "value": "very relevant"
            },
            "snippet": "LUCKILY for Grenfell, the full force of the storm experienced on Tuesday did not reach the to we. From reports from Glenelg and across to Bald Hills",
            "title": {
             "id": "474",
             "title": "The Grenfell Record and Lachlan District Advertiser (NSW : 1876 - 1951)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article121648670?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/121648670"
           },
           {
            "category": "Article",
            "date": "1950-06-18",
            "heading": "CYCLONE",
            "id": "59517703",
            "page": "3",
            "pageSequence": "3",
            "relevance": {
             "score": 237.1483154296875,
             "value": "very relevant"
            },
            "snippet": "Calcutta, Today: Death roll in the cyclonic storm over West Bengal during the last few days rose to 89",
            "title": {
             "id": "93",
             "title": "Sunday Times (Perth, WA : 1902 - 1954)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article59517703?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/59517703"
           },
           {
            "category": "Article",
            "date": "1893-07-12",
            "heading": "CYCLONE.",
            "id": "80024154",
            "page": "2",
            "pageSequence": "2",
            "relevance": {
             "score": 237.1483154296875,
             "value": "very relevant"
            },
            "snippet": "A cyclone has occurred at Lake Michigan through which a number of boats were swamped and 30 people drowned. The Chicago Exhibition was",
            "title": {
             "id": "270",
             "title": "Zeehan and Dundas Herald (Tas. : 1890 - 1922)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article80024154?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/80024154"
           },
           {
            "category": "Article",
            "date": "1900-04-13",
            "heading": "CYCLONE.",
            "id": "100756186",
            "page": "2",
            "pageSequence": "2",
            "relevance": {
             "score": 237.11956787109375,
             "value": "very relevant"
            },
            "snippet": "A WINDSTROM of cyclonic force occurred at Mullaley on Wednesday, and was most severely felt along the Gunnedah-Coonabara-bran-road. The post-office at Mullaley was",
            "title": {
             "id": "367",
             "title": "Goulburn Herald (NSW : 1881 - 1907)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article100756186?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/100756186"
           },
           {
            "category": "Article",
            "date": "1912-11-07",
            "edition": "WEEKLY.",
            "heading": "Cyclone.",
            "id": "66181696",
            "page": "2",
            "pageSequence": "2 S",
            "relevance": {
             "score": 237.11956787109375,
             "value": "very relevant"
            },
            "snippet": "About 3 a.m. on Thursday, a cyclone disturbance passed over this district, and, whilst no loss of human life occurred, in many instances damage was done to",
            "title": {
             "id": "208",
             "title": "South Bourke and Mornington Journal (Richmond, Vic. : 1877 - 1920; 1926 - 1927)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article66181696?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/66181696"
           },
           {
            "category": "Article",
            "date": "1980-01-05",
            "heading": "Cyclone",
            "id": "137006546",
            "page": "3",
            "pageSequence": "3",
            "relevance": {
             "score": 237.04788208007812,
             "value": "very relevant"
            },
            "snippet": "BRISBANE: The Brisbane Tropical Cyclone Warning Centre issued a cyclone watch message yesterday for gulf coastal areas from Aurukun to Port",
            "title": {
             "id": "11",
             "title": "The Canberra Times (ACT : 1926 - 1995)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article137006546?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/137006546"
           },
           {
            "category": "Article",
            "date": "1906-12-06",
            "heading": "Cyclone",
            "id": "173242189",
            "page": "2",
            "pageSequence": "2",
            "relevance": {
             "score": 237.04788208007812,
             "value": "very relevant"
            },
            "snippet": "A destructive cyclone struck the vicinity of the Guyra railway station yesterday afternoon and [?]ked a shed opposite Mr Whyburn's residence. Sheets of corrugated",
            "title": {
             "id": "888",
             "title": "Guyra Argus (NSW : 1902 - 1954)"
            },
            "troveUrl": "https://nla.gov.au/nla.news-article173242189?searchTerm=cyclone",
            "url": "https://api.trove.nla.gov.au/v3/newspaper/173242189"
           }
          ],
          "n": 20,
          "next": "https://api.trove.nla.gov.au/v3/result?q=cyclone&category=newspaper&encoding=json&s=AoIIQ20MQikxNzMyNDIxODk%3D",
          "nextStart": "AoIIQ20MQikxNzMyNDIxODk=",
          "s": "*",
          "total": 658101
         }
        }
       ],
       "query": "cyclone"
      },
      "text/plain": [
       "<IPython.core.display.JSON object>"
      ]
     },
     "execution_count": 24,
     "metadata": {
      "application/json": {
       "expanded": true,
       "root": "root"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display the results as nicely-formatted JSON\n",
    "JSON(data, expanded=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many results are there? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "658101"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[\"category\"][0][\"records\"][\"total\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Do something with the data\n",
    "\n",
    "As you can see, the API results are fairly complex. Individual item records are quite deeply nested. Let's run a simple script to display the basic details of each of our matching articles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loop through all the newspaper articles\n",
    "# The articles themselves are quite deeply nested, so we have to go down several levels to get them\n",
    "for article in data[\"category\"][0][\"records\"][\"article\"]:\n",
    "    # Display a string containing the date, title, newspaper, and page for each article\n",
    "    print(\n",
    "        f'{article[\"date\"]}, \"{article[\"heading\"]}\", {article[\"title\"][\"title\"]}, page {article[\"page\"]}'\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations!\n",
    "\n",
    "You've made your first Trove API request. Now let's move on to learn a bit [about Trove's zones](Working-with-zones.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "Created by [Tim Sherrratt](https://timsherratt.org) for the [GLAM workbench](https://glam-workbench.net/). Support this project by [becoming a GitHub sponsor](https://github.com/sponsors/wragge?o=esb)."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}