graphql_api.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "76fa365c-b6d9-479f-99d8-d0d53ec30e8e",
   "metadata": {},
   "source": [
    "# Querying Metadata with GraphQL\n",
    "\n",
    "In this notebook we explore searching for metadata from the GraphQL API. The GraphQL API provides a method to programmatically extract a JSON representation of the meta data from the API. It has the advantage over REST that it can prevent under and over fetching data.\n",
    "\n",
    "First we load some python dependancies that we will use as part of this notebook and set the variable `ENDPOINT_URL` to the location of the GraphQL API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ebba03b7-34c4-4b49-94d8-112c7176ac7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml\n",
    "import pandas as pd\n",
    "\n",
    "with open('config.yml') as file_handle:\n",
    "    config = yaml.load(file_handle, yaml.FullLoader)\n",
    "\n",
    "ENDPOINT_URL = config['graphql_api']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22ff9d86",
   "metadata": {},
   "source": [
    "Setup a `gql` client. This is used to query the graphql API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "bc504ba6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from gql import gql, Client\n",
    "from gql.transport.requests import RequestsHTTPTransport\n",
    "\n",
    "# Select your transport with a defined url endpoint\n",
    "transport = RequestsHTTPTransport(url=ENDPOINT_URL)\n",
    "# Create a GraphQL client using the defined transport\n",
    "client = Client(transport=transport, fetch_schema_from_transport=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a9a779a-3cc2-4557-8cb1-a9849f01a434",
   "metadata": {},
   "source": [
    "## Querying Shots with GraphQL\n",
    "\n",
    "With GraphQL you can query exactly what you want, rather than having to recieve the whole table from the database This is useful in cases where the whole table has many columns, but you are interested in just a subset of them.\n",
    "\n",
    "The GraphQL endpoint is located at `/graphql`. You can find the documentation and an interactive query explorer at the URL below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "dfb0f1c7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GraphQL API endpoint: http://209.97.185.27:8081/graphql\n"
     ]
    }
   ],
   "source": [
    "print(f\"GraphQL API endpoint: {ENDPOINT_URL}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa9ab9b8",
   "metadata": {},
   "source": [
    "Unlike the REST API which uses HTTP `GET` requests to return data, with GraphQL we use HTTP `POST` to post our query to the API.\n",
    "\n",
    "Here is a simple example of getting some shot data from the GraphQL API. We need to explicity state what information we want to return from the API. Here we are asking for:\n",
    " - the shot ID\n",
    " - the timestamp that the shot was taken\n",
    " - the preshot description\n",
    " - the divertor configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4970f970-2e6a-414a-8722-427f57c62964",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shot_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>preshot_description</th>\n",
       "      <th>divertor_config</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11695</td>\n",
       "      <td>2004-12-13T11:54:00</td>\n",
       "      <td>\\n0.1T TF SHOT\\n</td>\n",
       "      <td>conventional</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11696</td>\n",
       "      <td>2004-12-13T12:07:00</td>\n",
       "      <td>\\nSTANDARD 0.3T TF SHOT\\n</td>\n",
       "      <td>conventional</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11697</td>\n",
       "      <td>2004-12-13T12:19:00</td>\n",
       "      <td>\\nRAISE TO 0.5T\\n</td>\n",
       "      <td>conventional</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11698</td>\n",
       "      <td>2004-12-13T12:31:00</td>\n",
       "      <td>\\nRAISE TO .56T\\n</td>\n",
       "      <td>conventional</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>11699</td>\n",
       "      <td>2004-12-13T12:45:00</td>\n",
       "      <td>\\nRAISE TO .58T\\n</td>\n",
       "      <td>conventional</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   shot_id            timestamp        preshot_description divertor_config\n",
       "0    11695  2004-12-13T11:54:00           \\n0.1T TF SHOT\\n    conventional\n",
       "1    11696  2004-12-13T12:07:00  \\nSTANDARD 0.3T TF SHOT\\n    conventional\n",
       "2    11697  2004-12-13T12:19:00          \\nRAISE TO 0.5T\\n    conventional\n",
       "3    11698  2004-12-13T12:31:00          \\nRAISE TO .56T\\n    conventional\n",
       "4    11699  2004-12-13T12:45:00          \\nRAISE TO .58T\\n    conventional"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Write our GraphQL query.\n",
    "query = gql(\"\"\"\n",
    "query {\n",
    "    all_shots  {\n",
    "        shots {\n",
    "            shot_id\n",
    "            timestamp\n",
    "            preshot_description\n",
    "            divertor_config\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\"\"\")\n",
    "\n",
    "# # Query the API and get a JSON response\n",
    "result = client.execute(query)\n",
    "shots = result['all_shots']['shots']\n",
    "df = pd.DataFrame(shots)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c8a11b9",
   "metadata": {},
   "source": [
    "### Searching & Filtering Data\n",
    "We can also supply query parameters to GraphQL, such as limiting the number of returned values or filtering by value. Here we are limiting the first 3 values and we are selcting only shots from the M9 campaign."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "399c75c1-16ab-4a34-b968-5a9f3c6f99e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>shot_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>preshot_description</th>\n",
       "      <th>divertor_config</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>28390</td>\n",
       "      <td>2012-03-06T14:47:00</td>\n",
       "      <td>\\nBC5, 300 ms, 3 V. D2 plenum 1536 mbar. For r...</td>\n",
       "      <td>conventional</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>28391</td>\n",
       "      <td>2012-03-06T14:52:00</td>\n",
       "      <td>\\nBC5, 300 ms, 5 V. D2 plenum 1536 mbar. For r...</td>\n",
       "      <td>conventional</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>28392</td>\n",
       "      <td>2012-03-06T15:03:00</td>\n",
       "      <td>\\nHL11, 300 ms, 2 V. He plenum 1047.\\n</td>\n",
       "      <td>conventional</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   shot_id            timestamp  \\\n",
       "0    28390  2012-03-06T14:47:00   \n",
       "1    28391  2012-03-06T14:52:00   \n",
       "2    28392  2012-03-06T15:03:00   \n",
       "\n",
       "                                 preshot_description divertor_config  \n",
       "0  \\nBC5, 300 ms, 3 V. D2 plenum 1536 mbar. For r...    conventional  \n",
       "1  \\nBC5, 300 ms, 5 V. D2 plenum 1536 mbar. For r...    conventional  \n",
       "2             \\nHL11, 300 ms, 2 V. He plenum 1047.\\n    conventional  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Write our GraphQL query.\n",
    "query = gql(\"\"\"\n",
    "query ($campaign: String!) {\n",
    "    all_shots (limit: 3, where: {campaign: {eq: $campaign}}) {\n",
    "        shots {\n",
    "            shot_id\n",
    "            timestamp\n",
    "            preshot_description\n",
    "            divertor_config\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\"\"\")\n",
    "# Query the API and get a JSON response\n",
    "result = client.execute(query, variable_values={\"campaign\": \"M9\"})\n",
    "df = pd.DataFrame(result['all_shots']['shots'])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5aa79e2b",
   "metadata": {},
   "source": [
    "### Nested Queries\n",
    "\n",
    "One feature which makes GraphQL much more powerful that REST is that you may perform nested queries to gather different subsets of the data. For example, here we are going to query for all datasets with \"AMC\" in the name, and query for information about shots associated with this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "88bdab08",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'all_signal_datasets': {'signal_datasets': [{'name': 'AMC_P3U_COIL_CURRENT',\n",
       "    'url': 's3://mast/AMC_P3U_COIL_CURRENT.zarr',\n",
       "    'shots': [{'shot_id': 28412,\n",
       "      'timestamp': '2013-03-06T15:13:00',\n",
       "      'divertor_config': 'conventional'},\n",
       "     {'shot_id': 28415,\n",
       "      'timestamp': '2013-03-07T15:11:00',\n",
       "      'divertor_config': 'conventional'},\n",
       "     {'shot_id': 28416,\n",
       "      'timestamp': '2013-03-07T15:27:00',\n",
       "      'divertor_config': 'conventional'}]},\n",
       "   {'name': 'AMC_P2OL_FEED_CURRENT',\n",
       "    'url': 's3://mast/AMC_P2OL_FEED_CURRENT.zarr',\n",
       "    'shots': [{'shot_id': 23819,\n",
       "      'timestamp': '2009-12-09T10:04:00',\n",
       "      'divertor_config': 'conventional'},\n",
       "     {'shot_id': 23820,\n",
       "      'timestamp': '2009-12-09T10:24:00',\n",
       "      'divertor_config': 'conventional'},\n",
       "     {'shot_id': 23821,\n",
       "      'timestamp': '2009-12-09T10:48:00',\n",
       "      'divertor_config': 'conventional'}]},\n",
       "   {'name': 'AMC_ERROR_FIELD_05',\n",
       "    'url': 's3://mast/AMC_ERROR_FIELD_05.zarr',\n",
       "    'shots': [{'shot_id': 29516,\n",
       "      'timestamp': '2013-07-22T14:32:00',\n",
       "      'divertor_config': 'conventional'},\n",
       "     {'shot_id': 28412,\n",
       "      'timestamp': '2013-03-06T15:13:00',\n",
       "      'divertor_config': 'conventional'},\n",
       "     {'shot_id': 28415,\n",
       "      'timestamp': '2013-03-07T15:11:00',\n",
       "      'divertor_config': 'conventional'}]}]}}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Write our GraphQL query.\n",
    "query = gql(\"\"\"\n",
    "query ($signal: String!) {\n",
    "    all_signal_datasets (limit: 3, where: {name: {contains: $signal}}) {\n",
    "        signal_datasets {\n",
    "          name\n",
    "          url\n",
    "          shots (limit:3) {\n",
    "            shot_id\n",
    "            timestamp\n",
    "            divertor_config\n",
    "          }\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\"\"\")\n",
    "# Query the API and get a JSON response\n",
    "client.execute(query, variable_values={\"signal\": \"AMC\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e2ac419-db72-4a95-88a0-d5226b399376",
   "metadata": {},
   "source": [
    "### Pagination in GraphQL\n",
    "\n",
    "GraphQL queries are paginated. You may access other entries by including the page metadata and it's associated elements. Here's an example of getting paginated entries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "f4566e66-215c-4d41-aa3a-44fb9d6a0447",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'all_shots': {'shots': [{'shot_id': 28390, 'timestamp': '2012-03-06T14:47:00', 'preshot_description': '\\nBC5, 300 ms, 3 V. D2 plenum 1536 mbar. For reference.\\n', 'divertor_config': 'conventional'}, {'shot_id': 28391, 'timestamp': '2012-03-06T14:52:00', 'preshot_description': '\\nBC5, 300 ms, 5 V. D2 plenum 1536 mbar. For reference.\\n', 'divertor_config': 'conventional'}, {'shot_id': 28392, 'timestamp': '2012-03-06T15:03:00', 'preshot_description': '\\nHL11, 300 ms, 2 V. He plenum 1047.\\n', 'divertor_config': 'conventional'}], 'page_meta': {'next_cursor': 'Mw==', 'total_items': 2082, 'total_pages': 694}}}\n",
      "{'all_shots': {'shots': [{'shot_id': 28393, 'timestamp': '2012-03-06T15:09:00', 'preshot_description': '\\nHL11, 300 ms, 3 V. He plenum 1047.\\n', 'divertor_config': 'conventional'}, {'shot_id': 28394, 'timestamp': '2012-03-06T15:13:00', 'preshot_description': '\\nHL11, 300 ms, 5 V. He plenum 1047.\\n', 'divertor_config': 'conventional'}, {'shot_id': 28395, 'timestamp': '2012-03-06T15:21:00', 'preshot_description': '\\nHL11, 300 ms, 6 V. He plenum 1047.\\n', 'divertor_config': 'conventional'}], 'page_meta': {'next_cursor': 'Ng==', 'total_items': 2082, 'total_pages': 694}}}\n",
      "{'all_shots': {'shots': [{'shot_id': 28396, 'timestamp': '2012-03-06T15:31:00', 'preshot_description': '\\nHL11, 300 ms, 3 V. CH4 plenum 1032.\\n', 'divertor_config': 'conventional'}, {'shot_id': 28397, 'timestamp': '2012-03-06T15:38:00', 'preshot_description': '\\nHL11, 300 ms, 5 V. CH4 plenum 1032.\\n', 'divertor_config': 'conventional'}, {'shot_id': 28398, 'timestamp': '2012-03-06T15:43:00', 'preshot_description': '\\nHL11, 300 ms, 4 V. CH4 plenum 1032.\\n', 'divertor_config': 'conventional'}], 'page_meta': {'next_cursor': 'OQ==', 'total_items': 2082, 'total_pages': 694}}}\n"
     ]
    }
   ],
   "source": [
    "def do_query(cursor: str = None):\n",
    "    query = gql(\"\"\"\n",
    "    query ($cursor: String) {\n",
    "        all_shots (limit: 3, where: {campaign: {contains: \"M9\"}}, cursor: $cursor) {\n",
    "            shots {\n",
    "                shot_id\n",
    "                timestamp\n",
    "                preshot_description\n",
    "                divertor_config\n",
    "            }\n",
    "            page_meta {\n",
    "              next_cursor\n",
    "              total_items\n",
    "              total_pages\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "    \"\"\")\n",
    "    return client.execute(query, {'cursor': cursor})\n",
    "\n",
    "\n",
    "def iterate_responses():\n",
    "    cursor = None\n",
    "    while True:\n",
    "        response = do_query(cursor)\n",
    "        yield response\n",
    "        cursor = response['all_shots']['page_meta']['next_cursor']\n",
    "        if cursor is None:\n",
    "            return\n",
    "\n",
    "responses = iterate_responses()\n",
    "print(next(responses))\n",
    "print(next(responses))\n",
    "print(next(responses))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mast",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}