Metadata_Search_and_Discovery.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/supporting_fair_data_header.png\">\n",
    "\n",
    "In this notebook we demonstrate how the Globus platform can be used to create automated pipelines that can be used to make arbitrary data more Findable, Accessible, Interoperable, and Reusable. We demonstrate flexible access control, descriptive metadata, and use of persistent identifiers, as well as various ways to search and discover data based on these attributes.\n",
    "\n",
    "We will walk through the following data flow:\n",
    "1. Authenticate with Globus and get tokens for accessing various services\n",
    "1. Assemble a dataset and move the data to an endpoint with restricted access\n",
    "1. Define some metadata for our dataset\n",
    "1. Mint a persistent identifier for the data\n",
    "1. Index descriptive metadata in Globus Search such that is discoverable by other users\n",
    "\n",
    "The basic tutorial flow is illustrated below.  \n",
    "\n",
    "<img src=\"img/publication_flow.png\" alt=\"Automated data publication flow\" align=\"CENTER\" style=\"width: 85%;\"/>\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "To complete this tutorial you will need to make sure you are in the [Tutorial Users Group](https://app.globus.org/groups/50b6a29c-63ac-11e4-8062-22000ab68755)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import globus_sdk\n",
    "import minid  # for minting persistent identifiers (pip install minid)\n",
    "\n",
    "# ID of this tutorial notebook as registered with Globus Auth\n",
    "CLIENT_ID = \"d61ed2e0-b4f9-4fe9-9433-41e2528a807d\"\n",
    "\n",
    "# Collection for storing data\n",
    "publication_collection = \"a6f165fa-aee2-4fe5-95f3-97429c28bf82\"  # \"Globus Tutorials on ALCF Eagle\"\n",
    "http_hostname = \"g-fe1c1.fd635.8443.data.globus.org\"\n",
    "\n",
    "# Globus Group which can view datasets\n",
    "access_group = \"50b6a29c-63ac-11e4-8062-22000ab68755\"  # \"Tutorial Users\" group\n",
    "\n",
    "# Search index ID to store metadata; will be selected/created later\n",
    "search_index = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Authenticate\n",
    "\n",
    "Before implementing the automated data flow we must authenticate with Globus and request access tokens to use the transfer, search, and identifier services. Here we get the tokens avaialable in JupyterHub, and create clients for interacting with Globus services."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle, base64, os, pprint\n",
    "\n",
    "if os.getenv('GLOBUS_DATA'):\n",
    "    # Get Globus Auth token data from the JupyterHub environment\n",
    "    data = pickle.loads(base64.b64decode(os.getenv('GLOBUS_DATA')))\n",
    "    \n",
    "    # extract access token for each service\n",
    "    transfer_token = data['tokens']['transfer.api.globus.org']['access_token']\n",
    "    search_token = data['tokens']['search.api.globus.org']['access_token']\n",
    "    minid_token = data['tokens']['85114005-42e6-4671-a73a-0a40150c2b88']['access_token']\n",
    "\n",
    "else:\n",
    "    # not running in JupyterHub environment; need to authenticate user\n",
    "    native_auth_client = globus_sdk.NativeAppAuthClient(CLIENT_ID)\n",
    "\n",
    "    # start a flow with a specific set of requested scopes (levels of access to Globus apps/services)\n",
    "    # after login, you will be prompted to grant this notebook access to these services\n",
    "    transfer_scope = 'urn:globus:auth:scope:transfer.api.globus.org:all'\n",
    "    search_scope = 'urn:globus:auth:scope:search.api.globus.org:all'\n",
    "    minid_scope = 'https://auth.globus.org/scopes/identifiers.fair-research.org/writer'\n",
    "    native_auth_client.oauth2_start_flow(\n",
    "        requested_scopes=[\n",
    "            transfer_scope,\n",
    "            search_scope,\n",
    "            minid_scope\n",
    "        ]\n",
    "    )\n",
    "    print(\"Login Here:\\n\\n{0}\".format(native_auth_client.oauth2_get_authorize_url()))\n",
    "    print(\"\\nIMPORTANT NOTE: the link above can only be used once!\")\n",
    "    print(\"If login or a later step in the flow fails, you must execute this cell again to generate a new link.\")\n",
    "    \n",
    "    # add the code that you got from Globus below\n",
    "    auth_code = input('PASTE YOUR AUTH CODE HERE> ')\n",
    "\n",
    "    # and exchange it for a response object containing your token(s)\n",
    "    tokens = native_auth_client.oauth2_exchange_code_for_tokens(auth_code)\n",
    "\n",
    "    # extract access token for each service\n",
    "    transfer_token = tokens.by_scopes[transfer_scope]['access_token']\n",
    "    search_token = tokens.by_scopes[search_scope]['access_token']\n",
    "    minid_token = tokens.by_scopes[minid_scope]['access_token']\n",
    "    \n",
    "# see what the tokens look like\n",
    "print(\"Retrieved tokens:\")\n",
    "print(\"Transfer: %s\" % transfer_token)\n",
    "print(\"Search: %s\" % search_token)\n",
    "print(\"Minid: %s\" % minid_token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create clients to access each of the services\n",
    "# to pass tokens to clients, wrap them in GlobusAuthorizers and pass the results to client objects\n",
    "# these are generic objects which support multiple authentication methods - access Tokens are just one\n",
    "transfer_client = globus_sdk.TransferClient(\n",
    "    authorizer=globus_sdk.AccessTokenAuthorizer(transfer_token))\n",
    "search_client = globus_sdk.SearchClient(\n",
    "    authorizer=globus_sdk.AccessTokenAuthorizer(search_token))\n",
    "minid_client = minid.MinidClient(\n",
    "    authorizer=globus_sdk.AccessTokenAuthorizer(minid_token))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Assemble Dataset\n",
    "\n",
    "In the first stage of the flow we move the data to a location that is immuatable, accessible only to authorized users (i.e. those in the Tutorial Users group), and able to scale as needed. We use a Globus guest collection for this purpose, as it allows us to dynamically manage access to data. \n",
    "\n",
    "To isolate users' datasets from each other we create a unique directory on our guest collection. To avoid name conflcits, we will name the directory using a UUID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import uuid\n",
    "\n",
    "# use Globus Transfer to create a new directory\n",
    "share_path = '/metadata-search-and-discovery/' + str(uuid.uuid4()) + '/'\n",
    "r = transfer_client.operation_mkdir(publication_collection, path=share_path)\n",
    "\n",
    "print(\"Dataset path: %s\" % share_path)\n",
    "print(\"https://app.globus.org/file-manager?origin_id=%s&origin_path=%s\" % (publication_collection, share_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having created the directory we now need to populate it with our dataset. For simplicity, we will move sample Globus data from the \"Globus Tutorial Collection.\" You are welcome to use any data you like, just update the `source_collection` and `source_path`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the source collection and directory containing data to be published\n",
    "# (Globus Tutorial Collection 1):/home/share/godata/\n",
    "source_collection = '6c54cade-bde5-45c1-bdea-f4bd71dba2cc'\n",
    "source_path = '/home/share/godata/file1.txt'\n",
    "destination_filename = share_path + os.path.basename(source_path)\n",
    "\n",
    "# TransferData is a helper function for building good Transfer Task documents for the Globus Transfer Service\n",
    "transfer_data = globus_sdk.TransferData(\n",
    "    transfer_client,\n",
    "    source_collection,\n",
    "    publication_collection,\n",
    "    label='Tutorial copy data',\n",
    "    sync_level='checksum'\n",
    ")\n",
    "\n",
    "# you can add multiple files and directories to transfer -- for our case, just add one\n",
    "transfer_data.add_item(source_path, destination_filename)\n",
    "\n",
    "# submit the transfer and get a task document to describe it\n",
    "task_description = transfer_client.submit_transfer(transfer_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now wait for the transfer to complete using the Globus SDK `task_wait` function. To confirm that the data is transferred correctly we preform an `ls` operation on the guest collection. Note: in this example we also record the last file name in the publication directory so that we can associate metadata later in the tutorial. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Wait up to 100s, checking every second; note: we don't handle task failures here\n",
    "completed = transfer_client.task_wait(\n",
    "    task_description['task_id'],\n",
    "    timeout=100,\n",
    "    polling_interval=1\n",
    ")\n",
    "\n",
    "transferred_files = ''\n",
    "\n",
    "if not completed:\n",
    "    print('Transfer still running...')\n",
    "else:\n",
    "    for f in transfer_client.operation_ls(publication_collection, path=share_path):\n",
    "        transferred_files = f['name'] + ';' + transferred_files\n",
    "\n",
    "print(transferred_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the data are placed on a guest collection, and in a unique directory, we can share the data with individuals or groups of users. Below we share the data with the \"Tutorial Users Group\" so that other tutorial participants will be able to view and download files. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This access control rule...\n",
    "# ...grants Read access, permissions=\"r\"\n",
    "# ...to the Tutorial Users Group, access_group\n",
    "# ...on the directory we created above, 'share_path'\n",
    "rule_data = {\n",
    "    'DATA_TYPE': 'access',\n",
    "    'principal_type': 'group', \n",
    "    'principal': access_group,\n",
    "    'path': share_path,\n",
    "    'permissions': 'r'\n",
    "}\n",
    "\n",
    "# Add the access control rule to the guest collection\n",
    "result = transfer_client.add_endpoint_acl_rule(publication_collection, rule_data)\n",
    "print(result['message'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Create Metadata to Describe Dataset\n",
    "\n",
    "We will define simple metadata which describes our dataset. This metadata will be used for registering the identifier and also for loading into our search index to enable discovery of the published dataset.\n",
    "\n",
    "You should update the metadata below to reflect your publication. Add your name as a contributor and update the title, date, and keywords. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata = {\n",
    "    'title': 'My Globus Tutorial Dataset',\n",
    "    'contributors': ['John Smith', 'FrobozzCo', 'Zaphod Beeblebrox'],\n",
    "    'date': '2019-01-01',\n",
    "    'keywords': ['FCD#3', 'Blanket', 'Panic', ],\n",
    "    'files': [{\n",
    "        'url': \"https://%s%s\" % (http_hostname, share_path)\n",
    "    }]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  4. Associate an Identifier\n",
    "\n",
    "Next we associate a persistent and unambiguous identifier with the dataset. This allows others to refer to a permanent name rather than a potentially volatile storage location reference.\n",
    "\n",
    "When minting an identifier the following information must be provided:\n",
    "* One or more locations to access the data, such as a URL representing a particular path on a Globus collection\n",
    "* Metadata describing a mixture of publication-specific attributes (e.g., creator, checksum) and optionally extensible, user-defined attributes\n",
    "* Access policies governing which users can access the identifier\n",
    "\n",
    "Minids are public, simple, and lightweight identifiers that we can use for this example. We will also provide the checksum in this case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define a location for accessing the data\n",
    "dataset_location = \"https://%s%s\" % (http_hostname, share_path)\n",
    "\n",
    "dataset_identifier = minid_client.register(\n",
    "    locations=[dataset_location],\n",
    "    title=metadata['title'],\n",
    "    checksums=[{\n",
    "        'function': 'sha256',\n",
    "        'value': '2c8b08da5ce60398e1f19af0e5dccc744df274b826abe585eaba68c525434806'\n",
    "    }],\n",
    "    metadata={\n",
    "        'date': metadata['date'],\n",
    "        'contributors': metadata['contributors']\n",
    "    },\n",
    "    test=True,\n",
    ")\n",
    "\n",
    "metadata['identifier'] = dataset_identifier.data['identifier']\n",
    "\n",
    "print(\"Identifier %s\" % dataset_identifier.data['identifier'])\n",
    "print(\"location %s\" % dataset_identifier.data['location'])\n",
    "print(\"Metadata %s\" % dataset_identifier.data['metadata'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have minted the identifier we can resolve it to find out metadata and retrieve a link to the data. Irrespective of the service used to mint an identifier, you should ensure the scheme is also be registered with other resolvers, such as [nt2.net](https://n2t.net), the name 2 thing resolver.\n",
    "\n",
    "Note: Registration takes a few moments to propogate. If the identifier doesn't resolve, please wait a few seconds and try again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('https://n2t.net/{}'.format(metadata['identifier']))\n",
    "print('https://identifiers.fair-research.org/{}'.format(metadata['identifier']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Index Descriptive Metadata\n",
    "\n",
    "In this stage of the flow we aim to index the metadata that describes our published dataset. For this purpose we use Globus Search, a flexible, schema-agnostic search platform with fine grained access control on data and metadata. Globus Search provides powerful, free-text search capabilities via which others can discover our published dataset.\n",
    "\n",
    "Globus Search supports user-managed indexes in which an adminstrator may create an index and define policies regarding its use, including who can manage the index, ingest metadata, and query the index. \n",
    "\n",
    "Complete documentation for using Globus Search is available at https://docs.globus.org/api/search/.\n",
    "\n",
    "Run the code below to create your own personal Globus Search index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "index_name = \"tutorial-notebook-index\"\n",
    "\n",
    "# Fetch all indices to which the user has access\n",
    "indices = [si for si in search_client.get(\"/v1/index_list\").data['index_list']\n",
    "           if si['is_trial'] \n",
    "           and si['display_name'] == index_name\n",
    "           and 'owner' in si['permissions']\n",
    "          ]\n",
    "\n",
    "# If an index was found with the criteria above, re-use it\n",
    "# Otherwise, create a new index\n",
    "if indices:\n",
    "    tutorial_index = indices[0]\n",
    "    print('Found existing trial index!')\n",
    "else:\n",
    "    tutorial_index = search_client.create_index(\n",
    "        display_name=index_name,\n",
    "        description=\"A trial index for use in Globus tutorials\"\n",
    "    ).data\n",
    "    print('Index created for use in Globus tutorials.')\n",
    "\n",
    "search_index = tutorial_index['id']\n",
    "\n",
    "print(tutorial_index['display_name'])\n",
    "print(tutorial_index['description'])\n",
    "print(search_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing Data\n",
    "\n",
    "Globus Search supports scalable indexing of arbitrary entries into a selected index. An entry is comprised of three types of information:\n",
    "1. A subject, which represents a name or target for the entry (e.g., a URL for a Globus-accesible file or directory)\n",
    "1. Arbitrary metadata represented as a collection of attributes in nested JSON structure\n",
    "1. A visibility policy that defines which users or groups are able to view and query the subject and its metadata\n",
    "\n",
    "To index metadata we construct an JSON object that includes this information, and use the `ingest` function to add it to the index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "subject =  \"https://%s%s\" % (http_hostname, share_path)\n",
    "ingest_data = {\n",
    "    \"ingest_type\": \"GMetaEntry\",\n",
    "    \"ingest_data\": {\n",
    "        \"subject\": subject,\n",
    "        \"visible_to\": [\"public\"],\n",
    "        \"content\": metadata\n",
    "    }\n",
    "}\n",
    "search_client.ingest(search_index, ingest_data)\n",
    "print(\"Ingested Subject: %s\" % subject)\n",
    "print(metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Viewing Search Records\n",
    "\n",
    "Globus Search records can also be viewed at https://acdc.alcf.anl.gov/. Portals are a good way to visually present a large number of search results for users. \n",
    "\n",
    "The portal shown uses the same Globus SDK calls for generating queries that you will use below. Note the facets on the left, and the filter query params in the address bar. Each of these boil down into simple parameters passed into the Globus SDK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.parse import quote_plus\n",
    "subject_enc = quote_plus(quote_plus(subject))\n",
    "portal_url = f'https://acdc.alcf.anl.gov/globus-tutorial/{search_index}/detail/{subject_enc}/'\n",
    "print(f'Subject on ALCF Community Data Co-Op: {portal_url}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Search\n",
    "\n",
    "Globus Search implements a flexible query model that supports two types of queries: simple, free-text queries and complex, structured queries.\n",
    "\n",
    "Simple queries perform basic sub-string matching against any metadata fields that are visible to the querying user.\n",
    "As with web search, the results of a simple search are ordered based on the computed \"best match\" for the query. \n",
    "\n",
    "A simple query is as easy as passing a string to the `search` function.  The results are an ordered list of result objects. \n",
    "\n",
    "Update the following free text query to discover your dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query='john'\n",
    "\n",
    "search_results = search_client.search(index_id=search_index, q=query)\n",
    "\n",
    "print(\"Count: %s\" % search_results['count'])\n",
    "for i in search_results['gmeta']:\n",
    "    print(\"Subject: %s\" % i['subject'])\n",
    "    print(\"Entries: %s\" % json.dumps(i['entries']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Globus Search also supports an advanced query mode in which more precise queries can be expressed. For examples, queries that search specific attributes, range expressions, exact matches, and so forth.\n",
    "\n",
    "First we search for your published dataset using the minted identiifer, we then query for all publications with a specific contributor. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "search_results = search_client.search(search_index, q='identifier: \"%s\"' % metadata['identifier'], advanced=True)\n",
    "\n",
    "print(\"Count: %s\" % search_results['count'])\n",
    "for i in search_results['gmeta']:\n",
    "    print(\"Subject: %s\" % i['subject'])\n",
    "    print(\"Content: %s\" % i['entries'])  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "search_results = search_client.search(search_index, 'contributors: \"John Smith\"', advanced=True)\n",
    "\n",
    "print(\"Count: %s\" % search_results['count'])\n",
    "for i in search_results['gmeta']:\n",
    "    print(\"Subject: %s\" % i['subject'])\n",
    "    print(\"Content: %s\" % json.dumps(i['entries']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Complex queries\n",
    "\n",
    "Complex queries take the form of a structured JSON document, and are more commonly used when the queries is created programmatically. They may reference specific metadata fields, and may apply criteria such as value ranges, wildcards, and regular expressions. \n",
    "\n",
    "For example, to conduct the same free-text search as above&mdash;but to limit results to publications between 2010-2020&mdash;we can add a filter to the query.\n",
    "\n",
    "Note: We use the Globus SDK SearchQuery to construct complex queries. We also show the resulting JSON query object used to execute the query. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "structured_query = (globus_sdk.SearchQuery(q=query)\n",
    "                    .add_filter('date', [{'from': 2000, 'to': 2020}], type='range'))\n",
    "search_results = search_client.post_search(search_index, structured_query)\n",
    "\n",
    "structured_query_formatted = json.dumps(dict(structured_query), indent=2)\n",
    "print(f\"Structured Query Object: \\n{structured_query_formatted}\\n\")\n",
    "print(\"Count: %s\" % search_results['count'])\n",
    "for i in search_results['gmeta']:\n",
    "    print(\"Subject: %s\" % i['subject'])\n",
    "    print(\"Content: %s\\n\" % json.dumps(i['entries']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Complex queries may also specify facets&mdash;a method for generating categories and associated frequencies for particular metadata fields. For example, here is a query to produce keyword facets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "structured_query = (globus_sdk.SearchQuery(q='*').add_facet('Publication Keywords', 'keywords'))\n",
    "search_results = search_client.post_search(search_index, structured_query)\n",
    "\n",
    "structured_query_formatted = json.dumps(dict(structured_query), indent=2)\n",
    "print(f\"Structured Query Object: \\n{structured_query_formatted}\\n\")\n",
    "print(\"Results\\nCount: %s\" % search_results['count'])\n",
    "print(\"\\nFacets\")\n",
    "for i in search_results['facet_results']:\n",
    "    for j in i['buckets']:\n",
    "        print (\"%s (%s)\" % (j['value'], j['count']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Advanced indexing\n",
    "\n",
    "One of the benefits of the Globus Search model is that you can associate visibility policies with records and metadata. Here we demonstrate how you can add a new metadata entry to a record and make it visible only to a particular group of users. \n",
    "\n",
    "Update the metadata added below, and confirm that the queries now show the updated metadata. Note: When querying over these entities the results will collapse metadata for the same root subject. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "ingest_data = {\n",
    "    \"ingest_type\": \"GMetaEntry\",\n",
    "    \"ingest_data\": {\n",
    "        \"subject\": \"https://%s%s\" % (http_hostname, share_path),\n",
    "        \"id\": \"rating\",\n",
    "        \"visible_to\": ['urn:globus:groups:id:%s' % access_group],\n",
    "        \"content\": {\n",
    "            \"rating\": \"good\",\n",
    "        }\n",
    "    }\n",
    "}\n",
    "result = search_client.ingest(search_index, ingest_data)\n",
    "while search_client.get_task(result['task_id'])['state'] in ['PENDING', 'PROGRESS']:\n",
    "    print('Ingesting...')\n",
    "    time.sleep(1)\n",
    "print('Done.')\n",
    "\n",
    "search_results = search_client.search(search_index, q='identifier: \"%s\"' % metadata['identifier'], advanced=True)\n",
    "\n",
    "print(\"Count: %s\" % search_results['count'])\n",
    "for i in search_results['gmeta']:\n",
    "    print(\"Subject: %s\" % i['subject'])\n",
    "    print(\"Content: %s\" % i['entries'])                                   "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}