Registry.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Registry-powered Searches\n",
    "\n",
    "Archives register their data sets so that programs can discover them: http://vao.stsci.edu/keyword-search/\n",
    "\n",
    "The link above is a GUI, so you can play around with it. \n",
    "\n",
    "There is also an applications program interface (API) to this service, so that programs can send queries and retrieve search results automatically. The standard search API is a Table Access Protocol service on a relational data model for registries, which is a very low-level interface. Here we have created several simplified tools for common, useful queries into the registry.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests, io, astropy\n",
    "\n",
    "## For handling ordinary astropy Tables\n",
    "from astropy.table import Table, vstack\n",
    "\n",
    "## There are a number of relatively unimportant warnings that \n",
    "## show up, so for now, suppress them:\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "## our stuff\n",
    "import sys\n",
    "\n",
    "# Use the NASA_NAVO/astroquery\n",
    "from navo_utils.registry import Registry"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Registry query methods exist in an astroquery.vo.Registry() class with different levels of simplicity and power. For example, if you already know you want to search NED, you can get related service URLs as follows. Note that you may get *more* results than you expect, some of which should be easily differentiated by inspection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Registry:  sending query ADQL = \n",
      "          select res.waveband,res.short_name,cap.ivoid,res.res_description,\n",
      "          intf.access_url,res.reference_url,res_role.role_name as publisher,cap.cap_type as service_type\n",
      "          from rr.capability as cap\n",
      "            natural join rr.resource as res\n",
      "            natural join rr.interface as intf\n",
      "\t\t    natural join rr.res_role as res_role\n",
      "             where cap.cap_type='conesearch' and cap.ivoid like '%ned%' and res_role.base_role = 'publisher'\n",
      "\n",
      "Queried: http://vao.stsci.edu/RegTAP/TapService.aspx/sync\n",
      "\n",
      "Found 2 results:\n",
      "                                              access_url                                              \n",
      "------------------------------------------------------------------------------------------------------\n",
      "                                                https://irsa.ipac.caltech.edu/SCS?table=shelacomb&amp;\n",
      "http://ned.ipac.caltech.edu/cgi-bin/NEDobjsearch?search_type=Near+Position+Search&amp;of=xml_main&amp;\n",
      "ivo://ned.ipac/basic_data_near_position\n",
      "<TableColumns names=('waveband','short_name','ivoid','res_description','access_url','reference_url','publisher','service_type')>\n"
     ]
    }
   ],
   "source": [
    "results = Registry.query(source='ned', service_type='cone', verbose=True)\n",
    "print('Found {} results:'.format(len(results)))\n",
    "print(results[:]['access_url'])\n",
    "print(results[1]['ivoid'])\n",
    "print(results.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Registry.query() method takes arguments which we can use to further filter the results (passed to internal function _build_adql):  \n",
    "\n",
    "    service_type : \"conesearch\", \"simpleimageaccess\", \"simplespectralaccess\", \"tableaccess\". May be shortened to \"cone\", \"image\", \"spectr\", or \"table\" or \"tap\", respectively.\n",
    "    keyword      : any keyword contained in ivoid, title, or description\n",
    "    waveband     : waveband string. Multiple options may be comma-delimited i.e. \"optical, infrared\"\n",
    "    source       : any substring in ivoid\n",
    "    publisher    : the name of any publishing organization\n",
    "    order_by     : what field to order it by, but then you have to know the names, currently\n",
    "                    (\"waveband\",\"short_name\",\"ivoid\",\"res_description\",\"access_url\",\"reference_url\",\"publisher\", service_type\")\n",
    "    logic_string : any other string you want to add to the ADQL where clause, should start with \" and \"\n",
    "\n",
    "The results are returned by Registry.query() in an astropy table using the conversion function _astropy_table_from_votable_response(). \n",
    "\n",
    "\n",
    "\n",
    "## Waveband Allowed Terms and values in (A)\n",
    "* gamma-ray:\t \tless than 0.1\n",
    "* X-ray:\t \t0.1-100\n",
    "* EUV:\t \t100-1000\n",
    "* UV:\t \t1,000-3,000\n",
    "* Optical:\t \t3,000-10,000\n",
    "* Infrared:\t \t10,000-1,000,000\n",
    "* Millimeter:\t \t10^6 - 10^8\n",
    "* Radio:\t \tover 10^8\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Registry.query_counts() method takes arguments which we can use to see which keyword values might help us narrow down our search, or possibly give us too MANY results (these are passed to internal function _build_counts_adql):\n",
    "\n",
    "    field      : keyword field for which to see popular values: \"waveband\", \"publisher\", \"service_type\" currently supported.\n",
    "    minimum    : A minimum count of occurences for the keyword value to use as a cutoff (optional, defaults to 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Registry:  sending query ADQL = select * from (select role_name as publisher, count(role_name) as count_publisher from rr.res_role where base_role = 'publisher'  group by role_name) as count_table where count_publisher >= 15 order by count_publisher desc\n",
      "\n",
      "Queried: http://vao.stsci.edu/RegTAP/TapService.aspx/sync\n",
      "\n",
      "                         publisher                           count_publisher\n",
      "------------------------------------------------------------ ---------------\n",
      "                                                         CDS           17286\n",
      "                                           NASA/GSFC HEASARC            1041\n",
      "                          NASA/IPAC Infrared Science Archive             521\n",
      "                                            The GAVO DC team             159\n",
      "                   Space Telescope Science Institute Archive             101\n",
      "      WFAU, Institute for Astronomy, University of Edinburgh              99\n",
      "                                                     SVO CAB              35\n",
      "                                                         IA2              35\n",
      "                              Canadian Astronomy Data Centre              30\n",
      "Sternberg Astronomical Institute Virtual Observatory Project              29\n",
      "                                                       HELIO              27\n",
      "                                                        MSSL              24\n",
      "                                                   Astrogrid              24\n",
      "                                                         JHU              19\n",
      "                        The NASA/IPAC Extragalactic Database              18\n",
      "                                                         JVO              17\n",
      "                                       European Space Agency              17\n"
     ]
    }
   ],
   "source": [
    "results = Registry.query_counts('publisher', 15, verbose=True)\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a 'publisher' field to work from, we can get a narrowed down query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Registry:  sending query ADQL = \n",
      "          select res.waveband,res.short_name,cap.ivoid,res.res_description,\n",
      "          intf.access_url,res.reference_url,res_role.role_name as publisher,cap.cap_type as service_type\n",
      "          from rr.capability as cap\n",
      "            natural join rr.resource as res\n",
      "            natural join rr.interface as intf\n",
      "\t\t    natural join rr.res_role as res_role\n",
      "             where cap.cap_type='conesearch' and cap.ivoid like '%ned%' and res_role.base_role = 'publisher' and res_role.role_name like '%Extragalactic Database%'\n",
      "\n",
      "Queried: http://vao.stsci.edu/RegTAP/TapService.aspx/sync\n",
      "\n",
      "Found 1 results:\n",
      "                                              access_url                                              \n",
      "------------------------------------------------------------------------------------------------------\n",
      "http://ned.ipac.caltech.edu/cgi-bin/NEDobjsearch?search_type=Near+Position+Search&amp;of=xml_main&amp;\n"
     ]
    }
   ],
   "source": [
    "results = Registry.query(source='ned', publisher='Extragalactic Database', service_type='cone', verbose=True)\n",
    "print('Found {} results:'.format(len(results)))\n",
    "print(results[:]['access_url'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Note we will need to URL-decode the access_url information in our results, as the registry resource standard expects it be encoded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://ned.ipac.caltech.edu/cgi-bin/NEDobjsearch?search_type=Near+Position+Search&of=xml_main&\n"
     ]
    }
   ],
   "source": [
    "from html import unescape\n",
    "\n",
    "for result in results:\n",
    "    print(unescape(result['access_url']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix: Documentation on the Standards\n",
    "\n",
    "### Table Access Protocol \n",
    "* IVOA standard for RESTful web service access to tabular data\n",
    "* http://www.ivoa.net/documents/TAP/\n",
    "\n",
    "\n",
    "### Registry Relational Schema\n",
    "* IVOA standard for modeling registry metadata for querying with TAP\n",
    "* http://www.ivoa.net/documents/RegTAP/\n",
    "\n",
    "\n",
    "### Astronomical Query Data Language (2.0)\n",
    "* IVOA standard for querying astronomical data in tabular format, with geometric search support\n",
    "* http://www.ivoa.net/documents/latest/ADQL.html\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}