NB_Explore unkown datasets with SPARQL.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linked Data 'n' Music\n",
    "## SPARQL Exercises: Exploring unknown datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This Jupyter notebook aims to support a basic understanding of how to explore unknown Linked Data datasets with the query language SPARQL (https://www.w3.org/TR/rdf-sparql-query/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Notebook was created by [@musicenfanthen](https://github.com/musicEnfanthen) and is based on & inspired by [@Bob DuCharme](https://twitter.com/bobdc)'s description of the approach in his blog entry: \n",
    "- http://www.bobdc.com/blog/jupytersparql/ \n",
    "\n",
    "and the corresponding notebook \n",
    "\n",
    "- https://github.com/bobdc/misc/blob/master/JupyterSPARQL/Jupyter%20and%20SPARQL%20and%20Dort%20or%20Dordrecht.ipynb.\n",
    "\n",
    "It uses the `sparqlkernel` written and released by Paulo Villegas: https://github.com/paulovn/sparql-kernel."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To execute a block of code in this notebook, click in a cell and press `Shift+Enter`. For general information about Jupyter notebooks see http://jupyter.org/. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At first, we have to set up this Jupyter notebook and tell the `sparqlkernel` which SPARQL endpoint, query format or output format we want to use. The `sparqlkernel` provides some `magics` (keywords) for this, so it is pretty straightforward. You will find more information about the syntax of these `magics` here: https://github.com/paulovn/sparql-kernel#syntax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Possible music-related endpoints (in alphabetical order):\n",
    "\n",
    "\n",
    "**DoReMus**\n",
    "- Endpoint: http://data.doremus.org/sparql\n",
    "- Website:  https://www.doremus.org/\n",
    "\n",
    "\n",
    "**JazzCats**\n",
    "- Endpoint: http://cdhr-linkeddata.anu.edu.au/jazzcats-sparql/sparql\n",
    "- Website:  http://jazzcats.cdhr.anu.edu.au/\n",
    "\n",
    "\n",
    "**MIDI Linked Data** (could face timeouts)\n",
    "- Endpoint: http://virtuoso-midi.amp.ops.labs.vu.nl/sparql\n",
    "- Website:  https://midi-ld.github.io/\n",
    "\n",
    "\n",
    "**MusicOWL**\n",
    "- Endpoint: http://linkeddata.uni-muenster.de:7200\n",
    "- Website:  http://linkeddata.uni-muenster.de/musicportal/\n",
    "\n",
    "\n",
    "**RISM**\n",
    "- Endpoint: https://data.rism.info/sparql\n",
    "- Website:  https://opac.rism.info/\n",
    "\n",
    "\n",
    "**SLICKMEM**\n",
    "- Endpoint: https://blazegraph.linkedmusic.org/blazegraph/namespace/SLICKMEM/sparql\n",
    "- Website:  https://slickmem.linkedmusic.org\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the SPARQL endpoint (feel free to change to your liking)\n",
    "%endpoint http://cdhr-linkeddata.anu.edu.au/jazzcats-sparql/sparql\n",
    "\n",
    "# Don't show more than 50 results (even if more are fetched)\n",
    "%show 50\n",
    "\n",
    "# Set the data format requested from the SPARQL endpoint (%format JSON | XML | N3 | any | default)\n",
    "%format any\n",
    "\n",
    "# Set the output rendering shape (%display raw | table [withtypes] | diagram [svg|png] [withliterals])\n",
    "%display table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Short test if the selected endpoint is running and we are connecting to it properly (as recommended by Bob DuCharme).\n",
    "\n",
    "**Attention:** With very large datasets and a low timeout limit of the triple store, this query may lead to timeouts, but would work with more explicit requests (see below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT * \n",
    "WHERE {\n",
    "\n",
    "    ?s ?p ?o . \n",
    "\n",
    "}\n",
    "LIMIT 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. SPARQL Queries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1. Exploring unknown datasets: Concepts/Classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.0 Find all Concepts/Classes of the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation:\n",
    "\n",
    "General information about and specification of the SPARQL Query language can be found here: https://www.w3.org/TR/sparql11-query/ \n",
    "\n",
    "`SELECT`    = one of SPARQL's query forms; returns all, or a subset of, the variables bound in a query pattern match.\n",
    "\n",
    "`DISTINCT`  = ensure solutions in the sequence are unique, i.e. duplicates are merged into one result\n",
    "\n",
    "`*`         = shortcut to return all bound variables (`?xyz`) as results, here the results bound in `?Concept`\n",
    "\n",
    "`WHERE {…}` = so-called WHERE-clause, basic graph pattern to match against the data graph\n",
    "\n",
    "`[]`        = blank node, see [4.1.4 Syntax for Blank Nodes](https://www.w3.org/TR/sparql11-query/#QSynBlankNodes)\n",
    "\n",
    "`a`         = `rdf:type`\n",
    "\n",
    "`?Concept` = random variable name for any URI that matches the given pattern in the object position of a triple.\n",
    "\n",
    "`LIMIT`     = modifier to limit the result number (important!)\n",
    "\n",
    "\n",
    "**IMPORTANT: Use the `LIMIT` modifier in all of the following queries to avoid stressing the endpoint too much. You may need to increase your limit in certain queries according to the total number of query results e.g. in 2.1.1.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT DISTINCT * \n",
    "WHERE {                 \n",
    "\n",
    "    [] a ?Concept .     \n",
    "    \n",
    "}\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.1 Get total number of classes available in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "\n",
    "`(COUNT(DISTINCT ?Concept)`= count number of distinct values bound to variable `?Concept`\n",
    "\n",
    "You may need to increase your limit in subsequent queries according to the resulting total number to see all query results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT (COUNT(DISTINCT ?Concept) as ?count)\n",
    "WHERE   \n",
    "{ \n",
    "   \n",
    "    [] a ?Concept .\n",
    " \n",
    "}\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.2 Get the number of the individual concepts/classes in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "`(COUNT(*) AS ?n)` = bind the number of results of all bound variables to the variable `?n`\n",
    "\n",
    "`GROUP BY`         = group the results by the given value/variable\n",
    "\n",
    "`ORDER BY DESC`    = sort the results in descending order by the given value/variable (leave out `DESC` or use `ASC` for ascending order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT DISTINCT ?Concept (COUNT(*) AS ?n) \n",
    "WHERE {\n",
    "\n",
    "    [] a ?Concept .\n",
    "\n",
    "}\n",
    "GROUP BY ?Concept\n",
    "ORDER BY DESC(?n)\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.3 Get the number of these concepts/classes in the dataset plus a sample of every class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "\n",
    "see also: https://twitter.com/kidehen/status/1129143936839294976 \n",
    "\n",
    "`(SAMPLE(?s) AS ?sample)` = retrieve one random sample of the result set for the variable `?s` and bind the sample to the variable `?sample`\n",
    "\n",
    "`COUNT(1)`                = equivalent of `COUNT(*)`\n",
    "\n",
    "`?s`                      = random variable name for any URI that matches the given pattern in the subject position of a triple\n",
    "\n",
    "`FILTER(isIri(?s))`       = filter out only those solutions for the variable `?s` that are an IRI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT (SAMPLE(?s) AS ?sample) (COUNT(1) AS ?n) (?Concept AS ?entityType) \n",
    "WHERE {\n",
    "\n",
    "    ?s a ?Concept .\n",
    "    FILTER(isIri(?s)) .\n",
    "}\n",
    "GROUP BY ?Concept\n",
    "ORDER BY DESC(?n)\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.4 Get all classes/concepts with a given prefix/namespace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "\n",
    "see also: https://codyburleson.com/sparql-examples-select/ \n",
    "\n",
    "`FILTER( STRSTARTS(STR(?Concept),str(owl:)) )`       = filter out only those solutions for the variable `?Concept`  that start with the given string `owl:` (try to replace with `foaf:`, `rdfs:`, `mo:` or others)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PREFIX doremus: <http://data.doremus.org/ontology#>\n",
    "PREFIX ecrm: <http://erlangen-crm.org/current/>\n",
    "PREFIX foaf: <http://xmlns.com/foaf/0.1/>\n",
    "PREFIX mo: <http://purl.org/ontology/mo/>\n",
    "PREFIX owl: <http://www.w3.org/2002/07/owl#>\n",
    "PREFIX prov: <http://www.w3.org/ns/prov#>\n",
    "PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n",
    "PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n",
    "\n",
    "\n",
    "SELECT DISTINCT ?Concept\n",
    "WHERE {\n",
    "    \n",
    "     [] a ?Concept .\n",
    "     FILTER( STRSTARTS(STR(?Concept),str(owl:)) )\n",
    "    \n",
    "}\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.5 Get and count the namespaces of all classes/concepts used in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "\n",
    "see also: https://stackoverflow.com/a/20054523\n",
    "\n",
    "`COUNT (DISTINCT ?Concept)` = count only distinct values bound to variable `?Concept`, i.e. make sure all occurences of classes/concepts are counted only once\n",
    "\n",
    "`REPLACE(str(?Concept), \"(#|/)[^#/]*$\", \"$1\")` = REGEX expression that is used to remove the individual class name from the namespaces bound to `?Concept`\n",
    "\n",
    "`BIND(… AS ?ns)` = bind the results of the regex replacement to the variable `?ns`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT ?ns (COUNT (DISTINCT ?Concept) AS ?nsCount)\n",
    "WHERE   \n",
    "{ \n",
    "   [] a ?Concept .\n",
    "   BIND(REPLACE(str(?Concept), \"(#|/)[^#/]*$\", \"$1\") AS ?ns)\n",
    " \n",
    "}\n",
    "ORDER BY DESC(?nsCount)\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.6 Get all classes that are a subclass of another one"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "\n",
    "see also: https://nbviewer.jupyter.org/github/paulhoule/gastrodon/blob/master/notebooks/local/DBpedia_Schema_Queries.ipynb\n",
    "\n",
    "`rdfs:subClassOf` = subclass concept from the [RDF Schema](https://www.w3.org/TR/rdf-schema/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT DISTINCT ?subClass\n",
    "WHERE {\n",
    "\n",
    "    ?subClass rdfs:subClassOf ?class .\n",
    "   \n",
    "}\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.7 Get all classes that have subclasses. How many subclasses are there and what does a sample look like?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "\n",
    "see above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT ?class (COUNT(?subClass) AS ?numberOfSubClasses) (SAMPLE(?subClass) AS ?sSubClassSample) \n",
    "WHERE {\n",
    "\n",
    "    ?subClass rdfs:subClassOf ?class .\n",
    "\n",
    "}\n",
    "GROUP BY ?class\n",
    "ORDER BY DESC(?numberOfSubClasses)\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.8 Get all subclasses of a specific class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "\n",
    "see above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT DISTINCT *\n",
    "WHERE {\n",
    "\n",
    "    ?subClass rdfs:subClassOf rdf:Property .\n",
    "\n",
    "}\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1.9 Get all subclasses of a specific class recursively"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "\n",
    "`rdfs:subClassOf*` = property path operator in SPARQL 1.1 that will make the query engine recurse through multiple `rdfs:subClassOf` property links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT DISTINCT *\n",
    "WHERE {\n",
    "\n",
    "    ?subClass rdfs:subClassOf* rdf:Property .\n",
    "\n",
    "}\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  2.2 Exploring unknown datasets: Properties"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.0 Get all properties used in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "\n",
    "see above for classes/concepts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT DISTINCT ?p\n",
    "WHERE   \n",
    "{ \n",
    "   \n",
    "    ?s ?p ?o .\n",
    " \n",
    "}\n",
    "ORDER BY ?p\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.1 Get total number of properties used in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "\n",
    "`(COUNT(DISTINCT ?p)`= count number of distinct values bound to variable `?p`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT (COUNT(DISTINCT ?p) as ?pCount)\n",
    "WHERE   \n",
    "{ \n",
    "   \n",
    "    ?s ?p ?o .\n",
    " \n",
    "}\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.2 Get properties with a given prefix/namespace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "\n",
    "see above for classes/concepts\n",
    "\n",
    " `FILTER( STRSTARTS(STR(?p),str(owl:)) )`       = filter out only those solutions for the variable `?p` that start with the given string `owl:` (try to replace with `foaf:`, `rdfs:`, or `mo:`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PREFIX foaf: <http://xmlns.com/foaf/0.1/>\n",
    "PREFIX mo: <http://purl.org/ontology/mo/>\n",
    "PREFIX owl: <http://www.w3.org/2002/07/owl#>\n",
    "PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n",
    "PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>\n",
    "\n",
    "\n",
    "SELECT DISTINCT ?p\n",
    "WHERE {\n",
    "    \n",
    "     ?s ?p ?o .\n",
    "     FILTER( STRSTARTS(STR(?p),str(owl:)) )\n",
    "    \n",
    "}\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2.3 Get and count the namespaces of all properties used in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "\n",
    "see also: https://stackoverflow.com/a/20054523\n",
    "\n",
    "`COUNT (DISTINCT ?p)` = count only distinct values bound to variable `?p`, i.e. make sure all occurences of properties are counted only once\n",
    "\n",
    "`REPLACE(str(?p), \"(#|/)[^#/]*$\", \"$1\")` = REGEX expression that is used to remove the individual class name from the namespaces bound to `?p`\n",
    "\n",
    "`BIND(… AS ?ns)` = bind the results of the regex replacement to the variable `?ns`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Attention:** With very large datasets (e.g. DoReMus) this query may lead to timeouts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT ?ns (COUNT (DISTINCT ?p) AS ?nsCount)\n",
    "WHERE   \n",
    "{ \n",
    "   ?s ?p ?o .\n",
    "   BIND(REPLACE(str(?p), \"(#|/)[^#/]*$\", \"$1\") AS ?ns)\n",
    " \n",
    "}\n",
    "ORDER BY DESC(?nsCount)\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  2.3 Exploring unknown datasets: Datatypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beside classes and properties, we can also find literal datatypes in a dataset. Those include standard XML datatypes like `xsd:integer` or `xsd:datetime` or derived custom types."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3.0 Get all datatypes used in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "\n",
    "see above for classes/concepts\n",
    "\n",
    "see also: https://nbviewer.jupyter.org/github/paulhoule/gastrodon/blob/master/notebooks/local/DBpedia_Schema_Queries.ipynb\n",
    "\n",
    "`rdfs:Datatype` = concept of Datatype from the [RDF Schema](https://www.w3.org/TR/rdf-schema/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT DISTINCT ?dataType \n",
    "WHERE {\n",
    "\n",
    "    ?dataType a rdfs:Datatype .\n",
    "    \n",
    "} LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of the datasets presented at the beginning of this notebook, do not use any literal datatypes. In this case, try to switch to another SPARQL endpoint via `%endpoint xxx`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set another SPARQL endpoint if necessary\n",
    "%endpoint http://data.doremus.org/sparql"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT DISTINCT ?dataType \n",
    "WHERE {\n",
    "\n",
    "    ?dataType a rdfs:Datatype .\n",
    "    \n",
    "} LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3.1 Get total number of datatypes used in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "\n",
    "`(COUNT(DISTINCT ?dataType)`= count number of distinct values bound to variable `?dataType`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT (COUNT(DISTINCT ?dataType) as ?dtCount)\n",
    "WHERE {\n",
    "\n",
    "    ?dataType a rdfs:Datatype .\n",
    "    \n",
    "} LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3.2 Get and count the namespaces of all datatypes used in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "\n",
    "see also: https://stackoverflow.com/a/20054523\n",
    "\n",
    "`COUNT (DISTINCT ?dataType)` = count only distinct values bound to variable `?dataType`, i.e. make sure all occurences of classes/concepts are counted only once\n",
    "\n",
    "`REPLACE(str(?dataType), \"(#|/)[^#/]*$\", \"$1\")` = REGEX expression that is used to remove the individual class name from the namespaces bound to `?dataType`\n",
    "\n",
    "`BIND(… AS ?ns)` = bind the results of the regex replacement to the variable `?ns`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT ?ns (COUNT (DISTINCT ?dataType) AS ?dtCount)\n",
    "WHERE   \n",
    "{ \n",
    "   ?dataType a rdfs:Datatype .\n",
    "   BIND(REPLACE(str(?dataType), \"(#|/)[^#/]*$\", \"$1\") AS ?ns)\n",
    " \n",
    "}\n",
    "ORDER BY DESC(?dtCount)\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3.3 Get datatypes with a given prefix/namespace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "\n",
    "see above for classes/concepts\n",
    "\n",
    " `FILTER( STRSTARTS(STR(?datatype),str(rdf:)) )`       = filter out only those solutions for the variable `?datatype`  that start with the given string `rdf:` (try to replace with `dcterms:` etc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PREFIX dcterms: <http://purl.org/dc/terms/>\n",
    "PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n",
    "\n",
    "\n",
    "SELECT DISTINCT ?dataType \n",
    "WHERE {\n",
    "    \n",
    "    ?dataType a rdfs:Datatype .\n",
    "    FILTER( STRSTARTS(STR(?dataType),str(rdf:)))\n",
    "    \n",
    "    \n",
    "} LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3.4 Show all information about a specific datatype in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "\n",
    "see above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PREFIX dcterms: <http://purl.org/dc/terms/> \n",
    "PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>\n",
    "\n",
    "SELECT DISTINCT *\n",
    "WHERE {\n",
    "    \n",
    "    rdf:HTML ?p ?o .\n",
    "    \n",
    "} LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3.5 Which properties are used how often to describe datatypes?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explanation\n",
    "\n",
    "`(COUNT(*) AS ?n)` = bind the number of results of all bound variables to the variable `?n`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SELECT ?p (COUNT(*) as ?n)\n",
    "WHERE {\n",
    "\n",
    "    ?dataType a rdfs:Datatype ;\n",
    "              ?p ?o.\n",
    "} \n",
    "GROUP BY ?p\n",
    "ORDER BY DESC (?n)\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 Exploring a certain class/concept"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the SPARQL endpoint\n",
    "%endpoint http://cdhr-linkeddata.anu.edu.au/jazzcats-sparql/sparql"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3.0 Find all persons of the dataset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given there are any persons in the dataset at all, make sure that you use the correct entity type when you search for persons. For an overview of the entity types that are used in the dataset see 2.1.2.\n",
    "\n",
    "#### Explanation:\n",
    "\n",
    "`PREFIX foaf: <http://xmlns.com/foaf/0.1/>` = shortcut prefix for the [FOAF](http://xmlns.com/foaf/spec/) vocabulary and namespace\n",
    "\n",
    "`SELECT DISTINCT ?person` = return all unique values bound in the variable `?person` as results\n",
    "\n",
    "`foaf:Person`        = concept of Person from the [FOAF](http://xmlns.com/foaf/spec/) vocabulary (replace with the corresponding Person concept according to your dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PREFIX foaf: <http://xmlns.com/foaf/0.1/>\n",
    "\n",
    "SELECT DISTINCT ?person \n",
    "WHERE {\n",
    "    \n",
    "    ?person a foaf:Person .\n",
    "} \n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3.1 Count all persons in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given there are any persons in the dataset at all, make sure that you use the correct entity type when you search for persons. For an overview of the entity types that are used in the dataset see 2.1.2.\n",
    "\n",
    "#### Explanation:\n",
    "\n",
    "see 2.1.1\n",
    "\n",
    "`BIND(foaf:Person as ?personConcept )` = bind `foaf:Person` to the variable `?personConcept`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PREFIX foaf: <http://xmlns.com/foaf/0.1/>\n",
    "\n",
    "SELECT  ?personConcept (COUNT(?person) AS ?personCount)\n",
    "WHERE {\n",
    "    \n",
    "        ?person a foaf:Person .\n",
    "        BIND(foaf:Person as ?personConcept )\n",
    "}\n",
    "GROUP BY ?personConcept\n",
    "ORDER BY DESC (?personCount)\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3.2 Find all properties assigned to the Person concept (as subject) in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given there are any persons in the dataset at all, make sure that you use the correct entity type when you search for persons. For an overview of the entity types that are used in the dataset see 2.1.2.\n",
    "\n",
    "#### Explanation:\n",
    "\n",
    "`?p` = random variable name for any URI that matches the given pattern in the predicate position of a triple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PREFIX foaf: <http://xmlns.com/foaf/0.1/>\n",
    "\n",
    "SELECT DISTINCT ?p\n",
    "WHERE {\n",
    "    \n",
    "        ?person a foaf:Person .\n",
    "        ?person ?p ?o .\n",
    "\n",
    "}\n",
    "ORDER BY ?p\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3.3 Find and count all properties assigned to the Person concept (as subject) in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given there are any persons in the dataset at all, make sure that you use the correct entity type when you search for persons. For an overview of the entity types that are used in the dataset see 2.1.2.\n",
    "\n",
    "#### Explanation:\n",
    "\n",
    "`?p` = random variable name for any URI that matches the given pattern in the predicate position of a triple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PREFIX foaf: <http://xmlns.com/foaf/0.1/>\n",
    "\n",
    "SELECT ?p (COUNT(?p) AS ?pCount) \n",
    "WHERE\n",
    "{\n",
    "      ?person a foaf:Person .\n",
    "      ?person ?p ?o .\n",
    "}\n",
    "GROUP BY ?p\n",
    "ORDER BY DESC (?pCount)\n",
    "LIMIT 50\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3.4 Find and count all properties assigned to the Person concept (as object) in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given there are any persons in the dataset at all, make sure that you use the correct entity type when you search for persons. For an overview of the entity types that are used in the dataset see 2.1.2.\n",
    "\n",
    "#### Explanation:\n",
    "\n",
    "see 2.2.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PREFIX foaf: <http://xmlns.com/foaf/0.1/>\n",
    "\n",
    "SELECT DISTINCT ?p (COUNT(?p) AS ?pCount)\n",
    "WHERE {\n",
    "    \n",
    "        ?person a foaf:Person .\n",
    "        ?s ?p ?person .\n",
    "        \n",
    "}\n",
    "GROUP BY ?p\n",
    "ORDER BY DESC(?pCount)\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3.5 Find and count all properties assigned to the Person concept (as subject & object) in the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same query merging the two columns into one thus merging properties that are available for Persons in the position of subject and object.\n",
    "\n",
    "#### Explanation:\n",
    "\n",
    "`{…} UNION {…}` = keyword to form a disjunction of two graph patterns in terms of a logical OR; solutions to both sides of the UNION are included in the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PREFIX foaf: <http://xmlns.com/foaf/0.1/>\n",
    "\n",
    "SELECT DISTINCT ?p (COUNT(?p) AS ?pCount)\n",
    "WHERE\n",
    "{\n",
    "        ?person a foaf:Person .\n",
    "        { ?person ?p [] . }\n",
    "        UNION\n",
    "        { [] ?p ?person . }\n",
    "}\n",
    "GROUP BY ?p\n",
    "ORDER BY DESC (?pCount)\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of which type are possible subjects and objects of a property? And what does a sample look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "PREFIX mo: <http://purl.org/ontology/mo/>\n",
    "\n",
    "SELECT (SAMPLE(?s) AS ?perfomance) ?sType (COUNT(?s) AS ?pCount) ?oType (SAMPLE(?o) AS ?performer)\n",
    "WHERE\n",
    "{\n",
    "    \n",
    "      ?s mo:performer ?o .  # replace with a property that is in your dataset\n",
    "      ?s a ?sType .\n",
    "      ?o a ?oType .\n",
    "\n",
    "}\n",
    "GROUP BY ?sType ?oType\n",
    "ORDER BY DESC (?pCount)\n",
    "LIMIT 50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congratulations!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You made it through that Jupyter notebook. You should now have a basic idea of how to explore unknown/undocumented datasets with SPARQL. Feel free to examine another endpoint of your choice and play around with the classes/properties to your liking. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Enjoy and keep SPARQLing :)  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "SPARQL",
   "language": "sparql",
   "name": "sparql"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "sparql"
   },
   "mimetype": "application/sparql-query",
   "name": "sparql",
   "pygments_lexer": "sparql-nb"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}