Combine default graph and optional graphs via UNION to return all the types available in the dataset #1327

kinow · 2022-06-01T22:53:24Z

Reasons for creating this PR

Only the types of the default graph are returned in the REST API, unless the user enables the option in Jena that includes all graphs.

Link to relevant issue(s), if any

Closes The REST API types query relies on union default graph #678

Description of the changes in this PR

In the parts of the query where it references types (like rdfs label, or a subclass, etc) I have added a sibling UNION that includes the GRAPH ?g { same_statement }, to simulate what Jena does with that setting.

With the changes in this branch, after clearing the Local Storage in my browser, I can see all the types in both the REST response, and also in the UI (e.g. search auto-complete, see #1323 ), without needing to enable that option to merge the default graph in Jena Fuseki.

Known problems or uncertainties in this PR

Does it need a test? Does it sound like a valid solution? I thought about doing a simpler { { query_as_is } UNION { GRAPH ?g { query _as is } } } but since the query wasn't very long I opted for this current approach.

Checklist

phpUnit tests pass locally with my changes
I have added tests that prove my fix is effective or that my feature works (if not, explain why below)
The PR doesn't introduce unintended code changes (e.g. empty lines or useless reindentation)

codecov · 2022-06-01T22:58:32Z

Codecov Report

Merging #1327 (5d375f4) into master (5d193c2) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##             master    #1327   +/-   ##
=========================================
  Coverage     70.68%   70.68%           
- Complexity     1646     1647    +1     
=========================================
  Files            32       32           
  Lines          3786     3787    +1     
=========================================
+ Hits           2676     2677    +1     
  Misses         1110     1110

Impacted Files	Coverage Δ
model/Model.php	`80.81% <100.00%> (ø)`
model/sparql/GenericSparql.php	`91.28% <100.00%> (+0.01%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5d193c2...5d375f4. Read the comment docs.

osma · 2022-06-02T09:37:07Z

Thanks for giving this a shot @kinow , this has been a longstanding issue that has bitten many users (including yourself)!

However, I'm not sure this is the right approach, adding lots of UNION clauses to the query which seems overcomplicated to me. The intent of the types query is simply to list all the concept types across all the vocabularies. The result should be similar, if not identical, to the result of performing a types query separately for each vocabulary and merging them into one list.

There are at least a few ways of doing that that I can think of:

Relying on the union default graph (current status quo). This is really a hack and should be avoided. For one, it relies on the Fuseki setting. Second, this isn't really accurate as the dataset may include other graphs that are not configured as Skosmos vocabularies but may still contain concept type definitions that will end up in the results.
Adding a single GRAPH ?g { block around the whole query. I think that this should be equivalent to the above, at least in sensible scenarios (each graph contains both the type definition and its label). It's still a bit problematic since types from graphs that are not configured as vocabularies may be included.
Same as 2, but selecting only those graphs that are configured as vocabularies and using VALUES ?g to enumerate those, thus excluding irrelevant graphs.
Same idea as 3, but using FROM NAMED clauses for selecting individual graphs instead of GRAPH ?g with VALUES ?g.

Would you like to try some of the approaches 2-4? I think any of them would be an improvement over the current situation, as long as the query still performs well.

kinow · 2022-06-02T10:22:26Z

Would you like to try some of the approaches 2-4? I think any of them would be an improvement over the current situation, as long as the query still performs well.

Sounds like fun! I will give it a try later after or in between the other JQuery/Bootstrap work. Will be a good way to brush up SPARQL and Jena Fuseki :)

Do you have any suggestion on which item to try first (or to start thinking about), between options 2, 3, and 4?

Thanks!

osma · 2022-06-02T10:50:13Z

I think 3 or 4 would be most preferable as they don't suffer from the problem of irrelevant data being picked up from other possible named graphs in the dataset.

model/sparql/GenericSparql.php

kinow · 2022-06-05T03:46:25Z

model/sparql/GenericSparql.php

-    private function generateQueryTypesQuery($lang) {
-        $fcl = $this->generateFromClause();
+    private function generateQueryTypesQuery(string $lang): string {
+        $vocabs = $this->model->getVocabularies();


If there are no vocabs, then the query will return all the types from all the graphs. Would it be better to return an empty result int that case?

kinow · 2022-06-05T03:51:49Z

I decided to try FROM NAMED clauses since we already had the code for that. We actually called the function for that creating the $fcl variable, but without passing any vocabularies, so it wasn't returning anything.

Here's an example of what the query may look like. It is from my development environment with a few vocabularies configured.

PREFIX  skos: <http://www.w3.org/2004/02/skos/core#>
PREFIX  isothes: <http://purl.org/iso25964/skos-thes#>
PREFIX  rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX  finaf: <http://urn.fi/URN:NBN:fi:au:finaf:>

SELECT DISTINCT  ?type ?label ?superclass
FROM NAMED <http://skos.um.es/unescothes/>
FROM NAMED <http://zbw.eu/stw/>
FROM NAMED <http://vocabs.rossio.fcsh.unl.pt/tesauro/>
FROM NAMED <http://www.yso.fi/onto/yso/>
FROM NAMED finaf:
WHERE
  { GRAPH ?g
      { {   { BIND(skos:Concept AS ?type) }
          UNION
            { BIND(skos:Collection AS ?type) }
          UNION
            { BIND(isothes:ConceptGroup AS ?type) }
          UNION
            { BIND(isothes:ThesaurusArray AS ?type) }
          UNION
            { ?type rdfs:subClassOf/(rdfs:subClassOf)* skos:Concept }
          UNION
            { ?type rdfs:subClassOf/(rdfs:subClassOf)* skos:Collection }
        }
        OPTIONAL
          { ?type  rdfs:label  ?label
            FILTER langMatches(lang(?label), "en")
          }
        OPTIONAL
          { ?type  rdfs:subClassOf  ?superclass }
        FILTER EXISTS { ?s  a               ?type ;
                            skos:prefLabel  ?prefLabel
                      }
      }
  }

And the REST API response:

{
  "@context": {
    "skos": "http:\\/\\/www.w3.org\\/2004\\/02\\/skos\\/core#",
    "uri": "@id",
    "type": "@type",
    "rdfs": "http:\\/\\/www.w3.org\\/2000\\/01\\/rdf-schema#",
    "onki": "http:\\/\\/schema.onki.fi\\/onki#",
    "label": "rdfs:label",
    "superclass": {
      "@id": "rdfs:subClassOf",
      "@type": "@id"
    },
    "types": "onki:hasType",
    "@language": "en",
    "@base": "http:\\/\\/localhost:9090\\/rest\\/v1\\/"
  },
  "uri": "",
  "types": [
    {
      "uri": "http:\\/\\/www.w3.org\\/2004\\/02\\/skos\\/core#Concept",
      "label": "Concept"
    },
    {
      "uri": "http:\\/\\/zbw.eu\\/namespaces\\/zbw-extensions\\/Descriptor",
      "label": "Descriptor",
      "superclass": "http:\\/\\/www.w3.org\\/2004\\/02\\/skos\\/core#Concept"
    },
    {
      "uri": "http:\\/\\/zbw.eu\\/namespaces\\/zbw-extensions\\/Thsys",
      "label": "Thsys",
      "superclass": "http:\\/\\/www.w3.org\\/2004\\/02\\/skos\\/core#Concept"
    },
    {
      "uri": "http:\\/\\/www.w3.org\\/2004\\/02\\/skos\\/core#Collection",
      "label": "Collection"
    },
    {
      "uri": "http:\\/\\/purl.org\\/iso25964\\/skos-thes#ConceptGroup",
      "label": "Concept group"
    },
    {
      "uri": "http:\\/\\/purl.org\\/iso25964\\/skos-thes#ThesaurusArray",
      "label": "Array of sibling concepts"
    },
    {
      "uri": "http:\\/\\/www.yso.fi\\/onto\\/yso-meta\\/Concept",
      "label": "General concept",
      "superclass": "http:\\/\\/www.w3.org\\/2004\\/02\\/skos\\/core#Concept"
    },
    {
      "uri": "http:\\/\\/www.yso.fi\\/onto\\/yso-meta\\/Individual",
      "label": "Individual concept",
      "superclass": "http:\\/\\/www.w3.org\\/2004\\/02\\/skos\\/core#Concept"
    },
    {
      "uri": "http:\\/\\/www.yso.fi\\/onto\\/yso-meta\\/Hierarchy",
      "label": "Hierarchical concept",
      "superclass": "http:\\/\\/www.w3.org\\/2004\\/02\\/skos\\/core#Concept"
    }
  ]
}

kinow · 2022-06-05T04:43:13Z

tests/GenericSparqlTest.php

+      ),
+      'http://www.w3.org/2004/02/skos/core#Collection' => array(),
+      'http://purl.org/iso25964/skos-thes#ThesaurusArray' => array(),
+      'http://purl.org/iso25964/skos-thes#ConceptGroup' => array()


Had a look at testconfig.ttl, and I think the function shuold return these new types now as they are types used in vocabularies there. These extra values are added now since the query is searching types in all graphs for all configured vocabs.

sonarcloud · 2022-06-05T04:49:54Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
1 Security Hotspot
1 Code Smell

No Coverage information
0.0% Duplication

kinow · 2022-06-05T04:50:05Z

tests/ModelTest.php

+            'http://www.w3.org/2004/02/skos/core#Concept',
+            'http://www.w3.org/2004/02/skos/core#Collection',
+            'http://purl.org/iso25964/skos-thes#ThesaurusArray',
+            'http://purl.org/iso25964/skos-thes#ConceptGroup',


The only change in this array is the order of ThesaurusArray and ConceptGroup. Not sure why these two swapped places in this array.

kinow · 2022-06-05T04:57:28Z

The SonarCloud security issue reported is for a prefix URL, so that shouldn't be a blocker, I think. Ready for review again 👍

osma · 2022-09-12T12:30:09Z

model/sparql/GenericSparql.php

-  FILTER EXISTS {
-    ?s a ?type .
-    ?s skos:prefLabel ?prefLabel .
+  GRAPH ?g {


This looks wrong to me. You are already using $fcl i.e. a generated block of FROM clauses. There is no need for the GRAPH ?g { block.

osma · 2022-09-13T12:51:27Z

I can try to finish this off now that @kinow you are busy with the qtip-to-CSS PR #1324 ...

kinow · 2022-09-13T19:51:16Z

I can try to finish this off now that @kinow you are busy with the qtip-to-CSS PR #1324 ...

That'd be great, please :D Thank you @osma!

kinow force-pushed the fix-678 branch from f87aff8 to 1a4e395 Compare June 5, 2022 03:28

kinow commented Jun 5, 2022

View reviewed changes

model/sparql/GenericSparql.php Outdated Show resolved Hide resolved

kinow force-pushed the fix-678 branch from 1a4e395 to 6fec308 Compare June 5, 2022 03:44

kinow commented Jun 5, 2022

View reviewed changes

Use a FROM NAMED clause to query types per vocabs installed only

5d375f4

kinow force-pushed the fix-678 branch from b1bfc53 to 5d375f4 Compare June 5, 2022 04:48

kinow commented Jun 5, 2022

View reviewed changes

osma added this to Recent PRs from Skosmos users in Sprint Backlog III/2022 Jun 23, 2022

MikkoAleksanteri moved this from Recent PRs from Skosmos users to Selected product backlog items in Sprint Backlog III/2022 Sep 1, 2022

osma self-assigned this Sep 12, 2022

osma reviewed Sep 12, 2022

View reviewed changes

osma moved this from Selected product backlog items to Reviewed - further actions needed in Sprint Backlog III/2022 Sep 12, 2022

joelit moved this from Reviewed - further actions needed to Postponed in Sprint Backlog III/2022 Sep 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Combine default graph and optional graphs via UNION to return all the types available in the dataset #1327

Combine default graph and optional graphs via UNION to return all the types available in the dataset #1327

kinow commented Jun 1, 2022 •

edited

codecov bot commented Jun 1, 2022 •

edited

osma commented Jun 2, 2022

kinow commented Jun 2, 2022 •

edited

osma commented Jun 2, 2022

kinow Jun 5, 2022

kinow commented Jun 5, 2022

kinow Jun 5, 2022

sonarcloud bot commented Jun 5, 2022

kinow Jun 5, 2022

kinow commented Jun 5, 2022 •

edited

osma Sep 12, 2022

osma commented Sep 13, 2022

kinow commented Sep 13, 2022

Combine default graph and optional graphs via UNION to return all the types available in the dataset #1327

Are you sure you want to change the base?

Combine default graph and optional graphs via UNION to return all the types available in the dataset #1327

Conversation

kinow commented Jun 1, 2022 • edited

Reasons for creating this PR

Link to relevant issue(s), if any

Description of the changes in this PR

Known problems or uncertainties in this PR

Checklist

codecov bot commented Jun 1, 2022 • edited

Codecov Report

osma commented Jun 2, 2022

kinow commented Jun 2, 2022 • edited

osma commented Jun 2, 2022

kinow Jun 5, 2022

Choose a reason for hiding this comment

kinow commented Jun 5, 2022

kinow Jun 5, 2022

Choose a reason for hiding this comment

sonarcloud bot commented Jun 5, 2022

kinow Jun 5, 2022

Choose a reason for hiding this comment

kinow commented Jun 5, 2022 • edited

osma Sep 12, 2022

Choose a reason for hiding this comment

osma commented Sep 13, 2022

kinow commented Sep 13, 2022

kinow commented Jun 1, 2022 •

edited

codecov bot commented Jun 1, 2022 •

edited

kinow commented Jun 2, 2022 •

edited

kinow commented Jun 5, 2022 •

edited