diff --git a/Makefile b/Makefile index 5206455c5b..707c95d5b1 100644 --- a/Makefile +++ b/Makefile @@ -280,6 +280,13 @@ clean-local-tmp: @rm -rf .tmp @mkdir .tmp +.PHONY: clean-metals +clean-metals: ## clean SBT and Metals related stuff + @rm -rf .bloop + @rm -rf .bsp + @rm -rf .metals + @rm -rf target + clean: docs-clean clean-local-tmp clean-docker clean-sipi-tmp ## clean build artifacts @rm -rf .env diff --git a/docs/01-introduction/what-is-knora.md b/docs/01-introduction/what-is-knora.md index 2a4aac52c5..a2e3f9dc01 100644 --- a/docs/01-introduction/what-is-knora.md +++ b/docs/01-introduction/what-is-knora.md @@ -74,7 +74,7 @@ and can regenerate the original XML document at any time. DSP-API provides a search language, [Gravsearch](../03-apis/api-v2/query-language.md), that is designed to meet the needs of humanities researchers. Gravsearch supports DSP-API's -humanites-focused data structures, including calendar-independent dates and standoff markup, as well +humanities-focused data structures, including calendar-independent dates and standoff markup, as well as fast full-text searches. This allows searches to combine text-related criteria with any other criteria. For example, you could search for a text that contains a certain word and also mentions a person who lived in the same city as another person who is the diff --git a/docs/03-apis/api-v2/query-language.md b/docs/03-apis/api-v2/query-language.md index d41a36978c..48f36c0d9b 100644 --- a/docs/03-apis/api-v2/query-language.md +++ b/docs/03-apis/api-v2/query-language.md @@ -13,15 +13,15 @@ criteria) while avoiding their drawbacks in terms of performance and security (see [The Enduring Myth of the SPARQL Endpoint](https://daverog.wordpress.com/2013/06/04/the-enduring-myth-of-the-sparql-endpoint/)). It also has the benefit of enabling clients to work with a simpler RDF -data model than the one Knora actually uses to store data in the +data model than the one the API actually uses to store data in the triplestore, and makes it possible to provide better error-checking. Rather than being processed directly by the triplestore, a Gravsearch query -is interpreted by Knora, which enforces certain +is interpreted by the API, which enforces certain restrictions on the query, and implements paging and permission checking. The API server generates SPARQL based on the Gravsearch query submitted, queries the triplestore, filters the results according to the -user's permissions, and returns each page of query results as a Knora +user's permissions, and returns each page of query results as an API response. Thus, Gravsearch is a hybrid between a RESTful API and a SPARQL endpoint. @@ -80,14 +80,14 @@ If a gravsearch query times out, a `504 Gateway Timeout` will be returned. A Gravsearch query can be written in either of the two [DSP-API v2 schemas](introduction.md#api-schema). The simple schema is easier to work with, and is sufficient if you don't need to query -anything below the level of a Knora value. If your query needs to refer to +anything below the level of a DSP-API value. If your query needs to refer to standoff markup, you must use the complex schema. Each query must use a single schema, with one exception (see [Date Comparisons](#date-comparisons)). Gravsearch query results can be requested in the simple or complex schema; see [API Schema](introduction.md#api-schema). -All examples hereafter run with Knora started locally as documented in the section [Getting Started with DSP-API](../../04-publishing-deployment/getting-started.md). If you access another Knora-Stack, you can check the IRI of the ontology you are targeting by requesting the [ontologies metadata](ontology-information.md#querying-ontology-metadata). +All examples hereafter run with the DSP stack started locally as documented in the section [Getting Started with DSP-API](../../04-publishing-deployment/getting-started.md). If you access another stack, you can check the IRI of the ontology you are targeting by requesting the [ontologies metadata](ontology-information.md#querying-ontology-metadata). ### Using the Simple Schema @@ -100,8 +100,7 @@ PREFIX knora-api: PREFIX incunabula: ``` -In the simple schema, Knora values are represented as literals, which -can be used `FILTER` expressions +In the simple schema, DSP-API values are represented as literals, which can be used `FILTER` expressions (see [Filtering on Values in the Simple Schema](#filtering-on-values-in-the-simple-schema)). ### Using the Complex Schema @@ -115,7 +114,7 @@ PREFIX knora-api: PREFIX incunabula: ``` -In the complex schema, Knora values are represented as objects belonging +In the complex schema, DSP-API values are represented as objects belonging to subclasses of `knora-api:Value`, e.g. `knora-api:TextValue`, and have predicates of their own, which can be used in `FILTER` expressions (see [Filtering on Values in the Complex Schema](#filtering-on-values-in-the-complex-schema)). @@ -182,7 +181,7 @@ permission to see a matching dependent resource, the link value is hidden. ## Paging Gravsearch results are returned in pages. The maximum number of main -resources per page is determined by Knora (and can be configured +resources per page is determined by the API (and can be configured in `application.conf` via the setting `app/v2/resources-sequence/results-per-page`). If some resources have been filtered out because the user does not have permission to see them, a page could contain fewer results, or no results. @@ -195,9 +194,7 @@ one at a time, until the response does not contain `knora-api:mayHaveMoreResults ## Inference Gravsearch queries are understood to imply a subset of -[RDFS reasoning](https://www.w3.org/TR/rdf11-mt/). Depending on the -triplestore being used, this may be implemented using the triplestore's -own reasoner or by query expansion in Knora. +[RDFS reasoning](https://www.w3.org/TR/rdf11-mt/). This is done by the API by expanding the incoming query. Specifically, if a statement pattern specifies a property, the pattern will also match subproperties of that property, and if a statement specifies that @@ -205,15 +202,12 @@ a subject has a particular `rdf:type`, the statement will also match subjects belonging to subclasses of that type. If you know that reasoning will not return any additional results for -your query, you can disable it by adding this line to the `WHERE` clause: +your query, you can disable it by adding this line to the `WHERE` clause, which may improve query performance: ```sparql knora-api:GravsearchOptions knora-api:useInference false . ``` -If Knora is implementing reasoning by query expansion, disabling it can -improve the performance of some queries. - ## Gravsearch Syntax Every Gravsearch query is a valid SPARQL 1.1 @@ -244,8 +238,8 @@ clauses use the following patterns, with the specified restrictions: unordered set of triples. However, a Gravsearch query returns an ordered list of resources, which can be ordered by the values of specified properties. If the query is written in the complex schema, - items below the level of Knora values may not be used in `ORDER BY`. -- `BIND`: The value assigned must be a Knora resource IRI. + items below the level of DSP-API values may not be used in `ORDER BY`. +- `BIND`: The value assigned must be a DSP resource IRI. ### Resources, Properties, and Values @@ -269,7 +263,7 @@ must be represented as a query variable. #### Filtering on Values in the Simple Schema -In the simple schema, a variable representing a Knora value can be used +In the simple schema, a variable representing a DSP-API value can be used directly in a `FILTER` expression. For example: ``` @@ -279,7 +273,7 @@ FILTER(?title = "Zeitglöcklein des Lebens und Leidens Christi") Here the type of `?title` is `xsd:string`. -The following Knora value types can be compared with literals in `FILTER` +The following value types can be compared with literals in `FILTER` expressions in the simple schema: - Text values (`xsd:string`) @@ -295,7 +289,7 @@ performing an exact match on a list node's label. Labels can be given in differe If one of the given list node labels matches, it is considered a match. Note that in the simple schema, uniqueness is not guaranteed (as opposed to the complex schema). -A Knora value may not be represented as the literal object of a predicate; +A DSP-API value may not be represented as the literal object of a predicate; for example, this is not allowed: ``` @@ -304,9 +298,9 @@ for example, this is not allowed: #### Filtering on Values in the Complex Schema -In the complex schema, variables representing Knora values are not literals. +In the complex schema, variables representing DSP-API values are not literals. You must add something to the query (generally a statement) to get a literal -from a Knora value. For example: +from a DSP-API value. For example: ``` ?book incunabula:title ?title . @@ -479,7 +473,7 @@ within a single paragraph. If you are only interested in specifying that a resource has some text value containing a standoff link to another resource, the most efficient way is to use the property `knora-api:hasStandoffLinkTo`, whose subjects and objects -are resources. This property is automatically maintained by Knora. For example: +are resources. This property is automatically maintained by the API. For example: ``` PREFIX knora-api: @@ -623,7 +617,7 @@ CONSTRUCT { ### Filtering on `rdfs:label` -The `rdfs:label` of a resource is not a Knora value, but you can still search for it. +The `rdfs:label` of a resource is not a DSP-API value, but you can still search for it. This can be done in the same ways in the simple or complex schema: Using a string literal object: @@ -708,8 +702,8 @@ clause but not in the `CONSTRUCT` clause, the matching resources or values will not be included in the results. If the query is written in the complex schema, all variables in the `CONSTRUCT` -clause must refer to Knora resources, Knora values, or properties. Data below -the level of Knora values may not be mentioned in the `CONSTRUCT` clause. +clause must refer to DSP-API resources, DSP-API values, or properties. Data below +the level of values may not be mentioned in the `CONSTRUCT` clause. Predicates from the `rdf`, `rdfs`, and `owl` ontologies may not be used in the `CONSTRUCT` clause. The `rdfs:label` of each matching resource is always @@ -921,7 +915,7 @@ adding statements with the predicate `rdf:type`. The subject must be a resource and the object must either be `knora-api:Resource` (if the subject is a resource) or the subject's specific type (if it is a value). -For example, consider this query that uses a non-Knora property: +For example, consider this query that uses a non-DSP property: ``` PREFIX incunabula: @@ -992,7 +986,7 @@ CONSTRUCT { Note that it only makes sense to use `dcterms:title` in the simple schema, because its object is supposed to be a literal. -Here is another example, using a non-Knora class: +Here is another example, using a non-DSP class: ``` PREFIX knora-api: diff --git a/docs/05-internals/design/api-v2/gravsearch.md b/docs/05-internals/design/api-v2/gravsearch.md index 0dade5ef11..dad4aca225 100644 --- a/docs/05-internals/design/api-v2/gravsearch.md +++ b/docs/05-internals/design/api-v2/gravsearch.md @@ -128,7 +128,7 @@ pattern orders must be optimised by moving `LuceneQueryPatterns` to the beginnin - `ConstructToConstructTransformer` (extends `WhereTransformer`): instructions how to turn a triplestore independent Construct query into a triplestore dependent Construct query (implementation of inference). The traits listed above define methods that are implemented in the transformer classes and called by `QueryTraverser` to perform SPARQL to SPARQL conversions. -When iterating over the statements of the input query, the transformer class's transformation methods are called to perform the conversion. +When iterating over the statements of the input query, the transformer class' transformation methods are called to perform the conversion. ### Prequery @@ -152,7 +152,7 @@ Next, the Gravsearch query's WHERE clause is transformed and the prequery (SELEC The transformation of the Gravsearch query's WHERE clause relies on the implementation of the abstract class `AbstractPrequeryGenerator`. `AbstractPrequeryGenerator` contains members whose state is changed during the iteration over the statements of the input query. -They can then by used to create the converted query. +They can then be used to create the converted query. - `mainResourceVariable: Option[QueryVariable]`: SPARQL variable representing the main resource of the input query. Present in the prequery's SELECT clause. - `dependentResourceVariables: mutable.Set[QueryVariable]`: a set of SPARQL variables representing dependent resources in the input query. Used in an aggregation function in the prequery's SELECT clause (see below). @@ -288,29 +288,12 @@ to the maximum allowed page size, the predicate ## Inference -Gravsearch queries support a subset of RDFS reasoning -(see [Inference](../../../03-apis/api-v2/query-language.md#inference) in the API documentation +Gravsearch queries support a subset of RDFS reasoning (see [Inference](../../../03-apis/api-v2/query-language.md#inference) in the API documentation on Gravsearch). This is implemented as follows: -When the non-triplestore-specific version of a SPARQL query is generated, statements that do not need -inference are marked with the virtual named graph ``. +To simulate RDF inference, the API expands the prequery on basis of the available ontologies. For that reason, `SparqlTransformer.transformStatementInWhereForNoInference` expands all `rdfs:subClassOf` and `rdfs:subPropertyOf` statements using `UNION` statements for all subclasses and subproperties from the ontologies (equivalent to `rdfs:subClassOf*` and `rdfs:subPropertyOf*`). +Similarly, `SparqlTransformer.transformStatementInWhereForNoInference` replaces `knora-api:standoffTagHasStartAncestor` with `knora-base:standoffTagHasStartParent*`. -When the triplestore-specific version of the query is generated: - -- If the triplestore is GraphDB, `SparqlTransformer.transformKnoraExplicitToGraphDBExplicit` changes statements - with the virtual graph `` so that they are marked with the GraphDB-specific graph - ``, and leaves other statements unchanged. - `SparqlTransformer.transformKnoraExplicitToGraphDBExplicit` also adds the `valueHasString` statements which GraphDB needs - for text searches. - -- If Knora is not using the triplestore's inference (e.g. with Fuseki), - `SparqlTransformer.transformStatementInWhereForNoInference` removes ``, and expands unmarked - statements using `rdfs:subClassOf*` and `rdfs:subPropertyOf*`. - -Gravsearch also provides some virtual properties, which take advantage of forward-chaining inference -as an optimisation if the triplestore provides it. For example, the virtual property -`knora-api:standoffTagHasStartAncestor` is equivalent to `knora-base:standoffTagHasStartParent*`. If Knora is not using the triplestore's inference, `SparqlTransformer.transformStatementInWhereForNoInference` -replaces `knora-api:standoffTagHasStartAncestor` with `knora-base:standoffTagHasStartParent*`. # Optimisation of generated SPARQL @@ -320,8 +303,7 @@ Lucene queries to the beginning of the block in which they occur. ## Query Optimization by Topological Sorting of Statements -GraphDB seems to have inherent algorithms to optimize the query time, however query performance of Fuseki highly depends -on the order of the query statements. For example, a query such as the one below: +In Jena Fuseki, the performance of a query highly depends on the order of the query statements. For example, a query such as the one below: ```sparql PREFIX beol: @@ -370,8 +352,7 @@ The rest of the query then reads: ?letter beol:creationDate ?date . ``` -Since we cannot expect clients to know about performance of triplestores in order to write efficient queries, we have -implemented an optimization method to automatically rearrange the statements of the given queries. +Since users cannot be expected to know about performance of triplestores in order to write efficient queries, an optimization method to automatically rearrange the statements of the given queries has been implemented. Upon receiving the Gravsearch query, the algorithm converts the query to a graph. For each statement pattern, the subject of the statement is the origin node, the predicate is a directed edge, and the object is the target node. For the query above, this conversion would result in the following graph: @@ -384,17 +365,16 @@ topological sorting algorithm](https://en.wikipedia.org/wiki/Topological_sorting The algorithm returns the nodes of the graph ordered in several layers, where the root element `?letter` is in layer 0, `[?date, ?person1, ?person2]` are in layer 1, `[?gnd1, ?gnd2]` in layer 2, and the leaf nodes `[(DE-588)118531379, (DE-588)118696149]` are given in the last layer (i.e. layer 3). -According to Kahn's algorithm, there are multiple valid permutations of the topological order. The graph in the example - above has 24 valid permutations of topological order. Here are two of them (nodes are ordered from left to right with the highest - order to the lowest): +According to Kahn's algorithm, there are multiple valid permutations of the topological order. The graph in the example +above has 24 valid permutations of topological order. Here are two of them (nodes are ordered from left to right with the +highest order to the lowest): - `(?letter, ?date, ?person2, ?person1, ?gnd2, ?gnd1, (DE-588)118696149, (DE-588)118531379)` - `(?letter, ?date, ?person1, ?person2, ?gnd1, ?gnd2, (DE-588)118531379, (DE-588)118696149)`. -From all valid topological orders, one is chosen based on certain criteria; for example, the leaf should node should not +From all valid topological orders, one is chosen based on certain criteria; for example, the leaf node should not belong to a statement that has predicate `rdf:type`, since that could match all resources of the specified type. -Once the best order is chosen, it is used to re-arrange the query -statements. Starting from the last leaf node, i.e. +Once the best order is chosen, it is used to re-arrange the query statements. Starting from the last leaf node, i.e. `(DE-588)118696149`, the method finds the statement pattern which has this node as its object, and brings this statement to the top of the query. This rearrangement continues so that the statements with the fewest dependencies on other statements are all brought to the top of the query. The resulting query is as follows: @@ -423,8 +403,7 @@ CONSTRUCT { Note that position of the FILTER statements does not play a significant role in the optimization. -If a Gravsearch query contains statements in `UNION`, `OPTIONAL`, `MINUS`, or -`FILTER NOT EXISTS`, they are reordered +If a Gravsearch query contains statements in `UNION`, `OPTIONAL`, `MINUS`, or `FILTER NOT EXISTS`, they are reordered by defining a graph per block. For example, consider the following query with `UNION`: ```sparql diff --git a/docs/05-internals/design/api-v2/query-design.md b/docs/05-internals/design/api-v2/query-design.md index 1d023104f9..e4fb03b12f 100644 --- a/docs/05-internals/design/api-v2/query-design.md +++ b/docs/05-internals/design/api-v2/query-design.md @@ -7,10 +7,9 @@ ## Inference -Knora does not require the triplestore to perform inference, but may be able -to take advantage of inference if the triplestore provides it. +DSP-API does not require the triplestore to perform inference, as different triplestores implement inference quite differently, so that taking advantage of inference would require triplestore specific code, which is not well maintainable. Instead, the API simulates inference for each Gravsearch query, so that the expected results are returned. -In particular, Knora's SPARQL queries currently need to do the following: +Gravsearch queries currently need to do the following: - Given a base property, find triples using a subproperty as predicate, and return the subproperty used in each case. @@ -39,64 +38,35 @@ This query: - Finds the Knora values attached to the resource, and returns each value along with the property that explicitly attaches it to the resource. -In some triplestores, it can be more efficient to use RDFS inference than to use property path syntax, -depending on how inference is implemented. For example, Ontotext GraphDB does inference when -data is inserted, and stores inferred triples in the repository -([forward chaining with full materialisation](http://graphdb.ontotext.com/documentation/standard/reasoning.html)). -Moreover, it provides a way of choosing whether to return explicit or inferred triples. -This allows the query above to be optimised as follows, querying inferred triples but returning -explicit triples: +However, such a query is very inefficient. Instead, the API does inference on the query, so that the relevant information can be found in a timely manner. -```sparql -CONSTRUCT { - ?resource a ?resourceClass . - ?resource ?resourceValueProperty ?valueObject. -WHERE { - ?resource a knora-base:Resource . # inferred triple +For this, the query is analyzed to check which project ontologies are relevant to the query. If an ontology is not relevant to a query, then all class and property definitions of this ontology are disregarded for inference. - GRAPH { - ?resource a ?resourceClass . # explicit triple - } +Then, each statement that requires inference (i.e. that could be phrased with property path syntax, as described above) is cross-referenced with the relevant ontologies, to see which property/class definitions would fit the statement according to the rules of RDF inference. And each of those definitions is added to the query as a separate `UNION` statement. - ?resource knora-base:hasValue ?valueObject . # inferred triple +E.g.: Given the resource class `B` is a subclass of `A` and the property `hasY` is a subproperty of `hasX`, then the following query - GRAPH { - ?resource ?resourceValueProperty ?valueObject . # explicit triple - } +```sparql +SELECT { + ?res ?prop . +} WHERE { + ?res a . + ?res ?prop . +} ``` -By querying inferred triples that are already stored in the repository, the optimised query avoids property path -syntax and is therefore more efficient, while still only returning explicit triples in the query result. - -Other triplestores use a backward-chaining inference strategy, meaning that inference is performed during -the execution of a SPARQL query, by expanding the query itself. The expanded query is likely to look like -the first example, using property path syntax, and therefore it is not likely to be more efficient. Moreover, -other triplestores may not provide a way to return explicit rather than inferred triples. To support such -a triplestore, Knora uses property path syntax rather than inference. -See [the Gravsearch design documentation](gravsearch.md#inference) for information on how this is done -for Gravsearch queries. - -The support for Apache Jena Fuseki currently works in this way. However, Fuseki supports both forward-chaining -and backward-chaining rule engines, although it does not seem to have anything like -GraphDB's ``. It would be worth exploring whether Knora's query result -processing could be changed so that it could use forward-chaining inference as an optimisation, even if -nothing like `` is available. For example, the example query= could be written like -this: +can be rewritten as ```sparql -CONSTRUCT { - ?resource a ?resourceClass . - ?resource ?resourceValueProperty ?valueObject . -WHERE { - ?resource a knora-base:Resource . - ?resource a ?resourceClass . - ?resource knora-base:hasValue ?valueObject . - ?resource ?resourceValueProperty ?valueObject . +SELECT { + ?res ?prop . +} WHERE { + {?res a } UNION {?res a } . + {?res ?prop} UNION {?res ?prop} . +} + ``` -This would return inferred triples as well as explicit ones: a triple for each base class of the explicit -`?resourceClass`, and a triple for each base property of the explicit `?resourceValueProperty`. But since Knora knows -the class and property inheritance hierarchies, it could ignore the additional triples. ## Querying Past Value Versions diff --git a/webapi/src/main/scala/org/knora/webapi/messages/util/search/QueryTraverser.scala b/webapi/src/main/scala/org/knora/webapi/messages/util/search/QueryTraverser.scala index 1a2b56f677..0750435a49 100644 --- a/webapi/src/main/scala/org/knora/webapi/messages/util/search/QueryTraverser.scala +++ b/webapi/src/main/scala/org/knora/webapi/messages/util/search/QueryTraverser.scala @@ -5,6 +5,23 @@ package org.knora.webapi.messages.util.search +import akka.actor.ActorRef +import akka.pattern.ask +import akka.util.Timeout +import org.knora.webapi.InternalSchema +import org.knora.webapi.messages.IriConversions._ +import org.knora.webapi.messages.OntologyConstants +import org.knora.webapi.messages.SmartIri +import org.knora.webapi.messages.StringFormatter +import org.knora.webapi.messages.admin.responder.projectsmessages.ProjectADM +import org.knora.webapi.messages.admin.responder.projectsmessages.ProjectIdentifierADM +import org.knora.webapi.messages.store.cacheservicemessages.CacheServiceGetProjectADM +import org.knora.webapi.responders.v2.ontology.Cache + +import scala.concurrent.ExecutionContext +import scala.concurrent._ +import scala.concurrent.duration._ + /** * A trait for classes that visit statements and filters in WHERE clauses, accumulating some result. * @@ -58,14 +75,16 @@ trait WhereTransformer { /** * Transforms a [[StatementPattern]] in a WHERE clause into zero or more query patterns. * - * @param statementPattern the statement to be transformed. - * @param inputOrderBy the ORDER BY clause in the input query. + * @param statementPattern the statement to be transformed. + * @param inputOrderBy the ORDER BY clause in the input query. + * @param limitInferenceToOntologies a set of ontology IRIs, to which the simulated inference will be limited. If `None`, all possible inference will be done. * @return the result of the transformation. */ def transformStatementInWhere( statementPattern: StatementPattern, - inputOrderBy: Seq[OrderCriterion] - ): Seq[QueryPattern] + inputOrderBy: Seq[OrderCriterion], + limitInferenceToOntologies: Option[Set[SmartIri]] = None + )(implicit executionContext: ExecutionContext): Seq[QueryPattern] /** * Transforms a [[FilterPattern]] in a WHERE clause into zero or more query patterns. @@ -183,20 +202,117 @@ trait ConstructToSelectTransformer extends WhereTransformer { * or [[ConstructToSelectTransformer]]. */ object QueryTraverser { + private implicit val stringFormatter: StringFormatter = StringFormatter.getGeneralInstance + private implicit val timeout: Timeout = Duration(5, SECONDS) + + /** + * Helper method that analyzed an RDF Entity and returns a sequence of Ontology IRIs that are being referenced by the entity. + * If an IRI appears that can not be resolved by the ontology cache, it will check if the IRI points to project data; + * if so, all ontologies defined by the project to which the data belongs, will be included in the results. + * + * @param entity an RDF entity. + * @param map a map of entity IRIs to the IRIs of the ontology where they are defined. + * @param storeManager a reference to the storeManager to retrieve a [[ProjectADM]] by a shortcode. + * @return a sequence of ontology IRIs which relate to the input RDF entity. + */ + private def resolveEntity(entity: Entity, map: Map[SmartIri, SmartIri], storeManager: ActorRef): Seq[SmartIri] = + entity match { + case IriRef(iri, _) => { + val internal = iri.toOntologySchema(InternalSchema) + val maybeOntoIri = map.get(internal) + maybeOntoIri match { + // if the map contains an ontology IRI corresponding to the entity IRI, then this can be returned + case Some(iri) => Seq(iri) + case None => { + // if the map doesn't contain a corresponding ontology IRI, then the entity IRI points to a resource or value + // in that case, all ontologies of the project, to which the entity belongs, should be returned. + val shortcode = internal.getProjectCode + shortcode match { + case None => Seq.empty + case _ => { + // find the project with the shortcode + val projectFuture = + (storeManager ? CacheServiceGetProjectADM(ProjectIdentifierADM(maybeShortcode = shortcode))) + .mapTo[Option[ProjectADM]] + val projectMaybe = Await.result(projectFuture, 1.second) + projectMaybe match { + case None => Seq.empty + // return all ontologies of the project + case Some(project) => project.ontologies.map(_.toSmartIri) + } + } + } + } + } + } + case _ => Seq.empty + } + + /** + * Extracts all ontologies that are relevant to a gravsearch query, in order to allow optimized cache-based inference simulation. + * + * @param whereClause the WHERE-clause of a gravsearch query. + * @param storeManager a reference to the storeManager. + * @return a set of ontology IRIs relevant to the query, or `None`, if no meaningful result could be produced. + * In the latter case, inference should be done on the basis of all available ontologies. + */ + def getOntologiesRelevantForInference( + whereClause: WhereClause, + storeManager: ActorRef + )(implicit executionContext: ExecutionContext): Future[Option[Set[SmartIri]]] = { + // internal function for easy recursion + // gets a sequence of [[QueryPattern]] and returns the set of entities that the patterns consist of + def getEntities(patterns: Seq[QueryPattern]): Seq[Entity] = + patterns.flatMap { pattern => + pattern match { + case ValuesPattern(_, values) => values.toSeq + case UnionPattern(blocks) => blocks.flatMap(block => getEntities(block)) + case StatementPattern(subj, pred, obj, _) => List(subj, pred, obj) + case LuceneQueryPattern(subj, obj, _, _) => List(subj, obj) + case FilterNotExistsPattern(patterns) => getEntities(patterns) + case MinusPattern(patterns) => getEntities(patterns) + case OptionalPattern(patterns) => getEntities(patterns) + case _ => List.empty + } + } + + // get the entities for all patterns in the WHERE clause + val entities = getEntities(whereClause.patterns) + + for { + ontoCache <- Cache.getCacheData + // from the cache, get the map from entity to the ontology where the entity is defined + entityMap = ontoCache.entityDefinedInOntology + // resolve all entities from the WHERE clause to the ontology where they are defined + relevantOntologies = entities.flatMap(resolveEntity(_, entityMap, storeManager)).toSet + relevantOntologiesMaybe = + relevantOntologies match { + case Nil => None // if nothing was found, then None should be returned + case ontologies => + // if only knora-base was found, then None should be returned too + if (ontologies == Set(OntologyConstants.KnoraBase.KnoraBaseOntologyIri.toSmartIri)) + None + // in all other cases, it should be made sure that knora-base is contained in the result + else Some(ontologies + OntologyConstants.KnoraBase.KnoraBaseOntologyIri.toSmartIri) + } + } yield relevantOntologiesMaybe + } /** * Traverses a WHERE clause, delegating transformation tasks to a [[WhereTransformer]], and returns the transformed query patterns. * - * @param patterns the input query patterns. - * @param inputOrderBy the ORDER BY expression in the input query. - * @param whereTransformer a [[WhereTransformer]]. + * @param patterns the input query patterns. + * @param inputOrderBy the ORDER BY expression in the input query. + * @param whereTransformer a [[WhereTransformer]]. + * @param limitInferenceToOntologies a set of ontology IRIs, to which the simulated inference will be limited. If `None`, all possible inference will be done. * @return the transformed query patterns. */ def transformWherePatterns( patterns: Seq[QueryPattern], inputOrderBy: Seq[OrderCriterion], - whereTransformer: WhereTransformer - ): Seq[QueryPattern] = { + whereTransformer: WhereTransformer, + limitInferenceToOntologies: Option[Set[SmartIri]] = None + )(implicit executionContext: ExecutionContext): Seq[QueryPattern] = { // Optimization has to be called before WhereTransformer.transformStatementInWhere, because optimisation might // remove statements that would otherwise be expanded by transformStatementInWhere @@ -206,7 +322,8 @@ object QueryTraverser { case statementPattern: StatementPattern => whereTransformer.transformStatementInWhere( statementPattern = statementPattern, - inputOrderBy = inputOrderBy + inputOrderBy = inputOrderBy, + limitInferenceToOntologies = limitInferenceToOntologies ) case filterPattern: FilterPattern => @@ -218,7 +335,8 @@ object QueryTraverser { val transformedPatterns: Seq[QueryPattern] = transformWherePatterns( patterns = filterNotExistsPattern.patterns, whereTransformer = whereTransformer, - inputOrderBy = inputOrderBy + inputOrderBy = inputOrderBy, + limitInferenceToOntologies = limitInferenceToOntologies ) Seq(FilterNotExistsPattern(patterns = transformedPatterns)) @@ -227,7 +345,8 @@ object QueryTraverser { val transformedPatterns: Seq[QueryPattern] = transformWherePatterns( patterns = minusPattern.patterns, whereTransformer = whereTransformer, - inputOrderBy = inputOrderBy + inputOrderBy = inputOrderBy, + limitInferenceToOntologies = limitInferenceToOntologies ) Seq(MinusPattern(patterns = transformedPatterns)) @@ -236,7 +355,8 @@ object QueryTraverser { val transformedPatterns = transformWherePatterns( patterns = optionalPattern.patterns, whereTransformer = whereTransformer, - inputOrderBy = inputOrderBy + inputOrderBy = inputOrderBy, + limitInferenceToOntologies = limitInferenceToOntologies ) Seq(OptionalPattern(patterns = transformedPatterns)) @@ -247,7 +367,8 @@ object QueryTraverser { val transformedPatterns: Seq[QueryPattern] = transformWherePatterns( patterns = blockPatterns, whereTransformer = whereTransformer, - inputOrderBy = inputOrderBy + inputOrderBy = inputOrderBy, + limitInferenceToOntologies = limitInferenceToOntologies ) whereTransformer.leavingUnionBlock() transformedPatterns @@ -318,16 +439,24 @@ object QueryTraverser { /** * Traverses a SELECT query, delegating transformation tasks to a [[ConstructToSelectTransformer]], and returns the transformed query. * - * @param inputQuery the query to be transformed. - * @param transformer the [[ConstructToSelectTransformer]] to be used. + * @param inputQuery the query to be transformed. + * @param transformer the [[ConstructToSelectTransformer]] to be used. + * @param limitInferenceToOntologies a set of ontology IRIs, to which the simulated inference will be limited. If `None`, all possible inference will be done. * @return the transformed query. */ - def transformConstructToSelect(inputQuery: ConstructQuery, transformer: ConstructToSelectTransformer): SelectQuery = { + def transformConstructToSelect( + inputQuery: ConstructQuery, + transformer: ConstructToSelectTransformer, + limitInferenceToOntologies: Option[Set[SmartIri]] = None + )(implicit + executionContext: ExecutionContext + ): SelectQuery = { val transformedWherePatterns = transformWherePatterns( patterns = inputQuery.whereClause.patterns, inputOrderBy = inputQuery.orderBy, - whereTransformer = transformer + whereTransformer = transformer, + limitInferenceToOntologies = limitInferenceToOntologies ) val transformedOrderBy: TransformedOrderBy = transformer.getOrderBy(inputQuery.orderBy) @@ -348,14 +477,21 @@ object QueryTraverser { ) } - def transformSelectToSelect(inputQuery: SelectQuery, transformer: SelectToSelectTransformer): SelectQuery = + def transformSelectToSelect( + inputQuery: SelectQuery, + transformer: SelectToSelectTransformer, + limitInferenceToOntologies: Option[Set[SmartIri]] + )(implicit + executionContext: ExecutionContext + ): SelectQuery = inputQuery.copy( fromClause = transformer.getFromClause, whereClause = WhereClause( patterns = transformWherePatterns( patterns = inputQuery.whereClause.patterns, inputOrderBy = inputQuery.orderBy, - whereTransformer = transformer + whereTransformer = transformer, + limitInferenceToOntologies = limitInferenceToOntologies ) ) ) @@ -363,19 +499,22 @@ object QueryTraverser { /** * Traverses a CONSTRUCT query, delegating transformation tasks to a [[ConstructToConstructTransformer]], and returns the transformed query. * - * @param inputQuery the query to be transformed. - * @param transformer the [[ConstructToConstructTransformer]] to be used. + * @param inputQuery the query to be transformed. + * @param transformer the [[ConstructToConstructTransformer]] to be used. + * @param limitInferenceToOntologies a set of ontology IRIs, to which the simulated inference will be limited. If `None`, all possible inference will be done. * @return the transformed query. */ def transformConstructToConstruct( inputQuery: ConstructQuery, - transformer: ConstructToConstructTransformer - ): ConstructQuery = { + transformer: ConstructToConstructTransformer, + limitInferenceToOntologies: Option[Set[SmartIri]] = None + )(implicit executionContext: ExecutionContext): ConstructQuery = { val transformedWherePatterns = transformWherePatterns( patterns = inputQuery.whereClause.patterns, inputOrderBy = inputQuery.orderBy, - whereTransformer = transformer + whereTransformer = transformer, + limitInferenceToOntologies = limitInferenceToOntologies ) val transformedConstructStatements: Seq[StatementPattern] = inputQuery.constructClause.statements.flatMap { diff --git a/webapi/src/main/scala/org/knora/webapi/messages/util/search/SparqlTransformer.scala b/webapi/src/main/scala/org/knora/webapi/messages/util/search/SparqlTransformer.scala index 1ce3ccdd29..e941bd66e0 100644 --- a/webapi/src/main/scala/org/knora/webapi/messages/util/search/SparqlTransformer.scala +++ b/webapi/src/main/scala/org/knora/webapi/messages/util/search/SparqlTransformer.scala @@ -8,10 +8,20 @@ package org.knora.webapi.messages.util.search import org.knora.webapi._ import org.knora.webapi.exceptions.AssertionException import org.knora.webapi.exceptions.GravsearchException +import org.knora.webapi.exceptions.InconsistentRepositoryDataException +import org.knora.webapi.exceptions.NotFoundException import org.knora.webapi.messages.IriConversions._ import org.knora.webapi.messages.OntologyConstants import org.knora.webapi.messages.SmartIri import org.knora.webapi.messages.StringFormatter +import org.knora.webapi.responders.v2.ontology.Cache + +import scala.concurrent.Await +import scala.concurrent.ExecutionContext +import scala.concurrent.duration._ +import scala.util.Failure +import scala.util.Success +import scala.util.Try /** * Methods and classes for transforming generated SPARQL. @@ -31,11 +41,13 @@ object SparqlTransformer { override def transformStatementInWhere( statementPattern: StatementPattern, - inputOrderBy: Seq[OrderCriterion] - ): Seq[StatementPattern] = + inputOrderBy: Seq[OrderCriterion], + limitInferenceToOntologies: Option[Set[SmartIri]] = None + )(implicit executionContext: ExecutionContext): Seq[QueryPattern] = transformStatementInWhereForNoInference( statementPattern = statementPattern, - simulateInference = simulateInference + simulateInference = simulateInference, + limitInferenceToOntologies = limitInferenceToOntologies ) override def transformFilter(filterPattern: FilterPattern): Seq[QueryPattern] = Seq(filterPattern) @@ -64,9 +76,14 @@ object SparqlTransformer { override def transformStatementInWhere( statementPattern: StatementPattern, - inputOrderBy: Seq[OrderCriterion] - ): Seq[StatementPattern] = - transformStatementInWhereForNoInference(statementPattern = statementPattern, simulateInference = true) + inputOrderBy: Seq[OrderCriterion], + limitInferenceToOntologies: Option[Set[SmartIri]] = None + )(implicit executionContext: ExecutionContext): Seq[QueryPattern] = + transformStatementInWhereForNoInference( + statementPattern = statementPattern, + simulateInference = true, + limitInferenceToOntologies = limitInferenceToOntologies + ) override def transformFilter(filterPattern: FilterPattern): Seq[QueryPattern] = Seq(filterPattern) @@ -223,15 +240,19 @@ object SparqlTransformer { /** * Transforms a statement in a WHERE clause for a triplestore that does not provide inference. * - * @param statementPattern the statement pattern. - * @param simulateInference `true` if RDFS inference should be simulated using property path syntax. + * @param statementPattern the statement pattern. + * @param simulateInference `true` if RDFS inference should be simulated on basis of the ontology cache. + * @param limitInferenceToOntologies a set of ontology IRIs, to which the simulated inference will be limited. If `None`, all possible inference will be done. * @return the statement pattern as expanded to work without inference. */ def transformStatementInWhereForNoInference( statementPattern: StatementPattern, - simulateInference: Boolean - ): Seq[StatementPattern] = { + simulateInference: Boolean, + limitInferenceToOntologies: Option[Set[SmartIri]] = None + )(implicit executionContext: ExecutionContext): Seq[QueryPattern] = { + implicit val stringFormatter: StringFormatter = StringFormatter.getGeneralInstance + val ontoCache: Cache.OntologyCacheData = Await.result(Cache.getCacheData, 1.second) statementPattern.pred match { case iriRef: IriRef if iriRef.iri.toString == OntologyConstants.KnoraBase.StandoffTagHasStartAncestor => @@ -257,7 +278,8 @@ object SparqlTransformer { statementPattern.pred match { case iriRef: IriRef => // Yes. - val propertyIri = iriRef.iri.toString + val predIri = iriRef.iri + val propertyIri = predIri.toString // Is the property rdf:type? if (propertyIri == OntologyConstants.Rdf.Type) { @@ -269,49 +291,83 @@ object SparqlTransformer { throw GravsearchException(s"The object of rdf:type must be an IRI, but $other was used") } - val rdfTypeVariable: QueryVariable = createUniqueVariableNameForEntityAndBaseClass( - base = statementPattern.subj, - baseClassIri = baseClassIri - ) - - Seq( - StatementPattern( - subj = rdfTypeVariable, - pred = IriRef( - iri = OntologyConstants.Rdfs.SubClassOf.toSmartIri, - propertyPathOperator = Some('*') - ), - obj = statementPattern.obj - ), - StatementPattern( - subj = statementPattern.subj, - pred = statementPattern.pred, - obj = rdfTypeVariable + // look up subclasses from ontology cache + val superClasses = ontoCache.superClassOfRelations + val knownSubClasses = superClasses + .get(baseClassIri.iri) + .getOrElse({ + Set(baseClassIri.iri) + }) + .toSeq + + // if provided, limit the child classes to those that belong to relevant ontologies + val relevantSubClasses = limitInferenceToOntologies match { + case None => knownSubClasses + case Some(relevantOntologyIris) => + // filter the known subclasses against the relevant ontologies + knownSubClasses.filter { subClass => + ontoCache.classDefinedInOntology.get(subClass) match { + case Some(ontologyOfSubclass) => + // return true, if the ontology of the subclass is contained in the set of relevant ontologies; false otherwise + relevantOntologyIris.contains(ontologyOfSubclass) + case None => false // should never happen + } + } + } + + // if subclasses are available, create a union statement that searches for either the provided triple (`?v a `) + // or triples where the object is a subclass of the provided object (`?v a `) + // i.e. `{?v a } UNION {?v a }` + if (relevantSubClasses.length > 1) { + Seq( + UnionPattern( + relevantSubClasses.map(newObject => Seq(statementPattern.copy(obj = IriRef(newObject)))) + ) ) - ) + } else { + // if no subclasses are available, the initial statement can be used. + Seq(statementPattern) + } } else { // No. Expand using rdfs:subPropertyOf*. - val propertyVariable: QueryVariable = createUniqueVariableNameFromEntityAndProperty( - base = statementPattern.pred, - propertyIri = OntologyConstants.Rdfs.SubPropertyOf - ) - - Seq( - StatementPattern( - subj = propertyVariable, - pred = IriRef( - iri = OntologyConstants.Rdfs.SubPropertyOf.toSmartIri, - propertyPathOperator = Some('*') - ), - obj = statementPattern.pred - ), - StatementPattern( - subj = statementPattern.subj, - pred = propertyVariable, - obj = statementPattern.obj + // look up subproperties from ontology cache + val superProps = ontoCache.superPropertyOfRelations + val knownSubProps = superProps + .get(predIri) + .getOrElse({ + Set(predIri) + }) + .toSeq + + // if provided, limit the child properties to those that belong to relevant ontologies + val relevantSubProps = limitInferenceToOntologies match { + case None => knownSubProps + case Some(ontologyIris) => + knownSubProps.filter { subProperty => + // filter the known subproperties against the relevant ontologies + ontoCache.propertyDefinedInOntology.get(subProperty) match { + case Some(childOntologyIri) => + // return true, if the ontology of the subproperty is contained in the set of relevant ontologies; false otherwise + ontologyIris.contains(childOntologyIri) + case None => false // should never happen + } + } + } + + // if subproperties are available, create a union statement that searches for either the provided triple (`?a ?b`) + // or triples where the predicate is a subproperty of the provided object (`?a ?b`) + // i.e. `{?a ?b} UNION {?a ?b}` + if (relevantSubProps.length > 1) { + Seq( + UnionPattern( + relevantSubProps.map(newPredicate => Seq(statementPattern.copy(pred = IriRef(newPredicate)))) + ) ) - ) + } else { + // if no subproperties are available, the initial statement can be used + Seq(statementPattern) + } } case _ => diff --git a/webapi/src/main/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/AbstractPrequeryGenerator.scala b/webapi/src/main/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/AbstractPrequeryGenerator.scala index 9a030241d6..973d6f787f 100644 --- a/webapi/src/main/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/AbstractPrequeryGenerator.scala +++ b/webapi/src/main/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/AbstractPrequeryGenerator.scala @@ -574,7 +574,8 @@ abstract class AbstractPrequeryGenerator( protected def processStatementPatternFromWhereClause( statementPattern: StatementPattern, - inputOrderBy: Seq[OrderCriterion] + inputOrderBy: Seq[OrderCriterion], + limitInferenceToOntologies: Option[Set[SmartIri]] = None ): Seq[QueryPattern] = // Does this statement set a Gravsearch option? statementPattern.subj match { diff --git a/webapi/src/main/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/NonTriplestoreSpecificGravsearchToCountPrequeryTransformer.scala b/webapi/src/main/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/NonTriplestoreSpecificGravsearchToCountPrequeryTransformer.scala index 0c044e0dd5..838ae67dc3 100644 --- a/webapi/src/main/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/NonTriplestoreSpecificGravsearchToCountPrequeryTransformer.scala +++ b/webapi/src/main/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/NonTriplestoreSpecificGravsearchToCountPrequeryTransformer.scala @@ -7,9 +7,12 @@ package org.knora.webapi.messages.util.search.gravsearch.prequery import org.knora.webapi.ApiV2Schema import org.knora.webapi.feature.FeatureFactoryConfig +import org.knora.webapi.messages.SmartIri import org.knora.webapi.messages.util.search._ import org.knora.webapi.messages.util.search.gravsearch.types.GravsearchTypeInspectionResult +import scala.concurrent.ExecutionContext + /** * Transforms a preprocessed CONSTRUCT query into a SELECT query that returns only the IRIs and sort order of the main resources that matched * the search criteria. This query will be used to get resource IRIs for a single page of results. These IRIs will be included in a CONSTRUCT @@ -34,8 +37,9 @@ class NonTriplestoreSpecificGravsearchToCountPrequeryTransformer( def transformStatementInWhere( statementPattern: StatementPattern, - inputOrderBy: Seq[OrderCriterion] - ): Seq[QueryPattern] = + inputOrderBy: Seq[OrderCriterion], + limitInferenceToOntologies: Option[Set[SmartIri]] = None + )(implicit executionContext: ExecutionContext): Seq[QueryPattern] = // Include any statements needed to meet the user's search criteria, but not statements that would be needed for permission checking or // other information about the matching resources or values. processStatementPatternFromWhereClause( diff --git a/webapi/src/main/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/NonTriplestoreSpecificGravsearchToPrequeryTransformer.scala b/webapi/src/main/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/NonTriplestoreSpecificGravsearchToPrequeryTransformer.scala index e423b7e397..4e31c8c4ca 100644 --- a/webapi/src/main/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/NonTriplestoreSpecificGravsearchToPrequeryTransformer.scala +++ b/webapi/src/main/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/NonTriplestoreSpecificGravsearchToPrequeryTransformer.scala @@ -9,6 +9,7 @@ import org.knora.webapi._ import org.knora.webapi.exceptions.AssertionException import org.knora.webapi.exceptions.GravsearchException import org.knora.webapi.feature.FeatureFactoryConfig +import org.knora.webapi.messages.SmartIri import org.knora.webapi.messages.util.search._ import org.knora.webapi.messages.util.search.gravsearch.types.GravsearchTypeInspectionResult import org.knora.webapi.messages.util.search.gravsearch.types.GravsearchTypeInspectionUtil @@ -16,6 +17,8 @@ import org.knora.webapi.messages.util.search.gravsearch.types.NonPropertyTypeInf import org.knora.webapi.messages.util.search.gravsearch.types.PropertyTypeInfo import org.knora.webapi.settings.KnoraSettingsImpl +import scala.concurrent.ExecutionContext + /** * Transforms a preprocessed CONSTRUCT query into a SELECT query that returns only the IRIs and sort order of the main resources that matched * the search criteria and are requested by client in the input query's WHERE clause. This query will be used to get resource IRIs for a single @@ -45,19 +48,22 @@ class NonTriplestoreSpecificGravsearchToPrequeryTransformer( /** * Transforms a [[org.knora.webapi.messages.util.search.StatementPattern]] in a WHERE clause into zero or more query patterns. * - * @param statementPattern the statement to be transformed. - * @param inputOrderBy the ORDER BY clause in the input query. + * @param statementPattern the statement to be transformed. + * @param inputOrderBy the ORDER BY clause in the input query. + * @param limitInferenceToOntologies a set of ontology IRIs, to which the simulated inference will be limited. If `None`, all possible inference will be done. * @return the result of the transformation. */ override def transformStatementInWhere( statementPattern: StatementPattern, - inputOrderBy: Seq[OrderCriterion] - ): Seq[QueryPattern] = + inputOrderBy: Seq[OrderCriterion], + limitInferenceToOntologies: Option[Set[SmartIri]] = None + )(implicit executionContext: ExecutionContext): Seq[QueryPattern] = // Include any statements needed to meet the user's search criteria, but not statements that would be needed for permission checking or // other information about the matching resources or values. processStatementPatternFromWhereClause( statementPattern = statementPattern, - inputOrderBy = inputOrderBy + inputOrderBy = inputOrderBy, + limitInferenceToOntologies = limitInferenceToOntologies ) /** diff --git a/webapi/src/main/scala/org/knora/webapi/messages/util/search/gravsearch/types/GravsearchTypeInspectionUtil.scala b/webapi/src/main/scala/org/knora/webapi/messages/util/search/gravsearch/types/GravsearchTypeInspectionUtil.scala index 3e99e75594..c4664de2fe 100644 --- a/webapi/src/main/scala/org/knora/webapi/messages/util/search/gravsearch/types/GravsearchTypeInspectionUtil.scala +++ b/webapi/src/main/scala/org/knora/webapi/messages/util/search/gravsearch/types/GravsearchTypeInspectionUtil.scala @@ -12,6 +12,8 @@ import org.knora.webapi.messages.OntologyConstants import org.knora.webapi.messages.SmartIri import org.knora.webapi.messages.util.search._ +import scala.concurrent.ExecutionContext + /** * Utilities for Gravsearch type inspection. */ @@ -157,8 +159,9 @@ object GravsearchTypeInspectionUtil { private class AnnotationRemovingWhereTransformer extends WhereTransformer { override def transformStatementInWhere( statementPattern: StatementPattern, - inputOrderBy: Seq[OrderCriterion] - ): Seq[QueryPattern] = + inputOrderBy: Seq[OrderCriterion], + limitInferenceToOntologies: Option[Set[SmartIri]] = None + )(implicit executionContext: ExecutionContext): Seq[QueryPattern] = if (mustBeAnnotationStatement(statementPattern)) { Seq.empty[QueryPattern] } else { @@ -183,7 +186,7 @@ object GravsearchTypeInspectionUtil { * @param whereClause the WHERE clause. * @return the same WHERE clause, minus any type annotations. */ - def removeTypeAnnotations(whereClause: WhereClause): WhereClause = + def removeTypeAnnotations(whereClause: WhereClause)(implicit executionContext: ExecutionContext): WhereClause = whereClause.copy( patterns = QueryTraverser.transformWherePatterns( patterns = whereClause.patterns, diff --git a/webapi/src/main/scala/org/knora/webapi/responders/v2/SearchResponderV2.scala b/webapi/src/main/scala/org/knora/webapi/responders/v2/SearchResponderV2.scala index 48fcb95b0e..c3cb8617b9 100644 --- a/webapi/src/main/scala/org/knora/webapi/responders/v2/SearchResponderV2.scala +++ b/webapi/src/main/scala/org/knora/webapi/responders/v2/SearchResponderV2.scala @@ -12,6 +12,7 @@ import org.knora.webapi.exceptions.AssertionException import org.knora.webapi.exceptions.BadRequestException import org.knora.webapi.exceptions.GravsearchException import org.knora.webapi.exceptions.InconsistentRepositoryDataException +import org.knora.webapi.exceptions.TriplestoreTimeoutException import org.knora.webapi.feature.FeatureFactoryConfig import org.knora.webapi.messages.IriConversions._ import org.knora.webapi.messages.OntologyConstants @@ -44,6 +45,9 @@ import org.knora.webapi.responders.Responder.handleUnexpectedMessage import org.knora.webapi.util.ApacheLuceneSupport._ import scala.concurrent.Future +import scala.util.Failure +import scala.util.Success +import scala.util.Try class SearchResponderV2(responderData: ResponderData) extends ResponderWithStandoffV2(responderData) { @@ -177,23 +181,22 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand val searchTerms: LuceneQueryString = LuceneQueryString(searchValue) for { - countSparql <- Future( - org.knora.webapi.messages.twirl.queries.sparql.v2.txt - .searchFulltext( - searchTerms = searchTerms, - limitToProject = limitToProject, - limitToResourceClass = limitToResourceClass.map(_.toString), - limitToStandoffClass = limitToStandoffClass.map(_.toString), - returnFiles = false, // not relevant for a count query - separator = None, // no separator needed for count query - limit = 1, - offset = 0, - countQuery = true // do not get the resources themselves, but the sum of results - ) - .toString() - ) - - // _ = println(countSparql) + countSparql <- + Future( + org.knora.webapi.messages.twirl.queries.sparql.v2.txt + .searchFulltext( + searchTerms = searchTerms, + limitToProject = limitToProject, + limitToResourceClass = limitToResourceClass.map(_.toString), + limitToStandoffClass = limitToStandoffClass.map(_.toString), + returnFiles = false, // not relevant for a count query + separator = None, // no separator needed for count query + limit = 1, + offset = 0, + countQuery = true // do not get the resources themselves, but the sum of results + ) + .toString() + ) countResponse: SparqlSelectResult <- (storeManager ? SparqlSelectRequest(countSparql)).mapTo[SparqlSelectResult] @@ -242,27 +245,25 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand val searchTerms: LuceneQueryString = LuceneQueryString(searchValue) for { - searchSparql <- Future( - org.knora.webapi.messages.twirl.queries.sparql.v2.txt - .searchFulltext( - searchTerms = searchTerms, - limitToProject = limitToProject, - limitToResourceClass = limitToResourceClass.map(_.toString), - limitToStandoffClass = limitToStandoffClass.map(_.toString), - returnFiles = returnFiles, - separator = Some(groupConcatSeparator), - limit = settings.v2ResultsPerPage, - offset = offset * settings.v2ResultsPerPage, // determine the actual offset - countQuery = false - ) - .toString() - ) - - // _ = println(searchSparql) + searchSparql <- + Future( + org.knora.webapi.messages.twirl.queries.sparql.v2.txt + .searchFulltext( + searchTerms = searchTerms, + limitToProject = limitToProject, + limitToResourceClass = limitToResourceClass.map(_.toString), + limitToStandoffClass = limitToStandoffClass.map(_.toString), + returnFiles = returnFiles, + separator = Some(groupConcatSeparator), + limit = settings.v2ResultsPerPage, + offset = offset * settings.v2ResultsPerPage, // determine the actual offset + countQuery = false + ) + .toString() + ) prequeryResponseNotMerged: SparqlSelectResult <- (storeManager ? SparqlSelectRequest(searchSparql)) .mapTo[SparqlSelectResult] - // _ = println(prequeryResponseNotMerged) mainResourceVar = QueryVariable("resource") @@ -295,8 +296,6 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand } } - // println(valueObjectIrisPerResource) - // collect all value object IRIs val allValueObjectIris = valueObjectIrisPerResource.values.flatten.toSet @@ -317,13 +316,12 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand transformer = queryPatternTransformerConstruct ) - // println(triplestoreSpecificQuery.toSparql) - for { - searchResponse: SparqlExtendedConstructResponse <- (storeManager ? SparqlExtendedConstructRequest( - sparql = triplestoreSpecificQuery.toSparql, - featureFactoryConfig = featureFactoryConfig - )).mapTo[SparqlExtendedConstructResponse] + searchResponse: SparqlExtendedConstructResponse <- + (storeManager ? SparqlExtendedConstructRequest( + sparql = triplestoreSpecificQuery.toSparql, + featureFactoryConfig = featureFactoryConfig + )).mapTo[SparqlExtendedConstructResponse] // separate resources and value objects queryResultsSep: ConstructResponseUtilV2.MainResourcesAndValueRdfData = @@ -360,22 +358,21 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand FastFuture.successful(Map.empty[IRI, MappingAndXSLTransformation]) } - // _ = println(mappingsAsMap) - - apiResponse: ReadResourcesSequenceV2 <- ConstructResponseUtilV2.createApiResponse( - mainResourcesAndValueRdfData = mainResourcesAndValueRdfData, - orderByResourceIri = resourceIris, - pageSizeBeforeFiltering = resourceIris.size, - mappings = mappingsAsMap, - queryStandoff = queryStandoff, - calculateMayHaveMoreResults = true, - versionDate = None, - responderManager = responderManager, - settings = settings, - targetSchema = targetSchema, - featureFactoryConfig = featureFactoryConfig, - requestingUser = requestingUser - ) + apiResponse: ReadResourcesSequenceV2 <- + ConstructResponseUtilV2.createApiResponse( + mainResourcesAndValueRdfData = mainResourcesAndValueRdfData, + orderByResourceIri = resourceIris, + pageSizeBeforeFiltering = resourceIris.size, + mappings = mappingsAsMap, + queryStandoff = queryStandoff, + calculateMayHaveMoreResults = true, + versionDate = None, + responderManager = responderManager, + settings = settings, + targetSchema = targetSchema, + featureFactoryConfig = featureFactoryConfig, + requestingUser = requestingUser + ) } yield apiResponse } @@ -401,14 +398,16 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand for { // Do type inspection and remove type annotations from the WHERE clause. - typeInspectionResult: GravsearchTypeInspectionResult <- gravsearchTypeInspectionRunner.inspectTypes( - inputQuery.whereClause, - requestingUser - ) + typeInspectionResult: GravsearchTypeInspectionResult <- + gravsearchTypeInspectionRunner.inspectTypes( + inputQuery.whereClause, + requestingUser + ) - whereClauseWithoutAnnotations: WhereClause = GravsearchTypeInspectionUtil.removeTypeAnnotations( - inputQuery.whereClause - ) + whereClauseWithoutAnnotations: WhereClause = + GravsearchTypeInspectionUtil.removeTypeAnnotations( + inputQuery.whereClause + ) // Validate schemas and predicates in the CONSTRUCT clause. _ = GravsearchQueryChecker.checkConstructClause( @@ -426,15 +425,16 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand featureFactoryConfig = featureFactoryConfig ) - nonTriplestoreSpecificPrequery: SelectQuery = QueryTraverser.transformConstructToSelect( - inputQuery = inputQuery.copy( - whereClause = whereClauseWithoutAnnotations, - orderBy = Seq.empty[ - OrderCriterion - ] // count queries do not need any sorting criteria - ), - transformer = nonTriplestoreSpecificConstructToSelectTransformer - ) + nonTriplestoreSpecificPrequery: SelectQuery = + QueryTraverser.transformConstructToSelect( + inputQuery = inputQuery.copy( + whereClause = whereClauseWithoutAnnotations, + orderBy = Seq.empty[ + OrderCriterion + ] // count queries do not need any sorting criteria + ), + transformer = nonTriplestoreSpecificConstructToSelectTransformer + ) // Convert the non-triplestore-specific query to a triplestore-specific one. @@ -443,15 +443,22 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand simulateInference = nonTriplestoreSpecificConstructToSelectTransformer.useInference ) - triplestoreSpecificCountQuery = QueryTraverser.transformSelectToSelect( - inputQuery = nonTriplestoreSpecificPrequery, - transformer = triplestoreSpecificQueryPatternTransformerSelect - ) + ontologiesForInferenceMaybe <- + QueryTraverser.getOntologiesRelevantForInference( + inputQuery.whereClause, + storeManager + ) - // _ = println(triplestoreSpecificCountQuery.toSparql) + triplestoreSpecificCountQuery = + QueryTraverser.transformSelectToSelect( + inputQuery = nonTriplestoreSpecificPrequery, + transformer = triplestoreSpecificQueryPatternTransformerSelect, + ontologiesForInferenceMaybe + ) - countResponse: SparqlSelectResult <- (storeManager ? SparqlSelectRequest(triplestoreSpecificCountQuery.toSparql)) - .mapTo[SparqlSelectResult] + countResponse: SparqlSelectResult <- + (storeManager ? SparqlSelectRequest(triplestoreSpecificCountQuery.toSparql)) + .mapTo[SparqlSelectResult] // query response should contain one result with one row with the name "count" _ = if (countResponse.results.bindings.length != 1) { @@ -487,13 +494,15 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand for { // Do type inspection and remove type annotations from the WHERE clause. - typeInspectionResult: GravsearchTypeInspectionResult <- gravsearchTypeInspectionRunner.inspectTypes( - inputQuery.whereClause, - requestingUser - ) - whereClauseWithoutAnnotations: WhereClause = GravsearchTypeInspectionUtil.removeTypeAnnotations( - inputQuery.whereClause - ) + typeInspectionResult: GravsearchTypeInspectionResult <- + gravsearchTypeInspectionRunner.inspectTypes( + inputQuery.whereClause, + requestingUser + ) + whereClauseWithoutAnnotations: WhereClause = + GravsearchTypeInspectionUtil.removeTypeAnnotations( + inputQuery.whereClause + ) // Validate schemas and predicates in the CONSTRUCT clause. _ = GravsearchQueryChecker.checkConstructClause( @@ -515,11 +524,16 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand // TODO: if the ORDER BY criterion is a property whose occurrence is not 1, then the logic does not work correctly // TODO: the ORDER BY criterion has to be included in a GROUP BY statement, returning more than one row if property occurs more than once - nonTriplestoreSpecificPrequery: SelectQuery = QueryTraverser.transformConstructToSelect( - inputQuery = - inputQuery.copy(whereClause = whereClauseWithoutAnnotations), - transformer = nonTriplestoreSpecificConstructToSelectTransformer - ) + ontologiesForInferenceMaybe <- + QueryTraverser.getOntologiesRelevantForInference( + inputQuery.whereClause, + storeManager + ) + nonTriplestoreSpecificPrequery: SelectQuery = + QueryTraverser.transformConstructToSelect( + inputQuery = inputQuery.copy(whereClause = whereClauseWithoutAnnotations), + transformer = nonTriplestoreSpecificConstructToSelectTransformer + ) // variable representing the main resources mainResourceVar: QueryVariable = nonTriplestoreSpecificConstructToSelectTransformer.mainResourceVariable @@ -532,30 +546,51 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand // Convert the preprocessed query to a non-triplestore-specific query. - triplestoreSpecificPrequery = QueryTraverser.transformSelectToSelect( - inputQuery = nonTriplestoreSpecificPrequery, - transformer = triplestoreSpecificQueryPatternTransformerSelect - ) + triplestoreSpecificPrequery = + QueryTraverser.transformSelectToSelect( + inputQuery = nonTriplestoreSpecificPrequery, + transformer = triplestoreSpecificQueryPatternTransformerSelect, + limitInferenceToOntologies = ontologiesForInferenceMaybe + ) triplestoreSpecificPrequerySparql = triplestoreSpecificPrequery.toSparql _ = log.debug(triplestoreSpecificPrequerySparql) - prequeryResponseNotMerged: SparqlSelectResult <- (storeManager ? SparqlSelectRequest( - triplestoreSpecificPrequerySparql - )).mapTo[SparqlSelectResult] + start = System.currentTimeMillis() + tryPrequeryResponseNotMerged = Try(storeManager ? SparqlSelectRequest(triplestoreSpecificPrequerySparql)) + prequeryResponseNotMerged <- + (tryPrequeryResponseNotMerged match { + case Failure(exception) => { + exception match { + case timeoutException: TriplestoreTimeoutException => + log.error(s"Gravsearch timed out for query: $inputQuery") + } + throw exception + } + case Success(value) => value + }).mapTo[SparqlSelectResult] + duration = (System.currentTimeMillis() - start) / 1000.0 + _ = + if (duration < 3) { + log.debug(s"Prequery took: ${duration}s") + } else { + log.warn(s"Slow Prequery ($duration):\n$triplestoreSpecificPrequerySparql\nInitial Query:\n$inputQuery") + } pageSizeBeforeFiltering: Int = prequeryResponseNotMerged.results.bindings.size // Merge rows with the same main resource IRI. This could happen if there are unbound variables in a UNION. - prequeryResponse = mergePrequeryResults( - prequeryResponseNotMerged = prequeryResponseNotMerged, - mainResourceVar = mainResourceVar - ) + prequeryResponse = + mergePrequeryResults( + prequeryResponseNotMerged = prequeryResponseNotMerged, + mainResourceVar = mainResourceVar + ) // a sequence of resource IRIs that match the search criteria // attention: no permission checking has been done so far - mainResourceIris: Seq[IRI] = prequeryResponse.results.bindings.map { resultRow: VariableResultsRow => - resultRow.rowMap(mainResourceVar.variableName) - } + mainResourceIris: Seq[IRI] = + prequeryResponse.results.bindings.map { resultRow: VariableResultsRow => + resultRow.rowMap(mainResourceVar.variableName) + } mainQueryResults: ConstructResponseUtilV2.MainResourcesAndValueRdfData <- if (mainResourceIris.nonEmpty) { @@ -620,7 +655,8 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand val triplestoreSpecificMainQuery = QueryTraverser.transformConstructToConstruct( inputQuery = mainQuery, - transformer = queryPatternTransformerConstruct + transformer = queryPatternTransformerConstruct, + limitInferenceToOntologies = ontologiesForInferenceMaybe ) // Convert the result to a SPARQL string and send it to the triplestore. @@ -628,10 +664,11 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand log.debug(triplestoreSpecificMainQuerySparql) for { - mainQueryResponse: SparqlExtendedConstructResponse <- (storeManager ? SparqlExtendedConstructRequest( - sparql = triplestoreSpecificMainQuerySparql, - featureFactoryConfig = featureFactoryConfig - )).mapTo[SparqlExtendedConstructResponse] + mainQueryResponse: SparqlExtendedConstructResponse <- + (storeManager ? SparqlExtendedConstructRequest( + sparql = triplestoreSpecificMainQuerySparql, + featureFactoryConfig = featureFactoryConfig + )).mapTo[SparqlExtendedConstructResponse] // Filter out values that the user doesn't have permission to see. queryResultsFilteredForPermissions: ConstructResponseUtilV2.MainResourcesAndValueRdfData = @@ -683,20 +720,21 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand FastFuture.successful(Map.empty[IRI, MappingAndXSLTransformation]) } - apiResponse: ReadResourcesSequenceV2 <- ConstructResponseUtilV2.createApiResponse( - mainResourcesAndValueRdfData = mainQueryResults, - orderByResourceIri = mainResourceIris, - pageSizeBeforeFiltering = pageSizeBeforeFiltering, - mappings = mappingsAsMap, - queryStandoff = queryStandoff, - versionDate = None, - calculateMayHaveMoreResults = true, - responderManager = responderManager, - settings = settings, - targetSchema = targetSchema, - featureFactoryConfig = featureFactoryConfig, - requestingUser = requestingUser - ) + apiResponse: ReadResourcesSequenceV2 <- + ConstructResponseUtilV2.createApiResponse( + mainResourcesAndValueRdfData = mainQueryResults, + orderByResourceIri = mainResourceIris, + pageSizeBeforeFiltering = pageSizeBeforeFiltering, + mappings = mappingsAsMap, + queryStandoff = queryStandoff, + versionDate = None, + calculateMayHaveMoreResults = true, + responderManager = responderManager, + settings = settings, + targetSchema = targetSchema, + featureFactoryConfig = featureFactoryConfig, + requestingUser = requestingUser + ) } yield apiResponse } @@ -728,68 +766,69 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand // If an ORDER BY property was specified, determine which subproperty of knora-base:valueHas to use to get the // literal value to sort by. - maybeOrderByValuePredicate: Option[SmartIri] = maybeInternalOrderByPropertyIri match { - case Some(internalOrderByPropertyIri) => - val internalOrderByPropertyDef: ReadPropertyInfoV2 = - entityInfoResponse.propertyInfoMap( - internalOrderByPropertyIri - ) - - // Ensure that the ORDER BY property is one that we can sort by. - if ( - !internalOrderByPropertyDef.isResourceProp || internalOrderByPropertyDef.isLinkProp || internalOrderByPropertyDef.isLinkValueProp || internalOrderByPropertyDef.isFileValueProp - ) { - throw BadRequestException( - s"Cannot sort by property <${resourcesInProjectGetRequestV2.orderByProperty}>" - ) - } - - // Ensure that the resource class has a cardinality on the ORDER BY property. - if ( - !classDef.knoraResourceProperties.contains( - internalOrderByPropertyIri - ) - ) { - throw BadRequestException( - s"Class <${resourcesInProjectGetRequestV2.resourceClass}> has no cardinality on property <${resourcesInProjectGetRequestV2.orderByProperty}>" - ) - } - - // Get the value class that's the object of the knora-base:objectClassConstraint of the ORDER BY property. - val orderByValueType: SmartIri = - internalOrderByPropertyDef.entityInfoContent - .requireIriObject( - OntologyConstants.KnoraBase.ObjectClassConstraint.toSmartIri, - throw InconsistentRepositoryDataException( - s"Property <$internalOrderByPropertyIri> has no knora-base:objectClassConstraint" - ) - ) - - // Determine which subproperty of knora-base:valueHas corresponds to that value class. - val orderByValuePredicate = orderByValueType.toString match { - case OntologyConstants.KnoraBase.IntValue => - OntologyConstants.KnoraBase.ValueHasInteger - case OntologyConstants.KnoraBase.DecimalValue => - OntologyConstants.KnoraBase.ValueHasDecimal - case OntologyConstants.KnoraBase.BooleanValue => - OntologyConstants.KnoraBase.ValueHasBoolean - case OntologyConstants.KnoraBase.DateValue => - OntologyConstants.KnoraBase.ValueHasStartJDN - case OntologyConstants.KnoraBase.ColorValue => - OntologyConstants.KnoraBase.ValueHasColor - case OntologyConstants.KnoraBase.GeonameValue => - OntologyConstants.KnoraBase.ValueHasGeonameCode - case OntologyConstants.KnoraBase.IntervalValue => - OntologyConstants.KnoraBase.ValueHasIntervalStart - case OntologyConstants.KnoraBase.UriValue => - OntologyConstants.KnoraBase.ValueHasUri - case _ => OntologyConstants.KnoraBase.ValueHasString - } - - Some(orderByValuePredicate.toSmartIri) - - case None => None - } + maybeOrderByValuePredicate: Option[SmartIri] = + maybeInternalOrderByPropertyIri match { + case Some(internalOrderByPropertyIri) => + val internalOrderByPropertyDef: ReadPropertyInfoV2 = + entityInfoResponse.propertyInfoMap( + internalOrderByPropertyIri + ) + + // Ensure that the ORDER BY property is one that we can sort by. + if ( + !internalOrderByPropertyDef.isResourceProp || internalOrderByPropertyDef.isLinkProp || internalOrderByPropertyDef.isLinkValueProp || internalOrderByPropertyDef.isFileValueProp + ) { + throw BadRequestException( + s"Cannot sort by property <${resourcesInProjectGetRequestV2.orderByProperty}>" + ) + } + + // Ensure that the resource class has a cardinality on the ORDER BY property. + if ( + !classDef.knoraResourceProperties.contains( + internalOrderByPropertyIri + ) + ) { + throw BadRequestException( + s"Class <${resourcesInProjectGetRequestV2.resourceClass}> has no cardinality on property <${resourcesInProjectGetRequestV2.orderByProperty}>" + ) + } + + // Get the value class that's the object of the knora-base:objectClassConstraint of the ORDER BY property. + val orderByValueType: SmartIri = + internalOrderByPropertyDef.entityInfoContent + .requireIriObject( + OntologyConstants.KnoraBase.ObjectClassConstraint.toSmartIri, + throw InconsistentRepositoryDataException( + s"Property <$internalOrderByPropertyIri> has no knora-base:objectClassConstraint" + ) + ) + + // Determine which subproperty of knora-base:valueHas corresponds to that value class. + val orderByValuePredicate = orderByValueType.toString match { + case OntologyConstants.KnoraBase.IntValue => + OntologyConstants.KnoraBase.ValueHasInteger + case OntologyConstants.KnoraBase.DecimalValue => + OntologyConstants.KnoraBase.ValueHasDecimal + case OntologyConstants.KnoraBase.BooleanValue => + OntologyConstants.KnoraBase.ValueHasBoolean + case OntologyConstants.KnoraBase.DateValue => + OntologyConstants.KnoraBase.ValueHasStartJDN + case OntologyConstants.KnoraBase.ColorValue => + OntologyConstants.KnoraBase.ValueHasColor + case OntologyConstants.KnoraBase.GeonameValue => + OntologyConstants.KnoraBase.ValueHasGeonameCode + case OntologyConstants.KnoraBase.IntervalValue => + OntologyConstants.KnoraBase.ValueHasIntervalStart + case OntologyConstants.KnoraBase.UriValue => + OntologyConstants.KnoraBase.ValueHasUri + case _ => OntologyConstants.KnoraBase.ValueHasString + } + + Some(orderByValuePredicate.toSmartIri) + + case None => None + } // Do a SELECT prequery to get the IRIs of the requested page of resources. prequery = org.knora.webapi.messages.twirl.queries.sparql.v2.txt @@ -828,23 +867,22 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand for { // Yes. Do a CONSTRUCT query to get the contents of those resources. If we're querying standoff, get // at most one page of standoff per text value. - resourceRequestSparql <- Future( - org.knora.webapi.messages.twirl.queries.sparql.v2.txt - .getResourcePropertiesAndValues( - resourceIris = mainResourceIris, - preview = false, - withDeleted = false, - queryAllNonStandoff = true, - maybePropertyIri = None, - maybeVersionDate = None, - maybeStandoffMinStartIndex = maybeStandoffMinStartIndex, - maybeStandoffMaxStartIndex = maybeStandoffMaxStartIndex, - stringFormatter = stringFormatter - ) - .toString() - ) - - // _ = println(resourceRequestSparql) + resourceRequestSparql <- + Future( + org.knora.webapi.messages.twirl.queries.sparql.v2.txt + .getResourcePropertiesAndValues( + resourceIris = mainResourceIris, + preview = false, + withDeleted = false, + queryAllNonStandoff = true, + maybePropertyIri = None, + maybeVersionDate = None, + maybeStandoffMinStartIndex = maybeStandoffMinStartIndex, + maybeStandoffMaxStartIndex = maybeStandoffMaxStartIndex, + stringFormatter = stringFormatter + ) + .toString() + ) resourceRequestResponse: SparqlExtendedConstructResponse <- (storeManager ? SparqlExtendedConstructRequest( @@ -873,24 +911,21 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand } // Construct a ReadResourceV2 for each resource that the user has permission to see. - readResourcesSequence: ReadResourcesSequenceV2 <- ConstructResponseUtilV2.createApiResponse( - mainResourcesAndValueRdfData = - mainResourcesAndValueRdfData, - orderByResourceIri = mainResourceIris, - pageSizeBeforeFiltering = mainResourceIris.size, - mappings = mappings, - queryStandoff = maybeStandoffMinStartIndex.nonEmpty, - versionDate = None, - calculateMayHaveMoreResults = true, - responderManager = responderManager, - targetSchema = - resourcesInProjectGetRequestV2.targetSchema, - settings = settings, - featureFactoryConfig = - resourcesInProjectGetRequestV2.featureFactoryConfig, - requestingUser = - resourcesInProjectGetRequestV2.requestingUser - ) + readResourcesSequence: ReadResourcesSequenceV2 <- + ConstructResponseUtilV2.createApiResponse( + mainResourcesAndValueRdfData = mainResourcesAndValueRdfData, + orderByResourceIri = mainResourceIris, + pageSizeBeforeFiltering = mainResourceIris.size, + mappings = mappings, + queryStandoff = maybeStandoffMinStartIndex.nonEmpty, + versionDate = None, + calculateMayHaveMoreResults = true, + responderManager = responderManager, + targetSchema = resourcesInProjectGetRequestV2.targetSchema, + settings = settings, + featureFactoryConfig = resourcesInProjectGetRequestV2.featureFactoryConfig, + requestingUser = resourcesInProjectGetRequestV2.requestingUser + ) } yield readResourcesSequence } else { FastFuture.successful(ReadResourcesSequenceV2(Vector.empty[ReadResourceV2])) @@ -918,20 +953,19 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand val searchPhrase: MatchStringWhileTyping = MatchStringWhileTyping(searchValue) for { - countSparql <- Future( - org.knora.webapi.messages.twirl.queries.sparql.v2.txt - .searchResourceByLabel( - searchTerm = searchPhrase, - limitToProject = limitToProject, - limitToResourceClass = limitToResourceClass.map(_.toString), - limit = 1, - offset = 0, - countQuery = true - ) - .toString() - ) - - // _ = println(countSparql) + countSparql <- + Future( + org.knora.webapi.messages.twirl.queries.sparql.v2.txt + .searchResourceByLabel( + searchTerm = searchPhrase, + limitToProject = limitToProject, + limitToResourceClass = limitToResourceClass.map(_.toString), + limit = 1, + offset = 0, + countQuery = true + ) + .toString() + ) countResponse: SparqlSelectResult <- (storeManager ? SparqlSelectRequest(countSparql)).mapTo[SparqlSelectResult] @@ -975,25 +1009,25 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand val searchPhrase: MatchStringWhileTyping = MatchStringWhileTyping(searchValue) for { - searchResourceByLabelSparql <- Future( - org.knora.webapi.messages.twirl.queries.sparql.v2.txt - .searchResourceByLabel( - searchTerm = searchPhrase, - limitToProject = limitToProject, - limitToResourceClass = limitToResourceClass.map(_.toString), - limit = settings.v2ResultsPerPage, - offset = offset * settings.v2ResultsPerPage, - countQuery = false - ) - .toString() - ) - - // _ = println(searchResourceByLabelSparql) - - searchResourceByLabelResponse: SparqlExtendedConstructResponse <- (storeManager ? SparqlExtendedConstructRequest( - sparql = searchResourceByLabelSparql, - featureFactoryConfig = featureFactoryConfig - )).mapTo[SparqlExtendedConstructResponse] + searchResourceByLabelSparql <- + Future( + org.knora.webapi.messages.twirl.queries.sparql.v2.txt + .searchResourceByLabel( + searchTerm = searchPhrase, + limitToProject = limitToProject, + limitToResourceClass = limitToResourceClass.map(_.toString), + limit = settings.v2ResultsPerPage, + offset = offset * settings.v2ResultsPerPage, + countQuery = false + ) + .toString() + ) + + searchResourceByLabelResponse: SparqlExtendedConstructResponse <- + (storeManager ? SparqlExtendedConstructRequest( + sparql = searchResourceByLabelSparql, + featureFactoryConfig = featureFactoryConfig + )).mapTo[SparqlExtendedConstructResponse] // collect the IRIs of main resources returned mainResourceIris: Set[IRI] = @@ -1018,29 +1052,27 @@ class SearchResponderV2(responderData: ResponderData) extends ResponderWithStand } } - // _ = println(mainResourceIris.size) - // separate resources and value objects - mainResourcesAndValueRdfData = ConstructResponseUtilV2.splitMainResourcesAndValueRdfData( - constructQueryResults = searchResourceByLabelResponse, - requestingUser = requestingUser - ) - - //_ = println(queryResultsSeparated) - - apiResponse: ReadResourcesSequenceV2 <- ConstructResponseUtilV2.createApiResponse( - mainResourcesAndValueRdfData = mainResourcesAndValueRdfData, - orderByResourceIri = mainResourceIris.toSeq.sorted, - pageSizeBeforeFiltering = mainResourceIris.size, - queryStandoff = false, - versionDate = None, - calculateMayHaveMoreResults = true, - responderManager = responderManager, - targetSchema = targetSchema, - settings = settings, - featureFactoryConfig = featureFactoryConfig, - requestingUser = requestingUser - ) + mainResourcesAndValueRdfData = + ConstructResponseUtilV2.splitMainResourcesAndValueRdfData( + constructQueryResults = searchResourceByLabelResponse, + requestingUser = requestingUser + ) + + apiResponse: ReadResourcesSequenceV2 <- + ConstructResponseUtilV2.createApiResponse( + mainResourcesAndValueRdfData = mainResourcesAndValueRdfData, + orderByResourceIri = mainResourceIris.toSeq.sorted, + pageSizeBeforeFiltering = mainResourceIris.size, + queryStandoff = false, + versionDate = None, + calculateMayHaveMoreResults = true, + responderManager = responderManager, + targetSchema = targetSchema, + settings = settings, + featureFactoryConfig = featureFactoryConfig, + requestingUser = requestingUser + ) } yield apiResponse } diff --git a/webapi/src/main/scala/org/knora/webapi/responders/v2/ontology/Cache.scala b/webapi/src/main/scala/org/knora/webapi/responders/v2/ontology/Cache.scala index afd8273dba..69904d922b 100644 --- a/webapi/src/main/scala/org/knora/webapi/responders/v2/ontology/Cache.scala +++ b/webapi/src/main/scala/org/knora/webapi/responders/v2/ontology/Cache.scala @@ -69,19 +69,27 @@ object Cache extends LazyLogging { /** * The in-memory cache of ontologies. * - * @param ontologies a map of ontology IRIs to ontologies. - * @param subClassOfRelations a map of subclasses to their base classes. - * @param superClassOfRelations a map of base classes to their subclasses. - * @param subPropertyOfRelations a map of subproperties to their base proeprties. - * @param guiAttributeDefinitions a map of salsah-gui:Guielement individuals to their GUI attribute definitions. - * @param standoffProperties a set of standoff properties. + * @param ontologies a map of ontology IRIs to ontologies. + * @param subClassOfRelations a map of subclasses to their base classes. + * @param superClassOfRelations a map of base classes to their subclasses. + * @param subPropertyOfRelations a map of subproperties to their base properties. + * @param superPropertyOfRelations a map of base classes to their subproperties. + * @param guiAttributeDefinitions a map of salsah-gui:Guielement individuals to their GUI attribute definitions. + * @param classDefinedInOntology a map of class IRIs to the ontology where the class is defined + * @param propertyDefinedInOntology a map of property IRIs to the ontology where the property is defined + * @param entityDefinedInOntology a map of entity IRIs (property or class) to the ontology where the entity is defined + * @param standoffProperties a set of standoff properties. */ case class OntologyCacheData( ontologies: Map[SmartIri, ReadOntologyV2], subClassOfRelations: Map[SmartIri, Seq[SmartIri]], superClassOfRelations: Map[SmartIri, Set[SmartIri]], subPropertyOfRelations: Map[SmartIri, Set[SmartIri]], + superPropertyOfRelations: Map[SmartIri, Set[SmartIri]], guiAttributeDefinitions: Map[SmartIri, Set[SalsahGuiAttributeDefinition]], + classDefinedInOntology: Map[SmartIri, SmartIri], + propertyDefinedInOntology: Map[SmartIri, SmartIri], + entityDefinedInOntology: Map[SmartIri, SmartIri], standoffProperties: Set[SmartIri] ) { lazy val allPropertyDefs: Map[SmartIri, PropertyInfoContentV2] = ontologies.values @@ -305,6 +313,11 @@ object Cache extends LazyLogging { (propertyIri, OntologyUtil.getAllBaseDefs(propertyIri, directSubPropertyOfRelations).toSet + propertyIri) }.toMap + // Make a map in which each property IRI points to the full set of its subproperties. A property is also + // a superproperty of itself. + val allSuperPropertyOfRelations: Map[SmartIri, Set[SmartIri]] = + OntologyHelpers.calculateSuperPropertiesOfRelations(allSubPropertyOfRelations) + // A set of all subproperties of knora-base:resourceProperty. val allKnoraResourceProps: Set[SmartIri] = allPropertyIris.filter { prop => val allPropSubPropertyOfRelations = allSubPropertyOfRelations(prop) @@ -429,6 +442,13 @@ object Cache extends LazyLogging { } }.toSet + val classDefinedInOntology = classIrisPerOntology.flatMap { case (ontoIri, classIris) => + classIris.map(_ -> ontoIri) + } + val propertyDefinedInOntology = propertyIrisPerOntology.flatMap { case (ontoIri, propertyIris) => + propertyIris.map(_ -> ontoIri) + } + // Construct the ontology cache data. val ontologyCacheData: OntologyCacheData = OntologyCacheData( ontologies = new ErrorHandlingMap[SmartIri, ReadOntologyV2](readOntologies, key => s"Ontology not found: $key"), @@ -438,6 +458,16 @@ object Cache extends LazyLogging { new ErrorHandlingMap[SmartIri, Set[SmartIri]](allSuperClassOfRelations, key => s"Class not found: $key"), subPropertyOfRelations = new ErrorHandlingMap[SmartIri, Set[SmartIri]](allSubPropertyOfRelations, key => s"Property not found: $key"), + superPropertyOfRelations = + new ErrorHandlingMap[SmartIri, Set[SmartIri]](allSuperPropertyOfRelations, key => s"Property not found: $key"), + classDefinedInOntology = + new ErrorHandlingMap[SmartIri, SmartIri](classDefinedInOntology, key => s"Class not found: $key"), + propertyDefinedInOntology = + new ErrorHandlingMap[SmartIri, SmartIri](propertyDefinedInOntology, key => s"Property not found: $key"), + entityDefinedInOntology = new ErrorHandlingMap[SmartIri, SmartIri]( + propertyDefinedInOntology ++ classDefinedInOntology, + key => s"Property not found: $key" + ), guiAttributeDefinitions = new ErrorHandlingMap[SmartIri, Set[SalsahGuiAttributeDefinition]]( allGuiAttributeDefinitions, key => s"salsah-gui:Guielement not found: $key" diff --git a/webapi/src/main/scala/org/knora/webapi/responders/v2/ontology/OntologyHelpers.scala b/webapi/src/main/scala/org/knora/webapi/responders/v2/ontology/OntologyHelpers.scala index efcee878fb..c07617f661 100644 --- a/webapi/src/main/scala/org/knora/webapi/responders/v2/ontology/OntologyHelpers.scala +++ b/webapi/src/main/scala/org/knora/webapi/responders/v2/ontology/OntologyHelpers.scala @@ -2108,6 +2108,25 @@ object OntologyHelpers { baseClass -> baseClassAndSubClasses.map(_._2).toSet } + /** + * Given all the `rdfs:subPropertyOf` relations between properties, calculates all the inverse relations. + * + * @param allSubPropertiesOfRelations all the `rdfs:subPropertyOf` relations between properties. + * @return a map of IRIs of properties to sets of the IRIs of their subproperties. + */ + def calculateSuperPropertiesOfRelations( + allSubPropertiesOfRelations: Map[SmartIri, Set[SmartIri]] + ): Map[SmartIri, Set[SmartIri]] = + allSubPropertiesOfRelations.toVector.flatMap { case (subProp: SmartIri, baseProps: Set[SmartIri]) => + baseProps.map { baseProp => + baseProp -> subProp + } + } + .groupBy(_._1) + .map { case (baseProp: SmartIri, basePropAndSubProps: Vector[(SmartIri, SmartIri)]) => + baseProp -> basePropAndSubProps.map(_._2).toSet + } + /** * Given a class loaded from the triplestore, recursively adds its inherited cardinalities to the cardinalities it defines * directly. A cardinality for a subproperty in a subclass overrides a cardinality for a base property in diff --git a/webapi/src/main/scala/org/knora/webapi/store/triplestore/http/HttpTriplestoreConnector.scala b/webapi/src/main/scala/org/knora/webapi/store/triplestore/http/HttpTriplestoreConnector.scala index 72d7683ed1..49eab67cd0 100644 --- a/webapi/src/main/scala/org/knora/webapi/store/triplestore/http/HttpTriplestoreConnector.scala +++ b/webapi/src/main/scala/org/knora/webapi/store/triplestore/http/HttpTriplestoreConnector.scala @@ -791,13 +791,22 @@ class HttpTriplestoreConnector extends Actor with ActorLogging with Instrumentat (queryHttpClient, queryHttpPost) } - doHttpRequest( + val tryRequest = doHttpRequest( client = httpClient, request = httpPost, context = httpContext, processResponse = returnResponseAsString, simulateTimeout = simulateTimeout ) + tryRequest match { + case Failure(exception) => + exception match { + case TriplestoreTimeoutException(_, _) => + log.error(s"Triplestore timed out after query: $sparql") + } + tryRequest + case _ => tryRequest + } } /** diff --git a/webapi/src/test/scala/org/knora/webapi/messages/util/search/SparqlTransformerSpec.scala b/webapi/src/test/scala/org/knora/webapi/messages/util/search/SparqlTransformerSpec.scala index adb0760200..f8b7175320 100644 --- a/webapi/src/test/scala/org/knora/webapi/messages/util/search/SparqlTransformerSpec.scala +++ b/webapi/src/test/scala/org/knora/webapi/messages/util/search/SparqlTransformerSpec.scala @@ -19,6 +19,11 @@ class SparqlTransformerSpec extends CoreSpec() { protected implicit val stringFormatter: StringFormatter = StringFormatter.getGeneralInstance + private val thingIRI = "http://www.knora.org/ontology/0001/anything#Thing".toSmartIri + private val blueThingIRI = "http://www.knora.org/ontology/0001/anything#BlueThing".toSmartIri + private val hasOtherThingIRI = "http://www.knora.org/ontology/0001/anything#hasOtherThing".toSmartIri + private val hasTextIRI = "http://www.knora.org/ontology/0001/anything#hasText".toSmartIri + "SparqlTransformer" should { "create a syntactically valid base name from a given variable" in { @@ -70,7 +75,7 @@ class SparqlTransformerSpec extends CoreSpec() { val typeStatement = StatementPattern.makeExplicit( subj = QueryVariable("foo"), pred = IriRef(OntologyConstants.Rdf.Type.toSmartIri), - obj = IriRef("http://www.knora.org/ontology/0001/anything#Thing".toSmartIri) + obj = IriRef(thingIRI) ) val isDeletedStatement = StatementPattern.makeExplicit( subj = QueryVariable("foo"), @@ -79,7 +84,7 @@ class SparqlTransformerSpec extends CoreSpec() { ) val linkStatement = StatementPattern.makeExplicit( subj = QueryVariable("foo"), - pred = IriRef("http://www.knora.org/ontology/0001/anything#hasOtherThing".toSmartIri), + pred = IriRef(hasOtherThingIRI), obj = IriRef("http://rdfh.ch/0001/a-thing".toSmartIri) ) @@ -112,12 +117,12 @@ class SparqlTransformerSpec extends CoreSpec() { val typeStatement = StatementPattern.makeExplicit( subj = QueryVariable("foo"), pred = IriRef(OntologyConstants.Rdf.Type.toSmartIri), - obj = IriRef("http://www.knora.org/ontology/0001/anything#Thing".toSmartIri) + obj = IriRef(thingIRI) ) val hasValueStatement = StatementPattern.makeExplicit( subj = QueryVariable("foo"), - pred = IriRef("http://www.knora.org/ontology/0001/anything#hasText".toSmartIri), + pred = IriRef(hasTextIRI), obj = QueryVariable("text") ) val bindPattern = @@ -144,7 +149,7 @@ class SparqlTransformerSpec extends CoreSpec() { val hasValueStatement = StatementPattern.makeExplicit( subj = QueryVariable("foo"), - pred = IriRef("http://www.knora.org/ontology/0001/anything#hasText".toSmartIri), + pred = IriRef(hasTextIRI), obj = QueryVariable("text") ) val valueHasStringStatement = @@ -178,11 +183,11 @@ class SparqlTransformerSpec extends CoreSpec() { optimisedPatterns should ===(expectedPatterns) } - "expand an rdf:type statement to simulate RDFS inference" in { + "not simulate any RDF inference for a class, if there are no known subclasses" in { val typeStatement = StatementPattern.makeInferred( subj = QueryVariable("foo"), pred = IriRef(OntologyConstants.Rdf.Type.toSmartIri), - obj = IriRef("http://www.knora.org/ontology/0001/anything#Thing".toSmartIri) + obj = IriRef(blueThingIRI) ) val expandedStatements = SparqlTransformer.transformStatementInWhereForNoInference( statementPattern = typeStatement, @@ -191,36 +196,76 @@ class SparqlTransformerSpec extends CoreSpec() { val expectedStatements: Seq[StatementPattern] = Seq( StatementPattern( - subj = QueryVariable(variableName = "foo__subClassOf__httpwwwknoraorgontology0001anythingThing"), - pred = IriRef( - iri = OntologyConstants.Rdfs.SubClassOf.toSmartIri, - propertyPathOperator = Some('*') + subj = QueryVariable("foo"), + pred = IriRef(OntologyConstants.Rdf.Type.toSmartIri), + obj = IriRef(blueThingIRI), + namedGraph = None + ) + ) + + expandedStatements should equal(expectedStatements) + } + + "create a union pattern to simulate RDF inference for a class, if there are known subclasses" in { + val typeStatement = StatementPattern.makeInferred( + subj = QueryVariable("foo"), + pred = IriRef(OntologyConstants.Rdf.Type.toSmartIri), + obj = IriRef(thingIRI) + ) + val expandedStatements = SparqlTransformer.transformStatementInWhereForNoInference( + statementPattern = typeStatement, + simulateInference = true + ) + + val expectedUnionPattern = UnionPattern( + Seq( + Seq( + StatementPattern( + subj = QueryVariable("foo"), + pred = IriRef(OntologyConstants.Rdf.Type.toSmartIri), + obj = IriRef(thingIRI), + namedGraph = None + ) ), - obj = IriRef( - iri = "http://www.knora.org/ontology/0001/anything#Thing".toSmartIri, - propertyPathOperator = None + Seq( + StatementPattern( + subj = QueryVariable("foo"), + pred = IriRef(OntologyConstants.Rdf.Type.toSmartIri), + obj = IriRef(blueThingIRI), + namedGraph = None + ) ), - namedGraph = None - ), - StatementPattern( - subj = QueryVariable(variableName = "foo"), - pred = IriRef( - iri = OntologyConstants.Rdf.Type.toSmartIri, - propertyPathOperator = None + Seq( + StatementPattern( + subj = QueryVariable("foo"), + pred = IriRef(OntologyConstants.Rdf.Type.toSmartIri), + obj = IriRef("http://www.knora.org/ontology/0001/something#Something".toSmartIri), + namedGraph = None + ) ), - obj = QueryVariable(variableName = "foo__subClassOf__httpwwwknoraorgontology0001anythingThing"), - namedGraph = None + Seq( + StatementPattern( + subj = QueryVariable("foo"), + pred = IriRef(OntologyConstants.Rdf.Type.toSmartIri), + obj = IriRef("http://www.knora.org/ontology/0001/anything#ThingWithSeqnum".toSmartIri), + namedGraph = None + ) + ) ) ) - expandedStatements should ===(expectedStatements) + expandedStatements match { + case (head: UnionPattern) :: Nil => + head.blocks.toSet should equal(expectedUnionPattern.blocks.toSet) + case _ => throw new AssertionError("Simulated RDF inference should have resulted in exactly one Union Pattern") + } } - "expand a statement with a property IRI to simulate RDFS inference" in { + "not simulate any RDF inference for a property, if there are no known subproperties" in { val hasValueStatement = StatementPattern.makeInferred( subj = QueryVariable("foo"), - pred = IriRef("http://www.knora.org/ontology/0001/anything#hasText".toSmartIri), + pred = IriRef(hasTextIRI), obj = QueryVariable("text") ) val expandedStatements = SparqlTransformer.transformStatementInWhereForNoInference( @@ -230,26 +275,73 @@ class SparqlTransformerSpec extends CoreSpec() { val expectedStatements: Seq[StatementPattern] = Seq( StatementPattern( - subj = QueryVariable(variableName = "httpwwwknoraorgontology0001anythinghasText__subPropertyOf"), + subj = QueryVariable(variableName = "foo"), pred = IriRef( - iri = OntologyConstants.Rdfs.SubPropertyOf.toSmartIri, - propertyPathOperator = Some('*') - ), - obj = IriRef( - iri = "http://www.knora.org/ontology/0001/anything#hasText".toSmartIri, + iri = hasTextIRI, propertyPathOperator = None ), - namedGraph = None - ), - StatementPattern( - subj = QueryVariable(variableName = "foo"), - pred = QueryVariable(variableName = "httpwwwknoraorgontology0001anythinghasText__subPropertyOf"), obj = QueryVariable(variableName = "text"), namedGraph = None ) ) - expandedStatements should ===(expectedStatements) + expandedStatements should equal(expectedStatements) + } + + "create a union pattern to simulate RDF inference for a property, if there are known subproperties" in { + val hasValueStatement = + StatementPattern.makeInferred( + subj = QueryVariable("foo"), + pred = IriRef(hasOtherThingIRI), + obj = QueryVariable("text") + ) + val expandedStatements = SparqlTransformer.transformStatementInWhereForNoInference( + statementPattern = hasValueStatement, + simulateInference = true + ) + + val expectedUnionPattern: UnionPattern = UnionPattern( + Seq( + Seq( + StatementPattern( + subj = QueryVariable(variableName = "foo"), + pred = IriRef( + iri = "http://www.knora.org/ontology/0001/something#hasOtherSomething".toSmartIri, + propertyPathOperator = None + ), + obj = QueryVariable(variableName = "text"), + namedGraph = None + ) + ), + Seq( + StatementPattern( + subj = QueryVariable(variableName = "foo"), + pred = IriRef( + iri = hasOtherThingIRI, + propertyPathOperator = None + ), + obj = QueryVariable(variableName = "text"), + namedGraph = None + ) + ), + Seq( + StatementPattern( + subj = QueryVariable(variableName = "foo"), + pred = IriRef( + iri = "http://www.knora.org/ontology/0001/anything#hasBlueThing".toSmartIri, + propertyPathOperator = None + ), + obj = QueryVariable(variableName = "text"), + namedGraph = None + ) + ) + ) + ) + expandedStatements match { + case (head: UnionPattern) :: Nil => + head.blocks.toSet should equal(expectedUnionPattern.blocks.toSet) + case _ => throw new AssertionError("Simulated RDF inference should have resulted in exactly one Union Pattern") + } } } } diff --git a/webapi/src/test/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/NonTriplestoreSpecificGravsearchToCountPrequeryTransformerSpec.scala b/webapi/src/test/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/NonTriplestoreSpecificGravsearchToCountPrequeryTransformerSpec.scala index ded3180fa0..b6cf4ed71d 100644 --- a/webapi/src/test/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/NonTriplestoreSpecificGravsearchToCountPrequeryTransformerSpec.scala +++ b/webapi/src/test/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/NonTriplestoreSpecificGravsearchToCountPrequeryTransformerSpec.scala @@ -1,5 +1,6 @@ package org.knora.webapi.util.search.gravsearch.prequery +import akka.actor.ActorSystem import org.knora.webapi.CoreSpec import org.knora.webapi.exceptions.AssertionException import org.knora.webapi.feature.FeatureFactoryConfig @@ -13,10 +14,12 @@ import org.knora.webapi.messages.util.search.gravsearch.GravsearchQueryChecker import org.knora.webapi.messages.util.search.gravsearch.prequery.NonTriplestoreSpecificGravsearchToCountPrequeryTransformer import org.knora.webapi.messages.util.search.gravsearch.types.GravsearchTypeInspectionRunner import org.knora.webapi.messages.util.search.gravsearch.types.GravsearchTypeInspectionUtil +import org.knora.webapi.settings.KnoraDispatchers import org.knora.webapi.sharedtestdata.SharedTestDataADM import scala.collection.mutable.ArrayBuffer import scala.concurrent.Await +import scala.concurrent.ExecutionContext import scala.concurrent.duration._ private object CountQueryHandler { @@ -29,7 +32,7 @@ private object CountQueryHandler { query: String, responderData: ResponderData, featureFactoryConfig: FeatureFactoryConfig - ): SelectQuery = { + )(implicit executionContext: ExecutionContext): SelectQuery = { val constructQuery = GravsearchParser.parseQuery(query) diff --git a/webapi/src/test/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/NonTriplestoreSpecificGravsearchToPrequeryTransformerSpec.scala b/webapi/src/test/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/NonTriplestoreSpecificGravsearchToPrequeryTransformerSpec.scala index fefd813b1a..fad150283c 100644 --- a/webapi/src/test/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/NonTriplestoreSpecificGravsearchToPrequeryTransformerSpec.scala +++ b/webapi/src/test/scala/org/knora/webapi/messages/util/search/gravsearch/prequery/NonTriplestoreSpecificGravsearchToPrequeryTransformerSpec.scala @@ -20,6 +20,7 @@ import org.knora.webapi.util.ApacheLuceneSupport.LuceneQueryString import scala.collection.mutable.ArrayBuffer import scala.concurrent.Await import scala.concurrent.duration._ +import scala.concurrent.ExecutionContext private object QueryHandler { @@ -32,7 +33,7 @@ private object QueryHandler { responderData: ResponderData, settings: KnoraSettingsImpl, featureFactoryConfig: FeatureFactoryConfig - ): SelectQuery = { + )(implicit executionContext: ExecutionContext): SelectQuery = { val constructQuery = GravsearchParser.parseQuery(query) diff --git a/webapi/src/test/scala/org/knora/webapi/responders/v2/SearchResponderV2Spec.scala b/webapi/src/test/scala/org/knora/webapi/responders/v2/SearchResponderV2Spec.scala index 03df557c1e..3e310b697a 100644 --- a/webapi/src/test/scala/org/knora/webapi/responders/v2/SearchResponderV2Spec.scala +++ b/webapi/src/test/scala/org/knora/webapi/responders/v2/SearchResponderV2Spec.scala @@ -287,6 +287,23 @@ class SearchResponderV2Spec extends CoreSpec() with ImplicitSender { } } + "perform an extended search for a particular compound object (book)" in { + + val query = searchResponderV2SpecFullData.constructQueryForIncunabulaCompundObject + + responderManager ! GravsearchRequestV2( + constructQuery = query, + targetSchema = ApiV2Complex, + schemaOptions = SchemaOptions.ForStandoffWithTextValues, + featureFactoryConfig = defaultFeatureFactoryConfig, + requestingUser = SharedTestDataADM.anonymousUser + ) + + expectMsgPF(timeout) { case response: ReadResourcesSequenceV2 => + response.resources.length should equal(25) + } + } + } } diff --git a/webapi/src/test/scala/org/knora/webapi/responders/v2/SearchResponderV2SpecFullData.scala b/webapi/src/test/scala/org/knora/webapi/responders/v2/SearchResponderV2SpecFullData.scala index 8369f7f3d9..4490db4a12 100644 --- a/webapi/src/test/scala/org/knora/webapi/responders/v2/SearchResponderV2SpecFullData.scala +++ b/webapi/src/test/scala/org/knora/webapi/responders/v2/SearchResponderV2SpecFullData.scala @@ -2132,4 +2132,100 @@ class SearchResponderV2SpecFullData(implicit stringFormatter: StringFormatter) { ) ) ) + + val constructQueryForIncunabulaCompundObject: ConstructQuery = ConstructQuery( + constructClause = ConstructClause( + statements = Vector( + StatementPattern( + QueryVariable("page"), + IriRef("http://api.knora.org/ontology/knora-api/simple/v2#isMainResource".toSmartIri, None), + XsdLiteral("true", "http://www.w3.org/2001/XMLSchema#boolean".toSmartIri), + None + ), + StatementPattern( + QueryVariable("page"), + IriRef("http://api.knora.org/ontology/knora-api/simple/v2#seqnum".toSmartIri, None), + QueryVariable("seqnum"), + None + ), + StatementPattern( + QueryVariable("page"), + IriRef("http://api.knora.org/ontology/knora-api/simple/v2#hasStillImageFile".toSmartIri, None), + QueryVariable("file"), + None + ) + ), + querySchema = Some(ApiV2Simple) + ), + whereClause = WhereClause( + patterns = Vector( + StatementPattern( + QueryVariable("page"), + IriRef("http://www.w3.org/1999/02/22-rdf-syntax-ns#type".toSmartIri, None), + IriRef("http://api.knora.org/ontology/knora-api/simple/v2#StillImageRepresentation".toSmartIri, None), + None + ), + StatementPattern( + QueryVariable("page"), + IriRef("http://www.w3.org/1999/02/22-rdf-syntax-ns#type".toSmartIri, None), + IriRef("http://api.knora.org/ontology/knora-api/simple/v2#Resource".toSmartIri, None), + None + ), + StatementPattern( + QueryVariable("page"), + IriRef("http://api.knora.org/ontology/knora-api/simple/v2#isPartOf".toSmartIri, None), + IriRef("http://rdfh.ch/0803/861b5644b302".toSmartIri, None), + None + ), + StatementPattern( + IriRef("http://rdfh.ch/0803/861b5644b302".toSmartIri, None), + IriRef("http://www.w3.org/1999/02/22-rdf-syntax-ns#type".toSmartIri, None), + IriRef("http://api.knora.org/ontology/knora-api/simple/v2#Resource".toSmartIri, None), + None + ), + StatementPattern( + QueryVariable("page"), + IriRef("http://api.knora.org/ontology/knora-api/simple/v2#seqnum".toSmartIri, None), + QueryVariable("seqnum"), + None + ), + StatementPattern( + QueryVariable("seqnum"), + IriRef("http://www.w3.org/1999/02/22-rdf-syntax-ns#type".toSmartIri, None), + IriRef("http://www.w3.org/2001/XMLSchema#integer".toSmartIri, None), + None + ), + StatementPattern( + QueryVariable("page"), + IriRef("http://api.knora.org/ontology/knora-api/simple/v2#hasStillImageFile".toSmartIri, None), + QueryVariable("file"), + None + ), + StatementPattern( + QueryVariable("file"), + IriRef("http://www.w3.org/1999/02/22-rdf-syntax-ns#type".toSmartIri, None), + IriRef("http://api.knora.org/ontology/knora-api/simple/v2#File".toSmartIri, None), + None + ) + ), + positiveEntities = Set( + QueryVariable("page"), + QueryVariable("seqnum"), + QueryVariable("file"), + IriRef("http://www.w3.org/1999/02/22-rdf-syntax-ns#type".toSmartIri, None), + IriRef("http://api.knora.org/ontology/knora-api/simple/v2#isMainResource".toSmartIri, None), + IriRef("http://api.knora.org/ontology/knora-api/simple/v2#StillImageRepresentation".toSmartIri, None), + IriRef("http://api.knora.org/ontology/knora-api/simple/v2#Resource".toSmartIri, None), + IriRef("http://api.knora.org/ontology/knora-api/simple/v2#isPartOf".toSmartIri, None), + IriRef("http://rdfh.ch/0803/861b5644b302".toSmartIri, None), + IriRef("http://api.knora.org/ontology/knora-api/simple/v2#seqnum".toSmartIri, None), + IriRef("http://www.w3.org/2001/XMLSchema#integer".toSmartIri, None), + IriRef("http://api.knora.org/ontology/knora-api/simple/v2#hasStillImageFile".toSmartIri, None), + IriRef("http://api.knora.org/ontology/knora-api/simple/v2#File".toSmartIri, None) + ), + querySchema = Some(ApiV2Simple) + ), + querySchema = Some(ApiV2Simple) + ) + } diff --git a/webapi/src/test/scala/org/knora/webapi/util/StartupUtils.scala b/webapi/src/test/scala/org/knora/webapi/util/StartupUtils.scala index 1552e04dcc..3a02920c0b 100644 --- a/webapi/src/test/scala/org/knora/webapi/util/StartupUtils.scala +++ b/webapi/src/test/scala/org/knora/webapi/util/StartupUtils.scala @@ -31,7 +31,7 @@ trait StartupUtils extends LazyLogging { */ def applicationStateRunning(): Unit = { - implicit val timeout: Timeout = Timeout(5.second) + implicit val timeout: Timeout = Timeout(10.second) val state: AppState = Await.result(appActor ? GetAppState(), timeout.duration).asInstanceOf[AppState] if (state != AppStates.Running) {