dasch-swiss · benjamingeer · Apr 7, 2020 · Feb 19, 2020 · Feb 28, 2020 · Feb 28, 2020
diff --git a/docs/src/paradox/03-apis/api-v2/query-language.md b/docs/src/paradox/03-apis/api-v2/query-language.md
@@ -186,9 +186,21 @@ that refer to events that took place within a certain date range.
 
 Each matching resource is returned with the values that the user has
 permission to see. If the user does not have permission to see a matching
-main resource, it is replaced by a proxy resource called
-`knora-api:ForbiddenResource`. If a user does not have permission to see
-a matching dependent resource, only its IRI is returned.
+main resource, it is hidden in the results. If a user does not have
+permission to see a matching dependent resource, the link value is hidden.
+
+## Paging
+
+Gravsearch results are returned in pages. The maximum number of main
+resources per page is determined by Knora (and can be configured
+in `application.conf` via the setting `app/v2/resources-sequence/results-per-page`).
+If some resources have been filtered out because the user does not have
+permission to see them, a page could contain fewer results, or no results.
+If it is possible that more results are available in subsequent pages,
+the Gravsearch response will contain the predicate `knora-api:mayHaveMoreResults`
+with the boolean value `true`, otherwise it will not contain this predicate.
+Therefore, to retrieve all available results, the client must request each page
+one at a time, until the response does not contain `knora-api:mayHaveMoreResults`.
 
 ## Inference
 
@@ -227,8 +239,7 @@ clauses use the following patterns, with the specified restrictions:
 - `OFFSET`: the `OFFSET` is needed for paging. It does not actually
   refer to the number of triples to be returned, but to the
   requested page of results. The default value is 0, which refers
-  to the first page of results. The number of results per page is
-  defined in `app/v2` in `application.conf`.
+  to the first page of results.
 - `ORDER BY`: In SPARQL, the result of a `CONSTRUCT` query is an
   unordered set of triples. However, a Gravsearch query returns an
   ordered list of resources, which can be ordered by the values of

diff --git a/docs/src/paradox/05-internals/design/api-v2/gravsearch.md b/docs/src/paradox/05-internals/design/api-v2/gravsearch.md
@@ -228,32 +228,39 @@ the main query can specifically ask for more detailed information on these resou
 
 #### Generating the Main Query
 
-The classes involved in generating prequeries can be found in `org.knora.webapi.responders.v2.search.gravsearch.mainquery`.
-
-The main query is a SPARQL CONSTRUCT query. Its generation is handled by the method `GravsearchMainQueryGenerator.createMainQuery`.
-It takes three arguments: `mainResourceIris: Set[IriRef], dependentResourceIris: Set[IriRef], valueObjectIris: Set[IRI]`.
-From the given IRIs, statements are generated that ask for complete information on *exactly* these resources and values.
-For any given resource IRI, only the values present in `valueObjectIris` are to be queried.
-This is achieved by using SPARQL's `VALUES` expression for the main resource and dependent resources as well as for values.
+The classes involved in generating the main query can be found in
+`org.knora.webapi.responders.v2.search.gravsearch.mainquery`.
+
+The main query is a SPARQL CONSTRUCT query. Its generation is handled by the
+method `GravsearchMainQueryGenerator.createMainQuery`.
+It takes three arguments: `mainResourceIris: Set[IriRef], dependentResourceIris:
+Set[IriRef], valueObjectIris: Set[IRI]`. From the given Iris, statements are
+generated that ask for complete information on *exactly* these resources and
+values. For any given resource Iri, only the values present in
+`valueObjectIris` are to be queried. This is achieved by using SPARQL's
+`VALUES` expression for the main resource and dependent resources as well as
+for values.
 
 #### Processing the Main Query's results
 
-When processing the main query's results, permissions are checked and resources and values that the user did not explicitly ask for in the input query are filtered out. This is implemented in `MainQueryResultProcessor`.
-
-The method `getMainQueryResultsWithFullGraphPattern` takes the main query's results as an input and makes sure that the client has sufficient permissions on the results.
-A main resource and its dependent resources and values are only returned if the user has view permissions on all the resources and value objects present in the main query.
-Otherwise the method suppresses the main resource.
-To do the permission checking, the results of the main query are passed to `ConstructResponseUtilV2` which transforms a `SparqlConstructResponse` (a set of RDF triples)
-into a structure organized by main resource IRIs. In this structure, dependent resources and values are nested can be accessed via their main resource.
-`SparqlConstructResponse` suppresses all resources and values the user has insufficient permissions on.
-For each main resource, a check is performed for the presence of all resources and values after permission checking.
-
-The method `getRequestedValuesFromResultsWithFullGraphPattern` filters out those resources and values that the user does not want to be returned by the query.
-All the resources and values not present in the input query's CONSTRUCT clause are filtered out. This only happens after permission checking.
-
-The main resources that have been filtered out due to insufficient permissions are represented by the placeholder `ForbiddenResource`.
-This placeholder stands for a main resource that cannot be returned, nevertheless it informs the client that such a resource exists.
-This is necessary for a consistent behaviour when doing paging.
+To do the permission checking, the results of the main query are passed to
+`ConstructResponseUtilV2.splitMainResourcesAndValueRdfData`,
+which transforms a `SparqlConstructResponse` (a set of RDF triples)
+into a structure organized by main resource Iris. In this structure, dependent
+resources and values are nested and can be accessed via their main resource,
+and resources and values that the user does not have permission to see are
+filtered out. As a result, a page of results may contain fewer than the maximum
+allowed number of results per page, even if more pages of results are available.
+
+`MainQueryResultProcessor.getRequestedValuesFromResultsWithFullGraphPattern`
+then filters out values that the user did not explicitly ask for in the input
+query.
+
+Finally, `ConstructResponseUtilV2.createApiResponse` transforms the query
+results into an API response (a `ReadResourcesSequenceV2`). If the number
+of main resources found (even if filtered out because of permissions) is equal
+to the maximum allowed page size, the predicate
+`knora-api:mayHaveMoreResults: true` is included in the response.
 
 ## Inference
 
@@ -279,5 +286,6 @@ as an optimisation if the triplestore provides it. For example, the virtual prop
 `knora-api:standoffTagHasStartAncestor` is equivalent to `knora-base:standoffTagHasStartParent*`, but
 with GraphDB it is implemented using a custom inference rule (in `KnoraRules.pie`) and is therefore more
 efficient. If Knora is not using the triplestore's inference,
+
 `SparqlTransformer.transformStatementInWhereForNoInference` replaces `knora-api:standoffTagHasStartAncestor`
 with `knora-base:standoffTagHasStartParent*`.
diff --git a/knora-ontologies/knora-base.ttl b/knora-ontologies/knora-base.ttl
@@ -33,7 +33,7 @@
 
     :attachedToProject knora-admin:SystemProject ;
 
-    :ontologyVersion "knora-base v7" .
+    :ontologyVersion "knora-base v8" .
 
 
 
@@ -2558,19 +2558,6 @@
                     rdfs:comment "A resource containing a text file"@en .
 
 
-### http://www.knora.org/ontology/knora-base#ForbiddenResource
-
-:ForbiddenResource  rdf:type owl:Class ;
-
-                    rdfs:subClassOf :Resource ,
-                    [ rdf:type owl:Restriction ;
-                      owl:onProperty :hasComment ;
-                      owl:minCardinality "0"^^xsd:nonNegativeInteger
-                    ];
-
-                    rdfs:label """A ForbiddenResource is a proxy for a resource that the client has insufficient permissions to see."""@en ;
-
-                    rdfs:comment """A ForbiddenResource is a proxy for a resource that the client has insufficient permissions to see."""@en .
 
 ### http://www.knora.org/ontology/knora-base#XSLTransformation
 

diff --git a/upgrade/src/main/scala/org.knora.upgrade/Main.scala b/upgrade/src/main/scala/org.knora.upgrade/Main.scala
@@ -51,7 +51,8 @@ object Main extends App {
         PluginForKnoraBaseVersion(versionNumber = 4, plugin = new UpgradePluginPR1372, prBasedVersionString = Some("PR 1372")),
         PluginForKnoraBaseVersion(versionNumber = 5, plugin = new NoopPlugin, prBasedVersionString = Some("PR 1440")),
         PluginForKnoraBaseVersion(versionNumber = 6, plugin = new NoopPlugin), // PR 1206
-        PluginForKnoraBaseVersion(versionNumber = 7, plugin = new NoopPlugin) // PR 1403
+        PluginForKnoraBaseVersion(versionNumber = 7, plugin = new NoopPlugin), // PR 1403
+        PluginForKnoraBaseVersion(versionNumber = 8, plugin = new NoopPlugin) // PR 1615
     )
 
     /**

diff --git a/webapi/_test_data/all_data/anything-data.ttl b/webapi/_test_data/all_data/anything-data.ttl
@@ -1618,6 +1618,117 @@
   knora-base:hasPermissions "CR knora-admin:Creator|M knora-admin:ProjectMember|V knora-admin:UnknownUser";
   knora-base:attachedToUser <http://rdfh.ch/users/9XBCrDV3SRa7kS1WwynB4Q> .
 
+<http://rdfh.ch/0001/0JhgKcqoRIeRRG6ownArSw> a anything:Thing;
+  knora-base:attachedToUser <http://rdfh.ch/users/9XBCrDV3SRa7kS1WwynB4Q>;
+  knora-base:attachedToProject <http://rdfh.ch/projects/0001>;
+  knora-base:hasPermissions "V knora-admin:UnknownUser|M knora-admin:ProjectMember";
+  knora-base:creationDate "2020-04-07T09:12:56.710717Z"^^xsd:dateTime;
+  anything:hasOtherThingValue <http://rdfh.ch/0001/0JhgKcqoRIeRRG6ownArSw/values/UgSp5mXTTSKdI02ZU1KIAA>;
+  anything:hasOtherThing <http://rdfh.ch/0001/XTxSMt0ySraVmwXD-bD2wQ>;
+  anything:hasInteger <http://rdfh.ch/0001/0JhgKcqoRIeRRG6ownArSw/values/U1PwfNaVRQebbOSFWNdMqQ>;
+  rdfs:label "thing with one hidden thing";
+  knora-base:isDeleted false .
+
+<http://rdfh.ch/0001/0JhgKcqoRIeRRG6ownArSw/values/UgSp5mXTTSKdI02ZU1KIAA> a knora-base:LinkValue;
+  knora-base:valueHasUUID "UgSp5mXTTSKdI02ZU1KIAA"^^xsd:string;
+  rdf:subject <http://rdfh.ch/0001/0JhgKcqoRIeRRG6ownArSw>;
+  rdf:predicate anything:hasOtherThing;
+  rdf:object <http://rdfh.ch/0001/XTxSMt0ySraVmwXD-bD2wQ>;
+  knora-base:isDeleted false;
+  knora-base:attachedToUser <http://rdfh.ch/users/9XBCrDV3SRa7kS1WwynB4Q>;
+  knora-base:hasPermissions "V knora-admin:UnknownUser|M knora-admin:ProjectMember";
+  knora-base:valueCreationDate "2020-04-07T09:12:56.710717Z"^^xsd:dateTime;
+  knora-base:valueHasString "http://rdfh.ch/0001/XTxSMt0ySraVmwXD-bD2wQ";
+  knora-base:valueHasComment "link value pointing to hidden resource";
+  knora-base:valueHasRefCount "1"^^xsd:int .
+
+<http://rdfh.ch/0001/0JhgKcqoRIeRRG6ownArSw/values/U1PwfNaVRQebbOSFWNdMqQ> a knora-base:IntValue;
+  knora-base:attachedToUser <http://rdfh.ch/users/9XBCrDV3SRa7kS1WwynB4Q>;
+  knora-base:hasPermissions "V knora-admin:UnknownUser|M knora-admin:ProjectMember";
+  knora-base:valueHasUUID "U1PwfNaVRQebbOSFWNdMqQ"^^xsd:string;
+  knora-base:isDeleted false;
+  knora-base:valueCreationDate "2020-04-07T09:12:56.710717Z"^^xsd:dateTime;
+  knora-base:valueHasInteger 123454321;
+  knora-base:valueHasOrder 0;
+  knora-base:valueHasComment "visible int value in main resource";
+  knora-base:valueHasString "123454321" .
+
+<http://rdfh.ch/0001/XTxSMt0ySraVmwXD-bD2wQ> a anything:Thing;
+  knora-base:attachedToUser <http://rdfh.ch/users/BhkfBc3hTeS_IDo-JgXRbQ>;
+  knora-base:attachedToProject <http://rdfh.ch/projects/0001>;
+  knora-base:hasPermissions "V knora-admin:Creator";
+  knora-base:creationDate "2020-04-07T09:12:56.710717Z"^^xsd:dateTime;
+  anything:hasInteger <http://rdfh.ch/0001/XTxSMt0ySraVmwXD-bD2wQ/values/PVPAa37xR--K_wxQwlvSsg>;
+  rdfs:label "hidden thing";
+  knora-base:isDeleted false .
+
+<http://rdfh.ch/0001/XTxSMt0ySraVmwXD-bD2wQ/values/PVPAa37xR--K_wxQwlvSsg> a knora-base:IntValue;
+  knora-base:attachedToUser <http://rdfh.ch/users/BhkfBc3hTeS_IDo-JgXRbQ>;
+  knora-base:hasPermissions "V knora-admin:Creator";
+  knora-base:valueHasUUID "PVPAa37xR--K_wxQwlvSsg"^^xsd:string;
+  knora-base:isDeleted false;
+  knora-base:valueCreationDate "2020-04-07T09:12:56.710717Z"^^xsd:dateTime;
+  knora-base:valueHasInteger 123454321;
+  knora-base:valueHasOrder 0;
+  knora-base:valueHasComment "hidden int value in hidden resource";
+  knora-base:valueHasString "123454321" .
+
+<http://rdfh.ch/0001/l8f8FVEiSCeq9A1p8gBR-A> a anything:Thing;
+  knora-base:attachedToUser <http://rdfh.ch/users/9XBCrDV3SRa7kS1WwynB4Q>;
+  knora-base:attachedToProject <http://rdfh.ch/projects/0001>;
+  knora-base:hasPermissions "V knora-admin:UnknownUser|M knora-admin:ProjectMember";
+  knora-base:creationDate "2020-04-07T09:12:56.710717Z"^^xsd:dateTime;
+  anything:hasOtherThingValue <http://rdfh.ch/0001/l8f8FVEiSCeq9A1p8gBR-A/values/Nlcc7XWXQtmEITsIRQ5z4w>;
+  anything:hasOtherThing <http://rdfh.ch/0001/PHbbrEsVR32q5D_ioKt6pA>;
+  anything:hasInteger <http://rdfh.ch/0001/l8f8FVEiSCeq9A1p8gBR-A/values/a-40v6WiT4GHa79Kqwojjw>;
+  rdfs:label "thing with one deleted thing";
+  knora-base:isDeleted false .
+
+<http://rdfh.ch/0001/l8f8FVEiSCeq9A1p8gBR-A/values/Nlcc7XWXQtmEITsIRQ5z4w> a knora-base:LinkValue;
+  knora-base:valueHasUUID "Nlcc7XWXQtmEITsIRQ5z4w"^^xsd:string;
+  rdf:subject <http://rdfh.ch/0001/l8f8FVEiSCeq9A1p8gBR-A>;
+  rdf:predicate anything:hasOtherThing;
+  rdf:object <http://rdfh.ch/0001/PHbbrEsVR32q5D_ioKt6pA>;
+  knora-base:isDeleted false;
+  knora-base:attachedToUser <http://rdfh.ch/users/9XBCrDV3SRa7kS1WwynB4Q>;
+  knora-base:hasPermissions "V knora-admin:UnknownUser|M knora-admin:ProjectMember";
+  knora-base:valueCreationDate "2020-04-07T09:12:56.710717Z"^^xsd:dateTime;
+  knora-base:valueHasString "http://rdfh.ch/0001/XTxSMt0ySraVmwXD-bD2wQ";
+  knora-base:valueHasComment "link value pointing to deleted resource";
+  knora-base:valueHasRefCount "1"^^xsd:int .
+
+<http://rdfh.ch/0001/l8f8FVEiSCeq9A1p8gBR-A/values/a-40v6WiT4GHa79Kqwojjw> a knora-base:IntValue;
+  knora-base:attachedToUser <http://rdfh.ch/users/9XBCrDV3SRa7kS1WwynB4Q>;
+  knora-base:hasPermissions "V knora-admin:UnknownUser|M knora-admin:ProjectMember";
+  knora-base:valueHasUUID "a-40v6WiT4GHa79Kqwojjw"^^xsd:string;
+  knora-base:isDeleted false;
+  knora-base:valueCreationDate "2020-04-07T09:12:56.710717Z"^^xsd:dateTime;
+  knora-base:valueHasInteger 123454321;
+  knora-base:valueHasOrder 0;
+  knora-base:valueHasComment "int value in main resource";
+  knora-base:valueHasString "123454321" .
+
+<http://rdfh.ch/0001/PHbbrEsVR32q5D_ioKt6pA> a anything:Thing;
+  knora-base:attachedToUser <http://rdfh.ch/users/9XBCrDV3SRa7kS1WwynB4Q>;
+  knora-base:attachedToProject <http://rdfh.ch/projects/0001>;
+  knora-base:hasPermissions "V knora-admin:UnknownUser|M knora-admin:ProjectMember";
+  knora-base:creationDate "2020-04-07T09:12:56.710717Z"^^xsd:dateTime;
+  anything:hasInteger <http://rdfh.ch/0001/XTxSMt0ySraVmwXD-bD2wQ/values/PVPAa37xR--K_wxQwlvSsg>;
+  rdfs:label "deleted thing";
+  knora-base:isDeleted true;
+  knora-base:deleteDate "2020-04-07T14:59:28.960124Z"^^xsd:dateTime .
+
+<http://rdfh.ch/0001/PHbbrEsVR32q5D_ioKt6pA/values/7BIm9QAiQqKixcgXDWf12Q> a knora-base:IntValue;
+  knora-base:attachedToUser <http://rdfh.ch/users/9XBCrDV3SRa7kS1WwynB4Q>;
+  knora-base:hasPermissions "V knora-admin:UnknownUser|M knora-admin:ProjectMember";
+  knora-base:valueHasUUID "7BIm9QAiQqKixcgXDWf12Q"^^xsd:string;
+  knora-base:isDeleted false;
+  knora-base:valueCreationDate "2020-04-07T09:12:56.710717Z"^^xsd:dateTime;
+  knora-base:valueHasInteger 123454321;
+  knora-base:valueHasOrder 0;
+  knora-base:valueHasComment "int value in deleted resource";
+  knora-base:valueHasString "123454321" .
+
 <http://rdfh.ch/lists/0001/otherTreeList> a knora-base:ListNode;
   knora-base:isRootNode true;
   rdfs:label "Tree list root"@en;

diff --git a/webapi/_test_data/all_data/system-data.ttl b/webapi/_test_data/all_data/system-data.ttl
@@ -39,26 +39,3 @@
 	knora-base:listNodePosition 2 ;
 	rdfs:label "Maybe"@en ;
   rdfs:label "Vielleicht"@de .
-
-
-##########################################################
-#
-# Sweet and tasty but forbidden resource
-#
-# but of the tree of the knowledge of good and evil, thou shalt not eat of it
-#
-##########################################################
-
-<http://rdfh.ch/0000/forbiddenResource> a knora-base:ForbiddenResource ;
-
-                      rdfs:label "This resource is a proxy for a resource you are not allowed to see (may depend on the context: query path)" ;
-
-                      knora-base:isDeleted false ;
-
-                      knora-base:attachedToUser <http://rdfh.ch/users/root> ;
-
-                      knora-base:attachedToProject knora-admin:SystemProject ;
-
-                      knora-base:creationDate "2017-10-06T11:05:37Z"^^xsd:dateTime ;
-
-                      knora-base:hasPermissions "CR knora-admin:Creator|M knora-admin:ProjectMember|V knora-admin:KnownUser|V knora-admin:UnknownUser" .