Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Duplicated resources in Gravsearch result #1588

Closed
gfoo opened this issue Jan 28, 2020 · 50 comments · Fixed by #1626
Closed

Duplicated resources in Gravsearch result #1588

gfoo opened this issue Jan 28, 2020 · 50 comments · Fixed by #1626
Assignees

Comments

@gfoo
Copy link

gfoo commented Jan 28, 2020

Is that possible to get duplicated resources in a result of gravsearch query? In my test, 4 resources, one duplicated.

I suppose this is not normal because if I use the same query with the /count route I get the correct result, that is "schema:numberOfItems": 3.

@benjamingeer
Copy link

Should not be possible, can you make a small test case that we can reproduce?

@benjamingeer benjamingeer self-assigned this Jan 28, 2020
@tobiasschweizer
Copy link
Contributor

tobiasschweizer commented Jan 29, 2020

Maybe a bug in src/main/scala/org/knora/webapi/util/ConstructResponseUtilV2.scala? I guess if the count query is correct, then the prequery should not be the issue as it is issued in both cases (slightly modified though), but the processing of the main query's result.

@gfoo
Copy link
Author

gfoo commented Jan 31, 2020

The issue probably comes from the ORDER BY ASC(?hasName).

We have some sparql logs to provide to you, first, the gravsearch:

PREFIX knora-api: <http://api.knora.org/ontology/knora-api/v2#>
PREFIX knora-simple-api: <http://api.knora.org/ontology/knora-api/simple/v2#>
PREFIX ll: <http://0.0.0.0:3333/ontology/0113/lumieres-lausanne/v2#>
CONSTRUCT {
  ?Person knora-api:isMainResource true .
} WHERE {
  
  ?Person a ll:Person .
  
  ?Person ll:isModern ?isModern .

?isModern knora-api:booleanValueAsBoolean ?isModern_Value .
    FILTER(?isModern_Value = "false"^^xsd:boolean)
?Person ll:mayHaveBiography ?mayHaveBiography .

?mayHaveBiography knora-api:booleanValueAsBoolean ?mayHaveBiography_Value .
    FILTER(?mayHaveBiography_Value = "true"^^xsd:boolean)
?Person ll:noticeIsCreated ?noticeIsCreated .

?noticeIsCreated knora-api:booleanValueAsBoolean ?noticeIsCreated_Value .
    FILTER(?noticeIsCreated_Value = "true"^^xsd:boolean)
  
{

{
?Person ll:noticeCreationIsValid ?noticeCreationIsValid .

?noticeCreationIsValid knora-api:booleanValueAsBoolean ?noticeCreationIsValid_Value .
    FILTER(?noticeCreationIsValid_Value = "true"^^xsd:boolean)
}
UNION
{
?Person ll:noticeModificationIsValid ?noticeModificationIsValid .

?noticeModificationIsValid knora-api:booleanValueAsBoolean ?noticeModificationIsValid_Value .
    FILTER(?noticeModificationIsValid_Value = "true"^^xsd:boolean)
}

}

{

{
?Person ll:hasName ?hasName .

?hasName knora-api:valueAsString ?hasName_Value .
    FILTER knora-api:match(?hasName_Value, "Mirab*")
}
UNION
{
?Person ll:hasPlaceBirth ?hasPlaceBirth .

?hasPlaceBirth knora-api:valueAsString ?hasPlaceBirth_Value .
    FILTER knora-api:match(?hasPlaceBirth_Value, "Mirab*")
}
UNION
{
?Person ll:hasPlaceDeath ?hasPlaceDeath .

?hasPlaceDeath knora-api:valueAsString ?hasPlaceDeath_Value .
    FILTER knora-api:match(?hasPlaceDeath_Value, "Mirab*")
}
UNION
{
?Person ll:hasPlaceOrigin ?hasPlaceOrigin .

?hasPlaceOrigin knora-api:valueAsString ?hasPlaceOrigin_Value .
    FILTER knora-api:match(?hasPlaceOrigin_Value, "Mirab*")
}
UNION
{
?Person ll:hasBiography ?hasBiography .

?hasBiography knora-api:valueAsString ?hasBiography_Value .
    FILTER knora-api:match(?hasBiography_Value, "Mirab*")
}
UNION
{
?Person ll:hasCivilStatus ?hasCivilStatus .

?hasCivilStatus knora-api:valueAsString ?hasCivilStatus_Value .
    FILTER knora-api:match(?hasCivilStatus_Value, "Mirab*")
}
UNION
{
?Person ll:isInArchive ?isInArchive .

?isInArchive knora-api:valueAsString ?isInArchive_Value .
    FILTER knora-api:match(?isInArchive_Value, "Mirab*")
}
UNION
{
?Person ll:hasCommentWork ?hasCommentWork .

?hasCommentWork knora-api:valueAsString ?hasCommentWork_Value .
    FILTER knora-api:match(?hasCommentWork_Value, "Mirab*")
}
UNION
{
?Person ll:hasNote ?hasNote .
?hasNote a ll:Note .

?hasNote ll:hasDiffusionType ?hasDiffusionType_hasNote .

?hasDiffusionType_hasNote knora-api:booleanValueAsBoolean ?hasDiffusionType_hasNote_Value .
    FILTER(?hasDiffusionType_hasNote_Value = "true"^^xsd:boolean)
?hasNote ll:noteHasText ?noteHasText_hasNote .

?noteHasText_hasNote knora-api:valueAsString ?noteHasText_hasNote_Value .
    FILTER knora-api:match(?noteHasText_hasNote_Value, "Mirab*")
}

}

}
ORDER BY ASC(?hasName)

@gfoo
Copy link
Author

gfoo commented Jan 31, 2020

The sparql behind the scene that returns 4 resources:

SELECT DISTINCT ?Person (GROUP_CONCAT(DISTINCT(?hasNote); SEPARATOR='') AS ?hasNote__Concat) (GROUP_CONCAT(DISTINCT(?hasPlaceOrigin); SEPARATOR='') AS ?hasPlaceOrigin__Concat) (GROUP_CONCAT(DISTINCT(?noticeModificationIsValid); SEPARATOR='') AS ?noticeModificationIsValid__Concat) (GROUP_CONCAT(DISTINCT(?hasPlaceBirth); SEPARATOR='') AS ?hasPlaceBirth__Concat) (GROUP_CONCAT(DISTINCT(?noticeIsCreated); SEPARATOR='') AS ?noticeIsCreated__Concat) (GROUP_CONCAT(DISTINCT(?hasPlaceDeath); SEPARATOR='') AS ?hasPlaceDeath__Concat) (GROUP_CONCAT(DISTINCT(?isModern); SEPARATOR='') AS ?isModern__Concat) (GROUP_CONCAT(DISTINCT(?hasBiography); SEPARATOR='') AS ?hasBiography__Concat) (GROUP_CONCAT(DISTINCT(?noteHasText_hasNote); SEPARATOR='') AS ?noteHasText_hasNote__Concat) (GROUP_CONCAT(DISTINCT(?isInArchive); SEPARATOR='') AS ?isInArchive__Concat) (GROUP_CONCAT(DISTINCT(?hasName); SEPARATOR='') AS ?hasName__Concat) (GROUP_CONCAT(DISTINCT(?Person__http00003333ontology0113lumiereslausannev2hasNote__hasNote__LinkValue); SEPARATOR='') AS ?Person__http00003333ontology0113lumiereslausannev2hasNote__hasNote__LinkValue__Concat) (GROUP_CONCAT(DISTINCT(?mayHaveBiography); SEPARATOR='') AS ?mayHaveBiography__Concat) (GROUP_CONCAT(DISTINCT(?noticeCreationIsValid); SEPARATOR='') AS ?noticeCreationIsValid__Concat) (GROUP_CONCAT(DISTINCT(?hasDiffusionType_hasNote); SEPARATOR='') AS ?hasDiffusionType_hasNote__Concat) (GROUP_CONCAT(DISTINCT(?hasCommentWork); SEPARATOR='') AS ?hasCommentWork__Concat) (GROUP_CONCAT(DISTINCT(?hasCivilStatus); SEPARATOR='') AS ?hasCivilStatus__Concat)
 WHERE {
 ?Person <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/knora-base#Resource> .
 GRAPH <http://www.ontotext.com/explicit> {
     ?Person <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?Person <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/0113/lumieres-lausanne#Person> .
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#isModern> ?isModern .
 GRAPH <http://www.ontotext.com/explicit> {
     ?isModern <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?isModern <http://www.knora.org/ontology/knora-base#valueHasBoolean> ?isModern_Value .
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#mayHaveBiography> ?mayHaveBiography .
 GRAPH <http://www.ontotext.com/explicit> {
     ?mayHaveBiography <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?mayHaveBiography <http://www.knora.org/ontology/knora-base#valueHasBoolean> ?mayHaveBiography_Value .
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#noticeIsCreated> ?noticeIsCreated .
 GRAPH <http://www.ontotext.com/explicit> {
     ?noticeIsCreated <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?noticeIsCreated <http://www.knora.org/ontology/knora-base#valueHasBoolean> ?noticeIsCreated_Value .
 {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#noticeCreationIsValid> ?noticeCreationIsValid .
 GRAPH <http://www.ontotext.com/explicit> {
     ?noticeCreationIsValid <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?noticeCreationIsValid <http://www.knora.org/ontology/knora-base#valueHasBoolean> ?noticeCreationIsValid_Value .
 FILTER((?noticeCreationIsValid_Value = "true"^^<http://www.w3.org/2001/XMLSchema#boolean>))
 } UNION {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#noticeModificationIsValid> ?noticeModificationIsValid .
 GRAPH <http://www.ontotext.com/explicit> {
     ?noticeModificationIsValid <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?noticeModificationIsValid <http://www.knora.org/ontology/knora-base#valueHasBoolean> ?noticeModificationIsValid_Value .
 FILTER((?noticeModificationIsValid_Value = "true"^^<http://www.w3.org/2001/XMLSchema#boolean>))
 }
 {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasName> ?hasName .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasName <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasName <http://www.knora.org/ontology/knora-base#valueHasString> ?hasName__valueHasString .
 }
 ?hasName <http://www.knora.org/ontology/knora-base#valueHasString> ?hasName_Value .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasName_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
 }
 } UNION {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasPlaceBirth> ?hasPlaceBirth .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasPlaceBirth <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?hasPlaceBirth <http://www.knora.org/ontology/knora-base#valueHasString> ?hasPlaceBirth_Value .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasPlaceBirth_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
 }
 } UNION {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasPlaceDeath> ?hasPlaceDeath .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasPlaceDeath <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?hasPlaceDeath <http://www.knora.org/ontology/knora-base#valueHasString> ?hasPlaceDeath_Value .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasPlaceDeath_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
 }
 } UNION {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasPlaceOrigin> ?hasPlaceOrigin .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasPlaceOrigin <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?hasPlaceOrigin <http://www.knora.org/ontology/knora-base#valueHasString> ?hasPlaceOrigin_Value .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasPlaceOrigin_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
 }
 } UNION {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasBiography> ?hasBiography .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasBiography <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?hasBiography <http://www.knora.org/ontology/knora-base#valueHasString> ?hasBiography_Value .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasBiography_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
 }
 } UNION {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasCivilStatus> ?hasCivilStatus .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasCivilStatus <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?hasCivilStatus <http://www.knora.org/ontology/knora-base#valueHasString> ?hasCivilStatus_Value .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasCivilStatus_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
 }
 } UNION {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#isInArchive> ?isInArchive .
 GRAPH <http://www.ontotext.com/explicit> {
     ?isInArchive <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?isInArchive <http://www.knora.org/ontology/knora-base#valueHasString> ?isInArchive_Value .
 GRAPH <http://www.ontotext.com/explicit> {
     ?isInArchive_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
 }
 } UNION {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasCommentWork> ?hasCommentWork .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasCommentWork <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?hasCommentWork <http://www.knora.org/ontology/knora-base#valueHasString> ?hasCommentWork_Value .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasCommentWork_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
 }
 } UNION {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasNote> ?hasNote .
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasNoteValue> ?Person__http00003333ontology0113lumiereslausannev2hasNote__hasNote__LinkValue .
 GRAPH <http://www.ontotext.com/explicit> {
     ?Person__http00003333ontology0113lumiereslausannev2hasNote__hasNote__LinkValue <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/knora-base#LinkValue> .
 }
 GRAPH <http://www.ontotext.com/explicit> {
     ?Person__http00003333ontology0113lumiereslausannev2hasNote__hasNote__LinkValue <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 GRAPH <http://www.ontotext.com/explicit> {
     ?Person__http00003333ontology0113lumiereslausannev2hasNote__hasNote__LinkValue <http://www.w3.org/1999/02/22-rdf-syntax-ns#subject> ?Person .
 }
 GRAPH <http://www.ontotext.com/explicit> {
     ?Person__http00003333ontology0113lumiereslausannev2hasNote__hasNote__LinkValue <http://www.w3.org/1999/02/22-rdf-syntax-ns#object> ?hasNote .
 }
 ?hasNote <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/knora-base#Resource> .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasNote <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?hasNote <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/0113/lumieres-lausanne#Note> .
 ?hasNote <http://www.knora.org/ontology/0113/lumieres-lausanne#hasDiffusionType> ?hasDiffusionType_hasNote .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasDiffusionType_hasNote <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?hasDiffusionType_hasNote <http://www.knora.org/ontology/knora-base#valueHasBoolean> ?hasDiffusionType_hasNote_Value .
 ?hasNote <http://www.knora.org/ontology/0113/lumieres-lausanne#noteHasText> ?noteHasText_hasNote .
 GRAPH <http://www.ontotext.com/explicit> {
     ?noteHasText_hasNote <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?noteHasText_hasNote <http://www.knora.org/ontology/knora-base#valueHasString> ?noteHasText_hasNote_Value .
 FILTER((?hasDiffusionType_hasNote_Value = "true"^^<http://www.w3.org/2001/XMLSchema#boolean>))
 GRAPH <http://www.ontotext.com/explicit> {
     ?noteHasText_hasNote_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
 }
 }
 FILTER((?isModern_Value = "false"^^<http://www.w3.org/2001/XMLSchema#boolean>))
 FILTER((?mayHaveBiography_Value = "true"^^<http://www.w3.org/2001/XMLSchema#boolean>))
 FILTER((?noticeIsCreated_Value = "true"^^<http://www.w3.org/2001/XMLSchema#boolean>))
 }
 GROUP BY ?Person ?hasName__valueHasString
 ORDER BY ASC(?hasName__valueHasString) ASC(?Person)
 LIMIT 25

With @loicjaouen we suspect the GROUP BY ?Person ?hasName__valueHasString.

We get a correct result with a gravsearch query without ORDER BY ASC(?hasName) or by running the previous sparql query with a simple GROUP BY ?Person wihtout ?hasName__valueHasString

@gfoo
Copy link
Author

gfoo commented Jan 31, 2020

Concerning the previous sparql's query, do you really need to get all these columns GROUP_CONCAT... as result? The ?Person column should be sufficient?

@tobiasschweizer
Copy link
Contributor

Concerning the previous sparql's query, do you really need to get all these columns GROUP_CONCAT... as result? The ?Person column should be sufficient?

Gravsearch generates two queries from a given input query: a prequery and a main query (see https://docs.knora.org/paradox/05-internals/design/api-v2/gravsearch.html#transformation-of-a-gravsearch-query). The prequery returns Iris of main resources, dependent resources and values that matched the criteria given in the WHERE clause.

Per matching main resource, one row is returned. Iris of dependent resources and values are concatenated because there could be multiple matches.

This is achieved by grouping by the main resource. If I remember correctly, additional search criteria has to be included in Group By because it would not be bound, thus having no effect.

We get a correct result with a gravsearch query without ORDER BY ASC(?hasName) or by running the previous sparql query with a simple GROUP BY ?Person wihtout ?hasName__valueHasString

Do you think the problem is the results returned by the triplestore or the processing of the results in Knora?

@loicjaouen
Copy link
Collaborator

I am not sure what the triplestore is supposed to return, but we ask for:

SELECT DISTINCT ?Person
...
 GROUP BY ?Person ?hasName__valueHasString

and we receive duplicated ?Person (because they don't have the same ?hasName__valueHasString).

If sparql or graphdb says that it is so (there is one index per group by and you can't ask for a distinct on one of the variables of the group by) then it is up to knora to sort it out.

(then knowing why we have two different ?hasName__valueHasString values is another question)

@tobiasschweizer
Copy link
Contributor

Does ?Person ll:hasName ?hasName . have a cardinality greater than 1?

If so, I can confirm that Gravsearch won't work correctly:

https://github.com/dasch-swiss/knora-api/blob/70b0841f2edcc232a63cafb5348f0deb3e100e19/webapi/src/main/scala/org/knora/webapi/responders/v2/SearchResponderV2.scala#L433-L434

@gfoo
Copy link
Author

gfoo commented Feb 3, 2020

Does ?Person ll:hasName ?hasName . have a cardinality greater than 1?

equals to 1:

                        [
                           rdf:type owl:Restriction ;
                           owl:onProperty lumieres-lausanne:hasName ;
                           salsah-gui:guiOrder "1"^^xsd:integer ; 
                           owl:cardinality "1"^^xsd:nonNegativeInteger                        ] , 

@tobiasschweizer
Copy link
Contributor

But then this should be impossible

(then knowing why we have two different ?hasName__valueHasString values is another question)

@tobiasschweizer
Copy link
Contributor

Could you check if there is more than one ll:hasName for a person that is returned twice?

@loicjaouen
Copy link
Collaborator

loicjaouen commented Feb 3, 2020

we can look at that in a second stage, right now what do you think of having a:

SELECT DISTINCT ?Person
...
 GROUP BY ?Person

giving the right answer and a:

SELECT DISTINCT ?Person
...
 GROUP BY ?Person ?hasName__valueHasString

giving duplicates?

@tobiasschweizer
Copy link
Contributor

The only reason I can think if this that ?hasName__valueHasString has more than one occurrence.

Could you make a full resource request for the person resource for which you are getting multiple results?

@tobiasschweizer
Copy link
Contributor

Also could you please check which Iris are returned for ?hasName__Concat when doing

SELECT DISTINCT ?Person
...
 GROUP BY ?Person

@loicjaouen
Copy link
Collaborator

So unfortunately the data is not online right now.

The above query returns 4 Person

the duplicated resource is:

<http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now> a <http://www.knora.org/ontology/0113/lumieres-lausanne#Person>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#hasBiography> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/21G24A8URuWzY-1zz0q4mg>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#hasCivilStatus> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/bmFRKGcqSY2PoloEDGrXuQ>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#hasCommentWork> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/hQfRkxfNTEup55vAdfCWpg>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#hasName> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/RdkuXYVYTKivFL0TTeOdkw>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#hasNationality> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/loQiLtFrTt61w2-baphPSA>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#hasNote> <http://rdfh.ch/0113/O-kFyTo9QUSBRBrxnff1qA>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#hasNoteValue> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/TmeoQSz2Tq6poGUm_Bq1Ng>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#hasOldId> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/lBXEKqZ-RxGkxrknIHrHRQ>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#hasPlaceBirth> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/OEANncjVQSST4Cx6mn2MZw>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#hasPlaceDeath> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/oPVHETQsSHicbAu6Qo35bg>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#hasReligion> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/4_PVxTl8TWqU1EOHYGqhIQ>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#isApproximateBeginDate> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/73qU-5cvSTmgz5rt7WvPlg>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#isApproximateEndDate> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/TurrzU77TaijSZVT8ycIwg>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#isInSociety> <http://rdfh.ch/0113/c-9eEIl6QeKX73KMQI8ukg>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#isInSocietyValue> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/wKk8VtbuRRGtis2LnwlrAQ>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#isModern> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/fFaNMnk0RkCc9ajVM8DMRQ>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#mayHaveBiography> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/zMCFI3waQ8iuRk7WPTq4nw>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#noticeCreationIsValid> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/4T8SB1NsRGWiCM0ewG85tA>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#noticeHasValidDate> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/-OE9D7NcQpWIoALuTSBbkA>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#noticeIsCreated> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/kFYszbW6RJ-LgKoxmKfBKA>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#noticeModificationIsValid> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/wX8cFYEUQN6Xjgg25tyeRg>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#personHasBirthDate> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/-IFphG6vSR-4AfAzQrmcTw>;
  <http://www.knora.org/ontology/0113/lumieres-lausanne#personHasDeathDate> <http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/skS4afH7TVW3hmKbKEIJdw>;
  <http://www.knora.org/ontology/knora-base#attachedToProject> <http://rdfh.ch/projects/0113>;
  <http://www.knora.org/ontology/knora-base#attachedToUser> <http://rdfh.ch/users/lumieres-lausanne-b_la_kapossy>;
  <http://www.knora.org/ontology/knora-base#creationDate> "2010-07-07T10:52:15Z"^^xsd:dateTime;
  <http://www.knora.org/ontology/knora-base#hasPermissions> "CR http://rdfh.ch/groups/0113/lumieres-lausanne-administrator,http://rdfh.ch/groups/0113/lumieres-lausanne-director|D knora-admin:Creator|M http://rdfh.ch/groups/0113/lumieres-lausanne-phdstudent,http://rdfh.ch/groups/0113/lumieres-lausanne-researcher,http://rdfh.ch/groups/0113/lumieres-lausanne-student|V http://rdfh.ch/groups/0113/lumieres-lausanne-user,knora-admin:KnownUser,knora-admin:UnknownUser";
  <http://www.knora.org/ontology/knora-base#isDeleted> false;
  rdfs:label "Mirabeau, Victor de Riqueti, marquis de" .

where there is only one hasName predicate

@loicjaouen
Copy link
Collaborator

loicjaouen commented Feb 4, 2020

the duplicated resource comes once with an empty hasName__Concat the second with not empty one: http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/RdkuXYVYTKivFL0TTeOdkw

that's with the request:

SELECT DISTINCT ?Person (GROUP_CONCAT(DISTINCT(?hasNote); SEPARATOR='') AS ?hasNote__Concat) ...
...
 GROUP BY ?Person ?hasName__valueHasString
...

@loicjaouen
Copy link
Collaborator

the same the request with only a

 GROUP BY ?Person 

returns no duplicate and a non-empty hasName___Concat equal to http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/RdkuXYVYTKivFL0TTeOdkw

@loicjaouen
Copy link
Collaborator

for the record, that value is:

<http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/RdkuXYVYTKivFL0TTeOdkw> a <http://www.knora.org/ontology/knora-base#TextValue>;
  <http://www.knora.org/ontology/knora-base#attachedToUser> <http://rdfh.ch/users/lumieres-lausanne-b_la_kapossy>;
  <http://www.knora.org/ontology/knora-base#hasPermissions> "CR http://rdfh.ch/groups/0113/lumieres-lausanne-administrator,http://rdfh.ch/groups/0113/lumieres-lausanne-director|D knora-admin:Creator|M http://rdfh.ch/groups/0113/lumieres-lausanne-phdstudent|V http://rdfh.ch/groups/0113/lumieres-lausanne-researcher,http://rdfh.ch/groups/0113/lumieres-lausanne-student,http://rdfh.ch/groups/0113/lumieres-lausanne-user,knora-admin:KnownUser,knora-admin:UnknownUser";
  <http://www.knora.org/ontology/knora-base#isDeleted> false;
  <http://www.knora.org/ontology/knora-base#valueCreationDate> "2010-07-07T10:52:15Z"^^xsd:dateTime;
  <http://www.knora.org/ontology/knora-base#valueHasOrder> 0;
  <http://www.knora.org/ontology/knora-base#valueHasString> "Mirabeau, Victor de Riqueti, marquis de";
  <http://www.knora.org/ontology/knora-base#valueHasUUID> "oBj_g61TQtetL741cVrLEQ" .

@loicjaouen
Copy link
Collaborator

we are getting closer now...

@loicjaouen
Copy link
Collaborator

loicjaouen commented Feb 4, 2020

if we execute this sparql request:

PREFIX knora-base: <http://www.knora.org/ontology/knora-base#>
select ?p ?o ?v
FROM <http://www.ontotext.com/explicit>
where { 
    bind(<http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now> as ?s)
	?s ?p ?o .
    ?o knora-base:valueHasString ?v .
    GRAPH <http://www.ontotext.com/explicit> {
     ?v <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
    }
}

we get three properties matching: lumieres-lausanne:hasName, lumieres-lausanne:hasBiography, lumieres-lausanne:hasCivilStatus

@tobiasschweizer
Copy link
Contributor

Could give me a trig file to download (all ontologies and data in one file) so I can reproduce it locally?

@tobiasschweizer
Copy link
Contributor

Today is my day off, I can try it later this week.

@tobiasschweizer
Copy link
Contributor

tobiasschweizer commented Feb 5, 2020

I have the suspicion that you are getting the duplicates because the sort criterion is contained in a UNION. Please try the following query:

PREFIX knora-api: <http://api.knora.org/ontology/knora-api/v2#>
PREFIX knora-simple-api: <http://api.knora.org/ontology/knora-api/simple/v2#>
PREFIX ll: <http://0.0.0.0:3333/ontology/0113/lumieres-lausanne/v2#>
CONSTRUCT {
  ?Person knora-api:isMainResource true .
} WHERE {
  
  ?Person a ll:Person .
  
  ?Person ll:isModern ?isModern .

?isModern knora-api:booleanValueAsBoolean ?isModern_Value .
    FILTER(?isModern_Value = "false"^^xsd:boolean)
?Person ll:mayHaveBiography ?mayHaveBiography .

?mayHaveBiography knora-api:booleanValueAsBoolean ?mayHaveBiography_Value .
    FILTER(?mayHaveBiography_Value = "true"^^xsd:boolean)
?Person ll:noticeIsCreated ?noticeIsCreated .

?noticeIsCreated knora-api:booleanValueAsBoolean ?noticeIsCreated_Value .
    FILTER(?noticeIsCreated_Value = "true"^^xsd:boolean)

    ?Person ll:hasName ?hasName .

    ?hasName knora-api:valueAsString ?hasName_Value .
        FILTER knora-api:match(?hasName_Value, "Mirab*")

  
{

{
?Person ll:noticeCreationIsValid ?noticeCreationIsValid .

?noticeCreationIsValid knora-api:booleanValueAsBoolean ?noticeCreationIsValid_Value .
    FILTER(?noticeCreationIsValid_Value = "true"^^xsd:boolean)
}
UNION
{
?Person ll:noticeModificationIsValid ?noticeModificationIsValid .

?noticeModificationIsValid knora-api:booleanValueAsBoolean ?noticeModificationIsValid_Value .
    FILTER(?noticeModificationIsValid_Value = "true"^^xsd:boolean)
}

}

{



{
?Person ll:hasPlaceBirth ?hasPlaceBirth .

?hasPlaceBirth knora-api:valueAsString ?hasPlaceBirth_Value .
    FILTER knora-api:match(?hasPlaceBirth_Value, "Mirab*")
}
UNION
{
?Person ll:hasPlaceDeath ?hasPlaceDeath .

?hasPlaceDeath knora-api:valueAsString ?hasPlaceDeath_Value .
    FILTER knora-api:match(?hasPlaceDeath_Value, "Mirab*")
}
UNION
{
?Person ll:hasPlaceOrigin ?hasPlaceOrigin .

?hasPlaceOrigin knora-api:valueAsString ?hasPlaceOrigin_Value .
    FILTER knora-api:match(?hasPlaceOrigin_Value, "Mirab*")
}
UNION
{
?Person ll:hasBiography ?hasBiography .

?hasBiography knora-api:valueAsString ?hasBiography_Value .
    FILTER knora-api:match(?hasBiography_Value, "Mirab*")
}
UNION
{
?Person ll:hasCivilStatus ?hasCivilStatus .

?hasCivilStatus knora-api:valueAsString ?hasCivilStatus_Value .
    FILTER knora-api:match(?hasCivilStatus_Value, "Mirab*")
}
UNION
{
?Person ll:isInArchive ?isInArchive .

?isInArchive knora-api:valueAsString ?isInArchive_Value .
    FILTER knora-api:match(?isInArchive_Value, "Mirab*")
}
UNION
{
?Person ll:hasCommentWork ?hasCommentWork .

?hasCommentWork knora-api:valueAsString ?hasCommentWork_Value .
    FILTER knora-api:match(?hasCommentWork_Value, "Mirab*")
}
UNION
{
?Person ll:hasNote ?hasNote .
?hasNote a ll:Note .

?hasNote ll:hasDiffusionType ?hasDiffusionType_hasNote .

?hasDiffusionType_hasNote knora-api:booleanValueAsBoolean ?hasDiffusionType_hasNote_Value .
    FILTER(?hasDiffusionType_hasNote_Value = "true"^^xsd:boolean)
?hasNote ll:noteHasText ?noteHasText_hasNote .

?noteHasText_hasNote knora-api:valueAsString ?noteHasText_hasNote_Value .
    FILTER knora-api:match(?noteHasText_hasNote_Value, "Mirab*")
}

}

}
ORDER BY ASC(?hasName)

I am getting one result ("http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now").

@tobiasschweizer
Copy link
Contributor

When I run the following query

SELECT DISTINCT ?Person ?hasName__valueHasString
 WHERE {
 ?Person <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/knora-base#Resource> .
 GRAPH <http://www.ontotext.com/explicit> {
     ?Person <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?Person <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/0113/lumieres-lausanne#Person> .
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#isModern> ?isModern .
 GRAPH <http://www.ontotext.com/explicit> {
     ?isModern <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?isModern <http://www.knora.org/ontology/knora-base#valueHasBoolean> ?isModern_Value .
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#mayHaveBiography> ?mayHaveBiography .
 GRAPH <http://www.ontotext.com/explicit> {
     ?mayHaveBiography <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?mayHaveBiography <http://www.knora.org/ontology/knora-base#valueHasBoolean> ?mayHaveBiography_Value .
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#noticeIsCreated> ?noticeIsCreated .
 GRAPH <http://www.ontotext.com/explicit> {
     ?noticeIsCreated <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?noticeIsCreated <http://www.knora.org/ontology/knora-base#valueHasBoolean> ?noticeIsCreated_Value .
 {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#noticeCreationIsValid> ?noticeCreationIsValid .
 GRAPH <http://www.ontotext.com/explicit> {
     ?noticeCreationIsValid <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?noticeCreationIsValid <http://www.knora.org/ontology/knora-base#valueHasBoolean> ?noticeCreationIsValid_Value .
 FILTER((?noticeCreationIsValid_Value = "true"^^<http://www.w3.org/2001/XMLSchema#boolean>))
 } UNION {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#noticeModificationIsValid> ?noticeModificationIsValid .
 GRAPH <http://www.ontotext.com/explicit> {
     ?noticeModificationIsValid <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?noticeModificationIsValid <http://www.knora.org/ontology/knora-base#valueHasBoolean> ?noticeModificationIsValid_Value .
 FILTER((?noticeModificationIsValid_Value = "true"^^<http://www.w3.org/2001/XMLSchema#boolean>))
 }
 {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasName> ?hasName .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasName <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasName <http://www.knora.org/ontology/knora-base#valueHasString> ?hasName__valueHasString .
 }
 ?hasName <http://www.knora.org/ontology/knora-base#valueHasString> ?hasName_Value .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasName_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
 }
 } UNION {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasPlaceBirth> ?hasPlaceBirth .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasPlaceBirth <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?hasPlaceBirth <http://www.knora.org/ontology/knora-base#valueHasString> ?hasPlaceBirth_Value .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasPlaceBirth_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
 }
 } UNION {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasPlaceDeath> ?hasPlaceDeath .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasPlaceDeath <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?hasPlaceDeath <http://www.knora.org/ontology/knora-base#valueHasString> ?hasPlaceDeath_Value .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasPlaceDeath_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
 }
 } UNION {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasPlaceOrigin> ?hasPlaceOrigin .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasPlaceOrigin <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?hasPlaceOrigin <http://www.knora.org/ontology/knora-base#valueHasString> ?hasPlaceOrigin_Value .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasPlaceOrigin_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
 }
 } UNION {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasBiography> ?hasBiography .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasBiography <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?hasBiography <http://www.knora.org/ontology/knora-base#valueHasString> ?hasBiography_Value .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasBiography_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
 }
 } UNION {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasCivilStatus> ?hasCivilStatus .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasCivilStatus <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?hasCivilStatus <http://www.knora.org/ontology/knora-base#valueHasString> ?hasCivilStatus_Value .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasCivilStatus_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
 }
 } UNION {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#isInArchive> ?isInArchive .
 GRAPH <http://www.ontotext.com/explicit> {
     ?isInArchive <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?isInArchive <http://www.knora.org/ontology/knora-base#valueHasString> ?isInArchive_Value .
 GRAPH <http://www.ontotext.com/explicit> {
     ?isInArchive_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
 }
 } UNION {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasCommentWork> ?hasCommentWork .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasCommentWork <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?hasCommentWork <http://www.knora.org/ontology/knora-base#valueHasString> ?hasCommentWork_Value .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasCommentWork_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
 }
 } UNION {
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasNote> ?hasNote .
 ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasNoteValue> ?Person__http00003333ontology0113lumiereslausannev2hasNote__hasNote__LinkValue .
 GRAPH <http://www.ontotext.com/explicit> {
     ?Person__http00003333ontology0113lumiereslausannev2hasNote__hasNote__LinkValue <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/knora-base#LinkValue> .
 }
 GRAPH <http://www.ontotext.com/explicit> {
     ?Person__http00003333ontology0113lumiereslausannev2hasNote__hasNote__LinkValue <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 GRAPH <http://www.ontotext.com/explicit> {
     ?Person__http00003333ontology0113lumiereslausannev2hasNote__hasNote__LinkValue <http://www.w3.org/1999/02/22-rdf-syntax-ns#subject> ?Person .
 }
 GRAPH <http://www.ontotext.com/explicit> {
     ?Person__http00003333ontology0113lumiereslausannev2hasNote__hasNote__LinkValue <http://www.w3.org/1999/02/22-rdf-syntax-ns#object> ?hasNote .
 }
 ?hasNote <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/knora-base#Resource> .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasNote <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?hasNote <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/0113/lumieres-lausanne#Note> .
 ?hasNote <http://www.knora.org/ontology/0113/lumieres-lausanne#hasDiffusionType> ?hasDiffusionType_hasNote .
 GRAPH <http://www.ontotext.com/explicit> {
     ?hasDiffusionType_hasNote <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?hasDiffusionType_hasNote <http://www.knora.org/ontology/knora-base#valueHasBoolean> ?hasDiffusionType_hasNote_Value .
 ?hasNote <http://www.knora.org/ontology/0113/lumieres-lausanne#noteHasText> ?noteHasText_hasNote .
 GRAPH <http://www.ontotext.com/explicit> {
     ?noteHasText_hasNote <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
 }
 ?noteHasText_hasNote <http://www.knora.org/ontology/knora-base#valueHasString> ?noteHasText_hasNote_Value .
 FILTER((?hasDiffusionType_hasNote_Value = "true"^^<http://www.w3.org/2001/XMLSchema#boolean>))
 GRAPH <http://www.ontotext.com/explicit> {
     ?noteHasText_hasNote_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
 }
 }
 FILTER((?isModern_Value = "false"^^<http://www.w3.org/2001/XMLSchema#boolean>))
 FILTER((?mayHaveBiography_Value = "true"^^<http://www.w3.org/2001/XMLSchema#boolean>))
 FILTER((?noticeIsCreated_Value = "true"^^<http://www.w3.org/2001/XMLSchema#boolean>))
 }
 
 LIMIT 25

I get this result:

Screen Shot 2020-02-05 at 7 01 15 PM

"http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now" is the only resource that has a matching "hasName", the others don't (thus returning nothing for ?hasName__valueHasString).

But I don't understand why this is also the case for "http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now", as if its UNION once matched and once did not. @benjamingeer Scoping issue with UNION?

@benjamingeer
Copy link

What I've read is that a UNION is actually a subquery, and can't see variables bound outside its scope:

https://wiki.blazegraph.com/wiki/index.php/SPARQL_Bottom_Up_Semantics#UNIONs

But sometimes it seems to work anyway, e.g. in the UNION blocks in our full resource query (getResourcePropertiesAndValuesGraphDB.scala.txt).

@benjamingeer
Copy link

If I had the data I could play with it and try to figure out what's going on.

@tobiasschweizer
Copy link
Contributor

I have the data 🙂 will provide it tomorrow

@tobiasschweizer
Copy link
Contributor

tobiasschweizer commented Feb 6, 2020

@benjamingeer I shared the trig file with you on switchdrive

How I loaded it into my local graphDB:

  • unpack the file :-)
  • move it to the server files dir /Users/username/graphdb-import.
  • deactivate consistency checking (owlim:check-for-inconsistencies "false"; and I removed owlim:ruleset)
  • import it (takes about 10 minutes)
  • update Lucene index scripts/graphdb-knora-index-update.rq

@benjamingeer
Copy link

Why do I have to deactivate consistency checking? Is the issue caused by inconsistent data?

@tobiasschweizer
Copy link
Contributor

once you unpack the file, you'll see that it is very big (1.91 GB). I figured it would take too much time with consistency checking.

And yes, I spotted a problem with an empty integer value.

@loicjaouen
Copy link
Collaborator

so each union would have to ask for the common set of triplets otherwise we build different sub-graph for matching criteria?

a simplified version of the request on two properties, name and biography, we have:

SELECT DISTINCT ?Person ?hasName__valueHasString ?hasName ?hasName_Value
 WHERE {
     ?Person <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/knora-base#Resource> .

    {
        ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasName> ?hasName .
        ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasBiography> ?hasBiography .
        ?hasBiography <http://www.knora.org/ontology/knora-base#valueHasString> ?hasBiography_Value .
        GRAPH <http://www.ontotext.com/explicit> {
            ?hasName <http://www.knora.org/ontology/knora-base#valueHasString> ?hasName__valueHasString .
        }
        ?hasName <http://www.knora.org/ontology/knora-base#valueHasString> ?hasName_Value .
        GRAPH <http://www.ontotext.com/explicit> {
                ?hasName_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
            }
    } UNION {

        ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasName> ?hasName .
            ?hasName <http://www.knora.org/ontology/knora-base#valueHasString> ?hasName__valueHasString .
        ?hasName <http://www.knora.org/ontology/knora-base#valueHasString> ?hasName_Value .
        ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasBiography> ?hasBiography .
        GRAPH <http://www.ontotext.com/explicit> {
            ?hasBiography <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
        }
        ?hasBiography <http://www.knora.org/ontology/knora-base#valueHasString> ?hasBiography_Value .
        GRAPH <http://www.ontotext.com/explicit> {
            ?hasBiography_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
        }

    }
 }
 
 LIMIT 25

returning no duplicate

SELECT DISTINCT ?Person ?hasName__valueHasString ?hasName ?hasName_Value
 WHERE {
     ?Person <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/knora-base#Resource> .

    {
        ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasName> ?hasName .
        GRAPH <http://www.ontotext.com/explicit> {
            ?hasName <http://www.knora.org/ontology/knora-base#valueHasString> ?hasName__valueHasString .
        }
        ?hasName <http://www.knora.org/ontology/knora-base#valueHasString> ?hasName_Value .
        GRAPH <http://www.ontotext.com/explicit> {
                ?hasName_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
            }
    } UNION {

        ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasBiography> ?hasBiography .
        GRAPH <http://www.ontotext.com/explicit> {
            ?hasBiography <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
        }
        ?hasBiography <http://www.knora.org/ontology/knora-base#valueHasString> ?hasBiography_Value .
        GRAPH <http://www.ontotext.com/explicit> {
            ?hasBiography_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
        }

    }
 }
 
 LIMIT 25

returns duplicates and empty fields.

@benjamingeer
Copy link

@loicjaouen

so each union would have to ask for the common set of triplets otherwise we build different sub-graph for matching criteria?

I think so.

Could we have a data file just containing the data needed to reproduce this issue?

@loicjaouen
Copy link
Collaborator

it is a union of two subgraphs, the first one matches the name, the second the biography, if we set the both properties on both subgraphs, the union does it works and otherwise, the name is understood as a new value (empty) and adds a duplicated line for the same resource.

@loicjaouen
Copy link
Collaborator

loicjaouen commented Feb 6, 2020

maybe the problem is that we do a select and not a construct
(we then make a union on sets graphs that are not our original graph)

@benjamingeer
Copy link

maybe the problem is that we do a select and not a construct

I don't understand. I think a WHERE clause does the same thing whether it's in a SELECT or a CONSTRUCT.

@tobiasschweizer
Copy link
Contributor

Wouldn't it make sense to only use a non-optional property as a sort criterion?

@loicjaouen
Copy link
Collaborator

loicjaouen commented Feb 6, 2020

@benjamingeer did you get the data?

it makes a difference because select adds two paths of multiple lengths and construct adds triples and it fills the missing elements of the path with empty values.

the query:

SELECT DISTINCT ?Person ?hasName__valueHasString ?hasName ?hasName_Value
 WHERE {
     ?Person <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/knora-base#Resource> .

    {
        ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasName> ?hasName .
        GRAPH <http://www.ontotext.com/explicit> {
            ?hasName <http://www.knora.org/ontology/knora-base#valueHasString> ?hasName__valueHasString .
        }
        ?hasName <http://www.knora.org/ontology/knora-base#valueHasString> ?hasName_Value .
        GRAPH <http://www.ontotext.com/explicit> {
                ?hasName_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
            }
    } UNION {

        ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasBiography> ?hasBiography .
        GRAPH <http://www.ontotext.com/explicit> {
            ?hasBiography <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
        }
        ?hasBiography <http://www.knora.org/ontology/knora-base#valueHasString> ?hasBiography_Value .
        GRAPH <http://www.ontotext.com/explicit> {
            ?hasBiography_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
        }

    }
 }
 
 LIMIT 25
Person hasName__valueHasString hasName hasName_Value
http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now "Mirabeau Victor de Riqueti marquis de"
http://rdfh.ch/0113/9I_ISTYERz2eZIaegBX92Q "Mirabeau, Jean Antoine de Riqueti, chevalier de (1717-1794)" http://rdfh.ch/0113/9I_ISTYERz2eZIaegBX92Q/values/qjDSxsUiR4WopxCjNMnVeQ "Mirabeau, Jean Antoine de Riqueti, chevalier de (1717-1794)"
http://rdfh.ch/0113/Rfp7CBF0R0eOrBBRw3BoIQ "Mirabeau [-Castellane], Françoise de" http://rdfh.ch/0113/Rfp7CBF0R0eOrBBRw3BoIQ/values/hqnMU4-VS9aFbfT_g9jLxg "Mirabeau [-Castellane], Françoise de"
http://rdfh.ch/0113/mjslMOQ8QYCAOzS_oFXrwg "Mirabeau, Honoré Gabriel de Riqueti, comte de (1749-1791)" http://rdfh.ch/0113/mjslMOQ8QYCAOzS_oFXrwg/values/MYygv1_nQzyrEORni1Vs8g "Mirabeau, Honoré Gabriel de Riqueti, comte de (1749-1791)"
http://rdfh.ch/0113/wHll56bbSjCx11-LOb_qFw "Cabris [-Mirabeau], Marie Louise Elisabeth, marquise de (1752-1807)" http://rdfh.ch/0113/wHll56bbSjCx11-LOb_qFw/values/MgfjKKwISk-ReHLDrWP6qw "Cabris [-Mirabeau], Marie Louise Elisabeth, marquise de (1752-1807)"
http://rdfh.ch/0113/ceLZPIZDRF2ROHJ4SW5D9A
http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now
http://rdfh.ch/0113/TEk7jawGT5eck1JAL3WFcQ

@loicjaouen
Copy link
Collaborator

loicjaouen commented Feb 6, 2020

when filling the same properties on the two subgraphs of the union, we get:

SELECT DISTINCT ?Person ?hasName__valueHasString ?hasName ?hasName_Value
 WHERE {
     ?Person <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.knora.org/ontology/knora-base#Resource> .

    {
        ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasName> ?hasName .
        ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasBiography> ?hasBiography .
        ?hasBiography <http://www.knora.org/ontology/knora-base#valueHasString> ?hasBiography_Value .
        GRAPH <http://www.ontotext.com/explicit> {
            ?hasName <http://www.knora.org/ontology/knora-base#valueHasString> ?hasName__valueHasString .
        }
        ?hasName <http://www.knora.org/ontology/knora-base#valueHasString> ?hasName_Value .
        GRAPH <http://www.ontotext.com/explicit> {
                ?hasName_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
            }
    } UNION {

        ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasName> ?hasName .
            ?hasName <http://www.knora.org/ontology/knora-base#valueHasString> ?hasName__valueHasString .
        ?hasName <http://www.knora.org/ontology/knora-base#valueHasString> ?hasName_Value .
        ?Person <http://www.knora.org/ontology/0113/lumieres-lausanne#hasBiography> ?hasBiography .
        GRAPH <http://www.ontotext.com/explicit> {
            ?hasBiography <http://www.knora.org/ontology/knora-base#isDeleted> "false"^^<http://www.w3.org/2001/XMLSchema#boolean> .
        }
        ?hasBiography <http://www.knora.org/ontology/knora-base#valueHasString> ?hasBiography_Value .
        GRAPH <http://www.ontotext.com/explicit> {
            ?hasBiography_Value <http://www.ontotext.com/owlim/lucene#fullTextSearchIndex> "Mirab*"^^<http://www.w3.org/2001/XMLSchema#string> .
        }

    }
 }
 
 LIMIT 25
Person hasName__valueHasString hasName hasName_Value
http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now "Mirabeau, Victor de Riqueti, marquis de" http://rdfh.ch/0113/MKE_5sRNTy6N2H0JTV3Now/values/RdkuXYVYTKivFL0TTeOdkw "Mirabeau, Victor de Riqueti, marquis de"
http://rdfh.ch/0113/ceLZPIZDRF2ROHJ4SW5D9A "Constant, Benjamin" http://rdfh.ch/0113/ceLZPIZDRF2ROHJ4SW5D9A/values/1IAbjruyTVaaAaTR4zzbcA "Constant, Benjamin"
http://rdfh.ch/0113/TEk7jawGT5eck1JAL3WFcQ "Sacconay, Frédéric de" http://rdfh.ch/0113/TEk7jawGT5eck1JAL3WFcQ/values/ZGGhIPSJQNqgdZknzSGK8w "Sacconay, Frédéric de"

@benjamingeer
Copy link

@loicjaouen

@benjamingeer did you get the data?

Tobias said you gave him a 2 GB file and he had to turn off consistency checking to load it in a reasonable amount of time. I would like to be able to reproduce this issue with consistency checking turned on, in case the problem is actually caused by inconsistent data. Therefore I'm asking for a smaller data file, containing just the resources that you're searching for.

it makes a difference because select adds two paths of multiple lengths and construct adds triples and it fills the missing elements of the path with empty values.

I'm not sure I understand what you mean by path.

In both SELECT and CONSTRUCT, the WHERE clause can return solutions with unbound variables in a given result. The difference is in what happens to these solutions afterwards. In the results of a SELECT, unbound variables appear as empty cells. In the results of a CONSTRUCT, triple patterns containing unbound variables are not returned.

Without having your data, I can guess that in your first query above, the first branch of the UNION gets a resource with just its name, and the second branch gets the same resource with just its biography. These two solutions are different, so DISTINCT doesn't unify them. In the second query, both branches of the UNION get both the name and the biography, so the solutions have the same values, so DISTINCT unifies them. But would need the data to make sure.

@loicjaouen
Copy link
Collaborator

@benjamingeer

In the results of a SELECT, unbound variables appear as empty cells. In the results of a CONSTRUCT, triple patterns containing unbound variables are not returned.
[...]
the first branch of the UNION gets a resource with just its name, and the second branch gets the same resource with just its biography. These two solutions are different, so DISTINCT doesn't unify them. In the second query, both branches of the UNION get both the name and the biography, so the solutions have the same values, so DISTINCT unifies them.

exactly, that's what I meant

@benjamingeer
Copy link

So what should we do about this? Should we just add something about it to the documentation?

@loicjaouen
Copy link
Collaborator

loicjaouen commented Feb 6, 2020

Thinning out existing data in a 2G set is messy.
So figuring out the problem on that data set to eventually make up a simpler data set replicating the issue made more sense to me.
But if the issue is logically explained, there might be no sense in replicating it because it is just a school case of a sparql behavior.

What we do about it...

On this request, we have skipped the gravsearch to look directly at the sparql request, we will work out our gravsearch request to see if we get to the a working generated sparql (which we should).

The comment to add would be more something like "gravsearch generates sparql, make sure you understand sparql logic prior to using gravsearch", or you might rewrite sparql documentation in the gravsearch documentation.

For this specific case, I think there is a bit more to add because it is still strange to send a request and get different result whether we ask for the results or ask for a /count.

@benjamingeer
Copy link

"gravsearch generates sparql, make sure you understand sparql logic prior to using gravsearch"

That problem seems to happen again and again. There are already books about SPARQL, but maybe we could add a section mentioning common misunderstandings, surprising aspects of SPARQL, things like that.

I think there is a bit more to add because it is still strange to send a request and get different result whether we ask for the results or ask for a /count.

Is there a way we could change the generated SPARQL so they'd return the same result?

@benjamingeer
Copy link

benjamingeer commented Feb 6, 2020

We need the GROUP_CONCAT because the SELECT has to return only one row per main resource, to make paging work using SPARQL's OFFSET and LIMIT.

For now, can you change your Gravsearch query so that it binds the necessary variables in the UNION?

For a longer-term solution, I think this needs a bit more thought and will take some time. Maybe we could actually analyse the query and reject it if the same variables aren't bound in each branch of the UNION.

@loicjaouen
Copy link
Collaborator

loicjaouen commented Feb 6, 2020

sorry, I deleted my comment, it was something like:

Is there a way we could change the generated SPARQL so they'd return the same result?

in the select statement, by not adding the variables that end up as unbound for some of the union subgraphs:

- SELECT DISTINCT ?Person (GROUP_CONCAT(DISTINCT(?hasNote); SEPARATOR='')
+ SELECT DISTINCT ?Person

@gfoo
Copy link
Author

gfoo commented Feb 6, 2020

For a longer-term solution

don't forget this probably related issue #1575

@benjamingeer
Copy link

in the select statement, by not adding the variables that end up as unbound for some of the union subgraphs:

That's an interesting idea, definitely worth thinking about.

@benjamingeer
Copy link

For now can you change your Gravsearch query so it includes the additional criteria and works the way you want?

@gfoo
Copy link
Author

gfoo commented Feb 7, 2020

I don't think so right now. I will probably disable the ORDER BY

@subotic subotic added this to the Backlog milestone Feb 7, 2020
@loicjaouen loicjaouen linked a pull request Mar 10, 2020 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants