Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(api-v2): Add an RDF processing façade (DSP-1020) #1754

Merged
merged 36 commits into from Nov 17, 2020
Merged

feat(api-v2): Add an RDF processing façade (DSP-1020) #1754

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
533976c
feat: Start adding RDF API and Jena implementation.
Nov 6, 2020
32ac622
Merge branch 'main' into wip/DSP-1020-rdf-api
Nov 6, 2020
fb7b8c9
feat(rdf-api): Add RDF4J implementation.
Nov 9, 2020
34a4ad6
test(rdf-api): Start adding tests.
Nov 9, 2020
ae92ab5
test(rdf-api): Add tests.
Nov 9, 2020
648ffe1
Merge branch 'main' into wip/DSP-1020-rdf-api
Nov 9, 2020
472126f
style(rdf-api): Fix filename.
Nov 9, 2020
325e218
feat(api-v2): Add façade for RDF parsing/formatting (doesn't even com…
Nov 10, 2020
52f8427
fix: Make everything compile again.
Nov 11, 2020
e39e3ce
fix(rdf-api): Add tests, fix bugs.
Nov 11, 2020
f7c8b68
feat(store): Use RDF API in HttpTriplestoreConnector, cause lots of c…
Nov 11, 2020
d77cb8c
feat: Fix a lot of compile errors (many more to go)
Nov 12, 2020
e1bf122
fix: Fix many compile errors, many more to go.
Nov 12, 2020
1b8570d
test: Make tests compile again.
Nov 13, 2020
7d00c70
fix: Fix bugs.
Nov 13, 2020
7c29912
test(FeatureToggleR2RSpec): Fix test.
Nov 13, 2020
0f02ffe
test: Fix tests.
Nov 13, 2020
79edf9a
tests: Update E2E and R2R tests.
Nov 13, 2020
2e070fb
Merge branch 'main' into wip/DSP-1020-rdf-api
Nov 13, 2020
dbd8182
test: Fix tests.
Nov 13, 2020
949f0bd
fix(rdf-api): Fix bugs.
Nov 13, 2020
8887872
docs(rdf-api): Add design doc.
Nov 13, 2020
5fa52e3
test(rdf-api): Add test for prefixes and custom datatypes.
Nov 13, 2020
6fdb70e
refactor(rdf-api): Simplify code.
Nov 14, 2020
f5a2f5c
refactor(JsonLDUtil): Simplify and clarify code.
Nov 14, 2020
3eb9b22
Merge branch 'main' into wip/DSP-1020-rdf-api
Nov 16, 2020
16a81d1
chore(build): Clean up dependencies.
Nov 16, 2020
0187e96
style(RdfFormatUtil): Clarify code.
Nov 16, 2020
a0ea4a0
refactor(rdf-api): Always use the singleton instances of the node fac…
Nov 16, 2020
c200281
style(rdf-api): Fix typo in method name.
Nov 16, 2020
e6ff4ab
style(JsonLDUtil): Clarify comments.
Nov 17, 2020
52aa1c3
style(JsonLDUtil): Clarify method names and comments.
Nov 17, 2020
624c301
test(JsonLDUtil): Add test for circular reference.
Nov 17, 2020
5cc9534
test: Optimise imports.
Nov 17, 2020
6a4af8c
style(JsonLDUtil): Add comments.
Nov 17, 2020
263efa7
style(test): Rearrange code.
Nov 17, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
22 changes: 13 additions & 9 deletions webapi/src/main/scala/org/knora/webapi/messages/util/rdf/JsonLDUtil.scala
View file
Open in desktop
Expand Up @@ -1368,7 +1368,7 @@ object JsonLDUtil {
val objs: Vector[JsonLDValue] = predStatements.map(_.obj).map {
case resource: RdfResource =>
// The object is an entity. Recurse to get it and inline it here.
referencedEntityToJsonLDValue(
referencedRdfResourceToJsonLDValue(
resource = resource,
model = model,
topLevelEntities = topLevelEntities,
Expand Down Expand Up @@ -1431,20 +1431,23 @@ object JsonLDUtil {
/**
* Given an [[RdfResource]] that is referred to by another entity, make a [[JsonLDValue]] to
* represent the referenced resource. This will be either a complete entity for nesting, or just
* the referenced entity's IRI.
* the referenced resource's IRI.
*
* @param resource the resource to be converted.
* @param model the [[RdfModel]] that is being read.
* @param topLevelEntities the top-level entities that have been constructed so far.
* @param processedSubjects the subjects that have already been processed.
* @return a JSON-LD value representing the resource.
*/
private def referencedEntityToJsonLDValue(resource: RdfResource,
model: RdfModel,
topLevelEntities: collection.mutable.Map[RdfResource, JsonLDObject],
processedSubjects: collection.mutable.Set[RdfResource])
(implicit stringFormatter: StringFormatter): JsonLDValue = {
// Have we already made a top-level JSON-LD object for this entity?
private def referencedRdfResourceToJsonLDValue(resource: RdfResource,
model: RdfModel,
topLevelEntities: collection.mutable.Map[RdfResource, JsonLDObject],
processedSubjects: collection.mutable.Set[RdfResource])
(implicit stringFormatter: StringFormatter): JsonLDValue = {
// How we deal with circular references: the referenced resource is not yet in topLevelEntities, but it
// is already marked as processed. Therefore we will return its IRI rather than inlining it.

// Is this entity already in topLevelEntities?
topLevelEntities.get(resource) match {
case Some(jsonLDObject) =>
// Yes. Remove it from the top level so it can be inlined.
Expand All @@ -1459,9 +1462,10 @@ object JsonLDUtil {
iriToJsonLDObject(iriNode.iri)

case _ =>
// It's not a Knora ontology entity. Is it in the model, and have we not processed it yet?
// It's not a Knora ontology entity. See if it's in the model.
val resourceStatements: Set[Statement] = model.find(Some(resource), None, None)

// Is it in the model and not yet marked as processed?
if (resourceStatements.nonEmpty && !processedSubjects.contains(resource)) {
// Yes. Recurse to get it so it can be inlined.
entityToJsonLDObject(
Expand Down
71 changes: 61 additions & 10 deletions webapi/src/test/scala/org/knora/webapi/messages/util/rdf/JsonLDUtilSpec.scala
View file
Open in desktop
Expand Up @@ -24,6 +24,7 @@ import java.io.File
import org.knora.webapi.CoreSpec
import org.knora.webapi.feature._
import org.knora.webapi.messages.StringFormatter
import org.knora.webapi.messages.util.MessageUtil
import org.knora.webapi.messages.util.rdf._
import org.knora.webapi.util.FileUtil
import spray.json.{JsValue, JsonParser}
Expand All @@ -44,7 +45,7 @@ abstract class JsonLDUtilSpec(featureToggle: FeatureToggle) extends CoreSpec {

"The JSON-LD tool" should {
"parse JSON-LD text, compact it with an empty context, convert the result to a JsonLDDocument, and convert that back to text" in {
val inputStr =
val ontologyJsonLDInputStr =
"""
|{
| "knora-api:hasOntologies" : {
Expand Down Expand Up @@ -86,7 +87,7 @@ abstract class JsonLDUtilSpec(featureToggle: FeatureToggle) extends CoreSpec {
|}
""".stripMargin

val expectedOutputStr =
val ontologyCompactedJsonLDOutputStr =
"""
|{
| "http://api.knora.org/ontology/knora-api/v2#hasOntologies" : {
Expand Down Expand Up @@ -120,10 +121,10 @@ abstract class JsonLDUtilSpec(featureToggle: FeatureToggle) extends CoreSpec {
|}
""".stripMargin

val compactedJsonLDDoc: JsonLDDocument = JsonLDUtil.parseJsonLD(inputStr)
val compactedJsonLDDoc: JsonLDDocument = JsonLDUtil.parseJsonLD(ontologyJsonLDInputStr)
val formattedCompactedDoc = compactedJsonLDDoc.toPrettyString
val receivedOutputAsJsValue: JsValue = JsonParser(formattedCompactedDoc)
val expectedOutputAsJsValue: JsValue = JsonParser(expectedOutputStr)
val expectedOutputAsJsValue: JsValue = JsonParser(ontologyCompactedJsonLDOutputStr)
receivedOutputAsJsValue should ===(expectedOutputAsJsValue)
}

Expand All @@ -141,7 +142,7 @@ abstract class JsonLDUtilSpec(featureToggle: FeatureToggle) extends CoreSpec {
val expectedTurtle: String = FileUtil.readTextFile(new File("test_data/ontologyR2RV2/anythingOntologyWithValueObjects.ttl"))

// Parse the Turtle to an RDF4J Model.
val expectedModel: RdfModel = rdfFormatUtil.parseToRdfModel(expectedTurtle, Turtle)
val expectedModel: RdfModel = rdfFormatUtil.parseToRdfModel(rdfStr = expectedTurtle, rdfFormat = Turtle)

// Compare the parsed Turtle with the model generated by the JsonLDDocument.
outputModel should ===(expectedModel)
Expand All @@ -152,7 +153,7 @@ abstract class JsonLDUtilSpec(featureToggle: FeatureToggle) extends CoreSpec {
val turtle = FileUtil.readTextFile(new File("test_data/ontologyR2RV2/anythingOntologyWithValueObjects.ttl"))

// Parse it to an RDF4J Model.
val inputModel: RdfModel = rdfFormatUtil.parseToRdfModel(turtle, Turtle)
val inputModel: RdfModel = rdfFormatUtil.parseToRdfModel(rdfStr = turtle, rdfFormat = Turtle)

// Convert the model to a JsonLDDocument.
val outputJsonLD: JsonLDDocument = JsonLDUtil.fromRdfModel(inputModel)
Expand All @@ -167,7 +168,7 @@ abstract class JsonLDUtilSpec(featureToggle: FeatureToggle) extends CoreSpec {
val expectedJsonLD = FileUtil.readTextFile(new File("test_data/ontologyR2RV2/anythingOntologyWithValueObjects.jsonld"))

// Parse it to an RDF4J Model.
val jsonLDExpectedModel: RdfModel = rdfFormatUtil.parseToRdfModel(expectedJsonLD, JsonLD)
val jsonLDExpectedModel: RdfModel = rdfFormatUtil.parseToRdfModel(rdfStr = expectedJsonLD, rdfFormat = JsonLD)

// Compare that with the model generated by the JsonLDDocument.
jsonLDOutputModel should ===(jsonLDExpectedModel)
Expand All @@ -193,7 +194,7 @@ abstract class JsonLDUtilSpec(featureToggle: FeatureToggle) extends CoreSpec {
val expectedTurtle = FileUtil.readTextFile(new File("test_data/resourcesR2RV2/BookReiseInsHeiligeLand.ttl"))

// Parse it to an RDF4J Model.
val expectedModel: RdfModel = rdfFormatUtil.parseToRdfModel(expectedTurtle, Turtle)
val expectedModel: RdfModel = rdfFormatUtil.parseToRdfModel(rdfStr = expectedTurtle, rdfFormat = Turtle)

// Compare that with the model generated by the JsonLDDocument.
outputModel should ===(expectedModel)
Expand All @@ -204,7 +205,7 @@ abstract class JsonLDUtilSpec(featureToggle: FeatureToggle) extends CoreSpec {
val turtle = FileUtil.readTextFile(new File("test_data/resourcesR2RV2/BookReiseInsHeiligeLand.ttl"))

// Parse it to an RDF4J Model.
val inputModel: RdfModel = rdfFormatUtil.parseToRdfModel(turtle, Turtle)
val inputModel: RdfModel = rdfFormatUtil.parseToRdfModel(rdfStr = turtle, rdfFormat = Turtle)

// Convert the model to a JsonLDDocument.
val outputJsonLD: JsonLDDocument = JsonLDUtil.fromRdfModel(inputModel)
Expand All @@ -223,10 +224,60 @@ abstract class JsonLDUtilSpec(featureToggle: FeatureToggle) extends CoreSpec {
expectedJsonLDDocument.body should ===(outputJsonLD.body)

// Parse the same file to an RDF4J Model.
val jsonLDExpectedModel: RdfModel = rdfFormatUtil.parseToRdfModel(expectedJsonLD, JsonLD)
val jsonLDExpectedModel: RdfModel = rdfFormatUtil.parseToRdfModel(rdfStr = expectedJsonLD, rdfFormat = JsonLD)

// Compare that with the model generated by the JsonLDDocument.
jsonLDOutputModel should ===(jsonLDExpectedModel)
}

"correctly convert an RDF model to JSON-LD if it contains a circular reference" in {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent!

// A Turtle document with a circular reference.
val turtle =
"""@prefix foo: <http://example.org/foo#> .
|@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
|
|<http://rdfh.ch/foo1> a foo:Foo;
| rdfs:label "foo 1";
| foo:hasOtherFoo <http://rdfh.ch/foo2>.
|
|<http://rdfh.ch/foo2> a foo:Foo;
| rdfs:label "foo 2";
| foo:hasOtherFoo <http://rdfh.ch/foo1>.
|""".stripMargin

// Parse it to an RDF4J Model.
val inputModel: RdfModel = rdfFormatUtil.parseToRdfModel(rdfStr = turtle, rdfFormat = Turtle)

// Convert the model to a JsonLDDocument.
val outputJsonLD: JsonLDDocument = JsonLDUtil.fromRdfModel(inputModel)

// The output could be nested in two different ways:

val expectedWithFoo1AtTopLevel = JsonLDObject(value = Map(
"@id" -> JsonLDString(value = "http://rdfh.ch/foo1"),
"@type" -> JsonLDString(value = "http://example.org/foo#Foo"),
"http://www.w3.org/2000/01/rdf-schema#label" -> JsonLDString(value = "foo 1"),
"http://example.org/foo#hasOtherFoo" -> JsonLDObject(value = Map(
"@id" -> JsonLDString(value = "http://rdfh.ch/foo2"),
"@type" -> JsonLDString(value = "http://example.org/foo#Foo"),
"http://www.w3.org/2000/01/rdf-schema#label" -> JsonLDString(value = "foo 2"),
"http://example.org/foo#hasOtherFoo" -> JsonLDObject(value = Map("@id" -> JsonLDString(value = "http://rdfh.ch/foo1"))),
)),
))

val expectedWithFoo2AtTopLevel = JsonLDObject(value = Map(
"@id" -> JsonLDString(value = "http://rdfh.ch/foo2"),
"@type" -> JsonLDString(value = "http://example.org/foo#Foo"),
"http://www.w3.org/2000/01/rdf-schema#label" -> JsonLDString(value = "foo 2"),
"http://example.org/foo#hasOtherFoo" -> JsonLDObject(value = Map(
"@id" -> JsonLDString(value = "http://rdfh.ch/foo1"),
"@type" -> JsonLDString(value = "http://example.org/foo#Foo"),
"http://www.w3.org/2000/01/rdf-schema#label" -> JsonLDString(value = "foo 1"),
"http://example.org/foo#hasOtherFoo" -> JsonLDObject(value = Map("@id" -> JsonLDString(value = "http://rdfh.ch/foo2")))
))
))

assert(outputJsonLD.body == expectedWithFoo2AtTopLevel || outputJsonLD.body == expectedWithFoo1AtTopLevel)
}
}
}
2 changes: 1 addition & 1 deletion webapi/src/test/scala/org/knora/webapi/messages/util/rdf/jenaimpl/BUILD.bazel
View file
Open in desktop
Expand Up @@ -59,7 +59,7 @@ scala_test(
"//webapi:main_library",
"//webapi:test_library",
"@maven//:org_apache_jena_apache_jena_libs"
] + BASE_TEST_DEPENDENCIES_WITH_JSON,
] + BASE_TEST_DEPENDENCIES_WITH_JSON_LD,
)

scala_test(
Expand Down
2 changes: 1 addition & 1 deletion webapi/src/test/scala/org/knora/webapi/messages/util/rdf/rdf4jimpl/BUILD.bazel
View file
Open in desktop
Expand Up @@ -59,7 +59,7 @@ scala_test(
"//webapi:main_library",
"//webapi:test_library",
"@maven//:org_apache_jena_apache_jena_libs"
] + BASE_TEST_DEPENDENCIES_WITH_JSON,
] + BASE_TEST_DEPENDENCIES_WITH_JSON_LD,
)

scala_test(
Expand Down