enhancement(triplestore): Use N-Quads instead of TriG for repository …

…upgrade (DSP-1129) (#1767)

* enhancement(triplestore): Use N-Quads instead of TriG for repository upgrade.

* fix(store): Fix typo.

* style(test): Simplify code.

* fix(upgrade): Run repository update plugins in the correct order.

* refactor(upgrade): Simplify code.

* test(rdf-api): Add tests.

* refactor(rdf-api): Simplify the closing of streams.

* fix(json-ld): Reject input that results in an empty blank node.

* fix(json-ld): Accept empty blank nodes, but process them correctly.

* style(test): Optimise imports.

docs/05-internals/development/updating-repositories.md

@@ -49,14 +49,14 @@ it to `org.knora.webapi.store.triplestore.upgrade.RepositoryUpdater`.
 2. Consult `org.knora.webapi.store.triplestore.upgrade.RepositoryUpdatePlan` to see which
    transformations are needed.
 
-3. Download the entire repository from the triplestore into a TriG file.
+3. Download the entire repository from the triplestore into an N-Quads file.
 
-4. Read the TriG file into an `RdfModel`.
+4. Read the N-Quads file into an `RdfModel`.
 
 5. Update the `RdfModel` by running the necessary transformations, and replacing the
    built-in Knora ontologies with the current ones.
 
-6. Save the `RdfModel` to a new TriG file.
+6. Save the `RdfModel` to a new N-Quads file.
 
 7. Empty the repository in the triplestore.
 
@@ -115,6 +115,6 @@ has been carried out. If the requirement is not met, the plugin can throw
 ## Testing Update Plugins
 
 Each plugin should have a unit test that extends `UpgradePluginSpec`. A typical
-test loads a TriG file containing test data into a `RdfModel`, runs the plugin,
+test loads a file containing RDF test data into a `RdfModel`, runs the plugin,
 makes an `RdfRepository` containing the transformed `RdfModel`, and uses
 SPARQL to check the result.

webapi/src/main/scala/org/knora/webapi/RdfMediaTypes.scala

@@ -55,6 +55,13 @@ object RdfMediaTypes {
         fileExtensions = List("trig")
     )
 
+    val `application/n-quads`: MediaType.WithFixedCharset = MediaType.customWithFixedCharset(
+        mainType = "application",
+        subType = "n-quads",
+        charset = HttpCharsets.`UTF-8`,
+        fileExtensions = List("nq")
+    )
+
     /**
      * A map of MIME types (strings) to supported RDF media types.
      */
@@ -63,7 +70,8 @@ object RdfMediaTypes {
         `application/ld+json`,
         `text/turtle`,
         `application/trig`,
-        `application/rdf+xml`
+        `application/rdf+xml`,
+        `application/n-quads`
     ).map {
         mediaType => mediaType.toString -> mediaType
     }.toMap

webapi/src/main/scala/org/knora/webapi/messages/store/triplestoremessages/TriplestoreMessages.scala

@@ -78,16 +78,18 @@ case class SparqlConstructRequest(sparql: String,
 
 /**
  * Represents a SPARQL CONSTRUCT query to be sent to the triplestore. The triplestore's will be
- * written to the specified file in Trig format. A successful response message will be a [[FileWrittenResponse]].
+ * written to the specified file in a quad format. A successful response message will be a [[FileWrittenResponse]].
  *
  * @param sparql               the SPARQL string.
  * @param graphIri             the named graph IRI to be used in the TriG file.
  * @param outputFile           the file to be written.
+ * @param outputFormat         the output file format.
  * @param featureFactoryConfig the feature factory configuration.
  */
 case class SparqlConstructFileRequest(sparql: String,
                                       graphIri: IRI,
                                       outputFile: File,
+                                      outputFormat: QuadFormat,
                                       featureFactoryConfig: FeatureFactoryConfig) extends TriplestoreRequest
 
 /**
@@ -212,15 +214,17 @@ object SparqlExtendedConstructResponse {
 case class SparqlExtendedConstructResponse(statements: Map[SubjectV2, SparqlExtendedConstructResponse.ConstructPredicateObjects])
 
 /**
- * Requests a named graph, which will be written to the specified file in Trig format. A successful response
+ * Requests a named graph, which will be written to the specified file. A successful response
  * will be a [[FileWrittenResponse]].
  *
  * @param graphIri             the IRI of the named graph.
  * @param outputFile           the destination file.
+ * @param outputFormat         the output file format.
  * @param featureFactoryConfig the feature factory configuration.
  */
 case class NamedGraphFileRequest(graphIri: IRI,
                                  outputFile: File,
+                                 outputFormat: QuadFormat,
                                  featureFactoryConfig: FeatureFactoryConfig) extends TriplestoreRequest
 
 /**
@@ -343,7 +347,7 @@ case class CheckTriplestoreResponse(triplestoreStatus: TriplestoreStatus, msg: S
 case class UpdateRepositoryRequest() extends TriplestoreRequest
 
 /**
- * Requests that the repository is downloaded to a TriG file. A successful response will be a [[FileWrittenResponse]].
+ * Requests that the repository is downloaded to an N-Quads file. A successful response will be a [[FileWrittenResponse]].
  *
  * @param outputFile           the output file.
  * @param featureFactoryConfig the feature factory configuration.
@@ -356,7 +360,7 @@ case class DownloadRepositoryRequest(outputFile: File, featureFactoryConfig: Fea
 case class FileWrittenResponse()
 
 /**
- * Requests that repository content is uploaded from a TriG file. A successful response will be a
+ * Requests that repository content is uploaded from an N-Quads. A successful response will be a
  * [[RepositoryUploadedResponse]].
  *
  * @param inputFile a TriG file containing the content to be uploaded to the repository.

webapi/src/main/scala/org/knora/webapi/messages/util/rdf/JsonLDUtil.scala

@@ -1698,9 +1698,8 @@ object JsonLDUtil {
                 }
 
             case blankNode: BlankNode =>
-                // It's a blank node. It should be possible to inline it. If not, the input model is invalid;
-                // return an error.
-                inlineResource(throw InvalidRdfException(s"Blank node ${blankNode.id} was not found or is referenced in more than one place"))
+                // It's a blank node. It should be possible to inline it. If not, return an empty blank node.
+                inlineResource(JsonLDObject(Map.empty))
         }
     }
 }

webapi/src/main/scala/org/knora/webapi/messages/util/rdf/RdfFeatureFactory.scala

@@ -37,7 +37,7 @@ object RdfFeatureFactory extends FeatureFactory {
     // Jena singletons.
     private val jenaNodeFactory = new JenaNodeFactory
     private val jenaModelFactory = new JenaModelFactory(jenaNodeFactory)
-    private val jenaFormatUtil = new JenaFormatUtil(jenaModelFactory)
+    private val jenaFormatUtil = new JenaFormatUtil(modelFactory = jenaModelFactory, nodeFactory = jenaNodeFactory)
     private var jenaShaclValidator: Option[JenaShaclValidator] = None
 
     // RDF4J singletons.