From 521150fbf3dac5ce4c075e6b3ef3c537004a750d Mon Sep 17 00:00:00 2001 From: Ivan Subotic <400790+subotic@users.noreply.github.com> Date: Tue, 12 Apr 2022 13:08:37 +0200 Subject: [PATCH 1/2] refactor(cacheservice): use ZIO (DEV-546) (#2022) * refactor(cache): remove service from name * refactor(cacheservice): inmem (ongoing) * refactor(cacheservice): inmem (ongoing) * chore(docs): add structurizr (c4) * refactor(cache): inmem get user * refactor(cacheservice): in-mem version * refactor(cacheservice): redis version (ongoing) * refactor(cacheservice): add zio-config (ongoing) * refactor(cacheservice): redis implementation (ongoing) * refactor(cacheservice): redis implementation (ongoing) * refactor(cacheservice): redis implementation (ongoing) * refactor(cacheservice): add redis implementation * refactor(cacheservice): add redis serialization implementation (ongoing) * refactor: after spartan session (ongoing) * refactor(cacheservice): remove actor * refactor(cacheservice): fix tests (ongoing) * build: fix after merge * refactor(cacheservice): fix tests * refactor(cacheservice): add Redis Testcontainers layer * refactor(cacheservice): moving runtime initialization to main * refactor(cacheservice): moving runtime initialisation back where it was * docs: add ADR record * chore(deps): bump zio libraries * chore(deps): bump zio libraries * chore(deps): bump zio libraries * chore: remove unneeded file * refactor(cacheservice): ziofy redis testcontainers * cleanup * fix redis test container * fix redis test container * remove metrics logging * Update webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheServiceManager.scala Co-authored-by: irinaschubert * Update webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheServiceManager.scala Co-authored-by: irinaschubert * Update webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheServiceManager.scala Co-authored-by: irinaschubert * Update docs/architecture/decisions/0002-change-cache-service-manager-from-akka-actor-to-zlayer.md Co-authored-by: irinaschubert * cleanup * bumping dependencies * cleanup Co-authored-by: irinaschubert --- .adr-dir | 1 + .gitignore | 3 + .scalafmt.conf | 2 +- Makefile | 5 + build.sbt | 28 +- docs/architecture/README.md | 15 + .../0001-record-architecture-decisions.md | 19 + ...rvice-manager-from-akka-actor-to-zlayer.md | 19 + docs/architecture/workspace.dsl | 21 + .../dsp/schema/repo/SchemaRepoLive.scala | 4 +- .../dsp/schema/repo/SchemaRepoTest.scala | 8 +- project/Dependencies.scala | 125 +- .../knora/webapi/app/ApplicationActor.scala | 90 +- .../scala/org/knora/webapi/app/Main.scala | 86 +- .../org/knora/webapi/config/AppConfig.scala | 15 + .../scala/org/knora/webapi/core/Logging.scala | 28 + .../knora/webapi/exceptions/Exceptions.scala | 2 +- .../ProjectsMessagesADM.scala | 14 +- .../usersmessages/UsersMessagesADM.scala | 32 +- .../CacheServiceMessages.scala | 7 +- .../admin/ProjectsResponderADM.scala | 4 +- .../responders/admin/StoresResponderADM.scala | 6 +- .../responders/admin/UsersResponderADM.scala | 1358 +++++++++-------- .../org/knora/webapi/settings/package.scala | 6 +- .../org/knora/webapi/store/StoreManager.scala | 17 +- .../store/cacheservice/CacheService.scala | 27 - .../cacheservice/CacheServiceManager.scala | 150 +- .../store/cacheservice/api/CacheService.scala | 47 + .../{ => api}/CacheServiceExceptions.scala | 2 +- .../config/CacheServiceConfig.scala | 7 + .../impl/CacheServiceInMemImpl.scala | 220 +++ .../impl/CacheServiceRedisImpl.scala | 293 ++++ .../inmem/CacheServiceInMemImpl.scala | 175 --- .../redis/CacheServiceRedisImpl.scala | 397 ----- .../serialization/CacheSerialization.scala | 55 +- .../org/knora/webapi/util/ActorUtil.scala | 69 +- .../scala/org/knora/webapi/CoreSpec.scala | 6 +- .../org/knora/webapi/IntegrationSpec.scala | 32 +- .../knora/webapi/ManagersWithMockedSipi.scala | 14 +- .../org/knora/webapi/TestContainerRedis.scala | 10 +- .../webapi/responders/IriLockerSpec.scala | 4 +- .../admin/UsersResponderADMSpec.scala | 13 +- .../webapi/store/MockableStoreManager.scala | 8 +- .../CacheServiceManagerSpec.scala | 18 +- .../cacheservice/config/RedisTestConfig.scala | 16 + .../impl/CacheInMemImplSpec.scala | 116 ++ .../impl/CacheRedisImplSpec.scala | 92 ++ .../inmem/CacheServiceInMemImplSpec.scala | 69 - .../redis/CacheServiceRedisImplSpec.scala | 76 - .../CacheSerializationSpec.scala | 47 +- .../testcontainers/RedisTestContainer.scala | 33 + 51 files changed, 2095 insertions(+), 1816 deletions(-) create mode 100644 .adr-dir create mode 100644 docs/architecture/README.md create mode 100644 docs/architecture/decisions/0001-record-architecture-decisions.md create mode 100644 docs/architecture/decisions/0002-change-cache-service-manager-from-akka-actor-to-zlayer.md create mode 100644 docs/architecture/workspace.dsl create mode 100644 webapi/src/main/scala/org/knora/webapi/config/AppConfig.scala create mode 100644 webapi/src/main/scala/org/knora/webapi/core/Logging.scala delete mode 100644 webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheService.scala create mode 100644 webapi/src/main/scala/org/knora/webapi/store/cacheservice/api/CacheService.scala rename webapi/src/main/scala/org/knora/webapi/store/cacheservice/{ => api}/CacheServiceExceptions.scala (90%) create mode 100644 webapi/src/main/scala/org/knora/webapi/store/cacheservice/config/CacheServiceConfig.scala create mode 100644 webapi/src/main/scala/org/knora/webapi/store/cacheservice/impl/CacheServiceInMemImpl.scala create mode 100644 webapi/src/main/scala/org/knora/webapi/store/cacheservice/impl/CacheServiceRedisImpl.scala delete mode 100644 webapi/src/main/scala/org/knora/webapi/store/cacheservice/inmem/CacheServiceInMemImpl.scala delete mode 100644 webapi/src/main/scala/org/knora/webapi/store/cacheservice/redis/CacheServiceRedisImpl.scala create mode 100644 webapi/src/test/scala/org/knora/webapi/store/cacheservice/config/RedisTestConfig.scala create mode 100644 webapi/src/test/scala/org/knora/webapi/store/cacheservice/impl/CacheInMemImplSpec.scala create mode 100644 webapi/src/test/scala/org/knora/webapi/store/cacheservice/impl/CacheRedisImplSpec.scala delete mode 100644 webapi/src/test/scala/org/knora/webapi/store/cacheservice/inmem/CacheServiceInMemImplSpec.scala delete mode 100644 webapi/src/test/scala/org/knora/webapi/store/cacheservice/redis/CacheServiceRedisImplSpec.scala create mode 100644 webapi/src/test/scala/org/knora/webapi/testcontainers/RedisTestContainer.scala diff --git a/.adr-dir b/.adr-dir new file mode 100644 index 00000000000..da5cac6b772 --- /dev/null +++ b/.adr-dir @@ -0,0 +1 @@ +docs/architecture/decisions diff --git a/.gitignore b/.gitignore index 733b0e2c1e5..b2282fd28fd 100644 --- a/.gitignore +++ b/.gitignore @@ -18,6 +18,8 @@ webapi-it webapi-test upgrade/project sipi/test +/docs/architecture/.structurizr +/docs/architecture/workspace.json **/target/ *.aux @@ -31,6 +33,7 @@ sipi/test *.tdo *.toc *.bak +*.rdb .sbtrc dependencies.txt diff --git a/.scalafmt.conf b/.scalafmt.conf index bf360eab5bf..e94dd850831 100644 --- a/.scalafmt.conf +++ b/.scalafmt.conf @@ -1,6 +1,6 @@ version = "2.7.5" maxColumn = 120 -align.preset = some +align.preset = most align.multiline = false continuationIndent.defnSite = 2 assumeStandardLibraryStripMargin = true diff --git a/Makefile b/Makefile index dc05763536e..c8667706565 100644 --- a/Makefile +++ b/Makefile @@ -33,6 +33,11 @@ docs-install-requirements: ## install requirements docs-clean: ## cleans the project directory @rm -rf site/ +.PHONY: structurizer +structurizer: ## starts the structurizer and serves c4 architecture docs + @docker pull structurizr/lite + @docker run -it --rm -p 8080:8080 -v $(CURRENT_DIR)/docs/architecture:/usr/local/structurizr structurizr/lite + ################################# # Docker targets ################################# diff --git a/build.sbt b/build.sbt index 28bd1dacf94..db95f32dbfe 100644 --- a/build.sbt +++ b/build.sbt @@ -100,7 +100,7 @@ lazy val webapi: Project = Project(id = "webapi", base = file("webapi")) .settings(buildSettings) .enablePlugins(SbtTwirl, JavaAppPackaging, DockerPlugin, GatlingPlugin, JavaAgent, RevolverPlugin, BuildInfoPlugin) .settings( - webApiCommonSettings, + name := "webapi", resolvers ++= Seq( Resolver.bintrayRepo("hseeberger", "maven") ), @@ -112,11 +112,11 @@ lazy val webapi: Project = Project(id = "webapi", base = file("webapi")) .settings( // add needed files to production jar Compile / packageBin / mappings ++= Seq( - (rootBaseDir.value / "knora-ontologies" / "knora-admin.ttl") -> "knora-ontologies/knora-admin.ttl", - (rootBaseDir.value / "knora-ontologies" / "knora-base.ttl") -> "knora-ontologies/knora-base.ttl", - (rootBaseDir.value / "knora-ontologies" / "salsah-gui.ttl") -> "knora-ontologies/salsah-gui.ttl", - (rootBaseDir.value / "knora-ontologies" / "standoff-data.ttl") -> "knora-ontologies/standoff-data.ttl", - (rootBaseDir.value / "knora-ontologies" / "standoff-onto.ttl") -> "knora-ontologies/standoff-onto.ttl", + (rootBaseDir.value / "knora-ontologies" / "knora-admin.ttl") -> "knora-ontologies/knora-admin.ttl", + (rootBaseDir.value / "knora-ontologies" / "knora-base.ttl") -> "knora-ontologies/knora-base.ttl", + (rootBaseDir.value / "knora-ontologies" / "salsah-gui.ttl") -> "knora-ontologies/salsah-gui.ttl", + (rootBaseDir.value / "knora-ontologies" / "standoff-data.ttl") -> "knora-ontologies/standoff-data.ttl", + (rootBaseDir.value / "knora-ontologies" / "standoff-onto.ttl") -> "knora-ontologies/standoff-onto.ttl", (rootBaseDir.value / "webapi" / "scripts" / "fuseki-repository-config.ttl.template") -> "webapi/scripts/fuseki-repository-config.ttl.template" // needed for initialization of triplestore ), // use packaged jars (through packageBin) on classpaths instead of class directories for production @@ -124,7 +124,7 @@ lazy val webapi: Project = Project(id = "webapi", base = file("webapi")) // add needed files to test jar Test / packageBin / mappings ++= Seq( (rootBaseDir.value / "webapi" / "scripts" / "fuseki-repository-config.ttl.template") -> "webapi/scripts/fuseki-repository-config.ttl.template", // needed for initialization of triplestore - (rootBaseDir.value / "sipi" / "config" / "sipi.docker-config.lua") -> "sipi/config/sipi.docker-config.lua" + (rootBaseDir.value / "sipi" / "config" / "sipi.docker-config.lua") -> "sipi/config/sipi.docker-config.lua" ), // use packaged jars (through packageBin) on classpaths instead of class directories for test Test / exportJars := true @@ -140,12 +140,14 @@ lazy val webapi: Project = Project(id = "webapi", base = file("webapi")) javaAgents += Dependencies.aspectjweaver, fork := true, // run tests in a forked JVM Test / testForkedParallel := false, // run forked tests in parallel - Test / parallelExecution := false, // run non-forked tests in parallel + Test / parallelExecution := false, // run non-forked tests in parallel // Global / concurrentRestrictions += Tags.limit(Tags.Test, 1), // restrict the number of concurrently executing tests in all projects Test / javaOptions ++= Seq("-Dconfig.resource=fuseki.conf") ++ webapiJavaTestOptions, // Test / javaOptions ++= Seq("-Dakka.log-config-on-start=on"), // prints out akka config // Test / javaOptions ++= Seq("-Dconfig.trace=loads"), // prints out config locations - Test / testOptions += Tests.Argument("-oDF") // show full stack traces and test case durations + Test / testOptions += Tests.Argument("-oDF"), // show full stack traces and test case durations + // add test framework for running zio-tests + Test / testFrameworks ++= Seq(new TestFramework("zio.test.sbt.ZTestFramework")) ) .settings( // prepare for publishing @@ -189,8 +191,8 @@ lazy val webapi: Project = Project(id = "webapi", base = file("webapi")) name, version, "akkaHttp" -> Dependencies.akkaHttpVersion, - "sipi" -> Dependencies.sipiImage, - "fuseki" -> Dependencies.fusekiImage + "sipi" -> Dependencies.sipiImage, + "fuseki" -> Dependencies.fusekiImage ), buildInfoPackage := "org.knora.webapi.http.version" ) @@ -219,6 +221,10 @@ lazy val webapiJavaTestOptions = Seq( //"-XX:MaxMetaspaceSize=4096m" ) +////////////////////////////////////// +// DSP's new codebase +////////////////////////////////////// + lazy val apiMain = project .in(file("dsp-api-main")) .settings( diff --git a/docs/architecture/README.md b/docs/architecture/README.md new file mode 100644 index 00000000000..d0a7fb913f2 --- /dev/null +++ b/docs/architecture/README.md @@ -0,0 +1,15 @@ +# C4 Model and ADRs + +## Installation + +```bash +$ brew install adr-tools +``` + +## Usage + +Run the following command from the root directory to start the C4 model browser: + +```bash +$ make structurizer +``` diff --git a/docs/architecture/decisions/0001-record-architecture-decisions.md b/docs/architecture/decisions/0001-record-architecture-decisions.md new file mode 100644 index 00000000000..ed9d71ee405 --- /dev/null +++ b/docs/architecture/decisions/0001-record-architecture-decisions.md @@ -0,0 +1,19 @@ +# 1. Record architecture decisions + +Date: 2022-03-14 + +## Status + +Accepted + +## Context + +We need to record the architectural decisions made on this project. + +## Decision + +We will use Architecture Decision Records, as [described by Michael Nygard](http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions). + +## Consequences + +See Michael Nygard's article, linked above. For a lightweight ADR toolset, see Nat Pryce's [adr-tools](https://github.com/npryce/adr-tools). diff --git a/docs/architecture/decisions/0002-change-cache-service-manager-from-akka-actor-to-zlayer.md b/docs/architecture/decisions/0002-change-cache-service-manager-from-akka-actor-to-zlayer.md new file mode 100644 index 00000000000..4e038ad1e90 --- /dev/null +++ b/docs/architecture/decisions/0002-change-cache-service-manager-from-akka-actor-to-zlayer.md @@ -0,0 +1,19 @@ +# 2. Change Cache Service Manager from Akka-Actor to ZLayer + +Date: 2022-04-06 + +## Status + +Accepted + +## Context + +The `org.knora.webapi.store.cacheservice.CacheServiceManager` was implemented as an `Akka-Actor`. + +## Decision + +As part of the move from `Akka` to `ZIO`, it was decided that the `CacheServiceManager` and the whole implementation of the in-memory and Redis backed cache is refactored using ZIO. + +## Consequences + +The usage from other actors stays the same. The actor messages and responses don't change. diff --git a/docs/architecture/workspace.dsl b/docs/architecture/workspace.dsl new file mode 100644 index 00000000000..0bae6febb9e --- /dev/null +++ b/docs/architecture/workspace.dsl @@ -0,0 +1,21 @@ +workspace { + + model { + user = person "User" + softwareSystem = softwareSystem "Software System" + + user -> softwareSystem "Uses" + } + + views { + systemContext softwareSystem "Diagram1" { + include * + autoLayout + } + + theme default + } + + !adrs decisions + +} diff --git a/dsp-schema/repo/src/main/scala/dsp/schema/repo/SchemaRepoLive.scala b/dsp-schema/repo/src/main/scala/dsp/schema/repo/SchemaRepoLive.scala index dd618a186ae..7b9591fad28 100644 --- a/dsp-schema/repo/src/main/scala/dsp/schema/repo/SchemaRepoLive.scala +++ b/dsp-schema/repo/src/main/scala/dsp/schema/repo/SchemaRepoLive.scala @@ -14,6 +14,6 @@ case class SchemaRepoLive() extends SchemaRepo { } -object SchemaRepoLive extends (() => SchemaRepo) { - val layer: URLayer[Any, SchemaRepo] = (SchemaRepoLive.apply _).toLayer +object SchemaRepoLive { + val layer: URLayer[Any, SchemaRepo] = ZLayer.succeed(SchemaRepoLive()) } diff --git a/dsp-schema/repo/src/main/scala/dsp/schema/repo/SchemaRepoTest.scala b/dsp-schema/repo/src/main/scala/dsp/schema/repo/SchemaRepoTest.scala index ee83a9612e3..01bda642c7f 100644 --- a/dsp-schema/repo/src/main/scala/dsp/schema/repo/SchemaRepoTest.scala +++ b/dsp-schema/repo/src/main/scala/dsp/schema/repo/SchemaRepoTest.scala @@ -7,18 +7,18 @@ case class SchemaRepoTest() extends SchemaRepo { private var map: Map[UserID, UserProfile] = Map() def setTestData(map0: Map[UserID, UserProfile]): Task[Unit] = - Task { map = map0 } + Task.succeed { map = map0 } def getTestData: Task[Map[UserID, UserProfile]] = - Task(map) + Task.succeed(map) def lookup(id: UserID): Task[UserProfile] = - Task(map(id)) + Task.succeed(map(id)) def update(id: UserID, profile: UserProfile): Task[Unit] = Task.attempt { map = map + (id -> profile) } } object SchemaRepoTest extends (() => SchemaRepo) { - val layer: URLayer[Any, SchemaRepo] = (SchemaRepoTest.apply _).toLayer + val layer: URLayer[Any, SchemaRepo] = ZLayer.succeed(SchemaRepoTest()) } diff --git a/project/Dependencies.scala b/project/Dependencies.scala index 328292f4ad9..d738e71a4cb 100644 --- a/project/Dependencies.scala +++ b/project/Dependencies.scala @@ -9,46 +9,56 @@ import sbt.Keys._ import sbt.{Def, _} object Dependencies { - + val fusekiImage = "daschswiss/apache-jena-fuseki:2.0.8" // should be the same version as in docker-compose.yml - val sipiImage = "daschswiss/sipi:3.5.0" // base image the knora-sipi image is created from - + val sipiImage = "daschswiss/sipi:3.5.0" // base image the knora-sipi image is created from + // versions - val akkaHttpVersion = "10.2.9" - val akkaVersion = "2.6.19" - val jenaVersion = "4.4.0" - val metricsVersion = "4.0.1" - val scalaVersion = "2.13.8" - val ZioHttpVersion = "2.0.0-RC3" - val ZioPreludeVersion = "1.0.0-RC10" - val ZioVersion = "2.0.0-RC2" + val akkaHttpVersion = "10.2.9" + val akkaVersion = "2.6.19" + val jenaVersion = "4.4.0" + val metricsVersion = "4.0.1" + val scalaVersion = "2.13.8" + val ZioVersion = "2.0.0-RC5" + val ZioHttpVersion = "2.0.0-RC4" + val ZioJsonVersion = "0.3.0-RC3" + val ZioConfigVersion = "3.0.0-RC8" + val ZioSchemaVersion = "0.2.0-RC5" + val ZioLoggingVersion = "2.0.0-RC8" + val ZioZmxVersion = "2.0.0-RC4" + val ZioPreludeVersion = "1.0.0-RC13" // ZIO - all Scala 3 compatible - val zio = "dev.zio" %% "zio" % ZioVersion - val zioHttp = "io.d11" %% "zhttp" % ZioHttpVersion - val zioPrelude = "dev.zio" %% "zio-prelude" % ZioPreludeVersion - val zioTest = "dev.zio" %% "zio-test" % ZioVersion - val zioTestSbt = "dev.zio" %% "zio-test-sbt" % ZioVersion + val zio = "dev.zio" %% "zio" % ZioVersion + val zioHttp = "io.d11" %% "zhttp" % ZioHttpVersion + val zioJson = "dev.zio" %% "zio-json" % ZioJsonVersion + val zioPrelude = "dev.zio" %% "zio-prelude" % ZioPreludeVersion + val zioLoggingSlf4j = "dev.zio" %% "zio-logging-slf4j" % ZioLoggingVersion + val zioConfig = "dev.zio" %% "zio-config" % ZioConfigVersion + val zioConfigMagnolia = "dev.zio" %% "zio-config-magnolia" % ZioConfigVersion + val zioConfigTypesafe = "dev.zio" %% "zio-config-typesafe" % ZioConfigVersion + val zioTest = "dev.zio" %% "zio-test" % ZioVersion + val zioTestSbt = "dev.zio" %% "zio-test-sbt" % ZioVersion // akka - val akkaActor = "com.typesafe.akka" %% "akka-actor" % akkaVersion // Scala 3 compatible - val akkaHttp = "com.typesafe.akka" %% "akka-http" % akkaHttpVersion // Scala 3 incompatible - val akkaHttpCors = "ch.megard" %% "akka-http-cors" % "1.0.0" // Scala 3 incompatible + val akkaActor = "com.typesafe.akka" %% "akka-actor" % akkaVersion // Scala 3 compatible + val akkaHttp = "com.typesafe.akka" %% "akka-http" % akkaHttpVersion // Scala 3 incompatible + val akkaHttpCors = "ch.megard" %% "akka-http-cors" % "1.0.0" // Scala 3 incompatible val akkaHttpSprayJson = "com.typesafe.akka" %% "akka-http-spray-json" % akkaHttpVersion // Scala 3 incompatible - val akkaSlf4j = "com.typesafe.akka" %% "akka-slf4j" % akkaVersion // Scala 3 compatible - val akkaStream = "com.typesafe.akka" %% "akka-stream" % akkaVersion // Scala 3 compatible + val akkaSlf4j = "com.typesafe.akka" %% "akka-slf4j" % akkaVersion // Scala 3 compatible + val akkaStream = "com.typesafe.akka" %% "akka-stream" % akkaVersion // Scala 3 compatible // jena val jenaText = "org.apache.jena" % "jena-text" % jenaVersion // logging - val logbackClassic = "ch.qos.logback" % "logback-classic" % "1.2.11" - val scalaLogging = "com.typesafe.scala-logging" %% "scala-logging" % "3.9.4" // Scala 3 compatible + val logbackClassic = "ch.qos.logback" % "logback-classic" % "1.2.11" + val scalaLogging = "com.typesafe.scala-logging" %% "scala-logging" % "3.9.4" // Scala 3 compatible // Metrics - val aspectjweaver = "org.aspectj" % "aspectjweaver" % "1.9.4" - val kamonCore = "io.kamon" %% "kamon-core" % "2.5.0" // Scala 3 compatible - val kamonScalaFuture = "io.kamon" %% "kamon-scala-future" % "2.1.5" // Scala 3 incompatible + val aspectjweaver = "org.aspectj" % "aspectjweaver" % "1.9.4" + val kamonCore = "io.kamon" %% "kamon-core" % "2.5.0" // Scala 3 compatible + val kamonScalaFuture = "io.kamon" %% "kamon-scala-future" % "2.1.5" // Scala 3 incompatible // input validation val commonsValidator = @@ -61,34 +71,34 @@ object Dependencies { // caching val ehcache = "net.sf.ehcache" % "ehcache" % "2.10.9.2" - val jedis = "redis.clients" % "jedis" % "4.2.1" + val jedis = "redis.clients" % "jedis" % "4.2.1" // serialization val chill = "com.twitter" %% "chill" % "0.10.0" // Scala 3 incompatible // other - val diff = "com.sksamuel.diff" % "diff" % "1.1.11" - val gwtServlet = "com.google.gwt" % "gwt-servlet" % "2.9.0" - val icu4j = "com.ibm.icu" % "icu4j" % "70.1" - val jakartaJSON = "org.glassfish" % "jakarta.json" % "2.0.1" - val jodd = "org.jodd" % "jodd" % "3.2.6" - val rdf4jClient = "org.eclipse.rdf4j" % "rdf4j-client" % "3.4.4" - val rdf4jShacl = "org.eclipse.rdf4j" % "rdf4j-shacl" % "3.4.4" - val saxonHE = "net.sf.saxon" % "Saxon-HE" % "11.3" - val scalaGraph = "org.scala-graph" %% "graph-core" % "1.13.1" // Scala 3 incompatible - val scallop = "org.rogach" %% "scallop" % "4.1.0" // Scala 3 compatible - val swaggerAkkaHttp = "com.github.swagger-akka-http" %% "swagger-akka-http" % "1.6.0" // Scala 3 incompatible - val titaniumJSONLD = "com.apicatalog" % "titanium-json-ld" % "1.2.0" - val xmlunitCore = "org.xmlunit" % "xmlunit-core" % "2.9.0" + val diff = "com.sksamuel.diff" % "diff" % "1.1.11" + val gwtServlet = "com.google.gwt" % "gwt-servlet" % "2.9.0" + val icu4j = "com.ibm.icu" % "icu4j" % "70.1" + val jakartaJSON = "org.glassfish" % "jakarta.json" % "2.0.1" + val jodd = "org.jodd" % "jodd" % "3.2.6" + val rdf4jClient = "org.eclipse.rdf4j" % "rdf4j-client" % "3.4.4" + val rdf4jShacl = "org.eclipse.rdf4j" % "rdf4j-shacl" % "3.4.4" + val saxonHE = "net.sf.saxon" % "Saxon-HE" % "11.3" + val scalaGraph = "org.scala-graph" %% "graph-core" % "1.13.1" // Scala 3 incompatible + val scallop = "org.rogach" %% "scallop" % "4.1.0" // Scala 3 compatible + val swaggerAkkaHttp = "com.github.swagger-akka-http" %% "swagger-akka-http" % "1.6.0" // Scala 3 incompatible + val titaniumJSONLD = "com.apicatalog" % "titanium-json-ld" % "1.2.0" + val xmlunitCore = "org.xmlunit" % "xmlunit-core" % "2.9.0" // test - val akkaHttpTestkit = "com.typesafe.akka" %% "akka-http-testkit" % akkaHttpVersion // Scala 3 incompatible - val akkaStreamTestkit = "com.typesafe.akka" %% "akka-stream-testkit" % akkaVersion // Scala 3 compatible - val akkaTestkit = "com.typesafe.akka" %% "akka-testkit" % akkaVersion // Scala 3 compatible - val gatlingHighcharts = "io.gatling.highcharts" % "gatling-charts-highcharts" % "3.7.6" - val gatlingTestFramework = "io.gatling" % "gatling-test-framework" % "3.7.6" - val scalaTest = "org.scalatest" %% "scalatest" % "3.2.2" // Scala 3 compatible - val testcontainers = "org.testcontainers" % "testcontainers" % "1.16.3" + val akkaHttpTestkit = "com.typesafe.akka" %% "akka-http-testkit" % akkaHttpVersion // Scala 3 incompatible + val akkaStreamTestkit = "com.typesafe.akka" %% "akka-stream-testkit" % akkaVersion // Scala 3 compatible + val akkaTestkit = "com.typesafe.akka" %% "akka-testkit" % akkaVersion // Scala 3 compatible + val gatlingHighcharts = "io.gatling.highcharts" % "gatling-charts-highcharts" % "3.7.6" + val gatlingTestFramework = "io.gatling" % "gatling-test-framework" % "3.7.6" + val scalaTest = "org.scalatest" %% "scalatest" % "3.2.2" // Scala 3 compatible + val testcontainers = "org.testcontainers" % "testcontainers" % "1.16.3" val webapiLibraryDependencies = Seq( akkaActor, @@ -96,15 +106,14 @@ object Dependencies { akkaHttpCors, akkaHttpSprayJson, akkaHttpTestkit % Test, - akkaSlf4j % Runtime, + akkaSlf4j % Runtime, akkaStream, akkaStreamTestkit % Test, - akkaTestkit % Test, - chill, + akkaTestkit % Test, commonsValidator, diff, ehcache, - gatlingHighcharts % Test, + gatlingHighcharts % Test, gatlingTestFramework % Test, gwtServlet, icu4j, @@ -116,7 +125,7 @@ object Dependencies { kamonCore, kamonScalaFuture, logbackClassic % Runtime, - rdf4jClient % Test, + rdf4jClient % Test, rdf4jShacl, saxonHE, scalaGraph, @@ -129,8 +138,14 @@ object Dependencies { titaniumJSONLD, xmlunitCore % Test, zio, + zioConfig, + zioConfigMagnolia, + zioConfigTypesafe, + zioHttp, + zioJson, + zioLoggingSlf4j, zioPrelude, - zioTest % Test, + zioTest % Test, zioTestSbt % Test ) @@ -146,7 +161,7 @@ object Dependencies { zioPrelude ) - val schemaRepoLibraryDependencies = Seq() + val schemaRepoLibraryDependencies = Seq() val schemaRepoEventStoreServiceLibraryDependencies = Seq() - val schemaRepoSearchServiceLibraryDependencies = Seq() + val schemaRepoSearchServiceLibraryDependencies = Seq() } diff --git a/webapi/src/main/scala/org/knora/webapi/app/ApplicationActor.scala b/webapi/src/main/scala/org/knora/webapi/app/ApplicationActor.scala index ab069081099..cd57e769eba 100644 --- a/webapi/src/main/scala/org/knora/webapi/app/ApplicationActor.scala +++ b/webapi/src/main/scala/org/knora/webapi/app/ApplicationActor.scala @@ -5,8 +5,14 @@ package org.knora.webapi.app +import akka.actor.Actor +import akka.actor.ActorRef +import akka.actor.ActorSystem +import akka.actor.OneForOneStrategy +import akka.actor.Props +import akka.actor.Stash import akka.actor.SupervisorStrategy._ -import akka.actor.{Actor, ActorRef, ActorSystem, OneForOneStrategy, Props, Stash, Timers} +import akka.actor.Timers import akka.http.scaladsl.Http import akka.http.scaladsl.server.Directives._ import akka.http.scaladsl.server.Route @@ -14,48 +20,67 @@ import akka.stream.Materializer import akka.util.Timeout import ch.megard.akka.http.cors.scaladsl.CorsDirectives import ch.megard.akka.http.cors.scaladsl.settings.CorsSettings +import com.typesafe.config.ConfigFactory import com.typesafe.scalalogging.LazyLogging import kamon.Kamon +import org.knora.webapi.config.AppConfig import org.knora.webapi.core.LiveActorMaker -import org.knora.webapi.exceptions.{ - InconsistentRepositoryDataException, - MissingLastModificationDateOntologyException, - SipiException, - UnexpectedMessageException, - UnsupportedValueException -} -import org.knora.webapi.feature.{FeatureFactoryConfig, KnoraSettingsFeatureFactoryConfig} +import org.knora.webapi.exceptions.InconsistentRepositoryDataException +import org.knora.webapi.exceptions.MissingLastModificationDateOntologyException +import org.knora.webapi.exceptions.SipiException +import org.knora.webapi.exceptions.UnexpectedMessageException +import org.knora.webapi.exceptions.UnsupportedValueException +import org.knora.webapi.feature.FeatureFactoryConfig +import org.knora.webapi.feature.KnoraSettingsFeatureFactoryConfig import org.knora.webapi.http.directives.DSPApiDirectives import org.knora.webapi.http.version.ServerVersion import org.knora.webapi.messages.admin.responder.KnoraRequestADM +import org.knora.webapi.messages.admin.responder.projectsmessages.ProjectADM +import org.knora.webapi.messages.admin.responder.usersmessages.UserADM import org.knora.webapi.messages.app.appmessages._ import org.knora.webapi.messages.store.StoreRequest -import org.knora.webapi.messages.store.cacheservicemessages.{ - CacheServiceGetStatus, - CacheServiceStatusNOK, - CacheServiceStatusOK -} -import org.knora.webapi.messages.store.sipimessages.{IIIFServiceGetStatus, IIIFServiceStatusNOK, IIIFServiceStatusOK} +import org.knora.webapi.messages.store.cacheservicemessages.CacheServiceGetStatus +import org.knora.webapi.messages.store.cacheservicemessages.CacheServiceStatusNOK +import org.knora.webapi.messages.store.cacheservicemessages.CacheServiceStatusOK +import org.knora.webapi.messages.store.sipimessages.IIIFServiceGetStatus +import org.knora.webapi.messages.store.sipimessages.IIIFServiceStatusNOK +import org.knora.webapi.messages.store.sipimessages.IIIFServiceStatusOK import org.knora.webapi.messages.store.triplestoremessages._ -import org.knora.webapi.messages.util.{KnoraSystemInstances, ResponderData} +import org.knora.webapi.messages.util.KnoraSystemInstances +import org.knora.webapi.messages.util.ResponderData import org.knora.webapi.messages.v1.responder.KnoraRequestV1 +import org.knora.webapi.messages.v2.responder.KnoraRequestV2 +import org.knora.webapi.messages.v2.responder.SuccessResponseV2 import org.knora.webapi.messages.v2.responder.ontologymessages.LoadOntologiesRequestV2 -import org.knora.webapi.messages.v2.responder.{KnoraRequestV2, SuccessResponseV2} import org.knora.webapi.responders.ResponderManager import org.knora.webapi.routing._ import org.knora.webapi.routing.admin._ import org.knora.webapi.routing.v1._ import org.knora.webapi.routing.v2._ -import org.knora.webapi.settings.{KnoraDispatchers, KnoraSettings, KnoraSettingsImpl, _} +import org.knora.webapi.settings.KnoraDispatchers +import org.knora.webapi.settings.KnoraSettings +import org.knora.webapi.settings.KnoraSettingsImpl +import org.knora.webapi.settings._ import org.knora.webapi.store.StoreManager -import org.knora.webapi.store.cacheservice.inmem.CacheServiceInMemImpl +import org.knora.webapi.store.cacheservice.CacheServiceManager +import org.knora.webapi.store.cacheservice.impl.CacheServiceInMemImpl import org.knora.webapi.store.cacheservice.settings.CacheServiceSettings import org.knora.webapi.util.cache.CacheUtil import redis.clients.jedis.exceptions.JedisConnectionException +import zio.Runtime +import zio.ZIO +import zio.config.typesafe.TypesafeConfig +import zio.stm.TRef +import scala.concurrent.ExecutionContext +import scala.concurrent.Future import scala.concurrent.duration._ -import scala.concurrent.{ExecutionContext, Future} -import scala.util.{Failure, Success} +import scala.util.Failure +import scala.util.Success +import org.knora.webapi.store.cacheservice.config.RedisConfig +import zio.ZEnvironment +import zio.RuntimeConfig +import org.knora.webapi.core.Logging trait Managers { implicit val system: ActorSystem @@ -66,11 +91,22 @@ trait Managers { trait LiveManagers extends Managers { this: Actor => + /** + * Initializing the cache service manager, which is a ZLayer, + * by unsafe running it. + */ + lazy val cacheServiceManager: CacheServiceManager = + Runtime(ZEnvironment.empty, RuntimeConfig.default @@ Logging.live) + .unsafeRun( + (for (manager <- ZIO.service[CacheServiceManager]) + yield manager).provide(CacheServiceInMemImpl.layer, CacheServiceManager.layer) + ) + /** * The actor that forwards messages to actors that deal with persistent storage. */ lazy val storeManager: ActorRef = context.actorOf( - Props(new StoreManager(appActor = self, cs = CacheServiceInMemImpl) with LiveActorMaker) + Props(new StoreManager(appActor = self, csm = cacheServiceManager) with LiveActorMaker) .withDispatcher(KnoraDispatchers.KnoraActorDispatcher), name = StoreManagerActorName ) @@ -169,12 +205,12 @@ class ApplicationActor extends Actor with Stash with LazyLogging with AroundDire case _: Exception => Escalate } - private var appState: AppState = AppStates.Stopped + private var appState: AppState = AppStates.Stopped private var allowReloadOverHTTPState = false - private var printConfigState = false - private var ignoreRepository = true - private var withIIIFService = true - private val withCacheService = cacheServiceSettings.cacheServiceEnabled + private var printConfigState = false + private var ignoreRepository = true + private var withIIIFService = true + private val withCacheService = cacheServiceSettings.cacheServiceEnabled /** * Startup of the ApplicationActor is a two step process: diff --git a/webapi/src/main/scala/org/knora/webapi/app/Main.scala b/webapi/src/main/scala/org/knora/webapi/app/Main.scala index c62eea56ee2..fb1dea15f93 100644 --- a/webapi/src/main/scala/org/knora/webapi/app/Main.scala +++ b/webapi/src/main/scala/org/knora/webapi/app/Main.scala @@ -6,62 +6,48 @@ package org.knora.webapi.app import akka.actor.Terminated -import org.knora.webapi.messages.app.appmessages.{AppStart, SetAllowReloadOverHTTPState, SetLoadDemoDataState} +import org.knora.webapi.messages.app.appmessages.AppStart -import scala.concurrent.duration._ -import scala.concurrent.{Await, Future} +import zio.config.typesafe.TypesafeConfig +import com.typesafe.config.ConfigFactory +import org.knora.webapi.config.AppConfig + +import zio._ +import org.knora.webapi.core.Logging +import java.util.concurrent.TimeUnit /** * Starts Knora by bringing everything into scope by using the cake pattern. * The [[LiveCore]] trait provides an actor system and the main application * actor. */ -object Main extends App with LiveCore { - - val arglist = args.toList - - // loads demo data - if (arglist.contains("loadDemoData")) appActor ! SetLoadDemoDataState(true) - if (arglist.contains("--load-demo-data")) appActor ! SetLoadDemoDataState(true) - if (arglist.contains("-d")) appActor ! SetLoadDemoDataState(true) - - // allows reloading of data over HTTP - if (arglist.contains("allowReloadOverHTTP")) appActor ! SetAllowReloadOverHTTPState(true) - if (arglist.contains("--allow-reload-over-http")) appActor ! SetAllowReloadOverHTTPState(true) - if (arglist.contains("-r")) appActor ! SetAllowReloadOverHTTPState(true) - - if (arglist.contains("--help")) { - println(""" - | Usage: org.knora.webapi.Main - | or org.knora.webapi.Main -help - | - | Options: - | - | allowReloadOverHTTP, - | --allow-reload-over-http, - | -r Allows reloading of data over HTTP. - | - | -c Print the configuration on startup. - | - | --help Shows this message. - """.stripMargin) - } else { - /* Start the HTTP layer, loading data from the repository */ - appActor ! AppStart(ignoreRepository = false, requiresIIIFService = true) - - /** - * Adds shutting down of our actor system to the shutdown hook. - * Because we are blocking, we will run this on a separate thread. - */ - scala.sys.addShutdownHook( - new Thread(() => { - val terminate: Future[Terminated] = system.terminate() - Await.result(terminate, 30.seconds) - }) - ) - - system.registerOnTermination { - println("ActorSystem terminated") - } +object Main extends scala.App with LiveCore { + + /** + * Loads the applicaton configuration using ZIO-Config. ZIO-Config is capable to load + * the Typesafe-Config format. + */ + val config = TypesafeConfig.fromTypesafeConfig(ConfigFactory.load().getConfig("app"), AppConfig.config) + + /** + * Start server initialisation + */ + appActor ! AppStart(ignoreRepository = false, requiresIIIFService = true) + + /** + * Adds shutting down of our actor system to the shutdown hook. + * Because we are blocking, we will run this on a separate thread. + */ + scala.sys.addShutdownHook( + new Thread(() => { + import scala.concurrent._ + import scala.concurrent.duration._ + val terminate: Future[Terminated] = system.terminate() + Await.result(terminate, Duration(30.toLong, TimeUnit.SECONDS)) + }) + ) + + system.registerOnTermination { + println("ActorSystem terminated") } } diff --git a/webapi/src/main/scala/org/knora/webapi/config/AppConfig.scala b/webapi/src/main/scala/org/knora/webapi/config/AppConfig.scala new file mode 100644 index 00000000000..99c7d9b58b9 --- /dev/null +++ b/webapi/src/main/scala/org/knora/webapi/config/AppConfig.scala @@ -0,0 +1,15 @@ +package org.knora.webapi.config + +import org.knora.webapi.store.cacheservice.config.CacheServiceConfig +import zio.config.ConfigDescriptor +import zio.config._ + +import typesafe._ +import magnolia._ + + +final case class AppConfig(cacheService: CacheServiceConfig) + +object AppConfig { + val config: ConfigDescriptor[AppConfig] = descriptor[AppConfig].mapKey(toKebabCase) +} diff --git a/webapi/src/main/scala/org/knora/webapi/core/Logging.scala b/webapi/src/main/scala/org/knora/webapi/core/Logging.scala new file mode 100644 index 00000000000..6ed8e71c2e2 --- /dev/null +++ b/webapi/src/main/scala/org/knora/webapi/core/Logging.scala @@ -0,0 +1,28 @@ +package org.knora.webapi.core + +import zio.logging._ +import zio.logging.backend.SLF4J._ +import zio.logging.backend.SLF4J +import zio.LogLevel +import zio.RuntimeConfigAspect + +object Logging { + val logFormat = "[correlation-id = %s] %s" + def generateCorrelationId = Some(java.util.UUID.randomUUID()) + + val live: RuntimeConfigAspect = { + SLF4J.slf4j( + logLevel = LogLevel.Debug, + format = LogFormat.default, + _ => "dsp" + ) + } + + val console: RuntimeConfigAspect = { + zio.logging.console( + logLevel = LogLevel.Info, + format = LogFormat.default + ) + } + +} diff --git a/webapi/src/main/scala/org/knora/webapi/exceptions/Exceptions.scala b/webapi/src/main/scala/org/knora/webapi/exceptions/Exceptions.scala index 0dd8cfef885..f17a123a0d3 100644 --- a/webapi/src/main/scala/org/knora/webapi/exceptions/Exceptions.scala +++ b/webapi/src/main/scala/org/knora/webapi/exceptions/Exceptions.scala @@ -372,7 +372,7 @@ object InvalidApiJsonException { } /** - * Indicates that the during caching with the [[org.knora.webapi.store.cacheservice.CacheService]] something went wrong. + * Indicates that during caching with [[org.knora.webapi.store.cacheservice.api.CacheService]] something went wrong. * * @param message a description of the error. */ diff --git a/webapi/src/main/scala/org/knora/webapi/messages/admin/responder/projectsmessages/ProjectsMessagesADM.scala b/webapi/src/main/scala/org/knora/webapi/messages/admin/responder/projectsmessages/ProjectsMessagesADM.scala index e739f9f78ca..11b9f42a246 100644 --- a/webapi/src/main/scala/org/knora/webapi/messages/admin/responder/projectsmessages/ProjectsMessagesADM.scala +++ b/webapi/src/main/scala/org/knora/webapi/messages/admin/responder/projectsmessages/ProjectsMessagesADM.scala @@ -582,7 +582,7 @@ class ProjectIdentifierADM private ( maybeShortcode ).flatten.head - def hasType: ProjectIdentifierType.Value = + def hasType: ProjectIdentifierType = if (maybeIri.isDefined) { ProjectIdentifierType.IRI } else if (maybeShortcode.isDefined) { @@ -647,13 +647,11 @@ class ProjectIdentifierADM private ( * - Shortcode * - Shortname */ -object ProjectIdentifierType extends Enumeration { - - type ProjectIdentifierType - - val IRI: Value = Value(0, "iri") - val SHORTCODE: Value = Value(1, "shortcode") - val SHORTNAME: Value = Value(2, "shortname") +sealed trait ProjectIdentifierType +object ProjectIdentifierType { + case object IRI extends ProjectIdentifierType + case object SHORTCODE extends ProjectIdentifierType + case object SHORTNAME extends ProjectIdentifierType } /** diff --git a/webapi/src/main/scala/org/knora/webapi/messages/admin/responder/usersmessages/UsersMessagesADM.scala b/webapi/src/main/scala/org/knora/webapi/messages/admin/responder/usersmessages/UsersMessagesADM.scala index c63c012b026..b56fa442228 100644 --- a/webapi/src/main/scala/org/knora/webapi/messages/admin/responder/usersmessages/UsersMessagesADM.scala +++ b/webapi/src/main/scala/org/knora/webapi/messages/admin/responder/usersmessages/UsersMessagesADM.scala @@ -477,16 +477,16 @@ case class UserOperationResponseADM(user: UserADM) extends KnoraResponseADM { * @param sessionId The sessionId,. * @param permissions The user's permissions. */ -case class UserADM( +final case class UserADM( id: IRI, username: String, email: String, - password: Option[String] = None, - token: Option[String] = None, givenName: String, familyName: String, status: Boolean, lang: String, + password: Option[String] = None, + token: Option[String] = None, groups: Seq[GroupADM] = Vector.empty[GroupADM], projects: Seq[ProjectADM] = Seq.empty[ProjectADM], sessionId: Option[String] = None, @@ -570,8 +570,8 @@ case class UserADM( */ def isSelf(identifier: UserIdentifierADM): Boolean = { - val iriEquals = identifier.toIriOption.contains(id) - val emailEquals = identifier.toEmailOption.contains(email) + val iriEquals = identifier.toIriOption.contains(id) + val emailEquals = identifier.toEmailOption.contains(email) val usernameEquals = identifier.toUsernameOption.contains(username) iriEquals || emailEquals || usernameEquals @@ -590,8 +590,8 @@ case class UserADM( def fullname: String = givenName + " " + familyName def getDigest: String = { - val md = java.security.MessageDigest.getInstance("SHA-1") - val time = System.currentTimeMillis().toString + val md = java.security.MessageDigest.getInstance("SHA-1") + val time = System.currentTimeMillis().toString val value = (time + this.toString).getBytes("UTF-8") md.digest(value).map("%02x".format(_)).mkString } @@ -679,10 +679,10 @@ case class UserADM( */ sealed trait UserInformationTypeADM object UserInformationTypeADM { - case object Public extends UserInformationTypeADM - case object Short extends UserInformationTypeADM + case object Public extends UserInformationTypeADM + case object Short extends UserInformationTypeADM case object Restricted extends UserInformationTypeADM - case object Full extends UserInformationTypeADM + case object Full extends UserInformationTypeADM // throw InconsistentRepositoryDataException(s"User profile type not supported: $name") } @@ -692,8 +692,8 @@ object UserInformationTypeADM { */ sealed trait UserIdentifierType object UserIdentifierType { - case object Iri extends UserIdentifierType - case object Email extends UserIdentifierType + case object Iri extends UserIdentifierType + case object Email extends UserIdentifierType case object Username extends UserIdentifierType } @@ -745,7 +745,7 @@ sealed abstract case class UserIdentifierADM private ( /** * Tries to return the value as email. */ - def toEmail: IRI = + def toEmail: String = maybeEmail.getOrElse( throw DataConversionException(s"Identifier $value is not of the required 'UserIdentifierType.EMAIL' type.") ) @@ -759,7 +759,7 @@ sealed abstract case class UserIdentifierADM private ( /** * Tries to return the value as username. */ - def toUsername: IRI = + def toUsername: String = maybeUsername.getOrElse( throw DataConversionException(s"Identifier $value is not of the required 'UserIdentifierType.USERNAME' type.") ) @@ -773,7 +773,7 @@ sealed abstract case class UserIdentifierADM private ( /** * Returns the string representation */ - override def toString: IRI = + override def toString: String = s"UserIdentifierADM(${this.value})" } @@ -954,7 +954,7 @@ object UsersADMJsonProtocol "newPassword" ) implicit val usersGetResponseADMFormat: RootJsonFormat[UsersGetResponseADM] = jsonFormat1(UsersGetResponseADM) - implicit val userProfileResponseADMFormat: RootJsonFormat[UserResponseADM] = jsonFormat1(UserResponseADM) + implicit val userProfileResponseADMFormat: RootJsonFormat[UserResponseADM] = jsonFormat1(UserResponseADM) implicit val userProjectMembershipsGetResponseADMFormat: RootJsonFormat[UserProjectMembershipsGetResponseADM] = jsonFormat1(UserProjectMembershipsGetResponseADM) implicit val userProjectAdminMembershipsGetResponseADMFormat diff --git a/webapi/src/main/scala/org/knora/webapi/messages/store/cacheservicemessages/CacheServiceMessages.scala b/webapi/src/main/scala/org/knora/webapi/messages/store/cacheservicemessages/CacheServiceMessages.scala index 91c5ca75558..cfdd896e4d4 100644 --- a/webapi/src/main/scala/org/knora/webapi/messages/store/cacheservicemessages/CacheServiceMessages.scala +++ b/webapi/src/main/scala/org/knora/webapi/messages/store/cacheservicemessages/CacheServiceMessages.scala @@ -39,7 +39,7 @@ case class CacheServicePutString(key: String, value: String) extends CacheServic /** * Message requesting to retrieve simple string stored under the key. */ -case class CacheServiceGetString(key: Option[String]) extends CacheServiceRequest +case class CacheServiceGetString(key: String) extends CacheServiceRequest /** * Message requesting to remove anything stored under the keys. @@ -51,11 +51,6 @@ case class CacheServiceRemoveValues(keys: Set[String]) extends CacheServiceReque */ case class CacheServiceFlushDB(requestingUser: UserADM) extends CacheServiceRequest -/** - * Message acknowledging the flush. - */ -case class CacheServiceFlushDBACK() - /** * Queries Cache Service status. */ diff --git a/webapi/src/main/scala/org/knora/webapi/responders/admin/ProjectsResponderADM.scala b/webapi/src/main/scala/org/knora/webapi/responders/admin/ProjectsResponderADM.scala index bef65018ed2..2c38b2955de 100644 --- a/webapi/src/main/scala/org/knora/webapi/responders/admin/ProjectsResponderADM.scala +++ b/webapi/src/main/scala/org/knora/webapi/responders/admin/ProjectsResponderADM.scala @@ -1425,8 +1425,8 @@ class ProjectsResponderADM(responderData: ResponderData) extends Responder(respo * @param project a [[ProjectADM]]. * @return true if writing was successful. */ - private def writeProjectADMToCache(project: ProjectADM): Future[Boolean] = { - val result = (storeManager ? CacheServicePutProjectADM(project)).mapTo[Boolean] + private def writeProjectADMToCache(project: ProjectADM): Future[Unit] = { + val result = (storeManager ? CacheServicePutProjectADM(project)).mapTo[Unit] result.map { res => log.debug("writeProjectADMToCache - result: {}", result) res diff --git a/webapi/src/main/scala/org/knora/webapi/responders/admin/StoresResponderADM.scala b/webapi/src/main/scala/org/knora/webapi/responders/admin/StoresResponderADM.scala index befc4d0f744..896f658cbe9 100644 --- a/webapi/src/main/scala/org/knora/webapi/responders/admin/StoresResponderADM.scala +++ b/webapi/src/main/scala/org/knora/webapi/responders/admin/StoresResponderADM.scala @@ -14,7 +14,7 @@ import org.knora.webapi.messages.admin.responder.storesmessages.{ StoreResponderRequestADM } import org.knora.webapi.messages.app.appmessages.GetAllowReloadOverHTTPState -import org.knora.webapi.messages.store.cacheservicemessages.{CacheServiceFlushDB, CacheServiceFlushDBACK} +import org.knora.webapi.messages.store.cacheservicemessages.CacheServiceFlushDB import org.knora.webapi.messages.store.triplestoremessages.{ RdfDataObject, ResetRepositoryContent, @@ -84,8 +84,8 @@ class StoresResponderADM(responderData: ResponderData) extends Responder(respond )).mapTo[SuccessResponseV2] _ = log.debug(s"resetTriplestoreContent - load ontology done - {}", loadOntologiesResponse.toString) - redisFlushDB <- (storeManager ? CacheServiceFlushDB(systemUser)).mapTo[CacheServiceFlushDBACK] - _ = log.debug(s"resetTriplestoreContent - flushing Redis store done - {}", redisFlushDB.toString) + _ <- (storeManager ? CacheServiceFlushDB(systemUser)) + _ = log.debug(s"resetTriplestoreContent - flushing Redis store done.") result = ResetTriplestoreContentResponseADM(message = "success") diff --git a/webapi/src/main/scala/org/knora/webapi/responders/admin/UsersResponderADM.scala b/webapi/src/main/scala/org/knora/webapi/responders/admin/UsersResponderADM.scala index d18caf77f54..41582308d90 100644 --- a/webapi/src/main/scala/org/knora/webapi/responders/admin/UsersResponderADM.scala +++ b/webapi/src/main/scala/org/knora/webapi/responders/admin/UsersResponderADM.scala @@ -153,95 +153,95 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde ): Future[Seq[UserADM]] = for { _ <- Future( - if ( - !requestingUser.permissions.isSystemAdmin && !requestingUser.permissions - .isProjectAdminInAnyProject() && !requestingUser.isSystemUser - ) { - throw ForbiddenException("ProjectAdmin or SystemAdmin permissions are required.") - } - ) + if ( + !requestingUser.permissions.isSystemAdmin && !requestingUser.permissions + .isProjectAdminInAnyProject() && !requestingUser.isSystemUser + ) { + throw ForbiddenException("ProjectAdmin or SystemAdmin permissions are required.") + } + ) sparqlQueryString <- Future( - org.knora.webapi.messages.twirl.queries.sparql.admin.txt - .getUsers( - triplestore = settings.triplestoreType, - maybeIri = None, - maybeUsername = None, - maybeEmail = None - ) - .toString() - ) + org.knora.webapi.messages.twirl.queries.sparql.admin.txt + .getUsers( + triplestore = settings.triplestoreType, + maybeIri = None, + maybeUsername = None, + maybeEmail = None + ) + .toString() + ) usersResponse <- (storeManager ? SparqlExtendedConstructRequest( - sparql = sparqlQueryString, - featureFactoryConfig = featureFactoryConfig - )).mapTo[SparqlExtendedConstructResponse] + sparql = sparqlQueryString, + featureFactoryConfig = featureFactoryConfig + )).mapTo[SparqlExtendedConstructResponse] statements = usersResponse.statements.toList users: Seq[UserADM] = statements.map { case (userIri: SubjectV2, propsMap: Map[SmartIri, Seq[LiteralV2]]) => - UserADM( - id = userIri.toString, - username = propsMap - .getOrElse( - OntologyConstants.KnoraAdmin.Username.toSmartIri, - throw InconsistentRepositoryDataException( - s"User: $userIri has no 'username' defined." - ) - ) - .head - .asInstanceOf[StringLiteralV2] - .value, - email = propsMap - .getOrElse( - OntologyConstants.KnoraAdmin.Email.toSmartIri, - throw InconsistentRepositoryDataException(s"User: $userIri has no 'email' defined.") - ) - .head - .asInstanceOf[StringLiteralV2] - .value, - givenName = propsMap - .getOrElse( - OntologyConstants.KnoraAdmin.GivenName.toSmartIri, - throw InconsistentRepositoryDataException( - s"User: $userIri has no 'givenName' defined." - ) - ) - .head - .asInstanceOf[StringLiteralV2] - .value, - familyName = propsMap - .getOrElse( - OntologyConstants.KnoraAdmin.FamilyName.toSmartIri, - throw InconsistentRepositoryDataException( - s"User: $userIri has no 'familyName' defined." - ) - ) - .head - .asInstanceOf[StringLiteralV2] - .value, - status = propsMap - .getOrElse( - OntologyConstants.KnoraAdmin.Status.toSmartIri, - throw InconsistentRepositoryDataException( - s"User: $userIri has no 'status' defined." - ) - ) - .head - .asInstanceOf[BooleanLiteralV2] - .value, - lang = propsMap - .getOrElse( - OntologyConstants.KnoraAdmin.PreferredLanguage.toSmartIri, - throw InconsistentRepositoryDataException( - s"User: $userIri has no 'preferedLanguage' defined." - ) - ) - .head - .asInstanceOf[StringLiteralV2] - .value - ) - } + UserADM( + id = userIri.toString, + username = propsMap + .getOrElse( + OntologyConstants.KnoraAdmin.Username.toSmartIri, + throw InconsistentRepositoryDataException( + s"User: $userIri has no 'username' defined." + ) + ) + .head + .asInstanceOf[StringLiteralV2] + .value, + email = propsMap + .getOrElse( + OntologyConstants.KnoraAdmin.Email.toSmartIri, + throw InconsistentRepositoryDataException(s"User: $userIri has no 'email' defined.") + ) + .head + .asInstanceOf[StringLiteralV2] + .value, + givenName = propsMap + .getOrElse( + OntologyConstants.KnoraAdmin.GivenName.toSmartIri, + throw InconsistentRepositoryDataException( + s"User: $userIri has no 'givenName' defined." + ) + ) + .head + .asInstanceOf[StringLiteralV2] + .value, + familyName = propsMap + .getOrElse( + OntologyConstants.KnoraAdmin.FamilyName.toSmartIri, + throw InconsistentRepositoryDataException( + s"User: $userIri has no 'familyName' defined." + ) + ) + .head + .asInstanceOf[StringLiteralV2] + .value, + status = propsMap + .getOrElse( + OntologyConstants.KnoraAdmin.Status.toSmartIri, + throw InconsistentRepositoryDataException( + s"User: $userIri has no 'status' defined." + ) + ) + .head + .asInstanceOf[BooleanLiteralV2] + .value, + lang = propsMap + .getOrElse( + OntologyConstants.KnoraAdmin.PreferredLanguage.toSmartIri, + throw InconsistentRepositoryDataException( + s"User: $userIri has no 'preferedLanguage' defined." + ) + ) + .head + .asInstanceOf[StringLiteralV2] + .value + ) + } } yield users.sorted @@ -260,17 +260,17 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde ): Future[UsersGetResponseADM] = for { maybeUsersListToReturn <- getAllUserADM( - userInformationType = userInformationType, - featureFactoryConfig = featureFactoryConfig, - requestingUser = requestingUser - ) + userInformationType = userInformationType, + featureFactoryConfig = featureFactoryConfig, + requestingUser = requestingUser + ) result = maybeUsersListToReturn match { - case users: Seq[UserADM] if users.nonEmpty => - UsersGetResponseADM(users = users) - case _ => - throw NotFoundException(s"No users found") - } + case users: Seq[UserADM] if users.nonEmpty => + UsersGetResponseADM(users = users) + case _ => + throw NotFoundException(s"No users found") + } } yield result /** @@ -354,16 +354,16 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde ): Future[UserResponseADM] = for { maybeUserADM <- getSingleUserADM( - identifier = identifier, - userInformationType = userInformationType, - featureFactoryConfig = featureFactoryConfig, - requestingUser = requestingUser - ) + identifier = identifier, + userInformationType = userInformationType, + featureFactoryConfig = featureFactoryConfig, + requestingUser = requestingUser + ) result = maybeUserADM match { - case Some(user) => UserResponseADM(user = user) - case None => throw NotFoundException(s"User '${identifier.value}' not found") - } + case Some(user) => UserResponseADM(user = user) + case None => throw NotFoundException(s"User '${identifier.value}' not found") + } } yield result /** @@ -401,53 +401,53 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde for { // check if the requesting user is allowed to perform updates (i.e. requesting updates own information or is system admin) _ <- Future( - if (!requestingUser.id.equalsIgnoreCase(userIri) && !requestingUser.permissions.isSystemAdmin) { - throw ForbiddenException( - "User information can only be changed by the user itself or a system administrator" - ) - } - ) + if (!requestingUser.id.equalsIgnoreCase(userIri) && !requestingUser.permissions.isSystemAdmin) { + throw ForbiddenException( + "User information can only be changed by the user itself or a system administrator" + ) + } + ) // get current user information currentUserInformation: Option[UserADM] <- getSingleUserADM( - identifier = UserIdentifierADM(maybeIri = Some(userIri)), - userInformationType = UserInformationTypeADM.Full, - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser - ) + identifier = UserIdentifierADM(maybeIri = Some(userIri)), + userInformationType = UserInformationTypeADM.Full, + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser + ) // check if email is unique in case of a change email request emailTaken: Boolean <- userByEmailExists(userUpdateBasicInformationPayload.email, Some(currentUserInformation.get.email)) _ = if (emailTaken) { - throw DuplicateValueException( - s"User with the email '${userUpdateBasicInformationPayload.email.get.value}' already exists" - ) - } + throw DuplicateValueException( + s"User with the email '${userUpdateBasicInformationPayload.email.get.value}' already exists" + ) + } // check if username is unique in case of a change username request usernameTaken: Boolean <- userByUsernameExists(userUpdateBasicInformationPayload.username, Some(currentUserInformation.get.username)) _ = if (usernameTaken) { - throw DuplicateValueException( - s"User with the username '${userUpdateBasicInformationPayload.username.get.value}' already exists" - ) - } + throw DuplicateValueException( + s"User with the username '${userUpdateBasicInformationPayload.username.get.value}' already exists" + ) + } // send change request as SystemUser result <- updateUserADM( - userIri = userIri, - userUpdatePayload = UserChangeRequestADM( - username = userUpdateBasicInformationPayload.username, - email = userUpdateBasicInformationPayload.email, - givenName = userUpdateBasicInformationPayload.givenName, - familyName = userUpdateBasicInformationPayload.familyName, - lang = userUpdateBasicInformationPayload.lang - ), - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser, - apiRequestID = apiRequestID - ) + userIri = userIri, + userUpdatePayload = UserChangeRequestADM( + username = userUpdateBasicInformationPayload.username, + email = userUpdateBasicInformationPayload.email, + givenName = userUpdateBasicInformationPayload.givenName, + familyName = userUpdateBasicInformationPayload.familyName, + lang = userUpdateBasicInformationPayload.lang + ), + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser, + apiRequestID = apiRequestID + ) } yield result for { @@ -495,42 +495,42 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde for { // check if the requesting user is allowed to perform updates (i.e. requesting updates own information or is system admin) _ <- Future( - if (!requestingUser.id.equalsIgnoreCase(userIri) && !requestingUser.permissions.isSystemAdmin) { - throw ForbiddenException( - "User's password can only be changed by the user itself or a system administrator" - ) - } - ) + if (!requestingUser.id.equalsIgnoreCase(userIri) && !requestingUser.permissions.isSystemAdmin) { + throw ForbiddenException( + "User's password can only be changed by the user itself or a system administrator" + ) + } + ) // check if supplied password matches requesting user's password _ = if (!requestingUser.passwordMatch(userUpdatePasswordPayload.requesterPassword.value)) { - throw ForbiddenException("The supplied password does not match the requesting user's password.") - } + throw ForbiddenException("The supplied password does not match the requesting user's password.") + } // hash the new password encoder = new BCryptPasswordEncoder(settings.bcryptPasswordStrength) newHashedPassword = Password - .make(encoder.encode(userUpdatePasswordPayload.newPassword.value)) - .fold(e => throw e.head, value => value) + .make(encoder.encode(userUpdatePasswordPayload.newPassword.value)) + .fold(e => throw e.head, value => value) // update the users password as SystemUser result <- updateUserPasswordADM( - userIri = userIri, - password = newHashedPassword, - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser, - apiRequestID = apiRequestID - ) + userIri = userIri, + password = newHashedPassword, + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser, + apiRequestID = apiRequestID + ) } yield result for { // run the change password task with an IRI lock taskResult <- IriLocker.runWithIriLock( - apiRequestID, - userIri, - () => changePasswordTask(userIri, userUpdatePasswordPayload, requestingUser, apiRequestID) - ) + apiRequestID, + userIri, + () => changePasswordTask(userIri, userUpdatePasswordPayload, requestingUser, apiRequestID) + ) } yield taskResult } @@ -576,22 +576,22 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde // create the update request result <- updateUserADM( - userIri = userIri, - userUpdatePayload = UserChangeRequestADM(status = Some(status)), - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser, - apiRequestID = apiRequestID - ) + userIri = userIri, + userUpdatePayload = UserChangeRequestADM(status = Some(status)), + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser, + apiRequestID = apiRequestID + ) } yield result for { // run the change status task with an IRI lock taskResult <- IriLocker.runWithIriLock( - apiRequestID, - userIri, - () => changeUserStatusTask(userIri, status, requestingUser, apiRequestID) - ) + apiRequestID, + userIri, + () => changeUserStatusTask(userIri, status, requestingUser, apiRequestID) + ) } yield taskResult } @@ -635,12 +635,12 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde // create the update request result <- updateUserADM( - userIri = userIri, - userUpdatePayload = UserChangeRequestADM(systemAdmin = Some(systemAdmin)), - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser, - apiRequestID = apiRequestID - ) + userIri = userIri, + userUpdatePayload = UserChangeRequestADM(systemAdmin = Some(systemAdmin)), + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser, + apiRequestID = apiRequestID + ) } yield result @@ -669,16 +669,16 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde ): Future[Seq[ProjectADM]] = for { maybeUser <- getSingleUserADM( - identifier = UserIdentifierADM(maybeIri = Some(userIri)), - userInformationType = UserInformationTypeADM.Full, - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser - ) + identifier = UserIdentifierADM(maybeIri = Some(userIri)), + userInformationType = UserInformationTypeADM.Full, + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser + ) result = maybeUser match { - case Some(userADM) => userADM.projects - case None => Seq.empty[ProjectADM] - } + case Some(userADM) => userADM.projects + case None => Seq.empty[ProjectADM] + } } yield result @@ -697,17 +697,15 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde for { userExists <- userExists(userIri) _ = if (!userExists) { - throw BadRequestException(s"User $userIri does not exist.") - } + throw BadRequestException(s"User $userIri does not exist.") + } projects: Seq[ProjectADM] <- userProjectMembershipsGetADM( - userIri = userIri, - featureFactoryConfig = featureFactoryConfig, - requestingUser = requestingUser - ) - - result = UserProjectMembershipsGetResponseADM(projects = projects) - } yield result + userIri = userIri, + featureFactoryConfig = featureFactoryConfig, + requestingUser = requestingUser + ) + } yield UserProjectMembershipsGetResponseADM(projects) /** * Adds a user to a project. @@ -751,18 +749,18 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde // check if user exists userExists <- userExists(userIri) - _ = if (!userExists) throw NotFoundException(s"The user $userIri does not exist.") + _ = if (!userExists) throw NotFoundException(s"The user $userIri does not exist.") // check if project exists projectExists <- projectExists(projectIri) - _ = if (!projectExists) throw NotFoundException(s"The project $projectIri does not exist.") + _ = if (!projectExists) throw NotFoundException(s"The project $projectIri does not exist.") // get users current project membership list currentProjectMemberships <- userProjectMembershipsGetRequestADM( - userIri = userIri, - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser - ) + userIri = userIri, + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser + ) currentProjectMembershipIris: Seq[IRI] = currentProjectMemberships.projects.map(_.id) @@ -777,22 +775,22 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde } // create the update request - result <- updateUserADM( - userIri = userIri, - userUpdatePayload = UserChangeRequestADM(projects = Some(updatedProjectMembershipIris)), - featureFactoryConfig = featureFactoryConfig, - requestingUser = requestingUser, - apiRequestID = apiRequestID - ) - } yield result + updateUseResult <- updateUserADM( + userIri = userIri, + userUpdatePayload = UserChangeRequestADM(projects = Some(updatedProjectMembershipIris)), + featureFactoryConfig = featureFactoryConfig, + requestingUser = requestingUser, + apiRequestID = apiRequestID + ) + } yield updateUseResult for { // run the task with an IRI lock taskResult <- IriLocker.runWithIriLock( - apiRequestID, - userIri, - () => userProjectMembershipAddRequestTask(userIri, projectIri, requestingUser, apiRequestID) - ) + apiRequestID, + userIri, + () => userProjectMembershipAddRequestTask(userIri, projectIri, requestingUser, apiRequestID) + ) } yield taskResult } @@ -837,18 +835,18 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde // check if user exists userExists <- userExists(userIri) - _ = if (!userExists) throw NotFoundException(s"The user $userIri does not exist.") + _ = if (!userExists) throw NotFoundException(s"The user $userIri does not exist.") // check if project exists projectExists <- projectExists(projectIri) - _ = if (!projectExists) throw NotFoundException(s"The project $projectIri does not exist.") + _ = if (!projectExists) throw NotFoundException(s"The project $projectIri does not exist.") // get users current project membership list currentProjectMemberships <- userProjectMembershipsGetADM( - userIri = userIri, - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser - ) + userIri = userIri, + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser + ) currentProjectMembershipIris = currentProjectMemberships.map(_.id) // check if user is not already a member and if he is then remove the project from to list @@ -863,21 +861,21 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde // create the update request by using the SystemUser result <- updateUserADM( - userIri = userIri, - userUpdatePayload = UserChangeRequestADM(projects = Some(updatedProjectMembershipIris)), - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser, - apiRequestID = apiRequestID - ) + userIri = userIri, + userUpdatePayload = UserChangeRequestADM(projects = Some(updatedProjectMembershipIris)), + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser, + apiRequestID = apiRequestID + ) } yield result for { // run the task with an IRI lock taskResult <- IriLocker.runWithIriLock( - apiRequestID, - userIri, - () => userProjectMembershipRemoveRequestTask(userIri, projectIri, requestingUser, apiRequestID) - ) + apiRequestID, + userIri, + () => userProjectMembershipRemoveRequestTask(userIri, projectIri, requestingUser, apiRequestID) + ) } yield taskResult } @@ -900,35 +898,36 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde // ToDo: this is a bit of a hack since the ProjectAdmin group doesn't really exist. for { sparqlQueryString <- Future( - org.knora.webapi.messages.twirl.queries.sparql.v1.txt - .getUserByIri( - triplestore = settings.triplestoreType, - userIri = userIri - ) - .toString() - ) + org.knora.webapi.messages.twirl.queries.sparql.v1.txt + .getUserByIri( + triplestore = settings.triplestoreType, + userIri = userIri + ) + .toString() + ) userDataQueryResponse <- (storeManager ? SparqlSelectRequest(sparqlQueryString)).mapTo[SparqlSelectResult] groupedUserData: Map[String, Seq[String]] = userDataQueryResponse.results.bindings.groupBy(_.rowMap("p")).map { - case (predicate, rows) => predicate -> rows.map(_.rowMap("o")) - } + case (predicate, rows) => predicate -> rows.map(_.rowMap("o")) + } /* the projects the user is member of */ projectIris: Seq[IRI] = groupedUserData.get(OntologyConstants.KnoraAdmin.IsInProjectAdminGroup) match { - case Some(projects) => projects - case None => Seq.empty[IRI] - } + case Some(projects) => projects + case None => Seq.empty[IRI] + } maybeProjectFutures: Seq[Future[Option[ProjectADM]]] = projectIris.map { projectIri => - (responderManager ? ProjectGetADM( - identifier = ProjectIdentifierADM(maybeIri = Some(projectIri)), - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser - )).mapTo[Option[ProjectADM]] - } + (responderManager ? ProjectGetADM( + identifier = + ProjectIdentifierADM(maybeIri = Some(projectIri)), + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser + )).mapTo[Option[ProjectADM]] + } maybeProjects: Seq[Option[ProjectADM]] <- Future.sequence(maybeProjectFutures) - projects: Seq[ProjectADM] = maybeProjects.flatten + projects: Seq[ProjectADM] = maybeProjects.flatten } yield projects @@ -953,15 +952,15 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde for { userExists <- userExists(userIri) _ = if (!userExists) { - throw BadRequestException(s"User $userIri does not exist.") - } + throw BadRequestException(s"User $userIri does not exist.") + } projects: Seq[ProjectADM] <- userProjectAdminMembershipsGetADM( - userIri = userIri, - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser, - apiRequestID = apiRequestID - ) + userIri = userIri, + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser, + apiRequestID = apiRequestID + ) } yield UserProjectAdminMembershipsGetResponseADM(projects = projects) /** @@ -1004,19 +1003,19 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde // check if user exists userExists <- userExists(userIri) - _ = if (!userExists) throw NotFoundException(s"The user $userIri does not exist.") + _ = if (!userExists) throw NotFoundException(s"The user $userIri does not exist.") // check if project exists projectExists <- projectExists(projectIri) - _ = if (!projectExists) throw NotFoundException(s"The project $projectIri does not exist.") + _ = if (!projectExists) throw NotFoundException(s"The project $projectIri does not exist.") // get users current project membership list currentProjectAdminMemberships <- userProjectAdminMembershipsGetADM( - userIri = userIri, - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser, - apiRequestID = apiRequestID - ) + userIri = userIri, + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser, + apiRequestID = apiRequestID + ) currentProjectAdminMembershipIris: Seq[IRI] = currentProjectAdminMemberships.map(_.id) @@ -1032,21 +1031,21 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde // create the update request result <- updateUserADM( - userIri = userIri, - userUpdatePayload = UserChangeRequestADM(projectsAdmin = Some(updatedProjectAdminMembershipIris)), - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser, - apiRequestID = apiRequestID - ) + userIri = userIri, + userUpdatePayload = UserChangeRequestADM(projectsAdmin = Some(updatedProjectAdminMembershipIris)), + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser, + apiRequestID = apiRequestID + ) } yield result for { // run the task with an IRI lock taskResult <- IriLocker.runWithIriLock( - apiRequestID, - userIri, - () => userProjectAdminMembershipAddRequestTask(userIri, projectIri, requestingUser, apiRequestID) - ) + apiRequestID, + userIri, + () => userProjectAdminMembershipAddRequestTask(userIri, projectIri, requestingUser, apiRequestID) + ) } yield taskResult } @@ -1091,19 +1090,19 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde // check if user exists userExists <- userExists(userIri) - _ = if (!userExists) throw NotFoundException(s"The user $userIri does not exist.") + _ = if (!userExists) throw NotFoundException(s"The user $userIri does not exist.") // check if project exists projectExists <- projectExists(projectIri) - _ = if (!projectExists) throw NotFoundException(s"The project $projectIri does not exist.") + _ = if (!projectExists) throw NotFoundException(s"The project $projectIri does not exist.") // get users current project membership list currentProjectAdminMemberships <- userProjectAdminMembershipsGetADM( - userIri = userIri, - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser, - apiRequestID = apiRequestID - ) + userIri = userIri, + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser, + apiRequestID = apiRequestID + ) currentProjectAdminMembershipIris: Seq[IRI] = currentProjectAdminMemberships.map(_.id) @@ -1119,12 +1118,12 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde // create the update request result <- updateUserADM( - userIri = userIri, - userUpdatePayload = UserChangeRequestADM(projectsAdmin = Some(updatedProjectAdminMembershipIris)), - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser, - apiRequestID = apiRequestID - ) + userIri = userIri, + userUpdatePayload = UserChangeRequestADM(projectsAdmin = Some(updatedProjectAdminMembershipIris)), + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser, + apiRequestID = apiRequestID + ) } yield result for { @@ -1153,23 +1152,23 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde ): Future[Seq[GroupADM]] = for { maybeUserADM: Option[UserADM] <- getSingleUserADM( - identifier = UserIdentifierADM(maybeIri = Some(userIri)), - userInformationType = UserInformationTypeADM.Full, - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser - ) + identifier = UserIdentifierADM(maybeIri = Some(userIri)), + userInformationType = UserInformationTypeADM.Full, + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser + ) groups: Seq[GroupADM] = maybeUserADM match { - case Some(user) => - log.debug( - "userGroupMembershipsGetADM - user found. Returning his groups: {}.", - user.groups - ) - user.groups - case None => - log.debug("userGroupMembershipsGetADM - user not found. Returning empty seq.") - Seq.empty[GroupADM] - } + case Some(user) => + log.debug( + "userGroupMembershipsGetADM - user found. Returning his groups: {}.", + user.groups + ) + user.groups + case None => + log.debug("userGroupMembershipsGetADM - user not found. Returning empty seq.") + Seq.empty[GroupADM] + } } yield groups @@ -1188,10 +1187,10 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde ): Future[UserGroupMembershipsGetResponseADM] = for { groups: Seq[GroupADM] <- userGroupMembershipsGetADM( - userIri = userIri, - featureFactoryConfig = featureFactoryConfig, - requestingUser = requestingUser - ) + userIri = userIri, + featureFactoryConfig = featureFactoryConfig, + requestingUser = requestingUser + ) } yield UserGroupMembershipsGetResponseADM(groups = groups) /** @@ -1224,40 +1223,40 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde for { // check if user exists maybeUser <- getSingleUserADM( - UserIdentifierADM(maybeIri = Some(userIri)), - UserInformationTypeADM.Full, - featureFactoryConfig = featureFactoryConfig, - KnoraSystemInstances.Users.SystemUser, - skipCache = true - ) + UserIdentifierADM(maybeIri = Some(userIri)), + UserInformationTypeADM.Full, + featureFactoryConfig = featureFactoryConfig, + KnoraSystemInstances.Users.SystemUser, + skipCache = true + ) userToChange: UserADM = maybeUser match { - case Some(user) => user - case None => throw NotFoundException(s"The user $userIri does not exist.") - } + case Some(user) => user + case None => throw NotFoundException(s"The user $userIri does not exist.") + } // check if group exists groupExists <- groupExists(groupIri) - _ = if (!groupExists) throw NotFoundException(s"The group $groupIri does not exist.") + _ = if (!groupExists) throw NotFoundException(s"The group $groupIri does not exist.") // get group's info. we need the project IRI. maybeGroupADM <- (responderManager ? GroupGetADM( - groupIri = groupIri, - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser - )).mapTo[Option[GroupADM]] + groupIri = groupIri, + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser + )).mapTo[Option[GroupADM]] projectIri = maybeGroupADM - .getOrElse(throw InconsistentRepositoryDataException(s"Group $groupIri does not exist")) - .project - .id + .getOrElse(throw InconsistentRepositoryDataException(s"Group $groupIri does not exist")) + .project + .id // check if the requesting user is allowed to perform updates (i.e. project or system administrator) _ = if (!requestingUser.permissions.isProjectAdmin(projectIri) && !requestingUser.permissions.isSystemAdmin) { - throw ForbiddenException( - "User's group membership can only be changed by a project or system administrator" - ) - } + throw ForbiddenException( + "User's group membership can only be changed by a project or system administrator" + ) + } // get users current group membership list currentGroupMemberships = userToChange.groups @@ -1274,21 +1273,21 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde // create the update request result <- updateUserADM( - userIri = userIri, - userUpdatePayload = UserChangeRequestADM(groups = Some(updatedGroupMembershipIris)), - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser, - apiRequestID = apiRequestID - ) + userIri = userIri, + userUpdatePayload = UserChangeRequestADM(groups = Some(updatedGroupMembershipIris)), + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser, + apiRequestID = apiRequestID + ) } yield result for { // run the task with an IRI lock taskResult <- IriLocker.runWithIriLock( - apiRequestID, - userIri, - () => userGroupMembershipAddRequestTask(userIri, groupIri, requestingUser, apiRequestID) - ) + apiRequestID, + userIri, + () => userGroupMembershipAddRequestTask(userIri, groupIri, requestingUser, apiRequestID) + ) } yield taskResult } @@ -1313,23 +1312,23 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde for { // check if user exists userExists <- userExists(userIri) - _ = if (!userExists) throw NotFoundException(s"The user $userIri does not exist.") + _ = if (!userExists) throw NotFoundException(s"The user $userIri does not exist.") // check if group exists projectExists <- groupExists(groupIri) - _ = if (!projectExists) throw NotFoundException(s"The group $groupIri does not exist.") + _ = if (!projectExists) throw NotFoundException(s"The group $groupIri does not exist.") // get group's info. we need the project IRI. maybeGroupADM <- (responderManager ? GroupGetADM( - groupIri = groupIri, - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser - )).mapTo[Option[GroupADM]] + groupIri = groupIri, + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser + )).mapTo[Option[GroupADM]] projectIri = maybeGroupADM - .getOrElse(throw InconsistentRepositoryDataException(s"Group $groupIri does not exist")) - .project - .id + .getOrElse(throw InconsistentRepositoryDataException(s"Group $groupIri does not exist")) + .project + .id // check if the requesting user is allowed to perform updates (i.e. is project or system admin) _ = @@ -1342,10 +1341,10 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde // get users current project membership list currentGroupMemberships <- userGroupMembershipsGetRequestADM( - userIri = userIri, - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser - ) + userIri = userIri, + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser + ) currentGroupMembershipIris: Seq[IRI] = currentGroupMemberships.groups.map(_.id) @@ -1359,21 +1358,21 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde // create the update request result <- updateUserADM( - userIri = userIri, - userUpdatePayload = UserChangeRequestADM(groups = Some(updatedGroupMembershipIris)), - featureFactoryConfig = featureFactoryConfig, - requestingUser = requestingUser, - apiRequestID = apiRequestID - ) + userIri = userIri, + userUpdatePayload = UserChangeRequestADM(groups = Some(updatedGroupMembershipIris)), + featureFactoryConfig = featureFactoryConfig, + requestingUser = requestingUser, + apiRequestID = apiRequestID + ) } yield result for { // run the task with an IRI lock taskResult <- IriLocker.runWithIriLock( - apiRequestID, - userIri, - () => userGroupMembershipRemoveRequestTask(userIri, groupIri, requestingUser, apiRequestID) - ) + apiRequestID, + userIri, + () => userGroupMembershipRemoveRequestTask(userIri, groupIri, requestingUser, apiRequestID) + ) } yield taskResult } @@ -1409,108 +1408,111 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde } for { + + // get current user maybeCurrentUser <- getSingleUserADM( - identifier = UserIdentifierADM(maybeIri = Some(userIri)), - featureFactoryConfig = featureFactoryConfig, - requestingUser = requestingUser, - userInformationType = UserInformationTypeADM.Full, - skipCache = true - ) + identifier = UserIdentifierADM(maybeIri = Some(userIri)), + featureFactoryConfig = featureFactoryConfig, + requestingUser = requestingUser, + userInformationType = UserInformationTypeADM.Full, + skipCache = true + ) _ = if (maybeCurrentUser.isEmpty) { - throw NotFoundException(s"User '$userIri' not found. Aborting update request.") - } - - // we are changing the user, so lets get rid of the cached copy - _ = invalidateCachedUserADM(maybeCurrentUser) + throw NotFoundException(s"User '$userIri' not found. Aborting update request.") + } /* Update the user */ maybeChangedUsername = userUpdatePayload.username match { - case Some(username) => Some(username.value) - case None => None - } + case Some(username) => Some(username.value) + case None => None + } maybeChangedEmail = userUpdatePayload.email match { - case Some(email) => Some(email.value) - case None => None - } + case Some(email) => Some(email.value) + case None => None + } maybeChangedGivenName = userUpdatePayload.givenName match { - case Some(givenName) => - Some( - stringFormatter.toSparqlEncodedString( - givenName.value, - throw BadRequestException( - s"The supplied given name: '${givenName.value}' is not valid." - ) - ) - ) - case None => None - } + case Some(givenName) => + Some( + stringFormatter.toSparqlEncodedString( + givenName.value, + throw BadRequestException( + s"The supplied given name: '${givenName.value}' is not valid." + ) + ) + ) + case None => None + } maybeChangedFamilyName = userUpdatePayload.familyName match { - case Some(familyName) => - Some( - stringFormatter.toSparqlEncodedString( - familyName.value, - throw BadRequestException( - s"The supplied family name: '${familyName.value}' is not valid." - ) - ) - ) - case None => None - } + case Some(familyName) => + Some( + stringFormatter.toSparqlEncodedString( + familyName.value, + throw BadRequestException( + s"The supplied family name: '${familyName.value}' is not valid." + ) + ) + ) + case None => None + } maybeChangedStatus = userUpdatePayload.status match { - case Some(status) => Some(status.value) - case None => None - } + case Some(status) => Some(status.value) + case None => None + } maybeChangedLang = userUpdatePayload.lang match { - case Some(lang) => Some(lang.value) - case None => None - } + case Some(lang) => Some(lang.value) + case None => None + } maybeChangedProjects = userUpdatePayload.projects match { - case Some(projects) => Some(projects) - case None => None - } + case Some(projects) => Some(projects) + case None => None + } maybeChangedProjectsAdmin = userUpdatePayload.projectsAdmin match { - case Some(projectsAdmin) => Some(projectsAdmin) - case None => None - } + case Some(projectsAdmin) => Some(projectsAdmin) + case None => None + } maybeChangedGroups = userUpdatePayload.groups match { - case Some(groups) => Some(groups) - case None => None - } + case Some(groups) => Some(groups) + case None => None + } maybeChangedSystemAdmin = userUpdatePayload.systemAdmin match { - case Some(systemAdmin) => Some(systemAdmin.value) - case None => None - } + case Some(systemAdmin) => Some(systemAdmin.value) + case None => None + } updateUserSparqlString <- Future( - org.knora.webapi.messages.twirl.queries.sparql.admin.txt - .updateUser( - adminNamedGraphIri = OntologyConstants.NamedGraphs.AdminNamedGraph, - triplestore = settings.triplestoreType, - userIri = userIri, - maybeUsername = maybeChangedUsername, - maybeEmail = maybeChangedEmail, - maybeGivenName = maybeChangedGivenName, - maybeFamilyName = maybeChangedFamilyName, - maybeStatus = maybeChangedStatus, - maybeLang = maybeChangedLang, - maybeProjects = maybeChangedProjects, - maybeProjectsAdmin = maybeChangedProjectsAdmin, - maybeGroups = maybeChangedGroups, - maybeSystemAdmin = maybeChangedSystemAdmin - ) - .toString - ) + org.knora.webapi.messages.twirl.queries.sparql.admin.txt + .updateUser( + adminNamedGraphIri = OntologyConstants.NamedGraphs.AdminNamedGraph, + triplestore = settings.triplestoreType, + userIri = userIri, + maybeUsername = maybeChangedUsername, + maybeEmail = maybeChangedEmail, + maybeGivenName = maybeChangedGivenName, + maybeFamilyName = maybeChangedFamilyName, + maybeStatus = maybeChangedStatus, + maybeLang = maybeChangedLang, + maybeProjects = maybeChangedProjects, + maybeProjectsAdmin = maybeChangedProjectsAdmin, + maybeGroups = maybeChangedGroups, + maybeSystemAdmin = maybeChangedSystemAdmin + ) + .toString + ) - updateResult <- (storeManager ? SparqlUpdateRequest(updateUserSparqlString)).mapTo[SparqlUpdateResponse] + // we are changing the user, so lets get rid of the cached copy + _ <- invalidateCachedUserADM(maybeCurrentUser) + + // write the updated user to the triplestore + _ <- (storeManager ? SparqlUpdateRequest(updateUserSparqlString)).mapTo[SparqlUpdateResponse] - /* Verify that the user was updated. */ + /* Verify that the user was updated */ maybeUpdatedUserADM <- getSingleUserADM( - identifier = UserIdentifierADM(maybeIri = Some(userIri)), - featureFactoryConfig = featureFactoryConfig, - requestingUser = requestingUser, - userInformationType = UserInformationTypeADM.Full, - skipCache = true - ) + identifier = UserIdentifierADM(maybeIri = Some(userIri)), + featureFactoryConfig = featureFactoryConfig, + requestingUser = requestingUser, + userInformationType = UserInformationTypeADM.Full, + skipCache = true + ) updatedUserADM: UserADM = maybeUpdatedUserADM.getOrElse( @@ -1518,65 +1520,71 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde ) _ = if (userUpdatePayload.username.isDefined) { - if (updatedUserADM.username != userUpdatePayload.username.get.value) - throw UpdateNotPerformedException( - "User's 'username' was not updated. Please report this as a possible bug." - ) - } + if (updatedUserADM.username != userUpdatePayload.username.get.value) + throw UpdateNotPerformedException( + "User's 'username' was not updated. Please report this as a possible bug." + ) + } _ = if (userUpdatePayload.email.isDefined) { - if (updatedUserADM.email != userUpdatePayload.email.get.value) - throw UpdateNotPerformedException("User's 'email' was not updated. Please report this as a possible bug.") - } + if (updatedUserADM.email != userUpdatePayload.email.get.value) + throw UpdateNotPerformedException("User's 'email' was not updated. Please report this as a possible bug.") + } _ = if (userUpdatePayload.givenName.isDefined) { - if (updatedUserADM.givenName != userUpdatePayload.givenName.get.value) - throw UpdateNotPerformedException( - "User's 'givenName' was not updated. Please report this as a possible bug." - ) - } + if (updatedUserADM.givenName != userUpdatePayload.givenName.get.value) + throw UpdateNotPerformedException( + "User's 'givenName' was not updated. Please report this as a possible bug." + ) + } _ = if (userUpdatePayload.familyName.isDefined) { - if (updatedUserADM.familyName != userUpdatePayload.familyName.get.value) - throw UpdateNotPerformedException( - "User's 'familyName' was not updated. Please report this as a possible bug." - ) - } + if (updatedUserADM.familyName != userUpdatePayload.familyName.get.value) + throw UpdateNotPerformedException( + "User's 'familyName' was not updated. Please report this as a possible bug." + ) + } _ = if (userUpdatePayload.status.isDefined) { - if (updatedUserADM.status != userUpdatePayload.status.get.value) - throw UpdateNotPerformedException( - "User's 'status' was not updated. Please report this as a possible bug." - ) - } + if (updatedUserADM.status != userUpdatePayload.status.get.value) + throw UpdateNotPerformedException( + "User's 'status' was not updated. Please report this as a possible bug." + ) + } _ = if (userUpdatePayload.lang.isDefined) { - if (updatedUserADM.lang != userUpdatePayload.lang.get.value) - throw UpdateNotPerformedException("User's 'lang' was not updated. Please report this as a possible bug.") - } + if (updatedUserADM.lang != userUpdatePayload.lang.get.value) + throw UpdateNotPerformedException("User's 'lang' was not updated. Please report this as a possible bug.") + } _ = if (userUpdatePayload.projects.isDefined) { - if (updatedUserADM.projects.map(_.id).sorted != userUpdatePayload.projects.get.sorted) { - throw UpdateNotPerformedException( - "User's 'project' memberships were not updated. Please report this as a possible bug." - ) - } - } + if (updatedUserADM.projects.map(_.id).sorted != userUpdatePayload.projects.get.sorted) { + throw UpdateNotPerformedException( + "User's 'project' memberships were not updated. Please report this as a possible bug." + ) + } + } _ = if (userUpdatePayload.systemAdmin.isDefined) { - if (updatedUserADM.permissions.isSystemAdmin != userUpdatePayload.systemAdmin.get.value) - throw UpdateNotPerformedException( - "User's 'isInSystemAdminGroup' status was not updated. Please report this as a possible bug." - ) - } + if (updatedUserADM.permissions.isSystemAdmin != userUpdatePayload.systemAdmin.get.value) + throw UpdateNotPerformedException( + "User's 'isInSystemAdminGroup' status was not updated. Please report this as a possible bug." + ) + } _ = if (userUpdatePayload.groups.isDefined) { - if (updatedUserADM.groups.map(_.id).sorted != userUpdatePayload.groups.get.sorted) - throw UpdateNotPerformedException( - "User's 'group' memberships were not updated. Please report this as a possible bug." - ) - } + if (updatedUserADM.groups.map(_.id).sorted != userUpdatePayload.groups.get.sorted) + throw UpdateNotPerformedException( + "User's 'group' memberships were not updated. Please report this as a possible bug." + ) + } + + _ <- writeUserADMToCache( + maybeUpdatedUserADM.getOrElse( + throw UpdateNotPerformedException("User was not updated. Please report this as a possible bug.") + ) + ) } yield UserOperationResponseADM(updatedUserADM.ofType(UserInformationTypeADM.Restricted)) } @@ -1612,41 +1620,41 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde for { maybeCurrentUser <- getSingleUserADM( - identifier = UserIdentifierADM(maybeIri = Some(userIri)), - featureFactoryConfig = featureFactoryConfig, - requestingUser = requestingUser, - userInformationType = UserInformationTypeADM.Full, - skipCache = true - ) + identifier = UserIdentifierADM(maybeIri = Some(userIri)), + featureFactoryConfig = featureFactoryConfig, + requestingUser = requestingUser, + userInformationType = UserInformationTypeADM.Full, + skipCache = true + ) _ = if (maybeCurrentUser.isEmpty) { - throw NotFoundException(s"User '$userIri' not found. Aborting update request.") - } + throw NotFoundException(s"User '$userIri' not found. Aborting update request.") + } // we are changing the user, so lets get rid of the cached copy _ = invalidateCachedUserADM(maybeCurrentUser) // update the password updateUserSparqlString <- Future( - org.knora.webapi.messages.twirl.queries.sparql.admin.txt - .updateUserPassword( - adminNamedGraphIri = OntologyConstants.NamedGraphs.AdminNamedGraph, - triplestore = settings.triplestoreType, - userIri = userIri, - newPassword = password.value - ) - .toString - ) + org.knora.webapi.messages.twirl.queries.sparql.admin.txt + .updateUserPassword( + adminNamedGraphIri = OntologyConstants.NamedGraphs.AdminNamedGraph, + triplestore = settings.triplestoreType, + userIri = userIri, + newPassword = password.value + ) + .toString + ) updateResult <- (storeManager ? SparqlUpdateRequest(updateUserSparqlString)).mapTo[SparqlUpdateResponse] /* Verify that the password was updated. */ maybeUpdatedUserADM <- getSingleUserADM( - identifier = UserIdentifierADM(maybeIri = Some(userIri)), - featureFactoryConfig = featureFactoryConfig, - requestingUser = requestingUser, - userInformationType = UserInformationTypeADM.Full, - skipCache = true - ) + identifier = UserIdentifierADM(maybeIri = Some(userIri)), + featureFactoryConfig = featureFactoryConfig, + requestingUser = requestingUser, + userInformationType = UserInformationTypeADM.Full, + skipCache = true + ) updatedUserADM: UserADM = maybeUpdatedUserADM.getOrElse( @@ -1654,7 +1662,7 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde ) _ = if (updatedUserADM.password.get != password.value) - throw UpdateNotPerformedException("User's password was not updated. Please report this as a possible bug.") + throw UpdateNotPerformedException("User's password was not updated. Please report this as a possible bug.") } yield UserOperationResponseADM(updatedUserADM.ofType(UserInformationTypeADM.Restricted)) } @@ -1689,83 +1697,83 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde // check if username is unique usernameTaken: Boolean <- userByUsernameExists(Some(userCreatePayloadADM.username)) _ = if (usernameTaken) { - throw DuplicateValueException( - s"User with the username '${userCreatePayloadADM.username.value}' already exists" - ) - } + throw DuplicateValueException( + s"User with the username '${userCreatePayloadADM.username.value}' already exists" + ) + } // check if email is unique emailTaken: Boolean <- userByEmailExists(Some(userCreatePayloadADM.email)) _ = if (emailTaken) { - throw DuplicateValueException( - s"User with the email '${userCreatePayloadADM.email.value}' already exists" - ) - } + throw DuplicateValueException( + s"User with the email '${userCreatePayloadADM.email.value}' already exists" + ) + } // check the custom IRI; if not given, create an unused IRI customUserIri: Option[SmartIri] = userCreatePayloadADM.id.map(_.value.toSmartIri) - userIri: IRI <- checkOrCreateEntityIri(customUserIri, stringFormatter.makeRandomPersonIri) + userIri: IRI <- checkOrCreateEntityIri(customUserIri, stringFormatter.makeRandomPersonIri) // hash password - encoder = new BCryptPasswordEncoder(settings.bcryptPasswordStrength) + encoder = new BCryptPasswordEncoder(settings.bcryptPasswordStrength) hashedPassword = encoder.encode(userCreatePayloadADM.password.value) // Create the new user. createNewUserSparqlString = org.knora.webapi.messages.twirl.queries.sparql.admin.txt - .createNewUser( - adminNamedGraphIri = OntologyConstants.NamedGraphs.AdminNamedGraph, - triplestore = settings.triplestoreType, - userIri = userIri, - userClassIri = OntologyConstants.KnoraAdmin.User, - username = stringFormatter.toSparqlEncodedString( - userCreatePayloadADM.username.value, - errorFun = throw BadRequestException( - s"The supplied username: '${userCreatePayloadADM.username.value}' is not valid." - ) - ), - email = stringFormatter.toSparqlEncodedString( - userCreatePayloadADM.email.value, - errorFun = throw BadRequestException( - s"The supplied email: '${userCreatePayloadADM.email.value}' is not valid." - ) - ), - password = hashedPassword, - givenName = stringFormatter.toSparqlEncodedString( - userCreatePayloadADM.givenName.value, - errorFun = throw BadRequestException( - s"The supplied given name: '${userCreatePayloadADM.givenName.value}' is not valid." - ) - ), - familyName = stringFormatter.toSparqlEncodedString( - userCreatePayloadADM.familyName.value, - errorFun = throw BadRequestException( - s"The supplied family name: '${userCreatePayloadADM.familyName.value}' is not valid." - ) - ), - status = userCreatePayloadADM.status.value, - preferredLanguage = stringFormatter.toSparqlEncodedString( - userCreatePayloadADM.lang.value, - errorFun = throw BadRequestException( - s"The supplied language: '${userCreatePayloadADM.lang.value}' is not valid." - ) - ), - systemAdmin = userCreatePayloadADM.systemAdmin.value - ) - .toString + .createNewUser( + adminNamedGraphIri = OntologyConstants.NamedGraphs.AdminNamedGraph, + triplestore = settings.triplestoreType, + userIri = userIri, + userClassIri = OntologyConstants.KnoraAdmin.User, + username = stringFormatter.toSparqlEncodedString( + userCreatePayloadADM.username.value, + errorFun = throw BadRequestException( + s"The supplied username: '${userCreatePayloadADM.username.value}' is not valid." + ) + ), + email = stringFormatter.toSparqlEncodedString( + userCreatePayloadADM.email.value, + errorFun = throw BadRequestException( + s"The supplied email: '${userCreatePayloadADM.email.value}' is not valid." + ) + ), + password = hashedPassword, + givenName = stringFormatter.toSparqlEncodedString( + userCreatePayloadADM.givenName.value, + errorFun = throw BadRequestException( + s"The supplied given name: '${userCreatePayloadADM.givenName.value}' is not valid." + ) + ), + familyName = stringFormatter.toSparqlEncodedString( + userCreatePayloadADM.familyName.value, + errorFun = throw BadRequestException( + s"The supplied family name: '${userCreatePayloadADM.familyName.value}' is not valid." + ) + ), + status = userCreatePayloadADM.status.value, + preferredLanguage = stringFormatter.toSparqlEncodedString( + userCreatePayloadADM.lang.value, + errorFun = throw BadRequestException( + s"The supplied language: '${userCreatePayloadADM.lang.value}' is not valid." + ) + ), + systemAdmin = userCreatePayloadADM.systemAdmin.value + ) + .toString _ = log.debug(s"createNewUser: $createNewUserSparqlString") createNewUserResponse <- (storeManager ? SparqlUpdateRequest(createNewUserSparqlString)) - .mapTo[SparqlUpdateResponse] + .mapTo[SparqlUpdateResponse] // try to retrieve newly created user (will also add to cache) maybeNewUserADM: Option[UserADM] <- getSingleUserADM( - identifier = UserIdentifierADM(maybeIri = Some(userIri)), - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser, - userInformationType = UserInformationTypeADM.Full, - skipCache = true - ) + identifier = UserIdentifierADM(maybeIri = Some(userIri)), + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser, + userInformationType = UserInformationTypeADM.Full, + skipCache = true + ) // check to see if we could retrieve the new user newUserADM = @@ -1774,17 +1782,17 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde ) // create the user operation response - _ = log.debug("createNewUserADM - created new user: {}", newUserADM) + _ = log.debug("createNewUserADM - created new user: {}", newUserADM) userOperationResponseADM = UserOperationResponseADM(newUserADM.ofType(UserInformationTypeADM.Restricted)) } yield userOperationResponseADM for { // run user creation with an global IRI lock taskResult <- IriLocker.runWithIriLock( - apiRequestID, - USERS_GLOBAL_LOCK_IRI, - () => createNewUserTask(userCreatePayloadADM) - ) + apiRequestID, + USERS_GLOBAL_LOCK_IRI, + () => createNewUserTask(userCreatePayloadADM) + ) } yield taskResult } @@ -1839,20 +1847,20 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde ): Future[Option[UserADM]] = for { sparqlQueryString <- Future( - org.knora.webapi.messages.twirl.queries.sparql.admin.txt - .getUsers( - triplestore = settings.triplestoreType, - maybeIri = identifier.toIriOption, - maybeUsername = identifier.toUsernameOption, - maybeEmail = identifier.toEmailOption - ) - .toString() - ) + org.knora.webapi.messages.twirl.queries.sparql.admin.txt + .getUsers( + triplestore = settings.triplestoreType, + maybeIri = identifier.toIriOption, + maybeUsername = identifier.toUsernameOption, + maybeEmail = identifier.toEmailOption + ) + .toString() + ) userQueryResponse <- (storeManager ? SparqlExtendedConstructRequest( - sparql = sparqlQueryString, - featureFactoryConfig = featureFactoryConfig - )).mapTo[SparqlExtendedConstructResponse] + sparql = sparqlQueryString, + featureFactoryConfig = featureFactoryConfig + )).mapTo[SparqlExtendedConstructResponse] maybeUserADM: Option[UserADM] <- if (userQueryResponse.statements.nonEmpty) { @@ -1881,7 +1889,7 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde // log.debug("statements2UserADM - statements: {}", statements) - val userIri: IRI = statements._1.toString + val userIri: IRI = statements._1.toString val propsMap: Map[SmartIri, Seq[LiteralV2]] = statements._2 // log.debug("statements2UserADM - userIri: {}", userIri) @@ -1917,98 +1925,99 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde for { /* get the user's permission profile from the permissions responder */ permissionData <- (responderManager ? PermissionDataGetADM( - projectIris = projectIris, - groupIris = groupIris, - isInProjectAdminGroups = isInProjectAdminGroups, - isInSystemAdminGroup = isInSystemAdminGroup, - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser - )).mapTo[PermissionsDataADM] + projectIris = projectIris, + groupIris = groupIris, + isInProjectAdminGroups = isInProjectAdminGroups, + isInSystemAdminGroup = isInSystemAdminGroup, + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser + )).mapTo[PermissionsDataADM] maybeGroupFutures: Seq[Future[Option[GroupADM]]] = groupIris.map { groupIri => - (responderManager ? GroupGetADM( - groupIri = groupIri, - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser - )).mapTo[Option[GroupADM]] - } + (responderManager ? GroupGetADM( + groupIri = groupIri, + featureFactoryConfig = featureFactoryConfig, + requestingUser = KnoraSystemInstances.Users.SystemUser + )).mapTo[Option[GroupADM]] + } maybeGroups: Seq[Option[GroupADM]] <- Future.sequence(maybeGroupFutures) - groups: Seq[GroupADM] = maybeGroups.flatten + groups: Seq[GroupADM] = maybeGroups.flatten // _ = log.debug("statements2UserADM - groups: {}", MessageUtil.toSource(groups)) maybeProjectFutures: Seq[Future[Option[ProjectADM]]] = projectIris.map { projectIri => - (responderManager ? ProjectGetADM( - ProjectIdentifierADM(maybeIri = Some(projectIri)), - featureFactoryConfig = featureFactoryConfig, - requestingUser = KnoraSystemInstances.Users.SystemUser - )).mapTo[Option[ProjectADM]] - } + (responderManager ? ProjectGetADM( + ProjectIdentifierADM(maybeIri = Some(projectIri)), + featureFactoryConfig = featureFactoryConfig, + requestingUser = + KnoraSystemInstances.Users.SystemUser + )).mapTo[Option[ProjectADM]] + } maybeProjects: Seq[Option[ProjectADM]] <- Future.sequence(maybeProjectFutures) - projects: Seq[ProjectADM] = maybeProjects.flatten + projects: Seq[ProjectADM] = maybeProjects.flatten // _ = log.debug("statements2UserADM - projects: {}", MessageUtil.toSource(projects)) /* construct the user profile from the different parts */ user = UserADM( - id = userIri, - username = propsMap - .getOrElse( - OntologyConstants.KnoraAdmin.Username.toSmartIri, - throw InconsistentRepositoryDataException(s"User: $userIri has no 'username' defined.") - ) - .head - .asInstanceOf[StringLiteralV2] - .value, - email = propsMap - .getOrElse( - OntologyConstants.KnoraAdmin.Email.toSmartIri, - throw InconsistentRepositoryDataException(s"User: $userIri has no 'email' defined.") - ) - .head - .asInstanceOf[StringLiteralV2] - .value, - password = propsMap - .get(OntologyConstants.KnoraAdmin.Password.toSmartIri) - .map(_.head.asInstanceOf[StringLiteralV2].value), - token = None, - givenName = propsMap - .getOrElse( - OntologyConstants.KnoraAdmin.GivenName.toSmartIri, - throw InconsistentRepositoryDataException(s"User: $userIri has no 'givenName' defined.") - ) - .head - .asInstanceOf[StringLiteralV2] - .value, - familyName = propsMap - .getOrElse( - OntologyConstants.KnoraAdmin.FamilyName.toSmartIri, - throw InconsistentRepositoryDataException(s"User: $userIri has no 'familyName' defined.") - ) - .head - .asInstanceOf[StringLiteralV2] - .value, - status = propsMap - .getOrElse( - OntologyConstants.KnoraAdmin.Status.toSmartIri, - throw InconsistentRepositoryDataException(s"User: $userIri has no 'status' defined.") - ) - .head - .asInstanceOf[BooleanLiteralV2] - .value, - lang = propsMap - .getOrElse( - OntologyConstants.KnoraAdmin.PreferredLanguage.toSmartIri, - throw InconsistentRepositoryDataException(s"User: $userIri has no 'preferredLanguage' defined.") - ) - .head - .asInstanceOf[StringLiteralV2] - .value, - groups = groups, - projects = projects, - sessionId = None, - permissions = permissionData - ) + id = userIri, + username = propsMap + .getOrElse( + OntologyConstants.KnoraAdmin.Username.toSmartIri, + throw InconsistentRepositoryDataException(s"User: $userIri has no 'username' defined.") + ) + .head + .asInstanceOf[StringLiteralV2] + .value, + email = propsMap + .getOrElse( + OntologyConstants.KnoraAdmin.Email.toSmartIri, + throw InconsistentRepositoryDataException(s"User: $userIri has no 'email' defined.") + ) + .head + .asInstanceOf[StringLiteralV2] + .value, + password = propsMap + .get(OntologyConstants.KnoraAdmin.Password.toSmartIri) + .map(_.head.asInstanceOf[StringLiteralV2].value), + token = None, + givenName = propsMap + .getOrElse( + OntologyConstants.KnoraAdmin.GivenName.toSmartIri, + throw InconsistentRepositoryDataException(s"User: $userIri has no 'givenName' defined.") + ) + .head + .asInstanceOf[StringLiteralV2] + .value, + familyName = propsMap + .getOrElse( + OntologyConstants.KnoraAdmin.FamilyName.toSmartIri, + throw InconsistentRepositoryDataException(s"User: $userIri has no 'familyName' defined.") + ) + .head + .asInstanceOf[StringLiteralV2] + .value, + status = propsMap + .getOrElse( + OntologyConstants.KnoraAdmin.Status.toSmartIri, + throw InconsistentRepositoryDataException(s"User: $userIri has no 'status' defined.") + ) + .head + .asInstanceOf[BooleanLiteralV2] + .value, + lang = propsMap + .getOrElse( + OntologyConstants.KnoraAdmin.PreferredLanguage.toSmartIri, + throw InconsistentRepositoryDataException(s"User: $userIri has no 'preferredLanguage' defined.") + ) + .head + .asInstanceOf[StringLiteralV2] + .value, + groups = groups, + projects = projects, + sessionId = None, + permissions = permissionData + ) // _ = log.debug(s"statements2UserADM - user: {}", user.toString) result: Option[UserADM] = Some(user) @@ -2030,11 +2039,8 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde askString <- Future(org.knora.webapi.messages.twirl.queries.sparql.admin.txt.checkUserExists(userIri = userIri).toString) // _ = log.debug("userExists - query: {}", askString) - checkUserExistsResponse <- (storeManager ? SparqlAskRequest(askString)).mapTo[SparqlAskResponse] - result = checkUserExistsResponse.result - - } yield result + } yield checkUserExistsResponse.result /** * Helper method for checking if an username is already registered. @@ -2059,10 +2065,10 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde for { askString <- Future( - org.knora.webapi.messages.twirl.queries.sparql.admin.txt - .checkUserExistsByUsername(username = username.value) - .toString - ) + org.knora.webapi.messages.twirl.queries.sparql.admin.txt + .checkUserExistsByUsername(username = username.value) + .toString + ) // _ = log.debug("userExists - query: {}", askString) checkUserExistsResponse <- (storeManager ? SparqlAskRequest(askString)).mapTo[SparqlAskResponse] @@ -2092,10 +2098,10 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde for { askString <- Future( - org.knora.webapi.messages.twirl.queries.sparql.admin.txt - .checkUserExistsByEmail(email = email.value) - .toString - ) + org.knora.webapi.messages.twirl.queries.sparql.admin.txt + .checkUserExistsByEmail(email = email.value) + .toString + ) // _ = log.debug("userExists - query: {}", askString) checkUserExistsResponse <- (storeManager ? SparqlAskRequest(askString)).mapTo[SparqlAskResponse] @@ -2114,14 +2120,14 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde private def projectExists(projectIri: IRI): Future[Boolean] = for { askString <- Future( - org.knora.webapi.messages.twirl.queries.sparql.admin.txt - .checkProjectExistsByIri(projectIri = projectIri) - .toString - ) + org.knora.webapi.messages.twirl.queries.sparql.admin.txt + .checkProjectExistsByIri(projectIri = projectIri) + .toString + ) // _ = log.debug("projectExists - query: {}", askString) checkUserExistsResponse <- (storeManager ? SparqlAskRequest(askString)).mapTo[SparqlAskResponse] - result = checkUserExistsResponse.result + result = checkUserExistsResponse.result } yield result @@ -2140,7 +2146,7 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde // _ = log.debug("groupExists - query: {}", askString) checkUserExistsResponse <- (storeManager ? SparqlAskRequest(askString)).mapTo[SparqlAskResponse] - result = checkUserExistsResponse.result + result = checkUserExistsResponse.result } yield result @@ -2167,34 +2173,30 @@ class UsersResponderADM(responderData: ResponderData) extends Responder(responde * @return true if writing was successful. * @throws ApplicationCacheException when there is a problem with writing the user's profile to cache. */ - private def writeUserADMToCache(user: UserADM): Future[Boolean] = { - val result = (storeManager ? CacheServicePutUserADM(user)).mapTo[Boolean] - result.map { res => - log.debug("writeUserADMToCache - result: {}", result) - res - } - } + private def writeUserADMToCache(user: UserADM): Future[Unit] = for { + _ <- (storeManager ? CacheServicePutUserADM(user)) + _ <- Future(log.debug(s"writeUserADMToCache done - user: ${user.id}")) + } yield () /** * Removes the user from cache. */ - private def invalidateCachedUserADM(maybeUser: Option[UserADM]): Future[Boolean] = + private def invalidateCachedUserADM(maybeUser: Option[UserADM]): Future[Unit] = if (cacheServiceSettings.cacheServiceEnabled) { val keys: Set[String] = Seq(maybeUser.map(_.id), maybeUser.map(_.email), maybeUser.map(_.username)).flatten.toSet // only send to Redis if keys are not empty if (keys.nonEmpty) { - val result = (storeManager ? CacheServiceRemoveValues(keys)).mapTo[Boolean] + val result = (storeManager ? CacheServiceRemoveValues(keys)) result.map { res => log.debug("invalidateCachedUserADM - result: {}", res) - res } } else { // since there was nothing to remove, we can immediately return - FastFuture.successful(true) + FastFuture.successful(()) } } else { // caching is turned off, so nothing to do. - FastFuture.successful(true) + FastFuture.successful(()) } } diff --git a/webapi/src/main/scala/org/knora/webapi/settings/package.scala b/webapi/src/main/scala/org/knora/webapi/settings/package.scala index f04720dd8a5..a649eaf2d01 100644 --- a/webapi/src/main/scala/org/knora/webapi/settings/package.scala +++ b/webapi/src/main/scala/org/knora/webapi/settings/package.scala @@ -116,8 +116,8 @@ package object settings { val SipiConnectorActorName: String = "sipiConnector" - /* Redis */ - val RedisManagerActorName: String = "redisManager" - val RedisManagerActorPath: String = StoreManagerActorPath + "/" + RedisManagerActorName + /* Cache */ + val CacheServiceManagerActorName: String = "redisManager" + val CacheServiceManagerActorPath: String = StoreManagerActorPath + "/" + CacheServiceManagerActorName } diff --git a/webapi/src/main/scala/org/knora/webapi/store/StoreManager.scala b/webapi/src/main/scala/org/knora/webapi/store/StoreManager.scala index d6718e328e8..46c48608257 100644 --- a/webapi/src/main/scala/org/knora/webapi/store/StoreManager.scala +++ b/webapi/src/main/scala/org/knora/webapi/store/StoreManager.scala @@ -14,11 +14,14 @@ import org.knora.webapi.messages.store.cacheservicemessages.CacheServiceRequest import org.knora.webapi.messages.store.sipimessages.IIIFRequest import org.knora.webapi.messages.store.triplestoremessages.TriplestoreRequest import org.knora.webapi.settings.{KnoraDispatchers, KnoraSettings, KnoraSettingsImpl, _} -import org.knora.webapi.store.cacheservice.{CacheService, CacheServiceManager} +import org.knora.webapi.store.cacheservice.CacheServiceManager +import org.knora.webapi.store.cacheservice.api.CacheService import org.knora.webapi.store.iiif.IIIFManager import org.knora.webapi.store.triplestore.TriplestoreManager import scala.concurrent.ExecutionContext +import zio._ +import org.knora.webapi.util.ActorUtil /** * This actor receives messages for different stores, and forwards them to the corresponding store manager. @@ -28,7 +31,7 @@ import scala.concurrent.ExecutionContext * * @param appActor a reference to the main application actor. */ -class StoreManager(appActor: ActorRef, cs: CacheService) extends Actor with ActorLogging { +class StoreManager(appActor: ActorRef, csm: CacheServiceManager) extends Actor with ActorLogging { this: ActorMaker => /** @@ -74,18 +77,10 @@ class StoreManager(appActor: ActorRef, cs: CacheService) extends Actor with Acto IIIFManagerActorName ) - /** - * Instantiates the Redis Manager - */ - protected lazy val cacheServiceManager: ActorRef = makeActor( - Props(new CacheServiceManager(cs)).withDispatcher(KnoraDispatchers.KnoraActorDispatcher), - RedisManagerActorName - ) - def receive: Receive = LoggingReceive { case tripleStoreMessage: TriplestoreRequest => triplestoreManager forward tripleStoreMessage case iiifMessages: IIIFRequest => iiifManager forward iiifMessages - case cacheServiceMessages: CacheServiceRequest => cacheServiceManager forward cacheServiceMessages + case cacheServiceMessages: CacheServiceRequest => ActorUtil.zio2Message(sender(), csm receive cacheServiceMessages, log) case other => sender() ! Status.Failure(UnexpectedMessageException(s"StoreManager received an unexpected message: $other")) } diff --git a/webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheService.scala b/webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheService.scala deleted file mode 100644 index 27a2cde686a..00000000000 --- a/webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheService.scala +++ /dev/null @@ -1,27 +0,0 @@ -/* - * Copyright © 2021 - 2022 Swiss National Data and Service Center for the Humanities and/or DaSCH Service Platform contributors. - * SPDX-License-Identifier: Apache-2.0 - */ - -package org.knora.webapi.store.cacheservice - -import org.knora.webapi.messages.admin.responder.projectsmessages.{ProjectADM, ProjectIdentifierADM} -import org.knora.webapi.messages.admin.responder.usersmessages.{UserADM, UserIdentifierADM} -import org.knora.webapi.messages.store.cacheservicemessages.{CacheServiceFlushDBACK, CacheServiceStatusResponse} - -import scala.concurrent.{ExecutionContext, Future} - -/** - * Cache Service Interface - */ -trait CacheService { - def putUserADM(value: UserADM)(implicit ec: ExecutionContext): Future[Boolean] - def getUserADM(identifier: UserIdentifierADM)(implicit ec: ExecutionContext): Future[Option[UserADM]] - def putProjectADM(value: ProjectADM)(implicit ec: ExecutionContext): Future[Boolean] - def getProjectADM(identifier: ProjectIdentifierADM)(implicit ec: ExecutionContext): Future[Option[ProjectADM]] - def writeStringValue(key: String, value: String)(implicit ec: ExecutionContext): Future[Boolean] - def getStringValue(maybeKey: Option[String])(implicit ec: ExecutionContext): Future[Option[String]] - def removeValues(keys: Set[String])(implicit ec: ExecutionContext): Future[Boolean] - def flushDB(requestingUser: UserADM)(implicit ec: ExecutionContext): Future[CacheServiceFlushDBACK] - def ping()(implicit ec: ExecutionContext): Future[CacheServiceStatusResponse] -} diff --git a/webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheServiceManager.scala b/webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheServiceManager.scala index 17914c25d57..5129b9fd266 100644 --- a/webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheServiceManager.scala +++ b/webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheServiceManager.scala @@ -13,38 +13,48 @@ import org.knora.webapi.messages.admin.responder.projectsmessages.{ProjectADM, P import org.knora.webapi.messages.admin.responder.usersmessages.{UserADM, UserIdentifierADM} import org.knora.webapi.messages.store.cacheservicemessages._ import org.knora.webapi.settings.KnoraDispatchers -import org.knora.webapi.util.ActorUtil.future2Message +import org.knora.webapi.util.ActorUtil.zio2Message +import org.knora.webapi.store.cacheservice.api.CacheService import scala.concurrent.{ExecutionContext, Future} - -class CacheServiceManager(cs: CacheService) - extends Actor - with ActorLogging - with LazyLogging - with InstrumentationSupport { - - /** - * The Knora Akka actor system. - */ - protected implicit val _system: ActorSystem = context.system - - /** - * The Akka actor system's execution context for futures. - */ - protected implicit val ec: ExecutionContext = context.system.dispatchers.lookup(KnoraDispatchers.KnoraActorDispatcher) - - def receive: Receive = { - case CacheServicePutUserADM(value) => future2Message(sender(), putUserADM(value), log) - case CacheServiceGetUserADM(identifier) => future2Message(sender(), getUserADM(identifier), log) - case CacheServicePutProjectADM(value) => future2Message(sender(), putProjectADM(value), log) - case CacheServiceGetProjectADM(identifier) => future2Message(sender(), getProjectADM(identifier), log) - case CacheServicePutString(key, value) => future2Message(sender(), writeStringValue(key, value), log) - case CacheServiceGetString(key) => future2Message(sender(), getStringValue(key), log) - case CacheServiceRemoveValues(keys) => future2Message(sender(), removeValues(keys), log) - case CacheServiceFlushDB(requestingUser) => future2Message(sender(), flushDB(requestingUser), log) - case CacheServiceGetStatus => future2Message(sender(), ping(), log) - case other => - sender() ! Status.Failure(UnexpectedMessageException(s"RedisManager received an unexpected message: $other")) +import zio._ +import zio.metrics._ +import zio.metrics.Metric +import zio.metrics.MetricLabel +import java.time.temporal.ChronoUnit +import zio.metrics.MetricClient + +case class CacheServiceManager(cs: CacheService) { + + val cacheServiceWriteUserTimer = Metric + .timer( + name = "cache-service-write-user", + chronoUnit = ChronoUnit.NANOS + ) + + val cacheServiceWriteProjectTimer = Metric + .timer( + name = "cache-service-write-project", + chronoUnit = ChronoUnit.NANOS + ) + + val cacheServiceReadProjectTimer = Metric + .timer( + name = "cache-service-read-project", + chronoUnit = ChronoUnit.NANOS + ) + + def receive(msg: CacheServiceRequest) = msg match { + case CacheServicePutUserADM(value) => putUserADM(value) + case CacheServiceGetUserADM(identifier) => getUserADM(identifier) + case CacheServicePutProjectADM(value) => putProjectADM(value) + case CacheServiceGetProjectADM(identifier) => getProjectADM(identifier) + case CacheServicePutString(key, value) => writeStringValue(key, value) + case CacheServiceGetString(key) => getStringValue(key) + case CacheServiceRemoveValues(keys) => removeValues(keys) + case CacheServiceFlushDB(requestingUser) => flushDB(requestingUser) + case CacheServiceGetStatus => ping() + case other => ZIO.logError(s"CacheServiceManager received an unexpected message: $other") } /** @@ -57,20 +67,20 @@ class CacheServiceManager(cs: CacheService) * * @param value the stored value */ - private def putUserADM(value: UserADM): Future[Boolean] = tracedFuture("caches-service-write-user") { - cs.putUserADM(value) - } + private def putUserADM(value: UserADM): Task[Unit] = + for { + res <- cs.putUserADM(value) @@ cacheServiceWriteUserTimer.trackDuration + // _ <- cacheServiceWriteUserTimer.value.tap(value => ZIO.debug(value)) + } yield res /** * Retrieves the user stored under the identifier (either iri, username, * or email). * - * @param identifier the project identifier. + * @param id the project identifier. */ - private def getUserADM(identifier: UserIdentifierADM): Future[Option[UserADM]] = - tracedFuture("cache-service-get-user") { - cs.getUserADM(identifier) - } + private def getUserADM(id: UserIdentifierADM): Task[Option[UserADM]] = + cs.getUserADM(id) /** * Stores the project under the IRI and additionally the IRI under the keys @@ -82,65 +92,67 @@ class CacheServiceManager(cs: CacheService) * * @param value the stored value */ - private def putProjectADM(value: ProjectADM)(implicit ec: ExecutionContext): Future[Boolean] = - tracedFuture("cache-service-write-project") { - cs.putProjectADM(value) - } + private def putProjectADM(value: ProjectADM): Task[Unit] = + for { + res <- cs.putProjectADM(value) @@ cacheServiceWriteProjectTimer.trackDuration + // _ <- cacheServiceWriteProjectTimer.value.tap(value => ZIO.debug(value)) + } yield res /** * Retrieves the project stored under the identifier (either iri, shortname, or shortcode). * * @param identifier the project identifier. */ - private def getProjectADM( - identifier: ProjectIdentifierADM - )(implicit ec: ExecutionContext): Future[Option[ProjectADM]] = - tracedFuture("cache-read-project") { - cs.getProjectADM(identifier) - } + private def getProjectADM(id: ProjectIdentifierADM): Task[Option[ProjectADM]] = + for { + res <- cs.getProjectADM(id) @@ cacheServiceReadProjectTimer.trackDuration + // _ <- cacheServiceReadProjectTimer.value.tap(value => ZIO.debug(value)) + } yield res /** * Get value stored under the key as a string. * - * @param maybeKey the key. + * @param k the key. */ - private def getStringValue(maybeKey: Option[String]): Future[Option[String]] = - tracedFuture("cache-service-get-string") { - cs.getStringValue(maybeKey) - } + private def getStringValue(k: String): Task[Option[String]] = + cs.getStringValue(k) /** * Store string or byte array value under key. * - * @param key the key. - * @param value the value. + * @param k the key. + * @param v the value. */ - private def writeStringValue(key: String, value: String): Future[Boolean] = - tracedFuture("cache-service-write-string") { - cs.writeStringValue(key, value) - } + private def writeStringValue(k: String, v: String): Task[Unit] = + cs.putStringValue(k, v) /** * Removes values for the provided keys. Any invalid keys are ignored. * * @param keys the keys. */ - private def removeValues(keys: Set[String]): Future[Boolean] = - tracedFuture("cache-remove-values") { - cs.removeValues(keys) - } + private def removeValues(keys: Set[String]): Task[Unit] = + cs.removeValues(keys) /** - * Flushes (removes) all stored content from the Redis store. + * Flushes (removes) all stored content from the store. */ - private def flushDB(requestingUser: UserADM): Future[CacheServiceFlushDBACK] = - tracedFuture("cache-flush") { - cs.flushDB(requestingUser) - } + private def flushDB(requestingUser: UserADM): Task[Unit] = + cs.flushDB(requestingUser) /** * Pings the cache service to see if it is available. */ - private def ping(): Future[CacheServiceStatusResponse] = + private def ping(): Task[CacheServiceStatusResponse] = cs.ping() } + +object CacheServiceManager { + val layer: ZLayer[CacheService, Nothing, CacheServiceManager] = { + ZLayer { + for { + cache <- ZIO.service[CacheService] + } yield CacheServiceManager(cache) + } + } +} diff --git a/webapi/src/main/scala/org/knora/webapi/store/cacheservice/api/CacheService.scala b/webapi/src/main/scala/org/knora/webapi/store/cacheservice/api/CacheService.scala new file mode 100644 index 00000000000..95520279adf --- /dev/null +++ b/webapi/src/main/scala/org/knora/webapi/store/cacheservice/api/CacheService.scala @@ -0,0 +1,47 @@ +/* + * Copyright © 2021 - 2022 Swiss National Data and Service Center for the Humanities and/or DaSCH Service Platform contributors. + * SPDX-License-Identifier: Apache-2.0 + */ + +package org.knora.webapi.store.cacheservice.api + +import zio._ +import org.knora.webapi.messages.admin.responder.projectsmessages.{ProjectADM, ProjectIdentifierADM} +import org.knora.webapi.messages.admin.responder.usersmessages.{UserADM, UserIdentifierADM} +import org.knora.webapi.messages.store.cacheservicemessages.{CacheServiceStatusResponse} + +/** + * Cache Service Interface + */ +trait CacheService { + def putUserADM(value: UserADM): Task[Unit] + def getUserADM(identifier: UserIdentifierADM): Task[Option[UserADM]] + def putProjectADM(value: ProjectADM): Task[Unit] + def getProjectADM(identifier: ProjectIdentifierADM): Task[Option[ProjectADM]] + def putStringValue(key: String, value: String): Task[Unit] + def getStringValue(key: String): Task[Option[String]] + def removeValues(keys: Set[String]): Task[Unit] + def flushDB(requestingUser: UserADM): Task[Unit] + def ping(): Task[CacheServiceStatusResponse] +} + +/** + * Cache Service companion object using [[Accessible]]. + * To use, simply call `Companion(_.someMethod)`, to return a ZIO + * effect that requires the Service in its environment. + * + * Example: + * {{{ + * trait CacheService { + * def ping(): Task[CacheServiceStatusResponse] + * } + * + * object CacheService extends Accessible[CacheService] + * + * val example: ZIO[CacheService, Nothing, Unit] = + * for { + * _ <- CacheService(_.ping()) + * } yield () + * }}} + */ +object CacheService extends Accessible[CacheService] diff --git a/webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheServiceExceptions.scala b/webapi/src/main/scala/org/knora/webapi/store/cacheservice/api/CacheServiceExceptions.scala similarity index 90% rename from webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheServiceExceptions.scala rename to webapi/src/main/scala/org/knora/webapi/store/cacheservice/api/CacheServiceExceptions.scala index 65b6cb34bb2..24e0ddbec2a 100644 --- a/webapi/src/main/scala/org/knora/webapi/store/cacheservice/CacheServiceExceptions.scala +++ b/webapi/src/main/scala/org/knora/webapi/store/cacheservice/api/CacheServiceExceptions.scala @@ -3,7 +3,7 @@ * SPDX-License-Identifier: Apache-2.0 */ -package org.knora.webapi.store.cacheservice +package org.knora.webapi.store.cacheservice.api import org.knora.webapi.exceptions.CacheServiceException diff --git a/webapi/src/main/scala/org/knora/webapi/store/cacheservice/config/CacheServiceConfig.scala b/webapi/src/main/scala/org/knora/webapi/store/cacheservice/config/CacheServiceConfig.scala new file mode 100644 index 00000000000..36e35e68856 --- /dev/null +++ b/webapi/src/main/scala/org/knora/webapi/store/cacheservice/config/CacheServiceConfig.scala @@ -0,0 +1,7 @@ +package org.knora.webapi.store.cacheservice.config + +import zio._ + +final case class CacheServiceConfig(enabled: Boolean, redis: RedisConfig) + +final case class RedisConfig(server: String, port: Int) diff --git a/webapi/src/main/scala/org/knora/webapi/store/cacheservice/impl/CacheServiceInMemImpl.scala b/webapi/src/main/scala/org/knora/webapi/store/cacheservice/impl/CacheServiceInMemImpl.scala new file mode 100644 index 00000000000..7622bea4a92 --- /dev/null +++ b/webapi/src/main/scala/org/knora/webapi/store/cacheservice/impl/CacheServiceInMemImpl.scala @@ -0,0 +1,220 @@ +/* + * Copyright © 2021 - 2022 Swiss National Data and Service Center for the Humanities and/or DaSCH Service Platform contributors. + * SPDX-License-Identifier: Apache-2.0 + */ + +package org.knora.webapi.store.cacheservice.impl + +import akka.http.scaladsl.util.FastFuture +import org.knora.webapi.IRI +import org.knora.webapi.messages.admin.responder.projectsmessages.ProjectADM +import org.knora.webapi.messages.admin.responder.projectsmessages.ProjectIdentifierADM +import org.knora.webapi.messages.admin.responder.projectsmessages.ProjectIdentifierType +import org.knora.webapi.messages.admin.responder.usersmessages.UserADM +import org.knora.webapi.messages.admin.responder.usersmessages.UserIdentifierADM +import org.knora.webapi.messages.admin.responder.usersmessages.UserIdentifierType +import org.knora.webapi.messages.store.cacheservicemessages.CacheServiceStatusOK +import org.knora.webapi.messages.store.cacheservicemessages.CacheServiceStatusResponse +import org.knora.webapi.store.cacheservice.api.CacheService +import org.knora.webapi.store.cacheservice.api.EmptyKey +import org.knora.webapi.store.cacheservice.api.EmptyValue +import zio._ +import zio.stm._ + +import scala.concurrent.ExecutionContext +import scala.concurrent.Future + +/** + * In-Memory Cache implementation + * + * The state is divided into Refs used to store different types of objects. + * A ref in itself is fiber (thread) safe, but to keep the cumulative state + * consistent, all Refs need to be updated in a single transaction. This + * requires STM (Software Transactional Memory) to be used. + * + * @param users a map of users. + * @param projects a map of projects. + * @param lut a lookup table of username/email to IRI. + */ +case class CacheServiceInMemImpl( + users: TMap[String, UserADM], + projects: TMap[String, ProjectADM], + lut: TMap[String, String] // sealed trait for key type +) extends CacheService { + + /** + * Stores the user under the IRI (inside 'users') and additionally the IRI + * under the keys of USERNAME and EMAIL (inside the 'lut'): + * + * IRI -> byte array + * username -> IRI + * email -> IRI + * + * @param value the value to be stored + */ + def putUserADM(value: UserADM): Task[Unit] = + (for { + _ <- users.put(value.id, value) + _ <- lut.put(value.username, value.id) + _ <- lut.put(value.email, value.id) + } yield ()).commit.tap(_ => ZIO.logInfo(s"Stored UserADM to Cache: ${value.id}")) + + /** + * Retrieves the user stored under the identifier (either iri, username, or email). + * + * The data is stored under the IRI key. + * Additionally, the USERNAME and EMAIL keys point to the IRI key + * + * @param identifier the user identifier. + */ + def getUserADM(identifier: UserIdentifierADM): Task[Option[UserADM]] = + (identifier.hasType match { + case UserIdentifierType.Iri => getUserByIri(identifier.toIri) + case UserIdentifierType.Username => getUserByUsernameOrEmail(identifier.toUsername) + case UserIdentifierType.Email => getUserByUsernameOrEmail(identifier.toEmail) + }).tap(_ => ZIO.logInfo(s"Retrieved UserADM from Cache: ${identifier}")) + + /** + * Retrieves the user stored under the IRI. + * + * @param id the user's IRI. + * @return an optional [[UserADM]]. + */ + def getUserByIri(id: String): ZIO[Any, Nothing, Option[UserADM]] = + users.get(id).commit + + /** + * Retrieves the user stored under the username or email. + * + * @param usernameOrEmail of the user. + * @return an optional [[UserADM]]. + */ + def getUserByUsernameOrEmail(usernameOrEmail: String): ZIO[Any, Nothing, Option[UserADM]] = + (for { + iri <- lut.get(usernameOrEmail).some + user <- users.get(iri).some + } yield user).commit.unsome // watch Spartan session about error. post example on Spartan channel + + /** + * Stores the project under the IRI and additionally the IRI under the keys + * of SHORTCODE and SHORTNAME: + * + * IRI -> byte array + * shortname -> IRI + * shortcode -> IRI + * + * @param value the stored value + * @return [[Unit]] + */ + def putProjectADM(value: ProjectADM): Task[Unit] = + (for { + _ <- projects.put(value.id, value) + _ <- lut.put(value.shortname, value.id) + _ <- lut.put(value.shortcode, value.id) + } yield ()).commit.tap(_ => ZIO.logInfo(s"Stored ProjectADM to Cache: ${value.id}")) + + /** + * Retrieves the project stored under the identifier (either iri, shortname, or shortcode). + * + * The data is stored under the IRI key. + * Additionally, the SHORTCODE and SHORTNAME keys point to the IRI key + * + * @param identifier the project identifier. + * @return an optional [[ProjectADM]] + */ + def getProjectADM(identifier: ProjectIdentifierADM): Task[Option[ProjectADM]] = + (identifier.hasType match { + case ProjectIdentifierType.IRI => getProjectByIri(identifier.toIri) + case ProjectIdentifierType.SHORTCODE => getProjectByShortcodeOrShortname(identifier.toShortcode) + case ProjectIdentifierType.SHORTNAME => getProjectByShortcodeOrShortname(identifier.toShortname) + }).tap(_ => ZIO.logInfo(s"Retrieved ProjectADM from Cache: $identifier")) + + /** + * Retrieves the project stored under the IRI. + * + * @param id the project's IRI + * @return an optional [[ProjectADM]]. + */ + def getProjectByIri(id: String) = + projects.get(id).commit + + /** + * Retrieves the project stored under a SHORTCODE or SHORTNAME. + * + * @param shortcodeOrShortname of the project. + * @return an optional [[ProjectADM]] + */ + def getProjectByShortcodeOrShortname(shortcodeOrShortname: String) = + (for { + iri <- lut.get(shortcodeOrShortname).some + project <- projects.get(iri).some + } yield project).commit.unsome + + /** + * Store string or byte array value under key. + * + * @param key the key. + * @param value the value. + */ + def putStringValue(key: String, value: String): Task[Unit] = { + + val emptyKeyError = EmptyKey("The key under which the value should be written is empty. Aborting write to cache.") + val emptyValueError = EmptyValue("The string value is empty. Aborting write to cache.") + + (for { + key <- if (key.isEmpty()) Task.fail(emptyKeyError) else Task.succeed(key) + value <- if (value.isEmpty()) Task.fail(emptyValueError) else Task.succeed(value) + _ <- lut.put(key, value).commit + } yield ()).tap(_ => ZIO.logInfo(s"Wrote key: $key with value: $value to cache.")) + } + + /** + * Get value stored under the key as a string. + * + * @param maybeKey the key. + * @return an optional [[String]]. + */ + def getStringValue(key: String): Task[Option[String]] = + lut.get(key).commit.tap(value => ZIO.logInfo(s"Retrieved key: $key with value: $value from cache.")) + + /** + * Removes values for the provided keys. Any invalid keys are ignored. + * + * @param keys the keys. + */ + def removeValues(keys: Set[String]): Task[Unit] = + (for { + _ <- ZIO.foreach(keys)(key => lut.delete(key).commit) // FIXME: is this realy thread safe? + } yield ()).tap(_ => ZIO.logInfo(s"Removed keys from cache: $keys")) + + /** + * Flushes (removes) all stored content from the in-memory cache. + */ + def flushDB(requestingUser: UserADM): Task[Unit] = + (for { + _ <- users.foreach((k, v) => users.delete(k)) + _ <- projects.foreach((k, v) => projects.delete(k)) + _ <- lut.foreach((k, v) => lut.delete(k)) + } yield ()).commit.tap(_ => ZIO.logInfo("Flushed in-memory cache")) + + /** + * Pings the in-memory cache to see if it is available. + */ + def ping(): Task[CacheServiceStatusResponse] = + Task.succeed(CacheServiceStatusOK) +} + +/** + * Companion object providing the layer with an initialized implementation + */ +object CacheServiceInMemImpl { + val layer: ZLayer[Any, Nothing, CacheService] = { + ZLayer { + for { + users <- TMap.empty[String, UserADM].commit + projects <- TMap.empty[String, ProjectADM].commit + lut <- TMap.empty[String, String].commit + } yield CacheServiceInMemImpl(users, projects, lut) + }.tap(_ => ZIO.debug(">>> In-Memory Cache Service Initialized <<<")) + } +} diff --git a/webapi/src/main/scala/org/knora/webapi/store/cacheservice/impl/CacheServiceRedisImpl.scala b/webapi/src/main/scala/org/knora/webapi/store/cacheservice/impl/CacheServiceRedisImpl.scala new file mode 100644 index 00000000000..e80141d6065 --- /dev/null +++ b/webapi/src/main/scala/org/knora/webapi/store/cacheservice/impl/CacheServiceRedisImpl.scala @@ -0,0 +1,293 @@ +/* + * Copyright © 2021 - 2022 Swiss National Data and Service Center for the Humanities and/or DaSCH Service Platform contributors. + * SPDX-License-Identifier: Apache-2.0 + */ + +package org.knora.webapi.store.cacheservice.impl + +import com.typesafe.scalalogging.LazyLogging +import com.typesafe.scalalogging.Logger +import org.knora.webapi.exceptions.ForbiddenException +import org.knora.webapi.messages.admin.responder.projectsmessages.ProjectADM +import org.knora.webapi.messages.admin.responder.projectsmessages.ProjectIdentifierADM +import org.knora.webapi.messages.admin.responder.projectsmessages.ProjectIdentifierType +import org.knora.webapi.messages.admin.responder.usersmessages.UserADM +import org.knora.webapi.messages.admin.responder.usersmessages.UserIdentifierADM +import org.knora.webapi.messages.admin.responder.usersmessages.UserIdentifierType +import org.knora.webapi.messages.store.cacheservicemessages.CacheServiceStatusNOK +import org.knora.webapi.messages.store.cacheservicemessages.CacheServiceStatusOK +import org.knora.webapi.messages.store.cacheservicemessages.CacheServiceStatusResponse +import org.knora.webapi.store.cacheservice.api.CacheService +import org.knora.webapi.store.cacheservice.api.EmptyKey +import org.knora.webapi.store.cacheservice.api.EmptyValue +import org.knora.webapi.store.cacheservice.config.RedisConfig +import org.knora.webapi.store.cacheservice.serialization.CacheSerialization +import org.knora.webapi.store.cacheservice.settings.CacheServiceSettings +import redis.clients.jedis.Jedis +import redis.clients.jedis.JedisPool +import redis.clients.jedis.JedisPoolConfig +import zio._ + +case class CacheServiceRedisImpl(pool: JedisPool) extends CacheService { + + /** + * Stores the user under the IRI and additionally the IRI under the keys of + * USERNAME and EMAIL: + * + * IRI -> byte array + * username -> IRI + * email -> IRI + * + * @param user the user to store. + */ + def putUserADM(user: UserADM): Task[Unit] = + for { + bytes <- CacheSerialization.serialize(user) + _ <- putBytesValue(user.id, bytes) + _ <- putStringValue(user.username, user.id) + _ <- putStringValue(user.email, user.id) + } yield () + + /** + * Retrieves the user stored under the identifier (either iri, username, + * or email). + * + * The data is stored under the IRI key. + * Additionally, the USERNAME and EMAIL keys point to the IRI key + * + * @param identifier the user identifier. + */ + def getUserADM(identifier: UserIdentifierADM): Task[Option[UserADM]] = + identifier.hasType match { + case UserIdentifierType.Iri => getUserByIri(identifier.toIri) + case UserIdentifierType.Username => getUserByUsernameOrEmail(identifier.toUsername) + case UserIdentifierType.Email => getUserByUsernameOrEmail(identifier.toEmail) + } + + /** + * Retrieves the user stored under the IRI. + * + * @param id the user's IRI. + * @return an optional [[UserADM]]. + */ + def getUserByIri(id: String): Task[Option[UserADM]] = + (for { + bytes <- getBytesValue(id).some + user <- CacheSerialization.deserialize[UserADM](bytes).some + } yield user).unsome + + /** + * Retrieves the user stored under the username or email. + * + * @param usernameOrEmail of the user. + * @return an optional [[UserADM]]. + */ + def getUserByUsernameOrEmail(usernameOrEmail: String): Task[Option[UserADM]] = + (for { + iri <- getStringValue(usernameOrEmail).some + user <- getUserByIri(iri).some + } yield user).unsome + + /** + * Stores the project under the IRI and additionally the IRI under the keys + * of SHORTCODE and SHORTNAME: + * + * IRI -> byte array + * shortname -> IRI + * shortcode -> IRI + * + * @param value the stored value + */ + def putProjectADM(project: ProjectADM): Task[Unit] = + for { + bytes <- CacheSerialization.serialize(project) + _ <- putBytesValue(project.id, bytes) + _ <- putStringValue(project.shortcode, project.id) + _ <- putStringValue(project.shortname, project.id) + } yield () + + /** + * Retrieves the project stored under the identifier (either iri, shortname, or shortcode). + * + * @param identifier the project identifier. + */ + def getProjectADM(identifier: ProjectIdentifierADM): Task[Option[ProjectADM]] = + // The data is stored under the IRI key. + // Additionally, the SHORTNAME and SHORTCODE keys point to the IRI key + identifier.hasType match { + case ProjectIdentifierType.IRI => getProjectByIri(identifier.toIri) + case ProjectIdentifierType.SHORTCODE => getProjectByShortcodeOrShortname(identifier.toShortcode) + case ProjectIdentifierType.SHORTNAME => getProjectByShortcodeOrShortname(identifier.toShortname) + } + + /** + * Retrieves the project stored under the IRI. + * + * @param id the project's IRI + * @return an optional [[ProjectADM]]. + */ + def getProjectByIri(id: String): Task[Option[ProjectADM]] = + (for { + bytes <- getBytesValue(id).some + project <- CacheSerialization.deserialize[ProjectADM](bytes).some + } yield project).unsome + + /** + * Retrieves the project stored under a SHORTCODE or SHORTNAME. + * + * @param shortcodeOrShortname of the project. + * @return an optional [[ProjectADM]] + */ + def getProjectByShortcodeOrShortname(shortcodeOrShortname: String): Task[Option[ProjectADM]] = + (for { + iri <- getStringValue(shortcodeOrShortname).some + project <- getProjectByIri(iri).some + } yield project).unsome + + /** + * Store string value under key. + * + * @param key the key. + * @param value the value. + */ + def putStringValue(key: String, value: String): Task[Unit] = Task.attempt { + + if (key.isEmpty) + throw EmptyKey("The key under which the value should be written is empty. Aborting writing to redis.") + + if (value.isEmpty) + throw EmptyValue("The string value is empty. Aborting writing to redis.") + + val conn: Jedis = pool.getResource + try { + conn.set(key, value) + () + } finally { + conn.close() + } + + }.catchAll(ex => ZIO.logError(s"Writing to Redis failed: ${ex.getMessage}")) + + /** + * Get value stored under the key as a string. + * + * @param maybeKey the key. + */ + def getStringValue(key: String): Task[Option[String]] = { + // FIXME: make it resource safe, i.e., use Scope and add finalizers for the connection + for { + conn <- ZIO.attempt(pool.getResource) + value <- ZIO.attemptBlocking(conn.get(key)) + res <- if (value == "nil".getBytes) Task.succeed(None) + else Task.succeed(Some(value)) + _ = conn.close() + } yield res + }.catchAll(ex => ZIO.logError(s"Reading string from Redis failed: ${ex.getMessage}") *> Task.succeed(None)) + + /** + * Removes values for the provided keys. Any invalid keys are ignored. + * + * @param keys the keys. + */ + def removeValues(keys: Set[String]): Task[Unit] = Task.attemptBlocking { + + // del takes a vararg so I nee to convert the set to a swq and then to vararg + val conn: Jedis = pool.getResource + try { + conn.del(keys.toSeq: _*) + () + } finally { + conn.close() + } + + }.catchAll(ex => ZIO.logError(s"Removing keys from Redis failed: ${ex.getMessage}")) + + /** + * Store byte array value under key. + * + * @param key the key. + * @param value the value. + */ + private def putBytesValue(key: String, value: Array[Byte]): Task[Unit] = Task.attemptBlocking { + + if (key.isEmpty) + throw EmptyKey("The key under which the value should be written is empty. Aborting writing to redis.") + + if (value.isEmpty) + throw EmptyValue("The byte array value is empty. Aborting writing to redis.") + + val conn: Jedis = pool.getResource + try { + conn.set(key.getBytes, value) + () + } finally { + conn.close() + } + + }.catchAll(ex => ZIO.logError(s"Writing to Redis failed: ${ex.getMessage}")) + + /** + * Get value stored under the key as a byte array. If no value is found + * under the key, then a [[None]] is returned.. + * + * @param key the key. + */ + private def getBytesValue(key: String): Task[Option[Array[Byte]]] = + // FIXME: make it resource safe, i.e., use Scope and add finalizers for the connection + for { + conn <- ZIO.attempt(pool.getResource).onError(ZIO.logErrorCause(_)).orDie + value <- ZIO.attemptBlocking(conn.get(key.getBytes)) + res <- if (value == "nil".getBytes) Task.succeed(None) + else Task.succeed(Some(value)) + _ = conn.close() + } yield res + + /** + * Flushes (removes) all stored content from the Redis store. + */ + def flushDB(requestingUser: UserADM): Task[Unit] = Task.attemptBlocking { + + if (!requestingUser.isSystemUser) { + throw ForbiddenException("Only the system user is allowed to perform this operation.") + } + + val conn: Jedis = pool.getResource + try { + conn.flushDB() + () + } finally { + conn.close() + } + + } + .catchAll(ex => ZIO.logError(s"Flushing DB failed: ${ex.getMessage}")) + .tap(_ => ZIO.logDebug("Redis cache flushed.")) + + /** + * Pings the Redis store to see if it is available. + */ + def ping(): Task[CacheServiceStatusResponse] = Task.attemptBlocking { + + val conn: Jedis = pool.getResource + try { + conn.ping("test") + CacheServiceStatusOK + } finally { + conn.close() + } + }.catchAll(ex => ZIO.logError(s"Ping failed: ${ex.getMessage}") *> Task.succeed(CacheServiceStatusNOK)) +} + +object CacheServiceRedisImpl { + val layer: ZLayer[RedisConfig, Nothing, CacheService] = { + ZLayer { + for { + config <- ZIO.service[RedisConfig] + pool <- ZIO + .attempt(new JedisPool(new JedisPoolConfig(), config.server, config.port)) + .onError(ZIO.logErrorCause(_)) + .orDie // the Redis Client Pool + } yield CacheServiceRedisImpl(pool) + }.tap(_ => ZIO.debug(">>> Redis Cache Service Initialized <<<")) + } +} diff --git a/webapi/src/main/scala/org/knora/webapi/store/cacheservice/inmem/CacheServiceInMemImpl.scala b/webapi/src/main/scala/org/knora/webapi/store/cacheservice/inmem/CacheServiceInMemImpl.scala deleted file mode 100644 index 6fb9e798c28..00000000000 --- a/webapi/src/main/scala/org/knora/webapi/store/cacheservice/inmem/CacheServiceInMemImpl.scala +++ /dev/null @@ -1,175 +0,0 @@ -/* - * Copyright © 2021 - 2022 Swiss National Data and Service Center for the Humanities and/or DaSCH Service Platform contributors. - * SPDX-License-Identifier: Apache-2.0 - */ - -package org.knora.webapi.store.cacheservice.inmem - -import akka.http.scaladsl.util.FastFuture -import com.typesafe.scalalogging.LazyLogging -import org.knora.webapi.messages.admin.responder.projectsmessages.{ - ProjectADM, - ProjectIdentifierADM, - ProjectIdentifierType -} -import org.knora.webapi.messages.admin.responder.usersmessages.{UserADM, UserIdentifierADM, UserIdentifierType} -import org.knora.webapi.messages.store.cacheservicemessages.{ - CacheServiceFlushDBACK, - CacheServiceStatusOK, - CacheServiceStatusResponse -} -import org.knora.webapi.store.cacheservice.{CacheService, EmptyKey, EmptyValue} - -import scala.concurrent.{ExecutionContext, Future} - -object CacheServiceInMemImpl extends CacheService with LazyLogging { - - private var cache: scala.collection.mutable.Map[Any, Any] = - scala.collection.mutable.Map[Any, Any]() - - /** - * Stores the user under the IRI and additionally the IRI under the keys of - * USERNAME and EMAIL: - * - * IRI -> byte array - * username -> IRI - * email -> IRI - * - * @param value the stored value - */ - def putUserADM(value: UserADM)(implicit ec: ExecutionContext): Future[Boolean] = { - cache(value.id) = value - cache(value.username) = value.id - cache(value.email) = value.id - FastFuture.successful(true) - } - - /** - * Retrieves the user stored under the identifier (either iri, username, - * or email). - * - * @param identifier the user identifier. - */ - def getUserADM(identifier: UserIdentifierADM)(implicit ec: ExecutionContext): Future[Option[UserADM]] = { - // The data is stored under the IRI key. - // Additionally, the USERNAME and EMAIL keys point to the IRI key - val resultFuture: Future[Option[UserADM]] = identifier.hasType match { - case UserIdentifierType.Iri => FastFuture.successful(cache.get(identifier.toIri).map(_.asInstanceOf[UserADM])) - case UserIdentifierType.Username => { - cache.get(identifier.toUsername) match { - case Some(iriKey) => FastFuture.successful(cache.get(iriKey).map(_.asInstanceOf[UserADM])) - case None => FastFuture.successful(None) - } - } - case UserIdentifierType.Email => - cache.get(identifier.toEmail) match { - case Some(iriKey) => FastFuture.successful(cache.get(iriKey).map(_.asInstanceOf[UserADM])) - case None => FastFuture.successful(None) - } - } - resultFuture - } - - /** - * Stores the project under the IRI and additionally the IRI under the keys - * of SHORTCODE and SHORTNAME: - * - * IRI -> byte array - * shortname -> IRI - * shortcode -> IRI - * - * @param value the stored value - */ - def putProjectADM(value: ProjectADM)(implicit ec: ExecutionContext): Future[Boolean] = { - cache(value.id) = value - cache(value.shortcode) = value.id - cache(value.shortname) = value.id - FastFuture.successful(true) - } - - /** - * Retrieves the project stored under the identifier (either iri, shortname, or shortcode). - * - * @param identifier the project identifier. - */ - def getProjectADM(identifier: ProjectIdentifierADM)(implicit ec: ExecutionContext): Future[Option[ProjectADM]] = { - // The data is stored under the IRI key. - // Additionally, the SHORTNAME and SHORTCODE keys point to the IRI key - val resultFuture: Future[Option[ProjectADM]] = identifier.hasType match { - case ProjectIdentifierType.IRI => - FastFuture.successful(cache.get(identifier.toIri).map(_.asInstanceOf[ProjectADM])) - case ProjectIdentifierType.SHORTCODE => - cache.get(identifier.toShortcode) match { - case Some(iriKey) => FastFuture.successful(cache.get(iriKey).map(_.asInstanceOf[ProjectADM])) - case None => FastFuture.successful(None) - } - case ProjectIdentifierType.SHORTNAME => - cache.get(identifier.toShortname) match { - case Some(iriKey) => FastFuture.successful(cache.get(iriKey).map(_.asInstanceOf[ProjectADM])) - case None => FastFuture.successful(None) - } - } - resultFuture - } - - /** - * Store string or byte array value under key. - * - * @param key the key. - * @param value the value. - */ - def writeStringValue(key: String, value: String)(implicit ec: ExecutionContext): Future[Boolean] = { - - if (key.isEmpty) - throw EmptyKey("The key under which the value should be written is empty. Aborting writing to in-memory cache.") - - if (value.isEmpty) - throw EmptyValue("The string value is empty. Aborting writing to in-memory cache.") - - cache(key) = value - FastFuture.successful(true) - } - - /** - * Get value stored under the key as a string. - * - * @param maybeKey the key. - */ - def getStringValue(maybeKey: Option[String])(implicit ec: ExecutionContext): Future[Option[String]] = - maybeKey match { - case Some(key) => - FastFuture.successful(cache.get(key).map(_.asInstanceOf[String])) - case None => - FastFuture.successful(None) - } - - /** - * Removes values for the provided keys. Any invalid keys are ignored. - * - * @param keys the keys. - */ - def removeValues(keys: Set[String])(implicit ec: ExecutionContext): Future[Boolean] = { - - logger.debug("removeValues - {}", keys) - keys foreach { key => - cache remove key - } - - FastFuture.successful(true) - } - - /** - * Flushes (removes) all stored content from the in-memory cache. - */ - def flushDB(requestingUser: UserADM)(implicit ec: ExecutionContext): Future[CacheServiceFlushDBACK] = { - cache = scala.collection.mutable.Map[Any, Any]() - FastFuture.successful(CacheServiceFlushDBACK()) - } - - /** - * Pings the in-memory cache to see if it is available. - */ - def ping()(implicit ec: ExecutionContext): Future[CacheServiceStatusResponse] = - FastFuture.successful(CacheServiceStatusOK) - -} diff --git a/webapi/src/main/scala/org/knora/webapi/store/cacheservice/redis/CacheServiceRedisImpl.scala b/webapi/src/main/scala/org/knora/webapi/store/cacheservice/redis/CacheServiceRedisImpl.scala deleted file mode 100644 index 70ffee24083..00000000000 --- a/webapi/src/main/scala/org/knora/webapi/store/cacheservice/redis/CacheServiceRedisImpl.scala +++ /dev/null @@ -1,397 +0,0 @@ -/* - * Copyright © 2021 - 2022 Swiss National Data and Service Center for the Humanities and/or DaSCH Service Platform contributors. - * SPDX-License-Identifier: Apache-2.0 - */ - -package org.knora.webapi.store.cacheservice.redis - -import akka.http.scaladsl.util.FastFuture -import com.typesafe.scalalogging.{LazyLogging, Logger} -import org.knora.webapi.exceptions.ForbiddenException -import org.knora.webapi.messages.admin.responder.projectsmessages.{ - ProjectADM, - ProjectIdentifierADM, - ProjectIdentifierType -} -import org.knora.webapi.messages.admin.responder.usersmessages.{UserADM, UserIdentifierADM, UserIdentifierType} -import org.knora.webapi.messages.store.cacheservicemessages.{ - CacheServiceFlushDBACK, - CacheServiceStatusNOK, - CacheServiceStatusOK, - CacheServiceStatusResponse -} -import org.knora.webapi.store.cacheservice.serialization.CacheSerialization -import org.knora.webapi.store.cacheservice.settings.CacheServiceSettings -import org.knora.webapi.store.cacheservice.{CacheService, EmptyKey, EmptyValue} -import redis.clients.jedis.{Jedis, JedisPool, JedisPoolConfig} - -import scala.concurrent.{ExecutionContext, Future} - -class CacheServiceRedisImpl(s: CacheServiceSettings) extends CacheService with LazyLogging { - - /** - * The Redis Client Pool - */ - val pool: JedisPool = new JedisPool(new JedisPoolConfig(), s.cacheServiceRedisHost, s.cacheServiceRedisPort, 20999) - - // this is needed for time measurements using 'org.knora.webapi.Timing' - - implicit val l: Logger = logger - - /** - * Stores the user under the IRI and additionally the IRI under the keys of - * USERNAME and EMAIL: - * - * IRI -> byte array - * username -> IRI - * email -> IRI - * - * @param value the stored value - */ - def putUserADM(value: UserADM)(implicit ec: ExecutionContext): Future[Boolean] = { - val resultFuture = for { - bytes: Array[Byte] <- CacheSerialization.serialize(value) - result: Boolean <- writeBytesValue(value.id, bytes) - // additionally store the IRI under the username and email key - _ = writeStringValue(value.username, value.id) - _ = writeStringValue(value.email, value.id) - } yield result - - val recoverableResultFuture = resultFuture.recover { case e: Exception => - logger.warn("Aborting writing 'UserADM' to Redis - {}", e.getMessage) - false - } - - recoverableResultFuture - } - - /** - * Retrieves the user stored under the identifier (either iri, username, - * or email). - * - * @param identifier the user identifier. - */ - def getUserADM(identifier: UserIdentifierADM)(implicit ec: ExecutionContext): Future[Option[UserADM]] = { - // The data is stored under the IRI key. - // Additionally, the USERNAME and EMAIL keys point to the IRI key - val resultFuture: Future[Option[UserADM]] = identifier.hasType match { - case UserIdentifierType.Iri => - for { - maybeBytes: Option[Array[Byte]] <- getBytesValue(identifier.toIriOption) - maybeUser: Option[UserADM] <- maybeBytes match { - case Some(bytes) => CacheSerialization.deserialize[UserADM](bytes) - case None => FastFuture.successful(None) - } - } yield maybeUser - - case UserIdentifierType.Username => - for { - maybeIriKey: Option[String] <- getStringValue(identifier.toUsernameOption) - maybeBytes: Option[Array[Byte]] <- getBytesValue(maybeIriKey) - maybeUser: Option[UserADM] <- maybeBytes match { - case Some(bytes) => CacheSerialization.deserialize[UserADM](bytes) - case None => FastFuture.successful(None) - } - } yield maybeUser - - case UserIdentifierType.Email => - for { - maybeIriKey: Option[String] <- getStringValue(identifier.toEmailOption) - maybeBytes: Option[Array[Byte]] <- getBytesValue(maybeIriKey) - maybeUser: Option[UserADM] <- maybeBytes match { - case Some(bytes) => CacheSerialization.deserialize[UserADM](bytes) - case None => FastFuture.successful(None) - } - } yield maybeUser - } - - val recoverableResultFuture = resultFuture.recover { case e: Exception => - logger.warn("Aborting reading 'UserADM' from Redis - {}", e.getMessage) - None - } - - recoverableResultFuture - } - - /** - * Stores the project under the IRI and additionally the IRI under the keys - * of SHORTCODE and SHORTNAME: - * - * IRI -> byte array - * shortname -> IRI - * shortcode -> IRI - * - * @param value the stored value - */ - def putProjectADM(value: ProjectADM)(implicit ec: ExecutionContext): Future[Boolean] = { - val resultFuture = for { - bytes: Array[Byte] <- CacheSerialization.serialize(value) - result: Boolean <- writeBytesValue(value.id, bytes) - _ = writeStringValue(value.shortcode, value.id) - _ = writeStringValue(value.shortname, value.id) - } yield result - - val recoverableResultFuture = resultFuture.recover { case e: Exception => - logger.warn("Aborting writing 'ProjectADM' to Redis - {}", e.getMessage) - false - } - - recoverableResultFuture - } - - /** - * Retrieves the project stored under the identifier (either iri, shortname, or shortcode). - * - * @param identifier the project identifier. - */ - def getProjectADM(identifier: ProjectIdentifierADM)(implicit ec: ExecutionContext): Future[Option[ProjectADM]] = { - - // The data is stored under the IRI key. - // Additionally, the SHORTNAME and SHORTCODE keys point to the IRI key - val resultFuture: Future[Option[ProjectADM]] = identifier.hasType match { - case ProjectIdentifierType.IRI => - for { - maybeBytes <- getBytesValue(identifier.toIriOption) - maybeProject <- maybeBytes match { - case Some(bytes) => CacheSerialization.deserialize[ProjectADM](bytes) - case None => FastFuture.successful(None) - } - } yield maybeProject - case ProjectIdentifierType.SHORTCODE => - for { - maybeIriKey <- getStringValue(identifier.toShortcodeOption) - maybeBytes <- getBytesValue(maybeIriKey) - maybeProject: Option[ProjectADM] <- maybeBytes match { - case Some(bytes) => CacheSerialization.deserialize[ProjectADM](bytes) - case None => FastFuture.successful(None) - } - } yield maybeProject - case ProjectIdentifierType.SHORTNAME => - for { - maybeIriKey <- getStringValue(identifier.toShortnameOption) - maybeBytes <- getBytesValue(maybeIriKey) - maybeProject: Option[ProjectADM] <- maybeBytes match { - case Some(bytes) => CacheSerialization.deserialize[ProjectADM](bytes) - case None => FastFuture.successful(None) - } - } yield maybeProject - } - - val recoverableResultFuture = resultFuture.recover { case e: Exception => - logger.warn("Aborting reading 'ProjectADM' from Redis - {}", e.getMessage) - None - } - - recoverableResultFuture - } - - /** - * Store string or byte array value under key. - * - * @param key the key. - * @param value the value. - */ - def writeStringValue(key: String, value: String)(implicit ec: ExecutionContext): Future[Boolean] = { - - if (key.isEmpty) - throw EmptyKey("The key under which the value should be written is empty. Aborting writing to redis.") - - if (value.isEmpty) - throw EmptyValue("The string value is empty. Aborting writing to redis.") - - val operationFuture: Future[Boolean] = Future { - - val conn: Jedis = pool.getResource - try { - conn.set(key, value) - true - } finally { - conn.close() - } - - } - - val recoverableOperationFuture = operationFuture.recover { case e: Exception => - // Log any errors. - logger.warn("Writing to Redis failed - {}", e.getMessage) - false - } - - recoverableOperationFuture - } - - /** - * Get value stored under the key as a string. - * - * @param maybeKey the key. - */ - def getStringValue(maybeKey: Option[String])(implicit ec: ExecutionContext): Future[Option[String]] = { - - val operationFuture: Future[Option[String]] = maybeKey match { - case Some(key) => - Future { - val conn: Jedis = pool.getResource - try { - Option(conn.get(key)) - } finally { - conn.close() - } - } - case None => - FastFuture.successful(None) - } - - val recoverableOperationFuture = operationFuture.recover { case e: Exception => - // Log any errors. - logger.warn("Reading string from Redis failed, {}", e) - None - } - - recoverableOperationFuture - } - - /** - * Removes values for the provided keys. Any invalid keys are ignored. - * - * @param keys the keys. - */ - def removeValues(keys: Set[String])(implicit ec: ExecutionContext): Future[Boolean] = { - - logger.debug("removeValues - {}", keys) - - val operationFuture: Future[Boolean] = Future { - // del takes a vararg so I nee to convert the set to a swq and then to vararg - val conn: Jedis = pool.getResource - try { - conn.del(keys.toSeq: _*) - true - } finally { - conn.close() - } - } - - val recoverableOperationFuture = operationFuture.recover { case e: Exception => - // Log any errors. - logger.warn("Removing keys from Redis failed.", e.getMessage) - false - } - - recoverableOperationFuture - } - - /** - * Flushes (removes) all stored content from the Redis store. - */ - def flushDB(requestingUser: UserADM)(implicit ec: ExecutionContext): Future[CacheServiceFlushDBACK] = { - - if (!requestingUser.isSystemUser) { - throw ForbiddenException("Only the system user is allowed to perform this operation.") - } - - val operationFuture: Future[CacheServiceFlushDBACK] = Future { - - val conn: Jedis = pool.getResource - try { - conn.flushDB() - CacheServiceFlushDBACK() - } finally { - conn.close() - } - } - - val recoverableOperationFuture = operationFuture.recover { case e: Exception => - // Log any errors. - logger.warn("Flushing DB failed", e.getMessage) - throw e - } - - recoverableOperationFuture - } - - /** - * Pings the Redis store to see if it is available. - */ - def ping()(implicit ec: ExecutionContext): Future[CacheServiceStatusResponse] = { - val operationFuture: Future[CacheServiceStatusResponse] = Future { - - val conn: Jedis = pool.getResource - try { - conn.ping("test") - CacheServiceStatusOK - } finally { - conn.close() - } - } - - val recoverableOperationFuture = operationFuture.recover { case e: Exception => - CacheServiceStatusNOK - } - - recoverableOperationFuture - } - - /** - * Store string or byte array value under key. - * - * @param key the key. - * @param value the value. - */ - private def writeBytesValue(key: String, value: Array[Byte])(implicit ec: ExecutionContext): Future[Boolean] = { - - if (key.isEmpty) - throw EmptyKey("The key under which the value should be written is empty. Aborting writing to redis.") - - if (value.isEmpty) - throw EmptyValue("The byte array value is empty. Aborting writing to redis.") - - val operationFuture: Future[Boolean] = Future { - val conn: Jedis = pool.getResource - try { - conn.set(key.getBytes, value) - true - } finally { - conn.close() - } - } - - val recoverableOperationFuture = operationFuture.recover { case e: Exception => - // Log any errors. - logger.warn("Writing to Redis failed - {}", e.getMessage) - false - } - - recoverableOperationFuture - } - - /** - * Get value stored under the key as a byte array. If no value is found - * under the key, then a [[None]] is returned.. - * - * @param maybeKey the key. - */ - private def getBytesValue(maybeKey: Option[String])(implicit ec: ExecutionContext): Future[Option[Array[Byte]]] = { - - val operationFuture: Future[Option[Array[Byte]]] = maybeKey match { - case Some(key) => - Future { - val conn = pool.getResource - try { - Option(conn.get(key.getBytes)) - } finally { - conn.close() - } - } - case None => - FastFuture.successful(None) - } - - val recoverableOperationFuture = operationFuture.recover { case e: Exception => - // Log any errors. - logger.warn("Reading byte array from Redis failed - {}", e.getMessage) - None - } - - recoverableOperationFuture - } - -} diff --git a/webapi/src/main/scala/org/knora/webapi/store/cacheservice/serialization/CacheSerialization.scala b/webapi/src/main/scala/org/knora/webapi/store/cacheservice/serialization/CacheSerialization.scala index ffe56ce22a9..48b93392425 100644 --- a/webapi/src/main/scala/org/knora/webapi/store/cacheservice/serialization/CacheSerialization.scala +++ b/webapi/src/main/scala/org/knora/webapi/store/cacheservice/serialization/CacheSerialization.scala @@ -5,59 +5,50 @@ package org.knora.webapi.store.cacheservice.serialization -import com.twitter.chill.MeatLocker import org.knora.webapi.exceptions.CacheServiceException import org.knora.webapi.instrumentation.InstrumentationSupport import java.io.{ByteArrayInputStream, ByteArrayOutputStream, ObjectInputStream, ObjectOutputStream} import scala.concurrent.{ExecutionContext, Future} +import zio._ +import org.knora.webapi.messages.admin.responder.usersmessages.UserADM +import org.knora.webapi.messages.admin.responder.groupsmessages.GroupADM +import org.knora.webapi.messages.store.triplestoremessages.StringLiteralV2 + case class EmptyByteArray(message: String) extends CacheServiceException(message) -object CacheSerialization extends InstrumentationSupport { +object CacheSerialization { /** - * Serialize objects by using plain java serialization. Java serialization is not - * capable to serialize all our objects (e.g., UserADM) and that is why we use the - * [[MeatLocker]], which does some magic and allows our case classes to be - * serializable. + * Serialize objects by using Apache commons. * * @param value the value we want to serialize as a array of bytes. - * @tparam T the type parameter of our value. + * @tparam A the type parameter of our value. */ - def serialize[T](value: T)(implicit ec: ExecutionContext): Future[Array[Byte]] = tracedFuture("redis-serialize") { - - Future { - val boxedItem: MeatLocker[T] = MeatLocker[T](value) + def serialize[A](value: A): Task[Array[Byte]] = + ZIO.attempt { val stream: ByteArrayOutputStream = new ByteArrayOutputStream() - val oos = new ObjectOutputStream(stream) - oos.writeObject(boxedItem) + val oos = new ObjectOutputStream(stream) + oos.writeObject(value) oos.close() stream.toByteArray } - } /** - * Deserialize objects by using plain java serialization. Java serialization is not - * capable to serialize all our objects (e.g., UserADM) and that is why we use the - * [[MeatLocker]], which does some magic and allows our case classes to be - * serializable. + * Deserialize objects by using Apache commons. * - * @tparam T the type parameter of our value. + * @tparam A the type parameter of our value. */ - def deserialize[T](bytes: Array[Byte])(implicit ec: ExecutionContext): Future[Option[T]] = - tracedFuture("redis-deserialize") { - - Future { - if (bytes.isEmpty) { - None - } else { - val ois = new ObjectInputStream(new ByteArrayInputStream(bytes)) - val box = ois.readObject - ois.close() - Some(box.asInstanceOf[MeatLocker[T]].get) - } + def deserialize[A](bytes: Array[Byte]): Task[Option[A]] = + ZIO.attempt { + if (bytes.isEmpty) { + None + } else { + val ois = new ObjectInputStream(new ByteArrayInputStream(bytes)) + val value = ois.readObject + ois.close() + Some(value.asInstanceOf[A]) } - } } diff --git a/webapi/src/main/scala/org/knora/webapi/util/ActorUtil.scala b/webapi/src/main/scala/org/knora/webapi/util/ActorUtil.scala index c771e7718df..5bb7fa37e2a 100644 --- a/webapi/src/main/scala/org/knora/webapi/util/ActorUtil.scala +++ b/webapi/src/main/scala/org/knora/webapi/util/ActorUtil.scala @@ -5,23 +5,67 @@ package org.knora.webapi.util -import akka.actor.{ActorRef, Status} +import akka.actor.ActorRef +import akka.actor.Status import akka.event.LoggingAdapter import akka.http.scaladsl.util.FastFuture import akka.util.Timeout -import org.knora.webapi.exceptions.{ - ExceptionUtil, - RequestRejectedException, - UnexpectedMessageException, - WrapperException -} +import org.knora.webapi.core.Logging +import org.knora.webapi.exceptions.ExceptionUtil +import org.knora.webapi.exceptions.RequestRejectedException +import org.knora.webapi.exceptions.UnexpectedMessageException +import org.knora.webapi.exceptions.WrapperException +import zio._ -import scala.concurrent.{ExecutionContext, Future} +import scala.concurrent.ExecutionContext +import scala.concurrent.Future import scala.reflect.ClassTag -import scala.util.{Failure, Success, Try} +import scala.util.Failure +import scala.util.Success +import scala.util.Try object ActorUtil { + /** + * Transforms ZIO Task returned to the receive method of an actor to a message. Used mainly during the refactoring + * phase, to be able to return ZIO inside an Actor. + * + * It performs the same functionality as [[future2Message]] does, rewritten completely uzing ZIOs. + */ + def zio2Message[A](sender: ActorRef, zioTask: zio.Task[A], log: LoggingAdapter): Unit = + Runtime(ZEnvironment.empty, RuntimeConfig.default @@ Logging.live) + .unsafeRun( + zioTask.fold(ex => handleExeption(ex, sender), success => sender ! success) + ) + + /** + * The "throwable" handling part of `zio2Message`. It analyses the kind of throwable + * and sends the actor, that made the request in the `ask` pattern, the failure response. + * + * @param ex the throwable that needs to be handled. + * @param sender the actor that made the request in the `ask` pattern. + */ + def handleExeption(ex: Throwable, sender: ActorRef): ZIO[Any, Nothing, Unit] = + ex match { + case rejectedEx: RequestRejectedException => + // The error was the client's fault. Log the exception, and also + // let the client know. + ZIO.logDebug(s"This error is presumably the clients fault: $rejectedEx") *> ZIO.succeed( + sender ! akka.actor.Status.Failure(rejectedEx) + ) + + case otherEx: Exception => + // The error wasn't the client's fault. Log the exception, and also + // let the client know. + ZIO.logDebug(s"This error is presumably NOT the clients fault: $otherEx") *> ZIO.succeed( + sender ! akka.actor.Status.Failure(otherEx) + ) *> ZIO.fail(throw otherEx) + + case otherThrowable: Throwable => + // Don't try to recover from a Throwable that isn't an Exception. + ZIO.logDebug(s"Presumably something realy bad has happened: $otherThrowable") *> ZIO.fail(throw otherThrowable) + } + /** * A convenience function that simplifies and centralises error-handling in the `receive` method of supervised Akka * actors that expect to receive messages sent using the `ask` pattern. @@ -210,9 +254,10 @@ object ActorUtil { taskResult: TaskResult[T] <- nextTask.runTask(previousResult) recResult: TaskResult[T] <- taskResult.nextTask match { - case Some(definedNextTask) => runTasksRec(previousResult = Some(taskResult), nextTask = definedNextTask) - case None => FastFuture.successful(taskResult) - } + case Some(definedNextTask) => + runTasksRec(previousResult = Some(taskResult), nextTask = definedNextTask) + case None => FastFuture.successful(taskResult) + } } yield recResult } diff --git a/webapi/src/test/scala/org/knora/webapi/CoreSpec.scala b/webapi/src/test/scala/org/knora/webapi/CoreSpec.scala index 1652dfbdd61..78ff800beb9 100644 --- a/webapi/src/test/scala/org/knora/webapi/CoreSpec.scala +++ b/webapi/src/test/scala/org/knora/webapi/CoreSpec.scala @@ -166,10 +166,10 @@ abstract class CoreSpec(_system: ActorSystem) case Failure(e) => logger.error(s"Loading ontologies into cache failed: ${e.getMessage}") } - logger.info("Flush Redis cache started ...") + logger.info("CacheServiceFlushDB started ...") Try(Await.result(appActor ? CacheServiceFlushDB(KnoraSystemInstances.Users.SystemUser), 15 seconds)) match { - case Success(res) => logger.info("... flushing Redis cache done.") - case Failure(e) => logger.error(s"Flushing Redis cache failed: ${e.getMessage}") + case Success(res) => logger.info("... CacheServiceFlushDB done.") + case Failure(e) => logger.error(s"CacheServiceFlushDB failed: ${e.getMessage}") } } } diff --git a/webapi/src/test/scala/org/knora/webapi/IntegrationSpec.scala b/webapi/src/test/scala/org/knora/webapi/IntegrationSpec.scala index b410273a3a8..901daaee94e 100644 --- a/webapi/src/test/scala/org/knora/webapi/IntegrationSpec.scala +++ b/webapi/src/test/scala/org/knora/webapi/IntegrationSpec.scala @@ -77,14 +77,12 @@ abstract class IntegrationSpec(_config: Config) protected def waitForReadyTriplestore(actorRef: ActorRef): Unit = { logger.info("Waiting for triplestore to be ready ...") implicit val ctx: MessageDispatcher = system.dispatchers.lookup(KnoraDispatchers.KnoraBlockingDispatcher) - val checkTriplestore: ZIO[Any, Throwable, Unit] = for { - checkResult <- ZIO.fromTry( - Try( - Await - .result(actorRef ? CheckTriplestoreRequest(), 1.second.asScala) - .asInstanceOf[CheckTriplestoreResponse] - ) - ) + val checkTriplestore: Task[Unit] = for { + checkResult <- ZIO.attemptBlocking { + Await + .result(actorRef ? CheckTriplestoreRequest(), 1.second.asScala) + .asInstanceOf[CheckTriplestoreResponse] + } value <- if (checkResult.triplestoreStatus == TriplestoreStatus.ServiceAvailable) { @@ -98,11 +96,11 @@ abstract class IntegrationSpec(_config: Config) } } yield value - implicit val rt: Runtime[Clock with Console] = Runtime.default - rt.unsafeRun( - checkTriplestore + Runtime.default.unsafeRun( + (checkTriplestore .retry(ScheduleUtil.schedule) .foldZIO(ex => printLine("Exception Failed"), v => printLine(s"Succeeded with $v")) + ).provide(Console.live) ) } @@ -131,15 +129,3 @@ object ScheduleUtil { case (_, attempt, Decision.Continue(_)) => printLine(s"attempt #$attempt").orDie }) } - -//// ZIO helpers //// -object LegacyRuntime { - - val runtime: Runtime[Clock with Console] = Runtime.default - - /** - * Transforms a [[Task]] into a [[Future]]. - */ - def fromTask[Res](body: => Task[Res]): Future[Res] = - runtime.unsafeRunToFuture(body) -} diff --git a/webapi/src/test/scala/org/knora/webapi/ManagersWithMockedSipi.scala b/webapi/src/test/scala/org/knora/webapi/ManagersWithMockedSipi.scala index e61c8ea3f62..007d11afda7 100644 --- a/webapi/src/test/scala/org/knora/webapi/ManagersWithMockedSipi.scala +++ b/webapi/src/test/scala/org/knora/webapi/ManagersWithMockedSipi.scala @@ -12,9 +12,13 @@ import org.knora.webapi.messages.util.ResponderData import org.knora.webapi.responders.MockableResponderManager import org.knora.webapi.settings._ import org.knora.webapi.store.MockableStoreManager -import org.knora.webapi.store.cacheservice.inmem.CacheServiceInMemImpl +import org.knora.webapi.store.cacheservice.impl.CacheServiceInMemImpl import org.knora.webapi.store.cacheservice.settings.CacheServiceSettings import org.knora.webapi.store.iiif.MockSipiConnector +import org.knora.webapi.store.cacheservice.CacheServiceManager +import zio.Runtime +import zio.ZIO + /** * Mixin trait for running the application with mocked Sipi @@ -27,9 +31,15 @@ trait ManagersWithMockedSipi extends Managers { ) lazy val mockResponders: Map[String, ActorRef] = Map.empty[String, ActorRef] + lazy val cacheServiceManager: CacheServiceManager = Runtime.default + .unsafeRun( + (for (manager <- ZIO.service[CacheServiceManager]) + yield manager).provide(CacheServiceInMemImpl.layer, CacheServiceManager.layer) + ) + lazy val storeManager: ActorRef = context.actorOf( Props( - new MockableStoreManager(mockStoreConnectors = mockStoreConnectors, appActor = self, cs = CacheServiceInMemImpl) + new MockableStoreManager(mockStoreConnectors = mockStoreConnectors, appActor = self, csm = cacheServiceManager) with LiveActorMaker ), name = StoreManagerActorName diff --git a/webapi/src/test/scala/org/knora/webapi/TestContainerRedis.scala b/webapi/src/test/scala/org/knora/webapi/TestContainerRedis.scala index 16d7b1fb5e4..39b75b73aa0 100644 --- a/webapi/src/test/scala/org/knora/webapi/TestContainerRedis.scala +++ b/webapi/src/test/scala/org/knora/webapi/TestContainerRedis.scala @@ -16,13 +16,13 @@ import scala.jdk.CollectionConverters._ */ object TestContainerRedis { - val RedisImageName: DockerImageName = DockerImageName.parse("redis:5") - val RedisContainer = new GenericContainer(RedisImageName) - RedisContainer.withExposedPorts(6379) - RedisContainer.start() + val redisImageName: DockerImageName = DockerImageName.parse("redis:5") + val redisContainer = new GenericContainer(redisImageName) + redisContainer.withExposedPorts(6379) + redisContainer.start() private val portMap = Map( - "app.cache-service.redis.port" -> RedisContainer.getFirstMappedPort + "app.cache-service.redis.port" -> redisContainer.getFirstMappedPort ).asJava // all tests need to be configured with these ports. diff --git a/webapi/src/test/scala/org/knora/webapi/responders/IriLockerSpec.scala b/webapi/src/test/scala/org/knora/webapi/responders/IriLockerSpec.scala index 91ce8bc424d..736622fa363 100644 --- a/webapi/src/test/scala/org/knora/webapi/responders/IriLockerSpec.scala +++ b/webapi/src/test/scala/org/knora/webapi/responders/IriLockerSpec.scala @@ -76,10 +76,10 @@ class IriLockerSpec extends AnyWordSpecLike with Matchers { val testIri: IRI = "http://example.org/test2" val firstApiRequestID = UUID.randomUUID - val firstTestResult = Await.result(runRecursiveTask(testIri, firstApiRequestID, 3), 1.second) + val firstTestResult = Await.result(runRecursiveTask(testIri, firstApiRequestID, 3), 1.second) assert(firstTestResult == SUCCESS) val secondApiRequestID = UUID.randomUUID - val secondTestResult = Await.result(runRecursiveTask(testIri, secondApiRequestID, 3), 1.second) + val secondTestResult = Await.result(runRecursiveTask(testIri, secondApiRequestID, 3), 1.second) assert(secondTestResult == SUCCESS) } diff --git a/webapi/src/test/scala/org/knora/webapi/responders/admin/UsersResponderADMSpec.scala b/webapi/src/test/scala/org/knora/webapi/responders/admin/UsersResponderADMSpec.scala index 2f94e0f7a37..7c9fafb7296 100644 --- a/webapi/src/test/scala/org/knora/webapi/responders/admin/UsersResponderADMSpec.scala +++ b/webapi/src/test/scala/org/knora/webapi/responders/admin/UsersResponderADMSpec.scala @@ -47,13 +47,13 @@ class UsersResponderADMSpec extends CoreSpec(UsersResponderADMSpec.config) with private val timeout: FiniteDuration = 8.seconds - private val rootUser = SharedTestDataADM.rootUser + private val rootUser = SharedTestDataADM.rootUser private val anythingAdminUser = SharedTestDataADM.anythingAdminUser - private val normalUser = SharedTestDataADM.normalUser + private val normalUser = SharedTestDataADM.normalUser private val incunabulaUser = SharedTestDataADM.incunabulaProjectAdminUser - private val imagesProject = SharedTestDataADM.imagesProject + private val imagesProject = SharedTestDataADM.imagesProject private val imagesReviewerGroup = SharedTestDataADM.imagesReviewerGroup implicit val stringFormatter: StringFormatter = StringFormatter.getGeneralInstance @@ -380,7 +380,7 @@ class UsersResponderADMSpec extends CoreSpec(UsersResponderADMSpec.config) with "UPDATE the user's password (by himself)" in { val requesterPassword = Password.make("test").fold(e => throw e.head, v => v) - val newPassword = Password.make("test123456").fold(e => throw e.head, v => v) + val newPassword = Password.make("test123456").fold(e => throw e.head, v => v) responderManager ! UserChangePasswordRequestADM( userIri = SharedTestDataADM.normalUser.id, userUpdatePasswordPayload = UserUpdatePasswordPayloadADM( @@ -408,7 +408,7 @@ class UsersResponderADMSpec extends CoreSpec(UsersResponderADMSpec.config) with "UPDATE the user's password (by a system admin)" in { val requesterPassword = Password.make("test").fold(e => throw e.head, v => v) - val newPassword = Password.make("test654321").fold(e => throw e.head, v => v) + val newPassword = Password.make("test654321").fold(e => throw e.head, v => v) responderManager ! UserChangePasswordRequestADM( userIri = SharedTestDataADM.normalUser.id, @@ -576,10 +576,13 @@ class UsersResponderADMSpec extends CoreSpec(UsersResponderADMSpec.config) with "asked to update the user's project membership" should { "ADD user to project" in { + + // get current project memberships responderManager ! UserProjectMembershipsGetRequestADM(normalUser.id, defaultFeatureFactoryConfig, rootUser) val membershipsBeforeUpdate = expectMsgType[UserProjectMembershipsGetResponseADM](timeout) membershipsBeforeUpdate.projects should equal(Seq()) + // add user to images project (00FF) responderManager ! UserProjectMembershipAddRequestADM( normalUser.id, imagesProject.id, diff --git a/webapi/src/test/scala/org/knora/webapi/store/MockableStoreManager.scala b/webapi/src/test/scala/org/knora/webapi/store/MockableStoreManager.scala index 86100b3937e..bdaa9750d40 100644 --- a/webapi/src/test/scala/org/knora/webapi/store/MockableStoreManager.scala +++ b/webapi/src/test/scala/org/knora/webapi/store/MockableStoreManager.scala @@ -8,11 +8,13 @@ package org.knora.webapi.store import akka.actor.{ActorRef, Props} import org.knora.webapi.core.LiveActorMaker import org.knora.webapi.settings.{KnoraDispatchers, _} -import org.knora.webapi.store.cacheservice.CacheService +import org.knora.webapi.store.cacheservice.api.CacheService import org.knora.webapi.store.iiif.MockableIIIFManager +import zio.ZLayer +import org.knora.webapi.store.cacheservice.CacheServiceManager -class MockableStoreManager(mockStoreConnectors: Map[String, ActorRef], appActor: ActorRef, cs: CacheService) - extends StoreManager(appActor, cs) +class MockableStoreManager(mockStoreConnectors: Map[String, ActorRef], appActor: ActorRef, csm: CacheServiceManager) + extends StoreManager(appActor, csm) with LiveActorMaker { /** diff --git a/webapi/src/test/scala/org/knora/webapi/store/cacheservice/CacheServiceManagerSpec.scala b/webapi/src/test/scala/org/knora/webapi/store/cacheservice/CacheServiceManagerSpec.scala index 1e7294c6b3c..a1dc89ee23f 100644 --- a/webapi/src/test/scala/org/knora/webapi/store/cacheservice/CacheServiceManagerSpec.scala +++ b/webapi/src/test/scala/org/knora/webapi/store/cacheservice/CacheServiceManagerSpec.scala @@ -10,12 +10,10 @@ import org.knora.webapi._ import org.knora.webapi.messages.StringFormatter import org.knora.webapi.messages.admin.responder.projectsmessages.ProjectIdentifierADM import org.knora.webapi.messages.admin.responder.usersmessages.UserIdentifierADM -import org.knora.webapi.messages.store.cacheservicemessages.{ - CacheServiceGetProjectADM, - CacheServiceGetUserADM, - CacheServicePutProjectADM, - CacheServicePutUserADM -} +import org.knora.webapi.messages.store.cacheservicemessages.CacheServiceGetProjectADM +import org.knora.webapi.messages.store.cacheservicemessages.CacheServiceGetUserADM +import org.knora.webapi.messages.store.cacheservicemessages.CacheServicePutProjectADM +import org.knora.webapi.messages.store.cacheservicemessages.CacheServicePutUserADM import org.knora.webapi.sharedtestdata.SharedTestDataADM object CacheServiceManagerSpec { @@ -32,14 +30,14 @@ class CacheServiceManagerSpec extends CoreSpec(CacheServiceManagerSpec.config) { implicit protected val stringFormatter: StringFormatter = StringFormatter.getGeneralInstance - val user = SharedTestDataADM.imagesUser01 + val user = SharedTestDataADM.imagesUser01 val project = SharedTestDataADM.imagesProject - "The RedisManager" should { + "The CacheManager" should { "successfully store a user" in { storeManager ! CacheServicePutUserADM(user) - expectMsg(true) + expectMsg(()) } "successfully retrieve a user by IRI" in { @@ -59,7 +57,7 @@ class CacheServiceManagerSpec extends CoreSpec(CacheServiceManagerSpec.config) { "successfully store a project" in { storeManager ! CacheServicePutProjectADM(project) - expectMsg(true) + expectMsg(()) } "successfully retrieve a project by IRI" in { diff --git a/webapi/src/test/scala/org/knora/webapi/store/cacheservice/config/RedisTestConfig.scala b/webapi/src/test/scala/org/knora/webapi/store/cacheservice/config/RedisTestConfig.scala new file mode 100644 index 00000000000..f85e1aab9df --- /dev/null +++ b/webapi/src/test/scala/org/knora/webapi/store/cacheservice/config/RedisTestConfig.scala @@ -0,0 +1,16 @@ +package org.knora.webapi.store.cacheservice.config + +import zio._ +import org.knora.webapi.store.cacheservice.config.RedisConfig +import org.knora.webapi.testcontainers.RedisTestContainer + +object RedisTestConfig { + val hardcoded: ULayer[RedisConfig] = ZLayer.succeed(RedisConfig("localhost", 6379)) + val redisTestContainer: ZLayer[RedisTestContainer, Nothing, RedisConfig] = { + ZLayer { + for { + rtc <- ZIO.service[RedisTestContainer] + } yield RedisConfig("localhost", rtc.container.getFirstMappedPort()) + }.tap(_ => ZIO.debug(">>> Redis Test Config Initialized <<<")) + } +} diff --git a/webapi/src/test/scala/org/knora/webapi/store/cacheservice/impl/CacheInMemImplSpec.scala b/webapi/src/test/scala/org/knora/webapi/store/cacheservice/impl/CacheInMemImplSpec.scala new file mode 100644 index 00000000000..6050270102a --- /dev/null +++ b/webapi/src/test/scala/org/knora/webapi/store/cacheservice/impl/CacheInMemImplSpec.scala @@ -0,0 +1,116 @@ +/* + * Copyright © 2021 - 2022 Swiss National Data and Service Center for the Humanities and/or DaSCH Service Platform contributors. + * SPDX-License-Identifier: Apache-2.0 + */ + +package org.knora.webapi.store.cacheservice.impl + +import org.knora.webapi.UnitSpec +import org.knora.webapi.messages.StringFormatter +import org.knora.webapi.messages.admin.responder.projectsmessages.ProjectADM +import org.knora.webapi.messages.admin.responder.projectsmessages.ProjectIdentifierADM +import org.knora.webapi.messages.admin.responder.usersmessages.UserADM +import org.knora.webapi.messages.admin.responder.usersmessages.UserIdentifierADM +import org.knora.webapi.sharedtestdata.SharedTestDataADM +import org.knora.webapi.store.cacheservice.api.CacheService +import zio.test.Assertion._ +import zio.test.TestAspect.ignore +import zio.test.TestAspect.timeout +import zio.test._ +import zio.ZLayer + +/** + * This spec is used to test [[org.knora.webapi.store.cacheservice.impl.CacheServiceInMemImpl]]. + */ +object CacheInMemImplSpec extends ZIOSpec[CacheService] { + + StringFormatter.initForTest() + implicit val stringFormatter: StringFormatter = StringFormatter.getGeneralInstance + + private val user: UserADM = SharedTestDataADM.imagesUser01 + private val userWithApostrophe = UserADM( + id = "http://rdfh.ch/users/aaaaaab71e7b0e01", + username = "user_with_apostrophe", + email = "userWithApostrophe@example.org", + givenName = """M\\"Given 'Name""", + familyName = """M\\tFamily Name""", + status = true, + lang = "en" + ) + + private val project: ProjectADM = SharedTestDataADM.imagesProject + + /** + * Defines a layer which encompases all dependencies that are needed for + * for running the tests. + */ + val layer = ZLayer.make[CacheService](CacheServiceInMemImpl.layer) + + def spec = (userTests + projectTests + otherTests) + + val userTests = suite("CacheInMemImplSpec - user")( + test("successfully store a user and retrieve by IRI") { + for { + _ <- CacheService(_.putUserADM(user)) + retrievedUser <- CacheService(_.getUserADM(UserIdentifierADM(maybeIri = Some(user.id)))) + } yield assert(retrievedUser)(equalTo(Some(user))) + } + + test("successfully store a user and retrieve by USERNAME")( + for { + _ <- CacheService(_.putUserADM(user)) + retrievedUser <- CacheService(_.getUserADM(UserIdentifierADM(maybeUsername = Some(user.username)))) + } yield assert(retrievedUser)(equalTo(Some(user))) + ) + + test("successfully store a user and retrieve by EMAIL")( + for { + _ <- CacheService(_.putUserADM(user)) + retrievedUser <- CacheService(_.getUserADM(UserIdentifierADM(maybeEmail = Some(user.email)))) + } yield assert(retrievedUser)(equalTo(Some(user))) + ) + + test("successfully store and retrieve a user with special characters in his name")( + for { + _ <- CacheService(_.putUserADM(userWithApostrophe)) + retrievedUser <- CacheService(_.getUserADM(UserIdentifierADM(maybeIri = Some(userWithApostrophe.id)))) + } yield assert(retrievedUser)(equalTo(Some(userWithApostrophe))) + ) + ) + + val projectTests = suite("CacheInMemImplSpec - project")( + test("successfully store a project and retrieve by IRI")( + for { + _ <- CacheService(_.putProjectADM(project)) + retrievedProject <- CacheService(_.getProjectADM(ProjectIdentifierADM(maybeIri = Some(project.id)))) + } yield assert(retrievedProject)(equalTo(Some(project))) + ) + + test("successfully store a project and retrieve by SHORTCODE")( + for { + _ <- CacheService(_.putProjectADM(project)) + retrievedProject <- + CacheService(_.getProjectADM(ProjectIdentifierADM(maybeShortcode = Some(project.shortcode)))) + } yield assert(retrievedProject)(equalTo(Some(project))) + ) + + test("successfully store a project and retrieve by SHORTNAME")( + for { + _ <- CacheService(_.putProjectADM(project)) + retrievedProject <- + CacheService(_.getProjectADM(ProjectIdentifierADM(maybeShortname = Some(project.shortname)))) + } yield assert(retrievedProject)(equalTo(Some(project))) + ) + ) + + val otherTests = suite("CacheInMemImplSpec - other")( + test("successfully store string value")( + for { + _ <- CacheService(_.putStringValue("my-new-key", "my-new-value")) + retrievedValue <- CacheService(_.getStringValue("my-new-key")) + } yield assert(retrievedValue)(equalTo(Some("my-new-value"))) + ) + + test("successfully delete stored value")( + for { + _ <- CacheService(_.putStringValue("my-new-key", "my-new-value")) + _ <- CacheService(_.removeValues(Set("my-new-key"))) + retrievedValue <- CacheService(_.getStringValue("my-new-key")) + } yield assert(retrievedValue)(equalTo(None)) + ) + ) +} diff --git a/webapi/src/test/scala/org/knora/webapi/store/cacheservice/impl/CacheRedisImplSpec.scala b/webapi/src/test/scala/org/knora/webapi/store/cacheservice/impl/CacheRedisImplSpec.scala new file mode 100644 index 00000000000..980509ce7cb --- /dev/null +++ b/webapi/src/test/scala/org/knora/webapi/store/cacheservice/impl/CacheRedisImplSpec.scala @@ -0,0 +1,92 @@ +/* + * Copyright © 2021 - 2022 Swiss National Data and Service Center for the Humanities and/or DaSCH Service Platform contributors. + * SPDX-License-Identifier: Apache-2.0 + */ + +package org.knora.webapi.store.cacheservice.impl + +import org.knora.webapi.TestContainerRedis +import org.knora.webapi.UnitSpec +import org.knora.webapi.messages.StringFormatter +import org.knora.webapi.messages.admin.responder.projectsmessages.ProjectADM +import org.knora.webapi.messages.admin.responder.projectsmessages.ProjectIdentifierADM +import org.knora.webapi.messages.admin.responder.usersmessages.UserADM +import org.knora.webapi.messages.admin.responder.usersmessages.UserIdentifierADM +import org.knora.webapi.sharedtestdata.SharedTestDataADM +import org.knora.webapi.store.cacheservice.api.CacheService +import org.knora.webapi.store.cacheservice.config.RedisTestConfig +import org.knora.webapi.store.cacheservice.settings.CacheServiceSettings +import org.knora.webapi.testcontainers.RedisTestContainer +import zio._ +import zio.test.Assertion._ +import zio.test.TestAspect._ +import zio.test._ + +/** + * This spec is used to test [[org.knora.webapi.store.cacheservice.impl.CacheServiceRedisImpl]]. + */ +object CacheRedisImplSpec extends ZIOSpec[CacheService & zio.test.Annotations] { + + StringFormatter.initForTest() + implicit val stringFormatter: StringFormatter = StringFormatter.getGeneralInstance + + private val user: UserADM = SharedTestDataADM.imagesUser01 + private val project: ProjectADM = SharedTestDataADM.imagesProject + + /** + * Defines a layer which encompases all dependencies that are needed for + * for running the tests. + */ + val layer = ZLayer.make[CacheService & zio.test.Annotations]( + CacheServiceRedisImpl.layer, + RedisTestConfig.redisTestContainer, + zio.test.Annotations.live, + RedisTestContainer.layer + ) + + def spec = (userTests + projectTests) + + val userTests = suite("CacheRedisImplSpec - user")( + test("successfully store a user and retrieve by IRI") { + for { + _ <- CacheService(_.putUserADM(user)) + retrievedUser <- CacheService(_.getUserADM(UserIdentifierADM(maybeIri = Some(user.id)))) + } yield assert(retrievedUser)(equalTo(Some(user))) + } @@ ignore + + test("successfully store a user and retrieve by USERNAME")( + for { + _ <- CacheService(_.putUserADM(user)) + retrievedUser <- CacheService(_.getUserADM(UserIdentifierADM(maybeUsername = Some(user.username)))) + } yield assert(retrievedUser)(equalTo(Some(user))) + ) @@ ignore + + test("successfully store a user and retrieve by EMAIL")( + for { + _ <- CacheService(_.putUserADM(user)) + retrievedUser <- CacheService(_.getUserADM(UserIdentifierADM(maybeEmail = Some(user.email)))) + } yield assert(retrievedUser)(equalTo(Some(user))) + ) @@ ignore + ) + + val projectTests = suite("CacheRedisImplSpec - project")( + test("successfully store a project and retrieve by IRI")( + for { + _ <- CacheService(_.putProjectADM(project)) + retrievedProject <- CacheService(_.getProjectADM(ProjectIdentifierADM(maybeIri = Some(project.id)))) + } yield assert(retrievedProject)(equalTo(Some(project))) + ) + + test("successfully store a project and retrieve by SHORTCODE")( + for { + _ <- CacheService(_.putProjectADM(project)) + retrievedProject <- + CacheService(_.getProjectADM(ProjectIdentifierADM(maybeShortcode = Some(project.shortcode)))) + } yield assert(retrievedProject)(equalTo(Some(project))) + ) + + test("successfully store a project and retrieve by SHORTNAME")( + for { + _ <- CacheService(_.putProjectADM(project)) + retrievedProject <- + CacheService(_.getProjectADM(ProjectIdentifierADM(maybeShortname = Some(project.shortname)))) + } yield assert(retrievedProject)(equalTo(Some(project))) + ) + ) +} diff --git a/webapi/src/test/scala/org/knora/webapi/store/cacheservice/inmem/CacheServiceInMemImplSpec.scala b/webapi/src/test/scala/org/knora/webapi/store/cacheservice/inmem/CacheServiceInMemImplSpec.scala deleted file mode 100644 index 4d622175d49..00000000000 --- a/webapi/src/test/scala/org/knora/webapi/store/cacheservice/inmem/CacheServiceInMemImplSpec.scala +++ /dev/null @@ -1,69 +0,0 @@ -/* - * Copyright © 2021 - 2022 Swiss National Data and Service Center for the Humanities and/or DaSCH Service Platform contributors. - * SPDX-License-Identifier: Apache-2.0 - */ - -package org.knora.webapi.store.cacheservice.inmem - -import org.knora.webapi.UnitSpec -import org.knora.webapi.messages.StringFormatter -import org.knora.webapi.messages.admin.responder.projectsmessages.{ProjectADM, ProjectIdentifierADM} -import org.knora.webapi.messages.admin.responder.usersmessages.{UserADM, UserIdentifierADM} -import org.knora.webapi.sharedtestdata.SharedTestDataADM - -/** - * This spec is used to test [[org.knora.webapi.store.cacheservice.inmem.CacheServiceInMemImpl]]. - */ -class CacheServiceInMemImplSpec extends UnitSpec() { - - implicit protected val stringFormatter: StringFormatter = StringFormatter.getGeneralInstance - implicit val ec: scala.concurrent.ExecutionContext = scala.concurrent.ExecutionContext.global - - private val user: UserADM = SharedTestDataADM.imagesUser01 - private val project: ProjectADM = SharedTestDataADM.imagesProject - - private val inMemCache: CacheServiceInMemImpl.type = CacheServiceInMemImpl - - "The CacheServiceInMemImpl" should { - - "successfully store a user" in { - val resFuture = inMemCache.putUserADM(user) - resFuture map { res => res should equal(true) } - } - - "successfully retrieve a user by IRI" in { - val resFuture = inMemCache.getUserADM(UserIdentifierADM(maybeIri = Some(user.id))) - resFuture map { res => res should equal(Some(user)) } - } - - "successfully retrieve a user by USERNAME" in { - val resFuture = inMemCache.getUserADM(UserIdentifierADM(maybeUsername = Some(user.username))) - resFuture map { res => res should equal(Some(user)) } - } - - "successfully retrieve a user by EMAIL" in { - val resFuture = inMemCache.getUserADM(UserIdentifierADM(maybeEmail = Some(user.email))) - resFuture map { res => res should equal(Some(user)) } - } - - "successfully store a project" in { - val resFuture = inMemCache.putProjectADM(project) - resFuture map { res => res should equal(true) } - } - - "successfully retrieve a project by IRI" in { - val resFuture = inMemCache.getProjectADM(ProjectIdentifierADM(maybeIri = Some(project.id))) - resFuture map { res => res should equal(Some(project)) } - } - - "successfully retrieve a project by SHORTNAME" in { - val resFuture = inMemCache.getProjectADM(ProjectIdentifierADM(maybeShortname = Some(project.shortname))) - resFuture map { res => res should equal(Some(project)) } - } - - "successfully retrieve a project by SHORTCODE" in { - val resFuture = inMemCache.getProjectADM(ProjectIdentifierADM(maybeShortcode = Some(project.shortcode))) - resFuture map { res => res should equal(Some(project)) } - } - } -} diff --git a/webapi/src/test/scala/org/knora/webapi/store/cacheservice/redis/CacheServiceRedisImplSpec.scala b/webapi/src/test/scala/org/knora/webapi/store/cacheservice/redis/CacheServiceRedisImplSpec.scala deleted file mode 100644 index 9d3464eced7..00000000000 --- a/webapi/src/test/scala/org/knora/webapi/store/cacheservice/redis/CacheServiceRedisImplSpec.scala +++ /dev/null @@ -1,76 +0,0 @@ -/* - * Copyright © 2021 - 2022 Swiss National Data and Service Center for the Humanities and/or DaSCH Service Platform contributors. - * SPDX-License-Identifier: Apache-2.0 - */ - -package org.knora.webapi.store.cacheservice.redis - -import org.knora.webapi.messages.StringFormatter -import org.knora.webapi.messages.admin.responder.projectsmessages.{ProjectADM, ProjectIdentifierADM} -import org.knora.webapi.messages.admin.responder.usersmessages.{UserADM, UserIdentifierADM} -import org.knora.webapi.sharedtestdata.SharedTestDataADM -import org.knora.webapi.store.cacheservice.settings.CacheServiceSettings -import org.knora.webapi.{TestContainerRedis, UnitSpec} - -import scala.concurrent.ExecutionContext - -/** - * This spec is used to test [[org.knora.webapi.store.cacheservice.redis.CacheServiceRedisImpl]]. - * Adding the [[TestContainerRedis.PortConfig]] config will start the Redis container and make it - * available to the test. - */ -class CacheServiceRedisImplSpec extends UnitSpec(TestContainerRedis.PortConfig) { - - implicit protected val stringFormatter: StringFormatter = StringFormatter.getGeneralInstance - implicit val ec: ExecutionContext = ExecutionContext.global - - private val user: UserADM = SharedTestDataADM.imagesUser01 - private val project: ProjectADM = SharedTestDataADM.imagesProject - - private val redisCache: CacheServiceRedisImpl = new CacheServiceRedisImpl( - new CacheServiceSettings(TestContainerRedis.PortConfig) - ) - - "The CacheServiceRedisImpl" should { - - "successfully store a user" in { - val resFuture = redisCache.putUserADM(user) - resFuture map { res => res should equal(true) } - } - - "successfully retrieve a user by IRI" in { - val resFuture = redisCache.getUserADM(UserIdentifierADM(maybeIri = Some(user.id))) - resFuture map { res => res should equal(Some(user)) } - } - - "successfully retrieve a user by USERNAME" in { - val resFuture = redisCache.getUserADM(UserIdentifierADM(maybeUsername = Some(user.username))) - resFuture map { res => res should equal(Some(user)) } - } - - "successfully retrieve a user by EMAIL" in { - val resFuture = redisCache.getUserADM(UserIdentifierADM(maybeEmail = Some(user.email))) - resFuture map { res => res should equal(Some(user)) } - } - - "successfully store a project" in { - val resFuture = redisCache.putProjectADM(project) - resFuture map { res => res should equal(true) } - } - - "successfully retrieve a project by IRI" in { - val resFuture = redisCache.getProjectADM(ProjectIdentifierADM(maybeIri = Some(project.id))) - resFuture map { res => res should equal(Some(project)) } - } - - "successfully retrieve a project by SHORTNAME" in { - val resFuture = redisCache.getProjectADM(ProjectIdentifierADM(maybeShortname = Some(project.shortname))) - resFuture map { res => res should equal(Some(project)) } - } - - "successfully retrieve a project by SHORTCODE" in { - val resFuture = redisCache.getProjectADM(ProjectIdentifierADM(maybeShortcode = Some(project.shortcode))) - resFuture map { res => res should equal(Some(project)) } - } - } -} diff --git a/webapi/src/test/scala/org/knora/webapi/store/cacheservice/serialization/CacheSerializationSpec.scala b/webapi/src/test/scala/org/knora/webapi/store/cacheservice/serialization/CacheSerializationSpec.scala index fb0fb598855..2194858ce11 100644 --- a/webapi/src/test/scala/org/knora/webapi/store/cacheservice/serialization/CacheSerializationSpec.scala +++ b/webapi/src/test/scala/org/knora/webapi/store/cacheservice/serialization/CacheSerializationSpec.scala @@ -6,42 +6,35 @@ package org.knora.webapi.store.cacheservice.serialization import com.typesafe.config.ConfigFactory -import org.knora.webapi.UnitSpec import org.knora.webapi.messages.admin.responder.projectsmessages.ProjectADM import org.knora.webapi.messages.admin.responder.usersmessages.UserADM import org.knora.webapi.sharedtestdata.SharedTestDataADM -object CacheSerializationSpec { - val config = ConfigFactory.parseString(""" - akka.loglevel = "DEBUG" - akka.stdout-loglevel = "DEBUG" - """.stripMargin) -} +import zio.test._ +import zio.test.Assertion._ +import zio.test.TestAspect.{ignore, timeout} +import org.knora.webapi.store.cacheservice.api.CacheService /** * This spec is used to test [[CacheSerialization]]. */ -class CacheSerializationSpec extends UnitSpec(CacheSerializationSpec.config) { - - implicit val ec: scala.concurrent.ExecutionContext = scala.concurrent.ExecutionContext.global +object CacheSerializationSpec extends ZIOSpecDefault { - "serialize and deserialize" should { + private val user = SharedTestDataADM.imagesUser01 + private val project = SharedTestDataADM.imagesProject - "work with the UserADM case class" in { - val user = SharedTestDataADM.imagesUser01 + def spec = suite("CacheSerializationSpec")( + test("successfully serialize and deserialize a user") { for { - serialized <- CacheSerialization.serialize(user) - deserialized: Option[UserADM] <- CacheSerialization.deserialize[UserADM](serialized) - } yield deserialized shouldBe Some(user) - } - - "work with the ProjectADM case class" in { - val project = SharedTestDataADM.imagesProject - for { - serialized <- CacheSerialization.serialize(project) - deserialized: Option[ProjectADM] <- CacheSerialization.deserialize[ProjectADM](serialized) - } yield deserialized shouldBe Some(project) - } - - } + serialized <- CacheSerialization.serialize(user) + deserialized <- CacheSerialization.deserialize[UserADM](serialized) + } yield assert(deserialized)(equalTo(Some(user))) + } @@ ignore + + test("successfully serialize and deserialize a project") { + for { + serialized <- CacheSerialization.serialize(project) + deserialized <- CacheSerialization.deserialize[ProjectADM](serialized) + } yield assert(deserialized)(equalTo(Some(project))) + } + ) } diff --git a/webapi/src/test/scala/org/knora/webapi/testcontainers/RedisTestContainer.scala b/webapi/src/test/scala/org/knora/webapi/testcontainers/RedisTestContainer.scala new file mode 100644 index 00000000000..56af8938901 --- /dev/null +++ b/webapi/src/test/scala/org/knora/webapi/testcontainers/RedisTestContainer.scala @@ -0,0 +1,33 @@ +package org.knora.webapi.testcontainers + +import org.testcontainers.containers.GenericContainer +import org.testcontainers.utility.DockerImageName +import zio._ + +final case class RedisTestContainer(container: GenericContainer[Nothing]) + +object RedisTestContainer { + + /** + * A functional effect that initiates a Redis Testcontainer + */ + val aquireRedisTestContainer: Task[GenericContainer[Nothing]] = ZIO.attemptBlocking { + val RedisImageName: DockerImageName = DockerImageName.parse("redis:5") + val container = new GenericContainer(RedisImageName) + container.withExposedPorts(6379) + container.start() + container + }.orDie.tap(_ => ZIO.debug(">>> aquireRedisTestContainer executed <<<")) + + def releaseRedisTestContainer(container: GenericContainer[Nothing]): URIO[Any, Unit] = ZIO.attemptBlocking { + container.stop() + }.orDie.tap(_ => ZIO.debug(">>> releaseRedisTestContainer executed <<<")) + + val layer: ZLayer[Any, Nothing, RedisTestContainer] = { + ZLayer.scoped { + for { + tc <- ZIO.acquireRelease(aquireRedisTestContainer)(releaseRedisTestContainer(_)).orDie + } yield RedisTestContainer(tc) + }.tap(_ => ZIO.debug(">>> Redis Test Container Initialized <<<")) + } +} From f72e7a01e9396430e88c75c1ab3ec2743e6cf053 Mon Sep 17 00:00:00 2001 From: irinaschubert Date: Tue, 12 Apr 2022 18:02:15 +0200 Subject: [PATCH 2/2] docs(ingest): Add accepted file formats to documentation (DEV-677) (#2038) * add accepted file formats * Update data-formats.md * update sipi path * rename data-formats.md to file-formats.md * update index.md * fix footnote * reset scala setting --- .scalafmt.conf | 2 +- docs/01-introduction/data-formats.md | 22 ---------------------- docs/01-introduction/file-formats.md | 24 ++++++++++++++++++++++++ docs/01-introduction/index.md | 2 +- docs/01-introduction/what-is-knora.md | 12 ++++-------- docs/faq/index.md | 6 +++--- mkdocs.yml | 2 +- 7 files changed, 34 insertions(+), 36 deletions(-) delete mode 100644 docs/01-introduction/data-formats.md create mode 100644 docs/01-introduction/file-formats.md diff --git a/.scalafmt.conf b/.scalafmt.conf index e94dd850831..bf360eab5bf 100644 --- a/.scalafmt.conf +++ b/.scalafmt.conf @@ -1,6 +1,6 @@ version = "2.7.5" maxColumn = 120 -align.preset = most +align.preset = some align.multiline = false continuationIndent.defnSite = 2 assumeStandardLibraryStripMargin = true diff --git a/docs/01-introduction/data-formats.md b/docs/01-introduction/data-formats.md deleted file mode 100644 index 16df023bf65..00000000000 --- a/docs/01-introduction/data-formats.md +++ /dev/null @@ -1,22 +0,0 @@ - - -# Data Formats in DSP-API - -As explained in [What Is DSP and DSP-API (previous Knora)?](what-is-knora.md), the DSP stores data -in a small number of formats that are suitable for long-term preservation while -facilitating data reuse. - -The following is a non-exhaustive list of data formats and how their content -can be stored and managed by DSP-API: - -| Original Format | Format in DSP | -|----------------------------------------------|------------------------------------------------------------------------------------------------------------------------------| -| Text (XML, LaTeX, Microsoft Word, etc.) | [Knora resources](../03-apis/api-v2/editing-resources.md) (RDF) containing [Standoff/RDF](standoff-rdf.md) | -| Tabular data, including relational databases | [Knora resources](../03-apis/api-v2/editing-resources.md) | -| Data in tree or graph structures | [Knora resources](../03-apis/api-v2/editing-resources.md) | -| Images (JPEG, PNG, etc.) | JPEG 2000 files stored by [Sipi](https://github.com/dhlab-basel/Sipi) | -| Audio and video files | Audio and video files stored by [Sipi](https://github.com/dhlab-basel/Sipi) (in archival formats to be determined) | -| PDF | Can be stored by Sipi, but data reuse is improved by extracting the text for storage as [Standoff/RDF](standoff-rdf.md) | diff --git a/docs/01-introduction/file-formats.md b/docs/01-introduction/file-formats.md new file mode 100644 index 00000000000..173b5518b77 --- /dev/null +++ b/docs/01-introduction/file-formats.md @@ -0,0 +1,24 @@ + + +# File Formats in DSP-API + +Currently, only a limited number of file formats is accepted to be uploaded onto DSP. Some metadata is extracted from the files during the ingest but the file formats are not validated. Only image file formats are currently migrated into another format. Both, the migrated version of the file and the original are kept. + +The following table shows the accepted file formats: + +| Category | Accepted format | Converted during ingest? | +| --------------------- | ------------------------- | -------------------------------------------------------------------------- | +| Text, XML1 | TXT, XML, XSL, XSD | No | +| Tables | CSV, XLS, XLSX | No | +| 2D Images | JPEG, PNG, TIFF, JP2 | Yes, converted to JPEG 2000 by [Sipi](https://github.com/dasch-swiss/sipi) | +| Audio | MPEG (MP3), MP4, WAV | No | +| Video | MP4 | No | +| Office | PDF, DOC, DOCX, PPT, PPTX | No | +| Archives | ZIP, TAR, ISO, GZIP, 7Z | No | + + +1: If your XML files represent text with markup (e.g. [TEI/XML](http://www.tei-c.org/)), +the recommended approach is to allow Knora to store it as [Standoff/RDF](standoff-rdf.md). diff --git a/docs/01-introduction/index.md b/docs/01-introduction/index.md index 7da6fbba7f8..7f6bcb15a6a 100644 --- a/docs/01-introduction/index.md +++ b/docs/01-introduction/index.md @@ -6,6 +6,6 @@ # Introduction * [What Is DSP and DSP-API (previous Knora)?](what-is-knora.md) -* [Data Formats in DSP-API](data-formats.md) +* [File Formats in DSP-API](file-formats.md) * [Standoff/RDF Text Markup](standoff-rdf.md) * [An Example Project](example-project.md) diff --git a/docs/01-introduction/what-is-knora.md b/docs/01-introduction/what-is-knora.md index 8040cd1141d..2a4aac52c55 100644 --- a/docs/01-introduction/what-is-knora.md +++ b/docs/01-introduction/what-is-knora.md @@ -23,15 +23,11 @@ DSP solves this problem by keeping the data alive. You can query all the data in a DSP repository, not just the metadata. You can import thousands of databases into DSP, and run queries that search through all of them at once. -Another problem is that researchers use a multitude of different data formats, many of +Another problem is that researchers use a multitude of different file formats, many of which are proprietary and quickly become obsolete. It is not practical to maintain -all the programs that were used to create and read old data files, or even -all the operating systems that these programs ran on. - -Instead of preserving all these data formats, DSP supports -the conversion of all sorts of data to a [small number of formats](data-formats.md) -that are suitable for long-term preservation, and that maintain the data's meaning and -structure: +all the programs that were used to create and read old files, or even +all the operating systems that these programs ran on. Therefore, DSP only accepts a +certain number of [file formats](file-formats.md). - Non-binary data is stored as [RDF](http://www.w3.org/TR/2014/NOTE-rdf11-primer-20140624/), in a dedicated diff --git a/docs/faq/index.md b/docs/faq/index.md index b3a7f86d478..e21cd3f3b1d 100644 --- a/docs/faq/index.md +++ b/docs/faq/index.md @@ -5,11 +5,11 @@ # Frequently Asked Questions -## Data Formats +## File Formats -### What data formats does Knora store? +### What file formats does Knora store? -See [Data Formats in Knora](../01-introduction/data-formats.md). +See [File Formats in Knora](../01-introduction/file-formats.md). ### Does Knora store XML files? diff --git a/mkdocs.yml b/mkdocs.yml index a5dde90d529..505486de812 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -10,7 +10,7 @@ nav: - Introduction: - Index: 01-introduction/index.md - What is DSP?: 01-introduction/what-is-knora.md - - Data Formats in DSP-API: 01-introduction/data-formats.md + - File Formats in DSP-API: 01-introduction/file-formats.md - Standoff/RDF Text Markup: 01-introduction/standoff-rdf.md - An Example Project: 01-introduction/example-project.md - DSP Ontologies: