feat: use docker commands for stack handling (DEV-1530)

MANIFEST.in

@@ -4,3 +4,4 @@ include knora/dsplib/schemas/lists-only.json
 include knora/dsplib/schemas/resources-only.json
 include knora/dsplib/schemas/properties-only.json
 include knora/dsplib/schemas/data.xsd
+include knora/dsplib/docker

docs/dsp-tools-usage.md

@@ -254,61 +254,45 @@ In order to upload data incrementally the procedure described [here](dsp-tools-x
 
 
 
-## Start a DSP-stack on your local machine (for DaSCH-internal use only)
+## Start a DSP stack on your local machine 
+
+DSP-API is the heart of the DaSCH software - it is a server application for storing data from the Humanities.
+DSP-APP is our generic user interface, i.e. the viewer to look at the data. It's a server application, too.
 
 For testing purposes, it is sometimes necessary to run DSP-API and DSP-APP on a local machine. But the startup 
 and shutdown of API and APP can be complicated: Both repos need to be cloned locally, a `git pull` has to be executed 
-from time to time to stay up to date, and then there are several commands for each repository to remember. 
-
-Another challenge is the software that DSP depends upon: JDK, node, npm, Angular, etc. should be kept up to date. And
-it might happen that a dependency is replaced, e.g. JDK 11 Zulu by JDK 17 Temurin. An non-developer can quickly get lost
-in this jungle. 
-
-That's why dsp-tools offers some commands to facilitate the handling of API and APP. These commands 
-
- - clone the repos to `~/.dsp-tools`, and keep them up to date.
- - check every time if the dependencies are up to date, and give you advice how to update them, if necessary.
- - pass on the right commands to APP and API, even if the correct usage of these commands changes over time.
- - make sure that the repos don't get cluttered with old files over time.
- - log their activity in `~/.dsp-tools`, so you can check the logs for troubleshooting, if necessary.
-
-The only requirements for these commands are:
-
- - the initial installation of all software that you accomplished when you started working at DaSCH
- - Docker must be running (for DSP-API only)
-
-Please note that these commands were developed for DaSCH-internal use only. They only work on Macs that have the 
-required software installed that makes it possible to run the API and APP. We don't offer support or troubleshooting 
-for these commands.
-
+from time to time to stay up to date, and then there are several commands for each repository to remember. And your 
+local clone of the API might get cluttered with old files over time.
 
-### Start DSP-API
+Another challenge is the software that DSP depends on, and that should be kept up to date: JDK, node, npm, Angular, 
+etc. And it might happen that a dependency is replaced, e.g. JDK 11 Zulu by JDK 17 Temurin. A non-developer can quickly 
+get lost in this jungle. 
 
+That's why dsp-tools offers a command to facilitate the handling of API and APP:
 ```
-dsp-tools start-api
+dsp-tools start-stack
 ```
 
-This command makes a clone of the [DSP-API repository](https://github.com/dasch-swiss/dsp-api) into `~/.dsp-tools`. If
-it finds an existing clone there, it runs `git pull` instead. If the API is already running, it shuts down the old 
-instance, deletes all data that was in it, and starts a new one. If the dependencies are outdated or not installed, a 
-warning is printed to the console.
-
-
-### Shut DSP-API down
+This calls Docker commands to start up the latest deployed versions of API and the APP, i.e. the versions that are 
+running on [https://admin.dasch.swiss](https://admin.dasch.swiss). It's no longer necessary to have two local clones, 
+nor to execute `make` commands inside them. The only requirement for this command is that Docker must be running. It 
+can even be executed on a Windows machine without any of the software dependencies.
 
+When your work is done, shut down API and APP with
 ```
-dsp-tools stop-api
+dsp-tools stop-stack
 ```
 
-This command shuts DSP-API down, deletes all Docker volumes, and removes temporary files.
-
-
-### Start DSP-APP
+This deletes all Docker volumes, and removes all data that was in the database.
 
-```
-dsp-tools start-app
-```
+Some notes:
 
-This command makes a clone of the [DSP-APP repository](https://github.com/dasch-swiss/dsp-app) into `~/.dsp-tools`. If
-it finds an existing clone there, it runs `git pull` instead. Then, it installs the `npm` dependencies and runs DSP-APP.
-You must keep the terminal window open as long as you work with the APP. Then, you can press `Ctrl` + `C` to stop DSP-APP.
+ - As long as you want to keep the data in the database, don't execute `dsp-tools stop-stack`. It is possible to 
+   leave the API up for a long time. If you want to save power, you can pause Docker. When you resume it, the API 
+   will still be running, in the state how you left it.
+ - You can also send your computer to sleep while the DSP stack is running. For this, you don't even need to pause 
+   Docker.
+ - This command provides you with the latest released & deployed version of API and APP. If you want to run a 
+   development version, you need to have the software dependencies installed, clone the repos of DSP-API and DSP-APP,
+   and execute `make` commands therein.
+ - This command was developed for DaSCH-internal use only. We don't offer support or troubleshooting for it.

knora/dsp_tools.py

@@ -4,14 +4,9 @@
 import argparse
 import datetime
 import os
-import re
-import subprocess
 import sys
 from importlib.metadata import version
 
-import requests
-import yaml
-
 from knora.dsplib.utils.excel_to_json_lists import excel2lists, validate_lists_section_with_schema
 from knora.dsplib.utils.excel_to_json_project import excel2json
 from knora.dsplib.utils.excel_to_json_properties import excel2properties
@@ -22,6 +17,7 @@
 from knora.dsplib.utils.onto_get import get_ontology
 from knora.dsplib.utils.onto_validate import validate_project
 from knora.dsplib.utils.shared import validate_xml_against_schema
+from knora.dsplib.utils.stack_handling import start_stack, stop_stack
 from knora.dsplib.utils.xml_upload import xml_upload
 from knora.excel2xml import excel2xml
 
@@ -149,19 +145,14 @@ def program(user_args: list[str]) -> None:
     parser_excel2xml.add_argument('shortcode', help='Shortcode of the project that this data belongs to')
     parser_excel2xml.add_argument('default_ontology', help='Name of the ontology that this data belongs to')
 
-    # startup DSP-API
-    parser_stackup = subparsers.add_parser('start-api', help='Startup a local instance of DSP-API')
-    parser_stackup.set_defaults(action='start-api')
+    # startup DSP stack
+    parser_stackup = subparsers.add_parser('start-stack', help='Startup a local instance of the DSP stack (API and APP)')
+    parser_stackup.set_defaults(action='start-stack')
 
     # shutdown DSP-API
-    parser_stackdown = subparsers.add_parser('stop-api', help='Shut down the local instance of DSP-API, delete '
-                                                                'volumes, clean SIPI folders')
-    parser_stackdown.set_defaults(action='stop-api')
-
-    # startup DSP-APP
-    parser_dsp_app = subparsers.add_parser('start-app', help='Startup a local instance of DSP-APP')
-    parser_dsp_app.set_defaults(action='start-app')
-
+    parser_stackdown = subparsers.add_parser('stop-stack', help='Shut down the local instance of the DSP stack, and '
+                                                                'delete all data in it')
+    parser_stackdown.set_defaults(action='stop-stack')
 
 
     # call the requested action
@@ -236,29 +227,11 @@ def program(user_args: list[str]) -> None:
         excel2xml(datafile=args.datafile,
                   shortcode=args.shortcode,
                   default_ontology=args.default_ontology)
-    elif args.action == 'start-api' and not sys.platform.startswith('win'):
-        try:
-            response = requests.get("https://raw.githubusercontent.com/dasch-swiss/dsp-api/main/.github/actions/preparation/action.yml")
-            action = yaml.safe_load(response.content)
-            for step in action.get("runs", {}).get("steps", {}):
-                if re.search("(JDK)|(Java)", step.get("name", "")):
-                    distribution = step.get("with", {}).get("distribution", "").lower()
-                    java_version = step.get("with", {}).get("java-version", "").lower()
-        except:
-            distribution = "temurin"
-            java_version = "17"
-        subprocess.run(['/bin/bash', os.path.join(current_dir, 'dsplib/utils/start-api.sh'), distribution, java_version])
-    elif args.action == 'stop-api' and not sys.platform.startswith('win'):
-        subprocess.run(['/bin/bash', os.path.join(current_dir, 'dsplib/utils/stop-api.sh')])
-    elif args.action == 'start-app' and not sys.platform.startswith('win'):
-        try:
-            subprocess.run(['/bin/bash', os.path.join(current_dir, 'dsplib/utils/start-app.sh')])
-        except KeyboardInterrupt:
-            print("\n\n"
-                  "================================\n"
-                  "You successfully stopped the APP\n"
-                  "================================")
-            exit(0)
+    elif args.action == 'start-stack':
+        start_stack()
+    elif args.action == 'stop-stack':
+        stop_stack()
+
 
 
 def main() -> None:

knora/dsplib/docker/docker-compose.yml

@@ -0,0 +1,70 @@
+version: '3.7'
+
+services:
+
+  app:
+    image: daschswiss/dsp-app:v10.11.0-11-g4356dea  # after every deployment (fortnightly), check latest tag at https://hub.docker.com/r/daschswiss/dsp-app/tags
+    ports:
+      - "4200:4200"
+    networks:
+      - knora-net
+
+  db:
+    image: daschswiss/apache-jena-fuseki:2.0.10  # after every deployment (fortnightly), check latest tag at https://hub.docker.com/r/daschswiss/apache-jena-fuseki/tags
+    ports:
+      - "3030:3030"
+    networks:
+      - knora-net
+    environment:
+      - TZ=Europe/Zurich
+      - ADMIN_PASSWORD=test
+      - JVM_ARGS=-Xmx3G
+
+  sipi:
+    image: daschswiss/knora-sipi:24.0.8-18-gb8eaadf  # after every deployment (fortnightly), check latest tag at https://hub.docker.com/r/daschswiss/knora-sipi/tags
+    ports:
+      - "1024:1024"
+    volumes:
+      - .:/docker
+    networks:
+      - knora-net
+    environment:
+      - TZ=Europe/Zurich
+      - SIPI_EXTERNAL_PROTOCOL=http
+      - SIPI_EXTERNAL_HOSTNAME=0.0.0.0
+      - SIPI_EXTERNAL_PORT=1024
+      - SIPI_WEBAPI_HOSTNAME=api
+      - SIPI_WEBAPI_PORT=3333
+      - KNORA_WEBAPI_KNORA_API_EXTERNAL_HOST=0.0.0.0
+      - KNORA_WEBAPI_KNORA_API_EXTERNAL_PORT=3333
+    command: --config=/docker/sipi.docker-config.lua
+
+  api:
+    image: daschswiss/knora-api:24.0.8-18-gb8eaadf  # after every deployment (fortnightly), check latest tag at https://hub.docker.com/r/daschswiss/knora-api/tags
+    depends_on:
+      - sipi
+      - db
+    ports:
+      - "3333:3333"
+    networks:
+      - knora-net
+    environment:
+      - TZ=Europe/Zurich
+      - KNORA_AKKA_LOGLEVEL=DEBUG
+      - KNORA_AKKA_STDOUT_LOGLEVEL=DEBUG
+      - KNORA_WEBAPI_TRIPLESTORE_HOST=db
+      - KNORA_WEBAPI_TRIPLESTORE_DBTYPE=fuseki
+      - KNORA_WEBAPI_SIPI_INTERNAL_HOST=sipi
+      - KNORA_WEBAPI_TRIPLESTORE_FUSEKI_REPOSITORY_NAME=knora-test
+      - KNORA_WEBAPI_TRIPLESTORE_FUSEKI_USERNAME=admin
+      - KNORA_WEBAPI_TRIPLESTORE_FUSEKI_PASSWORD=test
+      - KNORA_WEBAPI_CACHE_SERVICE_ENABLED=true
+      - KNORA_WEBAPI_CACHE_SERVICE_REDIS_HOST=redis
+      - KNORA_WEBAPI_CACHE_SERVICE_REDIS_PORT=6379
+      - KNORA_WEBAPI_ALLOW_RELOAD_OVER_HTTP=true
+      - KNORA_WEBAPI_KNORA_API_EXTERNAL_HOST=0.0.0.0
+      - KNORA_WEBAPI_KNORA_API_EXTERNAL_PORT=3333
+
+networks:
+  knora-net:
+    name: knora-net

knora/dsplib/utils/stack_handling.py

@@ -0,0 +1,67 @@
+import re
+import subprocess
+import time
+
+import requests
+
+from knora.dsplib.models.helpers import BaseError
+
+
+def start_stack() -> None:
+    """
+    Start the Docker containers of DSP-API, and load some basic data models and data.
+    """
+    # start up the fuseki database
+    completed_process = subprocess.run("docker compose up db -d", shell=True, cwd="knora/dsplib/docker")
+    if not completed_process or completed_process.returncode != 0:
+        raise BaseError("Cannot start the API: Error while executing 'docker compose up db -d'")
+    time.sleep(5)
+
+    # inside fuseki, create the "knora-test" repository
+    url_prefix = "https://raw.githubusercontent.com/dasch-swiss/dsp-api/main"
+    repo_template = requests.get(f"{url_prefix}/webapi/scripts/fuseki-repository-config.ttl.template").text
+    repo_template = re.sub(r"@REPOSITORY@", "knora-test", repo_template)
+    response = requests.post(
+        url="http://0.0.0.0:3030/$/datasets",
+        files={"file": ("file.ttl", repo_template, "text/turtle; charset=utf8")},
+        auth=("admin", "test")
+    )
+    if not response.ok:
+        raise BaseError("Cannot start the API: Error when creating the 'knora-test' repository")
+
+    # load some basic ontos and data into the repository
+    graph_prefix = "http://0.0.0.0:3030/knora-test/data?graph="
+    ttl_files = [
+        (f"{url_prefix}/knora-ontologies/knora-admin.ttl", f"{graph_prefix}http://www.knora.org/ontology/knora-admin"),
+        (f"{url_prefix}/knora-ontologies/knora-base.ttl", f"{graph_prefix}http://www.knora.org/ontology/knora-base"),
+        (f"{url_prefix}/knora-ontologies/standoff-onto.ttl", f"{graph_prefix}http://www.knora.org/ontology/standoff"),
+        (f"{url_prefix}/knora-ontologies/standoff-data.ttl", f"{graph_prefix}http://www.knora.org/data/standoff"),
+        (f"{url_prefix}/knora-ontologies/salsah-gui.ttl", f"{graph_prefix}http://www.knora.org/ontology/salsah-gui"),
+        (f"{url_prefix}/test_data/all_data/admin-data-minimal.ttl", f"{graph_prefix}http://www.knora.org/data/admin"),
+        (f"{url_prefix}/test_data/all_data/permissions-data-minimal.ttl",
+         f"{graph_prefix}http://www.knora.org/data/permissions")
+    ]
+    for ttl_file, graph in ttl_files:
+        ttl_text = requests.get(ttl_file).text
+        response = requests.post(
+            url=graph,
+            files={"file": ("file.ttl", ttl_text, "text/turtle; charset: utf-8")},
+            auth=("admin", "test"),
+        )
+        if not response.ok:
+            raise BaseError(f"Cannot start the API: Error when creating graph '{graph}'")
+
+    # get sipi.docker-config.lua
+    docker_config_lua_text = requests.get(f"{url_prefix}/sipi/config/sipi.docker-config.lua").text
+    with open("knora/dsplib/docker/sipi.docker-config.lua", "w") as f:
+        f.write(docker_config_lua_text)
+
+    # startup all other components
+    subprocess.run("docker compose up -d", shell=True, cwd="knora/dsplib/docker")
+
+
+def stop_stack() -> None:
+    """
+    Shut down the Docker containers of DSP-API and delete all data that is in them.
+    """
+    subprocess.run("docker compose down", shell=True, cwd="knora/dsplib/docker")