feat: startup API and APP with dsp-tools (DEV-126) (#246)

docs/dsp-tools-usage.md

@@ -218,7 +218,6 @@ about it [here](./dsp-tools-excel2xml.md).
 
 
 
-
 ## Replace internal IDs with IRIs in XML file
 
 ```bash
@@ -238,3 +237,54 @@ Note that internal IDs and IRIs cannot be mixed. The input XML file has to be pr
 contains the mapping from internal IDs to IRIs. This JSON file is generated after each successful `xmlupload`.
 
 In order to upload data incrementally the procedure described [here](dsp-tools-xmlupload.md#incremental-xml-upload) is recommended.
+
+
+
+## Start a DSP-stack on your local machine (for DaSCH-internal use only)
+
+For testing purposes, it is sometimes necessary to have DSP-API and/or DSP-APP running on a local machine. The startup 
+and shutdown of API and APP can be complicated: Both repos need to be cloned locally, a `git pull` has to be executed 
+from time to time to stay up to date, and then there are several commands for each repository to remember. 
+That's why dsp-tools offers some commands to facilitate the handling of API and APP. 
+
+The only requirements for these commands are:
+
+ - the initial installation of all software that you accomplished when you started working at DaSCH
+ - Docker must be running
+
+It isn't necessary anymore to clone DSP-API and DSP-APP, to navigate to these repos and execute commands there.
+
+Please note that these commands were developed for DaSCH-internal use only. They only work on Macs that have the 
+required software installed that makes it possible to run the API and APP. We don't offer support or troubleshooting 
+for these commands.
+
+
+### Start DSP-API
+
+```
+dsp-tools start-api
+```
+
+This command makes a clone of the [DSP-API repository](https://github.com/dasch-swiss/dsp-api) into `~/.dsp-tools`. If
+it finds an existing clone there, it runs `git pull` instead. If the API is already running, it shuts down the old 
+instance and starts a new one. If the dependencies are outdated or not installed, a warning is printed to the console.
+
+
+### Shut DSP-API down
+
+```
+dsp-tools stop-api
+```
+
+This command shuts DSP-API down, deletes all Docker volumes, and removes temporary files.
+
+
+### Start DSP-APP
+
+```
+dsp-tools start-app
+```
+
+This command makes a clone of the [DSP-APp repository](https://github.com/dasch-swiss/dsp-app) into `~/.dsp-tools`. If
+it finds an existing clone there, it runs `git pull` instead. Then, it installs the `npm` dependencies and runs DSP-APP.
+You must keep the terminal window open as long as you work with the APP. Then, you can press `Ctrl` + `C` to stop DSP-APP.

docs/index.md

@@ -36,4 +36,5 @@ dsp-tools helps you with the following tasks:
   is already structured according to the DSP specifications.
 - [The module excel2xml](./dsp-tools-usage.md#use-the-module-excel2xml-to-convert-a-data-source-to-xml) provides helper
   methods that can be used in a Python script to convert data from a tabular format into XML.
-
+- [`dsp-tools start-api / stop-api / start-app`](./dsp-tools-usage.md#start-a-dsp-stack-on-your-local-machine-for-dasch-internal-use-only)
+  assist you in running a DSP software stack on your local machine.

knora/dsp_tools.py

@@ -3,6 +3,8 @@
 """
 import argparse
 import datetime
+import os
+import subprocess
 import sys
 from importlib.metadata import version
 
@@ -14,8 +16,8 @@
 from knora.dsplib.utils.onto_create_ontology import create_project
 from knora.dsplib.utils.onto_get import get_ontology
 from knora.dsplib.utils.onto_validate import validate_project
-from knora.dsplib.utils.xml_upload import xml_upload
 from knora.dsplib.utils.shared import validate_xml_against_schema
+from knora.dsplib.utils.xml_upload import xml_upload
 from knora.excel2xml import excel2xml
 
 
@@ -41,6 +43,7 @@ def program(user_args: list[str]) -> None:
     default_pw = 'test'
     dsp_tools_version = version('dsp-tools')
     now = datetime.datetime.now()
+    current_dir = os.path.dirname(os.path.realpath(__file__))
 
     # parse the arguments of the command line
     parser = argparse.ArgumentParser(description=f'dsp-tools (Version {dsp_tools_version}) DaSCH Service Platform data '
@@ -135,6 +138,20 @@ def program(user_args: list[str]) -> None:
     parser_excel2xml.add_argument('shortcode', help='Shortcode of the project that this data belongs to')
     parser_excel2xml.add_argument('default_ontology', help='Name of the ontology that this data belongs to')
 
+    # startup DSP-API
+    parser_stackup = subparsers.add_parser('start-api', help='Startup a local instance of DSP-API')
+    parser_stackup.set_defaults(action='start-api')
+
+    # shutdown DSP-API
+    parser_stackdown = subparsers.add_parser('stop-api', help='Shut down the local instance of DSP-API, delete '
+                                                                'volumes, clean SIPI folders')
+    parser_stackdown.set_defaults(action='stop-api')
+
+    # startup DSP-APP
+    parser_dsp_app = subparsers.add_parser('start-app', help='Startup a local instance of DSP-APP')
+    parser_dsp_app.set_defaults(action='start-app')
+
+
 
     # call the requested action
     args = parser.parse_args(user_args)
@@ -204,6 +221,12 @@ def program(user_args: list[str]) -> None:
         excel2xml(datafile=args.datafile,
                   shortcode=args.shortcode,
                   default_ontology=args.default_ontology)
+    elif args.action == 'start-api' and not sys.platform.startswith('win'):
+        output = subprocess.run(['/bin/bash', os.path.join(current_dir, 'dsplib/utils/start-api.sh')])
+    elif args.action == 'stop-api' and not sys.platform.startswith('win'):
+        output = subprocess.run(['/bin/bash', os.path.join(current_dir, 'dsplib/utils/stop-api.sh')])
+    elif args.action == 'start-app' and not sys.platform.startswith('win'):
+        output = subprocess.run(['/bin/bash', os.path.join(current_dir, 'dsplib/utils/start-app.sh')])
 
 
 def main() -> None:

knora/dsplib/utils/start-api.sh

@@ -0,0 +1,41 @@
+#! /bin/bash
+# make this file executable with chmod u+x (filename).sh
+set -u  # exit if an uninitialised variable is used (https://www.davidpashley.com/articles/writing-robust-shell-scripts/)
+set -e  # exit if any statement returns a non-true return value (https://www.davidpashley.com/articles/writing-robust-shell-scripts/)
+
+# only allow to run this command if Docker is running
+[[ $(docker stats --no-stream 2>/dev/null ) == "" ]] && printf "\e[31mERROR: Please start Docker before running DSP-API.\e[0m\n" && return
+
+# check dependencies
+echo "check for outdated dependencies..."
+[[ "$(brew outdated)" != "" ]] && printf "\e[31mWARNING: Some of your Homebrew formulas/casks are outdated. Please execute \"brew upgrade\"\e[0m\n"
+[[ "$(java --version)" =~ .*[Tt]emurin-17.* ]] || printf "\e[31mWARNING: Your JDK seems to be outdated. Please install JDK 17 Temurin.\e[0m\n"
+if echo -e "GET http://google.com HTTP/1.0\n\n" | nc google.com 80 -w 10 > /dev/null 2>&1; then
+    # pip should only be called if there is an internet connection
+    [[ "$(pip list --outdated)" =~ .*dsp-tools.* ]] && printf "\e[31mWARNING: Your version of dsp-tools is outdated. Please update it with \"pip install --upgrade dsp-tools\"\e[0m\n"
+    [[ "$(pip list --outdated)" != "" ]] && printf "\e[31mWARNING: Some of your pip packages are outdated. List them with \"pip list --outdated\" and consider updating them with \"pip install --upgrade (package)\"\e[0m\n"
+fi
+
+logfile="../dsp-api-stackup.log"
+
+cd ~
+mkdir -p .dsp-tools
+cd .dsp-tools
+if [[ ! -d dsp-api ]]; then
+    echo "git clone https://github.com/dasch-swiss/dsp-api.git" 2>&1 | tee -a "$logfile"
+    git clone https://github.com/dasch-swiss/dsp-api.git >>"$logfile" 2>&1
+fi
+cd dsp-api
+rm -f "$logfile"
+echo "make stack-down-delete-volumes..." 2>&1 | tee -a "$logfile"
+make stack-down-delete-volumes >>"$logfile" 2>&1
+if echo -e "GET http://google.com HTTP/1.0\n\n" | nc google.com 80 -w 10 > /dev/null 2>&1; then
+    # only pull if there is an internet connection
+    echo "git pull ..." 2>&1 | tee -a "$logfile"
+    git pull >>"$logfile" 2>&1
+fi
+echo "make init-db-test..." 2>&1 | tee -a "$logfile"
+make init-db-test >>"$logfile" 2>&1
+echo "make stack-up..." 2>&1 | tee -a "$logfile"
+make stack-up >>"$logfile" 2>&1
+echo "DSP-API is up and running."   # due to "set -e", this will only be printed if everything went well

knora/dsplib/utils/start-app.sh

@@ -0,0 +1,29 @@
+#! /bin/bash
+# make this file executable with chmod u+x (filename).sh
+set -u  # exit if an uninitialised variable is used (https://www.davidpashley.com/articles/writing-robust-shell-scripts/)
+set -e  # exit if any statement returns a non-true return value (https://www.davidpashley.com/articles/writing-robust-shell-scripts/)
+
+[[ "$(npm -g outdated)" =~ .*@angular/cli.* ]] || printf "\e[31mWARNING: Your Angular seems to be outdated. Please update it with \"npm update -g @angular/cli\"\e[0m\n"
+
+logfile="../dsp-app-startup.log"
+
+cd ~
+mkdir -p .dsp-tools
+cd .dsp-tools
+if [[ ! -d dsp-app ]]; then
+    echo "git clone https://github.com/dasch-swiss/dsp-app.git" 2>&1 | tee -a "$logfile"
+    git clone https://github.com/dasch-swiss/dsp-app.git >>"$logfile" 2>&1
+fi
+cd dsp-app
+rm -f "$logfile"
+
+if echo -e "GET http://google.com HTTP/1.0\n\n" | nc google.com 80 -w 10 > /dev/null 2>&1; then
+    # only pull if there is an internet connection
+    echo "git pull ..." 2>&1 | tee -a "$logfile"
+    git checkout HEAD package-lock.json >>"$logfile" 2>&1
+    git pull >>"$logfile" 2>&1
+fi
+echo "npm i --legacy-peer-deps ..." 2>&1 | tee -a "$logfile"
+npm i --legacy-peer-deps >>"$logfile" 2>&1
+echo "ng s ..." 2>&1 | tee -a "$logfile"
+npm run ng s 2>&1 | tee -a "$logfile"

knora/dsplib/utils/stop-api.sh

@@ -0,0 +1,19 @@
+#! /bin/bash
+# make this file executable with chmod u+x (filename).sh
+set -u  # exit if an uninitialised variable is used (https://www.davidpashley.com/articles/writing-robust-shell-scripts/)
+set -e  # exit if any statement returns a non-true return value (https://www.davidpashley.com/articles/writing-robust-shell-scripts/)
+
+[[ $(docker stats --no-stream 2>/dev/null ) == "" ]] && printf "\e[31mERROR: Docker is not running, so there is no DSP-API to shut down.\e[0m\n" && return
+
+logfile="../dsp-api-stackdown.log"
+
+cd ~
+[[ ! -d .dsp-tools/dsp-api ]] && return
+cd .dsp-tools/dsp-api
+rm -f "$logfile"
+echo "make stack-down-delete-volumes ..." 2>&1 | tee -a "$logfile"
+make stack-down-delete-volumes >>"$logfile" 2>&1
+echo "make clean-sipi-tmp ..." 2>&1 | tee -a "$logfile"
+make clean-sipi-tmp >>"$logfile" 2>&1
+echo "make clean-sipi-projects ..." 2>&1 | tee -a "$logfile"
+make clean-sipi-projects >>"$logfile" 2>&1