Merge branch 'master' of github.com:rtr-nettest/open-rmbt

rtr-nettest · Apr 19, 2023 · a9b68cc · a9b68cc
2 parents 38ff218 + 00c7b54
commit a9b68cc
Show file tree

Hide file tree

Showing 6 changed files with 120 additions and 77 deletions.
diff --git a/README.md b/README.md
@@ -6,24 +6,28 @@ Open-RMBT
 It consists of the following components:
 * Web site
 * JavaScript client
-* Android client
-* iOS client
+* Android client +
+* iOS client 
 * Measurement server
 * QoS measurement server (in this repository)
-* Control server 
-* Statistics server (in this repository)
+* Control server +
+* Statistics server +
 * Map server (in this repository)
 
++) These components are available in separate repositories. This repository still contains outdated and incompatible versions of these components. They shall not be used for production purposes. Plese contact us if you are uncertain about the compatiblity of repositories.
+
 *Open-RMBT* is released under the [Apache License, Version 2.0](LICENSE). It was developed
 by the [Austrian Regulatory Authority for Broadcasting and Telecommunications (RTR-GmbH)](https://www.rtr.at/).
 
-Related materials
------------------
+
+Related material
+----------------
 
 * [RMBT specification](https://www.netztest.at/doc/)
 * [RTR-NetTest/rmbt-server](https://github.com/rtr-nettest/rmbt-server) - Test Server for conducting measurements based on the RMBT protocol
 * [RTR-NetTest/rmbtws](https://github.com/rtr-nettest/rmbtws) - JavaScript client for conducting RMBT-based speed measurements
 * [RTR-NetTest/open-rmbt-control](https://github.com/rtr-nettest/open-rmbt-control) - Control server
+* [RTR-NetTest/open-rmbt-statistics](https://github.com/rtr-nettest/open-rmbt-statistics) - Statistics server
 * [RTR-NetTest/open-rmbt-ios](https://github.com/rtr-nettest/open-rmbt-ios) - iOS app
 * [RTR-NetTest/open-rmbt-android](https://github.com/rtr-nettest/open-rmbt-android) - Android app
 * [RTR-NetTest/rtr-nettest/open-rmbt-website](https://github.com/rtr-nettest/open-rmbt-website) - Web site
@@ -34,9 +38,9 @@ System requirements
 
 * 1-3 servers
 * Everything can be installed on a single server 
-* The test servers (RMBT and Websocket) should run on a physical machine
+* The test server (RMBT-Websocket) should run on a physical machine
 * Base system Debian 11 or newer (or similar) 
-* At least one static IPv4 address (IPv6 support recommended, more addresses allow to run more services on port 443)
+* At least a single static public IPv4 address (IPv6 support recommended, more addresses allow to run more services on port 443)
 
   *NOTE: other Linux distributions can also be used, but commands and package names may be different*
 
@@ -105,42 +109,17 @@ Installation
     * max_parallel_workers_per_gather
     * max_parallel_workers
 
-### Control-,  Map- and StatisticServer
+### MapServer
 
 1. Install:
-  * Apache Tomcat 8 or higher
+  * Apache Tomcat 9 or higher
   * nginx (optional, highly recommended)
-  * openjdk-8-jre (do not use a higher version)
+  * openjdk-11-jre (do not use a higher version)
   * libservlet3.1-java
-  * [Maxmind GeoLite2 database](https://dev.maxmind.com/geoip/geoip2/geolite2/)
-  * _Optional:_ For StatisticServer pdf export functionality
-    * [Prince](https://www.princexml.com/) or 
-    * [Weasyprint](https://weasyprint.org/). For weasyprint, install as user tomcat (not as root):
-        * python3
-        * python3-pip
-        * run `pip3 install weasyprint`
 
-2. Edit `/etc/tomcat8/context.xml` (substitute parts with `[]`), add to `<Context>`:
+2. Edit `/etc/tomcat9/context.xml` (substitute parts with `[]`), add to `<Context>`:
+
 
-   For control server:
-    ```xml
-    <Context>
-    <!-- [...] -->
-    <Resource 
-       name="jdbc/rmbt" 
-       auth="Container"
-       type="javax.sql.DataSource"
-       maxActive="200" maxIdle="10" maxWait="10000"
-       url="jdbc:postgresql://[db host]/rmbt"
-       driverClassName="org.postgresql.Driver"
-       username="rmbt_control" password="[db r/w pass]"
-       description="DB RW Connection" />
-    <Parameter name="RMBT_SECRETKEY" value="[rmbt qos secret key]" override="false" />
-    <!-- [...] -->
-    </Context>
-    ```
-    For statistic/map servers:
-
     ```xml
     <Context>
     <!-- [...] -->
@@ -157,24 +136,21 @@ Installation
     </Context>
 
     ```
-3. Build the servers
+3. Build the server:
 
-    The servers can be built with gradle:
+    The map server can be built with gradle:
     ```bash
-    ./gradlew :RMBTControlServer:war :RMBTMapServer:war :RMBTStatisticServer:war
+    ./gradlew :RMBTMapServer:war 
     ```
-    The war files are then located in `RMBT[Control|Map|Statistic]Server/build/lib`.
-
-4. Copy `RMBTControlServer.war`, `RMBTMapServer.war` and/or `RMBTStatisticServer.war` to `/var/lib/tomcat8/webapps/`
+    The war file is located in `RMBTMapServer/build/lib`.
 
-    In case the Java-Postgres connector is missing:
-    Add the package `libpostgresql-jdbc-java` and restart tomcat8.
-
-5. Run `service tomcat8 restart`
+4. Copy `RMBTMapServer.war` to `/var/lib/tomcat9/webapps/`
+
+5. Add the package `libpostgresql-jdbc-java` from [Postgresql JDBC](https://jdbc.postgresql.org/) and restart tomcat9.
 
 6. Optimize tomcat settings
 
-    Check the values in /etc/default/tomcat8
+    Check the values in /etc/default/tomcat9
     * JAVA_OPTS -Xmms MEM -Xmx MEM
 
 Get in Touch

diff --git a/RMBTMapServer/src/at/rtr/rmbt/mapServer/ShapeTiles.java b/RMBTMapServer/src/at/rtr/rmbt/mapServer/ShapeTiles.java
@@ -86,12 +86,12 @@ protected byte[] generateTile(final ShapeTileParameters params, final int tileSi
             //debugging hint: St_AsText allows human-readable representation of a geometry object
             final String sql = String.format(
                     "WITH box AS"
-                            //input from Browser is converted to 900913; has to be transformed to 31287 for use with bev data
+                            //input from Browser is converted to 3857; has to be transformed to 31287 for use with bev data
                     + " (SELECT ST_Transform(ST_SetSRID(ST_MakeBox2D(ST_Point(?,?),"
-                    + " ST_Point(?,?)), 900913), 31287) AS box)"
+                    + " ST_Point(?,?)), 3857), 31287) AS box)"
                     + " SELECT"
-                            //output has to be transformed to 900913 for Browsers
-                    + " ST_SnapToGrid(ST_Transform(ST_intersection(p.geom, box.box), 900913), ?,?,?,?) AS geom,"
+                            //output has to be transformed to EPSG:3857 for Browsers
+                    + " ST_SnapToGrid(ST_Transform(ST_intersection(p.geom, box.box), 3857), ?,?,?,?) AS geom,"
                     + " count(\"%1$s\") count,"
                     + " quantile(\"%1$s\",?) val" 
                     + " FROM box, bev_vgd p"

diff --git a/RMBTUtil/build.gradle b/RMBTUtil/build.gradle
@@ -7,7 +7,7 @@ repositories {
 
 
 dependencies {
-    testCompile group: 'junit', name: 'junit', version: '4.11'
+    testCompile group: 'junit', name: 'junit', version: '4.13.2'
     compile group: 'org.json', name: 'json', version: "$jsonVersion"
-    compile group: 'com.google.code.gson', name: 'gson', version: '2.8.0'
+    compile group: 'com.google.code.gson', name: 'gson', version: '2.10.1'
 }
diff --git a/common.gradle b/common.gradle
@@ -4,10 +4,10 @@
 ext {
     restletVersion = '2.4.3'
     servletVersion = '3.1.0'
-    jacksonVersion = '2.11.3'
-    catalinaVersion = '7.0.106'
-    jsonVersion = '20200518'
-    javaMelodyVersion = '1.86.0'
+    jacksonVersion = '2.14.2'
+    catalinaVersion = '7.0.109'
+    jsonVersion = '20220924'
+    javaMelodyVersion = '1.91.0'
     dbUtilsVersion = '1.7'
 }
 

diff --git a/...en_data/statistik_austria/get_atraster.sh → ...data/statistik_austria/get_atraster100.sh b/...en_data/statistik_austria/get_atraster.sh → ...data/statistik_austria/get_atraster100.sh
@@ -23,49 +23,46 @@ export LANG=C
 # Open data site:
 # https://www.data.gv.at/katalog/dataset/stat_regionalstatistische-rastereinheiten66c96
 # current release
-URL=http://data.statistik.gv.at/data/OGDEXT_RASTER_1_STATISTIK_AUSTRIA_L000100_LAEA.zip
-
+URL=https://data.statistik.gv.at/data/OGDEXT_RASTER_1_STATISTIK_AUSTRIA_L000100_LAEA.zip
 
 mkdir ~/open
 cd ~/open
+rm -rf STATISTIK_AUSTRIA_L000100_LAEA*
 
-rm STATISTIK_AUSTRIA_L000100_LAEA*
-
-wget $URL
-unzip *.zip
-rm *.zip
-
+wget $URL -O atraster100.zip
+unzip atraster100.zip && rm atraster100.zip
 
 # import as table atraster (takes some time)
 # -I create geo index
 # -s data is in "ETRS_1989_LAEA" - this is EPSG:3035 (ETRS89-extended / LAEA Europe) => code 3035
-# -d drop table if it exists
-shp2pgsql -I -s 3035 -d STATISTIK_AUSTRIA_L000100_LAEA.shp atraster | psql rmbt > /dev/null 
+# :3857 => projection to 3857 (web)
+# ((-d drop table ))
+shp2pgsql -I -s 3035:3857  STATISTIK_AUSTRIA_L000100_LAEA.shp atraster100 | psql rmbt > /dev/null 
 
 # typical record
-# INSERT INTO "atraster" ("id","name",geom) VALUES ('100mN27473E45458','CRS3035RES100mN2747300E4545800','01060000000100000001030000000100000005000000000000004257514100000000D2F5444100000000425751410000000004F64441000000005B5751410000000004F64441000000005B57514100000000D2F54441000000004257514100000000D2F54441');
+# INSERT INTO "atraster100" ("id","name",geom) VALUES ('100mN27473E45458','CRS3035RES100mN2747300E4545800','01060000000100000001030000000100000005000000000000004257514100000000D2F5444100000000425751410000000004F64441000000005B5751410000000004F64441000000005B57514100000000D2F54441000000004257514100000000D2F54441');
 # column name is redundant, can be dropped to save space
 
-
-
 # create index on id
 
 sql=$(cat <<EOF
 BEGIN;
-CREATE INDEX atraster_id_idx
+CREATE INDEX atraster100_id_idx 
   ON atraster(id);
+CREATE INDEX idx_the_geom_4326_atraster100 
+  ON public.atraster100 USING gist (public.st_transform(geom,4326));
 ANALYSE atraster;
-ALTER TABLE atraster OWNER TO rmbt;
-GRANT SELECT ON TABLE atraster TO rmbt_group_read_only;
+ALTER TABLE atraster100 OWNER TO rmbt;
+GRANT SELECT ON TABLE atraster100 TO rmbt_group_read_only;
 COMMIT;
-VACUUM atraster;
+SELECT pg_size_pretty( pg_total_relation_size('atraster') );
 EOF
 )
 echo -e $sql|psql rmbt
 
 
 # Example query to select a raster based on WGS84 coordinates
-# SELECT * from atraster where SELECT * from atraster where ST_intersects((ST_Transform(ST_SetSRID(ST_MakePoint(12.6939,47.074531),4326),3035)),geom);
+# SELECT * from atraster100 where ST_intersects((ST_Transform(ST_SetSRID(ST_MakePoint(12.6939,47.074531),4326),3857)),geom);
 
 
 echo done

diff --git a/install/postgresql/3rd_party_open_data/statistik_austria/get_atraster250.sh b/install/postgresql/3rd_party_open_data/statistik_austria/get_atraster250.sh
@@ -0,0 +1,70 @@
+#!/bin/bash
+#/*******************************************************************************
+# * Copyright 2019 Rundfunk und Telekom Regulierungs-GmbH (RTR-GmbH)
+# * 
+# * Licensed under the Apache License, Version 2.0 (the "License");
+# * you may not use this file except in compliance with the License.
+# * You may obtain a copy of the License at
+# * 
+# *   http://www.apache.org/licenses/LICENSE-2.0
+# * 
+# * Unless required by applicable law or agreed to in writing, software
+# * distributed under the License is distributed on an "AS IS" BASIS,
+# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# * See the License for the specific language governing permissions and
+# * limitations under the License.
+# ******************************************************************************/
+
+set -x
+export LANG=C
+
+# Import Austrian 250m x 250m raster
+
+# Open data site:
+# https://www.data.gv.at/katalog/dataset/stat_regionalstatistische-rastereinheiten66c96
+# current release
+URL=https://data.statistik.gv.at/data/OGDEXT_RASTER_1_STATISTIK_AUSTRIA_L000250_LAEA.zip
+
+mkdir ~/open
+cd ~/open
+rm -rf STATISTIK_AUSTRIA_L000250_LAEA*
+
+wget $URL -O atraster250.zip
+unzip atraster250.zip && rm atraster250.zip
+
+# import as table atraster (takes some time)
+# -I create geo index
+# -s data is in "ETRS_1989_LAEA" - this is EPSG:3035 (ETRS89-extended / LAEA Europe) => code 3035
+# :3857 => projection to 3857 (web)
+# ((-d drop table ))
+shp2pgsql -I -s 3035:3857  STATISTIK_AUSTRIA_L000250_LAEA.shp atraster250 | psql rmbt > /dev/null 
+
+# typical record
+# INSERT INTO "atraster250" ("id","name",geom) VALUES ('250mN27473E45458','CRS3035RES250mN2747300E4545800','01060000000250000001030000000250000005000000000000004257514250000000D2F5444250000000425751425000000004F64442500000005B5751425000000004F64442500000005B57514250000000D2F54442500000004257514250000000D2F54441');
+# column name is redundant, can be dropped to save space
+
+# create index on id
+
+sql=$(cat <<EOF
+BEGIN;
+CREATE INDEX atraster250_id_idx 
+  ON atraster(id);
+CREATE INDEX idx_the_geom_4326_atraster250 
+  ON public.atraster250 USING gist (public.st_transform(geom,4326));
+ANALYSE atraster;
+ALTER TABLE atraster250 OWNER TO rmbt;
+GRANT SELECT ON TABLE atraster250 TO rmbt_group_read_only;
+COMMIT;
+SELECT pg_size_pretty( pg_total_relation_size('atraster') );
+EOF
+)
+echo -e $sql|psql rmbt
+
+
+# Example query to select a raster based on WGS84 coordinates
+# SELECT * from atraster250 where ST_intersects((ST_Transform(ST_SetSRID(ST_MakePoint(12.6939,47.074531),4326),3857)),geom);
+
+
+echo done
+
+