wis2box-1.0b7-dev

This is in the WIP-release for the upcoming wis2box-1.0b7 release, as reference by the 'latest' documentation.

1.0b6

This is the sixth 1.0 beta release of wis2box. This release includes a number of new features, enhancements and bugfixes. This release implements the required updates to the WIS2 Topic Hierarchy regarding dataset topics. Users are strongly encouraged to update to this latest release for WIS2 compliance.

Enhancements/updates:

• Implementation of the WIS2 Topic Hierarchy update to merge country and centre-id (see the section below)
• UI: add option/toggle to display station points as clusters #588
• duplicate-data-check: wis2box will not publish the same data twice and instead print an error #600
• updated-data-check: wis2box will detect updates and add a link to indicate data is updated #565
• WNM: provide data inline in notification messages when data size less than 4096 bytes
• Storage guidelines are now 24GB
• Station list: barometer height is no longer calculated if not defined (left null/blank)
• updating of BUFR tools to their latest versions
• collect NGINX logs in loki

Bug fixes:

• BUFR handling:
  • "Passed array is too small" fix (#559)
  • fix TSI matching (#581)
• WCMP2
  • fix update date calculation
  • fix contact information
  • fix metadata message publication
• WTH
  • remove pywcmp in lieu of pywis-topics for topic handling

Choosing a valid centre-id for the pre-operational phase

The WIS2 Topic Hierarchy merges country and centre-id in a new centre-id that uniquely defines a WIS2-node.

If you plan to use wis2box-1.0b6 to run a WIS2-node in the pre-operational phase, please ensure to use your new centre-id starting with your TLD during the configuration process.

If you are unsure how to define the centre-id for your WIS2-node, please contact WMO.

New installation

Please follow the instructions provided at https://docs.wis2box.wis.wmo.int/en/latest/user/getting-started.html

Migrating existing installations

The following steps are required when upgrading from wis2box 1.0b5 to 1.0b6:

# stop current wis2box
cd ~/wis2box-1.0b5 && python3 wis2box-ctl.py stop

# download new release in home-directory
cd ~/ && wget https://github.com/wmo-im/wis2box/releases/download/1.0b6/wis2box-setup-1.0b6.zip

# unzip and go to new directory
unzip wis2box-setup-1.0b6.zip && cd wis2box-1.0b6

IMPORTANT: before starting the new wis2box-release your configuration files need to be updated to use the new WTH. You can use the migration-script to update your configuration files

# run migration script to update configuration
python3 migration/update_configuration.py

The migration-script will ask you to answer a series of questions to replace country-code and old centre-id with your new centre-id, as per the following example:

Enter your 3-letter country-code: ita
3-letter country-code: ita ? (y/n) y
Enter your old centre-id: maaike-test
Old centre-id: maaike-test ? (y/n) y
Enter your new centre-id: it-maaike-test
New centre-id: it-maaike-test ? (y/n) y
Updated file 'metadata-synop.yml'
Updated file 'data-mappings.yml'

Check and verify that your data-mappings.yml file contains the new topic hierarchy. An example can be found below:

data:
    it-maaike-test.data.core.weather.surface-based-observations.synop:
        plugins:
            txt:
                - plugin: wis2box.data.synop2bufr.ObservationDataSYNOP2BUFR
                  notify: true

Continue with the update:

# start wis2box
python3 wis2box-ctl.py start  # wait until the new images are downloaded

# login to wis2box and execute commands
python3 wis2box-ctl.py login

# create new data collection, replace with your filename
wis2box data add-collection /data/wis2box/metadata/discovery/metadata-synop.yml

# publish new discovery metadata, replace with your filename
wis2box metadata discovery publish /data/wis2box/metadata/discovery/metadata-synop.yml

Verify the wis2box UI to ensure that there are 2 entries per dataset (one for old and one for new topic hierarchy):

Continue with the update:

# reindex your data collection items, replace with your old and new discovery metadata ID
wis2box data reindex-collection-items urn:x-wmo:md:ita:maaike-test:surface-based-observations.synop urn:x-wmo:md:it-maaike-test:surface-based-observations.synop

# unpublish your old dataset using the old discovery metadata ID
wis2box metadata discovery unpublish urn:x-wmo:md:ita:maaike-test:surface-based-observations.synop

# associate stations to the new topic hierarchy
wis2box metadata station add-topic origin/a/wis2/it-maaike-test/data/core/weather/surface-based-observations/synop

# exit the wis2box-management container
exit

Finally you can clean the unused Docker images to preserve diskspace:

docker image prune -a

(Or docker image prune without -a depending on your docker version)

As a final step, you will need to update any data ingestion processes to copy to new corresponding folder in the wis2box-incoming bucket

(Old)

(New)

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0b6

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

1.0b5

This is the fifth 1.0 beta release of wis2box. This release includes a number of new features, enhancements and bugfixes.

Updates

wis2box-webapp (new container) provides an additional web application adding the following functionality:

• new station editor user interface with support to query OSCAR/Surface
• data entry via web-based form for SYNOP ASCII and CSV
• monitoring dashboard to via data notifications and download/inspect BUFR produced by wis2box

Updates to support wis2box configuration:

• new configuration creation script to allow the user to a create initial wis2box.env, data-mappings.yml and template discovery metadata files by answering a series of question on the command line
• environment files default.env and dev.env have been replaced by wis2box.env

Enhancements:

• add barometer height to station metadata management
• enable adding more user to default mosquitto broker
• fix to generate valid WNM (base64 integrity)
• fixes to generate valid WCMP2 (needs flushing of ES discovery-metadata)
• support for Docker Compose v2 (Note is it required or supported?)
• CSV publication (csv2bufr):
  • Improved logging and return of warnings and errors to the result object
  • BUFR templates now managed though the csv2bufr.templates module (note this will be moved to a stand alone module in a future release)
  • addition of minimal template for data from AWS weather stations

Bug fixes:

• fix dataset deletion workflow
• do not remove discovery metadata when running data cleaning
• BUFR publication: BUFR headers are now correctly set in extracted subsets. Previously the headers were not copied, leading to incorrect replication factors and data corruption
• GeoJSON API publication: code table entry description now correctly extracted and set in GeoJSON output. Previously only the first word of the description was extracted

important

• use the station editor in wis2box-webapp and ensure your stations are associated to a topic to ensure you can see the data in the "MAP" and "EXPLORE" option of wis2box-ui
• the cronjob running wis2box metadata station publish-collection in the wis2box-management every 10 minutes is now disabled

Migrating existing installations

The following steps are required when upgrading from wis2box 1.0b4 to 1.0b5:

# stop current wis2box
cd ~/wis2box-1.0b4 && python3 wis2box-ctl.py stop
# prune unused docker images
docker image prune -a
# delete api-config volume
docker volume rm wis2box_project_api-config
# download new release in home-directory
cd && wget https://github.com/wmo-im/wis2box/releases/download/1.0b5/wis2box-setup-1.0b5.zip
# unzip and go to new directory
unzip wis2box-setup-1.0b5.zip && cd wis2box-1.0b5
# run migration script to create wis2box.env
python3 migration/migration_create_wis2box_env.py
# check content of wis2box.env correctly copied your local updates in dev:env
cat wis2box.env
# run migration script to add barometer_height column to station_list.csv:
python3 migration/migration_update_station_list.py
# start wis2box
python3 wis2box-ctl.py start
# login to wis2box and execute commands
python3 wis2box-ctl.py login
# (inside wis2box-management container)
wis2box data add-collection /data/wis2box/surface-weather-observations.yml
wis2box metadata station publish-collection
# new images are downloaded (this may take awhile)

For installations older than 1.0b4, it is recommended to start from a clean/fresh installation.

To start a fresh installation, please clean your docker system after stopping your wis2box:

python3 wis2box-ctl.py stop
docker image prune -a
docker volume prune -a

...then delete the directory containing your current release and download the latest release and following the setup instructions.

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0b5

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

1.0b4

This is the fourth beta release of wis2box. This release includes a number of bugfixes.

Updates

• discovery metadata publication: fix message publication for discovery metadata (#454)
• metrics: do not display failure descriptions: metrics (#455)
• update to latest stable versions of csv2bufr, synop2bufr and pymetdecoder (#457)
• safeguard station metadata geometry z (#458)

Migrating existing installations

There are no migrations required when upgrading from 1.0b3.

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0b4

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

1.0b3

This is the third beta release of wis2box. This release includes a number of enhancements and bugfixes.

Updates

• new experimental metadata UI for station (#376) and discovery metadata (#154) management. This functionality is added to the release to facilitate initial stakeholder feedback, and is disabled by default. Future releases will provide the UI by default with detailed documentation on use
• wis2box-api enhancements
  • add metadata validation workflow
  • upgrade to Elasticsearch 8
• validation checks on topic matching for discovery metadata and API managements workflows (#431
• addition of data subscriber for integration of automated weather station (AWS) data (#430)
• improved user documentation for managing environment variables and configuration settings

Migrating existing installations

The upgrade to Elasticsearch 8 will update to your API data. If you are using wis2box 1.0b2, then you should be able to upgrade without issue. For versions previous to 1.0b2, a two-step upgrade is required as follows once 1.0b3 is installed:

python3 wis2box-ctl.py stop
# download wis2box 1.0b3
vi docker-compose.yml  # change elasticsearch version from 8.6.2 to 7.17.0 to ensure a clean upgrade from 7.16.2
python3 wis2box-ctl.py start
python3 wis2box-ctl.py stop
vi docker-compose.yml  # change elasticsearch version from 7.17.0 to 8.6.2
docker volume rm wis2box_project_api-config
python3 wis2box-ctl.py start

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0b3

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

1.0b2

This is the second beta release of wis2box. This release includes a number of enhancements and bugfixes.

Updates

• force error handling in bufr2geojson (#417)
• add optional FTP forwarder service (#297)
• add improved monitoring features (#329)
• run station publishing on startup and every 10 minutes (#413)
• fix date handling for BUFR pipeline (#412)
• allow csv2bufr templates to be local or external (#405)
• add standard data pipeline reporting (#329)
• update to ecCodes 2.28.0 (#381)
• add SSL deployment option (#140)
• update WIS2 notification message generation (#391)
• align data mappings workflow to stations (#388)
• set broker default message queue to 1000 (#377)
• add public user to broker for WIS2 compliance (#386)
• add WIS2 topic hierarchy validation (#228)

Migrating existing installations

Given the changes in various configurations, it is recommended to install 1.0b2 from scratch:

• delete wis2box Docker volumes
• update discovery metadata YAML configuration examples
• update station_list.csv from examples provided
• repeat runtime configuration

Documentation

Documentation for this release is available at https://docs.wis2box.wis.wmo.int/en/1.0b2

Feedback

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

1.0b1

This is the first beta release wis2box. This release includes a number of updates and bugfixes, including:

• new support for SYNOP to BUFR processing (#304)
• local station metadata management (#324)
• numerous documentation enhancements, including a user guide and reference documentation
• ensure API collections names are lower case
• streamlining of csv2bufr templates
• new WMO theme for documentation (#74)
• refactored and streamlined repository (#213)

User updates/changes (for existing installations):

• station metadata is managed in $WIS2BOX_DATADIR/metadata/station/station_list.csv and is managed by the user
• station metadata in the API needs to be flushed manually given model changes. From the wis2box-management container (i.e. python3 wis2box-ctl.py login):
  • curl -XDELETE http://elasticsearch:9200/stations
  • wis2box-ctl.py metadata station publish-collection
• existing data-mappings.yml files need to be updated to point the csv2bufr data pipeline to use template synop_bufr.json (replacing synop-bufr.json)

Documentation for 1.0.beta1 is available at https://docs.wis2box.wis.wmo.int/en/1.0b1

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

0.5.1

This is the 0.5.1 working release of wis2box. This is a bugfix released; updates include:

• fix BUFR multiple message handling (#331)
• remove Elasticsearch security warnings (#317)
• fix topics for message publishing (#338)
• improve data failure handling / message

Documentation for 0.5.1 is available at https://docs.wis2box.wis.wmo.int/en/0.5.1

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

0.5.0

This is the 0.5.0 working release of wis2box. Updated capabilities include:

• numerous performance updates (workflow, API, UI)
• updates to latest WIS2 notification message format
• discovery metadata publishing via broker
• addition of subscribe and download workflow (via pywis-pubsub)
• alignment of 3-letter country codes as per WIS2 topic hierarchy
• closing of non-proxied ports (API, Elasticsearch)
• addition of cadvisor
• addition of websockets support for broker
• movement of wis2box-auth to dedicated repository
• addition of custom data collection schemas in API
• support permalinks/bookmarks in UI
• addition of CSV export to UI
• fix to cronjobs for cleaning/archive of data
• numerous updates to user documentation

Documentation for 0.5.0 is available at https://docs.wis2box.wis.wmo.int/en/0.5.0

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int

0.4.0

This is the 0.4.0 working release of wis2box. Updated capabilities include:

• numerous UI/UX updates in support of user workflow
• numerous updates to user documentation
• numerous updates to WIS2 topic hierarchy workflow
• AsyncAPI support added to API
• numerous BUFR pipeline updates (WSI/TSI handling, workflow, publishing, updated BUFR tables)
• removal of WebDAV support (given default support for S3)
• configurable basemaps

All contributions are always welcome. Feedback is welcome via GitHub issues or discussions.

The wis2box team

https://docs.wis2box.wis.wmo.int