23.0.0

Features

Better Recovery from Rabbit unavailability

• Services and Ferryman have both been updated to provide more robust failure handling when RabbitMQ connections are lost. Pods will now intentionally fail when the channel is closed, canceled, or throws an error. The pods will then attempt to restart until they successfully connect to Rabbit again. (#1480, #1481)

Mongoose Update

• Rolled out Mongoose updates to support Mongo v6.x (#1470)

Secret Service

• Added FORM_AUTH request type (#1482)

Enhancements

Secret Service

• There is a preprocessor for getting data from Azure tokens (#1464)

Component Repository

• The Component Repository now allows the max request body size to be configured from the ENV (#1465)
• Allow longer component descriptions (#1467)

IAM

• Fetch users by username or id (#1478)

Fixes

• Update for Tenant Ownership of components (#1472)

• Fix for kubernetes permission errors (#1475 , #1477)

• Other minor bugfixes and docs improvements were also included. For a full changelog, see here

22.2.0

BREAKING CHANGES

Webhooks non-HMAC Authorization (#1425)

• Due to inconsistencies in Constant definitions between flow-repository and iam-utils, authentication of users via token or basic authorisation was failing.
• A fix has been provided for the current specific use case reported in #1422. A future update will help to unify constant definition across services.
• In order to complete this fix, one of the following actions must be taken:
  • Stop and start all Webhooks flows
  • Update all flows in the "webhooks.flows" collection in MongoDB to save the owner IDs as strings instead of BSON Object IDs

Improve Flow State Tracking and Fix Snapshots Service(#1461)

• Due to some updates made in early 2022, the Snapshots Service was broken, resulting either in Snapshots being deleted after a flow which should be persisted, or execution-scoped Snapshots not being cleaned up at the end of an execution. In order to preserve some of the changes made for Logic Flows, the following changes were made:
  • Adjust Flow State tracking in the Component Orchestrator to support multi-emit Component Functions
  • Add "end" event emission to the Ferryman to track completion of functions
  • Revert "snapshot" events to exclude storing the flowExecId, so that persistent Snapshots will not be deleted

In order to make use of these fixes, Components need to be re-built using the new Ferryman version, 3.0.0.

Features

New Prototypical Features

Analytics Service

• The Analytics Service serves as the central repository for all data analytics-related data and functions inside the OIH. It offers both a database for long-term storage of relevant data as well as an API for data retrieval and validation.

Enhancements

Flow Repository

• The Flow Repository API now supports bulk operations to save on API calls. Creating, updating, and deleting several flows at once can now be done in a single API call by passing array of objects. Doing so requires a separate user permission. (#1460 )

Component Repository

• The Component Repository API now offers an endpoint to restart all currently active global components, to allow for easier troubleshooting in case of loss of queue connection or similar issues. (#1459 )

Secret Service preprocessor for OIDC (#1415 #1416)

• Added secret service auth client preprocessor for mapping an Open ID Connect (OIDC) id_token into the secret value created from the auth client.

Fixes

• Added flowExecId, flowId, stepId, userId, tenant, and component function to component log messages for traceability. (#1419 )

• Other minor bugfixes and docs improvements were also included. For a full changelog, see here

22.1.0

Features

Production State

Component Generator (#15)

• Added the old recursion function to be able to get the schema object from every response
• Changed the way action and triggers are set up. The setup starts with the data object coming in the trigger and action which contains the operationId for the current function
• With that string we can query the component.json and get the callParams object which is now resides there
• We have the pathName, method and the requestContentType strings which are necessary for every swagger client call
• From now on only one trigger and one action is needed for every component as the necessary data for every case is stored in the component.json
• Extracted the Id Linking functionality from the action and trigger and added the function in the helpers.js file
• Cleaned up the generate.js file to match the functionality we wanted for the above
• Removed some more old comments and code from the action/trigger templates
• Created test files for the action and the trigger
• Added skip snapshot functionality in the triggers
• Thinned out the code from the trigger file to the helpers.js

Virtual/Versioned Components (#1385)

• Added a new model layer to the component-repository which provides for Virtual Components to be created, which allow for maintaining separate metadata and supports versioned components

Enhancements

Minor fixes and updates to Visual Flow Builder in Web-Ui (#1398)

Minor fixes and updates to Governance Service (#1405 & #1397)

Enhanced Governance Dashboard Filtering capabilities (#1402)

Fixes

Mongoose update to 6.x.x

• MongoDB decided to automatically update lower Atlas Clusters to MongoDB 5.0.
• As MongoDB is the default DB for the framework, we provided the necessary updates via these PRs: #1394 #1395 #1396

Fixed customPermissions in iam utils (#1404)

• fixed customPermissions by adding the customPermissions parameter to the createToken method
• readRaw permission does not have to be assigned to the user that starts the flow anymore

Updated Flow Repository API Docs (#1406)

• added credentials_id to Flow Repository API Docs to match current implementation
• credentials_id can be used to reference secrets in a flow

Fixed a bug where ferryman would not AutoSave Updates (#1411)

• the Ferryman function onUpdateSnapshot() attempted to access this.snapshotRoutingKey which caused an error
• Removed the fourth param, this.snapshotRoutingKey, from the call to self.amqpConnection.sendSnapshot() in Ferryman's onUpdateSnapshot() function

Other

Minikube Dev updates (#1359)

• Flow repository minikube: reduced the long delay for readiness and liveness probe

Updated Kubernetes Config Files (#1413)

• Updated Kubernetes config to include IAM Base URL in the Orchestrator for working with tokens. Also, unified and corrected the Secret Service URL naming.

22.0.0

Features

Production State

OIH Edge Extension L-Variant based on Apache Kafka (separate repo)

• Instead of one static Edge Flow it is now possible to start or stop source, analytics or sink components as Docker containers at any time and connect them to each other in a mesh structure
• a Pay-per-x component to demonstrate pay-per-use models used already by several machine vendors
• IIoT-Simulator, MQTT or OPCUA connectors can now be used simultaneously for multi-connectivity
• high performance data processing with Kafka streams in new analytics components
• buffering of messages inside Kafka topics and management via Kafka-UI
• new MongoDB sink component

REST API access to components (#1381 )

• components can now also be accessed via REST API
• this is a first step towards synchronous integrations - feedback and contributions welcome!

Development State

Visual Flow Builder (#1388)

• the WebUi now provides an interface to create flows through a visual builder instead of having to create it as 'json'

Governance Dashboard (#1372 )

• Added a beta suite of dashboard functionalities, intended to give users an overview of their data
• Users can now retrieve visualizations of their data distribution, as well as a detailed status and history of a single data object
• Governance can also check for potential governance risks in a user's current flow configurations

Support for Object Deletion

• best practices for the deletion of objects through a flow
• expanding on the Data-Hub's ID-Linking functionality
• created unified format for messages in flows that involve a delete action (#1244)

Enhancements

Improvements on Implementation of Tenancy

• started fixing the often inconsistent implementation of tenancy across the framework
• Related PR's (#1348, #1349, #1350, #1351)

remove hard-coded log level (#1345 )

• The hard-coded component LOG_LEVEL environment variable has been removed from the component orchestrator
• LOG_LEVEL for components will now be determined by container environment variable, or default to Ferryman info if not specified

Fixes

Ferryman Transformation Bugs (#1361 )

• fixed a bug where an incorrect payload was send to ferryman's transform function

npm audit / updates (#1371)

• updated most outdated npm versions
• significantly reduced the number of "critical" or "high" npm audit issues

Other

Add standalone proxy script to Dev Tools (#1357)

21.3.0

Features

Development Releases

Webhooks Authorization (#1317)

• Incoming Webhooks can now be authenticated in one of three ways:
  • HMAC Verification
  • Passing a Bearer Token
  • Using Basic Auth with the credentials of a User in IAM
• Information about how to implement Authorization can be found in the Webhooks README file

Policy Parsing Framework (#1268)

• The governance service now has a prototypical framework for automatically parsing policies attached to flow objects, which can be automatically called by the ferryman.
• This can be used to block the transmission of flow objects if they fail to adhere to certain user-defined conditions
• Further information can be found in the documentation: OIH Policy Introduction

Data Enrichment Functions (#1277)

• The data hub now offers a prototypical suite of functions to enrich stored data objects
• This includes scoring, filtering, formatting, and deduplicating data objects based on user-defined conditions

Enhancements

Move from yarn to npm7 (#1318)

• We decided to move to npm7 and updated all files accordingly
• Developers should now use Node 16 + npm> 7 as their environment
• The lockfile has been rewritten to enable reproducible builds
• All Docker images have been optimized accordingly and now use Node 16
• Many libraries have been updated

Service API Responses (#1247)

• The 404 response is now only used when a resource does not exist to align the system behaviour with REST specifications.

Auth Client Update (#1334)

• All fields of an Auth Client can be updated

Fixes

Rebound Queues (#1315)

• Starting with ferryman 2.0.0 rebound queues can be utilized
• The component orchestrator needs to have at least version 1.4.2 and, if existing, all deployments for components must be recreated (restart)

Get all Secrete for an Auth Client (#1335)

• Corresponding DAO has been fixed and now allows specific secrets to be obtained

Other

Removal of minikube folder

In order to simplify the maintenance of deployment and development tools, the /minikube folder has been removed. For local installation with minikube, please use the /dev-tools/minikube folder. Future improvements for local installation will be made there.

21.2.0

Features

Production State

Data Hub

• added POST /data/import API endpoint: Import multiple data records
• extended GET /data API endpoint: Filter and retrieve multiple data records
• see https://data-hub.openintegrationhub.com/api-docs

Tagging Service

• New optional service added as a separate repository, that allows group tags with all different types of objects (templates, components...).

Connector Repository (#1269)

• Expanded model allow storing information about component functions
• Added endpoint for automatically fetching function data from public component repositories

OIH Edge Extension

• Added Python OIH Edge Extension for connecting IIoT data sources to the OIH

Development State

Logic gateway

• A new global component "Logic Gateway" is currently in the alpha stage of the development and will be shipped shortly
• Logic Gateways can be placed in any flow to control the execution of nodes. A Logic Gateway requires a predefined rule with instructions, which can also reference any data emitted by any previous node in the given flow.
• Possible scenarios for Logic Gateways are: conditional execution, loops, branch switching, flow branch joins

IDS PoC

• Added IDS-SQL Adapter and OIH-IDS Gateway components together with the IDS-Wrapper for interaction with the IDS (International Data Spaces) ecosystem
• Added IDS-Connectivity chapter under "For Developers" section of OIH documentation describing the involved components, the basic concept of the PoC and giving an example scenario for testing

Enhancements

• Improvements to Data Hub
  • Allow Admins to vew more data objects
  • Better data handling in Ferryman
• Secret Service
  • Refactor to remove library (#1259)
  • Add lookups for secrets from auth clients (#1263)
• Flows can now be filtered by template (#1265)
• Ferryman can auto-save snapshots

Fixes

• Corrected minor bugs in Data Hub record passing
• Improve yarn audit severity by updating multiple packages (#1282)

Other

• Updated docker-compose implementation to include new services
• Updated minikube deployment for ARM Macs
• Add flow and build step for dev-tools/test-component in dev-tools/docker-compose (#1282)

21.1.0

Features

Session Authorization

The Secret Service now supports fetching authorisation tokens from external services that use Session Authorization types. This results in less need for showing passwords in plaintext. (#1242, #1243)

Enhancements

• Improved retries and handling of errors within message queues for ferryman (#1220)
• Completed updating all services to the standard request/response message format (#1120)

Fixes

• Removed references in developer documents to a non-controlled domain. (#1225)
• Allow for logging of more queue errors in multiple services (see #1233)

Other

Developer Tooling Improvements

• Added basic support for Apple Silicon devices on Minikube (without source code deploy) (#1238)

• Provide persistent volumes for InfluxDB and Grafana on Google Cloud (#1240)

• Various Dependency Updates

21.0.1

Fixes

• Component Orchestrator now uses correct backend-commons-lib version -> fixes local component handling
• Webhooks will now pass data into flow accurately (#1201)

21.0.0

⭐ See below for information on using release images in deployment

Features

ID Linking

We made multiple changes to enable linking of IDs across connected system to ensure that data records can be updated. Read more.

Orchestrator Replication

Added basic support for replicating the Component Orchestrator inside of the Kubernetes cluster. This included adding a new "preparing" state for Flows while initializing. Future releases will focus on improving the performance and scalability of replicated Orchestrators.

Enhancements

• Made preparations for upcoming new services (e.g. Raw Data Storage)
• Changed the message format used in multiple services as part of ongoing work described in #1078
• Updated event bus lib for better handling of service and component messages
• Updated service/lib test setup and packages
• Upon login, a User's record will no longer be marked as updated. (safeguard.lastLogin will be updated, but updatedAt will not be modifed)
• Connector transforms are now supported via Flow Configuration, and can be sent via the nodeSettings object

Fixes

• Removed race condition while starting and stopping local components in a replicated orchestrator setup
• Fixed problem in components where they could mistakenly think their publishChannel is full and wait perpetually for a "drain" event (#1139)
• Fixed some memory leaks caused by older versions of mongoose (#1186)
• Corrected problems in Scheduler Deployments

Other

Documentation Restructuring

Our documentation page has been restructured to make it easier to find information. We also worked on updating the content and will invest more time on documentation in the future. Please give us feedback and contribute if you find errors or have requests for more documentation on specific topics.

Release Image Versioning

Upon release, the appropriate Docker Hub versions of all Framework Services will be tagged with the release version. Deployments can then use the release version to ensure interoperability of service capabilities.

20.2.1

New Features

Transformations inside of Connectors

It is no longer necessary to include a separate Transformer Component inside of Flows for standard transformations. The Ferryman library now includes a transform()function to perform this functionality directly inside of a Connector (example).

Flow Templates

A new service has been added to the framework called Template Repository. This service contains "recipes" or templates from which Flows can be generated. This allows for the sharing of Flows without leakage of Secrets.

Enhancements

Improved Orchestrator Performance

Caching has been added to the Component Orchestrator to help it process flows more quickly.

Deployment Configuration Improvements

Deployment scripts for the Reference Implementation on Kubernetes have been updated. Configuration and Secrets have been pulled out of the Deployment files and moved to their own Config Maps and Kubernetes Secret files. This allows users to more easily implement their own deployments based on the Reference Implementation.

Failure Handling for Flows

In the case of failures when changing Flow states, especially when starting a Flow, inconsistencies could arise between the reported state in the Flow Repository and the Component Orchestrator. Failing Flows will now report their status and be transitioned back to Inactive. This also provides fixes for Flows which were "stuck" in a transitional state.

Fixes

• Data Hub: The Data Hub service has been updated to operate with the new framework features introduced in 20.2.0.
• Local Components: Fixes an issue where Local Components were not able to start.

Other

"help wanted" Label for Community Contributions

We want to make it easier for the community to contribute and know where help is needed. As a first step, we will start labeling issues with help wanted.

Developer Tooling

• Addition of Load Testing scripts in the dev-tools/
• Improved Installation Documentation

Out-factoring Logging Service

The Logging Service was removed from the core framework repository and moved into its own repository. This Service is tightly couple to execution on Google Cloud Platform and aggregated logs from its Cloud Logging service. The Framework is intended to remain agnostic from deployment platform, therefore this service was removed from the monorepo.

---

Libraries

Ferryman: v1.1.5
Component Orchestrator: v1.1.5
Attachment Storage Service: v0.0.4
Event Bus: v1.2.0
Secret Service: v1.7.1
Webhooks: v1.2.0
Scheduler: v1.2.0
IAM Utils: v1.6.0