v1.29-2 - New modules ModBus and Lua

Changes:

• In route definition, the I operator is replaced with D. Existing programs
  must be updated to prevent warning from being printed.
• Changed signatures for processor functions by adding a new last argument
  'method'. Existing scripts MAY BREAK and should be updated. Refer to
  the rrr.conf.5 manual page for correct function signatures.
• Set process and thread names on OS-es where this is possible. This can
  assist when debugging for instance CPU usage.

Features:

• Match generated messages in httpclient with their corresponding message triggering the query by using a report tag
• Route messages leaving an instance to specific reader instances by topic or array tags
• New parameter in httpclient to control timeout of low priority messages specifically
• New array value "hdlc" for parsing HDLC messages
• Ability in file module to write to files and devices among other new configuration parameters
• New conversion type "hchar2str" in mangler module to convert individual bytes to strings
• New OS object in javascript (operating-system) and ability to obtain hostname
• New ModBus TCP module
• New Lua module
• New method on Message object in JS push_tag_object which serializes objects into JSON. Receiving JS instances will automatically de-serialize the JSON added to messages this way.
• New parameter methods for cmodules (python3, js, perl5, lua and cmodule) allowing messages content to be identified before being processed. The processor function receives a method name.
• Support PATCH requests in httpclient and httpserver
• New function console.assert in JS
• Hook messages between modules and dump them using rrr_stats
• Hook events and show them using rrr_stats
• Set mode on file which are written by the file module
• Incrementer can use prefixes ORed with incremented ID
• Dump memory profiling using functionallity provided by jemalloc

Improvements:

• Timeout changes in httpclient and rotate deferred to msgdb PUT messages to ensure all are attempted and not the same messages time out repeatedly.
• In JS, refuse bare name module imports (require prefix like ./) and process relative paths correctly with respect to directory of importing script.
• In httpclient, print stats about dropped messages in periodic function instead of once for every message.
• Add warning when buffer entry count exceeds a certain configurable limit
• Add statistics data for message broker costumer buffers
• Print warning and disallow requests in HTTP-client if the endpoints has invalid characters.

Fixes:

• Raw data messages in JS module now working properly
• Crash in JS module upon GC due to null pointer access
• Httpclient sometimes spinning on CPU indefinitely after handshaking if there is no data to send on a socket.
• Readers were blocked in message broker when writer used do_again during slot write (fast writes from dummy module with buffer=no).
• Deadlock in rrr_stats due to printing messages during signal handling
• Soft error was ignored in socket client collection also when socket surely was closed, like when 0 bytes were read. This caused the socket not to be closed with following CPU spin due do constant POLLIN event. The soft error is now forced to propagate if bytes read is zero so that only data errors are ignored (which is also intention of higher levels).
• Connections were closed in IP module when byte by byte sync was used and the alignment used up all remaining buffered bytes.
• Non-working default socket name in msgdb module causing crash
• Incorrect tick timer in msgdb causing excessive CPU usage
• Memory leak of HV in Perl subroutine call helper function
• Stale connections in http-client when server accepts connections but does not respond
• Discrepancies in httpserver with respect to timeout and response when waiting for responses from sender modules.
• Incorrect header name for access-control-allow-methods in httpserver

Under the hood:

• Use events in file module
• Common send loop for file and ip modules

v1.27-1 - New JavaScript module

• Features

  • New module to run JavaScript programs
  • New parameter in ip module to generate empty messages when new TCP connections are accepted.
  • New parameter in httpclient to disable HTTP/2 upgrade headers

• Improvements:

  • Clarified inconsitency of manual with respect to configuration paramter mqtt_publish_rrr_message
  • Never send raw array data from mqttclient
  • Remove stale configuration cmodule parameters sleep_time_ms and nothing_happend_limit which had no effect when used.

• Fixes:

  • Handling of websocket topic filters in httpserver
  • Crash in httpclient when content-encoding: gzip was set from server with zero length body
  • Build with new major release of OpenSSL
  • Build with Python 3.11

v1.25-1 - New features and fixes

• Changes:

  • Argument '-f|--force', force read of unsupported array message version, is removed from rrr_msg utility.

• Features:

  • Add configurable delay between forks during startup
  • New rrr_sub binary which can read RRR messages in the payload of MQTT messages.
  • Support gzip compression in HTTP server and client
  • Default test page in HTTP server
  • Option in HTTP server to produce 404 responses to /favicon.ico requets

• Improvements:

  • Show IP address of remote host in error message when array data parsing fails in IP module
  • Extract boundary directive from content-type fields of type multipart/form-data in httpserver
  • New configuration parameter in httpclient to set content-type boundary directive in incoming messages.
  • If conversion from str to PyUnicode fails in python3 for array values, fallback to PyByteArray with a warning
  • Set CODE_FILE and CODE_LINE fields when logging to systemd-journald
  • More aggressive cleanup of larger allocations in mmap channel to reduce persistent memory usage in some situations.

• Fixes:

  • Misc. features in python RRR array handling API not working correctly, in particular handling of multibyte character sets
  • cmodule test failed if it found the dummy module on the filesystem with non-compatible version.

• Under the hood:

  • Cleanups in net transport framework

• Other:

  • Remove obsolete configuration parameter ip_persistent_connections

v1.23-2 - New features and fixes

• Features:

  • Option in httpclient to silence error messages for PUT queries for a specified amount of time
  • New topic_filter_invert option
  • Support message expiry interval in v5 in MQTT broker
  • Set retain flag and expiry interval in MQTT client
  • Set will messages in MQTT client
  • Control subscriptions in MQTT client by sending in command messages
  • Control the tidy interval of the cacher module per configuration parameter

• Improvements:

  • Files in message database stored in a tree structure based on their hash filename for performance.
  • Asynchrounous operation in message DB for better performance with very large DBs
  • Tidy and revive intervals in cacher is calculated from the last completion of each of these processes.

• Fixes:

  • Prevent HTTP client from getting killed of if there are a lot of connection errors
  • Deadlock in mmap channel due to incorrect use of lock() instead of trylock()
  • Lockup in cmodules in high traffic situations
  • Sometimes temporary lockup in HTTP-server if there is not immediate downstream response to an h2c upgrade.
  • Poor hard timeout management in MQTT broker causing low-traffic clients to be kicked
  • Fix buffer overflow in httpclient when wide characters were given to an urlencoder function.
  • Fix random crashes in message DB when zero length files were found.

• Under the hood:

  • Cleanups in MQTT framework

v1.21-1 - New features and fixes

• Changes:

  • Automatic restart of RRR deamon upon failure
  • Use JEmalloc instead of internal allocator

• Features:

  • Option in cacher to revive messages older than a specified age
  • Control maximum response size in httpclient
  • Control maximum request size in httpserver
  • Control content-type for raw requests in httpclient
  • Control which readers that will receive the different kinds of messages from cacher

• Improvements:

  • Performance for larger messages in misc. modules

• Fixes:

  • Busy-waiting on thread state change causing long startup times
  • Cacher getting killed off on large writes to message DB
  • Large messages sometimes failing to be sent due to incorrect return values in some send functions
  • Send allow-origin header in OPTIONS responses to satisfy preflight checks
  • Validate message header checksum in rrr_msg tool
  • Manual page for rrr_msg tool
  • Crash in rrr_msg as it tried to read from directories
  • Problems with compiler -pie flags on some platforms
  • Sed-replacement causing configure script not to work on non-GNU systems
  • Slowness when multiple mmap_channel instances were destroying block locks simultaneously

• Under the hood:

  • Common Message DB functions for httpclient and cacher
  • Use robust locking in process shared locks (mmap channel and shm frameworks)

v1.19-1 - Minor new features + Fixes

• Features:

  • HTTP server can extract values from JSON
  • Option in HTTP server to prevent any body from being parsed
  • Added memory cache in cacher module
  • Message taint option in httpclient to prevent messages from
    backfiring.

• Fixes:

  • Crash in MQTT client upon reconnect attempts due to incorrect
    cast.
  • Fix non-working will and retain messages in MQTT broker.
  • Incorrect default port numbers in MQTT broker
  • MQTT ping messages were not being sent
  • Sending in httpserver stopped for longer data if nothing else
    were added to be sent or read.
  • Messages were sorted in reverse order in IP module
  • Ensure correct sorting in IP module by using and index
    instead of timestamp.
  • Missing default port number in mqtt client
  • Incorrect byte counting in misc. send functions causing
    problems with longer messages.
  • Broken SQL generation in mysql module due to error in
    string builder.
  • Allow spaces after certain HTTP/1 header fields
  • Missing IP information in messages arriving on TCP in IP
    module.
  • Minor errors in debug messages.

• Under the hood:

  • Incref/decref abstraction added to FIFO buffer to reduce
    complexity in mqtt.
  • Misc. cleanups

v1.19-0 - Fixes and new Cacher module

• Features:

  • New cacher module (basic proxy server-like module)

• Fixes:

  • Prevent connection timeouts in message DB
  • Buld problems on Ubuntu Hirsute, Alpine and Void
  • Incorrect thread startup sequence sometimes causing race
    condition with log lock freezing threads (only with debug enabled).
  • Stream reset was faulty in UDP-stream protocol
  • Continuation after UDP-stream ID exhaustion
  • Broken async responses in HTTP2 incl. crashes
  • Incorrect handling of stream failures from remote in HTTP2
  • Upgrade to HTTP2 returned Bad Request
  • Blocking when httpserver with HTTP2 while waiting for response
    from other module
  • Data access after free when receiving http1 response (potential crash)
  • Processor modules could stop receiving messages after some time,
    fixed by converting them to use events.

• Improvements:

  • MMAP and Memory Allocator performance using shared memory
  • Adjustments in cmodules to prevent blocking under high message load
  • Removal of not needed mutexes
  • httpserver generates str type fields not blob
  • Better queue management in httpclient under high load
  • Improved performance in UDP-stream
  • New simplified buffer without locking where this is not needed

• Under the hood:

  • Improved misc. size and data length checks on 32 and 64 bits
  • Add memory allocator test
  • Module separations, standardizations, simplifications, code removal
  • Other cleanups

• Fixes on development versions only:

  • Fix rare race condition in allocator causing crash
  • Fix misc. bad behaviour causing mmap exhaustion despite them
    having more room.
  • Other fixes

v1.17-1 - Fixes and performance improvements

• HTTP fixes
  • Support HTTP/1.0
  • Respect Connection: close/keep-alive header
  • Improve redirection functionallity
  • Improve HTTP/2 performance by having a stream limit
  • Improve performance by changing misc. maintenance intervals
• Memory allocator
  • Implement custom memory allocator for messages
  • Long lived programs with message bursts no longer suffer from fragmentation and increased memory usage
• Create standalone build directory for cmodules
• Fix TLS connection handshake CPU spin
• Fix problems with TCP connections not being closed
• Fix issue with incorrect event amount counts
• Fix problems in test suite causing tests to fail
• Fix broken build when JSON-C was disabled
• Fix. misc build issues with optional libraries

v1.17-0 - Performance improvements with event driven modules

• Event driven modules for reduced latency and CPU usage
• New options to control persistent connections in IP module
• HTTP server is no longer threaded, all requests are handled by the same thread
• Improved performance in statistics engine

v1.16-0 - Support for HTTP/2 +++

• Buffers

  • All modules support duplication by setting "duplicate=yes"
  • New backstop mechanism preventing instances from getting messages generated by themselves, configurable.
  • Parameter to disable output buffer in instances to reduce latency

• HTTP

  • Support for HTTP/2, client and server will use this transparently whenever available. Implies extensive rewrite of HTTP.
  • Support for decoding and encoding JSON data to/from RRR arrays
  • Intermediate on-disk storage for PUT messages in the new Message DB, can store messages while a server is down
  • Raw modes removed in server and client, replaced with structured data which is compatible with both HTTP/1.1 and HTTP/2
  • HTTP raw forwarding not possible anymore, example removed
  • Parameter 'http_method' split into 'http_method' and 'http_format'

• New modules

  • New exploder module to split an RRR array message into multiple messages
  • New mangler module to change types of values in RRR array messages
  • New incrementer module to keep track of ID numbers
  • New msgdb (Message DB) module used by incrementer and httpclient

• File module

  • Configuration parameters for serial ports

• Cmodules

  • Configurable number of worker forks

• Arrays, types and message

  • New data type "vain" (a NULL type)
  • Strings may be of zero length
  • New tool rrr_msg to dump RRR messages stored on disk
  • Fix bugs when parsing istr and ustr input data

• Threads and forks

  • Changes in thread startup and shutdown
  • Simplification of signals + fix minor bugs with syscalls during forking causing program sometimes not to start

• MQTT

  • Fix to avoid crashes whener client performed retransmit

• Python

  • Support handling of message IP metadata