Prepare repository for Tenzir v4.0.0-rc2 (#3293)

This commit is was partially created with `/scripts/prepare-release`.

Here is a high-level summary of the changes:

* Updated /version.json to 4.0.0-rc2 and the expected rev-count of the
merge commit to 17173.
* Generated a new entry in the docs version selector list.
* Removed the docs for the previous release candidate.
* Moved all changelog entries from `/changelog/next` to
`/changelog/v4.0.0-rc2`.
* Moved all changelog entries from `/changelog/v4.0.0-rc1` to
`/changelog/v4.0.0-rc2`.
* Updated the python bindings version in `/python/pyproject.toml` to
4.0.0-rc2.

In addition to that, an wrong assertion that of a condition that could
actually be violated during regular operation was removed.

libvast/builtins/operators/serve.cpp

@@ -455,7 +455,9 @@ struct serve_manager_state {
           return;
         }
         const auto delivered = found->try_deliver_results(true);
-        VAST_ASSERT(delivered);
+        if (not delivered) {
+          VAST_DEBUG("failed to deliver results after timeout expired");
+        }
       });
     return found->get_rp;
   }

python/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "pyvast"
-version = "4.0.0-rc1"
+version = "4.0.0-rc2"
 description = "A security telemetry engine for detection and response"
 authors = ["Tenzir <engineering@tenzir.com>"]
 maintainers = ["Tenzir <engineering@tenzir.com>"]

version.json

@@ -4,14 +4,14 @@
     "annotated git tag without the leading 'v'.",
     "This value gets updated automatically by `scripts/prepare-release`."
   ],
-  "tenzir-version-fallback": "4.0.0-rc1",
+  "tenzir-version-fallback": "4.0.0-rc2",
   "tenzir-version-rev-count_COMMENT": [
     "This value stores the ancestor count of the tagged commit, calculated",
     "with `git rev-list --count HEAD`, then incremented by 1. This operates",
     "under the assumption that the release-preparing PR contains exactly one",
     "commit and is rebased before merging."
   ],
-  "tenzir-version-rev-count": 17059,
+  "tenzir-version-rev-count": 17174,
   "tenzir-partition-version_COMMENT": [
     "The partition version. This number must be bumped alongside the release",
     "version for releases that contain major format changes to the on-disk",

web/versioned_docs/version-Tenzir v4.0-rc2/connectors/kafka.md

@@ -0,0 +1,124 @@
+# kafka
+
+Loads bytes from and saves bytes to Kafka.
+
+## Synopsis
+
+Loader:
+
+```
+kafka [-t <topic>] [-c|--count <n>] [-e|--exit] [-o|--offset <offset>]
+      [-X|--set <key=value>,...]
+```
+
+Saver:
+
+```
+kafka [-t <topic>] [-k|--key <key>] [-T|--timestamp <time>]
+      [-X|--set <key=value>]
+
+```
+
+## Description
+
+The `kafka` loaders reads bytes from a Kafka topic. The `kafka` saver writes
+bytes to a Kafka topic.
+
+The implementation uses the official [librdkafka][librdkafka] from Confluent and
+supports all [configuration options][librdkafka-options]. You can specify them
+via `-X <key=value>,...`. We recommend putting your Kafka options into the
+dedicated `kafka.yaml` [plugin config file](../command-line.md#load-plugins).
+This way you can configure your all your environment-specific options once,
+independent of the per-connector invocations.
+
+[librdkafka]: https://github.com/confluentinc/librdkafka
+[librdkafka-options]: https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md
+
+The connector injects the following default librdkafka configuration values in
+case no configuration file is present, or when the configuration does not
+include them:
+
+- `bootstrap.servers`: `localhost`
+- `client.id`: `tenzir`
+- `group.id`: `tenzir`
+
+The default format for the `kafka` connector is [`json`](../formats/json.md).
+
+### `-t|--topic <topic>` (Loader, Saver)
+
+The Kafka topic use.
+
+Defaults to `tenzir`.
+
+### `-c|--count <n>` (Loader)
+
+Exit successfully after having consumed `n` messages.
+
+### `-e|--exit` (Loader)
+
+Exit successfully after having received the last message.
+
+Without this option, the loader waits for new messages after having consumed the
+last one.
+
+### `-o|--offset <offset>` (Loader)
+
+The offset to start consuming from. Possible values are:
+
+- `beginning`: first offset
+- `end`: last offset
+- `stored`: stored offset
+- `<value>`: absolute offset
+- `-<value>`: relative offset from end
+
+<!--
+- `s@<value>`: timestamp in ms to start at
+- `e@<value>`: timestamp in ms to stop at (not included)
+-->
+
+### `-X|--set <key=value>` (Loader, Saver)
+
+A comma-separated list of key-value configuration options for
+[librdkafka][librdkafka], e.g., `-X
+auto.offset.reset=earliest,enable.partition.eof=true`.
+
+The `kafka` operator passes the key-value pairs directly to
+[librdkafka][librdkafka]. Consult the list of available [configuration
+options][librdkafka-options] to configure Kafka according to your needs.
+
+We recommand factoring these options into the plugin-specific `kafka.yaml` so
+that they are indpendent of the `kafka` connector arguments.
+
+### `-k|--key <key>` (Saver)
+
+Sets a fixed key for all messages.
+
+### `-T|--timestamp <time>` (Saver)
+
+Sets a fixed timestamp for all messages.
+
+## Examples
+
+Read 100 JSON messages from the topic `tenzir`:
+
+```
+from kafka -c 100 read json
+```
+
+Read Zeek Streaming JSON logs from topic `zeek` starting at the beginning:
+
+```
+from kafka -t zeek -o beginning read zeek-json
+```
+
+Write the Tenzir version to topic `tenzir` with timestamp from the past:
+
+```
+version | to kafka -T 1984-01-01
+```
+
+Follow a CSV file and publish it to topic `data`:
+
+```
+from file -f /tmp/data.csv read csv | to kafka -t data
+```

web/versioned_docs/version-Tenzir v4.0-rc1/developer-guides/build-from-source.md → web/versioned_docs/version-Tenzir v4.0-rc2/developer-guides/build-from-source.md

@@ -44,6 +44,7 @@ Every [release](https://github.com/tenzir/tenzir/releases) of Tenzir includes an
 |✓|[robin-map](https://github.com/Tessil/robin-map)|>= 0.6.3|Fast hash map and hash set using robin hood hashing. (Bundled as subtree.)|
 |✓|[fast_float](https://github.com/FastFloat/fast_float)|>= 3.2.0|Required for parsing floating point numbers. (Bundled as submodule.)|
 ||[libpcap](https://www.tcpdump.org)||Required for building the `pcap` plugin.|
+||[librdkafka](https://github.com/confluentinc/librdkafka)||Required for building the `kafka` plugin.|
 ||[http-parser](https://github.com/nodejs/http-parser)||Required for building the `web` plugin.|
 ||[poetry](https://python-poetry.org)||Required for building the Python bindings.|
 ||[Doxygen](http://www.doxygen.org)||Required to build documentation for libtenzir.|

web/versioned_docs/version-Tenzir v4.0-rc2/operators/transformations/flatten.md

@@ -0,0 +1,67 @@
+# flatten
+
+Flattens nested data structures.
+
+## Synopsis
+
+```
+flatten [<separator>]
+```
+
+## Description
+
+The `flatten` operator removes any nested lists or records by merging lists and
+joining nested records with a separator. Flattening removes null values.
+
+:::info
+Unlike for most data models, flattening is an (almost) free operation in VAST's
+data model.
+:::
+
+### `<separator>`
+
+The separator string to join nested records with.
+
+Defaults to `.`.
+
+## Examples
+
+Consider the following data:
+
+```json
+{
+  "src_ip": "147.32.84.165",
+  "src_port": 1141,
+  "dest_ip": "147.32.80.9",
+  "dest_port": 53,
+  "event_type": "dns",
+  "dns": {
+    "type": "query",
+    "id": 553,
+    "rrname": "irc.freenode.net",
+    "rrtype": "A",
+    "tx_id": 0,
+    "grouped": {
+      "A": ["tenzir.com", null]
+    }
+  }
+}
+```
+
+The `flatten` operator removes any nesting from the data:
+
+```json
+{
+  "src_ip": "147.32.84.165",
+  "src_port": 1141,
+  "dest_ip": "147.32.80.9",
+  "dest_port": 53,
+  "event_type": "dns",
+  "dns.type": "query",
+  "dns.id": 553,
+  "dns.rrname": "irc.freenode.net",
+  "dns.rrtype": "A",
+  "dns.tx_id": 0,
+  "dns.grouped.A": ["tenzir.com"]
+}
+```

web/versions.json

@@ -1,5 +1,5 @@
 [
-  "Tenzir v4.0-rc1",
+  "Tenzir v4.0-rc2",
   "VAST v3.1",
   "VAST v3.0"
 ]