Comparing Beam and Bytewax
Setup
python -m venv venv
source venv/bin/activate
pip install -r beam-requirements.txt
beam-wordcount.py just makes sure that everything is running. Then, we can create
a fake parquet file by running python generate-parquet.py. This file, df.parquet
contains a list of user_ids and their features.
beam-parquet-proto.py uses Beam to scan this file (in reality we should use Kafka
and stream) and tries to convert each row to a protobuf which, with a given
probability p (default 0), is a wrong schema (-c proto) or a wrong field (-c field).
Unfortunately, this conversion is not done directly (parquet to proto), but uses an
intermediate JSON conversion, which can be a problem for some (complex) data types.
Some dependencies are clashing with each other, so I just created a new virtual environment.
python -m venv venv
source venv/bin/activate
pip install -r bw-requirements.txt
bw-wordcount.py implements the same wordcount, but uses reduce_window instead. A
reduce method is also available, but I haven't found a way to tell it when it stop...
But in a streaming case we need a window anyway.
bw-parquet-proto.py is basically a 1:1 translation of beam-parquet-proto.py: we can see it's a lot more "manual"
than Beam, everything needs to be specified, and again we use reduce_window to count. This also means that it's probably much more flexible than Beam is, but I'm doing things that are way too simple to test this statement.
Finally, we run a prototype pipeline using Kafka.
- Start Kafka locally:
docker compose up - Start the producer:
python kafka-producer.py - Start the Bytewax pipeline:
python kafka-bw.py
Bonus: for Slack notifications, a WEBHOOK_URL environment variable (in .env) is required.
This can be augmented with stateful anomaly detection and Slack messaging like so: https://github.com/awmatheson/junk-drawer/blob/main/data-quality/dataflow.py