[CORE-2736] Adds output batch compression for Data Transforms #18514
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
oleiman
force-pushed
the
xform/core-2736/compressed-batches
branch
2 times, most recently
from
4adf296
to
58df36c
Compare
|
/ci-repeat 1 |
|
/ci-repeat 1 |
|
new failures in https://buildkite.com/redpanda/redpanda/builds/49248#018f82f5-d2fc-4bd5-b204-a1774afd79bf: new failures in https://buildkite.com/redpanda/redpanda/builds/49248#018f82f5-d2ff-46d2-8840-d9e063a709e2: |
oleiman
commented
May 17, 2024
| "zstd" | ||
| ], | ||
| "required": false, | ||
| "description": "Output batch compression mode (TODO(oren): might want to limit which?)" |
There was a problem hiding this comment.
I wonder whether we want/need to make all of these available.
oleiman
force-pushed
the
xform/core-2736/compressed-batches
branch
2 times, most recently
from
d9f6691
to
f13207f
Compare
oleiman
requested review from
a team,
twmb,
r-vasquez,
gene-redpanda and
Deflaimun
as code owners
oleiman
requested review from
a team and
michael-redpanda
and removed request for
a team
rockwotj
reviewed
May 19, 2024
| @@ -149,6 +153,7 @@ The --var flag can be repeated to specify multiple variables like so: | |||
| cmd.Flags().StringSliceVarP(&fc.outputTopics, "output-topic", "o", []string{}, "The output topic to write the transform results to (repeatable)") | |||
| cmd.Flags().StringVar(&fc.functionName, "name", "", "The name of the transform") | |||
| cmd.Flags().Var(&fc.env, "var", "Specify an environment variable in the form of KEY=VALUE") | |||
| cmd.Flags().StringVar(&fc.compression, "compression", "", "Output batch compression type") | |||
There was a problem hiding this comment.
Shouldn't this be an explicit "none"?
| NAME INPUT TOPIC OUTPUT TOPIC RUNNING LAG | ||
| foo2bar foo bar 2 / 3 7 | ||
| scrubber pii cleaned, munged 0 / 1 99 | ||
| NAME INPUT TOPIC OUTPUT TOPIC RUNNING LAG COMPRESSION |
There was a problem hiding this comment.
Is this useful? We don't print env vars and this feels like that configuration. Maybe in the detail view? Doesn't feel like there is a user story where someone wants this
There was a problem hiding this comment.
good point, yeah I suppose not.
| @@ -81,3 +81,40 @@ Subsequent 'rpk transform list' operations will show transform processors as | |||
| } | |||
| return cmd | |||
| } | |||
|
|
|||
| func newSetCompressionCommand(fs afero.Fs, p *config.Params) *cobra.Command { | |||
There was a problem hiding this comment.
Probably confusing for users that this is not sticky across deploys
Feediver1
reviewed
May 20, 2024
Comment on lines
+102
to
+103
| at a cost. So, while it does occur asynchronously with respect to transform | ||
| execution, compression may introduce latency on the output topic. |
There was a problem hiding this comment.
Suggested change
| at a cost. So, while it does occur asynchronously with respect to transform | |
| execution, compression may introduce latency on the output topic. | |
| at a cost. So, while it occurs asynchronously with respect to transform | |
| execution, compression may introduce latency on the output topic. |
boost::lexical_cast<compression>("bogus) should raise bad_lexical cast.
Previous behavior: runtime_error("Fell off the end of a string-switch")
Failure to match the source string to one of the compression lexical cases
should set the fail bit on the istream instead of allowing the runtime_error
to escape.
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Also adds bool transform_metadata_patch::empty() Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Respecting transform_metadata::compression_mode Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
- tranform_from_json - deploy - set_compression Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
oleiman
force-pushed
the
xform/core-2736/compressed-batches
branch
from
f13207f
to
1e48f71
Compare
|
force push to sync w/ dev after a couple weeks of inactivity |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the ability to configure output batch compression on WASM transforms, either at deploy time or post facto, via metadata patch request. Includes rpk experience for same
TODO:
Backports Required
Release Notes
Improvements