ref: Initial refactor to facilitate streaming writes to clickhouse #5683
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
untitaker
changed the title
Codecov ReportAll modified and coverable lines are covered by tests ✅
✅ All tests successful. No failed tests found Additional details and impacted files@@ Coverage Diff @@
## master #5683 +/- ##
==========================================
- Coverage 89.94% 89.92% -0.02%
==========================================
Files 895 898 +3
Lines 43591 43453 -138
Branches 299 299
==========================================
- Hits 39207 39077 -130
+ Misses 4342 4334 -8
Partials 42 42 ☔ View full report in Codecov by Sentry. |
Swatinem
approved these changes
Mar 25, 2024
lynnagara
approved these changes
Mar 25, 2024
lynnagara
reviewed
Mar 25, 2024
| @@ -141,90 +137,3 @@ impl ProcessingStrategy<BytesInsertBatch> for ClickhouseWriterStep { | |||
| self.inner.join(timeout) | |||
| } | |||
| } | |||
|
|
|||
| #[derive(Clone)] | |||
There was a problem hiding this comment.
Do we need a test that actually writes a batch to clickhouse?
There was a problem hiding this comment.
right now those tests get in the way IMO
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We want to eventually stop buffering writes to clickhouse, and instead hold one long-running connection for the lifetime of a batch, streaming rows slowly in. This is already how Python works.
This initial refactor creates a new Batch type that holds onto the clickhouse client internally.