-
Notifications
You must be signed in to change notification settings - Fork 145
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prototype of stopping after N rows loaded from taps #8364
base: main
Are you sure you want to change the base?
Conversation
✅ Deploy Preview for meltano canceled.
|
for more information, see https://pre-commit.ci
@edgarrmondragon and team let us know if we can do anything to improve or test any specific use cases. |
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #8364 +/- ##
==========================================
- Coverage 91.50% 91.48% -0.03%
==========================================
Files 244 245 +1
Lines 19125 19168 +43
Branches 2139 2143 +4
==========================================
+ Hits 17501 17536 +35
- Misses 1344 1352 +8
Partials 280 280 ☔ View full report in Codecov by Sentry. |
@edgarrmondragon - I know it has been a while - but this is still of interest for us. What would be the next step? Resolving the failing checks? |
@dcowden ^ fyi |
Yeah, although the current implementation aborts the sync after any N lines have been emitted by the tap. Other than actual records, that count would include schema and state messages too. I'm fine with that, but just wanna make sure that is a good solution for other users' and yall's needs. That said, some requirements to get this across are probable:
The current pre-commit and coverage failures are ok at the moment, since I expect the refactoring and testing required by the above two points would address them. |
This is a sketch for how i introduced ability to limit rows from taps for the purposes of testing.
Warning: This code has not been executed. I have working code based on an older revision that i tested. So, i know the approach works with code very similar to this. I cleaned up the code to fit into latest meltano main, so that it would be easier to see the essence of the changes.
If this overall path is of interest, i could do more work-- i really would like thoughts about the overall approach before I do that. I dont know the meltano code well at all, so this might be a totally wrong way to do it.
Other notes that might be helpful: