-
Notifications
You must be signed in to change notification settings - Fork 69
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Controller][all] Do not truncate Kafka topic immediately for fatal d… #937
Merged
lluwm
merged 11 commits into
linkedin:main
from
lluwm:do_not_truncate_topic_for_FatalDataValidationException
May 6, 2024
Merged
[Controller][all] Do not truncate Kafka topic immediately for fatal d… #937
lluwm
merged 11 commits into
linkedin:main
from
lluwm:do_not_truncate_topic_for_FatalDataValidationException
May 6, 2024
Commits on Apr 9, 2024
-
[Controller][all] Do not truncate Kafka topic immediately for fatal d…
…ata validation error before EOP When the ingestion of a new store version is failed, today, we truncate the Kafka topic of the store version by updating its retention time to a small value (15 seconds), specified by DEPRECATED_TOPIC_RETENTION_MS. This is fine for regular failures but for fatal data validation errors, where it indicates critical issues happened during the batch push period (before EOP), truncating the topic too early can prevent us from finding the root cause, simply because the kafka data is gone. This change adjusts the retention time (to 5 days) when fatal data validation errors is identified, so that we can have enough time to investigate.
Configuration menu - View commit details
-
Copy full SHA for 131c0a5 - Browse repository at this point
Copy the full SHA 131c0a5View commit details
Commits on Apr 22, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 409ca11 - Browse repository at this point
Copy the full SHA 409ca11View commit details -
Configuration menu - View commit details
-
Copy full SHA for d5c586d - Browse repository at this point
Copy the full SHA d5c586dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 578b128 - Browse repository at this point
Copy the full SHA 578b128View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5016953 - Browse repository at this point
Copy the full SHA 5016953View commit details
Commits on Apr 24, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 3c03164 - Browse repository at this point
Copy the full SHA 3c03164View commit details
Commits on Apr 29, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 0c59977 - Browse repository at this point
Copy the full SHA 0c59977View commit details -
Configuration menu - View commit details
-
Copy full SHA for f42a5a4 - Browse repository at this point
Copy the full SHA f42a5a4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9dc550e - Browse repository at this point
Copy the full SHA 9dc550eView commit details
Commits on May 3, 2024
-
Configuration menu - View commit details
-
Copy full SHA for b319ed8 - Browse repository at this point
Copy the full SHA b319ed8View commit details
Commits on May 6, 2024
-
Addressed the comments on simplifying the impl of the new truncateKaf…
…kaTopic interface.
Configuration menu - View commit details
-
Copy full SHA for 0de06f0 - Browse repository at this point
Copy the full SHA 0de06f0View commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.