Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Controller][all] Do not truncate Kafka topic immediately for fatal d… #937

Commits on Apr 9, 2024

  1. [Controller][all] Do not truncate Kafka topic immediately for fatal d…

    …ata validation error before EOP
    
    When the ingestion of a new store version is failed, today, we truncate the Kafka topic of the
    store version by updating its retention time to a small value (15 seconds), specified by
    DEPRECATED_TOPIC_RETENTION_MS.
    
    This is fine for regular failures but for fatal data validation errors, where it indicates critical issues
    happened during the batch push period (before EOP), truncating the topic too early can prevent us from
    finding the root cause, simply because the kafka data is gone.
    
    This change adjusts the retention time (to 5 days) when fatal data validation errors is identified, so
    that we can have enough time to investigate.
    lluwm committed Apr 9, 2024
    Configuration menu
    Copy the full SHA
    131c0a5 View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. Configuration menu
    Copy the full SHA
    409ca11 View commit details
    Browse the repository at this point in the history
  2. Fix merge conflict

    lluwm committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    d5c586d View commit details
    Browse the repository at this point in the history
  3. fix merge conflict

    lluwm committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    578b128 View commit details
    Browse the repository at this point in the history
  4. fix merge issue

    lluwm committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    5016953 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2024

  1. Configuration menu
    Copy the full SHA
    3c03164 View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2024

  1. Configuration menu
    Copy the full SHA
    0c59977 View commit details
    Browse the repository at this point in the history
  2. Fix unit test

    lluwm committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    f42a5a4 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    9dc550e View commit details
    Browse the repository at this point in the history

Commits on May 3, 2024

  1. Configuration menu
    Copy the full SHA
    b319ed8 View commit details
    Browse the repository at this point in the history

Commits on May 6, 2024

  1. Configuration menu
    Copy the full SHA
    0de06f0 View commit details
    Browse the repository at this point in the history