Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DO NOT MERGE] Report per-type added/removed ordinal count in announcement metadata #550

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

Sunjeet
Copy link
Contributor

@Sunjeet Sunjeet commented Sep 29, 2021

Seeking feedback on this approach for reporting some measure of entropy from the producer side (applicable for Producer and IncrementalProducer alike) and without incurring much computation overhead (as opposed to adding a listener that iterates over data using DataAccessor to get record change counts).

This could be useful for-

  • powering alerts on too many changes in a rolling time window without having to spin up a consumer or custom producer side computation
  • reporting per-type change counts to power some history/diff views solely from metadata (without requiring a consumer for downloading data)

Some design considerations:

  • Ordinal added/removed counts are computationally cheap since they are available during delta computation but less useful as compared to record-level add/remove/modify counts which are readily available for IncrementalProducers and can be computed by producers/consumers for types with primary keys using DataAccessor.
  • Reporting per-type ordinals added and ordinals removed counts and a writestate-level total additions and total removal count (refer screenshot)
  • Reporting on the announcement metadata. This is convenient, since the counts are generated during delta generation and delta blob header has already been written at that point.
  • Only reported when delta is computed

Tests yet to be written.

Screen Shot 2021-09-29 at 8 28 59 AM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant