Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better logging on empty topics #76

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

bcarter97
Copy link
Member

Description

Currently we log Not loading data from empty topic if the lowest offset isn't <= the highest offset. This doesn't necessarily mean it's empty, but could instead mean messages have been deleted and the highest offset is > 0.

Additionally added some debug logging to show what offsets are deemed empty or not.

Example DEBUG logs from test

08:29:55.590 [io-compute-2] DEBUG u.s.f.kafka.topicloader.TopicLoader - Assigning partitions: load-state-topic-1-0,load-state-topic-1-1,load-state-topic-2-0,load-state-topic-2-1
08:29:55.642 [io-compute-4] DEBUG u.s.f.kafka.topicloader.TopicLoader - Seeking to offset 0 for partition load-state-topic-1-0
08:29:55.642 [io-compute-2] DEBUG u.s.f.kafka.topicloader.TopicLoader - Seeking to offset 0 for partition load-state-topic-1-1
08:29:55.642 [io-compute-7] DEBUG u.s.f.kafka.topicloader.TopicLoader - Seeking to offset 0 for partition load-state-topic-2-0
08:29:55.642 [io-compute-6] DEBUG u.s.f.kafka.topicloader.TopicLoader - Seeking to offset 0 for partition load-state-topic-2-1
08:29:55.643 [io-compute-6] DEBUG u.s.f.kafka.topicloader.TopicLoader - Non-empty offsets: (load-state-topic-1-0,LogOffset(highest=10, lowest=0)), (load-state-topic-1-1,LogOffset(highest=5, lowest=0))
08:29:55.643 [io-compute-6] DEBUG u.s.f.kafka.topicloader.TopicLoader - Empty offsets: (load-state-topic-2-0,LogOffset(highest=0, lowest=0)), (load-state-topic-2-1,LogOffset(highest=0, lowest=0))
08:29:55.643 [io-compute-6] INFO  u.s.f.kafka.topicloader.TopicLoader - Not loading data from load-state-topic-2-0 as lowest offset 0 <= highest offset 0
08:29:55.643 [io-compute-6] INFO  u.s.f.kafka.topicloader.TopicLoader - Not loading data from load-state-topic-2-1 as lowest offset 0 <= highest offset 0
08:29:56.876 [io-compute-4] INFO  u.s.f.kafka.topicloader.TopicLoader - Finished loading data from load-state-topic-1-1 at offset 4
08:29:57.981 [io-compute-9] INFO  u.s.f.kafka.topicloader.TopicLoader - Finished loading data from load-state-topic-1-0 at offset 9

Related Issues

  • Connects to #

Definition of Done:

The Below tasks should be completed before marking a PR as ready for review.

  • Added / Removed / Updated relevant tests
  • Scaladoc updated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant