chore: bump E2E tests for Kafka 3.7 #544
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jeqo
force-pushed
the
jeqo/bump-e2e-kafka-3.7
branch
3 times, most recently
from
6a70a49
to
c38861c
Compare
jeqo
changed the title
jeqo
commented
May 15, 2024
Comment on lines
-530
to
+588
| final ConsumerRecord<byte[], byte[]> record = consumer.poll(Duration.ofSeconds(1)).records(TP_0_0).get(0); | ||
| assertThat(record.offset()).isEqualTo(newStartOffset); | ||
|
|
||
| consumer.seekToBeginning(List.of(TP_0_0)); | ||
|
|
||
| // with current retention logic, active segment is rotated, and no data is available locally. | ||
| final List<ConsumerRecord<byte[], byte[]>> records = consumer.poll(Duration.ofSeconds(1)).records(TP_0_0); | ||
| assertThat(records).isEmpty(); |
There was a problem hiding this comment.
Main change: validating that the active segment is not rotated is not correct anymore.
jeqo
force-pushed
the
jeqo/bump-e2e-kafka-3.7
branch
2 times, most recently
from
d7548eb
to
730adc4
Compare
jeqo
commented
May 15, 2024
| @@ -49,7 +49,7 @@ public class RemoteLogMetadataTracker { | |||
| private static final String REMOTE_LOG_METADATA_TOPIC = "__remote_log_metadata"; | |||
|
|
|||
| private final KafkaConsumer<byte[], RemoteLogMetadata> consumer; | |||
| private List<TopicPartition> partitions; | |||
| private List<TopicPartition> metadataPartitions; | |||
There was a problem hiding this comment.
Separating "metadata" from "user" partitions.
jeqo
commented
May 15, 2024
| @@ -94,21 +109,23 @@ abstract class SingleBrokerTest { | |||
| static final TopicPartition TP_0_0 = new TopicPartition(TOPIC_0, 0); | |||
| static final TopicPartition TP_0_1 = new TopicPartition(TOPIC_0, 1); | |||
| static final TopicPartition TP_1_0 = new TopicPartition(TOPIC_1, 0); | |||
| static final List<TopicPartition> TOPIC_PARTITIONS = List.of(TP_0_0, TP_0_1, TP_1_0); | |||
| static final List<TopicPartition> USER_TOPIC_PARTITIONS = List.of(TP_0_0, TP_0_1, TP_1_0); | |||
There was a problem hiding this comment.
Same here, separating "metadata" from "user" partitions
| * from remote storage</li> | ||
| * <li>{@link SingleBrokerTest#remoteCleanupDueToRetention()}: wait for retention by size to remove data | ||
| * from remote storage</li> | ||
| * <li>{@link SingleBrokerTest#topicDelete()}: (unsupported) delete topic and validate data is removed |
There was a problem hiding this comment.
unsupported == not implemented yet?
There was a problem hiding this comment.
Correct, it should be implemented eventually: https://issues.apache.org/jira/browse/KAFKA-15166
e2e/src/test/java/io/aiven/kafka/tieredstorage/e2e/SingleBrokerTest.java
Outdated
Show resolved
Hide resolved
jeqo
force-pushed
the
jeqo/bump-e2e-kafka-3.7
branch
2 times, most recently
from
b1e47e4
to
c66e109
Compare
Let next phase to test topic T0 and avoid additional dependency between phases.
3.7 has changed how local retention impacts the rotation of active segment.
jeqo
force-pushed
the
jeqo/bump-e2e-kafka-3.7
branch
from
c66e109
to
bb6d8b3
Compare
giuseppelillo
approved these changes
May 21, 2024
giuseppelillo
approved these changes
May 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Kafka 3.7 have changed how local retention affects active segment: from now on, if local retention time is passed, then a new (empty) active segment is rollout.
This changes how the end-to-end tests validated deletion by retention.
Apart from this change, the PR includes some e2e test improvements: adding docs to make the sequence of steps on the end-to-end test easier to follow; some reorganization and renaming with the same purpose. See the commits for more detail.