kafka-config

Easy to use type-safe builders for kafka clients. Supports Java >= 8. kafka-config version aligns with kafka-client version.

Motivation

Configuring kafka consumers and producers is usually quite error-prone task because kafka client relies on java.util.Properties class. This introduces such issues as the lack of type-safety, possible mi-spelling of names, lack of auto-completion and easy to find javadoc. kafka-config tries to address these issues by providing simple builders to construct kafka configuration easier and safer. Also, listing and searching for all available configuration parameters is much easier because of auto-completion.

Supported clients:

• KafkaConsumer
• KafkaProducer
• KafkaStreams
• AdminClient

Examples

public static KafkaConsumer<String,String> createConsumer() {
  //provided configuration values are random, do not copy them
  Properties consumerProps = ConsumerConfigBuilder.create()
      .withBootstrapServers(BOOTSTRAP_SERVERS)
      .withGroupId("test-group")
      .withAllowAutoCreateTopics(true)
      .withAutoOffsetReset(OffsetResetStrategy.EARLIEST)
      .withCheckCRCs(true)
      .withEnableAutoCommit(false)
      .withDefaultApiTimeout(Duration.ofMinutes(1))
      .withAutoCommitInterval(Duration.ofSeconds(30))
      .withExcludeInternalTopics(false)
      .withFetchMaxBytes(1024 * 10)
      .withFetchMaxWait(Duration.ofSeconds(60))
      .withFetchMinBytes(1024)
      .withGroupInstanceId("groupId")
      .withHeartbeatInterval(Duration.ofSeconds(10))
      .withInterceptorClasses(TestConsumerInterceptor.class)
      .withIsolationLevel(IsolationLevel.READ_COMMITTED)
      .withKeyDeserializer(StringDeserializer.class)
      .withValueDeserializer(StringDeserializer.class)
      .withMaxPartitionFetchBytes(1024 * 100)
      .withMaxPollInterval(Duration.ofSeconds(30))
      .withMaxPollRecords(1000)
      .withPartitionAssignmentStrategy(RoundRobinAssignor.class)
      .withSessionTimeout(Duration.ofMinutes(5))
      .withClientDNSLookup(ClientDnsLookup.USE_ALL_DNS_IPS)
      .withClientId("clientId")
      .withClientRack("clientRack")
      .withConnectionsMaxIdle(Duration.ofSeconds(60))
      .withMetadataMaxAge(Duration.ofMinutes(5))
      .withMetricReporterClasses(JmxReporter.class)
      .withMetricsNumSamples(1000)
      .withMetricsRecordingLevel(Sensor.RecordingLevel.INFO)
      .withMetricsSampleWindow(Duration.ofSeconds(30))
      .withReceiveBufferBytes(1024)
      .withSendBufferBytes(1024)
      .withReconnectBackoff(Duration.ofSeconds(3))
      .withReconnectBackoffMax(Duration.ofMinutes(5))
      .withRequestTimeout(Duration.ofSeconds(10))
      .withRetryBackoff(Duration.ofSeconds(3))
      .buildProperties();
  return new KafkaConsumer<>(consumerProps);
}

@Test
void should_build_all_properties() {
    Map<String, Object> producerProps = ProducerConfigBuilder.create()
            .withBootstrapServers(BOOTSTRAP_SERVERS)
            .withAcks(Acks.ALL)
            .withBatchSize(10)
            .withCompressionType(CompressionType.SNAPPY)
            .withEnableIdempotence(true)
            .withBufferMemory(1024L * 1000L)
            .withDeliveryTimeout(Duration.ofMinutes(10))
            .withKeySerializer(StringSerializer.class)
            .withValueSerializer(StringSerializer.class)
            .withLinger(Duration.ofMinutes(1))
            .withMaxBlock(Duration.ofSeconds(10))
            .withPartitioner(DefaultPartitioner.class)
            .withMaxRequestSize(1024)
            .withRetries(10)
            .withTransactionalId("transactionalId")
            .withTransactionTimeout(Duration.ofSeconds(30))
            .withRequestTimeout(Duration.ofSeconds(30))
            .withClientId("clientId")
            .withMaxInFlightRequestsPerConnection(5)
            .withInterceptorClasses(TestProducerInterceptor.class)
            .buildMap();

    assertEquals(BOOTSTRAP_SERVERS, producerProps.get(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG));
    assertValid(producerProps);
}

Binaries

Binaries and dependency information for Maven, Ivy, Gradle and others can be found at http://search.maven.org.

Example for Gradle:

compile 'net.soundvibe:kafka-config:2.7.0'

and for Maven:

<dependency>
    <groupId>net.soundvibe</groupId>
    <artifactId>kafka-config</artifactId>
    <version>2.7.0</version>
</dependency>

Bugs and Feedback

For bugs, questions and discussions please use the Github Issues.

LICENSE

Copyright 2019 Linas Naginionis

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.