v1.0-RC1

New Features

• Added GET subject versions to the cached schema registry client.
• References librdkafka.redist 1.0.0-RC9
  • supports apline linux out-of-the-box.
  • fallback support (that excludes security features) for most linux distributions previously unsuppored out-of-the-box.
  • fixed a dependency issue on MacOS

Changes

• A new rebalance API.
  • SetRebalanceHandler has been split into SetPartitionsAssignedHandler and SetPartitionsRevokedHandler.
  • Calling of Assign/Unassign in these handlers is prohibited.
  • Partitions to read from / start offsets can be optionally specified manually via the return value from these handlers.
• The Message.PersistenceStatus property name has changed to Message.Status.
• Moved the GetWatermarkOffsets and QueryWatermarkOffsets methods from admin client to consumer.
• Context is now provided to serdes via a SerializationContext class instance.
• All errors (that don't correspond to application logic errors) that occur as the result of a call to the produce and consume methods are now reported via exceptions of type ProduceException and ConsumeException.

Fixes

• Corrected an error in the rd_kafka_event_type method signature which was causing incompatibility with mono.
• Audited exception use across the library and made changes in various places where appropriate.
• Removed unused CancellationToken parameters (we will add them back when implemented).
• Builder classes now return interfaces, not concrete classes.
• Removed the dependency on CompilerServices.Unsafe which was causing ProduceAsync to hang in some scenarios.
• Fixed a deadlock-on-dispose issue in AdminClient.
• Made Producer.ProduceAsync async.
• Fixed deadlock in BeginProduce and Consume when using async serdes in a single threaded context.

v1.0-beta3

New Features

• Revamped producer and consumer serialization functionality.
  • There are now two types of serializer and deserializer: ISerializer<T> / IAsyncSerializer<T> and IDeserializer<T> / IAsyncDeserializer<T>.
    • ISerializer<T>/IDeserializer<T> are appropriate for most use cases.
    • IAsyncSerializer<T>/IAsyncDeserializer<T> are async friendly, but less performant (they return Tasks).
  • Changed the name of Confluent.Kafka.Avro to Confluent.SchemaRegistry.Serdes (Schema Registry may support other serialization formats in the future).
  • Added an example demonstrating working with protobuf serialized data.
• Consumers, Producers and AdminClients are now constructed using builder classes.
  • This is more verbose, but provides a sufficiently flexible and future proof API for specifying serdes and other configuration information.
  • All events on the client classes have been replaced with corresponding Set...Handler methods on the builder classes.
    • This allows (enforces) handlers are set on librdkafka initialization (which is important for some handlers, particularly the log handler).
    • events allow for more than one handler to be set, but this is often not appropriate (e.g. OnPartitionsAssigned), and never necessary. This is no longer possible.
    • events are also not async friendly (handlers can't return Task). The Set...Handler appropach can be extend in such a way that it is.
• Avro serdes no longer make blocking calls to ICachedSchemaRegistryClient - everything is awaited.
  • Note: The Consumer implementation still calls async deserializers synchronously because the Consumer API is still otherwise fully synchronous.
• Reference librdkafka.redist 1.0.0-RC7
  • Notable features: idempotent producer, sparse connections, KIP-62 (max.poll.interval.ms).
  • Note: End of partition notification is now disabled by default (enable using the EnablePartitionEof config property).
• Removed the Consumer.OnPartitionEOF event in favor notifying of partition eof via ConsumeResult.IsPartitionEOF.
• Removed ErrorEvent class and added IsFatal to Error class.
  • The IsFatal flag is now set appropriately for all errors (previously it was always set to false).
• Added PersistenceStatus property to DeliveryResult, which provides information on the persitence status of the message.

Fixes

• Added Close method to IConsumer interface.
• Changed the name of ProduceException.DeliveryReport to ProduceException.DeliveryResult.
• Fixed bug where enum config property couldn't be read after setting it.
• Added SchemaRegistryBasicAuthCredentialsSource back into SchemaRegistryConfig (#679).
• Fixed schema registry client failover connection issue (#737).
• Improvements to librdkafka dependnecy discovery (#743).

v1.0-beta2

New Features

• References librdkafka 1.0.0-PRE1. Highlights:
  • Idempotent producer.
  • Sparse connections (broker connections are only held open when in use).

Enhancements / Fixes

• Fixed a memory leak in ProduceAsync #640 (regression from 0.11.x).

v1.0-beta

New Features

• Added an AdminClient, providing CreateTopics, DeleteTopics, CreatePartitions, DescribeConfigs and AlterConfigs.
• Can now produce / consume message headers.
• Can now produce user defined timestamps.
• Added IClient, IProducer and IConsumer interfaces (useful for dependency injection and mocking when writing tests).
• Added strongly typed configuration classes.
• Added a Handle property to all clients classes:
  • Producers can utilize the underlying librdkafka handle from other Producers (replaces the 0.11.x GetSerializingProducer method on the Producer class).
  • AdminClient can utilize the underlying librdkafka handle from other AdminClients, Producers or Consumers.
• IDeserializer now exposes message data via ReadOnlySpan<byte>, directly referencing librdkafka allocated memory. This results in considerable (up to 2x) performance and reduced memory.
• Most blocking operations now accept a CancellationToken parameter.
  • TODO: in some cases there is no backing implementation yet.
• .NET Specific configuration parameters are all specified/documented in the ConfigPropertyNames class.

Major Breaking API Changes

• The Message class has been re-purposed and now encapsulates specifically the message payload only.
  • ProduceAsync / BeginProduce now return a DeliveryReport object and Consumer.Consume returns a ConsumeResult object.
• The methods used to produce messages have changed:
  • Methods that accept a callback are now named BeginProduce (not ProduceAsync), analogous to similar methods in the standard library.
  • Callbacks are now specified as Action<DeliveryReport<TKey, TValue>> delegates, not implementations of IDeliveryHandler.
  • The IDeliveryHandler interface has been depreciated.
  • There are two variants of ProduceAsync and BeginProduce, the first takes a topic name and a Message. The second takes a TopicPartition and a message.
    • i.e. when producing, there is now clear separation between what is produced and where it is produced to.
  • The new API is more future proof.
  • ProduceAsync now calls SetException instead of SetResult on the returned Task, making error checking more convenient and less prone to developer mistakes.
• The feature to block ProduceAsync calls on local queue full has been removed (result in Local_QueueFull error). This should be implemented at the application layer if required.
• The non-serializing Producer and non-deserializing Consumer types have been removed (use generic types with byte[] instead), considerably reducing API surface area.
• The ISerializingProducer interface has been removed - you can achieve the same functionality by sharing client handles instead.
• The Consumer.Poll method and corresponding OnMessage event have been removed. You should use Consumer or ConsumerAsync instead.
• The Consumer.OnPartitionEOF event has been removed. You should inspect the ConsumeResult instance returned from Consumer.Consume instead.
• The Consumer.OnConsumeError has been removed. Consume errors are now exposed via a ConsumeException.
• The Consumer.Consume method now returns a ConsumeResult object, rather than a Message via an out parameter.
• Added Consumer.ConsumeAsync methods.
  • TODO: this is not finalized. there is an ongoing discussion relating to rebalence semantics.
• CommitAsync errors are now reported via an exception and method return values have correspondingly changed.
• ListGroups, ListGroup, GetWatermarkOffsets, QueryWatermarkOffsets, and GetMetadata have been removed from Producer and Consumer and exposed only via AdminClient.
  • TODO: these method signatures need work / extra capability and will be depreciated. we should try to update before 1.0.
• Added Consumer.Close and the Consumer.Dispose method no longer blocks.
  • TODO: yes it does! this needs fixing.
• Log handlers are now specified via configuration properties, not OnLog handlers. This allows logging to be enabled during client construction.
• Various methods that formerly returned TopicPartitionOffsetError / TopicPartitionError now return TopicPartitionOffset / TopicPartition and throw an exception in
  case of error (with a Result property of type TopicPartitionOffsetError / TopicPartitionError).

Minor Breaking API Changes

• Removed cast from Error to bool.
• Consumer.OffsetsForTimes if provided an empty collection will return an empty collection (not throw an exception).
• manualPoll argument has been removed from the Producer constructor and is now a configuration option.
• enableDeliveryReports argument has been removed from the Producer constructor and is now a configuration option.
• Removed methods with a millisecondsTimeout parameter (always preferring a TimeSpan parameter).
• Added Consume and ConsumeAsync variants without a timeout parameter (but with a CancellationToken parameter that is observed).
• Added A Producer.Flush method variant without a timeout parameter (but with a CancellationToken parameter that is observed).
• Added the SyslogLevel enumeration, which is used by the log handler delegate.

Minor Enhancements / Fixes

• When delivery reports are disabled, ProduceAsync will return completed Tasks rather than Tasks that will never complete.
• Avro serializers / deserializer now handle null values.
• Examples upgraded to target 2.1.
• Changed name of HasError to IsError
• Configuration options have been added to allow fine-grained control over of marshalling of values to/from librdkafka (for high performance usage).
  • headers, message keys and values, timestamps and the topic name.
• Improved XML API documentation.

v0.11.5

Maintenance Release

• References librdkafka version 0.11.5, inheriting associated bug fixes. For more information refer to https://github.com/edenhill/librdkafka/releases/tag/v0.11.5
• The Avro generic serializer now works with multiple topics (thanks @NikolayDekov).
• Fixed a GC cleanup ordering issue during client finalization.
• CentOS 7 compatibility out of the box.

Note: we are no longer adding features to the 0.11.x branches of the .NET Client (but will continue to do maintenance releases for the foreseeable future). All new feature work is being directed towards the upcoming 1.0 release, which contains breaking API changes.

v0.11.4

Enhancements and new features

• Avro Serializer and Deserializer
• Various performance improvements
• Moved to librdkafka v0.11.4

Fixes

• handle OperationCanceledException (#446)
• added single partition overloads for Assign (#417)
• Optimize topic handle lookup (#401)
• Structs and strings marshaling performance optimization (#403)
• use ThrowIfCancellationRequested to exit producer poll loop (#413)
• set produce.offset.report to true by default (#410)
• Load librdkafka automatically on Library.Version etc. (#392)

v0.11.3

Fixes

• Compatibility with Microsoft's dotnet core 2.0 docker images.

Enhancements

• Added IgnoreDeserializer.
• Added Consumer.Seek functionality.
• Added Consumer.StoreOffsets functionality.
• More flexible library loading functionality.

v0.11.2

This release of the .NET client references librdkafka v0.11.1 which contains a number of noteworthy critical bug fixes. For more information, refer to: https://github.com/edenhill/librdkafka/releases/tag/v0.11.1

v0.11.0

Fixes

• Fixed a deadlock situation that was occurring during disposal of Producer instances after using async variants of the ProduceAsync method (#92).
• Fixed a null check related bug in Consumer.MemberId (#222).

Enhancements

• References librdkafka.redist v0.11.0 which includes support for the new Kafka message format (MsgVersion 2) which makes this client transparently compatible use with the EOS (Exactly-Once-Semantics) supporting Java client released with Apache Kafka v0.11.0. For more information refer to the librdkafka v0.11.0 release notes.
• Any exceptions thrown whilst deserializing consumed messages are now caught and more conveniently exposed via OnConsumeError events (#175).
• Additional bootstrap servers can now be added after construction of a Producer or Consumer using the new AddBrokers method (#209).
• A strong name version of the library is now build which is available on nuget.org as Confluent.Kafka.StrongName (#168).
• Obsoleted overloads of various methods on Producer and Consumer that can block indefinitely and can't be cancelled, enforcing use of variants with a timeout (#184).

This release includes external PRs from @zikifer, @andreycha and @treziac - thanks for your help! Also thanks to @chiru1205 for extensive testing under high load scenarios and @treziac once more for his valuable code reviews.

v0.9.5

Maintenance Release

Critical Fixes

• Resolves high CPU bug on Windows #87

Fixes

• timestamps are now correct in delivery reports and consumed messages.
• native library packaging related fixes.

Enhancements

• API Documentation is now complete.
• NullDeserializer now throws an exception if data is not null.
• IEnumerable is used in place of ICollection in various methods.
• Internal improvements to timestamp related functionality.
• Unnecessary events removed from ISerializingProducer interface.
• Added long serializer and deserializer.
• string serializer/deserializer now handles null values.
• now works on .NET framework 4.5 and above.
• moved to MSBuild
• added WatermarkOffsets.ToString method.

Thanks to @bjornicus, @MrGlenThomas, @wangyanjun and especially @treziac for their contributions to this release.