Client
The .NET API was completely revamped with the release of version 1.0. The 1.x branch is now well ahead of 0.11.x in terms of reliability enhancements and bug fixes. It's also well ahead in terms of features. If you are still on the 0.11.x API we highly recommend upgrading. If you are using a version < 0.11.6, we highly recommend upgrading to at least 0.11.6 (which is API compatible).
https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
If you are in doubt, don't changes values away from the defaults - there is a lot of potential to cause unintended consequences. The choice of some of the default values is arguably not quite optimal (and they remain as they are for compatibility reasons), but on the whole they are good, and you won't go far wrong sticking to them.
The clients are thread safe, except the log callback may be called from any number of internal threads, so you should protect functionality in that with a lock if no threadsafe.
Also, you must not call any client methods after calling a client's Dispose method. This sometimes catches people out in multi-threaded setups.
This question is not as straight forward as it sounds. What does 'cluster down' mean? Is that all brokers down? Or just the leaders for the partitions we care about? Does it include the replicas? If all brokers are down is this maybe just a configuration error on the client, or a temporary networking problem?
If we propagate broker state information via the client, should we then make partition leader information available, and maybe consumer coordinator information? Should we provide everything you need to re-implement the capability the client already provides?
We take the approach that as a user you shouldn't care. You configure the message.timeout.ms and message.max.retries settings and let the client take care of the rest. At the end of the day, it typically boils down to the question: what amount of time will I allow for a message to be sent before it is deemed outdated?
Yep, this is a valid ask - you may commonly want to take different action on application startup than if the cluster becomes unavailable later. For example, it's common for mis-configurations to occur, and you might want to detect that happening.
At the moment this is a bit awkward to test for. You'll need to listen on the error handler for all-brokers down error and have logic to determine how soon this occurred after application startup.
How many connections should we expect to see from the .NET client into the kafka brokers? What factors determine the connection count? (#brokers, #topics, #partitions, #client consumer instances, other?)
Refer to: https://github.com/edenhill/librdkafka/wiki/FAQ#number-of-broker-tcp-connections
The number of open connections is determined by the number of brokers. The client writes / reads data directly from the broker that is the leader for the partition of interest and commonly a client will require connections to all brokers.
The worst case number of connections held open by librdkafka is: cnt(bootstrap.servers) + cnt(brokers in Metadata response). The minimum number of connections held open is cnt(brokers in Metadata response). Currently, librdkafka holds connections open to all brokers whether or not they are needed. In the future, we plan to enhance librdkafka so that disused connections are not maintained.
It's more efficient to fully utilize you kafka client instances (whether consumer or producer):
- Each client will maintain a connection with all the brokers in the cluster. This isn't an important factor though until you get to several thousands of open connections.
- If you create multiple clients, data won't be batched as efficiently as if you had just one. The primary benefit here is that there is a fairly high per-protocol request overhead. A reduction in the number of protocol requests accounts for the largest performance wins due to utilizing clients as fully as possible
- All operation on clients are thread safe (with some small exceptions documented in the API).
You can create multiple clients if you want. If you do, they won't share any data between them.
Mono is not currently supported. It should be a fairly small amount of work to make it work, we just haven't done it yet.