You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have encountered some issues with the API version negotiation that the .NET Kafka producer client does it with the Kafka broker. From the logs, it seems the negotiation fails and client gets stuck in a loop re-trying and failing to connect.
How to reproduce
var conf = new ProducerConfig
{
SecurityProtocol = SecurityProtocol.Plaintext,
Acks = Acks.All,
ClientId = "test",
Debug = "broker,topic,msg",
BootstrapServers = "b-1.*******************.kafka.us-east-1.amazonaws.com:9092"
};
Action<DeliveryReport<Null, string>> handler = r =>
Console.WriteLine(!r.Error.IsError
? $"Delivered message to {r.TopicPartitionOffset}"
: $"Delivery Error: {r.Error.Reason}");
using var p = new ProducerBuilder<Null, string>(conf).Build();
p.Produce("test", new Message<Null, string> { Value = "test" }, handler);
p.Flush();
Is there a compatibility issue between this NUGET package version and the Kafka version?
Logs
%7|1705530339.756|NOINFO|test#producer-1| [thrd:main]: Topic test metadata information unknown
%7|1705530339.756|NOINFO|test#producer-1| [thrd:main]: Topic test partition count is zero: should refresh metadata
%7|1705530339.757|CONNECT|test#producer-1| [thrd:main]: Cluster connection already in progress: refresh unavailable topics
%7|1705530339.757|CONNECT|test#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
%7|1705530340.758|NOINFO|test#producer-1| [thrd:main]: Topic test metadata information unknown
%7|1705530340.758|NOINFO|test#producer-1| [thrd:main]: Topic test partition count is zero: should refresh metadata
%7|1705530340.759|CONNECT|test#producer-1| [thrd:main]: Cluster connection already in progress: refresh unavailable topics
%7|1705530340.759|CONNECT|test#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
%7|1705530341.117|CONNECT|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:90
92/bootstrap: broker in state TRY_CONNECT connecting
%7|1705530341.117|STATE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:9092
/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1705530341.118|CONNECT|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:90
92/bootstrap: Connecting to ipv4#10.227.106.144:9092 (plaintext) with socket 1120
%7|1705530341.122|CONNECT|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:90
92/bootstrap: Connected to ipv4#10.227.106.144:9092
%7|1705530341.122|CONNECTED|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:
9092/bootstrap: Connected (#8)
%7|1705530341.123|FEATURE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:90
92/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1705530341.123|STATE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:9092
/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1705530341.125|FAIL|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:9092/
bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.
version.request) (after 1ms in state APIVERSION_QUERY) (_TRANSPORT): identical to last error: error log suppressed
%7|1705530341.125|FEATURE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:90
92/bootstrap: Updated enabled protocol features -ApiVersion to
%7|1705530341.125|STATE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:9092
/bootstrap: Broker changed state APIVERSION_QUERY -> DOWN
%7|1705530341.125|CONNECT|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: Cluster connection already in progress: broker down
%7|1705530341.125|STATE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:9092
/bootstrap: Broker changed state DOWN -> INIT
%7|1705530341.125|STATE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:9092
/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1705530341.125|RECONNECT|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1.*****************************.kafka.us-east-1.amazonaws.com:
9092/bootstrap: Delaying next reconnect by 7990ms
Checklist
Please provide the following information:
A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
Confluent.Kafka nuget version: 2.3.0
Apache Kafka version: 2.8.1 (AWS MSK)
Client configuration: Described above in the coding example
Operating system: Windows Server 2016 Datacenter
Provide logs (with "debug" : "..." as necessary in configuration).
Provide broker log excerpts.
Critical issue.
The text was updated successfully, but these errors were encountered:
Description
I have encountered some issues with the API version negotiation that the .NET Kafka producer client does it with the Kafka broker. From the logs, it seems the negotiation fails and client gets stuck in a loop re-trying and failing to connect.
How to reproduce
Is there a compatibility issue between this NUGET package version and the Kafka version?
Logs
%7|1705530339.756|NOINFO|test#producer-1| [thrd:main]: Topic test metadata information unknown
%7|1705530339.756|NOINFO|test#producer-1| [thrd:main]: Topic test partition count is zero: should refresh metadata
%7|1705530339.757|CONNECT|test#producer-1| [thrd:main]: Cluster connection already in progress: refresh unavailable topics
%7|1705530339.757|CONNECT|test#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
%7|1705530340.758|NOINFO|test#producer-1| [thrd:main]: Topic test metadata information unknown
%7|1705530340.758|NOINFO|test#producer-1| [thrd:main]: Topic test partition count is zero: should refresh metadata
%7|1705530340.759|CONNECT|test#producer-1| [thrd:main]: Cluster connection already in progress: refresh unavailable topics
%7|1705530340.759|CONNECT|test#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
%7|1705530341.117|CONNECT|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:90
92/bootstrap: broker in state TRY_CONNECT connecting
%7|1705530341.117|STATE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:9092
/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1705530341.118|CONNECT|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:90
92/bootstrap: Connecting to ipv4#10.227.106.144:9092 (plaintext) with socket 1120
%7|1705530341.122|CONNECT|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:90
92/bootstrap: Connected to ipv4#10.227.106.144:9092
%7|1705530341.122|CONNECTED|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:
9092/bootstrap: Connected (#8)
%7|1705530341.123|FEATURE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:90
92/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1705530341.123|STATE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:9092
/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1705530341.125|FAIL|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:9092/
bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.
version.request) (after 1ms in state APIVERSION_QUERY) (_TRANSPORT): identical to last error: error log suppressed
%7|1705530341.125|FEATURE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:90
92/bootstrap: Updated enabled protocol features -ApiVersion to
%7|1705530341.125|STATE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:9092
/bootstrap: Broker changed state APIVERSION_QUERY -> DOWN
%7|1705530341.125|CONNECT|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: Cluster connection already in progress: broker down
%7|1705530341.125|STATE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:9092
/bootstrap: Broker changed state DOWN -> INIT
%7|1705530341.125|STATE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:9092
/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1705530341.125|RECONNECT|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1.*****************************.kafka.us-east-1.amazonaws.com:
9092/bootstrap: Delaying next reconnect by 7990ms
Checklist
Please provide the following information:
The text was updated successfully, but these errors were encountered: