High CPU Usage on Consumer application

[ksdvishnukumar]

Description

Hello,

Im working on implementing kafka consumer and observed that the consumer caused high CPU utilization when multiple consumer application from different group is trying to read the message. Even though no messages are available in the partition seeing the CPU is more. All the applications waiting for the messages to receive and process.

I have a consumer close to 20 different applications. each application has 3 replicas (instances) subscribed to a single topic which has 10 partitions.

Am running the all the applications in AKS and application is trying to connect to Eventhub.

How to reproduce

Any simple application could re produce the scenario..

var consumerConfig = new ConsumerConfig
{
BootstrapServers = EventhubUrl,
EnableAutoCommit = false,
GroupId = consumerGroup,
AutoOffsetReset = AutoOffsetReset.Earliest,
CancellationDelayMaxMs = 200
};

using (var c = new ConsumerBuilder<Ignore, string>(consumerConfig).Build())
        {
            c.Subscribe(kafkaTopic);
            try
            {
                while (true)
                {
                    try
                    {
                        Console.WriteLine("Consumer running, waiting for messages");
                        var msg = c.Consume(cancellationToken);
                        Console.WriteLine($"Consumed message '{msg.Value}' at: '{msg.TopicPartitionOffset}'.");
                        c.Commit();
                    }
                    catch (ConsumeException e)
                    {
                        Console.WriteLine($"Error occured: {e.Error.Reason}");
                        Log.Error(e.Error.Reason);
                    }
                }
            }
            catch (Exception ex)
            {
                // Ensure the consumer leaves the group cleanly and final offsets are committed.
                c.Close();
                Log.Information(ex.Message);
            }
        }

The above issue happens only with when I use Subscribe method (1000 Mi CPU).

But if I use the Assign method to assign the Topic partition I don't see the CPU usage. Just to cross check and confirm with single application, consumer applications spins up with 10 consumer as a task with the single topic and different consumer group i see the CPU usage in 79 Mi CPU.

Checklist

Please provide the following information:

• A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• Confluent.Kafka nuget version.
  2.1.1
• Apache Kafka version.
• Client configuration.
• Operating system.
• Provide logs (with "debug" : "..." as necessary in configuration).
• Provide broker log excerpts.
• Critical issue.