NoSuchMethodError: AbstractKafkaAvroDeserializer.executeRules

[cvinacuasonoc]

On Pinot version 1.2.0-SNAPSHOT-3697552290f64b62e938152b891764adfb0ee031
We get a NoSuchMethodError when ingesting Avro messages from Kafka.
The same configuration works fine on Pinot version 1.0.0

1 2024-04-03T10:26:01.458004170+00:00 stderr F at java.base/java.lang.Thread.run(Thread.java:829)
2024-04-03 12:26:01.611 2024-04-03T10:26:01.458004170+00:00 stderr F at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager$PartitionConsumer.run(RealtimeSegmentDataManager.java:703)
2024-04-03 12:26:01.611 2024-04-03T10:26:01.458004170+00:00 stderr F at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.consumeLoop(RealtimeSegmentDataManager.java:465)
2024-04-03 12:26:01.611 2024-04-03T10:26:01.458004170+00:00 stderr F at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.processStreamEvents(RealtimeSegmentDataManager.java:576)
2024-04-03 12:26:01.611 2024-04-03T10:26:01.458004170+00:00 stderr F at org.apache.pinot.spi.stream.StreamDataDecoderImpl.decode(StreamDataDecoderImpl.java:54)
2024-04-03 12:26:01.611 2024-04-03T10:26:01.458004170+00:00 stderr F at org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder.decode(KafkaConfluentSchemaRegistryAvroMessageDecoder.java:53)
2024-04-03 12:26:01.611 2024-04-03T10:26:01.458004170+00:00 stderr F at org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder.decode(KafkaConfluentSchemaRegistryAvroMessageDecoder.java:121)
2024-04-03 12:26:01.610 2024-04-03T10:26:01.458004170+00:00 stderr F at org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder.decode(KafkaConfluentSchemaRegistryAvroMessageDecoder.java:111)
2024-04-03 12:26:01.610 2024-04-03T10:26:01.458004170+00:00 stderr F at io.confluent.kafka.serializers.KafkaAvroDeserializer.deserialize(KafkaAvroDeserializer.java:103)
2024-04-03 12:26:01.610 2024-04-03T10:26:01.458004170+00:00 stderr F at io.confluent.kafka.serializers.KafkaAvroDeserializer.deserialize(KafkaAvroDeserializer.java:108)
2024-04-03 12:26:01.610 2024-04-03T10:26:01.458004170+00:00 stderr F at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.deserialize(AbstractKafkaAvroDeserializer.java:188)
2024-04-03 12:26:01.610 2024-04-03T10:26:01.458004170+00:00 stderr F at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer$DeserializationContext.read(AbstractKafkaAvroDeserializer.java:532)
2024-04-03 12:26:01.610 2024-04-03T10:26:01.458004170+00:00 stderr F Exception in thread "sbts_edr_dlr__25__0__20240403T1024Z" java.lang.NoSuchMethodError: 'java.lang.Object io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.executeRules(java.lang.String, java.lang.String, org.apache.pinot.shaded.org.apache.kafka.common.header.Headers, java.lang.Object, io.confluent.kafka.schemaregistry.client.rest.entities.RuleMode, io.confluent.kafka.schemaregistry.ParsedSchema, io.confluent.kafka.schemaregistry.ParsedSchema, java.lang.Object)'
2024-04-03 12:26:01.61

[Jackie-Jiang]

This usually indicates a version conflict. How do you build the Pinot? Can you please check if you are using a different version of confluent Kafka?

[cvinacuasonoc]

yes, I think that is the case. Those kafka brokers are running an old version of Kafka: 2.4.1

[cvinacuasonoc]

I have moved the Kafka Topic to another Cluster running kafka_2.13-3.1.1 and the issue persists.

I have updated my Pinot deployment to "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba" on Kubernetes using the Helm attached.

This is the output of the "version" REST method:

{
"pinot-protobuf": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-kafka-2.0": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-avro": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-distribution": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-csv": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-s3": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-yammer": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-segment-uploader-default": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-batch-ingestion-standalone": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-confluent-avro": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-thrift": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-orc": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-clp-log": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-pulsar": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-azure": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-gcs": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-dropwizard": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-hdfs": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-adls": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-kinesis": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-json": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-minion-builtin-tasks": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-parquet": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba",
"pinot-segment-writer-file-based": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba"
}
pinot-dev-issue12785.txt

[TharinduDG]

I'm also getting this error when I build from master. 1.1 release branch works fine.

java.lang.NoSuchMethodError: 'java.lang.Object io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.executeRules(java.lang.String, java.lang.String, org.apache.pinot.shaded.org.apache.kafka.common.header.Headers, java.lang.Object, io.confluent.kafka.schemaregistry.client.rest.entities.RuleMode, io.confluent.kafka.schemaregistry.ParsedSchema, io.confluent.kafka.schemaregistry.ParsedSchema, java.lang.Object)'
        at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer$DeserializationContext.read(AbstractKafkaAvroDeserializer.java:532)
        at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.deserialize(AbstractKafkaAvroDeserializer.java:188)
        at io.confluent.kafka.serializers.KafkaAvroDeserializer.deserialize(KafkaAvroDeserializer.java:108)
        at io.confluent.kafka.serializers.KafkaAvroDeserializer.deserialize(KafkaAvroDeserializer.java:103)
        at org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder.decode(KafkaConfluentSchemaRegistryAvroMessageDecoder.java:111)
        at org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder.decode(KafkaConfluentSchemaRegistryAvroMessageDecoder.java:121)
        at org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder.decode(KafkaConfluentSchemaRegistryAvroMessageDecoder.java:53)
        at org.apache.pinot.spi.stream.StreamDataDecoderImpl.decode(StreamDataDecoderImpl.java:54)
        at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.processStreamEvents(RealtimeSegmentDataManager.java:576)
        at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.consumeLoop(RealtimeSegmentDataManager.java:466)
        at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager$PartitionConsumer.run(RealtimeSegmentDataManager.java:697)
        at java.base/java.lang.Thread.run(Thread.java:1583)

[Jackie-Jiang]

Just noticed this class: org.apache.pinot.shaded.org.apache.kafka.common.header.Headers
This means it is a shading issue. @xiangfu0 Can you help take a look? Will #12712 solve this?