Access denied error for IAM role based AWS MSK cluster while using confluent-kafka python library

[balajibreddi]

Description

Here is the overview of the application, it consumes from the upstream kafka cluster processes the kafka message and produces it to the downstream kafka cluster using the confluent-kafka python library(2.3.0).

We have changed the authentication type from SASL/SCRAM to IAM role-based and to do that we have added a trust relationship and also given all access(MSK, Apache api's) to applications in the policy attached of ECS service, but still, we see Access denied errors.

The weird behaviour we see in the application logs is it doesn't throw errors while processing kafka messages but if it sits idle for around 5 hours its starts throwing these errors, if any messages come in then it will not throw errors for the next 5 hours. This behaviour is odd.

Error: %3|1714955097.400|FAIL|8b68559c-dbf7-401b-ac6c-807523ee37ee#producer-1| [thrd:sasl_ssl://b-3.clusteranme.stinjb.c7.kafka.region.]: sasl_ssl://b-3.clusteranme.stinjb.c7.region.amazonaws.com:9098/3: SASL authentication error: [6ad8e7d6-f5f0-41c9-930f-26cc577779ed]: Access denied (after 346ms in state AUTH_REQ)

FYI: To generate an auth token we are using the aws_msk_iam_sasl_signer library from AWS to generate a token based on region and passing it to oauth_cb config parameter of Producer.

How to reproduce

Checklist

Please provide the following information:

• confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): 2.3.0
• MSK Apache Kafka broker version: 3.5.1
• Client configuration: {'security.protocol'="SASL_SSL",
  'client.id'=str(uuid.uuid4()),
  'bootstrap.servers'="b-1.clustername.stinjb.c7.kafka.us-east-2.amazonaws.com:9098,b-
  2.clustername.stinjb.c7.kafka.us-east-2.amazonaws.com:9098,b-3.clustername.stinjb.c7.kafka.us-east-2.amazonaws.com:9098",
  'sasl.mechanism'="OAUTHBEARER",
  'acks'=1,
  'oauth_cb'="Token from MSKAuthTokenProvider.generate_auth_token",
  'compression.type'="gzip",
  'reconnect.backoff.max.ms'=3000,
  'retries'=3,
  'request.timeout.ms'=15000)}`
• Operating system: Linux/X86_64