Bug while adding new ACLs

[HaruChebrolu]

Description

When we are trying to add new ACLs, its deleting old ones and replacing with new ones.

Versions

We are using mongey kafka provider.

https://registry.terraform.io/providers/Mongey/kafka/latest
github.com/IBM/sarama v1.43.0

Configuration

provider "kafka" {
bootstrap_servers = ["localhost:9092"]
ca_cert = file("../secrets/ca.crt")
client_cert = file("../secrets/terraform-cert.pem")
client_key = file("../secrets/terraform.pem")
}

resource "kafka_acl" "test" {
resource_name = "syslog"
resource_type = "Topic"
acl_principal = "User:Alice"
acl_host = "*"
acl_operation = "Write"
acl_permission_type = "Deny"
}

im using 0.6.0 version, using Amazon Managed Kafka. running terraform in an EC2
Terraform v1.5.7
on linux_amd64
output:
build 11-Apr-2024 09:45:06 �[1m # kafka_acl.acl[61]�[0m will be created
build 11-Apr-2024 09:45:06 �[0m �[32m+�[0m�[0m resource "kafka_acl" "acl" {
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m acl_host = ""
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m acl_operation = "Read"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m acl_permission_type = "Allow"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m acl_principal = "User:CN=test.user"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m id = (known after apply)
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m resource_name = "test_dev"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m resource_pattern_type_filter = "Literal"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m resource_type = "Group"
build 11-Apr-2024 09:45:06 }
build 11-Apr-2024 09:45:06
build 11-Apr-2024 09:45:06 �[1m # kafka_acl.acl[62]�[0m will be created
build 11-Apr-2024 09:45:06 �[0m �[32m+�[0m�[0m resource "kafka_acl" "acl" {
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m acl_host = ""
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m acl_operation = "Read"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m acl_permission_type = "Allow"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m acl_principal = "User:CN=test.user"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m id = (known after apply)
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m resource_name = "kafka_topic"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m resource_pattern_type_filter = "Prefixed"
build 11-Apr-2024 09:45:06 �[32m+�[0m�[0m resource_type = "Topic"
build 11-Apr-2024 09:45:06 }
build 11-Apr-2024 09:45:06
build 11-Apr-2024 09:45:06 �[1mPlan:�[0m 58 to add, 0 to change, 55 to destroy.

[puellanivis]

This doesn’t seem related at all to sarama here, which is merely the Golang interface with Kafka. Rather you’re looking for help with terraform code.

I have seen this sort of issue before, though, and I think it’s a fairly common sort of issue with terraform sometimes: hashicorp/terraform#19974 Common situations are when you change the ordering of a list, terraform cannot just mutate into that state, so it deletes and replaces rather than in-place updates.

[HaruChebrolu]

@puellanivis
When we are trying to create ACL from here, its not where comparing between comparing new ACLs with existing ACLs. Its trying to create all ACLs whatever we send as input.
https://github.com/IBM/sarama/blob/main/broker.go#L718

Don't we need to have a check here

[puellanivis]

I’m still confused. Are you creating those ACLs by calling sarama/Broker.CreateAcls or are you creating these ACLs via terraform? If you’re not creating them via terraform, why are you including that in the issue report?

[HaruChebrolu]

We are calling it via terraform provider which internally calls sarama package to create ACL.
Please check
here https://github.com/Mongey/terraform-provider-kafka/blob/main/kafka/kafka_acls.go#L216

[puellanivis]

Yes, but sarama’s sarama/Broker.CreateAcls isn’t deleting your ACLs. Mongey is calling into sarama’s DeleteAcls the same as it is calling into sarama’s CreateAcls. The logic on deciding what to delete and recreate, or what to modify is not in sarama’s code.

I strongly think you need to be reporting this to either Mongey or terraform itself. There’s nothing anyone here can do about what those codebases do with sarama’s code.