Drift in `cloudflare_ruleset` for `http_request_firewall_custom` phase for computed args `id`, `last_updated`, `ref`, `version` #2690

nitrocode · 2023-08-16T22:45:56Z

Confirmation

My issue isn't already found on the issue tracker.
I have replicated my issue using the latest version of the provider and it is still present.

Terraform and Cloudflare provider version

Terraform 1.5.5
Cloudflare provider 4.12.0 latest

Affected resource(s)

cloudflare_ruleset

Terraform configuration files

resource "cloudflare_ruleset" "custom" {
  kind    = "zone"
  name    = "default"
  phase   = "http_request_firewall_custom"
  zone_id = local.zone_id

  # ... 30 rules ...

  # e.g. pre-existing rule
  rules {
    enabled     = true
    action      = "log"
    description = "<snip>"
    expression  = <<-EOT
              <snip>
            EOT
  }

  # added additional rule which caused a drift. could be any rule
  rules {
    enabled     = true
    action      = "log"
    description = "<snip>"
    expression  = <<-EOT
              any(http.request.headers["custom-header"][*] matches ".?")
            EOT
  }

Link to debug output

none

Panic output

N/A

Expected output

No perma drift

Actual output

Perma drift

Steps to reproduce

Add a new rule or modify an existing one
terraform plan

Additional factoids

This happened after we bumped the provider from 3.x to 4.x

We deleted the cloudflare_ruleset and reimported.

If we do not make a change, we see No changes as expected. Any change we make, and we see a bunch of drift between rules, all affecting arguments that we do not set such as id and version

We have 30 something rules and we see these computed arguments changing for every rule.

in 4.12.0 cloudflare provider

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # cloudflare_ruleset.custom will be updated in-place
  ~ resource "cloudflare_ruleset" "custom" {
        id      = "<snip>"
        name    = "default"
        # (3 unchanged attributes hidden)

      ~ rules {
          ~ id           = "<snip>" -> (known after apply)
          ~ last_updated = "2023-08-18 16:36:15.176813 +0000 UTC" -> (known after apply)
          ~ ref          = "<snip>" -> (known after apply)
          ~ version      = "3" -> (known after apply)
            # (4 unchanged attributes hidden)
        }
      ~ rules {
          ~ id           = "<snip>" -> (known after apply)
          ~ last_updated = "2023-08-18 16:36:15.176813 +0000 UTC" -> (known after apply)
          ~ ref          = "<snip>" -> (known after apply)
          ~ version      = "15" -> (known after apply)
            # (4 unchanged attributes hidden)

          ~ action_parameters {
              + version  = (known after apply)
                # (3 unchanged attributes hidden)
            }

            # (1 unchanged block hidden)
        }
      + rules {
          + action       = "log"
          + description  = "log traffic with headers"
          + enabled      = true
          + expression   = <<-EOT
                any(http.request.headers["custom-header"][*] matches ".?")
            EOT
          + id           = (known after apply)
          + last_updated = (known after apply)
          + ref          = (known after apply)
          + version      = (known after apply)
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

In 4.7.1 cloudflare provider

      ~ rules {
          ~ id           = "<snip>" -> (known after apply)
          + last_updated = (known after apply)
          ~ version      = "3" -> (known after apply)
            # (4 unchanged attributes hidden)
        }
      ~ rules {
          ~ id           = "<snip>" -> (known after apply)
          + last_updated = (known after apply)
          ~ version      = "3" -> (known after apply)
            # (4 unchanged attributes hidden)

          ~ action_parameters {
              + version  = (known after apply)
                # (1 unchanged attribute hidden)
            }

            # (1 unchanged block hidden)
        }
      + rules {
          + action       = "log"
          + description  = "log traffic with headers"
          + enabled      = true
          + expression   = <<-EOT
                any(http.request.headers["custom-header"][*] matches ".?")
            EOT
          + id           = (known after apply)
          + last_updated = (known after apply)
          + ref          = (known after apply)
          + version      = (known after apply)
        }

We also cannot set a lifecycle ignore rule because this does not work for us

  lifecycle {
    ignore_changes = [
      "rules.*.id",
      "rules.*.version",
      "rules.*.last_updated",
      "rules.*.ref",
      # "rules[*].id",
      # "rules.id",
    ]
  }

I'm told this might work but it's not scaleable. We have over 30 rules. I tried this and still see the drift!

  lifecycle {
    ignore_changes = [
      rules[0].id,
      rules[0].version,
      rules[0].last_updated,
      rules[0].ref,
      rules[1].id,
      rules[1].version,
      rules[1].last_updated,
      rules[1].ref,
      rules[2].id,
      rules[2].version,
      rules[2].last_updated,
      rules[2].ref,
      rules[3].id,
      rules[3].version,
      rules[3].last_updated,
      rules[3].ref,
      # ...
      rules[n].id,
      rules[n].version,
      rules[n].last_updated,
      rules[n].ref,
    ]
  }

We did not see this issue in 3.x cloudflare provider

I don't see any DiffSuppressFunc for any of the above drifted computed arguments. Unsure if the drift needs to be suppressed or if it needs to be investigated.

https://github.com/cloudflare/terraform-provider-cloudflare/blob/adbc0accaafe0df39e78fcfd61d092573e13a25d/internal/framework/service/rulesets/resource.go

References

Related issues

Related pr

resource/cloudflare_ruleset: fix various attributes #2511 released in 4.8.0

From change log

BREAKING CHANGES:

resource/cloudflare_ruleset: Prevent the rule ID, version and last updated attributes from being set (resource/cloudflare_ruleset: fix various attributes #2511)

The text was updated successfully, but these errors were encountered:

github-actions · 2023-08-16T22:46:08Z

Community Note

Voting for Prioritization

Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

If you are interested in working on this issue, please leave a comment.
If this would be your first contribution, please review the contribution guide.

github-actions · 2023-08-16T22:46:31Z

Thank you for reporting this issue! For maintainers to dig into issues it is required that all issues include the entirety of TF_LOG=DEBUG output to be provided. The only parts that should be redacted are your user credentials in the X-Auth-Key, X-Auth-Email and Authorization HTTP headers. Details such as zone or account identifiers are not considered sensitive but can be redacted if you are very cautious. This log file provides additional context from Terraform, the provider and the Cloudflare API that helps in debugging issues. Without it, maintainers are very limited in what they can do and may hamper diagnosis efforts.

This issue has been marked with triage/needs-information and is unlikely to receive maintainer attention until the log file is provided making this a complete bug report.

jessegonzalez-life360 · 2023-08-19T06:26:36Z

I can provide this, but I'm not comfortable posting that here in a public forum. I'm an enterprise customer, can I create a case?

Rules, IPs, etc. would take a long while to redact.

Log uploaded to case id: #2901904

jessegonzalez-life360 · 2023-08-25T21:20:44Z

Thanks for the lifecycle hint. It can help with perma drift until a proper solution is identified.

github-actions · 2023-09-25T00:21:04Z

Marking this issue as stale due to 30 days of inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 7 days it will automatically be closed. Maintainers can also remove the lifecycle/stale label.
If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

DDuarte · 2023-10-10T08:16:01Z

Confirming this bug as well, any time there's a change to 1 rule, all the other rules in the ruleset get their mentioned fields updated in diff

jessegonzalez-life360 · 2023-11-01T17:12:55Z

@DDuarte and @nitrocode if you define a unique ref for the rules block, you should no longer see permanent drift.

Edit: Spoke too soon, the issue persists.

gabrielqs · 2023-11-14T15:08:51Z

+1 dealing with this issue.
given that we're migrating from multiple deprecated cloudflare_firewall_rules into a single new cloudflare_ruleset, this makes the plan outputs become really hard to read. currently using a list of 300 attributes being ignored by lifecycle rules 😅

vojkny · 2023-11-27T08:36:34Z

I tried to fix this something like:

    lifecycle {
      ignore_changes = ["rules[*].ref", "rules[*].last_updated"]
    }

However there is nothing like [*] in Terraform syntax, so this couldn't work.

nitrocode · 2023-11-28T05:31:29Z

@gabrielqs could you share what you use? I tried setting the ignore_changes and I could not get it to work.

github-actions · 2023-12-29T00:18:20Z

Marking this issue as stale due to 30 days of inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 7 days it will automatically be closed. Maintainers can also remove the lifecycle/stale label.
If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

gabrielqs · 2023-12-29T06:44:55Z

This is still a problem

gabrielqs · 2023-12-29T13:16:39Z

@nitrocode sorry, only seeing your message now. here's what we had to do to get around this:

  lifecycle {
    ignore_changes = [
      rules[0].id,
      rules[0].version,
      rules[0].last_updated,
      rules[0].ref,
      rules[0].action_parameters[0].version,
      rules[1].id,
      rules[1].version,
      rules[1].last_updated,
      rules[1].ref,
      rules[1].action_parameters[0].version,
      rules[2].id,
      rules[2].version,
      rules[2].last_updated,
      rules[2].ref,
      rules[2].action_parameters[0].version,
      ...

It is terrible, and we need to remind ourselves to add this every time we add a new rule, but at least the drift doesn't show up on every plan

nitrocode · 2023-12-30T16:29:05Z

I'm happy it works for you but that doesn't work for us. We still see the drift even with explicitly specifying each individual attributes per rule in ignore_changes.

Since these rules are all in a single terraform resource and we cannot get the plan in a reviewable state, we may have to consider moving these rules out of terraform and having them managed with a script that supplants all the rules upon pr merge.

cc @jacobbednarz friendly ping to get cloudflare's feedback here. This is the 2nd highest voted issue and ideally we should be able to manage and review changes within terraform without so much noise.

troymjones · 2024-01-22T17:57:12Z

I am providing a full TF_LOG=DEBUG for 2 scenarios, both of which are problematic when working with the Cloudflare ruleset resource. Please note that while there is a workaround (manually verifying them), this is a major issue for my team with a hundred rules in a zone and could result in a bad rule change making its way into the WAF.

Scenarios:

A rule is added in the middle of existing rules. Note that it looks like rules are drastically changing and a 'new' rule is being added to the bottom of the list. However, that 'new' rule is just an existing rule at the end. The rules are correctly being shown, but it is extremely confusing. When there are more than a couple rules, it becomes almost impossible to parse through the changes to make sure no mistakes were made. Also, all rules show changes to id, ref, version, etc, further complicating the issue. Link to gist
A rule is modified (ie name, expression, etc) but no other changes are made. This results in the plan showing all rules having changed. Again, if there are more than a couple rules, it makes it almost impossible to verify that the changes desired are what is shown in the plan. Link to gist

Within the rulesets schema, we have a handful of fields that are `Computed`. The expectation of this directive is that it is a value not known at run time and _may_ be provided by the remote. However, this premise will break down and not work correctly if the value is ever provided to the plan since the value is no longer "unknown". This subtle bug is one part of what is contributing to the additional output toil in #2690. To address this, I've removed the lines that modified these values and were being supplied to the plan operation. This takes the confusing and bloated output from looking like this: ``` Terraform will perform the following actions: # cloudflare_ruleset.rate_limiting_example will be updated in-place ~ resource "cloudflare_ruleset" "rate_limiting_example" { id = "87e1099f077f4e49bbfbe159217ff605" name = "restrict API requests count" # (4 unchanged attributes hidden) ~ rules { ~ id = "e3b97fdf354c4c24a7ac091c3537fbc5" -> (known after apply) ~ last_updated = "2024-01-31 04:02:55.651275 +0000 UTC" -> (known after apply) ~ ref = "e3b97fdf354c4c24a7ac091c3537fbc5" -> (known after apply) ~ version = "1" -> (known after apply) # (4 unchanged attributes hidden) # (1 unchanged block hidden) } + rules { + action = "block" + description = "rate limit for API" + enabled = true + expression = "(http.request.uri.path matches \"^/api0/\")" + id = (known after apply) + last_updated = (known after apply) + ref = (known after apply) + version = (known after apply) + ratelimit { + characteristics = [ + "cf.colo.id", + "ip.src", ] + mitigation_timeout = 600 + period = 60 + requests_per_period = 100 + requests_to_origin = false } } } Plan: 0 to add, 1 to change, 0 to destroy. ``` To a clearer and more concise output: ``` Terraform will perform the following actions: # cloudflare_ruleset.rate_limiting_example will be updated in-place ~ resource "cloudflare_ruleset" "rate_limiting_example" { id = "87e1099f077f4e49bbfbe159217ff605" name = "restrict API requests count" # (4 unchanged attributes hidden) ~ rules { id = "39ed6015367444e3905d838cadc1b9c2" + last_updated = (known after apply) + version = (known after apply) # (5 unchanged attributes hidden) # (1 unchanged block hidden) } + rules { + action = "block" + description = "rate limit for API" + enabled = true + expression = "(http.request.uri.path matches \"^/api0/\")" + id = (known after apply) + last_updated = (known after apply) + ref = (known after apply) + version = (known after apply) + ratelimit { + characteristics = [ + "cf.colo.id", + "ip.src", ] + mitigation_timeout = 600 + period = 60 + requests_per_period = 100 + requests_to_origin = false } } } Plan: 0 to add, 1 to change, 0 to destroy. ``` The second part of this confusing output is the `last_updated` and `version` output. The `last_updated` is pretty self explanatory however, there are some minor but important details about the `version` field here. Within ERE, it captures and is tied to a particular state of the ruleset rule at any point and is incremented when changes are detected to any parts of the rule. The naunce here is that ERE does not perform a diff of the current state and what is proposed. Instead, it autoincrements this field **even if the payload is identical**. So why is this important for the Terraform resource? Well, since we use explicit ordering via [`ListNestedObject`][1] schema attribute, we are sending all rules to the API even when only a single one changes to ensure we maintain the correctness the end user expects. Doing this causes the `last_updated` and `version` fields to always show as unknown values and result in updates. There are some plans to potentially look at de-duping the payloads to prevent spurious versions being created however, that won't help here until such a time that we individually manage `rules` as their own entities. Closes #2690 by improving the output and marking the `version` and `last_updated` as known values that will always be changing due to the way we update all the resources. [1]: https://github.com/cloudflare/terraform-provider-cloudflare/blob/f7c05af8be05e0ef9da72c10baa5a5bf8440e5d7/internal/framework/service/rulesets/schema.go#L104

eephillip · 2024-02-13T16:58:46Z

I tried to fix this something like:
    lifecycle {
      ignore_changes = ["rules[*].ref", "rules[*].last_updated"]
    }
However there is nothing like [*] in Terraform syntax, so this couldn't work.

Related hashicorp/terraform#24188

pranit-p · 2024-02-13T17:01:08Z

Yes @eephillip

But, this code is not workable

I tried to fix this something like:
    lifecycle {
      ignore_changes = ["rules[*].ref", "rules[*].last_updated"]
    }
However there is nothing like [*] in Terraform syntax, so this couldn't work.
Related hashicorp/terraform#24188

github-actions · 2024-03-15T00:21:32Z

Marking this issue as stale due to 30 days of inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 7 days it will automatically be closed. Maintainers can also remove the lifecycle/stale label.
If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

fcoelho · 2024-03-15T00:45:21Z

this is still an issue

gabrielqs · 2024-03-15T01:17:44Z

Still an issue

Anticmos · 2024-03-26T16:06:44Z

Still an Issue

Anticmos · 2024-03-26T16:09:51Z

@nitrocode can you provide the TF_LOG=DEBUG output so that they can remove needs-information tag by chance?

nitrocode · 2024-03-26T20:19:01Z

@Anticmos please see this comment where the debug log was attached to an internal cloudflare case #2901904 (see #2690 (comment) in Aug 2023)

nitrocode · 2024-03-26T20:20:53Z

@Anticmos feel free to add another one. 😄

jogarao96 · 2024-04-08T21:08:44Z

work around is that deleting the rule from cloudflare cli and recreating it through code. altough you might deleted at cloudflare gui we still have that rule exsits in cli version of cloudflare.
Fix is that

curl https://api.cloudflare.com/client/v4/zones/<ZONE_ID>/rulesets --header "X-Auth-Key:{you key here}" --header "X-Auth-Email:{your email here}" >rulesets.yml
Identify the thr rule with no discription but the phase you are trying at in my case http_request_firewall_custom
Delete that rule with cloudflare cli using curl --request DELETE https://api.cloudflare.com/client/v4/zones/<ZONE_ID>/rulesets/<RULESETID> --header "X-Auth-Key:{API KEY}" --header "X-Auth-Email:{EMAIL}" you can get rule id from above file
Re run terraform script to create a new rule which doesnt colide with any rule set ID

boudekerk · 2024-04-11T13:46:22Z

We're also running into this, as others have mentioned this kind of defeats the purpose of using terraform as the plan becomes unreadable. I.e. at least in the UI it's clear which rule is being edited before saving.

boudekerk · 2024-04-12T09:05:37Z

@gabrielqs We tried your workaround, but we got errors during apply when trying to lifecycle ignore versions and last_updated.
Is there anything special you did to get it to work?

I think one issue is that the action parameter version is always being added, but never ends up in the state, so is added again on each subsequent run.

gabrielqs · 2024-04-12T14:25:31Z

@boudekerk nothing spectial. the only caveat might be we're running a relatively old version of the cloudflare provider, so maybe there's a discrepancy between what that version is doing and yours?

nitrocode · 2024-04-12T22:08:14Z

@gabrielqs what version are you running? Maybe we can pin until this issue is resolved.

Is it between version 4 and 4.8.0?

github-actions · 2024-05-13T00:23:58Z

Marking this issue as stale due to 30 days of inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 7 days it will automatically be closed. Maintainers can also remove the lifecycle/stale label.
If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

nitrocode · 2024-05-13T03:22:16Z

Unstale

nitrocode added kind/bug Categorizes issue or PR as related to a bug. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 16, 2023

github-actions bot added triage/needs-information Indicates an issue needs more information in order to work on it. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 16, 2023

nitrocode changed the title ~~Drift in cloudflare_ruleset for http_request_firewall_custom phase~~ Drift in cloudflare_ruleset for http_request_firewall_custom phase for computed args id, last_updated, ref, version Aug 19, 2023

github-actions bot added the lifecycle/stale label Sep 25, 2023

This comment was marked as spam.

Sign in to view

github-actions bot removed the lifecycle/stale label Sep 26, 2023

This comment was marked as spam.

Sign in to view

github-actions bot added the lifecycle/stale label Dec 29, 2023

github-actions bot removed the lifecycle/stale label Dec 30, 2023

This comment was marked as spam.

Sign in to view

This comment has been minimized.

Sign in to view

Nmishin mentioned this issue Jan 27, 2024

Remove unneeded parameters from cloudflare_ruleset #3082

Closed

jacobbednarz mentioned this issue Jan 31, 2024

resource/cloudflare_ruleset: improve unknown value handling #3091

Closed

This comment has been minimized.

Sign in to view

github-actions bot added the lifecycle/stale label Mar 15, 2024

github-actions bot removed the lifecycle/stale label Mar 16, 2024

github-actions bot added the lifecycle/stale label May 13, 2024

github-actions bot removed the lifecycle/stale label May 14, 2024

Drift in cloudflare_ruleset for http_request_firewall_custom phase for computed args id, last_updated, ref, version #2690

Drift in cloudflare_ruleset for http_request_firewall_custom phase for computed args id, last_updated, ref, version #2690

Comments

nitrocode commented Aug 16, 2023 • edited

Confirmation

Terraform and Cloudflare provider version

Affected resource(s)

Terraform configuration files

Link to debug output

Panic output

Expected output

Actual output

Steps to reproduce

Additional factoids

References

github-actions bot commented Aug 16, 2023

Community Note

github-actions bot commented Aug 16, 2023

jessegonzalez-life360 commented Aug 19, 2023 • edited

jessegonzalez-life360 commented Aug 25, 2023

github-actions bot commented Sep 25, 2023

This comment was marked as spam.

This comment was marked as spam.

DDuarte commented Oct 10, 2023

This comment was marked as spam.

jessegonzalez-life360 commented Nov 1, 2023 • edited

gabrielqs commented Nov 14, 2023

vojkny commented Nov 27, 2023

nitrocode commented Nov 28, 2023

github-actions bot commented Dec 29, 2023

gabrielqs commented Dec 29, 2023

gabrielqs commented Dec 29, 2023

nitrocode commented Dec 30, 2023

troymjones commented Jan 22, 2024

This comment was marked as spam.

This comment has been minimized.

This comment has been minimized.

eephillip commented Feb 13, 2024 • edited

pranit-p commented Feb 13, 2024 • edited

github-actions bot commented Mar 15, 2024

fcoelho commented Mar 15, 2024

gabrielqs commented Mar 15, 2024

Anticmos commented Mar 26, 2024

Anticmos commented Mar 26, 2024

nitrocode commented Mar 26, 2024 • edited

nitrocode commented Mar 26, 2024

jogarao96 commented Apr 8, 2024 • edited

boudekerk commented Apr 11, 2024

boudekerk commented Apr 12, 2024 • edited

gabrielqs commented Apr 12, 2024

nitrocode commented Apr 12, 2024 • edited

github-actions bot commented May 13, 2024

nitrocode commented May 13, 2024

Drift in `cloudflare_ruleset` for `http_request_firewall_custom` phase for computed args `id`, `last_updated`, `ref`, `version` #2690

Drift in `cloudflare_ruleset` for `http_request_firewall_custom` phase for computed args `id`, `last_updated`, `ref`, `version` #2690

nitrocode commented Aug 16, 2023 •

edited

jessegonzalez-life360 commented Aug 19, 2023 •

edited

jessegonzalez-life360 commented Nov 1, 2023 •

edited

eephillip commented Feb 13, 2024 •

edited

pranit-p commented Feb 13, 2024 •

edited

nitrocode commented Mar 26, 2024 •

edited

jogarao96 commented Apr 8, 2024 •

edited

boudekerk commented Apr 12, 2024 •

edited

nitrocode commented Apr 12, 2024 •

edited