terraform apply does not list -replace option

[jason-swissre]

Terraform Version

v1.0.7

Expected Behavior

terraform apply documentation and terraform apply -help should talk about the -replace option

Actual Behavior

Neither the documentation for the apply command nor the command itself mention the -replace options. The only location this is mentioned is in the documentation for the terraform taint command where it says this option has been deprecated since Terraform v0.15.2.

Steps to Reproduce

Check terraform apply documentation or run terraform -apply -help

Links

https://www.terraform.io/cli/commands/taint
https://www.terraform.io/cli/commands/apply

[crw]

This does seem to be an oversight that persists into terraform 1.1.2. I am wondering if this flag was also deprecated? We should either document -replace if it is a valid flag, or remove the reference in the taint documentation if it is no longer a valid flag.

[jason-swissre]

I tried to use the flag and as far as I could tell it seemed to be calling taint in the end. I also couldn't figure out how to replace multiple resources at once. terraform apply would not accept multiple replace flags but putting multiple resources separated by comma didn't work either.

[laurapacilio]

Hi @jason-swissre thank you for submitting this!

I'm talking to engineering and working on getting the docs updated so that the usage of the -replace flag is clearer. Also, my current understanding is that you should be able to use the flag multiple times to replace multiple resources at once. Would you mind providing us with the command you used to attempt this? I'd like to pass that along to engineering so they can take a look at what's going on. 🙂

[laurapacilio]

Update - the PR that is now linked to this ticket contains some documentation changes (and the rationale behind them!) that will hopefully make this a bit clearer for future users. Thank you again for pointing this out to us!

After a bit more investigation, I'll open another PR to clarify the issue surrounding replacing multiple resources at once.

[jason-swissre]

Thanks for following up on this. I looked at the PR and it didn't actually address the issues I was having. If I type terraform apply -help then -replace doesn't show up there. It's also not in the documentation for terraform apply. Literally the only place I see any mention of -replace is in the documentation for taint.

[laurapacilio]

Hello! Thank you for checking out the PR. Let me provide some more background about why we took this approach.

The -replace option is discussed in detail on the plan page and is included in the plan command CLI output. But I know what you're asking is - why isn't it fully documented on the apply page where it's also relevant?

We completely agree with you that ideally it would be documented on apply as well. The issue is that we currently do not have a great way to show identical content in multiple places within the documentation, so we'd have to manually duplicate it. The worry is that this duplicated content would result in the lists of options getting out of sync, which is also confusing to users. However, we just migrated our website to a new platform that will allow us to make many improvements in the documentation - like reuse content across the site! Once we have that functionality, we will use it to address issues like the one you flagged here.

In the meantime, our approach has been to tell users in a few separate places on the apply page that apply also accepts all of the planning options and modes available on the plan page. Here's an example:

We also have a note in the CLI output for apply -help that says:

If you don't provide a saved plan file then this command will also accept
  all of the plan-customization options accepted by the terraform plan command.
  For more information on those options, run:
      terraform plan -help

This is why I left the content on the apply page and CLI output out of the first PR. However, that PR does provide a tiny additional fix - the mention of the -replace option links directly to where the flag is documented on the plan page. Previously, this content linked to the apply page, where the -replace option is not directly documented (confusing).

Once we learn more about your issues using the flag multiple times to replace multiple resources at once, I plan to open another PR with more improvements. I just want to batch the rest of the edits for this issue together. Would you mind providing us with the command you used to replace multiple resources at once? I'd like to pass that along to engineering so they can take a look. Once we understand what's going on there, the second PR will:

• Clarify whether you can/cannot replace multiple resources and once and help users understand how to do it.
• Update one or more of the callouts on the apply page so that they are more obvious. I think right now they might be easy to miss.
• Directly state on the taint page that apply accepts plan modes and options (like -replace). Then if people are googling -replace, and get to that page, they can understand the context a bit more.

Hopefully this is helpful - thank you again for bringing this to our attention! Let me know if you have other thoughts, questions, or ideas. 😄

[jason-swissre]

Ok, thanks for the update. That makes sense. Unfortunately I cannot find the commands anymore that I used to run the replace. We have a setup where only our build system actually has access to run terraform so the logs were in the CI/CD system but I can't find the entries where I was trying to use replace.

What I know is that I tried once with comma (e.g. terraform apply -replace "module.resource_type.main,module.other_resource_type.main") and one with each resource with a separate -replace (the command itself was generated by powershell but I saw the command line and everything looked correct).

[apparentlymart]

Hi @jason-swissre,

Unfortunately PowerShell is rather notorious for corrupting command lines when running external programs (that is, anything other than a real PowerShell cmdlet), particularly on Windows where the command line quoting/escaping rules are unique to each program. While I've not heard about problems with -replace before, it does seem likely that it would run into similar challenges as other command line options we've heard about before, since it includes dot characters which seem to have been one of the common themes in previously-reported problems. (For example, we've heard before about powershell rewriting -var-file=foo.tfvars such that the .tfvars part is presented as a separate argument, which therefore makes the syntax invalid as far as Terraform is concerned.)

I believe PowerShell/PowerShell#14692 is an PowerShell PR intended to address these problems, but it works by introducing a new opt-in setting to turn on better command line parsing for external programs, since it's a breaking change that may affect existing PowerShell scripts. The issue PowerShell/PowerShell#1995 describes the broader problems here, including some details about some classes of external programs that PowerShell cannot support even with that PR, but thankfully Terraform fits in to the category of "those programs that adhere to the quoting and escaping conventions used by Microsoft's C/C++ runtime" as discussed in PowerShell/PowerShell#15143, and so should work better with that new option enabled.

Because of all of this additional configuration complexity with PowerShell, we typically recommend that folks on Windows use the Windows Command Prompt instead of PowerShell, because that then avoids the incorrect parsing/escaping that PowerShell performs in that case and passes the entered command line more faithfully to Terraform.

With the weird PowerShell command line corruption aside, the correct way to force replacing multiple resource instances at once is to specify the -replace option multiple times, as you tried before:

terraform apply -replace="module.resource_type.main" -replace="module.other_resource_type.main"

[jason-swissre]

Hi @apparentlymart, thanks for the tips. In my case I'm certain it has nothing to do with the powershell script because to get our system working I took the exact same interpolations and created taint commands instead of -replace snippets. I also had the powershell output what it was sending and it looked correct. At some point I can try the -replace operation again to see what failure I get.

[apparentlymart]

Hi @jason-swissre,

If you are able to reproduce it, it can be helpful to run Terraform with the environment variable TF_LOG=trace set once you do so. It generates a lot of noisy output to help us with debugging, but for our purposes here the relevant information is close to the top, with log lines like this:

2022-01-04T10:30:29.736-0800 [INFO]  Terraform version: 1.1.2
2022-01-04T10:30:29.736-0800 [INFO]  Go runtime version: go1.17.2
2022-01-04T10:30:29.736-0800 [INFO]  CLI args: []string{"terraform", "plan"}

The "CLI args:" one is the relevant one, because it shows how the command line options got tokenized into a list of separate strings per argument. In previous situations where we've seen problems related to PowerShell, it's been visible here due to there being extra elements of the list which don't make sense. For example, for the -var-file problem I mentioned IIRC the observed behavior was for PowerShell to pass arguments like this:

[INFO]  CLI args: []string{"terraform", "apply", "-var-file=example", ".tfvars"}

...whereas Terraform is expecting to see the -var-file option all as one token, like this:

[INFO]  CLI args: []string{"terraform", "apply", "-var-file=example.tfvars"}

Likewise for the example command like I shared the expected tokenization (if run outside of PowerShell) would be the following:

[INFO]  CLI args: []string{"terraform", "apply", "-replace=module.resource_type.main", "-replace=module.other_resource_type.main"}

If your reproduction shows there being more than four elements in that rendered array then that would suggest incorrect tokenization by the shell. On the other hand, if you do see exactly four elements like the above and yet Terraform still fails to accept the command then that'd suggest a new bug we've not seen before, and so we'd love to hear more about it.

[laurapacilio]

Hi both - I've created a final PR that I think should close out this issue. :-) Let me know if you have feedback!

#30431

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.