Feature Request: Stop, don't destroy instance on user-data update

[hashibot]

This issue was originally opened by @jwaldrip as hashicorp/terraform#1887. It was migrated here as part of the provider split. The original body of the issue is below.

---

when updating user-data, its not required to destroy the instance. In fact, this can be devastating to a etcd cluster. What would be acceptable is for the machines to be stopped, userdata updated, and then started.

[hashibot]

This comment was originally opened by @JeanMertz as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

This is interesting. I don't think Terraform currently has a stop/start cycle, only destroy or don't destroy.

I can see a lot of value for this use-case (since we're also using user-data heavily), but I can also see this becoming complicated.

If I am not mistaken, on AWS an instance without an EBS backed storage device will simply reset the storage, so in that case the stop/start would have the same result as a destroy.

Here's the list of properties that AWS allows you to change in a stopped state:

You can modify the following attributes of an instance only when it is stopped:

• Instance type
• User data
• Kernel
• RAM disk

[hashibot]

This comment was originally opened by @sparkprime as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

So this particular issue is not a problem with GCE as you can update metadata without forceNew. However there is a similar problem if you want to change zone / scopes without losing the data on your disk.

The way I go about doing this is to use a separate disk resource in any situation where the content of the disk is precious in some way. E.g. obviously for databases this is true, but typically in a node of a load balanced cluster the disk is expendable. Having a separate disk allows Terraform to recreate the instance without losing the disk (as the disk is not recreated). I believe this is equivalent to a reboot cycle.

[hashibot]

This comment was originally opened by @Pryz as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

What about doing nothing when we update user_data ?

If you use Terraform and a configuration management solution, most of the time userdata are only for instance bootstrapping. So when you change userdata it's probably only for new instances not current ones.

[hashibot]

This comment was originally opened by @jwaldrip as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Disagree. I use CoreOS in production and manage everything with user data. Even existing instances.

[hashibot]

This comment was originally opened by @Pryz as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Ok, so what about having an argument to specify the strategy ? (Reboot, Recreate, None)

[hashibot]

This comment was originally opened by @sparkprime as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

For the sake of comparison: For google compute instance metadata, we started off with updateable metadata (i.e. don't recreate the instance, but update the metadata so the instance can see the new values). This is actually very useful because you can do a hanging get on the metadata server which means you can run an agent in your VM that gets notified of changes to the metadata. This allows you to control the VM (e.g. roll out new application software) by updating the metadata. I don't know if AWS let's you do that but I assume it does as it's really useful.

However if you don't have such an agent, and the only thing you have to work with (if you want to do truly declarative node config) is the startup-script (userdata on AWS), then you want changing the startup script to recreate the instance so it will re-run the startup script. For lots of people, especially stateless servers, that is plenty good enough. The only cost is the time taken to recreate the instance (the agent is instantaneous).

So, I added a metadata_startup_script field in the google instance that is the same as metadata.startup-script except it is ForceNew. One is not allowed to specify both. The user can therefore choose whether to have the ForceNew behavior or not.

You could do the same thing for aws_instance: Have an updateable_user_data field that was not ForceNew to allow people to make changes to it without recreating the instance and run agents inside the VM to pick up these changes. And keep the current user_data field for the simple non-agent case.

[hashibot]

This comment was originally opened by @Pryz as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

I like the idea of having two different user_data fields since it's for two totally different ways to manage instances.

[hashibot]

This comment was originally opened by @FergusNelson as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

I think it would be nice to have stop and start command line options in general. The use case would be a staging or testing environment that is not needed all the time; rather than destroy and recreate the environment every time it is needed it would be nice to be able to do

terraform stop
terraform start

And have terraform start the machines in the correct order with respect to the dependency graph.

[hashibot]

This comment was originally opened by @c4milo as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

As far as I understand the sole purpose of cloud-init or user-data scripts is to do early initialization of instances. From that perspective, it may not make sense to use it as a way to re-provision or re-configure instances since that's what tools like Puppet, Chef, Ansiable and Salt are for. Terraform was thought out as a way of creating and destroying infrastructure resources, and resource immutability is all over the place. Perhaps @mitchellh can shed some more light here.

[hashibot]

This comment was originally opened by @jwaldrip as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Sure, this is the case in some instances. But in our case, we use core-os
and use user-data to bootstrap the machines. We NEVER use puppet, chef,
salt etc, to instantiate the instance any further. Maybe this is bad
practice, but I know that user-data is subject to change. In addition we
cannot destroy our etcd instances to provision fresh user-data as we will
have a risk of losing quorom.

On Fri, Oct 23, 2015 at 11:13 AM, Camilo Aguilar notifications@github.com
wrote:

As far as I understand the sole purpose of cloud-init or user-data scripts
is to do early initialization of instances. From that perspective, it may
not make sense to use it as a way to re-provision or re-configure instances
since that's what tools like Puppet, Chef, Ansiable and Salt are for.
Terraform was thought out as a way of creating and destroying
infrastructure resources and resource immutability is all over the place.
Perhaps @mitchellh https://github.com/mitchellh can shed some more
light here.

—
Reply to this email directly or view it on GitHub
hashicorp/terraform#1887 (comment)
.

Jason Waldrip
m: 646-460-5959
e: jason@waldrip.net
http://www.facebook.com/jason.waldrip
http://www.linkedin.com/in/jasonwaldrip

[hashibot]

This comment was originally opened by @sparkprime as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

It's not bad practice, it works very well and is increasingly common.

[hashibot]

This comment was originally opened by @sparkprime as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Did you try using a separate disk resource? If you also use a statically assigned ip then I think that should be equivalent to a reboot?

[hashibot]

This comment was originally opened by @JamiKarvanen as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

I have the same use case as @jwaldrip. When using AWS's CloudFormation, it updates the user_data but doesn't re-create or reboot the instances. Amazon provides a cnf-hup script that detects user_data changes in instances and updates them accordingly, so it would be great to be able to only update the user_data with terraform and let AWS handle the updating.

[hashibot]

This comment was originally opened by @sparkprime as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Upgrading to "It's not bad practice, it works very well, is increasingly common, and is actively encouraged by AWS and Google" :)

[hashibot]

This comment was originally opened by @manojlds as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

No movement on this?

[hashibot]

This comment was originally opened by @sparkprime as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

I think it should be a fairly easy PR to add a new resource attribute for controlling userdata in a non force-new fashion. If someone were suitably motivated :)

[hashibot]

This comment was originally opened by @yogin as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

I'd be very interested in a way to prevent instances from being recreated when there's a change to userdata. I understand some people use it for different purposes, and I like the idea of specifying a strategy (recreate, none, ...).

In our case, we only use the userdata to bootstrap new instances, later on, chef will take over and do the rest, but occasionally the userdata template might get a small update, and if I could avoid having to recreate instances, that'd be amazing!

[hashibot]

This comment was originally opened by @mcortinas as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

No movement on this?

[hashibot]

This comment was originally opened by @modax as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

I'm no terraform developer but I had a quick look at the current terraform code and it seems that the only possible solution without (big) terraform core changes could be adding another attribute like live_user_data (since ForceNew value cannot depend on the context as far as can tell). The code might still end up being messy due to two attributes modifying the same thing (with regard to diff calculation). Hence I'm not sure if terraform team would accept such a PR.

[hashibot]

This comment was originally opened by @sparkprime as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

@modax As I said further up the thread, this is exactly what the google_compute_instance does and it works well

[hashibot]

This comment was originally opened by @markmaas as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Destroying and recreating instances when the user-data changes makes it so I can either:

• Not use user-data (And therefore cloud-config)
• Not use Terraform (Meaning I have to learn cloud-formation: ugh)

Since cloud-config is beeing used more and more to do the initial bootstrapping of an instance, I would think this is no longer an "enhancement" but more along the lines of "blocking"

[hashibot]

This comment was originally opened by @cheungpat as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

If you don’t want your instance to re-create when user-data changes, you can try including this key in ignore_changes.

[hashibot]

This comment was originally opened by @br0ch0n as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

I agree with @cheungpat that ignore_changes probably handles this ticket (once #5627 is done)

[hashibot]

This comment was originally opened by @gdubicki as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

I confirm that what @cheungpat suggests is a good workaround. If you use GCE provider, then to ignore startup script changes use this code:

lifecycle {
  ignore_changes = ["metadata_startup_script"]
}

But it would be even better if you could set it once, globally - not in each resource of google_compute_instance type. Is that possible?

UPDATE: AFAIK it's not possible and you can't even use a variable to set ignore_changes. :( See #10730.

[hashibot]

This comment was originally opened by @kirkmadera as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

This also applies to resizing AWS instances. We end up going into the AWS admin, stopping the instance, changing the instance type, then starting it again. Then we update terraform configs to match the new instance type and run terraform apply which syncs the terraform state to the new size.

Ideally we would just resize from Terraform. Resizing an AWS server also changes it's public and private ips. Running the resize from terraform would allow us to immediately change DNS as well if needed, rather than waiting for us to run terraform after the resize completes.

[hashibot]

This comment was originally opened by @stack72 as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

Dearest Friends, now that I have tackled the issue of changing instance_type and it will be part of terraform 0.8.8, I am going to start prototyping on this

Paul

[hashibot]

This comment was originally opened by @cnoffsin as hashicorp/terraform#1887 (comment). It was migrated here as part of the provider split. The original comment is below.

---

The use case for me here is I have to make type changes in AWS to a cloudera cluster.

The engineers shut it down gracefully so I could make the change.

I want to make the type change, but LEAVE them off. So that they can bring them up in the proper order.

[artkrz]

This would be very useful for me as well.

[bwolfehcm]

++^
This would be very helpful. I currently have this issue and I need a work around so it doesn't destroy my needed directory servers.
I'm not sure how the user data got out of sync either.

[jen20]

Yup. The focus here is that user data only runs once. Updating it on an existing instance does not trigger its execution.

This is not strictly true. To understand why, it's important to consider the difference between user data, and what any given image does with the user data. This is true across many different clouds, not just AWS.

In AWS, when you specify user data, it becomes available on the metadata endpoint for the instance. The fact that cloudinit does not respond to updates is entirely orthogonal to the ability to update this configuration at the provider - there are many images in use which do not use cloudinit at all, and instead pass other types of configuration data via user data. One such example is modern Windows images, which use ec2config instead of cloudinit.

However, Terraform does not have the concept of and update which requires a restart of a resource in this context - but it probably should learn to do so and allow opting in where it makes sense.

[cornfeedhobo]

I guess that's fair, but my point is that an admin is aware of this context and probably just wants to record changes intended for the next invocation of an instance, but I've started using lifecycle rules to ignore this change, which suits my use case. At this point, I'd rather y'all focus on cleaning up your backlog.

[glenjamin]

Would a PR to add a distinct aws_instance_user_data resource be likely to be accepted?

[dinvlad]

Updating instance_type (which also requires a similar stop-update-start cycle) has been supported for a while: #4838

Could we adopt a similar solution here? We had to engineer a different way around updating instance configuration (via SSM Document attachment), in large part because of this limitation in Terraform for user_data.

EDIT: for the context, we have a third-party proprietary application that has to pass some instance configuration via user_data, and it makes any updates to this configuration highly problematic. This application also can't tolerate an instance replacement, since it is tied to a particular instance ID for license activation..

[GoodMirek]

Yup. The focus here is that user data only runs once. Updating it on an existing instance does not trigger its execution.

This statement is not true on AWS. It is possible to add a tag persist, which ensures that user data is executed on each boot.

Example of user data with the tag persist:

<script>
net start codedeployagent
net start AmazonCloudWatchAgent
</script>
<persist>true</persist>

[iTaybb]

Any update?

[andreacab]

Any update?

[tfarmerpgr]

Any update on this one?

[shayki5]

Any update please? can someone check PR #18043?

[Flowlance]

Same issue here. Any update?

[rnmulchandani]

Faced this same issue. Updation to the user_data was destroying instances and recreating them. In this case, there was a loss of data.
To solve this issue, I added the user_data in the ignore_changes.
https://www.terraform.io/docs/language/meta-arguments/lifecycle.html#ignore_changes

[artkrz]

To solve this issue, I added the user_data in the ignore_changes.

How is that solving it? For You ignoring it is OK but one may want to change user data. Since AWS allows changes without destroying instances so should Terraform.

[dinvlad]

Yea it's good to have as a workaround but not really a solution, unfortunately.

[rnmulchandani]

@artkrz @dinvlad Yeah, I strongly agree. Terraform should support this.
For me, modifications to user_data should not affect resource after its creation. Hence it worked in my case.

[orihomie]

It seems its not working yet, I've just updated my user-data script's template's input variables and tried to update it.
I was expecting AWS instance's ip to stay the same, but it changed, so, I suppose my instance has been deleted and then created again. And overall, I see another bug: after updating my templates input data it still shows me the file generated with previous arguments.

[devopsrick]

Is it a public IP address? Those are released when you stop/start an instance so this would be expected. Only private IPs in a VPC or Elastic IPs are retained over stop/start.

[orihomie]

Doesn't that mean that my instance has been destroyed, does it?

[devopsrick]

Doesn't that mean that my instance has been destroyed, does it?

If it is a public IP address then no it doesn't mean that. If it is a private VPC IP address then it is possible. You can be sure by checking the instance id in the state files and seeing if that changed.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.