templating meta issue

[mwhooker]

Issues about wanting some kind of templating are opened frequently. This would be replacing variables inside static files with information from packer either during the file provisioner or inside floppy_files or the http_server

Here are some examples:

• [FEATURE] templating for floppy files #3961
• Parse files in http_directory as templates #1673
• Can user variables be used in external files i.e. autounattend.xml #2867
• [feature request] builder/file interpolation of user variables #4451

I don't believe packer should be in the business of providing templating, as it's out of scope of packer in my opinion. However, it does seem to be requested often.

I'd like to discuss here possible solutions, but right now I don't have any.

[WeatherGod]

if packer was written in python, the answer would be to use jinja2, which is an extremely powerful and easy-to-use templating language. Looks like someone tried a go port of it a couple years ago: https://github.com/jmoiron/jigo

[WeatherGod]

Some usecases to consider. Right now, I have a colleague who made a JS program to generate all of his packer configs, but the logic in that JS script is fairly straight-forward, and would be nice to simply keep all of that in one place. He also needs a separate JS script for each target json file, which is a maintenance headache.

I find myself in situations where my json file could benefit from comments so that others can understand it better. Another thing I find myself doing is copying-n-pasting file upload provisioner entries which could easily be solved with a jinja for-loop construct.

A possible problem with jinja2 templating of JSON files is the trailing comma problem with JSON. The templating would have to be smart enough to know whether or not a comma is needed (perhaps it would make sense to switch to YAML?).

[Roxyrob]

Another use case: passing packer variables (like builder_type, ...) to scripts in autounattend (windows) or kickstart/preseed (linux).

E.g.: vsphere-iso builder uses vmware tools to get IP and go next "Waiting for IP" step. Having builder_type variable (or something like provisioning phase PACKER_BUILDER_TYPE env variable) allow to use more generic script switching to step installing vmware tools at build time, avoiding code duplication.

I think having variables at earlier stages allow more flexibility and DRY code.

[Dr-Shadow]

I think that's really strange to not allow templating on cloud-config files where it's already supported on Terraform.
I have no luck injecting SSH private key dynamically because of this (hardcoded value).
EDIT : Nevermind, the documentation was not clear enough for me. We can use templatefile with http_content instead of http_directory which take static files. Example there : https://www.hashicorp.com/blog/using-template-files-with-hashicorp-packer

[nywilken]

EDIT : Nevermind, the documentation was not clear enough for me. We can use templatefile with http_content instead of http_directory which take static files. Example there : https://www.hashicorp.com/blog/using-template-files-with-hashicorp-packer

@Dr-Shadow thanks for the update here. Sounds like you were able to figure things out. But your comment on the documentation not being clear enough is concerning to me. If it is not clear to you, chances are it might not be clear to others.

Please feel free to open a PR with you suggested changes. We rely heavily on our docs and any help to improve them is very much appreciated. Cheers!

[michelzanini]

Hi @mwhooker,

Packer 1.7 supports the function templatefile. This already provides all the template functionality we could possibly require.

What is missing is a way of creating files on the destination using a templated content (the result of the templatefile function).

Terraform has a file provisioner https://www.terraform.io/docs/language/resources/provisioners/file.html that is very similar to Packer. See the way Terraform supports a content field. That filed is an open string, which can be the place to use templatefile.

Packer does not support such field. All we would need is to add that same content field to the Packer file provisioner. That way we could do the same as in Terraform, and it would look like this:

provisioner "file" {
  content = templatefile("files/foo.tpl", {
    my_var_1 = "value1"
    my_var_2 = "value2"
  })
  destination = "/tmp/foo.txt"
}

Just adding that content field that takes a string... is that something you would consider? It does not seem like a massive change to me...

Thanks.

[azr]

Hey @michelzanini, I think that is a good idea ! Thanks for suggesting 🙂 . Would you have the time to implement this feature ?

[michelzanini]

@azr Unfortunately, at the moment I am really busy and won't have the time.
I hope someone here can pick it up? It's possibly something small that can bring a good benefit.
Thanks.

[azr]

Hey @michelzanini, it was quicker than I though, and so I opened a new PR to get a new content field in the file provisioner here #11209

The original issues of this PRs all have been addressed and so given that #11209 is on its way, I think it's okay to close this issue here. Please open a new issue if you have another feature or bug ! 🙂 Thanks everyone !

[michelzanini]

Thank you!

[ekristen]

@azr this is not super clear to me, does this mean that there will be a way to dynamically generate the contents of floppy_files, cd_files, etc now? Looking at the PR and the file provisioner it doesn't look that way, which seems to me to be a part of this issue?

I really need to be able to use templatefile with floppy_files for windows builds, because autounattend.xml has to be passed in via floppy_files.

[michelzanini]

@ekristen , the change is to get the file provisioner a new content field. With a file provisioner you can move any file, and now, that file can be templated.

Other properties in other builders, such as floppy_files would not be contemplated by this change. Is it possible to use a file provisioner in place of floppy_files? If no, then that will also need a floppy_content or something similar in some other PR.

[ekristen]

@michelzanini that is what I thought, to my knowledge the file provisioner is only in the context of the VM like all other provisioners, copying content to the VM.

This issue was for templating in general, the PR fixes or adds support in a very narrow scope. I would suggest this issue gets re-opened until the other areas that are lacking support are addressed.

[michelzanini]

I would recommend that would be better to open another ticket just for floppy_files support... then it's a bit easier for someone to know what to fix. By the way, templating in general is already supported by the templatefile function. Now it's just a case of some places where it requires a file as input, that also a string is available. Like http_directory vs http_content etc...

[azr]

Thanks for opening a new issue ! I just want to clarify that we can use the templatefile function in HCL along with the cd_content option to do templating in cd files. There's also an http_content option that can serve strings as well.
For example, the docs for the cd_content field of vmare iso can be found here: https://www.packer.io/docs/builders/vmware/iso#cd_content this also works with virtualbox and other iso based builders.

Adding a floppy_file_content option would be a good idea too. I'll follow up over in #11210 .

[puetzk]

Just to note, cd_content doens't work in the vmware builder in the current release (it does in other builders, and vmware is fixed on the main branch per hashicorp/packer-plugin-vmware#30). Though there's a few more lingering bugs in hashicorp/packer-plugin-vmware#31.