Some basic questions and observations

[az-z]

Gentlemen,
a few initial questions as I'm going thru the tutorials and documentation. I appreciate your help and clarifications:

1. generate_file location:
   how do I specify the location of the file to be generated? The doc mentions "label", but I'm not sure how to use the labels block to configure the location.
2. terramate create stack vs generate:
   it was not obviously clear to me what the difference is. Especially with "create stack". Maybe some clarification in the documentation?
3. generate verbosity:
   doesn't seem to make any difference in the output produced:

terramate generate --verbose=10
Code generation report

Successes:

- /app/stacks/stack_1
	[+] hello_world.json

Hint: '+', '~' and '-' means the file was created, changed and deleted, respectively.

4. dryrun:
   the ability to validate the tm's configuration and "dry-run" generation/execution is a valuable flow. Is there a way to do this?
5. What is the practice for referring stack-level attributes? e.g.

stack {
  name        = "My stack"
  description = "My stack description"
}

generate_hcl "backend.tf" {
  content {
    backend "remote" {
      organization = global.org
          workspaces {
              name = "${global.account} _ ${stack.name}"
          }
      }
....
  }
}

Thank you for your time.

[mariux]

Hi @az-z,

Let me try to answer all your questions here.. ;)

1. generate_file location

Label is a term in HCL that you can add to a block. In the case of generate_file the label defines the name of the file to generate within the stack. To generate file_name.txt you can use the following syntax.

generate_file "file_name.txt" {
  content = "Hello from Terramate!"
}

2. terramate create vs terramate generate

• terramate create path/to/stack creates a new stack. It creates the directory and a stack.tm.hcl file. It also takes care of setting the stack ID to a new unique UUID.

• terramate generate runs Terramates code generation in all available stacks. So it takes care that all generate_hcl and generate_file blocks are executed.

3. generate verbosity

• We are still refactoring outputs and verbosity will change the output in future versions. At the moment it is not widely used and just existing in preparation of the refactoring and can be ignored for now. Verbosity will be available for multiple commands in the future.

4. dry run

• DRY run is on our roadmap already. Could you share what exactly would be the set of information you need and expect in dry run, so we better understand your use case.

5. What is the practice for referring stack-level attributes

• Stack levelmeta data can be accessed in terramate as terramate.stack.{something} e.g. terramate.stack.id the name as terramate.stack.name. In addition patch information and tag information is also available. A list of available meta data and values can be dumped using terramate experimental metadata command.

Please elaborate on any further questions or any details that are needed. I hope this answers your questions.

[az-z]

@mariux ,
thank you for the answers.

Please allow me to followup:

1. generate_file location - is there a way to specify the location of the file with a variable?
   e.g.

global.tm:

loc = "./policy"

stack_1/config.tm:

generate_file "${loc}/hello.json"{}

with the result :

app/stack_1/policy/hello.json

4. dry run - the use case - I see it as a huge time-saving to preview potentially unintended changes to the generated hcl.
   examples are :
   a. review if all files will be generated in the proper locations
   b. review if all variables and values have been resolved properly
   as i'm thinking about this, the combination of "verbosity" and "dry run" can be interesting.

I hope you do not mind me posting additional questions in this thread as I'm discovering the power of TM.

[i4ki]

Hi @az-z

generate_file location - is there a way to specify the location of the file with a variable?

No, at the moment you can't use variables in the block labels. This is a limitation of HCL, which treats the labels as plain strings. We already discussed making this work but we will need to deviate from HCL spec.

dry run

A dry run feature for terramate generate is indeed very desirable and is on our TODO list.

[az-z]

How common is it to repeat the same configuration across a number of clients/environmnets?

e.g. this is the prototype of my current environment:

├── terramate.tm
├── globals.tm
├── Clients
│   └── Client_1
│       ├── config.tm
│       ├── dev
│       │   ├── config.tm
│       │   └── stack_1
│       │       ├── backend.tf
│       │       └── config.tm
│       └── prod
│           ├── config.tm
│           └── stack_1
│               ├── backend.tf
│               └── config.tm

the immediate observation is that a new client ( /Clients/Client_2) will have the same (sub) set of envs that will need to be configured in the same way as for Client_1.
To solve the "cloning", I currently have a script that performs the setup by pulling from a template configuration. The limitation of this solution is that I have to backport changes into the template config after new requirements changed a setup for client_X.

[mariux]

cloning of stacks is currently only supported on a single stack. future implementations will support cloning a full sub-tree and also copying all the contained files.

This will support easy promotion between environments and in you case also between clients.

In addition we are working on interfaces to support easy comparison between sub-trees to detect any missed promotions or configuration changes.

We will create a more public roadmap soon so we can also allow voting on specific topics as we see the number of requests growing at the moment.

Feel free to create a feature request to support cloning of sub-trees so it is easier for you to keep track of the progress..

Current cloning is experimental and support a single stack: terramate experimental clone..

[az-z]

thank you for your offer to file the feature request. I have been debating the usefulness of such functionality in multi-account, multi-environment configurations. To be frank, I'm much concerned about the risks that such a feature introduces vs potential benefits.

A more interesting and beneficial functionality may be the automation of stack move - to have it automatically update all "after" and "before"s will be a big win. I'd be happy to put in a feature request for you unless you already have it on the map. (filed)

[az-z]

@mariux :
documentation:

here https://github.com/mineiros-io/terramate/blob/main/docs/config-overview.md#stack-block-schema :
tags list(string)
here : https://github.com/mineiros-io/terramate/blob/main/docs/stack.md#stacktags-setstringoptional
tags are set.

there is nothing in the documentation (?) suggesting that the following should not work:

stack {
  name        = "SMSMS"
  description = "My stack description"
  tags = [ "${global.env}" , "${terramate.stack.name}" ]
}

global.env is defined in the parent directory. Neither definitions make "generate" happy:

2023-03-28T00:09:40-04:00 ERR terramate schema error: loading from...
SMSMS-52/Clients/NYDSS/dev/smsms: eval expression: failed to evaluate "tags" attribute: There is no variable named "global". file=....SMS-IAC/SMSMS-52/Clients/NYDSS/dev/smsms/config.tm:4,15-21
2023-03-28T00:09:40-04:00 ERR terramate schema error: loading from.......
SMSMS-52/Clients/NYDSS/dev/smsms: eval expression: failed to evaluate "tags" attribute: There is no variable named "terramate". file=...SMS-IAC/SMSMS-52/Clients/NYDSS/dev/smsms/config.tm:4,33-38
2023-03-28T00:09:40-04:00 FTL looking up project root

The use case is: I want to deploy "stack" into "env". Hardcoding either doesn't seem like a good idea.

[mariux]

Stack tags are strings and can not be populated by globals or metadata as they are part of the meta data and also can be referenced in globals.

This will create cyclic dependencies and thus is not yet supported.

[az-z]

understand. As i'm thinking this use case over, it may not be such a great idea to deploy based on an artificial tag after all. A more stable approach may be to deploy based on the directory structure, as you have done already:
tm -C Clients/NYDSS/ run -- terraform plan

[az-z]

@mariux ,
if you don't mind explaining, why the "run" command looks so cumbersome:
tm run --[space] terraform plan

• the [space] between -- and 'terraform' is a pain point.
• what other none terraform commands can be executed?

[mariux]

The -- is common separator between one command and another to stop parsing options as tools executed might share same options as terramate.

terramate run terraform plan works perfectly fine. While terramate run -- terraform --help makes clear you want to execute terraform --help (in each stack) and not terramate --help.

At the moment we are still using a very strict parser for the command line, which also causes a lot of issues. As we will change it soon, the recommended way is doing it like any other linux tool by providing the -- to explicitly tell the parser that all following arguments are not considered options to the initial command..

You can also use -- in most commands to tell a command that the following should not be interpreted as a option e.g. cat -- --help will cat a file called --help and not print the help page of cat.

with terramate run you can execute any command - not only terraform.

e.g. you could also have a scenario where you do terramate run --tags k8s -- kubectl diff to run stacks that are not terraform but kubernetes. Or just execute terramate run ls to list all files in all stacks.

[az-z]

I completely missed the idea of allowing to execute any os command in the "run". My bad. This is clever.