[FEATURE] First class support for modules (HCL code generation and orchestration)

[cwe1ss]

The current main focus of Terramate is on stacks. Once a directory contains a stack {}-block, it can be used as a target for code generation, change detection and orchestration:

• You can use generate_hcl to generate any dynamic .tf file - e.g. to dynamically set backends, providers, or versions.
• You can use the terramate CLI to execute other programs in any stack - e.g. terramate run terraform init.

This feature request proposes extending the Terramate functionality to regular modules.

In contrast to a stack, a module is a reusable set of Terraform resources, that is not meant to be planned/applied directly. Instead, it is referenced by one or many stacks with the appropriate variables for the target environment.

While modules are no target for "terraform plan/apply", there are still scenarios where modules would benefit from Terramate functionality:

Generate HCL

One might want to use the same variable definitions for a set of modules. For example, multiple modules might need a context.tf (from Cloud Posse) which allows passing common variables between modules and sub-modules.

With Terramate's "Generate HCL" feature, one could declare this file once and have it generated for each module, making the solution more DRY.

* modules
  * module-a
    * _generated_context.tf
    * ...
  * module-b
    * _generated_context.tf
    * ...
  * context.tm.hcl
* stacks
  * ...

# context.tm.hcl
generate_hcl "_generated_context.tf" {
  content {
    variable "global_prefix" {
      type = string
    }
  }
}

The demonstrated _generated_context.tf content is static, but there are also use-cases for generating dynamic HCL. Each module must declare its required_providers with a source and minimum version. By using the generate_hcl feature of Terramate, one could centralize the generation of these declarations to keep version numbers in one location and to keep the code DRY. This means, that modules could also benefit from using globals with their lazy evaluation.

Generate File (with local file names)

It is already possible to generate regular files in arbitrary directories by setting context = root - e.g.

generate_file "/file.txt" {
    context = root
    content = "something"
}

However, this only supports setting an "absolute" path starting from the repository root ("/some/folder/file.txt"), so it is not possible to place a tm.hcl-file inside a module and generate a file within the same module without specifying the directory (which is error-prone if the directory is renamed)

# This attempted workaround to create a .tf file within the module is NOT possible
generate_file "_generated_context.tf" {
    context = root
    content = <<-EOT
      variable "my_variable" {
        type = string
      }
    EOT
}

Terramate Run

There are multiple scenarios where it would be good to use "terramate run" to invoke another program in all modules - e.g.

• terraform validate to validate the modules
• terraform-docs to create documentation for all modules

Describe the solution you'd like

I don't know the internals of Terramate and I don't have a lot of experience with it yet, so I can only guess what a possible solution might be from a user's perspective:

• Introduce a first class module {}-block that inherits most of the features from stack (HCL code generation, globals, ...).
• Introduce a module-context (analog to the stack-context)
• By having a separate module-block/concept it can be documented accordingly with its differences to stacks

Introducing this into the CLI is probably tricky. I think, the nicer integration would be to introduce "stack"/"module" sub-commands, but this would obviously be a breaking change.

With sub-commands, instead of terramate create we could have terramate stack create and terramate module create and adjust the logic/documentation to its needs.

Other commands like fmt, generate (which already operate on all child directories) would not need stack/module sub-commands.

Another option would be to introduce e.g. a --module option to integrate the logic into existing commands - e.g. terramate create --module modules/my-module. However, this option might not work with other options (e.g. --after, --before) and it would make the documentation more complex, since it would have to describe the differences.

Describe alternatives you've considered

• Treating modules as stacks by including a stack {} block
  • HCL code generation is possible when you declare a stack {} block within your modules. To ensure modules aren't included when running any terramate statements, one can use tags to exclude modules.
  • However, this solution conflicts with the idea of "stacks" and their documentation. (they are not "runnable")
• Using generate_file with context = root
  • It is possible to generate files in arbitrary directories with generate_file and context = root. However,
    • this requires the file path to be absolute, so one can not generate a file in the "current directory"
    • this doesn't support globals or any other HCL features (like dynamic blocks, ...)
    • this doesn't support hierarchical code generation

Additional context
Related Discord discussion

[mariux]

Hi @cwe1ss, as discussed in discord, this use case makes sense for repositories managing local modules that want to benefit from code generation.

Adding a possibility to orchestrate different commands by default or context seems reasonable too.

Thanks for creating the detailed issue here, please give us some time to investigate different ways of making the desired functionality available.

The proposed solutions make sense and will be considered and I will share a draft proposal asap.

[cwe1ss]

The proposed solutions make sense and will be considered and I will share a draft proposal asap.

This definitely is not critical as there are some workarounds so please don't feel any pressure! I appreciate your work on this tool and I really have been liking it more and more over the last few weeks, so thank you very much! ❤️

[g13013]

It would be awesome to be able to do terramate run -C modules -- terraform test :)

maybe also consider providing a metadata to be able to distinguish between modules and stacks when using generate_file and generate_hcl

[mariux]

We are planning new feature to make this possible. It will not be possible as a first class feature but the combination of different features can lead to the desired outcome.

• Terramate Scripts - a new experimental feature allowing to specify scripts in specific sub-trees e.g. a terramate script run module test could be defined in a modules sub-tree and would execute commands only within this subtree even if called on top level. the same way in a stacks/ subtree a terramate script run terraform apply could be made available that will not be defined in modules and thus not include the modules/ subtree in execution.

• As mentioned before tags can be used to limit execution further when using terramate run. An upcoming feature will allow to create tags on any level and they will inherit through the hierarchy to stacks, removing the need to define the same tags for every stack. This will make it easy to tag all stacks with a terraform tag and all modules with a module tag.

This combination will allow to either use terramate scripts for predefined actions or run any command but limit execution to non-modules via terramate run --no-tags module.

[mariux]

It would be awesome to be able to do terramate run -C modules -- terraform test

this is possible as of today. With experimental terramate scripts you can even define a script inside the modules directory to limit it's scope.

[cwe1ss]

I personally made peace with the fact that my reusable modules also are "stacks" and the mentioned combination of scripts & tag inheritance will make it much easier to differentiate between these reusable modules and actual runnable stacks.

Having a separate module-concept (basically as an alias to stack) is probably not worth the additional complexity, so I personally consider this topic solved for my use-cases.

I leave it up to you to decide if this issue should stay open.

[g13013]

It would be awesome to be able to do terramate run -C modules -- terraform test

this is possible as of today. With experimental terramate scripts you can even define a script inside the modules directory to limit it's scope.

If we put the scripts inside a modules directory, would we be able to detect the changes with --changed ? this is real need here.

[mariux]

If we put the scripts inside a modules directory, would we be able to detect the changes with --changed ? this is real need here.

Yes, --changed will be fully supported. As a script only runs, where it is defined it would be an intersection of stacks that are changed and stacks the script is available for that would execute the code defined in the script.

Or in other words, on changes stacks a script is not defined for, the script will not run.