Directory structure advice

[ottramst]

Hey all!

Thanks for the great project!
I have a small question with which I'm struggling a bit.

I have this directory structure:

.
├── environments
│   ├── common
│   │   └── config.tm.hcl
│   ├── prod
│   │   └── config.tm.hcl
│   └── staging
│       └── config.tm.hcl
├── services
│   ├── backend.tm.hcl
│   ├── config.tm.hcl
│   ├── demo-service
│   │   ├── common
│   │   │   ├── _backend.tf
│   │   │   ├── _k8s_namespace.tf
│   │   │   ├── _providers.tf
│   │   │   ├── _psql_db.tf
│   │   │   ├── _vars.tf
│   │   │   ├── _versions.tf
│   │   │   └── stack.tm.hcl
│   │   ├── config.tm.hcl
│   │   ├── prod
│   │   │   ├── _backend.tf
│   │   │   ├── _k8s_namespace.tf
│   │   │   ├── _providers.tf
│   │   │   ├── _psql_db.tf
│   │   │   ├── _vars.tf
│   │   │   ├── _versions.tf
│   │   │   └── stack.tm.hcl
│   │   └── staging
│   │       ├── _backend.tf
│   │       ├── _k8s_namespace.tf
│   │       ├── _providers.tf
│   │       ├── _psql_db.tf
│   │       ├── _vars.tf
│   │       ├── _versions.tf
│   │       └── stack.tm.hcl
│   ├── modules.tm.hcl
│   ├── providers.tm.hcl
│   ├── vars.tm.hcl
│   └── versions.tm.hcl
└── terramate.tm.hcl

The idea behind this is to have a monorepo that houses the Terraform configuration for microservices. Each service being in services/<SERVICE_NAME> and each environment for the service being in services/<SERVICE_NAME>/<ENV_NAME>.

A set of common globals for a single environment is found in environments/<ENV_NAME>/config.tm.hcl. In those files I have globals that configure the providers to be used in the context of different environments, such as Kubernetes provider configuration that is pointed to different clusters etc.

To have some common set of globals for all environments I utilize the services/<SERVICE_NAME>/config.tm.hcl, with a set of globals that each environment uses. After that I use the services/<SERVICE_NAME>/<ENV_NAME>/stack.tm.hcl to import in the globals from the environment specific directory in environments/<ENV_NAME>/config.tm.hcl like so

import {
  source = "/environments/<ENV_NAME>/config.tm.hcl"
}

The issue I'm seeing is that Terramate then ignores the globals I have set in services/<SERVICE_NAME>/config.tm.hcl.
Any advice on this kind of approach and/or advice on how to achieve "service-level" common globals?

Thanks in advance!

[katcipis]

Hi @ottramst,

Glad to know you are enjoying Terramate !

Importing globals should not affect any other globals defined on your hierarchy if there are no name clashes, the setup you are describing should just work.

For example, if on services/<SERVICE_NAME>/config.tm.hcl you have:

globals {
  a = "a"
}

And on environments/<ENV_NAME>/config.tm.hcl you have:

globals {
  b = "b"
}

On the stack services/<SERVICE_NAME>/<ENV_NAME>/stack.tm.hcl, after importing environments/<ENV_NAME>/config.tm.hcl you should observe the two globals (merge behavior):

a = "a"
b = "b"

What could be happening is that if any globals have the same name, then the imported globals will override/shadow the inherited globals. The order of overriding being stack ->(overrides) imported ->(overrides) inherited, so even the stack itself can redefine and override globals that where imported/inherited.

I hope this helps, if the globals are indeed different and they are just being "lost" then that would be a bug, if you could provide a little more details on the contents of the files maybe we can reproduce the issue (so far the described behavior seems to be working).

[mariux]

Hi @ottramst

On first sight without knowing any details about the contents and information you want to share i would suggest trying if the following would make sense for your desired setup:

Instead of having a list of services that run in various environments, why not care about environments running a bunch of services each. It's the reverse of your setup that i would propose:

├── environments/
│   ├── config.tm.hcl   # config for all environments
│   ├── backend.tm.hcl  # code generation for backend
│   ├── ...             # other code generation for every stack etc.
│   ├── prod/
│   │   ├── config.tm.hcl # prod config overriding the common config
|   |   ├── backing-service/
|   |   |    ├── databases/
│   |   |    └── buckets/
│   │   └── services/
│   │       ├── demo-serviceA/
│   │       │   ├── main.tf
│   │       │   ├── ...
│   │       │   └── stack.tm.hcl
│   │       ├── demo-serviceB/
│   │       │   ├── main.tf
│   │       │   ├── ...
│   │       │   └── stack.tm.hcl
│   │       └── demo-serviceC/
│   │           ├── main.tf
│   │           ├── ...
│   │           └── stack.tm.hcl
│   └── staging/
│       ├── config.tm.hcl # staging config overriding the common config
|       ├── backing-service/
|       |    ├── databases/
│       |    └── buckets/
│       └── services/
│           ├── demo-serviceA/
│           │   ├── main.tf
│           │   ├── ...
│           │   └── stack.tm.hcl
│           ├── demo-serviceB/
│           │   ├── main.tf
│           │   ├── ...
│           │   └── stack.tm.hcl
│           └── demo-serviceC/
│               ├── main.tf
│               ├── ...
│               └── stack.tm.hcl
└── terramate.tm.hcl

This allows for several nice features:

• You can share configuration per environment easily to all services.
• It might match you actual cloud structure where you have a separate account (AWS) or project (Google Cloud) per environment.
• You can just run staging changes (plan/apply) easily with terramate -C environments/staging run -- terraform plan
  • You could run CI/CD with different credentials based on environments
• If using GitHub you can set up different code owners on /environments/staging/* and /environments/prod/*

I hope this helps a bit. Please let us know if you need follow up details ;)

[ottramst]

Hey guys!

First off, thanks for the suggestions!

I'll give a bit more background on this. So the first thing - why we implemented a "service-based" set of stacks instead of "env-based"?
We wanted to have shared configuration across each of the service environments - for example global.service_name is a good candidate for such a requirement. This way the name doesn't have to be defined twice and thus lowering the level of human error.

Next - there's always a static set of configuration for each environment, which doesn't change regardless of the service. For example - Kubernetes provider context is always staging-cluster for all services that have a staging environment configured and so on.

I understand that there are tradeoffs and gains for both approaches, but in terms of allowing the end user add more services to this structure, the "service-based" approach seems a bit less error-prone and easier. I will also mention that I understand that Terramate is not all-mighty and my desires to implement the structure this way may be a bit utopian.

---

But, let me try to explain what I'm currently observing.

Dir/files structure:

.
├── ./environments
│   ├── ./environments/dev.tm.hcl
│   └── ./environments/staging.tm.hcl
├── ./services
│   ├── ./services/config.tm.hcl
│   ├── ./services/foobar
│   │   ├── ./services/foobar/config.tm.hcl
│   │   ├── ./services/foobar/dev
│   │   │   └── ./services/foobar/dev/stack.tm.hcl
│   │   └── ./services/foobar/staging
│   │       └── ./services/foobar/staging/stack.tm.hcl
└── ./terramate.tm.hcl

• Contents of services/config.tm.hcl

globals {
  service_name  = ""
  environment   = ""
}

• Contents of environments/staging.tm.hcl

globals {
  environment = "staging"
}

• Contents of environments/dev.tm.hcl

globals {
  environment = "dev"
}

• Contents of services/foobar/config.tm.hcl

globals {
  service_name = "foobar"
}

• Contents of services/foobar/staging/stack.tm.hcl

stack {
  name = "foobar staging"
}

import {
  source = "../../../environments/staging.tm.hcl"
}

• Contents of services/foobar/dev/stack.tm.hcl

stack {
  name = "foobar dev"
}

import {
  source = "../../../environments/dev.tm.hcl"
}

However running terramate generate on this structure ends up with the next error:

Fatal failure preparing for code generation.
Error details: <generated-code>:2,3-14: terramate schema error: listing stacks: failed to parse directory /.../terramate/environments: attribute "environment" redeclared in file "/.../terramate/environments/staging.tm.hcl" (first defined in "/.../terramate/environments/dev.tm.hcl")

[katcipis]

Hi @ottramst,

It seems that on terramate/environments you have two files:

• dev.tm.hcl
• staging.tm.hcl

That define the same global attribute "environment". Terramate implements a similar semantic as Terraform where all files are "merged" and then handled as a single configuration (more details here). Just as would happen with Terraform local block, if you redefine a global you get an error since there is no clear semantic to how to handle this, there is no explicit ordering on HCL/Terramate files, so there is no way for one name to shadow/override the other.

I realize this is somewhat confusing/non-intuitive because you imported a single file, not a directory, but Terramate by default parses all directories in a project that have any .tm/*.tm.hcl files, that is why the error happens even before doing any importing, when Terramate is just trying to list all available stacks.

If the objective of environments is to only be imported and never handled directly by Terramate one solution to allow this would be to skip the environments directory entirely. Terramate will not try to parse it anymore unless you import specific files in that case. This feature will be release later today, you can check it here.

Hope this was helpful, let me know if I got something wrong.

[ottramst]

Hey!

All is clear and understood. Thank you for all the insights and help!
I think I have enough to continue for now

[katcipis]

Glad to help 👍

[katcipis]

@ottramst the release v0.1.38 includes support to skip directories. You can check details here.