How to create reusable base profile with injected secrets?

[Toizi]

Hey, I'm currently trying to switch to resticprofile from a custom systemd setup with the goal of making it reusable across machines. I therefore want to put the base configuration into version control and customize it per machine with a file that contains secrets and potentially overrides some of the base.

The way I tried this was to create a profiles.toml similar to this:

version = "1"
includes = ["config.toml"]

[shared]
  [shared.backup]
    source = "/"
    password-command = "echo {{ $PASSWORD }}"
    run-after-fail = "echo failed"

[rest]
  inherit = "shared"
  repository = "{{ $REST_URL }}"

[local]
  inherit = "shared"
  repository = "local:/mnt/backup-{{ .Profile.Name }}/restic"
  run-before = [
    "echo mounting /mnt/backup-{{ .Profile.Name }}",
    "mkdir -p /mnt/backup-{{ .Profile.Name }}",
    "mount /dev/disk/by-uuid/{{ $LOCAL_BACKUP_UUID }} /mnt/backup-{{ .Profile.Name }}"
  ]
  run-finally = [
    "echo unmounting /mnt/backup-{{ .Profile.Name }}",
    "umount /mnt/backup-{{ .Profile.Name }}"
  ]

And a corresponding config.toml that defines the variables:

version = "1"

{{ $PASSWORD := "1234" }}
{{ $REST_URL := "rest:some_url_with_password" }}
{{ $LOCAL_BACKUP_UUID := "1234" }}

[shared.backup]
  run-after-fail = ""

But this doesn't seem to work at all. Is there a nice way to implement this?
So far, the only workaround I found was cating together some defaults.toml, template.toml and config.toml files with custom variables for each of the sections I want to customize. However, even that has some limitations that I'd rather not live with.

[creativeprojects]

Hi!

Thanks for your message 😄

This is very much of the chicken and egg problem. Either we resolve the templates before the resulting TOML, or we try to parse the TOML before resolving the templates.

In that case, we do parse the template before the TOML (which contain the include statement) and therefore the template cannot find the variable declaration (yet). This is a less than ideal situation, and this is why we decided introducing mixins (https://creativeprojects.github.io/resticprofile/configuration/inheritance/index.html#mixins)

In the meantime, you can use environment variables as such:

profiles.toml

version = "1"
includes = ["config.toml"]

[shared]
  [shared.backup]
    source = "/"
    password-command = "echo {{ .Env.PASSWORD }}"
    run-after-fail = "echo failed"

[rest]
  inherit = "shared"
  repository = "{{ .Env.REST_URL }}"

[local]
  inherit = "shared"
  repository = "local:/mnt/backup-{{ .Profile.Name }}/restic"
  run-before = [
    "echo mounting /mnt/backup-{{ .Profile.Name }}",
    "mkdir -p /mnt/backup-{{ .Profile.Name }}",
    "mount /dev/disk/by-uuid/{{ .Env.LOCAL_BACKUP_UUID }} /mnt/backup-{{ .Profile.Name }}"
  ]
  run-finally = [
    "echo unmounting /mnt/backup-{{ .Profile.Name }}",
    "umount /mnt/backup-{{ .Profile.Name }}"
  ]

At that stage the template doesn't care if the environment variable is empty, the syntax is valid. These variables will be populated when merging the configuration after the include.

config.toml

version = "1"

[shared.env]
  PASSWORD = "1234"
  REST_URL = "rest:some_url_with_password"
  LOCAL_BACKUP_UUID = "1234"

Alternatively you can try to use the mixins 😄

[Toizi]

Hey @creativeprojects, thanks for the response. I tried your suggestion with the environment variables but it doesn't seem to work.
The output of resticprofile --name local show is the following:

global:
    ionice-class:             2
    default-command:          snapshots
    restic-arguments-filter:  true
    restic-lock-retry-after:  1m0s
    restic-stale-lock-age:    1h0m0s
    min-memory:               100
    send-timeout:             30s

profile local:
    run-before:   echo mounting /mnt/backup-local
                  mkdir -p /mnt/backup-local
                  mount /dev/disk/by-uuid/<no value> /mnt/backup-local
    run-finally:  echo unmounting /mnt/backup-local
                  umount /mnt/backup-local
    repository:   local:/mnt/backup-local/restic

    env:
        LOCAL_BACKUP_UUID:  1234
        PASSWORD:           ×××
        REST_URL:           rest:some_url_with_password

    backup:
        password-command:  echo <no value>
        run-after-fail:
        source:            /

So the env vars are populated but not inside the actual configuration. Am I doing something wrong here?

Regarding the mixins, do you personally believe they will be included in the version 2 format close to their current state or are they still very experimental?
Of course I'm not taking that as a guarantee, but I'd like to gauge the likelihood of having to rewrite everything.

[creativeprojects]

So the env vars are populated but not inside the actual configuration. Am I doing something wrong here?

No sorry, it looks like I was wrong 🤔

Regarding the mixins, do you personally believe they will be included in the version 2 format close to their current state or are they still very experimental?
Of course I'm not taking that as a guarantee, but I'd like to gauge the likelihood of having to rewrite everything.

I think at this stage, these are very unlikely to change much. Many people are using them already 👍🏻

[creativeprojects]

I found a trick: as opposed to local variables, go templates are all resolved together.
This might look a bit convoluted, but it actually works defining a template instead of a variable:

To make it work, we have to declare a template before using it. The main profiles.toml file will be the generated one, while config.toml is the generic one.

profile.toml

version = "1"
includes = ["config.toml"]

{{define "PASSWORD"}}1234{{end}}
{{define "REST_URL"}}rest:some_url_with_password{{end}}
{{define "LOCAL_BACKUP_UUID"}}1234{{end}}

config.toml

version = "1"

[shared]
  [shared.backup]
    source = "/"
    password-command = "echo {{template "PASSWORD"}}"
    run-after-fail = "echo failed"

[rest]
  inherit = "shared"
  repository = "{{template "REST_URL"}}"

[local]
  inherit = "shared"
  repository = "local:/mnt/backup-{{ .Profile.Name }}/restic"
  run-before = [
    "echo mounting /mnt/backup-{{ .Profile.Name }}",
    "mkdir -p /mnt/backup-{{ .Profile.Name }}",
    "mount /dev/disk/by-uuid/{{template "LOCAL_BACKUP_UUID"}} /mnt/backup-{{ .Profile.Name }}"
  ]
  run-finally = [
    "echo unmounting /mnt/backup-{{ .Profile.Name }}",
    "umount /mnt/backup-{{ .Profile.Name }}"
  ]

[jkellerer]

@Toizi, just for your info, restic and custom command hooks interpret environment variables, therefore:

[local.env]
  RESTIC_REPOSITORY = "local:some-repo"
  RESTIC_PASSWORD = "1234"
  LOCAL_BACKUP_UUID = "1234"

[local]
  run-before = [
    "echo mount /dev/disk/by-uuid/${LOCAL_BACKUP_UUID} /mnt/backup-{{ .Profile.Name }}"
  ]

does echo (or run if echo is removed):

mount /dev/disk/by-uuid/1234 /mnt/backup-local

See also: https://restic.readthedocs.io/en/latest/040_backup.html#environment-variables

[Toizi]

Thank you both @creativeprojects and @jkellerer.

Inspired by your suggestions I improved the cating together some files setup I already had.
It's not super elegant but it seems to work well so far without being too verbose and decently supports all of my use cases.
In case you're interested, the setup can be found at https://github.com/Toizi/config/tree/master/resticprofile_shared and is supposed to be used with the wrapper script https://github.com/Toizi/config/blob/master/.my_scripts/resticp.

I'll mark the last suggestion by @jkellerer as the answer since that's really the simplest approach if injecting secrets is the main goal.

[creativeprojects]

I see. It's quite annoying you have to concatenate the configuration files manually before running resticprofile though 😐