dotenv support for variables and parameter expansion is unclear

[polarathene]

Expected Behavior

Either:

• Variables support parameter expansion.
• Docs clarify limitations / support with using variables (which has plenty of other gotchas).

Current Behavior

dotenv does not detail support of .env file format, it appears that the following applies:

• Values can be wrapped in single quotes which prevents evaluating any variable.
• Double-quote wrapped values is the same as no quote wrapping.
• A variable can be $VAR or ${VAR}, but will only support matching another ENV key defined earlier within that same .env file.
• Not all ENV keys defined earlier are valid matches. There is a constrained pattern (A-Z + _) so an ENV key that uses a-z won't match, instead the characters outside the valid key pattern are handled as plain-text when used in a variable in a value, this includes special chars like : / - / = (used with parameter expansion).
  • Despite an defining an earlier xyz=42 key, it is not a valid key pattern to reference, which results in (${xyz} or $xyz) => (${xyz} or $xyz) output as plain-text instead of resolved. But teller env will happily output xyz=42.
  • $VAR can have additional characters directly after it that are not considered part of the variable name. So if there is an existing VAR=hello and you define TEST=$VARworld this will resolve to TEST=helloworld.
  • If $VARworld cannot find a VAR key, it is resolved as empty, resulting in a value of world.
  • It is almost the same behaviour with ${VAR}, except when non-key content exists within the braces. TEST=${VARwo}rld (with valid VAR=hello key) would resolve into TEST=hellowo}rld. Not what you'd expect normally.
  • For ${VAR}, parameter expansion is not supported. As shown above, anything not matching the valid key pattern will be treated as plain-text. ${VAR:=hello}world (with no VAR key set) resolves to :hello}world, and with it set resolves to hello:=hello}world.
  • ${VAR} syntax is buggy if not following the internal variable naming convention/pattern for keys (it should not attempt any nesting/expansion):
    • ${VAR can be resolved as $VAR and any remaining content up to and including the closing } is output as plain-text.
    • Whereas for (_FOO=FOO, BAR=BAR, FOOBAR=baz):
      • abc${FOO${BAR}}xyz => abcFOOBAR}xyz (removed ${ for both but left the outer }).
      • And abc$FOO${BAR}xyz => abcFOOBARxyz (correct).
      • abc${$FOO$BAR}xyz => abc${FOOBARxyz (removing the } but leaving the ${).
• Using carry_env: true has no effect on the processing of an .env file (with a variable value that references an ENV carry_env would bring in). Only relevant to teller run, not providers? (although ProcessEnv provider can achieve similar with env_sync unaffected by carry_env: false).

Possible Solution

Perhaps the above notes about using variables in values is not intentionally supported by Teller?

There's a few caveats there. The gist of it is:

• $VARxyz is treated as $VAR for evaluation, which can be made clearer in intent with braces for scope ${VAR}xyz.
• Variable names must use a restricted charset (Uppercase + _?), you cannot have a xyz=hello key and use TEST=$xyz.
• When there is no valid key match (key must be defined earlier in the file), the variable evaluates as empty / ''.
• Single-quote wrapping of a value will prevent evaluating any variables it contains.
• Parameter expansion (broken) and carry_env: true (no effect) are not supported for variables in values of the .env file to leverage.

The DotEnv provider docs could probably touch on these rules, or communicate that variables should not be used, or not officially supported.

Steps to Reproduce

1. Use the dotenv provider with a value containing a variable.
2. When the variable meets a valid key rule it'll work. But as detailed above, there's plenty of gotchas.
3. Variables attempting to use parameter expansion fail, outputting plain-text with trailing }, while resolving ${VAR to a possible key value or empty.

Context

I wanted to try set defaults for some ENV as fallback via parameter expansion, or carry_env (also tried with separate processenv provider, but teller env outputs duplicate ENV, while teller yaml / teller json are racey and alternate on provider source).

Specifications

• Version: 1.5.6
• Platform: Linux x86_64

[polarathene]

The variables support is probably not intentional, given that teller is focused on managing secrets.

---

Feedback

Not sure if this is the right place for this. The variables issue raised above is related in the context of how I was trying to leverage teller for projects where secrets was not the primary use-case. The feedback below might be helpful to anyone else considering similar (or interested maintainers, if they find more time for the project).

I had an interest for using Teller beyond just secrets within a project to source via providers: dotenv + processenv + filesystem, along with the useful teller env (sh, yaml, json) outputs feature. But it seems teller is not tailored as well for that as my initial impression thought 😅

• remap is great, but not equivalent to an alias. I can't have an ENV that could be defined as that ENV, but alternatively source from a fallback value (supporting ENV with releases for backwards-compatibility / deprecation phase).
• teller env / teller sh would output duplicate duplicate keys common between them. While all output types were inconsistent in the provider ordering. Defining source + sink between them did not help establish a preferred source. This was an issue notably when one provider such as processenv had an empty value from no ENV present, while another provider had the same key defined but the output still chose to be racey, resulting in an empty value for the key.
• No way to set a default (except for opts, but those can only be used in path values AFAIK, not actual defaults to fallback to for a key).
• env / env_sync cannot easily leverage remap to remove/add a prefix (aka namespace). Would be useful in processenv at least, possibly a valid use-case for path in env_sync as a filtering out a subset of the environment which would be valid with opts. I needed retrieve ENV grouped under several prefix for example, the child process however didn't need the extra context of the prefix.
• The severity default cannot easily be inverted to none when the environment contains only a few secrets. I was interested in the redaction feature, especially for bug reports where logs and environment vars could be shared with teller env (but an explicit severity set for secrets). While probably unlikely, I noticed the redaction feature would redact ENV keys when paired with output of teller env if the key (or any subslice) represented a secret value... which risks giving away information it intended to hide 😅 (arguably weak secrets were chosen in that scenario, such as password or admin)

Beyond that, it doesn't look like maintenance / releases is too active, let alone further development from SpectralOps? I'm guessing it does the job well enough for most and there is limited resources available for teller beyond the kind received over the past year?

• This sort of issue isn't too reassuring. Breakages like that happen, but the lack of engagement for over a month is not encouraging given the pattern with older issues.
• The project has CI setup (albeit a bit outdated at present):
  • Follows semver, but has not been tagging new releases to go with the PRs that do get merged (approaching one year, with feature and fix 4 months ago sitting in master).
  • There doesn't appear to be any interest improving the CI workflow to make teller more easily available, just GH Releases (via archives) and brew?

[jondot]

Thanks for the feedback, we've used much of it to reshape how variable mapping works.
Today with Teller 2.0, we are being more explicit about mapping and kv-routing.
Regarding dotenv interpolated variables in Teller 2.0, its not part of a library that we can use, we'd be happy to see this as a PR from someone who's willing to implement.
Closing this for now but happy to see new feedback on Teller 2.0