Provide an option or alternative for --set and/or --set-string to take a value literally

[maxhov]

I am experiencing an escaping problem with Helm. For instance, if I want to set a password in a chart, I run into the problem that \ characters are removed.

Example:

1. Take a chart that sets a k8s secret where the value of the secret is base64 encoded by Helm (b64enc)
2. Run helm upgrade --install --set password=Passw\Ord! somechart
3. The resulting base64 encoded string decodes to PasswOrd! instead of Passw\Ord!

I looked into possible solutions on my end and did the following:

1. Singly escaping the \ resulting in Passw\\0rd! does not give the desired output, both \ are removed, resulting in PasswOrd!.
2. Wrapping with either ticks (') or quotes (") results \ being removed, resulting in PasswOrd!.

What did work though is escaping the backslash two times Passw\\\Ord!, which results in Passw\Ord!.

Expected behaviour

I would expect that \\ would result in \ instead of both being removed.

Desired outcome

It should be possible for passwords to contain special characters, such as \.

Possible solution:

Wrapped variable values in the --set argument should be taken literally (e.g. --set password='Passw\Ord!' should result in Passw\Ord!.

Output of helm version:
2.8.2

Output of kubectl version:
Client: 1.10.1
Server: 1.9.6

Edit: updated character naming to be more consistent on what is meant

[mattfarina]

It appears this is the expected escape behavior. For the code location see https://github.com/kubernetes/helm/blob/b6660cd5c9a9ee35ff24034fcc223b7eaeed3ee2/pkg/strvals/parser.go#L309

For Helm 2 we should document this as changing it would cause a backward incompatibility.

[bacongobbler]

labelling this as a good first issue if someone wants to do a deep dive around the --set parser and document their findings.

[mattfarina]

@technosophos can you provide a little history on why \\ is the escape character rather than \? Git blames you for this so I thought I'd ask.

[technosophos]

The escape character for the --set parser is \.

Char '\\' cited in the code above is the \ character, escaped per Go's escaping rules.

Keep in mind that you have the shell interpreting things here, too. So often you have to double escape things because the shell will interpret \ as an escape sequence. Thus, you might need to do \\ to escape the \ character for the shell. And in some cases you will need to \\\\ to escape things. Wrapping the value in single quotes (--set name='value') will also have an impact in most shells. But the bottom line is that you need to consider all the different things that are interpreting the set value between when you execute helm and when the final value is injected into Helm's values.

[muratg]

I was playing with stable/registry chart today, and hit this issue. I was passing htpasswd file contents to --set. $ character was the culprit.

Workaround was just escaping it, but finding the root cause took a little while!

[smurfralf]

I will work on this as my first contribution to documentation.

[bacongobbler]

closed via #4485

[muratg]

@smurfralf I believe you'll need to escape $ character in your example as well as \.

cc @bacongobbler

[muratg]

Also, I'm not sure if I agree with the resolution. It's great that the documentation is updated (thanks @smurfralf!) but shouldn't the real fix be just taking quoted values literally? Even with the docs, this will continue to confuse folks.

[bacongobbler]

fair enough.

[maxhov]

I would be very much in favor of actually taking quoted values for —set (variants) literally.
Especially in CI/CD setups escaping (secret) values can be extra challenging, as one might be unaware of the actual value being processed. To overcome this, quite some extra code/parsing of the values would be required in the pipeline before the value can be offered to Helm.

[muratg]

I agree with @0x6D6178.

edit: If backwards compatibility can't be broken, perhaps you can add a flag to --set (like --literal) to at least enable this behavior? Otherwise everyone will need to implement the workarounds that @0x6D6178 mentions above.

[moikot]

I'm not entirely sure that the workaround proposed by @0x6D6178 is going to cover 100% of the cases. How about a password including comma? Do I understand correctly that a comma needs to be escaped using backslash?

[smurfralf]

Its not clear to me in the comments above what "quoted values" and "literally" mean. Since this is a CLI we have to contend with the shell no matter what. So chars like dollar sign ($), quote (") and tick(') are going to have to be escaped in some way by the user regardless. The use of ticks to quote for the shell also serves to protect against comma (,) and equals (=) delimiter parsing. So isn't the only thing being debated is whether the yaml interpretation of backslashes should be continued? I would argue yes, in case someone wants to use non-printable characters in the value.

EDIT: checking on @moikot feedback, looks like my typo in the example (leaving out the comma) was more than just a typo - it does affect behavior. To get the desired value you have to backslash the comma as the comment above mentions. The tick marks don't stop the splitting of the values on the comma. Back to the drawing board...

[moikot]

@smurfralf You absolutely right, the proposed solution (e.g. --set password="Passw\Ord!" should result in Passw\Ord!) is not gonna work since it's not up to Helm to decide how to treat '\O' in double quotes.

But I'm afraid there is another aspect as well, even if you use a value provided in a variable, you are not safe since a comma (',') and an opening curly brace ('{') has a special meaning in --set and --set-string and you have to escape them. For example:

export QQQ="bar,baz"
helm lint --set foo=${QQQ}

Gives you an error

Error: failed parsing --set data: key "baz" has no value

and in case of '{'

export QQQ="{bar"
helm lint --set foo=${QQQ}

Gives you an error

Error: failed parsing --set data: list must terminate with '}'

There is another issue reported about the escaping and I think it's more correct #4406.

[maxhov]

Bash does not interpret strings that are wrapped with ticks ('), whereas it does for some characters when the string is wrapped with quotes (") (see https://stackoverflow.com/questions/6697753/difference-between-single-and-double-quotes-in-bash#6697781). I would expect that a string is not interpreted by Helm when I supply a tick wrapped string in the --set option, just like the behaviour of Bash.

Currently the behaviour is:

helm lint --set foo='{bar'

Currently results in:

Error: failed parsing --set data: list must terminate with '}'

Whereas I would expect the value to be taken literally.

However, I understand the additional functionality that --set provides (e.g. allowing lists, multiple values etc.), so maybe it would be an idea to introduce a new CLI option (as #4406 also suggests), for instance something like --set-literal that does not interpret any of the characters that are supplied by the user. Would that be feasible?

[anisimovsergey]

@0x6D6178 Command line processors, in general, remove all the special characters (double quotes (") or single quote (')) before passing the resulting value to a program, so Helm doesn't get a chance to distinguish between them.

[brianharwell]

Any update on adding --set-literal or something similar?

[pscheid92]

Hello all together 👋 I would like to give it a try.
I just opened a draft PR containing my experiments (for visibility; no working implementation, yet).

I added a --set-literal argument and duplicated the existing parser code; stripped away some functions/methods and added a small test case. I first have to familiarize myself with the existing parser code.

Currently, escaping is coupled heavily with the parser in parser.go, therefore a duplicated parser in literal_parser.go.
Maybe we can merge both eventually, we will see.

[github-actions]

This issue has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.

[dennisotugo]

This is still an issue. Really nothing to add but hopeful for the PR merge

[PSanetra]

@bacongobbler in the comment #4030 (comment) you did include this list as the set of characters which should be escaped when passing a string to set-string:

[...]

=
{
[
,
.
]
}

[...]

In this list the character \ is missing, which should be escaped, too. Can you edit your comment and also include \?

[brianonn]

I ran into this issue today trying to set a literal json string to use it in a template (to set an env var). We get the json string from our TUF repository root key and I just want to set it as a json literal string in an environment variable in the k8s container spec.
--set or --set-string both try to parse the string. I can really use a --set-literal

[bradacina]

+1 for the pull request

[bmitchinson]

+1 for the pull request please

[Aitozi]

+1 for the this feature :)

[gm42]

https://helm.sh/docs/intro/using_helm/#the-format-and-limitations-of---set

Since this page still does not list the characters that must be escaped, nor does the CLI, shall I make a PR to address the issue?

[dennisotugo]

+1

[Amirilw]

+1

[manoj-bandara]

+1 for a feature like --set-literal

this issue of not taking literals in --set is causing many problems for us

[bkalcho]

+1 for --set-literal

This will be really useful. When you need to pass stringified JSON through GitHub actions to helm it is hell, as you need to escape the GH YAML templating engine, bash, and at the end Helm interpretation.

[natalicot]

+1 for a feature like --set-literal

[amkartashov]

I expect --set-string to actually do its job. Why on Earth there is a need for escaping? It's a string, there is no need to parse it in a first place.