Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support mounting yaml and json files as is? #120

Open
ljani opened this issue Sep 29, 2021 · 8 comments · May be fixed by #520
Open

Support mounting yaml and json files as is? #120

ljani opened this issue Sep 29, 2021 · 8 comments · May be fixed by #520

Comments

@ljani
Copy link

ljani commented Sep 29, 2021

My use case: I'd like to pass a sops encrypted .yaml configuration file to my app after decryption as is, instead of picking a single key from the file. So, sops-nix is a little too clever for my use case and it should focus only on decrypting the file, not parsing it.

In other words, I use this file ./secrets/my-config.yaml in my flake:

my-secret1: ENC[AES256_GCM,data:tkyQPQODC3g=,iv:yHliT2FJ74EtnLIeeQtGbOoqVZnF0q5HiXYMJxYx6HE=,tag:EW5LV4kG4lcENaN2HIFiow==,type:str]
my-secret2: ENC[AES256_GCM,data:tkyQPQODC3g=,iv:yHliT2FJ74EtnLIeeQtGbOoqVZnF0q5HiXYMJxYx6HE=,tag:EW5LV4kG4lcENaN2HIFiow==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
...

It should be mounted as /run/secrets/my-config.yaml:

my-secret1: hello
my-secret2: hello

Then I could pass the /run/secrets/my-config.yaml to my app to read any secrets from.

I could do this using a format=binary file, but then I have to remember to use format=binary everywhere, including when invoking sops. Alternatively, I could use a different file extension, but then I'd lose syntax highlight. Yet another alternative would be to use a literal multi line yaml string, but then again I'd lose highlight.

I wonder if this is already supported, because the README.md mentions home-assistant-secrets.yaml, which is similar to what I'm looking for? But I think that was defined as something like this:

home-assistant-secrets.yaml: |
    my-secret1: hello
    my-secret2: hello

(Notice how the renderer fails to highlight the yaml keys my-secret1 and my-secret2).

Does this make any sense to you?

@Mic92
Copy link
Owner

Mic92 commented Sep 29, 2021

Currently I don't see any other way but using either the multi-line variant or encrypting the yaml file as a binary, in which case you would still not be able to use sops as an editor of the file. Personally I use multline yaml values to encode yaml secrets.
Notice that this a limitation in sops itself, where sops-nix has little control over.

@ljani
Copy link
Author

ljani commented Sep 29, 2021

Notice that this a limitation in sops itself, where sops-nix has little control over.

Does it? To me it seems it's sops-nix doing the unmarshalling.

A hack would be to write decryptSecret to something like this (search for SkipUnmarshal), but I have not tested this:

func decryptSecret(s *secret, sourceFiles map[string]plainData) error {
	sourceFile := sourceFiles[s.SopsFile]
	if sourceFile.data == nil || sourceFile.binary == nil {
		plain, err := decrypt.File(s.SopsFile, string(s.Format))
		if err != nil {
			return fmt.Errorf("Failed to decrypt '%s': %w", s.SopsFile, err)
		}
		if s.Format == Binary || s.SkipUnmarshal {
			sourceFile.binary = plain
		} else {
			if s.Format == Yaml {
				if err := yaml.Unmarshal(plain, &sourceFile.data); err != nil {
					return fmt.Errorf("Cannot parse yaml of '%s': %w", s.SopsFile, err)
				}
			} else {
				if err := json.Unmarshal(plain, &sourceFile.data); err != nil {
					return fmt.Errorf("Cannot parse json of '%s': %w", s.SopsFile, err)
				}
			}
		}
	}
	if s.Format == Binary || s.SkipUnmarshal {
		s.value = sourceFile.binary
	} else {
		val, ok := sourceFile.data[s.Key]

		if !ok {
			return fmt.Errorf("The key '%s' cannot be found in '%s'", s.Key, s.SopsFile)
		}
		strVal, ok := val.(string)
		if !ok {
			return fmt.Errorf("The value of key '%s' in '%s' is not a string", s.Key, s.SopsFile)
		}
		s.value = []byte(strVal)
	}
	sourceFiles[s.SopsFile] = sourceFile
	return nil
}

@ljani
Copy link
Author

ljani commented Sep 29, 2021

Yet another way would be to reserialize nested keys back to yaml/json and set that string into s.value, but I'm not sure if I like that

@Mic92
Copy link
Owner

Mic92 commented Sep 30, 2021

I think I would prefer format=binary over adding another option which does the same thing. At the end of the day you would need an option to treat it differently.

@ljani
Copy link
Author

ljani commented Sep 30, 2021

Yeah, I understand and thus I'm trying to come up with a good idea. What about supporting setting s.Key to nil? In other words, if the secret key is not set, use the whole file:

func decryptSecret(s *secret, sourceFiles map[string]plainData) error {
	sourceFile := sourceFiles[s.SopsFile]
	if sourceFile.data == nil || sourceFile.binary == nil {
		plain, err := decrypt.File(s.SopsFile, string(s.Format))
		if err != nil {
			return fmt.Errorf("Failed to decrypt '%s': %w", s.SopsFile, err)
		}
		if s.Format == Binary || s.Key == nil {
			sourceFile.binary = plain
		} else {
			if s.Format == Yaml {
				if err := yaml.Unmarshal(plain, &sourceFile.data); err != nil {
					return fmt.Errorf("Cannot parse yaml of '%s': %w", s.SopsFile, err)
				}
			} else {
				if err := json.Unmarshal(plain, &sourceFile.data); err != nil {
					return fmt.Errorf("Cannot parse json of '%s': %w", s.SopsFile, err)
				}
			}
		}
	}
	if s.Format == Binary || s.Key == nil {
		s.value = sourceFile.binary
	} else {
		val, ok := sourceFile.data[s.Key]

		if !ok {
			return fmt.Errorf("The key '%s' cannot be found in '%s'", s.Key, s.SopsFile)
		}
		strVal, ok := val.(string)
		if !ok {
			return fmt.Errorf("The value of key '%s' in '%s' is not a string", s.Key, s.SopsFile)
		}
		s.value = []byte(strVal)
	}
	sourceFiles[s.SopsFile] = sourceFile
	return nil
}

EDIT: Err, the current way of defining secrets does not really support that. To do this, secrets should be indexed by their file paths, not their keys.
EDIT2: I think I probably should investigate systemd's LoadCredential= and LoadCredentialEncrypted= a bit more as their functionality is more closer how I imagine secrets should be handled. As far as I understand, those could be plugged to use sops as well.

@shajra
Copy link

shajra commented May 6, 2023

I think I would prefer format=binary over adding another option which does the same thing. At the end of the day you would need an option to treat it differently.

I ran into wanting the same request personally. I do agree that functionally, you get some of the same things with a binary encoding. But the biggest thing you don't get with a binary encoding is one of SOPS's selling points, which is a nice textual diffs in source control. So I don't think it's completely fair to say it's the same thing, though it's close.

So that said, is this feature something we think we can get upstreamed? That would inform the effort and approach I take personally in trying to get it to work.

@miooochi
Copy link

miooochi commented Feb 8, 2024

Hi, I came across with similar use cases. Any updates on this feature request?

@musjj
Copy link

musjj commented Feb 19, 2024

Would love to have this too. It's really useful for passing a complex configuration file as-is to a program, while still retaining the benefits of SOPS (diff-able plain text).

@thomaslepoix thomaslepoix linked a pull request Mar 25, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants