WIP: GPG remote trusted key updating #2260
Conversation
Currently the verifier only imports all the GPG keys when verifying data, but it would also be useful for inspecting the trusted keys.
In order to use the GPG verifier, it needs to be seeded with GPG keys after instantation. Currently this is only used for verifying data, but it will also be used for getting a list of trusted GPG keys in a subsequent commit.
This function enumerates the trusted GPG keys for a remote and returns an array of `GVariant`s describing them. This is useful to see which keys are collected by ostree for a particular remote. The same information can be gathered with `gpg`. However, since ostree allows multiple keyring locations, that's only really useful if you have knowledge of how ostree collects GPG keyrings. The format of the variants is documented in `OSTREE_GPG_KEY_GVARIANT_FORMAT`. This format is primarily a copy of selected fields within `gpgme_key_t` and its subtypes. The fields are placed within vardicts rather than using a more efficient tuple of concrete types. This will allow flexibility if more components of `gpgme_key_t` are desired in the future.
This provides a wrapper for the `ostree_repo_remote_get_gpg_keys` function to show the GPG keys associated with a remote. This is particularly useful for validating the GPG key updates have been applied.
This will be used to implement the PGP Web Key Directory (WKD) URL generation. This is a slightly cleaned up implementation[1] taken from the zbase32 author's original implementation[2]. It provides a single zbase32_encode API to convert a set of bytes to the zbase32 encoding. I believe this should be acceptable for inclusion in ostree. The license in the source files is BSD style while the original repo LICENSE file claims the Creative Commons CC0 1.0 Universal license, which is public domain. 1. https://github.com/dbnicholson/libbase32/tree/for-ostree 2. https://github.com/zooko/libbase32
Calculate the advanced and direct update URLs for the key discovery portion[1] of the OpenPGP Web Key Directory specification, and include the URLs in the key listing in ostree_repo_remote_get_gpg_keys(). These URLs can be used to locate updated GPG keys for the remote. 1. https://tools.ietf.org/html/draft-koch-openpgp-webkey-service-08#section-3.1
If the key UID contains a valid email address, include the GPG WKD update URLs in GVariant returned by ostree_repo_remote_get_gpg_keys().
This provides a wrapper for the `ostree_repo_remote_update_gpg_keys` API to update a remote's GPG trusted keys using the PGP Web Key Directory protocol.
In order to test `ostree_remote_update_gpg_keys`, we need to be able to fetch the keys from a local test server. This inherently requires introducing a backdoor to the update process. If the _OSTREE_GPG_UPDATE_LOCAL_PORT environment variable is set, change the server to http://127.0.0.1:<port> after validating that the port is numerical. This should keep any attack local by not allowing the URL to be changed to an arbitrary remote server.
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: dbnicholson The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@dbnicholson: PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
dbnicholson
changed the title
|
For Fedora(host systems) we use For flatpak in general, wouldn't it be a lot simpler to sign |
|
To be clear I'm not opposed to this, I just want to be sure that the simple solutions aren't sufficient. |
This is the same as Endless - key updates are delivered in OS updates via What happens when the OS update is no longer signed by that trusted key because it's expired or is compromised? There needs to be an out of band mechanism to update the local trusted key. Currently that out of band mechanism would be manual and technical - you'd send out a bunch of emails and make a web page explaining the situation with the new trusted key and instructions on how to add it to the local key store. This is a pretty high barrier. It requires the user to be looking for that type of information and to know how to handle a security sensitive process correctly. Turning that manual process into an automated process is the goal of this work. For flatpak, there's already a mechanism for delivering keys via the |
|
Just for some other background, here's a page on the gnupg wiki discussing different key update methods - https://wiki.gnupg.org/OpenPGPEmailSummit201607/KeyDiscoveryComparison. There are some other pages on the wiki discussing these types of things. |
|
Lastly, I sent an email to the mailing list in https://mail.gnome.org/archives/ostree-list/2021-January/msg00000.html if you want to have a more general discussion there. |
If it's compromised...I don't see how we can do anything? I need to dig into this a bit but what's the trust model for the web service? Is it just "TLS with keys from ca-certificates"? This WKD stuff seems to mostly be about authenticating the email address in GPG keys, which seems orthogonal? I guess let me say it this way: You've added a lot of new code here but I am missing a brief document that describes how one might set this up and use it. And a specific question: if ostree had had this support originally when your key expired, would it have saved you? How? I feel like this key expired case is actually a special case of a more general one where one wants an "out of band hotfix" process for an ostree-based system. I believe Firefox has something like this - a mechanism they can use for truly critical fires. A "hotfix system" wouldn't need to be implemented in ostree at all really, it could be a separate systemd service that pulls content however it wants and validates it integrity however and basically supports live-updating the system. Of course, the question is how the integrity of those patches is validated...and if you're signing those correctly then presumably you're also signing your ostree commits correctly. |
|
Can you split out the non-WKD patches to a separate PR? |
Sure. No promises on when I'll get to that, but there are some things in here that are not specifically for WKD. |
If it's compromised but you can easily (or automatically) fetch an updated key from a trusted location, then you can carry on. For example, suppose my ostree signing key got compromised. I can generate a new key that has the same ID (email address) and publish it on a web server in a WKD compliant location. The client checks for an updated key at that location, sees there's something new, updates the local keyring and we carry on. It's basically the same process as if I'd written an email saying where to get my new key from and what to do with it except that it can happen automatically because the URL is predictable. The trust model is TLS here and the fact that the key can only be fetched from a fixed URL within the domain it's email address is a part of. To spoof that process, an attacker would need to send a fake DNS reply and provide a valid TLS certificate.
The email address determines the URL where the update can come from, so I was trying to be a bit paranoid about ensuring that was done correctly. This goes in 2 directions - determining the URL to make the request to and then only taking the key for that desired email address in the received keyring.
Sure, I can write something up on how you'd do it. https://wiki.gnupg.org/WKDHosting basically explains how you'd populate the server side, but I'll follow up with something more concrete. It definitely would have saved us. After updating the expiration date on our key, we would have published the new key on our WKD server. Then the clients would fetch the updated key and signatures for our commits would be valid again.
They feel a little different to me. Being able to provide a hotfix because the OS we published is itself broken or compromised sounds very useful. We have more than once released commits that could no longer boot or update and the only thing we could resort to was publishing some instructions for people to run manually. But this issue isn't with the OS itself, it's with the trust path. Hotfixing the payload vs hotfixing the delivery mechanism seems different to me. That said, there isn't any requirement for this to be in ostree. You could have a service that occasionally checks for and updates the trusted keys. The nice thing about having it in ostree is that any ostree application can make use of it without depending on an outside service. |
I wouldn't assume any security with DNS in general, so this is basically TLS. And if you're trusting that absolutely...what's the point of GPG signing in the first place over just fetching the commit over that same TLS connection? |
Because they validate different things. TLS validates the transport while GPG validates the payload. Having separate validation of the payload means that you can use an untrusted transport such as any non-TLS network protocol or a local filesystem. Like would happen for peer to peer distribution. Likewise, payload verification means that if someone can insert a rogue commit into the remote repo it would still fail to be validated by clients. Why do you sign the ostree git tags? Why did you develop git-evtag? Presumably we're all using a secure transport to access the remote repo (HTTPS or SSH), so by the same logic we can assume that any object in the git repo is valid. How do I get your public key to validate the signature on those tags? I wouldn't take it from an object in the repo and then use it to validate a tag on the repo. I would get it from some other trusted source. TLS is also the security mechanism used for the transport of the trusted keys here, but I don't think that TLS also being the typical repo object transport security mechanism really has any bearing. They're trying to secure and validate access to different things even though they happen to use the same mechanism in this work. Indeed I am trusting TLS here just like we all do every day on the internet and just like the WKD developers determined would be acceptable. |
|
I'm not arguing against GPG signing, you are right transport and offline integrity are different things. I've glanced though the WKD stuff but haven't done a deep dive - it may be somewhere there and I'm just missing it. It seems to mostly be a protocol about discovery, leaving the "how to choose to trust this key" outside of the scope. You are here basically automatically trusting keys found this way, right? So that to me seems to negate the whole point of using GPG in the first place if we unconditionally load new keys over TLS. Perhaps one middle ground is that we only try to fetch keys from WKD in the case of expired keys by default? And in that case, require the new key be cross-signed from the old one? |
|
Sorry for taking so long to respond.
In the case of ostree, we're saying that we already trust the IDs in the keys for the remote's trusted keyring. You or someone on your behalf has already imported my key with dbn@endlessos.org to trust my repo at the remote URL. What WKD does is give you a URL to find a new version of my key that's at a fixed URL tied to my domain. So, in case my key expires or I need to revoke it there's a known place to get it from that can't really be spoofed unless I lose control of my domain or TLS breaks down. So, it's not totally arbitrary, but it does preclude that you trust the IDs that are in your trusted keyrings. Perhaps there could be a remote flag that says whether the key can be updated.
I think revoked is the bigger issue, which you can't tell by looking at the key you currently have. I think cross signing could work. Another option would be that the primary key fingerprint is unchanged. In which case you'd do all your key management on subkeys. |
That's really what I'm focusing on - in exactly what circumstances can a key be updated via this mechanism, and what does it require? If we're focusing on the expired key problem, then requiring new keys to be cross signed seems like a good default.
Right; this is a common pattern in the GPG world for this. Here's another random idea though, what if we patched ostree to warn loudly if a key it verifies is e.g. going to expire in a month or even 3 months? |
|
Is it okay to rely GPG keys lookup on SHA-1 algorithm used in protocol? Shouldn't it be updated to more secure algorithms? It does not have support for different hash algorithms it seems to me. |
|
Sorry I keep forgetting to reply on this. For the expiration warnings, I think they could be a little useful. On the server side I probably would have noticed that before the expiration and done something about it. On the client side I can't see it really being useful, though. The warnings wouldn't be seen unless a CLI was in use, and even if they were users can't do anything to fix the issue. Still wouldn't do anything about a key that needed to be revoked, though. I think that the really scary part here is the fully automated part. So, maybe let's not do that right now. If ostree can just help you see what keys are there and what the WKD URLs are, then anyone could essentially write their own automation around it. Providing a way to fetch and filter the returned key would be nice, but not really necessary. That's really not different than what someone could do now manually - figure out the email address from a remote's key, turn it into a WKD URL, download it and import it back into ostree. What I'm proposing now is having ostree just make parts of that easier to do. I think your initial "hey can you split out these general parts" would cover most of it. Whether someone wants to trust the WKD URL itself or what's returned from downloading it can be someone else's business. |
|
I cleaned up and broke out the non-updating parts to #2401. I think with that in place the actual key updating and associated policy and security decisions can be made by someone else. |
cgwalters
added
difficulty/hard
|
@dbnicholson: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
This is a work in progress that I started a while back and would like some comments on. In order to securely update the trusted GPG keys for a remote, it adds an implementation of the OpenPGP Web Key Directory client side. This lets a remote publish updated GPG keys at a known location and have ostree fetch them to update its local keyring.
There are still parts that need some polish, but I believe it mostly works. I'm mostly posting this to get feedback. The impetus for this work is that at Endless we had one of our trusted keys expire, which would have prevented people from updating. We had a workaround by having a 2nd key in the trusted keyring, but we would have preferred a reliable way to push an update for the expiring key.