I think this makes sense, and has been requested in #1250. Additionally we've discussed the design of this in #1201.
As mentioned by @bmw it's easy to implement this in a --deploy-hook #1201 (comment)
That said, I agree with @pde that it's probably best for this to be a default rather than associated with an installer or a flag: #1201 (comment)
@jmtd started implementing this: jmtd@a7a8941 but stopped since they found it easier just to fix it in their case.
My fear with this being a flag is that you might be able to request renewal without that flag and end up in a situation where your renewed cert is now missing the cert file that you've referenced in your jabber config.
That said, I'd be pretty happy with this being something you could set in cli.ini so that all of your renewals made this file, but it wouldn't be a surprise to you that you have multiple files that contain your privkey, because you had to write your cli.ini file. This also means that if jabber ever wanted to rely on certbot they could ship this cli.ini file and it'd work out of the box.
I think we'd be happy to take a PR, but @bmw would have the most useful information on what we'd want this feature to look like.

Thanks for the response and links. So, there is a possible additional case for lighthttpd that requires only privkey and cert to be bundled but not chain... Actually, even jabberd2 can take an additional chain file (I'm not sure whether it could be equally happy with and without chain in the combined one). We should try and list all the funny formats to figure out whether there are any more cases (different concatenation order required or something) but hopefully we can make do with just two files (privkey + cert and privkey + cert + chain). Question of naming those files got even more complicated. privkeycert.pem and privkeyfullchain.pem, maybe? (The latter sounds gross but is more or less consistent with current naming convention and it's intuitively easy to guess what's inside and in which order.)

I'd say let's generate those files by default unless various servers need more than these two formats.

And if there are too many possibilities, we should make a flag that sets extra formats required (something like --extra-formats=privkeycert,privkeyfullchain,certprivkeychain,whatnot and that could include formats other than pem actually, if we ever need them for some reason.)

I'm gonna do some research on formats servers expect and report back in a few days.

Awesome - let me know what conclusions you come to!

While I think we should do something here, I think implementing this will be somewhat controversial within the Certbot team. Here's a post from @schoen, the most active member of the Certbot team on Let's Encrypt's community forum, saying that using the wrong file is the most common post-issuance problem. I feel like adding more files here (at least by default), will just exacerbate this issue. That's not to say we can't do this, just that it should be designed carefully to avoid confusion from sysadmins of any skill level.

@SwartzCr, is there are a reason to keep both this issue and #1250 open or can they be consolidated into a single issue?

I'd be happy with having all cert bundling issues bundled into one issue about cert bundling

Brad, I agree, it should be designed carefully. That's why I opened this ticket to discuss design and reach agreement on things instead of starting coding right away.

I appreciate that and your interest in working on this topic all together. Again, I definitely think we should do something here.

However we do it, I'm pretty sure some kind of built-in bundling should produce fever errors than hooks or, worse, wrapping renew command into a script that rewrites bundled files and does other unconditional things (which was a common pattern with older versions, before renew-hook).

I completely agree with this.

Despite my concerns with creating these files by default, if we can do this in a way where we feel confident we'll avoid confusion for most people, I'm fine with this approach.

If we don't think we can make the differences between the files clear though, I think the user should be able to configure which files get created. If we go this route, it's slightly unfortunate from a consistency point of view that we currently create fullchain.pem (which is just the concatenation of cert.pem and chain.pem), but we're unable to change this being the default behavior in the near future.

My instinct is that we could add extra files if and only if the user requests them with a CLI flag. There would be some architectural decisions to make about how these options are controlled by Authenticators and Installers. Conceptually it makes sense to make the new files at auth time, but installers (for instance, a hypothetical jabberd installer) might sometimes want to enable these options on the user's behalf, with somewhat tricky consequences.

I think that was basically a restatement of this comment. We should close either that old issue or this new one as a dupe perhaps :)

Per #5643 , this would also be useful for courier-imap.

Here's a post from @schoen, the most active member of the Certbot
team on Let's Encrypt's community forum, saying that using the wrong
file is the most common post-issuance problem. I feel like adding more
files here (at least by default), will just exacerbate this issue.

Stumbled on this exact problem, ran into this bug.

A far greater problem than using the wrong file is not having a file at all - which totally destroys the point of letsencrypt existing in the first place.

Both jabberd and certbot have taken opposing ideological stands, and the people who lose out are end users like us who find certificate deployment way harder than it needs to be. Certbot already provides four different files, providing a fifth option (and a PKCS12 option) is in no way making the situation worse.

This script is a workaround, but warning - it creates a file in the live directory instead of symlinks, which is non ideal.

`[root@localhost ~]# cat /etc/letsencrypt/renewal-hooks/deploy/fullchainprivkey
#!/bin/bash

cat "$RENEWED_LINEAGE/fullchain.pem" "$RENEWED_LINEAGE/privkey.pem" > "$RENEWED_LINEAGE/fullchainprivkey.pem"

`

Not to be a Bumping Betty, but is there any progress on this?

I see the point of having a discussion, but arguments have been made, counter-arguments have been made... then a year of silence.

Does anyone have the authority to make a final decision? It's a simple implementation, desperately needed for people who use lighttpd and similar, and it's lingered in the discussion phase for three years.

There has not been. In the duplicate issue about this I said:

If someone would like to propose a design and PR for this, please do so! I'd recommend sharing your design first to get feedback before you spend time implementing it.

Also, while this is already referenced above, if anyone needs help coming up with a workaround for now, you can see my comment at #1201 (comment).

OK... cool. I'm not a python expert, but I know how to concat strings. 😃 Never contributed to open source before either but I'll give it a whirl.

I have one question: What should the new file be called? Some workarounds I've seen cat the file into ssl.pem or web.pem. Is there a naming standard? I think minfrin's suggestion of fullchainprivkey.pem is obvious, albeit a little long. Having the name be so descriptive isn't totally necessary either since there'll be a note in the README.txt anyway.

That'd be great @keith24!

I recommend you look through the discussion in this issue, #1250, and #1201 and write up a short proposal for how this work (e.g. is the file created by default?) and include a suggestion or two for the name of the file. Like you alluded too, concatenating strings is pretty easy, but there's been a lot of concerns about different aspects of the design here.

OK. I read through those discussions. Here are some questions that came up, and my proposals:

1. Question Is creating a new file the best implementation?
   Proposal: Yes, because:

• Several webservers require it
• Kludgey workarounds involving cat with cron/hooks should not be necessary.
• Making an installer (which performs the concatenation) for each of those servers will likely never be implemented.

2. Question Should the new file be created by default?
   Proposal: Probably, because:

• The file wouldn't be very big and generating it wouldn't take very long
• Switching from a server that doesn't require it to one that does would not require reconfiguring certbot

3. Question What should the file be called?
   Proposal: fullchainprivkey.pem, because:

• The contents are obvious even without checking the readme
• I'm not creative and can't think of anything better

Are there other concerns I missed? Does anyone have an issue with the proposals? I see arguments going both ways but I'd like to see something actually happen rather than an eternal debate.

Regarding point 3: These are longer, but seems more clear to me: fullchainplusprivkey.pem or fullchainandprivkey.pem

That said, I certainly wouldn't cry over your original suggestion. I'd far rather have the file than not. :-D

Regarding points 1 & 2: I'm in total agreement.

Just a little contribution. Since key is appended before the fullchain, name should be privkeyandfullchain.pem, or privkeyplusfullchain.pem. Also, since the file is a concatenation of two existing files with well defined specs, you could save some characters to express the concatenation: privkey_fullchain.pem.

Yeah, I also thought about fullchainwithkey.pem. It actually might be easier to put "fullchain" second, like privkeyfullchain.pem because an "f" is taller than a "p" and that makes it easier to read, like camelCase.

That raises another concern... Does the order matter? Do any of the servers expect the key or the chain to be first in the file? All the workarounds I've seen put the chain first...

OK, the key goes first. I like the underscore... Its readable too.

The file wouldn't be very big and generating it wouldn't take very long

Our concern here isn't the computational complexity of catting 2 files. It's a UX and project design concern for the overall project.

• Project design/maintainability: Do we want to support every use case by default? One other issue like this is Java keystore (.jks) / jetty support? #1701, for JKS support. Would this lead to a large blow-up of files in the /live/ directory? Depending on how many requests of this we see, it might make sense to make this a configurable option going forwards. I'll do a quick dig into how many similar issues there are and get back to you!
• UX: @bmw says:

Here's a post from @schoen, the most active member of the Certbot team on Let's Encrypt's community forum, saying that using the wrong file is the most common post-issuance problem. I feel like adding more files here (at least by default), will just exacerbate this issue.

Although I personally disagree with this point-- I think that the name privkey_fullchain.pem is very clear, and that the "using the wrong file" issue is primarily due to the confusion between the names cert.pem and fullchain.pem (the latter being the correct file to use in most cases), and not necessarily due to the sheer quantity of files in the live directory.

While we'd still like help designing and implementing this, I'm assigning @sydneyli as she volunteered to help from our end with the design here.

@Harvie
tell me the file in which a more beautiful version of this 3 year long procrastination has to go in.

I think it would be better to send a patches with the fullchain support to all the legacy programs. As far I see some of them like lighttpd already support this.

This problem (and problems like it) annoyed me so much I made this to solve it:

https://redwax.eu/rt/

Suck in the certs generated by certbot, write out the certs/chain/root/keys in whatever weird combination required as a one liner.

Test it here: https://source.redwax.eu/svn/dist/rt/dev/, binaries here: https://copr.fedorainfracloud.org/coprs/redwax/rt/

Sorry for the stupid question: what was the initial purpose to store priv key, public cert and chain separately? You anyway can't establish TLS connection without all of them.
Just to encrypt/sign some data it may be enough only a priv key but this is a rare use case.
Also the other reason that I see is to be able to give the pub key to client but for the let'sencrypt this is not needed because the cert already signed by CA.

Having the three files (ok two if use fullchain) increases complexity.
Apps needs for at least two config keys, at least two read syscall etc.
In the same time it's totally fine to have all needed certs and key in a single file. If some application doesn't support this we may just send a patch to the app. Not so many of them.
Please explain to me, I'll appreciate

If some application doesn't support this we may just send a patch to the app. Not so many of them.

But you can't really get complete list of such applications and you cannot be sure their community will allow for easy 3rd party contributions. Not everything is FOSS. If you fix it in certbot, you only have to fix it in single place and the fix will be easy. Otherwise you have to fix it in every app that is out there, which seems rather hard.

Imagine that 5 years from now you will need some obscure application which has not been fixed yet, because you haven't used it so far. If we fix now in certbot, there will be no problem. Otherwise you will have to get that fixed once again in some weird app and live for two years with custom build of that app until it gets merged to Debian.

Thank you @Harvie, I totally agree with you. I don't insist to force everyone to use a single file with all certs. But if I understood correctly this would be a good "default" option. Even more: the lighttpd, postfix and some other apps like emailrelay initially expected such a single file with all needed included.

maybe that filename should contain _deprecated_ keyword in it's name, so author of that software might understand that it's not reccomended way of using certs and he might want to fix that. but users will not have to struggle.

I just googled and it looks like mentioned here postfix, lighttpd and ejaberd now supports specifying both privkey and fullchain separately without a concatenation. So the issue is not that important anymore.

The question is remains if the single all-in-one cert file makes sense. Except of the single read syscall to read all at once.

For the Apache HTTPD you can actually use such all-in-one file for the SSLCertificateKeyFile config. But there said:

The private key may also be combined with the certificate in the file given by SSLCertificateFile, but this practice is highly discouraged. If it is used, the certificate files using such an embedded key must be configured after the certificates using a separate key file.

I think that reasons why it's discouraged was the same as mentioned here. Still this is an open question Why store Apache SSL certificate and private key in separate files?.

Interesting that looks like such pubpriv pem files aren't specified anywhere.

For Windows and ISS web server you can import only a .pfx file which is PKCS12 archive that contains fullchain and privkey.
We can create the pfx file with sudo openssl pkcs12 -export -in /etc/letsencrypt/live/domain.com/fullchain.pem -inkey /etc/letsencrypt/live/domain.com/privkey.pem -out domain.com.pfx.

Such a file has an advantage that it's recognized by Explorer and you can easily import it by a double click. Also you may set a password for it. This is a binary format so it uses less space. But it can't be just written somewhere in configs. I don't think that's so important.

BTW Windows Explorer doesn't know about the .pem extension but it can show details for .crt files. Hope they'll extend their support for the .pem extension too.
But even if rename privkey.pem to privkey.crt it won't be opened in Windows i.e. this works only for a public certs.
As a workaround for this usability problem of Windows we may use a Keystore Explorer program.

Confirming: lighttpd has supported ssl.privkey since lighttpd 1.4.53, released Jan 2019 (3 1/2 years ago).

Aside: lighttpd has also supported TLS-ALPN-01 verification challenge (https://wiki.lighttpd.net/HowToSimpleSSL) since Jan 2019, but you have to use dehydrated since certbot client is still lacking support for TLS-ALPN-01 (#6724).

I've just come across this thread after searching for a way of doing this for my SIP server - flexisip. It requires a combined cert and private key file for TLS connections.

@RaddishIoW oh. you must be new here :-D

Haha - yes, reading the thread it would seem this is not something that will happen soon..

I did however go on to discover post-deploy hooks, so am using a shell script to cat the cert and key together after renewals automatically. :-)

@RaddishIoW instead you can ask or send a PR to the flexisip.
Here I leaved a comment there BelledonneCommunications/flexisip#144 (comment)

I asked an author of EmailRelay to do that and he implemented easily.

I still don't get this "we're right and everyone else is wrong" idea... Why everyone else should be adapting to behaviour that can easily be fixed in single point?

So, I have moved my old certbot-extra-formats into a standalone repo.
I published it on pypi and on the AUR.

After 7 years could you make a decision to implement this or not?

Personally I think FOSS should make our lives easier not harder.

@mrx23dot which app are using that still can't accept two files?
We better to send a PR to them instead.

@mrx23dot which app are using that still can't accept two files? We better to send a PR to them instead.

All the tutorial for lighttpd was written using the merged pem, so this will resurface again and again, even though it has been fixed as I read.
Also not every distro ships with latest releases.

I don't know about you but I wouldn't waste a minute more on this, there are more important things to do.

There is no security implication generating:
domain.com/merged_depricated.pem
since the private key is next to it, worst case it won't be used.

If you insist:
certbot --no_merged

@stokito wrote:

@mrx23dot which app are using that still can't accept two files? We better to send a PR to them instead.

The problem is not restricted to the question whether the application can or cannot accept two files. Having everything in a single file has other benefits, too.

For example, Postfix can accept two files (or even three files with key, own cert and chain), but a single file is recommended due to the option to exchange the file atomically. I quote from #9915 which has just been marked as duplicate:

For certain application [...] a single file which contains everything [...] would be beneficial. For example, Postfix picks up a new certificate (or private key for that matter) without reloading the daemon. However, it is necessary that the information is updated in an atomic manner. If the file private key and the certificates are stored in individual files, it might happen that Postfix reads a private key and certificate which do not match (of course depending on circumstances and timing).

See Postfix Configuration Parameters – smtpd_tls_chain_files:

Storing the private key in the same file as the corresponding certificate is more reliable. With the key and certificate in separate files, there is a chance that during key rollover a Postfix process might load a private key and certificate from separate files that don't match.

And yes, I can try to implement my own shell script and then call it from the post deployment hook, but as Certbot already creates four files it shouldn't be that difficult to create another fifth file and solve that aspect at its root.

The problem may occur only if:

1. LE renewal started
2. The key was stored to a file (but the cert file not yet)
3. The Postfix restarting and reads a new key and old cert

This is highly unlikely and other servers doesn't care.
To avoid the problem you must add a LE renewal hook that will trigger Postfix reload. Then even if the postfix failed to load a cert in first place it will re-read it again successfully.

@mrx23dot

I don't know about you but I wouldn't waste a minute more on this

If you came here and asking for a help and someone proposed a help to you then better to spend a minute. I already contributed to one server to implement the separate key and cert files support and can help with others.

If you are using lighttpd you already have a solution and you can save your precious minutes.

not every distro ships with latest releases

But even if the LE will implement this you anyway have to upgrade to use new feature.

@stokito wrote

The problem may occur only if: [...] The Postfix restarting and reads a new key and old cert

Not quite. Postfix does not need to be restarted to read the new key and old cert.

Postfix uses multiple processes to serve client requests. A Postfix "worker" process (term coined by me) lives for a certain amount of requests. After that the process dies and a new worker process is spawned. When such a worker process is started, it picks up the current key and cert file. That is why the documentation says

Storing the private key in the same file as the corresponding certificate is more reliable.

Per default each worker process serves 100 requests. See Postfix Configuration Parameters – max_use. Other software uses similar approaches (for example Apache with the worker or event threading model).

And no, with a high-volume mail server which handle several 100 mails per minute, the chances are not so low that a new worker process is spawned just while Certbot renews the certificate.

To avoid the problem you must add a LE renewal hook that will trigger Postfix reload. Then even if the postfix failed to load a cert in first place it will re-read it again successfully.

I do that right now, but this solution is far from optimal, because this means that there is a short period of time during which Postfix does not respond to new connections. It would be much better if certificate renewal was atomic.

You better to report this to Postfix as a bug. They should ether change their weird pseudo unix architecture (unlikely) or check a timestamp of both files and wait for the cert to be updated. Instead of waiting they can use an old key even if it's file was overwritten. basically you can use an old cert for days before it gets expired.
This is just ideas out of head at first sight, they should know a proper solution.

See, even the one file write is not atomic. You may use inotify to see that actually a lot of things may happen before the flush call.

That's why the only guaranteed signal is a renewal hook.

See, even the one file write is not atomic.

As long as you create temporary file and then mv it to destination (which often times is the only correct thing to do) it is extremely atomic. (unlike moving two files).

And yes, I can try to implement my own shell script and then call it from the post deployment hook, but as Certbot already creates four files it shouldn't be that difficult to create another fifth file and solve that aspect at its root.

Everyone things everyone else should solve the problem.

I eventually solved it like this:

  redwax-tool --pem-in="/etc/pki/tls/private/*" \
              --pem-in="/etc/pki/tls/certs/ca-bundle.trust.crt" \
              --pem-in="/etc/pki/tls/certs/*.pem" \
              --filter=verify \
              --filter-hostname="${tls_client_dns}" \
              --filter-current \
              --filter-expiry=ignore-leaf \
              --parameter-out --order-out=key-first \
              --cert-out --chain-out --key-out \
              --pem-out="/run/postfix/private/smtp.crt" \
              --order-out=all \
              --no-cert-out --no-chain-out --no-key-out --trust-out \
              --pem-out="/run/postfix/ca-certs/ssl-cacert.pem"

Point --pem-in at wherever the certs and keys are. Postfix certs, keys, ca certs are written key first in files specified.