Problems using Consul as certificate store (x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate)

[hdecolle]

Hi, this is more a question, than an issue.

I want to use Consul as certificate store with Fabio. Fabio is started by nomad with the exec driver. Consul is configured to use mTLS, which means I have a custom CA. Consul stores at the moment self signed certificates, later on they will come from LetsEncrypt.

At the moment I use fabio 1.3.0, nomad 1.3.0 and consul 1.12.0.

Fabio is started this way:

-log.level TRACE -insecure 
-registry.consul.tls.insecureskipverify=true
-registry.consul.addr https://localhost:8501
-registry.consul.kvpath /system/fabio/config
-registry.consul.noroutehtmlpath /system/fabio/noroute.html
-registry.consul.tls.cafile /etc/gravity/consul/certificates/consul-agent-ca.pem
-registry.consul.tls.certfile /etc/gravity/consul/certificates/fabio.pem
-registry.consul.tls.keyfile /etc/gravity/consul/certificates/fabio-key.pem
-proxy.cs "cs=cluster;type=consul;cert=https://localhost:8501/v1/kv/system/fabio/https-certificates"
-proxy.addr ":9997;cs=cluster,:9999;proto=http"

With this configuration I see the following line in the logs:
[WARN] cert: Error fetching certificates from system/fabio/https-certificates. consul: list: Get "https://localhost:8501/v1/kv/system/fabio/https-certificates?consistent=&recurse=": x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "XYZ")

I can access consul, with the binary and with curl without problems.

I have the following entries as SAN in the certificate client.xyz.consul,localhost,127.0.0.1

I tried also adding the hostname where fabio is running and its public IP, but I still get the same error. It would help to see the exact call that fabio makes to consul.

Has someone an idea how I could fix this?
Thank you and thanks for fabio.

[hdecolle]

Ups. I use fabio 1.6.0. Sorry.

[nathanejohnson]

So as it stands currently, none of the consul.registry config items influence the behavior of the consul certificate store's consul api client. So using self-signed certificates with loading certs from consul is not currently possible. One workaround would be to talk to a local consul agent over http and configure encryption on the consul cluster for node-to-node communication. Another option would be to store the certificate information inside of vault and use the vault cert store backend instead. It would be fairly easy to add some options to override the certificate bundle use for the consul cert backend api client, but that doesn't exist today.

[nathanejohnson]

so it might be sufficient to specify the CA using the SSL_CERT_FILE environment variable. @hdecolle can you try this and see if that helps?

https://pkg.go.dev/crypto/x509#SystemCertPool

[nathanejohnson]

disregard mentions of a PR for this issue, that's for #872, not this one.

[hdecolle]

Thanks for the suggestion, but it did not work. I have got the same error.

Vault can't be used at the moment, but it is the end goal.

At the moment I have the following workaround:

• Fabio is reading the certs from a file store
• When Fabio is deployed via Nomad, a sidecar job starts a consul-template which watches the consul kv path for changes and writes all certificates from the Consul KV path to the disk

So as it stands currently, none of the consul.registry config items influence the behavior of the consul certificate store's consul api client

I thought the following are used to access consul and when I think about it, those do work, because if they would not, then fabio would not be able to route anything. Or is fabio calling the "consul" binary, which already has a working connection to the server?

-registry.consul.tls.cafile /etc/gravity/consul/certificates/consul-agent-ca.pem
-registry.consul.tls.certfile /etc/gravity/consul/certificates/fabio.pem
-registry.consul.tls.keyfile /etc/gravity/consul/certificates/fabio-key.pem

So using self-signed certificates with loading certs from consul is not currently possible.
So is the issue then that I have a self-signed certificates in the consul cert store?

Which would mean if I only store certificates which are signed from LetsEncrypt, that it would work without at problem? I did not yet try that.

Then my test with the SSL_CERT_FILE was invalid because I have used the Consul CA for that.

The self signed-certs are not created with a CA because the idea was to use them only for the time where no certificates from LetsEncrypt where available.

[nathanejohnson]

So you don't actually need a nomad sidecar in this scenario, you can actually do inline templates with substitutions from consul values, just like you could with vault secrets.

https://www.nomadproject.io/docs/job-specification/template#inline-template

If you do go this route, I'd recommend change_mode as noop, since fabio picks up changes to file stores without needing a restart.

Nomad basically has consul-template built in.

Ideally, the consul certificate store would use the same config values passed to the registry (particularly CA stuff). I will make this as a note for the next release. But I think your workaround is sane, though you could skip sidecar step as I mentioned above (assuming I fully understand your scenario).

[hdecolle]

Thanks for the suggestion.

I tried it with inline templates first, but as I understand it, fabio needs each "domain" as a single certificate even though it is a san certificate.

At the moment each domain has an entry in the KV "directory" that fabio should use. I also update those paths with a different service. But since fabio can't access the kv path in my setup, I needed a way to download each certificate to the disk where fabio is running.

For that to work the inline template must write more than one file, and I did not find a way. So I used a nomad sidecar and consul-template for that.

[nathanejohnson]

You can have as many inline templates as you want within the nomad job spec, that render to as many files as you want. Also, fabio is smart enough to map all of the SAN names in a certificate, so you shouldn't need a file per domain name. Wildcards also work.

https://github.com/fabiolb/fabio/blob/v1.6.1/cert/store.go#L100

[hdecolle]

Is there any news regarding this issue?

Now I have one file for each SAN certificate. It works great thanks for the tip. Also only one entry per SAN certificate in the consul kv store now.

The rest of my workaround is still the same. Now I have fabio in version 1.6.3, noamd in 1.5.5 and consul in 1.15.2.

I know that nomad can render multiple inline templates, this was the solution that I had previously but then I have to change the job file each time a new SAN certificate will be added. I would like to avoid that.