Unable to load correct certificates if 1 invalid one is in consul k/v

[finwo]

Summary

When there's a single invalid certificate in the consul certificate store in use by fabio, all the other certificates in the store will not be refreshed.

Upon a fresh start of fabio (like a server reboot), this causes no single ssl endpoint to function anymore.

Steps to reproduce

We're running fabio using the fabiolb/fabio:latest image from dockerhub on a nomad cluster as system job (running on every worker node).

First off, here's the used config loaded into /etc/fabio/fabio.properties within the container from a nomad template:

proxy.addr = :{{env "NOMAD_PORT_http"}};proto=http,:{{env "NOMAD_PORT_https"}}:proto=https;cs=cssl
ui.addr = :{{env "NOMAD_PORT_ui"}}
registry.consul.register.enabled = false
{{with nomadVar "nomad/jobs/fabio"}}
registry.consul.addr = {{.consul_addr}}
proxy.cs = cs=cssl;type=consul;cert=http://{{.consul_addr}}/v1/kv/certs
{{end}}

The certificates are loaded into /certs within consul as bundles, so /certs/example.com.pem contains the full chain including the private key.

To replicate the issue, insert a single certificate where the private key does not match the certificate. This will result in fabio not loading any certificates.

Expected result

Fabio to load all valid certificates, and ignore/skips the invalid certificates, allowing domains where valid certificates are available for to continue to function.

Actual result

Fabio refuses to load or update any certificate, causing all services it's loadbalancing to fail upon ssl initialization because it fails to find a valid certificate for the servername indicated by SNI.