SSL/TLS Passthrough

[M-Gregoire]

Hello everyone,

I'm trying to do a SSL/TCP Passthrough with Marathon-LB 1.14.2, as described in HA Proxy documentation (https://www.haproxy.com/documentation/haproxy/deployment-guides/tls-infrastructure/#ssl-tls-pass-through).

This would allow for Marathon-LB to expose the certificate exposed by the service and not have to provide any certificates to Marathon-LB itself.

Based on https://github.com/mesosphere/marathon-lb/blob/master/Longhelp.md I've created the following template:

frontend marathon_https_in
  bind *:443
  mode tcp
  tcp-request inspect-delay 5s
  tcp-request content accept if {{ req_ssl_hello_type 1 }}

After doing so, I've added the following labels to my container:

  "labels": {
    "HAPROXY_0_REDIRECT_TO_HTTPS": "true",
    "HAPROXY_0_MODE": "tcp",
    "HAPROXY_GROUP": "external",
    "HAPROXY_0_BACKEND_NETWORK_ALLOWED_ACL": "0.0.0.0/0",
    "HAPROXY_0_FRONTEND_HEAD": "\nfrontend {backend}\n bind {bindAddr}:{servicePort}{sslCert}{bindOptions}\n mode tcp\n",
    "HAPROXY_0_HTTPS_FRONTEND_ACL": "  acl acl_{backend} req_ssl_sni -i {hostname}\n use_backend {backend} if acl_{backend}\n",
    "HAPROXY_0_VHOST": "my.awesome.domain"
  }

However, I hit the following error:

[ALERT] 114/140233 (273) : http frontend 'marathon_http_in' (/tmp/tmpclb9vgp5:56) tries to use incompatible tcp backend 'nginx_10011' (/tmp/tmpclb9vgp5:88) in a 'use_backend' rule (see 'mode').
[ALERT] 114/140233 (273) : Fatal errors found in configuration.

Could anyone explain me how I can expose the certificate of my service through Marathon-LB without terminating the SSL session?

Thank you for any help you might provide.

[raghu999]

I think your issue is with "HAPROXY_0_REDIRECT_TO_HTTPS": "true" try removing that parameter and test

[M-Gregoire]

Thanks a lot for taking the time to answer me.

Removing "HAPROXY_0_REDIRECT_TO_HTTPS": "true" does seems to fix the frontend<->backend issue but I now get the following error:

Traceback (most recent call last):
  File "/marathon-lb/marathon_lb.py", line 2030, in do_reset
    self.__group_https_by_vhost)
  File "/marathon-lb/marathon_lb.py", line 1854, in regenerate_config
    app_map_array, config_file, group_https_by_vhost)
  File "/marathon-lb/marathon_lb.py", line 574, in config
    duplicate_map)
  File "/marathon-lb/marathon_lb.py", line 1289, in generateHttpVhostAcl
    haproxy_dir=haproxy_dir
KeyError: 'backend'

After investigating, I found that this error is caused by the following line:
"HAPROXY_0_HTTPS_FRONTEND_ACL": " acl acl_{backend} req_ssl_sni -i {hostname}\n use_backend {backend} if acl_{backend}\n",.

I'm a bit surprised as the example from https://github.com/mesosphere/marathon-lb/blob/master/Longhelp.md#haproxy_https_frontend_acl does make use of the {backend} key and HAPROXY_0_FRONTEND_HEAD also uses it without any issue.

Removing "HAPROXY_0_HTTPS_FRONTEND_ACL" label does make marathon happy but leads to a secure connection failed (PR_END_OF_FILE_ERROR) when accessing the vhost (which seems normal).

[raghu999]

Seems like that is because of your template mode tcp you are enabling the tcp on the 443 port but your config is referring to work on https.

frontend marathon_https_in
  bind *:443
  mode tcp
  tcp-request inspect-delay 5s
  tcp-request content accept if {{ req_ssl_hello_type 1 }}

Just use the default template and then try enabling your app with below labels

"labels": {
    "HAPROXY_0_REDIRECT_TO_HTTPS": "true",
    "HAPROXY_0_MODE": "tcp",
    "HAPROXY_GROUP": "external",
    "HAPROXY_0_BACKEND_NETWORK_ALLOWED_ACL": "0.0.0.0/0",
    "HAPROXY_0_VHOST": "my.awesome.domain"
  }

[M-Gregoire]

Thank you again for your answer.

I've deleted my HAPROXY_HTTPS_FRONTEND_HEAD template in marathon and set only the labels from your post.
I have no more errors in my Marathon logs and I am able to access nginx through the vhost, however, I'm presented the certificate from Marathon, not the nginx one so this is not a passthrough (If I access nginx with https://<slave-ip>:<service-port> I'm greeted with the correct certificate).

Could this be a bug in marathon-lb?

[raghu999]

How are you configuring your certificate? Marathon-LB expects the certificate to be formatted in a certain manner with new lines try sed ':a;N;$!ba;s/\n/\\n/g' key.pem and add the cert to ssl-cert environment variable if you are using dcos or pass it to --ssl-certs if you are running marathon-lb as a package

[M-Gregoire]

I'm sorry I'm not sure I understand. If I'm doing SSL/TCP Passthrough with Marathon-lb, it shouldn't be aware of the certificate, It should just pass the TCP traffic without even looking at what's it's routing right?

Why would the certificate have to be added to ssl-cert? The whole point of doing SSL/TCP passthrough is to let the container manage his own certificate, without implicating marathon-lb.

[raghu999]

I am not sure how you are routing that traffic if I am not wrong you are pointing your domain to a CNAME of marathon-lb. From what I understood, All the traffic is being routed to my.awesome.domain which is a vhost of marathon-lb so the first SSL check happens on marathon-lb. If you are not interested in adding the cert to marathon-lb you can also configure HAPROXY_{n}_SSL_CERT option to enable a TLS/SSL to a service port

[M-Gregoire]

Thanks a lot for all your suggestions. I've looked at all of them but it seems I might have been a bit unclear on what I'm trying to achieve. I'm sorry if this is the case.

As you correctly imagined, my.awesome.domain is a CNAME of marathon-lb.

My goal is to do load balancing to my nginx container - currently only one - using SNI extension, without marathon-lb doing anything other than routing the packet.

The setup I'm trying to achieve is very similar to https://www.haproxy.com/blog/enhanced-ssl-load-balancing-with-server-name-indication-sni-tls-extension/.

This would allow me to keep the certificates in the nginx container only, without providing any certificate to marathon-lb nor HAProxy.

The way I see it is something like this (Although I might be misunderstanding something):

1. I request my.awesome.domain, this request is sent to Marathon-lb.

The requested hostname is in clear in HTTPS (https://stackoverflow.com/a/8277348) "but in HTTPS, a TLS handshake takes place first, before the HTTP conversation can begin (HTTPS still uses HTTP – it just encrypts the HTTP messages). Without SNI, then, there is no way for the client to indicate to the server which hostname they're talking to." (Citation from https://www.cloudflare.com/learning/ssl/what-is-sni/)

2. So, using SNI, Marathon-lb knows which hostname is requested (my.awesome.domain).

3. Using the VHOST label ("HAPROXY_0_VHOST": "my.awesome.domain") defined on my nginx container, marathon-lb matches my request to the correct container.

4. The HTTPS request is sent to nginx without ever being opened or checked by Marathon-lb.

Hopefully this makes sense.