Replacing OpenSSL implementation in nginx

[kevinburke1]

Hi, in the absence of a better place to write about this/get feedback, I figured I'd post here. Whether or not this is eligible for a bounty I think it would probably be a good thing for the ecosystem and wanted to try to make progress on it.

I looked last night at the feasibility of replacing the OpenSSL implementation in nginx. My initial idea was to use mesalink - github.com/mesalock-linux/mesalink - since this would allow you to compile in rustls without needing to change the nginx source code at all. If I got it working, I could put together a prototype, benchmark it, test it in our development environment and then write up instructions on how to build it.

Unfortunately nginx uses the SSL_CTX_set_options OpenSSL API, which is not currently supported by mesalink, and it doesn't look like the mesalink library has momentum, see mesalock-linux/mesalink#51 for details.

I guess my options right now would be

• Try to add support for SSL_CTX_set_options in mesalink
• Look for a different drop-in OpenSSL replacement. I'm not sure this exists, searching Github for SSL_CTX_set_options in Rust source code mostly reveals things that let you call OpenSSL or BoringSSL - the C sources - from Rust which doesn't seem like what we want.
• Edit the nginx sources to add a TLS compatibility layer, and then add a new TLS backend, similar to the approach here: curl/curl@246399a. I am probably not proficient enough to be able to do this.

Mainly I am looking for advice at this point - I am good at fighting build systems and getting stuff to compile and less good at understanding what's currently possible to do in the crypto ecosystem, and at writing C code - I've probably written under 2,000 lines of C in my life.

[alex]

So, my view is that the healthiest long term direction is the curl approach of a TLS abstraction layer (well, really ideal would be just switching, but if you still care about supporting OpenSSL you need a layer). That direction is likely to lead you to simpler APIs, and an even longer-term opportunity to not have to support the OpenSSL API. The problem with the MesaLink approach IMO is that you have to try to shoehorn everything into OpenSSL which, whatever it's merits or demerits, is not designed to be an abstract TLS interface, it's an interface to how OpenSSL works! That means there's always going to be pain, and I'm amazed mesalink works as well as it does.

[reaperhulk]

It has been our experience that work like this is far more effective when you can convince upstream to work with you. This allows a long term path for driving improvements without having to color exclusively within the lines of an existing C API (with the accompanying requirement to be bug-compatible). MesaLink is a fascinating approach to a "zero cost" Rust replacement, but approaches like this are very difficult to expand into a sustainable model (e.g. could you convince a Linux distribution to ship this with their nginx?).

That's not to say there's no value in this effort, but I'd encourage you to try to engage with upstream about developing a strategy for incorporating memory safe code into the tree.

[kevinburke1]

OK, I sent a message to the development mailing list and will see what they say: http://mailman.nginx.org/pipermail/nginx-devel/2021-February/013806.html

[kevinburke1]

From Maxim Dounin:

We already support three distinct SSL libraries: OpenSSL,
LibreSSL, and BoringSSL. It is unlikely that support for other
libraries will be considered: that's already way more than it's
comfortable to support.

We are, however, open for patches to improve portability of the
SSL code, such as providing ngx_ssl_() wrapper functions in (very
few, actually) places where SSL library interfaces are used
directly rather than via appropriate ngx_ssl_() wrappers.

[reaperhulk]

This seems like a reasonable first step. Defining abstracted interfaces would allow a Rust TLS implementation without the drawbacks we've been talking about, and once a rust backend is available it becomes possible to have a distribution choose to ship it while remaining entirely compatible with upstream nginx.