openssl bug when using Node.js 10.x or Node.js 11.x but Node.js 8.x is fine

[hsgreen]

I observed a very strange bug when I recently updated from using Node.js 8 to 11 (I also then tried 10.x and it had the same issue as 11.x).

I'm running in a Centos 7 (latest) docker container and install using these instructions.

I have a custom node module that I have built (I rebuild each time I install a new Node.js version), that uses another custom library that I have built that uses OpenSSL. The version of OpenSSL at the system level is standard Centos 7 OpenSSL 1.0.2k-fips. Within this custom library linked in as a shared library from my custom node module it makes the following openssl call:

m_certificate = X509_new();

On all versions of node this call succeeds and m_certificate is not NULL. However, the problem is on Node.js 10.x and 11.x one of the critical members of that object is still set to NULL after that call. This is the cert_info member. When cert_info is NULL it causes subsequent methods to set/update this certificate to seg fault, such as X509_time_adj_ex(X509_get_notBefore(m_certificate), 0, 0, &nowTime.tv_sec) because nearly all these OpenSSL methods/macros blindly call m_certificate->cert_info->....

I thought maybe it was a bug in the system installed OpenSSL, but when I downgraded to use the latest 8.x Node.js version, then the problem went away. It must be something with the OpenSSL configuration or build when using Node.js 10 and 11.

Is there anything in Node.js 10 and 11 that interacts with OpenSSL differently than 8.x or does the pre-built binary distros use a statically linked OpenSSL version that may be different than 8.x and may introduce this odd failure to allocate the cert_info member when calling X509_new()?

[bnoordhuis]

Your issue/question is essentially "can I use two copies of openssl in the same process?" and it's not the first time it's come up, check this repo and nodejs/help.

The answer is 'yes' but you need to be very meticulous about initializing your copy of openssl properly and ensuring that you never mix calls or data structures.

The first thing I'd do is check with a debugger whether that call to X509_new() calls into node or libcrypto.so and double-check if that is the expected location.

[bnoordhuis]

Is there anything in Node.js 10 and 11 that interacts with OpenSSL differently than 8.x or does the pre-built binary distros use a statically linked OpenSSL version that may be different than 8.x

To answer your specific question: yes, v10.x and v11.x ship openssl 1.1.0j, v8.x openssl v1.0.2q.

[hsgreen]

Thanks that makes sense. I had installed a custom openssl of the same system version built with debug symbols to try and troubleshoot, but it always seemed to go to the Node binary install openssl calls. I'll have to look around to figure out how to build my standalone library that needs openssl using a different openssl version than what node is using and to get them to all play nice. Thanks for confirming and answering!

[hsgreen]

I think this may actually be a more serious design flaw in NodeJS after doing more research and experimentation on this issue.

I found this other thread that is suffering the same problem here.

It seems that the current approach in NodeJS to export the OpenSSL library symbols prevents any native addons from using ANY library on the system that depends on OpenSSL unless the OpenSSL libraries have 100% ABI compatibility.

This would mean I can't make a native addon that uses the system libcurl without risk of segfaults if I upgrade NodeJS to a version beyond the one that has ABI compatibility with the system OpenSSL.

It seems a little ridiculous to expect developers to have to rebuild their custom libraries and all those libraries dependent libraries that use OpenSSL to match the same OpenSSL version used in NodeJS in order to be used as an addon every time you update NodeJS version that includes a new OpenSSL ABI.

Would it be so hard for NodeJS to just use the system provided openssl library and write code like the rest of us that accommodates varying API version differences of OpenSSL across its versions?

[mscdex]

Would it be so hard for NodeJS to just use the system provided openssl library and write code like the rest of us that accommodates varying API version differences of OpenSSL across its versions?

You can already build node against a system copy of OpenSSL, provided it's compatible with node's usage of the OpenSSL API.

[bnoordhuis]

It seems that the current approach in NodeJS to export the OpenSSL library symbols prevents any native addons from using ANY library on the system that depends on OpenSSL unless the OpenSSL libraries have 100% ABI compatibility.

No, you can do that, you just need to be careful with how you link and in what order (and of course the other things I mentioned in my first comment.)

Whether you should is another question. Ask yourself if there are compelling reasons not to link against node's bundled copy of openssl.

Would it be so hard for NodeJS to just use the system provided openssl library and write code like the rest of us that accommodates varying API version differences of OpenSSL across its versions?

We used to do that in the past. There's a reason we moved away from that. :-)

[hsgreen]

No, you can do that, you just need to be careful with how you link and in what order (and of course the other things I mentioned in my first comment.)

Could you point to a reference detailing what you mean by "careful with how you link and in what order"? Due to OpenSSL being a C library the only solutions I've found involve complete remapping of the symbols, which then goes back to essentially requiring a rebuild of all the other system installed libraries we may want in a node addon that happen to use the system OpenSSl library as well.

I even compiled the same version of OpenSSL that the system uses as a static library with debug symbols, linked to my shared library. When running my shared library in a stand alone application it works fine (I can step into the OpenSSL methods etc). When I go and use the shared library in a node addon, it still uses the OpenSSL symbols exposed by the node binary. The interesting thing is even when I'm in gdb and call info symbol SSL_library_init from a break point inside my shared library, gdb says it is a symbol in my shared library. But when I go to step into the OpenSSL methods it jumps over like no debug symbols are available (effectively using the nodejs binary exported symbols for OpenSSL).

I certainly appreciate how most of this is due to the design of OpenSSL, being a C only library, etc. I also realize the pitfalls of using the host system OpenSSL (way out of date, full of security vulnerabilities, etc). I just wonder if you could actually point to the specific method you had in mind that would be a solution to this for having different ABI incompatible versions of OpenSSL in use in this case.

I do realize one could recompile NodeJS to match the system OpenSSL. I'm just trying to find a way that maximizes compatibility between standard system libraries from the base OS maintained repos and the pre-built distributions of nodejs for those systems. One of the things I appreciate about NodeJS is how easy it is to install/distribute and update with the pre-built binary distributions and npm. It would be unfortunate to have to rebuild NodeJS for every minor update that comes out.

Whether you should is another question. Ask yourself if there are compelling reasons not to link against node's bundled copy of openssl.

The compelling reason is it would then require custom builds of a slew of other 3rd party libraries that I can otherwise just use from installation via the system package manager and I have other applications that also use my custom library outside of node on the same system.

We used to do that in the past. There's a reason we moved away from that. :-)

Again if there are "careful" ways to link to make this work, wouldn't there be a "careful" way that the standard NodeJS distribution could be linked to OpenSSL to avoid exposing all the OpenSSL symbols?

[bnoordhuis]

I regret that I don't have time right now to answer in full. I'll try to come back to this later but a quick comment apropos this:

I even compiled the same version of OpenSSL that the system uses as a static library with debug symbols, linked to my shared library. [..] When I go and use the shared library in a node addon, it still uses the OpenSSL symbols exposed by the node binary.

That sounds impossible; probably something went wrong during the build.

If you linked your addon.node to a static libcrypto.a, then nm addon.node should print no undefined openssl symbols (no symbols marked U or u and with no address.)

[hsgreen]

That sounds impossible; probably something went wrong during the build.

If you linked your addon.node to a static libcrypto.a, then nm addon.node should print no undefined openssl symbols (no symbols marked U or u and with no address.)

I think maybe I was not clear in describing what happened.

1. Linked a libcrypto.a that I built with debug symbols to my shared library "A"
2. When I run a test executable that uses shared library "A" through gdb I can successfully step into the openssl methods from libcrypto.a
3. THEN I linked my shared library "A" as a shared library to my node addon "B".
4. When I run node in gdb using my addon "B", I can step into my shared library code, but when that code is about to call a method from OpenSSL, gdb shows the symbol as being in my shared library "A", however if I try to "step" into the OpenSSL method in this case it skips over it as if it was using the symbol from the node binary.

Perhaps I need to explicitly link the node addon "B" to the same libcrypto.a file, even though it is already technically inside my shared library "A"?

But even this solution is not optimal because it does not address the use of other 3rd party libraries that my node addon "B" would like to use, that also depend on the system openSSL library, like libcurl.

So regardless, there should be something that can be done to the node build process that prevents it from exporting the OpenSSL symbols that it has statically linked to.