openssl_fopen() causes segmentation fault (access violation) under some conditions on Windows

[michael-o]

This has been verified with OpenSSL 3.0.11, but seems to happen with any OpenSSL version on Windows.
The actual, lengthly report comes from https://lists.apache.org/thread/m1dbj3w1x1oqftqsbj7jbnvkm2073x1o
The code in question is here: https://github.com/apache/tomcat-native/blob/4eaa5c93c632f1ea80e889b5458d5b95f57b59a2/native/src/sslcontext.c#L711 where the first argument is not NULL while the second one is NULL.

Issue: Upon calling this function (SSL_add_file_cert_subjects_to_stack()) OpenSSL causes a "EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000001e5288f772e, pid=17588, tid=0x0000000000003aec". The reason is that the NULL pointer is passed along and reaches openssl_fopen(). While fopen() on Windows and POSIX-like properly sets errno on NULL pointer. The problem is the Windows code:

openssl/crypto/o_fopen.c

Lines 41 to 42 in a6afe2b

	# if defined(_WIN32) && defined(CP_UTF8)
	int sz, len_0 = (int)strlen(filename) + 1;

strlen() crashes upon a NULL pointer.

From my PoV:

• The behavior is inconsistent across platforms
• The higher level functions do not tell whether NULL pointer are valid input or not

Maybe the easiest way would be a NULL check and pass along to fopen() where it can set the errno properly.

[t8m]

I do not think that calling fopen with NULL pathname is actually valid. Yes, it might not crash for some implementations but it is not an explicitly documented behavior.

I believe we should check this in SSL_add_file_cert_subjects_to_stack() and fail if the second argument is NULL.

[michael-o]

I do not think that calling fopen with NULL pathname is actually valid. Yes, it might not crash for some implementations but it is not an explicitly documented behavior.

At least both Windows and FreeBSD gave me a reasonable errno, but true if it is not documented do not rely on it.

[paulidale]

OpenSSL doesn't set errno anywhere else for errors. It has it's own error mechanism. Why should this case be different?

[michael-o]

OpenSSL doesn't set errno anywhere else for errors. It has it's own error mechanism. Why should this case be different?

I was solely referring to fopen() on those two platforms and its behavior with a NULL argument. No OpenSSL code involved.