SSL Library Error: error:0A000126:SSL routines::unexpected eof while reading

[vinay-mummadi-digicert]

hi team, have been facing this error after upgrading to openssl 3.1.2. earlier we were on openssl 1.1.1u and it was stable. this is the details from the log.

[Tue Oct 17 05:58:16.863055 2023] [proxy:debug] [pid 26272:tid 140171114653440] proxy_util.c(2587): AH00943: AJP: has released connection for (m2be-q2-ap:9009)
[Tue Oct 17 05:58:16.996852 2023] [ssl:debug] [pid 26271:tid 140171156616960] ssl_engine_io.c(1151): [client IP4:55711] AH02001: Connection closed to child 15 with standard shutdown (server a.lab.net:443)
[Tue Oct 17 05:58:17.559642 2023] [ssl:info] [pid 26271:tid 140171165009664] [client IP6:5060] AH01964: Connection to child 14 established (server a.lab.net:443)
[Tue Oct 17 05:58:17.559786 2023] [ssl:info] [pid 26271:tid 140171165009664] (70014)End of file found: [client IP6:5060] AH02008: SSL library error 1 in handshake (server a.lab.net:443)
[Tue Oct 17 05:58:17.559806 2023] [ssl:info] [pid 26271:tid 140171165009664] SSL Library Error: error:0A000126:SSL routines::unexpected eof while reading
[Tue Oct 17 05:58:17.559811 2023] [ssl:info] [pid 26271:tid 140171165009664] [client IP6:5060] AH01998: Connection closed to child 14 with abortive shutdown (server a.lab.net:443)
[Tue Oct 17 05:58:18.424232 2023] [ssl:info] [pid 26271:tid 140171328984832] [client IP5:47602] AH01964: Connection to child 3 established (server a.lab.net:443)
[Tue Oct 17 05:58:18.424413 2023] [ssl:info] [pid 26271:tid 140171328984832] (70014)End of file found: [client IP5:47602] AH02008: SSL library error 1 in handshake (server a.lab.net:443)
[Tue Oct 17 05:58:18.424427 2023] [ssl:info] [pid 26271:tid 140171328984832] SSL Library Error: error:0A000126:SSL routines::unexpected eof while reading

as we're struck , i'm posting it in the QnA section
please suggest . please let me know if you need any additional information on this . thanks.

[mattcaswell]

This usually happens when the remote peer abruptly shuts down the connection without sending a "close_notify" alert which is what you are supposed to do.

1.1.1 and earlier treated this incorrectly. In such a case SSL_read() would fail and SSL_get_error() would return SSL_ERROR_SYS - but with nothing in errno. This was actually incorrect behaviour - it should be treated as SSL_ERROR_SSL with an appropriate error in the error stack. But making that change in 1.1.1 branch was too disruptive so it was introduced in 3.x instead.

So, this behaviour on behalf of the peer has probably always been there, but you are just noticing it now because how OpenSSL reacts to it has changed.

One option for dealing with this is to set the SSL_OP_IGNORE_UNEXPECTED_EOF option. See:

https://www.openssl.org/docs/man3.1/man3/SSL_set_options.html

This causes OpenSSL to treat an unexpected eof as if the peer had closed down gracefully - but make sure you read the docs above around truncation attacks.

If you have control over the remote peer, then another option is to change the remote peer so that it closes down connections gracefully.

[vinay-mummadi-digicert]

Thanks Matt for your response. Where is this setting required. is it in any of the config files? Please suggest. Thanks and regards, Vinay9008855944 

[mattcaswell]

If this is in your own application, then you can modify it to call SSL_set_options on the SSL object.

Otherwise it is possible to set this via the OpenSSL config file. A minimal OpenSSL config file to turn this option on might look like this (untested):

config_diagnostics = 1

openssl_conf = default_conf

[ default_conf ]

ssl_conf = ssl_sect

[ssl_sect]

system_default = ssl_default_sect

[ssl_default_sect]
Options = SSL_OP_IGNORE_UNEXPECTED_EOF

[vinay-mummadi-digicert]

Thanks Matt. I changed the file openssl-3.1.2/apps/openssl.cnf . the other .cnf files were in demo and test dir’s and I ignored them. I added the options line as below. I’ve built the rpm and testing in progress.i think this is what you suggested. I’ll keep you posted .  Thanks again for your guidance.

[mattcaswell]

Better is just to modify the installed config file rather than change it in the source.

Or create a whole new OpenSSL config file specific for your application. Before starting your application ensure the OPENSSL_CONF environment variable is set and pointing to the location of the config file.

[vinay-mummadi-digicert]

Hi Matt,   we tried the changes in the installed config file, restarted the httpd service which uses this openssl and getting the same errors. We added these as suggested. ```` openssl_conf = default_conf [ default_conf ] ssl_conf = ssl_sect [ssl_sect] system_default = ssl_default_sect [ssl_default_sect] Options = SSL_OP_IGNORE_UNEXPECTED_EOF ```` The .cnf file also has this entry # Use this in order to automatically load providers.openssl_conf = openssl_init we tried with both  openssl_init, default_conf values for openssl_conf.  is there any way we can echo the value used for options to confirm that out settings are taken into account? please suggest  thanks,Vinay9008855944

[mattcaswell]

If you are still seeing the errors then the config option did not take effect. This could be for one of three reasons that I can think of:

1. There is an error in the config file that prevents the changes from taking effect

2. The application is using some other config file than the one that you changed

3. The application itself is overriding the config options or explicitly causing the config file not to be loaded.

For (1) you can zip the config file and attach it here if you like. I can take a look to see if it seems sane.

For (2) I am assuming you are using the "system" OpenSSL installation and you are modifying the system OpenSSL config file. You should check whether the application is using the system OpenSSL or its own one. You might also want to check whether the application has some custom OPENSSL_CONF environment variable setting pointing to some other config file.

For (3) you would have to raise an issue with whoever supplied your application.

[vinay-mummadi-digicert]

I’ve attached the cnf file. The changes are from line 39.Pls note that line 17 I’ve commented the “openssl_conf = “ line as it’s repeated after line 39 This is overview of our current appln. I generate the openssl rpm, with changes related to our app path. I then generate the apache httpd related rpm pointing to this new openssl. When the httpd service is started for our webservices framework, the errors are noted. Pls let me know if you need any more details. Thanks-- Vinay9008855944  

[vinay-mummadi-digicert]

Hi Matt, Did you get a chance to review the .cnf file. pls suggest corrections if possible. We’ve noted this additional error in the log.Additional InformationErrors : index="ent" "SSL library error 1 in handshake" "ws. auth.com:443"Customre getting following errorsTC_HttpCommunication::process()[TC_HttpCommunication.cpp:934]:## Error : The connection with the server has been reset or terminated, or an incompatible SSL protocol was encountered. For example, WinHTTP version 5.1 does not support SSL2 unless the client specifically enables it Thanks, Vinay9008855944  

[t8m]

@vinay-mummadi-digicert unfortunately your two last comments which you posted by e-mail are practically illegible due to formatting issues. Please do not use e-mail or possibly this particular e-mail client as it completely messes up the formatting and it is completely unclear what is the new text you've posted and what is the original comment you're replying to.

[vinay-mummadi-digicert]

understood. i'll start posting in the ticket. i've added both 1,2 mails here.
cnf.zip
mattcaswell

part 1: I’ve attached the cnf file.
The changes are from line 39.Pls note that line 17 I’ve commented the “openssl_conf = “ line as it’s repeated after line 39
This is overview of our current appln.
I generate the openssl rpm, with changes related to our app path. I then generate the apache httpd related rpm pointing to this new openssl. When the httpd service is started for our webservices framework, the errors are noted. Pls let me know if you need any more details.

part 2: We’ve noted this additional error in the log.Additional InformationErrors : index="ent" "SSL library error 1 in handshake" "ws. auth.com:443"Customre getting following errorsTC_HttpCommunication::process()[TC_HttpCommunication.cpp:934]:## Error : The connection with the server has been reset or terminated, or an incompatible SSL protocol was encountered. For example, WinHTTP version 5.1 does not support SSL2 unless the client specifically enables it

Thanks-- Vinay
9008855944

[mattcaswell]

The changes are from line 39.Pls note that line 17 I’ve commented the “openssl_conf = “ line as it’s repeated after line 39
This is overview of our current appln.

The config file is not quite right. You've pasted the "openssl_conf =" line into the "[ new_oids ]" section - so it won't get noticed. You need to move all the stuff you pasted in from line 39 to be at line 32 instead.

[vinay-mummadi-digicert]

i moved the content to above "[ new_oids ]" . we're observing the same error.
any option to debug or print the settings, to see if the .cnf file changes are incorporated ?

any other changes to try, please suggest.

[AntonZhernosek]

Having a similar issue connecting to a sql server from a dockerized container with Debian 12 and OpenSSL 3.0.11
Would appreciate the feedback very much because the errors I get look related. I've tried the suggested changes to the .cnf above and it doesn't seem to change anything

[gbryant200]

I'm having the same issue on Ubuntu 22.04

openssl version -a

OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)
built on: Wed Jan 31 18:43:23 2024 UTC
platform: debian-amd64
options: bn(64,64)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/openssl-l59e1T/openssl-3.0.2=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_TLS_SECURITY_LEVEL=2 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
OPENSSLDIR: "/usr/lib/ssl"
ENGINESDIR: "/usr/lib/x86_64-linux-gnu/engines-3"
MODULESDIR: "/usr/lib/x86_64-linux-gnu/ossl-modules"
Seeding source: os-specific
CPUINFO: OPENSSL_ia32cap=0xfeda32035f8bffff:0x800d19e67a9

It seems like the Options = SSL_OP_IGNORE_UNEXPECTED_EOF isn't doing anything.

I tried to run this from the command line: SSL_CONF_cmd(ctx,"Options", "SSL_OP_IGNORE_UNEXPECTED_EOF") but received and unexpected token ctx, "Options"

this fails as well:
openssl s_client -ignore_unexpected_eof -connect github.com:443

from what I can tell, at least for me, this only happens connecting to github.

Any ideas/help would be GREATLY appreciated.

[riwalker]

has anyone found a fix for EOF with OpenSSL 3.x?
seeing the same with Mosquitto, in Ubuntu 22.04