SFTP failed with the error -41 working with Openssl

[depenghu]

I have built libssh2-1.11.0 with Openssl3, the SFTP server is SSHD, the default algorithm is aes-256-gcm, the crypt method is "aes256-gcm@openssh.com".
In the _libssh2_cipher_crypt() function in openssl. c, the value of the "algo" parameter should be EVP_aes_256_gcm, so "is_aesgcm" should be 1, but in fact "algo" is not EVP_aes_256_gcm or EVP_aes_128_gcm, so "is_aesgcm" here is 0 and the decrypted data is incorrect, and the p->packet_length in _libssh2_transport_read() is a randomly large number and returns the error LIBSSH2_ERROR_OUT_OF_BOUNDARY.

I'm not sure why the function address of EVP_aes_256_gcm is different between compilation and runtime, when I call the "algo" function the return value seems to be correct, the nid is 901(NID_aes_256_gcm).
This only happens on Windows (VS2013).
Is this something to do with the build options?

[depenghu]

Changed the following lines:

    const int is_aesgcm = (algo == EVP_aes_128_gcm) ||
                         (algo == EVP_aes_256_gcm);

to :

    EVP_CIPHER* c = (EVP_CIPHER*)algo();
    const int is_aesgcm = (c->nid == NID_aes_256_gcm) || (c->nid == NID_aes_128_gcm);

This error went away.

[rmsh1216]

Can you provide your sftp demo?

[depenghu]

I have built libssh2 in conjunction with CURL, I can reproduce it with the curl command like "curl -k -v -u user:pswd sftp://localhost:22"

[mason-zhu]

I have exactly the same issue now. Really appreciate your help and suggestions.
I work for a product, which consumes libssh2 to connect to SFTP server.
I have this issue on Windows client now after we upgraded libssh2 from 1.10.0 to 1.11.0.

Symptoms:
I built libssh2 1.11.0 with openssl 1.1.1.
SFTP server is deployed on Linux with default configurations.
When using libssh2 1.10.0, we can successfully connect to SFTP server on Windows/Linux client.
After upgrading to libssh2 1.11.0, I can still successfully connect to SFTP server on Linux client.

However, for API call libssh2_session_handshake()
I get error message " “<(-41) Failed to get response to ssh-userauth request" on Windows( Windows Server 2022) client now.
The root cause is same as depengu mentioned:
decrypted data is incorrect, and the p->packet_length in _libssh2_transport_read() is a randomly large number and returns the error LIBSSH2_ERROR_OUT_OF_BOUNDARY.

My code is same as this example:
https://libssh2.org/examples/sftp_write.html
libssh2_session_handshake() returns -41 on Windows now.

My libssh2 1.11.0 Windows build log:

for FILE in `find q:/win64_x64/release/obj/tp.libssh2 -name NMakefile`; do \
    RANDOM_FILE_NAME=${FILE}_$$_${RANDOM}; \
    cat $FILE | \
    sed 's#^CFLAGS=#CFLAGS=/D_WINDLL /D_ITERATOR_DEBUG_LEVEL=0 #g' | \
    sed 's#crypto.lib#libeayM64.lib#g' | \
    sed 's#ssl.lib#ssleayM64.lib#g' | \
    sed 's#zlib.lib#zdll.lib /link /MAP /MANIFEST:EMBED#g' | \
    tee \
    > $RANDOM_FILE_NAME ; \
    mv -f $RANDOM_FILE_NAME $FILE ; \
    chmod 775 $FILE ; \
done
cd q:/win64_x64/release/obj/tp.libssh2/libssh2-*/src && perl -pe 's/\s+\\\n//g' Makefile.inc | perl -pe 's/\s\s+/ /g' > Makefile.inc.new
cd q:/win64_x64/release/obj/tp.libssh2/libssh2-*/src && mv Makefile.inc.new Makefile.inc
cd q:/win64_x64/release/obj/tp.libssh2/libssh2-1.11.0 && Q:/src/im.lac.bts.make.tools/export/RunVisualStudioCommand.pl "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" 64 nmake -f NMakefile WITH_ZLIB=1 OPENSSL_PATH=q:/win64_x64/release/obj/tp.libssh2/deps ZLIB_PATH=q:/win64_x64/release/obj/tp.libssh2/deps TARGET=Release

Q:\win64_x64\release\obj\tp.libssh2\libssh2-1.11.0>call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" amd64 
**********************************************************************
** Visual Studio 2019 Developer Command Prompt v16.11.8
** Copyright (c) 2021 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'

Microsoft (R) Program Maintenance Utility Version 14.29.30138.0
Copyright (C) Microsoft Corporation.  All rights reserved.

	rc /Iinclude /Fo"Release\libssh2.res" src\libssh2.rc
Microsoft (R) Windows (R) Resource Compiler Version 10.0.10011.16384

Copyright (C) Microsoft Corporation.  All rights reserved.


	cl -c /D_WINDLL /D_ITERATOR_DEBUG_LEVEL=0 /D_WINDLL /D_ITERATOR_DEBUG_LEVEL=0 /D_WINDLL /D_ITERATOR_DEBUG_LEVEL=0 /nologo /GL /Zi /EHsc /D_WINDLL /D_ITERATOR_DEBUG_LEVEL=0 /Oi /O2 /Oy /GF /Y- /MD /DNDEBUG /Iinclude /DLIBSSH2_OPENSSL /Iq:/win64_x64/release/obj/tp.libssh2/deps\include /DLIBSSH2_HAVE_ZLIB /Iq:/win64_x64/release/obj/tp.libssh2/deps\include /Fo"Release\\" src\agent.c src\bcrypt_pbkdf.c src\channel.c src\comp.c src\crypt.c src\crypto.c src\global.c src\hostkey.c src\keepalive.c src\kex.c src\knownhost.c src\mac.c src\misc.c src\packet.c src\pem.c src\publickey.c src\scp.c src\session.c src\sftp.c src\transport.c src\userauth.c src\userauth_kbd_packet.c src\version.c 
agent.c
bcrypt_pbkdf.c
channel.c
comp.c
crypt.c
crypto.c
global.c
hostkey.c
keepalive.c
kex.c
knownhost.c
mac.c
misc.c
packet.c
pem.c
publickey.c
scp.c
session.c
sftp.c
transport.c
Compiling...
userauth.c
userauth_kbd_packet.c
version.c
	cl -o libssh2.dll /D_WINDLL /D_ITERATOR_DEBUG_LEVEL=0 /D_WINDLL /D_ITERATOR_DEBUG_LEVEL=0 /D_WINDLL /D_ITERATOR_DEBUG_LEVEL=0 /nologo /GL /Zi /EHsc /D_WINDLL /D_ITERATOR_DEBUG_LEVEL=0 /Oi /O2 /Oy /GF /Y- /MD /DNDEBUG /Iinclude /DLIBSSH2_OPENSSL /Iq:/win64_x64/release/obj/tp.libssh2/deps\include /DLIBSSH2_HAVE_ZLIB /Iq:/win64_x64/release/obj/tp.libssh2/deps\include /LD Release/agent.obj Release/bcrypt_pbkdf.obj Release/channel.obj Release/comp.obj Release/crypt.obj Release/crypto.obj Release/global.obj Release/hostkey.obj Release/keepalive.obj Release/kex.obj Release/knownhost.obj Release/mac.obj Release/misc.obj Release/packet.obj Release/pem.obj Release/publickey.obj Release/scp.obj Release/session.obj Release/sftp.obj Release/transport.obj Release/userauth.obj Release/userauth_kbd_packet.obj Release/version.obj Release\libssh2.res  q:/win64_x64/release/obj/tp.libssh2/deps\lib\libeayM64.lib q:/win64_x64/release/obj/tp.libssh2/deps\lib\ssleayM64.lib q:/win64_x64/release/obj/tp.libssh2/deps\lib\zdll.lib /link /MAP /MANIFEST:EMBED ws2_32.lib user32.lib advapi32.lib gdi32.lib
cl : Command line warning D9035 : option 'o' has been deprecated and will be removed in a future release
   Creating library libssh2.lib and object libssh2.exp
Generating code
Finished generating code

[fborges22]

I am too working in an application that is using the LIBSSH2 for providing the SFTP functionality. Having the same issue on Windows.

[willco007]

@depenghu what is the value of algo when it fails?

[depenghu]

@willco007
algo : 0x00007FF89EC5F5B0
EVP_aes_256_gcm : 0x00007FF85BF92590
EVP_aes_128_gcm : 0x00007FF85BF96A05

[willco007]

@dfandrich Any ideas on this one?

[mason-zhu]

Does anyone know the ETA of the official fix? Is there any workaround for the issue? I need these information in my project. Thanks.