mod_sftp has low performance

[vincdrouin]

What I Did

I compared SFTP upload performabce through sshd (openssh 8.5) & proftpd mod_sftp (1.3.8 / 1.3.8a).
The SFTP client used is FileZilla.

Upload rate to an sshd server is 106 Mio/s.
Upload rate to a proftpd server is only 64 Mio/s.

The host is the same, and I don't run both servers simultaneously.

What I Expected/Wanted

I expected upload rate to a proftpd server to be similar.

ProFTPD Version and Configuration

proftpd 1.3.8 & 1.3.8a
Configuration key points:

• SFTPCiphers aes256-ctr aes192-ctr aes128-ctr
• SFTPDigests hmac-sha2-512 hmac-sha2-256

These parameters are the same in sshd_config.
If I remove SFTPCiphers from proftpd configuration, the upload rate goes up to 80Mio/s which is a bit better but still way lower than sshd.

[severach]

FileZilla 3.66.1 does not offer hmac-sha2-512 so you are using hmac-sha2-256. Use SSHAudit to find out what your client supports. Check the SFTPLog to verify your ciphers and ensure that compression is off.

With FileZilla compression on my transfers were 5-15MiB/s. I turned compression off to get better performance. Compression must be turned off if you're getting 60. Other clients do not use compression by default.

Test bed:
Gigabit LAN, not over Internet.
Client Linux x64: E3-2146G AES-NI
Client Windows x64: E3-1245 v2 AES-NI
Server: E3-1245 v3 AES-NI, proftpd 1.3.9rc2 (git)
File size: 6,326,614,695
Direction: put to server

All measurements are taken from client before the first rekey. FileZilla rekeys at 10% and does not remove the rekey time from the speed. sftp does not rekey.

Client: Linux FileZilla 3.66.1 x64, compression off to proftpd
aes128-gcm@openssh.com: 100MiB/s, exactly spec for a Gigabit network.
aes256-gcm@openssh.com: 95MiB/s
chacha20-poly1305@openssh.com: 90Mib/s
aes256-ctr+hmac-sha2-256: 75MiB/s

Client: Linux OpenSSH_9.5p1 x64 to proftpd
sftp -o Compression=no -o Ciphers=chacha20-poly1305@openssh.com: 110MB/s
sftp -o Compression=no -o Ciphers=aes128-gcm@openssh.com: 110MB/s
sftp -o Compression=no -o Ciphers=aes256-gcm@openssh.com: 105MB/s
sftp -o Compression=no -o Ciphers=aes256-ctr -o Macs=hmac-sha2-256: 109MB/s

Client: Linux PuTTY psftp 0.79 x64, does not show a speed. Tested with time:
# time psftp -l xxx -pw xxx -b psup.txt "rm file; put file; quit"
aes256-ctr+hmac-sha2-256: 6,326,614,695/3m45.621s/1024/1024 = 27MiB/s
aes256-gcm@openssh.com: 6,326,614,695/3m35.621s/1024/1024 = 28MiB/s
aes128-gcm@openssh.com: 6,326,614,695/3m32.621s/1024/1024 = 28MiB/s
chacha20-poly1305@openssh.com: 6,326,614,695/3m35.621s/1024/1024 = 28MiB/s

Client: Windows x64 WinSCP 6.2beta x86 to proftpd
aes256-ctr+hmac-sha2-256: 60MiB/s
aes256-gcm@openssh.com: 20MiB/s
aes128-gcm@openssh.com: 20MiB/s
chacha20-poly1305@openssh.com: 24Mib/s

Client: Windows x64 Bitvise 9.31 x86 to proftpd
chacha20-poly1305@openssh.com: 115MB/s
aes256-gcm@openssh.com: 115MB/s
aes128-gcm@openssh.com: 115MB/s
aes256-ctr+hmac-sha2-256-etm: 75MB/s

A sftp benchmark site I can't find now found that AES+HMAC was slower than AES+UMAC and AES-GCM which were fast and equal speed. AES-CTR might go faster if paired with umac-128-etm@openssh.com which FileZilla and Bitvise do not offer. For better performance try an AEAD cipher AES-GCM or CHACHA20.

proftpd sftp performance isn't slower than the fastest client. All performance problems are at the client. Clients that use OpenSSL and libsodium are fast. Clients that use their own code are not.

These are the conf lines I used. Bitvise and OpenSSH can specify protocols so don't need these lines.

    # Test FileZilla 3.66.1 performance,default is aes256-gcm@openssh.com
    #SFTPClientMatch "^FileZilla.*"              sftpCiphers aes256-gcm@openssh.com
    #SFTPClientMatch "^FileZilla.*"              sftpCiphers aes128-gcm@openssh.com
    #SFTPClientMatch "^FileZilla.*"              sftpCiphers aes256-ctr
    #SFTPClientMatch "^FileZilla.*"              sftpCiphers chacha20-poly1305@openssh.com
    #SFTPClientMatch "^FileZilla.*"              sftpDigests hmac-sha2-256

    # Test WinSCP 6.2beta performance, default is aes256-ctr+hmac-sha2-256
    #SFTPClientMatch "^WinSCP.*"              sftpCiphers aes256-gcm@openssh.com
    #SFTPClientMatch "^WinSCP.*"              sftpCiphers aes128-gcm@openssh.com
    #SFTPClientMatch "^WinSCP.*"              sftpCiphers aes256-ctr
    #SFTPClientMatch "^WinSCP.*"              sftpCiphers chacha20-poly1305@openssh.com
    #SFTPClientMatch "^WinSCP.*"              sftpDigests hmac-sha2-256

    # Test PuTTY psftp 0.79 performance, default is aes256-ctr+hmac-sha2-256
    #SFTPClientMatch "^PuTTY.*"              sftpCiphers aes256-gcm@openssh.com
    #SFTPClientMatch "^PuTTY.*"              sftpCiphers aes128-gcm@openssh.com
    #SFTPClientMatch "^PuTTY.*"              sftpCiphers aes256-ctr
    #SFTPClientMatch "^PuTTY.*"              sftpCiphers chacha20-poly1305@openssh.com
    #SFTPClientMatch "^PuTTY.*"              sftpDigests hmac-sha2-256

---

Did some more testing, same clients, more servers.

Test Bed:
Windows Server: E3-1245 v2 AES-NI (same as Windows client above so Windows client had to move to another machine).
Alternate Windows Client: E3-1246 v3 AES-NI

Linux OpenSSH 9.5
Windows nsoftware SFTP server 2022 22.0.8669
Windows Bitvise Server 9.31

Added test for aes256-ctr + umac-128-etm@openssh.com where available.

This is a repeat of the above test in table form.

proftpd	FileZilla	OpenSSH	WinSCP	PuTTY	Bitvise
chacha20-poly1305@	90	110	24	28	115
aes256-gcm@	95	105	20	28	115
aes128-gcm@	100	110	20	28	115
aes256-ctr+hmac-sha2-256	75	109	60	27	75
aes256-ctr+umac-128-etm@		105

OpenSSH	FileZilla	OpenSSH	WinSCP	PuTTY	Bitvise
chacha20-poly1305@	110	107	22	99	115
aes256-gcm@	110	110	20	104	115
aes128-gcm@	110	110	19	108	115
aes256-ctr+hmac-sha2-256	110	108	52	107	72
aes256-ctr+umac-128-etm@		108

nSoftware	FileZilla	OpenSSH	WinSCP	PuTTY	Bitvise
chacha20-poly1305@	27	32	23	20	14
aes256-gcm@	24	28	20	18	14
aes128-gcm@	28	32	20	20	14
aes256-ctr+hmac-sha2-256	48	55	48	27	14
aes256-ctr+umac-128-etm@		23

nSoftware is a slow starter. Takes till about 30% to get the max speed.

Bitvise	FileZilla	OpenSSH	WinSCP	PuTTY	Bitvise
chacha20-poly1305@	110	108	29	48	110
aes256-gcm@	110	110	25	52	110
aes128-gcm@	110	109	25	49	110
aes256-ctr+hmac-sha2-256	80	86	80	36	80

Here are the lines I used in /etc/ssh/sshd_config

  # Performance testing
  #Ciphers chacha20-poly1305@openssh.com
  #Ciphers aes256-gcm@openssh.com
  #Ciphers aes128-gcm@openssh.com
  #Ciphers aes256-ctr
  #MACS umac-128-etm@openssh.com
  #MACS hmac-sha2-256

[jlusterlvx]

I'd like to chime in with similar findings, I'm seeing slow speeds with sftp versus scp. Given that sftp as a subsystem is more secure, I would prefer not to drop it for speed. (I saw a post from RedHat that they are dropping the scp default subsystem )

Client OpenSSH 7

Try 1: Regular Usage

scp -o Ciphers=aes256-ctr -o Macs=hmac-sha2-256 workdir/small-512.dat <USER>@<PROFTPD>:

small-512.dat 51% 264MB 94.5MB/s 00:02 ETA
small-512.dat 100% 512MB 23.3MB/s 00:22

OpenSSH 8

Try 2: Regular usage

scp -o Ciphers=aes256-ctr -o Macs=hmac-sha2-256 workdir/small-512.dat <USER>@<PROFTPD>:
small-512.dat 12% 65MB 3.7MB/s 02:02 ETA
[CTRL-C Cancelled]

Try 3: Addl '-O' speeds up significantly

scp -O -o Ciphers=aes256-ctr -o Macs=hmac-sha2-256 workdir/small-512.dat <USER>@<PROFTPD>:
small-512.dat 100% 512MB 17.3MB/s 00:29

Thanks in advance, I'm hoping there is a straightforward fix.

[cfiehe]

Problem maybe related to #1288 (comment).

[severach]

On the previous test the speed limit imposed by the gigabit network made OpenSSH sshd and proftpd look about the same. I improved the test and found that OpenSSH has a significant performance advantage.

Test methodology:

# perftest.sh
# We want to know the speed of an sftp server which may be faster than disks and networks.

# For this we must:
# Use 127.0.0.1 to avoid network speed limits. Download must be on same machine as server.
# get sparse file to avoid disk read speed limits and caching
# get to /dev/null to avoid disk write limits and caching
# Use OpenSSH sftp as the client. It is the fastest and we can specify algos on the command line
# We'll use another client if someone can find one.

Plenty of OpenSSH Benchmarks but they all missed at least one of the above rules. Like Famzah, this test also demonstrates the dramatic differences between processors and platforms. My E-2146G desktop runs ahead of pack pushing more than 10Gbe speeds using the latest and best algorithms, no arcfour needed.

There is a performance difference between OpenSSH and proftpd when you get above 1Gbe. It would be good to find out how OpenSSH goes so fast.

Some sftp clients and servers are so slow that gigabit Ethernet and hard drives are not bottlenecks. Some clients perform differently with different servers of similar speed. Most clients are speed optimized for OpenSSH server. OpenSSH sftp can specify algos on the command line, logs results, and consistently outperformed all other clients. OpenSSH sftp client was used for all tests.

aes-gcm and umac are the speed leaders. Some algos are so slow that faster processors make no noticeable improvement.

sha1 is faster than md5.

My Haswell laptop keeps up with Haswell workstations, handily beating earlier workstations.

Windows performance was substantially lower than Linux performance with both the Windows 10 build and the CYGWin build.

Note that chacha20 and aes-gcm do not use a separate MAC. Ignore the column header. All measurements are as reported by OpenSSH sftp client: Megabytes/second (not megabits).

Entries are sorted on their aes256-gcm speed.

Header CPU	Note	a256 gcm	a128 gcm	cha	a256c u128	a256c h256
E3-1245 V2	svr_noftware2022 cli_cyg9.5	30	34	36	49	75
E3-1245 V2	svr_win8.1 cli_cyg9.5	116	118	83		47
G1620T	ssh_9.5 -aesni	128	141	171	135	88
E3-1245 v3	svr_proftpd cli_ssh9.5	142	144	126	136	106
E3-1245 V2	svr_cyg9.5 cli_win8.1	143	152	122		52
E3-1245 V2	svr_bitvise9.31 cli_cyg9.5	152	156	150		88
E3-1245 v3	svr_proftpd cli_ssh9.5 mitigations=off	153	154	135	149	113
E3-1245 V2	ssh_win8.1	164	156	115		46
i3-3220	ssh_9.5 -aesni	194	235	271	122	116
i3-3220	ssh_9.4 -aesni	206	249	282	222	135
E3-1245 V2	ssh_cyg9.5	224	232	175	181	139
E3-1245 V2	ssh_cyg9.5 bad	229	225	178	181	142
E5-2695 v2	ssh8.8	250	263	195	240	127
i5-4310M	ssh9.5 +aesni	450	483	312	408	177
i5-2400	ssh_9.5 +aesni	492	518	316	521	189
E3-1245 V2	ssh7.5	497	512	217	425	220
E31245	ssh_9.5	526	518	314	525	205
E3-1246 v3	ssh7.4 bad	541	664	208	409	249
E3-1245 V2	ssh_9.5	552	600	348	600	235
i3-4160	ssh_9.5 +aesni	560	595	357	506	251
E3-1246 v3	ssh7.4	563	610	202	430	234
E3-1245 v3	ssh9.5	825	895	495	729	291
E-2146G	ssh9.5	966	989	511	768	333
E3-1245 v3	ssh9.5 mitigations=off	976	1000	524	741	308
E-2146G	ssh8.8	1100	1100	586	692	333
E-2146G	ssh7.5	1200	1300	329	777	367
E-2146G	ssh9.5 mitigations=off	1400	1400	682	982	383

Here's a couple interesting full tables.

Lenovo ThinkServer TS140, E3-1245 v3, Arch Linux
Client: OpenSSH_9.5p1, OpenSSL 3.1.4 24 Oct 2023
Server: OpenSSH_9.5p1, OpenSSL 3.1.4 24 Oct 2023
Linux 5.10.202

OpenSSH	umac-128-etm@	hmac-sha2-512-etm@	hmac-sha2-256-etm@	hmac-sha1	hmac-md5
chacha20-poly1305@	495
aes256-gcm@	825
aes128-gcm@	895
aes256-ctr	729	385	291	485	423
aes192-ctr	749	383	297	468	409
aes128-ctr	757	384	300	485	418
aes256-cbc	340	236	203	272	246
aes192-cbc	368	250	217	292	265
aes128-cbc	405	270	225	325	283
3des-cbc	26	26	25	26	26

Lenovo ThinkServer TS140, E3-1245 v3, Arch Linux
Client: OpenSSH_9.5p1, OpenSSL 3.1.4 24 Oct 2023
Server: ProFTPD Version 1.3.9rc2 OpenSSL 3.1.4
Linux 5.10.203

proftpd	umac-128-etm@	hmac-sha2-512-etm@	hmac-sha2-256-etm@	hmac-sha1	hmac-md5
chacha20-poly1305@	126
aes256-gcm@	142
aes128-gcm@	144
aes256-ctr	136	115	106	125	116
aes192-ctr	135	115	106	122	117
aes128-ctr	136	115	107	125	116
aes256-cbc	93	82	76	87	85
aes192-cbc	95	84	78	90	87
aes128-cbc	101	87	80	92	88
3des-cbc	21	20	20	21	21

Included are the full set of results and the test program. The test program generates these markdown tables. The test program is a shell script. It runs in Linux or CYGWIN bash. It includes the Windows CMD lines to generate sparse files.

sftp_perftest.zip

[Castaglia]

@severach thanks for the thorough testing! I'm wondering what differences we might see, specifically with newer OpenSSH clients, now that mod_sftp implements the limits@openssh.com SFTP extension (see Issue #1798), which is used by newer OpenSSH clients such that they use larger SFTP READ, WRITE packet lengths (and more outstanding SFTP requests).