Upload within Parallel.ForEach() #1533
Comments
|
I generally love reports about TLS Session Resume failures, however: A "real" session resume problem normally does not even allow the first file transfer to work. So, this log is really really interesting. But, please look at all the closed issues marked with the "Session Resume" tab and you will note that this looks somewhat fishy. Is access to the FileZilla (set to debug level) log a viable option, can you post it? Normally, using TLS 1.2, session resume can work, but not always. Strange to see it working and then not working here (at the same time). On the other hand, multithreading FTP operations with each on having its own FtpClient is perfectly acceptable. A number of things come to mind:
*1) I have a parallel up- and download test program I run regularly, it does 50 operation simultaneously (albeit agains a proFTPd server). No problems. Since you can reproduce this and you have supplied the code, I can try the same here (also against a FileZilla) and that will also give us another angle to analyze this. |
|
Thanks for your response. I was hoping for a "You're doing it wrong" answers, so I'm willing to try anything.
Thanks again. |
|
Attached Server Log (from my local FileZilla server 1.8.1) |
|
Good, so you can run this on your own server. Later I will do some testing also and we will see where that gets us. Pity about GnuTLS, what is the actual things preventing use? |
|
OK, I have been playing around with parallel uploads. I get the exact same symptoms as you with FileZilla, but I would not necessarily say it is FileZilla's fault (it can be very strict and sometimes ornery, but all accords to the RFCs instead of "strange" custom behaviour). Here are my results: Encryption OFF: Encryption: Explicit SslStream Encryption: Explicit FluentFTP.GnuTLS Conclusion: .NET SslStream Session Resume is requested by FileZilla. The way SslStream TLS1.2 implements Session Resume is not correct. FileZilla(GnuTLS) rejects this. proFTPd seemingly does not request Session Resume for TLS1.2 and thus it works. To use TLS1.3, you would need at least Win11 and .NET6+ (there are rumours of a late WIN10, but YMMV), or use FluentFTP.GnuTLS. The first upload, or sequential uploads work, because in those cases the session data used for the Session Resume is unique and not (incorrectly) shared between the sessions, I can only surmise. And now go figure :-) - like trying to see at what level this is happening: Running two separate executables at the same time, for example, I have never tested that. |
FanDjango
added
cantfix
FluentFTP.GnuTls Candidate
SSL Session Resume
|
Thanks for checking out my scenario. Very much appreciated. I've got signoff to try FluentFTP.GnuTLS now, and I've tried. With limited success. Once again, with no threads/parallelism it works fine. However, there seems to be issues with concurrency. The issues seems to be inside GetCachedHostAddresses
Sometimes it outright refuses to do it saying "Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct." Other times it simply says that a key already exists with that hostname. Are the settings in your tests the same as those in the repository here for the test? In case I'm missing out something that magically makes this work. I've looked through the issues and there were historically settings for concurrency, but they don't seem to be there now. Thanks! |
First of all, this error is a totally different one from the one you initially reported. I wonder how you got that to happen, given that the Status.CachedHostIpads is local to the ftpClient object. I looked at your code and I don't see anything obvious, here is mine: Random r = new Random();
await Task.WhenAll(Enumerable.Repeat(1, 5).Select(async _ => {
using (var client = new AsyncFtpClient("host", "****", "********")) {
client.Config.EncryptionMode = FtpEncryptionMode.Explicit;
client.Config.DataConnectionType = FtpDataConnectionType.PASV;
client.Config.DownloadDataType = FtpDataType.Binary;
client.Config.ValidateAnyCertificate = true;
client.Config.Navigate = FtpNavigate.SemiAuto;
//client.Config.SslSessionLength = 0;
client.LegacyLogger = FTPLogEvent;
//client.Config.CustomStream = typeof(GnuTlsStream);
//client.Config.CustomStreamConfig = new GnuConfig() {
// //SecuritySuite = GnuSuite.Normal,
// //SecurityOptions = new List<GnuOption> {
// // new GnuOption(GnuOperator.Include, GnuCommand.Protocol_All),
// // new GnuOption(GnuOperator.Exclude, GnuCommand.Protocol_Tls10),
// // new GnuOption(GnuOperator.Exclude, GnuCommand.Protocol_Tls11),
// // },
// //SecurityProfile = GnuProfile.None,
// //AdvancedOptions = new List<GnuAdvanced> {
// // GnuAdvanced.NoExtensions,
// // },
// SetALPNControlConnection = string.Empty,
// SetALPNDataConnection = string.Empty,
// //LoadLibraryDllNamePrefix = "D:\\temp\\",
// LogLevel = 0,
// LogMessages = GnuMessage.All,
//};
await client.Connect();
await client.UploadFile("D:\\testdir\\t.t", "\\home\\****\\t" + r.Next() + ".cs", FtpRemoteExists.Overwrite, true);
await client.Disconnect();
}
}
)); |
|
This runs 5 parallel uploads of one local file to different names on the server and with or without GnuTLS depending on uncommenting the config statements. It is .NET 6 on my side at the moment and I have not tested any others. |
|
Meanwhile you ponder that, I do have a few tricks up my sleeve to dig deeper into your original reported problem, which I can reproduce - namely to place a lock around SslStream initial authorization. Will do this later on... |
|
One more thing:
This is outside any SslStream concurrency issues - and comon code for non-encrypted - you should always also check your code with a non-encrypted connection (if at all possible) as a cross-check. |
|
Cheers for the update. Yes, it is a new issue, it came about when trying FluentFTP.GnuTLS as suggested. Apologies for muddling up a issue tracking system with a support line :) I will try the things suggested but I have to travel now and will not be back until next week. So I'm not deliberately being rude if you don't hear back from me soon. Thanks. |
FTP Server OS: Windows
FTP Server Type: FileZilla
Client Computer OS: Windows
FluentFTP Version: 49.0.2
Framework: .NET 8 and/or .NET Frankwork 4.8
I generally love FluentFTP. However, when used within a Parallel.ForEach() loop, the server allows the first file upload. Then responds with.
FTP Session 133 hidden developmenttesting [Error] TLS session of data connection not resumed.
FTP Session 133 hidden developmenttesting [Response] 425 Unable to build data connection: TLS session of data connection not resumed.
Each iteration of the loop is getting their own FtpClient. So is it not safe to use inside threads like this?
Here's a simplified version that reproduces this:
Loop:
Method:
Logs :
The text was updated successfully, but these errors were encountered: