Intermittent SSL handshake error when calling S3

[AlfredoDiaz90]

Current Behavior

We are experiencing intermittent errors ("SSL Handshake failed with OpenSSL error - SSL_ERROR_ZERO_RETURN") when interacting with the S3 client from ECS in one of our applications. The error seems random and occasional, and it may occur while loading or getting objects from S3.

This is an example of an exception we got while getting an object from S3, right after loading it:

{
    "message": "The SSL connection could not be established, see inner exception.",
    "data": {},
    "innerException": {
        "ClassName": "System.Security.Authentication.AuthenticationException",
        "Message": "Authentication failed, see inner exception.",
        "Data": null,
        "InnerException": {
            "message": "SSL Handshake failed with OpenSSL error - SSL_ERROR_ZERO_RETURN.",
            "data": {},
            "stackTrace": "   at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, Byte[] recvBuf, Int32 recvOffset, Int32 recvCount, Byte[]& sendBuf, Int32& sendCount)
            \n   at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteContext& context, SecurityBuffer inputBuffer, SecurityBuffer outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)",
            "source": "System.Net.Security",
            "hResult": -2146233088
        },
        "HelpURL": null,
        "StackTraceString": "   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
        \n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
        \n   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)\n--- End of stack trace from previous location where exception was thrown ---
        \n   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
        \n   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
        \n--- End of stack trace from previous location where exception was thrown ---
        \n   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)",
        "RemoteStackTraceString": null,
        "RemoteStackIndex": 0,
        "ExceptionMethod": null,
        "HResult": -2146233087,
        "Source": "System.Private.CoreLib",
        "WatsonBuckets": null
    },
    "stackTrace": "   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
    \n   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    \n   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
    \n   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
    \n   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    \n   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
    \n   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
    \n   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
    \n   at MyNameSpace.AwsS3Context.GetContentVersionId(Guid id) in /filePath/AwsS3Context.cs:line XX
    \n   at MyNameSpace.AwsS3Context.PutEncryptedContentAsync(Guid id, Stream content) in /filePath/AwsS3Context.cs:line XX
    \n   at MyNameSpace.ManagerClass.SaveContentAsync(Guid id, Stream content) in /filePath/ManagerClass.cs:line XX
    \n   at MyNameSpace.ControllerClass.CreateAsync() in /filePath/ControllerClass.cs:line XX
    \n   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
    \n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
    \n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
    \n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
    \n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
    \n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
    \n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
    \n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
    \n   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
    \n   at MyNameSpace.APIClass.Startup.<Configure>b__8_1(HttpContext context, Func`1 next) in /filePath/Startup.cs:line XX
    \n   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)",
    "source": "System.Net.Http",
    "hResult": -2146233087
}

S3 client is configured as below:

// At startup
services.AddAWSService<IAmazonS3>();
     
// S3 client class
public class AwsS3Context : IAwsS3Context
{
    private readonly IAmazonS3 awsS3Client;
        
    public AwsS3Context(IAmazonS3 awsS3Client)
    {
        this.awsS3Client = awsS3Client;
    }
        
    public async Task<string> GetContentVersionId(Guid id)
    {
        var getS3Request = new GetObjectRequest
                  {
            BucketName = "bucket_name",
            Key = id.ToString()
        };
 
        using (var s3Item = await awsS3Client.GetObjectAsync(getS3Request).ConfigureAwait(false))
        {
            return s3Item.VersionId;
        }
    } 
    
    public async Task<string> PutEncryptedContentAsync(Guid id, Stream content)
    {
        var config = new TransferUtilityConfig();
 
        try
        {
            using (var awsTransferUtility = new TransferUtility(awsS3Client, config))
            {
                var request = new TransferUtilityUploadRequest
                {
                    BucketName = "bucket_name",
                    Key = id.ToString(),
                    InputStream = content,
                    ServerSideEncryptionMethod = ServerSideEncryptionMethod.AES256 
                };
 
                await awsTransferUtility.UploadAsync(request, CancellationToken.None).ConfigureAwait(false);
                return await GetContentVersionId(Guid);
            }
        }
        finally
        {
            content?.Dispose();
        }
    }
}

Expected Behavior

Intermittent SSL handshake errors when calling S3 are not expected.

Steps to Reproduce (for bugs)

We are not able to reproduce this error consistently. In order to test, we tried isolating the application in a dev environment provisioning a single container and sending it a regular load for some days, adjusting the request rate and size. After this we got the mentioned exception a couple of times but root cause or conditions that produce the error are not evident.

Context

This is a simple API implemented with ECS that receive different types of files to internally save them in an S3 bucket, providing a resource id to its clients.

Your Environment

• AWSSDK.Core version used: AWSSDK.Core (3.3.104.34)
• Service assembly and version used: AWSSDK.S3 (3.3.110.32)
• Targeted .NET platform: .Net Core 2.1 (Microsoft.NETCore.App 2.1.0)
• Visual Studio version: VS 2019 (16.4.1)
• Operating System and version: Alpine 3.11 (From image mcr.microsoft.com/dotnet/core/aspnet:2.1-alpine)

.NET Core Info

• .NET Core version used for development: 2.1
• .NET Core version installed in the environment where application runs: 2.1

[GKrivosheev-rms]

Seeing the same problem intermittently while reading data from S3

[GlebObodovsky]

Have the same type of issue when working with Amazon.S3.Transfer.TransferUtility of AWSSDK.S3, v.3.3.110.50

[ymwymw]

Getting the same error intermittently. Running on Lambda.

Exception Message: SSL Handshake failed with OpenSSL error - SSL_ERROR_ZERO_RETURN.
Stack Trace:
  at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
  at System.Threading.Tasks.ValueTask`1.get_Result()
  at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
  at System.Threading.Tasks.ValueTask`1.get_Result()
  at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
  at System.Threading.Tasks.ValueTask`1.get_Result()
  at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
  at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
  at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
  at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
  at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
  at internal method...

[samiechan]

Got the same problem.
Occurs constantly when downloading a large set of files (about 80 files of different sizes) on a low spec EC2 machine.
In my case, the problem does not occur if the number of files is small (e.g. 10).
I'm using TransferUtility class with ConcurrentServiceRequests = 3 and DownloadAsync method.
Limiting the number of threads (1 thread = 1 file download) using SemaphoreSlim (e.g. maxThreadCount = 3) is a workaround for me.

UPDATE
In AWS TransferUtilityConfig documentation it is said that ConcurrentServiceRequests property determines how many active threads or the number of concurrent asynchronous web requests will be used to upload/download the file. I found in the SDK code that this property is only used in multipart upload method.

[ashishdhingra]

Got the same problem.
Occurs constantly when downloading a large set of files (about 80 files of different sizes) on a low spec EC2 machine.
In my case, the problem does not occur if the number of files is small (e.g. 10).
I'm using TransferUtility class with ConcurrentServiceRequests = 3 and DownloadAsync method.
Limiting the number of threads (1 thread = 1 file download) using SemaphoreSlim (e.g. maxThreadCount = 3) is a workaround for me.

UPDATE
In AWS TransferUtilityConfig documentation it is said that ConcurrentServiceRequests property determines how many active threads or the number of concurrent asynchronous web requests will be used to upload/download the file. I found in the SDK code that this property is only used in multipart upload method.

Hi @samiechan,

Thanks for sharing your findings. Could you please share the EC2 instance configuration (OS, EC2 instance type, etc.) which would help us reproduce the issue?

Thanks,
Ashish

[samiechan]

Got the same problem.
Occurs constantly when downloading a large set of files (about 80 files of different sizes) on a low spec EC2 machine.
In my case, the problem does not occur if the number of files is small (e.g. 10).
I'm using TransferUtility class with ConcurrentServiceRequests = 3 and DownloadAsync method.
Limiting the number of threads (1 thread = 1 file download) using SemaphoreSlim (e.g. maxThreadCount = 3) is a workaround for me.
UPDATE
In AWS TransferUtilityConfig documentation it is said that ConcurrentServiceRequests property determines how many active threads or the number of concurrent asynchronous web requests will be used to upload/download the file. I found in the SDK code that this property is only used in multipart upload method.

Hi @samiechan,

Thanks for sharing your findings. Could you please share the EC2 instance configuration (OS, EC2 instance type, etc.) which would help us reproduce the issue?

Thanks,
Ashish

OS: Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-1035-aws x86_64)
EC2 instance type: t2.medium
Storage type: EBS (1000 GiB)
Download file size varies from 150 KB to 2 GB.

Let me know, if you need more information.

[ashishdhingra]

OS: Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-1035-aws x86_64)
EC2 instance type: t2.medium
Storage type: EBS (1000 GiB)
Download file size varies from 150 KB to 2 GB.

Let me know, if you need more information.

Hi @samiechan,

I tried to reproduce the issue using the t2.micro Ubuntu instance by download the files repeatedly for 3 minutes (to simulate 80 files download) using the code below:

using System;
using System.IO;
using System.Reflection;
using Amazon.S3;
using Amazon.S3.Transfer;
using NLog;
using NLog.AWS.Logger;
using NLog.Config;
using NLog.Targets;

namespace SSLTestConsole
{
    class Program
    {
        static readonly string bucketName = "<<bucket-name>>";
        static readonly string downloadDirectory = Path.Combine(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location), "DownloadedFiles");

        static void Main(string[] args)
        {
            ConfigureNLog();

            Logger logger = LogManager.GetCurrentClassLogger();

            logger.Info($"Checking if local download directory '{downloadDirectory}' exists.");
            if (!Directory.Exists(downloadDirectory))
            {
                logger.Info($"Creating local download directory '{downloadDirectory}'");
                Directory.CreateDirectory(downloadDirectory);
            }

            AmazonS3Client amazonS3Client = new AmazonS3Client(Amazon.RegionEndpoint.USEast2);

            logger.Info($"Executing ListObjectsV2Async on bucket '{bucketName}'");
            var listObjevctsV2Response = amazonS3Client.ListObjectsV2Async(new Amazon.S3.Model.ListObjectsV2Request() {
                BucketName = bucketName
            }).Result;
            logger.Info($"Found {listObjevctsV2Response.KeyCount} objects.");

            DateTime endDateTime = DateTime.Now.AddMinutes(3);
            while (DateTime.Now < endDateTime)
            {
                foreach (var s3Object in listObjevctsV2Response.S3Objects)
                {
                    string fileName = Path.GetFileNameWithoutExtension(s3Object.Key).Replace(" ", "_") + "_" + DateTime.Now.ToFileTime() + "." + Path.GetExtension(s3Object.Key);
                    logger.Info($"Downloading '{s3Object.Key}' to '{downloadDirectory}' as '{fileName}'  using TransferUtility");

                    TransferUtility transferUtility = new TransferUtility(new TransferUtilityConfig());
                    transferUtility.DownloadAsync(new TransferUtilityDownloadRequest()
                    {
                        BucketName = bucketName,
                        Key = s3Object.Key,
                        FilePath = Path.Combine(downloadDirectory, fileName)
                    }).GetAwaiter().GetResult();
                }
            }
        }

        static void ConfigureNLog()
        {
            var config = new LoggingConfiguration();

            var consoleTarget = new ColoredConsoleTarget();
            config.AddTarget("console", consoleTarget);

            var awsTarget = new AWSTarget()
            {
                LogGroup = "NLog.ProgrammaticConfigurationExample",
                Region = "us-east-2"
            };
            config.AddTarget("aws", awsTarget);

            config.LoggingRules.Add(new LoggingRule("*", LogLevel.Debug, consoleTarget));
            config.LoggingRules.Add(new LoggingRule("*", LogLevel.Debug, awsTarget));

            LogManager.Configuration = config;
        }
    }
}

.csproj file

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="AWS.Logger.NLog" Version="2.0.1" />
    <PackageReference Include="AWSSDK.S3" Version="3.5.7.8" />
  </ItemGroup>
</Project>

But somehow the issue is not reproducible. I have the maximum file size of 200MB, not sure if it would make a difference.

Would it be possible for you to share the sample code for reproduction? Also, how are credentials configured and how much time elapses before you get SSL_ERROR_ZERO_RETURN error.

Thanks,
Ashish

[ashishdhingra]

There was a retry logic added for intermittent SSL_ERROR_ZERO_RETURN issue in AWSSDK.Core 3.7.0.17. Closing this issue.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[khteh]

There was a retry logic added for intermittent SSL_ERROR_ZERO_RETURN issue in AWSSDK.Core 3.7.0.17. Closing this issue.

Why is "retry" a solution?