Connection was closed before we received a valid response from endpoint URL

[mikeball]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• [ x] I've gone though the User Guide and the API reference
• [x ] I've searched for previous similar issues and didn't find any solution. Duplicate of (Connection was closed before we received a valid response from endpoint URL #5090) but it was closed due to staleness and was not resolved. The bug still exists.

Describe the bug
Unable to upload zip file lambda. Core underlying issue appears to be inability to open an HTTPS connection to https://lambda.us-east-2.amazonaws.com

SDK version number

aws --version
aws-cli/2.0.46 Python/3.7.3 Linux/5.4.0-45-generic exe/x86_64.elementary.5

Platform/OS/Hardware/Device
Operating System: elementary OS 5.1.7 Hera
Kernel: Linux 5.4.0-45-generic
Architecture: x86-64

To Reproduce (observed behavior)

> aws lambda publish-layer-version --layer-name openj9-runtime --zip-file fileb://runtime.zip --region us-east-2 --debug

Expected behavior
Lambda function zip should be uploaded, instead get timeout error.

Logs/output

2020-09-07 10:53:54,760 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (3): lambda.us-east-2.amazonaws.com:443
2020-09-07 10:54:55,118 - MainThread - botocore.hooks - DEBUG - Event needs-retry.lambda.PublishLayerVersion: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7f54ca7f2c18>>
2020-09-07 10:54:55,119 - MainThread - botocore.retries.standard - DEBUG - Max attempts of 3 reached.
2020-09-07 10:54:55,119 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2020-09-07 10:54:55,120 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "urllib3/connectionpool.py", line 677, in urlopen
  File "urllib3/connectionpool.py", line 392, in _make_request
  File "http/client.py", line 1229, in request
  File "botocore/awsrequest.py", line 92, in _send_request
  File "http/client.py", line 1275, in _send_request
  File "http/client.py", line 1224, in endheaders
  File "botocore/awsrequest.py", line 119, in _send_output
  File "botocore/awsrequest.py", line 203, in send
  File "http/client.py", line 977, in send
  File "ssl.py", line 1015, in sendall
  File "ssl.py", line 984, in send
socket.timeout: The write operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "botocore/httpsession.py", line 263, in send
  File "urllib3/connectionpool.py", line 727, in urlopen
  File "urllib3/util/retry.py", line 379, in increment
  File "urllib3/packages/six.py", line 734, in reraise
  File "urllib3/connectionpool.py", line 677, in urlopen
  File "urllib3/connectionpool.py", line 392, in _make_request
  File "http/client.py", line 1229, in request
  File "botocore/awsrequest.py", line 92, in _send_request
  File "http/client.py", line 1275, in _send_request
  File "http/client.py", line 1224, in endheaders
  File "botocore/awsrequest.py", line 119, in _send_output
  File "botocore/awsrequest.py", line 203, in send
  File "http/client.py", line 977, in send
  File "ssl.py", line 1015, in sendall
  File "ssl.py", line 984, in send
urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "awscli/clidriver.py", line 386, in main
  File "awscli/clidriver.py", line 558, in __call__
  File "awscli/clidriver.py", line 738, in __call__
  File "awscli/clidriver.py", line 867, in invoke
  File "awscli/clidriver.py", line 879, in _make_client_call
  File "botocore/client.py", line 229, in _api_call
  File "botocore/client.py", line 535, in _make_api_call
  File "botocore/client.py", line 554, in _make_request
  File "botocore/endpoint.py", line 102, in make_request
  File "botocore/endpoint.py", line 156, in _send_request
  File "botocore/endpoint.py", line 200, in _do_get_response
  File "botocore/endpoint.py", line 269, in _send
  File "botocore/httpsession.py", line 294, in send
botocore.exceptions.ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL: "https://lambda.us-east-2.amazonaws.com/2018-10-31/layers/openj9-runtime/versions".
2020-09-07 10:54:55,122 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

Connection was closed before we received a valid response from endpoint URL: "https://lambda.us-east-2.amazonaws.com/2018-10-31/layers/openj9-runtime/versions".

Additional context

No VPN is in use. Able to open https://lambda.us-east-2.amazonaws.com/ in browser.

Ping to endpoint works

ping lambda.us-east-2.amazonaws.com
PING lambda.us-east-2.amazonaws.com (3.16.204.232) 56(84) bytes of data.
64 bytes from ec2-3-16-204-232.us-east-2.compute.amazonaws.com (3.16.204.232): icmp_seq=1 ttl=226 time=76.1 ms

Curl to endpoint works

curl https://lambda.us-east-2.amazonaws.com
<MissingAuthenticationTokenException>
  <Message>Missing Authentication Token</Message>
</MissingAuthenticationTokenException>

CLI is able to successfully connect and list functions

aws lambda list-functions --region us-east-2
{
    "Functions": []
}

Also of note, the size of the runtime.zip file is 21.1MB compressed, and 41.7MB uncompressed.

[kdaily]

Hi @mikeball,

Sorry to hear of the error. I also tried to reproduce on Linux and Mac but it worked OK. I also tested with netcat (nc):

$ nc -vz lambda.us-east-2.amazonaws.com 443
lambda.us-east-2.amazonaws.com [3.16.205.211] 443 (https) open

In general this is a firewall or connectivity issue, as noted in the linked issue. Are you able to reproduce on another machine on the same network as this one (either with the same OS or different OS)?

[mikeball]

Hi @kdaily,

Pretty sure it's not a firewall or network issue, as the machine can make curl https, browser https connections, ping connections, etc.. and on a freshly installed and fully up to date pretty plain jane ubuntu based distro.

Doing a netcat test doesn't seem relevant at all, as clearly the machine can make an https connections to AWS, it's able to list functions. But yet lambda zip uploads don't work?

My suspicion is that it's a python version, or underlying library version conflict of some sort, or an issue with python multi-threading around file uploads.

[kdaily]

@mikeball, thanks.

The AWS CLI V2 uses it's own isolated Python and packages, so unlikely to be a version conflict. Can you confirm if this happens for you on a different machine on this network? I'll look into it more myself.

[kdaily]

Hi @mikeball, I also came across this issue which is related:

#3842

Setting a longer --cli-connect-timeout helped to resolve.

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.