Error: Connection was closed before we received a valid response from endpoint URL #3842

moldstadt · 2019-01-10T13:24:05Z

I am receiving the above error when trying to upload my code to Lambda using the cli:

aws lambda update-function-code --function-name <api> --zip-file fileb://api.zip

This was working fine a couple of days ago, but today it just keeps failing.

Here is a piece of the debug logs, where the error is thrown:

2019-01-10 15:08:06,708 - MainThread - botocore.hooks - DEBUG - Event request-created.lambda.UpdateFunctionCode: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x000002439C15CE10>>
2019-01-10 15:08:06,708 - MainThread - botocore.hooks - DEBUG - Event choose-signer.lambda.UpdateFunctionCode: calling handler <function set_operation_specific_signer at 0x000002439A2B69D8>
2019-01-10 15:08:06,744 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2019-01-10 15:08:06,744 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
PUT
/2015-03-31/functions/################################/code

host:lambda.us-east-1.amazonaws.com
x-amz-date:20190110T130806Z

host;x-amz-date
33a30325ede6631c605d7dd8fc83cb0200f609b587e0ff6821ae2b88e9d6ffea
2019-01-10 15:08:06,745 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20190110T130806Z
20190110/us-east-1/lambda/aws4_request
003e24e57b3ce08238427bf808241b76bcd70c30b3cafc1bf5e3c18f92a6f8da
2019-01-10 15:08:06,745 - MainThread - botocore.auth - DEBUG - Signature:
###########################################
2019-01-10 15:08:06,746 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/#############################/code, headers={'User-Agent': b'aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7', 'X-Amz-Date': b'20190110T130806Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=############################/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=###################################', 'Content-Length': '15340895'}>
2019-01-10 15:09:08,100 - MainThread - botocore.hooks - DEBUG - Event needs-retry.lambda.UpdateFunctionCode: calling handler <botocore.retryhandler.RetryHandler object at 0x000002439AB59E10>
2019-01-10 15:09:08,101 - MainThread - botocore.retryhandler - DEBUG - retry needed, retryable exception caught: Connection was closed before we received a valid response from endpoint URL: "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/##############################/code".
Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "http\client.py", line 1239, in request
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "http\client.py", line 1285, in _send_request
  File "http\client.py", line 1234, in endheaders
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "http\client.py", line 986, in send
  File "ssl.py", line 965, in sendall
  File "ssl.py", line 935, in send
  File "ssl.py", line 636, in write
socket.timeout: The write operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\httpsession.py", line 242, in send
    decode_content=False,
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\util\retry.py", line 343, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\packages\six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "http\client.py", line 1239, in request
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "http\client.py", line 1285, in _send_request
  File "http\client.py", line 1234, in endheaders
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "http\client.py", line 986, in send
  File "ssl.py", line 965, in sendall
  File "ssl.py", line 935, in send
  File "ssl.py", line 636, in write
urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 269, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 317, in __call__
    caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 174, in _get_response
    http_response = self._send(request)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 218, in _send
    return self.http_session.send(request)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\httpsession.py", line 273, in send
    endpoint_url=request.url
botocore.exceptions.ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL: "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###########################/code".
2019-01-10 15:09:08,128 - MainThread - botocore.retryhandler - DEBUG - Retry needed, action of: 0.23947896318623496
2019-01-10 15:09:08,129 - MainThread - botocore.endpoint - DEBUG - Response received to retry, sleeping for 0.23947896318623496 seconds

It retries a few times but continually fails. It finally fails with the following error:

Connection was closed before we received a valid response from endpoint URL: "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/##############################/code".

What is the problem?

The text was updated successfully, but these errors were encountered:

moldstadt · 2019-01-11T03:59:57Z

I am still experiencing this problem. Any suggestions?

justnance · 2019-01-11T05:24:40Z

@moldstadt - Thank you for reaching out. Since this use to work as expected it is possible the file size has increased and exceeded the Lambda limitation. Please let us know if the file size is greater than 50 MB.

Also, thanks for providing the debug output in your initial response. It appears only part of the debug was provided and I'm not able to determine the CLI version in use. If the file size has not exceeded the Lamdba limits and this issue continues, please provide a more of the debug output.

Also, you can try the following:

enable history
rerun the command to reproduce the timeout.
run aws history show to see if there is any other status codes that may help us understand the problem better.

moldstadt · 2019-01-14T18:23:40Z

@justnance Thanks for the feedback.

The zip file is only 11MB (67MB uncompressed).

The CLI version is 1.16.17.

Here is a more complete debug log:

call aws lambda update-function-code --function-name ############################### --zip-file fileb://API.zip --debug
2019-01-11 12:41:02,455 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7
2019-01-11 12:41:02,455 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['lambda', 'update-function-code', '--function-name', '###############################', '--zip-file', 'fileb://API.zip', '--debug']
2019-01-11 12:41:02,457 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x000002CF1562E7B8>
2019-01-11 12:41:02,457 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:02,457 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x000002CF14F877B8>
2019-01-11 12:41:02,458 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:02,459 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x000002CF14FAB730>
2019-01-11 12:41:02,460 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:02,460 - MainThread - botocore.session - DEBUG - Loading variable credentials_file from defaults.
2019-01-11 12:41:02,460 - MainThread - botocore.session - DEBUG - Loading variable config_file from defaults.
2019-01-11 12:41:02,461 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:02,461 - MainThread - botocore.session - DEBUG - Loading variable metadata_service_timeout from defaults.
2019-01-11 12:41:02,462 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:02,462 - MainThread - botocore.session - DEBUG - Loading variable metadata_service_num_attempts from defaults.
2019-01-11 12:41:02,663 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:02,663 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x000002CF154D0AE8>
2019-01-11 12:41:02,665 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:02,665 - MainThread - awscli.customizations.history - DEBUG - Enabling CLI history
2019-01-11 12:41:02,675 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:02,675 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2019-01-11 12:41:02,677 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\data\lambda\2015-03-31\service-2.json
2019-01-11 12:41:02,680 - MainThread - botocore.hooks - DEBUG - Event service-data-loaded.lambda: calling handler <function register_retries_for_service at 0x000002CF14E6C1E0>
2019-01-11 12:41:02,681 - MainThread - botocore.handlers - DEBUG - Registering retry handlers for service: lambda
2019-01-11 12:41:02,684 - MainThread - botocore.hooks - DEBUG - Event building-command-table.lambda: calling handler <function add_waiters at 0x000002CF156349D8>
2019-01-11 12:41:02,722 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('function-name', <awscli.arguments.CLIArgument object at 0x000002CF15F5CFD0>), ('zip-file', <awscli.arguments.CLIArgument object at 0x000002CF15F58400>), ('s3-bucket', <awscli.arguments.CLIArgument object at 0x000002CF15F625F8>), ('s3-key', <awscli.arguments.CLIArgument object at 0x000002CF15F620B8>), ('s3-object-version', <awscli.arguments.CLIArgument object at 0x000002CF15F62048>), ('publish', <awscli.arguments.BooleanArgument object at 0x000002CF15F620F0>), ('no-publish', <awscli.arguments.BooleanArgument object at 0x000002CF15F62160>), ('dry-run', <awscli.arguments.BooleanArgument object at 0x000002CF15F62278>), ('no-dry-run', <awscli.arguments.BooleanArgument object at 0x000002CF15F62240>), ('revision-id', <awscli.arguments.CLIArgument object at 0x000002CF15F62358>)])
2019-01-11 12:41:02,723 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.lambda.update-function-code: calling handler <function add_streaming_output_arg at 0x000002CF1562ED08>
2019-01-11 12:41:02,724 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.lambda.update-function-code: calling handler <function _modify_zipfile_docstring at 0x000002CF14FAB8C8>
2019-01-11 12:41:02,725 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.lambda.update-function-code: calling handler <function add_cli_input_json at 0x000002CF14FABD08>
2019-01-11 12:41:02,727 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.lambda.update-function-code: calling handler <function unify_paging_params at 0x000002CF15649BF8>
2019-01-11 12:41:02,767 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\data\lambda\2015-03-31\paginators-1.json
2019-01-11 12:41:02,768 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.lambda.update-function-code: calling handler <function add_generate_skeleton at 0x000002CF154C9950>
2019-01-11 12:41:02,773 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.lambda.update-function-code: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x000002CF15F624A8>>
2019-01-11 12:41:02,774 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.lambda.update-function-code: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x000002CF15F623C8>>
2019-01-11 12:41:02,776 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.function-name: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000002CF144BB908>
2019-01-11 12:41:02,777 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.lambda.update-function-code: calling handler <function validate_is_zip_file at 0x000002CF14FAB6A8>
2019-01-11 12:41:02,777 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.lambda.update-function-code: calling handler <awscli.argprocess.ParamShorthandParser object at 0x000002CF151F1BE0>
2019-01-11 12:41:02,778 - MainThread - awscli.arguments - DEBUG - Unpacked value of '###############################' for parameter "function_name": '###############################'
2019-01-11 12:41:02,780 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.zip-file: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000002CF144BB908>
2019-01-11 12:41:02,793 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.lambda.update-function-code: calling handler <function validate_is_zip_file at 0x000002CF14FAB6A8>
2019-01-11 12:41:02,885 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.lambda.update-function-code: calling handler <awscli.argprocess.ParamShorthandParser object at 0x000002CF151F1BE0>
2019-01-11 12:41:07,698 - MainThread - awscli.arguments - DEBUG - Unpacked value of b'PK\x03\x04\x14\...'
2019-01-11 12:41:53,427 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.s3-bucket: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000001961C94B9E8>
2019-01-11 12:41:53,427 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.s3-key: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000001961C94B9E8>
2019-01-11 12:41:53,428 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.s3-object-version: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000001961C94B9E8>
2019-01-11 12:41:53,431 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.publish: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000001961C94B9E8>
2019-01-11 12:41:53,432 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.dry-run: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000001961C94B9E8>
2019-01-11 12:41:53,433 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.revision-id: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000001961C94B9E8>
2019-01-11 12:41:53,433 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000001961C94B9E8>
2019-01-11 12:41:53,434 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.lambda.update-function-code.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x000001961C94B9E8>
2019-01-11 12:41:53,434 - MainThread - botocore.hooks - DEBUG - Event calling-command.lambda.update-function-code: calling handler <bound method CliInputJSONArgument.add_to_call_parameters of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x000001961E3E24A8>>
2019-01-11 12:41:53,434 - MainThread - botocore.hooks - DEBUG - Event calling-command.lambda.update-function-code: calling handler <bound method GenerateCliSkeletonArgument.generate_json_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x000001961E3E23C8>>
2019-01-11 12:41:53,434 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:53,434 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:53,435 - MainThread - botocore.session - DEBUG - Loading variable region from config file with value 'us-east-1'.
2019-01-11 12:41:53,435 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:53,435 - MainThread - botocore.session - DEBUG - Loading variable ca_bundle from defaults.
2019-01-11 12:41:53,435 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:53,435 - MainThread - botocore.session - DEBUG - Loading variable api_versions from defaults.
2019-01-11 12:41:53,435 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2019-01-11 12:41:53,436 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2019-01-11 12:41:53,436 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2019-01-11 12:41:53,437 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2019-01-11 12:41:53,438 - MainThread - botocore.loaders - DEBUG - Loading JSON file: C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\data\endpoints.json
2019-01-11 12:41:53,447 - MainThread - botocore.session - DEBUG - Loading variable profile from defaults.
2019-01-11 12:41:53,448 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x000001961D257730>
2019-01-11 12:41:53,450 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.lambda: calling handler <function add_generate_presigned_url at 0x000001961D19FF28>
2019-01-11 12:41:53,452 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None
2019-01-11 12:41:53,455 - MainThread - botocore.endpoint - DEBUG - Setting lambda timeout as (60, 60)
2019-01-11 12:41:53,457 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: lambda
2019-01-11 12:41:53,475 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.lambda.UpdateFunctionCode: calling handler <function generate_idempotent_uuid at 0x000001961D2E6AE8>
2019-01-11 12:41:53,607 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=UpdateFunctionCode) with params: {'url_path': '/2015-03-31/functions/###################################/code', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7'}, 'body': b'{"ZipFile": "UEsDBBQAAAAIABZ2Kk6MMc..."}', 'url': 'https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###############################/code', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x000002CF16D57128>, 'has_streaming_input': False, 'auth_type': None}}
2019-01-11 12:43:18,141 - MainThread - botocore.hooks - DEBUG - Event request-created.lambda.UpdateFunctionCode: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x000002CF16D57198>>
2019-01-11 12:43:18,148 - MainThread - botocore.hooks - DEBUG - Event choose-signer.lambda.UpdateFunctionCode: calling handler <function set_operation_specific_signer at 0x000002CF14E669D8>
2019-01-11 12:43:18,193 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2019-01-11 12:43:18,193 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
PUT
/2015-03-31/functions/###############################/code

host:lambda.us-east-1.amazonaws.com
x-amz-date:20190111T104318Z

host;x-amz-date
33a30325ede6631c605d7dd8fc83cb0200f609b587e0ff6821ae2b88e9d6ffea
2019-01-11 12:43:18,194 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20190111T104318Z
20190111/us-east-1/lambda/aws4_request
92c08cff0013f4216f1867da86748d396d5302483b73049c0a81c6356794ce6c
2019-01-11 12:43:18,195 - MainThread - botocore.auth - DEBUG - Signature:
######################################################
2019-01-11 12:43:18,195 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###############################/code, headers={'User-Agent': b'aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7', 'X-Amz-Date': b'20190111T104318Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=################/20190111/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=######################################################', 'Content-Length': '15340895'}>
2019-01-11 12:44:19,789 - MainThread - botocore.endpoint - DEBUG - Exception received when sending HTTP request.
Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "http\client.py", line 1239, in request
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "http\client.py", line 1285, in _send_request
  File "http\client.py", line 1234, in endheaders
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "http\client.py", line 986, in send
  File "ssl.py", line 965, in sendall
  File "ssl.py", line 935, in send
  File "ssl.py", line 636, in write
ssl.SSLWantWriteError: The operation did not complete (write) (_ssl.c:2037)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\httpsession.py", line 242, in send
    decode_content=False,
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\util\retry.py", line 343, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\packages\six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "http\client.py", line 1239, in request
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "http\client.py", line 1285, in _send_request
  File "http\client.py", line 1234, in endheaders
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "http\client.py", line 986, in send
  File "ssl.py", line 965, in sendall
  File "ssl.py", line 935, in send
  File "ssl.py", line 636, in write
urllib3.exceptions.SSLError: The operation did not complete (write) (_ssl.c:2037)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 174, in _get_response
    http_response = self._send(request)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 218, in _send
    return self.http_session.send(request)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\httpsession.py", line 260, in send
    raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###############################/code The operation did not complete (write) (_ssl.c:2037)
2019-01-11 12:44:19,816 - MainThread - botocore.hooks - DEBUG - Event needs-retry.lambda.UpdateFunctionCode: calling handler <botocore.retryhandler.RetryHandler object at 0x000002CF15F580F0>
2019-01-11 12:44:19,817 - MainThread - botocore.retryhandler - DEBUG - retry needed, retryable exception caught: SSL validation failed for https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###############################/code The operation did not complete (write) (_ssl.c:2037)
Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "http\client.py", line 1239, in request
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "http\client.py", line 1285, in _send_request
  File "http\client.py", line 1234, in endheaders
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "http\client.py", line 986, in send
  File "ssl.py", line 965, in sendall
  File "ssl.py", line 935, in send
  File "ssl.py", line 636, in write
ssl.SSLWantWriteError: The operation did not complete (write) (_ssl.c:2037)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\httpsession.py", line 242, in send
    decode_content=False,
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\util\retry.py", line 343, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\packages\six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "http\client.py", line 1239, in request
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "http\client.py", line 1285, in _send_request
  File "http\client.py", line 1234, in endheaders
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "http\client.py", line 986, in send
  File "ssl.py", line 965, in sendall
  File "ssl.py", line 935, in send
  File "ssl.py", line 636, in write
urllib3.exceptions.SSLError: The operation did not complete (write) (_ssl.c:2037)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 269, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 317, in __call__
    caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 174, in _get_response
    http_response = self._send(request)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 218, in _send
    return self.http_session.send(request)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\httpsession.py", line 260, in send
    raise SSLError(endpoint_url=request.url, error=e)
botocore.exceptions.SSLError: SSL validation failed for https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###############################/code The operation did not complete (write) (_ssl.c:2037)
2019-01-11 12:44:19,820 - MainThread - botocore.retryhandler - DEBUG - Retry needed, action of: 0.8963882468378392
2019-01-11 12:44:19,820 - MainThread - botocore.endpoint - DEBUG - Response received to retry, sleeping for 0.8963882468378392 seconds
2019-01-11 12:44:20,717 - MainThread - botocore.hooks - DEBUG - Event request-created.lambda.UpdateFunctionCode: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x000002CF16D57198>>
2019-01-11 12:44:20,718 - MainThread - botocore.hooks - DEBUG - Event choose-signer.lambda.UpdateFunctionCode: calling handler <function set_operation_specific_signer at 0x000002CF14E669D8>
2019-01-11 12:44:20,799 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2019-01-11 12:44:20,800 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
PUT
/2015-03-31/functions/###############################/code

host:lambda.us-east-1.amazonaws.com
x-amz-date:20190111T104420Z

host;x-amz-date
33a30325ede6631c605d7dd8fc83cb0200f609b587e0ff6821ae2b88e9d6ffea
2019-01-11 12:44:20,801 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20190111T104420Z
20190111/us-east-1/lambda/aws4_request
87579068e9d38b28b9617bb66917d6b7c3f70ffe36f0356bdb252a87d0910c5c
2019-01-11 12:44:20,802 - MainThread - botocore.auth - DEBUG - Signature:
######################################################
2019-01-11 12:44:20,802 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###############################/code, headers={'User-Agent': b'aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7', 'X-Amz-Date': b'20190111T104420Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=################/20190111/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=######################################################', 'Content-Length': '15340895'}>

...

2019-01-11 12:48:36,132 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "http\client.py", line 1239, in request
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "http\client.py", line 1285, in _send_request
  File "http\client.py", line 1234, in endheaders
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "http\client.py", line 986, in send
  File "ssl.py", line 965, in sendall
  File "ssl.py", line 935, in send
  File "ssl.py", line 636, in write
socket.timeout: The write operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\httpsession.py", line 242, in send
    decode_content=False,
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\util\retry.py", line 343, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\packages\six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\urllib3\connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "http\client.py", line 1239, in request
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "http\client.py", line 1285, in _send_request
  File "http\client.py", line 1234, in endheaders
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "http\client.py", line 986, in send
  File "ssl.py", line 965, in sendall
  File "ssl.py", line 935, in send
  File "ssl.py", line 636, in write
urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\awscli\clidriver.py", line 207, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\awscli\clidriver.py", line 344, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\awscli\clidriver.py", line 516, in __call__
    call_parameters, parsed_globals)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\awscli\clidriver.py", line 636, in invoke
    client, operation_name, parameters, parsed_globals)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\awscli\clidriver.py", line 648, in _make_client_call
    **parameters)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\client.py", line 320, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\client.py", line 610, in _make_api_call
    operation_model, request_dict)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 136, in _send_request
    success_response, exception):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 205, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 251, in __call__
    caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 317, in __call__
    caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 174, in _get_response
    http_response = self._send(request)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\endpoint.py", line 218, in _send
    return self.http_session.send(request)
  File "C:\Program Files\Amazon\AWSCLI\runtime\lib\site-packages\botocore\httpsession.py", line 273, in send
    endpoint_url=request.url
botocore.exceptions.ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL: "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###############################/code".
2019-01-11 12:48:36,151 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

Connection was closed before we received a valid response from endpoint URL: "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/###############################/code".

Output from aws history:

AWS CLI command entered
at time: 2019-01-11 12:41:02.670
with AWS CLI version: aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7
with arguments: ['lambda', 'update-function-code', '--function-name', '####################################################', '--zip-file', 'fileb://API.zip', '--debug']

[0] API call made
at time: 2019-01-11 12:42:29.095
to service: lambda
using operation: UpdateFunctionCode
with parameters: {
    "FunctionName": "####################################################",
    "ZipFile": "<Byte sequence>"
}

[0] HTTP request sent
at time: 2019-01-11 12:43:18.195
to URL: https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/####################################################/code
with method: PUT
with headers: {
    "Authorization": "AWS4-HMAC-SHA256 Credential=####################/20190111/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=###############################################################",
    "Content-Length": "15340895",
    "User-Agent": "aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7",
    "X-Amz-Date": "20190111T104318Z"
}
with body: {
    "ZipFile": "UEsDBBQAAAAIABZ2Kk6MM..."
}

[0] HTTP request sent
at time: 2019-01-11 12:44:20.802
to URL: https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/####################################################/code
with method: PUT
with headers: {
    "Authorization": "AWS4-HMAC-SHA256 Credential=####################/20190111/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=###############################################################",
    "Content-Length": "15340895",
    "User-Agent": "aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7",
    "X-Amz-Date": "20190111T104420Z"
}
with body: {
    "ZipFile": "UEsDBBQAAAAIABZ2Kk6MM..."
}

[0] HTTP request sent
at time: 2019-01-11 12:45:24.212
to URL: https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/####################################################/code
with method: PUT
with headers: {
    "Authorization": "AWS4-HMAC-SHA256 Credential=####################/20190111/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=###############################################################",
    "Content-Length": "15340895",
    "User-Agent": "aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7",
    "X-Amz-Date": "20190111T104524Z"
}
with body: {
    "ZipFile": "UEsDBBQAAAAIABZ2Kk6MM..."
}

[0] HTTP request sent
at time: 2019-01-11 12:46:29.277
to URL: https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/####################################################/code
with method: PUT
with headers: {
    "Authorization": "AWS4-HMAC-SHA256 Credential=####################/20190111/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=###############################################################",
    "Content-Length": "15340895",
    "User-Agent": "aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7",
    "X-Amz-Date": "20190111T104629Z"
}
with body: {
    "ZipFile": "UEsDBBQAAAAIABZ2Kk6MM..."
}

[0] HTTP request sent
at time: 2019-01-11 12:47:34.508
to URL: https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/####################################################/code
with method: PUT
with headers: {
    "Authorization": "AWS4-HMAC-SHA256 Credential=####################/20190111/us-east-1/lambda/aws4_request, SignedHeaders=host;x-amz-date, Signature=###############################################################",
    "Content-Length": "15340895",
    "User-Agent": "aws-cli/1.16.17 Python/3.6.0 Windows/10 botocore/1.12.7",
    "X-Amz-Date": "20190111T104734Z"
}
with body: {
    "ZipFile": "UEsDBBQAAAAIABZ2Kk6MM..."
}

AWS CLI command exited
at time: 2019-01-11 12:48:36.154
with return code: 255

I am still experiencing the problem today. Please advise further.

Edit: Updating to the latest version of aws cli (1.16.88) yields the same results.

moldstadt · 2019-01-21T09:25:30Z

I'm still experiencing this problem. What could the problem be?

I have tried re-uploading an unchanged project of about the same size, and that works fine.

justnance · 2019-01-25T16:42:40Z

@moldstadt - Thank you for your feedback and patience. Based on the debug logs and the history output it appears there is a network issue or timeout. Is there a difference in the network between the changed and unchanged project?

no-response · 2019-03-26T18:18:33Z

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

nosajio · 2019-03-26T18:52:19Z

@moldstadt did you ever manage to resolve this? I'm a couple of hours into debugging the same issue 😣

Thanks!

Fuiste · 2019-03-26T22:10:49Z

I am having this issue as well when trying to deploy new lambda code.

MimoMoreno · 2019-03-27T13:36:01Z

I am having a similar issue when deploying to elastic beanstalk. I am working with PowerShell on Windows with the following version: EB CLI 3.14.13 (Python 3.6.2)

The repeated error is:
(DEBUG) ebcli.lib.aws : Making api call: (s3, upload_part) to region: us-west-2 with args: {...}

nosajio · 2019-03-27T13:39:51Z

Still having the same issue today. Other lambda commands are usable through the cli, but update-function-code continues to return the same error:

Connection was closed before we received a valid response from endpoint URL: "https://lambda.eu-west-2.amazonaws.com/2015-03-31/functions/test-fn/code".

I can confirm this error happens in all regions I've tested uploading code to (us-east-1, us-east-2, eu-west-2).

I've also reinstalled awscli with no change. current version: aws-cli/1.16.132 Python/3.7.2 Darwin/18.0.0 botocore/1.12.122

Hopefully this issue will be reopened as it seems like more people are experiencing this.

Fuiste · 2019-03-27T13:45:44Z

For what it's worth, my issue seemed to resolve itself after a few hours.

The command that finally worked was adding the --debug flag to the update function code command, but I'm fairly certain this had little to do with the resolution.

For posterity sake, these are the things I tried, which did not solve the issue:

Running with a different Access Key/Secret
Running as a different IAM user
Running on a different network
Running on a newer version of the AWS CLI (1.16.130)
Restarting my computer

It is worth noting as well that my team was able to run the command (even using my IAM credentials) while this problem was occurring, so it's very likely to be an issue with the computer, and not the AWS CLI.

moldstadt · 2019-03-29T19:12:58Z

I'm still experiencing this problem. Some times it works, other times it doesn't.

…

On Wed, 27 Mar 2019, 15:46 William Pelrine ***@***.***> wrote: For what it's worth, my issue seemed to resolve itself after a few hours. The command that finally worked was adding the --debug flag to the update function code command, but I'm fairly certain this had little to do with the resolution. For posterity sake, these are the things I tried, which *did not* solve the issue: 1. Running with a different Access Key/Secret 2. Running as a different IAM user 3. Running on a different network 4. Running on a newer version of the AWS CLI (1.16.130) 5. Restarting my computer It is worth noting as well that my team was able to run the command (even using my IAM credentials) while this problem was occurring, so it's very likely to be an issue with the computer, and not the AWS CLI. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3842 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AVoOPyfE30twp3uA1Vzo0SsaHfNd-6uWks5va3YhgaJpZM4Z5fYF> .

ijkeleher · 2019-04-01T12:45:19Z

Also experiencing this problem

Have tried running with --debug flag to no effect.

@moldstadt are you a mac osx user? and did you install with homebrew?

nosajio · 2019-04-09T18:46:27Z

@moldstadt @inci90 and anyone else experiencing the above problem, I was able to resolve it by connecting to another wifi network.

This is weird but the upload always works with/without the --debug flag when I tether to my phone network. But for some reason it fails on my home network. I'm curious about what could be causing this...

justnance · 2019-04-22T15:25:09Z

It appears this issue is related to a local computer or network issue and not related to the CLI. I'm closing this issue for now but can reopen if there are specific steps we can do to reproduce the issue.

ijkeleher · 2019-04-25T00:16:00Z

The issue has disappeared for me now. Can confirm it appears to be a network issue.

I moved to another wifi network and the issue fixed itself.

justnance · 2019-04-25T20:41:41Z

@inci90 - Thanks for the update.

wswardiii · 2020-07-25T18:53:00Z

Just commenting to say that none of the proposed solutions here worked for me.
My .zip file is 16.6 mb and adding the --cli-connect-timeout 6000 gave me a signature expired error.
Switching to a different network did nothing either.

Is there an official solution provided by the cli for this?

I have had to retry the upload a number of times to get around this. I believe that one needs a fast enough upload speed to usually avoid this message. Otherwise, it is a persistent issue, because the signature seems to expire DURING the upload.

mowliv · 2020-09-26T16:02:12Z

Turning off Mac bitdefender AV didn't help but --cli-connect-timeout 6000 fixed it for me. File size only 10 MB, rural DSL internet cxn (15 down, 1 up).

life-efficient · 2020-11-06T11:18:42Z

Looks like just a timeout due to a slow network and relatively large upload.

dskvr · 2020-11-17T22:55:43Z

I'm having this issue as well, and none of the answers address my issue.

This started happening when I enabled accelerate on s3 buckets.
I have properly configured accelerate to true
This obscure error only occurs on Windows while using git-bash
The logic is ran in a matrix, macos and ubuntu run the same logic with no error.
The download fails at the exact same place every time, suggesting this is not a timeout isssue.
Unless buckets with accelerated transfers use Lambda on the backend, I fail to see how Lambda would have anything to do with this. Especially considering the exact same logic is used on two other platforms.

I will switch over the cmd for windows only, as it will probably fix the issue, but it makes everything a pain.

Always fails at roughly 42 MiB, short of the 50mb limit suggested above.

Completed 41.2 MiB/136.6 MiB (1.3 MiB/s) with 1 file(s) remaining   
upload failed: .\***.tar.gz Connection was closed before we received a valid response from endpoint URL: "***.tar.gz".
fatal error: An error occurred (404) when calling the HeadObject operation: Key "***.tar.gz" does not exist
Completed 41.5 MiB/136.6 MiB (1.3 MiB/s) with 1 file(s) remaining

mowliv · 2020-11-18T16:56:06Z

In case it helps someone, I tried this and it works for when I'm in a rural area. I changed the python library code to retry several times. For some reason, in my rural location it always goes on the 4th retry. But when I was at a hotel with bad wifi, it still wouldn't go.

Change file /lib/python3.8/http/client.py - here's the whole send() method, with my change marked with a comment:

def send(self, data):
    """Send `data' to the server.
    ``data`` can be a string object, a bytes object, an array object, a
    file-like object that supports a .read() method, or an iterable object.
    """

    if self.sock is None:
        if self.auto_open:
            self.connect()
        else:
            raise NotConnected()

    if self.debuglevel > 0:
        print("send:", repr(data))
    if hasattr(data, "read") :
        if self.debuglevel > 0:
            print("sendIng a read()able")
        encode = self._is_textIO(data)
        if encode and self.debuglevel > 0:
            print("encoding file using iso-8859-1")
        while 1:
            datablock = data.read(self.blocksize)
            if not datablock:
                break
            if encode:
                datablock = datablock.encode("iso-8859-1")
            self.sock.sendall(datablock)
        return
    try:
        # mowliv Edit 10/31/20
        count = 0
        while True:
            try:
                self.sock.sendall(data)
                break
            except socket.timeout:
                count += 1
                if count >= 8:
                    raise
                else:
                    print(f"mowliv: ssl send - retry {count}")

    except TypeError:
        if isinstance(data, collections.abc.Iterable):
            for d in data:
                self.sock.sendall(d)
        else:
            raise TypeError("data should be a bytes-like object "
                            "or an iterable, got %r" % type(data))

LarryKlugerDS · 2020-11-25T07:44:23Z

--cli-connect-timeout 6000 also solved the problem for me

Home network with corp security proxy that gives a slow connect time.
22M zip payload

Version:
aws-cli/2.1.4 Python/3.7.4 Darwin/18.7.0 exe/x86_64

Other aws lambda commands work. Just aws lambda update-function-code times out.

LarryKlugerDS · 2020-11-25T08:03:11Z

Please upvote / comment on the feature request to make --cli-connect-timeout 6000 be the default:

See #5754

Nohclu · 2021-01-08T10:54:39Z

Hey, I ran into this issue also and found that updating botocore to the latest version resolved it for me.
Hope this can be of use to someone.

mowliv · 2021-01-08T16:17:43Z

Interesting, thanks. I ended up setting up an intermediate host at Vultr.com. I change code locally, commit to github, update the code on the intermediate host, and upload from there in 5 secs to AWS. The heavy lifting upload happens there, sidestepping this issue.

…

On Fri, Jan 8, 2021 at 2:54 AM Nohclu ***@***.***> wrote: Hey, I ran into this issue also and found that updating botocore to the latest version resolved it for me. Hope this can be of use to someone. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3842 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABDDXTJWPLPZPXMYW3U3CHTSY3P75ANCNFSM4GPF6YCQ> .

billyjsubs · 2021-02-17T17:44:57Z

My only solution was to downgrade the modules used to version that work. Awscli 1.18.177 and botocore 1.19.17 will not work (and higher) ....tried all of the workarounds here. Would love this to be fixed in the latest awscli:-

pip install awscli==1.18.176
pip install botocore==1.19.16

necro351 · 2021-02-28T22:59:28Z

Hi all, thanks for all the insight on this thread.

This is a bug, why is it closed? The default timeout is too small and the error says 'Connection was closed...' has nothing to do with the actual problem: the operation was arbitrarily cancelled before the upload finished because it was too slow.

The error should say: "operation was cancelled before upload finished (increase timeout?)" at the very least.

LarryKlugerDS · 2021-03-01T05:16:34Z

AWS closed it "as designed" in one of the dup issue threads. And stated that increasing the default might break other developers' uses of the SDK.
In this thread, @justnance claims it can't be reproduced, see comment above

I think both decisions are mistaken, but this ain't a democracy. /Larry

ItayGoren · 2021-04-04T09:31:01Z

I'm still experiencing this problem. Some times it works, other times it
doesn't.
same here
we just retry on failure and it works most of the times

fedecaccia · 2021-04-23T14:32:42Z

I have been experimenting the same problem, and I found the solution by changing the internet network from my local wifi to my mobile connection.

debu99 · 2021-05-26T09:56:17Z

add --region fixed my issue

kngmat004 · 2021-05-26T15:45:44Z

Has this been resolved yet? Im still seeing this issue regardless of whether I use --cli-connect-timeout 6000 or not

osmanvee · 2021-05-26T17:44:45Z

Facing something similar:

botocore.exceptions.HTTPClientError: An HTTP Client raised and unhandled exception: Failed to parse: https://sts.amazonaws.com/
2021-05-26 12:31:08,359 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

An HTTP Client raised and unhandled exception: Failed to parse: https://sts.amazonaws.com/

--region didn't fix my issue either. I have Python

Eugeniu-N · 2021-06-09T15:06:15Z

I'm having the same issue when trying to upload multiple 40+ mb functions at the same time. If I do one by one they are uploaded successfully but doing 4 at the time will return the error for all of them. With the --cli-connect-timeout 6000 flag they all get uploaded successfully

alexisperrier · 2021-06-21T09:55:34Z

For what it's worth

rural network 15mb down 1mb up
payload 19M
aws lambda create-function --function-name ...

What did not work:

tried with and without a VPN (own) to remove potential ISP related issues
changed zone from us-east-1 to eu-west-1 (where my laptop is)

What finally worked:

adding --cli-connect-timeout 600
(600 instead of 6000, did not want to wait that long)

So problem appeared to be caused by a weak internet connection

BrunoDuarte · 2021-08-18T18:32:20Z

I had this problem many other times, and it was always resolved when I restarted my internet modem. Don't ask me why!

kamyker · 2022-01-28T02:24:46Z

Same issue but I have 400mb upload. Everything was fine 4 days ago.

Trying to upload using browser but it's stuck for few minutes...

abdoulsn · 2022-02-24T15:09:36Z

Solved this by just moving from wifi X to wifi Y.

stargazejin · 2022-04-21T10:39:07Z

Mine happened when I'm uploading with mobile hotspot with 1Mbps, adding --cli-connect-timeout 600 worked.

sayedazharsabri · 2022-05-27T18:02:25Z

Adding --cli-connect-timeout 6000 is worked for me

thanh-doan-cko · 2022-06-16T08:01:57Z

I am having the same problem.

So basically, when I start localstack docker then it is working. Then I can run some aws commands. However, as soon as I started to use one of my service which will consume localstack then it does not work any more.

I tried to add --cli-connect-timeout, upgrading boto3, botocore but it does not work at all.

But when I shutdown my laptop then it is working again. Hence the issues happen again.

I am using aws-cli/2.7.7 Python/3.9.11 Windows/10 exe/AMD64 prompt/off and here is the issue I got

2022-06-16 08:54:27,021 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "urllib3\connectionpool.py", line 703, in urlopen
  File "urllib3\connectionpool.py", line 449, in _make_request
  File "<string>", line 3, in raise_from
  File "urllib3\connectionpool.py", line 444, in _make_request
  File "http\client.py", line 1377, in getresponse
  File "http\client.py", line 320, in begin
  File "http\client.py", line 281, in _read_status
  File "socket.py", line 704, in readinto
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "awscli\botocore\httpsession.py", line 358, in send
  File "urllib3\connectionpool.py", line 785, in urlopen
  File "urllib3\util\retry.py", line 525, in increment
  File "urllib3\packages\six.py", line 769, in reraise
  File "urllib3\connectionpool.py", line 703, in urlopen
  File "urllib3\connectionpool.py", line 449, in _make_request
  File "<string>", line 3, in raise_from
  File "urllib3\connectionpool.py", line 444, in _make_request
  File "http\client.py", line 1377, in getresponse
  File "http\client.py", line 320, in begin
  File "http\client.py", line 281, in _read_status
  File "socket.py", line 704, in readinto
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

Any ideas for that issue?

rwebber · 2023-05-03T17:58:07Z

Incase it is helpful to anyone, I was able to move past this issue by Pausing file uploads in OneDrive (Windows system).
It appears OneDrive was making the upload to slow, so things timed out.

justnance self-assigned this Jan 11, 2019

justnance added guidance Question that needs advice or information. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. labels Jan 11, 2019

justnance added investigating This issue is being investigated and/or work is in progress to resolve the issue. and removed guidance Question that needs advice or information. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. labels Jan 24, 2019

justnance added guidance Question that needs advice or information. and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Jan 25, 2019

justnance added the closing-soon This issue will automatically close in 4 days unless further comments are made. label Mar 19, 2019

no-response bot closed this as completed Mar 26, 2019

no-response bot removed the closing-soon This issue will automatically close in 4 days unless further comments are made. label Mar 29, 2019

no-response bot reopened this Mar 29, 2019

justnance closed this as completed Apr 22, 2019

kdaily mentioned this issue Oct 1, 2020

Connection was closed before we received a valid response from endpoint URL #5525

Closed

devangmehta123 mentioned this issue Oct 11, 2020

Issue with 'chalice deploy' of pymysql[rsa] aws/chalice#1515

Closed

LarryKlugerDS mentioned this issue Nov 25, 2020

Make default: --cli-connect-timeout 6000 #5754

Open

lisovskey mentioned this issue Dec 22, 2020

Max retries and connection timeout arguments capitalone/locopy#109

Closed

Error: Connection was closed before we received a valid response from endpoint URL #3842

Error: Connection was closed before we received a valid response from endpoint URL #3842

Comments

moldstadt commented Jan 10, 2019 • edited

moldstadt commented Jan 11, 2019

justnance commented Jan 11, 2019

moldstadt commented Jan 14, 2019 • edited

moldstadt commented Jan 21, 2019

justnance commented Jan 25, 2019

no-response bot commented Mar 26, 2019

nosajio commented Mar 26, 2019 • edited

Fuiste commented Mar 26, 2019

MimoMoreno commented Mar 27, 2019

nosajio commented Mar 27, 2019

Fuiste commented Mar 27, 2019

moldstadt commented Mar 29, 2019 via email

ijkeleher commented Apr 1, 2019

nosajio commented Apr 9, 2019

justnance commented Apr 22, 2019

ijkeleher commented Apr 25, 2019

justnance commented Apr 25, 2019

wswardiii commented Jul 25, 2020

mowliv commented Sep 26, 2020

life-efficient commented Nov 6, 2020

dskvr commented Nov 17, 2020 • edited

mowliv commented Nov 18, 2020

LarryKlugerDS commented Nov 25, 2020

LarryKlugerDS commented Nov 25, 2020 • edited

Nohclu commented Jan 8, 2021

mowliv commented Jan 8, 2021 via email

billyjsubs commented Feb 17, 2021

necro351 commented Feb 28, 2021

LarryKlugerDS commented Mar 1, 2021

ItayGoren commented Apr 4, 2021

fedecaccia commented Apr 23, 2021

debu99 commented May 26, 2021

kngmat004 commented May 26, 2021

osmanvee commented May 26, 2021

Eugeniu-N commented Jun 9, 2021

alexisperrier commented Jun 21, 2021 • edited

BrunoDuarte commented Aug 18, 2021

kamyker commented Jan 28, 2022

abdoulsn commented Feb 24, 2022

stargazejin commented Apr 21, 2022

sayedazharsabri commented May 27, 2022

thanh-doan-cko commented Jun 16, 2022 • edited

rwebber commented May 3, 2023

moldstadt commented Jan 10, 2019 •

edited

moldstadt commented Jan 14, 2019 •

edited

nosajio commented Mar 26, 2019 •

edited

dskvr commented Nov 17, 2020 •

edited

LarryKlugerDS commented Nov 25, 2020 •

edited

alexisperrier commented Jun 21, 2021 •

edited

thanh-doan-cko commented Jun 16, 2022 •

edited