SAM - Init failed error=fork/exec /var/task/bootstrap: no such file or directory

[denizdogan]

Git repo for reproducing: https://github.com/denizdogan/deno-lambda-bug-repro

Check/run ./local.sh to see exactly what I'm doing.

---

Does the SAM example actually work? It behaves "flaky" for me, and I'm not sure if I'm misunderstood something fundamental here, or if it's my Mac, or my Docker, or something else.

Every once in a while, when I try to sam invoke my Lambda, I get this error:

05 May 2023 10:05:24,350 [ERROR] (rapid) Init failed error=fork/exec /var/task/bootstrap: no such file or directory InvokeID=

When it does "work", it seems that no response is being picked up by whatever SAM is doing under the hood:

Building image....................
2023-05-05 12:06:55,911 | Using local image: public.ecr.aws/lambda/provided:al2-rapid-arm64.

2023-05-05 12:06:55,911 | Mounting /Users/deniz/Code/denizdogan/deno-lambda-bug-repro as /var/task:ro,delegated, inside runtime container
2023-05-05 12:06:56,153 | Starting a timer for 10 seconds for function 'HelloWorldFunction'
2023-05-05 12:07:06,156 | Function 'HelloWorldFunction' timed out after 10 seconds
END RequestId: 79a245cc-4e8a-4f0d-85f1-6e689c39e838
REPORT RequestId: 79a245cc-4e8a-4f0d-85f1-6e689c39e838	Init Duration: 0.07 ms	Duration: 10000.00 ms	Billed Duration: 10000 ms	Memory Size: 128 MB	Max Memory Used: 128 MB
2023-05-05 12:07:06,934 | Container was not created. Skipping deletion
2023-05-05 12:07:06,935 | Cleaning all decompressed code dirs
2023-05-05 12:07:06,935 | No response from invoke container for HelloWorldFunction

Here's two runs where I changed literally nothing in between. The first time I get the bootstrap error, the second time it actually seems to run, but no response is being picked up:

~/Code/denizdogan/deno-lambda-bug-repro main*
❯ ./local.sh
deno 1.32.5 (release, aarch64-apple-darwin)
v8 11.2.214.9
typescript 5.0.3
SAM CLI, version 1.82.0
{
  "version": "1.82.0",
  "system": {
    "python": "3.8.16",
    "os": "macOS-13.3.1-arm64-arm-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "20.10.23",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  }
}
/Users/deniz/Code/denizdogan/deno-lambda-bug-repro/template.yaml is a valid SAM Template
2023-05-05 12:05:20,295 | Config file location: /Users/deniz/Code/denizdogan/deno-lambda-bug-repro/samconfig.toml
2023-05-05 12:05:20,296 | Config file '/Users/deniz/Code/denizdogan/deno-lambda-bug-repro/samconfig.toml' does not exist
2023-05-05 12:05:20,297 | Using SAM Template at /Users/deniz/Code/denizdogan/deno-lambda-bug-repro/template.yaml
2023-05-05 12:05:20,311 | Using config file: samconfig.toml, config environment: default
2023-05-05 12:05:20,311 | Expand command line arguments to:
2023-05-05 12:05:20,311 | --template_file=/Users/deniz/Code/denizdogan/deno-lambda-bug-repro/template.yaml --force_image_build --event=event.json --container_host_interface=host.docker.internal --function_logical_id=HelloWorldFunction --no_event --layer_cache_basedir=/Users/deniz/.aws-sam/layers-pkg --container_host=localhost
2023-05-05 12:05:20,311 | local invoke command is called
2023-05-05 12:05:20,312 | No Parameters detected in the template
2023-05-05 12:05:20 Requesting to create CFN template arn:aws:serverlessrepo:us-east-1:390065572566:applications/deno/1.32.5 in serverless application repo...
2023-05-05 12:05:21 Requested to create CFN template arn:aws:serverlessrepo:us-east-1:390065572566:applications/deno/1.32.5 in serverless application repo.
2023-05-05 12:05:21,293 | There is no customer defined id or cdk path defined for resource Deno, so we will use the resource logical id as the resource id
2023-05-05 12:05:21,293 | There is no customer defined id or cdk path defined for resource HelloWorldFunction, so we will use the resource logical id as the resource id
2023-05-05 12:05:21,293 | There is no customer defined id or cdk path defined for resource ServerlessHttpApi, so we will use the resource logical id as the resource id
2023-05-05 12:05:21,294 | Unable to resolve property Layers: [{'Fn::GetAtt': ['Deno', 'Outputs.LayerArn']}]. Leaving as is.
2023-05-05 12:05:21,294 | 0 stacks found in the template
2023-05-05 12:05:21,294 | No Parameters detected in the template
2023-05-05 12:05:21 Requesting to create CFN template arn:aws:serverlessrepo:us-east-1:390065572566:applications/deno/1.32.5 in serverless application repo...
2023-05-05 12:05:22 Requested to create CFN template arn:aws:serverlessrepo:us-east-1:390065572566:applications/deno/1.32.5 in serverless application repo.
2023-05-05 12:05:22,097 | There is no customer defined id or cdk path defined for resource Deno, so we will use the resource logical id as the resource id
2023-05-05 12:05:22,097 | There is no customer defined id or cdk path defined for resource HelloWorldFunction, so we will use the resource logical id as the resource id
2023-05-05 12:05:22,098 | There is no customer defined id or cdk path defined for resource ServerlessHttpApi, so we will use the resource logical id as the resource id
2023-05-05 12:05:22,098 | Unable to resolve property Layers: [{'Fn::GetAtt': ['Deno', 'Outputs.LayerArn']}]. Leaving as is.
2023-05-05 12:05:22,098 | 3 resources found in the stack
2023-05-05 12:05:22,099 | layer "{'Fn::GetAtt': ['Deno', 'Outputs.LayerArn']}" is not recognizable, it might be using intrinsic functions that we don't support yet. Skipping.
2023-05-05 12:05:22,099 | Found Serverless function with name='HelloWorldFunction' and CodeUri='.'
2023-05-05 12:05:22,099 | --base-dir is not presented, adjusting uri . relative to /Users/deniz/Code/denizdogan/deno-lambda-bug-repro/template.yaml
2023-05-05 12:05:22,115 | Found one Lambda function with name 'HelloWorldFunction'
2023-05-05 12:05:22,115 | Invoking hello.handler (provided.al2)
2023-05-05 12:05:22,115 | No environment variables found for function 'HelloWorldFunction'
2023-05-05 12:05:22,116 | Loading AWS credentials from session with profile 'None'
2023-05-05 12:05:22,126 | Resolving code path. Cwd=/Users/deniz/Code/denizdogan/deno-lambda-bug-repro, CodeUri=/Users/deniz/Code/denizdogan/deno-lambda-bug-repro
2023-05-05 12:05:22,126 | Resolved absolute path to code is /Users/deniz/Code/denizdogan/deno-lambda-bug-repro
2023-05-05 12:05:22,126 | Code /Users/deniz/Code/denizdogan/deno-lambda-bug-repro is not a zip/jar file
Building image....................
2023-05-05 12:05:24,146 | Using local image: public.ecr.aws/lambda/provided:al2-rapid-arm64.

2023-05-05 12:05:24,146 | Mounting /Users/deniz/Code/denizdogan/deno-lambda-bug-repro as /var/task:ro,delegated, inside runtime container
2023-05-05 12:05:24,344 | Starting a timer for 10 seconds for function 'HelloWorldFunction'
START RequestId: 917ca73e-5374-4c65-a4c6-a4960aeec017 Version: $LATEST
05 May 2023 10:05:24,350 [ERROR] (rapid) Init failed error=fork/exec /var/task/bootstrap: no such file or directory InvokeID=
2023-05-05 12:05:34,347 | Function 'HelloWorldFunction' timed out after 10 seconds
END RequestId: a280b54d-c928-4ea7-a86e-b55f93c2ca55
REPORT RequestId: a280b54d-c928-4ea7-a86e-b55f93c2ca55	Init Duration: 0.07 ms	Duration: 10000.00 ms	Billed Duration: 10000 ms	Memory Size: 128 MB	Max Memory Used: 128 MB
2023-05-05 12:05:35,132 | Container was not created. Skipping deletion
2023-05-05 12:05:35,133 | Cleaning all decompressed code dirs
2023-05-05 12:05:35,133 | No response from invoke container for HelloWorldFunction
2023-05-05 12:05:35,134 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-05-05 12:05:35,182 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'f1e15fdb-5489-4c25-8c77-31b03ba12c98', 'installationId': '675e8f86-7d81-46be-8196-8db31b92f765', 'sessionId': 'cf777a7c-0974-42e1-a8b9-ce99f7409799', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.16', 'samcliVersion': '1.82.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin': None, 'projectName': '175a66beadd7eff7c9d5bce825ac4df4254d77af7f4f496409c32f6200554f7e', 'initialCommit': None}, 'duration': 14823, 'exitReason': 'success', 'exitCode': 0}}]}
2023-05-05 12:05:35,887 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

~/Code/denizdogan/deno-lambda-bug-repro main* 19s
❯ ./local.sh
deno 1.32.5 (release, aarch64-apple-darwin)
v8 11.2.214.9
typescript 5.0.3
SAM CLI, version 1.82.0
{
  "version": "1.82.0",
  "system": {
    "python": "3.8.16",
    "os": "macOS-13.3.1-arm64-arm-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "20.10.23",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  }
}
/Users/deniz/Code/denizdogan/deno-lambda-bug-repro/template.yaml is a valid SAM Template
2023-05-05 12:06:52,035 | Config file location: /Users/deniz/Code/denizdogan/deno-lambda-bug-repro/samconfig.toml
2023-05-05 12:06:52,035 | Config file '/Users/deniz/Code/denizdogan/deno-lambda-bug-repro/samconfig.toml' does not exist
2023-05-05 12:06:52,037 | Using SAM Template at /Users/deniz/Code/denizdogan/deno-lambda-bug-repro/template.yaml
2023-05-05 12:06:52,050 | Using config file: samconfig.toml, config environment: default
2023-05-05 12:06:52,050 | Expand command line arguments to:
2023-05-05 12:06:52,050 | --template_file=/Users/deniz/Code/denizdogan/deno-lambda-bug-repro/template.yaml --force_image_build --event=event.json --container_host_interface=host.docker.internal --function_logical_id=HelloWorldFunction --no_event --layer_cache_basedir=/Users/deniz/.aws-sam/layers-pkg --container_host=localhost
2023-05-05 12:06:52,050 | local invoke command is called
2023-05-05 12:06:52,052 | No Parameters detected in the template
2023-05-05 12:06:52 Requesting to create CFN template arn:aws:serverlessrepo:us-east-1:390065572566:applications/deno/1.32.5 in serverless application repo...
2023-05-05 12:06:52 Requested to create CFN template arn:aws:serverlessrepo:us-east-1:390065572566:applications/deno/1.32.5 in serverless application repo.
2023-05-05 12:06:52,983 | There is no customer defined id or cdk path defined for resource Deno, so we will use the resource logical id as the resource id
2023-05-05 12:06:52,983 | There is no customer defined id or cdk path defined for resource HelloWorldFunction, so we will use the resource logical id as the resource id
2023-05-05 12:06:52,983 | There is no customer defined id or cdk path defined for resource ServerlessHttpApi, so we will use the resource logical id as the resource id
2023-05-05 12:06:52,984 | Unable to resolve property Layers: [{'Fn::GetAtt': ['Deno', 'Outputs.LayerArn']}]. Leaving as is.
2023-05-05 12:06:52,984 | 0 stacks found in the template
2023-05-05 12:06:52,984 | No Parameters detected in the template
2023-05-05 12:06:53 Requesting to create CFN template arn:aws:serverlessrepo:us-east-1:390065572566:applications/deno/1.32.5 in serverless application repo...
2023-05-05 12:06:53 Requested to create CFN template arn:aws:serverlessrepo:us-east-1:390065572566:applications/deno/1.32.5 in serverless application repo.
2023-05-05 12:06:53,880 | There is no customer defined id or cdk path defined for resource Deno, so we will use the resource logical id as the resource id
2023-05-05 12:06:53,880 | There is no customer defined id or cdk path defined for resource HelloWorldFunction, so we will use the resource logical id as the resource id
2023-05-05 12:06:53,880 | There is no customer defined id or cdk path defined for resource ServerlessHttpApi, so we will use the resource logical id as the resource id
2023-05-05 12:06:53,881 | Unable to resolve property Layers: [{'Fn::GetAtt': ['Deno', 'Outputs.LayerArn']}]. Leaving as is.
2023-05-05 12:06:53,881 | 3 resources found in the stack
2023-05-05 12:06:53,881 | layer "{'Fn::GetAtt': ['Deno', 'Outputs.LayerArn']}" is not recognizable, it might be using intrinsic functions that we don't support yet. Skipping.
2023-05-05 12:06:53,881 | Found Serverless function with name='HelloWorldFunction' and CodeUri='.'
2023-05-05 12:06:53,881 | --base-dir is not presented, adjusting uri . relative to /Users/deniz/Code/denizdogan/deno-lambda-bug-repro/template.yaml
2023-05-05 12:06:53,902 | Found one Lambda function with name 'HelloWorldFunction'
2023-05-05 12:06:53,902 | Invoking hello.handler (provided.al2)
2023-05-05 12:06:53,902 | No environment variables found for function 'HelloWorldFunction'
2023-05-05 12:06:53,902 | Loading AWS credentials from session with profile 'None'
2023-05-05 12:06:53,913 | Resolving code path. Cwd=/Users/deniz/Code/denizdogan/deno-lambda-bug-repro, CodeUri=/Users/deniz/Code/denizdogan/deno-lambda-bug-repro
2023-05-05 12:06:53,913 | Resolved absolute path to code is /Users/deniz/Code/denizdogan/deno-lambda-bug-repro
2023-05-05 12:06:53,914 | Code /Users/deniz/Code/denizdogan/deno-lambda-bug-repro is not a zip/jar file
Building image....................
2023-05-05 12:06:55,911 | Using local image: public.ecr.aws/lambda/provided:al2-rapid-arm64.

2023-05-05 12:06:55,911 | Mounting /Users/deniz/Code/denizdogan/deno-lambda-bug-repro as /var/task:ro,delegated, inside runtime container
2023-05-05 12:06:56,153 | Starting a timer for 10 seconds for function 'HelloWorldFunction'
2023-05-05 12:07:06,156 | Function 'HelloWorldFunction' timed out after 10 seconds
END RequestId: 79a245cc-4e8a-4f0d-85f1-6e689c39e838
REPORT RequestId: 79a245cc-4e8a-4f0d-85f1-6e689c39e838	Init Duration: 0.07 ms	Duration: 10000.00 ms	Billed Duration: 10000 ms	Memory Size: 128 MB	Max Memory Used: 128 MB
2023-05-05 12:07:06,934 | Container was not created. Skipping deletion
2023-05-05 12:07:06,935 | Cleaning all decompressed code dirs
2023-05-05 12:07:06,935 | No response from invoke container for HelloWorldFunction
2023-05-05 12:07:06,936 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-05-05 12:07:06,986 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'c5b97140-a187-4a15-8919-8ee6def6965e', 'installationId': '675e8f86-7d81-46be-8196-8db31b92f765', 'sessionId': '85b0811c-c556-4879-bc7d-bad1b48316ac', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.16', 'samcliVersion': '1.82.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN', 'gitOrigin': None, 'projectName': '175a66beadd7eff7c9d5bce825ac4df4254d77af7f4f496409c32f6200554f7e', 'initialCommit': None}, 'duration': 14885, 'exitReason': 'success', 'exitCode': 0}}]}
2023-05-05 12:07:07,686 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)