Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: Unable to run sam local invoke on Apple Silicon #7061

Open
nataizya opened this issue May 13, 2024 · 14 comments
Open

Bug: Unable to run sam local invoke on Apple Silicon #7061

nataizya opened this issue May 13, 2024 · 14 comments
Labels
area/build sam build command area/local area/rust Related to Support of Rust/Cargo/Cargo Lambda in SAM CLI

Comments

@nataizya
Copy link

nataizya commented May 13, 2024

Description:

I am trying to setup a rust lambda function and to test, I am running sam local invoke. This fails with:

START RequestId: b2283bbf-ccb9-4863-ba4f-e81bc28e36ad Version: $LATEST
13 May 2024 21:35:39,195 [ERROR] (rapid) Init failed error=Runtime exited with error: signal: illegal instruction InvokeID=
13 May 2024 21:35:39,199 [ERROR] (rapid) Invoke failed error=Runtime exited with error: signal: illegal instruction InvokeID=26f73d6f-5680-4e27-9663-00bef9017d5e
13 May 2024 21:35:39,211 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure

The same command works fine on an Intel Mac.

Steps to reproduce:

On an Apple Silicon (e.g. M2) machine, setup a Rust lambda:

  1. Run sam init
  2. Select AWS Quick Start Templates
  3. Select Hello World Example
  4. Select rust (provided.al2)
  5. Select N for the following options.
  6. The app config will therefore be:
 -----------------------
    Generating application:
    -----------------------
    Name: sam-app
    Runtime: rust (provided.al2)
    Architectures: x86_64
    Dependency Manager: cargo
    Application Template: hello-world
    Output Directory: .
    Configuration file: sam-app/samconfig.toml
  1. Run sam build <--- this succeeds
  2. Run sam local invoke

Observed result:

2024-05-13 23:46:14,648 | Config file location: /Users/nataizya/Documents/Playground/Rust/SAM/sam-app/samconfig.toml          
2024-05-13 23:46:14,651 | Loading configuration values from [default.['local', 'invoke'].parameters]                          
(env.command_name.section) in config file at '/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/samconfig.toml'...        
2024-05-13 23:46:14,652 | Configuration values successfully loaded.                                                           
2024-05-13 23:46:14,652 | Configuration values are: {'stack_name': 'sam-app'}                                                 
2024-05-13 23:46:14,656 | Using SAM Template at                                                                               
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/template.yaml                                            
2024-05-13 23:46:14,702 | Using config file: samconfig.toml, config environment: default                                      
2024-05-13 23:46:14,703 | Expand command line arguments to:                                                                   
2024-05-13 23:46:14,704 | --template_file=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/template.yaml  
--no_event --layer_cache_basedir=/Users/nataizya/.aws-sam/layers-pkg --container_host=localhost                               
--container_host_interface=127.0.0.1                                                                                          
2024-05-13 23:46:14,705 | local invoke command is called                                                                      
2024-05-13 23:46:14,708 | No Parameters detected in the template                                                              
2024-05-13 23:46:14,727 | Sam customer defined id is more priority than other IDs. Customer defined id for resource           
HelloWorldFunction is HelloWorldFunction                                                                                      
2024-05-13 23:46:14,728 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use  
the resource logical id as the resource id                                                                                    
2024-05-13 23:46:14,729 | 0 stacks found in the template                                                                      
2024-05-13 23:46:14,730 | No Parameters detected in the template                                                              
2024-05-13 23:46:14,746 | Sam customer defined id is more priority than other IDs. Customer defined id for resource           
HelloWorldFunction is HelloWorldFunction                                                                                      
2024-05-13 23:46:14,746 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use  
the resource logical id as the resource id                                                                                    
2024-05-13 23:46:14,747 | 2 resources found in the stack                                                                      
2024-05-13 23:46:14,748 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'           
2024-05-13 23:46:14,748 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to                           
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/template.yaml                                            
2024-05-13 23:46:14,779 | Found one Lambda function with name 'HelloWorldFunction'                                            
2024-05-13 23:46:14,779 | Invoking bootstrap (provided.al2)                                                                   
2024-05-13 23:46:14,780 | No environment variables found for function 'HelloWorldFunction'                                    
2024-05-13 23:46:14,780 | Loading AWS credentials from session with profile 'None'                                            
2024-05-13 23:46:14,788 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build,      
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction                               
2024-05-13 23:46:14,789 | Resolved absolute path to code is                                                                   
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction                                       
2024-05-13 23:46:14,790 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build,      
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction                               
2024-05-13 23:46:14,790 | Resolved real code path to                                                                          
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction                                       
2024-05-13 23:46:14,791 | Code /Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction is not 
a zip/jar file                                                                                                                
2024-05-13 23:46:20,664 | Local image is up-to-date                                                                           
2024-05-13 23:46:20,685 | Checking free port on 127.0.0.1:5726                                                                
2024-05-13 23:46:20,695 | Using local image: public.ecr.aws/lambda/provided:al2-rapid-x86_64.                                 
                                                                                                                              
2024-05-13 23:46:20,697 | Mounting /Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction as 
/var/task:ro,delegated, inside runtime container                                                                              
2024-05-13 23:46:20,949 | Starting a timer for 3 seconds for function 'HelloWorldFunction'                                    
2024-05-13 23:46:20,953 | Getting lock for the key localhost-5726                                                             
2024-05-13 23:46:20,955 | Waiting to retrieve the lock (localhost-5726) to start invocation                                   
2024-05-13 23:46:21,068 | Getting lock for the key localhost-5726                                                             
2024-05-13 23:46:21,072 | Waiting to retrieve the lock (localhost-5726) to start invocation                                   
START RequestId: 2d02aba6-328d-4a67-b94f-91b4058b407e Version: $LATEST
13 May 2024 21:46:21,114 [ERROR] (rapid) Init failed error=Runtime exited with error: signal: illegal instruction InvokeID=
13 May 2024 21:46:21,118 [ERROR] (rapid) Invoke failed error=Runtime exited with error: signal: illegal instruction InvokeID=3bde647e-a338-4ff7-b860-7e24e6d8ff10
13 May 2024 21:46:21,120 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure
2024-05-13 23:46:21,122 | Failed to deserialize response from RIE, returning the raw response as is                           

2024-05-13 23:46:22,157 | [Container state] OOMKilled False                                                                   
2024-05-13 23:46:22,288 | Cleaning all decompressed code dirs                                                                 
2024-05-13 23:46:22,291 | Telemetry endpoint configured to be                                                                 
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                        
2024-05-13 23:46:22,404 | Telemetry endpoint configured to be                                                                 
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                        
2024-05-13 23:46:22,406 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '88d70caf-438d-43c5-bdb9-cd8b910a8459',
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': 'eab61e43-2617-4a4b-8b64-239884235d30',                
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'awsProfileProvided': False,   
'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN',
'gitOrigin': None, 'projectName': '593ab2ca51e925b9f6c2f258bc55ed5926cf6d2c78239a685d65907e4ec7edd3', 'initialCommit': None}, 
'duration': 7588, 'exitReason': 'success', 'exitCode': 0}}]}                                                                  
2024-05-13 23:46:22,407 | Unable to find Click Context for getting session_id.                                                
2024-05-13 23:46:22,411 | Sending Telemetry: {'metrics': [{'events': {'requestId': 'a5e10d43-7cbc-43b6-be11-7dabe9d0a741',    
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': 'eab61e43-2617-4a4b-8b64-239884235d30',                
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'commandName': 'sam local      
invoke', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id':
'ba1472d1be8a4258bce10fcb3747afbc', 'time_stamp': '2024-05-13 21:46:14.647', 'exception_name': None}, {'event_name':          
'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '4d7d2b65326940d7aebc4265646dfc81', 'time_stamp': '2024-05-13  
21:46:14.702', 'exception_name': None}]}}}]}                                                                                  
2024-05-13 23:46:23,369 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read  
timed out. (read timeout=0.1)                                                                                                 
2024-05-13 23:46:23,375 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read  
timed out. (read timeout=0.1)                                                                                                 
2024-05-13 23:46:23,379 | Telemetry endpoint configured to be                                                                 
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                        
2024-05-13 23:46:23,381 | Sending Telemetry: {'metrics': [{'runtimeMetric': {'requestId':                                     
'a53adc53-679e-478a-9032-a01c08bee7d2', 'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId':                
'eab61e43-2617-4a4b-8b64-239884235d30', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion':   
'1.116.0', 'runtimes': ['provided.al2']}}]}                                                                                   
2024-05-13 23:46:24,182 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read  
timed out. (read timeout=0.1)  

Expected result:

Successfully run the sam local invoke command and run the lambda function.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: MacOS 14.4.1 (23E224)
  2. sam --version: SAM CLI, version 1.116.0
  3. AWS region: n/a
{
  "version": "1.116.0",
  "system": {
    "python": "3.8.13",
    "os": "macOS-14.4.1-x86_64-i386-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "24.0.7",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

Add --debug flag to command you are running

@nataizya nataizya added the stage/needs-triage Automatically applied to new issues and PRs, indicating they haven't been looked at. label May 13, 2024
@mndeveci
Copy link
Contributor

Thanks for reporting this issue.

Can you try changing your runtime to provided.al2023 to see if it works? I think there was a change in the rust builder we used under the hood, they changed default runtime settings to be aligned with newer runtimes, which broke the provided.al2 ones.

We've merged a fix to align with the change above, however it is not been released yet. If you confirm that changing to newer runtime works, we can assume that new SAM CLI release should fix the issue.

@mndeveci mndeveci added blocked/more-info-needed More info is needed from the requester. If no response in 14 days, it will become stale. area/local area/rust Related to Support of Rust/Cargo/Cargo Lambda in SAM CLI area/build sam build command and removed stage/needs-triage Automatically applied to new issues and PRs, indicating they haven't been looked at. labels May 14, 2024
@nataizya
Copy link
Author

nataizya commented May 14, 2024

Thanks for responding @mndeveci.

I did try it with provided.al2023 as well and faced the same issue.

Logs:

2024-05-13 23:44:07,411 | Config file location: /Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/samconfig.toml        
2024-05-13 23:44:07,415 | Loading configuration values from [default.['local', 'invoke'].parameters]                          
(env.command_name.section) in config file at '/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/samconfig.toml'...      
2024-05-13 23:44:07,416 | Configuration values successfully loaded.                                                           
2024-05-13 23:44:07,416 | Configuration values are: {'stack_name': 'sam-app-2'}                                               
2024-05-13 23:44:07,420 | Using SAM Template at                                                                               
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/template.yaml                                          
2024-05-13 23:44:07,480 | Using config file: samconfig.toml, config environment: default                                      
2024-05-13 23:44:07,481 | Expand command line arguments to:                                                                   
2024-05-13 23:44:07,481 | --template_file=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/template.yaml
--no_event --layer_cache_basedir=/Users/nataizya/.aws-sam/layers-pkg --container_host=localhost                               
--container_host_interface=127.0.0.1                                                                                          
2024-05-13 23:44:07,482 | local invoke command is called                                                                      
2024-05-13 23:44:07,486 | No Parameters detected in the template                                                              
2024-05-13 23:44:07,505 | Sam customer defined id is more priority than other IDs. Customer defined id for resource           
HelloWorldFunction is HelloWorldFunction                                                                                      
2024-05-13 23:44:07,506 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use  
the resource logical id as the resource id                                                                                    
2024-05-13 23:44:07,507 | 0 stacks found in the template                                                                      
2024-05-13 23:44:07,508 | No Parameters detected in the template                                                              
2024-05-13 23:44:07,524 | Sam customer defined id is more priority than other IDs. Customer defined id for resource           
HelloWorldFunction is HelloWorldFunction                                                                                      
2024-05-13 23:44:07,524 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use  
the resource logical id as the resource id                                                                                    
2024-05-13 23:44:07,525 | 2 resources found in the stack                                                                      
2024-05-13 23:44:07,526 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'           
2024-05-13 23:44:07,526 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to                           
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/template.yaml                                          
2024-05-13 23:44:07,539 | Found one Lambda function with name 'HelloWorldFunction'                                            
2024-05-13 23:44:07,540 | Invoking bootstrap (provided.al2023)                                                                
2024-05-13 23:44:07,540 | No environment variables found for function 'HelloWorldFunction'                                    
2024-05-13 23:44:07,541 | Loading AWS credentials from session with profile 'None'                                            
2024-05-13 23:44:07,548 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build,    
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction                             
2024-05-13 23:44:07,549 | Resolved absolute path to code is                                                                   
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction                                     
2024-05-13 23:44:07,550 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build,    
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction                             
2024-05-13 23:44:07,550 | Resolved real code path to                                                                          
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction                                     
2024-05-13 23:44:07,551 | Code /Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction is   
not a zip/jar file                                                                                                            
2024-05-13 23:44:13,273 | Local image is up-to-date                                                                           
2024-05-13 23:44:13,292 | Checking free port on 127.0.0.1:8347                                                                
2024-05-13 23:44:13,299 | Using local image: public.ecr.aws/lambda/provided:al2023-rapid-x86_64.                              
                                                                                                                              
2024-05-13 23:44:13,301 | Mounting /Users/nataizya/Documents/Playground/Rust/SAM/sam-app-2/.aws-sam/build/HelloWorldFunction  
as /var/task:ro,delegated, inside runtime container                                                                           
2024-05-13 23:44:13,698 | Starting a timer for 3 seconds for function 'HelloWorldFunction'                                    
2024-05-13 23:44:13,701 | Getting lock for the key localhost-8347                                                             
2024-05-13 23:44:13,702 | Waiting to retrieve the lock (localhost-8347) to start invocation                                   
START RequestId: 55fe8a55-ca1d-436c-a3ed-eec2c3fda782 Version: $LATEST
13 May 2024 21:44:13,759 [ERROR] (rapid) Init failed error=Runtime exited with error: signal: illegal instruction InvokeID=
13 May 2024 21:44:13,762 [ERROR] (rapid) Invoke failed InvokeID=261e53eb-7616-45bf-96fc-b24359daf3d3 error=Runtime exited with error: signal: illegal instruction
13 May 2024 21:44:13,763 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure
2024-05-13 23:44:13,767 | Failed to deserialize response from RIE, returning the raw response as is                           

2024-05-13 23:44:14,790 | [Container state] OOMKilled False                                                                   
2024-05-13 23:44:14,925 | Cleaning all decompressed code dirs                                                                 
2024-05-13 23:44:14,927 | Telemetry endpoint configured to be                                                                 
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                        
2024-05-13 23:44:15,027 | Telemetry endpoint configured to be                                                                 
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                        
2024-05-13 23:44:15,028 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'dc3cb305-75d5-4d40-b3b7-d281511cda09',
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': '699a40ef-6b4d-474d-8610-c8a5640b79f5',                
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'awsProfileProvided': False,   
'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN',
'gitOrigin': None, 'projectName': '9e7669abfc305db5598caf057dd1af7044b0710b8c9f99f979d86cc7bb8b69ca', 'initialCommit': None}, 
'duration': 7446, 'exitReason': 'success', 'exitCode': 0}}]}                                                                  
2024-05-13 23:44:15,028 | Unable to find Click Context for getting session_id.                                                
2024-05-13 23:44:15,032 | Sending Telemetry: {'metrics': [{'events': {'requestId': '48b8995e-d5d3-43ca-9aba-def06169a914',    
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': '699a40ef-6b4d-474d-8610-c8a5640b79f5',                
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'commandName': 'sam local      
invoke', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id':
'4a072847aba84d9eae8720975249e7d7', 'time_stamp': '2024-05-13 21:44:07.411', 'exception_name': None}, {'event_name':          
'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': 'd0e7eb37fe8247958066a2e7d2574cd1', 'time_stamp': '2024-05-13  
21:44:07.480', 'exception_name': None}]}}}]}                                                                                  
2024-05-13 23:44:15,977 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read  
timed out. (read timeout=0.1)                                                                                                 
2024-05-13 23:44:15,978 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read  
timed out. (read timeout=0.1)                                                                                                 
2024-05-13 23:44:15,986 | Telemetry endpoint configured to be                                                                 
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                        
2024-05-13 23:44:15,988 | Sending Telemetry: {'metrics': [{'runtimeMetric': {'requestId':                                     
'd840b3ea-0bfe-4bde-8fe8-33eaee4876d5', 'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId':                
'699a40ef-6b4d-474d-8610-c8a5640b79f5', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion':   
'1.116.0', 'runtimes': ['provided.al2023']}}]}                                                                                
2024-05-13 23:44:16,792 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read  
timed out. (read timeout=0.1)                                                                                                 
nataizya@Nataizyas-MacBook-Air sam-app-2 % cd ../sam-app
nataizya@Nataizyas-MacBook-Air sam-app % sam local invoke --debug 
2024-05-13 23:46:14,648 | Config file location: /Users/nataizya/Documents/Playground/Rust/SAM/sam-app/samconfig.toml          
2024-05-13 23:46:14,651 | Loading configuration values from [default.['local', 'invoke'].parameters]                          
(env.command_name.section) in config file at '/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/samconfig.toml'...        
2024-05-13 23:46:14,652 | Configuration values successfully loaded.                                                           
2024-05-13 23:46:14,652 | Configuration values are: {'stack_name': 'sam-app'}                                                 
2024-05-13 23:46:14,656 | Using SAM Template at                                                                               
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/template.yaml                                            
2024-05-13 23:46:14,702 | Using config file: samconfig.toml, config environment: default                                      
2024-05-13 23:46:14,703 | Expand command line arguments to:                                                                   
2024-05-13 23:46:14,704 | --template_file=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/template.yaml  
--no_event --layer_cache_basedir=/Users/nataizya/.aws-sam/layers-pkg --container_host=localhost                               
--container_host_interface=127.0.0.1                                                                                          
2024-05-13 23:46:14,705 | local invoke command is called                                                                      
2024-05-13 23:46:14,708 | No Parameters detected in the template                                                              
2024-05-13 23:46:14,727 | Sam customer defined id is more priority than other IDs. Customer defined id for resource           
HelloWorldFunction is HelloWorldFunction                                                                                      
2024-05-13 23:46:14,728 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use  
the resource logical id as the resource id                                                                                    
2024-05-13 23:46:14,729 | 0 stacks found in the template                                                                      
2024-05-13 23:46:14,730 | No Parameters detected in the template                                                              
2024-05-13 23:46:14,746 | Sam customer defined id is more priority than other IDs. Customer defined id for resource           
HelloWorldFunction is HelloWorldFunction                                                                                      
2024-05-13 23:46:14,746 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use  
the resource logical id as the resource id                                                                                    
2024-05-13 23:46:14,747 | 2 resources found in the stack                                                                      
2024-05-13 23:46:14,748 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'           
2024-05-13 23:46:14,748 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to                           
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/template.yaml                                            
2024-05-13 23:46:14,779 | Found one Lambda function with name 'HelloWorldFunction'                                            
2024-05-13 23:46:14,779 | Invoking bootstrap (provided.al2)                                                                   
2024-05-13 23:46:14,780 | No environment variables found for function 'HelloWorldFunction'                                    
2024-05-13 23:46:14,780 | Loading AWS credentials from session with profile 'None'                                            
2024-05-13 23:46:14,788 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build,      
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction                               
2024-05-13 23:46:14,789 | Resolved absolute path to code is                                                                   
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction                                       
2024-05-13 23:46:14,790 | Resolving code path. Cwd=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build,      
CodeUri=/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction                               
2024-05-13 23:46:14,790 | Resolved real code path to                                                                          
/Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction                                       
2024-05-13 23:46:14,791 | Code /Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction is not 
a zip/jar file                                                                                                                
2024-05-13 23:46:20,664 | Local image is up-to-date                                                                           
2024-05-13 23:46:20,685 | Checking free port on 127.0.0.1:5726                                                                
2024-05-13 23:46:20,695 | Using local image: public.ecr.aws/lambda/provided:al2-rapid-x86_64.                                 
                                                                                                                              
2024-05-13 23:46:20,697 | Mounting /Users/nataizya/Documents/Playground/Rust/SAM/sam-app/.aws-sam/build/HelloWorldFunction as 
/var/task:ro,delegated, inside runtime container                                                                              
2024-05-13 23:46:20,949 | Starting a timer for 3 seconds for function 'HelloWorldFunction'                                    
2024-05-13 23:46:20,953 | Getting lock for the key localhost-5726                                                             
2024-05-13 23:46:20,955 | Waiting to retrieve the lock (localhost-5726) to start invocation                                   
2024-05-13 23:46:21,068 | Getting lock for the key localhost-5726                                                             
2024-05-13 23:46:21,072 | Waiting to retrieve the lock (localhost-5726) to start invocation                                   
START RequestId: 2d02aba6-328d-4a67-b94f-91b4058b407e Version: $LATEST
13 May 2024 21:46:21,114 [ERROR] (rapid) Init failed error=Runtime exited with error: signal: illegal instruction InvokeID=
13 May 2024 21:46:21,118 [ERROR] (rapid) Invoke failed error=Runtime exited with error: signal: illegal instruction InvokeID=3bde647e-a338-4ff7-b860-7e24e6d8ff10
13 May 2024 21:46:21,120 [ERROR] (rapid) Invoke DONE failed: Sandbox.Failure
2024-05-13 23:46:21,122 | Failed to deserialize response from RIE, returning the raw response as is                           

2024-05-13 23:46:22,157 | [Container state] OOMKilled False                                                                   
2024-05-13 23:46:22,288 | Cleaning all decompressed code dirs                                                                 
2024-05-13 23:46:22,291 | Telemetry endpoint configured to be                                                                 
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                        
2024-05-13 23:46:22,404 | Telemetry endpoint configured to be                                                                 
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                        
2024-05-13 23:46:22,406 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '88d70caf-438d-43c5-bdb9-cd8b910a8459',
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': 'eab61e43-2617-4a4b-8b64-239884235d30',                
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'awsProfileProvided': False,   
'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'metricSpecificAttributes': {'projectType': 'CFN',
'gitOrigin': None, 'projectName': '593ab2ca51e925b9f6c2f258bc55ed5926cf6d2c78239a685d65907e4ec7edd3', 'initialCommit': None}, 
'duration': 7588, 'exitReason': 'success', 'exitCode': 0}}]}                                                                  
2024-05-13 23:46:22,407 | Unable to find Click Context for getting session_id.                                                
2024-05-13 23:46:22,411 | Sending Telemetry: {'metrics': [{'events': {'requestId': 'a5e10d43-7cbc-43b6-be11-7dabe9d0a741',    
'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId': 'eab61e43-2617-4a4b-8b64-239884235d30',                
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.116.0', 'commandName': 'sam local      
invoke', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id':
'ba1472d1be8a4258bce10fcb3747afbc', 'time_stamp': '2024-05-13 21:46:14.647', 'exception_name': None}, {'event_name':          
'SamConfigFileExtension', 'event_value': '.toml', 'thread_id': '4d7d2b65326940d7aebc4265646dfc81', 'time_stamp': '2024-05-13  
21:46:14.702', 'exception_name': None}]}}}]}                                                                                  
2024-05-13 23:46:23,369 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read  
timed out. (read timeout=0.1)                                                                                                 
2024-05-13 23:46:23,375 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read  
timed out. (read timeout=0.1)                                                                                                 
2024-05-13 23:46:23,379 | Telemetry endpoint configured to be                                                                 
https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics                                                        
2024-05-13 23:46:23,381 | Sending Telemetry: {'metrics': [{'runtimeMetric': {'requestId':                                     
'a53adc53-679e-478a-9032-a01c08bee7d2', 'installationId': '200cdfea-e657-47bc-954f-69c7ded8a763', 'sessionId':                
'eab61e43-2617-4a4b-8b64-239884235d30', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion':   
'1.116.0', 'runtimes': ['provided.al2']}}]}                                                                                   
2024-05-13 23:46:24,182 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read  
timed out. (read timeout=0.1)

@mndeveci
Copy link
Contributor

Thanks for confirmation!

Can you also confirm what happens when you deploy this function? As far as we can see, when this function is deployed it works as expected, however running locally fails with the error that you mentioned above.

From our side we validated following scenarios;

  • A: Building on Apple Silicon Macbook works (sam build), locally invoking fails (sam local invoke)
  • B: Building on Intel Macbook works (sam build), locally invoking succeeds (sam local invoke)
  • C: Deploying the artifact from A works as expected, deployed lambda function can be invoked
  • D: Using the built artifact from A on an Intel Macbook works for sam local invoke

@mndeveci
Copy link
Contributor

@nataizya we've tried with a Docker alternative, and it worked without issues. I believe this is coming from Docker Desktop's Apple Silicon distribution since it also works on Intel machines.

@nataizya
Copy link
Author

I suspect that the deployment will work as well though I havent tested that.

Can you elaborate on what exactly you mean by trying it with a Docker alternative? What commands would I be using to test that?

@mndeveci
Copy link
Contributor

Before going further down the list, you may try to uninstall/remove Docker Desktop completely and install it from their official installers, to see if it is going to fix your problem (I've seen that sometimes a clean installation fixes their issues).

I can't open an issue on their repository since they are asking more details which I can't provide since I am using an Intel Macbook. If you can, I will recommend creating an issue in this repository here; https://github.com/docker/for-mac/issues

Can you elaborate on what exactly you mean by trying it with a Docker alternative? What commands would I be using to test that?

There are alternative tools which can replace Docker Desktop, and they provide similar experience to use containers on your machine. Some of those are podman or rancher.

@aquanow-jeffen
Copy link

Local invocation works for me after change to Architectures: - arm64. I had to remove all the containers before run the invoke command.

@nataizya
Copy link
Author

Setting the architecture to arm64 allows it to work. Thanks!!!

@mndeveci
Copy link
Contributor

Please keep in mind that, if you change Architecture value in the template.yaml it will also change the architecture of the deployed function. Architecture of the developer's local machine can be different from what they are planning to deploy in their account. In this case, I believe Docker Desktop for Apple Silicon has some issues with executing cross architecture code.

@nataizya
Copy link
Author

Noted. I will try and make use of podman or rancher. But for now I can at least test the function locally by temporarily switching to arm64 in the template.yaml.

@mndeveci mndeveci added stage/needs-investigation Requires a deeper investigation and removed blocked/more-info-needed More info is needed from the requester. If no response in 14 days, it will become stale. labels May 17, 2024
@mndeveci
Copy link
Contributor

mndeveci commented May 17, 2024

As another workaround (which might be better since this doesn't require architecture change), you can also build Rust x86 function with SAM_BUILD_MODE=debug sam build which will build in debug mode, and it looks like it works with sam local invoke.

!!! Please don't use SAM_BUILD_MODE=debug when doing a deployment as it removes the --release flag for Rust compiler. !!!

@mndeveci
Copy link
Contributor

I think we found the root cause (thanks to @lucashuy).

There was a change in cargo-lambda package which is used for building Rust functions under the hood. This PR changed the default target cpu (microarch) to be haswell. It looks like this is causing issues when executable built with that flag. If it is changed to x86_64 then it works fine.

However the reason for the change above was to make it more optimized for the target lambda runtime. So this code works when it is deployed (as intended) however when it is tried to be emulated on Apple Silicon machines, it doesn't work (possible) missing support from Docker Desktop.

We don't want to change default target since this was selected due to optimizations, for now we would ask customers to use workaround to build with SAM_BUILD_MODE=debug sam build when testing their lambda functions locally.

@nataizya
Copy link
Author

Thanks @mndeveci .

I can confirm that using SAM_BUILD_MODE=debug sam build does work for local testing.

@lucashuy lucashuy removed the stage/needs-investigation Requires a deeper investigation label Jun 3, 2024
@lucashuy
Copy link
Contributor

lucashuy commented Jun 3, 2024

Thanks all for working with us on this issue. Just to sum up for future folks popping into this issue: SAM CLI builds Rust Functions for "release mode", meaning that specific architecture level optimizations are applied when building the binary. Unfortunately, these optimizations do not work when invoking them in an x86 container, on an arm64 Mac.

Workaround

Before building the Lambda function, set the SAM_BUILD_MODE=debug environment variable before running sam build, like such: SAM_BUILD_MODE=debug sam build

Please note that this is only recommended when building and testing locally. We advise customers to switch back to a release build by unsetting or removing the SAM_BUILD_MODE environ variable.

Whats next

Our team is going to prioritize adding a disclaimer message saying the essentially the same thing inside of sam local.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/build sam build command area/local area/rust Related to Support of Rust/Cargo/Cargo Lambda in SAM CLI
Projects
None yet
Development

No branches or pull requests

4 participants