Added more detail for a working SAM template example for a serverless lambda

[wattry]

• Added a makefile and deno directory
• Tested SAM template deployment

[hayd]

Does the SAM example deployment no longer work?
(It should really be added to the test suite. I'd missed that it wasn't tested.)

I don't understand the need for this mv and empty makefile. Is that required/why does that help? Could you explain? Thanks!

Edit: I tested the existing code with sam (version 0.44) and it worked... so I am confused at the issue! (I would definitely like to include it in the test suite somehow.)

Edit2: I tested this and sam build --template template.yml --region us-west-2 still errors for me, is that expected?

[wattry]

Does the SAM example deployment no longer work?
(It should really be added to the test suite. I'd missed that it wasn't tested.)

I don't understand the need for this mv and empty makefile. Is that required/why does that help? Could you explain? Thanks!

Edit: I tested the existing code with sam (version 0.44) and it worked... so I am confused at the issue! (I would definitely like to include it in the test suite somehow.)

Edit2: I tested this and sam build --template template.yml --region us-west-2 still errors for me, is that expected?

With regards to Edit1, I am relatively new to AWS and Deno so I have no problem figuring out how to add this to the test suit when I have some free time.

With regards to Edit2: So that is my bad, I removed something in the makefile which apparently is required. I will push up a fix for this.

Edit > I don't understand the need for this mv and empty makefile. Is that required/why does that help? Could you explain? Thanks!

Does the move make sense in the context of the issue?

[hayd]

Can you move these back up a directory? (I think it should work without the /deno/ change.)

If not, I can fix it later. Thanks again for the investigation. 👍

---

What I think we really need here is something like (Makefile):

build-HelloWorldFunction:
	DENO_DIR=.deno_dir deno cache hello.ts
	cp -R .deno_dir/gen/file$(ARTIFACTS_DIR)/ .deno_dir/LAMBDA_TASK_ROOT
	cp -R .deno_dir/gen/file$(ARTIFACTS_DIR)/ .deno_dir/LAMBDA_TASK_ROOT

But, at least for me (on OSX), this fails due to make using /private/var directory and confusing deno's compilation, which is worrisome. This means that compilation will not be cached.

[hayd]

I've got it:

build-HelloWorldFunction:
        DENO_DIR=.deno_dir deno cache hello.ts
        cp -R .deno_dir/gen/file$(CURDIR)/ .deno_dir/LAMBDA_TASK_ROOT

[wattry]

Please correct me if my understanding of SAM is incorrect, does using CodeUri: . bundle everything in the current directory into the deployment?

Is there a specific reason the Lambda needs to have the .npmignore or README.md files stored? I suppose that was my rationale behind the ./deno directory.

[hayd]

sam respects .npmignore, so those wouldn't be included in the deployment.

[wattry]

sam respects .npmignore, so those wouldn't be included in the deployment.

That's good to know! Thanks.

[wattry]

I've got it:

build-HelloWorldFunction:
        DENO_DIR=.deno_dir deno cache hello.ts
        cp -R .deno_dir/gen/file$(CURDIR)/ .deno_dir/LAMBDA_TASK_ROOT

Is this indentation purposeful, since I see you've used 2 spaces in the rest of the codebase?

[hayd]

At least my Makefile required tabs 😞

[wattry]

Weird, I'm using VSCodium and it did not complain. Even weirder my build succeeded:

Building function 'HelloWorldFunction'
Running CustomMakeBuilder:CopySource
Running CustomMakeBuilder:MakeBuild
Current Artifacts Directory : ~/deno-lambda/example-sam/.aws-sam/build/HelloWorldFunction

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided

Edit: I am also using OSX

Edit1: It looks like the ~/deno-lambda/example-sam/.aws-sam/build/HelloWorldFunction directory is empty.

[hayd]

and still sam local invoke doesn't work:

{"errorType":"exitError",
 "errorMessage":"RequestId: 981552a1-0545-12ff-bcf5-8ecb4bf88539
                 Error: Couldn't find valid bootstrap(s): [/var/task/bootstrap /opt/bootstrap]"}

strange. Are we going down the wrong path here? Very confusing.

Note: sam deploy still works.

I don't really understand! 😢

[wattry]

and still sam local invoke doesn't work:

{"errorType":"exitError",
 "errorMessage":"RequestId: 981552a1-0545-12ff-bcf5-8ecb4bf88539
                 Error: Couldn't find valid bootstrap(s): [/var/task/bootstrap /opt/bootstrap]"}

strange. Are we going down the wrong path here? Very confusing.

Note: sam deploy still works.

I don't really understand! 😢

I gave up on SAM local functionality a while ago. There are too many moving parts and it's not stable enough in complex builds.

I just had a laugh because I spent a good chunk of yesterday trying to compile deno for AWS and it lead me to your issue and eventually this repo. I didn't connect your account to the issue until right now.

[hayd]

😄 . Yeah compiling is for sure a pain, we publish a binary in releases.

I guess the question here is what is the point of sam build ? (If it doesn't work locally.)
It's frustrating this doesn't give better errors...

---

Actually, I take back that sam deploy still works, I get:

UPDATE_FAILED                      AWS::Lambda::Function              HelloWorldFunction                 Uploaded file must be a non-
                                                                                                         empty zip (Service:
                                                                                                         AWSLambdaInternal; Status Code:
                                                                                                         400; Error Code:
                                                                                                         InvalidParameterValueException;
                                                                                                         Request ID: 17dfc38d-fb08-4a63-b
                                                                                                         cf4-9815ada55316)

which sounds like the same issue (.aws-sam/build/HelloWorldFunction directory is empty.)

I'm optimistic that if we can work this out then local invoke will work...

[wattry]

@hayd yeah they are handy! I was a little hesitant to have someone else's lambda layer executing possibly sensitive API requests. However, this is so far in dev that it's unlikely we will be moving from Node to Deno anytime soon.

So I am relatively new to the SAM world. We tried running local templates but it just created headaches because some of the intrinsic functions relied on resources that were created in other accounts or that did not work for the resources that we needed yet.

I used a tool called dive to check the image and it looks like the lambda layer isn't getting pulled down/ created. That's what is throwing the error you posted, we had issues referencing layers locally, last I checked you needed an ARN version.

[hayd]

I think it actually may be this issue aws/aws-sam-cli#1489

Since the lambda layer is referenced by an SAR application. That would be annoying. I am loath to publish a lambda layer explicitly as I have no idea how much that could end up costing.

[wattry]

I think it actually may be this issue awslabs/aws-sam-cli#1489

Since the lambda layer is referenced by an SAR application. That would be annoying. I am loath to publish a lambda layer explicitly as I have no idea how much that could end up costing.

Agreed, I see now I have conflated two issues here. Most likely due to my limited work with SAM and Deno. The issue I originally was trying to solve is that build requires a runtime at build time, however we are not creating the runtime for the lambda we are accessing the runtime from a lambda layer.

The makefile inclusion is a workaround I was using to trick SAM into thinking it was building the runtime locally but I was getting it from the reference to the SAR. However, that is not actually pulling down the layer and on top of that, I see now in your docs that you need to compile the deno function and dependencies. This means that the deploy and the function work but it must be compiling the code itself.

What we could do is provide the lambda layer as an input parameter? Then if we can get the makefile to actually do the work you provided we would likely be able to run it locally.

[wattry]

The first problem was solved by just referencing the AWS lambda layer ARN as an input parameter. Until I can figure out how better to do this I'm going to use this method, I'm very open to suggestions. Now the layer gets pulled down and it is a top of the base image.

After much head bashing (excuse the pun) I have come to this makefile:

build-HelloWorldFunction:
				DENO_DIR=.deno_dir deno cache hello.ts
				mkdir .deno_dir/LAMBDA_TASK_ROOT
				cp -R .deno_dir/gen/file$(CURDIR)/ .deno_dir/LAMBDA_TASK_ROOT
				cp -R .deno_dir/ $(ARTIFACTS_DIR)/.deno_dir

The reason why the original makefile did not work is that cp cannot recursively make the nested directory .deno_dir/LAMBDA_TASK_ROOT so in a long-winded way I solved that issue by making the directory before the copy.

Wonderfully now you can see the docker image now has the bootstrap file, the deno executable, and the compiled code.

This is all great, however, when I run the deploy the ARTIFACTS_DIR is not uploaded to the lambda and it is empty.

I followed the following example and copied the example-sam project and was able to get this to work. However, they are not using a custom lambda layer as the runtime they are instead building the custom bootstrap as part of the lambda build.

From my observations, it seems that if I manually compile the code for the deno application it will still pick up the uncompiled file and compile it. This leads me to believe that since the lambda is hot the then compiled code is executed on subsequent requests. This may be a design limitation on SAM's part or something that needs to be requested.

Edit1: I deployed the rust application to AWS and I noticed there was a function name property on the SAM template, which was not on the HelloWorld template once I added that the compiled code was pushed to AWS.

Edit 2: I have isolated the issue to the bootstrap file, I'm not sure yet how to debug it but because I'm passing transpiled code to the lambda there is no handler file that may be what is causing the error I am seeing in the lambda.

[wattry]

@hayd I was able to get this working in a repo I made. I didn't want to throw up changes to the pull request that you didn't agree with. Let me know if you have any comments/thoughts on that.

[hayd]

Thanks. I'm going to sit on this for a while.

It's a shame that SAR layer doesn't "just work" since IME it's kinda a pain to manage layer versions (especially when they can be reuploaded frequently, it's annoying they don't share more efficiently). It's nice that SAR references by version string rather than meaningless incremental number. I'm not happy you would have to upload the deno binary everytime (?). /rant

I feel like there has to be a better way.

---

Just to step back: what is the benefit of building vs only deploying?

[wattry]

Just to step back: what is the benefit of building vs only deploying?

I apologise for the length here.

TLDR;

The build step is not required for Deno because all dependencies are listed in the src code so at runtime deno will simply download and transpile. Use the makefile at the bottom of this post and update the bootstrap file with suggestions.

Reasoning

The build step is crucial and language-specific. Let me illustrate this with a Node project since this is my background. Ignore the runtime for the moment, assume we have the need for a specific version of Node because of some dependency.

1. SAM uses the subfile and 'manifest' file i.e. nodejs and package.json respectively to decide on a workflow you can see this here.
2. On sam build an npm install creates node_modules in .aws-sam/build and copies the src code. A new template.yaml is created with paths relative to the build code.
3. Asam deploy before you run the build will use the original template.yml which will push up the src code directory (this is crucial to understand why Deno will still work)
4. Once the code is built you can run sam deploy --guided, using the new template.yaml in .aws-sam/build the function directory & contents are pushed to an AWS S3 bucket.
5. Assuming the application is built and deployed it can be invoked remotely.
6. Delete the .aws-sam directory and invoke locally there will be a dependency error, deploy it and the same will happen in AWS.

Deno has a different execution cycle due to the way package management works, or lack thereof.

1. Deno dependencies are cached and transpiled on deno run, thus if you don't sam build the runtime will. This is why a deploy still works.
   2.DENO_DIR=deno_dir deno cache hello.ts results in transpiled code referenced by absolute file paths in a nested path based on where it was transpiled.
2. Thus when we do the cp in the makefile, deno will never be able to actually use the .deno_dir transpiled code because those paths don't exist.
3. I saw some logic in the bootstrap file that seemed to deal with this, however, I do not believe this will call the compiled code because of the absolute pathing, correct me if I'm wrong.

We can use the bundle subcommand to wrap all dependencies and transpile them in the makefile then store it as JS:

build-HelloWorldFunction:
        DENO_DIR=.deno_dir deno bundle hello.ts hello.js
	mv hello.js hello.ts $(ARTIFACTS_DIR)

We also need to have a logical test in the bootstrap script that tests to see if there is a .js file first then fall back on the .ts file else throw an error. Secondly, the cp logic can be removed, and lastly, the hello.ts can be removed from the bundled layer. Or alternatively, you could provide an alternative output bundle.

To test this I ran a sam build then manually updated the hello.ts file with a console.log("This shouldn't run"). Then invoked the lambda, the console message was not logged but it was still successful thus I know it used the bundled code.

I know that this is a long post so I'll give you a while to digest it, if I have any incorrect assumptions please correct me.

[hayd]

There is HANDLER_EXT env variable which you can set to js https://github.com/hayd/deno-lambda#configuration

I am erring (for the moment) two template files

1. using SAR (which only deploy works, with the caveat you need to deno cache first - or perhaps this can be added as a makefile too).
2. where the makefile downloads (from github releases) and expands the layer, so build and local invoke work. Note: Potentially it should cache based on the version number.

In Example 2 it looks like you can define the makefile inside the template, that way the version is in the template which is IMO desirable. (Oh, it looks like you already use that feature above 👍 )

I still think this is a bug in SAM (that they don't support SAR layers).