Is It possible to sequence the creation of permissions for a Lambda with multiple API endpoints in a SAM template?

[mmcglone]

I have a SAM template that contains a Lambda with multiple (around 20) API endpoints. For the most part, this setup has been working well for me. However, lately when I deploy this template, CFN intermittently receives one or two 500 errors from Lambda. According to AWS support, the errors are due to a race condition connected to the concurrent calls to AddPermissions made by CFN. In a standard CFN template, I could resolve this issue by sequencing the creation of the Lambda permissions with "DependsOn". Is there a way to sequence the creation of the Lambda permissions in my SAM template without splitting the API endpoints across multiple Lambdas?

[kaihendry]

Not sure what you mean by endpoints, so you're not using a catchall like https://github.com/wolfeidau/sam-golang-multi-lambda/blob/main/template.yaml#L27 ?

[mmcglone]

@kaihendry Good question. Thanks. No, I'm not currently using a catchall like that example. In saying that the Lambda has multiple "endpoints" I meant that in the SAM template the AWS::Serverless::Function has multiple Events of type Api. The Events have different Path and Method values. I'm wondering whether I can keep those distinct Events on that AWS::Serverless::Function and find a way of sequencing the corresponding creation of Lambda permissions in my SAM template.

[mmcglone]

@kaihendry Is there a public explanation of why having separate paths and methods would increase cold starts? Or is this just something shown to be true through latency testing?

[kaihendry]

I'm assuming the separate paths are wired to separate functions in your case?

[mmcglone]

It's one Lambda Function with one Node handler with all of the different http paths/methods being handled by the same Node handler.

[kaihendry]

I'm puzzled why you don't use a catchall then? Perhaps I am completely misunderstanding you. Can you perhaps have a gist showing your template and handler code?

[mmcglone]

That may be where I end up. It's just not where I started. Thanks

[theburningmonk]

I'm not sure there's a way to add "DependsOn" to the LambdaPermission resources that SAM generates, but I think there are two solutions available here without forcing you to make drastic code changes.

1. Use a {proxy} or {proxy+} greedy path, which sounds like it's essentially what you're doing anyway, to route all API Gateway routes to one function. So as far as API Gateway is concerned, this is one integration and one endpoint and one Lambda Permission.
2. Create multiple Lambda functions all pointing to the same Node handler module, one for each endpoint, or group them in some way, e.g. UserFunction handles /user/.... paths

[mmcglone]

I was expecting that the answer would likely be "No", as you suggest. I'm thinking option 1 is probably the way to go. Thanks very much.

[awood45]

There is a way to add a DependsOn, see my comment below. I'm just not sure it's the issue in this case.

[awood45]

I'd have to know more about your template to say how you'd structure the "DependsOn" statements but the names of our generated Lambda Permission objects are predictable. For API Gateway to Lambda, the naming structure is "${ServerlessFunctionLogicalId}${ServerlessFunctionApiEventLogicalId}Permission${ApiGatewayStageName}"

So you might have "HelloWorldFunctionHelloWorldEventPermissionProd" or similar. You can do a DependsOn around this generated name (and you can check against current deployed templates, which are stable).

I'd be curious where these errors are coming from though. Are you seeing this with a brand new template? New deployments of an existing template?

[awood45]

If you can, could you provide a reproducing example template? I don't need your exact template if there's anything sensitive, but I want to eliminate the possibility that something is misconfigured.

[mmcglone]

Thanks. It seemed to me that (ideally) I shouldn't have to change anything either.

This is the error message from CFN "An error occurred and the request cannot be processed. (Service: Lambda, Status Code: 500, Request ID: xxxx)" (RequestToken: xxxx, HandlerErrorCode: GeneralServiceException)"

Let me look into making a sanitized template available to you. Also, I have a case number for the support ticket if that's helpful.

[awood45]

That would be, feel free to shoot it to alexwood@amazon.com

[theburningmonk]

@awood45 If you have 20 endpoints all pointed to the same function, would SAM generate just one Lambda permission resource, or would it generate 20?

[mmcglone]

@theburningmonk I can confirm that it generates 20