Unsupported intrinsic functions

[hoffa]

The problem

Some properties have limited support for intrinsic functions (e.g. Ref, Fn::GetAtt, Fn::If, etc.).

Using unsupported intrinsic functions in such properties can cause issues with deployment.

Why it happens

AWS SAM is a AWS CloudFormation macro; it receives a SAM template as input (as-is, along with the intrinsic functions), and returns a CloudFormation template which is then deployed by CloudFormation.

This means that SAM is unable to resolve some intrinsic functions. For example, an !If condition with a Ref to a stack parameter is resolvable (SAM has access to the stack parameters), but a Ref to a stack resource is not (as the transform happens before deployment). Furthermore, SAM supports limited intrinsic function resolution for only some properties.

It's not an issue for properties that are passed as-is to properties of the underlying CloudFormation resources, but it becomes an issue when SAM must know the value for its transform logic.

Workarounds

Add the AWS::LanguageExtensions transform

The AWS::LanguageExtensions transform will resolve intrinsic functions if the value is known when Transforms are run.

Replace:

Transform: AWS::Serverless-2016-10-31

With:

Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31

The AWS SAM CLI currently doesn't process AWS::LanguageExtensions locally, so it won't work where local transforms are needed.

Note

There is a known issue where AWS::Serverless-2016-10-31 and AWS::LanguageExtensions can conflict; see aws-cloudformation/cfn-language-discussion#109.

Use a pass-through property, if available

Pass-through properties are passed directly to the underlying CloudFormation resources, hence intrinsic functions work. Check the "AWS CloudFormation compatibility" notice under properties to see whether the property is passed as-is to an underlying CloudFormation resource.

For example for the Schedule event type, use the State property instead of Enabled.

Use raw CloudFormation

If nothing else works, you can always switch to using the underlying CloudFormation resources directly. Since they are not processed by SAM, CloudFormation will be able to resolve the intrinsic functions.

You can get the transformed CloudFormation template of a stack <my-stack> using:

aws cloudformation get-template --query TemplateBody --change-set-name "$(aws cloudformation describe-stacks --query 'Stacks[0].ChangeSetId' --output text --stack-name <my-stack>)"

Which you can then use to replace the affected resources.

See also #3007 (comment) for other ways of transforming a SAM template into a CloudFormation template.

[hoffa]

@tariromukute The issue seems to be that the intrinsic function must be resolved before deployment (local paths are unsupported by CloudFormation; under the hood SAM CLI uploads the assets and creates an intermediary template with the final S3 URIs), and (1) the SAM transform does not attempt to resolve the intrinsic function locally, and (2) the SAM CLI does not support running AWS::LanguageExtensions locally. The limitation is briefly mentioned in #2533 (comment).

[tariromukute]

Thank you @hoffa for the explanation. I went through the comment, however I wasn't able to understand what the workaround for this issue is. How can I use aws cloudformation directly to deploy a project that I initialised with sam init?

[hoffa]

@tariromukute You have to use SAM CLI to deploy any templates with local paths (such as most sam init templates), as SAM CLI needs to upload the assets first.

As for intrinsic functions that should resolve to local paths, unless the property implements some ad-hoc intrinsic function resolution, there's no great workaround apart from hardcoding. This is something we could explore in SAM CLI so that e.g. it also resolves intrinsic functions before passing to the SAM transform.

[tariromukute]

Hi @hoffa, thank you for clarifying the above. Is there an issue or work being done on this (we could explore in SAM CLI so that e.g. it also resolves intrinsic functions before passing to the SAM transform) or there is need for this to be created. This will make development and deployment easier. We tend to have javascript developers that do not have a lot of cloud experience and knowledge. Allowing SAM applications to be set up end-to-end locally using docker-compose (configure SAM app and local stack etc) will enable the developers to work on the lambda code without needing to know or deal with the cloud and SAM configurations.

[hoffa]

@tariromukute If I'm understanding correctly the problem you're encountering is !Ref to local paths in AWS::Serverless::LayerVersion not working. I was able to reproduce it, and confirmed it works with CodeUri in AWS::Lambda::Function. I've created aws/aws-sam-cli#4767 for the issue. There's not much else that can be done on the SAM transform side as the intrinsic resolution to the local path must happen before the SAM transform receives it. While not ideal, as workaround in the meantime you could use sed or similar to replace the text.

[tariromukute]

@tariromukute If I'm understanding correctly the problem you're encountering is !Ref to local paths in AWS::Serverless::LayerVersion not working. I was able to reproduce it, and confirmed it works with CodeUri in AWS::Lambda::Function. I've created aws/aws-sam-cli#4767 for the issue. There's not much else that can be done on the SAM transform side as the intrinsic resolution to the local path must happen before the SAM transform receives it. While not ideal, as workaround in the meantime you could use sed or similar to replace the text.

Awesome. Thanks @hoffa for reproducing the issue and for creating an issue for it. I will try the sed route on the CI pipeline.

[garretwilson]

This breaks CodeUri when deploying AWS lambda from a prebuilt JAR file using !Sub (see #271). Let's say I'm deploying some lambda function I built with Maven, and I don't want to hard-code the JAR version in my SAM template.

Resources:
  FooFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: !Sub "target/foo-bar-functions-${Ver}.jar"
      Handler: com.example.FooBar::foo

SAM gives me an error:

'CodeUri' requires Bucket and Key properties to be specified.

If I try the workaround noted here of adding AWS::LanguageExtensions, that SAM will at least upload my JAR to the S3 bucket. But then when it fails with:

'CodeUri' is not a valid S3 Uri of the form 's3://bucket/key' with optional versionId query parameter.

The problem is that SAM is not updating the CodeUri value before it uploads the template to S3, as it would if the value were a literal string. Shouldn't SAM know the dereferenced value at the time of uploading the template? Hasn't the template already been processed? There must be a bug where SAM forgets it needs to update the CodeUri value before uploading the template if !Ref was used. I filed bug aws/aws-sam-cli#5249.

[smaly-amazon]

If the AWS::LanguageExtensions transform is the solution, is that transform available in all regions where the serverless transform is available?

[davidchatz]

I've been using the language extension to workaround this issue with using !If in AutoPublishAlias, but this does not work with the new sam list command, which outputs: Error: [InvalidResourceException('SnsTopicConsumer', "'AutoPublishAlias' must be a string or a Ref to a template parameter")] ('SnsTopicConsumer', "'AutoPublishAlias' must be a string or a Ref to a template parameter")

[GavinZZ]

Hi @davidchatz, feel free to create an issue in aws-sam-cli repository on this.

[ozcsdog]

I am coming from #1616.
Is there any way to set string to whole Step Functions TimeoutSeconds? Just like TimeoutSecondsPath for Task.
Now the whole Step Functions TimeoutSeconds is only for int, and can't get string from path.