SAM deploy doesn't modify uploaded CodeUri if !Sub is used even with AWS::LanguageExtensions

[garretwilson]

I'm using SAM 1.84.0 on Windows 10.

Let's say I'm deploying some lambda function I built with Maven:

Resources:
  FooFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: target/foo-bar-functions-1.2.3.jar
      Handler: com.example.FooBar::foo

SAM recognizes that CodeUri points to a local file, so it uploads JAR file to S3 and also uploads the template itself to S3. But first it modifies the template to reference the new JAR file location. The template uploaded winds up looking like this when it gets to S3:

Resources:
  FooFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: s3://aws-sam-cli-managed-default-samclisourcebucket-…/…
      Handler: com.example.FooBar::foo

Obviously I don't want to hard-code the version 1.2.3 in my SAM template. In pure CloudFormation it is no problem for me to simply pass in a Ver parameter from the command line, indicating the version to use (e.g. the latest Maven version, or some version in the past, or whatever):

Resources:
  FooFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: !Sub "target/foo-bar-functions-${Ver}.jar"
      Handler: com.example.FooBar::foo

But because of this aws/serverless-application-model#2533, SAM gives me an error:

'CodeUri' requires Bucket and Key properties to be specified.

This general problem has been discussed in aws/serverless-application-model#2533. It seems that SAM gets all confused if there are functions used in the values that SAM needs right away. So the workaround noted in aws/serverless-application-model#2533 was to add AWS::LanguageExtensions.

Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31

That gets me a little further: SAM will at least upload my JAR to the S3 bucket. But then when it fails with:

'CodeUri' is not a valid S3 Uri of the form 's3://bucket/key' with optional versionId query parameter.

The template uploaded has:

      CodeUri:
        Fn::Sub: target/foo-bar-functions-${Ver}.jar

I believe this is not simply the same as aws/serverless-application-model#2533 ; rather, it would seem to be a SAM bug. If SAM with AWS::LanguageExtensions can figure out the JAR file that !Sub "target/foo-bar-functions-${Ver}.jar" references, it should be able to realize that the value for CodeUri should be updated, just as if it were a literal string.

In other words, this is not just a problem of not having values available at the time of deployment. As you can see, with AWS::LanguageExtensions SAM already knows what the correct value is for CodeUri not an S3 bucket but a local file, because it stopped giving the error 'CodeUri' requires Bucket and Key properties to be specified. The problem is that when SAM then uploads the template to S3, it doesn't realize that the CodeUri still needs updating. Something about using !Sub with CodeUri is preventing SAM from replacing the local reference as it would if the value were a literal string.

(I would guess that somewhere in the code there is an erroneous "sanity-check" line that say something like "if (CodeUri is a literal string and CodeUri points to a local value) {update CodeUri value to S3 form}". This bug fix may be a simple as removing the erroneous "CodeUri is a literal string" check. This is all just a guess based upon behavior; I haven't looked at the code.)

This is severe; it's forcing me to hard-code versions into my template, and I haven't yet found a workaround.

[hoffa]

Thanks for reporting this! Moving over to SAM CLI repository since this is related to local intrinsic function resolution; similar to #4767.

Meanwhile, while not pretty you could use sed or similar to replace the version before deployment.

[hoffa]

Thanks for the thorough summary, indeed it looks like when !Sub is used, SAM CLI passes it as-is to CloudFormation, which in turn resolves it to the local path that it doesn't understand, hence the error about S3 URI format.

So ideally SAM CLI could resolve the local path before deployment, and then package the corresponding files as usual.

[jfuss]

@garretwilson I understand what you are trying to do but I don't understand the flow you have.

Are you using sam build? Or are you manually putting the CodeUri value into the template and wanting sam package to resolve and upload?

Can't you generate a jar without the version and then not need this at all?

Generally the issue here: SAM CLI has had to implement intrinsic resolution on the client side and we do this in some area but sam package is not one. Currently we resolve intrinsic in memory for some commands and do not have the ability to resolve them per Property.

sam build takes away all of this thought. So even if you produce a versioned artifact, the template will be updated correctly and sam package will just work how you want without any need to pass versions in.

If you don't want sam build, the way forward here is to generate a non versioned artifact but I would recommend just using sam build.

[garretwilson]

Are you using sam build?

No. I'm only using sam deploy after I build my project with Maven. I don't want to use sam build. My multi-module Maven build is working just great.

Please read Building serverless Java applications with the AWS SAM CLI, in particular the section, "Deploying the application without building with AWS SAM".

It's a JAR file. sam deploy knows how to upload JAR files. It shouldn't break if I pass a parameter to CloudFormation to indicate the JAR filename. Sincerely, this is not a complicated scenario.

SAM CLI has had to implement intrinsic resolution on the client side and we do this in some area but sam package is not one.

I'm not using sam package that I know of. I'm using sam deploy. Sorry if I wasn't clear on that. (I also apologize in advance if sam deploy calls sam package without my knowing it. I'm new to SAM, and wasn't expecting to have to dig under the covers so soon.)

Can't you generate a jar without the version and then not need this at all?

But why? Why? Why? Are we merging Merkle trees with Bloom filters in LSM trees? No, we're … doing simple string interpolation. Why do I have to add workarounds for something so simple and straightforward?

SAM, CloudFormation, and CI/CD are built on the foundation of immutable, versioned artifacts. Why should SAM force me to start un-versioning my artifacts? This project will likely be deploying the actual JAR files to a registry such a Maven Central. Another project will pull them down using the normal Maven dependency resolution. All these JAR files will be versioned. Why should I have to build a workaround layer just to un-version things? Why should SAM force me to hack and undo industry best practices?

In CloudFormation when I define my ECS Fargate task container definition, I can easily use parameter interpolation for the name of the container image. Why should I expect the same thing to break just because I "stepped up" to use SAM?

Perhaps I'm not explaining the scenario well, because in my mind this could hardly be simpler, and is probably just a straightforward bug for a situation the developer didn't consider. Let me ask a few questions:

• When I use sam deploy and I put Transform: AWS::LanguageExtensions in my template, before the SAM CLI uploads the JAR and template to S3, does it have access to the interpolated value of CodeUri: !Sub "target/foo-bar-functions-${Ver}.jar"?
• If sam deploy has access to the interpolated value of CodeUri: !Sub "target/foo-bar-functions-${Ver}.jar", then why is this scenario any different than a hard-coded string CodeUri: target/foo-bar-functions-1.2.3.jar?

Maybe in all of this you respond, "but interpolation of parameters is not supposed to take place until the template is read from S3 and used to create a stack". But step back a second: sam deploy … is already doing a sort of interpolation before it even uploads the template to S3 when it manually converts CodeUri: target/foo-bar-functions-1.2.3.jar to CodeUri: s3://aws-sam-cli-managed-default-samclisourcebucket-…/…. So simply checking to see if the string should be interpolated before modifying the CodeUri value before upload is really no different. The logic is no different. It would look like the following. The logic in bold in step 3 is the only different thing.

1. Look at the CodeUri value. (already implemented)
2. If the CodeUri value is an s3://… URI, do nothing; skip to step 5. (already implemented)
3. Intepolate the CodeUri as needed.
4. Convert the CodeUri to a s3://… URI based upon the location at which you'll store the JAR file on S3. (already implemented)
5. Upload the possibly tweaked template. (already implemented)
6. Upload the JAR file. (already implemented)
7. …

Where in the source code does the SAM CLI convert CodeUri: target/foo-bar-functions-1.2.3.jar to CodeUri: s3://aws-sam-cli-managed-default-samclisourcebucket-…/… before it uploads the template? I'll take a look at the source code and see if I can give you a few more pointers on how to fix this.

[garretwilson]

An even better design would avoid the need for interpolation altogether in the modified template: simply keep the original name of the artifact foo-bar-functions-1.2.3.jar when you upload the file to the SAM S3 bucket, instead of renaming it to some hash. Thus if CodeUri specified !Sub "target/foo-bar-functions-${Ver}.jar", then you would dereference that to find target/foo-bar-functions-1.2.3.jar, and upload that without renaming it. Then you would convert !Sub "target/foo-bar-functions-${Ver}.jar" to !Sub "s3://aws-sam-cli-managed-default-samclisourcebucket-…/foo-bar-functions-${Ver}.jar", similar to what you do already, but leaving the parameter in the string. Thus when CloudFormation pulls in the modified template, it would interpolate s3://aws-sam-cli-managed-default-samclisourcebucket-…/foo-bar-functions-${Ver}.jar to s3://aws-sam-cli-managed-default-samclisourcebucket-…/foo-bar-functions-1.2.3.jar and it would work just as intended.

Then someone might say, "But then what if we overwrite an existing artifact? How will we tell them apart without renaming them to hashes?" And that is the whole point of version numbers! That is the whole point of immutable, versioned builds. That is the whole point of Docker and Amazon Elastic Container Registry (ECR) and ECS Fargate using version numbers for artifacts. In this case SAM CLI is attempting to do a quick-and-dirty implementation of an artifact repository and rely on hashes to keep versions apart. But it makes it a pain for the industry best-practices approach of immutable builds based upon versioned artifacts.

If you add a flag "don't rename artifact when uploading", that would solve this problem more elegantly and be really easy to implement—although you would need to dereference the CodeUri value once just to find the JAR file to upload.

The other approach is to write a dereferenced value in CodeUri for the modified template if the JAR file was local, as I explained above.

I'm still looking for where this happens in the code. If you happen to know, pass it along and I'll try to give more code-oriented suggestions.

[garretwilson]

I just thought of a potential workaround based upon my last idea: if I specify an S3 URI for CodeUri, will SAM simply skip uploading the JAR file and upload the template keeping the CodeUri as it is? If so, I think I could simply put the final S3 bucket path in the value to begin with, but keeping the ${Ver} interpolation, like this:

CodeUri: !Sub "s3://aws-sam-cli-managed-default-samclisourcebucket-…/foo-bar-functions-${Ver}.jar"

Then I could do something like this in the script that invokes SAM (to which the version is initially passed), manually uploading the JAR file to S3 so that it won't be renamed by SAM:

aws s3 cp target/foo-bar-functions-$ver s3://aws-sam-cli-managed-default-samclisourcebucket-…/
sam deploy …

The idea here is to sidestep SAM CLI's implementation of the JAR upload because SAM gets confused with interpolation. Can the CLI figure out that it doesn't need to upload the JAR file if I specify an S3 URI, and just continue normally with deployment?

Once the template is uploaded and get processed, I'm assuming that CodeUri: !Sub "s3://aws-sam-cli-managed-default-samclisourcebucket-…/foo-bar-functions-${Ver}.jar" will get interpolated correctly to e.g. CodeUri: s3://aws-sam-cli-managed-default-samclisourcebucket-…/foo-bar-functions-1.2.3.jar and find the JAR the script uploaded manually. But I haven't tried it yet.

This workaround has some downsides. Now the JAR file is no longer managed by SAM, so when I use sam delete … it probably won't delete the JAR. This may not be a big problem (and is certainly a smaller problem than it not working at all). Still, ideally SAM would just get fixed so I don't have to write code to substitute for the parts of its deploy functionality that don't work, and I can go back to letting SAM manage the things it's supposed to be managing.

[jfuss]

@garretwilson There is a lot here. So this will likely be a longer post and will probably not respond to each thing above. I am going to focus on 1) getting mutual understanding of SAM CLI and what happens 2) your direct request 3) side affects like sam delete. Happy to go into more details in other areas but don't want to this diverge completely.

Mutual understanding of SAM CLI and what happens

I am going to ignore some of the commands (like sam build) and focus on sam deploy. There are two parts to sam deploy 1) uploading local artifacts (aka sam package) 2) calling CloudFormation CreateChangeSet and ExecuteChangeSet.

Uploading Artifacts (aka sam package)

The first step of sam deploy is to read the template and see if there are any local artifacts. This reads the template directly and does not do intrinsic resolution for a couple reasons 1) this process is run locally 2) we do not have access to CloudFormation's logic for resolution 3) we view the template as a versioned artifact meaning doing intrinsic resolution and acting on it could cause wrong results. sam package's purpose it to upload artifact and zip them first if needed. So while Fn::Sub: target/foo-bar-functions-${Ver}.jar is valid CloudFormation and CloudFormation would resolve it, that happens to late in the process (part of deploy on the service side) and the package step needs the specific artifact defined in the template (aka no intrinsic logic run).

The last step of sam package is to update the template with the location of the artifact we uploaded. This could be ECR or S3.

After the package step is run and artifacts are updated in the template, we deploy the updated templates through CreateChangeSet and ExecuteChangeSet APIs. This is where we mostly pass through things to CloudFormation (like parameter-overrides) and CloudFormation does it's thing to get the stack deployed. This include calling Service APIs and doing their resolution logic.

Your direct request

Technically, this is something we could introduce however is poses risk which I don't think is worth it and there are other ways to achieve this. We generally stay away from resolving intrinsics locally. It leads to higher cost of maintenance and no guarantee what we do would match CloudFormation's logic. On top of this, as CloudFormation expands (AWS::LanguageExtensions is a good example of that), we have to keep up. This is main reason we try to stay away from resolution locally, which is what you would need/are asking for.

We cannot just support Sub either. We would need to support the full intrinsic suite. Otherwise, customers to other variations are left out (Joins, SSM parameters, etc). Sub can also be embedded with other Intrinsics. We could support a subset but leaves us with requests to support more and more over time.

We could provide some other way to allow you to achieve sometime similar more directly but we already have ways to support this which I cover below.

Side affects like sam delete

sam delete will just work. You only provide the stack-name so it doesn't matter if you deployed that stack with sam deploy or not. The command only cares about a stack. This goes for other commands too, like sam list, sam logs, sam traces, etc and is generally true through the cli.

So there should be no concerns here.

So where does this leave the request

If you do not want to change artifact names, that is fine and reasonable. Your other alternatives are to

1. generate the artifacts to different folders (assuming you have multiple artifacts under target/) and add those folders as the CodeUri. sam package should detect the folder and zip that contents. (please verify here, as I am going off memory on what the final artifact is here).
2. You can do what you suggesting and upload the artifacts yourself. Letting you control everything you want. You can still use sam deploy it just wouldn't upload the artifacts. For this to work, you would need to update the template somehow or do some subs. You risk eventual consistency in this process though and could be brittle: What happens if you upload artifact with version X but your deploy command uses version Y.

The experience is going to be better if you edit your flow to match the tool and what is supported. If you do not wish to, you will need to do one of the alternatives.

I will leave this open and happy to have a deeper conversation but at this point in time, I am not inclined to add support for this within SAM CLI.

[garretwilson]

@jfuss thank you for taking the time to provide this well-organized explanation. I have some strong feelings about both the design decisions that were made as well as the current decision not to address the shortcomings in the tool itself, but I'll just leave it at that for now.

I have great news: the workaround idea I had yesterday is working marvelously—in fact better than I had even envisioned.

First let's take a step back and think about why any of this even matters. Let's say I'm deploying container images in AWS ECS Fargate. One day I upload and deploy my-container-image-2.3.0 using AWS ECR. The next day I upload my-container-image-2.4.0 and tell my ECS service to update. The next day I do the same with my-container-image-2.5.0. Oops! That breaks all over the place. Even worse, it turns out that because of an interaction with another service, the entire 2.x line has been erroneously charging our best customer (she was on vacation in Paris and didn't notice it), so we have to roll back to v1.8.8! It turns out that my-container-image-1.8.8 is still in ECR, so we simply tell the service to redeploy using v1.8.8. Then we fix the other service, but the fix only works with v2.4.0 of our service—no problem, my-container-image-2.4.0 is still in ECR, so we simply deploy the service specifying v2.4.0 as the CloudFormation Ver parameter.

Now let's start using SAM. What if we have this in our SAM template (which is basically what the SAM problem in this ticket forces us to do) and we find out we have to roll back to last week's version?

Resources:
  FooFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: target/foo-bar-functions.jar
      Handler: com.example.FooBar::foo

The hashes sam deploy uses on S3 are opaque—we have no idea what the versions are. Plus even if we did we would need to do some manual modification of SAM templates to directly reference the hash blob on S3. Or we can find a tag in Git for the previous version, check out that tag, rebuild everything, and do a redeploy.

That would be a mess. That would not be DevOps. That would not be appropriate for 2023.

So now for the workaround. The SAM template still requires AWS::LanguageExtensions:

Transform:
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31

The template has a Ver parameter for the version:

Parameters:
  Ver:
    Description: The version of the functions, such as "1.2.3".
    Type: String
    AllowedPattern: '[\w.+-]+'
    ConstraintDescription: The service version must use only word characters, dots, plus signs, and dashes.

The template simply uses the JAR file from S3 as-is, but a substitution for the version parameter is passed, like this:

Resources:
  TestIngestFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: !Sub "s3://my-sam-bucket/foo-bar-functions-${Ver}.jar"
      Handler: com.example.FooBar::foo

That works like a charm. Now I can use sam deploy to simply pass any version number I want in the Ver parameter. Since sam deploy sees that this is an S3 URI, it doesn't touch the CodeUri, and the parameter is resolved after the change set is created (i.e. at "runtime"). I can roll back. I can roll forward. And the artifacts are no longer opaque on S3—they have real version numbers! And since sam deploy didn't upload the JAR, it leaves it alone. I can have years worth of versions on S3 and simply redeploy with a single command that looks something like this:

sam deploy \
    --profile $awsProfile \
    --stack-name my-stack \
    --s3-bucket my-sam-bucket \
    --capabilities CAPABILITY_NAMED_IAM \
    --parameter-overrides Ver=$ver

Your next question may be, "But how do you get the artifacts to S3 if you don't rely on sam deploy to upload them?" I'm so glad you asked. The script has a line like this:

aws s3 cp $artifactPath s3://${s3BucketName}/${s3ObjectName} --profile $awsProfile

"So you re-upload the file each time? Isn't that a lot of needless bandwidth? With the hash filename method, it wouldn't upload the file again if it hadn't changed." Right. We can do the same thing here, but it's even better because of version numbers. The most standards-based best-practices modern approach is to use versioned, immutable artifacts. Thus if an artifact is uploaded already with the same version number, you simply don't upload it again—you don't need to, because you can be confident that it hasn't changed! It looks like this:

s3BucketName=my-sam-bucket
sourceDirectory=target
artifactBaseName=foo-bar-functions
artifactName=$artifactBaseName-${ver}.jar
artifactPath=$sourceDirectory/${artifactName}

# SNAPSHOT processing goes here; see below

s3ObjectName=$artifactBaseName-${ver}.jar
if (( $(aws s3api head-object --bucket $s3BucketName --key $s3ObjectName --profile $awsProfile \
    >/dev/null 2>/dev/null; echo $?) == 254 )); then
  echo "Uploading $artifactName ..."
  aws s3 cp $artifactPath s3://${s3BucketName}/${s3ObjectName} --profile $awsProfile
else
  echo "$s3ObjectName already exists in S3 bucket; skipping upload."
fi

Oh, but it gets better. What if you're working with a -SNAPSHOT version (e.g. foo-bar-functions-1.2.3-SNAPSHOT.jar)? These type of versioned artifacts are meant to change. Apache Maven pioneered the idea that a version ending in -SNAPSHOT is considered to be in private development, and thus should not be considered final and immutable—one foo-bar-functions-1.2.3-SNAPSHOT.jar and another foo-bar-functions-1.2.3-SNAPSHOT.jar may be different builds from non-identical code, with different behavior. We don't want to rely on one foo-bar-functions-1.2.3-SNAPSHOT.jar on S3 being the same as was just built.

So what do we do? Do we simply overwrite -SNAPSHOT versions on S3? But then there's a problem: SAM+CloudFormation, if we pass the same foo-bar-functions-1.2.3-SNAPSHOT.jar to the Ver parameter, won't see that anything has changed and won't redeploy the stack. And even though we've replaced the file in S3, the deployed lambda doesn't know that it should "refresh" from the new code! What do we do?

"Ah ha! That's why we used hashes for the filenames in sam deploy, so we wouldn't have this problem", someone might gloat. Right, but this is a corner case we're addressing, and using hashes across the board breaks the happy path of immutable, versioned artifacts, as I explained above. Yes, hashes allow us to address this corner case, but we can do so in a very surgical manner that actually enhances the functionality rather than break other functionality by forcing the technique across all filenames.

In other words, we use hashes only with -SNAPSHOT versions, and we append the hash—we don't replace the entire filename. (After all, foo-bar-functions-1.0.0-SNAPSHOT.jar and foo-bar-functions-3.0.0-SNAPSHOT.jar are two very different things semantically, and we still want to keep them separate.) The code is simple. Simply insert the following into the script above where it says "SNAPSHOT processing goes here":

if [[ $ver == *-SNAPSHOT ]]; then
  hash=($(shasum -a 256 $artifactPath))
  ver="${ver}+${hash:0:16}"
  echo "Using modified snapshot version: ${ver}"
fi

Things to note:

• I'm using SHA-256 for robustness, but truncating the hash string to just the first 16 characters, which should be sufficient to avoid collisions while being easier to work with. Whether someone uses ${hash:0:8}, ${hash:0:10}, ${hash:0:32}, or the entire ${hash} is based upon personal preference—all such variations would probably be fine.
• I'm using a plus sign + for appending the hash, following the Semantic Versioning rules for a "build metadata" component. In particular build metadata (as opposed to a pre-release version appended with a dash -) does not imply any order and has no prohibition against starting with zero. And of course "build metadata" is exactly what this is, so that's the most appropriate thing to use.

And there you have it. Versions such as foo-bar-functions-1.2.3-SNAPSHOT.jar will be treated as immutable, versioned artifacts. SNAPSHOT versions will have a hash appended to the version, e.g. foo-bar-functions-1.2.3-SNAPSHOT+xxxxxxxx….jar, and the Ver parameter passed to the SAM/CloudFormation template will also be modified to 1.2.3-SNAPSHOT+xxxxxxxx…. Furthermore files will only be uploaded when needed. Here's what the artifacts will look like on S3:

• foo-bar-functions-1.2.1.jar
• foo-bar-functions-1.2.2.jar
• foo-bar-functions-1.2.2-SNAPSHOT+xxxxxxxx….jar
• foo-bar-functions-1.2.2-SNAPSHOT+xxxxxxxx….jar
• foo-bar-functions-1.2.3.jar
• foo-bar-functions-1.2.4.jar

It's elegant. It works like a modern CD/CI tool should. It works like developers expect who are used to working with Maven, Docker, Node.js, AWS ECS+ECR, etc.

The only downside I can think of is that I had to spend my morning writing it.

Otherwise I have no complaints, and it makes no difference to me now whether this ticket is addressed. Again thanks for reading the ticket, and thanks for taking the time to respond. Just please keep sam deploy from breaking this workaround in the future (e.g. keep supporting AWS::LanguageExtensions at least to the level you do now). Cheers.

[jfuss]

@garretwilson Going to close this as the solution is the best way forward if you want control over the template at deploy time.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.