sam deploy should accept parameter file just like aws cloudformation *-stack

[rudpot]

Description:

I would like to store configuration is version control and make deployment steps immutable. Instead of changing parameter overrides on the command line I would like to provide template parameters from a file.

There is an idiom for this in aws cloudformation create-stack --parameters file://.

Steps to reproduce the issue:

1. sam deploy --parameter-overrides file://params.json

Observed result:

Error: Invalid value for '--parameter-overrides':  file://params.json is not in valid format. It must look something like 'ParameterKey=KeyPairName,ParameterValue=MyKey ParameterKey=InstanceType,ParameterValue=t1.micro' or 'KeyPairName=MyKey InstanceType=t1.micro'

Expected result:

	Deploying with following values
	===============================
	Stack name                 : ...
	Region                     : ...
	Confirm changeset          : ...
	Deployment s3 bucket       : ...
	Capabilities               : ...
	Parameter overrides        : {'MyParamOverride': 'MyParamValue'}

[jfuss]

Why not use the samconfig.toml file? https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html

[rudpot]

Why not use the samconfig.toml file? https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html

I am trying to set a parameter that is specified in the template. There is no hint in the documentation that this should be possible but following your suggestion it tried the following:

Parameter defined in the template:

Parameters:
  AllowedSrcIp:
    Type: "String"
    Description: "IP address of the client used for testing. If in doubt, deploy the code and the output will guide you"
    Default: "192.168.0.1"

Updated samconfig.toml with the parameters directly:

version = 0.1
[default]
[default.deploy]
[default.deploy.parameters]
stack_name = "api-unsecured-sam"
s3_bucket = "aws-sam-cli-managed-default-samclisourcebucket-10ziytzq60hp8"
s3_prefix = "api-unsecured-sam"
region = "us-west-2"
confirm_changeset = false
capabilities = "CAPABILITY_IAM"
AllowedSrcIp = "205.251.233.178"

The deploy works but doesn't honor the new parameter. Looking at the structure of the config I also tried this line:

parameter_overrides = ParameterKey=AllowedSrcIp,ParameterValue=205.251.233.178

Which breaks the deployment entirely:

sam deploy
Usage: sam deploy [OPTIONS]
Try 'sam deploy --help' for help.

Error: Missing option '--stack-name', 'sam deploy --guided' can be used to provide and save needed parameters for future deploys.

And this:

parameter_overrides = "ParameterKey=AllowedSrcIp,ParameterValue=205.251.233.178"

And this:

parameter_overrides = "AllowedSrcIp=205.251.233.178"

both of which do work but are really hard to maintain if you have more than a couple parameters or if your parameters need complex quoting. In fact it turns out that if you were to enter the data interactively as a result of using sam deploy --guided it will create something like this which highlights the issue:

parameter_overrides = "VpcIdParameter=\"vpc-0000111122223333\" VpcAllowedSecurityGroupIdParameter=\"sg-0000111122223333\" VpcEndpointSubnetIdsParameter=\"subnet-0000111122223333 \""

[what-name]

As @rudpot mentioned, the following parameter_overrides value is rather tedious to maintain with more than a couple entries:

parameter_overrides = "VpcIdParameter=\"vpc-0000111122223333\" VpcAllowedSecurityGroupIdParameter=\"sg-0000111122223333\" VpcEndpointSubnetIdsParameter=\"subnet-0000111122223333 \""

+1 for a simpler approach.

[rojomisin]

work around until a file can be used is to wrap w/ shell or make

...
		--parameter-overrides \
			DBUsername=$(DBUSER) \
			DBPassword=\"$(shell get-db-pass)\" \
			Subnets=$(SUBNETS) \
			VpcId=$(VPC_ID)

load up a config file with k/v pairs and use make's include

ifndef CONFIG
include config/$(CONFIG)
endif

make sam-deploy CONFIG=dev

[jveldboom]

We use a similar pattern to @rojomisin but use a standard environment variable file format.

.sam-params

DBUsername=admin
DBPassword=password123
Subnets=subnet-12345
VpcId=vpc-1234

Then in our Makefile, we cat that file and pass them in as --parameter-overrides

sam deploy --parameter-overrides $(shell cat .sam-params)

[jaridspokes]

I've used the above workaround for now, but it would be great to get some consistency with cloudformation in this regard.

[benkehoe]

#2253 proposes parameter overrides should get their own section in samconfig.toml (and should be individually overridden by --parameter-overrides)

[ghost]

The trick by @jveldboom works just as well in a shell-script, except with --parameter-overrides $(cat path-to-file).

[HFR1994]

I'm using jq as an alternative.... Having the following structure

// parameters.json

[
  {
    "ParameterKey": "ApplicationName",
    "ParameterValue": "Foo"
  },
  {
    "ParameterKey": "ApplicationKey",
    "ParameterValue": "Bar"
  }
]

I can do

SAM_PARAMETERS=$( cat ${DIR}/parameters.json | jq -r '.[] | "\(.ParameterKey)=\(.ParameterValue)"' )
sam deploy ... --parameter-overrides $SAM_PARAMETERS

The good thing is that file can be shared with Cloudformation templates with:

aws cloudformation deploy ... --parameter-overrides "file://${DIR}/parameters.json"

[whereisaaron]

samconfig.toml is yet another different and, and as others pointed out, worse way to maintain AWS parameters, using yet another file format on top of YAML and JSON! Why are we trying so hard to make life more complicated 😄

cloudformation deploy has added support for --parameter-overrides file://my-params.json in aws/aws-cli#5443, including the also different and incompatible CodePipeline parameters format.

if sam is a wrapper for cloudformation deploy which now, or soon, supports --parameter-overrides files://my-params.json, can sam pick up on that and resolve this issue?

[mountHouli]

I need to use a SAM parameter to upload a private RSA key in PEM format to AWS SecretsManager.

I've been trying all kinds of things on the CLI (including everything above) to get this to work. I ended up giving up and writing this workaround in my python code:

import base64
import boto3

# AWS CLI can't (easily) take params with newlines or spaces. Thus, we base64 encode our RSA PEM
# to make it single line before we upload it to Secrets Manager. See docs/design-notes.md
def get_lighthouse_rsa_key(awsSecretArn):  # pragma: no cover
    smClient = boto3.client("secretsmanager")
    base64EncodedPem = smClient.get_secret_value(SecretId=awsSecretArn)["SecretString"]

    resultBytes = base64.b64decode(base64EncodedPem)

    return resultBytes.decode()

I would be thankful if the SAM CLI maintainers incorporate @whereisaaron 's suggestion above on April 5th, 2021.

Note: Another part of the problem plaguing with (most) PEM files is the fact that CloudFormation parameters can be 4096 bytes, max. To support PEM files (at least those that are more than 4096 bytes) CloudFormation would need to provide an attribute to each parameter to override the 4096 byte max. Perhaps an attribute like MaxBytes: AN_INTEGER. But that's a different topic than this thread.

[PalituxD]

I resolved this scenario using options below:

"scripts": { "invoke": "sam ... --parameter-overrides \"$(jq -j 'to_entries[] | \"\\(.key)='\\\\\\\"'\\(.value)'\\\\\\\"''\\ '\"' params.json)\"" }

Or

sam ... --parameter-overrides "$(jq -j 'to_entries[] | "\(.key)='\\\"'\(.value)'\\\"''\ '"' params.json)"

[manraog]

I'm using jq as an alternative.... Having the following structure

// parameters.json

[
  {
    "ParameterKey": "ApplicationName",
    "ParameterValue": "Foo"
  },
  {
    "ParameterKey": "ApplicationKey",
    "ParameterValue": "Bar"
  }
]

I can do

SAM_PARAMETERS=$( cat ${DIR}/parameters.json | jq -r '.[] | "\(.ParameterKey)=\(.ParameterValue)"' )
sam deploy ... --parameter-overrides $SAM_PARAMETERS

The good thing is that file can be shared with Cloudformation templates with:

aws cloudformation deploy ... --parameter-overrides "file://${DIR}/parameters.json"

Line breaks didn't let me use this option

Result:

ApplicationName=Foo
ApplicationKey=Bar

I had to change it to this

SAM_PARAMETERS=$( cat params.json | jq -r '[ .[] | "\(.ParameterKey)=\(.ParameterValue)" ] | join(" ")' )

Result:

ApplicationName=Foo ApplicationKey=Bar

[yigiterinc]

I would love to see SAM having a built-in easier way to do this. It becomes a maintenance hell unless the developers implement workarounds like @HFR1994 and @jveldboom suggested.

[JaimySmets]

Perhaps an easier workaround than custom wrapper scripts, is to use TOML's multiline support in the samconfig.toml file.

[default.deploy.parameters]
parameter_overrides = """
    ApplicationName='Foo'
    ApplicationKey='Bar'
    """

Which it seems to parse just fine during sam deploy:

Deploying with following values
===============================
...
Parameter overrides          : {"ApplicationName": "Foo", "ApplicationKey": "Bar"}

Although I agree putting it in a separate file would look cleaner, at least this way it remains the default sam deploy.

[benkehoe]

@JaimySmets as mentioned in #2253 (comment) it also works with a list

parameter_overrides=[
  "ApplicationName=Foo",
  "ApplicationKey=Bar"
]

either way it doesn't solve for the parameters needing to be managed separately from the samconfig.toml file

[jhonatanacelas]

How should configure it in a pipeline without load credentials to git if I need the value of the Parameters in build phase, because the lambda download an private pypi package which require authentication?

[liamfit]

I'm using jq as an alternative.... Having the following structure
// parameters.json

[
  {
    "ParameterKey": "ApplicationName",
    "ParameterValue": "Foo"
  },
  {
    "ParameterKey": "ApplicationKey",
    "ParameterValue": "Bar"
  }
]

I can do

SAM_PARAMETERS=$( cat ${DIR}/parameters.json | jq -r '.[] | "\(.ParameterKey)=\(.ParameterValue)"' )
sam deploy ... --parameter-overrides $SAM_PARAMETERS

The good thing is that file can be shared with Cloudformation templates with:

aws cloudformation deploy ... --parameter-overrides "file://${DIR}/parameters.json"

Line breaks didn't let me use this option

Result:

ApplicationName=Foo
ApplicationKey=Bar

I had to change it to this

SAM_PARAMETERS=$( cat params.json | jq -r '[ .[] | "\(.ParameterKey)=\(.ParameterValue)" ] | join(" ")' )

Result:

ApplicationName=Foo ApplicationKey=Bar

This worked great for me until I started trying to use parameters whose values were lists:

[
  {
    "ParameterKey": "VpcId",
    "ParameterValue": "vpc-xxx"
  },
  {
    "ParameterKey": "VpcPrivateSubnets",
    "ParameterValue": [
      "subnet-xxx",
      "subnet-xxx",
      "subnet-xxx"
    ]
  },
  {
    "ParameterKey": "LambdaSecurityGroup",
    "ParameterValue": "sg-xxx"
  }
]