Sceptre http handler causes extra characters in template

[zaro0508]

We get extra &#39 and &#34 characters in our template only when it's downloaded using the http sceptre template handler

environment

• sceptre ver 2.7.1
• python ver 3.7
• MAC/OS Monterey ver 12.0.1

Steps to reproduce:

Create the following sceptre template config.

template:
  type: "http"
  url: "https://raw.githubusercontent.com/Sage-Bionetworks/aws-infra/v0.3.7/templates/ssm-parameters.j2"
stack_name: test-stack
sceptre_user_data:
  Prefix: /nba/
  Parameters:
    - Name: TeamToRoleArnMap
      Value: >-
        '[
          {"division":"east","team":"blazers"},
          {"mascot":"beavers","home":"oregon"}
        ]'

generate cloudformation template..

➜  sceptre generate develop/test-stack.yaml
---

Description: "Setup parameters in the SSM parameter store"
AWSTemplateFormatVersion: "2010-09-09"
Resources:
  TeamToRoleArnMap:
      Type: AWS::SSM::Parameter
      Properties:
        Name: /nba/TeamToRoleArnMap
        Value: '[
  {"division":"east","team":"blazers"},
  {"mascot":"beavers","home":"oregon"}
]'
        Type: 'String'

Actual behaviour:
Notice the &#39 and &#34 characters in the Value property.

Expected behavior:
Now download the jinja template https://raw.githubusercontent.com/Sage-Bionetworks/aws-infra/v0.3.7/templates/ssm-parameters.j2 and save it to the local template folder. Generate the cloudformation using the local jinja file and it will generate without those extra characters

template:
  path: "ssm-parameters.j2"
stack_name: test-stack
sceptre_user_data:
  Prefix: /nba/
  Parameters:
    - Name: TeamToRoleArnMap
      Value: >-
        '[
          {"division":"east","team":"blazers"},
          {"mascot":"beavers","home":"oregon"}
        ]'

➜  sceptre generate develop/test-stack.yaml
---
Description: "Setup parameters in the SSM parameter store"
AWSTemplateFormatVersion: "2010-09-09"
Resources:
  TeamToRoleArnMap:
      Type: AWS::SSM::Parameter
      Properties:
        Name: /nba/TeamToRoleArnMap
        Value: '[
  {"division":"east","team":"blazers"},
  {"mascot":"beavers","home":"oregon"}
]'
        Type: 'String'

[jfalkenstein]

That's weird. Have you figured out what is causing this?

[zaro0508]

Nope, haven't had time. My current workaround is to download the file with the sceptre before_launch hook.

[jfalkenstein]

My first impulse would be to look at encoding issues here. Perhaps we're not properly decoding the response or it's not encoded in utf-8 but instead something different.

[zaro0508]

I have finally figured this out...

The Sceptre jinja environment's autoescape setting is enabled by default therefore jinja is auto escaping the characters when rendering the template.

There are two ways to get around this problem:

• Use the jinja safe filter in the jinja template to mark the variable as safe (i.e. {{ VAR|safe }})
• Disable the jinja environment's autoescape setting in the Sceptre config.yaml file:

j2_environment:
   autoescape: False

[iainelder]

Ran into this now. Thanks @zaro0508 for helping to understand the behavior and how to work around it.

I made a Sceptre project with different combinations of template type and autoescape setting. The GitHub workflow generates the YAML output for each one.

https://github.com/iainelder/sceptre-http-jinja2-bug/actions/runs/6593703005/job/17916559230

Case: file-autoescape-false/test.yaml
---
Unsafe: "Hello, world!"
Safe: "Hello, world!"

Case: file-autoescape-true/test.yaml
---
Unsafe: "Hello, world!"
Safe: "Hello, world!"

Case: file/test.yaml
---
Unsafe: "Hello, world!"
Safe: "Hello, world!"

Case: http-autoescape-false/test.yaml
---
Unsafe: "Hello, world!"
Safe: "Hello, world!"

Case: http-autoescape-true/test.yaml
---
Unsafe: "Hello, world!"
Safe: "Hello, world!"

Case: http-github/test.yaml
---
Unsafe: "Hello, world!"
Safe: "Hello, world!"

Case: http-local/test.yaml
---
Unsafe: "Hello, world!"
Safe: "Hello, world!"