Skip to content

Commit

Permalink
lang/funcs: Experimental "templatestring" function
Browse files Browse the repository at this point in the history 
This function complements the existing "templatefile" to deal with the
unusual situation of rendering a template that comes from somewhere
outside of the current module's source code, such as from a data resource
result.

We have some historical experience with the now-deprecated
hashicorp/template provider and its template_file data source, where we
found that new authors would find it via web search and assume it was
"the way" to render templates in Terraform, and then get frustrated
dealing with the confusing situation of writing a string template that
generates another string template for a second round of template rendering.

To try to support those who have this unusual need without creating another
attractive nuisance that would derail new authors, this function imposes
the artificial extra rule that its template argument may only be populated
using a single reference to a symbol defined elsewhere in the same module.
This is intended to entice folks trying to use this function for something
other than its intended purpose to refer to its documentation (once
written) and then hopefully learn what other Terraform language feature
they ought to have used instead.

The syntax restriction only goes one level deep, so particularly-determined
authors can still intentionally misuse this function by adding one level
of indirection, such as by building template source code in a local value
and then passing that local value as the template argument. The restriction
is in place only to reduce the chances of someone _misunderstanding_ the
purpose of this function; we don't intend to prevent someone from actively
deciding to misuse it, if they have a good reason to do so.

This new function inherits the same restriction as templatefile where it
does not allow recursively calling other template-rendering functions.
This is to dissuade from trying to use Terraform templates "at large",
since Terraform's template language is not designed for such uses. It would
be better to build a Terraform provider that wraps a more featureful
template system like Gonja if someone really does need advanced templating,
beyond Terraform's basic goals of being able to build small configuration
files, etc.

Because this function's intended purpose is rendering templates obtained
from elsewhere, this function also blocks calls to any of Terraform's
functions that would read from the filesystem of the computer where
Terraform is running. This is a small additional measure of isolation to
reduce the risk of an attacker somehow modifying a dynamically-fetched
template to inspire Terraform to write sensitive data from the host
computer into a location accessible to the same attacker, or similar.

This is currently only a language experiment and so will not yet be
available in stable releases of Terraform. Before stabilizing this and
committing to supporting it indefinitely we'll want to gather feedback on
whether this function actually meets the intended narrow set of use-cases
around dynamic template rendering.
  • Loading branch information
@apparentlymart
apparentlymart committed Apr 29, 2024
1 parent 2ac2f40 commit edf335e
Show file tree
Hide file tree
Showing 6 changed files with 633 additions and 92 deletions.
7 changes: 7 additions & 0 deletions internal/lang/funcs/descriptions.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -420,6 +420,13 @@ var DescriptionList = map[string]descriptionEntry{
Description: "`templatefile` reads the file at the given path and renders its content as a template using a supplied set of template variables.",
ParamDescription: []string{"", ""},
},
"templatestring": {
Description: "`templatestring` takes a string from elsewhere in the module and renders its content as a template using a supplied set of template variables.",
ParamDescription: []string{
"a simple reference to a string value containing the template source code",
"object of variables to expose in the template scope",
},
},
"textdecodebase64": {
Description: "`textdecodebase64` function decodes a string that was previously Base64-encoded, and then interprets the result as characters in a specified character encoding.",
ParamDescription: []string{"", ""},
Expand Down
90 changes: 17 additions & 73 deletions internal/lang/funcs/filesystem.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ package funcs
import (
"encoding/base64"
"fmt"
"io/ioutil"
"io"
"os"
"path/filepath"
"unicode/utf8"
Expand All @@ -17,6 +17,8 @@ import (
homedir "github.com/mitchellh/go-homedir"
"github.com/zclconf/go-cty/cty"
"github.com/zclconf/go-cty/cty/function"

"github.com/hashicorp/terraform/internal/collections"
)

// MakeFileFunc constructs a function that takes a file path and returns the
Expand Down Expand Up @@ -69,20 +71,7 @@ func MakeFileFunc(baseDir string, encBase64 bool) function.Function {
// As a special exception, a referenced template file may not recursively call
// the templatefile function, since that would risk the same file being
// included into itself indefinitely.
func MakeTemplateFileFunc(baseDir string, funcsCb func() map[string]function.Function) function.Function {

params := []function.Parameter{
{
Name: "path",
Type: cty.String,
AllowMarked: true,
},
{
Name: "vars",
Type: cty.DynamicPseudoType,
},
}

func MakeTemplateFileFunc(baseDir string, funcsCb func() (funcs map[string]function.Function, fsFuncs collections.Set[string], templateFuncs collections.Set[string])) function.Function {
loadTmpl := func(fn string, marks cty.ValueMarks) (hcl.Expression, error) {
// We re-use File here to ensure the same filename interpretation
// as it does, along with its other safety checks.
Expand All @@ -99,65 +88,20 @@ func MakeTemplateFileFunc(baseDir string, funcsCb func() map[string]function.Fun
return expr, nil
}

renderTmpl := func(expr hcl.Expression, varsVal cty.Value) (cty.Value, error) {
if varsTy := varsVal.Type(); !(varsTy.IsMapType() || varsTy.IsObjectType()) {
return cty.DynamicVal, function.NewArgErrorf(1, "invalid vars value: must be a map") // or an object, but we don't strongly distinguish these most of the time
}

ctx := &hcl.EvalContext{
Variables: varsVal.AsValueMap(),
}

// We require all of the variables to be valid HCL identifiers, because
// otherwise there would be no way to refer to them in the template
// anyway. Rejecting this here gives better feedback to the user
// than a syntax error somewhere in the template itself.
for n := range ctx.Variables {
if !hclsyntax.ValidIdentifier(n) {
// This error message intentionally doesn't describe _all_ of
// the different permutations that are technically valid as an
// HCL identifier, but rather focuses on what we might
// consider to be an "idiomatic" variable name.
return cty.DynamicVal, function.NewArgErrorf(1, "invalid template variable name %q: must start with a letter, followed by zero or more letters, digits, and underscores", n)
}
}

// We'll pre-check references in the template here so we can give a
// more specialized error message than HCL would by default, so it's
// clearer that this problem is coming from a templatefile call.
for _, traversal := range expr.Variables() {
root := traversal.RootName()
if _, ok := ctx.Variables[root]; !ok {
return cty.DynamicVal, function.NewArgErrorf(1, "vars map does not contain key %q, referenced at %s", root, traversal[0].SourceRange())
}
}

givenFuncs := funcsCb() // this callback indirection is to avoid chicken/egg problems
funcs := make(map[string]function.Function, len(givenFuncs))
for name, fn := range givenFuncs {
if name == "templatefile" || name == "core::templatefile" {
// We stub this one out to prevent recursive calls.
funcs[name] = function.New(&function.Spec{
Params: params,
Type: func(args []cty.Value) (cty.Type, error) {
return cty.NilType, fmt.Errorf("cannot recursively call templatefile from inside templatefile call")
},
})
continue
}
funcs[name] = fn
}
ctx.Functions = funcs

val, diags := expr.Value(ctx)
if diags.HasErrors() {
return cty.DynamicVal, diags
}
return val, nil
}
renderTmpl := makeRenderTemplateFunc(funcsCb, true)

return function.New(&function.Spec{
Params: params,
Params: []function.Parameter{
{
Name: "path",
Type: cty.String,
AllowMarked: true,
},
{
Name: "vars",
Type: cty.DynamicPseudoType,
},
},
Type: func(args []cty.Value) (cty.Type, error) {
if !(args[0].IsKnown() && args[1].IsKnown()) {
return cty.DynamicPseudoType, nil
Expand Down Expand Up @@ -426,7 +370,7 @@ func readFileBytes(baseDir, path string, marks cty.ValueMarks) ([]byte, error) {
}
defer f.Close()

src, err := ioutil.ReadAll(f)
src, err := io.ReadAll(f)
if err != nil {
return nil, fmt.Errorf("failed to read file: %w", err)
}
Expand Down
26 changes: 15 additions & 11 deletions internal/lang/funcs/filesystem_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,13 @@ import (
"path/filepath"
"testing"

"github.com/hashicorp/terraform/internal/lang/marks"
homedir "github.com/mitchellh/go-homedir"
"github.com/zclconf/go-cty/cty"
"github.com/zclconf/go-cty/cty/function"
"github.com/zclconf/go-cty/cty/function/stdlib"

"github.com/hashicorp/terraform/internal/collections"
"github.com/hashicorp/terraform/internal/lang/marks"
)

func TestFile(t *testing.T) {
Expand Down Expand Up @@ -149,13 +151,13 @@ func TestTemplateFile(t *testing.T) {
cty.StringVal("testdata/recursive.tmpl"),
cty.MapValEmpty(cty.String),
cty.NilVal,
`testdata/recursive.tmpl:1,3-16: Error in function call; Call to function "templatefile" failed: cannot recursively call templatefile from inside templatefile call.`,
`testdata/recursive.tmpl:1,3-16: Error in function call; Call to function "templatefile" failed: cannot recursively call templatefile from inside another template function.`,
},
{
cty.StringVal("testdata/recursive_namespaced.tmpl"),
cty.MapValEmpty(cty.String),
cty.NilVal,
`testdata/recursive_namespaced.tmpl:1,3-22: Error in function call; Call to function "core::templatefile" failed: cannot recursively call templatefile from inside templatefile call.`,
`testdata/recursive_namespaced.tmpl:1,3-22: Error in function call; Call to function "core::templatefile" failed: cannot recursively call templatefile from inside another template function.`,
},
{
cty.StringVal("testdata/list.tmpl"),
Expand Down Expand Up @@ -187,14 +189,16 @@ func TestTemplateFile(t *testing.T) {
},
}

templateFileFn := MakeTemplateFileFunc(".", func() map[string]function.Function {
return map[string]function.Function{
"join": stdlib.JoinFunc,
"core::join": stdlib.JoinFunc,
"templatefile": MakeFileFunc(".", false), // just a placeholder, since templatefile itself overrides this
"core::templatefile": MakeFileFunc(".", false), // just a placeholder, since templatefile itself overrides this
}
})
funcs := map[string]function.Function{
"join": stdlib.JoinFunc,
"core::join": stdlib.JoinFunc,
}
funcsFunc := func() (funcTable map[string]function.Function, fsFuncs collections.Set[string], templateFuncs collections.Set[string]) {
return funcs, collections.NewSetCmp[string](), collections.NewSetCmp[string]("templatefile")
}
templateFileFn := MakeTemplateFileFunc(".", funcsFunc)
funcs["templatefile"] = templateFileFn
funcs["core::templatefile"] = templateFileFn

for _, test := range tests {
t.Run(fmt.Sprintf("TemplateFile(%#v, %#v)", test.Path, test.Vars), func(t *testing.T) {
Expand Down

0 comments on commit edf335e

Please sign in to comment.