terraform-provider-dataprocessor

Avoid ugly terraform logic and code to transform data. This Terraform provider helps you with the data processing in a clean and easy way by using tools like JQ, YQ and Go plugins.

Processors

JQ

The famous and well known JQ processor for your JSON inputs.

YQ

The famous and well known YQ processor for your YAML inputs.

Go plugins v1

Check examples:

• FS check: Checks files exist on disk. Shows how you can access the FS outside the plugin.
• Complex validation: Validate Prometheus Rules. Shows how to create advanced logic plugins.
• Data structure transformation: Transforms a data structure into another. Shows how to transform data for easier consumption by different terraform providers.
• Filtering: Filters a list of usernames based on a regex. Shows how to filter terraform data to avoid HCL complex logic.
• Remote plugin: Uses a plugin that is hosted in github. Shows how plugins can be shared and create plugin repos.
• Simple validation: Validates the length of a string. Shows that simple validation plugins can be powerful (like small functions), perfect to be used as a remote plugin.

The processor for everything 🎉, is the most powerful of all. You can use almost (e.g unsafe package is banned) all the Go standard library. These are the requirements to create a plugin:

• Written in Go.
• No external dependencies, only Go standard library.
• Implemented in a single file (or string block).
• Implement the plugin API (Check the examples to know how to do it).
  • The Filter function should be called:ProcessorPluginV1.
  • The Filter function should have this signature: ProcessorPluginV1(ctx context.Context, inputData string, vars map[string]string) (result string, error error).

This is the simplest plugin that you could create, a noop:

package tfplugin

import "context"

func ProcessorPluginV1(ctx context.Context, inputData string, vars map[string]string) (string, error) {
 return inputData, nil
}

However you can do complex things like loading JSON, HTTP requests, using timers, complex regex validations, templating...

Go plugins are implemented with Yaegi, so they are portable and can run anywhere terraform can run.

Use cases

• Generate, filter, mutate... JSON data.
• Retrieve JSON data from APIs, process and use it on other Terraform providers resources.
• Remove ugly HCL code in favor of a more clean and powerful data processing approach.
• Data validation (including complex cases).

Requirements

• Terraform >=1.x.

Terraform cloud

The provider its compatible with Terraform cloud workers, its focused on portability and thats why it doesn't require any binary CLI.

Development

To install your plugin locally you can do make install, it will build and install in your ${HOME}/.terraform/plugins/...

Note: The installation is ready for OS_ARCH=linux_amd64, so you make need to change the Makefile if using other OS.

Example:

cd ./examples/local
rm -rf ./.terraform ./.terraform.lock.hcl
cd -
make install
cd -
terraform plan