Add new experimental preprocessor

[anbraten]

#2473

Can be enabled by using WOODPECKER_EXPERIMENTAL_GO_PREPROCESSOR=true
The legacy pre-processor will run as well. If it should be disabled this can be done by setting: WOODPECKER_EXPERIMENTAL_DISABLE_LEAGAY_PREPROCESSOR=true

Both settings are un-documented and this PR will be hidden from changelogs, so we can test them first with ci.woodpecker-ci.org and elaborate if we should continue this path.

[6543]

This will break usage where we use e.g. string replace or shortening within yaml!

[anbraten]

This will break usage where we use e.g. string replace or shortening within yaml!

What do you mean?

If we continue, it will be opt-in anyway see: newPreprocessor := true!

[qwerty287]

I also had an idea recently about how we can prevent parsing YAMLs multiple times with the env var problem.

Currently, we perform env-replacement on the whole YAML file.

However, the usecases for this are limited: Mainly replacing in image, settings, environment etc.

Why don't we perform the env-replacement directly on these strings?

I.e. with this config:

steps:
  - name: x
    image: some-img:${sometag}

do this:

1. Parse YAML
2. For all strings where you can use env var replacement, get the string and perform replacing there, in the example above: take the string from image and replace on some-img:${sometag}.

If supported for all properties that are used together with env vars, this could improve the YAML parsing process a lot without reducing usability of env replacement.

What do you think about this approach? Am I missing important downsides?

[anbraten]

What do you think about this approach? Am I missing important downsides?

In general an interesting approach. Could be a bit tedious to implement for all properties / yaml value types.

Something I would really like to be able to do is a basic tenary operation to do sth like:

steps:
  - name: docker build
    image: buildx
    settings:
      dry-run: {{ if eq .env.CI_PIPELINE_EVENT "pull_request"  }}true{{ else }}false{{ end }}

But I can also image options where it would be helpful to write sth like:

steps:
  - name: docker build
    image: buildx
    {{ if eq .env.CI_PIPELINE_EVENT "pull_request"  }}
    depends_on:
      - test
    {{ end }}
    settings:
      ...

[anbraten]

What do you think about adding this pre-processor behind an un-documented feature flag in addition to the ${{ ... }} syntax for some time to test it, so we can simply remove it at any time again if we notice that it is too powerful / complicated, ...

[qwerty287]

Even if it's experimental, this should be moved to flags (there's a Hidden parameter on flags)

[qwerty287]

It should be possible to use string manipulation: https://github.com/drone/envsubst?tab=readme-ov-file#supported-functions (using https://pkg.go.dev/text/template#Template.Funcs)

Also, it looks like there are default functions (https://pkg.go.dev/text/template#hdr-Functions) and they also include aliases for fmt.Println, so on a public instance, you can print anything to the log, this shouldn't be allowed.

[qwerty287]

And I don't think this must be excluded from docs and changelog - there just should be a clear message that it's experimental. The more users use and test it, the more feedback we get 😉

[lafriks]

Personally I'm very skeptical about this change, helm charts are using a similar approach and it has its problems, especially quirks with spaces etc, also it adds much complexity imho

Also it makes it invalid yaml file... For that should be used a different format, ex. starlak etc