RFC: melange test --test-append

[imjasonh]

With the recent addition of melange test, packages can be tested for basic functionality, which is great.

We may have a set of common tests we want to run on many packages, which we could codify in a tests/ directory, and pass to melange test <pkg> --test-append=tests/ or melange test <pkg> --test-append=tests/foo.yaml --test-append=bar.yaml

The YAML file could include any extra environment to append, and pipeline steps to append:

environment:
    contents:
      packages:
        - apk-tools
  pipeline:
    - runs: |
        apk add --simulate <pkg>

If this was run on a file which already contained tests, e.g.:

test:
  environment:
    contents:
      packages:
        - wolfi-base
  pipeline:
    - runs: |
        az --version

...then the resulting pipeline would be as if the file specified this:

test:
  environment:
    contents:
      packages:
        - apk-tools
        - wolfi-base
  pipeline:
    - runs: |
        az --version
    - runs: |
        apk add --simulate <pkg>

That <pkg> makes me think we might need some basic templating, like ${{test.package}} or something, so that the test pipeline is reusable.

WDYT @vaikas

[vaikas]

So you can already do the:
melange test test.yaml <pkg>

https://github.com/chainguard-dev/melange/blob/main/docs/TESTING.md#specifying-package-to-test--reusing-tests

Meaning that the tests do not have to be in the same file as the package definition. IIRC however, there's some cruft that needs to be in there because the melange yaml struct always requires some fields, for a good reason in general, but in this case it means a little added cruft. We can ofc change this, but just a note.
As far as being able to specify multiple files or a directory, I think I'd prefer the explicit specification of files over specifying the directory. Thinking being that you can easily create a loop for files and create the command line from it. If it's a blob directory, I could then see needing a way to exclude some tests down the line, so I'd rather not do it that way. Then just following that little further, you can already do what you propose by having a for loop over melange test from those files and invoking it multiple times.
So, in your example, you could already do this:

melange test tests/foo.yaml <pkg>
melange test tests/bar.yaml <pkg>

Because the pkg is already added to the test container, doing something like apk add seems unnecessary, but I'm curious if there are other cases where you would need to know the name of the package under test? Not saying there isn't, but just curious if we can come up with another use case.