ci-matrix: add partition-tests-inpackage #277
Conversation
|
@dnephin Could you push this to a branch and tag it? I would very much like to run this in CI and validate. |
| if opts.partitionTestsInPackage != "" { | ||
| p.Packages = opts.partitionTestsInPackage | ||
| p.Description = fmt.Sprintf("partition %d with %d tests", p.ID, len(bucket.Items)) | ||
| p.Tests = fmt.Sprintf("-run='^%v$'", strings.Join(bucket.Items, "$,^")) |
There was a problem hiding this comment.
I believe this should be:
| p.Tests = fmt.Sprintf("-run='^%v$'", strings.Join(bucket.Items, "$,^")) | |
| p.Tests = fmt.Sprintf("-run ^%v$", strings.Join(bucket.Items, "|")) |
See:
There was a problem hiding this comment.
Oh ya, the join needs to be a pipe not a comma for sure! Good catch.
I think you're right that we don't need the ^ and $ around each test name, but I'll need to test that out again. go test has some strange handling for regex (ex: golang/go#39904). We'll need to add ( and ) if we remove those.
Both = and should work for the separator. The = is nice because you can pass it as a single quoted arg instead of it being two separate arguments.
I believe the whole string does need to be quoted with single quotes so that the pipes and $ are not interpreted by the shell. In your case that may not be a problem because you're running it from python, but most of the time I expect this to be read from bash.
There was a problem hiding this comment.
Yeah, I wish there were a simpler (& more efficient) way to provide a list of tests.
I don't think the whole string needs to be quoted - or you should let the user do the quoting. Easier for them to add quotes of the appropriate kind, and GitHub Actions allows plenty of ways to inject a variable or string into a script, e.g.: the following will ensure that any special characters are handled correctly:
env:
TESTS: ${{ inputs.tests }}
run:
echo $TESTSThere was a problem hiding this comment.
My primary goal with this command is to make it easy to integrate into a github workflow. Having to set a value into an env var just to use it does not make it easy. I expect someone to be able to do something like this and not have to worry about escaping or formatting the values.
I see your use case is quite different. You have a lot of code already in place, and you're looking for a tool to perform the test bucketing.
I think for your use case we could add a --format flag to this command. The default would be --format=github-action-matrix would be a JSON output that you can use directly in a github actions matrix. For your use case we could do --format=json, which would output the package list and test list as an array (instead of a space separated string, or a -run flag). That should make it easier for you to consume the output, while still supporting my primary goal of making it easy to use in a github workflow.
dnephin
force-pushed
the
ci-matrix
branch
2 times, most recently
from
255c099
to
d0adc24
Compare
|
That's weird, I deleted the base branch and instead of re-targeting Edit: ok, I was able to get it back by restoring the branch |
dnephin
force-pushed
the
ci-matrix-by-file
branch
from
d09768c
to
2adbb4a
Compare
TODO: finish the test case for the new flag