-
-
Notifications
You must be signed in to change notification settings - Fork 580
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test: Run colima tests native instead of github, fixes #5598 #5886
Conversation
Download the artifacts for this pull request:
See Testing a PR |
Both colima tests (vz and qemu) ran in about an hour and a half (and succeeded). Yay! |
This reverts commit 3b47419.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great to me!
And I think we can add comments to all buildkite config yml files with clear info about where it can be run at the top of the file, without using the branches:
stanza, to understand the expected behavior for each test:
For example:
add to .buildkite/macos-orbstack.yml
# See https://buildkite.com/ddev/macos-orbstack/settings/repository
# Runs on master, PRs, and 3rd party PRs
This is not ideal, of course, but it gives us a better understanding of the test, and when we return to it in six months or a year, we can already see how it should behave.
d56ccba
to
6fe7779
Compare
Added suggested comments to buildkite builds and made some adjustments so we don't see builds that don't happen. I think we may not be using buildkite entirely right and may be able to improve how we're doing it. |
Right now we're running 3 tests on every PR (colima, colima_vz, orbstack) that require tb-macos-arm64-(5-6-7), so they're fully occupied for one PR. On master, we're running 4 (colima, colima_vz, orbstack, rancher) I think we'll probably want to dial back one of the colima ones soon. |
The Issue
We have never been able to get predictable results from colima tests on Github Actions. Maybe the VMs are too small
Also, we were never able to do normal https tests, so there are a bunch of things that require http, we might be able to remove those.
How This PR Solves The Issue
Run the builds on our larger buildkite test machines.
Detail
Testing by running TestExtraPortExpose with:
GOTEST_SHORT=true make testpkg TESTARGS="-run TestExtraPortExpose -count=1"
It seems to work after a
colima start
and then acolima restart
. And then it stays working. So weird.