You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The regression tests are not kept up to date very well. Moreover, the timings can be very different when run on gitlab's CI runner versus some user's computer. See #505.
My proposal: when the runner fails due to a difference between case-studies-regression
and case-studies,
it puts case-studies somewhere (e.g., using this mechanism https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts) and outputs a command for the user to obtain this data and overwrite their local case-studies-regression with it. The user confirms that the difference is okay but committing this artefact to the same PR.
The text was updated successfully, but these errors were encountered:
rkunnema
changed the title
Regression tests
Store updated regression tests computed by runner
Nov 10, 2022
Ideally, whenever regressionTests.py returns an error code, the runner outputs:
The regression tests returned an error code. The analysis results are stored in
$PLACE
Please inspect the regression test's output carefully (CONTRIBUTING.md lists
the criteria). If your modifications are justified, update and commit the directory
case-studies-regression in this PRs branch with the following commands:
$COMMANDS
The regression tests are not kept up to date very well. Moreover, the timings can be very different when run on gitlab's CI runner versus some user's computer. See #505.
My proposal: when the runner fails due to a difference between
case-studies-regression
and
case-studies
,it puts
case-studies
somewhere (e.g., using this mechanism https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts) and outputs a command for the user to obtain this data and overwrite their local case-studies-regression with it. The user confirms that the difference is okay but committing this artefact to the same PR.The text was updated successfully, but these errors were encountered: