Please support something like "allow-failure" for a given job

[mvo5]

Edit from @vanZeben:

As this is a GitHub Actions platform feature request, discussion for this feature has been moved to https://github.com/orgs/community/discussions/15452, please go and put your support behind that community discussion instead of this issue.

---

We use github actions in our "snapd" project and we love them.

One small feature we would love to see is a way to mark a test job as "allow-failure" (or a term along these lines) [0]. This would simply mean that the overall /pulls overview page would show the PR as with the little green tick-mark (and maybe in the tooltip 5/6 OK, 1 ignored). It would still show as a failure in the details view (maybe with a different icon?).

Our use-case is that we have some CI environments that fail frequently because of external factors like repository mirrors that are our of sync etc. We still want to run the CI on these systems but not get distracted too much by these out-of-our-control issues.

Hope this makes sense.

Thanks!
Michael

[0] E.g.

jobs:
  spread:
    runs-on: ubuntu-latest
    allow-failure: true
    steps:
    - do-some-flaky-stuff

[thboop]

Hey @mvo5 Thanks for the feature request!

We do support marking a step to allow failure via continue-on-error.

We also support marking specific checks as required
That will allow you to merge pr's without a particular job succeeding.

I think the latter should solve your issue at the job level, is there a reason that doesn't well for you?

[mvo5]

Hey @thboop. Thanks for your quick reply!

Yeah, it's really just about the little green tick at the pull-request overview page. AFAICT when one job (even if it's not required) fails the overview PR list will show this PR as failed. Having a way to mark certain jobs as not rquired would still show the pulls as green (or yellow?) instead of red.

But I do understand this is a bit of a specific request, so feel free to close it if you think it's a bit too odd. We had it with our old CI system and I liked it.

[fingolfin]

I don't think it is that specific. People have requested this before: https://github.community/t5/GitHub-Actions/continue-on-error-allow-failure-UI-indication/td-p/37033

I just came here via Google as I was surprised I couldn't find anything like this in the documentation. It's standard with e.g. Travis CI

[jhenstridge]

I guess one way to handle this would be for jobs with an allow-failure flag set to generate a neutral conclusion on failure instead of failure. As I understand it, this would let the overall check suite conclusion be success if such a job fails.

[jadephilipoom]

I'm also an Actions user who would love to see this feature. My team's project has a long build and a few jobs that are flaky and frequently take a while to fix. We do just ignore failures on those for merging PRs, but being able to make the green tick agree with that convention would be really nice.

[jakesylvestre]

This would be great

[ghostsquad]

in the interim, what if there was a step that simply posted the "allowed failures" in a comment (that same comment updated each time the workflow ran). Similar CodeCov.

according to the docs, this is a detectable condition:

https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#steps-context

steps..outcome	string	The result of a completed step before continue-on-error is applied. Possible values are success, failure, cancelled, or skipped. When a continue-on-error step fails, the outcome is failure, but the final conclusion is success.
steps..conclusion	string	The result of a completed step after continue-on-error is applied. Possible values are success, failure, cancelled, or skipped. When a continue-on-error step fails, the outcome is failure, but the final conclusion is success.

[henryiii]

I thought that jobs.<job_id>.continue-on-error sounded exactly like this feature, but it doesn't seem to work like that. For me, at least, setting a job to "continue-on-error" still causes a red x for the workflow; it doesn't seem to have any effect at all.

[dopplershift]

By default, when any job within a workflow fails, all the other jobs are cancelled (unless 'fail-fast' is False).continue-on-error specifies that a certain job failing should not trigger cancelling other jobs. (I would definitely love some way to keep a job from signalling a failing check.)

[ramsey]

Not only does continue-on-error not work as expected (as mentioned by everyone on this thread), but the example in the documentation appears to result in a syntax error that I cannot figure out how to get around. As a result, I've been trying all sorts of permutations to figure out if this syntax error is causing continue-on-error not to work as expected.

Maybe it is?

Here's the documentation example verbatim:

runs-on: ${{ matrix.os }}
continue-on-error: ${{ matrix.experimental }}
strategy:
  fail-fast: false
  matrix:
    node: [11, 12]
    os: [macos-latest, ubuntu-18.04]
    experimental: [false]
    include:
      - node: 13
        os: ubuntu-18.04
        experimental: true

My version looks similar:

unit-tests:
  name: "Unit Tests"

  runs-on: ${{ matrix.operating-system }}
  continue-on-error: ${{ matrix.experimental }}

  strategy:
    fail-fast: false
    matrix:
      dependencies:
        - "lowest"
        - "highest"
      php-version:
        - "7.4"
        - "8.0"
      operating-system:
        - "ubuntu-latest"
      experimental: [false]
      include:
        - php: "8.0"
          composer-options: "--ignore-platform-reqs"
          experimental: true

When this runs, it fails at the unit-tests job, due to the following error. (See the failure here.)

Error when evaluating 'runs-on' for job 'unit-tests'. (Line: 121, Col: 14): Unexpected value ''

Somehow, it's getting an empty string for runs-on, or at least, that's what this error seems to indicate to me.

When I remove the line that defines experimental: [false] (and only this line), the build runs successfully (see here).

So, maybe the successful build is not actually taking into account the continue-on-error property? Does this syntax error help point to some kind of culprit that could resolve this issue for everyone on this thread?

[nschonni]

@ramsey your issue doesn't seem related to the thread. Your job is failing because your include doesn't have a value for operating-system

[ramsey]

Why does my include work without operating-system when I remove experimental: [false] from the matrix?

[ramsey]

@nschonni You were right. Turns out, I had my matrix messed up. Sorry for the churn here, y'all. 😀

[szepeviktor]

This is one solution

      - name: Install dependencies
        id: composer-run
        continue-on-error: true
        run: composer update --${{ matrix.dependency-version }} --prefer-dist --no-interaction --no-suggest

      - name: Execute tests
        if: steps.composer-run.outcome == 'success' && steps.composer-run.conclusion == 'success'
        run: vendor/bin/phpunit

continue-on-error causes difference in outcome and conclusion!

[dersimn]

I‘d need something like this to allow a whole workflow to be optional (not getting a notification on failure).

It‘s a workflow with a task like „Check if parent Docker Image was updated“ with a nightly run. If there was no update the Job fails and I’ll get a notification about failure, but in this case the job is designed to fail (most of the time) and should not spam my mails.

This one here: https://github.com/dersimn/HelloARM-GitHub-Workflow/blob/master/.github/workflows/auto-build-on-base-image-change.yml

[sigprof]

@dersimn
Instead of making check-job fail when no update is needed, you can add an output to report the result of the check without failing the job:

  check-job:
    outputs:
      needs-updating: ${{ steps.check.outputs.needs-updating }}

Then remove the Check result step and use if: needs.check-job.outputs.needs-updating != 'false' in the trigger-build job; this way you won't have failures when no updating is needed (the trigger-build job would just be marked as skipped).