bug: identify and silence owl-bot post-processor failures when PRs attempt to push to a branch on which the post-processor has already run

[sofisl]

See #2304 and #2306

To https://github.com/googleapis/python-billing.git
 ! [rejected]        release-v1.3.2 -> release-v1.3.2 (fetch first)
error: failed to push some refs to 'https://github.com/googleapis/python-billing.git'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.

This happens when a PR is created, then the post-processor runs on it, then the PR author attempts to push another commit to that branch without first pulling down the changes. However, ultimately, the post-processor runs fine on Github since the user can force-push in order to retrigger the post-processor. So, perhaps there is a way of silencing these errors, or perhaps a way of delaying the post-processor for a few seconds to ensure another commit pushed to that branch recently does not cause this error.

[sofisl]

cc @parthea

[bcoe]

@parthea brought up in the meeting that the user facing nuisance that arrises as a result of this, is this:

• an update to a PR is made after an OwlBot post processor is already running.
• this update will have its own OwlBot post processor run.
• the first post processor fails, because it can't push at the branch with changes, updates status to failed.
• failed status remains until second post-processor completes successfully.

The user facing aspect is the failure that is displayed until the second post processor completes.

If an actual fix for this problem is too difficult, one potential stop gap would be putting the OwlBot status back into a pending state, whenever a new post processor run starts.

[parthea]

I'd like to bump the priority on this issue as it's preventing python from adding owlbot as a required check. I had to manually add the owlbot:run label to trigger owlbot to run again after a push failure. See googleapis/python-api-core#259

[chingor13]

@danielbankhead are you looking into this one? If not, I'll reassign as it's now 🚨

[danielbankhead]

I should be able to get to this by Friday - feel free to grab if you'd like

[SurferJeffAtGoogle]

The first solution I'd try is to perform a git pull immediately before this git push:

repo-automation-bots/packages/owl-bot/cloud-build/update-pr.yaml

Line 90 in a939e26

- name: 'gcr.io/cloud-builders/git'

It will only succeed if there are no conflicts, but if there are conflicts, the PR will need human attention anyway and failure is a good signal.

[sofisl]

I am still seeing these errors in our post-processor:

https://pantheon.corp.google.com/cloud-build/builds;region=global/c9932256-7f4f-46b6-af3b-fcd3783066fb;step=0?project=repo-automation-bots

https://pantheon.corp.google.com/cloud-build/builds;region=global/95b6be11-b593-48e4-9cc8-3cf33013eb56;step=11?project=repo-automation-bots

https://pantheon.corp.google.com/cloud-build/builds;region=global/27f5ad62-0751-49b3-a130-9fa00a1adb97;step=11?project=repo-automation-bots

https://pantheon.corp.google.com/cloud-build/builds;region=global/9a088e73-3545-4993-995f-6506590deabb;step=11?project=repo-automation-bots

https://pantheon.corp.google.com/cloud-build/builds;region=global/9a088e73-3545-4993-995f-6506590deabb;step=11?project=repo-automation-bots

Although it has been greatly reduced!