Investigate go-git concurrency issues

[nerdalert]

Consistently seeing 3/3 on retries. Either bump them up or dig into the issue. Not a big deal to just add more retries imo since it's just a local thing but worth a little time investigating.

Example:

2024-04-04T18:54:19.938Z	INFO	cmd/generate.go:246	Processing job	{"job": "5"}
2024-04-04T18:54:20.037Z	INFO	cmd/generate.go:310	Retrying fetching updates, attempt 2/3	{"job": "5", "pr_number": "4", "work_dir": "/data", "origin": "origin"}
2024-04-04T18:54:22.097Z	INFO	cmd/generate.go:310	Retrying fetching updates, attempt 3/3	{"job": "5", "pr_number": "4", "work_dir": "/data", "origin": "origin"}

[russellb]

Another option would be to bail on go-git and use a wrapper for the real git.

https://github.com/ldez/go-git-cmd-wrapper?tab=readme-ov-file

[nerdalert]

@mingxzhao reported a back to back error in this job instructlab/taxonomy#80 :

Apr 18 15:49:24 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[226392]: 2024-04-18T15:49:24.581Z        INFO        cmd/generate.go:647        Job took 34s to run
Apr 18 15:49:24 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[226392]: 2024-04-18T15:49:24.633Z        INFO        cmd/generate.go:518        Job done        {"job": "163", "pr_number": "715", "work_dir": "/home/fedora/instruct-lab-bot", "origin": "origin", "out_dir": "/home/fedora/instruct-lab-bot/precheck-pr-715-8e87e8e15b71c0fe3e9aa7aced29211003d5c138"}
Apr 18 15:57:55 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[226392]: 2024-04-18T15:57:55.883Z        INFO        cmd/generate.go:341        Processing job 164        {"job": "164"}
Apr 18 15:57:56 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[226392]: 2024-04-18T15:57:56.034Z        INFO        cmd/generate.go:561        Retrying fetching updates, attempt 2/3        {"job": "164", "pr_number": "80", "work_dir": "/home/fedora/instruct-lab-bot", "origin": "origin"}
Apr 18 15:57:58 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[226392]: 2024-04-18T15:57:58.167Z        INFO        cmd/generate.go:561        Retrying fetching updates, attempt 3/3        {"job": "164", "pr_number": "80", "work_dir": "/home/fedora/instruct-lab-bot", "origin": "origin"}
Apr 18 15:58:00 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[226392]: 2024-04-18T15:58:00.660Z        ERROR        cmd/generate.go:389        git operations error: could not fetch PR branch: some refs were not updated
Apr 18 15:58:00 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[226392]: github.com/instruct-lab/instruct-lab-bot/worker/cmd.(*Worker).processJob
Apr 18 15:58:00 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[226392]:         /home/fedora/worker-intruct-lab-bot/instruct-lab-bot/worker/cmd/generate.go:389
Apr 18 15:58:00 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[226392]: github.com/instruct-lab/instruct-lab-bot/worker/cmd.glob..func1.4
Apr 18 15:58:00 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[226392]:         /home/fedora/worker-intruct-lab-bot/instruct-lab-bot/worker/cmd/generate.go:196
Apr 18 17:40:59 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[226392]: 2024-04-18T17:40:59.883Z        INFO        cmd/generate.go:341        Processing job 165        {"job": "165"}
Apr 18 17:41:00 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[226392]: 2024-04-18T17:41:00.382Z        ERROR        cmd/generate.go:389        git operations error: could not fetch PR branch: some refs were not updated
Apr 18 17:41:00 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[226392]: github.com/instruct-lab/instruct-lab-bot/worker/cmd.(*Worker).processJob
Apr 18 17:41:00 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[226392]:         /home/fedora/worker-intruct-lab-bot/instruct-lab-bot/worker/cmd/generate.go:389
Apr 18 17:41:00 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[226392]: github.com/instruct-lab/instruct-lab-bot/worker/cmd.glob..func1.4
Apr 18 17:41:00 ip-172-31-13-79.us-east-2.compute.internal instruct-lab-bot-worker[226392]:         /home/fedora/worker-intruct-lab-bot/instruct-lab-bot/worker/cmd/generate.go:196

[nerdalert]

Here was another incident of this: instructlab/taxonomy#1060