Analysis step in a transient state

[maiste]

Sometimes, when running the analysis step, it wouldn't rebuild, get stuck and would always show the same error message. To solve that locally, the solution is to clean the local state and rerun the CI.

It needs more investigation but it could be solved by the solver-service.

[tmcgilchrist]

An example from ocaml/ocaml-file-format https://ci.ocamllabs.io:8100/job/2022-12-13/165403-git-fetch-7502d0

2022-12-13 16:54.03: New job: git fetch https://github.com/ocaml/opam-file-format.git#refs/pull/47/head (91af0c0e6148bda1e268c35dc8f62159d254649e)
2022-12-13 16:54.03: Exec: "git" "-C" "/var/lib/ocurrent/var/git/opam-file-format.git-86eb760e672d02d33591ec88eb8e890f599af36acbd25c5eac00005e3f372f69" 
                           "branch" "-f" "fetch-91af0c0e6148bda1e268c35dc8f62159d254649e" 
                           "91af0c0e6148bda1e268c35dc8f62159d254649e"
fatal: Not a valid branch point: '91af0c0e6148bda1e268c35dc8f62159d254649e'.
2022-12-13 16:54.03: Exec: "git" "-C" "/var/lib/ocurrent/var/git/opam-file-format.git-86eb760e672d02d33591ec88eb8e890f599af36acbd25c5eac00005e3f372f69" 
                           "fetch" "--recurse-submodules=false" "-q" 
                           "-f" "https://github.com/ocaml/opam-file-format.git" 
                           "refs/pull/47/head"
2022-12-13 16:54.03: Exec: "git" "-C" "/var/lib/ocurrent/var/git/opam-file-format.git-86eb760e672d02d33591ec88eb8e890f599af36acbd25c5eac00005e3f372f69" 
                           "branch" "-f" "fetch-91af0c0e6148bda1e268c35dc8f62159d254649e" 
                           "91af0c0e6148bda1e268c35dc8f62159d254649e"
fatal: Not a valid branch point: '91af0c0e6148bda1e268c35dc8f62159d254649e'.
2022-12-13 16:54.03: Job failed: Command "git" "-C" "/var/lib/ocurrent/var/git/opam-file-format.git-86eb760e672d02d33591ec88eb8e890f599af36acbd25c5eac00005e3f372f69" 
"branch" "-f" "fetch-91af0c0e6148bda1e268c35dc8f62159d254649e" "91af0c0e6148bda1e268c35dc8f62159d254649e" exited with status 128

The git commit in question does not belong to any branch on this repository, and may belong to a fork outside of the repository. Looking at the PR for that job ocaml/opam-file-format#47 that commit is available.

Rerunning the fetch step just now gives:

2022-12-13 22:08.46: New job: git fetch https://github.com/ocaml/opam-file-format.git#refs/pull/47/head (91af0c0e6148bda1e268c35dc8f62159d254649e)
2022-12-13 22:08.46: Exec: "git" "-C" "/var/lib/ocurrent/var/git/opam-file-format.git-86eb760e672d02d33591ec88eb8e890f599af36acbd25c5eac00005e3f372f69" 
                           "branch" "-f" "fetch-91af0c0e6148bda1e268c35dc8f62159d254649e" 
                           "91af0c0e6148bda1e268c35dc8f62159d254649e"
fatal: Not a valid branch point: '91af0c0e6148bda1e268c35dc8f62159d254649e'.
2022-12-13 22:08.46: Exec: "git" "-C" "/var/lib/ocurrent/var/git/opam-file-format.git-86eb760e672d02d33591ec88eb8e890f599af36acbd25c5eac00005e3f372f69" 
                           "fetch" "--recurse-submodules=false" "-q" 
                           "-f" "https://github.com/ocaml/opam-file-format.git" 
                           "refs/pull/47/head"
2022-12-13 22:08.47: Exec: "git" "-C" "/var/lib/ocurrent/var/git/opam-file-format.git-86eb760e672d02d33591ec88eb8e890f599af36acbd25c5eac00005e3f372f69" 
                           "branch" "-f" "fetch-91af0c0e6148bda1e268c35dc8f62159d254649e" 
                           "91af0c0e6148bda1e268c35dc8f62159d254649e"
2022-12-13 22:08.47: Exec: "git" "-C" "/var/lib/ocurrent/var/git/opam-file-format.git-86eb760e672d02d33591ec88eb8e890f599af36acbd25c5eac00005e3f372f69" 
                           "reset" "--hard" "-q" "91af0c0e6148bda1e268c35dc8f62159d254649e"
2022-12-13 22:08.47: Exec: "git" "-C" "/var/lib/ocurrent/var/git/opam-file-format.git-86eb760e672d02d33591ec88eb8e890f599af36acbd25c5eac00005e3f372f69" 
                           "submodule" "sync"
2022-12-13 22:08.47: Exec: "git" "-C" "/var/lib/ocurrent/var/git/opam-file-format.git-86eb760e672d02d33591ec88eb8e890f599af36acbd25c5eac00005e3f372f69" 
                           "submodule" "deinit" "--force" "--all"
2022-12-13 22:08.47: Exec: "git" "-C" "/var/lib/ocurrent/var/git/opam-file-format.git-86eb760e672d02d33591ec88eb8e890f599af36acbd25c5eac00005e3f372f69" 
                           "submodule" "update" "--recursive" "--init"
2022-12-13 22:08.47: Job succeeded

[novemberkilo]

Sometimes this requires that the worker's cache be wiped. Rebuilding a step from the UI results in the same worker being picked (unless it has been paused or taken down). Is it possible to improve the results of a rebuild by choosing a different worker for the rebuild?

[mtelvers]

This might occasionally make the rebuild work where it previously failed, but on the whole, this would be a significant performance hit. Better to try to detect the issue on the worker and get it to fix/delete the cache. Such as what @kit-ty-kate did here ocurrent/ocluster@318a69f