[Feature] GitVersion Fails with GitVersion.BugException on New Commits in Running Pipeline

[tyb-dev]

Describe the bug
GitVersion fails with a GitVersion.BugException when pushing new commits to a running pipeline branch. The error message suggests that the HEAD moves unexpectedly during repo normalization.

Expected Behavior

GitVersion should complete successfully without exceptions when new commits are pushed to the branch that has a running pipeline.

Actual Behavior

When a new commit is pushed to a running pipeline branch, GitVersion fails with a GitVersion.BugException error. Re-executing pipeline on past commits on a branch immediately results in failure as well.

NFO [10/06/23 11:14:08:47] Applicable build agent found: 'GitLabCi'.
INFO [10/06/23 11:14:08:48] Working directory: /builds/group/git-test
INFO [10/06/23 11:14:08:49] Project root is: /builds/group/git-test/
INFO [10/06/23 11:14:08:49] DotGit directory is: /builds/group/git-test/.git
INFO [10/06/23 11:14:08:49] Branch from build environment: branch
INFO [10/06/23 11:14:08:49] Begin: Normalizing git directory for branch 'branch'
  INFO [10/06/23 11:14:08:51] One remote found (origin -> 'https://gitlab-ci-token:*******@code.example.com/group/git-test.git').
  INFO [10/06/23 11:14:08:52] Skipping fetching, if GitVersion does not calculate your version as expected you might need to allow fetching or use dynamic repositories
  INFO [10/06/23 11:14:08:52] Creating local branch branch pointing at d55344d
  INFO [10/06/23 11:14:08:57] Creating local branch from remote tracking 'refs/remotes/origin/main'.
  INFO [10/06/23 11:14:08:58] HEAD points at branch 'refs/heads/branch'.
  INFO [10/06/23 11:14:08:58] End: Normalizing git directory for branch 'branch' (Took: 83.34ms)
  ERROR [10/06/23 11:14:08:59] An unexpected error occurred:
GitVersion.BugException: GitVersion has a bug, your HEAD has moved after repo normalisation.
To disable this error set an environmental variable called IGNORE_NORMALISATION_GIT_HEAD_MOVE to 1
Please run `git log --graph --format="%h %cr %d" --decorate --date=relative --all --remotes=* -n 100` and submit it along with your build log (with personal info removed) in a new issue at https://github.com/GitTools/GitVersion
   at GitVersion.GitPreparer.NormalizeGitDirectory(Boolean noFetch, String currentBranchName, Boolean isDynamicRepository) in D:\a\GitVersion\GitVersion\src\GitVersion.Core\Core\GitPreparer.cs:line 192
   at GitVersion.GitPreparer.NormalizeGitDirectory(String targetBranch, Boolean isDynamicRepository) in D:\a\GitVersion\GitVersion\src\GitVersion.Core\Core\GitPreparer.cs:line 134
   at GitVersion.GitPreparer.PrepareInternal(GitVersionOptions gitVersionOptions) in D:\a\GitVersion\GitVersion\src\GitVersion.Core\Core\GitPreparer.cs:line 69
   at GitVersion.GitPreparer.Prepare() in D:\a\GitVersion\GitVersion\src\GitVersion.Core\Core\GitPreparer.cs:line 48
   at GitVersion.GitVersionCalculateTool.CalculateVersionVariables() in D:\a\GitVersion\GitVersion\src\GitVersion.Core\Core\GitVersionCalculateTool.cs:line 43
   at GitVersion.GitVersionExecutor.RunGitVersionTool(GitVersionOptions gitVersionOptions) in D:\a\GitVersion\GitVersion\src\GitVersion.App\GitVersionExecutor.cs:line 66
  INFO [10/06/23 11:14:08:59] Attempting to show the current git graph (please include in issue): 
  INFO [10/06/23 11:14:08:59] Showing max of 100 commits
  INFO [10/06/23 11:14:08:60] * d55344d 6 seconds ago  (HEAD -> branch, origin/branch)
* d2c12a9 2 minutes ago  (origin/main, refs/pipelines/12604226, main)
* 746426c 3 minutes ago 

Possible Fix

N/A

Steps to Reproduce

1. Create a new GitLab project.
2. Commit file: gitlab-ci.yml.
3. Create a new branch and start a pipeline.
4. While the pipeline is running, commit and push a new change to the branch.

Example: After creating a branch the pipeline executes the job "versioning1" and "versioning2" successfully. After adding an additional commit to the branch, the running pipeline's remaining job "versioning3" fails. Although GitLab performs a detached HEAD checkout, git clone picks up the new unrelated commit and GitVersion is unable to normalize.

variables:
  GIT_DEPTH: 0  # Disables shallow clone

.default_template: &gitversion_template
  tags: 
    - x86_64
    - linux
  image: 
    name: gittools/gitversion:latest
    entrypoint: [""]
  script:
    - /tools/dotnet-gitversion

stages:
  - versioning1
  - versioning2
  - versioning3
  - versioning4

gitversion1:
  <<: *gitversion_template
  stage: versioning1

gitversion2:
  <<: *gitversion_template
  stage: versioning2

gitversion3:
  <<: *gitversion_template
  stage: versioning3

gitversion4:
  <<: *gitversion_template
  stage: versioning4

Context

Our team is unable to commit while another pipeline is running on the same branch.

IGNORE_NORMALISATION_GIT_HEAD_MOVE suppresses the exception, however the code running after GitVersion will work on the wrong commit. /nonormalize avoids the issue, however, it is not clear whether there are side-effects. We suspect there is a reason for normalization.

Your Environment

We use an on-prem GitLab instance. Once the error is established on the GitLab project, the error can be reproduced locally without GitLab. The prerequisite is that there is a remote defined and two environment variables are set:

git clone https://code.example.com/group/git-test.git
cd git-test
git checkout 8ef5fe10b126b1dca847e725e55d432bfec174a2
set GITLAB_CI=true
set CI_COMMIT_REF_NAME=branch
`` 

- Version Used: 5.12.0
- Operating System and version: Windows 10, 11, Ubuntu 22.04, Debian 12.1
- Link to your project: N/A, empty project with `.gitlab-ci.yml` file was provided above
- Link to your CI build: N/A, the build output has been provided above

[tyb-dev]

Note that the issue also occurs when re-running a pipeline on old commits, it is not limited to race condition cases only.

[HHobeck]

I found the following definition what normalization does:

Normalization of a git directory turns all remote branches into local branches,
turns pull request refs into a real branch and a few other things.
This is designed to be run *only on the build server* which checks out repositories in different ways.
It is not recommended to run normalization against a local repository

If we take this definition my naive assumption would be:

1. The build server fetches the source from a remote git-repository in a separated workspace
2. Normalize the local repository and
3. Calculate the next version

If you get such an error on step two only when you have no sequential pipeline runs I would say the problem is not located in GitVersion right? Because GitVersion needs a stable workspace with no interdifferences from ohter build runs.

Have you checked that the build pipeline configuration:

1. Ensures that only one pipeline run accesses the workspace on the build agent at the same time
2. The workspace will be cleared before fetching
3. A clean fetch will be performed with depth 0

[HHobeck]

Note that the issue also occurs when re-running a pipeline on old commits, it is not limited to race condition cases only.

If this is the case then please create an integration test to reproduce it in an isolated environment. Maybe this example can help: #3351 (comment)

Thank you!

[nluzgin]

Got the same issue

RROR [01/23/24 12:02:29:80] An unexpected error occurred: GitVersion.BugException: GitVersion has a bug, your HEAD has moved after repo normalisation after step 'EnsureLocalBranchExistsForCurrentBranch' To disable this error set an environmental variable called IGNORE_NORMALISATION_GIT_HEAD_MOVE to 1 Please run git log --graph --format="%h %cr %d" --decorate --date=relative --all --remotes=* -n 100 and submit it along with your build log (with personal info removed) in a new issue at https://github.com/GitTools/GitVersion at GitVersion.GitPreparer.EnsureRepositoryHeadDuringNormalisation(String occasion, String expectedSha) in /_/src/GitVersion.Core/Core/GitPreparer.cs:line 203 at GitVersion.GitPreparer.NormalizeGitDirectory(Boolean noFetch, String currentBranchName, Boolean isDynamicRepository) in /_/src/GitVersion.Core/Core/GitPreparer.cs:line 164 at GitVersion.GitPreparer.NormalizeGitDirectory(String targetBranch, Boolean isDynamicRepository) in /_/src/GitVersion.Core/Core/GitPreparer.cs:line 1[34](https://gitlab.services.mts.ru/devsecops3000Pro/public/api/python_ascapi/-/jobs/25372814#L34) at GitVersion.GitPreparer.PrepareInternal(GitVersionOptions gitVersionOptions) in /_/src/GitVersion.Core/Core/GitPreparer.cs:line 69 at GitVersion.GitPreparer.Prepare() in /_/src/GitVersion.Core/Core/GitPreparer.cs:line 48 at GitVersion.GitVersionCalculateTool.CalculateVersionVariables() in /_/src/GitVersion.Core/Core/GitVersionCalculateTool.cs:line 44 at GitVersion.GitVersionExecutor.RunGitVersionTool(GitVersionOptions gitVersionOptions) in /_/src/GitVersion.App/GitVersionExecutor.cs:line 66 INFO [01/23/24 12:02:29:80] Attempting to show the current git graph (please include in issue): INFO [01/23/24 12:02:29:80] Showing max of 100 commits INFO [01/23/24 12:02:29:82] * 9b50e2e 25 minutes ago (origin/master)

git log --graph --format="%h %cr %d" --decorate --date=relative --all --remotes=* -n 100
cmd.txt

IGNORE_NORMALISATION_GIT_HEAD_MOVE == 1 result a 0.0.1 version </3

UPD:
+++ Ensures that only one pipeline run accesses the workspace on the build agent at the same time
+++ The workspace will be cleared before fetching
+++ A clean fetch will be performed with depth 0

UPD[2]:

No push force or something like this have been done with repo

UPD[3]:
Found new gitlab agent with cache
After disabling problem was solved =)

[DefinitelyADev]

Is there a workaround? For me it only fails on pipelines that didn't start with a tag. For example merge request pipelines.
Cleaning runner caches had no effect.

Edit:
Added cache: [] had no effect

[nluzgin]

@DefinitelyADev did you disable cache for gitlab runner or disable cache for job?
My case is disable for runner

[DefinitelyADev]

@nluzgin i tried both.