Changeset detection broke between 866.vdea_7dcd3008e and 874.v659a_b_70f5e69

[friesoft]

Jenkins and plugins versions report

Environment

Bitbucket Server: 8.9.9
Jenkins: 2.444 # also not working with 2.445
OS: Linux - 3.10.0-1160.105.1.el7.x86_64
Java: 17.0.10 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
authentication-tokens:1.53.v1c90fd9191a_b_
authorize-project:1.7.1
basic-branch-build-strategies:81.v05e333931c7d
bitbucket:241.v6d24a_57f9359
bootstrap4-api:4.6.0-6
bootstrap5-api:5.3.2-3
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1148.vce12cfcdf090
build-timeout:1.32
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.0.2
cloudbees-bitbucket-branch-source:866.vdea_7dcd3008e # downgraded from 874.v659a_b_70f5e69
cloudbees-folder:6.921.vfb_b_224371fb_4
command-launcher:107.v773860566e2e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-95.v22a_d30ee5d36
credentials:1319.v7eb_51b_3a_c97b_
credentials-binding:657.v2b_19db_7d6e6d
data-tables-api:1.13.8-2
display-url-api:2.200.vb_9327d658781
durable-task:550.v0930093c4b_a_6
echarts-api:5.4.3-2
email-ext:2.104
emailext-template:1.5
extended-choice-parameter:376.v2e02857547b_a_
font-awesome-api:6.5.1-2
git:5.2.1
git-client:4.6.0
git-server:114.v068a_c7cc2574
gson-api:2.10.1-15.v0d99f670e0a_7
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
instance-identity:185.v303dc7c645f9
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.16.1-373.ve709c6871598
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jjwt-api:0.11.5-77.v646c772fddb_0
jobConfigHistory:1229.v3039470161a_d
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery:1.12.4-1
jquery3-api:3.7.1-1
jsch:0.2.16-86.v42e010d9484b_
json-path-api:2.9.0-33.v2527142f2e1d
junit:1259.v65ffcef24a_88
lockable-resources:1240.v43673a_a_7944a_
mailer:463.vedf8358e006b_
matrix-auth:3.2.1
matrix-project:822.824.v14451b_c0fd42
mercurial:1260.vdfb_723cdcc81
mina-sshd-api-common:2.12.0-90.v9f7fb_9fa_3d3b_
mina-sshd-api-core:2.12.0-90.v9f7fb_9fa_3d3b_
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-config-history:1.6
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:704.vc58b_8890a_384
pipeline-input-step:491.vb_07d21da_1a_fb_
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2175.v76a_fff0a_2618
pipeline-model-definition:2.2175.v76a_fff0a_2618
pipeline-model-extensions:2.2175.v76a_fff0a_2618
pipeline-rest-api:2.34
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2175.v76a_fff0a_2618
pipeline-stage-view:2.34
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.8.0
popper-api:1.16.1-3
popper2-api:2.11.6-4
rebuild:330.v645b_7df10e2a_
resource-disposer:0.23
reverse-proxy-auth-plugin:1.7.7
scm-api:683.vb_16722fb_b_80b_
script-security:1326.vdb_c154de8669
simple-theme-plugin:176.v39740c03a_a_f5
snakeyaml-api:2.2-111.vc6598e30cc65
ssh-credentials:308.ve4497b_ccd8f4
sshd:3.322.v159e91f6a_550
structs:337.v1b_04ea_4df7c8
timestamper:1.26
token-macro:400.v35420b_922dcb_
trilead-api:2.133.vfb_8a_7b_9c5dd1
variant:60.v7290fc0eb_b_cd
workflow-aggregator:596.v8c21c963d92d
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1042.ve7b_140c4a_e0c
workflow-cps:3867.v535458ce43fd
workflow-durable-task-step:1331.vc8c2fed35334
workflow-job:1400.v7fd111b_ec82f
workflow-multibranch:783.va_6eb_ef636fb_d
workflow-scm-step:415.v434365564324
workflow-step-api:657.v03b_e8115821b_
workflow-support:865.v43e78cc44e0d
ws-cleanup:0.45

What Operating System are you using (both controller, and any agents involved in the problem)?

Red Hat Enterprise Linux Server 7.9

Reproduction steps

We are using the following function to gather changed yaml files:

@NonCPS
def gatherChangedFiles() {
    def changeLogSets = currentBuild.changeSets
    def files = []
    for (int i = 0; i < changeLogSets.size(); i++) {
        def entries = changeLogSets[i].items
        for (int j = 0; j < entries.length; j++) {
            def entry = entries[j]
            def fileList = new ArrayList(entry.affectedFiles)
            for (int k = 0; k < fileList.size(); k++) {
                def file = fileList[k]
                if (!"delete".equals(file.editType.name) && file.path.endsWith(".yaml")) {
                    files.add(file.path)
                }
            }
        }
    }
    files = files.unique()
    return files
}

This can be reproduced by a simple git commit in the repo which is being built, modifying one of the yaml files (function filters for Files ending with ".yaml"). With the old version this triggered an entry in the array, using the new version the array stays empty.

Expected Results

Using 874.v659a_b_70f5e69 the returned files variable (from our function) is empty. We would expect to still get all changed files from the function mentioned above

Actual Results

After upgrading from 866.vdea_7dcd3008e to 874.v659a_b_70f5e69 changeset detection doesn't work anymore.

Anything else?

No response

Are you interested in contributing a fix?

No response

[lifeofguenter]

potentially @andrey-fomin if you have a look please?

[kenden]

We are also seeing something strange after upgrading this plugin from 866.vdea_7dcd3008e to 874.v659a_b_70f5e69.
I'm adding here because it seems related. It you don't think so, let me know and I'll create another issue.

A multi-branch pipeline with "Discover pull requests from origin" used to show this at the beginning of the logs:

git fetch --tags --force --progress -- ssh://git@bitbucket.redactedserver.com/repo_example.git +refs/heads/*:refs/remotes/origin/* 
git fetch --tags --force --progress -- ssh://git@bitbucket.redactedserver.com/repo_example.git +refs/pull-requests/38/from:refs/remotes/origin/PR-38
git fetch --tags --force --progress -- ssh://git@bitbucket.redactedserver.com/repo_example.git +refs/heads/main:refs/remotes/upstream/main

and after the upgrade it just shows:

git fetch --tags --force --progress -- ssh://git@bitbucket.redactedserver.com/repo_example.git +refs/heads/branch_of_pr_38:refs/remotes/origin/branch_of_pr_38

We don't have local git information about pull requests, which causes issues in scripts we use.

This PR: https://github.com/jenkinsci/bitbucket-branch-source-plugin/pull/796/files
seems to be the cause: there are related changes to file BitbucketGitSCMBuilder.java

[mtheiss]

This change also leads to SonarQube being unable to capture the changes and always reporting 0 changed files.

SCM collecting changed files in the branch
Could not find ref 'master' in refs/heads, refs/remotes, refs/remotes/upstream or refs/remotes/origin
SCM collecting changed files in the branch (done) | time=3ms

[andrey-fomin]

I'll check

[andrey-fomin]

I can't reproduce this issue. For me change log is empty for the first build and contains some changes in next builds if branch is updated.

Do you use default strategy to compute change log agains previous build? Or do you use ChangelogToBranch extension? I can guess that new fetching implementation doesn't work properly with ChangelogToBranch extension.
@friesoft @kenden

[friesoft]

We do nothing else beside of calling the mentioned function. We always got (and rely on) the change delta from last build

I've attached our job config in case this is of any help

config.xml

<?xml version='1.1' encoding='UTF-8'?>
<jenkins.branch.OrganizationFolder plugin="branch-api@2.1046.v0ca_37783ecc5">
  <actions/>
  <description></description>
  <displayName>Konfigurationen</displayName>
  <properties>
    <jenkins.branch.OrganizationChildHealthMetricsProperty>
      <templates>
        <com.cloudbees.hudson.plugins.folder.health.WorstChildHealthMetric plugin="cloudbees-folder@6.773.vd2dcc704ee7e">
          <nonRecursive>false</nonRecursive>
        </com.cloudbees.hudson.plugins.folder.health.WorstChildHealthMetric>
      </templates>
    </jenkins.branch.OrganizationChildHealthMetricsProperty>
    <jenkins.branch.OrganizationChildOrphanedItemsProperty>
      <strategy class="jenkins.branch.OrganizationChildOrphanedItemsProperty$Inherit"/>
    </jenkins.branch.OrganizationChildOrphanedItemsProperty>
    <jenkins.branch.OrganizationChildTriggersProperty>
      <templates>
        <com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger plugin="cloudbees-folder@6.773.vd2dcc704ee7e">
          <spec>H H/4 * * *</spec>
          <interval>86400000</interval>
        </com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger>
      </templates>
    </jenkins.branch.OrganizationChildTriggersProperty>
    <com.cloudbees.hudson.plugins.folder.properties.FolderCredentialsProvider_-FolderCredentialsProperty plugin="cloudbees-folder@6.773.vd2dcc704ee7e">
      <domainCredentialsMap class="hudson.util.CopyOnWriteMap$Hash">
        <entry>
          <com.cloudbees.plugins.credentials.domains.Domain plugin="credentials@1189.vf61b_a_5e2f62e">
            <specifications/>
          </com.cloudbees.plugins.credentials.domains.Domain>
          <java.util.concurrent.CopyOnWriteArrayList>
            <com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl plugin="credentials@1189.vf61b_a_5e2f62e">
              <id>36c31674-7dc1-4af2-9447-677b5393c8a1</id>
              <description></description>
              <username><entfernt></username>
              <password><entfernt></password>
              <usernameSecret>false</usernameSecret>
            </com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
          </java.util.concurrent.CopyOnWriteArrayList>
        </entry>
      </domainCredentialsMap>
    </com.cloudbees.hudson.plugins.folder.properties.FolderCredentialsProvider_-FolderCredentialsProperty>
    <jenkins.branch.NoTriggerOrganizationFolderProperty>
      <branches>.*</branches>
      <strategy>NONE</strategy>
    </jenkins.branch.NoTriggerOrganizationFolderProperty>
  </properties>
  <folderViews class="jenkins.branch.OrganizationFolderViewHolder">
    <owner reference="../.."/>
  </folderViews>
  <healthMetrics/>
  <icon class="jenkins.branch.MetadataActionFolderIcon">
    <owner class="jenkins.branch.OrganizationFolder" reference="../.."/>
  </icon>
  <orphanedItemStrategy class="com.cloudbees.hudson.plugins.folder.computed.DefaultOrphanedItemStrategy" plugin="cloudbees-folder@6.773.vd2dcc704ee7e">
    <pruneDeadBranches>true</pruneDeadBranches>
    <daysToKeep>-1</daysToKeep>
    <numToKeep>-1</numToKeep>
    <abortBuilds>false</abortBuilds>
  </orphanedItemStrategy>
  <triggers>
    <com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger plugin="cloudbees-folder@6.773.vd2dcc704ee7e">
      <spec>H H/4 * * *</spec>
      <interval>86400000</interval>
    </com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger>
  </triggers>
  <disabled>false</disabled>
  <navigators>
    <com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator plugin="cloudbees-bitbucket-branch-source@791.vb_eea_a_476405b">
      <serverUrl>https://<entfernt>/bitbucket</serverUrl>
      <credentialsId>36c31674-7dc1-4af2-9447-677b5393c8a1</credentialsId>
      <repoOwner>PROJECT</repoOwner>
      <traits>
        <jenkins.scm.impl.trait.RegexSCMSourceFilterTrait plugin="scm-api@621.vda_a_b_055e58f7">
          <regex>(config_dev|config_test)</regex>
        </jenkins.scm.impl.trait.RegexSCMSourceFilterTrait>
        <com.cloudbees.jenkins.plugins.bitbucket.BranchDiscoveryTrait>
          <strategyId>1</strategyId>
        </com.cloudbees.jenkins.plugins.bitbucket.BranchDiscoveryTrait>
        <com.cloudbees.jenkins.plugins.bitbucket.OriginPullRequestDiscoveryTrait>
          <strategyId>1</strategyId>
        </com.cloudbees.jenkins.plugins.bitbucket.OriginPullRequestDiscoveryTrait>
        <com.cloudbees.jenkins.plugins.bitbucket.ForkPullRequestDiscoveryTrait>
          <strategyId>1</strategyId>
          <trust class="com.cloudbees.jenkins.plugins.bitbucket.ForkPullRequestDiscoveryTrait$TrustTeamForks"/>
        </com.cloudbees.jenkins.plugins.bitbucket.ForkPullRequestDiscoveryTrait>
      </traits>
    </com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator>
  </navigators>
  <projectFactories>
    <org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProjectFactory plugin="workflow-multibranch@716.vc692a_e52371b_">
      <scriptPath>Jenkinsfile</scriptPath>
    </org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProjectFactory>
  </projectFactories>
  <buildStrategies>
    <jenkins.branch.buildstrategies.basic.ChangeRequestBuildStrategyImpl plugin="basic-branch-build-strategies@71.vc1421f89888e">
      <ignoreTargetOnlyChanges>true</ignoreTargetOnlyChanges>
      <ignoreUntrustedChanges>false</ignoreUntrustedChanges>
    </jenkins.branch.buildstrategies.basic.ChangeRequestBuildStrategyImpl>
    <jenkins.branch.buildstrategies.basic.NamedBranchBuildStrategyImpl plugin="basic-branch-build-strategies@71.vc1421f89888e">
      <filters>
        <jenkins.branch.buildstrategies.basic.NamedBranchBuildStrategyImpl_-RegexNameFilter>
          <regex>^.*$</regex>
          <caseSensitive>false</caseSensitive>
        </jenkins.branch.buildstrategies.basic.NamedBranchBuildStrategyImpl_-RegexNameFilter>
      </filters>
    </jenkins.branch.buildstrategies.basic.NamedBranchBuildStrategyImpl>
  </buildStrategies>
  <strategy class="jenkins.branch.DefaultBranchPropertyStrategy">
    <properties class="empty-list"/>
  </strategy>
</jenkins.branch.OrganizationFolder>

[sha-root]

Hi.

I confirm a similar problem. I configured additional "ref specs" for git fetch - to use in "git diff <source_branch>...<target_branch>" for generating of FILES_CHANGELIST in first PR build when changeset is empty.
And it stoped work after 866.vdea_7dcd3008e. I downgraded plugin versions to 866.vdea_7dcd3008e and it has been working again.

Before updating to the latest versions:

`using GIT_SSH to set credentials

git fetch --no-tags --force --progress -- git@bitbucket.org:ORG/PROJECT.git +refs/heads/SOURCEBRANCH:refs/remotes/origin/PR-0001 +refs/heads/:refs/remotes/origin/ # timeout=10`

After plugin updating to 866 higher version, it removes fetching of additional ref specs:

`using GIT_SSH to set credentials

git fetch --no-tags --force --progress -- git@bitbucket.org:ORG/PROJECT.git +refs/heads/feature/SOURCEBRANCH:refs/remotes/origin/feature/SOURCEBRANCH # timeout=10`

And I get an error in my diff command

git diff --name-only origin/develop...origin/feature/SOURCEBRANCH fatal: ambiguous argument 'origin/develop...origin/feature/SOURCEBRANCH': unknown revision or path not in the working tree.

Additional Ref Specs are set through git-plugin behaviour "Specify ref specs"

[kenden]

@andrey-fomin We use the default strategy also.

See the config.xml of a job with the issue

<?xml version='1.1' encoding='UTF-8'?>
<org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject plugin="workflow-multibranch@756.v891d88f2cd46">
  <actions/>
  <description>Build pipeline for project redacted</description>
  <displayName>job-name-redacted</displayName>
  <properties/>
  <folderViews class="jenkins.branch.MultiBranchProjectViewHolder" plugin="branch-api@2.1128.v717130d4f816">
    <owner class="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject" reference="../.."/>
  </folderViews>
  <healthMetrics>
    <com.cloudbees.hudson.plugins.folder.health.WorstChildHealthMetric plugin="cloudbees-folder@6.858.v898218f3609d">
      <nonRecursive>false</nonRecursive>
    </com.cloudbees.hudson.plugins.folder.health.WorstChildHealthMetric>
  </healthMetrics>
  <icon class="jenkins.branch.MetadataActionFolderIcon" plugin="branch-api@2.1128.v717130d4f816">
    <owner class="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject" reference="../.."/>
  </icon>
  <orphanedItemStrategy class="com.cloudbees.hudson.plugins.folder.computed.DefaultOrphanedItemStrategy" plugin="cloudbees-folder@6.858.v898218f3609d">
    <pruneDeadBranches>true</pruneDeadBranches>
    <daysToKeep>5</daysToKeep>
    <numToKeep>-1</numToKeep>
    <abortBuilds>false</abortBuilds>
  </orphanedItemStrategy>
  <triggers>
    <com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger plugin="cloudbees-folder@6.858.v898218f3609d">
      <spec>H H/4 * * *</spec>
      <interval>86400000</interval>
    </com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger>
  </triggers>
  <disabled>false</disabled>
  <sources class="jenkins.branch.MultiBranchProject$BranchSourceList" plugin="branch-api@2.1128.v717130d4f816">
    <data>
      <jenkins.branch.BranchSource>
        <source class="com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource" plugin="cloudbees-bitbucket-branch-source@848.v42c6a_317eda_e">
          <id>name_redacted</id>
          <serverUrl>https://bitbucket.companyname.com</serverUrl>
          <credentialsId>service_account_credentials</credentialsId>
          <repoOwner>REDACTED</repoOwner>
          <repository>repo_name_redacted</repository>
          <traits>
            <com.cloudbees.jenkins.plugins.bitbucket.BranchDiscoveryTrait>
              <strategyId>1</strategyId>
            </com.cloudbees.jenkins.plugins.bitbucket.BranchDiscoveryTrait>
            <com.cloudbees.jenkins.plugins.bitbucket.ForkPullRequestDiscoveryTrait>
              <strategyId>1</strategyId>
              <trust class="com.cloudbees.jenkins.plugins.bitbucket.ForkPullRequestDiscoveryTrait$TrustTeamForks"/>
            </com.cloudbees.jenkins.plugins.bitbucket.ForkPullRequestDiscoveryTrait>
            <com.cloudbees.jenkins.plugins.bitbucket.OriginPullRequestDiscoveryTrait>
              <strategyId>1</strategyId>
            </com.cloudbees.jenkins.plugins.bitbucket.OriginPullRequestDiscoveryTrait>
            <jenkins.plugins.git.traits.WipeWorkspaceTrait plugin="git@5.2.0"/>
            <jenkins.plugins.git.traits.CloneOptionTrait plugin="git@5.2.0">
              <extension class="hudson.plugins.git.extensions.impl.CloneOption">
                <shallow>false</shallow>
                <noTags>false</noTags>
                <honorRefspec>false</honorRefspec>
              </extension>
            </jenkins.plugins.git.traits.CloneOptionTrait>
            <com.cloudbees.jenkins.plugins.bitbucket.SSHCheckoutTrait>
              <credentialsId>service_account_ssh_key</credentialsId>
            </com.cloudbees.jenkins.plugins.bitbucket.SSHCheckoutTrait>
          </traits>
        </source>
        <strategy class="jenkins.branch.DefaultBranchPropertyStrategy">
          <properties class="java.util.Arrays$ArrayList">
            <a class="jenkins.branch.BranchProperty-array">
              <jenkins.branch.BuildRetentionBranchProperty>
                <buildDiscarder class="hudson.tasks.LogRotator">
                  <daysToKeep>-1</daysToKeep>
                  <numToKeep>10</numToKeep>
                  <artifactDaysToKeep>-1</artifactDaysToKeep>
                  <artifactNumToKeep>-1</artifactNumToKeep>
                </buildDiscarder>
              </jenkins.branch.BuildRetentionBranchProperty>
            </a>
          </properties>
        </strategy>
      </jenkins.branch.BranchSource>
    </data>
    <owner class="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject" reference="../.."/>
  </sources>
  <factory class="org.jenkinsci.plugins.workflow.multibranch.WorkflowBranchProjectFactory">
    <owner class="org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject" reference="../.."/>
    <scriptPath>Jenkinsfile</scriptPath>
  </factory>
</org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject>

[andrey-fomin]

@sha-root I believe your problem should be solved by #815. Please can you check release 876.v857269a_5f439

@kenden I don't understand why pattern +refs/heads/*:refs/remotes/origin/* is used for cloning. I don't see RefSpecsSCMSourceTrait in your configuration.

The only thing I can assume that your builds are executed in different folders (for example, multiple worker nodes or parallel builds). If branch was overwritten with forced push then commit from previous build can be missing in the current build folder. Fetching all branches with +refs/heads/*:refs/remotes/origin/* can increase the probability that commit from the previous build is available in the current build folder. But it can't guarantee it. If commit from previous build is missing then change log will be empty and message First time build. Skipping changelog. should be in the output.

@kenden do you use force pushes? Please can you try again with 876.v857269a_5f439?

[sha-root]

@andrey-fomin
Thanks! 🚀
Release 876.v857269a_5f439 has fixed the problem with additional Ref Specs

[kenden]

@andrey-fomin Thanks.
We updated to 876.v857269a_5f439. We still don't get: +refs/pull-requests/XX/from:refs/remotes/origin/PR-XX like we used to:

git fetch --tags --force --progress -- ssh://git@bitbucket.redactedserver.com/repo_example.git +refs/pull-requests/38/from:refs/remotes/origin/PR-38

We allow force pushed for pull requests. But we protect the base branch (main).

But made changed to our scripting to not depend on it. So this is not a problem for us anymore.