Nightly tests are run with outdated `/_assets/scripts/run_unit_tests.sh` #4993

osmaczko · 2024-03-26T17:22:34Z

Nightly tests are run with outdated /_assets/scripts/run_unit_tests.sh.

https://ci.status.im/job/status-go/job/tests-nightly/167/execution/node/72/log/

03:49:09  + nix-shell --show-trace --run 'make test-unit V=1' --impure --no-out-link --option sandbox relaxed /home/jenkins/workspace/status-go/tests-nightly/shell.nix
03:49:11  ./_assets/scripts/run_unit_tests.sh
03:49:12  �[0;32mTesting:�[0m github.com/status-im/status-go/abi-spec
...

In this certain build, g show 2aa306ef4:./_assets/scripts/run_unit_tests.sh script should also print the Iteration info, but it doesn't. Expected output is:

03:49:00  + nix-shell --show-trace --run 'make test-unit V=1' --impure --no-out-link --option sandbox relaxed /home/jenkins/workspace/status-go/tests-nightly/shell.nix
03:49:01  ./_assets/scripts/run_unit_tests.sh
03:49:03  �[0;32mTesting:�[0m github.com/status-im/status-go/abi-spec Iteration:1
03:49:03  �[0;32mTesting:�[0m github.com/status-im/status-go/abi-spec Iteration:2
03:49:03  �[0;32mTesting:�[0m github.com/status-im/status-go/abi-spec Iteration:3
03:49:03  �[0;32mTesting:�[0m github.com/status-im/status-go/abi-spec Iteration:4
03:49:03  �[0;32mTesting:�[0m github.com/status-im/status-go/abi-spec Iteration:5
...

The text was updated successfully, but these errors were encountered:

yakimant · 2024-03-27T10:13:38Z

I've rerun the same #167 build on another agent - linux-02 and it's fine:
https://ci.status.im/job/status-go/job/tests-nightly/168

Hm, it ran on a bit newer commit unfortunately - 30fe620ff07392b5b06a7562fd0a8c2b8c61a4b7

But newer build doesn't have these iterations too:
https://ci.status.im/job/status-go/job/tests-nightly/169/

Looks like an issue with linux-03. Will check the workdir.

yakimant · 2024-03-27T10:28:03Z

Workdir is /home/jenkins/workspace/status-go/tests-nightly

Iteration output was added in a068b64

yakimant · 2024-03-27T10:29:25Z

❯ ansible -i ansible/inventory/misc linux-03.he-eu-hel1.ci.devel,linux-02.he-eu-hel1.ci.devel -a 'grep Iteration /home/jenkins/workspace/status-go/tests-nightly/_assets/scripts/run_unit_tests.sh'
linux-03.he-eu-hel1.ci.devel | FAILED | rc=1 >>
non-zero return code
linux-02.he-eu-hel1.ci.devel | CHANGED | rc=0 >>
  echo -e "${GRN}Testing:${RST} ${package} Iteration:${iteration}"

yakimant · 2024-03-27T10:37:18Z

what the heck?

❯ ansible -i ansible/inventory/misc linux-03.he-eu-hel1.ci.devel,linux-02.he-eu-hel1.ci.devel -a 'sudo git --git-dir /home/jenkins/workspace/status-go/tests-nightly/.git log -1'
linux-03.he-eu-hel1.ci.devel | CHANGED | rc=0 >>
commit 01b3f8ace43a385f2e85d2c8885d314bc14ec77d
Author: frank <lovefree103@gmail.com>
Date:   Thu Feb 22 09:44:14 2024 +0800

    fix flaky test TestMarkMessagesSeenMarksNotificationsRead (#4781)

    * fix flaky test TestMarkMessagesSeenMarksNotificationsRead

    * address review feedback
linux-02.he-eu-hel1.ci.devel | CHANGED | rc=0 >>
commit 004bd4e79a783a32f5238a80ea8d490275596180
Author: Anthony Laibe <491074+alaibe@users.noreply.github.com>
Date:   Tue Mar 5 10:33:12 2024 +0100

    fix: gas estimation error cause network error

So they are both stuck in the past. linux-02 is just new enough to have a068b64

yakimant · 2024-03-27T10:49:08Z

linux-02 latest build:
https://ci.status.im/job/status-go/job/tests-nightly/168
should have revision 30fe620ff07392b5b06a7562fd0a8c2b8c61a4b7 (Mar 26)
in reality 004bd4e79a783a32f5238a80ea8d490275596180 (Mar 5)

linux-03 latest build:
https://ci.status.im/job/status-go/job/tests-nightly/169
should have revision 1a2880b3658f62f3019bd4a318de2aaf32b329d7 (Mar 26)
in reality 01b3f8ace43a385f2e85d2c8885d314bc14ec77d (Feb 22)

yakimant · 2024-03-27T10:51:01Z

Checkout log from linux-02:

The recommended git tool is: git
using credential jenkins-status-im-fine-grained
Fetching changes from the remote Git repository
Cleaning workspace
 > git rev-parse --resolve-git-dir /home/jenkins/workspace/status-go/tests-nightly/.git # timeout=10
 > git config remote.origin.url https://github.com/status-im/status-go # timeout=10
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Fetching upstream changes from https://github.com/status-im/status-go
 > git --version # timeout=10
 > git --version # 'git version 2.34.1'
using GIT_ASKPASS to set credentials IMPORTANT: GitHub fine-grained token for status-im-auto.
 > git fetch --tags --force --progress -- https://github.com/status-im/status-go +refs/heads/*:refs/remotes/origin/* # timeout=10
Checking out Revision 1a2880b3658f62f3019bd4a318de2aaf32b329d7 (origin/develop)
Commit message: "Fix/community tags indices (#4992)"
 > git rev-parse origin/develop^{commit} # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1a2880b3658f62f3019bd4a318de2aaf32b329d7 # timeout=10
 > git rev-list --no-walk 30fe620ff07392b5b06a7562fd0a8c2b8c61a4b7 # timeout=10

yakimant · 2024-03-27T11:02:54Z

Running these commands manually changed the commit succesfully.
(git checkout -f 30fe620ff07392b5b06a7562fd0a8c2b8c61a4b7)

❯ ansible -i ansible/inventory/misc linux-03.he-eu-hel1.ci.devel,linux-02.he-eu-hel1.ci.devel -a 'sudo git --git-dir /home/jenkins/workspace/status-go/tests-nightly/.git log -1'
linux-03.he-eu-hel1.ci.devel | CHANGED | rc=0 >>
commit 1a2880b3658f62f3019bd4a318de2aaf32b329d7
Author: Igor Sirotin <sirotin@status.im>
Date:   Tue Mar 26 20:02:12 2024 +0000

    Fix/community tags indices (#4992)

    * fix: strict order of community tags

    * make tags containers private

    * fix RandomCommunityTags implementation
linux-02.he-eu-hel1.ci.devel | CHANGED | rc=0 >>
commit 30fe620ff07392b5b06a7562fd0a8c2b8c61a4b7
Author: Patryk Osmaczko <osmaczkopatryk@gmail.com>
Date:   Tue Mar 26 17:00:17 2024 +0100

    chore: extend tests timeout

    Due to an increased volume of tests, numerous test jobs have been timing
    out. The timeout has been extended by 10 minutes.

❯ ansible -i ansible/inventory/misc linux-03.he-eu-hel1.ci.devel,linux-02.he-eu-hel1.ci.devel -a 'grep Iteration /home/jenkins/workspace/status-go/tests-nightly/_assets/scripts/run_unit_tests.sh'
linux-02.he-eu-hel1.ci.devel | CHANGED | rc=0 >>
  echo -e "${GRN}Testing:${RST} ${package} Iteration:${iteration}"
linux-03.he-eu-hel1.ci.devel | CHANGED | rc=0 >>
  echo -e "${GRN}Testing:${RST} ${package} Iteration:${iteration}"

yakimant · 2024-03-27T11:19:38Z

stdout/stdin of git commands is not printed unfortunately:
https://raw.githubusercontent.com/MarkEWaite/git-client-plugin/master/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java

yakimant · 2024-03-27T11:23:07Z

New build succesfully checked out, but maybe because I did fetch.
Need to wait for more commits, or next nightly to verify.

yakimant · 2024-03-27T11:26:38Z

Wow, interesting thing!

https://ci.status.im/job/status-go/job/tests-nightly/171/

changed from 1a2880b3658f62f3019bd4a318de2aaf32b329d7 to 98c3be55b937582374e60adb651543c5f1348c29
and then back to 1a2880b3658f62f3019bd4a318de2aaf32b329d7 during the build

yakimant · 2024-03-27T11:31:11Z

https://ci.status.im/job/status-go/job/tests-nightly/172

Wed Mar 27 11:27:48 UTC 2024
1a2880b3658f62f3019bd4a318de2aaf32b329d7
...
Wed Mar 27 11:28:06 UTC 2024
1a2880b3658f62f3019bd4a318de2aaf32b329d7
Wed Mar 27 11:28:07 UTC 2024
98c3be55b937582374e60adb651543c5f1348c29
...
98c3be55b937582374e60adb651543c5f1348c29
Wed Mar 27 11:28:38 UTC 2024
1a2880b3658f62f3019bd4a318de2aaf32b329d7
Wed Mar 27 11:28:39 UTC 2024
...

this one:

[2024-03-27T11:28:35.125Z] + nix-shell --show-trace --run 'make migration-check' --impure --no-out-link --option sandbox relaxed /home/jenkins/workspace/status-go/tests-nightly/shell.nix
[2024-03-27T11:28:37.544Z] bash _assets/scripts/migration_check.sh
[2024-03-27T11:28:37.544Z] HEAD is now at 98c3be55b feat: l1 gas price estimation when placing l2 transaction
[2024-03-27T11:28:37.544Z] From https://github.com/status-im/status-go
[2024-03-27T11:28:37.544Z]  * branch                develop    -> FETCH_HEAD
[2024-03-27T11:28:37.544Z] Already up to date.
[2024-03-27T11:28:37.544Z] Previous HEAD position was 98c3be55b feat: l1 gas price estimation when placing l2 transaction
[2024-03-27T11:28:37.544Z] HEAD is now at 1a2880b36 Fix/community tags indices (#4992)

yakimant · 2024-03-27T11:46:53Z

That's the code checking out incorrectly:

status-go/_assets/scripts/migration_check.sh

Lines 21 to 23 in 98c3be5

    
           git checkout origin/develop 
        
           git pull origin develop 
        
           git checkout -

yakimant · 2024-03-27T11:47:22Z

@andremedeiros, @osmaczko, please have a look

jakubgs · 2024-03-27T11:55:57Z

According to Andrea:

we need to check migrations against the main branch

But changing the locally checked out branch is the wrong way to do this.

We either need a separate checkout(slow) or use the ability of git checkout to checkout only specific files we need.

osmaczko · 2024-03-27T11:56:36Z

That's the code checking out incorrectly:

status-go/_assets/scripts/migration_check.sh

Lines 21 to 23 in 98c3be5

git checkout origin/develop

git pull origin develop

git checkout -

Oh, I see, nice catch @yakimant 🙏

osmaczko · 2024-03-27T12:05:09Z

According to Andrea:

we need to check migrations against the main branch

But changing the locally checked out branch is the wrong way to do this.

We either need a separate checkout(slow) or use the ability of git checkout to checkout only specific files we need.

Wouldn't git fetch origin develop be enough?

yakimant · 2024-03-27T12:06:28Z

Will the fix take a long time? Is it worth to disable this functionality meanwhile?

fixes: #4993

jakubgs · 2024-03-27T12:13:39Z

The fix is most probably just to use:

git fetch origin develop
git checkout origin/develop -- protocol/migrations/sqlite

So it doesn't change the branch, just checks out develop for those files.

jakubgs · 2024-03-27T12:16:10Z

And of course clean it up after the check is performed.

cammellos · 2024-03-27T12:48:06Z

git fetch origin develop

git fetch origin develop is what we had before,
but wasn't working (the check would pass even if it should fail), that's why we explicitly checkout and pulled (which fixed the issue)

git checkout origin/develop -- protocol/migrations/sqlite

Not sure that makes sense, we want to check against develop, so we don't need to checkout those files.

One thing I don't understand, 1) Why the nightly tests run migration checks (or do they?) or 2) Are we sharing workspaces and is PR impacting this?

If the nightly run is runing migration check, it should not as it doesn't make sense, if it's PR impacting this, then I guess is a different story. If anyone wants to try to see why git fetch origin develop didn't work, that's also great

mitigates: #4993

osmaczko assigned yakimant Mar 26, 2024

osmaczko added the E:Tests label Mar 26, 2024

osmaczko added a commit that referenced this issue Mar 27, 2024

fix: prevent migration_check checking-out git branch

1fa21b6

fixes: #4993

osmaczko mentioned this issue Mar 27, 2024

fix: prevent migration check on nightly tests #4995

Merged

osmaczko added a commit that referenced this issue Mar 27, 2024

fix: prevent migration_check checking-out local branch

86a1faf

fixes: #4993

osmaczko added a commit that referenced this issue Mar 27, 2024

fix: prevent migration check on nightly tests

da3b397

mitigates: #4993

osmaczko added a commit that referenced this issue Mar 28, 2024

fix: prevent migration check on nightly tests

9648e22

mitigates: #4993

osmaczko added a commit that referenced this issue Mar 28, 2024

fix: prevent migration check on nightly tests

c98acf7

mitigates: #4993

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nightly tests are run with outdated `/_assets/scripts/run_unit_tests.sh` #4993

Nightly tests are run with outdated `/_assets/scripts/run_unit_tests.sh` #4993

osmaczko commented Mar 26, 2024

yakimant commented Mar 27, 2024 •

edited

yakimant commented Mar 27, 2024

yakimant commented Mar 27, 2024

yakimant commented Mar 27, 2024 •

edited

yakimant commented Mar 27, 2024

yakimant commented Mar 27, 2024 •

edited

yakimant commented Mar 27, 2024 •

edited

yakimant commented Mar 27, 2024

yakimant commented Mar 27, 2024

yakimant commented Mar 27, 2024

yakimant commented Mar 27, 2024 •

edited

yakimant commented Mar 27, 2024 •

edited

yakimant commented Mar 27, 2024

jakubgs commented Mar 27, 2024

osmaczko commented Mar 27, 2024

osmaczko commented Mar 27, 2024

yakimant commented Mar 27, 2024

jakubgs commented Mar 27, 2024

jakubgs commented Mar 27, 2024

cammellos commented Mar 27, 2024

Nightly tests are run with outdated /_assets/scripts/run_unit_tests.sh #4993

Nightly tests are run with outdated /_assets/scripts/run_unit_tests.sh #4993

Comments

osmaczko commented Mar 26, 2024

yakimant commented Mar 27, 2024 • edited

yakimant commented Mar 27, 2024

yakimant commented Mar 27, 2024

yakimant commented Mar 27, 2024 • edited

yakimant commented Mar 27, 2024

yakimant commented Mar 27, 2024 • edited

yakimant commented Mar 27, 2024 • edited

yakimant commented Mar 27, 2024

yakimant commented Mar 27, 2024

yakimant commented Mar 27, 2024

yakimant commented Mar 27, 2024 • edited

yakimant commented Mar 27, 2024 • edited

yakimant commented Mar 27, 2024

jakubgs commented Mar 27, 2024

osmaczko commented Mar 27, 2024

osmaczko commented Mar 27, 2024

yakimant commented Mar 27, 2024

jakubgs commented Mar 27, 2024

jakubgs commented Mar 27, 2024

cammellos commented Mar 27, 2024

Nightly tests are run with outdated `/_assets/scripts/run_unit_tests.sh` #4993

Nightly tests are run with outdated `/_assets/scripts/run_unit_tests.sh` #4993

yakimant commented Mar 27, 2024 •

edited

yakimant commented Mar 27, 2024 •

edited

yakimant commented Mar 27, 2024 •

edited

yakimant commented Mar 27, 2024 •

edited

yakimant commented Mar 27, 2024 •

edited

yakimant commented Mar 27, 2024 •

edited