Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

teuthology/suite: initialize lua prng using run's seed #1926

Merged
merged 1 commit into from May 1, 2024
Merged

teuthology/suite: initialize lua prng using run's seed #1926

merged 1 commit into from May 1, 2024

Conversation

batrick
Copy link
Member

@batrick batrick commented Mar 19, 2024 •
edited

When a script may use Lua's prng, we want it to produce the same sequence during a rerun.

Looks like:

2024-03-19 16:33:08,391.391 INFO:teuthology.suite:Using random seed=8131
2024-03-19 16:33:08,392.392 INFO:teuthology.suite.run:kernel sha1: distro
2024-03-19 16:33:08,716.716 DEBUG:teuthology.repo_utils:git ls-remote https://git.ceph.com/ceph-ci.git wip-batrick-testing-20240318.181317 -> b6ae24918b03b1b8a19c9263a50655bba80bb810
2024-03-19 16:33:08,716.716 INFO:teuthology.suite.run:ceph sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810
2024-03-19 16:33:08,717.717 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810
2024-03-19 16:33:09,010.010 DEBUG:teuthology.packaging:looking for centos/8 x86_64 default
2024-03-19 16:33:09,010.010 DEBUG:teuthology.packaging:build: centos/8 arm64 default
2024-03-19 16:33:09,010.010 DEBUG:teuthology.packaging:build: centos/8 x86_64 crimson
2024-03-19 16:33:09,011.011 DEBUG:teuthology.packaging:build: centos/9 x86_64 default
2024-03-19 16:33:09,011.011 DEBUG:teuthology.packaging:build: centos/9 x86_64 crimson
2024-03-19 16:33:09,011.011 DEBUG:teuthology.packaging:build: ubuntu/22.04 x86_64 default
2024-03-19 16:33:09,011.011 DEBUG:teuthology.packaging:build: ubuntu/20.04 x86_64 default
2024-03-19 16:33:09,011.011 DEBUG:teuthology.packaging:build: centos/8 x86_64 default
2024-03-19 16:33:09,014.014 INFO:teuthology.suite.run:ceph version: 19.0.0-2206.gb6ae2491
2024-03-19 16:33:09,142.142 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/batrick/ceph wip-batrick-testing-20240318.181317 -> a76be9e33cc5b22f5135f29be5c73cd2ad978d87
2024-03-19 16:33:09,142.142 INFO:teuthology.suite.run:ceph branch: wip-batrick-testing-20240318.181317 a76be9e33cc5b22f5135f29be5c73cd2ad978d87
2024-03-19 16:33:09,144.144 DEBUG:teuthology.repo_utils:Resetting repo at /cephfs/home/pdonnell/src/github.com_batrick_ceph_wip-batrick-testing-20240318.181317 to origin/wip-batrick-testing-20240318.181317
2024-03-19 16:33:09,393.393 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/teuthology lua-prng -> af04998bf273ab00c53c2ed79d93564778800149
2024-03-19 16:33:09,393.393 INFO:teuthology.suite.run:teuthology branch: lua-prng af04998bf273ab00c53c2ed79d93564778800149
2024-03-19 16:33:09,417.417 DEBUG:teuthology.suite.run:Suite fs in /cephfs/home/pdonnell/src/github.com_batrick_ceph_wip-batrick-testing-20240318.181317/qa/suites/fs
2024-03-19 16:33:09,417.417 DEBUG:teuthology.suite.run:subset = (483, 512)
2024-03-19 16:33:09,418.418 DEBUG:teuthology.suite.run:no_nested_subset = False
2024-03-19 16:33:09,418.418 INFO:teuthology.suite.build_matrix:Subset=483/512
2024-03-19 16:33:09,742.742 INFO:teuthology.suite.run:Suite fs in /cephfs/home/pdonnell/src/github.com_batrick_ceph_wip-batrick-testing-20240318.181317/qa/suites/fs generated 265 jobs (not yet filtered or merged)
2024-03-19 16:33:09,742.742 DEBUG:teuthology.suite.merge:configuring Lua randomseed to 8131

@batrick batrick mentioned this pull request Mar 19, 2024
Merged
14 tasks
@batrick batrick force-pushed the lua-prng branch 3 times, most recently from cddd8b7 to af04998 Compare March 19, 2024 16:31
@batrick batrick requested a review from zmc March 21, 2024 01:15
@batrick
Copy link
Member Author

batrick commented Mar 21, 2024

@zmc here's a run I did with this:

https://pulpito.ceph.com/?branch=wip-batrick-testing-20240320.145742

and here's a ceph dependency:

https://github.com/ceph/ceph/pull/56293/files#diff-fa652d3fa199c184ebad839aa71892f6de074f08adc97f2dc00831c963f49a9c

(I think this is pretty cool!)

I don't think the jenkins failures are related. can you confirm?

@zmc
Copy link
Member

zmc commented Mar 22, 2024

@batrick this is cool! I do see a legitimate CI failure:
E TypeError: %d format: a number is required, not NoneType

@batrick
teuthology/suite: initialize lua prng using run's seed
d3eebb9 
When a script may use Lua's prng, we want it to produce the same sequence
during a rerun.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
@batrick
Copy link
Member Author

batrick commented Mar 23, 2024

@zmc thanks, I think I fixed it to make jenkins happy but it's pretty wonky.

@kamoltat kamoltat self-requested a review May 1, 2024 14:45
kamoltat
kamoltat approved these changes May 1, 2024
View reviewed changes
Copy link
Member

@kamoltat kamoltat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@batrick thanks for making this change, logic seems correct to me and the all CI passed.

@kamoltat
Copy link
Member

kamoltat commented May 1, 2024 •
edited

Tested branch this on my teuthology clone:

./virtualenv/bin/teuthology-suite -v --ceph wip-ksirivad-fix-64802 --ceph-repo https://github.com/ceph/ceph-ci.git --suite-repo https://github.com/kamoltat/ceph.git --machine-type smithi --rerun ksirivad-2024-04-18_21:18:05-rados-wip-ksirivad-fix-64802-distro-default-smithi --rerun-statuses pass --priority 50 --limit 1 --num 15 --force-priority --dry-run
2024-05-01 15:06:31,470.470 INFO:teuthology.suite:Using stored seed=6409
2024-05-01 15:06:31,470.470 INFO:teuthology.suite:Using stored subset=(90000, 100000)
2024-05-01 15:06:31,471.471 INFO:teuthology.suite.run:kernel sha1: distro
2024-05-01 15:06:31,667.667 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci.git wip-ksirivad-fix-64802 -> 2ee77fb1b2bba60efcad9e9d8006801fcbdda379
2024-05-01 15:06:31,668.668 INFO:teuthology.suite.run:ceph sha1: 2ee77fb1b2bba60efcad9e9d8006801fcbdda379
2024-05-01 15:06:31,668.668 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=2ee77fb1b2bba60efcad9e9d8006801fcbdda379
2024-05-01 15:06:31,984.984 DEBUG:teuthology.packaging:looking for centos/8 x86_64 default
2024-05-01 15:06:31,985.985 DEBUG:teuthology.packaging:build: centos/8 arm64 default
2024-05-01 15:06:31,985.985 DEBUG:teuthology.packaging:build: centos/8 x86_64 crimson
2024-05-01 15:06:31,985.985 DEBUG:teuthology.packaging:build: centos/9 arm64 default
2024-05-01 15:06:31,985.985 DEBUG:teuthology.packaging:build: centos/8 x86_64 default
2024-05-01 15:06:31,988.988 INFO:teuthology.suite.run:ceph version: 19.0.0-3117.g2ee77fb1
2024-05-01 15:06:32,117.117 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/kamoltat/ceph.git wip-ksirivad-fix-64802 -> 6aac7e010fa9758fd9517c037a33d3db5ee87e5a
2024-05-01 15:06:32,117.117 INFO:teuthology.suite.run:ceph branch: wip-ksirivad-fix-64802 6aac7e010fa9758fd9517c037a33d3db5ee87e5a
2024-05-01 15:06:32,123.123 DEBUG:teuthology.repo_utils:Setting repo remote to https://github.com/kamoltat/ceph.git
2024-05-01 15:06:32,131.131 INFO:teuthology.repo_utils:Fetching github.com_kamoltat_ceph_wip-ksirivad-fix-64802 from origin
2024-05-01 15:06:32,461.461 DEBUG:teuthology.repo_utils:Resetting repo at /cephfs/home/ksirivad/src/github.com_kamoltat_ceph_wip-ksirivad-fix-64802 to origin/wip-ksirivad-fix-64802
2024-05-01 15:06:32,585.585 DEBUG:teuthology.suite.run:Check file /cephfs/home/ksirivad/src/github.com_kamoltat_ceph_wip-ksirivad-fix-64802/qa/.teuthology_branch exists
2024-05-01 15:06:32,766.766 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/teuthology main -> 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17
2024-05-01 15:06:32,767.767 INFO:teuthology.suite.run:teuthology branch: main 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17
2024-05-01 15:06:32,803.803 DEBUG:teuthology.suite.run:Suite rados in /cephfs/home/ksirivad/src/github.com_kamoltat_ceph_wip-ksirivad-fix-64802/qa/suites/rados
2024-05-01 15:06:32,803.803 DEBUG:teuthology.suite.run:subset = (90000, 100000)
2024-05-01 15:06:32,803.803 DEBUG:teuthology.suite.run:no_nested_subset = None
2024-05-01 15:06:32,803.803 INFO:teuthology.suite.build_matrix:Subset=90000/100000
2024-05-01 15:07:31,113.113 INFO:teuthology.suite.run:Suite rados in /cephfs/home/ksirivad/src/github.com_kamoltat_ceph_wip-ksirivad-fix-64802/qa/suites/rados generated 314 jobs (not yet filtered or merged)
2024-05-01 15:07:31,114.114 DEBUG:teuthology.suite.merge:configuring Lua randomseed to 6409
2024-05-01 15:07:31,114.114 DEBUG:teuthology.suite.merge:merging config cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_repos}
2024-05-01 15:07:31,118.118 DEBUG:teuthology.suite.merge:postmerge script running:

Works fine for me

@batrick batrick merged commit 1ae7ad8 into ceph:main May 1, 2024
8 checks passed
@batrick batrick deleted the lua-prng branch May 1, 2024 15:09
@batrick batrick restored the lua-prng branch May 1, 2024 15:09
@batrick batrick deleted the lua-prng branch May 1, 2024 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kamoltat kamoltat kamoltat approved these changes

@zmc zmc Awaiting requested review from zmc

Assignees

@batrick batrick

Labels
improvement
Projects
None yet
Milestone
No milestone
3 participants
@batrick @zmc @kamoltat