Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove phase0~capella state transition tests from testgen #3719

Open
wants to merge 4 commits into
base: dev
Choose a base branch
from

Conversation

hwwhww
Copy link
Contributor

@hwwhww hwwhww commented Apr 24, 2024

Address #3634

Remove phase0-capella state transition tests:

  • epoch_processing
  • finality
  • fork_choice
  • forks
  • genesis
  • light_client
  • operations
  • random
  • rewards
  • sanity
  • sync
  • transition

Note:

  1. The tests written in phase0 folder will still be included in the Deneb test vectors.
  2. We will still run tests on CI with minimal presets.

@ralexstokes
Copy link
Member

the idea is generally that these test vectors don't change, esp once we have gotten to a particular fork (e.g. phase0-capella test data doesn't change with respect to deneb)

and so we want to drop this test generation from the produced test corpus -- so that it becomes smaller

I think this is fine -- one thing to not is that assuming the test outputs don't change, then you can go to a previous release of the spec tests and get the data still; so this just changes the way we handle the data, and doesn't change what test data is available

it sounds like your intention is no change to how it currently works for deneb, and in either case we will still run all tests in CI (w/ minimal preset)

one thing I would also suggest is that we just ensure we do run the "full" test suite once per spec release, just so we can ensure the tests all pass -- and then under this PR we would just skip generating the test artifacts

Copy link
Member

@ralexstokes ralexstokes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice work! I think just touching the generators instead of anything related to the tests themselves (e.g. how the decorators work) is the right way to do this, as it makes it clear no safety properties of the spec (testing) itself changed, and just changes how the tests are generated

the "old" tests should not change from release to release and so this makes the test gen much lighter and we can point to previous spec test releases if consumers want that data

@hwwhww hwwhww mentioned this pull request Apr 25, 2024
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants