Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Testing Framework #1951

Open
wants to merge 156 commits into
base: develop
Choose a base branch
from
Open

New Testing Framework #1951

wants to merge 156 commits into from

Conversation

islas
Copy link
Collaborator

@islas islas commented Dec 19, 2023

TYPE: enhancement

KEYWORDS: testing, regression, test framework

SOURCE: internal

DESCRIPTION OF CHANGES:
Problem:
The current regression suite code is complex, requires maintenance of multiple alternate repositories, and takes involved effort to add a new test making community contribution limited at best. Likewise, the complexity of the system reduces the likelihood of independent local testing of changes, leading to a development cycle of one-off commits done to reinvoke testing to see if meaningful commits fix the issues.

Solution:
This new proposed regression suite addresses these shortcomings in a number of discrete ways:

  1. Modularize the testing framework to an generalized independent repo usable by any repo seeking to set up tests that can run locally, on HPC systems, and within any CI/CD framework
  2. Write WRF-specific test scripts inside the WRF repo and in a manner that does not rely on specific layouts/hardware/etc. so long as WRF can compile and run on intended system (i.e. able to be run locally)
  3. Write CI/CD tests in a simple and generally CI/CD framework-agnostic method where definitions of these also reside within the WRF repo
  4. Utilize HPC resources in a safe manner to increase breadth of testing to allow testing of many more compilers and on similar hardware to the general use case of WRF

LIST OF MODIFIED FILES:
A .ci/env/cheyenne.sh
A .ci/env/derecho.sh
A .ci/env/helpers.sh
A .ci/env/hostenv.sh
A .ci/env/wsl.sh
A .ci/missingTests.md
A .ci/tests/SCRIPTS
A .ci/tests/build.sh
A .ci/tests/runNamelists.sh
A .ci/wrf_arw_tests.json
A .ci/wrf_chem_tests.json
A .ci/wrf_em_ideal_tests.json
A .ci/wrf_em_real_tests.json
A .github/workflows/ci.yml
M .gitignore
M .gitmodules
M arch/postamble
M configure
M external/io_netcdf/diffwrf.F90
A hpc-workflows
M main/depend.common

TESTS CONDUCTED:

  1. All tests were conducted on cheyenne and derecho to replicate as close as possible a one-to-one matching of the current regression suite

RELEASE NOTE:
Introduce a modularized testing framework that allows testing locally and natively on HPC systems that lives within the WRF repository

@islas islas requested review from a team as code owners December 19, 2023 20:29
@islas
Copy link
Collaborator Author

islas commented Dec 19, 2023

Modifications to configure are to be superseded by #1743
Modifications to main/depend.common are to be superseded by #1950

@islas islas changed the title Test workflow New Testing Framework Dec 19, 2023
@weiwangncar
Copy link
Collaborator

The regression test results:

Test Type              | Expected  | Received |  Failed
= = = = = = = = = = = = = = = = = = = = = = = =  = = = =
Number of Tests        : 23           24
Number of Builds       : 60           57
Number of Simulations  : 158           150        0
Number of Comparisons  : 95           86        0

Failed Simulations are: 
None
Which comparisons are not bit-for-bit: 
None

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants