Rethink CCPP framework development cycle and branch structure

[peverwhee]

Summary

Now that feature/capgen has been merged into main, we have the opportunity (and perhaps necessity) to change the development cycle to:

1. Speed up the development process;
2. Ensure changes do not break UFS or CAM-SIMA (or any future host model who wants to use the framework and is willing to put in the resources to consistently test it); and
3. Make the framework more accessible to other potential partners/users.

NCAR Wish List

1. Tags for PRs merged into main and proposed development branch; this ensures that older versions of CAM-SIMA will be stable and new updates can be fully tested before being integrated.
2. Some way to get quick bug fixes into the framework’s (proposed) development branch if necessary (possibly skirting the full UFS or CAM-SIMA testing cycle on occasion if deemed necessary)
3. No long-lived branches dedicated to specific host applications, due to concerns that this may cause the framework to dis-unify as hosts add features that work for that particular host model but not the other CCPP-enabled host models.

Proposal

• New “development” branch that is merged into main on some regular schedule (during which time the full UFS, CAM-SIMA, etc. testing suite is run by the maintainers of those models)
  • The development branch will still require unit tests and potentially a subset of host model regression tests to pass before accepting a PR, to ensure no major failure occurs while still allowing for a relatively rapid response to PRs.
• Framework development branch is tagged when minimal testing completed
  • Host models may use development tags, but should be aware that full testing has not been completed
• Framework main branch is tagged when all host model and UFS testing completed
  • Host model will use release tags whenever possible

“Regular” workflow:

1. New framework feature developed (“feature/new” in diagram)
2. feature/new merged into framework development branch (passed minimal testing)
3. Tag (ccpp_dev_tag_3 in diagram) made for new feature on development branch
4. Tag tested in host model development branch
5. Testing identified bug; bugfix branch created for CCPP-framework
6. Bugfix merged into framework development branch (passed minimal testing)
7. Tag (ccpp_dev_tag_n in diagram) made for bugfix
8. Bugfix tag tested in host model development branch
9. Host model A testing passed
10. Host model A can use "ccpp_dev_tag_n"
11. Framework development merged into main after complete testing suite performed - NCAR and UFS testing (merged at regular cadence)
12. Framework release tag made
13. Host model release branch made
14. End result: host model release branch points to framework release tag

[climbfuji]

I like this. The development branch should be named develop following standard practices.

[dustinswales]

This all seems sensible to me, but isn't the governance of development outside of "main" none of our business?

[peverwhee]

After further discussion at the framework meeting (see Meeting Minutes for details), we have three options:

• Some version of what is proposed in this issue: a shared "develop" branch that is merged into "main" with some regular cadence
• Separate develop branches for each host model
• Status quo - main branch only

In an effort to keep track of people's ideas, I've started a pro/con chart here. I have populated it with themes and ideas I pulled from our meeting, but feel free to edit with your thoughts. I will post a screenshot of the final list here in this issue when we're "done" for record-keeping; here's what it looks like now:

[mkavulich]

@peverwhee Thanks again for listing this all out and organizing the pros and cons. After some time to digest the topic, I've come up with some (mildly rambly) thoughts:

Off the bat, I am leaning towards solution 2 (separate branches for each host model). Specifically, a modified proposition that would use forked repositories rather than simple branches within the main repository: this would bring ccpp-framework more in line with how ccpp-physics is handled, and would allow CESM/SIMA and other individual projects to work at their own development pace and only take contributions from the main branch/repository when they are ready.

My reason for wanting forks is that this will allow for issues and PRs specific to each dycore to be appropriately separated so that A. Issues and PRs will be seen more easily by the people most able to fix them, and B. The main repository will remain less cluttered and easier to maintain.

In order for this to actually work, we would need buy-in from all groups maintaining forks that

1. They will follow all CCPP standards in their fork as defined in the CCPP Technical Documentation
2. They will run all tests from the main branch on their own fork on a regular basis (preferably these would pass for all changes) and
3. They will offer in-kind help to ccpp-framework code managers for any trouble merging changes to the main branch, and make reasonable efforts to adopt any requested changes from ccpp-framework code managers.

As a counterpoint to the testing requirements from the main branch, groups with forks would also have the right to insist on any tests they require being run on all development to the main branch. In fact, the more automated testing at more steps, the better!

As for what kind of cadence to merge changes back into the main repository, I don't really know what makes sense.

[peverwhee]

Updated pro/con list with contributions from @mkavulich

[nusbaume]

Hi @mkavulich et al., after discussion from the group here at NCAR/CGD, we just wanted to share our overall thoughts:

• As mentioned in previous meetings, our biggest fear is needing a critical bug fix for some release version of a host model, and then having that be postponed for weeks or months because the PR is waiting on testing from all of the participating host models. Right now, the UFS is the only host model doing offline testing that could hold up a PR, but in the future a PR could be held up by CAM-SIMA testing as well.

  One way to avoid this problem is to simply ensure that all testing of the main branch is done within a certain time period (which is easily done if everything is automated), or, if that isn't doable, that PRs can occasionally be labeled "urgent" and fast-tracked within each host model's testing workflow.

  However, if speeding up testing (or at least ensuring a hard timeline) isn't feasible, then we'll basically have to have more than one branch, with main still being the branch that has the most rigorous testing (i.e. every participating host institution signs off).
  It should also be noted that the overall testing burden will likely continuously increase as more host models add their own required tests to the main branch, so this may become a problem in the future even if we greatly improve our efficiency for the current set of tests.

• Adding new branches/forks for each host model is certainly feasible, but our concern is that it will quickly lead to code divergence, especially if host models find that there isn't really a good reason to push certain features back to main (e.g. "This feature will only ever be used in our host model, so no need to push it back").

  Also I think it's important to note that the CCPP-framework arguably shouldn't follow the organization of CCPP-physics. Host models will always have particular science goals or objectives that may cause their physics schemes to be specific to a particular host institution (even if they are still inter-operable from a CCPP software standpoint). However, the goal of the framework is that all host models are using literally the exact same framework code. If we collectively end up using both different physics AND different frameworks then I think the core goal of interoperability will become almost impossible.

• On the other hand, having a development branch within the NCAR/CCPP-framework repo will both reduce the risk of code divergence and allow for hashes/tags to be created without necessarily waiting for testing from every host model (while still doing at least some minimal testing via the current automated testing system we have now).

  Having a development branch may occasionally cause noticeably difficult merges, especially if development is moving quickly, but to be honest the current workflow of only having main will have the exact same problem, as will having separate branches or forks whenever they are synced to main. In other words, I don't think there really is a way to avoid that particular problem.

Given this, assuming there is no way to significantly speed up testing, we are leaning towards having a shared development branch. Of course we are always happy to discuss this further if folks still have concerns. Thanks!