author: Pey Lian Lim, Leah Wasser, William Jamieson, Derek Homeier, Moritz Günther
date-created: 2023 July 13
date-last-revised: 2024 January 29
date-accepted: 2024 January 29
type: Process
status: Discussion
Since the early days of the Astropy Project, we had recognized the need to accommodate and develop an internal process for acknowledging Astropy Affiliated Packages (hereafter, for brevity, Affiliated) – packages that are developed outside of the core package, but interoperate with it and are extending its functionality. As our ecosystem thrives and grows, so does the wider scientific Python ecosystem, which resulted in similar processes like JOSS and pyOpenSci software peer review (hereafter, pyOpenSci). Between the two, while JOSS has existed longer, pyOpenSci is more closely aligned with what we already do with Astropy Affiliated Packages. This APE overhauls our existing Affiliated process to partner with pyOpenSci in order to share resources and gain wider exposure.
The Affiliated page has existed since 2013 (i.e, 245a237) though the concept could have been incepted even earlier. At that time it included 10 packages, 3 of which were highlighted as "featured" packages and 4 others considered still in (early) development. JOSS was established in 2016 (source: Wikipedia) but it was perceived as a different enough system that there was no effort to consolidate with them. Meanwhile, pyOpenSci was founded in 2018 with the peer review beginning in March 2019, partnership with the JOSS soon after, and with Pangeo in Fall 2022 using Sloan Foundation funding.
Fast forward to July 2023, there are 47 accepted Affiliated packages and 7 undergoing the review process, plus 10 that have been elevated to the status of Coordinated package. The latter provide crucial extension functionality and are maintained by Astropy team members to ensure their continued development and availability. We have two editors (Derek Homeier and William Jamieson as of July 2023) and recruit reviewers from the Astropy community. Editors identify community members with sufficient experience in the Astropy ecosystem and practices to review packages, usually asking reviewers with scientific domain experience related to a potentially affiliated package. This process mirrors what is commonly done for scientific journals, where editors invite scientific peers of a study to be reviewers. Given that reviewers are anonymous by default, we are unable to provide a fully transparent review process (even though results are publicly posted and discussed, they are posted second-hand by the editors).
Unlike scientific articles, which are static after publication, software continues to develop, adding features and fixing bugs; otherwise, it will become obsolete and might be incompatible with future versions of its dependencies (e.g., NumPy or Python). Thus, a process of "re-review" for Affiliated packages has been envisioned, but never executed because we have no resources nor infrastructure to re-review the 47 accepted packages to ensure that they still meet the Affiliated standards. Therefore, there is a need to restructure our Affiliated process to be sustainable. We are confident that a partnership with pyOpenSci is the best solution currently available.
- Larger exposure for Affiliated packages within the scientific Python ecosystem, not just Astropy.
- If they are in scope for JOSS, they can be fast-tracked through the JOSS review process and get a publication via the partnership between pyOpenSci and JOSS.
- pyOpenSci's objective way of initial triage will automatically weed out packages that do not even meet basic standards (e.g., not installable), reducing our editors' burdens.
- We can be sure the process is consistent with other projects that have also partnered with pyOpenSci.
- We can share infrastructure with pyOpenSci, reducing our own maintenance burden.
- We can share reviewer pool with pyOpenSci, providing our reviewers opportunity to connect with peers beyond Astropy.
- Even when a package ends up not being accepted into Affiliated but satisfies basic pyOpenSci criteria, its application effort is not entirely wasted, as it could still be listed under pyOpenSci (and JOSS) only without our Affiliated status.
pyOpenSci already has an established software peer review process and it is community-driven. It is compatible with Astropy but it is up to us to decide if a package reviewed and accepted by pyOpenSci also qualifies as Affiliated.
Yes, we think so! We will still keep the Astropy Affiliated Packages page, but now instead of hosting the content ourselves, we grab an RSS/XML feed from pyOpenSci for the packages marked as Affiliated. To keep the history, we will move the existing listing to a "legacy affiliation" page and keep it static.
Coordinated packages are separate from this due to their tighter integration with the Astropy project's team structure, so they will remain listed on this page without a connection to pyOpenSci.
pyOpenSci has its own funding structure beyond the control of Astropy. As of July 2023, it is fully funded for the next 2 years. Given the number of projects currently looking into a similar partnership with pyOpenSci (SunPy, PlasmaPy, pyHeliophysics, etc.) and that it has a full-time staff dedicated to seeking out more funding, we find that the benefits outweigh the risks.
| Step | pyOpenSci | Astropy (pre-APE 22) |
|---|---|---|
| Editorial infrastructure for a single review | Editor-in-Chief, editor (domain specific), 2 reviewers (1 can be non-domain specific) | Editor, reviewer |
| Editorial team | pyOpenSci Editorial Board | 2 editors that rotate, see Astropy team roles |
| Review style | Open review | Reviewer is anonymous by default |
| Post-review checks | We will be creating a set of checks to see if the package is still maintained and in what state. We might be able to work with Astropy (and other communities) on what this looks like. | Astropy has static badges that need to be manually updated. |
As of July 2023, Astropy had the following pre-APE 22 Affiliated Package review guidelines and followed this pre-APE 22 Affiliated Package review process, with templates.
Meanwhile, pyOpenSci has its own submission guide for package authors, accompanying GitHub issue templates, and listing of accepted packages. As of July 2023, they are in the process of revamping their process to handle collaborations with different communities (e.g., Astropy). They are also:
- creating a community-specific label for package filtering for that community on their website package listing;
- updating the submission template so a potential Astropy affiliated package can select Astropy for the actual review;
- adding an Astropy label on their GitHub repository; and
- creating a new editorial bot to add community-specific tags.
We do not think so! In the pre-APE 22 process, the reviewer was anonymous by default because we modeled the process after traditional astronomical journals. Even then, the reviewer had the option to reveal themselves if they wanted to. Given that the Astropy ecosystem is open-source anyway, it makes sense for us to move away from the journal-style tradition and embrace a more open process that pyOpenSci uses.
We still need them! However, instead of managing the whole process themselves, they will now be part of the pyOpenSci Editorial Board and perform their duties within the pyOpenSci process as laid out in this APE.
A package may be "rejected" in different stages of the review, either by the pyOpenSci Editor-in-Chief or the editor, including but not limited to causes such as:
- the package is not in technical or domain scope;
- the package fails one of the pre-review checks done by the Editor-in-Chief;
- the package maintainer stops responding to review comments; or
- the package maintainer is unwilling to ensure the package can be used and maintained.
While in practice, rejection is rare, a rejection does not have to come from the Astropy community directly and will be objective.
A package that fails the Astropy-specific criteria may still be accepted into the pyOpenSci ecosystem (and also published by JOSS) if it is in scope; an example of this might be a well-documented and developed Python package that is useful to the scientific community that fails to relate well to the Astropy ecosystem.
Note: This does not exist yet and probably will not happen before this APE is accepted. As of July 2023, pyOpenSci had a somewhat manual process to check in on an accepted package.
One core goal of pyOpenSci is to support scientific Python packages that are maintained over time. Due to the lack of resources, it does not do a full re-review of packages (involving Editor-in-Chief, editor, and 2 reviewers). However, it has plans to set up automated checks that track the "health" and maintenance level of a package over time, with the goal of identifying packages that have become "orphans" (i.e, unmaintained). Once a package is identified as needing additional maintenance, it will be flagged.
"Health" might include frequency of commits, releases, CI status, etc. Some, if not most, of this data will be collected using devstats and repo-review from Scientific Python. This information will be displayed on public dashboards hosted on the pyOpenSci website; we may also grab those same badges, where applicable, for a similar dashboard on Astropy website focused only in our Affiliated packages.
However, automation can only go so far; for instance, if automation is deployed in GitHub Actions, then any package hosted outside of GitHub would be excluded from these checks. In that case, manual intervention or alternate implementation might still be necessary.
If the package is no longer maintained, one of the following can happen:
- If it is a widely used package and the maintainer wants to see it live on, pyOpenSci will try to help the maintainer build a new maintainer team;
- Otherwise, pyOpenSci will gracefully sunset it from the list of maintained/accepted packages.
After sunsetting, if a package becomes active again, it is up to the package maintainer to contact Astropy or pyOpenSci in order to have it be actively listed once more. An example scenario that might happen is when the sole maintainer goes on a hiatus longer than the time frame set in maintainer responsiveness and then comes back to a sunsetted package.
pyOpenSci currently has a list of reviewers who have signed up for this task using the reviewer sign-up form. Because they utilize 2 reviewers for each package, they generally try to find a reviewer with domain-specific expertise, while the other with or without. Sometimes, the second reviewer will focus instead on general usability, ease of installation, documentation quality, or packaging infrastructure. For every review, they target a diversity of contributors to ensure that they have a mix of varying gender, cultural, etc., identities.
If we decide to partner with pyOpenSci, our reviewers would sign up using the pyOpenSci form above, specifying:
- astronomy domain expertise, and
- membership in the Astropy community.
pyOpenSci does not publicly list all the people who signed up (i.e, a reviewer is only public during the review process) but the pyOpenSci Editorial Board has access to the list.
pyOpenSci will respect the Astropy-specific criteria for someone to be a reviewer for Affiliated request, namely:
- familiarity with the Astropy project,
- ability to judge whether a package integrates well with the Astropy ecosystem (as per pre-APE 22 guidelines), and
- having domain expertise in the area of the package (e.g., galaxy evolution).
pyOpenSci and Astropy both ask editors/reviewers disclose any potential conflict of interest (COI) prior to agreeing to review a package. In the event where COI occurs, with this partnership, we would follow the pyOpenSci COI process. pyOpenSci invites the Astropy community to review this language to ensure it meets our needs.
Astropy currently does not enforce any concrete turnaround time. Reviews typically come within weeks, but the response time for submitted packages to address the review is very non-uniform from "within days" to "years".
If we partner with pyOpenSci, Astropy would need to adhere to their expected timeline for each step, as laid out in An Overview Of the Peer Review Process. For example, editor is expected to find reviewers within 2-3 weeks and a peer review should be completed within 3 weeks after that. This is to ensure that the package maintainers have a good experience with the review and things do not languish over a long period of time.
Generally, the editor role should not take a huge amount of time, but it is important for an editor, once the review starts, to check in on the review periodically (every few weeks and more often during wrap-up).
The Astropy community has followed packaging guidelines published in the OpenAstronomy packaging guide for a few years, and Astropy package template before that.
pyOpenSci is also developing a community-driven packaging guide that covers modern best practices and recommendations for scientific Python packaging. This guide has a stringent community review process. Reviewers for this guide represent members of PyPA, core Python, Anaconda (conda/conda-forge), and core packages for front- and back-end tools (e.g., flit, PDM, hatch/hatchling, etc.).
If we partner with pyOpenSci, we encourage our Affiliated packages to continue to follow the OpenAstronomy packaging guide for packaging structure, which ensures a packaging style consistent with the majority of existing Affiliated packages as well as much of the core scientific Python environment Astropy relies on, in particular Numpy and Scipy. However maintainers will find the pyOpenSci guide to be a useful reference, given it provides community-wide guidelines and is targeted to those newer to packaging (e.g., just creating the initial package for their code). Editors and reviewer shall not let a particular choice of packaging influence their rating of a package.
Regardless, the Editor-in-Chief checks are the bare minimum for a package being able to go through the pyOpenSci review process.
The pre-APE 22 Astropy Affiliated Packages listing are listed by alphabetical order with hard-coded badges. While these badges are color-coded to give a at-a-glance status of packages, they become outdated over time and now could even be misleading (i.e., they really are showing the status at the time of acceptance, not the current status). pyOpenSci is willing to work with Astropy and the broader scientific community to develop a more consistent standard of evaluating the "health state" of a package, using universally accepted metrics for such measurements (see "Re-review" of packages); therefore, getting rid of the need for static badges altogether.
pyOpenSci is willing to create the following specifically for Astropy if we agree to this partnership:
- A feed of Affiliated packages that we could ingest and use to rebuild our own Astropy Affiliated Packages listing. This way, the packages would be listed both over at pyOpenSci and at Astropy websites.
- An Astropy-specific page on the pyOpenSci website that would be a dedicated link for only Affiliated packages. We could then include some other information about the Astropy ecosystem and branding (logo, colors, etc). A listed package would also have a link back to the review that was done.
- A link back to Astropy website from pyOpenSci, with the understanding that Astropy would do vice versa. This is similar to what pyOpenSci has done with Pangeo.
Issues:
- Affiliated Package: Consolidate with pyOpenSci?
- Metrics for community partnership landing pages to track health (as noted above in "Re-review" of packages, this issue is not required to be resolved for this APE to be accepted)
Pull requests:
- Feat: add text associated with astropy partnership
- FEAT: Start at community mockup / astropy
- Add: more specific community partners page with FAQ
- TODO: A future PR by William Jamieson to update our Affiliated guidelines and process, after consolidating what we have with that of pyOpenSci.
If we decide to move forward with this partnership, these are the proposed steps:
- Do a trial review period of some packages to see how it goes to make the Astropy community feel more comfortable with the changes (also see The trial period below).
- Based on the trial, our current Affiliated editors would finalize and publish the updated guidelines and process, including new COI policy, packaging guidelines, and so forth. These new guidelines need to be compatible with pre-APE 22 Affiliated Package review guidelines and pre-APE 22 Affiliated Package review process, with templates. At the same time, we should link to the actively developed community-driven packaging guide over at pyOpenSci and Scientific Python, in addition to the OpenAstronomy packaging guide already implemented in Astropy.
- Come up with a plan to transition already accepted packages over to the pyOpenSci review process. This could be a process that happens over time in that new packages just go through the new review process and get the value of this partnership (pyOpenSci and JOSS) through that transition. Then, we could slowly look at the older packages and evaluate their current health states to determine whether another review is warranted (also see "Re-review" of packages). We will encourage these older packages that are still actively maintained to also go through the new process, pointing them to the benefits mentioned in What do we gain from a partnership with pyOpenSci?
- For the new packages, have our project website ingest a RSS/XML feed from pyOpenSci for cross-listing (also see Listing on websites). We would also link back to pyOpenSci on our page. Meanwhile, we would keep the pre-APE 22 listing on a "legacy affiliation" page that will be kept alive during the lifetime of the Astropy project.
- Work with pyOpenSci to cross-list Affiliated editors, between their editors listing and our roles page. For example, pyOpenSci could mark Astropy editors on the pyOpenSci Editorial Board with Astropy logo.
- Work with pyOpenSci to hash out a more concrete plan on how to get a previously sunsetted package to be listed actively again if the package is revived.
- Work with pyOpenSci to come up with a process to swap Affiliated editors in and out of pyOpenSci Editorial Board, since our Affiliated editor assignment is not permanent.
The goal is to have a migration that is not too disruptive to current process. Once this APE is accepted, new package submissions will go through the pyOpenSci process. Packages currently under review can choose to continue under the old process (and then be treated and listed like packages accepted as Affiliated before this APE), or be transferred to the new process and follow pyOpenSci procedures.
Note: Communication is very important at this stage!
This trial is only done for new Affiliated requests that are early enough in the process as not to have work duplicated for package maintainers, reviewers, and editors.
Editors will give qualifying packages the option to try out this new process. Regardless of the outcome of this APE, if the package is accepted during this trial, the acceptance stands; That is, if this APE falls through but the package used the proposed process here, it still counts as Affiliated and does not have to re-apply.
Reviewers who agree to participate will use the reviewer sign-up form for pyOpenSci. If a package agrees but the chosen reviewer declines this trial, the editor assigned will find a new reviewer.
Somewhat compatible:
- Packaging guidelines are pretty similar.
- Existing Affiliated editors would join the pyOpenSci Editorial Board.
- There would be cross-listing between Affiliated editors over at pyOpenSci and our roles page.
- There would be cross-listing between Affiliated packages over at pyOpenSci and Affiliated page.
Not backward compatible:
- Reviewers can no longer be anonymous and have to sign up via reviewer sign-up form. The whole review process is open.
- Instead of static color badges, there will be a link to full pyOpenSci review for that package. More dynamic badges might come later (see "Re-review" of packages).
- Inactive packages will be sunsetted if revival is not an option.
We keep the status quo; no changes needed but we are also not tapping into similar effort in a wider scientific Python community.
After significant discussion in the PR, this APE ended up with clear support by all the commenters, and was unanimously supported by the Coordination Committee (aside from the two co-authors on the commitee, who abstained due to being co-authors). Hence it was accepted.