More structured recommendations for review issues?

[sneakers-the-rat]

We recommend that people raise issues on reviewed repos with comments/questions rather than in the review themselves, but could we also give a bit more structure to those?

One obvious suggestion would be to have a sort of standard taxonomy of things that a given issue is about:

• Docs
• Tests
• Installation
• CI
• Bugs
• Performance

etc. We could make shortcodes for the various items in the review checklist, so people could just use like [docs] in the issue title and we know it corresponds to that checklist item.

Another is to indicate the "status" of an issue, specifically I have noticed some lack of clarity for both reviewers and authors re: whether an issue is a blocker for the review, just a suggestion, or a question. Reviewers aren't sure if they are able to 'go beyond' the checklist and say something specifically "I need this to be documented, I need tests for this, etc." and they are also unsure if they should raise issues just for asking questions (the ones i have spoken with were concerned that would be interpreted as a blocking requirement when it was just a question). On the other side, authors are uncertain if they need to wontfix something that's just a comment, unsure if they can close issues, etc.

I think having a recommended set of terms would make this clearer - obviously people can go beyond them/not use them, but merely having them I think gives a clearer indication of the kind of things that should be raised as issues.

• Question
• Optional Suggestion
• Required/Blocking (whatever we want to call this)

Some examples from me on a recent review where i was trying to signpost this more clearly: https://github.com/neuroanatomy/thresholdmann/issues?q=author%3Asneakers-the-rat

[jedbrown]

I like the intent, but note that many projects (especially the large/more mature ones) have their own taxonomy, labeling conventions, and issue templates. It feels out of our lane to prescribe that they use our convention or write/document new conventions solely for the JOSS review. (That is, the rest of their project documentation may apply to thousands of people with a time scale of many years, the JOSS review is relevant to 2-3 people over a timespan of weeks to months.)

[sneakers-the-rat]

many projects (especially the large/more mature ones) have their own taxonomy, labeling conventions, and issue templates.

Oh shoot I wasnt clear enough - I meant this not as a "requirement" but as like a set of prompting examples. If there isnt more structure/the authors dont have preferences, give more of a prompt to reviewers so they know what they can do