Reviewers List

[oliviaguest]

I would like to propose to either make the reviewers list private (or at least not on the website on the page it currently is) and/or to make all reviewers add themselves, perhaps?

Most of the people I ask to review and who do a very good job (of course) are not on the list and thus do not get the same level of recognition. Having a public list without having it updated doesn't seem to add much except to say "look at all these people"? Or am I missing something?

[khinsen]

The list of reviewers on the Web site is more precisely the list of people who have volunteered to be on that list. It's mainly of interest to editors looking for a suitable reviewer. We could probably easily (meaning automatically) compile a list of people who actually reviewed ReScience papers from the article metadata.

I agree that the current presentation is a bit misleading. I don't think we should make anything private, and we don't even have any private space for ReScience, but we can probably do a better job with the presentation.

[oliviaguest]

Great. So let's move to maybe two lists: a list with active reviewers and a list with people who volunteer(ed)?

[rougier]

The idea of the public list of reviewers was to give some recognition to reviewers. I used to add stars behind each name that point to the review they made but in the end it was too much work. Also, once people have registered, it's hard to know if they are reactive or not. For some of them, we do not even have an email address and the only wait to notify them is through the GitHub interface. If they don't receive such notifications through mail and do not connect to GitHub on a regular basis, they won't receive anything.

I agree for the two lists but I'm not sure how to display them.

[oliviaguest]

I think ideally we should have a list of who has reviewed on the website. We can also have a "private" list (or not and have it on website too?) of who can review — by private I mean one that can just be a .csv on GitHub and doesn't need to be public-facing on the website. Just an idea, but either way: I think it would be nice to do the former list.

[rougier]

I agree and if I remember well, @ctb initiated such a list in .csv format and of course, I don't remember where it is now. But most probably it is outdated.

[oliviaguest]

@khinsen how easy is it to make the "people who have reviewed" list? can I help?

[khinsen]

I can't really say how easy it is because I have no experience with Web scraping, but here's an outline of what it takes:

1. For the first years (when articles were submitted as pull requests), traverse https://github.com/ReScience-Archives, and scan the headers of the article.md files for the reviewer names.
2. For recent articles, iterate over the directories in https://github.com/rescience/articles, and scan article.yaml for reviewer names.
3. Clean up and format the list, taking care of the usual problems of misspelt names, non-unique names, etc. (though the list is short enough to do this by hand).

[oliviaguest]

Oh, I thought you were suggesting using the GitHub API. I misunderstood:

We could probably easily (meaning automatically) compile a list of people who actually reviewed ReScience papers from the article metadata.

What do we want to do that's a bit less time-/working memory-demanding? Because I can't offer anything that involved — I don't have time to write a scraper, right now (sorry).

Do we want to take the current list offline from the website itself? I personally believe yes, because I think it disincentivises people (in the current set-up) from reviewing who think it's an in-group thing only (based on private communications).

[khinsen]

Maybe there is a GitHub API that would make this easier. I know nothing about GitHub APIs!

As for making the reviewer less prominent (but keeping it public in one of our GitHub repositories), I agree. Pinging @rougier @benoit-girard for their opinion!

[oliviaguest]

NBD but I assume the GitHub API is how bots like @whedon — see: docs for all the features, if you haven't already — work.

[ctb]

I don't remember what I did back when with respect to a list, but I do have GitHub API experience - see http://ivory.idyll.org/blog/2019-github-project-reporting.html

@oliviaguest I would be happy to whip up a Python approach, if it suits, and then iterate with you. I'm not very fast at the iterations tho so if this is time sensitive I might not be a good choice.

[ctb]

(by "iterate" I mean ask for ~5 minutes of your attention periodically to go, oh yeah, that's useful, vs what the heck is that? gut checks, not serious time investments.)

[oliviaguest]

Sure — thanks. BTW I think if so it should be all EiCs involved?

[khinsen]

Just use this issue for follow-ups! And thanks @ctb for offering to help!

[ctb]

digging into it. Some notes --

• The GitHub API is overkill for anything other than getting the list of https://github.com/ReScience-Archives repos (and even that would have been faster by hand, only 17 repos :). Nonetheless here is the list of URLs: urls.txt
• the old stuff doesn't need to be automated, we'll only do that once anyway
• "For recent articles" in https://github.com/rescience/articles the extraction is worth automating because we will want to rerun that regularly.
• cross-checking the reviewer names and so on is straightforward too
• I'll probably binderize all of this so no one has to worry about installing anything.

Inclusive (but still some FP) list of article md files from archives is:
articles-list.txt

[ctb]

Reviewers for old stuff, from ReScience-Archives -
reviewers-archives.txt

[ctb]

for

10.5281_zenodo.3162114/article.yaml
10.5281_zenodo.3234524/article.yaml
10.5281_zenodo.3763416/article.yaml
10.5281_zenodo.3538217/article.yaml
10.5281_zenodo.3162890/article.yaml
10.5281_zenodo.3528175/article.yaml
10.5072_zenodo.416801/article.yaml
10.5281_zenodo.3069619/article.yaml
10.5281_zenodo.3161734/article.yaml
10.5281_zenodo.3158244/article.yaml
10.5281_zenodo.3160540/article.yaml
10.5281_zenodo.3630224/article.yaml
10_5281_zenodo.3538217/article.yaml

I find the following reviewers:

Andrea Caso
Andrew P. Davison
Anonymous reviewers
Christian Jarvers
Frédéric Gava
Georgios Detorakis
Justin Kitzes
Laura J. Graham
None
Pierre de Buyl
Qihong Lu
Xavier Hinaut

there are a bunch of None in here, looks like reviewers aren't always indicated in the article.yaml.

[khinsen]

Normally we have two reviewers, so there are two reviewer slots in the metadata file. For the ten-year challenge we decided to simplify to a single reviewer, and then the second slot is None. I hope that's the only explanation required for those Nones!

[ctb]

no reviewers found for:

"ReScience (R)evolution" - 10.5281_zenodo.3069619/article.yaml

"ICLR Reproducibility Challenge 2019" - 10.5281_zenodo.3158244/article.yaml

which seems ok?

There are also several anonymous reviewers.

[ctb]

SO. Where do we want to take this? Do we want to find github IDs for reviewers? Is there other metadata parsing and indexing foo that should be done? Are people interested in the code? (It's pretty simple stuff, now that I've got it worked out :)

[oliviaguest]

I think for me the answer to this question is probably what will answer your question:

Do we want to take the current list offline from the website itself? I personally believe yes, because I think it disincentivises people (in the current set-up) from reviewing who think it's an in-group thing only (based on private communications).

[rougier]

T oasnwer @ctb question, editorial and article for the Ten Years Reproducibility Challenge have only 1 reveiwer. For the NeurIPS challenge, reviewers are anonymous (through open review).

[ctb]

When we are speaking of "the web site" list of reviewers, this is what that refers to, right? https://github.com/rescience/rescience.github.io/blob/sources/04-board.md

[khinsen]

@ctb Yes, that's all there is for now.