Choosing stopping criteria before starting the review process

[juliandwain]

Hi,

Thanks for this amazing project!
I have two questions, the first one is more a general question, and the second one is an idea I got while using ASReview.
I wanted to share this idea and get your feedback on it!

1. Is there a prospective date when v1 will be released? (is there a timeline or something similar?) (in my case, I am looking forward to the idea proposed in Possibility to obtain csv file preserving the complete history of the reviewing process #1089.

Now onto the idea.
In #557, stopping criteria are categorized by estimating the number of relevant papers or using heuristics.
(VAN HAASTRECHT ET AL. 2021) propose a simple estimate how to guess the number of relevant papers:

$$ R \approx N \cdot {r \over r+i}$$

Herein, $R$ is the estimated number of relevant papers, $N$ is the total number of records in the dataset, $r$ is a pre-defined number of papers to mark as relevant, and $i$ denotes the number of papers marked as irrelevant at the current stage.
The screening can be stopped if a pre-defined percentage (e.g. 95 %) of the estimated number of relevant papers $R$ has been marked relevant.
There certainly are more methods to guess the number of relevant papers.

However, my suggestions would be that the user can decide what criteria to follow, i.e., estimating the number of relevant papers or using heuristics (a combination of both could also be beneficial) before starting the review process.
ASReview could then provide an overview of how close the user is to the stopping criteria, e.g.,

• heuristics (I think this is already sufficiently implemented, but when choosing a time-based method, then maybe a timer can be useful)
• estimating the number of relevant papers: herein, considering the formula from above, what might be useful is showing how much the number of papers to be marked as relevant r and the estimated number of relevant papers R are different from another - what is then shown to the user is how close one is to the defined percentage from above

I am happy to hear your thoughts on this and I hope that this topic is/was not already discussed elsewhere!

Best,
J

References

VAN HAASTRECHT ET AL. 2021
van Haastrecht, M.; Sarhan, I.; Yigit Ozkan, B.; Brinkhuis, M.; Spruit, M.: SYM-BALS: A Systematic Review Methodology Blending Active Learning and Snow-balling. Frontiers in Research Metrics and Analytics 6 (2021), S. 1‑14.

[Rensvandeschoot]

Is there a prospective date when v1 will be released? (is there a timeline or something similar?)

Yes! we aim at June 7-10... see the milestones overview for topics we are still working on.

[Rensvandeschoot]

@MaxvanHaastrecht: How would you propose to define r??

[MaxvanHaastrecht]

Hey @j0sien, the Knee algorithm has a parameter 'k', the minimum number of relevant papers you want to find, which you have to choose before starting. The main reason for using this parameter is that since you are comparing two slopes (below the knee and above the knee), you want to make sure that the 'below the knee' part is not too short to avoid any extreme scenarios where you would stop very early on.

To answer your question, for the knee algorithm it doesn't matter much what your starting 'random' set is. You could screen randomly using ASReview until you have reached this number k of relevant papers. You could also start once you have found (e.g.) at least one relevant paper, so that at least you know ASReview can do its job. I am assuming k>1, Cormack & Grossman suggest k=10, but this depends a bit on your dataset. The knee algorithm is then applied dynamically while you screen, meaning once every few iterations (or whenever you think the slope is starting to flatten off) you would need to check whether and when you have reached the 'knee'.

Hope this helps!

[tychovt99]

Dear @MaxvanHaastrecht , thank you for the ASReview Knee Method code. I'm trying to use it to find a stopping criterion for my current systematic review, but I unfortunately do not have very much experience in Python.

I was wondering how you downloaded the .csv file you used in this code, as the columns are very different from my dataset that I download from ASReview. For example, I do not have "category" and "Relevant by ASReview Lab". My dataset includes authors, title, doi, abstract, keywords, included, and asreview_ranking. From what I understand, 'included' indicates whether I have checked a paper as relevant or irrelevant within ASReview. I'm not sure how I can translate my dataset's columns into your code, do you think you could give me some advice on this? Any help would be appreciated.

Cheers,
Tycho

[MaxvanHaastrecht]

Hey @tychovt99, the Python script was made to work based on the CSV @Rensvandeschoot provided in this comment: #1115 (reply in thread). If you go to the Analytics tab of the project you're working on and download the CSV of the Progress graph (see attached image), you should get a CSV with a "category" and "Relevant records" column. These correspond to the "category" and "Relevant by ASReview Lab" columns from the original example.

To be clear, calculating the knee stopping criterion is something you do during the review process. In practice, you might want to stop every 20 records or so and see if your stopping criterion has been satisfied. This avoids having to do the calculation after every decision you make.

[tychovt99]

Thank you for the quick and informative reply! I managed to find the CSV file this way and it seems to work well now. Also thank you for the suggestion on stopping, my supervisor and I were considering stopping after batches of 5% of the total records.

[Rensvandeschoot]

@tychovt99 see also the proposed stopping heuristic by @j0sien in this thread: #1300