Building Citizen Science Quality Control Exploratory Data Analysis & Other Tools

[ericearl]

Project info

Title:

Building Citizen Science Quality Control Exploratory Data Analysis & Other Tools

Project leads:

From the Oregon Health & Science University (OHSU) Developmental Cognition and Neruoimaging (DCAN) Labs in Portland, OR, USA:

• Eric Earl, https://github.com/ericearl
• Anders Perrone, https://github.com/perronea

Timezone: UTC -7 hours

Hub: The Americas

Maximum number of participants: 6 (+ 2 leads)

Description:

"Big data in biomedical research needs individual inspection, but this doesn’t scale for large datasets."

~ Swipes For Science

Swipes For Science is a web application to expedite visual quality control and can easily be set up and customized for any researcher’s personal needs. It was designed by Anisha Keshavan, PhD, et al, and inspired by the Tinder app. Scientists of any skill level can create an account and swipe left or right to judge an image as Pass or Fail. Each vote is recorded into a live online database that can be visualized for efficient data inclusion or exclusion in a study. While this platform can be simply leveraged by experts on a small scale to make the quality control process more efficient and organized it may also be used to crowd source quality control by “citizen scientists” for large-scale datasets with proven accuracy that matches expert raters.

We have a unique opportunity at this OHBM Brainhack 2020 for you to both setup your own Swipes For Science site and to create exploratory data analysis tools with anonymized data collected from an existent Swipes For Science site. The dataset we’ve included is for QC of outputs from the lightly modified DCAN Labs HCP minimal preprocessing pipeline.

You can imagine answering questions like:

• For all the images associated with one subject's data, should that entire subject be considered a Pass or Fail for inclusion in further analysis?
• For all ratings provided by citizen scientist ABC, how reliably does ABC vote compared to the gold standard data?
• Among all citizen scientists, who are the best raters?
• What is the inter-rater reliability on image XYZ?
• How can we use deep learning to align crowd sourced QC data with gold standard expert evaluation.
  ... and more!

Link to project: https://github.com/DCAN-Labs/SwipesForScienceToolkit

Mattermost handle: earl

Goals for the OHBM Brainhack:

1. Show interested contributors how to set up a SwipesForScience site
2. Share with contributors anonymized data from our SwipesForScience site
3. Create Exploratory Data Analysis Tools for SwipesForScience data

Good first issues:

• Import data from JSON format and summarize in a digestible format like CSV.
• Report basic statistics on subjects and/or users.
• Visualize swipe data.

Skills:

The more you have beyond the below skills, the more you could do with the project:

• (beginner) Data transformation (munging)
• (beginner) Python
• (beginner) Git/GitHub
• (beginner) Data visualizations
• (confirmed) Basic statistics (mean, variance, etc)
• (barely) NPM/Vue/JavaScript/HTML/CSS

Chat channel: https://mattermost.brainhack.org/brainhack/channels/hbm-swipes

Image for the OHBM brainhack website

Project submission

Submission checklist

Once the issue is submitted, please check items in this list as you add under 'Additional project info'

Please include the following above (all required):

• Link to your project: could be a code repository, a shared document, etc. See here
• Include your Mattermost handle (i.e. your username). If you do not have an account, please sign up here.
• Goals for the OHBM Brainhack: describe what you want to achieve during this brainhack. See here.
• Flesh out at least 2 "good first issues": those are tasks that do not require any prior knowledge about your project, could be defined as issues in a GitHub repository, or in a shared document, cf here.
• Skills: list skills that would be particularly suitable for your project. We ask you to include at least one non-coding skill, cf. here.
• Chat channel: A link to a chat channel that will be used during the OHBM Brainhack. This can be an existing channel or a new one. We recommend using the Brainhack space on mattermost, cf. here.
• Video channel: Please create a video channel that will be used during the OHBM Brainhack and share it in your chat channel above. This can be an existing channel or a new one. For instance a jitsi meet room, cf. here.
• Provide an image of your project for the OHBM brainhack website

You can also include information about (all optional):

• Number of participants, cf. here
• Twitter-size summary of your project pitch, cf. here
• Set up a kanban board on your repository to better divide the work and keep track of things, cf here
• Project snippet for the OHBM Brainhack website, cf. here

We would like to think about how you will credit and onboard new members to your project. We recommend reading references from this section. If you'd like to share your thoughts with future project participants, you can include information about (recommended):

• Specify how will you acknowledge contributions (e.g. listing members on a contributing page).
• Provide links to onboarding documents if you have some.

[welcome]

(Image: CC-BY license, The Turing Way Community, & Scriberia. Zenodo. http://doi.org/10.5281/zenodo.3332808)
🎉 Welcome to the 2020 OHBM Hackathon! 🎉 We're really excited to have your input on this repository and welcome your idea! 💖 If you haven't done so already, please make sure you check out our Code of Conduct. Did you issue a new hackathon project? Please check if you have provided all relevant information and an image for the website.

[llevitis]

hey @ericearl - thanks for adding the project! Everything looks great, and I just ask that you add 2 concrete good first issues :)

[complexbrains]

Dear @ericearl Would you mind creating and sharing a video chat room link and in your Mattermost channel and pinning it to the channel please?

Once you are done with that please let us know, we will complete the revision and your project will be ready to go! Thank you 🤗

[ericearl]

@llevitis and @complexbrains the requested edits have been made. Hopefully it is satisfactory. Thank you!