We’ve now gone through all the steps of creating a more reproducible research project and workflow. If it worked, we should be able to recreate our thought process and analyses. As we discussed at the beginning, the actual test of this comes when you, or another collaborator or colleague, go back to a project a few weeks or a few months later and try to figure out what you did. We can’t really wait for our own memories to decay, so we’ll use each other as stand-ins for our future selves who have probably forgotten a lot of details. So, please pair up with the group beside you, and share the URL for your projects. Look over each others' projects. Take note of things they did well and things they could potentially do better so that you can quickly understand what they did and how they did it.
Activity: Look over another groups project and critique the transparency/documentation
Alright, hopefully you guys had time to get through each others projects. To kick things off I’ll say a few things about my own project that I did well, and a few things that I could have done better.
-
Done Well
- All my data, code, and materials are in one central place so I haven’t left any files out that could get lost
- I’ve done at least a bit of code commenting so I have information about what I did in plain english
- I documented the basic ideas and rationals for the project so I will be aware if this changes overtime
-
Could do better
- My data files are SPSS files, so not everyone will be able to use them easily (and I might not be able to use them in the future) because SPSS format is proprietary and it is expensive software. It would be be great if I also uploaded the data in an open format such as csv files so that I or anyone else would always be able to use them.
- I have a private data subcomponent because, presumable, my IRB said that I couldn’t share the de-anonymized data. However, I don’t make a note of that anywhere. I could improve my documentation by noting that in the wiki. Also, I could create a truly anonymous version of the data that can be public and then tell people how they can request the complete version of the datafile (e.g., once they have IRB approval to work with the data).
For the projects you looked over, what were other things that you noticed people doing well? Were there things you noticed that might have helped improve the documentation or transparency of the project?
Activity: Q&A depending on the feedback that comes in above.
We covered a lot of information today. Reproducible research practices aren’t all or nothing. Even small improvements to organization and documentation can help increase the transparency and reproducibility of research. Some people will make a full workflow overhaul straight out of the gate, others will make a series of small, incremental changes to their research processes. Both of these approaches are fine, it’s all about what works best to integrating reproducible practices into your own workflow. Figuring out how to apply examples to your particular needs can be challenging. So, before I open it up to questions and discussion here is where you can get help after this workshop.
- If you want to review any of the workshop content, it is all available here:
- If you have specific questions, the Center for Open Science provides completely free, one-on-one stats and methods consulting over email and google hangouts
- If you have questions about the OSF, there is a helpdesk manned by a real person who can help you through any issues or questions you might have