There is often much hang-wringing about research quality. Researchers are familiar with many of the issues: the file-drawer effect, HARKing, p-hacking, over-emphasis on novelty/surprise. We know some possible solutions to these problems, but putting them into practice is difficult when they are in conflict with existing practices. For example, registered reports involve reviewing a paper at the design stage, before data collection. This avoids a focus on novelty of results and allows meaningful changes to the study in response to reviews. Making sure to accept replication studies can make our knowledge base sturdier, shoring up existing foundations before moving on to new studies.
So: what if we took these great ideas and combined them into a journal special issue? Aleata Hubbard Cheuoua, Eva Marinus and I will be guest-editing a special issue of the Computer Science Education journal that will only accept replication studies that are published as registered reports. This means you will submit your introduction, motivation, and research/analysis design for a study which has not yet taken place. This will be reviewed, and if you get a positive decision, the paper is accepted in-principle, regardless of the results. The study must be a replication, which means much of your research design will already be decided for you. So the upfront effort is low — although don’t forget that if you get accepted, you must actually do the study!
Because this is a new way of doing things for our area, we are issuing a call for papers and a call for reviewers; there will be training for the reviewers. The full text of the call will appear soon on the CSE journal website, but in the mean time here is the call and a draft of our further details webpage. If you have any queries, please do contact Aleata, Eva or me.
Academic Computing 