By:

Em. prof. dr. Paul. A. Kirschner, Editor-in-Chief, Open University of the Netherlands / Thomas More University of Applied Sciences, and

Dr. Jeroen Janssen, Associate Editor, Utrecht University, Netherlands

The science of learning and instruction is a rapidly evolving field, with a variety of theoretical and methodological approaches. It is also sometimes burdened by two gadflies, namely the so-called ‘replication crisis’ where attempts to replicate accepted theories and results may not replicate (for whatever reason) and ‘positive’ publication bias where articles that fail to confirm hypotheses and/or that don’t produce statistically significant results are seen to have a lesser chance of getting published. In an attempt to combat both of these problems, and increase trust in research, we at the Journal of Computer Assisted Learning have chosen to explicitly invite authors to submit Registered Reports – a new format of empirical articles that is designed to improve the transparency and reproducibility of hypothesis-driven research.

The call to invite authors to submit their Registered Reports went out in February 2018. Since then, we have received a growing number of Registered Reports submissions, and we are now delighted to publish our first collection of Registered Reports. To us, this proves that the learning and instruction community is responding positively to this new format.

Registered Reports differ from conventional empirical articles by performing part of the review process before researchers even collect and analyze data. The Introduction and Method section (including hypotheses and all relevant materials) are peer-reviewed prior to data collection. High quality pre-registered protocols that meet strict editorial criteria are then offered in principle acceptance, which guarantees publication of the results, provided that the authors adhere to their pre-registered protocol.

Daniel Ansari (University of Western Ontario, Canada) and Judit Gervain (Université Paris Descartes, Paris) in a blog about the introduction of Registered Reports in Developmental Science, noted the following:

“Registered Reports (RR) place an emphasis on the adequacy of methods and analysis plan for studies deemed to be informative. They thus benefit both the submitting authors and the discipline. If the rationale and the methods of the planned studies are sound, and accepted during the review process, authors can expect their work to be published, unless they deviate from the accepted methods, even if the results are weak, null, or different than predicted. The discipline gains, because the publication process will be more transparent and is therefore more likely to curtail questionable research practices. This increases the reliability and reproducibility of data. … [Also, in] RR format interpretations are less likely to be modified post hoc to fit the initial predictions.”

Three arguments for Registered Reports

For studies with a clear hypothesis, the Registered Reports format has three key strengths compared with traditional research publishing. First, it prevents publication bias by ensuring that editorial decisions are made on the basis of the theoretical importance and methodological rigor of a study, before research outcomes are known. Second, by requiring authors to pre-register their study methods and analysis plans in advance, it prevents common forms of research bias including p-hacking (mis-use of data analysis to find patterns that are statistically significant) and HARKing (Hypothesizing After Results are Known or hindsight bias) while still welcoming unregistered analyses that are clearly labelled as exploratory. Third, because protocols are accepted in advance of data being collected, the format provides an incentive for researchers to conduct important replication studies and other novel, resource-intensive projects (e.g. involving multi-site consortia) — projects that would otherwise be too risky to undertake where the publishability of the outcome is contingent on the results.

It is encouraging to see the positive reaction to Registered Reports from researchers in their various roles as editors, authors and peer reviewers. For more information and answers frequently asked questions about Registered Reports see https://cos.io/rr/

Procedure

• Authors are invited to submit ‘Stage 1 manuscripts’ which outline the rationale and method of the proposed study or studies (see Guidelines for Authors).
• All submitted papers will be peer-reviewed for theoretical significance and methodological quality. In-Principle Acceptance (IPA) will be given to high quality submissions. Data collecting can only commence after a Stage 1 manuscript has been accepted.
• Once the study (or studies) is complete, the ‘Stage 2 manuscripts’ will be also be peer-reviewed to see whether they are consistent with the pre-registered Stage 1 protocol. Editorial decisions will not be based on the perceived importance, novelty or conclusiveness of the results.
• The journal operates double blind peer review as a means of tackling real or perceived bias in the review process, so the authors must provide their title page as a separate file from their main document. Title page includes the complete title of the paper, affiliation and contact details for the corresponding author (both postal address and email address).

Five tips for preparing a Registered Report for submission

The following tips have been offered by Ansari and Gervain. They are also useful for authors considering to submit a registered report to the Journal of Computer Assisted Learning

1. Carefully read the submission guidelines. They are extremely detailed and informative.
2. Ensure that you have all the resources necessary to carry out the research you have proposed. You don’t want to receive in-principle acceptance of a Stage 1 Registered Report, only to find out that you do not have the resources to collect the data! Hopefully in the near future, funding agencies will begin to support Stage 1 manuscripts that have received IPA.
3. Make sure that you are setting yourself a realistic timeline that takes into account variability in the time it will take to get your manuscript reviewed at both Stage 1 and 2. This is especially important for graduate students and post-doctoral fellows who may have pressing deadlines for finalizing their data collection so that they can defend, move to a new position etc.
4. Be very precise in your hypotheses and analysis plan. Registered Reports are all about spending a lot of time thinking and discussing your research plans before you collect the data. Being as precise as possible and having a very detailed analysis plan can save you a lot of time both during peer review as well as the analysis after you have collected the data and before you prepare your Stage 2 manuscript.
5. Make sure that your research ethics approval will allow you to share your data on an open data repository, such as the Open Science Framework. Open data are a requirement for the publication of a Registered Report.