The Limitations of Self-Testing in Learning

The recently published article, Testing Effects for Self-Generated Versus Experimenter-Provided Questions (Myers, Hausman, & Rhodes, 2024), looks at self-testing (i.e., retrieval practice) as a study strategy and generative learning strategy. They evaluated whether learners benefited from generating and answering their own test questions compared to answering experimenter (e.g., teacher)-provided questions or simply rereading the study materials. Across three experiments, the findings reveal the limitations of self-testing when self-generated questions are involved.

Self-testing is widely regarded as a robust learning strategy, supported by extensive research demonstrating the “testing effect”—the idea that retrieving information enhances memory better than passive review. This effect is often linked to practices like answering provided questions or engaging with structured quizzes. However, in real-world educational contexts, students are often left to create their own questions without formal guidance. This study aimed to assess whether this type of self-testing leads to meaningful learning benefits.

Three Studies

In a first experiment the researchers looked at whether answering self-generated questions after a delay (so as to incorporate retrieval practice) was better than answering questions immediately after generating them. Participants in the study read two passages, one about monetary policy (549 words) and one about ice ages (1,052 words), and either created and answered questions simultaneously or created questions to answer later. Contrary to expectations, there was no significant advantage for delayed answering. Furthermore, participants often failed to generate questions that aligned with key test material, diminishing the potential benefits of self-testing.

The second experiment compared self-testing with delayed answering to answering experimenter-provided questions and rereading the material. Again, self-generated testing showed no significant advantage over the other methods. Interestingly, answering questions provided by the experimenter was consistently more effective, particularly for factual recall. This raised questions again about the alignment between self-generated questions and test material. It also questioned the efficacy of unstructured question creation.

The third and final experiment introduced a longer interval between study and testing. This change mimicked real educational scenarios. In such scenarios, assessments often follow days or weeks of study. Even with this delay, the students who answered questions provided by the experimenter outperformed self-testing, particularly for factual recall. Again, self-generated testing was less effective.

Insights

One critical insight from the study is that self-testing often fails to yield significant benefits because students struggle to create questions that effectively target material on subsequent assessments. Exploratory analyses revealed that participants’ self-generated questions frequently focused on irrelevant details or overly broad concepts. This misalignment undermined the benefits of retrieval practice.

Note: The results of all three experiments, especially the lack of concordance between the self-generated questions and the test that the students received about what they had read shows that generating good questions as a study strategy is a skill that the student needs to acquire. The experimenters assumed that the students could generate high-quality questions. I made a similar assumption in the past in a study with Kim Dirkx and Liesbeth Kester. We focused on the effect of summarising as a learning strategy. In Myers et al. , the students were university undergraduates, and in our research, they were juniors and seniors in high school. When our study showed that summarising didn’t lead to the expected results we analysed the summaries and saw that they were pitiful. We had assumed that high school juniors and seniors could write a good summary of a short text, but that wasn’t the case. They lacked the skills needed to do it. This is true for all study strategies. Learners need to be explicitly taught what they are and how to do it, be given the opportunity to practise it a lot in different contexts (i.e., school subjects), and receive good feedback from their teachers (and peers?) on their attempts.

Another issue according to the authors might be the cognitive load involved in generating questions. The dual task of creating and later answering questions might overwhelm learners, especially without clear guidance. In contrast, answering provided questions allows students to focus solely on retrieval, maximising the benefits of the testing effect.

The findings challenge the common educational advice that students should “test themselves” without offering structured materials or training on effective question creation. While self-testing has potential, its benefits are heavily contingent on the quality and relevance of the questions generated. This underscores the importance of providing students with carefully crafted practice questions or explicit training on how to develop their own.

The study concludes that self-testing, when implemented through unstructured self-generated questions, is not inherently beneficial. For educators and learners, the implications are clear: structured practice materials and guidance are critical for maximising the benefits of testing as a learning strategy. Without these, self-testing may fail to deliver the significant learning gains that we hope to achieve.

Myers, S. J., Hausman, H., & Rhodes, M. G. (2024). Testing effects for self-generated versus experimenter-provided questions. Journal of experimental psychology. Applied, 30(2), 241–257. https://doi.org/10.1037/xap0000487