You’ve often heard from me about the problem that I have with using both self-report as a measure when doing research as well as using teacher-report as a measure. My reasons are simple, namely that such reports are subjective and not objective and that they are usually neither valid nor reliable. These two problems are compounded by the results of this study, namely that a person can only make a judgement if they really understand the concept. This, for example, is a prominent problem with the most often used measure of cognitive load; the one item scale by Paas and colleagues. The learner is often asked to judge how much mental effort a task required. Unfortunately, most people (1) have no basis with which they can judge their mental effort as they do for physical effort (e.g., whether or not they are out of breath, are sweating, have an increased heart rate, feel their arms or legs getting tired or cramping) and (2) confuse how much mental effort a task required with how difficult the task was, how much trouble they had when carrying out the task, and so forth.

The authors of this paper (Charlotte Dignath and Lara Sprenger) encountered this problem with respect to teacher assessment of self-regulated learning in their students. In the article ‘Can you only diagnose what you know? The relation between teachers’ self-regulation of learning concepts and their assessment of students’ self-regulation’ they conclude:

We conducted an examination of educators’ conceptualization of SRL, identified three patterns to classify these conceptualizations, and investigated how such conceptualization is associated with teachers’ assessment of SRL. We found that, in particular, teachers who conceptualize SRL as student autonomy and self-directedness might be at risk for using cues that are not diagnostic of SRL when attempting to identify their students’ self-regulation skills. This raises questions about the impact that teachers’ conceptualization of SRL has on their assessment accuracy for SRL, which might, in turn, affect teachers’ adaptive teaching and promoting SRL in the classroom.

The answer to the question is the article’s title is: No

Here the abstract:

Self-regulation of learning (SRL) positively affects achievement and motivation. Therefore, teachers are supposed to foster students’ SRL by providing them with strategies. However, two preconditions have to be met: teachers need to diagnose their students’ SRL to take instructional decisions about promoting SRL. To this end, teachers need knowledge about SRL to know what to diagnose. Only little research has investigated teachers’ knowledge about SRL and its assessment yet. Thus, the aim of this study was to identify teachers’ conceptions about SRL, to investigate their ideas about how to diagnose their students’ SRL, and to test relationships between both. To this end, we developed two systematic coding schemes to analyze the conceptions about SRL and the ideas about assessing SRL in the classroom among a sample of 205 teachers. The coding schemes for teachers’ open answers were developed based on models about SRL and were extended by deriving codes from the empirical data and produced satisfactory interrater reliability (conceptions about SRL: κ = 0.85, SE = 0.03; ideas about assessing SRL: κ = 0.63, SE = 0.05). The results showed that many teachers did not refer to any regulation procedure at all and described SRL mainly as student autonomy and self-directedness. Only few teachers had a comprehensive conception of the entire SRL cycle. We identified three patterns of teachers’ conceptualizations of SRL: a motivation-oriented, an autonomy-oriented, and a regulation-oriented conceptualization of SRL. Regarding teachers’ ideas about assessing their students’ SRL, teachers mainly focused on cues that are not diagnostic of SRL. Yet, many teachers knew about portfolios to register SRL among students. Finally, our results suggest that, partly, teachers’ ideas about assessing SRL varied as a function of their SRL concept: teachers with an autonomy-oriented conceptualization of SRL were more likely to use cues that are not diagnostic of SRL, such as unsystematic observation or off-task behavior. The results provide insights into teachers’ conceptions of SRL and of its assessment. Implications for future research in the field of SRL will be drawn, in particular about how to support teachers in diagnosing and fostering SR among their students.

Source: Can You Only Diagnose What You Know? The Relation Between Teachers’ Self-Regulation of Learning Concepts and Their Assessment of Students’ Self-Regulation