Opportunities to reframe moderation practices in the wake of the COVID-19 pandemic
CHARLES DARR AND ROSEMARY HIPKINS
The COVID-19 pandemic has disrupted some traditional assessment practices and drawn attention to the range of evidence students produce in their day-to-day learning. In this edition of Assessment News, Charles Darr and Rosemary Hipkins argue that using this evidence to make sound summative judgements requires a robust understanding of what it means to make a valid judgement, along with a clear commitment to professional learning through social moderation.
Introduction
For many teachers, especially but not only in the secondary sector, the COVID-19 pandemic has disrupted the traditional strategy of using a set task or an end-of-unit test as the preferred form of summative assessment. Some professional learning advisers have recognised the need to support teachers to identify features that make students’ independent work suitable for assessment purposes. The box “What format will the evidence be in?” shows one small section of a pamphlet produced by two science advisers to support the teachers they are working with (Cleary & Kenton, 2020).
“What format will the evidence be in?” implies that a wide variety of tasks can produce evidence that teachers can subsequently use for summative assessment purposes. But how can we demonstrate that the assessment judgements we make based on the evidence are sound?
Making valid judgements
Making a valid judgment involves being able to defend a claim that the judgment is accurate. This applies whether that judgement is made in the context of NCEA, against a curriculum level, or against some other specified standard. At least three important ingredients are required to make a strong validity claim regarding a judgement (Darr, 2019).
First, it is important to have a strong understanding of what the judgement entails. For an achievement standard in NCEA this means understanding what is at the heart of the standard and how the expectation for performance varies at Achieved, Merit, and Excellence levels. When making a judgement against a curriculum level, it involves understanding the capabilities that the level encompasses and how well the student is expected to be able to demonstrate those capabilities.
Secondly, it is important to be clear about what can count as evidence that students know and can do the things we have judged them to be capable of. What should we be able to observe when students have met the standard or level?
The third ingredient is a sense of how much evidence is required to support the judgment. If the evidence is scanty then judgements may be weak, and the claim the judgement is sound easily rebutted.
Every one of these three ingredients must draw on teachers’ professional expertise. That’s because they involve understandings that cannot necessarily be easily described or codified in the words of a standard or a curriculum level descriptor and its supporting materials. This is not a bad thing—teaching professionals are very capable of developing shared norms around these understandings. And once such norms have been developed, they can lead to high levels of agreement when the understandings are applied in a range of real-life classroom contexts.
The importance of professional learning through moderation
Developing shared norms does not occur in a vacuum. It generally comes about through social interactions in appropriate practice settings. In the assessment world, taking part in activities that lead to shared understanding around judgements and judgement-making is often referred to as social moderation.
Social moderation involves teachers working together to discuss and explore what standards mean, what appropriate evidence looks like, and how much evidence is sufficient. When teachers become involved in this kind of moderation, they can generate shared understandings and agreements about how to interpret and use evidence to support judgements. When moderation conversations are productively focused, they can lead to strong agreement regarding what constitutes a valid judgement. Once an agreement is firmly established and understood, it can be transferred to new learning contexts—that is, it constitutes valuable professional learning to take forward.
It is important to maintain a clear focus: no-one wants to take part in moderation conversations that go around and around without reaching shared views about what is important and why. This is where a robust “decision framework” can help. A decision framework helps facilitate the judgement process by alerting participants to features in the evidence that are important to observe, and providing richly illustrated descriptions of achievement at different levels. The Progress and Consistency Tool (PaCT) is an example of a decision framework. The PaCT provides a common frame of reference which helps teachers to systematically approach evidence and make judgements about achievement.
Although they might look different to the PaCT, frameworks developed to support NCEA judgements would help teachers and students to understand the essential differences associated with Achieved, Merit, and Excellence levels, and provide exemplification of evidence of learning that corresponded to achievement at these different levels. Decision frameworks would also alert teachers to the kind of indicators they could observe over time and in rich learning contexts. Armed with such insights, teachers would not need to rely on final tests or one-off tasks. Evidence that students create, and that teachers get to see day-to-day when students are involved in learning, could take on new meaning. Even if this evidence was initially collected for formative purposes, it could be reinterpreted to support a summative judgement. In this way, teaching and learning could be in the forefront of valid and reliable assessment processes, and there would not be the same need to rely on one-off summative assessment events.
Decision frameworks could and should be developed to support NCEA moderation conversations, especially as new, larger achievement standards are introduced. We see huge potential here in the NCEA review: newer, bigger achievement standards will require teachers to rethink curriculum purposes and assessment practices simultaneously as the new standards come on-stream. This challenge underscores the importance of opportunities for moderation conversations that foreground deep professional learning as an essential cornerstone of quality curriculum design and assessment assurance processes.
Final thoughts
As we were writing this column, students were about to head back to school at Level 2. It was a time of uncertainty and some apprehension. Another series of “spikes” of COVID-19 infections could yet see students sent home again. Whether this happens or not, ways of doing and being a teacher or a learner might never seem the same again. In this fluid context our hope is that practices for systematically gathering, documenting, and judging evidence of achievement produced as part of meaningful and engaging learning will become more deeply embedded into teachers’ professional work at all levels of schooling.
References
Cleary, S. & Kenton, C. (2020). Capturing naturally occurring evidence: What, why & how? https://ihiresearch.sharepoint.com/:b:/g/EbSG_9CIEzROgQRahiWdK9sBpsa6RZuwVInUsBSoI8iphw?e=fjcdKa
Darr, C. (2019). Approaching classroom assessment after the NCEA review. Set: Research Information for Teachers, (3), 57–61. https://doi.org/10.18296/set.0153
Booth, B., Dixon, H., & Hill, M. (2016). Assessment capability for New Zealand teachers and students: Challenging but possible. Set: Research Information for Teachers, (2), 28–35. http://dx.doi.org/10.18296/set.0043
Wylie, C., & Bonne, L. (2014). Primary and intermediate schools in 2013: Main findings from the NZCER national survey. New Zealand Council for Educational Research. http://www.nzcer.org.nz.ezproxy.otago.ac.nz/research/publications/primary-and-intermediate-schools-2013