“Do games help learning? Where’s the evidence?” Simple questions with not-so-simple answers

By Rachel Bolstad

Of the many questions about games and learning that we’re exploring in our project, perhaps two obvious questions to ask are: “Do games actually help learning?” And “What’s the research evidence for this?”

They might seem like  simple questions, but the answers are far from simple. The short answer is “it depends”. The long-version answers require us to sit down to unpack what kinds of learning we’re interested in, what kinds of games we’re talking about, and how they’re being used.  Our research approach has focussed on cultivating a rich understanding of the New Zealand classroom contexts in which games are used for learning. We’re especially interested in the motivations, intentions, and experiences of game-using (and game-designing) teachers and students. We want to address questions about the learning value and impact of games within these specific and interesting contexts.

However, we’re certainly not the first researchers to look into the learning potential of games, and for the last 18 months we’ve also been accumulating a large body of existing NZ and international research literature to see what others have found. In my next few blog posts, I’m going to explore some of what this literature has to say, and discuss why answers to the simple question “do games help learning?” remain fascinatingly complex.

So, do games help learning?  What’s the evidence?

Since this blog post is about evidence, I’ll begin by looking at some recent systematic reviews. Systematic reviews are a kind of evidence-based literature review that aims to rigorously review and analyse dozens of existing studies, usually with the goal of finding out definitive answers to questions like “does [input X] improve [outcome Y]” or “how much does [input X] improve [outcome Y]?”.

I’ll be honest, I do have some problems with systematic reviews, which I’ll discuss at the end of the post. I should also mention that the research field around games and learning is incredibly broad and diverse, and systematic reviews are only one small corner of that field. However, since they bring together many individual pieces of research with the aim of trying to find generalisable conclusions, they can be a useful starting point. Let’s look at four different systematic reviews which have sought to evaluate the effect on learning of, respectively, 1) digital games, 2) game-based learning, 3) gamification, and 4) digital simulations.

Do digital games improve learning?

One systematic review by Clark et al. (2014) [1] analysed 69 published studies, each of which used either randomised controlled trials (RCTs) or quasi-experimental design, involving learners aged between 6 and 25, learning with games versus learning in non-game conditions. The review only looked at digital games (notably excluding digital games that aim to teach young people how to create or programme games – these will be the focus of future blogs). The various studies reviewed by Clark et al. focussed on a range of learning outcomes, including:

• cognitive outcomes (e.g. knowledge, creativity)
• intrapersonal outcomes (e.g. intellectual openness, conscientiousness), and
• interpersonal outcomes (e.g. teamwork, collaboration, leadership).

After thoroughly reviewing all the data, they concluded that digital games did enhance student learning, relative to the non-game control conditions. Furthermore, games that had been specifically designed or augmented to support learning were slightly better than standard versions of those games, and learning outcomes were better when students had multiple game-play sessions. So yes, digital games can support learning, but the authors themselves conclude that “the important question is not if but how games can support learning” (Clark et al., 2014, p. 14, emphasis added). For example, what are the design features of games that are particularly good for particular kinds of learning? How, and how often, can or should gameplay be used to support those kinds of learning? What else, besides playing the games, can help develop or strengthen the learning?

Does game-based learning support engagement and learning?

A second systematic review by Abdul Jabbar and Felicia (2015) [2]  evaluated 91 studies involving learners aged between 8 and 14, and included many kinds of game-based learning (GBL), including digital and non-digital games. The authors were particularly interested in looking at the various design features of games to see whether they could connect particular features to enhanced learner/player motivation, engagement, and learning.

The authors note that their analysis was very difficult because game-based learning is such a diverse field of study. The most clear-cut finding they produced was that “there are no specific rules” to what makes a game or gameplay engaging and motivating for learners. They also noted (no surprises here?) that engagement with a game has something to do with the individual person, as well as having something to do with the design features of the games themselves, and that students’ cognitive and emotional involvement in gameplay is part of what leads to their engagement and learning. “Effective” game-based learning seems to happen when “[players act] as enthusiastic, confident, and strategic learners to access and understand content and to achieve their goals, triggered and supported by multiple elements” (Abdul Jabbar & Felicia, 2015, p. 767).

Based on their review, the authors make some general recommendations for the design of game-based learning activities, including:

• Game design must be accompanied with multiple learning tools and interesting tasks and materials that facilitate and help students to explore and complete gaming and learning activities in accordance with their needs and abilities.
• GBL activities should be matched to students’ abilities, interests, game type preferences, providing just the right amount of challenge, keep players focussed, not too frustrated, and give them opportunities to see what they’re succeeding at, and so on.
• Gameplay should be supported with appropriate feedback and scaffolding (which can take various forms, depending on students’ learning requirements).

If you’re starting to think that every systematic review and meta-analysis of game-based learning might conclude with a variation on “Yes, studies suggest games can improve learning, but it depends on a range of contextual factors including…and other questions we ought to look into are….”, then that makes two of us. To bring it home, let’s look at the findings of two more systematic reviews. What do you think they might find?

Does gamification work?

A third study by Hamari et al. (2014) [3] involved a meta-analysis of studies involving gamification of learning – that is, learning which is designed using game-like elements, to “invok[e] the same psychological  experiences as games (generally) do”. Their meta-analysis of 24 studies concludes that “gamification provides positive effects [on learning], however, the effects are greatly dependent on the context in which the gamification is being implemented, as well as on the users using it”. 

Do digital simulations improve learning?

A fourth systematic review by D'Angelo et al. [4] took a more narrow focus, looking not at games per se but computer simulations that lay somewhere between simple visualizations and full-blown digital games, and their effect on student learning in STEM (science, technology, engineering, and mathematics). Reviewing 59 studies, they found that computer-based simulations tended to lead to better achievement compared with non-simulation instruction, and that simulations supplemented or modified with something else (e.g., simulation plus scaffolding) provided modest improvements in learning over non-modified simulations.

While their meta-analysis only included studies with quantitative data that could be used to calculate the “effect size” of using simulations (versus not using them), the authors noted that additional qualititative information gathered in some of  the studies  – but not looked at in the review – included “a vast amount of evidence that could be used to better understand and support the findings reported in this study”.

Conclusion

I said earlier that I have some problems with the systematic review and meta-analysis methodology, perhaps by now it might be clear why. The approach originates in the field of clinical medicine, but has spilled over into other fields of professional practice including education and social development.[5] Specific criticisms of this approach in the latter fields include: that it prioritises certain research methodologies above others (often based on the “gold standard” of the randomised controlled trial in medicine), that it is unsuitable for research involving complex interventions with multiple outcomes, and that it does not allow sufficient room for theory to play a role.[6] 

When it comes to research on games for learning, my main problem with the systematic reviews and meta-analyses cited above is that they tend to start by focussing on “the games” (or gamification, or simulations, or game-based learning) and their effectiveness for learning, rather than focussing on the diverse contexts in which those games are being played, used, talked about, and experienced. Yet each systematic review seems to conclude that these diverse contexts (with all their complex, messy variability)  might be where the real juicy goodness lies.

In fact, the use of games in and for learning is full of complex, messy variability. It’s in the diversity of kinds of games and game-forms that could be used for learning. It’s in the infinite number of ways that teachers and learners might use games in a classroom setting. It’s in the vast array of learning goals and intentions that might underpin teachers’ use of games, and in the huge range of things that one might learn from games (including learning that wasn’t necessarily planned for). It’s in the diversity amongst people, and how individual engagement, enjoyment, or learning from any particular kind of game might vary according to individual tastes, capabilities, and so on.  In my view, the best way to deal with all this complex variability is to dive right in and embrace it. And in order to make sense of it all, we need to have some good theories to guide us.

That’s what I’ll start looking at in forthcoming blogs. Instead of the question “Do games help learning?”, I’ll consider “Which theories suggest games might help learning?” and "How can these theories help us plan for learning with games in our schools and classrooms?"

References

1. Clark, D., Tanner-Smith, E., and Killingsworth, S. (2014). Digital games, design, and learning: A systematic review and meta-analysis (Executive Summary). Menlo Park, CA, SRI International. 

2. Abdul Jabbar, A. and Felicia, P.  (2015). 'Gameplay engagement and learning in game-based learning: A systematic review'. Review of Educational Research. 85: 740-779.

3. Hamari, J.,  Koivisto, J., and Sarsa, H.  (2014). Does Gamification Work? - A Literature Review of Empirical Studies on Gamification. Proceedings of the 47th Hawaii International Conference on System Sciences, Hawaii, USA, January 6-9, 2014.

4. D’Angelo, C.,  Rutstein, D., Harris, C., Bernard, R., Borokhovski, E., and Haertel, G. (2014). Simulations for STEM Learning: Systematic Review and Meta-Analysis. Menlo Park, CA, SRI International. 

5. See Solesbury, W. (2001). Evidence-based policy: whence it came and where it's going. Queen Mary University of London: ESRC UK Centre for Evidence Based Policy and Practice. Online

6. See Boaz, A., Ashby, D., & Young, K. (2002). Systematic reviews: What have they got to offer evidence based policy and practice? Queen Mary University of London: ESRC UK Centre for Evidence Based Policy and Practice.