You are here

Report card for integrated learning systems (SuccessMaker): Is there evidence of improved literacy and numeracy outcomes?

Judy M. Parr
Abstract: 

The integrated learning system SuccessMaker is in many New Zealand schools. However, the author’s review of these systems concludes that few have proven effective in producing substantial gains in reading and language; gains in certain areas of maths do not appear to be transferred to other tasks; and system-generated progress data requires validation. Autonomous learning is not viable for many students, and curriculum integration is also an issue.

Journal issue: 

Report card for integrated learning systems (SuccessMaker):

Is there evidence of improved literacy and numeracy outcomes?

Judy M. Parr Image School of Education, University of Auckland

For a variety of reasons, including political and economic ones, computers are in our schools. Yet research has been unable conclusively to link their appearance with enhanced learning outcomes. In fact, compared to many other school-based factors and educational innovations, such as bilingual programs, peer tutoring and parent involvement, overall, the use of computers ranks below average in enhancing learning (e.g. Hattie, 1999).

However, the use of computers in educational settings can take many forms, and some are more effective in terms of learning than others. Recently, one particular form of software has become a major spending category for educational technology. Integrated learning systems are in over a fifth of US elementary schools and account for about half the current dollar investments in educational software (Bailey, 1993).

An integrated learning system (ILS) delivers a curriculum content through modules providing tutorial, practice and assessment, with coverage across a range of curriculum subjects and levels of ability. It includes a student record system and an extensive management system that interprets student responses to a task, updates records, chooses pathways through the curriculum content, delivers appropriate sequences of learning modules, and provides feedback to students and teachers about level and rate of achievement on tasks in a particular module (Brown, 1997).

An ILS called SuccessMaker is now in a number of New Zealand schools. As in other countries, these are often schools where students are failing to master basic maths and language skills during the primary years (Becker, 1994).

Evidence of learning outcomes from ILS

In a recent report to the Ministry of Education (Parr, 2000), the author reviewed the international evidence on the effectiveness of ILSs with respect to literacy and numeracy outcomes. (The report also touched on alternative interventions and cost effectiveness, but these aspects are not discussed here.) It covered the findings of large-scale studies from the United States and the United Kingdom. The differences among ILS products, learning contexts, modes of integration and research designs make the interpretation and comparison of various study findings complex. In the review of the international literature, a deliberate decision was taken not to consider attitude when assessing an ILS in relation to learning outcomes. There is a simple reason for this. According to Wood, Underwood and Avis (1999), the positive responses of teachers and the fact that students enjoy working on an ILS bear no necessary relation to outcome data. They consider such ‘consumer satisfaction’ measures to be very unreliable indicators of effectiveness in enhancing learning.

Three major US studies focus on whether an ILS is useful in boosting literacy and numeracy outcomes. Becker (1992a) reviews the findings of over 30 different studies of ILS, 11 of which were of SuccessMaker. He concluded that some of the latter studies substantially over-report effectiveness. In some cases this is because the studies fail to control for a general upward drift in standardised test scores. Others employ non-standard and upward biasing statistical procedures. The outcomes from studies supplied by the vendor were substantially better than those conducted by local education authorities or independent researchers. Becker stresses the need to be wary of analyses that incorporate studies of widely varying quality without any reanalyses, and not to accept uncritically the studies supplied by vendors.

A second, more recent, large-scale evaluation of an ILS was done in New York with the Waterford Integrated Learning System (WILS)(Miller, 1997). This evaluation concluded that there was no sustained, significant effect on student test scores (Finn, Stevens, Stufflebeam & Walberg, 1997). The third study was an ILS initiative in West Virginia known as the Basic Skills/ Computer Education program (Mann, Shakeshaft, Becker & Kottkamp, 1999). It found that use of the ILS accounted for 11% of the total variance in the basic skills achievements of the students.

In the United Kingdom, several independent research teams conducted a large three-phase study of ILSs with different criteria for assessing outcomes across the phases (BECTA, 1998). SuccessMaker was used by about 60% of the schools. In the early phase, where the assessment outcome measures were focused on basic or core knowledge and skills in English and maths, the most general finding was that SuccessMaker had significant effects on achievements in basic maths skills (NCET, 1994) and no clear effect on English skills. However, in phases one and two, there was no assessment of more general conceptual understanding or knowledge application.

Phase three examined how the results from ILS generalised to different performance measures, namely, standardised tests and external examinations. In terms of performance on standardised tests, Year 5 SuccessMaker students performed the same as controls. In Year 6, the ILS students performed marginally better in maths, but not in English. Looking at exam performance for Year 9 students, ILSs were found to have small, but consistent and significant, negative effects on achievement. Overall, the predictive power of the ILS system-generated measures of performance to external measures of achievement was weak (BECTA, 1998).

The major discrepancy across the phases in the British study concerns the effect of SuccessMaker on maths. The conclusion was that it was ‘good at supporting the learning of basic or core knowledge, skills and procedures [e.g. arithmetic procedures, calculating decimals and fractions and measurement] but less effective in promoting the knowledge and practices that go into the interpretation or translation of tasks into recognisable problems for solution’ (BECTA, 1998, p.20). In the British studies, there was evidence of significant differences across schools, although the data do not provide any conclusive evidence about factors that might explain this.

Implications in considering the use of an ILS such as SuccessMaker

The general pattern of results from these international studies has several implications in considering the use of SuccessMaker in New Zealand. Data about the preconditions for effective use of SuccessMaker was obtained through semi-structured interviews with 14 of the 17 New Zealand primary schools funded for SuccessMaker through the Ministry of Education’s special funding available for literacy and numeracy initiatives in low decile schools.

1.&&In the reading and language area, independent evaluations in the UK (eg. BECTA, 1998) conclude that SuccessMaker does not seem to have produced worthwhile learning gains. However, it is not alone. When various computer-based reading programs are compared with alternative programs in terms of outcomes and cost effectiveness, the conclusion is that few have proven to be effective in producing substantial gains. Most ILSs are expensive and the return in terms of achievement is ‘modest at best’ (Schacter, 1999). Schacter’s analysis shows a number of school-wide reading programs to be both more effective and cheaper than computer-based reading programs.

2.&&In maths, there is clear evidence of gains from SuccessMaker in certain areas, notably those relating to basic knowledge and skills. But maths learnt on the program does not appear to be transferred to curriculum based assessment tasks.

The British research team used local curriculum and national assessment tests as, presumably, valid measures of what children have learned within their national curriculum. There was no evidence that an ILS was effective in supporting the development of numeracy or literacy using such measures.

In attempting to explain this, the U.K. evaluation team suggested that there may be a lack of curriculum validity in ILSs, which becomes more pronounced at later stages than early on, when core skills are being acquired. Alternatively, it could be that the basic skills are curriculum appropriate, but additional teaching and learning is necessary to make them relevant and applicable to situations where knowledge has to be applied, or where a problem is not well defined and requires conceptualisation. It seems as if the type of knowledge and skills taught and the way they are assessed on SuccessMaker and other ILSs may not match what the curriculum values and the way this is assessed.

For Becker (1992b), the lack of social interaction or meaningful interaction with the materials is such that genuine learning is inhibited. For Wood, Underwood and Avis (1999), learners seem not to gain conceptual understanding of the procedural skills they acquire because goals such as flexibility of reasoning or problem-solving skills, with more than one system of signs, are not the goals of the designers of current ILSs. It is interesting that more open-ended and problem solving use of computers, such as in the project Computer Supported Intentional Learning Environments (Bereiter & Scardamalia, 1996), has consistently shown learning advantages on standardised measures of reading, language and vocabulary, as well as on other measures such as depth of understanding and reflection.

3.&&The demonstrated lack of generalisability of ILS results gives rise to a cautionary note about using the normative data generated by the ILS as a more general, developmental measure of achievement. Like the use of behavioural or attitudinal indicators as evidence of effectiveness, the use of program-generated data is also an inaccurate way to measure generalised learning outcomes.

However, it was these two indicators – program-generated data and attitudinal or behavioural indicators – that the majority of New Zealand schools in the report used to form their largely positive judgement of SuccessMaker. At this point, only one school had assessed progress by using standardised tests, external to the program, and their evaluation was not positive. Generally, schools used informal observation – either the personal observation of key personnel involved, or teacher reports to this person – as well as data obtained from the ILS management system.

One school questioned, quite rightly, the real meaning of the system-generated data. This consists of gains data (months of school learning per hour on the program) and outcomes data (percentage of correct answers, or a diagnostic profile of strengths and weaknesses). The school questioned how the gains were worked out, how accurate they were, and, most importantly, how they related to the curriculum.

The assessment tasks of SuccessMaker, for example, indicate only mastery of what the program teaches and of how it is taught. Mastery of these may or may not indicate progress on important curricular goals. The progress indicated from data generated by SuccessMaker needs to be validated in terms of the New Zealand curriculum by other, widely accepted measures of progress. Research (e.g. Wood, Underwood & Avis, 1999) goes further and questions the model the program uses to characterise learner knowledge.

4.&&Other related findings from the research also have significant implications. One finding is that outcomes are variable; another is that particular conditions and circumstances of the context of use may be important.

Preconditions for effective use of an ILS

The preconditions for effective use of ILSs were a major focus of the interviews with New Zealand schools. An overarching theme was the need for some other intervention, notably by teachers, to ensure learning. Most key personnel interviewed stressed that SuccessMaker is not a stand-alone intervention, and that it could not be maximally effective without both skilled involvement in and oversight of the project, and without personal assistance being available to students as they worked. Considerable time and knowledge were seen as necessary to ‘utilise, assimilate and unpack’ the SuccessMaker software in order to gain the most from it. In these responses emphasising the centrality of personnel, several interlinked ideas were operating.

Teacher help for students

The first idea counters the simplistic notion (often found in marketing) that a program such as an ILS allows autonomous tutoring and individual pacing, with the student interacting with the machine and the content. Only a few New Zealand schools ran the program without a person dedicated to interacting with the children working on SuccessMaker. Even in those few schools, help was usually available nearby or on call.

Skilled personnel were seen as necessary to place children accurately, monitor their progress, and make the necessary adjustments to tailor the program to the needs of the individual. This aspect also included making decisions about which students to place on the program. As a number of schools noted, SuccessMaker ‘works for some’; however, these may not be the children who are, for example, more than two years behind their peers, or those with learning difficulties.

Teacher help for teachers

Another idea countering any notion of SuccessMaker being a stand-alone intervention focused on the need for an experienced teacher to assist with integration and work with other teachers to see where SuccessMaker materials may fit with classroom units of work. The maximum effect is gained if there is a connection between work on the ILS and classroom work.

However, school opinion was divided regarding integration. Although teachers did not generally see SuccessMaker as standing alone in terms of the student operating independently, many did see it as standing alone in terms of the curriculum. This was because ‘it was not built on the classroom program’ or ‘because the links to the curriculum were not there’.

Many teachers did not raise the issue of match or integration as a concern, because they saw the function of SuccessMaker as providing the ‘very basic, mainline stuff’. However, perhaps because of the form learning takes on an ILS, research shows that generalisation does not readily occur. This means that the teacher needs to work on integrating the different basic skills and helping students to apply them in a broader context.

Autonomous learning with an ILS like SuccessMaker is not viable for most students, as there needs to be a skilled hand operating in placement, monitoring, intervention and adjustment to optimise learning. A theory of learning in which the individual interacts with a resource, such as a book or machine, is no longer accepted as a complete or accurate theory of effective classroom instruction or learning. Instead we now place more emphasis on the socially constructed nature of all learning. Nor is it reflective of more general classroom practice. Successful implementation of an ILS may depend on the extent to which the learning theories guiding a teacher’s classroom practice are consistent with the learning theories adopted by the software developers. The term ‘integrated learning system’ is in fact a misnomer, because it is the teacher, not the program, who accomplishes integration with the classroom curriculum and pedagogy.

References

Bailey, G.D. (1993). Computer-based integrated learning systems. Englewood Cliffs, NJ: Educational Technology Publications.

Becker, H.J. (1992a). Computer-based integrated learning systems in the elementary and middle grades: A critical review and synthesis of evaluation reports. Journal of Educational Computing Research, 8, pp.1-41.

Becker, H.J. (1992b). A model for improving the performance of integrated learning systems: Mixed individualized/group/whole class lessons, cooperative learning, and organizing time for teacher-led remediation of small groups. Educational Technology, 32, pp.6-15.

Becker, H.J. (1994). Mindless or mindful use of integrated learning systems. International Journal of Educational Research, 21, pp.65-79.

BECTA. (1998). The UK ILS evaluations: Final report. Coventry, UK: British Educational Communications and Technology Agency.

Brown, J. (1997). When is a system an ILS? In J. Underwood & J. Brown (eds), Integrated learning systems: Potential into practice. London: Heinemann.

Finn, J.C.E., Stevens, F.I., Stufflebeam, D.L., & Walberg, H.J. (1997). A meta-evaluation. International Journal of Educational Research, 27, pp.159-174.

Hattie, J. (1999). Influences on student learning. Paper presented at Inaugural Professorial Lecture, University of Auckland, Auckland, New Zealand.

Mann, D., Shakeshaft, C., Becker, J., & Kottkamp, R. (1999). West Virginia’s basic skills/ computer education program (BS/CE): An analysis of student achievement. Santa Monica, CA: Milken Family Foundation.

NCET. (1994). Integrated learning systems: A report of the pilot evaluation of ILS in the UK. Coventry, UK: National Council for Educational Technology.

Parr, J.M. (2000). A review of the literature on computer-assisted learning, particularly integrated learning systems, and outcomes with respect to literacy and numeracy. Report to the Ministry of Education. Auckland: University of Auckland.

Scardamalia, M., & Bereiter, C. (1996). Engaging students in a knowledge society. Educational Leadership, 54, pp.6-10.

Schacter, J. (1999). Reading programs that work: A review of programs from kindergarten to 4th grade. Santa Monica, CA: Milken Family Foundation.

Wood, D., Underwood, J., & Avis, P. (1999). Integrated learning systems in the classroom. Computers and Education, 33, pp.91-108.

DR JUDY PARR is a senior lecturer in the School of Education at the University of Auckland. Her research has been largely concerned with aspects of literacy, principally the development of writing, but also with technology, in particular the relationship between literacy and technology.

Email: jm.parr@auckland.ac.nz