You are here

English-language learners and validity

Charles Darr
Journal issue: 

English-language learners and validity

CHARLES DARR

Introduction

Susan Tung and Margaret Kitchen (Tung with Kitchen, this issue) argue that results from a standardised test, such as the Progressive Achievement Test of Mathematics (PAT: Mathematics) have limited validity for students who are English-language learners (ELLs). In this edition of Assessment News I look at the issue of validity and how it applies to using a test like PAT: Mathematics with ELLs. I make five main points:

1.Language complexity can affect the validity of test results for all learners, especially ELLs.

2.While PAT: Mathematics was carefully developed to minimise language demands, the test questions involve applying mathematical ideas in context, necessitating a degree of English comprehension and New Zealand cultural knowledge.

3.PAT: Mathematics can have a range of uses, and some uses are potentially more valid for ELL students.

4.Basing National Standards reporting solely on a standardised test result is not appropriate for ELLs or other learners.

5.Both test developers and teachers have a role to play in ensuring the valid use of test results.

What is validity?

At its heart, validity is an evaluation of the extent to which we can justify the inferences and decisions we make on the basis of test results (Kane, 2006). The important point here is that validity depends on how we use test results: the same set of test results can have high, moderate, or low validity depending on what they are being used for.

Test developers and teachers both have an important part to play when it comes to validity. Tests should be developed so that they are optimal for the uses envisaged and the range of learners who might be assessed. This includes providing advice and guidance related to the valid use of tests. Tung and Kitchen (this volume) note that direct advice related to the use of the test with ELL students is lacking in the PAT: Mathematics teacher manual.

Teachers should make professional judgements about the valid use of tests. This involves having a strong understanding of the test they intend to use, a clear idea of what the test results will be used for, and a knowledge of the learner who will be tested, including their level of language competency.

Understanding an assessment tool such as PAT: Mathematics involves being familiar with the test materials, including the style of questions, the learning concepts covered, and the use of language. The test manual should be read, and the measures reported by the tool understood by the teacher, so that results can be used to make appropriate inferences and decisions.

The influence of language on validity

Research indicates that language factors do affect test performance, particularly for ELLs (Abedi, 2006). Altering the wording of test items can change the difficulty of the items without necessarily changing what the item is fundamentally testing. Abedi (2006) notes:

rewording a verbal problem can make semantic relations more explicit, without affecting the underlying semantic and content structure; thus the reader is more likely to construct a proper problem representation and solve the problem correctly. (p. 380)

Variables that affect test performance, but which have nothing to do with the focus of the assessment, create what is known as construct-irrelevant variance. In a mathematics test, unnecessary language complexity will introduce construct-irrelevant variance when a learner’s ability to comprehend the question interferes with their ability to demonstrate mathematical knowledge and skill.

How does language complexity affect comprehension?

Table 1 draws on Abedi (2006) to list eleven linguistic features that can affect learners’ ability to comprehend test questions. These kinds of features can slow students down, lead them to misrepresent problems and introduce cognitive load. Cognitive load refers to the amount of information a student must keep in working memory while solving a problem. Increased cognitive load can affect a student’s ability to co-ordinate the multiple steps that might be needed to solve a problem efficiently. Language features such as the ones listed can affect any student who struggles to make meaning from written text. ELL students are particularly susceptible.

Language and PAT: Mathematics

PAT: Mathematics was developed with an expectation that most questions would present students with a contextualised mathematical problem. That is, part of the demand of PAT: Mathematics items would involve students having to make sense of a problem situation in order to choose and apply appropriate mathematical strategies and knowledge. This is a fundamental idea in the way The New Zealand Curriculum (Ministry of Education, 2007) presents mathematics as a learning area. Students are expected to apply mathematical ideas in meaningful ways and to use mathematics and statistics to model situations.

Care was taken within the PAT: Mathematics development process to ensure that the language used in the test was straightforward and the contexts chosen accessible. Examination of language features, such as sentence length and vocabulary, were part of a careful review process. Where appropriate, pictures were used to illustrate the context and support comprehension. The review process involved multiple reviews of the test items, including a cultural review that looked at how accessible contexts and wording were for different cultural groups. Each item was piloted with groups of students and trialled with nationally representative samples. Only items that performed satisfactorily were selected for inclusion in the test.

While processes like these can go some way to minimising language complexity, they cannot control it completely for all learners. For many ELLs the language of the test will be a confounding factor affecting a teacher’s ability to use the test results to make appropriate inferences and decisions about their achievement levels. Table 2 lists some test uses where PAT: Mathematics results for ELLs are likely to have reduced validity compared with the results for first-language speakers who are competent readers. In these situations a PAT: Mathematics result may underestimate what the ELL is capable of.

TABLE 1. LINGUISTIC FEATURES WHICH CAN MAKE TEST ITEMS MORE COMPLEX

Use of low-frequency vocabulary
Word length
Sentence length
Use of passive voice
Long question phrases
Use of prepositional phrases
Use of noun phrases with several modifiers
Use of subordinate phrases
Use of abstract terms
Use of negations in sentences
Complex sentence structure

TABLE 2. USES OF PAT: MATHEMATICS THAT ARE LIKELY TO BE LESS VALID FOR ELLS



Use of the test Threat to validity for ELLs
Inferring or making decisions about a student’s development of mathematical ideas. Inclusion of contexts and language that could be unfamiliar. Students may have been able to select and apply appropriate mathematics if they could comprehend the question, or if they had more time to comprehend the question.
Comparing a student’s understanding of mathematics with that of other students. An ELL’s ability to meet the mathematical demands of a question may have been confounded by the language used. Other students’ ability to use their mathematical skills and knowledge may not have been confounded with their ability to unpack the language of the questions.
Comparing a student’s understanding of mathematics with a standard or a criterion. Difficulties comprehending questions may not allow the ELL’s knowledge of mathematics to be compared with the mathematical competencies described by the criterion.

These threats to validity mean that using a standardised test result to make important decisions about the mathematical competencies of ELLs can be problematic. It would be unwise, for instance, to use a result from PAT: Mathematics to determine how well an ELL is doing against the mathematics National Standards. Given the breadth of the National Standards and the level of precision of a single test result, this point is true for all learners, not just ELLs.

Tung and Kitchen (this volume) suggest that in the current National Standards context schools use normed and standardised assessment tools with ELL students to report against National Standards. Using normed and standardised tests is not a legislated requirement and schools can and should make appropriate professional decisions about how, when, for what purpose, and for whom they use standardised tests.

When might you use a test like PAT: Mathematics with an English-language learner?

When a teacher is aware of the validity issues, knows the student well and can control how a test such as PAT: Mathematics will be used, a standardised test may still prove useful as an assessment tool for ELLs. For instance, using a test such as PAT: Mathematics might provide insights into:

the kinds of words and phrases that are creating comprehension issues, including important mathematical terms that the student will need to master over time

the extent to which known mathematical competencies are blunted by written language demands, for instance where a student can operate with the mathematics when the problem is presented as an equation or in their first language

the extent to which the student is progressing in their ability to solve mathematical problems presented in an English-language context—over time, as the learner’s ability to read and comprehend questions written in English improves, we would expect to see their PAT score rise.

These kinds of uses of PAT: Mathematics will often involve looking at ELL students’ performance on individual items, or interacting with each student as they do the test, or both. When using scale scores or norms, teachers will need to take into account the amount and nature of any interaction that occurs when interpreting the results.

Conclusion

All assessment tools and approaches have strengths and weaknesses. This is one of the reasons why schools and teachers in New Zealand are expected to use professional judgement about when and how different assessment approaches will be used. A critical part of this is being able to make judgements about validity.

Tung and Kitchen’s article is a reminder for test developers and teachers to take validity issues for all learners, including ELLs, seriously. Developers should use processes which lead to tests that are as inclusive as possible, and they should provide clear information about how the test can be used and with whom. Teachers should feel empowered to make professional judgements to decide for whom, how, when, and for what purpose different assessment tools should be used.

References

Abedi, J. (2006). Language issues in item development. In S. M. Downing & T. M. Haladyna (Eds), Handbook of test development (pp. 377–398). London: Lawrence Erlbaum.

Kane, M. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (pp. 1–16). Westport, CT: Praeger.

Ministry of Education. (2007). The New Zealand curriculum. Wellington: Learning Media.

Tung, S. (with M. Kitchen). (2013). English-language learners and the Progressive Achievement Test of Mathematics. set: Research Information for Teachers, 2, 51–58.

CHARLES DARR is manager of assessment design and reporting at the New Zealand Council for Educational Research.

Email: charles.darr@nzcer.org.nz