Kohia ngä taikäkä – Seek the heartwood:
Issues of validity in translating NEMP assessment tasks
 

The purpose of the National Education Monitoring Project (NEMP) is to provide a national picture of what New Zealand children know and can do. The project team consults widely and works with curriculum advisory panels to develop and select assessment tasks that focus on important dimensions or the ‘heartwood’ of curriculum areas. Since 1999, NEMP has also been assessing students in te reo Mäori mainly using tasks originally developed to be administered in English. The transfer of assessment tasks to Mäori has raised a number of issues.


It has not been possible to bring a blanket solution to some of these problems. Within each curriculum area different issues and concerns have arisen. This paper examines the process of NEMP task development in four different learning areas and analyses some of the issues considered in their development for mainstream students and the subsequent transferral to Mäori Immersion settings.

Key words:
Educational Assessment, Translation, Test Validity



INTRODUCTION


“Assessment is an inexact matter” (Harlen, 1994) and much of the work of those involved in developing assessment tasks centres around refining the inexact tools. The inexactness of the assessment process is exacerbated when tasks that have been developed in English and within a mainstream educational climate are translated and transferred to Mäori Immersion settings. In the four years that assessments in te reo Mäori have been undertaken by NEMP, there have been many lessons learnt by task developers and reviewers. This paper examines the process used by the National Education Monitoring Project (NEMP) in the development, translation and use of assessment tasks in Mäori Immersion settings. The “learning journeys” that have occurred as four different tasks have been developed are described, and some of the issues considered are discussed.




NATIONAL EDUCATION MONITORING PROJECT


New Zealand’s National Education Monitoring Project (NEMP) commenced in 1993 with the task of assessing and reporting on the achievement of primary school children in all areas of the curriculum. Since 1999, parallel assessments of samples of Year 8 students have been conducted in te reo Mäori. NEMP provides a national ‘snapshot’ on “how well overall national standards are being maintained, and where improvements might be needed” (Ministry of Education, 1993a). The assessing and reporting procedures used by NEMP are designed to provide a rich picture of what children can do, resulting in a detailed national picture of student achievement. A number of the procedures used by NEMP to gather this rich data are unique for a national assessment project.

NEMP aims to address coverage of the National Curriculum Framework over a four-year cycle, rather than restrict itself to pre-selected priority areas such as mathematics or literacy. The first cycle of assessments began in 1995 and was completed in 1998, the second cycle running from 1999 to 2002. A third cycle will begin in 2003. The assessment tasks are designed to emphasise aspects of the curriculum that are particularly important to life in the community. They endeavour to “achieve a balanced coverage of important skills, knowledge and understandings within the various curriculum strands, but without attempting to slavishly follow the finer details of current curriculum statements” (Crooks & Flockton, 2002)

Care is taken to use tasks and approaches that interest and motivate students and a variety of assessment task formats are used in order to get the broadest possible coverage of learning outcomes. The assessment task formats used include:

  • One-to-one interviews: students work individually with a teacher, the session being recorded on videotape.
  • Stations: Four students, working independently, move around a series of stations where tasks have been set up. This session is not videorecorded.
  • Team: four students work collaboratively, supervised by a teacher, on group activities; the session being recorded on videotape.
  • Paper and pencil activities: students work independently on paper and pencil activities, that is; short answer, extended written responses or multiple choice questions.
  • Practical activities: students work independently on practical activities such as making art works or physical performances for physical education.

No marking or scoring of student work is undertaken while the tasks are being administered, but all work is returned to the NEMP office for marking by senior tertiary education students and teachers. In this manner, a considerable amount of information can be gathered without placing too many demands on individual students, different students attempt different tasks. The students selected in the main sample are divided into three groups. The immersion students are divided into two groups. These two groups work in Mäori, with two of the sets of tasks used in the main sample.

Each year, random samples of students are selected nationally at two class levels: Year 4 (8 - 9 years old) and Year 8 (12 -13 years old). The main national samples (approximately 1440 children at each Year level) represent approximately 2.5% of the children at those levels in New Zealand schools. Additional samples of 120 children at each level allow the achievement of Pacific students to be assessed and reported. Since 1999, at the Year 8 level only, a special sample of 120 children learning in Mäori immersion settings is selected to take part in assessments conducted in te reo Mäori. About 60% of this sample is drawn from immersion schools (mainly Kura Kaupapa Mäori), while the other 40% are learning in immersion classes (located in mainstream schools, but having upwards of 80% of instruction conducted in Mäori). Their achievement is then compared with the achievement of Mäori students in general education and the results given in a separate report each year. The inclusion of assessment tasks in te reo Mäori in NEMP has not been unproblematic. The procedures and practices used have been, and continue to be, scrutinised and refined. The 1999 NEMP assessments are believed to be the first assessments conducted at a national level using tasks originally developed to be administered nationally in English. Some significant difficulties were experienced in that first year and substantial improvements to the sampling, translation and assessment procedures were implemented in 2000. The process has continued to be evaluated and refined.

The assessment tasks used in NEMP come from a variety of sources. They can be developed from ideas proposed by teachers participating in regional task development workshops, by the curriculum advisory panels that are convened for each curriculum area, from a review of national and international assessment materials, or developed by NEMP staff. A small proportion of the assessment tasks used each year (for both assessments conducted in English and those conducted in Mäori) are developed from ideas proposed by educators working in Mäori Immersion education as being particularly appropriate for these children.

The initial task ideas are developed and trialled by NEMP staff then subjected to careful scrutiny by the advisory panel for that particular area each of which includes at least one Mäori Immersion educator. All the tasks are then further scrutinised by those attending a combined meeting of the NEMP Mäori Immersion Education Advisory Committee and the NEMP Mäori Reference Group (the latter focussing on the interests of Mäori students who will be assessed in English).


THE TRANSLATION PROCESS


In 1999, tasks were translated by a group of translators working independently of each other and NEMP staff. A task was translated from English to Mäori by one translator, then back-translated (from Mäori to English) by another translator. Congruence between the two English versions was then checked. At the end of this process, the English and Mäori versions were sent to Te Taura Whiri (The Mäori Language Commission) for checking and guidance on improving. After the assessment took place, concern was raised that the Mäori version of the tasks used language more appropriate for adults than children; using more words and being linguistically more complex. The translation process was therefore altered, and for subsequent translations, six translators (working in two teams of three) have worked in the NEMP office, able to consult with NEMP staff. A process of back-translation between the two groups, with overview from senior translators within the team has incorporated the need for ensuring that the language is more natural and child-focused. After the initial translation, the tasks are trialled in a Kura Kaupapa Mäori, and further adjustments made if required.

It also became apparent as the assessments were being conducted in 1999 that a limited understanding of te reo Mäori affected the performance of at least 30 percent of the students. From 2000 on, only students reported by their schools to have completed five or more years in Mäori Immersion education are included in the sample. International research (for example, Cummins, 1984; Lacelle-Peterson, 2000) has suggested that at least five years of immersion in a language is required before performance on assessments in that language is not significantly undermined by language difficulties.

NEMP is committed to providing all students every opportunity to perform to their best ability (Flockton, 1999). Tasks are rigorously scrutinised during the process of development and trialling to ensure they relate strongly to student experience, have a high level of interest for the student, are based in authentic contexts, and that they allow engagement of all students, regardless of ability. Considerable effort is spent in shaping the language of the task in order that it be readily understood. Particular attention is given to the specific wording of questions. Clearly, these issues are become even more important when the assessment of students in Mäori Immersion settings are considered.
This paper examines some of the issues that are considered in each stage of the development of a task. Four tasks are reported here: a speaking task, a technology task, a science task and a social studies task. In this reporting, some of the issues that have been raised, and NEMP’s attempts to address them, are discussed.

next page

top of page    |    return to Probe Studies - INDEX   |    return to Probe Studies menu
  For further information and contact details for the Author    |    Contact USEE