ngä taikäkä – Seek the heartwood:
Issues of validity in translating NEMP assessment tasks
of the National Education Monitoring Project (NEMP) is to provide
a national picture of what New Zealand children know and can do.
The project team consults widely and works with curriculum advisory
panels to develop and select assessment tasks that focus on important
dimensions or the ‘heartwood’ of curriculum areas.
Since 1999, NEMP has also been assessing students in te reo Mäori
mainly using tasks originally developed to be administered in
English. The transfer of assessment tasks to Mäori has raised
a number of issues.
It has not been possible to bring a blanket solution to some of
these problems. Within each curriculum area different issues and
concerns have arisen. This paper examines the process of NEMP
task development in four different learning areas and analyses
some of the issues considered in their development for mainstream
students and the subsequent transferral to Mäori Immersion
Educational Assessment, Translation, Test Validity
“Assessment is an inexact matter” (Harlen, 1994) and much
of the work of those involved in developing assessment tasks centres around
refining the inexact tools. The inexactness of the assessment process
is exacerbated when tasks that have been developed in English and within
a mainstream educational climate are translated and transferred to Mäori
Immersion settings. In the four years that assessments in te reo Mäori
have been undertaken by NEMP, there have been many lessons learnt by task
developers and reviewers. This paper examines the process used by the
National Education Monitoring Project (NEMP) in the development, translation
and use of assessment tasks in Mäori Immersion settings. The “learning
journeys” that have occurred as four different tasks have been developed
are described, and some of the issues considered are discussed.
EDUCATION MONITORING PROJECT
New Zealand’s National Education Monitoring Project (NEMP) commenced
in 1993 with the task of assessing and reporting on the achievement
of primary school children in all areas of the curriculum. Since 1999,
parallel assessments of samples of Year 8 students have been conducted
in te reo Mäori. NEMP provides a national ‘snapshot’
on “how well overall national standards are being maintained,
and where improvements might be needed” (Ministry of Education,
1993a). The assessing and reporting procedures used by NEMP are designed
to provide a rich picture of what children can do, resulting in a detailed
national picture of student achievement. A number of the procedures
used by NEMP to gather this rich data are unique for a national assessment
NEMP aims to address coverage of the National Curriculum Framework over
a four-year cycle, rather than restrict itself to pre-selected priority
areas such as mathematics or literacy. The first cycle of assessments
began in 1995 and was completed in 1998, the second cycle running from
1999 to 2002. A third cycle will begin in 2003. The assessment tasks
are designed to emphasise aspects of the curriculum that are particularly
important to life in the community. They endeavour to “achieve
a balanced coverage of important skills, knowledge and understandings
within the various curriculum strands, but without attempting to slavishly
follow the finer details of current curriculum statements” (Crooks
& Flockton, 2002)
Care is taken to use tasks and approaches that interest and motivate
students and a variety of assessment task formats are used in order
to get the broadest possible coverage of learning outcomes. The assessment
task formats used include:
interviews: students work individually with a teacher, the
session being recorded on videotape.
Stations: Four students, working independently, move
around a series of stations where tasks have been set up. This session
is not videorecorded.
Team: four students work collaboratively, supervised
by a teacher, on group activities; the session being recorded on videotape.
Paper and pencil activities: students work independently
on paper and pencil activities, that is; short answer, extended written
responses or multiple choice questions.
students work independently on practical activities such as making
art works or physical performances for physical education.
marking or scoring of student work is undertaken while the tasks are
being administered, but all work is returned to the NEMP office for
marking by senior tertiary education students and teachers. In this
manner, a considerable amount of information can be gathered without
placing too many demands on individual students, different students
attempt different tasks. The students selected in the main sample are
divided into three groups. The immersion students are divided into two
groups. These two groups work in Mäori, with two of the sets of
tasks used in the main sample.
Each year, random samples of students are selected nationally at two
class levels: Year 4 (8 - 9 years old) and Year 8 (12 -13 years old).
The main national samples (approximately 1440 children at each Year
level) represent approximately 2.5% of the children at those levels
in New Zealand schools. Additional samples of 120 children at each level
allow the achievement of Pacific students to be assessed and reported.
Since 1999, at the Year 8 level only, a special sample of 120 children
learning in Mäori immersion settings is selected to take part in
assessments conducted in te reo Mäori. About 60% of this sample
is drawn from immersion schools (mainly Kura Kaupapa Mäori), while
the other 40% are learning in immersion classes (located in mainstream
schools, but having upwards of 80% of instruction conducted in Mäori).
Their achievement is then compared with the achievement of Mäori
students in general education and the results given in a separate report
each year. The inclusion of assessment tasks in te reo Mäori in
NEMP has not been unproblematic. The procedures and practices used have
been, and continue to be, scrutinised and refined. The 1999 NEMP assessments
are believed to be the first assessments conducted at a national level
using tasks originally developed to be administered nationally in English.
Some significant difficulties were experienced in that first year and
substantial improvements to the sampling, translation and assessment
procedures were implemented in 2000. The process has continued to be
evaluated and refined.
The assessment tasks used in NEMP come from a variety of sources. They
can be developed from ideas proposed by teachers participating in regional
task development workshops, by the curriculum advisory panels that are
convened for each curriculum area, from a review of national and international
assessment materials, or developed by NEMP staff. A small proportion
of the assessment tasks used each year (for both assessments conducted
in English and those conducted in Mäori) are developed from ideas
proposed by educators working in Mäori Immersion education as being
particularly appropriate for these children.
The initial task ideas are developed and trialled by NEMP staff then
subjected to careful scrutiny by the advisory panel for that particular
area each of which includes at least one Mäori Immersion educator.
All the tasks are then further scrutinised by those attending a combined
meeting of the NEMP Mäori Immersion Education Advisory Committee
and the NEMP Mäori Reference Group (the latter focussing on the
interests of Mäori students who will be assessed in English).
In 1999, tasks were translated by a group of translators working independently
of each other and NEMP staff. A task was translated from English to Mäori
by one translator, then back-translated (from Mäori to English) by
another translator. Congruence between the two English versions was then
checked. At the end of this process, the English and Mäori versions
were sent to Te Taura Whiri (The Mäori Language Commission) for checking
and guidance on improving. After the assessment took place, concern was
raised that the Mäori version of the tasks used language more appropriate
for adults than children; using more words and being linguistically more
complex. The translation process was therefore altered, and for subsequent
translations, six translators (working in two teams of three) have worked
in the NEMP office, able to consult with NEMP staff. A process of back-translation
between the two groups, with overview from senior translators within the
team has incorporated the need for ensuring that the language is more
natural and child-focused. After the initial translation, the tasks are
trialled in a Kura Kaupapa Mäori, and further adjustments made if
It also became apparent as the assessments were being conducted in 1999
that a limited understanding of te reo Mäori affected the performance
of at least 30 percent of the students. From 2000 on, only students reported
by their schools to have completed five or more years in Mäori Immersion
education are included in the sample. International research (for example,
Cummins, 1984; Lacelle-Peterson, 2000) has suggested that at least five
years of immersion in a language is required before performance on assessments
in that language is not significantly undermined by language difficulties.
NEMP is committed to providing all students every opportunity to perform
to their best ability (Flockton, 1999). Tasks are rigorously scrutinised
during the process of development and trialling to ensure they relate
strongly to student experience, have a high level of interest for the
student, are based in authentic contexts, and that they allow engagement
of all students, regardless of ability. Considerable effort is spent in
shaping the language of the task in order that it be readily understood.
Particular attention is given to the specific wording of questions. Clearly,
these issues are become even more important when the assessment of students
in Mäori Immersion settings are considered.
This paper examines some of the issues that are considered in each stage
of the development of a task. Four tasks are reported here: a speaking
task, a technology task, a science task and a social studies task. In
this reporting, some of the issues that have been raised, and NEMP’s
attempts to address them, are discussed.
of page | return
to Probe Studies - INDEX | return
to Probe Studies menu
further information and contact details for the Author
| Contact USEE