This chapter presents a concise outline of the rationale and operating procedures for national monitoring, together with some information about the reactions of participants in the 2001 assessments. Detailed information about the samples of students and schools is available in the appendix.

Purpose of national monitoring
The New Zealand Curriculum Framework (1993, p26) states that the purpose of national monitoring is to provide information on how well overall national standards are being maintained, and where improvements might be needed.

The focus of the National Education Monitoring Project (NEMP) is on the educational achievements and attitudes of New Zealand primary and intermediate school children. NEMP provides a national “snapshot” of children’s knowledge, skills and motivation, and a way to identify which aspects are improving, staying constant, or declining. This information allows successes to be celebrated and priorities for curriculum change and teacher development to be debated more effectively, with the goal of helping to improve the education which children receive.

Assessment procedures and tasks are selected to provide a rich picture of what children can do and to optimize value to the educational community. The result is a quite detailed national picture of student achievement. It is neither feasible nor appropriate, given the purpose and the approach used, to release information about individual students or schools.

Monitoring at two class levels
National monitoring assesses and reports what children know and can do at two levels in primary and intermediate schools: year 4 (ages 8-9) and year 8 (ages 12-13). Because this report focuses only on year 8 students, no further details of the assessments of year 4 students are included here.

National samples of students
National monitoring information is gathered using carefully selected random samples of students. The main national sample of 1440 year 8 children represents about 2.5 percent of the year 8 children in New Zealand schools. These students are educated predominantly in English. A special sample of 120 year 8 children learning in Mäori immersion schools or classes is also selected. They are educated entirely or predominantly in Mäori. This report compares the achievement of Mäori students in these two samples (educated in English or Mäori).

Three sets of tasks at each level So that a considerable amount of information can be gathered without placing too many demands on individual students, different students attempt different tasks. The 1440 students selected in the main sample are divided into three groups of 480 students, comprising four students from each of 120 schools. The 120 students in the Mäori immersion sample, drawn from 13 schools, are divided into two groups of 60 students. These two groups of 60 students attempt, in Mäori, two of the sets of tasks used with the main sample. The third set of tasks used with the main sample is not used in the Mäori immersion assessments.

Timing of assessments
The assessments take place in the second half of the school year, between August and November. The year 8 assessments in English occur first, over a five–week period. The year 8 assessments in Mäori follow, over a similar period. Each student participates in about four hours of assessment activities spread over one week.

Specially trained teacher administrators
The assessments are conducted by experienced teachers, usually working in their own region of New Zealand. The four teachers working with Mäori immersion students are experienced with such students. All teachers are selected from a national pool of applicants, attend a week of specialist training in Wellington led by senior Project staff, and then work in pairs to conduct assessments of 60 children over five weeks. Their employing school is fully funded by the Project to employ a relief teacher during their secondment.

Four-year assessment cycle
Each year, the assessments cover about one quarter of the national curriculum for primary schools. The New Zealand Curriculum Framework is the blueprint for the school curriculum. It places emphasis on seven essential learning areas, eight essential skills, and a variety of attitudes and values. National monitoring aims to address all of these areas, rather than restrict itself to preselected priority areas.

The first four–year cycle of assessments began in 1995 and was completed in 1998. The second cycle runs from 1999 to 2002. Assessments of Mäori immersion students have been included only from 1999. Similar cycles of assessment are expected to be repeated in subsequent four year periods.

About one third of the tasks (link tasks) are kept constant from one cycle to the next. This re-use of tasks allows trends in achievement across a four–year interval to be observed and reported.

Important learning outcomes assessed
The assessment tasks emphasize aspects of the curriculum which are particularly important to life in our community, and which are likely to be of enduring importance to students. Care is taken to achieve balanced coverage of important skills, knowledge and understandings within the various curriculum strands, but without attempting to slavishly follow the finer details of current curriculum statements. Such details change from time to time, whereas national monitoring needs to take a long-term perspective if it is to achieve its goals.

Wide range of task difficulty
National monitoring aims to show what students know and can do. Because children at any particular class level vary greatly in educational development, tasks spanning multiple levels of the curriculum need to be included if all children are to enjoy some success and all children are to experience some challenge. Many tasks include several aspects, progressing from aspects most children can handle well to aspects that are more challenging.

Engaging task approaches Special care is taken to use tasks and approaches that interest students and stimulate them to do their best. Students’ individual efforts are not reported and have no obvious consequences for them. This means that worthwhile and engaging tasks are needed to ensure that students’ results represent their capabilities rather than their level of motivation. One helpful factor is that extensive use is made of equipment and supplies which allow students to be involved in “hands-on” activities. Presenting some of the tasks on video or computer also allows the use of richer stimulus material, and standardizes the presentation of those tasks.

Positive student reactions to tasks
At the conclusion of each assessment session, students complete evaluation forms in which they identify tasks that they particularly enjoyed and tasks that did not appeal. Averaged across all tasks in the 2001 assessments, 70 percent of the year 8 students in the main sample indicated that they particularly enjoyed the tasks. The students in Mäori immersion settings were even more positive, with 74 percent indicating that they particularly enjoyed the tasks. The students’ parents and teachers also reacted very positively to the tasks and assessment approaches.

Appropriate support for students
A key goal in Project planning is to minimise the extent to which student strengths or weaknesses in one area of the curriculum might unduly influence their assessed performance in other areas. For instance, skills in reading and writing often play a key role in success or failure in paper-and-pencil tests in areas such as science, social studies, or even mathematics. In national monitoring, many tasks are presented orally by teachers, on videotape, or on computer. Similarly, many answers are given orally or by demonstration rather than in writing. Where reading or writing skills are required to perform tasks in areas other than reading and writing, teachers are happy to help students to understand these tasks or to communicate their responses. Teachers are working with no more than four students at a time, so are readily available to help individuals.

To further free teachers to concentrate on providing appropriate guidance and help to students, so that the students achieve their best efforts, teachers are not asked to record judgements on the work the students are doing. All marking and analysis is done later, when the students’ work has reached the Project office in Dunedin. Some of the work comes on paper, but much of it arrives recorded on videotape. In 2001, almost half of the students’ work came in that form. The video recordings give a detailed picture of what both the student and teacher did and said, allowing rich analysis of both process and task achievement.

Four task approaches used
In 2001, four task approaches were used. Each student was expected to spend about an hour working in each format. The four approaches were:

       • One-to-one interview. Each student worked individually with a teacher,
            with the whole session recorded on videotape.
       • Stations. Four students, working independently, moved around a series of stations
           where tasks had been set up. This session was not videotaped.
       • Team. Four students worked collaboratively, supervised by a teacher, on some tasks.
          This was recorded on videotape.
       • Independent. Four students worked individually on some paper-and-pencil tasks.

Professional development benefits for teacher administrators
The teacher administrators reported that they found their training and assessment work very stimulating and professionally enriching. Working so closely with interesting tasks administered to 60 children in at least five schools offered valuable insights. Some teachers have reported major changes in their teaching and assessment practices as a result of their experiences working with the Project.

Marking arrangements
The marking and analysis of the students’ work occurs in Dunedin. The marking process includes extensive discussion of initial examples and careful checks of the consistency of marking by different markers.

Tasks which can be marked objectively or with modest amounts of professional experience usually are marked by senior tertiary students, most of whom have completed two to four years of preservice preparation for primary school teaching. More than 40 student markers worked on the 2001 tasks, most employed 5 hours per day for periods ranging between 5 weeks and 7 weeks. Some of these students were appropriately qualified to mark work presented in Mäori.

The tasks that require higher levels of professional judgement are marked by teachers selected from throughout New Zealand. In 2001, approximately half of the teachers who applied were appointed: a total of 166. Most teachers worked either mornings or afternoons for one week. One to three teachers in each marking group were suitably qualified to mark work presented in Mäori. Teacher professional development through participation in the marking process is another substantial benefit from national monitoring. In evaluations of their experiences on a four point scale (“dissatisfied” to “highly satisfied”),75 to 97 percent of the teachers who marked student work in 2001 chose “highly satisfied” in response to questions about:

       • the extent to which marking was professionally satisfying and interesting;
       • its contribution to professional development in the area of assessment;
       • whether they would recommend NEMP marking work to colleagues;
       • whether they would be happy to do NEMP marking again.

Analysis of results
The results are analysed and reported task by task. Results achieved by the Mäori students in the main sample are compared with results achieved by students in the Mäori immersion sample. Because of the small numbers of students in the latter sample, no analysis by subgroups (such as boys and girls) is included in this report.

Funding arrangements
National monitoring is funded by the Ministry of Education, and organised by the Educational Assessment Research Unit at the University of Otago, under the direction of Associate Professor Terry Crooks and Lester Flockton. The current contract runs until 2003. The cost is about $2.5 million per year, less than one tenth of a percent of the budget allocation for primary and secondary education. Almost half of the funding is used to pay for the time and expenses of the teachers who assist with the assessments as task developers, teacher administrators or markers.

Review by International Scholars.
In June 1996, three scholars from the United States and England, with distinguished international reputations in the field of educational assessment, accepted an invitation from the Project directors to visit the Project. They conducted a thorough review of the progress of the Project, with particular attention to the procedures and tasks used in 1995 and the results emerging. At the end of their review, they prepared a report which concluded as follows:

The National Education Monitoring Project is well conceived and admirably implemented. Decisions about design, task development, scoring, and reporting have been made thoughtfully. The work is of exceptionally high quality and displays considerable originality. We believe that the Project has considerable potential for advancing the understanding of and public debate about the educational achievement of New Zealand students. It may also serve as a model for national and/or state monitoring in other countries.
(Professors Paul Black, Michael Kane & Robert Linn, 1996)

A further review was conducted late in 1998 by another distinguished panel (Professors Elliot Eisner, Caroline Gipps and Wynne Harlen). Amid very helpful suggestions for further refinements and investigations, they commented that:

We want to acknowledge publicly that the overall design of NEMP is very well thought through. … The vast majority of tasks are well designed, engaging to students and consistent with good assessment principles in making clear to students what is expected of them.

Further information   
A more extended description of national monitoring, including detailed information about task development procedures, is available in:  

       Flockton, L. (1999). School-wide Assessment: National Education Monitoring Project.
       Wellington: New Zealand Council for Educational Research.