Main
samples
In
2001, 2869 children from 254 schools were in the main samples
to participate in national monitoring. About half were in year
4, the other half in year 8. At each level, 120 schools were
selected randomly from national lists of state, integrated and
private schools teaching at that level, with their probability
of selection proportional to the number of students enrolled
in the level. The process used ensured that each region was fairly
represented. Schools with fewer than four students enrolled at
the given level were excluded from these main samples, as were
special schools and Mäori immersion schools (such as Kura
Kaupapa Mäori ).
Early in
May 2001, the Ministry of Education provided computer files
containing lists of eligible schools with year 4 and year 8
students, organised by region and district, including year
4 and year 8 roll numbers drawn from school statistical returns
based on enrolments at 1 March 2001.
From these
lists, we randomly selected 120 schools with year 4 students
and 120 schools with year 8 students. Schools with four students
in year 4 or 8 had about a one percent chance of being selected,
while some of the largest intermediate (year 7 and 8) schools
had a more than 90 percent chance of inclusion. In the six cases
where the same school was chosen at both year 4 and year 8 level,
a replacement year 4 school of similar size was chosen from the
same region and district, type and size of school.
Additional
samples
From 1999
onwards, national monitoring has included additional samples
of students to allow the performance of special categories of
students to be reported.
To allow results for Pacific students to be compared with those
of Mäori students and other students, 10 additional schools
were selected at year 4 level and 10 at year 8 level. These
were selected randomly from schools that had not been selected
in the main sample, had at least 15 percent Pacific students
attending the school, and had at least 12 students at the relevant
year level.
To allow results for Mäori students learning in Mäori
immersion programmes to be compared with results for Mäori
children learning in English, 10 additional schools were selected
at year 8 level only. They were selected from Mäori immersion
schools (such as Kura Kaupapa Mäori ) that had at least
4 year 8 students, and from other schools that had at least
4 year 8 students in classes classified as Level 1 immersion
(80 to 100 percent of instruction taking place in Mäori).
Only students that the schools reported to be in at least their
fifth year of immersion education were included in the sampling
process.
Pairing
small schools
At the year 8 level, 9 of the 120 chosen schools in the main
sample had less than 12 year 8 students. For each of these
schools, we identified the nearest small school meeting our
criteria to be paired with the first school. Wherever possible,
schools with 8 to 11 students were paired with schools with
4 to 7 students, and vice versa. However, the travelling distances
between the schools were also taken into account. Four of the
10 schools in the year 8 Mäori immersion sample also needed
to be paired with other schools of the same type.
Similar pairing procedures were followed at the year 4 level.
Five pairs were required in the main sample of 120 schools.
Contacting schools
Late in May, we attempted to telephone the principals or acting
principals of all schools in the year 8 samples (excluding
the 15 schools in the Mäori immersion sample). We made
contact with all schools within a week.
In our telephone calls with the principals, we briefly explained
the purpose of national monitoring, the safeguards for schools
and students, and the practical demands that participation
would make on schools and students. We informed the principals
about the materials which would be arriving in the school (a
copy of a 20 minute NEMP videotape plus copies for all staff
and trustees of the general NEMP brochure and the information
booklet for sample schools). We asked the principals to consult
with their staff and Board of Trustees and confirm their participation
by the end of June.
A similar procedure was followed at the end of July with the
principals of the schools selected in the year 4 samples, and
they were asked to respond to the invitation by the end of
August. The principals of the 14 schools in the Mäori
immersion sample at year 8 level were contacted in the middle
of August and asked to respond by the middle of September.
They were sent brochures in both Mäori and English.
Response
from schools
Of the 288 schools originally invited to participate, 282 agreed.
Three schools in the year 8 sample declined to participate:
an intermediate school because of major building work, an independent
school because of a clash with a drama production involving
all year 8 students, and a very small paired school because
of the high level of teaching in Mäori in that school.
The first two were replaced within their districts by schools
of similar size. The paired school was not replaced: instead,
additional pupils were selected from the other school in the
pair. An independent school in the Year 4 sample declined to
participate, and was replaced by a school of similar size in
the same district. In the Mäori Immersion sample, a school
chose not to participate and was replaced by a nearby school,
while a very small paired school lost students and was replaced
by selecting additional students from its paired school.
Sampling of students
With their confirmation of participation, each school sent
a list of the names of all year 4 or year 8 students on their
roll. Using computer generated random numbers, we randomly
selected the required number of students (12, or 4 plus 8 in
a pair of small schools), at the same time clustering them
into random groups of four students. The schools were then
sent a list of their selected students and invited to inform
us if special care would be needed in assessing any of those
children (e.g. children with disabilities or limited skills
in English).
At the year 8 level, we received 110 comments from schools
about particular students. In 58 cases, we randomly selected
replacement students because the children initially selected
had left the school between the time the roll was provided
and the start of the assessment programme in the school, or
were expected to be away throughout the assessment week, or
had been included in the roll by mistake. The remaining 52
comments concerned children with special needs. Each such child
was discussed with the school and a decision agreed. Nine students
were replaced because they were very recent immigrants or overseas
students who had extremely limited English language skills.
Eight students were replaced because they had disabilities
or other problems of such seriousness that it was agreed that
the students would be placed at risk if they participated.
Participation was agreed upon for the remaining 35 students,
but a special note was prepared to give additional guidance
to the teachers who would assess them.
In the corresponding operation at year 4 level, we received
123 comments from schools about particular students. Thirty-six
students originally selected were replaced because they had
left the school, were not actually year 4 students, or were
expected to be away throughout the assessment week. Two students
were replaced because they attended a satellite school more
than 60 minutes travel from the main school. Ten students were
replaced because of their NESB status and very limited English.
Sixteen students were replaced because they had disabilities
or other problems of such seriousness the students appeared
to be at risk if they participated. Special notes for the assessing
teachers were made about 59 children retained in the sample.
Communication
with parents
Following these discussions with the school, Project staff
prepared letters to all of the parents, including a copy of
the NEMP brochure, and asked the schools to address the letters
and mail them. Parents were told they could obtain further
information from Project staff (using an 0800 number) or their
school principal, and advised that they had the right to ask
that their child be excluded from the assessment.
At the year 8 level, we received a number of phone calls including
several from students wanting more information about what would
be involved. Three children were replaced as a result of these
contacts, one at the child’s request and two at the parents’ request
(one family would not allow their child to view videos or use
computers on religious grounds, the other simply requested
that their child not participate).
At the year 4 level we also received several phone calls from
parents. Some wanted details confirmed or explained (notably
about reasons for selection). Three children were replaced
at parents’ request.
Practical
arrangement with schools
On the basis of preferences expressed by the schools, we then
allocated each school to one of the five assessment weeks available
and gave them contact information for the two teachers who
would come to the school for a week to conduct the assessments.
We also provided information about the assessment schedule
and the space and furniture requirements, offering to pay for
hire of a nearby facility if the school was too crowded to
accommodate the assessment programme. This proved necessary
in several cases.
Results of the sampling process
As a result of the considerable care taken, and the attractiveness
of the assessment arrangements to schools and children, the
attrition from the initial sample was quite low. Only about
two percent of selected schools did not participate, and less
than two percent of the originally sampled children had to
be replaced for reasons other than their transfer to another
school or planned absence for the assessment week. The sample
can be regarded as very representative of the population from
which it was chosen (all children in New Zealand schools at
the two class levels except the one to two percent in special
schools or schools with less than four year 4 or year 8 children).
Of course, not all the children in the samples actually could
be assessed. Five year 8 students and 13 year 4 students left
school at short notice and could not be replaced. A parent
withdrew one year 8 student too late to be replaced. One NESB
year 8 student was judged by the teacher administrators to
be too limited in English language skills, and another was
a year 7 student. A further 16 year 8 students, 11 year 4 students,
and 2 Mäori immersion students were absent from school
throughout the assessment week. Some others were absent from
school for some of their assessment sessions, and a small percentage
of performances were lost because of malfunctions in the video
recording process. Some of the students ran out of time to
complete the schedules of tasks. Nevertheless, for many tasks
over 95 percent of the student sample were assessed. No task
had less than 90 percent of the student sample assessed. Given
the complexity of the Project, this is a very acceptable level
of participation.
Composition
of the sample
Because of the sampling approach used, regions were fairly
represented in the sample, in approximate proportion to the
number of school children in the regions.
|
84