MethodologyAs a trained administrator of the National Education Monitoring Project I initiated my research project with a good knowledge of the monitoring programme and its potential value for teachers.

In the first section of this chapter I will provide contextual background information about NEMP and their monitoring processes.

I will then report on the key aspects of my methodology, specifically:

w Criteria for tasks
w Selection of tasks
w Task description

National Education Monitoring Project
New Zealand’s National Education Monitoring Project started in 1993 with the aim of providing a national “snapshot” of children’s knowledge and skills. Because NEMP carry out their curriculum assessments in four year cycles, the information gained is useful in identifying trends in student achievement.

The main goal of national monitoring is to provide detailed information about what children can do so that patterns of performance can be recognised, successes celebrated and desirable changes to educational practices and resources identified and implemented.
Flockton & Crooks (1997p.4)

The Aims of NEMP
There are two aims of national monitoring; the first is to provide feedback to policy makers, planners and educators in order to guide curriculum development and resourcing. The second is to provide the public i.e.; government, tax payers and parents with information about how the education system is performing. These two issues – supporting the teaching and learning process and accountability underpin the purpose of National Education monitoring and add validity to my decision to use NEMP materials for this research.

The Monitoring Process – Sampling
NEMP monitors student achievement at two levels, year 4 and year 8 in all of the essential curriculum areas. The 1440 year 4 students and 1440 year 8 students are randomly selected nationwide and represent 2.5 percent of the children at those levels in New Zealand schools. Not all students will attempt the same assessment tasks; the 1440 students selected in the main sample in each group are then divided into three groups of 480 students in order to be able to collect a wide amount of information.

Ethics
The process of involving students in the National Monitoring Project requires parental permission and provides the availability of an 0800 number as well as extensive information packs for schools and parents. The students are identified by numbers in the videos and pencil and paper tasks. There is no record of the names of the students or the schools evident in the viewing of the tapes.

The issue of ethics is well considered and covered under the NEMP umbrella. The Probe study that has been undertaken as part of this research stipulates confidentiality for any information that is not otherwise considered as part of the public domain. (Clause 7. sub contracting agreement NEMP Probe Study 2002).

Task Approaches
Four different task approaches are used by NEMP in order to allow for differing learning styles and assessment information requirements. Each student was expected to spend about an hour working in each format.

The Four types of tasks are:

w One to One interview – student works individually through a series of tasks
w Stations – students worked independently moving through a series of activity stations
w Team – Four students worked collaboratively sometimes supervised by
               a teacher on tasks
w Independent – Students worked individually on paper and pencil tasks

The one to one tasks and team tasks are both videoed and involved specific opportunities for students to explain their reasons for answers. This was an easy medium for me to observe and record my information so the choices for tasks were limited to these two task types.

Criteria for Tasks
The marking of NEMP tasks requires specific criteria for each activity, depending on what skills or knowledge are being assessed. For the purposes of this research I considered it necessary to specify a marking checklist that looked at the skills of social decision-making (SDM) in isolation. SDM skills have clearly set achievement objectives in the New Zealand Curriculum, however they are often difficult to assess, because of the very real and necessary overlap into co-operative skills. Throughout the literature, there are several “indicators” that stand unaccompanied and are applicable to social decision making only.

The three aspects that are common to all descriptors of the social decision- making processes are:

w Identifying a problem.
w Suggesting some strategies to solve the problem.
w Making a decision to help prevent/fix/minimise the problem.

The lists of skills that describe the Social decision-making process all start with the identification of a problem. All of the literature mentions a decision- making process that involves critical thinking and evaluating.

The Curriculum document describes the process as:

“Students identify and clarify a social issue then suggest a range of
possibilities/ strategies to address this issue.”
Social Studies in the New Zealand Curriculum. (1997 p.18)

More specifically the Position Paper 1997 (p6) lists these skills as:

w Problem solving.
w Decision-making.
w Critical reflection.
w Evaluation of options.
w Co-operation and leadership.

Similarly The Participatory Research Network (1982) cited in Keown, (1999) involves pupils in a sequence including:

w Problem identification.
w Analysis.
w Planning.
w Action.
w Evaluation and reflection.

The three most important aspects that could pragmatically be isolated assessed and collated were: identifying, suggesting and decision- making. All three are fundamental to social decision-making but each skill can be assessed as single component.

These are the skills that are clearly defined in the process indicators of the curriculum document at levels 1 –4.

Students will demonstrate skills as they make decisions about
possible social action by

w Identifying issues and problems (identifying)
w Develop solutions (suggesting)
w Make choices about possible action (decision-making)

SSNZC (1997 p 52-53)

Through my involvement on the social studies contract I had developed a school programme to ensure coverage of the three social studies processes. This in-depth knowledge of the processes in the social studies curriculum enabled me to recognise and select which skills to focus on in order to give consistency and accuracy in this research. All three of the above skills were also present in all of the literature that was reviewed in order to clarify a definition of what social decision making entailed.

Selection of Tasks
I made contact with the NEMP office in Dunedin, following the acceptance of my research proposal. I notified them of the purpose of my study and the support that I would require to carry out my research. Specifically, these were examples of assessment tasks that might indicate student achievement in social decision-making. A list of 44 possible tasks was supplied. (See Appendix 2)

A date for viewing these tasks was set and I travelled to the NEMP offices in Dunedin to view the samples over a three-day period. I viewed a sample of each task along with its marking sheet that was supplied with the task. Notes were made as to how many times the skills of identifying, suggesting and decision- making arose. In order for a task to be considered for selection all three of these skills had to be noted.

Additional notes were made about the possibility for other problems, suggestions and decisions that may not have been mentioned but may well be with another sample of the same task. It was important for the tasks chosen to present as many opportunities for presentation of skills as possible. Each of the tasks then had a running commentary .

The following task illustrates an example of the types of responses that I was able to identify.

This task showed a video clip of a school canteen where there were teachers pushing in ahead of children, orders were not kept for people and students were pushing in front of each other.

After watching the video clip the students were asked the following questions:

All of the questions in this task provide opportunity to identify problems, suggest solutions and decide on a way that the problems could be fixed.

Some typical responses under each of these three headings were:

Problems identified
   w Teachers pushing in
   w Orders not being kept for people
   w People pushing in
   w Some people going without lunch

Suggestions
   w 2 separate lines
   w teachers in own line
   w Extra food
   w Kinder canteen lady
   w A queue with iron bars

Decision making
Teachers having to place their order in staffroom every morning, a student will collect the total order and money for the canteen.

Other aspects not mentioned
   w Rules are needed
   w Parents could be involved

The task provided me with enough information about what students can do but it also highlighted a lack of skill development in the process of social decision-making. The decision to have teachers place their order in the staffroom did not come from the original list of possible solutions and therefore did not provide the best solution for the bigger problem. This was the type of information that had implications for skill development within the process of social decision-making and would therefore be useful to me in the context of my research. When analysing the task commentary sheets it was obvious that some didn’t have all three aspects of the decision- making process. Only tasks that presented students with opportunities for identifying, suggesting and deciding were kept. This then resulted in 14 tasks that needed to be further analysed.

These were:

1. Working together
2. Equal and Different
3. Roller Blades 14. Disaster
4. Saikaloni
5. Tree troubles
6. We Need a Leader
7. Refugees
8. Drinking Fountain
9. A Good Team Member
10. Children and Teachers
11. Ripeka
12. Playground
13. School Canteen

From there these were sorted according to which activities provided equal opportunity for defining, suggesting and decision making to be evident. Final choices involved two1-1 tasks and two team tasks. These were “Saikaloni,” “We Need a Leader,” “Playground” and “Tree Troubles.”

The 1-1 tasks We Need a Leader & Saikaloni were administered to both year 8 and year 4 pupils so that would give a good comparison of skill levels between the year groups.

The two team tasks were different for year 8 and year 4. Playground was chosen for the year 4 task and Tree Troubles for year 8. Both of the team tasks were link tasks which meant that the identical task had been administered to a group of year 8 students (or year 4 in the case of the year 4 task) in the 1997 NEMP cycle. This would give some indication of changes in achievement over time and could be used to add to the total amount of information as well as using the results to assess trends in achievement.

Sampling
From the four tasks chosen I requested a random sample of 50 for each activity. In total, that meant 200 representations of student responses. The supervisor of this research suggested 50 as a manageable number when looking specifically at any one task. The sample of 50 was made up of a comparison of 25 year 4 and year 8 responses for two of the activities, and a comparison of 25 1997 results against 25 2001 results in the same year groups. It would also give an indication of the achievement of year 4 students compared with year 8 students and look at any trends in achievement over time. Within the scope of that selection it would be possible to gain an accurate indication of the level of social decision-making skills that students are achieving at in terms of the wider definition of social decision-making. National Monitoring uses carefully selected random samples of students and national samples equate to about 6% of children in New Zealand Schools. The 200 samples chosen only represent a small percentage of the 2880 children tested each year but they were indicative of children in any New Zealand school.

Data Gathering
NEMP arranged the random sampling of 50 tapes in each of the 4 tasks and sent those and the viewing equipment to the researcher.

The tapes were viewed task by task over a five week period. The marking sheets that were created for each task provided space for the number of each tape in order to ensure that none were repeated or omitted. A positive or negative column under each of the three indicators provided space for recording whether the responses provided evidence of any of the skills. This made it easy to total the positive responses at the completion of each set of tasks. (See Appendix 3)

Quantitative and Qualitative Methodology
I couldn’t help but perceive the differences in co-operative skills and other learning strategies that were apparent when viewing the tapes. It became obvious that certain tasks lent themselves to allow students to identify problems more easily than others.
Judging from the students’ interaction and the quality of the responses it was also evident that some tasks were easier for year 8 students to achieve than the year 4 students. Although the results are collated and presented quantitatively, the anecdotal comments that were noted on the sheets will be discussed in the results.

An assistant was used to evaluate student performance against the criteria and to cross check results with the researcher. This provided a degree of collaborative marking which helped clarify the interpretations of the tasks over the five weeks it took to view the selection of tapes.

Description of the Four Tasks

This was a video presentation of a girl who was upset because the teacher couldn’t pronounce her name properly. She was named Saikaloni because she was born in a cyclone in which her uncle was killed. When the teacher tried to pronounce her name – “Saikaloni” she would get tongue-tied and mispronounce it. The rest of the class would laugh and this made Saikaloni upset and hurt. She used to like school but this has made her hate it.

The student was asked to identify the problems in this scenario. A typical response was to mention that Saikaloni had lost her uncle or a relative in the cyclone. The issue of the teacher mispronouncing her name was usually identified through the fact that Saikaloni disliked children laughing at her. When suggesting ways to help Saikaloni, most of the year 4 responses mentioned involving her mother. There was evidence throughout all of the responses regarding empathy towards others feelings. Some of the terms used like “The Principal should talk to the class about name calling and bullying” and “Saikaloni should tell them to stop laughing and tell them she doesn’t like it” are evident of the awareness of bullying and strategies to stop such behaviour in school cultures. One year 8 student out of the total 50 suggested that right at the beginning of the year the teacher should check her roll and learn the names of her students. Other suggestions about this issue were focused around how to prevent this sort of thing happening in the future.

A number of both year 8 and year 4 students mentioned a co-operative approach with the teacher asking Saikaloni about her name and how to pronounce The name Saikaloni, the girl in the video and the place where the cyclone occurred are all Samoan. This may have had some bearing in the achievement or lack of achievement in students being able to relate to this situation. This was highlighted by a Samoan boy in year 4, who very quickly associated his own culture and memories of time in Samoa with this scenario.

Year 4 did not score over 50% in the decision making aspect of this activity. Only 2 year 4 students were able to make a decision about how to stop this problem happening again. 14 year 8 students were able to make decisions that would be helpful in providing a solution to the problem.

This was a video clip that showed a group of children trying to organise themselves into a sports practice. The end result shows children looking exasperated and one child stating the fact that they needed a leader. Students were asked why the children needed a leader. Even though this obvious part of the problem was identified the students had to identify specific problems that arose from the fact that there wasn’t any leadership. Responses recognised that the practice wasn’t happening and children were fighting and arguing with each other. The suggestions aspect of this scenario focused on students saying what they thought a leader could do in this situation. “Tell them what to do’ was evident in both year groups. “Help organise them into teams” “Make sure that there is fair play.” There was a clear gender perspective inherent in this tape as the majority of the answers regarding what a leader could do came quickly from both year 4 and year 8 boys. There was an added understanding amongst the boys that the teams should be playing games that gave the children catching and throwing practice. In general boys were more confident in this task than girls. The decision making aspect of this task involved a degree of critical thinking as the students had to recognise and state the qualities needed in a person to be a good leader. Less than 50% of both year 8 and year 4 students were able to do this.

This was a video of a playground full of children clambering over playground equipment. Students were asked to identify the problem that they saw. Over 50% of students in both 1997 and 2001 were able to do this. “Children not taking turns” “Fighting over the playground equipment” Suggestions as how to fix this problem were much higher in the 2001 group than the 1997 group. There were lots of general ideas like “respect other people” and “be safe” in the 1997 group. In the 2001 group there was a noticeable improvement in the way the students related to each other – the co-operative rule of facing everyone in the group and eye contact was much more evident and there was mention amongst students of using “co-operative skills.” The 2001 group made more references to taking turns, student or class supervision and or adult supervision. There were three references made to rules that were already in place in schools in order to prevent this type of situation.

The second part of the video required the students to make a decision about who would do what in a scenario where one child had fallen off the equipment and appeared to be unconscious. This was where the students did not achieve as well as they had in previous aspects of this task. There seemed to be a problem with assigning roles to individuals, most students said that they would go and get a teacher, and some said one student could stay with the hurt child. Although the students were clearly asked what each person in the group would do in this situation (implying that there should be a different role for each group member), and were given time to discuss and assign these roles, there was little evidence of students thinking through the consequences and importance of these roles in a real life situation. See Appendix 4 for the NEMP summary of this task in the 2001 Assessment publication.

The group of students were shown a video clip of some children and teachers talking about a tree that was growing very close to their classroom. The teacher was complaining that the tree was noisy as it scraped against the side of the classroom and the students complained that its branches were blocking light from coming into the class. There were other viewpoints considered in this task. The local Kaumatua who said that the tree was planted in scared ground, the community who had planted the tree to honour those who had taken part in the war, the caretaker of the school who was sick of tidying up the leaves that it shed and the board of trustees member who pointed out that the tree provided a sun safe shade. The students in the group were asked to take on one of these roles and argue their case. They then had to come to some agreement as to what to do with the tree.

This was an interesting situation because the students already had some direction as to where to go with their discussion and many of them enlarged and built on their role plays. In both year groups the consequences of the decisions were considered as they returned to the original problems that were identified. There was a noticeable increase in reflective thinking processes in the 2001 group. Comments like “If we cut the tree down that will make the teachers and students happy but what about the community? And the cost – who will cut it down?” “ Perhaps we could sell the wood as a fundraiser?” “If we get the care taker to cut it back regularly then we will be able to keep the tree and help the teachers and students have more light and less noise in their classroom.”

Other issues that came out of this discussion were

w The role of the caretaker – is it in his job description to keep the grounds tidy?
w The tree keeps the school looking beautiful
w The waste of power that is used on heating and lighting in the classroom because
   of the tree.
w The board of trustees are under financial pressure
w The rising rate of skin cancer

All very valid points, but they made for difficult decision- making. Most groups came to a decision to keep the tree and get it trimmed, but some decided to replant it elsewhere. This scenario provided the students with a starting point and a model for their thinking processes. Both 1997 and 2001 scored over 50% in their totals with the 2001 year achieving a slightly higher rate in identifying specific problems and suggesting solutions. See Appendix 5 for the NEMP summary of this task in the 2001 assessment publication.

next page