The National Education Monitoring Project (NEMP) Mathematics studies
in 1997 and 2001 provide information about children's achievement
in a range of task items (Crooks and Flockton, 2002; Flockton and
Crooks, 1998). The tasks that focus on numeracy are found mostly
(but not exclusively) in the Items from the 1997 Number and Money
chapters and from the 2001 Number chapter. There are also numberrelated
items from the Measurement, Algebra and Statistics chapters. The
current Numeracy Projects have a focus on children's strategies
and is a topic of some discussion within mathematics education (Walls,
2004). This highlights a tension between knowledge and strategies
in the Numeracy Framework used in the current Numeracy projects.
The NEMP Mathematics Framework (Crooks and Flockton, 2002, p.10)
also includes connections between areas of Knowledge and Process
and Skills.
The
Commentary section in each NEMP Report provides brief details
about children's errors and strategies after each Item. Further
analysis of the task items may provide greater detail about children's
mathematical and strategic reasoning as well as errors or misconceptions
within the context of the tasks and among groups of children.
This
Probe Study report sets out the method used to further analyse four
task items from a small sample of the 2001 NEMP Mathematics data.
The results of the data analysis are set out for each task and include
the categories of strategies and errors for each item. Finally,
the findings for each item and implications for research are briefly
discussed.

Assessment
tasks in mathematics are designed to provide children with opportunities
to solve mathematical problems and to demonstrate their knowledge
of mathematics. Yet research suggests that assessment of children's
understanding in mathematics is complex. Tasks can be presented
in written form or verbally, with or without equipment. Problems
can also be posed in everyday contexts, referred to as contextualized
problems. These are designed to make a link with children's experiences
including cultural backgrounds and assumed to be motivation for
engagement in mathematical tasks. Tasks set in a context, however,
can be problematic for children and can pose either a distraction
or barrier for children in their attempts to solve the task (Boaler,
1993: Sullivan, Zevenbergen and Moulsey, 2002: Zevenbergen, 2002).
The language
of the task also poses difficulties for children whether in written
or oral form. The description of a context situation usually requires
more text for children to read, placing greater demands on children
(Eley and Cargill, 2002). They must be able to decode both the specialized
terms in mathematics as well as the linguistic forms embedded in
the questions (Eley and Cargill, 2003). School mathematics involves
words that are not necessarily specific for mathematics but are
signifiers of the type of task requirement. For example a question
that asks “how many more than …?” includes a signifier 'more than'
that provides a cue for the child about the type of task and possible
solution strategies (Cooper and Dunne, 2000; Zevenbergen, 2000).
In New Zealand,
further analysis of NEMP items has raised questions about the context
and/or format of mathematical tasks. For example, a station task
set in a context did not necessarily promote recognition. Although
it was a familiar context of a family pizza dinner, the Year 4 children
in particular relied on a prompt from the interviewer to continue
with the questions in the task (Anthony and Walshaw, 2003a). The
year 4 children were also more likely to talk about the features
of the pizza context rather than the mathematical structure (Anthony
and Walshaw, 2003b). Written tasks were found to be less popular
with children than the onetoone or multichoice tasks (Eley and
Cargill, 2002). Question formats were examined and found that students
were less successful with 'short answer' formats as these provided
the least support to children without teacher clarification or equipment
(Eley and Cargill, 2002). Children's explanations in onetoone
tasks described their strategies and/or reasoning and were more
extended responses, generally resulting in higher scores than for
other kinds of task formats (Eley and Cargill, 2002: Eley and Cargill,
2003). The onetoone tasks had drawbacks, however, because some
children gave as many responses as possible rather than taking time
to articulate their thinking. Children may have felt watched with
an interviewer present and were less likely to check answers. The
multichoice format was somewhat contradictory as many children used
the choice of answers provided to work out or guess the correct
answer yet it was not popular as a format with the children (Eley
and Cargill, 2002).
The children's
verbal responses to tasks provide information about their mathematical
thinking. This has become a common research tool as children's strategies
or reasoning can be identified or inferred from what they say. For
example, some seven to twelve year old students were found to retain
the use of counting strategies rather than using number facts for
addition and subtraction (Gray, 1991; Gray and Pitta, 1996;). The
less successful students were found to have an overreliance on
counting strategies which was inferred as a tenuous knowledge of
number facts.
Errors in written
tasks can also be a source of information about children's mathematical
reasoning. Errors can be based on misconceptions, plausible beliefs
about numbers but applied to inappropriate situations (Hart, 1981;
Johnson, 1989; Maurer, 1987). Recurring stable errors are also known
as bugs, from a metaphor used in computing. Errors with decimal
numbers have been a topic of much research in both primary and secondary
schools (Irwin, 1999; Moloney and Stacey, 1996; Steinle and Stacey,
2001). Decimal misconceptions have been attributed to a variety
of factors: application of whole number knowledge to decimal numbers
(Steinle and Stacey, 2001), differing interpretations of some real
life situations such as money, emerging knowledge of place value,
or not enough emphasis on multiplicative thinking (Irwin, 1999).
This probe study
used four task items from the 2001 NEMP data to investigate the
following research questions: 
The
process of data analysis centred around immersion in the data, category
generation, refining category analysis and peer review. These aspects
were considered important course experiences for the researchers.
The research
team familiarised themselves with the marking schedules for each
task supplied by NEMP. They examined the task, became familiar with
the marking approach, and generated further analysis categories
guided by the course lecturer. Further categories focused on errors,
and associated possible misconceptions or perturbations in mathematical
activity. The research team refined the analysis protocol for each
of the three written tasks, outlined in the next section. Each researcher
used the analysis protocols for their sample of eight children,
recording results in a grid for each task. This was followed by
peer review of the analysis in order to check for consistency and
to add further clarity to the analysis. Peer review involved a member
of the research team analysing another's data set and recording
their results. This process of comparing analysis decisions and
discussing similarities and differences provided opportunities for
clarifying and 'sharpening' the analysis protocols, resulting in
greater consistency between researchers.
One of the constraints
of analysing children's written work is that researchers are unable
to use direct observation or ask children to self report their strategies.
Children's strategies can only be inferred from the written record
and, while inferences are influenced by research, interpretations
can still be somewhat speculative. When known strategies could not
be inferred or peer review revealed multiple interpretations, then
the research team resolved to categorise these items as unknown.
For the video
task (36 and 29), the researchers generated a transcript
of what was said by the child and the interviewer, adding supplementary
information observed on the video. The transcripts were analysed
using the NEMP marking protocol and peer reviewed for one child.
Finally, the research team put together a profile for each of their
eight children based on the survey data and the analysis of the
four task items. 