A study of changes in mathematics achievement using data from
the first and second cycles of NEMP assessment
 
   

5. CHANGES IN ACHIEVEMENT FROM 1997 TO 2001

5.1

INTRODUCTION

The principal objective of this study is to investigate the changes in children’s achievement, attitude and values using the data from the two NEMP assessments.
In order to measure changes in achievement, about one third of the tasks are kept constant from one cycle to the next. In the first assessment these tasks are called link tasks and are not reported on in detail to avoid biasing the second assessment results. In the second assessment these tasks are called trend tasks and the performance of students in the two assessments can be compared.

5.2 THE REPORTING OF CHANGES ACHIEVEMENT IN THE NEMP REPORTS   The changes in performance from 1997 to 2001 are, of course, discussed in the 2001 NEMP report. Most of the commentaries associated with the trend tasks contain comments on these changes, and there are also paragraphs at the beginning of each chapter which give some statistics relating to the changes.
It is useful initially to look at these comments in order to decide where further examination is justified.

  5.2.1 Comments in the NEMP report concerning change in achievement on individual trend tasks

The table below indicates the nature of the comments on each of the trend tasks in the 2001 report.

TREND TASK
COMMENT
Number
Subtraction Facts For both year 4 and year 8 there was a small improvement.
Division Facts A small improvement in high scores.
Addition examples At both year levels the 2001 students performed very similarly to the 1997 students.
Multiplication Examples Small improvement for both years.
Calculator Ordering Results very similar.
Pizza Pieces Year 4 a little better, no consistent pattern for year 8.
Girls and Boys 2001 students had a small but consistent advantage in parts 1 to 6 and a larger advantage in parts 7 and 8.
Equivalents On average, 4% more succeeded with each conversion than in 1997. Although this improvement was slight, it occurred for all 12 conversions.
Measurement
Apples No comment.
Better Buy Little change at either level.
Measures Little change.
Bean Estimates 2001 year 4 students did a little better, but year 8 students did markedly worse.
Video Recorder Overall performance of both year 4 and year 8 students was slightly lower in 2001.
Measurement Items B No comment.
Measurement Items C No comment.
Party Time No comment.
What’s the time? Very similar in 1997 and 2001.
Supermarket shopping The 2001 students were a little less successful.
Two boxes Only 30% calculated the volume correctly compared to 29% in 1997, but more students used the correct units in 2001 than in 1997.
Geometry 
Hedgehog Year 4 students did equally well or slightly better in 2001, but year 8 a little worse.
Cut Cube Year 4 students performed very similarly but year 8 noticeably lower.
Flat shapes Recording dimension better in 1997 otherwise similar.
Algebra and Statistics
Train Trucks 2001 year 4 a little more successful, year 8 results more mixed.
Algebra/Logic Items A Only one tend item for year 4; 15% more successful in 2001. Very little change for year 8.
Algebra/Logic Items B Year 4 substantially better, year 8 performed similarly.
Statistics Items A Year 8 results almost identical. Not administered to year 4.
Statistics Items B Results very similar.
Farmyard Race 2001 students performed a little better.
Photo Line-Up 2001 students performed similarly to the 1997 students.

 

  5.2.2 Comments in the NEMP report on changes in performance in content areas

Each chapter in the report contains a paragraph on the changes in performance in its content area. The summary comments are highlighted.

Number
There was evidence of modest improvement between 1997 and 2001, especially for year 4 students. Averaged across 59 trend task components attempted by year 4 students in both years, 5 percent more students succeeded in 2001 than in 1997. Gains occurred on 51 of the 59 components. At year 8 level, with 106 trend task components included, 3 percent more students succeeded in 2001 than in 1997. Gains occurred on 85 of the 106 components. (see Number 2001 - Introduction)

Measurement
There was little evidence of change between 1997 and 2001. Averaged across 41 trend task components attempted by year 4 students in both years, 2 percent more students succeeded in 2001 than in 1997. Gains occurred on 25 of the 41 components. At year 8 level, with 45 trend task components included, 2 percent fewer students succeeded in 2001 than in 1997. Gains occurred on 15 of the 45 components. (Measurement 2001 - Introduction)

Geometry
There was little evidence of change between 1997 and 2001 for year 4 students, but a small decline for year 8 students. Averaged across 13 trend components attempted by year 4 students in both years, 2 percent more succeeded in 2001 than in 1997. Gains occurred on 10 of the 13 components. At year 8 level, with 22 trend task components included, 5 percent fewer students succeeded in 2001 than in 1997. Gains occurred in 3 of the 22 components. (Geometry 2001 - Introduction)

Algebra and Statistics
There was evidence of substantial improvement between 1997 and 2001 for year 4 students, but little change over the same period for year 8 students. Averaged across 15 trend task components attempted by year 4 students in both years, 9 percent more students succeeded in 2001 than in 1997. Gains occurred in 14 of the 15 components. At year 8 level, with 28 trend task components included, 1 percent more students succeeded in 2001 than in 1997. Gains occurred on 16 of the 28 components. (Algebra and Statistics 2001 - Introduction)

  5.2.3 Discussion The above comments are certainly useful in giving a broad indication of change. However, there does seem to be a case for a closer look at some of the data. The individual trend tasks commentaries suffer from the difficulty of having to summarise, in a single sentence, the data from a number of component tasks which might differ quite markedly in difficulty, performance expectation and achievement objective.

The chapter summaries have the same problem. Without further information, it is not easy to draw useful conclusions for classroom practice, for the professional development of teachers, or for curriculum development, from these summaries.

We need to know the nature of those components on which there were significant gains in performance over the period and those for which there were losses.

5.3 THE NATURE OF TREND TASK COMPONENTS ON WHICH THERE WERE THE HIGHEST GAINS OR LOSSES IN ACHIEVEMENT  The most obvious way to consider the importance of any changes in achievement is by using statistical significance. The authors of the report use this when considering subgroup performance but not in considering changes in performance from one cycle of assessment to the next. The data in the report is not sufficient for the researcher to be able to statistically determine the significance of any changes in achievement.

However, the number of students in each analysis is quite large (approximately 450), so statistical significance tests run the risk of drawing undue attention to small differences.

Consequently, it was decided to just concentrate on identifying those trend task components on which there were the greatest changes in achievement between the two assessments. A relatively arbitrary figure of 6 percent or more gain or loss was chosen to identify the components. This accounts for about 30% of all gains and losses. Of 128 components at year 4 there were 30 in this category. At year 8, 31 of 201 components had changes of this magnitude.

  5.3.1 The trend task components on which there were changes in achievement of at least six percent

In two of the trend tasks, Subtraction Facts and Division Facts, only summary data is given rather than data on individual components. For these tasks, the mean score, estimated from the table given, was allocated to each of the components. On each of the other tasks component scores were available.

The following table indicates those trend task components on which there was a change in achievement of at least 6%. The achievement objective and the performance expectation of each component is also given.

COMPONENT
CHANGE %
AO
PE
NUMBER
YEAR 4
YEAR 8
   
Addition Examples (2)
-6
N2-8
RP
Multiplication Examples
(2)
+8
N3
RP
(3)
+11
N3
RP
(4)
+6
N3
RP
(5)
+7
N3
RP
(6)
+8
N3
RP
(7)
+8
N3
RP
Girls and Boys (4)
+7
N2-10
SP
(6)
+7
N2-10
SP
(7)
+12
N2-10
SP
(8)
+15
N2-10
SP
Equivalents
(2ii)
+8
N4-5
RP
  (4ii)
+8
N4-4
RP

The table suggests a quite marked improvement in performance at both levels. The one component on which the year 4 achievement decreased was a column addition of five single digit numbers. This invites an algorithmic approach which would be discouraged in the current numeracy projects (Ministry of Education, 2003, page 9). All the items on which year 4 students showed improvement of this magnitude were at level 2 in the curriculum. Those for year 8 were at levels 3 and 4.

COMPONENT
CHANGE %
AO
PE
MEASUREMENT
YEAR 4
YEAR 8
   
Apples (1)
+8
-6
N2-10
RP
(4)
-13
N3-6
CP
Measures (4)
 
+6
M3-1
CP
  (5)
+6
M3-1
CP
  (6)
-7
M2-1
RP
Bean Estimate (1)
 
-10
M3-1
CP
  (2)
+7
-7
M3-1
CP
  (3)
-13
M3-1
CP
Lump Balance
(2)
+11
M3-2
C
(4)
-7
M3-2
SP
Video Recorder (1)
-12
-9
M4-5
K
Measurement Items B (6)
+12
M3-1
CP

Measurement Items C
(3)
+6
N3-7
RP
  (9)
+8
M4-5
CP
(11)
-12
M3-1
K
Party Time
(2)
+7
M2-2
RP
Supermarket Shopping (1)
-8
N3-6
RP

The chapter summary for measurement stated that there was little evidence of change between 1997 and 2001. The data above would suggest an improvement at year 4 and a decrease in performance at year 8. The components on which year 4 students showed improvement came from levels 2, 3 and 4 of the curriculum, as did those on which year 8 students showed a decrease in performance.

COMPONENT
CHANGE %
AO
PE
GEOMETRY
YEAR 4
YEAR 8
   
Hedgehog
(1)
+7
G2-4
RP
(3)
+7
-6
G2-4
RP
(4)
-14
G2-4
RP
  (5)
-7
G3
RP
  (6)
-10
G3
RP
  (8)
-6
G3
RP
Cut Cube
(1)
-11
G3
SP
  (2)
-9
G3
SP
  (4)
-11
G3
SP
  (5)
-6
-13
G3
SP
Flat shapes
(1ii)
-6
M4-1
RP
  (2)
-7
G2-1
K
  (3v)
-16
M4-1
RP

The data suggests little change at year 4 but a fairly substantial decline at year 8. Again the decline at year 8 was in components from levels 2, 3 and 4 of the curriculum.

COMPONENT
CHANGE %
AO
PE
ALGEBRA AND STATISTICS
YEAR 4
YEAR 8
   
Train Trucks
(3)
+7
A2-1
CP
(4)
+8
A2-1
CP
(7)
+12
S2-4
SP
(8)
+10
S3-6
CP
(13)
+8
S3-6
SP
Algebra/Logic Items A (2)
+15
A2-3
K
Algebra/Logic Items B
(14)
+14
A4-4
CP
(17)
+15
A3-5
CP
(18)
+14
A2-3
K
(20)
+13
A2-3
K
Farmyard Race
- strategy good/vg +9
MP
SP
- cooperation good +8
MP
C
Photo Line-Up
- cooperation good  
-7
MP
C

The last three components relate to the Mathematical Processes strand of the curriculum and cannot be linked to a particular level.

The table confirms the chapter summary of a substantial improvement at year 4 and little change at year 8.

  5.3.2 Summary by content area

The table below indicates the number of trend task components in each chapter of the NEMP report on which there was a change in performance of at least 6 percent.

   
YEAR 4
YEAR 8
   
+
-
+
-
Number
6
1
6
0
Measurement
8
3
1
8
Geometry
2
1
0
12
Algebra/Statistics
9
0
3
1
TOTAL
 
25
5
10
21

The gains at year 4 were spread over each of the content areas. The picture at year 8 is very different with the gains concentrated in number and algebra/statistics and the losses in measurement and geometry.

  5.3.3 Summary by achievement objective levels
   
YEAR 4
YEAR 8
   
+
-
+
-
Level 2
12
2
2
4
Level 3
9
2
6
13
Level 4
2
1
2
3
Math Pr
2
0
0
1
TOTAL
 
25
5
10
21

There does not seem to be a discernable pattern in the levels of the components on which there were the greatest changes in achievement.

  5.3.4 Summary by performance expectation

The table below gives the number of trend task components on which there was a change in performance of at least 6 percent and which required the indicated performance expectation.

   
YEAR 4
YEAR 8
   
+
-
+
-
Knowing
3
2
0
2
Routine procedures
7
2
6
9
Complex procedures  
6
0
4
4
Solving problems  
7
1
0
5
Justifying and proving
0
0
0
0
Communicating
2
0
0
1
TOTAL
 
25
5
10
21

For year 4 students, the number of gains was greater than the number of losses for each of the performance expectations. At year 8 there were no performance expectations for which the number of gains was greater than the number of losses.

5.4 CONCLUSIONS

Based on an examination of those trend task components for which there was the greatest change in achievement between 1997 and 2001, the following conclusions are suggested:

Year 4
w Major gains in achievement outweighed major losses by a ratio of 5:1.
w The gains were spread across all content areas and performance expectations.

Year 8
w Major losses in achievement outweighed major gains by a ratio of 2:1.
w The gains were concentrated in the areas of number and algebra/statistics and the losses in measurement and geometry.
w There were no performance expectations for which the number of gains was greater than he number of losses. The two performance expectations for which the difference was greatest were routine procedures and solving problems.

next page

top of page    |    return to Probe Studies - INDEX   |    return to Probe Studies menu
  For further information and contact details for the Author    |    Contact USEE