## Servicios Personalizados

## Revista

## Articulo

## Indicadores

## Links relacionados

## Compartir

## Estudios pedagógicos (Valdivia)

##
*versión On-line* ISSN 0718-0705

### Estud. pedagóg. n.28 Valdivia 2002

#### http://dx.doi.org/10.4067/S0718-07052002000100005

Estudios Pedagógicos, Nº 28, 2002, pp. 89-107
PROFILE OF CHILEAN ACHIEVEMENT IN THE TIMSS 1999 DATA REPRESENTATION SUB-SCALE
International Study Center, Boston College, Lynch School of Education, Beacon 188, Chestnut Hill, MA 02467, U.S.A. E-mail: ramiremb@bc.edu
For the first time, Chile participated in IEA’s Trends in Mathematics and Science Study (TIMSS 1999), testing a nationally representative sample of 5907 eighth graders, and collecting vast background information about the students, their teachers and their schools. This article analyzes student’s achievement in data representation, the sub-scale that showed the best relative math results for Chile. The effect of five item-related variables was analyzed: format, cognitive skill, sub-content area, curricular intentions, and curricular implementation. Conclusions indicate that the better performance in data representation can be mainly explained because of a “street mathematics” phenomenon combined with an item-format effect.
## ResumenPor primera vez, Chile participó en el Estudio de Tendencias en Matemáticas y Ciencias de IEA (TIMSS 1999), evaluando los conocimientos de una muestra representativa a nivel nacional de 5907 alumnos de octavo básico, y recogiendo valiosa información de contexto sobre los alumnos, sus profesores y sus colegios. Este artículo analiza los resultados obtenidos en representación de datos, la subescala de matemáticas con los mejores resultados relativos para Chile. Aquí se analiza el efecto de cinco características de las preguntas incluidas en la prueba: formato, habilidad cognitiva, contenido, intención curricular e implementación curricular. Las conclusiones apuntan a que el mejor rendimiento en representación de datos puede explicarse por un fenómeno de aprendizaje informal de las matemáticas combinado con un efecto de formato de las preguntas.
## INTRODUCTIONIn TIMSS 1999, Chile ranked 35 out of 38 countries in the mathematics test. Its average achievement was 392 scale points
The following sections explain how each of these questions were addressed, and present the major findings of the study. Those results are then discussed in a broader perspective in the final chapter of this paper. ## METHOD
All this information was captured in the Data from the TIMSS Curricular Analysis Study was also used in this report. Even though Chile did not participate in the original study (Schmidt, Raizen, Britton, Bianchi & Wolfe 1997), the same methodology was applied in a data collection effort carried on in 1999. Basically, the method consisted of an analytical approach where curricular documents and textbooks were coded according to the content and performance expectations presented in each “block”. Blocks were the basic unit of analysis in the documents, usually ranging in length from a simple statement (e.g. a curricular standard) to a short paragraph (e.g. an instructional activity). This information was used to support statements about the curricular intentions and the most likely math experiences to which 8
The items were then sorted from easiest to hardest, and graphed in a scatterplot with Items 1 to 21 presented in the x-axis and the percent of correct responses in the y-axis. A data representation achievement profile was then obtained, with the item plots forming a smooth slope of difficulty level for the Chilean performance. This same procedure was repeated several times in different figures, just changing the plots symbols in order to represent a third variable. This third variable was directly related to each one of the hypotheses under analysis: item format, performance expectations, content sub-areas, curricular intentions and curricular implementation. The Chilean and international percents of correct responses for the 21 items under analysis were directly taken from the TIMSS data almanac files. Information about the multiple categorization systems used to describe the items of the test – format, cognitive skills, content sub-areas, and curricular intentions – was also directly retrieved from the item information files contained in the TIMSS database. The classification of items by cognitive skill required some reorganization of the original categories. The international report mentions five performance expectations: “knowing”, “using routine procedures”, “investigating & problem solving”, “mathematical reasoning”, and “communicating” (Mullis For the purposes of this report, the second classification system was used because of its direct relation with the tasks presented in the data representation items. However, some changes were introduced in order to make the classification more meaningful for the content area under analysis. Specifically, five categories were rearranged into only two categories: “recalling mathematical objects & properties” was merged with “representing” into one group; “solving” was merged with “predicting” and with “solving, describing & discussing”, into another group. The computations of the percent of correct responses for students who were taught the topic and those who were not yet taught the topic –curriculum implementation status– required the combination of students’ achievement scores and data from the teachers’ questionnaire. Teachers were asked when students in their mathematics class were taught each topic in the tests. In the “Data representation, analysis and probability” section, they were specifically asked about the three sub-content areas that form the data representation sub-scale: – Representation & interpretation of data in graphs, charts, and tables. For each of these topics, teachers were asked if the topic was: a) Taught before this year; b) Taught 1 to 5 periods this year; c) Taught more than 5 periods this year; d) Not yet taught; e) Do not know. Their answers were reclassified into two dichotomous groups: already taught (which includes points one to three) and not yet taught (which included the fourth point only). The “Do not know” answers –that accounted for 5% of the responses– were excluded from the analyses. In each sub-content area, students were classified according to their teachers’ answers. For example, a student could be reported as having been taught representation and interpretation of data and arithmetic mean, but not yet taught simple probabilities. In this case, this student was counted as “already taught” in the first two sub-areas, but as “not yet taught” probabilities. Then, within each sub-content area, separate percents of correct responses were calculated for every item. ## RESULTS
Several comments can be made about the graph above. First of all, it is worth noting the wide rage of Chilean difficulty levels for the data representation items. While 71.60% of the students answered Item 1 correctly, only 8.15% got the correct answer in Item 21 Secondly –and directly related to the previous paragraph– comparing the differences in item difficulty for Chile and the international segment, these differences tend to increase from the easiest to the hardest items. For example, while in Item 1 both lines are relatively close one to another (reflecting a difference of 6.60 points between Chile and the international average), in Item 21 the lines are much more apart (reflecting differences of 40.10 points). One clear exception to this pattern was Item 5, which presented almost the same difficulty level for Chile and the international composite – with 1.70 percent points of difference favoring the latter.
Is there something special about Item 5 that can explain why it is out of the pattern when comparing the Chilean and international profiles? Item 5 is a multiple-choice question that asks about the likely result of a fifth coin toss; it is a probability question (sub-content area) that 55.70% of the Chilean students answered correctly. Even though probability was not a topic intended to be taught to 8 Item 5 required the students to recall mathematical objects and properties, forming part of the more basic performance expectation level included in the test. It could be hypothesized that the Chilean performance equaled the international segment because of the basic cognitive skills demanded by this problem. But this is not a good explanation, because the item would be easier not only for the Chilean students but for all the students of the participating countries. A more plausible explanation refers to the familiarity of the problem presented in the item. That the probability of tossing a coin is always the same –no matter what the previous results were– is something very probable to be learned informally, as part of the “street mathematics” curriculum. This line of argumentation will be expanded in this paper. Third, overall, most difficult items for Chilean students were also the most difficult items for the students across all TIMSS countries, and vice versa. What changed drastically between one group and the other was their mean achievement in data representation. This is graphically depicted by the distance separating both profile lines. On average, 17.25 percent points separate both lines. These are strong and significant differences ( The fourth and final comment is devoted to Item 21. This question is definitely out of the slope pattern depicted by the Chilean line. On average, when passing from one item to another, the difficulty level changes two or three percent points. Nonetheless, in the case of Item 21, its percent of correct answers was 15 percent points less than the item with the next difficulty level. The international profile does not replicate this pattern. A somewhat drastic
In order to see if there is a The figure shows that 19 out of 21 items in the data representation content area have a multiple-choice format. This is a substantially higher proportion compared to the three-fourths of items with this format in the entire test. Of the open-ended questions presented in this content area, only one corresponds to a short-answer type (Item 6), and another one to an extended response item (Item 21). The only short-answer item belongs to the first third of item difficulty (group of easiest items). Item 6 fits perfectly well the profile line, showing no special pattern in relation to the rest of the multiple-choice items. A very different situation is observed in the case of the only extended-response question included in the sub-scale. Item 21 clearly falls below the projected line of the slope flow, presenting a percent of correct responses considerably lower than the rest of the items (8.15%). This drastic drop in the achievement profile line is not observed in the case of the international profile.
Considering that Item 21 is the only one with an extended response format, the possibility of a A direct analysis of Item 21 allows not only an appreciation of its major format complexity, but also raises questions about its classification in the data representation content area. Which one is the most demanding task to solve this item: to understand the message of the two advertisements or to perform the operations required to solve the questions? If this second option were true, the item would be better classified in the fraction & number sense category. While Item 21 describes what the most able students can do, Item 3 –the other released data representation item– describes what the students pertaining to the lower quarter benchmarking can do (Mullis
Chilean students obtained 67.40% of correct responses in this question, compared to 79.20% obtained by the international composite. The difference is somewhat smaller than the average for all items of the sub-scale (11.80 versus 17.25 percent points).
As shown in the figure, using complex procedure items are spread out all over the distribution, ranging from Item 1 to Item 20. This is not surprising considering that more than half of the data representation items are part of this group. The other higher order set of items –solving & predicting– is somewhat more unified, with Items 21, 19, 18, and 17 forming part of a clear cluster with the lowest From a developmental perspective, these results fit well the expectations: items aimed to measure higher order skills have lower
In spite of this emphasis on basic skills, the old Chilean curriculum covered a wide range of skills. Knowing, representing, using routine procedures, using more complex procedures, understanding mathematical problems, solving problems, and communicating were all mentioned in the framework; predicting was the only skill not intended to be taught (Ministerio de Educación 1999). Do the predicting items show a lower Textbooks can be understood as a link between intended and implemented curriculum. They serve as an interface between the national standards/objectives and the lessons put in practice by teachers at a classroom level. In Chile, textbooks are widely used by teachers, being for many the curriculum Consistent with the old curriculum, the Chilean textbook devoted extensive percent of its overall space to basic skills as knowing (41-50%) and using routine procedures (31-40%) Another focus of analysis is concerned with the number of items presented per cognitive skill. As shown in the previous figure, there is an uneven distribution of items per category. While in the two higher order skills there are six solving & predicting items and 11 using complex procedures items, in the two lower order skills there are just two items in recalling & representing and another two in performing routine procedures. This unequal distribution of items can be justified by the higher order cognitive skill emphasis of TIMSS in the assessment of 8
– Representation & interpretation of data in tables, charts, and graphs. In order to see if the items belonging to the same sub-content clustered together in the Chilean achievement profile, each one of the 21 data representation questions were classified according to these categories and then plotted in the figure 6 below. As shown in the figure, the number of items varied considerably depending upon the topic under analysis. While in representation & interpretation of data there were 13 items, means and ranges only counted with one. Probability was just in the middle, with seven items addressing this topic. Item 2 was the only means & ranges item, and it was among the easiest items in the subscale, with 70% correct responses. Representation & interpretation of data items are spread all over the range of achievement. In fact, both the easiest and the most difficult items are part of this sub-content category. Considering that more than half of the data representation items are aimed to measure representation & interpretation of data, it is not really surprising that there is not a special pattern in their distribution. The probability items appear slightly clustered together and inclined towards the lower bound of the achievement line. This is especially evident for Items 17, 18, and 19. As we will see in the next section, this pattern can be easily linked to the topics intended to be taught in the old Chilean curriculum.
In order to see if intended items clustered together among the easiest questions in the subscale, the 21 data representation questions were plotted in the achievement profile line, marking them according to their curricular status (figure 7). A clearer achievement pattern arises. According to expectations, intended items clustered together towards the easiest items, whereas not intended items clustered towards the most difficult ones. Intended items prevail from Item 1 to 14. Only three not intended items are counted among the half of easier items in the subscale: Items 5, 7, and 11. Against expectations, the most difficult item of all (Item 21) is an intended one. Once more, this is the extended response item whose classification in the data representation sub-scale was questioned. Both factors taken together –item format and topic– could explain why this item stands so far apart from the remaining intended questions in the achievement profile line.
Turning now to the unintended items, they mostly range from Items 15 to 20, presenting a percent of correct responses below 39. Separated from this group stand Items 5, 7 and 11, presenting scores even above the Chilean data representation mean of 44%. A comparison of Figures 6 and 7 reveals that not intended items include all the probability questions, plus two representation & interpretation of data items. Probability is, in fact, a topic only introduced at grade 12 in the old national curriculum. Nevertheless, representation & interpretation of data is a topic intended to be introduced from grade 6 throughout grade 8 (Ministerio de Educación 1999). Why were two items of this sub-content area (Items 16 and 20) classified as not intended? None of them required to predict –the only cognitive skill not intended in the curriculum. Even though it is always a possibility that a special type of graph or topic– beyond the curricular expectations for 8th grade students – was used in these items, this inconsistency raises questions about the validity and reliability of the data Consistently with the curricular intentions, the remaining 11 representation & interpretation of data items are presented as intended in figure 7, as well as the only item aimed to measure means & ranges (Item 2). Means & ranges was expected to be introduced early in primarily education and was supposed to be reinforced throughout grade 8 (Ministerio de Educación 1999). These curricular intentions supported the high percent of correct responses (70%) given to Item 2. In order to know if the differences in the percent of correct responses between intended and not intended items were significant, an independent sample t test of the mean was run. Since at this point there are reasonable explanations why Item 21 is so out of the pattern of the intended items, this question was excluded from the analysis. Intended items were correctly answered by almost 53% of the Chilean students, as compared to just 37,64% of correct responses obtained in the not intended items. These differences prove to be significant at
Against expectations, the figure shows no substantial differences between students who were taught the topics versus those who were not. For both groups, the achievement profile lines follow a quite similar pattern, with slight variations from item to item. In the best of cases, these differences favored the students who were already taught the topic; this actually happened in 10 out of 21 items. In the remaining 11 items, eight graders not yet taught the topic did better than their counterparts already taught. Even thought these differences are not significant, on average, students not yet taught the topics obtained better results than those who were already taught (45.01% versus 43.26%, respectively). Surprisingly, for all seven probability questions (Items 5, 7, 11, 15, 17, 18, and 19), students not yet taught the topic obtained higher percent of correct answers than those who were already taught the topic. These items also presented the seven widest differences in the percent of correct answers between both groups, ranging from 11 to 3.70 percent points in favor of the not yet taught group. ## CONCLUSIONS AND DISCUSSIONThis report presents a series of analyses aimed to foster understanding of the Chilean performance in the data representation sub-scale of the TIMSS 1999 assessment. The findings suggest a If this holds true, a validity question concerning the Chilean relative performance in the five content areas arises. Is it by chance that algebra, the sub-scale with the biggest proportion of extended response items (7 out of 35), was also the content area with the weakest results for this country? In order to avoid this kind of uncertainty, it is recommended to keep more or less the same proportion of items with a different format across the test sub-scales. The analysis of items by Turning now to the The analysis of the released items raised concern about their classification in the different content categories. Strictly speaking, if Item 21 were to measure data representation skills, its questions should be focused on the information explicitly presented in the ads, and not in the inferences the students can make through mathematical operations. Considering that the content area of fraction & number sense –together with algebra– presented the weakest results for Chile, one is lead to think that the presence of mathematical operations in Item 21 could be explaining, at least in part, the unexpectedly low p value of this question. The analysis of items by In Chile, the overall percent of correct responses in math items across the entire test was 31%, whereas its average percent of correct responses in intended items was only 32% (Mullis A reasonable explanation for these discrepancies could be that, when we talk about different scores for intended and not intended items for Chile, we are really talking about differences between two ways of learning mathematics. Representation & interpretation of data is a more “common sense” topic, from which students can pick up the rudiments either in other curricular subjects or from the media, in particular newspaper and TV (Howson 2001). Probability is more strictly tied to school mathematics. Thus, the better performance of Chilean students in representation & interpretation of data may be better explained because of their “street” experiences with this subject, and not so much because of the instruction received in the schools. Item 5 is a special case in this respect. Even though it is a probability topic, the problem presented is extremely familiar to 8 This idea is supported by the findings of the Finally, some comments about the What kind of pedagogical guidance was providing the old curriculum? Why did teachers not follow the old framework? A plausible explanation is because it was an outdated document. Mathematics has sufficiently changed since 1980, the year this curricular framework was introduced. It is reasonable to think that teachers were not following it in order to give way to new math tendencies. In fact, these new tendencies were especially promoted in the context of the development of the new curriculum. Even though this new framework was ready since 1996, the new curriculum was not supposed to be introduced at grade 8 until the school year of 2002, when a new detailed program of study, derived from the framework, will be ready for use at the school level. Thus, it is reasonable to think that, during these years, teachers have been involved in some kind of confusion and uncertainties about which curricular framework to use: the “old” or the “new”. After the introduction of the new curriculum, it should be recommended to closely monitor its implementation in the Chilean classrooms. Maybe this could help in the promotion of better instruction and learning capabilities. |