SciELO - Scientific Electronic Library Online

 
vol.39 número2Vouchers, elección y política públicaElección de escuelas e información índice de autoresíndice de assuntospesquisa de artigos
Home Pagelista alfabética de periódicos  

Serviços Personalizados

Journal

Artigo

Indicadores

Links relacionados

Compartilhar


Estudios de economía

versão On-line ISSN 0718-5286

Estudios de Economía vol.39 no.2 Santiago dez. 2012

http://dx.doi.org/10.4067/S0718-52862012000200002 

 

Artículos

 

Voucher system and school effectiveness: Reassessing school performance difference and parental choice decision-making*

Efectividad escolar y sistema de vouchers: Reexaminando diferencias de desempeño público/privado y elección parental

 

ALEJANDRO CARRASCO** ERNESTO MARTÍN***

** Faculty of Education, Pontificia Universidad Católica de Chile and CEPPE-UC, Center of Research on Policies and Practices in Education.

*** Faculty of Education and Faculty of Mathemathics, Pontificia Universidad Católica de Chile, Measurement Center MIDE UC, and CEPPE-UC, Center of Research on Policies and Practices in Education.

 


Abstract

This paper discusses the potential contribution of employing school effectiveness methodological approach within the ongoing research debate on school choice issues. Using the first approach, we estimate the effectiveness of a sample of Chilean schools after controlling by a baseline at the student level. In order to avoid the endogeneity of such a baseline with respect to the school effect, we use a longitudinal data set (SIMCE 2004 and SIMCE 2006) from which a natural pseudo-experiment is defined in such a way that the baseline is by design uncorrelated with the school effect. Thereafter, we investigate possible relationships between parental school choice (as declared in public standardized surveys) and the schools classified by their effectiveness. The main conclusions of this paper are, on the one hand, that there is not remarkable difference between municipal (public) and subsidised schools in terms of their effectiveness analyzed under value-added; and, on the other hand, that there is no relation between parental school choice preferences and school effectiveness.

Key words: School choice, Private-public schools, Value-added.

JEL Classification: 120.


Resumen

Este trabajo discute la potencial contribución de las metodologias de efectividad escolar al debate, en permanente desarrollo, sobre elección de escuelas. Usando dicha aproximaci0n metodol0gica, estimamos la efectividad de una muestra de escuelas chilenas, controlando por una línea de base a nivel de estudiantes. A fin de evitar la endogeneidad de dicha línea de base con respecto al efecto escuela, usamos datos longitudinales (las aplicaciones 2004 y 2006 del SIMCE) a partir de los cuales es posible definir un cuasi-experimento natural tal que, por diseno, dicha linea de base tiene correlaci0n nula con el efecto escuela. A partir de esto, investigamos posibles relaciones entre la elección parental de escuelas (según lo declarado en cuestionarios estandarizados) y las escuelas clasificadas de acuerdo a su efectividad. Las principales conclusiones de este trabajo son dos: en primer lugar, que no hay una notable diferencia entre escuelas públicas y subvencionadas en relaci0n a sus respectivas efectividades cuando esta se analiza con modelos de valor agregado; en segundo lugar, que no hay una relación entre las preferencias de elección parental de escuelas y la efectividad escolar.

Palabras clave: Elección de escuelas, Escuelas públicas y subvencionadas, Valor agregado.


 

1. INTRODUCTION

The motivation of this paper is to explore the contribution of recent advances in school effectiveness research (e.g. value-added methods) on the on-going voucher system research debate. The research progress in this field has offered mixed evidence mainly in relation to both private/public performance gains and the nature of parental choice preferences (see Bellei, 2007; Drago and Paredes, 2011). This paper claims that value-added models might be a methodological alternative in analysing the educational impact of market-driven mechanisms in education and offers empirical examples in such direction.

Chile was one of the first countries worldwide, at the beginning of the 1980s, to introduce voucher system reforms. The fundamental principle that encouraged the school choice reform in the 1980s was to incentive competition among schools as a mechanism for improving the quality of schooling. It was also expected that parental freedom to choose (following school quality) would lead to pressure schools to improve their performance and efficiency raising national educational standards in particular those of most disadvantaged students in the worst performing schools. Such theoretical premises have become worthy hypotheses to be empirically tested and called the attention of researchers during last two decades. Chilean market-driven mechanisms have commonly been analysed around four core questions: (i) the impact on socioeconomic segmentation; (ii) the impact of vouchers on the efficiency of the system; (iii) the difference public-private schools in terms of school performance on standardised tests; and (iv) the factors influencing parental choice. Research findings have been controversial and inconclusive.

This paper is focusing on the previous questions (iii) and (iv) using value-added models in order to address two central methodological aspects. Firstly, albeit in Chile effectiveness seems to be equated to average SIMCE performance, the paper estimates the effectiveness of schools using value-added models and comparing such effectiveness with average SIMCE scores. Concretely, one of the most striking problems in value-added models, it is the need that prior attainment should fulfill the property of exogeneity. This condition is a crucial challenge to attribute actual effectiveness to one school and this paper addresses it using a particular SIMCE's panel data set.

Secondly, while this contribution is empirically based on a reduced sample of secondary schools receiving students from primary schooling in 2005 (SIMCE 2004-2006), it follows recent advances in school choice research to overcome the limited use of cross-sectional data set to offer claims about school effects (Chumacero, Paredes and Valin, 2011). In addition, overcoming this limit to assess the student progress in two different time periods, Lara, Mizala and Reppeto (2009) used SIMCE data set (2004-2006), employing propensity score and CIC methods. However, they used a reduced sample of schools, including only students coming from municipal/public schools and going to private schools to analyse its performance (i.e., municipal schools excluded). Likewise, Chumacero, Paredes and Valin (2011) used similar data set (SIMCE 2004-2006) but to assess the impact of shared funding on student performance. Thus, attempting goes further, this paper used similar data set (SIMCE 2004-2006), including near to 48% of students moving from one school to another since this panel data set offers sound conditions to undertake reliable value-added analysis in order to take into account the specific contribution of school practices on student performance when using prior attainment scores.

This paper is organized as follows: section a discusses the relevance of bringing conceptual and methodological tools from school effectiveness research to the school choice debate. Section 3 presents the conceptual issues underlying school value-added analysis. Section 4 describes the specific data sets and discusses the results of a value-added analysis. Section 5 offers evidence on the public/private performance gap, whereas section 6 does similarly on parental choice decisions. This papers ends, section 7, with a general discussion on the results and implications.

2. THE SEARCHOF SCHOOL EFFECTS M MARKET-DRIVEN POLICY MECHANISMS

There are two important shortcomings identified in school choice research in Chile: the misleading estimation of actual school quality and the absence of longitudinal dataset to compare students' learning over time.

Firstly, following a core of international literature, value-added models have stronger advantages to estimate and compare the school differences in terms of quality or effectiveness. Most research on Chilean voucher system has not considered such educational assumption. At least theoretically the ultimate policy purpose of market-based educational reforms is to increase the quality of education in terms of academic gains of students, measured by standardised tests. As a mechanism of public policy, the voucher scheme would foster greater competition among public and private subsidised schools to capture students, which would turn out to be a strong stimulus for schools in order to improve their 'school effectiveness', that is, to offer better learning experiences, independent of students' prior attainment and social backgrounds. At the very core of the system, market-based mechanisms would allow schools to get rid of bureaucracy limits or political influences in order to focus on better management and classroom practices (i.e., school-based practices), which a non-competitive system restrains. As a result of such a market-based system, more effective schools will remain, whereas ineffective schools will disappear. Taking into account these set of educational assumptions underlying this policy, it seems plausible to evaluate it from a school effectiveness approach. School effectiveness research (hereafter, SER) has had a strong development from 80's to date (Rutters et al., 1979; Teddlie and Stringfield, 1993; Teddlie and Reynolds, 2000; Townsend, 2007), producing a large international corpus of literature on school effectiveness. One of the most important SER's contributions has been the development of HLM and value-added models to assess school effectiveness (Goldstein, 2001; Creemers, Kyriakides and Sammons, 2010). A typical way in which most of this research measures the school performance is to compare the progress that students make between two test occasions using value-added analysis; see, among many others, Steele, Vignoles and Jenkkins (2007), OECD (2008), Braun, Chudowsky and Koening (2010) and EPI Briefing Paper (2010). Concretely, in order to calculate value-added, a measurement of the value of both outputs and inputs is required. The basic idea of a value-added is to calculate the relative value of the inputs and the value of the outputs relative to the inputs. Value-added models are, therefore, characterized by the inclusion of the previous attained score of the students (the input) as a control variable. It can also be remarked that the value-added analysis seems to be meaningful from an educational point of view due to the explicit inclusion of students' learning in value-added models.

Secondly, although Chilean voucher scheme has been largely studied, it has not been examined using longitudinal data. Research has used mainly cross-sectional data. In the past, this limitation has been due to the lack of longitudinal data for the Chilean educational system. However, it is now possible to have access to longitudinal data at the student level. In fact, in 2004, the SIMCE test was applied to student from the 8th level of primary school. Two years later, the same students were measured by the 2006 application of the SIMCE test.1 These data sets allow undertaking value-added analyses of the Chilean educational system and, therefore, to estimate the value-added indicators of the schools on the basis of student progress. However, it is important to keep in mind that these value-added indicators represent the effectiveness of the schools under specific structural hypotheses conditions, only. Such hypotheses can empirically be justified using the 2004 and 2006 SIMCE applications; for details, see section 3 below.

Two analytical strategies will be followed in the rest of the paper. On the one hand, a common methodological criticism within literature attempting to examine the performance differences between private/public schools is the selection bias.

In doing so, this paper argues that 'prior attainment', as most SER literature indicates, has a more significant explicative power (see section 5). On the other hand, focusing on parental choice decision-making research has not compared what has been denominated the 'stated' parental preferences of chosen schools to the actual school performance measured in terms of real school value-added to understand to what extent parental choices pressure to schools to improve their performance, as theory would expect (see section 6).

In order to assess the potential contribution of value-added models to school choice research, and prior to present the results in sections 5 and 6, a careful discussion about the value-added model approach and empirical results for the Chilean case are addressed in section 3 and 4.

3. VALUE-ADDED MODELS: CONCEPTUAL ASPECTS

3.1. Structural hypotheses underlying HLM models

Value-added models are typically specified through a multilevel model, also called hierarchical linear model (hereafter, HLM). This is mainly motivated by the hierarchical structure of the data, namely that students are nested into schools. HLM models intend to explain a dependent variable not only by observable explanatory factors, but also by an unobservable factor (typically called school effect) defined at the school level; see, among many others, Goldstein (2001) and Tekwe et al. (2004).

In order to make explicit the structural hypotheses underlying HLM models, denote by Yij the contemporaneous score of student i belonging to school j, where i = 1,... nj and j = 1,...J. Other explanatory factors are typically collected, either at the school level or at the student level, and we denote by Tij the vector of these explanatory factors. This vector contains a constant (the intercept) and the prior attainment score of each student. If school-level variables are present in the vector Tij , these components do not depend on i. Finally, let Tj = (T1j,..., Tnjj) the matrix collecting the factors of all students belonging to school j.

HLM models are specified through a marginal-conditional decomposition of the following type (Snijders and Bosker, 1999; Goldstein, 2001; Raudenbush and Bryk, 2001):

where β is an unknown parameter and, conditionally on (Tj θj), the contemporaneous scores of students belonging to school j are mutually independent.

This last condition on mutual independence, called local independence (Lazarsfeld, 1950), leads to understand the meaning of the school-effect: it means that, conditionally on Tj, all the within relationships between the contemporaneous scores Yij's should be accounted for by the way in which each score Yj alone is related to the unobservable variable θj. The school effect θj captures, therefore, the heterogeneity that is present in the contemporaneous scores and that is not fully explained by the factors Tj.

A key property implied by the previous specification is that the covariates Tj are uncorrelated with the school effect θj. This means that the covariates are exogenous (Engle, Hendry and Richard, 1983) with respect to the school effect and, consequently, the school effect can be interpreted as the contribution of the school to the contemporaneous score Yj after controlling by the covariates Tj.

3.2. Features of the data to be used to perform a value-added analysis

The concept of value-added seeks at measuring the gain or the loss of being in a given school with respect to an "average" school; see Raudenbush and Willms (1995), Raudenbush (2004) and Timmermans, Doolaard and Wolf (2011). Following Manzi, San Martín and Van Bellegem (2012), a structural definition of the value-added of school j is given by.

The first term represents an average of the expected score conditional on observed explanatory factors and the school effect. The second term is an average of the expected score conditional on the observed explanatory factors only. This last term integrates out the school effect and, consequently, can be interpreted as the expected score of a student who would be treated by an average school. This average school is characterized by the observable explanatory factors which are considered in the model. Thus, a positive (respectively, a negative) value-added means that a school can take students beyond (respectively, below) of what have achieved if they would have attended the average school.

If (some of) the explanatory factors are endogenous (that is, correlated with the school effect), the value-added of a school is no longer equivalent to the school effect. In value-added analysis, this endogeneity could arise when some students have been treated by the same school during the two test occasions. If the prior attainment score is measured when students were already treated by a school, it is likely that such prior score measure contains the effect of the school. In the Chilean case, this problem is not exceptional: there exists national panel data at the individual level (SIMCE 2005 and 2009 applications; SIMCE 2007 and 2011 applications) in which most of the measurements were taken while students were at the same school in both tests occasions.

If the explanatory factors are exogenous, the value-added of a school is exactly equivalent to the school effect. In the Chilean case, it is also possible to have a large sample where the prior attainment score is exogenous with respect to the school effect. This paper takes advantage of this possibility offered by the Chilean case and focuses its value-added analysis on a sample of schools such that the prior attainment score is exogenous with respect to the school effect. The endogenous case will be considered in future research.

The previous considerations lead to define specific criteria under which a data set can be chosen in such a way that a value-added analysis is possible without resorting on endogeneity problems. These criteria are the following. First, the exogeneity of the prior attainment score is ensured if a full mobility of students between the two tests occasions is ensured. Second, a full mobility of students offers the possibility to look for possible relationships between parental school choice and school effectiveness. Third, a fair comparison between the effectiveness of schools is ensured if the average school (which is the reference in a value-added analysis) is characterized by explanatory factors determining selectivity process.

4. VALUE-ADDED ANALYSIS OF THE CHILEAN EDUCATIONAL SYSTEM

4.1. Defining the sample of schools for the empirical analysis

Taking into account the previous discussion, we use the 2004-2006 SIMCE applications since one of the features of this panel data is students' migration from one school to another between 2004 and 2006. This mobility is mainly due to the fact that in 2004 students finished the primary educational cycle at the 8th grade. Table 1 shows that 70% of students moved between 2004 and 2006. It can also be appreciated that both the 2004 and 2006 scores of students who moved is, in average, lower than the scores of students who did not move.


TABLE 1
STUDENTS SCHOOL MOVEMENT BETWEEN 2004 AND 2006

In order to avoid the endogeneity of the prior attainment score, we selected schools such that 100% of their students are new; that is, students who in 2004 attended a school different from the current school in 2006. Of the 1,886 initial schools with at least a0 students, the 31.9% of them (that is, 601 schools) satisfy this condition, which attend 79,093 students. This sample is composed of 9a public schools of type I (that is, schools which are financed by the state and administered by county corporations), a66 public schools of type II (that is, schools financed by the state and administered by county governments) and a43 subsidized schools (that is, schools which receive the voucher, but the administration depends on private organizations)2; private schools are excluded from this data set due to the small student mobility. Moreover, 193 schools have a socio-economic status (thereafter, SES) A, which corresponds to the lower SES; a84 schools have a SES B; 104 schools have a SES C; and a0 schools have a SES D3.

A selectivity mechanism widely used by schools, at least in 2006, is the selectivity by ability. During the SIMCE application, the Ministry of Education also applies a questionnaire which is supposed to be answered by parents at home. In their questionnaire, parents are asked whether a test of knowledge on their child was organized when they applied for the school. Schools, which use this mechanism of selection, are free to decide to whom to apply such a test. Using this information, we compute the proportion of parents declaring that their children were selected by ability; in the sequel, this variable is called Select. If Select is at least equal to 0.5, it is said that the school is selective. Under this criterion, 39.1% of schools under study are selective: 51% of these schools are subsidized and 49% are public.

4.2. Results from a value-added analysis for the Chilean case

Using the sample of 601 schools, a value-added analysis was performed by fitting a heteroscedastic multilevel model, in which the variance of the school effect and the variance of the idiosyncratic error depend on SES.4 According to the discussion developed in Section 3, the explanatory factors included in the value-added models are the following: The prior attainment score (Mat04j). The socio-economic group (SES) as defined by the SIMCE office; it corresponds to a categorical variable with five levels: A, B, C, D, and E, being category A the reference. Selectivity of a school (Select). The average of Mat04j over students in each school j in 2006, denoted as Avmat04; this factor controls selectivity mechanism which are not reported by parents.

Six different specifications of value-added models were fitted (see Table 2). Considering the AIC criterion, the HLMab specification should be chosen as the better model. This specification includes factors that control selectivity, namely Select and Avmat04j. The prior attainment score Mat04j is an individual variable that explains the 45% of the individual variance for each SES, as well as more than the 50% of the between-variance, being more than 70% for the higher socio-economic levels. The marginal effect of Mat04ij is equal to 0.8, whereas the marginal effect of Avmat004 is equal to 0.6. It is relevant to mention that Avmat04j is a compositional effect which helps to control the selectivity in the sense that if Select is excluded from the value-added models, Avmat04j still controls selectivity mechanisms. This is concluded from that fact that the fixed effects corresponding to Select decreases in the presence of Avmat04j. Moreover, the fixed effects corresponding to the SES are dramatically affected by the presence of Select and Avmat04j.


TABLE 2
RESULTS FROM VALUE-ADDED ANALYSIS

Let us finish this section by mentioning that the explanatory factors other than the prior attainment score Mat04j and the compositional effect Avmat04j are exogenous with respect to the school effect Bj. This was verified by computing the γ-factor proposed by Loockwood and McCaffrey (2007). This factor is equal to the ratio p/[1+p(T-1)], where ρ is the intra-class correlation and T is the number of students by class. For each socio-economic group, yT ~ 1, which means exogeneity of the explanatory factors.

Remark. A possible objection to value-added analysis is the presence of ceiling scores. If both the previous attainment score and the contemporaneous score of a student are almost at the bottom of the scale, it could be said that the value-added of the school is near to 0. However, as it can be concluded from identity (3), the value-added of school is the same for all students attending a school.

Thus, in the improbable scenario that all students of a school obtained ceiling scores at both measurements, and that all such students are characterized by the same covariates, then it is plausible to expect a zero value-added. In empirical studies, as the one discussed in this paper, this is clearly a situation far to be realistic. It is plausible that some students get ceiling scores, but being the value-added of a school the same for all the students, such ceiling scores will not produce a zero value-added.

5. COMPARING PERFORMANCE BETWEEN SUBSIDISED AND PUBLIC SCHOOLS FROM A VALUE-ADDED ASSESSMENT

5.1. Effectiveness of subsidised and public schools: results and discussion

A relevant question, especially for the Chilean context, is to know whether subsidised schools are more or less effective than public schools. While some have found gains in favour of private subsidized schools (Sapelli and Vial, 2002; Anand, Mizala, and Repetto, 2006); other after controlling for selection bias and peer effects have found a positive and small private-public gap (Carnoy and McEwan, 2000; Drago and Paredes, 2011; McEwan, 2003).

Novel partial evidence can be provided by using the value-added analysis developed in the previous section. It is partial because we are using a sample of 601 schools (from which private schools are excluded by design), but its strength is mainly based on the exogeneity of the prior attainment score. Figure 1 shows the boxplots of the value-added indicators for each type of school, namely public schools (type I and II) and subsidized schools. Their inter-quartile range is practically equivalent, although for subsidized schools the first quartile is lesser than the first quartile of public schools. For public schools type I, the mean of the value-added is equal to 1.16 (std. dev. 1a.9), for public schools type II the mean is equal to 0.06 (std. dev. 11.6) and for subsidized schools the mean is -0.51 (std. dev. 14.07). The conclusion is, therefore, straightforward: for each type of schools, their effectiveness is essentially independent from the type of school.


FIGURE 1

The previous result provides evidence to claim that the school performance is independent of the school administrative dependence. That is, it seems that subsidised private schools do not perform better than public schools, and reciprocally, at least for the sample under study. This confirms previous findings but it does by means of, and adding to, a different conceptual and methodological strategy to estimate school effectiveness.

5.2. Coherence between school effectiveness estimated under SIMCE scores and under value-added: results and discussion

Regarding school effectiveness, it is relevant to ask how coherent are the performance of schools when it is estimated either with their value-added indicators, or with their average 2006 SIMCE scores. If schools are ranked using both their value-added indicators and their average SIMCE scores, a global measure of agreement is given by the Spearman, which is equal to 0.43. A more precise way to depict this degree of agreement is to classify schools into four types of performance according to the quartiles of the distribution of value-added indicators as well as the distribution of averages SIMCE scores. Thus, for instance, if the value-added indicator of a school is smaller than the first quartile, the school is of type 1; and so on. Similarly, for the average SIMCE score.

Table 3 summarizes the percentage of schools being classified by their value-added and their average SIMCE score. Globally, 37.3% of schools maintain their classifications; 3a.9% of schools appear to be more effectives when their effectiveness is computed using a value-added model instead of being approached by their average SIMCE score. Contrary, a9.8% of schools appear to be less effective when their effectiveness is computed using a value-added model instead of being approached by their average SIMCE score. Thus, two thirds of schools change their classification, whereas one third maintains it.


TABLE 3
CLASSIFICATION OF SCHOOLS ACCORDING TO VALUE-ADDED AND AVERAGE SIMCE SCORE

The previous misclassification can be described by type of school. Regarding schools which maintain their classifications, the percentages are the following: 34.8% of public schools type I; 37.a% of public schools type II; and 38.a% of subsidized schools. Regarding schools which appear to be more effective when their effectiveness is computed using a value-added model instead of being approached by their average SIMCE score, the percentages are the following: 4a.4% of public schools type I; 34.6% of public schools type II; and a7.6% of subsidised schools. Finally, regarding schools which appear to be less effective when their effectiveness is computed using a value-added model instead of being approached by their average SIMCE score, the percentage are the following: aa.8% of public scores type I; a8.a% of public schools type II; and 34.a% of subsidised schools.

It can be concluded that most of the public schools are proportionally more effective in value-added terms than the pseudo-effectiveness measured by the average SIMCE scores; whereas most of the subsidised schools are proportionally less effective in value-added terms than the pseudo-effectiveness measured by the average SIMCE scores5.

6. PARENTAL CHOICE: ARE ACTUALLY PARENTS CHOOSING THE MOST 'EFFECTIVE' SCHOOLS?

6.1. Parental choice research in Chile

Parental choice practices are a key component of the voucher scheme system because it is assumed that parents (or a group of them) choose a school based on its quality. Chilean parental choice research, mostly developed since 2000's, has accordingly focused its attention on this assumption, namely whether parents predominantly choose based on 'education quality' criteria or, rather, they make decisions based on aspects like distance, price, or average socioeconomic level of schools. Most of this research has been conducted using surveys or secondary information about some general attributes of chosen schools.

Following Carnoy and McEwan (2000), Elacqua and Fabrega (2004) conducted a pioneer research through a survey asking retrospectively to parents reasons to choose a school. They elaborated a 'choice set' of family preferences, which then were compared with characteristics of the school chosen by families. Using the contrast between 'stated' and 'revealed' preferences, this study was a step forward from previous mere surveys in which reasons to choose schools were asked to parents (CEP, 1997). Subsequently, following similar strategy, Elacqua, Schneider and Buckley (2006) using data from a survey in the Metropolitan area of Santiago of what they called indicators of 'actual' behaviour, compared parents' choice sets to a set of school attributes parents declared they were looking for when choosing a school. Elacqua and Fabrega (2004) and Elacqua et al. (2006) found that parents decisions are based mainly on the social composition of the student body, instead of, as expected, the academic attributes of schools. However, Gallego and Hernando (2007), using a different methodology and distance considerations, found different findings pointing out that quality is a crucial aspect in family choices.

Chumacero, Gómez and Paredes (2011) improve the previous research due to Gallego and Hernando (2007). As a matter of fact, Chumacero et al. (2011) not only use the distance of what they call 'anecdotic' data provided by surveys, but also actual data about parental decisions in order to consider distance and quality as two related factors. Using both the CASEN database and neighbourhood blocks data (Mapcity), Chumacero et al. (2011) analysed the actual parental choice decisions, concluding that quality, price and distance are key factors when parents choose schools. Following a similar methodological strategy, Chumacero, Gomez and Paredes (2012) went further focusing at this time on the information provided by the SIMCE test on school choice. Contrasting the 1996 to 2003 SIMCE data sets, they conclude that the information revealed by publishing SIMCE results was relevant, but that previous to that, parents in 1996 acted "as if' they knew the school quality they chose.

6.2. Parents' choice when actual school effectiveness is available

A contribution to parental choice research can be provided by the value-added analysis developed in this paper. Two reasons support this possibility: on the other hand, our value-added analysis is based on a data set characterized by full students' mobility between 2004 and 2006; on the other hand, the stated preferences of parents, leading to choose a school for their children in 2006, are available. Taking into account these data, we explore to what extent effective schools were chosen by parents who declared had chosen a school due to its academic quality. This question is of speculative character since the value-added indicators are not publicly available, but it can be considered as an empirical evaluation of the stated preferences of parents regarding school quality.

In each SIMCE application, the Ministry of Education includes a survey to be responded by families. Parents are explicitly asked to declare the three main reasons leading to choose the actual school for their children. These reasons are the following: Proximity; Other members of the family are were in those school; School prestige; Good results in the SIMCE national test; Good results in the PSU test (the national university entrance test); Good socio-economic and cultural status of the students; Costs of the schools are achievable; The school is the only one in the commune.

The value-added analysis developed previously is based on a data set mainly characterized by a full students' mobility 2004 and 2006. Consequently, it is possible to use the information regarding parental preferences to choose a school, as collected by the SIMCE survey in 2006. In spite of the explicit requirement of choosing the three main reasons, most of the respondents choose one, two, three or more answers. This information is summarized in Table 4 (where the percentages are computed with respect to the 79,073 students).


TABLE 4
REASONS BY WHICH PARENTS CHOOSE A SCHOOL
(According to the 2006 survey)

The most mentioned reasons are good results in the SIMCE national test; good results in the PSU test; good socio-economic and cultural status of the students; and that the school is the only one in the county. These responses are correlated between them, which suggests to undertaking a factorial analysis in order to simplify this information.

A principal component analysis was accordingly performed, and the rotated solution with two factors was chosen. The first factor, which explains the 51.5% of the variability, groups the following questions: good results in the SIMCE national test; good results in the PSU test; school prestige; and good socio-economic and cultural status of the students. The second factor, which explains the 48.9% of the variability, groups the following questions: proximity; the school is the only one in the county; and costs of the schools are achievable. Thus, the first factor is called multiform quality, because of a diversity and inseparable set of symbolic and educational aspects to choose schools, are grouping in it. Whereas the second factor represents common material and spatial access defining parental choices. Using standard procedures, the factor scores were also estimated. The following aggregate variables at the school level were subsequently computed: the mean of parents declaring that they choose a school by quality; the mean of parents declaring that they choose a school by access; the mean of factor 1 scores; the mean of factor a scores; the factor 1 score at the first quartile and at the third quartile; the factor a score at the first quartile and at the third quartile; the interquartile range for factor score 1; and, finally, the interquartile range for factor a score.

In order to find some possible relation between effectiveness and school choice, the previous variables were correlated with both the value-added indicators of schools and the average 2006 SIMCE score. These correlations were computed for the overall sample and for the subgroups defined by public schools and subsidized schools. No correlation was statistically significant, except when average SIMCE scores were considered; see Table 5.


TABLE 5
CORRELATIONS (P-VALUE) BETWEEN PARENTS 'SCHOOL-CHOICE INDICATORS AND SCHOOL EFFECTIVENESS INDICATORS

Regarding the absence of relationship between value-added indicators and parents' school choice, it can be said that the value-added is not public information and, consequently, no parental reaction to such information could be expected. However, taking into account that parents declare their choice at the end of 2006 (that is, contemporaneously to the final period with respect to the effectiveness of a school is estimated), it would be expected that, in a retrospective view, the effectiveness of schools (as estimated by a value-added models) had been perceived by parents. In any case, we did not find relationships between the actual effectiveness of schools and both multiform quality factor and access factor.

Regarding the absence of relationship between average 2006-SIMCE scores and parents' school choice, a plausible explanation might be the following: when parents declare that their school choices are motivated by quality, it should be considered that, by the end of 2006, the only public information about the SIMCE performance of secondary schools was that informed in 2003. In fact, this year schools were measured at the 10 grade at secondary level. However, as can be verified in Table 5, parent's motivation to choosing schools are not only uncorrelated with actual effectiveness, but also it is uncorrelated with both the average 2006 SIMCE score and the average 2003 SIMCE scores. It should be mentioned that the 2003 SIMCE information is available for 494 of the 601 schools of the sample under study. These negative results might illustrate how unreliable is the information related with school choice which is collected by the SIMCE questionnaire. And also, the uninformed way under which parental school choice is practiced in Chilean school system. In other words, voucher scheme and parental choice mechanism seem to work on opaque basis in terms of school performance information available for families.

7. IMPLICATIONS AND FINAL REMARKS

Research has been controversial in responding if educational markets work in reality. However, no research has accurately measured school effectiveness in terms of value-added models as this paper does. The panel data set used here, and particularly the sample of schools, allow us to undertake an empirical investigation about the possible relationships between school effectiveness and parental school choice. This is due to the fact that in 2004 parents chose schools and in principle used public information about the SIMCE. Our analysis is limited because it is based on a sample of schools from the 2004-2006 SIMCE data sets (30% of schools). Although it implied a partial examination of school choice issues, the advantage of these data sets is that it satisfies a key requirement, namely the exogeneity of the prior attainment score.

This paper concludes that, firstly, school performance measured in terms of value-added is independent of the administrative nature of schools. While they could have impacted on a diverse sort of benefits, when using pseudo-effectiveness measures (average SIMCE scores) the gains of subsidized schools over municipal schools are overestimated. Although this finding has been reported before, this is one of the first empirical studies employing longitudinal dataset and analysing them using more accurate measurement of school effects (i.e. VA).

Secondly, parental choice seems not to be influenced by academic reasons as we found inconsistencies between actual schools' performance and parents' preferences who declare they chose them by multiform quality reasons where academic reasons are included. While Chumacero et al. (2012) found, employing utility models, that revealed preferences (or actual choice) is informed by academic reasons, we found utilising a different strategy that there is no relationship between school effectiveness (as estimated with VA models) and multiform quality and access factors. Similarly, there is no relationship between school performance (as represented by the SIMCE outcomes) and both parental choice factors. This finding may be explained by the fact that parental information on choice preferences as collected by the SIMCE questionnaire might to some extent be unreliable to measure actual quality. Then, it seems that declared preferences about choice are not informative. An alternative interpretation is that actual school effectiveness seems to be unperceived by society, families, and communities whose decisions to assess school quality are multiform and multidimensional in nature. As a consequence, parental choice seems to be functioning on a set of diverse school attributes rather than on school academic quality only as commonly hypothesised by economic theory. Then, it could be suggested that a value-added system of public information would contribute to support parental choice. However, this is a fairly preliminary hypothesis which would require, on the one hand, implementing a national accountability system based on value-added models, and, intensive research to assess whether actual school effectiveness information turns out to be more relevant than multiform quality factors shaping parental choices, on the other.

Finally, these results suggest that previous research on school choice has employed a limited definition of school quality to test the contribution of market-based reforms on school improvement. The available evidence on the impact of school choice policies has not accurately been measuring school effectiveness. In general, such research has measured performance in terms of SIMCE average scores using: cross-sectional dataset without capturing learning over time, using data at school level, comparing school progress over time without individual measures, or measuring school effects under endogeneity problems.

Markets-based reforms have a strong role in the functioning of Chilean educational system, which make necessary based-evidence policy decisions. This paper contributes with additional analytical strategies for future research to assess the contribution of voucher mechanisms in education.

NOTES

* Previous stages of this work were presented during the workshop Elección de Escuelas, Competencias, y Resultados (September 12, 2011, Santiago, Chile) where we received worthy comments from Cristian Bellei on an early draft. We thank the Guest Editor, Ricardo Paredes, for helpful discussions leading to improve this paper. The authors also are grateful to CONICYT for the partial financial support of the FONDECYT Project N° 1110315 Schools Effectiveness and Value Added Models: From Quantitative Analysis to Qualitative Outcomes. The authors are also grateful to the SIMCE Office of the Chilean Ministry of Education for providing access to the database. All opinions and conclusions expressed in this paper are of exclusive authors' responsibility.

1 It is also the case with the 2005 (4-th level of primary school) and 2009 (8-th level of primary school) SIMCE applications, as well as with the 2007 (4-th level of primary school) and 2011 (8-th level of primary school) SIMCE applications.It is also the case with the 2005 (4-th level of primary school) and 2009 (8-th level of primary school) SIMCE applications, as well as with the 2007 (4-th level of primary school) and 2011 (8-th level of primary school) SIMCE applications.

2 It should be mentioned that this sample excludes the emblematic public schools, as the Instituto Nacional José Miguel Carrera or the Liceo Carmela Carvajal de Pratt. These emblematic and exceptional public schools are known to have highest SIMCE scores at national level.

3 At the individual level, it seems relevant to mention that 17.64% of students move from public to subsidised schools between 2004 and 2006; 16.12% of students move from subsidised to public schools between 2004 and 2006; and 66.a4% of students maintain their type of schools. Regarding the SES of school, 9.18% of students increase the SES; 46.3a% of students decrease the SES; and 44.5% of students maintain the SES. In other words, for the students attending this sample of 601 schools, it can be said that the social mobility is not a key feature.

4 The estimations as well as the computation of the value-added indicators were obtained using the Proc MIXED from SAS.

5 It has been internationally accepted that raw school performance data is a biased and unfair measure of school effectiveness. Instead, value-added models are being increasingly used for educational and accountability purposes (see San Martín y Carrasco 2012, for a further discussion)

REFERENCES

Anand, P., A. Mizala and A. Repeto (2006). "Using School Scholarships to Estimate the Effect of Government Subsidized Private Education on Academic Achievement in Chile," Documentos de Trabajo 220, Centro de Economía Aplicada, Universidad de Chile.         [ Links ]

Bellei, C. (2007). "Expansión de la educación privada y mejoramiento de la educación en Chile: Evaluación a partir de la evidencia", Revista Pensamiento Educativo, 40, pp. 285-311.         [ Links ]

Braun, H., N. Chudowsky and J. Koening (2010). Getting Value Out of Value Addded: Report of a worfahop.Washington D.C.: The National Academics.         [ Links ]

Carnoy, M. & P. McEwan (2000). "The Effectiveness and Efficiency of Private Schools in Chile's Voucher System". Educational Evaluation and Policy Analysis. Vol. 22 (3), 213-239.         [ Links ]

CEP (1997). Estudio Nacional de Opinión Pública, noviembre-diciembre 1996. Incluye tema especial: "Educación en Chile: ¿Qué piensan los padres?", Centro de Estudios Públicos, N° 268, junio.         [ Links ]

Chumacero, R., R. Paredes y A. Valin (2011). Financiamiento Compartido y Desempeno Escolar. Manuscrito. Escuela de Ingeniería, Pontificia Universidad Católica de Chile.         [ Links ]

Creemers, B., L. Kyriakides and P. Sammons (2010). Methodological advances in educational effectiveness research. Routledge.         [ Links ]

Drago, J. y R. Paredes (2011). "The quality gap in Chile's education system". Santiago: CEPAL Review, 104, August, pp. 161-174.         [ Links ]

Elacqua, G. y R. Fabrega (2004). "El consumidor de la educación: actor olvidado de la libre elección de escuelas en Chile". En: PREAL Educación y brechas de equidad en América Latina. Vol. 2, pp. 353-398.         [ Links ]

Elacqua, G., M. Schneider and J. Buckley (2006). School Choice in Chile: Is it Class or the Classroom? Journal of Policy Analysis and Management, vol. 25 (3), pp. 577-601.         [ Links ]

Engle, R. E., D. F. Hendry and J. F. Richard (1983). Exogeneity. Econometrica, 51, 277-304.         [ Links ]

EPI Briefing Paper (2010). Problems with the Use of Student Test Scores to Evaluate Teachers. Washington D.C.: Economic Policy Institute.         [ Links ]

Gallego, F. y A. Hernando (2007). "School Choice in Chile: Looking at the Demand Side". Manuscrito. Instituto de Economía, PUC.         [ Links ]

Goldstein, H. (2001). Multilevel Statistical Models, London: Edward Arnold.         [ Links ]

Gómez, D., R. Chumacero y R. Paredes (2012). "School Choice and Information", Estudios de Economia 39 (2), 115-122.         [ Links ]

Lazarsfeld, P. F. (1950). The Logical and Mathematical Foundation of Latent Structural Analysis. In: S. S. Stout (ed.), Measurement and Prediction (pp. 362-412). Science Editions.         [ Links ] Lookcwood, J. R. and D. F. McCaffrey (2007). Controlling for individual heterogeneity in longitudinal models, with applications to student achievements. Electronic Journal of Statistics, 1, 223-252.         [ Links ]

Manzi, J., E. San Martín and S. Van Bellegem (2011). School System Evaluation by Value-Added Analysis Under Endogeneity. Conditionally accepted in Psychometrika.         [ Links ]

McEwan, P. (2003). "Peer effect on student achievement: evidence from Chile", Economics of Education Review, vol. 22, N° 2, Amsterdam, Elsevier.         [ Links ]

McEwan, P. and M. Carnoy (2000). "The effectiveness and efficiency of private schools in Chile's voucher system", Educational Evaluation and Policy Analysis, 22 (3), pp. 213-239.         [ Links ]

Lara, B., A. Mizala y A. Reppeto (2009). The effectiveness ofprivate voucher education: evidence from structural school switches. Centro de Economía Aplicada, Universidad de Chile.         [ Links ]

OECD (2008). Measuring Improvements in Learning Outcomes. Best Practices to Asses the Value-Added of Schools. Organization for Economic Cooperation and Development OECD.         [ Links ]

Raudenbush, S. W., J. D. and Willms (1995). The Estimation of School Effects. Journal of Educational and Behavioral Statistics 20, 307-335.         [ Links ]

Rutter, M., B. Maughan, P. Mortimore and J. Ouston (1979). Fifteen thousand hours: secondary schools and their effect on children. Cambridge MA: Harvard University Press.         [ Links ]

San Martín, E. y A. Carrasco (2012). "Clasificación de escuelas en la nueva institucionalidad educativa: contribución de modelos de valor agregado para una responsabilización justa", Temas de la Agenda Pública, Centro de Políticas Públicas UC, año 7, N° 53, julio.         [ Links ]

Sapelli, C. and B. Vial (2002). "The performance of private and public schools in the Chilean voucher system", Cuadernos de Economia, 39 (118), Pontificia Universidad Católica de Chile, Santiago.         [ Links ]

Steele, F., A. Vignoles and A. Jenkins (2007). The effect of school resources on pupil attainment: a multilevel simultaneous equation modelling approach. Journal of the Royal Statistical Society, Series A, 170, 801-824.         [ Links ]

Teddlie, C. and D. Reynolds (eds.) (2000). International Handbook of School Effectiveness Research. London and New York: Falmer.         [ Links ]

Teddlie, C. and S. Stringfield (1993). Schools Make a Difference: Lessons Learned from 210 Year Study of School Effects, New York: Teacher College Press.         [ Links ]

Tekwe, C.D., R.L. Carter, C.X. Ma, J. Algina, M.E. Lucas, J. Roth (2004). An Empirical Comparison of Statistical Models for Value-Added Assessment of School Performance. Journal of Educational and Behavioral Statistic 29, 11-36.         [ Links ]

Timmermans, A., S. Doolaard and I. Wolf (2011). Conceptual and empirical differences among various value-added models for accountability. School Effectiveness and School Improvements, 22, 393-413.         [ Links ]

Creative Commons License Todo o conteúdo deste periódico, exceto onde está identificado, está licenciado sob uma Licença Creative Commons