Catálogo de cuentas

The student data analyzed in this dissertation (test scores, background characteristics, etc.) was collected in a large-scale observational study of nationally representative samples, i.e., as part of the TIMSS 1995, 1999 and 2003 cycles, without random assignment to treatment and control conditions.

Randomized controlled trials (RCTs) and quasi-experimental studies are terms

often used in the educational research literature since the Education Sciences Reform Act of 2002 demanded “the transformation of education into an evidence-based field”

(Schneider, Carnoy, Kilpatrick, Schmidt, & Shavelson, 2007, p. 5). Random assignment of participants to treatment and control groups provides RCTs with “the most powerful design for detecting treatment effects” (Schneider et al., 2007, p.11). The authors continue to explain that “the random assignment of participants to treatment conditions assures that treatment group assignment is independent of the pretreatment characteristics of group members; thus differences between groups can be attributed to treatment effects rather than to the pretreatment characteristics” (p.11).

On the other hand, quasi-experiments represent a group of “comparative studies that carefully attempt to isolate the effect of an intervention through means other than randomization” (Schneider et al., 2007, p. 4). An example of such studies is the case of

large-scale observational studies of nationally representative samples, such as TIMSS, Progress in International Reading Literacy Study (PIRLS), or the National Education Longitudinal Study of 1988-2000 (NELS). When analyzed with proper statistical techniques, such datasets may be collected even from settings where RCT may be logistically, financially, and even ethically not possible. Schneider et al. (2007) point out to the numerous advantages of the large-sale observational studies, including their generalizability to specific populations, their usefulness in making plausible hypotheses about the causes of achievement gaps, etc.

One fundamental problem that stems from the lack of random assignment to treatment conditions in observational studies is known as sample selection bias Schneider et al. (2007). Also known as the endogeneity problem in econometrics literature, or

reversed causality in sociology and psychology literatures, selection bias is manifested by

the fact that there may be relevant observed or unobserved variables that are correlated with both the outcome and predictor variables. According to Gustafsson (2006), the term selection bias describes a situation in which “groups of students who received different treatments were not comparable in terms of their level of performance (or other

characteristics related to the outcome) before they received the treatment” (p. 6). Schneider et al. (2007) maintain that “[without random assignment] those who participate in a program may differ systematically from those who do not, which can bias the estimated treatment effect” (p. 40). Therefore, due to selection bias and the potential presence of unobserved variables, which may alter the estimated parameters of the

treatment effect, any causal inference drawn from observational studies – such as TIMSS studies and, hence, this dissertation, as well - may be weak.

Kim and Frees (2006) warned about the importance of the omission of relevant predictor variables in a multilevel modeling framework because “the consequences of omitted variables will be more complex and more dangerous in the multilevel case, as the effects of omitted variables at one level can pervade all levels of the model” (p. 661). They explain how the omitted variable causes bias because it may introduce a correlation between the error term and the other measured predictor variables. Moreover, these authors maintain that all known causes of the above correlation, i.e., “(1) measurement errors in the explanatory variables; (2) self-selection (an exogenous choice influences both the dependent and explanatory variables); and (3) simultaneity (the dependent and explanatory variables are jointly determined)” (p. 661), are all examples of problems caused by omitted variables.

Gustafsson (2006) explained that when the estimated relationship between an independent variable and a dependent variable is interpreted in causal terms, “it is assumed that there are no other independent variables which are correlated with the independent variable in focus, and which have not already been included in the model” (p. 7). Further, he asserted that the presence of such omitted variables will cause “bias in the estimated causal relationship in case that they are correlated with the residual of the outcome variable” (p. 7). Hence, the study can erroneously make causal inferences about variables others than the ones which are actually involved.

One option would be to measure and analyze all potentially relevant variables, but, obviously, this would be virtually impossible, no matter how strong the theory behind the selection process would be. Although some studies employ a rich set of student characteristics, it cannot be assumed that selection bias has been eliminated. Braun et al. (2006b) asserted that it would be impossible to determine how patterns of self-selection may affect the estimated student and school- effects, without more information about previous student characteristics, such as prior achievement. These authors explained that researchers “have attempted to address the problem of selection bias in observational studies by utilizing auxiliary information about both students and schools in order to generate so-called adjusted comparisons that (it is hoped) are less subject to selection bias” (p.1). Model specifications were tested to get a sense of the sensitivity of these estimated comparisons to various assumptions.

While discussing the limitations of inferences made from data collected in large- scale assessments in the context of examining within-country trends in achievement, Gustafsson (2006) proposed a different approach to solve the combined problems of selection bias and omitted variables. He points out that by relating within-country change over time in explanatory variables to within-country change in achievement, country characteristics are kept constant and consequently, they cannot be a source for omitted variables.

Other attempts to solve the problem of weak causal inferences drawn from

(examples include Altonji, Elder & Taber, 2002; 2005) and propensity scores matching (for example Harding, 2003).

Over time, the statistical technique known as sensitivity analysis emerged as a promising solution to this problem. Sensitivity analysis (SA) was defined by Saltelli (2006) as “the study of how the variation in the output of a model (numerical or

otherwise) can be apportioned to different sources of uncertainty in the model input” (p. 2). In the beginning, sensitivity analysis was created to assess the uncertainties in the independent variables and model parameters (i.e., standard error of the estimates). Later, SA evolved to incorporate model conceptual uncertainty, i.e., uncertainty in model structures, assumptions and specifications, and became closely linked to uncertainty analysis (Saltelli, 2006).

In their seminal paper, Rosenbaum and Rubin (1983) maintained that, “despite their obvious limitations, observational studies are still a valuable source of information about causal effects” (p. 212). Using as an example data from clinical study on coronary artery disease, they proposed a sensitivity analysis to assess how robust the estimated effects are to deviations from the assumptions of a randomly equivalent groups design. Montgomery et al. (1986) employed the approach proposed by Rosenbaum and Rubin (1983) in longitudinal study of the links between breast-feeding practices and infant mortality using data from a large scale demographic survey conducted in Malaysia. These authors were concerned with the estimated effect of breast-feeding practices on infant survival rates may be biased because preliminary analyses revealed that children who were healthier at birth were more likely to be breast-fed by their mothers. On the other

hand, healthier newborns had higher survival rates anyway, hence the possibility of selection bias. Montgomery and his colleagues attempted to overcome this potential problem by using child weight at birth as a “proxy indicator” for the omitted variable which was responsible for the unmeasured selection bias. The results of their sensitivity analysis indicated that the relationship between breast-feeding practices in Malaysia and infant subsequent survival remained strong, even after accounting for selection bias.