La teoría de las limitaciones TOC: Sistema OPT

Independent of how reliable the assumptions imposed on the VAM are, it might be possible to measure the accuracy of TE predictions under different type of ed- ucational contexts and estimation strategies. Even if VAM estimators are slightly biased, it might be useful, at least, to analyse whether their predictions of TE correctly rank teachers.

In absence of the true teacher effects, and therefore teacher rankings, some authors have tried either to identify non-random assignment settings or to use com- plex simulated data to assess VAM estimators performance. Building their own data allows authors to define different educational contexts and set a benchmark of TEs consistently estimated. From this benchmark it is possible to evaluate how VAM estimators differ in their TE predictions under different simulated contexts.

Dieterle et al. (2015) highlight the importance of the type of non-random

assignments observed in the data when VAMs are estimated. Focusing mainly on student-to-teacher assignment, the authors distinguish between student sorting and the teacher assignment process as two different sources of endogeneity. Stu- dents might be randomly grouped into classrooms or discretionally sorted based on observables and unobservable characteristics, such as previous scores and in- dividual ability respectively. On the other hand, teachers might be randomly assigned to classrooms, or sorted based on observable characteristics and unob- served teacher effects.

The authors state that common VAM specifications can easily deal with cases of student sorting when it is based on observable characteristics to re- searchers. However, the problem gets more complicated when student sorting is based on unobservable variables. Here, typical VAM estimators would perform differently depending on the assumptions imposed on the model specification.

Using data from an anonymous large state in the US, the authors are able to identify different types of non-random assignment and test the capability of some VAM estimators to predict true TEs. They found evidence of non-random assignment within schools estimating the probability that a student is assigned to a particular teacher given a set of 5 observed characteristics.31 The results show that there is enough evidence of non-random assignment of students to classrooms, and it is worth to consider it in the VAM estimation.

Additionally, Dieterle et al.(2015) also analysed the potential non-random assignment of teachers into specific types of classrooms within schools. Regressing

31_{This approach is taken fromClotfelter et al. (2006) where they also test the independence}

of assignment based on 6 observable characteristics. An alternative methodology to test for non-random assignment is used by Aaronson et al. (2007), and it consists in comparing scores distribution between current classes and sorted counterfactuals based on previous scores or gains.

some teacher characteristics on classroom average characteristics, the results show some evidence of race matching between teacher-students, and classrooms with higher academic performance in previous years with more experienced teachers. However, most of the student sorting and teacher assignment might vary across schools, grades and years.

The authors separate the data into different scenarios and compare TE es- timates for two type of VAM estimators. The methodologies to estimate TE are OLS and MLE estimations, obtaining posterior standard deviation of TE from EB estimations (MLE-EB). The OLS estimator includes teacher dummies to control for teacher fixed effects τj,g, while the MLE-EB relies on the independence as-

sumption of teacher random effects τj,g with respect to other covariates and other

unobserved factors, as is stated in A4.

The VAM specifications vary depending on the assumptions made on the rate of persistence λ. The specification proposed here slightly differs from our

Model 1and 3.32

(λ= 1) :Ai,g−Ai,g−1 =x0i,gβg+ +τj,g+ζt+αi+εi,g −εi,g−1 (2.42)

(0< λ₆1) : Ai,g =λAi,g−1+x0i,gβg+τj,g+ζt+αi+υi,g (2.43)

The estimations show that there are not significant differences between TE estimates from OLS and MLE-EB estimators when there is not student sorting. The high levels of correlation between TE ranking drops when the comparison are made within estimators (OLS, MLE-EB) but using different VAM specifications (e.g. λ = 1 and 0 < λ ₆ 1). However, there are some differences between OLS and MLE-EB estimators classifying high quality teachers in the top quantile, if there is evidence of grouping in the sample. Independent from the type of VAM estimator and VAM specification, when there is non-random assignment of student to teacher the risk of misclassification increases. Therefore, it is not straightforward to recommend a particular VAM specification or VAM estimator when there is presence of non-random assignment.

Continuing the investigation of the most appropriate VAM specifications and estimation methodologies,Guarino et al.(2014b) evaluate the performance of VAM estimators on simulated data. Under this approach, it is possible to replicate artificially different non-random assignment scenarios presented by Dieterle et al.

(2015).

The simulated data structure allows one to create cases with different com-

32_{This VAM includes a year effects factorζ}

t, and the individual heterogeneity is not affected

binations of student grouping (no sorting, sorting based on baseline scores, and sorting based on previous scores), and type of student-to-teacher assignment (ran- dom, non-random with respect to previous scores).

Guarino et al. (2014b) test estimation methodologies that have been ex-

plained in previous subsections, such us OLS, AR, and FGLS, besides others. The VAM specification is shown in equation (2.44) with 0< λ₆1.

Ai,g, =λAi,g−1+Tj,g+αi+εi,g−λεi,g−1 (2.44)

.

The authors claim that due to the simulation design of the data generation process, where the number of students per class remains constant, the dynamic OLS estimator (OLS estimator with 0< λ <1) provides TE estimates which are proportional to those obtained from the MLE-EB estimator. Therefore, in their paper they only show results for the dynamic OLS.

The analysis compares true TE (from the simulated data) with predicted TE under different VAM specifications and estimation strategies. Based on the assumption that there is a random assignment of students and teachers to schools

A2, the authors find that there is not any estimation methodology which is prefer- able in all contingent scenarios, regarding student sorting and teacher assignment process within schools.

However, in terms of correlation between predicted and real TE, and mis- classification of Value-Added estimates above or below the average, the FGLS, AR and OLS (with lagged scores) estimators perform similarly well when the scenarios involve teacher random assignment to classrooms. Even when students are sorted based on previous scores (statically or dynamically) or unobserved heterogeneities, this group of estimators performs well.33

Generally speaking, the OLS (with lagged scores), and proportionally the MLE-EB estimators are useful methodologies to predict TE across several scenar- ios. Despite the lack of consensus regarding the reliability and consistency of most of the VAM observed in the literature, it has been shown with simulated data that there are some VAM estimators which do well when predicting TE under certain conditions, while there are others which predict poorly in most of the settings.

Hence, Guarino et al. (2014b)’s results suggest that it might not be nec- essary to hold all underlying assumptions of VAM estimators to get good predic- tions of TE. Therefore, it is not fully recommendable to rely on validation tests for choosing the most appropriate VAM estimation strategy.

33_{It is called static sorting when student grouping is based on a fixed value such a baseline}

test scores, while dynamic sorting is when grouping keeps changing depending on previous test scores.

In this line,Guarino et al.(2015) extend the analysis initiated byRothstein

(2010) and Kinsler (2012) with respect to validation tests of VAM estimations. Taking advantage of the simulated data employed in Guarino et al. (2014b), the authors implement two typical validation tests on several VAM estimators for all possible student to teacher assignment scenarios.

As true TE are generated from a simulation process, it is possible to check whether VAM estimations with high predictive power are ruled out by common proposed statistical tests. Both tests are based on whether the underlying assump- tions of VAM estimators hold. Then, if the strict exogeneity A1 is violated, the VAM would be no longer consistent, and therefore not recommendable to predict TE.

The first validation test presented byGuarino et al.(2015) is an adaptation of the Hausman test, which basically tests for correlation between unobserved student heterogeneity and teacher assignment. The adaptation comes because the testing is based on a Wald test which is robust for serial correlation and heteroskedasticity of error terms. The second validation test is known in the literature as the Rothstein’s falsification test (Rothstein(2010)), where the impact of future TE on current academic performance is taken as evidence of non-random assignment of teacher to classrooms.

If the Hausman validation test rejects the FGLS estimator, this means that estimators from the FE approach would be appropriate as TEs are correlated with other covariates. However, under student non-random grouping but with teacher random assignment, the rejection rate for the FGLS is above 60% while its teacher ranking correlation with true TE is higher than the correlation ranking observed for WG estimates. However, when the rate of persistence is assumed to be partial (0< λ <1), the strict exogeneityA1 would be violated with the FGLS estimator, and that implies a full rejection of the FGLS, even though if it has higher prediction power than the WG estimator.

If we apply the Rothstein’s falsification test, the FGLS is also rejected in most of the scenarios, repeating the same mistakes observed from the Hausman test when the FGLS performs even better than other consistent estimators of TEs. If there is sorting of students but teachers are randomly assigned to classroom, the dynamic OLS presents lower rejection rates than FGLS, but it increases sig- nificantly when student grouping is based on unobserved heterogeneity, although its rejection rates are never higher than the FGLS in this case.

Guarino et al. (2015) conclude that when falsification tests rejects the null

of random assignment implying that the VAM estimator is inconsistent, it still might be the case that the TEs estimates rank the teachers correctly.

sible non-experimental scenarios. Hence, it is crucial to understand the context where TE are estimated, as it has been shown that it is possible to define effec- tive VAM estimators which predict satisfactorily teacher quality rankings, even if fundamental assumptions of the structural model are violated. Nevertheless, because there is still a degree of uncertainty and variation among estimators when teachers are sorted into groups, we have to be prudent before using TE measures to execute any policy at individual teacher level.