CAPÍTULO IV. ORGANIZACIÓN Y COORDINACIÓN ESCOLAR EN EDUCACIÓN
IV.2. La estructura de los institutos españoles
∀g ∈ G, cov(ci, y∗i|gi = g) = 0 where g are observable characteristics as in the previous section.
Proposition 5: The Distribution of True Answers Under Conditional Consis-tency Independence Let Yc(g) = Y0(g)+1−YY0(g)1(g). Under Assumptions 1-4 and Conditional Consistency Independence, E[yi∗] = Eg[Yc(g)].
Proof By the law of iterated expectations, E[y∗i] = Eg[E[yi∗|gi = g]]. Proposition 2.1 implies that for any g, Yc(g) = E[yi∗|gi = g, ci = 1]. Conditional consistency independence then implies that Yc(g) = E[yi∗|gi = g].24 Substituting this into the first expression yields
the result.
To implement the approach suggested by Proposition 5 in practice, one would first cal-culate Yc(g) for each observable characteristic (or as a continuous function of observable characteristics), and then take a weighted sum of Yc(g), using weights equal to the distri-bution of g in the full population (denoted f(g) in the previous section). The technique is analogous to post-stratification weights frequently employed in survey analysis (Holt and Smith, 1979), which correct for the fact that some respondents are more likely to select into the sample than others. In our setting, consistency weights correct for the fact that some respondents are more likely to select into the consistent subgroup of the population – the subgroup whose true responses to the survey can be inferred.
Carrying the analogy further, for conventional post-survey non-response weights to elim-inate selection bias, it must be the case that respondents’ propensity to participate in the survey is uncorrelated with unobservable correlates of the variable being investigated. Our
As described below, this difference complicates the task of adapting the standard sample selection techniques to this setting.
23An alternative approach, analogous to instrumental variable methods, for when this assumption is not credible is described in Goldin and Reck (2015).
24Recall that two-valued variables are independent if they have zero covariance. One can also show formally that E[y∗i] = Yc−cov(cp(ci=1)i,yi∗). Conditioning this on g and taking its expectation also leads to the desired result.
conditional consistency independence assumption guarantees exactly this; it will fail when respondents’ consistency is related to the distribution of y in unobservable ways. As such, the more individual characteristics the researcher can observe that are potentially corre-lated with a respondent’s consistency, the more confident the researcher can be that using consistency weights will recover the full population average.
7 Conclusion
In this paper we have developed a straightforward framework to study a ubiquitous problem in survey research: the sensitivity of responses to seemingly-arbitrary features of the survey’s design. We showed how the conventional approach to dealing with this problem does not actually solve it, and proposed several practical alternatives. Many of these techniques focus on the distribution of answers of consistent respondents and the characteristics of consistent respondents; others attempt to go further and recover the distribution of “true answers” in the population.
Once one adopts the formal framework we propose for analyzing survey response incon-sistencies, several parallels emerge to classic ideas about identification and inference. As ever, the degree to which interesting parameters can be identified from the data depend on the assumptions the researcher is willing to impose. With very little structure, one can recover information on the responses of consistent respondents – those unaffected by framing – and the characteristics of consistent individuals. Recovering true answers in the population requires assuming that true answers exist in the first place, and assuming away unobserved framing effects. In both of these cases, the sharpness of the identification depends on whether framing effects are assumed to be monotonic. Finally, assuming that consistency across frames and true answers are conditionally independent permits point identification of the distribution of true answers in the full population.
Two limitations of our work deserve consideration in future research and applications.
First, we have focused on the relatively simple setting of binary survey variables with two frames; applying the approaches in settings with additional frames or answer choices requires further assumptions. It is difficult to imagine that the difficulty with randomizing across frames and pooling the data will be eliminated in a more complicated setting, but credible alternatives may be somewhat more difficult to obtain. Second, our approach is aimed at eliminating the bias induced by framing effects, but other sources of bias could still be a problem. Generalizing the approach proposed here to non-binary survey questions and
to settings characterized by other types of bias – such as random choice, forgetfulness, or selection effects – are important directions for future research.
References
Abadie, Alberto. 2003. “Semiparametric Instrumental Variable Estimation of Treatment Response Models.” Journal of Econometrics, 113(2): 231–263.
Chong, Dennis, and James Druckman. 2007. “Framing Theory.” Annual Review of Political Science, 10: 103–106.
Deaton, Angus. 2012. “The Financial Crisis and the Well-Being of Americans.” Oxford Economic Papers, 64(1): 1–26.
Fischhoff, Baruch. 1991. “Value Elicitation: Is There Anything in There?” American Psychologist, 46(8): 835.
Gallup Organization. 2012. “Gallup News Service Poll: Monthly Indicators Update.”
Roper Center Public Opinion Archives Survey Dataset.
Goldin, Jacob, and Daniel Reck. 2015. “Preference Identification Under Inconsistent Choice.”
Holbrook, Allyson, Jon Krosnick, David Moore, and Roger Tourangeau. 2007.
“Response Order Effects in Dichotomous Categorical Questions Presented Orally.” Public Opinion Quarterly, 71(3): 325–348.
Holland, Paul. 1986. “Statistics and Causal Inference.” Journal of the American Statistical Association, 81(396): 945–960.
Holt, D. Tim, and T.M. Fred Smith. 1979. “Post Stratification.” Journal of the Royal Statistical Society, 142(1): 33–46.
Horowitz, Joel L., and Charles F. Manski. n.d.. “Identification and Robustness with Contaminated and Corrupted data.” Econometrica, 63.
Imbens, Guido W., and Joshua D. Angrist. 1994. “Identification and Estimation of Local Average Treatment Effects.” Econometrica, 62(2): 467–475.
Krosnick, John, and Lee R. Fabrigar. Forthcoming. The Handbook of Questionnaire Design. Oxford University Press.
LeBoeuf, Robyn, and Eldar Shafir. 2003. “Deep Thoughts and Shallow Frames: On the Susceptibility to Framing Effects.” Journal of Behavioral Decision Making, 16(2): 77–92.
Manski, Charles. 1989. “Anatomy of the Selection Problem.” The Journal of Human Re-sources, 24(3).
Manski, Charles. 2003. Partial Identification of Probability Distributions (Springer Series in Statistics. Springer.
Moore, David W. 2002. “Measuring New Types of Question-Order Effects: Additive and Subtractive.” Public Opinion Quarterly, 66(1): 80–91.
Presser, Stanley, Mick Couper, Judith Lessler, and Elizabeth Martin. 2004. Meth-ods for Testing and Evaluating Survey Questions. Wiley.
Schuman, Howard, and Stanley Presser. 1981. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. SAGE.
Schwarz, Norbert. 1990. “Assessing Frequency Reports of Mundane Behaviors: Contribu-tions of Cognitive Psychology to Questionnaire Construction.” Review of Personality and Social Psychology, 11: 98–119.
Schwarz, Norbert, and Daphna Oyserman. 2001. “Asking Questions About Behav-ior: Cognition, Communication, and Questionnaire Construction.” American Journal of Evaluation, 22(2): 127–160.
Sudman, Seymour, Norman M. Bradburn, and Norbert Schwarz. 1995. Thinking about Answers : The application of Cognitive Processes to Survey Methodology. Jossey-Bass.
Tversky, Amos, and Daniel Kahneman. 1981. “The Framing of Decisions and the Psychology of Choice.” Science, 211(4481): 453–458.