Data were gathered from six classes of undergraduate students at different stages of their degree in order to test the hypotheses above. The advantage of using relatively homogenous participants in experimental design is that it makes controlling for confounding factors through random assignment more effective (Kirk, 1982). While this is a standard approach taken by researchers investigating voter bias, it is worth taking a moment to address the debate around the use of student samples in experimental research. Criticism of the use of student samples typically borrows from Sears’ (1986) paper which pushes back against the prevailing norm at that time whereby researchers in social psychology used samples of these individuals in laboratory settings in order to say something about the general population. Sears pointed out that undergraduates are people who have “less-crystalized attitudes, less-formulated senses of self, stronger cognitive skills, stronger tendencies to comply with authority, and more unstable peer group relationships” (ibid., p. 515), all of which make them unlike the general public. Particularly relevant for this chapter is the argument that where participants have social and political attitudes that are less crystalized they may be easier to manipulate. While Sears’ paper calls for results employing this narrow data base in a laboratory setting to be treated with caution, some have pushed this critique further, calling into question the external validity of any such results generated using a student sample. The practice of using these individuals in research has persisted but the reflexive critique that it occasions has only become stronger. There are good reasons to believe that it is on the whole this critique is exaggerated.
Firstly, even Sears (1986) did not intend his critique as a wholesale rejection of the use of student samples. Rather it is important to think about whether students are different from the general population “in ways that are relevant to the study in question” (Kam, 2005, p.176). Where a task is particularly complex, there is some concern that the higher than average cognitive ability that is expected to be present in undergraduate samples will impact the results of a study in an important way (Bol, St-Vincent & Lavoie, 2016). In a similar vein, Kam (2005) notes that where level of political
95 awareness, strength of party identification, and interpretation of concepts differs across student and non-student populations this has the potential to limit the external validity of experiments investigating voting behaviour.
Nevertheless student samples are appropriate to use when “there is theoretical and/or empirical reason to believe that the effect of the treatment would be the same (or closely similar) across student and nonstudent samples” (Kam et al., 2007, p.420) and there is good empirical evidence that – with respect to framing experiments at least – there are very few differences with respect to treatment effects across student and heterogeneous samples. Kühberger’s (1998) meta-analysis of 136 experimental studies (totalling approximately 30,000 participants) on loss aversion, finds that type of sample was not a predictor of receptivity to the framing effect, though in some cases the magnitude of the effect was very slightly larger in the student samples. Bosch-Domenech et al.’s (2002) comparison of newspaper and laboratory beauty-contest experiments21 conducted among thousands
of participants across several European countries also finds comparable effects across student and non-student samples. Bol, St-Vincent and Lavoie’s (2016) comparison of voting experiments across Canada and Europe similarly finds that results for the student and heterogeneous samples are not statistically different from one another except where the experiment is cognitively complex.
Druckman and Kam (2011) argue that concern with external validity has led us to focus almost entirely on who is studied when in fact the conditions under which the composition of the sample should be an issue are rather more limited than critics would have us believe. Where the treatment effect is expected to be the same across different populations then the nature of the sample that researchers use to establish that there is an effect is in the main irrelevant. However, where the effect of any treatment is moderated by particular individual-level characteristics, the unrepresentativeness of a sample should lead us to be cautious about the conclusions that we are drawing. Druckman and Kam (ibid.) go so far as to argue that the use of student samples does not pose an intrinsic challenge to an experimental study’s external validity and instead the burden of proof should lie with the critics of the practice rather than the researchers who employ such samples.
So, does the use of an undergraduate sample represent a challenge to the external validity of any results that this chapter will present? I argue that it does not. As is the case generally it is crucial to be cautious when attempting to generalise findings from a single cross-sectional study with a limited sample to the population at large. However, it is not immediately clear that this study poses a particular
21 Beauty contest experiments take the form of a competition whereby participants must guess a value between 0 and 100 that is closest to what the mean value of the respondents will be in order to win a prize (a modification of the original contest introduced by John Maynard Keynes).
96 challenge to generalisability by virtue of its use of a student sample. Firstly, as discussed, one of the main differences that is highlighted between undergraduates and the population at large is a higher- than-average cognitive ability that can lead to differences across student and non-student samples where the study that they are involved in is cognitively demanding. This study is not cognitively demanding in this way. The study procedure is outlined below and the treatment and survey are included in the appendix to the chapter, so I will only note that the text is short and modelled after a news item that participants are likely to come across in the course of their normal lives. The survey questions are also of the format and wording used in surveys of the general voting population (such as the Irish National Election Study). In addition, as was discussed in the previous section, this study seeks to address one of the key shortcomings of previous work, the artificiality of the setting which requires participants to make inferences from very limited information. As such, the study includes information about candidate party identification and policy positions which voters are likely to have when evaluating candidates. The study then is not one that requires a high degree of cognitive skills in order to be able to participate.
Secondly, critics tend to argue that student samples are made up of individuals that are highly manipulable and therefore when we use such individuals we are likely to uncover treatment effects where none exists in the general population. And with respect to this study in particular, critics are likely to argue that students should be more liberal than the general population, so therefore we should not find any evidence of bias against women. These two arguments against the use of a student sample for this study predict biased results in opposing directions. If, as the former point suggests, undergraduates are easy to manipulate, then this study should lead us to conclude that there is gender bias where none exists in the general population. If, as the latter argument suggests, younger individuals are less likely to rely on traditional gender stereotypes, this study most likely represents a conservative test of the hypotheses above (see Higgle et al., 1997). If the critics are correct, it is not clear which of these two effects we should expect to dominate, or if they will simply cancel each other out.
We will return to this topic of the use of student samples in experiments in the discussion section, but I will conclude here by saying that while it is true that the individuals in this study will not be typical of the national population on a variety of demographics, it is not immediately clear that this represents an insurmountable obstacle to the study’s generalisability.
Though undergraduates are employed as the sample in the study, it was important that different types of undergraduates be included since students who are interested in politics might be unrepresentative of the wider demographic. Only three classes of students, or 263 respondents, were taking a Political
97 Science module, two of these modules were introductory and less than thirty percent of those students enrolled in the modules had elected to study Political Science. The three other groups were made up of Computer Science, Engineering, and Sociology students. Incorporating these other groups into the study also helped to achieve near parity in the gender composition of the sample.
Since revealing the purpose of the study to participants would allow them to conceal the very biases that are being tested for, a methodology that did not alert participants to the true nature of the experiment was used (see Smith, Paul & Paul, 2007). Participants were told that they were part of a study examining campaign communication. Each participant was given a dossier relating to a fictitious candidate containing information regarding the candidate’s background and policy positions on various issues then asked to fill out the accompanying questionnaire. Since one of the aims of the project was to make the scenario as realistic as possible, participants were informed that the candidates would be standing for the first time at the 2016 Irish general election in an urban constituency outside of the capital.
Respondents were asked to read the dossier and then to answer a series of questions about their perceptions of the candidate based on the information provided. Though four experimental conditions were employed, with candidates differing with respect to party identification and sex, each participant evaluated only one candidate. The information received by the participants was identical and varied only with respect to these treatments. Assignment to the experimental conditions took place on a random basis within each of the six classes. This study design is therefore concerned with the study of between- rather than within-person differences in evaluations.
As was noted above, there are four experimental conditions in total, each varying with respect to candidate sex and party. A pilot study was carried out in January of 2016 among a sample of 90 first year Political Science undergraduates to assess the treatments and variables being used prior to running the full study. It also included two additional experimental conditions in order to incorporate non-party candidates.22 Ultimately, it was deemed unfeasible to create experimental conditions for
more than two parties. Incorporating each of the main parties in the Irish political system into the study would have involved recruiting substantially more participants. However, it was important to include parties of both the left and the right in the design of the experiment, for the reason outlined above. The government of the moment provided the opportunity to feature parties of both the left and right in the study design. The Fine Gael/Labour coalition that existed at the time of the election enabled the inclusion of these parties without fear that differences in evaluations would result from a
22 Section A3.2.1 in the chapter’s appendix provides the pilot treatment, and Section A3.2 provides the procedure for the study as well as the data analysis conducted.
98 government versus opposition effect, which could impact evaluations, particularly where anti- government sentiment is strong. The four experimental conditions along with the number of participants assigned to each group are represented in Table 3.2.
After the parties were chosen, it was necessary to construct their biographies and issue positions. The Irish Times Nealon’s Guides data from the last three general elections was employed in order to construct a short biography for a likely Dáil candidate. Issue areas covered in the dossiers relate to stereotypically female (childcare) and male (economic growth, employment) policy areas, as well as one gender-neutral area covering constituency service (improving road quality). Reproductive rights was the second feminine issue area according to which respondents were asked to evaluate the competence of their candidate. However, for reasons that will be discussed below, information relating to the candidate’s position on that issue was not included in the dossier.
Table 3.2. Assignment of experimental conditions Sex
Party Male Female Total
Fine Gael 104 102 206
Labour Party 102 112 214
Total 206 214 420
It is important to say a few words relating to the construction of the issue positions. Since the design of the study necessitated that each respondent receive an informational dossier containing exactly the same information, regardless of candidate sex or party, it was important to find issue positions that are common across members of both Fine Gael and Labour. To this end, a search was made of campaign literature employed at the 2011 election, and of recent statements made to the press by existing TDs of both parties.23 Issues were selected that would not only meet the criteria of being
gendered but would also have positions that could be accommodated by both parties.
With respect to the ‘male’ issue areas, this was an easier task: support for economic growth and employment is common across Irish political parties. The non-gendered issue position relates to constituency service, more specifically ‘improving road quality’. It is worth discussing this issue in
23 The Irish Election Literature wordpress blog, as well as articles from The Irish Times, Irish Independent, and various other newspapers were employed.
99 greater detail. Because voters are able to choose between candidates across parties as well as within the same party, it is a feature of the Irish electoral system that TDs engage in intra-party competition. Politicians need to distinguish themselves from their opponents (Heitshusen, Young & Wood, 2005) and therefore engage in constituency-based or related activities such as publicizing “the work they are undertaking on behalf of their district...[and] being seen in the district” (Norton, 2002, p.5). While the aim of constituency-based activities is of course to ensure re-election, the extent to which politicians have scope to provide particular services varies across countries. In Ireland there is little opportunity for direct patronage (providing goods and services) and so it has long been argued that politicians tend to engage in brokerage services for their constituents which help them to navigate state bureaucratic processes (Carty, 1981; Kelly, 1987). Komito (1984) for example argued that in Ireland clientialism functions in the following way; Irish politicians do not necessarily secure votes directly from their clients but instead enhance their reputation and standing through their activities. There is then an attempt by politicians to create a climate of obligation and gratitude through activities like “attendance at funerals, testimonial dinners, social events, and public meetings” (ibid., p.181) as well as lobbying for the interests of the constituency as a whole (O’Halpin, 2002; Suiter & O’Malley, 2014). Constituency service then constitutes an important part of the job description of TDs (Heitshusen, Young & Wood, 2005; O’Leary, 2011) and where they have a say over how public money is allocated – for example when they are ministers – they will target their own constituencies (Suiter & O’Malley, 2014). There is an expectation on the part of the electorate then that TDs will lobby for their constituencies. For example, in addition to the sports and school building grants examined by Suiter and O’Malley (2014), there have been controversies over primary care centres and other hospital projects targeted at ministers’ constituencies (Carty,2012; Irish Mirror, 2013). For the purposes of this study, road quality has been selected as a good that candidates might be expected to deliver to their constituency. This good was selected since unlike other goods (e.g. hospitals or schools) everyone in the constituency should directly benefit from improved road quality and also because there are notable cases in Irish politics where independent TDs have struck deals with the government in order to secure improvements in this area in exchange for their support (e.g. Mildred Fox and Jackie Healy-Rae [Tynan, 1997; Irish Times, 2000; O’Sullivan, 2012]).
The issue that represented the biggest challenge was that of reproductive rights. In the pilot study, the information sheet provided to respondents included a section detailing the fictitious candidate’s position on the issue:
Finally, when asked whether [SHE/HE] would support a repeal of the Eighth Amendment to the Constitution (this amendment explicitly makes abortion illegal in Ireland, except where the life of the mother is at risk) [CLARE/MICHAEL] had this to say: “I have never been in favour of abortion
100 on demand. I think that it is important to respect the life of the child as
well as the mother. However, I do think that what we have in place now is much too restrictive. I think it is very difficult to ask women who have been assaulted or whose babies will not survive... To ask those women to carry a baby to term is difficult.”
[HE/SHE] concluded: “I would support holding a referendum on the issue. I think that it should be up to the Irish people to decide whether [the Amendment] should be removed from the Constitution. And I think that they should have a say about what it would be replaced with.”
The issue position above is one that can be accommodated by both parties. While it is true that the Labour Party is among the most committed to increasing access to abortion in Ireland, the broad position of the party cannot be described as particularly radical and in fact a range of positions exist within it. Fine Gael, as a Christian Democratic party (see Gallagher & Marsh, 2004), has had a much less liberal record in this area. However, a diversity of opinions on this issue exists within the party. When he was Minister for Health, Dr. James Reilly, openly expressed his view that existing abortion laws are too restrictive and that the Eighth Amendment to the Constitution which explicitly bans abortion – except where the life of the mother is at risk – should be repealed, though his successor in that department, Dr. Leo Varadkar, stated a few months before the general election that he was in favour of keeping the Eighth Amendment in the Constitution (Ó Cionnaith, 2015b). Additionally, the Fine Gael-Labour coalition passed the Protection of Life During Pregnancy Act 2013, the first piece of legislation to enact into law the Supreme Court ruling in 1992 Attorney General v. X case which stated that a woman could undergo an abortion where her life is at risk from a continuation of the pregnancy, even where such a threat arises from the risk of suicide. Both parties therefore share overlapping positions on this issue and since reproductive rights is generally thought of as a core women’s issue, it