The City Council of Padua has asked for volunteers to take care of visiting English schoolchildren. Volunteers have to fill in a card. Mr Rossi and Mrs Bianchi are about
she is wrong, and states that men will volunteer as well. Mrs Bianchi counters that if that is the case, then the men will be married.
Cards filled in by the volunteers show gender on one side and marital status on the other. Given the four cards below, circle the two cards that you must turn over to check whether or not Mrs Bianchi is correct that the male volunteers will be married.
The correct answer for the F’s test is 7 and the correct answer for the card selection task is cards 1 and 3 (‘male’ and ‘unmarried’, respectively).
The two CRT validation items, relating to delayed financial gain, were taken from Frederick’s (2005) original study and in that study series had p-values less than .0001 in a chi-square test involving groups scoring high and low on the CRT. The total sample size was 3428. Oechssler et al. (2009) also replicated the correlation of CRT with these two items. These items were therefore selected as likely to be the most robust validation items for use in the present study. The items were modified slightly to fit the UK audience (the originals were in dollars), and are given in full on the next page.
Male Female Unmarried Married
1) Which would you prefer to receive?
a) £100 now
b) £140 this time next year
2) Which would you prefer to receive?
a) £3000 now
b) £3800 this time next month
The number of validation items included was kept small so as not make the questionnaire too long, especially as they were also used in some of the main research programme studies in order to gather additional data. However, considering the apparent robustness of the items this was considered to be sufficient. Loss avoidance items were also included, both as filler items so that participants would be less likely to catch on to the delay items, and as a check, since loss avoidance is known to be a robust effect (Silberberg et al., 2008).
Finally, the revised belief-basis measure and belief rating questions were used as per the other studies in the main research programme. The free-text response questions were omitted for brevity of the questionnaire due to time available and self-prediction of belief change was also omitted to avoid introducing any unnecessary bias.
Scoring
Belief in each topic was measured in the same way as in Study 1.
Belief-basis on each topic was measured in the same way as in Study 3.
Original CRT was scored by assigning one point for each correct answer to the three original CRT questions. These points were added together for an overall score between 0 and 3. Where 0 indicates a low tendency to inhibit default responses and 3 indicates a high tendency to inhibit default responses.
New CRT was scored by assigning one point for each correct answer to the two new
Where 0 indicates a low tendency to inhibit default responses and 2 indicates a high tendency to inhibit default responses.
CRT5 was scored as the sum of original CRT and new CRT (i.e. using all five items).
This gave a score ranging from 0 to 5, where 0 indicates a low tendency to inhibit default responses and 5 indicates a high tendency to inhibit default responses.
CRT validation measure, delayed reward, was calculated by assigning one point for choosing to delay on each of the two delayed financial gain questions, giving a score between 0 and 2, where 0 indicates a greater preference for smaller financial rewards now and 2 indicates a greater preference for greater financial rewards later. That is to say, a higher score on this measure indicates a greater tendency to accept a delay in return for a greater reward later on.
Procedure
The time sequence of the study is illustrated in Figure 23. Participants from each cohort were given the questionnaire which had a briefing and consent sheet on the front (Appendix 5). The signed consent sheet was detached from the questionnaire after data collection to maintain anonymity of the participants. Participants also had the right to withdraw from the study. After signing the consent form they completed the questionnaire. Once everyone in the room had completed the questionnaire they were briefed on the nature of the study. This included an extended discussion of related topics, as part of the students’ visit to Goldsmiths. As usual (BPS, 2018) ethical guidelines were followed. This study was approved by the ethics committee at Goldsmiths College, University of London.
Figure 23 - Time sequence and measures for Study 4
CRT5 &
validation itemsa
Topic statement
Rate belief in topicb
Rate reasons for
belief in topiccaCRT5 – one point for each correct answer, added together for a range of 0-5 (low to high tendency to inhibit default responding). Two validation items, on delayed reward, providing a range of 0-2 (low to high preference to accept a delay for a greater reward).
bBelief in topic rated from 1-9 (none to completely).
cBelief-basis – calculated from 6 ratings of reasons for belief (1-9, from not at all to completely). Mean rating for the three intuitive items is subtracted from the mean rating for the three rational items, giving an overall rating from 1-9 (from intuitive belief-basis to rational belief-basis)
Questionnaire
For 5 topics
Results
As in the previous studies graphs of the belief-basis in each topic for believers and non-believers are presented, compared alongside the results from Griffin and Ohlsson‘s (2001) original study. Once again, there is a close visual match, although disbelievers show a little more variation from Griffin and Ohlsson’s sample than believers do.
Figure 24 - Comparison between Study 4 and Griffin and Ohlsson’s (2001) USA sample for believers' belief-bases (numbers of contributing participants are given for each topic, with USA sample on the right; error bars not available for USA sample)
Figure 25 - Comparison between Study 4 and Griffin and Ohlsson’s (2001) USA sample for disbelievers' belief-bases (numbers of contributing participants are given for each topic, with USA sample on the right; error bars not available for USA sample)
Prior to statistical analysis of the results, the integrity of the belief-basis measure was checked. The overall Cronbach’s alpha was .492. Factor analysis without rotation is relatively clear on the expected split, but with item 4 perhaps a little less clear (not a negative correlation). However, rotation makes it clear that 1, 3, 4 are all on a single factor. After rotation, Cronbach’s alpha for intuitive was 0.537 and for rational it was 0.685.
Cronbach’s alpha for the CRT5 is given in Table 21, along with the same analysis conducted on the data from the other studies in the wider study program. This value
0.00
ranged from .413 to .655 across the four studies. The value of .413 is a little on the low side, even for the original 3-item CRT, but overall the results are not too far off the range found in the literature where researchers have attempted to extend the CRT.
Indeed, the Cronbach’s alpha is included here chiefly as a standard measure, but for low-item measures Cronbach’s alpha is liable to lead to lower alpha values simply due to the low number of items and this factor needs to be taken into account when using it as a measure of internal consistency (Lance et al., 2016; Tavakol & Dennick, 2011). For example, Toplak et al. (2014) goes so far as to state that .72 for their 7-item test indicates
“substantial reliability”. Meanwhile, Thomson and Oppenheimer (2016) report Cronbach’s alpha for the original CRT alpha as .642 and for their four new items as .511, with the combined 7-item scale achieving .705. Primi, Morsanyi, Chiesi, Donati, and Hamilton (2016) reviewed CRT studies and found a range from .57 to .74 for the original CRT, whereas Frederick (2005) does not cite Cronbach’s alpha at all in his original paper on the CRT. Similarly, Erceg and Bubić (2017) do not cite the Cronbach’s alphas for their scoring of CRT in ‘five different ways’, even though they studiously present tables of the reliability metrics for six other measures that they correlated with CRT. This included stating the mean, SD, min, and max, in addition to the Cronbach’s alpha, for each of the 6 measures. Finally, it must be noted that the items added to the original CRT were not simply added to strengthen the existing scale in terms of measuring exactly the same thing. They were added to mitigate the numeracy bias of the original scale and this means that there will inevitably be some difference in what the new and original items are measuring. However, as will be seen, the items do intercorrelate, indicating that they are related and not measuring completely unrelated constructs.
Table 21 - Cronbach's Alpha for CRT5 in each study from the wider research programme
Study 3* Study 4* Study 5* Study 6*
Cronbach’s Alpha
.548 .413 .596 .655
* significant at p < .05
Hypothesis 1 predicted that scores on the original CRT would correlate positively with