The number of correlations based on unrelated items (2750 so-called baseline
correlations) might seem disproportionate relative to the number of correlations based on reversed items (29 correlations) and non-reversed related items (71 correlations). This is a consequence of the fact that any filler item could be correlated with any other filler item, while the other types of correlations were much more selective by design. The apparent imbalance of contentwise unrelated items to contentwise related items is not problematic. By using the dummy specification that reflected the different types of correlations, and by creating interaction terms of these dummies with each of the effects, separate effects were estimated for the different categories of items, and all estimates had their own appropriate standard errors. At the same time, ARS was being controlled for in a highly reliable way (based on the many baseline
correlations), such that the main effect and the effect moderated by distance of ARS could be assessed independently of the item-interaction effects. Further, the
correlations were based on a large number of respondents (over 3000) which
enhanced their stability and reliability (Zimmerman, Zumbo and Williams 2003), and the items were randomly assigned to positions in the questionnaire. These factors made it possible not to include extraordinarily large numbers of reversals in the questionnaire, which might have led respondents to become acutely aware of the set- up, possibly even leading them to see the task as a ‘reversal examination’ rather than an ordinary questionnaire.
The specific curve of reversed item correlations as a function of inter-item distance was attributed to a unipolar response model. The varied contents of the questionnaire in the current study renders implausible an otherwise appealing alternative
positively related items and are then confronted with a reversed item, careless reading might lead some respondents to misinterpret the reversed item as a same direction item (Schmitt and Stults 1985). However, for this effect to occur, it seems that many similar items should occur in an uninterrupted series (cf. Drolet and Morrison 2001). Though this was the case in studies in which a negative item method effect has been observed (e.g. Marsh 1996; Motl and Distefano 2002), it was not in the current study. Since the length of the questionnaire used for this study was limited to 76 items, most inter-item distances were quite small. The median inter-item distance in the data is 23. It would be useful to further study the current phenomena using longer questionnaires. Possibly, the effect of distance on r fades out completely after a given distance. The current data are too limited in scope to find out.
For now, good fit was obtained using the natural logarithm of (distance + 1). Though the natural logarithm is an often-used transformation (Greene 2003, p. 11-13;
Tabachnick and Fidell 1996, p.80-82), other specifications are also possible, and some of these possibilities are shortly reviewed below. Note that the substantive findings were found to be robust over different specifications.
As an exploratory exercise several specifications of the regression model were tested: (1) a strictly linear model; (2) a model with quadratic effects of distance (and its interaction terms); (3) a spline regression, where the effect of distance (and its interaction terms) was allowed to be different in the inter-item distance range of 0-10 versus 11-76. However, the different specifications resulted in the same substantive conclusions, where (1) there is a significantly positive base correlation (the intercept) in the range of .05 to .08, which is slowly declining towards zero over distance, (2) a negative correlation between reverse-direction items which grows in strength
correlation between same-direction items which also more pronouncedly declines over inter-item distance.
In addition to further quantitative research, it would be most interesting to further validate the current findings by means of cognitive interviews (DeMaio and Rothgeb 1996; Jobe and Mingay 1989). Specifically, it would be enlightening to study
respondents’ processing of unrelated items, same direction items and reversed items in a controlled setting. Using questionnaires similar to the one used in the current study, respondents could be asked to think aloud as they process the meaning of items and retrieve information. It would be especially relevant to observe the extent to which respondents refer to previous items and how respondents use the intended scoring direction of the items (non-reversed or reversed) and inter-item distance as input for the comprehension process. Another interesting probing technique would be to ask respondents to paraphrase reversed items, i.e. to word these items in the
respondents’ own words. This would be indicative of whether or not respondents refer to related concepts when processing reversed items.
Finally, a study is planned that approaches the issues investigated here from a different perspective. The current study used a between-item design with a one-time random assignment of items to locations. In a follow-up research, a between-subject design will be used. In this study, item content will be kept constant by investigating a pair of reversed items and a pair of non-reversed items. Item location of item i and i’ will be randomized over respondents. The following regression model will be tested: xi’ = α + β1 ARS + β2 xi + β3 (xi * LN_DISTii’) + ε, where xi’ and xi are the observed scores on item i and i’, ARS is a measure of acquiescence measured over a set of heterogeneous filler items, LN_DISTii’ is the natural logarithm of the distance +1 between item i and i’, and α, β1, β2 and β3 are the regression intercept and weights. α
corresponds to the mean of xi’, β1 to the effect of ARS, β2 is expected to be negative and corresponds to the extent to which the extremity of a respondent’s position on the construct underlying both items is identical in size (but opposite in direction) for i and i’, and β3 captures the effect of distance on this relation.