The previous analysis offers a compelling lens through which to view ‘Lord’s Paradox’ (147), a peculiarity that has evaded statisticians since its original formulation in 1967 (154-156, 162). The paradox is summarised below:
A university is interested in investigating whether the diet provided in the dining halls has an effect on students’ weight over the course of the year, and whether there are any sex differences in these effects. It hires two statisticians to answer this question. The first
statistician examines the mean weight of the girls at the beginning of the year and at the end of the year and finds that these are identical; moreover, the frequency distribution of weight for girls at the end of the year is the same as it was at the beginning. He finds the same to be true for boys, and thus concludes that there is no evidence for any differential effect on the two sexes. The second statistician conducts an analysis of covariance (i.e. a follow-up adjusted for baseline analysis). He finds that the slope of the regression line of final weight on initial weight is essentially the same for the two sexes, but that the difference between the intercepts is statistically highly significant. The second statistician therefore concludes that boys showed significantly more gain in weight than the girls when proper allowance is made for differences in initial weight between the two sexes.16
Which statistician is correct? The conclusions of the two statisticians appear to contradict one another, leading Lord to conclude that ‘there simply is no logical or statistical procedure that can be counted on to make proper allowances for uncontrolled pre-existing differences between groups’ (147). However, the causal lens adopted in the previous sections can help to resolve this question.
4.7.1 Considering the paradox within a causal framework
One of the primary challenges in interpreting Lord’s paradox stems from the fact that baseline weight is a mediator for the effect of sex on final weight. Thus, although baseline weight represents a ‘pre-existing difference’ between boys and girls according to Lord, it is actually a consequence of the exposure rather than a cause of it. Therefore, it is fundamentally different from the experimental setting that has historically been considered in the analysis of change – a setting in which pre-existing differences occur before the exposure.17
16 We note that the research question is itself ill-defined, since the diet is a fixed condition that is
applied to all students, male and female. Therefore, the diet can have no identifiable causal effect on the students’ weights. Consequently, the question appears to actually be about the differential effect of sex on weight change.
17 Additional confusion has been created by the fact that in several subsequent reinterpretations of
In Figure 4.7 we draw the scenario described by Lord as a path diagram, where 𝑆 represents sex, 𝑊0 represents initial weight, 𝑊1 represents final weight, and 𝐷 represents diet. This path
diagram is equivalent to the one considered in Scenario 3 (§4.4.3). Figure 4.7 Path diagram representing Lord’s Paradox (147)
The exposure sex (𝑆) is measured once at baseline, and the outcome weight (𝑊) is measured once at baseline (𝑊0) and once at follow-up (𝑊1). The change-score ∆𝑊 is constructed from
𝑊1− 𝑊0. The diet (𝐷) is depicted in grey because, although it affects 𝑊1, it is not truly a variable; all students are subjected to it, and it thus does not have any identifiable causal effect on 𝑊1.
The first statistician, in comparing the average change scores between boys and girls, has essentially conducted a change-score analysis and found the effect of sex on ‘change’ in weight to be zero. In other words, the first statistician estimated the total effect of sex on weight change-score (i.e. 𝑝𝑊1𝑆+ 𝑝𝑊0𝑆∙ 𝑝𝑊1𝑊0− 𝑝𝑊0𝑆, in Figure 4.7). By contrast, the second statistician conducted a straightforward follow-up adjusted for baseline analysis, and thus estimated the direct effect of sex on final weight (i.e. 𝑝𝑊1𝑆, in Figure 4.7).
It is therefore not surprising that the two statisticians came to different conclusions, and Lord himself did not see how this problem might be resolved, as the effects estimated are expected to differ by 𝑝𝑊0𝑆∙ (𝑝𝑊1𝑊0− 1). Only under one of the following conditions would the two agree:
1. Where 𝑆 and 𝑊0 were uncorrelated (i.e. the randomised experimental setting), such
that 𝑝𝑊0𝑆= 0; or
2. Where 𝑊0 and 𝑊1 were perfectly correlated (i.e. the deterministic case in which no exogenous change exists), such that 𝑝𝑊1𝑊0− 1 = 0.
4.7.2 Identifying the most useful estimand
Identifying the most useful and/or important estimand in this scenario can help us to identify which statistician was correct.
In conducting a change-score analysis, Statistician 1 did not in fact estimate the effect of sex on ‘change’ in weight; this is because ‘change’ is fully encapsulated in the follow-up weight whereas a change score conflates information from both baseline and follow-up (as argued in Section 4.3).18 Therefore, Statistician 2 can claim to be in possession of a more meaningful
answer to the original query, since the direct effect represents a valid estimand in this scenario.
However, we might consider an equally valid solution (presented by a fictional ‘Statistician 3’) to be the total effect of sex on final weight, obtained via a follow-up unadjusted for baseline analysis. This effect captures all changes in weight which result either directly or indirectly from sex. Moreover, this estimate is not susceptible to the potential biases introduced by conditioning on a mediator (i.e. baseline weight, which is likely to share many causes with follow-up weight).