In the previous section we described the inappropriateness of the general linear hypothesis to test the average total effect. Following the concerns formulated by Crager (1987, p. 895), we will now present a discussion of inference based on the conditional variance of the effect function, i. e., conditional on the values of the covariate: “An often-recommended solution to the problem of random covariates is to use fixed - covariate ANCOVA and to consider the analysis conditional on the set of observed covariate values. This leaves in doubt what the true significance level of the hypothesis tests would be with the sampling distribu-tion of the covariate taken into account. Since the variance of ANCOVA parameter estimators is a funcdistribu-tion of the covariate values, there is cause for concern.”
As mentioned in the introduction of this chapter (see section 3.1), interactions are treated as a problem of non-parallel regression slopes in the ANCOVA literature. The typical treatment of non-parallel regression slopes is known as probing interactions, strongly inspired by the work of Rogosa (1980), who considered for a simple linear parameterization of the covariate-treatment regression [see Equation (2.3) on page 27] the difference between the sample regression lines as a function of the values of the covariate Z . With obtained estimates of the regression coefficients, the value of the effect function can be computed for each value z of the covariate Z as
ˆ
g1(Z = z) = ˆγ10+ ˆγ11· z. (3.15)
3.2 General Linear Model 63
This point estimate is labeled as the simple slope, a linear combination of the coefficients estimated by ordi-nary least-squares from a (moderated) multiple regression model and the values of the covariates (Aiken &
West, 1996, p. 25). Note that the values of the covariate are considered in Equation (3.15) as fixed and known, only the regression coefficients are treated as estimates (see fixed-X assumption in subsection 3.2.2.2).
Conditional Variance of the Effect Function and Center of Accuracy Following the idea of an effect de-scription for a given value Z = z, Rogosa (1980) derived an estimator for the variance of the conditional effect as a transformation of the estimated variance of the regression coefficients dVar (ˆγ10), dVar (ˆγ11) and the covariance dCov (ˆγ10, ˆγ11). This variance of the effect function for a given value is due to the model-based variability of the estimated regression coefficients (see also Maxwell & Delaney, 2004):
Vard¡
This estimator of the variance of the effect function for a value Z = z is also the starting point for the so-called center of accuracy. For a simple linear parameterized effect function with a single univariate covariate Z the center of accuracy is
bzC A= −Cov (ˆγd 10, ˆγ11)
Var (ˆγd 10) , (3.17)
and in general, the center of accuracy can be obtained by differentiating the estimator of the variance for the conditional effects, i. e., dVar¡
b g1(z)¢
, with respect to the covariate and setting the resulting derivative equal to zero. For the value bzC Aof the covariate Z obtained from Equation (3.17), Rogosa (1980) has shown that the (conditional) variance of the effect function is minimal by deriving the following decomposition:
Vard¡
gˆ1(Z = z)¢
= Vard¡
gˆ1(zC A)¢
+ dVar (ˆγ11) · (z − bzC A)2. (3.18)
Rogosa (1980) also noted that in case of an interaction, the classical ANCOVA, which assumes parallel re-gression lines, compares conditional means at the value Z = bzC Aof the covariate, which are weighted aver-ages across the groups (see also R. J. A. Little et al., 2000). The vertical distance between the two non-parallel regression lines at Z = bzC A equals the estimated difference of the adjusted means for an ANCOVA model with a common slope. In other words, if the interaction term is omitted although the regression slopes are not parallel, the (conditional) treatment effect for the value Z = bzC Aof the covariate is estimated.
3.2 General Linear Model 64
Average Distance and Average Total Effect In line with Rubin (1977), who defined the average distance as “the sum of the vertical difference of the population within-group regressions weighted by the population distribution of Z ”, Rogosa (1980) gave an equivalent formulation for a covariate-treatment regression as moderated regression model:
∆(Z ) = 1 N
XN i=1
¡γ10+ γ11· zi
¢. (3.19)
This average distance yields an unbiased estimator of the average total effect (for instance, based on least-squares estimated regression coefficients) provided that the covariate-treatment regression is unbiased and the linear parameterization of the covariate-treatment regression holds (with ˆµZ= N1PN
i=1zi as estimated mean of the covariate):
ATEd10= 1 N
XN i=1
¡g1(Z = zi)¢
= ˆγ10+ ˆγ11· ˆµZ= 1 N
XN i=1
¡ˆγ10+ ˆγ11· zi
¢. (3.20)
In other words, as described also by Finney, Mitchell, Cronkite, and Moos (1984), the average total effect in Equation (3.20) is estimated as the average of the simple slopes across all units (see also Aiken & West, 1996).
We describe this relationship here in great detail to point out that the average total effect, although defined in a very general manner by Steyer et al. (in press), results in a quantity well known in the literature on the traditional treatment of the non-parallel regression slopes (Rogosa, 1980). Nevertheless, to obtain a valid standard error for the estimator in Equation (3.20) the properties of the ordinary least-squares estimation must be taken into account. This next logical step was not carried out by Rogosa (1980, p. 308, nomenclature changed): “The statistical methods considered in this article condition on the values of Z that are observed, as is conventional in regression analysis. The data can be thought of as consisting of N Z , Y pairs that are a random sample from the population bivariate distribution of Z and Y . The values of Z are not chosen or fixed in advance. However, inferences from these data are restricted to sub populations having the same values or configuration of Z because inferences from the linear model are conditional on the observed values of Z .” Because of the product of the regression coefficient ˆγ4and the estimated mean of the covariate ˆµZ
the estimator in Equation (3.20) is clearly not conditional on the covariate (see also subsection 3.2.3). This can also be verified simply by re-writing the definition of the average total effect for a discrete covariate Z in the following way:
ATE10= E¡ g1(Z )¢
=X
z
g1(Z = z)P(Z = z). (3.21)
Obviously, this formulation of the expectation of the effect function (or the equivalent notation as integral over the distribution of a continuous covariate) incorporates the unconditional distribution of the covariate Z .
3.2 General Linear Model 65
Note that Rogosa (1980) also developed a modified test statistic based on the adjusted mean difference for a classical ANCOVA model without interaction terms, which is proposed to be at least less influenced by violations of the assumption of equal regression slopes. Among others, Harwell and Serlin (1988) reported that this test statistic performs poorly in simulation studies (see for a summary Harwell, 2003). Therefore, we will not consider such an approach in this thesis.
In the remaining part of this subsection we present two of the traditional methods for probing inter-action terms in order to show that all these methods are valid only for observed values of Z = z, i. e., under the fixed-X assumption. None of these techniques can be applied to derive valid unconditional inference about the average total effect.
Pick-a-point Technique The pick-a-point approach (Rogosa, 1980) involves plotting and testing the con-ditional effects at selected and meaningful values of the covariate (e. g., high, medium, and low). For or-dinary least-squares estimates of the regression parameters and their variances and covariance, tests at particular values Z = z can be performed with the help of Equation (3.18) [see, for example, Rogosa, 1980;
Cohen et al., 2003; Bauer & Curran, 2005]:
t = gˆ1(Z = z)
£Vard¡
gˆ1(Z = z)¢¤1/2. (3.22)
The application of the pick-a-point approach is suggested when an interaction between covariates and treatment is significant and if it makes substantive sense to compare groups at a particular value z of the co-variate Z . The confidence interval for the comparison at Z = z is: C IZ =z= ˆg1(Z = z)±tCrit£
Vard¡
gˆ1(Z = z)¢¤1/2
(see, e. g., Jaccard, Turrisi, & Wan, 1990). While the application of the pick-a-point approach is simple due to available software implementations (see, e. g., O’Connor, 1988), typically only a small number of arbitrarily selected values are evaluated. Additionally, the selected z-values may even reside outside of the observed range of the regressor (e. g., one standard deviation above the mean of a skewed covariate, see Bauer & Cur-ran, 2005). It might, therefore, be more interesting to examine the full range of the covariate as described in the next paragraph.
Johnson-Neyman Technique An alternative approach is the evaluation of the effect function ˆg1(Z = z) at each level of the covariate with the goal of determining regions of statistically significant (conditional) differences. The main idea behind this so-called Johnson-Neyman Technique (J-N) can be described as a reversion of the t-test presented in Equation (3.22). Now, the critical tCrit-value (i. e., ±1.96 for sufficiently large samples) is taken as a known constant, and the values z∗1/2of the covariate Z which solve the equation are searched for:
0 = tCrit2 ·£ Vard¡
gˆ1(Z = z)¢¤
− gˆ1(Z = z)2. (3.23)
3.2 General Linear Model 66
The quadratic Equation (3.23) can be solved by plugging in Equation (3.15) and Equation (3.18), which leads to two real valued roots z∗1/2under certain conditions (see Rogosa, 1981, for a discussion). Bauer and Curran (2005) described a step-by-step derivation of the formula for the case of a simple linear parameterized effect function. The obtained two values z1/2∗ are the boundaries of the so-called region of significance.12
Potthoff (1964) extended the J-N procedure by controlling the type-I-error rate for a large number of possible values of the covariate Z = z, at which a test can be performed. Larholt and Sampson (1995) analyzed the robustness of the regions of significance with respect to heterogeneity of residual variances for equal and unequal sample sizes. Furthermore, generalizations of this approach to a J-N type procedure for hierarchical linear models are given by various authors (see, for example, Tate, 2004; Miyazaki & Maier, 2005; Bauer & Curran, 2005).
The Johnson-Neyman technique as a method for probing interaction terms is also easily accessible due to various software implementations (see, for example, Karpman, 1983, for a SPSS implementation, Hunka
& Leighton, 1997, for a generalization to more complicated effect functions in Mathematica, and Preacher, Curran, & Bauer, 2006, for a collection of online resources that implement these techniques for different models). A closely related strategy for describing the conditional effects is the computation of confidence bands, that is the computation or plotting of the confidence interval for the whole range of the covariate:
C B(z) = gˆ1(Z = z) ± tCrit£ Vard¡
gˆ1(Z = z)¢¤1/2
. (3.24)
These confidence bands are narrowest at the value Z = bzC A because of the property presented in Equa-tion (3.18).
Summary The common strategies for probing interaction terms were described in this subsection in more detail in order to show that they share the same idea of making inference about conditional effects given a fixed value z of the covariate Z . All methods rely on the estimated variance of the simple slope for a fixed value of the covariate dVar¡
ˆ
g1(Z = z)¢
from Equation (3.16). Nevertheless, Rogosa (1980) was already aware of the average distance (in the tradition of Rubin, 1977), and he formulated an appropriate estimator for the linear parameterized analysis of covariance with interaction terms.
In the next subsection we shall present an analytical proof that the unconditional variance of the ATE–
estimator is underestimated by the estimated variance obtained from the general linear model (i. e., from the ordinary least-squares regression).
12The same difficulties as for the pick-a-point technique with respect to the interpretation of the obtained boundaries of the region of significance occur when z∗1/2are outside of the observed range of the regressor Z .
3.2 General Linear Model 67