In the following two subsections we describe the performance of the adjusted standard errors for the average total effect estimator based on regression estimates and based on predictive simulations. As for all other re-gression models with correctly specified mean-models (i. e., covariate-treatment rere-gressions without first order misspecifactions, see Long & Trivedi, 1992), the ATE–estimator based on regression estimates and predictive simulations is unbiased.
Absolute Bias B( dATE10) of the ATE–Estimator
Regression Estimate vs. Predictive Simulation, Grouped by Sample Size
−0.10 0.00 0.10
−0.100.000.050.10
N=100
Regression Estimate
Predictive Simulation
−0.10 0.00 0.10
−0.100.000.050.10
N=250
Regression Estimate
Predictive Simulation
−0.10 0.00 0.10
−0.100.000.050.10
N=400
Regression Estimate
Predictive Simulation
−0.10 0.00 0.10
−0.100.000.050.10
N=1000
Regression Estimate
Predictive Simulation
Figure 4.15: Absolute bias of the ATE–estimator: Scatter plots for a comparison of the regression esti-mates vs. predictive simulations, grouped by sample size N
Absolute Bias and Mean Squared Error of the ATE–Estimator A comparison of the ATE–estimator’s ab-solute biases for regression estimates (x–axis) and predictive simulations (y–axis, presented in Figure 4.15) with the absolute biases of the GLH / mean-centering approach and the absolute biases of the multi-group structural equation models (presented in Figure 4.1 on page 117) reveal some minor random derivations of the average total effect estimator based on the predictive simulations which are due to the simulation
4.5 Results for the General Linear Model 136
procedure. This additional small variability of the ATE–estimator does not depend on the amount of inter-action.24We find no noteworthy differences between the regression estimates and the predictive simulation approach with respect to the mean squared error of the average total effect estimator (see Figure 4.16). The magnitude of the mean squared error decreases — as expected — with increasing sample sizes (notice the different scales within Figure 4.16) but this is true for both methods (all points are on the identity line).
Mean Squared Error MSE£ ATEd10¤
of the ATE–Estimator Regression Estimate vs. Predictive Simulation, Grouped by Sample Size
0 1 2 3 4 5
012345
N=100
Regression Estimate
Predictive Simulation
0.0 0.5 1.0 1.5 2.0
0.00.51.01.52.0
N=250
Regression Estimate
Predictive Simulation
0.0 0.4 0.8
0.00.20.40.60.81.0
N=400
Regression Estimate
Predictive Simulation
0.0 0.4 0.8
0.00.20.40.60.81.0
N=1000
Regression Estimate
Predictive Simulation
Figure 4.16: Mean squared error of the ATE–estimator: Scatter plots for a comparison of the regression estimates vs. predictive simulations, grouped by sample size N
Type-I-Error Rate The obtained empirical type-I-error rates for tests of the hypothesis ATE = 0 based on the regression estimates with adjusted standard errors as suggested by Schafer and Kang (2008) are surpris-ingly good for equal group sizes: Figure 4.17 compares the normal approximation (z–test, x–axis) and the t –test (y–axis) for all conditions of the simulation study I with P (X = 1) = 0.5 (presented as separate scatter plots for each level of the interaction parameter γ11used for data generation). The rejection frequencies obtained from both test statistics for almost all studied conditions with equal group sizes are within the confidence bands (marked by gray and red horizontal and vertical lines), regardless of the interaction pa-rameter γ11.25
Two phenomena can be observed for conditions with unequal group sizes: On the one hand, inflated empirical type-I-error rates for small sample sizes (N = 100) are obvious.26 On the other hand, it is inter-esting to notice the (slightly) different behavior of the t–test compared to the normal approximation based
24A direct comparison of the ATE–estimator obtained from the predictive simulations and the ATE–estimator obtained from the regression estimates is included in the digital appendix as additional Figure 26 on page 37.
25Furthermore, the empirical distribution of rejection frequencies, which is provided as the additional Figure 27 on page 38 of the digital appendix, reveals that there are no observable systematic small sample differences between the z–test and the t –test for condi-tions with equal group sizes.
26Note that this inflation is more obvious for datasets generated with small interaction effects for conditions with unequal group sizes (see the additional Figure 28 on page 39 and Figure 29 on page 40 of the digital appendix).
4.5 Results for the General Linear Model 137
Type-I-Error Rate for the Hypothesis ATE = 0 Regression Estimates (Normal Approximation vs. t–Test) Equal Group Size [P(X = 1) = 0.5], Grouped by Interaction
2 4 6 8 10
246810
γ11=0.5
Regression Estimate (Normal Approximation)
Regression Estimate (t − Test)
2 4 6 8 10
246810
γ11=1
Regression Estimate (Normal Approximation)
Regression Estimate (t − Test)
2 4 6 8 10
246810
γ11=2.5
Regression Estimate (Normal Approximation)
Regression Estimate (t − Test)
2 4 6 8 10
246810
γ11=5
Regression Estimate (Normal Approximation)
Regression Estimate (t − Test)
2 4 6 8 10
246810
γ11=7.5
Regression Estimate (Normal Approximation)
Regression Estimate (t − Test)
2 4 6 8 10
246810
γ11=10
Regression Estimate (Normal Approximation)
Regression Estimate (t − Test)
N=100 N=250 N=400 N=1000
Figure 4.17: Type-I-error rate: Scatter plots for a comparison of the regression estimates based on a normal approximation and based on a t–test, grouped by interaction γ11[P(X = 1) = 0.5]
z–test (in particular for conditions with strong interaction effects and small sample sizes). Whereas the dis-tribution of the rejection frequencies for the z–test is closer to the desired symmetric disdis-tribution around the nominal 5 % level than is the distribution of rejection frequencies for the t–test for conditions of the simulation study I with N = 100 and P(X = 1) = 0.2 , the reverse is true for conditions of the simulation
4.5 Results for the General Linear Model 138
Type-I-Error Rate for the Hypothesis ATE = 0 Regression Estimate (Normal Approximation)
Figure 4.18: Type-I-error rate: Level plots for the regression estimates based on the normal approxi-mation [R2X |Z= 0.75 vs. R2X |Z= 0.1; N = 1000 and γ01= 5]
study with P(X = 1) = 0.8:27Here the normal approximation is worse for small sample sizes (N = 100). The rejection frequencies of the z–test and the t–test are identical for large sample sizes (N = 1000).28
27This is obvious from a comparison of the upper part of the additional Figure 30 on page 41 of the digital appendix to the lower part of the same figure.
28See also the additional Figure 31 on page 42 of the digital appendix for the corresponding level plots.
4.5 Results for the General Linear Model 139
Figure 4.18 summarizes the findings for the regression estimates (z–test) regarding the empirical type-I-error rates for N = 1000, γ01= 5 and two different values of the dependancy between X and Z as level plots for equal and unequal group sizes. A comparison of this figure with the corresponding figures for the GLH / mean-centering approach (see Figure 4.6 and Figure 4.7), as well as a comparison to the level plots generated from the results of the heteroscedasticity-consistent estimators (see, for instance, Figure 4.12) reveal that there is no observable systematic inflation of the empirical type-I-error rates for the regression estimates due to heterogeneity of residual variance (and unequal group sizes), or due to the amount of the covariate-treatment interaction.
Mean of the Estimated Standard Errors vs.
Standard Deviation of the Estimates Regression Estimates, Grouped by Interaction
Unequal Group Size [P(X = 1) = 0.2]
0.0 0.5 1.0 1.5 2.0
0.00.51.01.52.0
γ11=0.5
SD(ATE10) S.E.(ATE10)
0.0 0.5 1.0 1.5 2.0
0.00.51.01.52.0
γ11=1
SD(ATE10) S.E.(ATE10)
0.0 0.5 1.0 1.5 2.0
0.00.51.01.52.0
γ11=2.5
SD(ATE10) S.E.(ATE10)
0.0 0.5 1.0 1.5 2.0
0.00.51.01.52.0
γ11=5
SD(ATE10) S.E.(ATE10)
0.0 0.5 1.0 1.5 2.0
0.00.51.01.52.0
γ11=7.5
SD(ATE10) S.E.(ATE10)
0.0 0.5 1.0 1.5 2.0
0.00.51.01.52.0
γ11=10
SD(ATE10) S.E.(ATE10)
N=100 N=250
N=400 N=1000
RX2|Z=0.75 RX2|Z=0.5
RX2|Z=0.25 RX2|Z=0.1
Figure 4.19: Mean of the estimated standard errors vs. standard deviation of the estimated average total effects, regression estimates, grouped by interaction γ11[P(X = 1) = 0.2]
4.5 Results for the General Linear Model 140
Relative Bias RB£
S.E.( dd ATE10)¤
of the Standard Error of the ATE–Estimator Regression Estimate, Grouped by Sample Size and Group Size
N=100 and P(X=1)=0.2
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.087
N=100 and P(X=1)=0.5
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.018
N=100 and P(X=1)=0.8
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.071
N=250 and P(X=1)=0.2
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.035
N=250 and P(X=1)=0.5
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.008
N=250 and P(X=1)=0.8
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.024
N=400 and P(X=1)=0.2
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.020
N=400 and P(X=1)=0.5
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.003
N=400 and P(X=1)=0.8
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.018
N=1000 and P(X=1)=0.2
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.007
N=1000 and P(X=1)=0.5
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: 0.000
N=1000 and P(X=1)=0.8
−0.4 −0.2 0.0 0.2 0.4
050100150 mean: −0.005
Figure 4.20: Relative bias of the standard error of the ATE–estimator: Histograms for the regression estimates, grouped by sample size N and group size P(X = 1)
Bias of the Standard Error of the ATE–Estimator The standard errors for the average total effect estimator corrected with the formulas given by Schafer and Kang (2008) are unbiased for almost all conditions of simulation study I. This is displayed in Figure 4.19 for unequal group sizes [P(X = 1) = 0.2]. Each symbol in the six scatter plots represents the empirical standard derivation of the ATE–estimates for one condition of the simulation study on the x–axis [i. e., SD¡
d ATE10¢
] and the average of the calculated standard errors for the ATE–estimator on the y–axis [i. e., dS.E.¡
ATEd10¢
] for the same condition of the simulation study’s design.
The dotted lines (in colors corresponding to the different sample sizes used for generating the datasets) show the results of a simple linear regression of the averaged standard errors on the standard deviation. For
4.5 Results for the General Linear Model 141
equal group sizes the plotted regression lines differ only slightly from the diagonal line, meaning that the observer variability of the standard error is almost unbiasedly estimated with the adjusted standard errors.29 For unequal sample sizes where the treatment group is smaller than the control group [P(X = 1) = 0.2], the relative bias RB£
S.E.( dd ATE10)¤
is apparently larger.30 The conditional distributions of the RB£
S.E.( dd ATE10)¤
for the regression estimate approach, approxi-mated as histograms, are presented in Figure 4.20, grouped by sample size and group size. Obviously, the standard error is biased for unequal group sizes and small sample sizes, but this bias vanishes if the sample size increases.31
Summary Surprisingly good results were observed for the adjusted standard errors of the ATE–estimator obtained from the ordinary least-squares estimated regression estimates. Although Schafer and Kang (2008) did not explicitly mention stochasticity of covariates in their derivation of the formulas for the adjusted standard errors, we verified the appropriateness of these standard errors empirically for the conditions studied in part I of the Monte Carlo simulation.