To recap, the general linear hypothesis can be stated as H0: Cβ = d. Here, C is
a p× (k + 1) matrix, d is a p × 1 vector and both C and d are user specified and
depend on the application at hand. Although k+ 1 is the number of regression
coefficients, p is the number of restrictions under H0 on these coefficients. (For
those readers with knowledge of advanced matrix algebra, p is the rank of C.) This null hypothesis is tested against the alternative Ha : Cβ = d. This may be
obvious, but we do require p≤ k + 1 because we cannot test more constraints
than free parameters.
To understand the basis for the testing procedure, we first recall some of the basic properties of the regression coefficient estimators described in Section 3.3. Now, however, our goal is to understand properties of the linear combinations of regression coefficients specified by Cβ. A natural estimator of this quan- tity is Cb. It is easy to see that Cb is an unbiased estimator of Cβ, because E Cb= CE b = Cβ. Moreover, the variance is Var (Cb) = CVar (b) C =
σ2CX X −1C . To assess the difference between d, the hypothesized value
of Cβ, and its estimated value, Cb, we use the following statistic:
F-ratio=
(Cb− d) CX X −1C −1(Cb− d)
psfull2 . (4.1)
Here, s2
fullis the mean square error from the full regression model. Using the theory
of linear models, it can be checked that the statistic F -ratio has an F -distribution with numerator degrees of freedom df1 = p and denominator degrees of freedom
df2= n − (k + 1). Both the statistic and the theoretical distribution are named
for R. A. Fisher, a renowned scientist and statistician who did much to advance statistics as a science in the early half of the twentieth century.
Like the normal and the t-distribution, the F -distribution is a continuous distribution. The F -distribution is the sampling distribution for the F -ratio and is proportional to the ratio of two sum of squares, each of which is positive or zero. Thus, unlike the normal distribution and the t-distribution, the F -distribution takes on only nonnegative values. Recall that the t-distribution is indexed by a single degree-of-freedom parameter. The F -distribution is indexed by two degree of freedom parameters: one for the numerator, df1, and one for the denominator, df2. Appendix A3.4 provides additional details.
Appendix A3.4 provides additional details about the
F-distribution, including a graph and distribution table.
The test statistic in equation (4.1) is complex in form. Fortunately, there is an alternative that is simpler to implement and to interpret; this alternative is based on the extra sum of squares principle.
Procedure for Testing the General Linear Hypothesis
(i) Run the full regression and get the error sum of squares and mean square error, which we label as (Error SS)fulland sfull2 , respectively.
(ii) Consider the model assuming the null hypothesis is true. Run a regres- sion with this model and get the error sum of squares, which we label (Error SS)reduced.
(iii) Calculate
F-ratio= (Error SS)reduced− (Error SS)full
psfull2 . (4.2)
(iv) Reject the null hypothesis in favor of the alternative if the F -ratio exceeds an F -value. The F -value is a percentile from the F -distribution with
df1= p and df2 = n − (k + 1) degrees of freedom. The percentile is
one minus the significance level of the test. Following our notation with the t-distribution, we denote this percentile as Fp,n−(k+1),1−α, where α is
the significance level.
This procedure is commonly known as an F -test.
Section 4.7.2 provides the mathematical underpinnings. To understand the extra-sum-of-squares principle, recall that the error sum of squares for the full model is determined to be the minimum value of
SS(b∗0, . . . , b∗k)= n i=1 yi− b∗0+ · · · + b∗kxi,k 2 . Here, SS(b∗
0, . . . , b∗k) is a function of b∗0, . . . , bk∗, and (Error SS)fullis the minimum
over all possible values of b∗
0, . . . , bk∗. Similarly, (Error SS)reducedis the minimum
error sum of squares under the constraints in the null hypothesis. Because there are fewer possibilities under the null hypothesis, we have
4.2 Statistical Inference for Several Coefficients 117
To illustrate, consider our first special case, where H0: βj = 0. In this case,
the difference between the full and the reduced models amounts to dropping a variable. A consequence of equation (4.3) is that, when adding variables to a regression model, the error sum of squares never goes up (and, in fact, usually goes down). Thus, adding variables to a regression model increases R2, the
coefficient of determination.
When adding variables to a regression model, the error sum of squares never goes up. The R2 statistic never goes down.
How large a decrease in the error sum of squares is statistically significant? Intuitively, one can view the F -ratio as the difference in the error sum of squares divided by the number of constraints, ((Error SS)reduced− (Error SS)full)/p, and
then rescaled by the best estimate of the variance term, the s2, from the full
model. Under the null hypothesis, this statistic follows an F -distribution, and we can compare the test statistic to this distribution to see whether it is unusually large.
Using the relationship Regression SS = Total SS − Error SS, we can reex-
press the difference in the error sum of squares as
(Error SS)reduced− (Error SS)full= (Regression SS)full
− (Regression SS)reduced.
This difference is known as a Type III sum of squares. When testing the importance of a set of explanatory variables, xk+1, . . . , xk+p,in the presence of x1, . . . , xk,
you will find that many statistical software packages compute this quantity directly in a single regression run. The advantage of this is that it allows the analyst to perform an F -test using a single regression run instead of two regres- sion runs, as in our four-step procedure described previously.
Example: Term Life Insurance, Continued. Before discussing the logic and
the implications of the F -test, let us illustrate the use of it. In the term life insurance example, suppose that we want to understand the impact of marital status. Table4.3presented a mixed message in terms of t-ratios; sometimes they were statistically significant and sometimes not. It would be helpful to have a formal test to give a definitive answer, at least in terms of statistical significance. Specifically, we consider a regression model using LNINCOME, EDUCATION, NUMHH, MAR0, and MAR2 as explanatory variables. The model equation is
y= β0+ β1LNINCOME+ β2EDUCATION+ β3NUMHH + β4MAR0+ β5MAR2.
Our goal is to test H0: β4= β5= 0.
(i) We begin by running a regression model with all k+ p = 5 variables.
The results were reported in Table4.2, where we saw that (Error SS)full=
615.62 and s2
(ii) The next step is to run the reduced model without MAR0 and MAR2. This was done in Table 3.3 of Chapter 3, where we saw that (Error SS)reduced=
630.43.
(iii) We then calculate the test statistic
F-ratio= (Error SS)reduced− (Error SS)full
psfull2 =
630.43− 615.62
2× 2.289 = 3.235.
(iv) The fourth step compares the test statistic to an F -distribution with df1 = p= 2 and df2 = n − (k + p + 1) = 269 degrees of freedom. Using a
5% level of significance, it turns out that the 95th percentile is F -value≈
3.029. The corresponding p-value is Pr(F > 3.235)= 0.0409. At the
5% significance level, we reject the null hypothesis H0 : β4 = β5 = 0.
This suggests that it is important to use marital status to understand term life insurance coverage, even in the presence of income, education, and number of household members.
Some Special Cases
The general linear hypothesis test is available when you can express one model as a subset of another. For this reason, it useful to think of it as a device for comparing “smaller”to “larger”models. However, the smaller model must be a subset of the larger model. For example, the general linear hypothesis test cannot be used to compare the regression functions E y = β0+ β7x7 versus E y= β0+ β1x1+ β2x2+ β3x3+ β4x4. This is because the former, smaller function is
not a subset of the latter, larger function.
The general linear hypothesis can be used in many instances, although its use is not always necessary. For example, suppose that we wish to test H0 : βk = 0.
We have already seen that this null hypothesis can be examined using the t- ratio test. In this special case, it turns out that (t-ratio)2= F -ratio. Thus, these
tests are equivalent for testing H0: βk= 0 versus Ha : βk= 0. The F -test has
the advantage that it works for more than one predictor, whereas the t-test has the advantage that one can consider one-sided alternatives. Thus, both tests are considered useful.
Dividing the numerator and denominator of equation (4.2) by Total SS, the test statistic can also be written as
F-ratio= R2full− R2reduced /p 1− R2 full /(n− (k + 1)). (4.4)
The interpretation of this expression is that the F -ratio measures the drop in the coefficient of determination, R2.
The expression in equation (4.2) is particularly useful for testing the adequacy of the model, our Special Case 5. In this case, p= k, and the regression sum of
squares under the reduced model is zero. Thus, we have
F-ratio= (Regression SS)full /k sfull2 = (Regression MS)full (Error SS)full .
4.2 Statistical Inference for Several Coefficients 119
This test statistic is a regular feature of the ANOVA table for many statistical packages.
For example, in our term life insurance example, testing the adequacy of the model means evaluating H0 : β1 = β2 = β3 = β4 = β5 = 0. From Table4.2, the F-ratio is 68.66/2.29 = 29.98. With df1 = 5 and df2 = 269, we have that the F -
value is approximately 2.248 and the corresponding p-value is Pr(F > 29.98)≈
0. This leads us to reject strongly the notion that the explanatory variables are not useful in understanding term life insurance coverage, reaffirming what we learned in the graphical and correlation analysis. Any other result would be surprising.
For another expression, dividing by Total SS, we may write
F-ratio= R 2
1− R2
n− (k + 1)
k .
Because both F -ratio and R2 are measures of model fit, it seems intuitively
plausible that they are related in some fashion. A consequence of this relationship is the fact that as R2 increases, so does the F -ratio and vice versa. The F -ratio
is used because its sampling distribution is known under a null hypothesis, so we can make statements about statistical significance. The R2 measure is used
because of the easy interpretations associated with it.
4.2.3 Estimating and Predicting Several Coefficients