S(Pt-Pt)2
(t=l,...,n).
equal to 1. Once each forecast is multiplied by 3 and added to a constant a, the new set of forecast has undergone what has been called
an ’optimal linear correction’ (Theil, 1961) of forecasts.
We can interpret the two changes with the help of mean square prediction error. A decomposition of Mp is given by
Mp = <Pt-At)2 + a2(Ut)
(4.19)
When the forecast is corrected for bias, pt=A^. This will yield a mean square error for the corrected forecast
Mp = O2(Ut)
when the forecasts are corrected for slope, i.e. 3 = 1 in
At = a + 3Pt + Ut
This will imply that the forecast error U^is uncorrelated with the forecast values P.. In this case the variance of the residuals
t
2 . ' 2 Cf (V^.) is equal to the variance of the forecast error G (U^_).
2 2
Otherwise, G (U^.)>a (V^). Based on this, Mincer and Zarnotwitz (1969) 2 2
M_ = a2(vt) = c2(ut)
Thus, an unbiased and efficient forecast will have ct=O, and 1
P
In general, M^ > a2(Ut) > a2(V^)
(4.20)
(4.21)
In the case of corrected forecasts, the new mean square error is .2,„ x _ _2
m’ = <r(v.) < a (u.) < m
p t t p (4.22)
Granger and Newbold (1973) find the definition of efficiency in the Mincer and Zarnowitz sense (3=1) unacceptable. In particular, they point out the case of a process where the actual realizations are
generated by a random walk model,
At ' At-1 + ®t (4.23)
where is zero mean, finite variance, white noise. Then the Mincer and Zarnowitz criterion of efficiency would find that all the one step predictors of A^, defined below
= At_. ; j=l, 2, 3, .... (4.24)
are efficient. In theory a regression of A or P for any j, will have a zero intercept and unit slope. The efficiency criterion
is thus not able to distinguish between different predictors for different values of j in the above example.
Granger and Newbold (1973) have suggested that as a criterion for ’efficiency’ the following definition may be used
MSE for optimum predictor using information set A
n
MSE for any predictor P^ E
In practice, however, the optimum predictor for any given information set will not be known in general. The concept of
’relative efficiency’ defined subsequently may be more useful in practice.
4.3.2 Relative Efficiency and Combination of Forecasts
The concept of relative efficiency is defined by Mincer and
Zarnowitz as •
. MSE of P
= MSE of P2 ■
where P^ is the predictor in question and P2 is an alternative ’benchmark’ forecast. The criterion has been suggested as a ’rate of return’ index inasmuch as it takes the return to be inversely proportional to the mean square error of P1 and the cost to be inversely proportional to the mean square error of the ’benchmark' prediction. The size of RE indicates whether the prediction is superior to benchmark or not. If RE>1, the forecast is prima facie inferior. If RE<1, the forecast is superior to the benchmark.
Granger and Newbold (1973) argue the size of RE cannot indicate what the improvement in forecast performance would be if greater effort or a larger information set was used in constructing the forecasts. In particular, they suggest that useful insights in forecast evaluation can be obtained by combining forecasts. For two forecasts, P^ and P2, if they are both unbiased, and if the forecast errors, e^ and e2 are bivariate stationary, combinations of the following form are appropriate.
It can be established that the sample variance of the error of composite forecasts (S2) is minimised by taking
S1 + S2 * 2rSlS2
where S^2 and S22 are sample variances of respective forecast errors
of predictors and P^. If K is so chosen, then
SQ2 < min (S^ , S2 )
If the condition O<K<1 is met ’K’ itself can be taken as a measure of relative efficiency. If this condition is not met, or otherwise, the following criterion of ’conditional efficiency’ of P^ with respect to Pj may be considered
CE (P1/P2) = S^/Sj2 if (Sx2 « S22 )
If P1 is the optimum forecast for a given information set, then CE = 1. The conditional efficiency is taken to be a measure of extra information contained in the pair of predictors rather than if just one is used.
The usefulness of this concept of conditional efficiency is limited by the assumption of stationarity. To some extent non-stationarity can be removed by differencing. Altneratively, K may be allowed to vary over time.
The question as to how ’benchmark’ forecasts should be constructed is also important. The convention has been to use naive models like ’no-change’ or ’constant-change’ extrapolation. More recently, sophisticated autoregressive and/or moving average models have also been used. In practice, if one decides to use extrapolative models, it does seem that parameters of the model should be so chosen as to best utilise the available information contained in the past history of the predicted series.
If forecasts based on different information sets are compared, the problem arises as to how the results should be interpreted.
Usually, a multi-equation model forecast will use the past history of many variables and an extrapolative forecast, the past history of a
single variable. Granger and Newbold suggest that appropriate models for comparison are constructed by using the same set of information as in the original model. This calls for building multi-variate time- series models even if the specifications do not conform to economic theory. If benchmarks are constructed in this manner, the criterion of conditional or relative efficiency then boils down to the condition that a forecast is adequate if it cannot be significantly improved by combining with the benchmark forecasts.
4.3.3. Decompositions of Mean Square Error
Theil (1958) has suggested that the mean square error
M P
= j 2 (Pt - At)2
can be decomposed in two ways, viz.,
- M P = (P - A)2 + (Sp - SA>2 + 2(l-r)SpSA (4.26) and M P
= (P - A)2 + (Sp -' rSA)2 + (l-r2)SA2
(4.27)
where P and A are the sample means of predictions and realization, Sp and are their standard deviations and r is the correlation
coefficient between them. The division of the terms on the right-hand side by the mean square error gives rise to the following quantities which have been called ’inequality proportions’
UM = (P - A)/fe 7 P mean proportion us = (SP - SA)2A variance proportion uc = 2(l-r)spsA/Mp covariance proportion UR = (Sp - rSA)2/«p slope proportion UD = (l-r2)SA/Mp disturbance proportion We have, UM + US + UC 1 and UM + UR + UD 1
The terms thus provide information on the relative importance of one source of error rather than another.
ahead predictor based on the information s<
The mean proportion has a positive value if P # A. This is due therefore to ’bias’. The variance proportion is zero when the two standard deviations are equal. It is therefore interpreted as error due to incomplete variation and would arise because of a forecaster’s neglect of causes of fluctuations in the two series. The covariance term is zero when r=l or when Sp.r = SASp, when the
covariance of predictions and realizations takes its maximum value which is the product of the two standard deviations. Any positive value for this term is, therefore, due to incomplete variation. Errors of this type stand less chance of correction.
Similarly, the terms in the second decomposition are interpreted respectively, as those due to bias, slope, and a residual or a
disturbance component. Granger and Newbold observe that this decomposition is not so useful as the first one.
As an example, they consider the case where A is generated by a first order autoregressive process,
At = <*A + £t '0«x<l (4.28)
where e is a zero-mean white noise process. The optimal one-step (A . j>l) is given by
‘-“J i
(4.29) Pt = aAt-l
For this prediction, as N ■* 00,
= 0, US = —■■■-a - nC
S C
If one varies a from 0 to 1, U and U can take any values apart from S C S C
the restrictions 0<U , U <1,U + U =1. This makes the .interpretation of these proportions impossible. The difficulty arises because for an optimal predictor one does not expect S_ and S. to be equal.
Ir A w
In the first decomposition, however, for the same example of first order autoregressive process, and (J^ tend to zero for the optimum predictor and so UD tends to unity.
More useful information can be obtained by an examination of forecast errors. Errors should be tested for randomness and their autocorrelation properties should be explored. First order auto correlation can be tested by the von Neumann ratio and higher order and other forms of serial correlation can be tested by direct
regressions of current errors on lagged errors.
M-. 4- Evaluation of Multi-equation Econometric Models
The procedures outlined above relate to 'single variable forecasts produced in any manner. We now consider the requirements for evaluating the forecasting performance of a multi-equation model. This can be done by a comparison with realizations against predicted values, a comparison with other ’benchmark1 methods like autoregressive or single equation models, and a comparison with other multi-equation models producing forecasts for similar variables.
4.4.1 Variable-by-variable comparisons
The established practice, although not entirely satisfactory, is to evaluate the performance of a model by applying techniques
outlined in previous sections variable-by-variable for all or a subset of the endogenous variables of the model. Thus, indicators like the mean square error and the inequality coefficients are constructed for
each variable. Such a procedure can indicate variables for which
the model does better and those for which the model is not satisfactory. However, except in cases where one model is overwhelmingly superior to another, this procedure cannot be used to rank the prediction performance of different models. If one model gets a few less or more scores, it • cannot be established categorically that it is inferior or superior to
another model. • '
4.4.2 Construction of Composite Performance Indices
One way of ranking models according to prediction performance is to construct composite indices which can refer to all variables together. There are, however, a number of difficulties involved. If one takes an average of an indicator like the mean square error, one has to contend with at least three problems: (i) the units of variables may be
different, some being measured in percentages, some in nominal currency units etc., (ii) the weight which should be given to one variable rather than another, and (iii) how many variables should be included in the composite index. This last problem is important, as by a judicous decomposition of variables in the identities any number of variables can be constructed and the prediction performance can be made to look better.
The problem of units of measurement can be resolved,to some extent, by either defining the mean square error in terms of percentages, or choosing an index like Theil’s second inequality
index where the problem of units has been taken care of. Thus if we have predictions for n variables for m periods P^^_,
(i=l, ...,n, t=l, ...,m), and corresponding realizations A^, , then the mean square error may be defined with respect to percentage changes P| and where
P. - P. i,t i,t-l Pi,t-1 Ai,t Ai,t-1 A. . l ,t-l
A composite standardised mean square error can then be written as
CSMSQ = — mn n Z i=l m Z t=l (Pl i,t - <t>2
The problem of weights is however more intractable. What relative weights should be given to prediction error of one variable vis a vis another calls for a cost function in errors of all variables in question. In practice it will be difficult to obtain such a
function. If it is decided to give arbitrary weights to the errors, some guidelines can be obtained from the purpose of the forecasts, viz., whether the model is designed to forecast primarily the national
income components or prices etc. Similarly, errors can be inter
temporally weighted depending on whether the forecasts are in a short-, medium- or long term perspective.
4.4.3 Information Measures of Predictive Accuracy . Information measures of predictive accuracy (suggested by
Theil (1961, 1966) are applicable in the case where positive fractions adding to unity are predicted and where it is assumed that the cost function is of the log-linear type. The measure is useful in
input-output models and models which forecast national income components. If both national income and its components are forecasted, one can
define variables such that all components are positive and calculate predicted income-share in each case. Assuming that P. (i=l,...n)
1 s
are n forecasted shares and are corresponding realized shares, for any given period of forecast, then the information measure
n
I(A : P) = £ A. log A.
1=1 p?
i
can indicate the power of the model to forecast components of income or output jointly. The lower the value of I(A : P), the better is the forecasting power.
4.4.4 Comparisons with Other Models
Multi-equation model forecasts are usually compared with time-series models and forecasts generated from other multi-equation models.
Requirements of using a similar information set have been considered in section 4.3.2. It is generally observed that adequately built time- series models, either univariate or multi-variate, perform well in terms of predictions at least for the short-run. In this context, a multi equation model may be considered quite satisfactory if it performs at
least as well as the competing time-series model. However, even if it performs worse, the extra information it may contain for purposes of policy analysis in its multi-variate specification, may render it useful.
Comparisons among different multi-equation models are also beset with problems. The problems of weighting and the number of variables to be used for comparisons have already been noted. Proper
comparisons also cannot be made when the size, estimation techniques and sample-periods of different models differ.
One way round this problem is to re-estimate competing models under similar conditions of information and estimation as far as
possible. Thus, models may be re-estimated for a common sample period and with a common estimation technique. An important study of this kind for some U.S. models is by Cooper (1972).
However, apart from being a costly procedure, there are other problems in this exercise. Some estimation techniques are more
appropriate for one set of data and specification, and some for others, especially if non-linearities are involved. Re-estimation with a common technique may, therefore, discriminate against some models.
Also, a mechanical application of the same set of rules violates against the essence of model-building where the analyst reacts by testing and retesting his hypothesis with a given set of data. If data were to change, it is expected that his reactions would also have changed. Howrey et al (1974) have called for a ’tender loving care' (TLC) in the re-estimation of models. Frequently, very small changes in the original model when re-estimated with new data can substantially improve its performance. In particular, since economic theory does not tell
much about precise lag-structures and functional forms which are a matter of sample experimentation, the re-estimated model may deviate from the original model in these matters in the light of revised information. Howrey et al (1974) suggest that- in view of the (i) different requirements of degrees of freedom and (ii) different characteristics of collinearity among variables in different models, the requirements of a homogenous sample period and estimation
techniques may be compromised.
4.4.5 Pre-release Period and Pseudo-forecasts
Before models are released, a standard practice is to ’save' some observations in the available sample, estimate the model with the remaining observations, and use the ’saved1 observations for a test on forecasting performance. Such forecasts have been termed ’pseudo forecasts’. This may be a useful and unavoidable procedure in the model-building stage but a proper evaluation of the performance can only be done after the model has been released. Available data in the pre-release period are expected to affect the model-builder and he would normally make sure that for these the model provides a good fit. Christ (1975) has argued that if one is
’to discriminate between the (inferior) models that have chosen to fit the random or non-enduring features of the economy, and the (superior) models that have been chosen to fit primarily the systematic and enduring features of the economy’
4.4.6 Diagnostic Checks; Role of Exogenous Variables
In the post-release period, an interesting diagnostic check is to allocate the prediction error between structural mispecification and incorrectly predicted exogenous variables. One such decomposition has been suggested by Theil (1961).
Theoretically, one can think of the role of the use of incorrect exogenous values in the following way. Suppose the predictions are made by a linear system of equation such that a vector Yp of
predictions is generated by using a vector Zp of already predicted
exogenous variables. If H^ is the estimated coefficient matrix of the equation system, the reduced-form of the system may be written an
YP = HeZP + UP ... (4.30)
where Up are the reduced-form predicted errors. If Z is the
corresponding vector of ’true’ values of the exogenous variables, and II the ’true’ coefficient matrix, assuming that the true system is also •• linear, we can write the true vector Y of predicted variables as
Y = HZ + U ... (4.31)
where (J is the vector of ’true’ reduced-form errors. The prediction error is then given by
yp
-
y= n zp - nz + up - u
... (4.32) e or yp-
y= (n - n)z + (zp - z)n + (n - n) (zp - z) + (up - u)
e e « • « (4.33)The first term in (4.33) arises because of the discrepancy between and II and is, therefore, due to an incorrectly specified model. The second term arises because of the deviation of Zp from Z and is, therefore,
due to incorrectly predicted exogenous variables. The third term is a mixture of the two but is of second order of importance, i.e., if the first two terms are small, the third term will be negligible.
It is difficult to imagine that these three components of the
forecast errors could.be computed because although Z will be known ex-post, we will rarely know the ’true’ equation system II. What we will know is the vector of the true values of the predicted variables, i.'e., Y.
Hence it may be better to rewrite (4.32) in the following form:
yp
-
y = (n z-
y) + n (zp - z) + (up - u) ....
(3.34)
e e