To determine the performance of the Markov switching models in generating a relatively best measure of time-varying systematic risk, the different tech- niques are formally ranked based on their in-sample performance. The first two criteria used to evaluate and compare the respective in-sample forecasts are the mean absolute error (MAE) and the mean squared error (MSE):
MAEi = 1 τ τ−1 X t=0 |Rˆit−Rit| τ , MSEi = 1 τ τ−1 X t=0 ( ˆRit−Rit)2 τ ,
whereτ is the number of forecast observations and ˆRit = ˆβitR0tdenotes the se-
ries of return forecasts for sectori, calculated as the product of the conditional beta series estimated over the entire sample and the series of market returns which is assumed to be known in advance. The forecast quality is inversely related to the size of these two error measures.
While the mean error criteria can be used to evaluate the average forecast performance over a specified period of time for each model, and each sector individually, they do not allow for an analysis of forecast performances across sectors. From a practical perspective, it is interesting to observe how closely the rank order of forecasted sector returns corresponds to the order of realized sector returns at any time. Spearman’s rank correlation coefficient (ρS
t), a
non-parametric measure of correlation that can be used for ordinal variables in a cross-sectional context, is applied as the third evaluation criteria. After ranking the forecasted and observed sector returns separately for each point of time, where the sector with the highest return ranks first,ρS
t can be computed as ρSt = 1− 6 PIt i=1Dit2 It(It2−1) ,
with Dit being the difference between the corresponding ranks for each sector
and It being the number of pairs of sector ranks, each at timet.
The first step is the analysis of the in-sample forecasts using the first two cri- teria MAE and MSE. Compared to the Markov switching approaches, the degree of inferiority of OLS is remarkably low while the two KF techniques
clearly outperform their competitors. Within the Markov switching frame- work, the MS betas led to lower average errors than the MSM technique. Considering the third criterion, while the highest in-sample rank correlations are observed for the MR (ρS = 0.46) and the RW model (0.26), the MSM
model (0.16) and the the MS (0.17) do only slightly better than OLS (0.15). Figure 4.2 illustrates how the average in-sample rank correlations develop over time for the various modeling techniques.
Figure 4.2: In-sample rank correlation coefficients
0 200 400 600 800 −0.1 0.0 0.1 0.2 0.3 number of observations
average rank correlation
100 200 300 400 500 600 700 800 900 −0.1 0.0 0.1 0.2 0.3 OLS KFRW MS MSM
To sum up, the in-sample comparison suggests that the two Markov switching models are outperformed by their competitors.
While the in-sample analysis is useful to assess the various techniques’ ability to fit the data, their out-of-sample forecast performance is more relevant from a practical point of view. For that purpose, 100 beta and return forecasts based on 100 samples of 520 weekly observations are estimated for each technique. Within this rolling window forecast procedure, the sample is rolled forward by one week while the sample size is kept constant at 520. The first sample, starting on 24 March 1993 and ending on 5 March 2003, is used to calculate
the out-of-sample conditional beta forecasts on 12 March 2003 based on the chosen modeling technique. The 100th beta forecast is then generated based on the last sample starting on 15 February 1995 and ending on 26 January 2005.
Without going into all the details, it can be observed that the KF approaches again offer the best forecast performance, while the two Markov switching approaches yield the worst results. These findings are broadly confirmed in a cross-sectional setting as shown in Figure 4.3, where the Markov switching techniques (each 0.21) produce the worst forecasts.
Figure 4.3: Out-of-sample rank correlation coefficients
0 20 40 60 80 100 0.2 0.3 0.4 0.5 0.6 number of observations
average rank correlation
0 10 20 30 40 50 60 70 80 90 100 0.2 0.3 0.4 0.5 0.6 OLS KFRW MS MSM
4.5
Conclusion
The results of this study indicate that the out-of-sample forecast performances of the two proposed Markov switching models is inferior to that of any time- varying alternative and also to OLS. One reasonable explanation is that the Markov switching models were limited to two-state models, a very common
assumption in the switching CAPM framework. The undesirable consequence of this limitation, which leads to reduced flexibility compared to other ap- proaches, results in an insufficient fit to the evolution of beta during the TMT bubble and the subsequent crash at the stock markets.
One possible solution is to significantly increase the number of states. However, as the number of parameters of the TPM increases quadratically with the num- ber of states, this approach complicates the estimation procedure significantly. One possibility to reduce the number of variables could be a structuring of the TPM. For many models, the main contribution to the likelihood is obtained by the elements on the TPM’s main diagonal. The off-diagonal elements could, for instance, be expressed in polynomial dependence on the main-diagonal element of the respective row. However, this approach is subject to future research.