2.1 Introduction
The very nature of data collected in different fields as as diverse as economics, finance, biology, medicine, and engineering leads one nat-urally to a consideration of time series models. Samples taken from all of these disciplines are typically observed over a sequence of time periods. Often, for example, one observes hourly or daily or monthly or yearly data, even tick-by-tick trade data, and it is clear from exam-ining the histories of such series over a number of time periods that the adjacent observations are by no means independent. Hence, the usual techniques from classical statistics, developed primarily for in-dependent identically distributed (iid) observations, are not applicable.
Clearly, we can not hope to give a complete accounting of the the-ory and applications of time series in the limited time to be devoted to this course. Therefore, what we will try to accomplish, in this presentation is a considerably more modest set of objectives, with more detailed references quoted for discussions in depth. First, we will attempt to illustrate the kinds of time series analyses that can arise in scientific contexts, particularly, in economics and finance, and give examples of applications using real data. This necessarily
15
will include exploratory data analysis using graphical displays and numerical summaries such as the autocorrelation and cross correla-tion funccorrela-tions. The use of scatter diagrams and various linear and nonlinear transformations also will be illustrated. We will define classical time series statistics for measuring the patterns described by time series data. For example, the characterization of consistent trend profiles by dynamic linear or quadratic regression models as well as the representation of periodic patterns using spectral analysis will be illustrated. We will show how one might go about examining plausible patterns of cause and effect, both within and among time series. Finally, some time series models that are particularly useful such as regression with correlated errors as well as multivariate au-toregressive and state-space models will be developed, together with unit root, co-integration, and nonlinear time series models, and some other models. Forms of these models that appear to offer hope for applications will be emphasized. It is recognized that a discussion of the models and techniques involved is not enough if one does not have available the requisite resources for carrying out time series compu-tations; these can be formidable. Hence, we include a computing package, called R.
In this chapter, we will try to minimize the use of mathematical notation throughout the discussions and will not spend time devel-oping the theoretical properties of any of the models or procedures.
What is important for this presentation is that you, the reader, can gain a modest understanding as well as having access to some of the principal techniques of time series analysis. Of course, we will refer to Hamilton (1994) for additional references or more complete dis-cussions relating to an application or principle and will discuss them in detail.
2.2 Stationary Time Series
We begin by introducing several environmental and economic as well as financial time series to serve as illustrative data for time series methodology. Figure 2.1 shows monthly values of an environmental series called the Southern Oscillation Index (SOI) and associated recruitment (number of new fish) computed from a model by Pierre Kleiber, Southwest Fisheries Center, La Jolla, California. Both series are for a period of 453 months ranging over the years 1950 − 1987.
The SOI measures changes in air pressure that are related to sea surface temperatures in the central Pacific. The central Pacific Ocean warms up every three to seven years due to the El Ni˜no effect which has been blamed, in particular, for foods in the midwestern portions of the U.S.
Both series in Figure 2.1 tend to exhibit repetitive behavior, with regularly repeating (stochastic) cycles that are easily visible. This periodic behavior is of interest because underlying processes of interest may be regular and the rate or frequency of oscillation char-acterizing the behavior of the underlying series would help to identify them. One can also remark that the cycles of the SOI are repeating at a faster rate than those of the recruitment series. The recruit series also shows several kinds of oscillations, a faster frequency that seems to repeat about every 12 months and a slower frequency that seems to repeat about every 50 months. The study of the kinds of cycles and their strengths will be discussed later. The two series also tend to be somewhat related; it is easy to imagine that somehow the fish population is dependent on the SOI. Perhaps there is even a lagged relation, with the SOI signalling changes in the fish population.
The study of the variation in the different kinds of cyclical
behav-ior in a time series can be aided by computing the power spectrum which shows the variance as a function of the frequency of oscilla-tion. Comparing the power spectra of the two series would then give valuable information relating to the relative cycles driving each one.
One might also want to know whether or not the cyclical variations of a particular frequency in one of the series, say the SOI, are asso-ciated with the frequencies in the recruitment series. This would be measured by computing the correlation as a function of frequency, called the coherence. The study of systematic periodic variations
0 100 200 300 400
−1.0−0.50.00.51.0
Southern Oscillation Index
0 100 200 300 400
020406080100
Recruit
Figure 2.1: Monthly SOI (left) and simulated recruitment (right) from a model (n=453 months, 1950-1987).
in time series is called spectral analysis. See Shumway (1988) and Shumway and Stoffer (2001) for details.
We will need a characterization for the kind of stability that is exhibited by the environmental and fish series. One can note that the two series seem to oscillate fairly regularly around central values (0 for SOI and 64 for recruitment). Also, the lengths of the cycles and their orientations relative to each other do not seem to be changing drastically over the time histories.
In order to describe this in a simple mathematical way, it is
con-venient to introduce the concept of a stationary time series.
Suppose that we let the value of the time series at some time point t be denoted by {xt}. Then, the observed values can be represented as x1, the initial time point, x2, the second time point and so forth out to xn, the last observed point. A stationary time series is one for which the statistical behavior of xt1, xt2, . . . , xtk is identical to that of the shifted set xt1+h, xt2+h, . . . , xtk+h for any collection of time points t1, t2, . . ., tk and for any shift h. This means that all of the multivariate probability density functions for subsets of variables must agree with their counterparts in the shifted set for all values of the shift parameter h. This is called strictly strong station-ary, which can be regarded as a mathematical assumption.
The above version of stationarity is too strong for most applications and is difficult or impossible to be verified statistically in applica-tions. Therefore, to relax this mathematical assumption, we will use a weaker version, called weak stationarity or covariance sta-tionarity, which requires only that first and second moments satisfy the constraints. This implies that
E(xt) = µ and E[(xt+h − µ)(xt − µ)] = γx(h), (2.1) where E denotes expectation or averaging over the population densi-ties and h is the shift or lag. This implies, first, that the mean value function does not change over time and that γx(h), the population covariance function, is the same as long as the points are separated by a constant shift h. Estimators for the population covariance are important diagnostic tools for time correlation as we shall see later.
When we use the term stationary time series in the sequel, we mean weakly stationary as defined by (2.1). The autocorrelation function (ACF) is defined as a scaled version of (2.1) and is written as
ρx(h) = γx(h)/γx(0), (2.2)
which is always between −1 and 1. The denominator of (2.2) is the mean square error or variance of the series since γx(0) = E[(xt−µ)2].
Exercise: For given time series {xt}nt=1, how do you check whether the time series {xt} is weakly or strong stationary? Thank about this problem.
Example 1.1: We introduce in this example a simple example of a time domain model to be considered in detail later. A simple moving average model assumes that the series xt is generated from linear combinations of independent or uncorrelated “shocks” wt, sometimes called white noise1 (WN), to the system. For example, the simple first order moving average series
xt = wt − 0.9 wt−1
is stationary when the inputs {wt} are assumed independent with E(wt) = 0 and E(w2t) = 1. It can be easily verified that E(xt) = 0 and γx(h) = 1 + 0.92 if h = 0, −0.9 if h = ±1, 0 if h > 1 (please verify this). We can see what such a series might look like by drawing random numbers wt from a standard normal distribution and then computing the values of xt. One such simulated series is shown in Figure 2.2 for n = 200 values; the series resembles vaguely the real data in the bottom panel of Figure 2.1.
Many of our techniques are based on the idea that a suitably modified time series can be regarded as stationary (weakly). This requires first that the mean value function be constant as in (2.1).
Several simple commonly occurring nonstationary time series can be illustrated by letting this assumption be violated. For example, the
1white noise is defined as a sequence of uncorrelated ransom variables with mean zero and same variance.
0 50 100 150 200
−3−2−10123
Simulated MA(1)
Figure 2.2: Simulated MA(1) with θ1 = 0.9.
series yt = t + xt, where xt is the moving average series of Example 1.1, will be nonstationary because E(xt) = t and the constant mean assumption of (2.1) is clearly violated.
Four techniques for modifying the given series to improve the ap-proximation to stationarity are detrending, differencing, trans-formations, and linear filtering as discussed below. A simple example of a nonstationary series is also given later.
2.2.1 Detrending
One of the dominant features on many economic and business time series is the trend. Such a trend can be upward or downward, it can be steep or not, and it can be exponential or approximately linear. Since a trend should be definitely somehow be incorporated in a time series model, simply because it can be exploited for out-of-sample forecasting, an analysis of trend behavior typically requires quite some research input. The discussion later will show that the type of trend has an important impact on forecasting.
The general version of the nonstationary time series given above is
to assume a general trend of the form yt = Tt + xt, particularly, the linear trend Tt = β1 + β2t. If one looks for a method of modifying the above series to achieve stationarity, it is natural to consider the residual
"
xt = yt −T#t = yt − β#1 − β#2t
as a plausible stationary series where β#1 and β#2 are the estimated intercept and slope of the least squares line for yt as a function of t. The use of the residual or detrended series is common and the process of constructing residual is known as detrending.
Example 1.2: To illustrate the presence of trends in economic data, consider the five graphs in Figure 2.3, which are the annual indices of
1955 1960 1965 1970 1975 1980 1985
5678
agriculture commerce consumption industry transport
Figure 2.3: Log of annual indices of real national output in China, 1952-1988.
real national output (in logs) in China in five different sectors for the sample period 1952 − 1988. These sectors are agriculture, industry, construction, transportation, and commerce.
From this figure, it can be observed that the five sectors have grown over the year at different rates, and also that the five sectors seem to have been affected by the, likely, exogenous shocks to the Chinese economy around 1958 and 1968. These shocks roughly correspond
to the two major political movements in China: the Great-Leap-Forward around 1958 until 1962 and the Cultural Revolution from 1966 to 1976. It also appears from the graphs that these political movements may not have affected each of the five sectors in a similar fashion. For example, the decline of the output in the construction sector in 1961 seems much larger than that in the industry sector in the same year. It also seems that the Great-Leap-Forward shock already had an impact on the output in the agriculture sector as early as 1959. To quantify the trends in the five Chinese output series, one might consider a simple regression model with a linear trend as mentioned earlier or some more complex models.
Example 1.3: As another more interesting example, consider the global temperature series given in Figure 2.4. There appears to be an increasing trend in global temperature which may signal global
o
1900 1950 2000
−1.0−0.50.00.5
Original Data with Linear and Nonlinear Trend
Figure 2.4: Monthly average temperature in degrees centigrade, January, 1856 - February 2005, n = 1790 months. The straight line (wide and green) is the linear trend y = −9.037 + 0.0046 t and the curve (wide and red) is the nonparametric estimated trend.
warming or it may be just a normal fluctuation. Fitting a straight line relating time t to temperature in degrees Centigrade by simple least squares leads to β#1 = −9.037, β#2 = 0.0046 and a detrended series shown in the left panel of Figure 2.5. Note that the detrended series
1900 1950 2000
−0.50.00.5
Detrended: Linear
1900 1950 2000
−0.6−0.4−0.20.00.20.40.6
Detrended: Nonlinear
Figure 2.5: Detrended monthly global temperatures: left panel (linear) and right panel (nonlinear).
still contains a trend like bulge that is highest at about t = 60 years.
In this case the slope of the line is often used to argue that there is a global warming trend and that the average increase is approximately 0.83 degrees F per 100 years. It is clear that the residuals in Figure 2.5 still contain substantial correlation and the ordinary least squares model may not be appropriate.
There may also be other functional forms that do a better job of detrending; for example, quadratic or logarithmic representations are common or nonparametric approach can be used (We will discuss this approach in detail later); see the detrended series shown in the right panel of Figure 2.5. Detrending is particularly essential when one is estimating the covariance function and power spectrum.
2.2.2 Differencing
A common method for achieving stationarity in nonstationary cases is with the first difference
∆yt = yt − yt−1,
where ∆ is called the differencing operator. The use of differenc-ing as a method for transformdifferenc-ing to stationarity is common also for series with trend. For example, in the trend in Example 1.3, the differenced series would be ∆yt = b + xt − xt−1, which is stationary because the difference xt − xt−1 can be shown to be stationary.
Example 1.4: The first difference of the global temperature se-ries is shown in Figure 2.6 and we see that the upward linear trend has disappeared as has the trend like bulge that remained in the
1900 1950 2000
−0.50.00.51.0
Differenced Time Series
Figure 2.6: Differenced monthly global temperatures.
detrended series. Higher order differences are defined as successive applications of the operator ∆. For example, the second difference is ∆2yt = ∆∆yt so that∆2yt = yt − 2 yt−1 + yt−2. If the model also contains a quadratic trend term c t2, it is easy to show that taking the second difference reduces the model to a stationary form.
The trends in Figures 2.3 and 2.4 are all of the familiar type, that is, many economic time series display an upward moving trend. It is however not necessary for a trend to move upwards to be called a trend. It is also that a trend is less smooth and may display slowly changing tendencies which once in a while change directions.
Example 1.5: An example of such a trending pattern is given in the top left panel of Figure 2.7 and the first difference in the top right
50 60 70 80 90
100150200250 Data
0 10 20 30 40
0204060
First difference
0 10 20 30 40
−20−100102030
Second order difference
Figure 2.7: Annual stock of motor cycles in the Netherlands, 1946-1993.
panel of Figure 2.7 and the second order difference in the bottom left panel of Figure 2.7, where the annual stock of motor cycles in The Netherland is displayed, for 1946 − 1993, with the first order and second order differencing time series. From the figures in the right top and bottom panels, we can see that the differencing might not work well for this example. One way to describe this changing trends is to allow the parameters to change over time, driven by some exogenous shocks (macroeconomic variables)2, for example, oil shock in 1974.
2.2.3 Transformations
A transformation that cuts down the values of larger peaks of a time series and emphasizes the lower values may be effective in reducing
2see the paper by Cai (2006)
nonstationary behavior due to changing variance. An example is the logarithmic transformation yt = log(xt), where log denotes the exponential-base logarithm.
Example 1.6: For example, the data shown in Figure 2.8 represent quarterly earnings per share for the American Company Johnson &
0 20 40 60 80
051015
J&J Earnings
0 20 40 60 80
012
transformed log(earnings)
Figure 2.8: Quarterly earnings for Johnson & Johnson (4th quarter, 1970 to 1st quarter, 1980, left panel) with log transformed earnings (right panel).
Johnson from the from the fourth quarter of 1970 to the first quar-ter of 1980. It is easy to note some very nonstationary behavior in this series that cannot be eliminated completely by differencing or detrending because of the larger fluctuations that occur near the end of the record when the earnings are higher. The right panel of Fig-ure 2.8 shows the log-transformed series and we note that the latter peaks have been attenuated so that the variance of the transformed series seems more stable. One would have to eliminate the trend still remaining in the above series to obtain stationarity. For more details on the current analyses of this series, see the later analyses and the papers by Burman and Shumway (1998) and Cai and Chen (2006).
A general transformation is the well-known Box-Cox transfor-mation; see Hamilton (1994, p.126), Shumway (1988), and Shumway
and Stoffer (2000), defined in terms of arbitrary power xαt for some α in a certain range, which can be chosen based on some optimal criterion such as the smallest mean squared error.
2.2.4 Linear Filters
The first difference is a linear combination of the values of the series at two lags, say 0 and 1 and has the effect of retaining the faster oscillations and attenuating or reducing the slower oscillations. We may define more general linear filters to do other kinds of smoothing or roughening of a time series to enhance signals and attenuate noise.
Consider the general linear combination of past and future values of a time series given as
yt = $∞
j=−∞aj xt−j
where aj, j = 0, ±1, ±2, . . ., define a set of fixed filter coefficients to be applied to the series of interest. An example is the first difference where a0 = 1, a1 = −1, aj = 0 otherwise. Note that the above {yt} is also called a linear process in probability literature.
Example 1.7: To give a simple illustration, consider the twelve month moving average aj = 1/12, j = 0, ±1, ±2, ±3, ±4, ±5,
±6 and zero otherwise. The result of applying this filter to the SOI index is shown in Figure 2.9. It is clear that this filter removes some higher oscillations and produces a smoother series. In fact, the yearly oscillations have been filtered out (see the bottom panel in Figure 2.9) and a lower frequency oscillation appears with a cycling rate of about 42 months. This is the so-called El Ni˜no effect that accounts for all kinds of phenomena. This filtering effect will be examined further later on spectral analysis since it is extremely important to know exactly how one is influencing the periodic oscillations by filtering.
0 100 200 300 400
−1.0−0.50.00.51.0
0 100 200 300 400
−0.50.00.5
Figure 2.9: The SOI series (black solid line) compared with a 12 point moving average (red thicker solid line). The top panel: original data and the bottom panel: filtered series.
To summarize, the graphical examination of time histories can point the way to further analyses by noting periodicities and trends that may be present. Furthermore, looking at time his-tories of transformed or filtered series often gives an intuitive idea as
To summarize, the graphical examination of time histories can point the way to further analyses by noting periodicities and trends that may be present. Furthermore, looking at time his-tories of transformed or filtered series often gives an intuitive idea as