Introduction to Time Series Analysis Introduction to Time Series Analysis

Copyright(© MTS-2002GG): You are free to use and modify these slides for educational purposes, but please if you improve this material send us your new version.

Introduction to Time Series Analysis Introduction to Time Series Analysis

Gloria González-Rivera

University of California, Riverside and

Jesús Gonzalo U. Carlos III de Madrid

• Sample Space: , the set of possible outcomes of some random experiment

• Outcome: , a single element of the Sample Space

• Event: , a subset of the Sample Space

• Field: , the collection of Events we will be considering

• Random Variables: , a function from the Sample Space  to a State Space S

• State Space: S, a space containing the possible values of a random variables –common choices are the integers N, reals R, k-vectors R^k, complex numbers C, positive reals R₊, etc

• Probability: , obeying the three rules that you must very well know

• Distribution: is the Borel sets (intervals, etc)

Brief Review of Probability Brief Review of Probability







} { Ω  



 E

} E

:

E  

 F 

S :

Z  

] 1 , 0 [ :

P F

} R A

: A { where

,]

1 , 0 [

:   

 B B

•Random Vectors: Z= (Z₁, Z₂ , ..., Z_n) is a n-dimensional random vector if its components Z₁ , ..., Z_n are one-dimensional real-valued random variables

If we interpret t=1, ..., n as equidistant instants of time, Z_t can stand for the outcome of an experiment at time t . Such a time series may, for example, consists of Toyota share prices Z_t at n succeeding days.

The new aspect now, compared to a one-dimensional radnom variable, is that now we can talk about the dependence structure of the random vector.

•Distribution function F_Z of Z : It is the collection of the probabilities

Brief Review (cont) Brief Review (cont)

n}) z ) n( Z ,..., z1 ) 1( Z : ({

P

n) n z

Z ,..., z1 Z1

( P ) z Z( F



















We suppose that the exchange rate €/$ at every fixed instant t between 5p.m and 6p.m. this afternoon is random. Therefore we can interpret it as a realization Z_t() of the random variable Z_t, and so we observe Z_t(), 5<t<6. In order to make a guess at 6 p.m.

about the exchange rate Z₁₉() at 7 p.m. it is reasonable to look at the whole evolution of Z_t( between 5 and 6 p.m. A

mathematical model describing this evolution is called a stochastic process.

Stochastic Processes

Stochastic Processes

R Z

Z

_t

(  ),

_t

:  

Suppose that (1) For a fixed t

Changing the time index, we can generate several random variables:

) (

),...

( ),

(

2

1

  

tn

t

t

Z Z

Z

(2) For fixed 

This is just a random variable.

R T

Z

_

: 

This is a realization or sample function

This collection of random variables is called a STOCHASTIC PROCESSA realization of the stochastic process is called a TIME SERIES

From which a realization is:

n t

z .. ,..

² t

z ,

¹ t

z

A stochastic process is a collection of time indexed random variables defined on some space 

) ,

T t

), t (

Z ( ) T t

t, Z

(     

Stochastic Processes (cont)

Stochastic Processes (cont)

Examples of stochastic processes Examples of stochastic processes

E1: Let the index set be T={1, 2, 3} and let the space of outcomes () be the possible outcomes associated with tossing one dice:



Define

Z(t, )= t + [value on dice]² t

Therefore for a particular say ₃={3}, the realization or path would be (10, 20, 30).

Q1: Draw all the different realizations (six) of this stochastic process.

Q2: Think on an economic relevant variable as an stochastic process and write down an example similar to E1 with it. Specify very clear the sample space and the “rule”

that generates the stochastic process.

E2: A Brownian Motion B=(B_t, t [0, infty]):

• It starts at zero: B_o=0

• It has stationary, independent increments

• For evey t>0, B_t has a normal N(0, t) distribution

• It has continuous sample paths: “no jumps”.

Distribution of a Stochastic Process Distribution of a Stochastic Process

In analogy to random variables and random vectors we want to introduce non-random characteristics of a stochastic process such as its distribution, expectation, etc. and describe its dependence structure. This is a task much more complicated that the description of a random vector. Indeed, a non-trivial stochastic process Z=(Z_t, t  T) with infinite index set T is an infinite-dimensional object; it can be inderstood as the infinite collection of the random variables Z_t, t  T. Since the values of Z are functions on T, the distribution of Z should be defined on subsets of a certain “function space”, i.e.

P(X  A), A  F,

where F is a collection of suitable subsets of this space of functions. This approach is possible, but requires advanced mathematics, and so we will try something simpler.

The finite-dimensional distributions (fidis) of the stochastic process Z are the distributions of the finite-dimensional vectors

(Z_t1,..., Z_tn), t₁, ..., t_n T, for all possible choices of times t₁, ..., t_n  T and every n  1.

Stationarity Stationarity

Consider the joint probability distribution of the collection of random variables

) ,...

, (

) ,...

,

( z

t1

z

t2

z

tn

P Z

t1

z

t1

Z

t2

z

t2

Z

tn

z

tn

F    

1^st order stationary process if

k t any for

z F z

F( _t1)  ( _t1k ) ₁,

n-order stationary process if

k t t any for

z z

F z

z

F( _t , _t ) ( _{t k}, _{t k}) ₁, ₂,

2 1

2

1   

k t t any for

z z

F z

z

F _{t t t k t k n}

n

n ) ( ... ) , ,

...

( ₁

1

1   

Definition.

A process is strongly (strictly) stationary if it is a n-order stationary process for any n.

2^nd order stationary process if

Moments Moments

2 t 2 t

t ) Z t ,

Z ) cov(

t 2 1 , t (

t )]

Z t t )(

Z t [(

E t )

Z t ,

Z ( Cov

dz t t )

z ( 2 f t ) Z t

2 ( t ) Z t

( 2 E

) t Z t ( Var

dz t t )

z ( t f t Z

t ) Z ( E

2 1

2 1

2 2

1 1

2 1









































 

Moments (cont) Moments (cont) For strictly stationary

process:

2

2













t t

because

F z

_t

 F z

_{t k}

 

_t

 

_{t k}

 

1 1

1

1

) ( )

(

provided that

  ) ( ,   ) (

t 2

Z E

t

Z E

k k

t k t t

t

k t k t t

t

k t

t t

k t

t t

t t

k t

t

k t

k t

t t

z z

z z

z z

F z

z F



















































) ,

( )

, (

) , (

then ,

and

let

) ,

( )

, (

) ,

cov(

) ,

cov(

) ,

( )

, (

2 1

2 1

2 1

2 1

2 1

2 1

2 1

2 1

The correlation between any two random variables depends on the time difference

Weak Stationarity Weak Stationarity

A process is said to be n-order weakly stationary if all its joint moments up to order n exist and are time invariant.

Covariance stationary process (2^nd order weakly stationary):

• constant mean

• constant variance

• covariance function depends on time difference between R.V.

Autocovariance and Autocorrelation Functions Autocovariance and Autocorrelation Functions

For a covariance stationary process:

0 2

2

) var(

) var(

) ,

cov(

) ,

( ) (

) (







 







k k

k t t

k t k t

t s s

t t t

Z Z

Z Z Z

Z Cov

Z Var

Z E



















] 1 , 1 [ k

:

(ACF) function

ation autocorrel

k :

R k

:

function ance

autocovari k :















Properties of the autocorrelation function Properties of the autocorrelation function

1.

2.

3.

1

then )

var(

If 

₀

 Z

_t



₀



0 k

1

t, coefficien n

correlatio a

is Since















_k

k

k t

k t

k k t k

t k

k k

k k

Z Z

E

Z Z

E























































) )(

(

) )(

(

since

_{( )}

Partial Autocorrelation Function (conditional correlation) Partial Autocorrelation Function (conditional correlation) This function gives the correlation between two random variables that are k periods apart when the in-between linear dependence (between t and t+k ) is removed.

) ,...

| ,

( by given

is PACF the

, variables random

two be

and Let

1

1  







k t t

k t t

k t t

Z Z

Z Z

Z Z



Motivation Think about a regression model

(without loss of generality, assume that E(Z)=0)

k j ... kk

2 j 2 k 1

j 1 k j

ns expectatio take

) 2 (

j k Zt k et j k Zt Zt ... kk

j k Zt 2 k Zt 2 k j

k Zt 1 k Zt 1 k k

Zt j k Zt

j k Zt by multiply (1)

1 j j

k Zt with ed

uncorrelat k is

et where

k et Zt

... kk 2 k Zt 2 k 1

k Zt 1 k k

Zt

 



 





 













 



 

 





 

 





 

 









 





 



 

 

 

 

 

k j kk

j k

j

k 









      



_{j 1 1 2 2}

...

Dividing by the variance of the process:

k j 1,2,...

0 1

1

2 1

1 2

1 0

1 1

...

...

...































kk k

k k

k kk k

k kk k



























Yule-Walker equations

0 33 1

32 2

31 3

1 33 0

32 1

31 2

2 33 1

32 0

31 1

0 22 1

21 2

1 22 0

21 1

1 11

0 11 1

3 2 1















































































































k k k

1 1

1

1

1 2 1

1

22











 



1 1

1

1 1

1 2

1 1

2 1

3 1 2

2 1

1 1

33



























 



E4: Z_t=

Y_t if t is even

Y_t+1 if t is odd

where Y_t is a stationary time series. Is Z_t weak stationary?

E5: Define the process

S_t = X₁+ ... + X_n , where X_i is iid (0, ^Show that for h>0 Cov (S_t+h, S_t) = t ^ and therefore S_t is not weak stationary.

Examples of stochastic processes Examples of stochastic processes

Examples of stochastic processes

Examples of stochastic processes (cont)(cont)

E6: White Noise Process

A sequence of uncorrelated random variables is called a white noise process.

 

0 for

0 )

, (

) (

) 0 (normally

)

( :

2













k

a a Cov

a Var

a E a

k t t

a t

a a

t t













 







 







 

0 0

0 1

0 0

0 1

0

0

0

ation autocorrel

and ance

Autocovari

2

k k

k k

k k

kk k

a k





 

. . . .

1 2 3 4 k

k

Dependence: Ergodicity Dependence: Ergodicity

• See Reading 1 from Leo Breiman (1969) “Probability and Stochastic Processes: With a View Toward Applications”

• We want to allow as much dependence as the Law of Large Numbers (LLN) let us do it

• Stationarity is not enough as the following example shows:

E7: Let {U_t} be a sequence of iid r.v uniformly distributed on [0, 1] and let Z be N(0,1) independent of {U_t}.

Define Y_t=Z+U_t . Then Y_t is stationary (why?), but

The problem is that there is too much dependence in the sequence {Y_t}. In fact the correlation between Y₁ and Y_t is always positive for any value of t.

2 Z 1

Yn

2 ) 1

Yt ( no E

n

1 t

Yt n

n 1 Y





















Ergodicity for the mean Ergodicity for the mean

Need to distinguishing between:

1. Ensemble average 2. Time average

Objective: estimate the mean of the process

Which estimator is the most appropriate? Ensemble average Problem: It is impossible to calculate

Under which circumstances we can use the time average?

m Z z

m i



i



¹

n Z z

n t



t



¹

  ^Z

_t

Is the time average an unbiased and consistent estimator of the mean?

) ( Z

_t

 E



Ergodicity for the mean (cont) Ergodicity for the mean (cont)

Reminder. Sufficient conditions for consistency of an estimator.

0 ˆ )

var(

lim

and

ˆ ) (

lim 

T









 T T

T

E   

1. Time average is asymptotically unbiased



^



^



t t

t n

Z n E

z

E 1 ( ) ¹  

) (

2. Time average is consistent for the mean





























































   

  

 







)]

(

) (

) [(

) (

) , 1 cov(

) var(

0 )

2 ( )

1 (

2 1

0 1

1 1

2 0 0

1

2 2 1

0

1 1

2 0

1 1

2



















 





 

 











n n

n n

n t

n t t

t

n t

n s

s t n

t n s

s t

n n

Z n n Z

z

Ergodicity for the mean (cont) Ergodicity for the mean (cont)

Finite

0

0 k

) k n 1 k

n ( 0 nlim

) z nlim var(

1 n

) 1 n

(

k k

) k n 1 k

n ( k 0

) k n

2 ( n ) 0

z var(









 



 















 





 





 

A covariance-stationary process is ergodic for the mean if





 ( )

lim z E Z

_t

p

A sufficient condition for ergodicity for the mean is

k as

k  0  



Ergodicity under Gaussanity Ergodicity under Gaussanity

If

  ^Z

_t is a stationary Gaussian process,



 

k

k

is sufficient to ensure ergodicity for all moments Ergodicity for second moments Ergodicity for second moments

A sufficient condition to ensure ergodicity for second moments

is

  

k



k

Where are We?

Where are We?

The Prediction Problem as a Motivating Problem:

Predict Z_t+1 given some information set I_t at time t.

The conditional expectation can be modeled in a parametric way or in a non-parametric way. We will choose in this course the former.

Parametric models can be linear or non-linear. We will choose in this course the former way too. Summarizing the models we are going to study and use in this course will be

Parametric and linear models

t ] I 1 | Zt

[ 1 E

Zt : Solution

]2 1 Zt

1 Zt

[ E Min

 



 





Some Problems Some Problems

P1: Let {Z_t} be a sequence of uncorrelated real-valued variables with zero means and unit variances, and define the “moving average”

for constants ___Show that Y is weak stationary and find its autocovariance function P2: Show that a Gaussian process is strongly stationary if and only if it is weakly stationary

P3: Let X be a stationary Gaussian process with zero mean, unit variance, and autocovariance function c. Find the autocovariance functions of the process

i Zt

r 0 i

i

Yt 









} t

3 : ) t ( X 3 {

X and } t

2 : ) t ( X 2 {

X          

Appendix: Transformations Appendix: Transformations

•Goal: To lead to a more manageable process

•Log transformation reduces certain type of

heteroskedasticity. If we assume _t=E(X_t) and V(X_t) = k ²_t, the delta method shows that the variance of the log is roughly constant:

•Differencing eliminates the trend (not very informative about the nature of the trend)

•Differencing + Log = Relative Change

k t )

Z ( 2Var t )

/ 1 ( t )

Z (log(

Var )

Z ( 2Var )

'( f )) Z ( f (

Var      

1 Zt

1 Zt Zt

1 ) Zt

1 Zt Zt

1 log(

1) Zt

Zt log(

1) Zt log(

t ) Z

log( 

 

 

 



 

 

