Metas y Fines Posibles

The procedure introduced by Arnoldi in 1951 starts by building an orthogonal basis of the Krylov subspace_Km. In exact arithmetic, one variant of the algorithm

is as follows.

ALGORITHM6.1 Arnoldi

1. Start:Choose a vectorv1of norm 1. 2. Iterate:forj= 1,2, . . . , mcompute:

hij = (Avj, vi), i= 1,2, . . . , j, (6.3) wj =Avj− j X i=1 hijvi, (6.4) hj+1,j =kwjk2, if hj+1,j = 0 stop (6.5) vj+1=wj/hj+1,j. (6.6)

The algorithm will stop if the vectorwj computed in (6.4) vanishes. We will come back to this case shortly. We now prove a few simple but important properties of the algorithm. A first observation is that the algorithm is a form of classical Gram-Schmidt orthogonalization process whereby a new vector (Avj) is formed and then orthogonalized against all previousvi’s. As the next proposition shows, the resulting basis is a basis of the Krylov subspave_Km.

Proposition 6.5 The vectorsv1, v2, . . . , vmform an orthonormal basis of the sub- space_Km= span{v1, Av1, . . . , Am−1v1}.

Proof. The vectors vj, i = 1,2, . . . mare orthonormal by construction. That they span_Kmfollows from the fact that each vectorvjis of the formqj−1(A)v1

whereqj−1is a polynomial of degreej−1. This can be shown by induction on j as follows. Clearly, the result is true whenj = 1, sincev1 = q0(A)v1with

q0(t)≡1. Assume that the result is true for all integers_≤j and considervj+1. We have hj+1vj+1=Avj− j X i=1 hijvi=Aqj−1(A)v1− j X i=1 hijqi−1(A)v1 (6.7)

which shows thatvj+1can be expressed asqj(A)v1whereqj is of degreejand completes the proof.

Proposition 6.6 Denote byVmthen×mmatrix with column vectorsv1, . . . , vm

and byHmthem×mHessenberg matrix whose nonzero entries are defined by the algorithm. Then the following relations hold:

AVm = VmHm+hm+1,mvm+1eHm, (6.8) VH

mAVm = Hm. (6.9)

Proof. The relation (6.8) follows from the following equality which is readily

derived from (6.6) and (6.4):

Avj =

j+1 X i=1

hijvi, j = 1,2, . . . , m . (6.10)

Relation (6.9) follows by multiplying both sides of (6.8) byVH

m and making use

of the orthonormality of_{v1, . . . , vm}.

The situation is illustrated in Figure 6.1. As was noted earlier the algorithm may break down in case the norm of wj vanishes at a certain step j. In this situation the vector vj+1 cannot be computed and the algorithm stops. There remains to determine the conditions under which this situation occurs.

Proposition 6.7 Arnoldi’s algorithm breaks down at stepj(i.e.,wj= 0in (6.4)) if and only if the minimal polynomial ofv1is of degreej. Moreover, in this case the subspace_Kj is invariant and the approximate eigenvalues and eigenvectors

are exact.

Proof. If the degree of the minimal polynomial isj,thenwj must be equal to zero. Indeed, otherwisevj+1 can be defined and as a result_Kj+1 would be of

Vm +wmeH m = A Hm Vm

Figure 6.1: The action ofAonVmgivesVmHmplus a rank one matrix.

which is not true. To prove the converse, assume thatwj = 0. Then the degree

µ of the minimal polynomial ofv1 is such thatµ ≤ j. Moreover, we cannot haveµ < jotherwise by the previous proof the vectorwµwould be zero and the algorithm would have stopped at the earlier step numberµ. The rest of the result follows from Proposition 4.3 seen in Chapter 4.

The approximate eigenvaluesλ(m)i provided by the projection process onto Km are the eigenvalues of the Hessenberg matrix Hm. The Ritz approximate eigenvector associated withλ(m)_i is defined byu(m)_i =Vmy(m)_i wherey(m)_i is an eigenvector associated with the eigenvalueλ(m)_i . A number of the Ritz eigenval- ues, typically a small fraction ofm,will usually constitute good approximations of corresponding eigenvaluesλi ofAand the quality of the approximation will usually improve asmincreases. We will examine these ‘convergence’ properties in detail in later sections. The original algorithm consists of increasingmuntil all desired eigenvalues ofAare found. This is costly both in terms of computation and storage. For storage, we need to keepmvectors of lengthnplus anm×m

Hessenberg matrix, a total of approximatelynm+m2_{/2. Considering the com-}

putational cost of the j-th step, we need to multiplyvjbyA,at the cost of2×N z,

whereN zis number of nonzero elements inA, and then orthogonalize the result againstjvectors at the cost of4(j+ 1)n,which increases with the step numberj. On the practical side it is crucial to be able to estimate the residual norm inexpensively as the algorithm progresses. This turns out to be quite easy to do for Arnoldi’s method and, in fact, for all the Krylov subspace methods described in this chapter. The result is given in the next proposition.

Proposition 6.8 Letyi(m)be an eigenvector ofHmassociated with the eigenvalue λ(m)i andu

(m)

i the Ritz approximate eigenvectoru (m) i =Vmy (m) i . Then, (A−λ(m)i I)u (m) i =hm+1,meHmy (m) i vm+1

and, therefore, k(A−λ(m)i I)u (m) i k2=hm+1,m|eHmy (m) i |.

Proof. This follows from multiplying both sides of (6.8) byyi(m): AVmy(m)i = VmHmy (m) i +hm+1,meHmy (m) i vm+1 = λ(m)_i Vmy_i(m)+hm+1,meHmy (m) i vm+1. Hence, AVmy(m)_i −λ(m)_i Vmy(m)_i =hm+1,meH my (m) i vm+1.

In simpler terms, the proposition states that the residual norm is equal to the last component of the eigenvectory(m)i multiplied byhm+1,m. In practice, the residual

norms, although not always indicative of actual errors, are quite helpful in deriving stopping procedures.

6.2.2

Practical Implementations

The description of the Arnoldi process given earlier assumed exact arithmetic. In reality, much is to be gained by using the Modified Gram-Schmidt or the House- holder algorithm in place of the standard Gram-Schmidt algorithm. With the mod- ified Gram-Schmidt alternative the algorithm takes the following form.

ALGORITHM6.2 Arnoldi - Modified Gram-Schmidt 1. Start.Choose a vectorv1of norm 1.

2. Iterate.Forj= 1,2, . . . , mdo: (a) w:=Avj; (b) Fori= 1,2, . . . , jdo: hij = (w, vi), w:=w−hijvi; (c) hj+1,j =kwk2; (d) vj+1=w/hj+1,j.

There is no difference in exact arithmetic between this algorithm and Al- gorithm 6.1. Although this formulation is numerically superior to the standard Gram Schmidt formulation, we do not mean to imply that the above Modified Gram-Schmidt is sufficient for all cases. In fact there are two alternatives that are implemented to guard against large cancellations during the orthogonalization process.

The first alternative is to resort to double orthogonalization. Whenever the final vector obtained at the end of the second loop in the above algorithm has been computed, a test is performed to compare its norm with the norm of the initialw(which is_kAvjk2). If the reduction falls below a certain threshold, an

indication that sever cancellation might have occurred, a second orthogonalization is made. It is known from a result by Kahan that additional orthogonalizations are superfluous (see for example Parlett [148]).

The second alternative is to resort to a different technique altogether. In fact one of the most reliable orthogonalization techniques, from the numerical point of view, is the Householder algorithm. This has been implemented for the Arnoldi process by Walker [220]. We do not describe the Householder algorithm here but we would like to compare the cost of each of the three versions.

In the table shown below, GS stands for Gram-Schmidt, MGS for Modified Gram-Schmidt, MGSR for Modified Gram-Schmidt with Reorthogonalization, and HO for Householder.

GS MGS MGSR HO Flops m2_{n m}2_{n 2m}2_{n 2m}2_n−2 3m 3 Storage (m+ 1)n (m+ 1)n (m+ 1)n (m+ 1)n−12m 2

A few comments are in order. First, the number of operations shown for MGSR are for the worst case situation when a second orthogonalization is needed every time. This is unlikely to take place and in practice the actual number of operations is much more likely to be close to that of the simple MGS. Concerning storage, the little gain in storage requirement in the Householder version comes from the fact that the Householder transformation requires vectors whose length diminishes by1at every step of the process. However, this difference is negligible relative to the whole storage requirement given that usuallym ≪ n. Moreover, the implementation to take advantage of this little gain may become rather com- plicated. In spite of this we do recommend implementing Householder orthog- onalization for developing general purpose reliable software packages. A little additional cost in arithmetic may be more than offset by the gains in robustness in these conditions.

Example 6.1. Consider the matrix Mark(10) used in the examples in the pre- vious two Chapters. Table 6.1 shows the convergence of the rightmost eigenvalue obtained by Arnoldi’s method. Comparing the results shown in Table 6.1 with those of the examples seen in Chapter 4, it is clear that the convergence is much faster than the power method or the shifted power method.

As was mentioned earlier the standard implementations of Arnoldi’s method are limited by their high storage and computational requirements asmincreases. Suppose that we are interested in only one eigenvalue/eigenvector ofA,namely

m ℜe(λ) ℑm(λ) Res. Norm 5 0.9027159373 0.0 0.316D+00 10 0.9987435899 0.0 0.246D-01 15 0.9993848488 0.0 0.689D-02 20 0.9999863880 0.0 0.160D-03 25 1.000000089 0.0 0.135D-05 30 0.9999999991 0.0 0.831D-08

Table 6.1: Convergence of rightmost eigenvalue computed from a simple Arnoldi algorithm forA= Mark(10).

m ℜe(λ) ℑm(λ) Res. Norm 10 0.9987435899D+00 0.0 0.246D-01 20 0.9999523324D+00 0.0 0.144D-02 30 0.1000000368D+01 0.0 0.221D-04 40 0.1000000025D+01 0.0 0.508D-06 50 0.9999999996D+00 0.0 0.138D-07

Table 6.2: Convergence of rightmost eigenvalue computed from a restarted Arnoldi procedure forA= Mark(10).

the eigenvalue of largest real part ofA. Then one way to circumvent the diffi- culty is to restart the algorithm. After a run withmArnoldi vectors, we compute the approximate eigenvector and use it as an initial vector for the next run with Arnoldi’s method. This process, which is the simplest of this kind, is iterated to convergence.

ALGORITHM6.3 Iterative Arnoldi

1. Start:Choose an initial vectorv1and a dimensionm. 2. Iterate:Performmsteps of Arnoldi’s algorithm.

3. Restart: Compute the approximate eigenvector u(m)1 associated with the rightmost eigenvalueλ(m)1 .

If satisfied stop, else setv1≡u(m)1 and go to 2.

Example 6.2. Consider the same matrix Mark(10) as above. We now use a restarted Arnoldi procedure for computing the eigenvector associated with the eigenvalue with algebraically largest real part. We use m = 10. Comparing

the results of Table 6.2 with those of the previous example indicates a loss in performance, in terms of total number of matrix-vector products. However, the number of vectors used here is 10 as opposed to 50, so the memory requirement is much less.