Simultaneous singular value decomposition

(1)

Contents lists available atScienceDirect

Linear Algebra and its Applications

journal homepage:w w w . e l s e v i e r . c o m / l o c a t e / l a a

Simultaneous singular value decomposition

Takanori Maehara

∗

, Kazuo Murota

Department of Mathematical Informatics, Graduate School of Information Science and Technology, University of Tokyo, Tokyo 113-8656, Japan

A R T I C L E I N F O A B S T R A C T Article history:

Received 5 May 2009 Accepted 4 January 2011 Available online 1 February 2011 Submitted by R.A. Brualdi Keywords:

Singular value decomposition Block-diagonalization Matrix∗-algebra Bimodule Eigenvalue

We consider the following problem: Given a set ofm×nreal (or complex) matricesA1, . . . ,AN, find anm×morthogonal (or uni-tary) matrixPand ann×northogonal (or unitary) matrixQsuch thatP∗A1Q, . . . ,P∗ANQare in a common block-diagonal form with possibly rectangular diagonal blocks. We call this thesimultaneous singular value decomposition(simultaneous SVD). The name is mo-tivated by the fact that the special case withN=1, where a single matrix is given, reduces to the ordinary SVD. With the aid of the theory of∗-algebra and bimodule it is shown that a finest simul-taneous SVD is uniquely determined. An algorithm is proposed for finding the finest simultaneous SVD on the basis of recent algorithms of Murota–Kanno–Kojima–Kojima and Maehara–Murota for simul-taneous block-diagonalization of square matrices under orthogonal (or unitary) similarity.

1. Introduction

Singular value decomposition (SVD) is one of the most fundamental tools in dealing with noisy data. It is useful, for instance, in least squares method, principal component analysis, and matrix approximations. Mathematically, the singular value decomposition of anm

×

nreal matrixAis to transformAto a diagonal matrix, with nonnegative diagonal elements, through a transformation of the formPAQwith anm

×

morthogonal matrixPand ann

×

northogonal matrixQ. Singular value decomposition can also be defined for a complex matrixA, where a unitary transformationP∗AQwith unitary matricesPandQis employed.

In this paper we consider such decompositions for a family of matrices, which we call the simulta-neous singular value decomposition. We distinguish two cases, decompositions over

R

and over

C

:

∗Corresponding author.

(2)

Problem

[

R

]

: Given a set ofm

×

nreal matricesA1

, . . . ,

AN, find anm

×

morthogonal matrixPand ann

×

northogonal matrixQsuch thatPA1Q

, . . . ,

PANQare in a common block-diagonal form. Problem

[

C

]

: Given a set ofm

×

ncomplex matricesA1

, . . . ,

AN, find anm

×

munitary matrixP and ann

×

nunitary matrixQsuch thatP∗A1Q

, . . . ,

P∗ANQare in a common block-diagonal form. Naturally we are interested in a “finest" decomposition having diagonal blocks that cannot be decom-posed further into smaller blocks.

Obviously, the special case withN

=

1, where a single matrix is given, reduces to the ordinary singular value decomposition. In this special case we obtain a (genuine) diagonal matrix, which means that a family of orthogonal one-dimensional subspaces are identified as special directions of impor-tance, and the singular vectors are the bases for these subspaces. For multiple matrices, we cannot hope for simultaneous diagonalization but we look for a common block-diagonal form, where the di-agonal blocks are possibly rectangular matrices. This means that we are to identify a family of mutually orthogonal subspaces characteristic to the given family of matrices. It may be said that the diagonal blocks in our decomposition are higher dimensional extensions of singular values, which are scalars (or 1

×

1 matrices).

This paper shows, with the theory of

∗

-algebra and bimodule, that a finest block-diagonal decompo-sition exists and is uniquely determined. We call this the simultaneous singular value decompodecompo-sition of the given family of matrices. Moreover, structure theorems will be established in both cases (see Theorems2and7). As an immediate corollary of the structure theorems we obtain a necessary and sufficient condition for the simultaneous diagonalization under the transformationPAiQ orP∗AiQ (see Corollaries5and9).

Our construction of simultaneous SVD is a natural extension of the well-known fact that the SVD of a single (real) matrixAcan be constructed from the eigenvalue decompositions ofAAandAA. In place of the eigenvalue decompositions ofAAandAA, we use the Wedderburn-type canonical decom-positions of the

∗

-algebra generated byAiAj (i

,

j

=

1

, . . . ,

N) and the

∗

-algebra generated byAi Aj (i

,

j

=

1

, . . . ,

N). Then using the theoretical framework of bimodule we can derive the desired simul-taneous SVD. In the structure theorems for simulsimul-taneous SVD there is a substantial difference between

R

and

C

, which stems from the difference in the structure theorems of matrix

∗

-algebra over

R

and

C

. An algorithm is proposed for finding the simultaneous SVD. This is built upon recent algorithms of Murota–Kanno–Kojima–Kojima [1] and Maehara–Murota [2–4] for simultaneous block-diagonalization of square matrices, i.e., for finding, given a set of square matricesB1

, . . . ,

BN, an or-thogonal (or unitary) matrixPsuch thatP∗B1P

, . . . ,

P∗BNPare in a common block-diagonal form.

In the literature of semidefinite programming, group representation theory and matrix

∗

-algebra have been attracting research interest as effective tools for exploiting algebraic structures due to sym-metry, sparsity, etc. [1,5–10]. Typically, we are given a family of symmetric (or Hermitian) matrices B1

, . . . ,

BNsuch that eachB

=

Bi is endowed with invariance to a finite groupGin the sense of T

(

g

)

∗BT

(

g

)

=

B

(

g

∈

G

)

with respect to an orthogonal (or unitary) representation T. Then the problem is to find an orthogonal (or unitary) matrixPsuch thatP∗B1P, . . . ,P∗BNPare in the same block-diagonal form. In contrast, the simultaneous SVD of the present paper corresponds to equivari-ance in the sense ofS

(

g

)

∗AT

(

g

)

=

A

(

g

∈

G

)

with respect to orthogonal (or unitary) representations SandT. A standard result in group representation theory affords a canonical decomposition for such matrices. Our contribution is to generalize this by means of bimodule, and also to give an algorithm for the decomposition.

The structure theorems of

∗

-algebras form the foundation of the decomposition method for semi-definite programs. It is hoped that the structure theorems established in this paper trigger a new direction in some area of optimization or data science.

2. Structure theorem for simultaneous SVD over

C

Problem

[

C

]

is considered in this section. As a preliminary the structure theorem of matrix

∗

-algebras is described in Section2.1and the simultaneous SVD is constructed in Section2.2.

(3)

2.1. Matrix

∗

-algebra over

C

We denote byMm,n

=

Mm,n

(

C

)

the set ofm

×

ncomplex matrices, and putMn

=

Mn,n. A subsetT ofMnis said to be a

∗

-subalgebra (or a matrix

∗

-algebra) over

C

ifIn

∈

Tand [A

,

B

∈

T

;

α, β

∈

C

⇒

α

A

+

β

B

,

AB

,

A∗

∈

T]. We say that a matrix

∗

-algebraT is simple ifThas no ideal other than

{

O

}

and T itself, where an ideal ofT means a submoduleIofT such that [A

∈

T

,

B

∈

I

⇒

AB

,

BA

∈

I]. A linear subspaceWof

C

nis said to be invariant with respect toT, orT-invariant, ifAW

⊆

Wfor everyA

∈

T. We say thatT is irreducible if noT-invariant subspace other than

{

0

}

and

C

nexists. Two

∗

-algebrasT1andT2are said to be isomorphic if there exists a

∗

-isomorphism (i.e., a bijection preserving sum, product, conjugate, and scalar product) betweenT1andT2. Note that two isomorphic

∗

-algebras are considered “equal” in the theory of

∗

-algebra.

The following is a standard result in

∗

-algebra (e.g. [11, Chapter X]). Note that for a matrix

∗

-algebra Tand a unitary matrixP, the setP∗TP

= {

P∗AP

:

A

∈

T

}

is another matrix

∗

-algebra isomorphic toT. Theorem 1. LetT be a

∗

-subalgebra ofMn

(

C

).

(A) There exist a unitary matrix Q and simple

∗

-subalgebrasTjofMnˆj

(

C

)

for somen

ˆ

j

(

j

=

1

,

2

, . . . , )

such that Q∗TQ

= {

diag

(S

1

,

S2

, . . . ,

S

)

:

Sj

∈

Tj

(j

=

1

,

2

, . . . , )

}

.

(B) IfT is simple, there exist a unitary matrix P and an irreducible

∗

-subalgebraTofMn¯

(

C

)

for some

¯

n such that P∗TP

= {

diag

(

B

,

B

, . . . ,

B

)

:

B

∈

T

}

. (C) IfT is irreducible,T

=

Mn

(

C

)

.

2.2. Construction of simultaneous SVD over

C

For

∗

-algebrasTL

(

⊆

Mm

(

C

))

andTR

(

⊆

Mn

(

C

))

we call a submoduleAofMm,n

(

C

)

a matrix

(

TL

,

TR

)

-bimodule over

C

if [A

∈

A

,

L

∈

TL

,

R

∈

TR

⇒

LAR

∈

A].

Given a family ofm

×

ncomplex matricesA1

, . . . ,

ANwe consider three algebraic structures: (i) Matrix

∗

-algebraTLgenerated byAiA∗j (i,j

=

1

, . . . ,

N).

(ii) Matrix

∗

-algebraTRgenerated byA∗iAj(i

,

j

=

1

, . . . ,

N). (iii) Matrix

(

TL

,

TR

)

-bimoduleAgenerated byA1

, . . . ,

AN.

Note thatTLandTRare determined byA; that is,TLandTRare

∗

-algebras generated, respectively, by

AA∗andA∗A. It is mentioned that ifAi

=

O(i

=

1

, . . . ,

N), we haveA

= {

O

}

, and then bothTL andTRare

∗

-algebras generated by zero matrices, which means thatTL

=

C

ImandTR

=

C

In, since a

∗

-algebra (in our present definition) always contains the identity matrix. Such a degenerate case needs to be included as it may possibly occur as a result of our decomposition.

The fundamental fact underlying our approach is that decomposing the given matricesA1

, . . . ,

AN by means of a transformation of the formP∗AiQis equivalent to decomposing every elementAofA byP∗AQ. Accordingly we assume that we are given a matrix

(

TL

,

TR

)

-bimoduleA

(

⊆

Mm,n

(

C

))

such thatTLandTRare

∗

-algebras generated, respectively, byAA∗andA∗A. Note that no reference is made to the generatorsA1

, . . . ,

ANin this setting.

The following theorem shows that the simultaneous SVD, i.e., the finest decomposition under P∗A1Q

, . . . ,

P∗ANQcan be constructed from the decompositions of

∗

-algebrasAA∗andA∗Ain the sense of Theorem1. Note that this construction generalizes the construction of the SVD of a single matrixAthrough the eigenvalue decompositions ofAA∗andA∗A.

Theorem 2. LetA

⊆

Mm,n

(

C

)

,A

= {

O

}

, be a matrix

(

TL

,

TR

)

-bimodule over

C

such thatTLandTRare

∗

-algebras generated, respectively, byAA∗andA∗A.

(A) There exist unitary matrices P and Q and a natural number

such that

(4)

Here eachAjis a matrix

(

TLj

,

TRj

)

-bimodule, andTLjandTRjare simple matrix

∗

-algebras generated byAjA∗j andA∗jAj, respectively.

(B) IfTLandTRare simple, there exist unitary matrices P and Q and a natural number

μ

such that P∗TLP

=

Iμ

⊗

TL

,

P∗AQ

=

Iμ

⊗

A

,

Q∗TRQ

=

Iμ

⊗

TR

.

HereAis a matrix

(

TL

,

TR

)

-bimodule, andTLandTRare irreducible matrix

∗

-algebras generated byAA∗andA∗A, respectively.

(C) IfTLandTRare irreducible, there exist unitary matrices P and Q such that P∗TLP

=

Mm

(

C

),

P∗AQ

=

Mm,n

(

C

),

Q∗TRQ

=

Mn

(

C

).

Proof. See Section 4.

Corollary 3. A finest block-diagonal decomposition over

C

of given complex matrices A1

, . . . ,

ANexists and its form is uniquely determined, i.e., the number and the sizes of the blocks are uniquely determined.

Proof. Take any minimal block-diagonalization of given matrices, by which we mean a decomposition with diagonal blocks that cannot be decomposed further. Then, by Theorem2, the

∗

-algebrasTLand

TRgenerated, respectively, byAiA∗j (i

,

j

=

1

, . . . ,

N) and byA∗iAj(i

,

j

=

1

, . . . ,

N), are decomposed accordingly into minimal components, and the number and the sizes of the blocks in these decomposi-tions correspond to those in the block-diagonal decomposition ofA1

, . . . ,

AN. Finally we note that, by the theory of matrix

∗

-algebras, the number and the sizes of the blocks in the minimal decomposition of

∗

-algebra are uniquely determined. This proves the corollary.

Example 4. We illustrate the simultaneous SVD by way of a simple example of two 4

×

8 matrices

A1= ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ 1 −1 −1 −1 −3 3 3 3 1 −1 1 1 13 −13 −7 −7 3 −3 −3 −3 −1 1 1 1 −13 13 7 7 −1 1 −1 −1 ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ , A2= ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ 9 −9 5 5 3 −3 1 1 1 −1 −3 −3 15 −15 −19 −19 −3 3 −1 −1 −9 9 −5 −5 −15 15 19 19 −1 1 3 3 ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ .

Although the theorem is stated for complex matrices, we have chosen real matrices for the sake of presentation and treat them as complex matrices. We have

P∗A1Q=2 √ 2× ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ 1 1 2 2 0 0 0 0 3 3 4 4 0 0 0 0 0 0 0 0 1 1 3 3 0 0 0 0 2 2 4 4 ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ , P∗A2Q=2 √ 2× ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ 5 2 5 2 0 0 0 0 3 5 3 5 0 0 0 0 0 0 0 0 4 7 4 7 0 0 0 0 7 1 7 1 ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦

with suitable unitary matricesPandQ. We have

=

2,

μ

=

1 in Theorem2, and accordingly both P∗A1QandP∗A2Qbelong toM2,4

(

C

)

⊕

M2,4

(

C

)

.

(5)

As an immediate corollary we obtain a necessary and sufficient condition for complex matrices A1

, . . . ,

ANto have the same set of singular vectors in the conventional sense. This means that the diagonal forms in the SVD ofA1

, . . . ,

ANcan be obtained through a single pair of unitary matricesP andQvalid for all the matricesA1

, . . . ,

AN.

Corollary 5. For complex matrices A1

, . . . ,

AN, there exist unitary matrices P and Q such that P∗AiQ

(

i

=

1

, . . . ,

N

)

are diagonal if and only if AiAj∗

(

i

,

j

=

1

, . . . ,

N

)

are all normal and commute with each other, and A∗_iAj

(

i

,

j

=

1

, . . . ,

N

)

are all normal and commute with each other.

Proof. The matricesA1

, . . . ,

ANcan be transformed into a diagonal form if and only if the bimoduleA generated by those matrices can be transformed into a diagonal form (i.e., decomposed into 1

×

1 bi-modules). By the structure theorem (Theorem2), the latter is equivalent to the condition that bothTL andTR, the

∗

-algebras generated, respectively, byAi∗Aj(i

,

j

=

1

, . . . ,

N) and byAiA∗j (i

,

j

=

1

, . . . ,

N), can be transformed into a diagonal form (i.e., decomposed into 1

×

1

∗

-subalgebras). According to the theory of

∗

-algebra,TLcan be transformed into a diagonal form if and only if the set of square matri-cesA∗_iAj(i,j

=

1

, . . . ,

N) can be transformed simultaneously into a diagonal form, whereas a standard result of linear algebra says that a set of square matrices can be transformed simultaneously into a di-agonal form if and only if they are all normal and pairwise commute. Similarly forTR. This proves the corollary.

3. Structure theorem for simultaneous SVD over

R

Problem

[

R

]

is considered in this section. The structure theorem of

∗

-algebras is modified for

R

in Section3.1and the simultaneous SVD over

R

is constructed in Section3.2.

3.1. Matrix

∗

-algebra over

R

Matrix

∗

-algebra over

R

and the associated concepts such as irreducibility are defined similarly as in §2.1, where “unitary” is replaced by “orthogonal.” The structure theorem, however, needs a revision for irreducible components, as stated in Theorem6below (see, e.g. [9,1]).

Let

H

denote the quaternion field, i.e.,

H

= {

a

+

ι

b

+

j

c

+

kd

:

a

,

b

,

c

,

d

∈

R

}

with the multiplication defined as:

ι

=

j

k

= −

k

j

,

j

=

k

ι

= −

ι

k,k

=

ιj

= −

jι

,

ι

2

=

j

2

=

k2

= −

1. We regard

C

as a subset of

H

by identifying

ι

with the imaginary unit in

C

.

We define three types of matrices: the set ofm

×

nreal matricesMm,n

=

Mm,n

(

R

)

, the real representation of complex matricesCm,n

⊂

M2m,2n

(

R

)

defined by

Cm,n

=

⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ C

(

z11

)

· · ·

C

(

z1n

)

... ... ...

C

(

zm1

)

· · ·

C

(

zmn

)

⎤ ⎥ ⎥ ⎥ ⎥ ⎦

:

z11

,

z12

, . . . ,

zmn

∈

C

⎫ ⎪ ⎪ ⎪ ⎪ ⎬ ⎪ ⎪ ⎪ ⎪ ⎭ with C(a

+

ιb)

=

⎡ ⎣a

−

b b a ⎤ ⎦

_,

and the real representation of quaternion matricesHm,n

⊂

M4m,4n

(

R

)

defined by

Hm,n

=

⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ H(h11

)

· · ·

H(h1n

)

... ... ...

H(hm1

)

· · ·

H(hmn

)

⎤ ⎥ ⎥ ⎥ ⎥ ⎦

:

h11

,

h12

, . . . ,

hmn

∈

H

⎫ ⎪ ⎪ ⎪ ⎪ ⎬ ⎪ ⎪ ⎪ ⎪ ⎭

(6)

with H(a

+

ιb

+

j

c

+

kd)

=

⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ a

−

b

−

c

−

d b a

−

d c c d a

−

b d

−

c b a ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦

.

We putMn

=

Mn,n,Cn

=

Cn,n,Hn

=

Hn,nfor notational simplicity.

Theorem 6. LetT be a

∗

-subalgebra ofMn

=

Mn

(

R

)

.

(A) There exist an orthogonal matrix Q and simple

∗

-subalgebrasTj of Mˆnj

(

R

)

for somen

ˆ

j

(

j

=

1

,

2

, . . . , )

such that QTQ

= {

diag

(

S1

,

S2

, . . . ,

S

)

:

Sj

∈

Tj

(

j

=

1

,

2

, . . . , )

}

.

(B) IfT is simple, there exist an orthogonal matrix P and an irreducible

∗

-subalgebraTofMn_¯

(

R

)

for somen such that P

¯

TP

= {

diag

(

B

,

B

, . . . ,

B

)

:

B

∈

T

}

.

(C) IfT is irreducible, there exists an orthogonal matrix P such that PTP

=

Mn

,

Cn/2orHn/4. 3.2. Construction of simultaneous SVD over

R

The simultaneous SVD over

R

can be constructed in parallel with the case over

C

. The result, however, has a significant difference due to the difference between the statements in (C) of Theorems

1and6.

For

∗

-algebrasTL

(

⊆

Mm

(

R

))

andTR

(

⊆

Mn

(

R

))

we call a submoduleAofMm,n

(

R

)

a matrix

(

TL

,

TR

)

-bimodule over

R

if [A

∈

A

,

L

∈

TL

,

R

∈

TR

⇒

LAR

∈

A].

Given a family ofm

×

nreal matricesA1

, . . . ,

ANwe consider three algebraic structures: (i) Matrix

∗

-algebraTLgenerated byAiAj(i

,

j

=

1

, . . . ,

N).

(ii) Matrix

∗

-algebraTRgenerated byAi Aj(i

,

j

=

1

, . . . ,

N). (iii) Matrix

(

TL

,

TR

)

-bimoduleAgenerated byA1

, . . . ,

AN.

Note thatTL andTRare determined fromAas the

∗

-algebras generated, respectively, byAAand

AA. IfAi

=

O(i

=

1

, . . . ,

N), we haveA

= {

O

}

, and thenTL

=

R

ImandTR

=

R

In.

The fundamental fact is, again, that decomposing the given matricesA1

, . . . ,

ANby means of a transformation of the formPAiQ is equivalent to decomposing every element AofAbyPAQ. Accordingly we assume that we are given a matrix

(

TL

,

TR

)

-bimoduleA

(

⊆

Mm,n

(

R

))

such that

TLandTRare

∗

-algebras generated, respectively, byAAandAA.

The following theorem shows that the simultaneous SVD, i.e., the finest decomposition under PA1Q

, . . . ,

PANQ can be constructed from the decompositions of

∗

-algebrasAAandAAas given in Theorem6. Note that this construction generalizes the construction of the SVD of a single matrixAthrough the eigenvalue decompositions ofAAandAA.

Theorem 7. LetA

⊆

Mm,n

(

R

),

A

= {

O

}

, be a matrix

(

TL

,

TR

)-bimodule over

R

such thatTLandTRare

∗

-algebras generated, respectively, byAAandAA.

(A) There exist orthogonal matrices P and Q and a natural number

such that

PTLP

=

TL1

⊕ · · · ⊕

TL

,

PAQ

=

A1

⊕ · · · ⊕

A

,

QTRQ

=

TR1

⊕ · · · ⊕

TR

.

(

TLj

,

TRj

)

∗

-algebras generated byAjAj andAj Aj, respectively.

(B) IfTLandTRare simple, there exist orthogonal matrices P and Q and a natural number

μ

such that PTLP

=

Iμ

⊗

TL

,

PAQ

=

Iμ

⊗

A

,

QTRQ

=

Iμ

⊗

TR

.

(7)

HereAis a matrix

(

TL

,

TR

)

∗

-algebras generated byAAandAA, respectively.

(C) IfTLandTRare irreducible, there exist orthogonal matrices P and Q such that PTLP

=

Dm_ˆ

,

PAQ

=

Dmˆ,nˆ

,

QTRQ

=

Dn_ˆ

.

HereD

=

M,C, orH, and

(

m

ˆ

,

n

ˆ

)

=

(

m

,

n

)

ifD

=

M;

(

m

ˆ

,

n

ˆ

)

=

(

m

/

2

,

n

/

2

)

ifD

=

C; and

(

m

ˆ

,

n

ˆ

)

=

(

m

/

4

,

n

/

4

)

ifD

=

H.

Proof. The proof is given in Section 4.

Corollary 8. A finest block-diagonal decomposition over

R

of given real matrices A1

, . . . ,

ANexists and its form is uniquely determined, i.e., the number and the sizes of the blocks are uniquely determined.

Proof. The proof is similar to that for Corollary3.

As an immediate corollary we obtain a necessary and sufficient condition for real matrices A1

, . . . ,

ANto have the same set of singular vectors in the conventional sense. Compare this with its

C

-version given in Corollary5, where commutativity is explicitly required.

Corollary 9. For real matrices A1

, . . . ,

AN, there exist orthogonal matrices P and Q such that PAiQ

(

i

=

1

, . . . ,

N

)

are diagonal if and only if AiAj , Ai Aj

(

i

,

j

=

1

, . . . ,

N

)

are symmetric matrices.

Proof. Just as in the proof of Corollary5for

C

, real matricesA1

, . . . ,

ANare transformed into a diagonal form if and only if each of the two sets of matricesAiAj (i,j

=

1

, . . . ,

N) andAi Aj(i,j

=

1

, . . . ,

N) are transformed into a diagonal form, whereas a set of real square matrices can be transformed si-multaneously into a diagonal form if and only if they are all symmetric and pairwise commute. Here we note a further property of real matrices, which is not true for complex matrices. In the case of

R

, symmetricity implies commutativity, as follows. WhenAiAj is symmetric by the assumption, we have AiAj

=

(A

iAj

)

=

AjAi . Similarly, we haveAi Aj

=

Aj Ai. Using these relations repeatedly, we obtain

(

AiAj

)(

AkAl

)

=

AiAkAjAl

=

AkAi AlAj

=

(

AkAl

)(

AiAj

)

and

(A

i Aj

)(A

kAl

)

=

Ai AkAj Al

=

AkAiAl Aj

=

(A

kAl

)(A

i Aj

).

Example 10. Here we show an example to demonstrate the difference of irreducible components over

R

and

C

. Consider two 2

×

4 matrices

Aj

=

⎡ ⎣

α

j

−

β

j

γ

j

−

δ

j

β

j

α

j

δ

j

γ

j ⎤ ⎦

_(j

₌

1

,

2

),

where

α

j

, β

j

, γ

j

, δ

j

(

j

=

1

,

2

)

are algebraically independent real numbers. The two matrices are irreducible over

R

, but they can be decomposed into two 1

×

2 blocks over

C

. Indeed, we have

P∗Aj ⎡ ⎣P O O P ⎤ ⎦S

=

⎡ ⎣

α

+

ιβ γ

+

ιδ

0 0 0 0

α

−

ιβ γ

−

ιδ

⎤ ⎦

₍

j

=

1

,

2

)

for P

=

√

1 2 ⎡ ⎣ 1 1

−

ι ι

⎤ ⎦

_,

S

=

⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ 1 0 0 0 0 0 1 0 0 1 0 0 0 0 0 1 ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦

.

(8)

4. Proof of the structure theorems

In this section, we will prove the structure theorems (Theorems2and7). We prove Theorem7for

R

only since the proof of Theorem2for

C

is similar and easier.

We first prove the following lemma, which shows the relation between the block-diagonalization ofAand the block-diagonalizations ofTLandTR. This is an extension of the fact that the ordinary SVD of a matrixAcan be constructed from the eigenvalue decompositions ofAAandAA.

Lemma 11. The following are equivalent:

(1) Adoes not have a nontrivial block-diagonalization. (2) BothTLandTRare irreducible.

Proof. IfAhas a nontrivial block-diagonalization, at least one ofTLorTRhas also a nontrivial block-diagonalization. Suppose, for example, thatAcan be decomposed as

PAQ

=

⎡ ⎣A1 O O A2 ⎤ ⎦

_(A

_∈

_A

_).

Then, sinceTLis generated byAA, the generators ofTLare written asAA(A

,

A

∈

A), where bothPAQ andPAQare decomposed as above. Hence we have

P

(

AA

)

P

=

⎡ ⎣A1A1 O O A2A2 ⎤ ⎦

_.

This implies thatTLhas a nontrivial block-diagonalization.

To prove the converse, we may assume thatTRis reducible; otherwise we transpose all matrices. In this case,TRhas a nontrivial invariant subspaceW

⊂

R

n. LetU

=

span

(

AW

)

⊆

R

m. We take an orthogonal basis forW,W⊥andU,U⊥. Then we claim that for allA

∈

A, we have

PAQ

=

←W→ ←W⊥→ ↑ U A₁ O ↓ ↑ U⊥ O A2 ↓

wherePis an orthogonal basis transformation forUandU⊥, andQ is an orthogonal basis transfor-mation forWandW⊥. (Note that ifU

= {0}

orU⊥

= {0}

, the corresponding part disappears but we still say that such decomposition is nontrivial.) Because of the definition ofU, the lower-left part is clearly zero. To prove that the upper-right part is zero, it is sufficient to checkuAv

=

0 for allv

∈

W⊥ andu

∈

U. By the definition ofU, we haveu

=

jBjwjfor someBj

∈

Aandwj

∈

W. Therefore uAv

=

_jw_jB_j Av. SinceABj

∈

TR, we haveABjwj

∈

W. ThereforeuAv

=

0.

Structure theorem (A).There exist orthogonal matrices P and Q and a natural number

such that PTLP

=

TL1

⊕ · · · ⊕

TL

,

PAQ

=

A1

⊕ · · · ⊕

A

,

QTRQ

=

TR1

⊕ · · · ⊕

TR

.

(

TLj

,

TRj

)

∗

-algebras generated by

(9)

Proof. Take any minimal block-diagonalization ofA, by which we mean a decomposition with diagonal blocks that cannot be decomposed further. Specifically, letA

=

i∈IA(i)be such a decomposition. Then eachA

∈

Ais a block-diagonal matrixA

=

diag

(

A(i)

|

i

∈

I

)

withA(i)

∈

A(i)_{. Then, by} Lemma11,TL

=

AAandTR

=

AAare decomposed accordingly into irreducible components. Then by collecting isomorphic irreducible components, i.e., by partitioning the index setIinto equivalence classes based on isomorphism, we obtain the decomposition in the above form, where

is equal to the number of equivalent classes inI.

Structure theorem (C).IfTLandTRare irreducible, there exist orthogonal matrices P and Q such that PTLP

=

Dmˆ

,

PAQ

=

Dmˆ,nˆ

,

QTRQ

=

Dnˆ

.

HereD

=

M,C, orH, and

(

m

ˆ

,

n

ˆ

)

=

(

m

,

n

)

ifD

=

M;

(

m

ˆ

,

n

ˆ

)

=

(

m

/

2

,

n

/

2

)

ifD

=

C; and

(

m

ˆ

,

n

ˆ

)

=

(

m

/

4

,

n

/

4

)

ifD

=

H.

Proof. By the structure theorem for matrix

∗

-algebras (Theorem6), there exist orthogonal matricesP andQsuch thatPTLP

=

Dm_ˆ andQTRQ

=

D_n_ˆ. Therefore we can assume, without loss of generality,

TL

=

Dmˆ andTR

=

D_ˆ_n.

Letd

=

1, 2 or 4 forD

=

M,CorHrespectively, and letd

=

1, 2 or 4 forD

=

M,CorH respectively. Putm

ˆ

=

m

/

dandn

ˆ

=

n

/

d. We divideA

∈

Aintom

ˆ

× ˆ

nblocks of sized

×

d, whose

(i,

j)block is denotedA_[i,j]. Similarly, we divideL

∈

TLintom

ˆ

× ˆ

mblocks of sized

×

dandR

∈

TRinto

ˆ

n

× ˆ

nblocks of sized

×

d.

SinceTL

=

Dm_ˆ, it contains the matrix, sayELi, of which theith diagonal block isIdand the other blocks areOd. Similarly,TRhas the matrix, sayERj, of which thejth diagonal block isIdand the other blocks areOd. Therefore, for allA

∈

A,Ahas the matrixELiAERj, of which the

(

i

,

j

)

block isA[i,j]and the other blocks areOd,d. Noting thatTLandTRcontain block-wise permutation matrices, we see that for allA

,

A

∈

A,A_[i,j]A_[k,l]

∈

DandA[i,j]A[k,l]

∈

D.

Pick a nonzero matrixA

∈

A, and letA_[i,j]be one of the nonzero blocks ofA. SinceA[i,j]A_[i,j]

∈

D and a symmetric matrix inDis necessarily a scalar matrix, we haveA_[i,j]A_[i,j]

=

α

Idfor some

α >

0. Similarly, we also haveA_[_i_,_j_]A_[i,j]

=

α

Idfor some

α

>

0. These imply thatA[i,j]has full row rank and full column rank. Therefore we haved

=

d, andD

=

Din particular. Note also

α

=

α

.

Next, we construct an orthogonal transformation from the nonzero matrixA

∈

A(chosen above). LetP

=

diag

(

A_[i,j]

, . . . ,

A[i,j]

)/

√

α

, which is an orthogonal matrix. We claim the following equalities:

PTLP

=

Dm_ˆ

,

PA

=

Dmˆ,nˆ

.

The first equality is clear sinceTL

=

Dm_ˆ andP

∈

TL. The second equality can be shown as follows: For allA

∈

A, the

(

k

,

l

)

block ofPAisA_[_i_,_j_]A_[_k_,_l_]

/

√

α

, which is an element ofD. ThereforePA

∈

Dm_ˆ,ˆn, and hencePA

=

Dm_ˆ,nˆ.

Structure theorem (B).IfTLandTRare simple, there exist orthogonal matrices P and Q and a natural number

μ

such that

PTLP

=

Iμ

⊗

TL

,

PAQ

=

Iμ

⊗

A

,

QTRQ

=

Iμ

⊗

TR

.

HereAis a matrix

(

TL

,

TR

)

∗

-algebras generated byAA andAA, respectively.

Proof. It turns out to be convenient to prove the above claim by showing PTLP

=

TL

⊗

Iμ

,

PAQ

=

A

⊗

Iμ

,

QTRQ

=

TR

⊗

Iμ

.

Note thatTL

⊗

IμandIμ

⊗

TL, for example, are connected by permutations of row and columns. The proof goes in a similar way as the proof of structure theorem (C).

(10)

By the structure theorem for matrix

∗

-algebras (Theorem6), there exist orthogonal matricesPand Q such thatPTLP

=

Dm_ˆ

⊗

IμandQTRQ

=

D_ˆn

⊗

Iμ. Therefore we can assume, without loss of generality,TL

=

Dm_ˆ

⊗

IμandTR

=

Dn_ˆ

⊗

Iμ. Note thatDis common in these equalities by structure theorem (C).

Letd

=

1, 2 or 4 forD

=

M,CorHrespectively. Putm

ˆ

=

m

/

d

μ

andn

ˆ

=

n

/

d

μ

. We divideA

∈

A intom

ˆ

× ˆ

nblocks of sized

μ

×

d

μ

, of which the

(

i

,

j

)

block is denotedA_[i,j]. Similarly, we divideL

∈

TL intom

ˆ

× ˆ

mblocks of sizedμ

×

dμandR

∈

TRinton

ˆ

× ˆ

nblocks of sizedμ

×

dμ.

SinceTL

=

Dm_ˆ

⊗

Iμ, it contains the matrix, sayELi, of which theith diagonal block isIdμand the other blocks areOdμ. Similarly, theTRhas the matrix, sayERj, of which thejth diagonal block isIdμ and the other blocks areOdμ. Therefore, for allA

∈

A,Ahas the matrixELiAERj, of which the

(i,

j) block isA_[i,j]and the other blocks areOdμ,dμ. Noting thatTLandTRcontain block-wise permutation matrices, we see that for allA,A

∈

A,A_[i,j]A_[k,l]

∈

D

⊗

IμandA[i,j]A[k,l]

∈

D

⊗

Iμ.

Pick a nonzero matrixA

∈

A, and letA_[i,j]be one of the nonzero blocks ofA. SinceA[i,j]A_[i,j]

∈

D

⊗

Iμ and a symmetric matrix inDis necessarily a scalar matrix, we haveA_[i,j]A_[i,j]

=

α

Idμfor some

α >

0. Similarly, we also haveA_[_i_,_j_]A_[i,j]

=

α

Idμ for some

α

>

0. These imply thatA[i,j]has full row rank and full column rank. Therefore we have

μ

=

μ

. Note also

α

=

α

.

Next, we construct an orthogonal transformation from the nonzero matrixA

∈

A(chosen above). LetP

=

diag

(

A_[i,j]

, . . . ,

A[i,j]

)/

√

α

, which is an orthogonal matrix. We claim the following equalities: PTLP

=

Dmˆ

⊗

Iμ

,

PA

=

Dmˆ,nˆ

⊗

Iμ

.

The first equality is clear sinceTL

=

Dmˆ

⊗

IμandP

∈

TL. The second equality can be shown as follows: For allA

∈

A, the

(

k

,

l

)

block ofPAisA_[_i_,_j_]A_[_k_,_l_]

/

√

α

, which is an element ofD

⊗

I_μ. Therefore PA

∈

Dm_ˆ,nˆ

⊗

Iμ, and hencePA

=

Dmˆ,ˆn

⊗

Iμ.

5. Algorithms

The proofs of the structure theorems (Theorems2and7) for simultaneous SVD are constructive, so that they can readily be turned into algorithms.

In this section, we describe an algorithm for Problem [

R

] only, whereas an algorithm for Prob-lem [

C

] is similar and simpler, and hence omitted. The algorithm assumes subroutines for the de-composition of

∗

-algebras into simple and irreducible components. Such algorithms for

∗

-algebras are indeed available; see Murota–Kanno–Kojima–Kojima [1] and Maehara–Murota [2–4] as well as Eberly–Giesbrecht [12].

The decomposition in Part (A) of Theorem7can be carried out by the following algorithm. Recall that TLis the

∗

-algebra generated byAiAj

(

i

,

j

=

1

, . . . ,

N

)

andTRis generated byAi Aj

(

i

,

j

=

1

, . . . ,

N

)

.

Algorithm 12

Step 1: Find an orthogonal matrixPthat decomposes the

∗

-algebraTLinto simple components as in Theorem6(A). Also find an orthogonal matrixQthat decomposes the

∗

-algebraTR into simple components.

Step 2: Find permutationsΠLandΠRsuch thatΠL

(

PAiQ

)

ΠRfori

=

1

, . . . ,

Nare in the same block-diagonal form, sayA

¯

i1

⊕ · · · ⊕ ¯

Ai.

For eachk

=

1

, . . . ,

,Ak,TLkandTRkare generated byA

¯

ik(i

=

1

, . . . ,

N),A

¯

ikA

¯

jk(i

,

j

=

1

, . . . ,

N), andA

¯

_ikA

¯

jk(i

,

j

=

1

, . . . ,

N), respectively. The validity of this algorithm is guaranteed by the fact that the orthogonal matrix denoted as “Q” in Theorem6(A) for

∗

-algebras is unique up to a permutation of simple components and transformations within simple components.

The decompositions in Parts (B) and (C) of Theorem7can be carried out by the following algorithm, which should be applied to eachAkobtained in Part (A). To simplify notation we omit the subscriptk

(11)

and assume thatAsatisfies the premise in (B) thatTLandTRare simple

∗

-algebras with multiplicity

μ

of irreducible components. We defined

=

1, 2, or 4 according to whetherD

=

M,C, orHin (C).

Algorithm 13

Step 1: Find an orthogonal matrixPthat decomposes the

∗

-algebraTL into irreducible com-ponents as in Theorem6(B). Also find an orthogonal matrixQ that decomposes the

∗

-algebraTRinto irreducible components.

Step 2: Pick a nonzero matrixAifrom among the input matrices, and regard it as ad

μ

×

d

μ

block-matrix. LetBbe one of the nonzero blocks ofAi, whereBism

/(

d

μ)

×

n

/(

d

μ)

ifAi ism

×

n.

Step 3: SetP

=

diag

(

B

,

B

, . . . ,

B

)/

c, wherecis a constant such thatc2I

=

BB.

Step 4: Find permutationsΠLandΠRsuch thatΠL

(

PAi

)

ΠRfori

=

1

, . . . ,

Nare in the same block-diagonal form.

The performance of this algorithm depends strongly on the performance of the subroutines. For the decomposition of

∗

-algebras into simple and irreducible components, the algorithm of Maehara– Murota [3,4] is robust against numerical errors and hence suitable as the subroutine.

Acknowledgments

This work is supported by a Grant-in-Aid for Scientific Research from the Ministry of Education, Culture, Sports, Science and Technology of Japan and by the Global COE “The Research and Training Center for New Development in Mathematics.”

References

[1] K. Murota, Y. Kanno, M. Kojima, S. Kojima, A numerical algorithm for block-diagonal decomposition of matrix∗-algebras with application to semidefinite programming, Japan J. Indust. Appl. Math. 27 (2010) 125–160.

[2] T. Maehara, K. Murota, A numerical algorithm for block-diagonal decomposition of matrix∗-algebras with general irreducible components, Japan J. Indust. Appl. Math. 27 (2010) 263–293.

[3] T. Maehara, K. Murota, Error controlling algorithm for simultaneous block-diagonalization and its application to independent component analysis, JSIAM Lett. 2 (2010) 131–134.

[4] T. Maehara, K. Murota, Algorithm for error-controlled simultaneous block-diagonalization of matrices, Submitted for publication. [5] E. de Klerk, D.V. Pasechnik, A. Schrijver, Reduction of symmetric semidefinite programs using the regular∗-representation,

Math. Program. Ser. B 109 (2007) 613–624.

[6] E. de Klerk, R. Sotirov, Exploiting group symmetry in semidefinite programming relaxations of the quadratic assignment, Math. Program. Ser. A 122 (2010) 225–246.

[7] K. Gatermann, P.A. Parrilo, Symmetry groups, semidefinite programs, and sums of squares, J. Pure Appl. Algebra 192 (2004) 95–128.

[8] Y. Kanno, M. Ohsaki, K. Murota, N. Katoh, Group symmetry in interior-point methods for semidefinite program, Optim. Eng. 2 (2001) 293–320.

[9] M. Kojima, S. Kojima, S. Hara, Linear algebra for semidefinite programming, Research Report B-290, Tokyo Institute of Technology, October 1994; also in RIMS Kokyuroku 1004, Kyoto University, 1997, pp. 1–23.

[10] F. Vallentin, Symmetry in semidefinite programs, Linear Algebra Appl. 430 (2009) 360–369.

[11] J.H.M. Wedderburn, Lectures on Matrices, American Mathematical Society, New York, 1934; Dover, Mineola, NY, 2005. [12] W. Eberly, M. Giesbrecht, Efficient decomposition of separable algebras, J. Symbolic Comput. 37 (2004) 35–81.