SVD和LSI教程（2）：计算奇异值

（1）SVD与LSI教程（1）：理解SVD和LSI

（2）SVD和LSI教程（2）：计算奇异值

（3）SVD与 LSI教程（3）： 计算矩阵的全部奇异值

（4）SVD 与 LSI 教程（4）： LSI计算

（5）SVD 与 LSI教程（5）：LSI关键字研究与协同理论

/**********************作者信息****************/

Dr. E. Garcia

Mi Islita.com

Email | Last Update: 01/07/07

/**********************作者信息****************/

Revisiting Matrix Transposition

In Part I of this tutorial you learned about the fundamental equation of the Singular Value Decomposition algorithm:

Equation 1: A = USV^T

In Equation 1, the columns of U are the eigenvectors of the AA^T matrix and the columns of V are the eigenvectors of the A^TA matrix. V^T is the transpose of V and S is a diagonal matrix. By definition the nondiagonal elements of diagonal matrices are zero. The diagonal elements of Sare a special kind of values of the original matrix. These are termed the singular values of A.

When Professor Gene Golub developed the SVD technique back in 1965 (1), one of his goals was to determine the singular values and pseudo-inverse of a matrix, compute the rank of a matrix by counting the number of nonzero singular values, and to expose hidden properties and features of matrices under SVD (1-4).

Decomposing a matrix with Equation 1 is often referred to as computing its "Full SVD". Evidently, to compute an SVD we need to know about matrix transposition, so let's revisit this subject first.

As mentioned in Matrix Tutorial 1, a transponse matrix A^T is obtained by converting rows into columns and columns into rows. An example is given below.

Figure 1. A matrix and its transpose.

We can use these matrices to construct two new matrices. This is done by premultiplying A by A^T and A^T by A.

Figure 2. "Left" and "right" matrices.

That wasn't that hard. Right? You are now half-way to become an above the average search engine marketer.

A Practical Notation

Note that I have labeled these matrices using a practical left-right notation. This notation is just a mnemonic (memorization aid) based on the relative position of A and as follows:

• AA^T is a left matrix since A is at the left of A^T
• A^TA is a right matrix since A is at the right of A^T

This matrix is also obtained by making the transformation:

Equation 2: (A^TA)^T = A^T(A^T)^T = A^TA

Now for the sake of consistency I'm going to refer to their eigenvectors as the left and right eigenvectors. This left-right labeling system is trivial but very convenient. When plotted in the same space, these eigenvectors end at the left and right of each other. Examples will be provided.

So, what kind of features can be exposed by decomposing a matrix?

The Frobenius Norm

Before proceeding any further, let's introduce an important feature of any matrix: the Frobenius Norm, also known as the Euclidean Norm. The Frobenius Norm of a matrix is defined as the square root of the sum of the absolute squares of its elements.

Essentially, take a matrix, square its elements, add them together and square root the result. The computed number is the Frobenious Norm of the matrix.

Since a column vector is a one-column matrix and a row vector is a one-row matrix, the Frobenius Norm of these matrices equals the length (L) of the vectors. Thus, normalized unit vectors are vectors normalized in terms of their Frobenius Norm.

Great, but how does this relate to matrix decomposition?

Computing A^T, AA^T, and A^TA

To answer this question let's consider the example given in Figure 1 and Figure 2. Defining A as in the figure and computing A^T, AA^T, and A^TA, the following becomes evident.

Figure 3. Some exposed features.

We first note from Figure 3 that:

1. A and A^T have same trace, determinant and Frobenius Norm
2. AA^T and A^TA have same trace, determinant and Frobenius Norm
3. in the absolute sense, the square root of the determinant of AA^T or A^TA equals the determinant of A^T or A^T.

Now if you recall from Matrix Tutorial 3: Eigenvalues and Eigenvectors, we mentioned that a matrix A and its transpose A^T respond to the same characteristic equation and, therefore, have identical eigenvalues. Figure 4 confirms this for the example at hand.

Figure 4. Common characteristic equations and eigenvalues.

Note that the product of the eigenvalues (in the absolute sense) equals the determinant of the corresponding matrix. Note also that A^TA and AA^T have the same characteristic equation and eigenvalues, too. This is not surprising --considering the transformation given in Equation 2. To make this clear, I am providing the full proof below.

Figure 5. Characteristic equation and eigenvalues for AA^T and A^TA.

These eigenvalues, being common to both matrices, are quite unique. So, what do we do with these?

Computing S

In the SVD technique, once eigenvalues from either the left or right matrix are obtained these are ordered in decreasing order (in the absolute sense) like this

| c₁ | > | c₂ | > | c₃ | ... > | c_n |

Then, taking their square roots yields

s₁ > s₂ > s₃ ... > s_n

which are the infamous singular values of the original matrix. These can be zero or nonzero values.

The number of nonzero singular values is defined as the Rank of a Matrix. Before SVD was introduced by Golub and Kahan (1), the old way of computing the Rank of a Matrix consisted in counting the number of rows or columns (whichever the lower number) of the echolon form of the original matrix.

Retaking the example at hand, the singular values of A are

s₁ = (c₁)^1/2 = (40)^1/2 = 6.32...
s₂ = (c₂)^1/2 = (10)^1/2 = 3.16...

Thus, by virtue of Figure 5 we have demonstrated that one can safely compute the singular values of A with either A^TA or AA^T.

Despite the information uncovered from Figure 3, 4 and 5, there is no simple relation between eigenvalues from the A, A^Tpair and the AA^T, A^TA pair. One could ask: which properties or features of AA^T and A^TA can be trace back to A?

Well, let's take a closer look at the singular values. These can be trace back to the original matrix A in three different ways:

1. the products of the singular values equals the absolute determinant of A; i.e., (s₁ )(s₂) = | -20 | (ignoring rounding errors).
2. the Frobenius Norm of A is the square root of the sum of the absolute squares of the singular values. Thus, only nonzero singular values contribute to a Frobenius Norm. In this case, Frobenius Norm = (s₁² + s₂²)^1/2 = (6.32² + 3.16²)^1/2 = 20 = | - 20 | (ignoring rounding errors).
3. this is a matrix of Rank 2 since only two nonzero singular values were obtained, which turned out to be the only singular values of A.

Proceeding with the decomposition, now that we have computed the singular values we construct matrix S by placing values in descending order along its main diagonal, like this:

Figure 6. Singular matrix of A and its singular values, ordered in decreasing order.

Since this is a diagonal matrix, by definition its nondiagonal elements are zero. So now you know how to compute S. In Part III of this tutorial you will learn of an alternate way of computing S, just to convince yourself of the nature of this matrix. You will also learned how to compute U and V^T.

Summary

We have shown that given a matrix A this can be decomposed into several matrices so their features can be exposed. In particular, we have shown that its singular matrix can be computed by the following procedure:

1. compute its transpose A^T and A^TA.
2. determine the eigenvalues of A^TA and sort these in descending order, in the absolute sense. Square roots these to obtain the singular values of A.
3. Construct diagonal matrix S by placing singular values in descending order along its diagonal.

In Part 3 of the tutorial I will show you how to complete the decomposition and reconstruction of A by computing U and V^T. Believe it or not, this is easy to do.

Tutorial Review

Consider the following matrix

1. compute the Frobenius Norm, determinant, characteristic equation, and eigenvalues of A.
2. express AA^T and A^TA.
3. compute the Frobenius Norm, determinant, characteristic equation, and eigenvalues of AA^T and A^TA.
4. calculate singular values and express S.

References

1. Calculating the Singular Values and Pseudo-Inverse of a Matrix, by G. Golub and W. Kahan, J. SIAM, Numer. Anal. Ser. B, p. 205-224, Vol 2, No. 2 (1965).
2. Using Linear Algebra for Intelligent Information Retrieval. SIAM Review 37(4): 573-595. Berry, M., S. Dumais, and G. O'Brien. (1995).
3. The Singular Value Decomposition: Its Computation and Some Applications, V. C. Klema, A. J. Laub, IEEE Transactions on Automatic Control, Vol. AC-25, NO. 2, April (1980).
4. An Improved Algorithm for Computing the Singular Value Decomposition, T. Chan, ACM Transactions on Mathematical Software, Vol 8, No. 1 (1982).