Singular Value Decomposition
This is the first in a short series of posts on an application of matrix theory to the mechanics of continuous media. I’m planning my Vector and Tensor Analysis course for the winter quarter, and wanted to include a discussion of stress in elastic media as an application of tensors interesting to the engineering students. Identifying the prerequisite material needed for this topic and to what extent I can expect the students to have been exposed to it through our curriculum has been an interesting exercise. The physical interpretation of the stress tensor (which I will define in a later post) comes through the real polar decomposition of a matrix. The real polar decomposition can be deduced from the singular value decomposition which can be deduced from the real spectral theorem. Here’s how that last step goes.
Theorem. (Singular Value Decomposition) Every real matrix can be factored as where is an $m\times m$ orthogonal matrix, is an matrix with non-negative real numbers on the diagonal (called the singular values of ) and is an orthogonal matrix.
With an understanding of the spectral theorem and how to interpret matrix multiplication one can actually derive a procedure for computing this factorization. The important things to remember are:
- if is an matrix then is a symmetric matrix and is a symmetric $m\times m$ matrix because of the two properties of the transpose that and ;
- what it means for a matrix to be orthogonal is that is square and , i.e., is invertible and ;
- the condition also says that the columns of are orthonormal vectors since the -entry of the product is precisely the dot product of row of (i.e., column of ) with column of ;
- if is a diagonal matrix of size then is the result of multiplying the rows of by the corresponding diagonal entries of , while if is diagonal of size $n\times n$ then is the result of multiplying the columns of by the corresponding entries of .
Let’s suppose for a moment that with , , and as in the statement of the theorem. Then
by the orthogonality of and . Due to the structure of , the matrix is an diagonal matrix whose first diagonal entries are the squares of the diagonal entries of . Likewise, the matrix is an matrix whose first diagonal entries are the squares of the diagonal entries of (in the same order). The matrix equation then says that the columns of are eigenvectors of . Similarly, the equation then says that the columns of are eigenvectors of .
Fortunately, the real spectral theorem guarantees that the eigenvectors of the symmetric matrices and are orthogonal, and thus can be chosen to be orthonormal. If is an normalized eigenvector of with associated eigenvalue (i.e., ) then being the dot product of a vector with itself. Similarly, the eigenvalues of are non-negative. So it is certainly possible that the non-negative diagonal entries of and are the eigenvalues of and , respectively. But, it must be the case that the eigenvalue of is equal to the eigenvalue of for in order for our factorization to work since both must be equal to the square of the diagonal entry of . This is where a clever observation saves the day. The matrix converts eigenvectors of into eigenvectors of ! Indeed, if is an eigenvector of with eigenvalue , then
Hence, if then it is an eigenvalue of with the same eigenvalue ! Of course, if but is non-zero (being an eigenvector of ) then and so the associated eigenvalue of is zero. Likewise, if is an eigenvector of and is non-zero then is an eigenvector of with the same eigenvalue. It follows that at most of the eigenvalues of and are nonzero and, in fact, of them are the same. The square roots of these joint eigenvalues of and are called the singular values of .
The upshot is that we can compute a set of normalized eigenvectors for and order them in descending order of their associated eigenvalues (so that those in the kernel occur at the end) and use them to form the columns of the orthogonal matrix . We can compute a set of normalized eigenvectors for and order them in descending order of their associated eigenvalues to form the columns of the orthogonal matrix . Then, if is the matrix whose non-diagonal entries are zero but with diagonal entries equal to the singular values of (also listed in descending order) we are guaranteed that . This constructs the singular value decomposition of .