In linear algebra, eigendecomposition (also known as eigenvalue decomposition or EVD) is a factorization of a matrix into a canonical form given by , where is a diagonal matrix containing the eigenvalues of on the diagonal, and is a matrix whose columns are the corresponding eigenvectors of . Only diagonalizable matrices can be factorized in this way. When the matrix being factorized is a normal or real symmetric matrix, the decomposition is called "spectral decomposition", derived from the spectral theorem.
Fundamental theory of matrix eigenvectors and eigenvalues
A nonzero vector of dimension is an eigenvector of a square matrix if it satisfies a linear equation of the form
for some scalar . Then, is called the eigenvalue corresponding to . Geometrically speaking, the eigenvectors of are the vectors that merely elongates or shrinks, and the amount that each of them elongates/shrinks by is the corresponding eigenvalue. The above equation is called the eigenvalue equation or the eigenvalue problem.
This yields an equation for the eigenvalues:
We call this equation the characteristic equation of ; it is an th-order polynomial equation in the unknown . We call the characteristic polynomial of .
The present article only considers algebraically closed fields of scalars. Then, the characteristic equation will have distinct solutions, where . The set of solutions, that is, of eigenvalues, is called the spectrum of .
We can factor as
and the characteristic equation becomes
The integer is termed the algebraic multiplicity of the eigenvalue . The algebraic multiplicities sum to :
For each eigenvalue , we have a specific eigenvalue equation in the unknown :
There will be linearly independent solutions to each eigenvalue equation. The linear combinations of the solutions (except the one which gives the zero vector) are the eigenvectors associated with the eigenvalue . The integer is termed the geometric multiplicity of . It is important to keep in mind that the algebraic multiplicity and geometric multiplicity may or may not be equal, but we always have . The simplest case is of course when . The total number of linearly independent eigenvectors, , can be calculated by summing the geometric multiplicities:
The eigenvectors can be indexed by eigenvalues, using a double index, with being the th eigenvector for the th eigenvalue. The eigenvectors can also be indexed using the simpler notation of a single index , with .
Eigendecomposition of a matrix Let be a square matrix with linearly independent eigenvectors (where ). Then can be factored as
where is the square matrix whose th column is the eigenvector of , and is the diagonal matrix whose diagonal elements are the corresponding eigenvalues, . Note that only diagonalizable matrices can be factorized in this way. For example, the defective matrix (which is a shear matrix) cannot be diagonalized.
The eigenvectors are usually normalized, but they do not have to be. A non-normalized set of eigenvectors can also be used as the columns of . That can be understood by noting that the magnitude of the eigenvectors in gets canceled in the decomposition by the presence of . If one of the eigenvalues has multiple linearly independent eigenvectors (that is, the geometric multiplicity of is greater than ), then the eigenvectors for this eigenvalue can be chosen to be mutually orthogonal. However, if two eigenvectors belong to two different eigenvalues, it may be impossible for them to be orthogonal to each other. For instance, the matrix cannot be diagonalized in an orthogonal basis (see Example below). One special case is when is a normal matrix; then by the spectral theorem, it is always possible to diagonalize in an orthonormal basis .
The decomposition can be derived from the fundamental property of eigenvectors:
The linearly independent eigenvectors with nonzero eigenvalues form a basis (not necessarily orthonormal) for all possible products , for , which is the same as the image (or range) of the corresponding matrix transformation, and also the column space of the matrix . The number of linearly independent eigenvectors with nonzero eigenvalues is equal to the rank of the matrix , and also the dimension of the image (or range) of the corresponding matrix transformation, as well as its column space.
The linearly independent eigenvectors with an eigenvalue of zero form a basis (which can be chosen to be orthonormal) for the null space (also known as the kernel) of the matrix transformation .
Example The real matrix
may be diagonalized as , where is a non-singular matrix to be determined:
Then
for some real numbers and .
Multiplying both sides of the equation on the left by ,
The above equation can be decomposed into two simultaneous equations:
Factoring out the eigenvalues and ,
Letting
gives us a system of two vector equations:
This can be represented by a single vector equation involving two solutions as eigenvalues:
where represents any of the eigenvalues, or , and represents the corresponding eigenvector, or .
Shifting to the left-hand side and factoring out,
Since is non-singular, it is essential that is nonzero. Therefore,
Thus
giving us the solutions of the eigenvalues for the matrix as or . The resulting diagonal matrix from the eigendecomposition of is thus .
Putting the solutions back into the above simultaneous equations,
Solving the equations, we have
Thus the matrix required for the eigendecomposition of is
Finally, the diagonalization of is:
In this example, and , eigenvectors associated with distinct eigenvalues, cannot be orthogonal to each other. Indeed:
which would make singular.
Matrix inverse via eigendecomposition
If a matrix can be eigendecomposed and if none of its eigenvalues are zero, then is invertible and its inverse is given by
If is a symmetric matrix, since is formed from the eigenvectors of , then is guaranteed to be an orthogonal matrix; therefore . Furthermore, because is a diagonal matrix, its inverse is easy to calculate:
Practical implications When eigendecomposition is used on a matrix of measured, real data, the inverse may be less valid when all eigenvalues are used unmodified in the form above. This is because as eigenvalues become relatively small, their contribution to the inversion is large. Those near zero or at the "noise" of the measurement system will have undue influence and could hamper solutions (detection) using the inverse.
Two mitigations have been proposed: truncating small or zero eigenvalues, and extending the lowest reliable eigenvalue to those below it. See also Tikhonov regularization as a statistically motivated but biased method for rolling off eigenvalues as they become dominated by noise.
The first mitigation method is similar to a sparse sample of the original matrix, removing components that are not considered valuable. However, if the solution or detection process is near the noise level, truncating may remove components that influence the desired solution.
The second mitigation extends the eigenvalue so that lower values have much less influence over inversion, but do still contribute, such that solutions near the noise will still be found.
The reliable eigenvalue can be found by assuming that eigenvalues of extremely similar and low value are a good representation of measurement noise (which is assumed low for most systems).
If the eigenvalues are rank-sorted by value, then the reliable eigenvalue can be found by minimization of the Laplacian of the sorted eigenvalues:
where the eigenvalues are subscripted with an to denote being sorted. The position of the minimization is the lowest reliable eigenvalue. In measurement systems, the square root of this reliable eigenvalue is the average noise over the components of the system.
Functional calculus The eigendecomposition allows for much easier computation of power series of matrices. If is given by
then we know that
Because is a diagonal matrix, functions of are very easy to calculate:
The off-diagonal elements of are zero; that is, is also a diagonal matrix. Therefore, calculating reduces to just calculating the function on each of the eigenvalues.
A similar technique works more generally with the holomorphic functional calculus, using
from above. Once again, we find that
Examples
which are examples for the functions . Furthermore, is the matrix exponential.
Decomposition for spectral matrices
Spectral matrices are matrices that possess distinct eigenvalues and a complete set of eigenvectors. This characteristic allows spectral matrices to be fully diagonalizable, meaning they can be decomposed into simpler forms using eigendecomposition. This decomposition process reveals fundamental insights into the matrix's structure and behavior, particularly in fields such as quantum mechanics, signal processing, and numerical analysis.
The spectral theorem states that a complex square matrix is unitarily diagonalizable if and only if it is normal. In the real case, every symmetric matrix has an orthogonal basis of eigenvectors. The form is useful because it reduces many matrix operations including powers and matrix functions, to operations on the diagonal entries of Λ.
Normal matrices A complex-valued square matrix is normal (meaning , where is the conjugate transpose of ) if and only if it can be decomposed as , where is a unitary matrix (meaning ) and is a diagonal matrix. The columns of form an orthonormal basis and are eigenvectors of with corresponding eigenvalues .
For example, consider the normal matrix .
The eigenvalues are and .
Possible normalized eigenvectors corresponding to these eigenvalues are and .
The factorization is , where
, , and .
The verification is .
This example illustrates the process of diagonalizing a normal matrix by finding its eigenvalues and eigenvectors, forming the unitary matrix , the diagonal matrix , and verifying the decomposition.
Real symmetric matrices As a special case, for every real symmetric matrix, the eigenvalues are real and the eigenvectors can be chosen real and orthonormal. Thus a real symmetric matrix can be decomposed as , where is an orthogonal matrix whose columns are the real, orthonormal eigenvectors of , and is a diagonal matrix whose entries are the eigenvalues of .
Diagonalizable matrices Diagonalizable matrices can be decomposed using eigendecomposition, provided they have a full set of linearly independent eigenvectors. They can be expressed as , where is a matrix whose columns are eigenvectors of , and is a diagonal matrix consisting of the corresponding eigenvalues of .
Positive definite matrices Positive definite matrices are matrices for which all eigenvalues are positive. They can be decomposed as using the Cholesky decomposition, where is a lower triangular matrix.
Unitary and Hermitian matrices Orthogonal matrices satisfy (real case) and unitary matrices satisfy (complex case), where denotes the transpose of and denotes the conjugate transpose of . They diagonalize using unitary transformations.
Hermitian matrices satisfy , where denotes the conjugate transpose of . They can be diagonalized using unitary or orthogonal matrices.
Useful facts
Useful facts regarding eigenvalues The product of the eigenvalues is equal to the determinant of : Note that each eigenvalue is raised to the power , its algebraic multiplicity. The sum of the eigenvalues is equal to the trace of : Note that each eigenvalue is multiplied by , its algebraic multiplicity. If the eigenvalues of are , and is invertible, then the eigenvalues of are simply . If the eigenvalues of are , then the eigenvalues of are simply , for any holomorphic function and any for which is well-defined.
Useful facts regarding eigenvectors If is a Hermitian matrix and has full-rank, then the basis of eigenvectors may be chosen to be mutually orthogonal. The eigenvalues are real.