Code360 powered by Coding Ninjas X Naukri.com. Code360 powered by Coding Ninjas X Naukri.com
Table of contents
1.
Introduction
2.
Singular Value Decomposition
3.
Eigen Value Decomposition
4.
Difference between singular value decomposition and eigenvalue decomposition
5.
Mathematics behind singular value decomposition and eigenvalue decomposition
6.
Frequently Asked Questions
6.1.
Is SVD applicable to all matrices?
6.2.
Is SVD persistent in all matrices? 
6.3.
Is SVD essential for persistent matrix reduction? 
6.4.
What happens if the eigenvalues are nearly equal?
7.
Conclusion
Last Updated: Mar 27, 2024
Easy

Singular Value Decomposition and Eigenvalue Decomposition

Author Tashmit
0 upvote
Master Python: Predicting weather forecasts
Speaker
Ashwin Goyal
Product Manager @

Introduction

Singular value decomposition, also known as SVD, is the most vital concept of linear algebra applied in Unsupervised machine learning. On the other hand, Eigenvalue decomposition is the factorization of a matrix into canonical form. In this article, we will be studying the difference between the two. 

Singular Value Decomposition

The singular value decomposition is the process of factorizing a matrix A into the product of three matrices A = UDVT. The column of U and V is orthonormal, and the matrix D is diagonal with accurate positive entries, known as singular value decomposition. The decomposition is valid for a variety of activities.

First, data matrix A is often near a low-rank matrix, and it is essential to locate a low-rank matrix that is a reasonable approximation to the data matrix. We can extract the matrix B of rank k that best approximates A in the singular value decomposition; we can do this for any rank k.

Source: Link

Get the tech career you deserve, faster!
Connect with our expert counsellors to understand how to hack your way to success
User rating 4.7/5
1:1 doubt support
95% placement record
Akash Pal
Senior Software Engineer
326% Hike After Job Bootcamp
Himanshu Gusain
Programmer Analyst
32 LPA After Job Bootcamp
After Job
Bootcamp

Eigen Value Decomposition

Eigenvalue decomposition is a form of matrix decomposition. When we wish to calculate a product of matrices equal to the initial matrix, it is known as decomposing a matrix. If we decompose the initial matrix into the product of its eigenvalues and eigenvectors, it is eigenvalue decomposition. Decomposing a matrix regarding eigenvalues and its eigenvectors gives valuable insights into the properties of the matrix. Specific matrix calculations like computing the power of the matrix become much easier when we use the eigendecomposition of the matrix.

Source: Link

Difference between singular value decomposition and eigenvalue decomposition

Let us understand the difference with the help of an example.

Let's consider the following eigendecomposition A=PDP−1 and SVD A=UΣV* 

The difference between them is:

  • In the eigenvalue decomposition matrix P, the vectors are not certainly orthogonal. Therefore, the change of basis isn't a simple rotation. Besides the vectors in the matrices U and V in the SVD are orthonormal, representing rotations (and possibly flips).
  • The non-diagonal matrices in Singular value decomposition, i.e., U and V, are not necessarily the inverse of one another. They are generally not related to each other at all. In the eigenvalue decomposition, the non-diagonal matrices P and P-1 are inverse.
  • In singular value decomposition, the entries are non-negative and natural in the summation of the diagonal matrix.
  • Most of the time, the Singular Value Decomposition exists for any rectangular or square matrix. On the other hand, the eigenvalue decomposition can only exist for square matrices, and even among square matrices, sometimes it doesn't exist.

Mathematics behind singular value decomposition and eigenvalue decomposition

For Eigenvalue decomposition:

Let’s say we’re given a non-defective square matrix A, which has a set of n linearly independent eigenvectors. A can be represented as the diagonal matrix Λ if we express domain and range on the basis of eigenvectors:

A = XΛX-1

The expression can now be represented in the form Ax= b

b’ = Λx’

Where,

B’ is X-1

X’ is X-1x

Now, for Singular value decomposition:

As long as we use the correct domain and range spaces, the decomposition of the Singular value generalizes the concept of expressing a matrix as a diagonal matrix to any arbitrary matrix. Having a look at the Ax = b example again, we can now define b based on the left singular vector:

B’ = U*b

And x, i.e., columns of V based on right singular vectors.

X’ = V*x

By solving,

b= Ax⟶U*b = U*Ax =U*U∑V*x⟶b’ = ∑x’

Reducing this,

∑x’ = b’

The mathematical difference between singular value decomposition and eigenvalue decomposition are:

  • The eigenvalue decomposition utilizes only the basis, which means the eigenvectors. On the other hand, the singular value decomposition uses two different bases, i.e., left and right singular vectors. 
  • In SVD, the eigenbasis is orthonormal, and the basis of eigenvalue decomposition is not orthogonal.
  • Every (!) matrix has an SVD; it does not need to be square or fulfil other requirements. On the other hand, not even every square matrix has an eigendecomposition, which is a fundamental difference that makes the SVD very powerful.

Frequently Asked Questions

Is SVD applicable to all matrices?

Unlike the more often used spectral decomposition in Linear Algebra, singular value decomposition is specified for all matrices (rectangular or square). 

Is SVD persistent in all matrices? 

The SVD is always true for every rectangular or square matrix, but the eigendecomposition is only valid for square matrices; it isn't always true among even square matrices. 

Is SVD essential for persistent matrix reduction? 

Significant is the fraction of a vector's linear transformation corresponding to a substantial single value. Positive SVD can be utilized to establish the productive rank of matrix A by counting smaller values as zeros, in addition to what you mention in your query. 

What happens if the eigenvalues are nearly equal?

PCA will not choose the principal components if all eigenvectors are the same because all main features will be similar.

Conclusion

In this article, we have extensively discussed Singular Value Decomposition and EigenValue Decomposition and its difference. We hope that this blog has helped you enhance your knowledge regarding Singular Value Decomposition and EigenValue Decomposition and if you would like to learn more, check out our articles here. Do upvote our blog to help other ninjas grow. Happy Coding!

Previous article
Singular Value Decomposition
Next article
What is Neural Networks?
Live masterclass