Applying PCA on MNIST dataset

Implementation

In this section, we will reduce the 784 dimensions of the MNIST dataset to 2 dimensions and plot the corresponding principal components obtained.

Let's start by importing the basic libraries.

import numpy as np 
import pandas as pd
import seaborn as sns

You can also try this code with Online Python Compiler

Run Code

Now let’s load the data into a pandas dataframe. Give the path to the train.csv file in your system for the read_csv function.

data=pd.read_csv("D:/mnist/train.csv") #load the data into a pandas dataframe
data.head(5) #Show the first 5 rows

You can also try this code with Online Python Compiler

Run Code

The output is given below(first five rows)
 

Now let’s drop the label column so that we only have the features i.e the pixel values in our dataset.

label=data['label'] # save label data for later use
data.drop('label', axis = 1, inplace = True)

You can also try this code with Online Python Compiler

Run Code

Before applying PCA, it is essential to standardize our dataset, i.e., the mean should be 0, and the standard deviation should be 1 for each variable. We can scale our data with the help of the StandardScaler method in sklearn.

from sklearn.preprocessing import StandardScaler
data_standardized = StandardScaler().fit_transform(data)

You can also try this code with Online Python Compiler

Run Code

Now we need to compute the covariance matrix, which helps us determine the relationship between the dimensions. The covariance matrix for a matrix can be calculated by multiplying the matrix with its transpose. 

covMatrix = np.matmul(data_standardized.T ,data_standardized) # matrix multiplication in numpy

You can also try this code with Online Python Compiler

Run Code

Now we will compute the eigenvalues and the eigenvector, which would help us determine the principal components. We will use the linear algebra library in scipy(linalg) to compute the eigenvalues.

from scipy.linalg import eigh
values, vector = eigh(covMatrix,eigvals=(782,783))
values

You can also try this code with Online Python Compiler

Run Code

The eigh function returns eigenvalues in ascending order. Therefore if we need the highest two, we have to give the tuple (783,784), which means that values[0] will contain the 2nd principal component and values[1] will contain the 1st principal component. 

We will transpose the eigenvector to make it easy to use it further.

print("The shape before",vector.shape)
vector=vector.T
print("The new shape",vector.shape)

You can also try this code with Online Python Compiler

Run Code

Now we will project the original data on the vector plane formed using vector multiplication.

projectedData = np.matmul(vector, data_standardized.T)

You can also try this code with Online Python Compiler

Run Code

Let’s stack our projected data with labels so that we have our final dataframe ready for visualization.

reducedData = np.vstack((projectedData, label)).T #Stack with labels
reducedData = pd.DataFrame(reducedData, columns = ['pca_1', 'pca_2', 'label'])

You can also try this code with Online Python Compiler

Run Code

Now lets plot the data with the help of seaborn Facetgrid method.

sns.FacetGrid(reducedData, hue = 'label', size = 8) \
  .map(sns.scatterplot, 'pca_1', 'pca_2') \
  .add_legend()

You can also try this code with Online Python Compiler

Run Code

Therefore we have converted our dataset from 784 dimensions to 2 dimensions.

Frequently Asked Questions

1. What is the need for standardization of data in PCA?
   PCA gives more emphasis to variables with high variance. Therefore, if the dimensions are not scaled, we will get inconsistent results. For example, the value for one variable might lie in the range 50-100 and the other one 5-10. In this case, PCA will give more weight to the first variable. Such issues can be resolved by standardizing the dataset before applying PCA
    
2. What is FacetGrid in seaborn?
   FacetGrid is used for plotting conditional relationships. The basic workflow is to initialize the FacetGrid object with the dataset, and the variables used to structure the grid. Then one or more plotting functions can be applied to each subset by calling FacetGrid.map() or FacetGrid.map_dataframe(), and then the other customizations can also be done. I recommend you to look at the documentation for more details.
    
3. How can I plot an image from the MNIST dataset?
   After loading the data, we can easily plot the image with the help of the pixel values. The code snippet is given below.
    

import matplotlib.pyplot as plt
index=1234 #Select a random index
fig_data = np.array(data.iloc[index]).reshape(28,28) #Reshape it into the size 28x28
plt.imshow(fig_data, interpolation = None, cmap = 'gray') #Use imshow function from pyplot to plot the figure
plt.show()
print(label[index])

You can also try this code with Online Python Compiler

Run Code

Key Takeaways

We learned that we could apply PCA on high-dimensional datasets and reduce them to low dimensions. We have also learned how to use python effectively to load our datasets, apply PCA on them, and prepare our data for future machine learning models.