Code360 powered by Coding Ninjas X Naukri.com. Code360 powered by Coding Ninjas X Naukri.com
Table of contents
1.
Introduction
2.
Enter Visualizations
2.1.
Visualizing Intermediate Layer Activations
2.2.
Visualizing Convnet Filters
2.3.
Visualizing Heatmaps of class activations
3.
Visualizing Convolutional Layers
4.
Pre-fit VGG Model
5.
How to Visualize Filters
6.
How to Visualize Feature Maps
7.
Frequently Asked Questions
7.1.
What do convolutional neural networks' filters do?
7.2.
How do convolutional neural networks learn the filters?
7.3.
How many filters does a convolutional neural network have?
7.4.
Why do we employ several filters in convolutional neural networks?
8.
Conclusion
Last Updated: Mar 27, 2024

Visualizing Convolutional Neural Networks with Filters

Author Aditi
0 upvote
Master Python: Predicting weather forecasts
Speaker
Ashwin Goyal
Product Manager @

Introduction

Deep learning neural networks tend to be opaque; thus, even if they can make accurate predictions, it is often unclear how or why a particular prediction was formed. Convolutional neural networks keep the spatial relationships for what the model learns because its internal structures are made to operate on two-dimensional picture data. In particular, the model's two-dimensional filters may be examined and visualized to determine the features it will detect, and convolutional layer activation maps can be reviewed to see precisely which features were recognized for a specific input image.

In this article, you will learn how to create straightforward visualizations for filters and feature maps in a convolutional neural network.

Also Read About, Resnet 50 Architecture

Enter Visualizations

Whether it's a tree-based model or a sizable neural network, visualizing the output of your machine learning model is a terrific method to understand how it's progressing. Most people only consider the training error (accuracy) and validation error while training deep networks (accuracy). There is so much more that we can view and thus learn about network architecture when it comes to deep CNN networks like Inception. Judging these two aspects does offer us a sense of how our network is functioning at each epoch.

Visualizing Intermediate Layer Activations

We need to understand how our deep convolutional neural network model perceives the input image by examining the output of its intermediate layers to comprehend how our model can classify the input image. By doing this, we can discover more about how these layers function.

Some filters function as edge detectors, while others focus on a specific area of the flower, such as the center, and yet others serve as background detectors. Since the pattern caught by the convolution kernel becomes sparser as you go deeper, it is simpler to observe this behavior of convolution layers in the initial layers. However, it is possible that such patterns may not even exist in your image, in which case they would not be collected.

Visualizing Convnet Filters

Visualizing the convolution layer filters is another approach to understanding what your convolution network is looking for in the photos. By presenting the network layer filters, you may discover the pattern that each filter will respond to. Starting with a blank input image, this can be accomplished by applying Gradient Descent on the value of a convnet to maximize the response of a particular filter.

Visualizing Heatmaps of class activations

When predicting the class labels for photos, your model may occasionally predict the incorrect label for your class, meaning the probability of the correct label is not always at its highest. When this happens, it will be beneficial if you can see what parts of the image your convnet is analyzing to get the class labels.

Class Activation Map (CAM) visualization is the umbrella term for this class of methods. Making heatmaps of class activations across input photos is one of the CAM approaches. A class activation heatmap is a 2D grid of scores computed for each place in an input image and assigned to a specific output class that shows how significant each location is concerning that output class.

Get the tech career you deserve, faster!
Connect with our expert counsellors to understand how to hack your way to success
User rating 4.7/5
1:1 doubt support
95% placement record
Akash Pal
Senior Software Engineer
326% Hike After Job Bootcamp
Himanshu Gusain
Programmer Analyst
32 LPA After Job Bootcamp
After Job
Bootcamp

Visualizing Convolutional Layers

Opaque is a standard description of neural network models. This indicates that they do poorly elaborates on the rationale behind a given judgment or prediction. Convolutional neural networks are intended to process picture input, and given that, they should be easier to understand than other varieties of neural networks. The models are made up of tiny linear filters and the output of applying filters known as activation maps, or more generally, feature maps. Visualizing filters and feature maps is possible. For example, we can create and comprehend tiny filters like line detectors. A learned convolutional neural network's filters may be visualized to help explain the model's operation.

The feature maps produced by applying filters to input images and feature maps produced by earlier layers may shed light on the internal representation of a particular input that the model possesses at a certain stage. In this article, we will examine both of these methods for convolutional neural network visualization.

Pre-fit VGG Model

For visualization, we require a model. We can employ a pre-fit previous state-of-the-art image classification model rather than creating a model from the start.

For the ILSVRC (stands for ImageNet Large Scale Visual Recognition Challenge), several examples of effective picture classification models created by various research teams are provided by Keras. The VGG-16 model, which won the 2014 competition, is one example. Because it has a straightforward uniform structure of sequentially ordered convolutional and pooling layers, is deep (16 learned layers), and performs well, this model is a strong choice for visualization because it will capture valuable information in the filters and feature maps that result.

# load vgg model
from keras.applications.vgg16 import VGG16
# load the model
model = VGG16()
# summarize the model
model.summary()

When we run the example, the model weights are loaded into memory, and a summary of the loaded model is printed. The weights will be obtained from the internet and saved in your home directory if this is your first time loading the model. These files weigh about 500 megabytes and, depending on your internet connection speed, can take a while to download.

The layers' names are clear, they are arranged into blocks, and names for each block's layers use integer indices. We can utilize the pre-fit model we have created as the foundation for visualizations at this point.

How to Visualize Filters

Plotting the learned filters is maybe the most straightforward visualization to complete. The learned filters are only weights in the context of neural networks. Still, because of their unique two-dimensional structure, the weight values have a spatial relationship, making it worthwhile to display each filter as a two-dimensional image (or could be). The first step is reviewing the model's filters to determine what we have to work with.

The output shape of each layer, such as the shape of the generating feature maps, is summarized in the model summary written in the preceding section. Only the total number of weights per layer is shown, with no indication of the form of the filters (weights) in the network. The model provides access to all of the model's layers characteristics of layers.

Convolutional layers have naming conventions like block# conv#, where the '#' is an integer, and each layer has a layer.name property. As a result, we may examine each layer's name and disregard any that do not contain the word "conv."

# summarize filter shapes
for layer in model.layers:
    # check for convolutional layer
    if 'conv' not in layer.name:
        continue

The weights in each convolutional layer come in two groups. The block of filters is one, and the block of bias values is another. Through the layer.get weights() function, you can access these. These weights are retrievable, and we can summarize their shape after that.

# get filter weights
filters, biases = layer.get_weights()
print(layer.name, filters.shape)

 

Summarizing the model filters complete example is provided below.

# summarize filters in each convolutional layer
from keras.applications.vgg16 import VGG16
from matplotlib import pyplot
# load the model
model = VGG16()
# summarize filter shapes
for layer in model.layers:
    # check for convolutional layer
    if 'conv' not in layer.name:
        continue
    # get filter weights
    filters, biases = layer.get_weights()
    print(layer.name, filters.shape)

When the example is run, a list of layer specifics is printed, including the layer name and the form of the layer's filters.

block1_conv1 (3, 3, 3, 64)
block1_conv2 (3, 3, 64, 64)
block2_conv1 (3, 3, 64, 128)
block2_conv2 (3, 3, 128, 128)
block3_conv1 (3, 3, 128, 256)
block3_conv2 (3, 3, 256, 256)
block3_conv3 (3, 3, 256, 256)
block4_conv1 (3, 3, 256, 512)
block4_conv2 (3, 3, 512, 512)
block4_conv3 (3, 3, 512, 512)
block5_conv1 (3, 3, 512, 512)
block5_conv2 (3, 3, 512, 512)
block5_conv3 (3, 3, 512, 512)

 

We can see that 33 filters, which are compact and potentially simple to understand, are used by all convolutional layers. A convolutional neural network's architectural issue is that a filter's depth must coincide with the filter's input depth (e.g., the number of channels). We can see that each filter has a depth of three for the input image, with three channels for red, green, and blue (here, we are working with a channel-last format). The first layer's filters are retrievable as follows:

# retrieve weights from the second hidden layer
filters, biases = model.layers[1].get_weights()

The weight values will probably have modest positive and negative values near 0.0. To make them simpler to visualize, we can normalize their values to the range 0–1.

# normalize filter values to 0-1 so we can visualize them
f_min, f_max = filters.min(), filters.max()
filters = (filters - f_min) / (f_max - f_min)

 

The first six filters out of the total 64 in the block may now be listed, and each filter's three channels can be plotted. Each filter is plotted using the matplotlib library as a new row of subplots and a new column for each filter channel or depth.

# plot first few filters
n_filters, ix = 6, 1
for i in range(n_filters):
    # get the filter
    f = filters[:, :, :, i]
    # plot each channel separately
    for j in range(3):
        # specify subplot and turn of axis
        ax = pyplot.subplot(n_filters, 3, ix)
        ax.set_xticks([])
        ax.set_yticks([])
        # plot filter channel in grayscale
        pyplot.imshow(f[:, :, j], cmap='gray')
        ix += 1
# show the figure
pyplot.show()

 

The VGG16 model's full example of charting the first six filters from the hidden convolutional layer is shown below.

# cannot easily visualize filters lower down
from keras.applications.vgg16 import VGG16
from matplotlib import pyplot
model = VGG16()
filters, biases = model.layers[1].get_weights()

f_min, f_max = filters.min(), filters.max()
filters = (filters - f_min) / (f_max - f_min)
n_filters, ix = 6, 1
for i in range(n_filters):
    # get the filter
    f = filters[:, :, :, i]
    # plot each channel separately
    for j in range(3):
        # specify subplot and turn of axis
        ax = pyplot.subplot(n_filters, 3, ix)
        ax.set_xticks([])
        ax.set_yticks([])
        # plot filter channel in grayscale
        pyplot.imshow(f[:, :, j], cmap='gray')
        ix += 1
# show the figure
pyplot.show()

 

When the example is run, a figure is produced that contains six rows of three images, one column for each channel and one row for each filter. In some instances, we can see that (the first row), the filter is the same across all of the channels, whereas, in other cases, the filters vary (the last row). Small or inhibitory weights are shown by dark squares, whereas light squares show big or excitatory weights. This understanding allows us to observe that the first row's filters pick up on a gradient from light in the top left to dark in the bottom right. Source

gradient image

Despite having a visualization, we can only make out the first six of the first convolutional layer's 64 filters. It is possible to see all 64 filters in a single photograph. Unfortunately, this does not scale. If we want to start looking at filters in the second convolutional layer, we can see that there are once again 64 filters, but each has 64 channels to match the input feature maps. It may be difficult to discern any detail in (6464) 4,096 subplots needed to view all 64 channels in a row for all 64 filters.

How to Visualize Feature Maps

The activation maps, also known as feature maps, record what happens when the filters are applied to an input, either the input image or another feature map. Understanding what input features are recognized or preserved in the feature maps is the goal of seeing a feature map for a particular input image. In contrast to feature maps close to the model's output, which would be expected to capture more generic features, feature maps close to the input would likely identify small or fine-grained detail.

We require input for the VGG16 model that may be used to generate activations so that we can investigate the visualization of feature maps. We'll make use of a clear image of a bird.

Bird Image(Robin by Chris Heald)

Bird Image(Robin by Chris Heald)

Next, to extract the correct layer output, we need a better understanding of the shape of the feature maps produced by each convolutional layer and the layer index number. The sample below will list every layer in the model, print the size of each convolutional layer's output or feature map, and display the layer index.

# summarize feature map size for each conv layer
from matplotlib import pyplot
from keras.applications.vgg16 import VGG16

model = VGG16()

for i in range(len(model.layers)):
    layer = model.layers[i]
    # check for convolutional layer
    if 'conv' not in layer.name:
        continue
    # summarize output shape
    print(i, layer.name, layer.output.shape)

 

When we run the example, we only observe the convolutional layers and not the other output forms that we saw in the model summary.

1 block1_conv1 (?, 224, 224, 64)
2 block1_conv2 (?, 224, 224, 64)
4 block2_conv1 (?, 112, 112, 128)
5 block2_conv2 (?, 112, 112, 128)
7 block3_conv1 (?, 56, 56, 256)
8 block3_conv2 (?, 56, 56, 256)
9 block3_conv3 (?, 56, 56, 256)
11 block4_conv1 (?, 28, 28, 512)
12 block4_conv2 (?, 28, 28, 512)
13 block4_conv3 (?, 28, 28, 512)
15 block5_conv1 (?, 14, 14, 512)
16 block5_conv2 (?, 14, 14, 512)
17 block5_conv3 (?, 14, 14, 512)

 

Using this knowledge, we can create a new model that has only a portion of the layers included in the entire VGG16 model. The output of a particular convolutional layer, which we know would be the activation of the layer or the feature map, would be different from the model's input layer, which would remain the same as in the original model.

A new model that generates a feature map from the first convolutional layer (index 1), for instance, can be defined as follows after loading the VGG model.

# redefine model to output 
model = Model(inputs=model.inputs, outputs=model.layers[1].output)

 

The feature map for the first convolutional layer for a particular input image will be provided when making a prediction using this model. Implementing this now.

 After defining the model, we must load the bird image at the 224X224 resolution that the model requires.

The image PIL object must then be expanded from a 3D array to a 4D array with the dimensions [samples, rows, cols, channels], where we only have one sample, then transformed to a NumPy array of pixel data.

# convert the image to an array
img = img_to_array(img)
# expand dimensions 
img = expand_dims(img, axis=0)

 

After that, the pixel values must be scaled properly for the VGG model. Now is the time to obtain the feature map. Calling the model makes it simple for us to do this. Using the predict() function with the provided single image as input.

# prepare the image 
img = preprocess_input(img)
# get a feature map for the first hidden layer
feature_maps = model.predict(img)

We are aware that the outcome will be a 224x224x64 feature map. All 64 two-dimensional images can be plotted as an 8 by 8 square.

# plot all 64 maps in 8x8 
square = 8
ix = 1
for _ in range(square):
    for _ in range(square):
        ax = pyplot.subplot(square, square, ix)
        ax.set_xticks([])
        ax.set_yticks([])
        pyplot.imshow(feature_maps[0, :, :, ix-1], cmap='gray')
        ix += 1
pyplot.show()

 

The entire code example for viewing the feature map for the first convolutional layer in the VGG16 model for an input image of a bird is provided below.

# plot feature map of 
#first conv layer for a given image
from keras.applications.vgg16 import preprocess_input
from keras.applications.vgg16 import VGG16
from keras.preprocessing.image import img_to_array
from keras.preprocessing.image import load_img
from matplotlib import pyplot
from keras.models import Model
from 
numpy import expand_dims
model = VGG16()

model = Model(inputs=model.inputs, outputs=model.layers[1].output)
model.summary()
img = load_img('bird.jpg', target_size=(224, 224))

img = img_to_array(img)
# expand dimensions 
img = expand_dims(img, axis=0)
# prepare the image
img = preprocess_input(img)
# get a feature map for the first hidden layer
feature_maps = model.predict(img)
# plot all 64 maps in 8x8 
square = 8
ix = 1
for _ in range(square):
    for _ in range(square):
        # specify subplot and turn of axis
        ax = pyplot.subplot(square, square, ix)
        ax.set_xticks([])
        ax.set_yticks([])
        pyplot.imshow(feature_maps[0, :, :, ix-1], cmap='gray')
        ix += 1
pyplot.show()

 

Running the example first provides an overview of the new, more compact model that processes an image to produce a feature map. Remember that while being considerably smaller than the VGG16 model, this one employs the same weights (filters) in the first convolutional layer. The 64 feature maps are then represented as subplots in a figure. We can observe that a variety of variations of the bird image with various features highlighted are produced as a result of applying the filters in the first convolutional layer. Source

bird image with various features

This is an intriguing outcome that mostly meets our expectations. The example could be updated to plot the feature maps produced by the output of further particular convolutional layers. Another strategy would be to make images of all the feature maps that were output from each block of the model in a single run. The image has a pooling layer at the end of five main blocks (block1, block2, etc.). The last convolutional layer in each block has the layer indices [2, 5, 9, 13, 17]. 
 

A list of feature maps will be produced after making a prediction using this new model. We are aware that deeper layers include considerably more feature maps than 64, like 256 or 512, such as depth or the number of channels. However, for consistency, we can limit the number of feature maps displayed to 64.

# plot the output from each block
square = 8
for fmap in feature_maps:
    # plot all 64 maps in 8x8 squares
    ix = 1
    for _ in range(square):
        for _ in range(square):
            # specify subplot and turn of axis
            ax = pyplot.subplot(square, square, ix)
            ax.set_xticks([])
            ax.set_yticks([])
            # plot filter channel in grayscale
            pyplot.imshow(fmap[0, :, :, ix-1], cmap='gray')
            ix += 1
    # show the figure
    pyplot.show()

 

By combining these modifications, we are now able to produce five distinct plots—one for each of the five blocks in the VGG16 model—for our bird shot. Below is a list of everything in it.

# visualize feature maps output 
# from each block in the vgg model
from keras.applications.vgg16 import VGG16
from keras.preprocessing.image import load_img
from keras.applications.vgg16 import preprocess_input
from keras.models import Model
from keras.preprocessing.image import img_to_array
from matplotlib import 
pyplot
from keras.models import Model
from numpy import expand_dims
model = VGG16()
# redefine model to output 
ixs = [2, 5, 9, 13, 17]
outputs = [model.layers[i].output for i in ixs]
model = Model(inputs=model.inputs, outputs=outputs)
img = load_img('bird.jpg', target_size=(224, 224))
img = img_to_array(img)
# expand dimensions 
img = expand_dims(img, axis=0)
# prepare the image 
img = preprocess_input(img)
# get a feature map for the first hidden layer
feature_maps = model.predict(img)
# plot the output from each block
square = 8
for fmap in feature_maps:
    # plot all 64 maps in 8x8 squares
    ix = 1
    for _ in range(square):
        for _ in range(square):
            # specify subplot and turn of axis
            ax = pyplot.subplot(square, square, ix)
            ax.set_xticks([])
            ax.set_yticks([])
            # plot filter channel in grayscale
            pyplot.imshow(fmap[0, :, :, ix-1], cmap='gray')
            ix += 1
    # show the figure
    pyplot.show()

 

Five plots displaying the feature maps from the VGG16 model's five main blocks are produced after running the example. We can observe that as we move deeper into the model, the feature maps display less and less detail, whereas those closer to the model's input capture a lot of fine detail in the image. As the model abstracts the features from the image into broader ideas that may be utilized to generate a classification, this pattern was to be expected. We typically lose the ability to decipher these deeper feature maps, even when it is not immediately apparent from the final image that the model spotted a bird. Source

Block 1 - Visualization of the Feature Maps Extracted

Block 1 - Visualization of the Feature Maps Extracted

Block 2 - Visualization of the Feature Maps Extracted

Block 2 - Visualization of the Feature Maps Extracted

Block 3 - Visualization of the Feature Maps Extracted

Block 3 - Visualization of the Feature Maps Extracted

Block 4 - Visualization of the Feature Maps Extracted

Block 4 - Visualization of the Feature Maps Extracted

Block 5 - Visualization of the Feature Maps Extracted

Block 5 - Visualization of the Feature Maps Extracted

Frequently Asked Questions

What do convolutional neural networks' filters do?

The function of a filter is to serve as a single template or pattern that, when convolved across the input, identifies correspondences between the stored template and various locations/regions in the input image.

How do convolutional neural networks learn the filters?

This is achieved by being familiar with the various levels of visual abstraction. The CNN often recognizes broad patterns in the first few hidden layers, such as edges; as we go deeper into the CNN, these learned abstractions get more precise, such as textures, patterns, and (parts of) things.

How many filters does a convolutional neural network have?

Filters help us better grasp what CNN is seeking to discover. AlexNet's initial convolution layer discovered these 96 filters.

Why do we employ several filters in convolutional neural networks?

It is more convenient to employ all necessary filters at once because convolving individual filters separately will lengthen calculation time.

Conclusion

In this article, we have extensively discussed the convolutional neural network. We have also explained how to enter visualizations in CNN, visualize convolutional layers, pre-fit the VGG model, visualize filters, and more in detail.

We hope this blog has helped you enhance your visualizing Convolutional Neural Networks with filter knowledge. If you would like to learn more, check out our articles on Learn neural networks, artificial neural networksUnderstanding CNN, and the basics of python. Practice makes a man perfect. To practice and improve yourself in the interview, you can check out Top 100 SQL problemsInterview experienceCoding interview questions, and the Ultimate guide path for interviews.

Do upvote our blog to help other ninjas grow. Happy Coding!

CN thank you
Previous article
GoogLeNet Model
Next article
Guided Backpropagation
Live masterclass