Neutral Style Transfer

Working Of NST

According to the basic theory behind Neural Style Transfer, "It is possible to decouple the style representation, and content representations in a CNN trained during a computer vision task (e.g., image recognition task)."

We already know many cutting-edge models such as AlexNet, VGG, RESNET, and many others. These models have one thing in common: they were trained on the ImageNet Dataset (14 million images with 1000 classes), which allows them to understand the ins and outs of each image. By separating the content and style parts of an image and giving a loss function to maximize the needed outcome, we can use the quality of these models.

As previously stated, we define a pre-trained convolutional model and loss functions that visually blend two images, so we'll need the following inputs.

A Content Picture is an image that will be used to impart style.

A Style Image - the look and feel we wish to convey

A created input image – the final content and the appropriate style image.

We'll look at several intermediate layers in our model to gain our image's content and style representations. As you travel deeper into the map, the middle levels reflect feature maps that grow higher organized. We're utilizing the VGG19 network architecture, a pre-trained image classification network. These intermediate layers are required to define our photos' content and style representation. We will match the corresponding style and content target representations for an input image at these intermediate layers.

Now the question arises why intermediate layers?

Why Intermediate Layers?

You might be asking why we can define style and content representations using intermediate outputs from our pre-trained image classification network. This phenomenon can be explained at a high level by the fact that a network needs understand the image to do image classification (which our network has been trained to do). This entails taking the raw image as input pixels and constructing an internal representation using transformations that convert the primary image pixels into a complex understanding of the image's properties. This is also one of the reasons why convolutional neural networks generalize well: they can capture invariances and define features inside classes unaffected by background noise and other annoyances. As a result, somewhere between where the raw image is fed in and where the categorization label is produced, the model acts as a complicated feature extractor, allowing us to define the content and style of incoming images by accessing intermediary layers.

Loss Functions

To obtain the intended image, we must define a loss function that optimizes the losses to achieve the desired output. The concept of per-pixel losses will be used here.

Per Pixel Loss is a statistic for determining the differences between photographs at the pixel level. The output pixel values are compared to the input pixel values. (Perpetual loss functions are another option, which we shall examine briefly later in this blog.) Each pixel loss can have its own set of difficulties when displaying every relevant trait. That's where the problem of repeated losses comes into play. We'll be concentrating on the following loss terms:

• Content Loss
• Style Loss

Content Loss

It ensures that the content we desire in the resulting image is efficiently captured. The higher tiers of the network, according to CNN, capture information about the content, but the lower levels are more focused on particular pixel values. Our concept of content loss is reasonably straightforward. We'll send the requested content image and base input image to the network. Our model's intermediate layer outputs (from the layers defined above) will be returned. The Euclidean distance between the two intermediate representations of those images is then calculated. Content loss is calculated by:

def get_content_loss(base_content, target):
     return tf.reduce_sum(tf.square(base_content - target))

Style Loss

Because computation involves numerous levels, defining the loss function for style takes more time than defining the loss function for content. The correlation between the feature maps per layer determines the style information. For computing style loss, we use the Gram Matrix. What is a gram matrix, exactly?

The gram matrix is a metric for capturing the distribution of features across a collection of feature maps in a layer. So, while you're computing or limiting the style loss, you're also ensuring that the amount of feature distribution in both the styles and generated images is the same. The aim is to create style and generated picture gram matrices, and compute their difference. The Gram matrix (Gij) is the sum of a layer's ith and jth feature maps multiplied by their height and breadth. 

Img_Src

The content loss is implemented by:

def gram_matrix(M):
     M=tf.transpose(M, (2,0,1))
     features=tf.reshape(M, (tf.shape(M)[0], -1))
     gram=tf.matmul(features, tf.transpose(features))
     return gram

 

def style_loss(style, target):
      S=gram_matrix(style)
      C=gram_matrix(target)
      channels=3
      size=img_rows*img_columns
      return tf.reduce_sum(tf.squares(S-C)) / (4.0 * (channels ** 2)*(size ** 2))

As a result, we'll compute a weighted sum of both the calculated content and style losses to arrive at the final loss.

 

The above code completes the final integration of losses by traveling through the layers and determining the final loss by taking a weighted summation in the second last line. Finally, we'll need to define an optimizer (Adam or SGD) to minimize the network's loss.

Implementation

Importing the Libraries

import tensorflow_hub as hub
import tensorflow as tf
from matplotlib import pyplot as plt
import numpy as np
import cv2

Importing the VGG model

model = hub.load('https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2')

def load_image(img_path):
    img = tf.io.read_file(img_path)
    img = tf.image.decode_image(img, channels=3)
    img = tf.image.convert_image_dtype(img, tf.float32)
    img = img[tf.newaxis, :]
    return img

Loading the content and style image

content_image = load_image('content1.jfif')
style_image = load_image('style2.jpg')

plt.imshow(np.squeeze(content_image))
plt.show()

plt.imshow(np.squeeze(style_image))
plt.show()

Generated image after NST

stylized_image = model(tf.constant(content_image), tf.constant(style_image))[0]
plt.imshow(np.squeeze(stylized_image))
plt.show()

Saving the image

cv2.imwrite('generated_img.jpg', cv2.cvtColor(np.squeeze(stylized_image)*255, cv2.COLOR_BGR2RGB))

Also read, Sampling and Quantization

FAQ’s

1. In the Neural Style Transfer, what is the style loss?

Ans. Total variation loss imposes local spatial continuity between the pixels of the combined image, resulting in visual coherence. Deep learning is retained in the style loss, determined using a deep convolutional neural network.

2. In terms of style transfer, what is a gram matrix?

Ans. By using a gram matrix to combine characteristics extracted from convolutional neural networks. The Gram Matrix can be calculated using the following formula: V. gram=V. gram=V. gram=V. gram=V Multiply with the transpose of V, which is an arbitrary vector.

3. What is total variation loss, and how does it affect you?

Ans. The total variation loss is the sum of the absolute differences for contiguous pixel-values in the input images. This metric determines how much noise is there in the photographs.

4. Is supervised learning possible with Neural Style Transfer?

Ans. The neural style transfer inputs two images (x) and trains a network to produce a new, synthesized image as a supervised learning task (y).

Key Takeaways

Let us brief the article.

Firstly, we saw the basics of neural style transfer and how it can create effects. Later, we saw the architecture of NST, followed by the working of the NST. Moving on, we saw different types of losses in NST, and lastly, we saw the basic implementation of NST.

I hope you all like this article. 

I hope you all like this article. Want to learn more about Data Analysis? Here is an excellent course that can guide you in learning. Can also refer to our Machine Learning course.

Happy Learning Ninjas!