Batch Normalization - Introduction

Batch Normalization

Batch Norm is a normalizing technique between layers of a Neural Network rather than in the raw data. Instead of using the entire data set, it is done in mini-batches. Its purpose is to facilitate learning by speeding up training and utilizing higher learning rates.

Normalization is usually applied to input data, but it makes sense to keep internal data flow within the network normalized.

Within the input values passed between the layers of a neural network, BN is the internal enforcer of normalization. The covariate shift to the activations inside the layers is usually limited by internal normalization.

As previously stated, the BN approach works by applying a sequence of operations to the data that enters the BN layer.

We may define the normalizing formula of Batch Norm as follows, using the technique described in the preceding section:

Where:

m_z : mean of the neuron’s output 

s_z : standard deviation of the neuron’s output

How Batch Normalization is Applied

A standard feed-forward Neural Network is shown in the image below: The inputs are xi, the neuron’s output is z, the activation functions output is a, and the network's output is y:

Source

Before applying the activation function, Batch Norm – indicated by a red line in the image – is applied to the neurons' output. 

A neuron without a Batch Norm is often computed as follows with the model learning the parameters w and b:
 

Where,

g(): the linear transformation of the neuron

w: weights of the neuron

b: bias of the neuron

f(): activation function

Adding Batch Norm, it looks as:

Where,

m_z : mean of the neuron’s output 

s_z : standard deviation of the neuron’s output

β: learning parameters of Batch Norm

γ: learning parameters of Batch Norm

The components are parameter vectors that are used to scale(γ) and shift(β) the vector that contains the results of the previous operations. The values of the scaling and shifting parameter vectors are learnable parameters. BN ensures that the learnable parameters are the best values for proper normalization of each mini-batch during neural network training.

Benefits of Batch Normalization

• Hyperparameter tuning is less sensitive in this model. While higher learning rates have previously resulted in non-valuable models, higher LRs are now acceptable.
• Gradients' dependency on the scale of the parameters or their underlying values is reduced.
• Within the neural network, there is less covariate shift.
• Incorporating Batch Normalization into deep neural networks reduces training time.

Frequently Asked Questions

Q1) What is bias?

The gap between the current model's average prediction and the actual results we need to anticipate is known as bias.

Q2) What is Variance?

The distribution (or clustering) of the model outputs on a data point is known as a variance. The higher the conflict, the more probable the model is focused on training data and does not generalize data that has never been observed.

Q3) Is it possible to train a neural network model by setting all biases to 0?

Yes, even if all of the biases are set to zero, the neural network model has a chance of learning.

Q4) Why is it critical to include nonlinearities in a neural network?

Without non-linearities, no matter how many layers are present, a neural network will behave like a perceptron, with the output being linearly dependent on the input.

Key Takeaways

The BN algorithm isn't nearly as hard as it appears. With the development of current machine learning libraries like TensorFlow and PyTorch, the complexity of BN implementation within a neural network is abstracted. This makes BN implementation in a neural network a breeze, found here.