Do you think IIT Guwahati certified course can help you in your career?
Introduction
Deep learning is a compelling field of computer science used to solve many real-life problems such as object detection and product recommendation with great accuracy. It involves various complex algorithms which try to imitate the functioning of the human brain. To get high accuracy, we must train our model on a large amount of training data. It may seem that the more we train our model on the training data, the more will be the accuracy. But if you train your model for long, the prediction accuracy decreases. This is due to a problem known as overfitting. Let us look at this problem and how we can tackle this problem by using early stopping.
Overfitting is a common problem in the realm of deep learning. It happens when our model fits very closely to our training data. It will lead to excellent training accuracy on the trained dataset. But outside the dataset, the predictions will be poor. Our model will just memorize the training data and will not perform well for any unseen data.
above is the visual representation of an underfit, optimal, and overfit model. We want our model to be optimal. An optimal model will perform well on the training dataset and will also make good predictions on the test dataset. An overfit model will perform excellently for the training dataset but will make poor predictions on the test dataset. An underfit model will perform poorly on both the training dataset and the test dataset. Let us see how we handle the problem of overfitting in deep learning.
Regularization
Overfitting is a nightmare for Machine learning practitioners. The general technique that we use in deep learning to remove the problem of overfitting is known as Regularization. Many reasons can cause overfitting. It may be due to an overly complex model that completely fits the training data or training our model for longer than needed.
Some regularization techniques reduce overfitting by adding a penalty term or by shrinking the coefficients to ensure that our model does not become overly complex, while some regularization techniques ensure that our model is not trained for more epochs than required. Early stopping is one of the regularization techniques which solves the problem of overfitting caused due to excessive training of our model.
Early stopping
By training our model for a longer amount of time, we mean training our model for a large number of epochs. In deep learning, an epoch is defined as the number of times our model is trained on the entire dataset. It is crucial to choose the number of epochs so that it leads to an optimal model. Choosing a very low number for the epochs will lead to an underfit model, and choosing a very high number of epochs will lead to overfitting.
For choosing an optimal number of epochs, we can use early stopping. We can choose an arbitrarily high number of epochs, and early stopping will stop the training as soon as we start to overfit our data. Let us look at the idea behind early stopping.
Working of early stopping
In early stopping, During the training of our model, we also check the prediction accuracy of our model on a validation dataset. We keep track of the prediction accuracy on the validation dataset, and as soon as the validation accuracy starts to decrease, we stop our training preventing overfitting. How early stopping is achieved:
Source
Here, We can see how early stopping is achieved during the training of our model. We keep track of our training and validation error, and we stop once the minimum validation error is achieved. If we keep on training our model, the training error decreases, but the validation error increases, indicating overfitting.
Implementation of early stopping using Keras
To get the validation error while training, we use callback. A callback is a set of functions that provides a way to interact with the model while training.
We can implement early stopping by using the EarlyStopping callback in Keras.
First, we need to import EarlyStopping:
from keras.callbacks import EarlyStopping
To initialize early stopping. We have to define various arguments first. Let us see an example code of the EarlyStopping function with some important arguments.
earlystop = EarlyStopping(monitor = 'val_loss',
mode= ‘min’, patience = 25, verbose = 1,
restore_best_weights = True,
min_delta = 1)
monitor: “monitor” argument helps us select the performance measure that we want to track, such as validation loss or validation accuracy.
mode: mode can be set to ‘min’ or ‘max’ to monitor the minimum or maximum value, respectively. By default, it is set to ‘auto’ and automatically detects what we want to achieve, either maximizing the accuracy or minimizing the loss.
patience: Sometimes, the first time, we reach our lowest point in the validation loss the first time. It is not the best performance of our model. We can use ‘patience’ to set the number of epochs with no improvement after which the training is stopped.
verbose: To know the epoch number on which the training was stopped, we set the verbose to 1.
restore_best_weights: To keep the best weights once the training is stopped, we set this to True.
min_delta: Even a slight improvement in the model will be counted as relevant, but in actual practice, it may lead to trivial improvements, and the model keeps on training without any significant improvement.To counter this, we can set a value of min_delta. It will consider the improvement only after a certain threshold of improvement is reached. For example, min_delta = 1 will consider the improvement only if 1% of improvement is reached.
EarlyStopping will stop the training once the monitored performance measure stops improving according to the set arguments.
Below is an example of early stopping used while training. Checkpoint is created when the validation loss decreases. The training is stopped when we see no improvement in the validation loss after the given patience. The weights and biases at the checkpoint are saved to get the best fit model.
It gives us the better accuracy of our model by preventing overfitting
It helps us to train our model in less number of epochs
FAQs
What are some of the other regularization methods other than early stopping? Yes, there are other regularization methods like L1 regularization, L2 regularization, dropout, etc.
When should we use early stopping? We can always use early stopping to get the best result from our model with the minimum number of epochs.
What is ModelCheckpoint in Keras? Model checkpoint is used in Keras to save the best model achieved during our training.
Key takeaways
This blog gave us a complete understanding of Early stopping. What it is and how it solves the problem of Overfitting. We also learned how we could implement it in our model using Keras. There are other methods of regularization. To learn more, check out our courses on Coding Ninjas.
84+ registered