Bidirectional LSTM

How do LSTM works?

The Lengthy Short Term Memory architecture was inspired by an examination of error flow in current RNNs, which revealed that long time delays were inaccessible to existing designs due to backpropagated error, which either blows up or decays exponentially.

An LSTM layer is made up of memory blocks that are recurrently linked. These blocks can be thought of as a differentiable version of a digital computer's memory chips. Each one has recurrently connected memory cells as well as three multiplicative units – the input, output, and forget gates – that offer continuous analogs of the cells' write, read, and reset operations. 

Must read topic, Spring Boot Architecture

What is Bi-LSTM?

Bidirectional LSTM networks function by presenting each training sequence forward and backward to two independent LSTM networks, both of which are coupled to the same output layer. This means that the Bi-LSTM contains comprehensive, sequential information about all points before and after each point in a particular sequence. 

In other words, rather than encoding the sequence in the forward direction only, we encode it in the backward direction as well and concatenate the results from both forward and backward LSTM at each time step. The encoded representation of each word now understands the words before and after the specific word.

Below is the basic architecture of Bi-LSTM.

^source

Working of Bi-LSTM

Let us understand the working of Bi-LSTM using an example. Consider the sentence “I will swim today”. The below image represents the encoded representation of the sentence in the Bi-LSTM network.

So when forward LSTM occurs, “I” will be passed into the LSTM network at time t = 0, “will” at t = 1, “swim” at t = 2, and “today” at t = 3. In backward LSTM “today” will be passed into the network at time t = 0, “swim” at t = 1, “will” at t = 2, and “I” at t = 3. In this way, results from both forward and backward LSTM at each time step are calculated.

Bi-LSTM in keras

To implement Bi-LSTM in keras, we need to import the Bidirectional class and LSTM class provided by keras.

First, let us understand the syntax of the LSTM layer. There is one mandatory argument in the LSTM layer, i.e., the number of LSTM units in a particular layer.

tf.keras.layers.LSTM(units)

You can also try this code with Online Python Compiler

Run Code

LSTM layer accepts many other arguments like activation, recurrent activation, use_bias, kernel_initializer, recurrent_initializer, bias_initializer, etc. But all these arguments have some default value, so the user doesn't need to specify all the parameters until he wants to change them. (For more information, visit here)

Now, to implement the Bi-LSTM, we just need to wrap the LSTM layer inside the Bidirectional class.

tf.keras.layers.Bidirectional(LSTM(units))

You can also try this code with Online Python Compiler

Run Code

The Bidirectional layer accepts other arguments like merge_mode, backward_layer. (For more details, visit here)

FAQs

1. What is the difference between GRU and LSTM?

A: GRU has two gates, i.e., reset and update gate, whereas LSTM has three gates, i.e, input, output, and forget gate. GRU is preferred in small datasets, whereas LSTM is preferred while handling larger datasets.

2. Why is Bi-LSTM better than LSTM?

A: At every time step, LSTM calculates the results of forwarding LSTM, but in the case of Bi-Direction results from both forward and backward LSTM at each time step are calculated.

3. What are the limitations of Bi-LSTM?

A: Bi-LSTM takes more time to train than normal LSTM networks. Also, they acquire more memory to train. They are easy to overfit and dropout implementation is hard in Bi-LSTM.

4. What is a bidirectional layer?

A: Bidirectional recurrent neural networks (BRNN) connect two hidden layers of opposite directions to the same output. With this form of generative deep learning, the output layer can simultaneously get information from past (backward), and future (forward) states.

Key Takeaways

In this article, we have covered the following topics:

• Introduction and need of LSTM network
• The architecture of the LSTM unit
• Working of LSTM and Bi-LSTM
• Implementation of Bi-LSTM in keras

Want to learn more about Machine Learning? Here is an excellent course that can guide you in learning. 

Happy Coding!