Sentence Prediction with BERT

NSP In Code

Let's look into how we can use code to show NSP.

Along with the bert-base-uncased model, we'll be using HuggingFace's transformers and PyTorch. Let's start by importing and initializing everything:=

CODE-

from transformers import BertTokenizer, BertForNextSentencePrediction
import torch
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForNextSentencePrediction.from_pretrained('bert-base-uncased')
text = ("After Abraham Lincoln won the November 1860 presidential election on an "
        "anti-slavery platform, an initial seven slave states declared their "
        "secession from the country to form the Confederacy.")
text2 = ("War broke out in April 1861 when secessionist forces attacked Fort "
         "Sumter in South Carolina, just over a month after Lincoln's "
         "inauguration.")

You can also try this code with Online Python Compiler

Run Code

It's worth noting that we have two strings: text for sentence A and text2 for sentence B. Keeping them distinct allows our tokenizer to accurately process both of them, as we'll see in a moment.

We must now take the following three steps:

• Tokenization
• Make a label for categorization.
• Make a loss calculation.

To begin, we'll look at tokenization.

Tokenization 

We use our initialized tokenizer to do tokenization, passing both text and text2.

inputs = tokenizer(text, text2, return_tensors='pt')
inputs.keys()

You can also try this code with Online Python Compiler

Run Code

output

dict_keys(['input_ids', 'token_type_ids', 'attention_mask'])

You can also try this code with Online Python Compiler

Run Code

Few things to keep in mind regarding NSP. First, our two sentences are blended into the same collection of tensors — yet BERT can tell that they are two separate statements in several ways.

• Between the two sentences, a [SEP] token is added. In our input ids tensor, this separator token is represented by 102.
• The token type ids tensor holds segment ids that indicate which token belongs to which segment. Sentence A has a value of 0, and sentence B has a value of 1.

Create class label 

The next step is simple: we need to generate a new labels tensor that determines if phrase B comes after sentence A.

labels = torch.LongTensor([0])
labels

You can also try this code with Online Python Compiler

Run Code

output

tensor([0])

You can also try this code with Online Python Compiler

Run Code

IsNextSentence has a value of 0 and NotNextSentence has a value of 1. We'll also need to utilize the torch. LongTensor is a tensor format.

Calculate loss 

Finally, we figure out how much we've lost. We begin by running our model over our inputs and labels.

outputs = model(**inputs, labels=labels)
outputs.keys()
outputs.loss
outputs.loss.item()

You can also try this code with Online Python Compiler

Run Code

output

odict_keys(['loss', 'logits'])
tensor(3.2186e-06, grad_fn=<NllLossBackward>)
3.2186455882765586e-06

You can also try this code with Online Python Compiler

Run Code

Our model will return the loss tensor, which we'll focus on during training — which we'll get to shortly.

Prediction

We may not need to train our model and instead want to use it to make inferences. We wouldn't have a labels tensor in this scenario, so we'd change the last section of our method to extract the logits tensor as follows:

outputs = model(**inputs)
outputs.keys()
And take the argmax to get our prediction:
torch.argmax(outputs.logits)

You can also try this code with Online Python Compiler

Run Code

output

odict_keys(['logits'])
tensor(0)

You can also try this code with Online Python Compiler

Run Code

The activation for the IsNextSentence class in index 0 and the activation for the NotNextSentence class in index one will be returned by our model as a logits tensor.

To return our model's forecast, we take the argmax of the output logits. It returns 0 in this situation, indicating that BERT believes sentence B follows sentence A. (correct).

FAQs

1. Can BERT predict the next sentence?

BERT is fine-tuned on three methods for the following sentence prediction task: We have sentences as input and just one class label output in the first kind, as in the following task: MNLI (Multi-Genre Natural Language Inference) is a technique for inferring between different types of natural language. It's a massive classification project.

2. How do you predict the BERT model?

Because BERT is a bidirectional model, it tries to look in both left-right and right-left directions. To deduce the meaning of a masked word For prediction, BERT considers both the following and preceding tokens of the masked word.

3. Can BERT be used for text generation?

The instructor BERT model is no longer required once the student model has been trained, and only the student model is used to generate the text. This means that Distill-BERT does not need any additional resources during generation.

4. What is BERT ML?

BERT is a machine learning framework for natural language processing that is open source (NLP). BERT is a program that uses surrounding text to help computers grasp the meaning of ambiguous words in the text.

Key Takeaways

So that's the end of the article.

In this article, we have extensively discussed Sentence Prediction with BERT.

Isn't Machine Learning exciting!! We hope that this blog has helped you enhance your knowledge regarding Sentence Prediction with BERT and if you would like to learn more, check out our articles on the MACHINE LEARNING COURSE. Do upvote our blog to help other ninjas grow. Happy Coding!