Code360 powered by Coding Ninjas X Code360 powered by Coding Ninjas X
Last Updated: Mar 27, 2024

Noisy Channel Model

Leveraging ChatGPT - GenAI as a Microsoft Data Expert
Prerita Agarwal
Data Specialist @
23 Jul, 2024 @ 01:30 PM


The Noisy Channel model is related to language modelling. So what is the noisy channel model, and why is it so important in natural language processing?


So let's take a hypothetical system that takes an input written in English. Let's call it X, and then we have some encoder that randomly misinterprets the information,  converting it to Y, which is some garbled English. So the output will be what we call spoken English or Y. The garbling is just converting the original English words into an audio signal corresponding to spoken English.   Let us look at more examples. We can convert grammatical English X to English with grammatical mistakes Y. We can convert English X to bitmaps or characters Y. In a more general case, the noisy channel model determines the connection between a joint probability of two sequences as the product of the first sequence times the likelihood of the second one given the first one. We can think of it as a process of encoding and decoding. So given our foreign language string, we want to guess the English version. We assume that 'e' was converted to 'f' using an encoder. Then, we want to build a decoder that converts 'f' into 'e.' Here, I am using this notation e’ (with an apostrophe) to indicate that this is our estimate, which doesn't mean that it's the original. It's just our best guess as to what ‘e’ is. 

So how do we determine e.? Well, ‘e’ will be the value that maximises the conditional probability of the English sentence given the following sentence. So, according to the Bayesian formula, we can write the expression that maximises the product of the conditional probability f given e times the probability of ‘e’

P (f|e) Is a noisy channel model known as a translation model, and P(e) is the language model. A language model takes a sequence of words in a given language, and it tells us how likely it is that a particular series is a valid sentence. 

Now, I will introduce the translation model, which essentially tells us that given a particular sentence in English, what's the probability that this specific sentence in the foreign language corresponds to the ruling in English.

So let's take an example from French that translates "la Maison Blanche" Into English. So here are some possible translations. The first one that we're going to solve is "cat plays the piano."  The probability that this foreign string "la Maison Blanche"  matches the English line "cat plays the piano" is very low, so we will put the minus in that box. What about the English language probability of a "cat plays piano.?"  Again, this is not a proper sentence, so we will give it a low score. The next one is a little bit better. We have the possible translation as "House White the ."This is not a grammatical sentence, but it matches the right words in French. So we will give it a positive score for the translation model and a negative score for the language model. Next, there is a little bit better "the House White," again, this is not a grammatical sentence in English but matches the French words. So we will give it the same set of scores as that of the previous example. 


Similarly, as shown here in the table, we can assign the scores to the rest of the sentences. 

The idea of the translation model is to come up with all the possible sequences of words in English of the right length to try to figure out, based on the two probabilities in the two columns, which one is the best translation of the original sentence. So a good translation will have a high score on their language model and a high score on the translation model. If many phrases have high scores on both of the columns, we're just going to multiply those two probabilities and pick the one with the highest product. 

Use Cases

Handwriting recognition- The input will be the bitmap, and the output will be an  English sentence. 

Text generation

Text summarization

Machine translation

Spelling correction-  In spelling correction, we can use it to model the probability that a specific type of mistake will be made. 

Get the tech career you deserve, faster!
Connect with our expert counsellors to understand how to hack your way to success
User rating 4.7/5
1:1 doubt support
95% placement record
Akash Pal
Senior Software Engineer
326% Hike After Job Bootcamp
Himanshu Gusain
Programmer Analyst
32 LPA After Job Bootcamp
After Job

Frequently Asked Questions

  1. What is a noisy channel model?
    Noisy channel model is a model In natural language processing that is used in handwriting recognition, machine translation text summarization, etc. 
  2. What is Natural Language Processing?
    Natural Language Processing in the form of artificial intelligence that gives machines the sense of reading and understanding human language.
  3. What are the types of statistical language models? 
    1. N-gram 
    2. Exponential 
    3. continuous space
  4. Give some common examples of language models.
    1. Speech recognition
    2. Machine translation
    3. Sentiment analysis
    4. Text suggestion
    5. Passing tool
  5. What are speech recognition language models?
    Speech recognition model is used to translate the human speech or spoken words into the text.

Key Takeaways

The Noisy channel model is used worldwide, and smart speakers like Alexa also use the technique to recognize the spoken words. To learn more interesting natural language processing techniques, hit the link.

Topics covered
Use Cases
Frequently Asked Questions
Key Takeaways