Independent and Identically Distributed Random Variable

Independent Random Variables

Let us understand the term independent random variables with an example. Let us consider a fair coin. You flip a fair coin that could result either in heads or tails. Consider another fair die. You roll a die that could result in values from one to six. Now let us define X as a result of flipping a fair coin and Y as a result of rolling a die. The probability of getting head or tail after flipping a coin is ½, and the probability of getting any number after rolling a die is ⅙.

If a question is framed, the die and coin are rolled and flipped simultaneously. What is the probability of getting (4, H) on die and coin, respectively?

As total sample space is of 12 values ((1, H), (2, H), (3, H), (4, H), (5, H), (6, H), (1, T), (2, T), (3, T), (4, T), (5, T), (6, T)). Therefore the probability will be 1/12.

We can write it as P(4 ^ H) = P(4).P(H)

        1/12 = ⅙ * ½ 

        1/12 = 1/12

As we can see that the probability of getting head or tail is independent of getting any value on the die. So they are independent random variables.

In technical terms, we can understand the definition as:

If X and Y are two random variables with probability mass function of p_x and p_y. X and Y are said to be independent random variables if and only if P(X ^ Y) = P(X)*P(Y) for all pairs of (X, Y).

Identically Distributed Random Variables

Let X and Y be two Random Variables. The probability mass function of X is p_x, and the probability mass function of Y is p_y. X and Y are said to be Identically distributed random variables if and only if p_x = p_y.

For example, considering a biased one rupee coin and a biased two rupee coin(both are different), both the coins' probability mass function is the same. Hence it is identically distributed random variables.

Independent and Identically Distributed Random Variable

Consider the random variables X and Y. If the X and Y are independent and both are identically distributed, then the two random variables are independent and identically distributed.

We will consider two cases to understand independent and identically distributed random variables. Consider a pot having 10 distinct balls(all balls of different colors) where the probability of drawing each ball is the same. We draw a total of 7 balls with replacement from the pot. The probability of getting a ball x on the 5th draw will be independent of the previous outcomes, i.e., the total balls in the pot at the time of the 5th draw will be 10, and the probability of getting ball x will be 1/10. So drawing a ball at any instance is totally independent of previous outcomes. The probability distribution remains the same throughout the event, so these events are independent and identically distributed. But what if the balls are drawn without replacement? Then the probability of drawing a ball depends on the previous outcomes. So these are not independent, and also, the distribution of probability is not identical.

Examples of independent and identically distributed random variables

• Tossing a coin 25 times and recording how many times head occurs.
• Choosing 15 cards from a deck of cards.(Putting the card back in the deck after choosing)
• Rolling a die for 20 times and recording how many times 5 occurs.

FAQs

1. Why is IID important in statistics?
   IID samples have the important property that the larger the sample becomes, the greater the sample's probability of closely resembling the population.
    
2. How do you find IID?
   IID must satisfy two conditions, i.e., it must be independent and identically distributed.
    
3. Is IID a normal distribution?
   IID data need not be normally distributed.

Key Takeaways

In this article, we have discussed:

• Independent random variables
• Identically distributed random variables
• Independent and identically distributed random variables

Want to learn more about Machine Learning? Here is an excellent course that can guide you in learning. 
Happy Coding!