Pratidin
2026 · Electrical Engineering Optional Paper II · 10 marksMainsQuestion 7bi

A discrete memoryless source X has four symbols x₁, x₂, x₃, x₄ with probabilities P(x₁) = 0·4, P(x₂) = 0·3, P(x₃) = 0·2, P(x₄) = 0·1 respectively.
I. Calculate H(x).
II. Find the amount of information contained in the messages x₁, x₂, x₁, x₃ and x₄, x₃, x₃, x₂ and compare with the H(x) obtained in part (I).

10 marks · directive: Calculate and compare

Answer

The core of a good answer

H(X) = −Σ p log₂ p ≈ 1.846 bits/symbol, so a four symbol message carries 4H ≈ 7.39 bits on average; the message x₁x₂x₁x₃ carries 6.70 bits (below average, as it uses likely symbols) and x₄x₃x₃x₂ carries 9.70 bits (above average, as it uses unlikely symbols).

An outline written by upsc.help, not an official UPSC answer; every fact checked by a separate review. How we work

What the question asks

The question separates two ideas: entropy, which is the average information per symbol, and self information of a particular message, which depends on its probability. For a memoryless source the information of a sequence is the sum of the self informations of its symbols.

Answer outline

Introduction

Self information I(xᵢ) = −log₂ P(xᵢ) bits. Since the source is memoryless, P(message) is the product of symbol probabilities and its information is the sum.

Step 1: Self information of each symbol

Step 2: Entropy, part I

Step 3: Message x₁, x₂, x₁, x₃

Step 4: Message x₄, x₃, x₃, x₂

Step 5: Comparison with H(X)

Conclusion

H(X) ≈ 1.846 bits/symbol; message one carries 6.70 bits and message two 9.70 bits, compared with an average of 7.39 bits for four symbols. Check: the maximum entropy for four symbols is log₂ 4 = 2 bits, and H is below it because the probabilities are unequal.

Examples and data to cite

Facts

Facts to remember

Sources used for the explanation

Write a timed answer in the app