For the General Linear Model
Y = Xβ + u
where u ~ N(0, σ²Iₙ)
obtain the Maximum Likelihood Estimator (MLE) β̂ of β. Further, let e = (Y − Xβ̂), and X is a matrix of order n × k with rank of X, k < n. If e′e = u′Mu, where M = (I − X(X′X)⁻¹X′), show that e′e/σ² follows chi-square distribution with (n − k) degrees of freedom (d.f.).
10 marks · directive: Show
The core of a good answer
Maximising the normal log-likelihood −(n/2) log(2πσ²) − (Y − Xβ)′(Y − Xβ)/(2σ²) gives β̂ = (X′X)⁻¹X′Y (identical to OLS) and σ̂² = e′e/n; since M is symmetric and idempotent with trace n − k, writing M = PΛP′ with Λ = diag(1, ..., 1, 0, ..., 0) and w = P′u/σ ~ N(0, Iₙ) gives e′e/σ² = u′Mu/σ² = w₁² + ... + wₙ₋ₖ² ~ χ²(n − k).
An outline written by upsc.help, not an official UPSC answer; every fact checked by a separate review. How we work
What the question asks
Two parts: derive the MLE by differentiating the log-likelihood, then prove the distribution of the residual sum of squares using the properties of M (symmetric, idempotent, rank n − k) and the theorem on idempotent quadratic forms in normal variables.
Answer outline
Introduction
Y is n × 1, X is n × k fixed with rank k, β is k × 1, u ~ Nₙ(0, σ²I). So Y ~ Nₙ(Xβ, σ²I).
Step 1: Likelihood
- L(β, σ²) = (2πσ²)^(−n/2) exp[−(Y − Xβ)′(Y − Xβ)/(2σ²)]
- log L = −(n/2) log 2π − (n/2) log σ² − S(β)/(2σ²), S(β) = Y′Y − 2β′X′Y + β′X′Xβ
Step 2: MLE of β and σ²
- ∂ log L/∂β = (X′Y − X′Xβ)/σ² = 0 gives the normal equations X′Xβ̂ = X′Y.
- Rank k makes X′X invertible: β̂ = (X′X)⁻¹X′Y; the Hessian −X′X/σ² is negative definite, so a maximum.
- ∂ log L/∂σ² = 0 gives σ̂² = (Y − Xβ̂)′(Y − Xβ̂)/n = e′e/n (biased; e′e/(n − k) is unbiased).
Step 3: e′e = u′Mu
- e = Y − X(X′X)⁻¹X′Y = MY = M(Xβ + u) = Mu, because MX = X − X = 0.
- M′ = M and M² = M − 2X(X′X)⁻¹X′ + X(X′X)⁻¹X′X(X′X)⁻¹X′ = M, so e′e = u′M′Mu = u′Mu.
Step 4: Rank of M
- Eigenvalues of an idempotent matrix are 0 or 1, and rank = trace.
- tr M = tr Iₙ − tr[X(X′X)⁻¹X′] = n − tr[(X′X)⁻¹X′X] = n − tr Iₖ = n − k.
Step 5: Distribution
- M symmetric, so M = PΛP′ with P orthogonal and Λ = diag(1, ..., 1, 0, ..., 0) having n − k ones.
- Let z = u/σ ~ N(0, Iₙ) and w = P′z; then w ~ N(0, P′P) = N(0, Iₙ), so w₁, ..., wₙ are i.i.d. N(0, 1).
- e′e/σ² = z′Mz = w′Λw = Σᵢ₌₁ⁿ⁻ᵏ wᵢ².
- A sum of n − k squares of independent standard normals is χ² with n − k d.f.
Conclusion
β̂_MLE = (X′X)⁻¹X′Y and e′e/σ² ~ χ²(n − k); hence E(e′e) = (n − k)σ², so s² = e′e/(n − k) is unbiased, and since MX = 0 makes e independent of β̂, this result underlies the t and F tests in regression.
Examples and data to cite
- In simple regression (k = 2) the result gives RSS/σ² ~ χ²(n − 2), the basis of the t test for the slope with n − 2 d.f.
- Cochran's theorem generalises the argument to a partition of Y′Y into several idempotent quadratic forms.
Facts to remember
- MLE of β under normal errors coincides with OLS: β̂ = (X′X)⁻¹X′Y.
- M = I − X(X′X)⁻¹X′ is symmetric, idempotent, MX = 0, rank n − k.
- If z ~ N(0, I) and A is symmetric idempotent of rank r, then z′Az ~ χ²(r).
- MLE of σ² is e′e/n; unbiased estimator is e′e/(n − k).