For the general linear model Y = X β + u (Y: n×1, X: n×k, β: k×1, u: n×1) with E(u) = 0 and E(uu′) = σ² Iₙ, where X (n×k) is a matrix of fixed numbers with rank k (k < n), obtain the ordinary least squares estimator β̂ of β and the dispersion matrix of β̂. Also, prove that β̂ is the best linear unbiased estimator (BLUE) of β.
20 marks · directive: Prove
The core of a good answer
Minimising S(β) = (Y − Xβ)′(Y − Xβ) gives the normal equations X′Xβ̂ = X′Y and β̂ = (X′X)⁻¹X′Y; since β̂ = β + (X′X)⁻¹X′u, E(β̂) = β and D(β̂) = (X′X)⁻¹X′E(uu′)X(X′X)⁻¹ = σ²(X′X)⁻¹; and for any other linear unbiased estimator β̃ = [(X′X)⁻¹X′ + D]Y unbiasedness forces DX = 0, so D(β̃) = σ²(X′X)⁻¹ + σ²DD′, which exceeds D(β̂) by a non-negative definite matrix: β̂ is BLUE (Gauss-Markov theorem).
An outline written by upsc.help, not an official UPSC answer; every fact checked by a separate review. How we work
What the question asks
Three tasks: derive OLS by matrix calculus, derive its dispersion matrix, and prove the Gauss-Markov theorem in matrix form. State clearly where each assumption (fixed X, full rank, zero mean errors, spherical errors) is used.
Answer outline
Introduction
Assumptions: X non-stochastic with rank k (so X′X is non-singular), E(u) = 0, E(uu′) = σ²Iₙ (homoscedastic, uncorrelated errors). No normality is needed.
Step 1: OLS estimator
- S(β) = Y′Y − 2β′X′Y + β′X′Xβ
- ∂S/∂β = −2X′Y + 2X′Xβ = 0, giving X′Xβ̂ = X′Y.
- Rank k ensures (X′X)⁻¹ exists: β̂ = (X′X)⁻¹X′Y.
- ∂²S/∂β∂β′ = 2X′X, positive definite, so this is the minimum.
Step 2: Unbiasedness
- β̂ = (X′X)⁻¹X′(Xβ + u) = β + (X′X)⁻¹X′u.
- E(β̂) = β + (X′X)⁻¹X′E(u) = β (X fixed).
Step 3: Dispersion matrix
- D(β̂) = E[(β̂ − β)(β̂ − β)′] = (X′X)⁻¹X′E(uu′)X(X′X)⁻¹
- = σ²(X′X)⁻¹X′X(X′X)⁻¹ = σ²(X′X)⁻¹.
- σ² is estimated unbiasedly by s² = e′e/(n − k), giving the estimated dispersion s²(X′X)⁻¹.
Step 4: Gauss-Markov proof
- Let β̃ = CY be any linear estimator; write C = (X′X)⁻¹X′ + D.
- E(β̃) = CXβ = β + DXβ; unbiased for all β requires DX = 0.
- β̃ − β = Cu, so D(β̃) = σ²CC′ = σ²[(X′X)⁻¹X′ + D][X(X′X)⁻¹ + D′].
- Cross terms vanish because DX = 0 (and X′D′ = 0): D(β̃) = σ²(X′X)⁻¹ + σ²DD′.
- DD′ is non-negative definite, so D(β̃) − D(β̂) ≥ 0; equality only if D = 0, i.e. β̃ = β̂.
Step 5: Scalar form
- For any fixed vector ℓ, Var(ℓ′β̃) − Var(ℓ′β̂) = σ²ℓ′DD′ℓ = σ²‖D′ℓ‖² ≥ 0; in particular each Var(β̃ⱼ) ≥ Var(β̂ⱼ).
- So β̂ has minimum variance in the class of linear unbiased estimators: it is BLUE.
Conclusion
β̂ = (X′X)⁻¹X′Y, D(β̂) = σ²(X′X)⁻¹, and β̂ is BLUE under the classical assumptions; normality is not required, but if E(uu′) ≠ σ²I the theorem fails and generalised least squares (Aitken) is BLUE instead.
Examples and data to cite
- In simple regression yᵢ = α + βxᵢ + uᵢ, the result gives Var(β̂) = σ²/Σ(xᵢ − x̄)².
- Aitken (1935) extended the theorem to non-spherical errors: β̂_GLS = (X′Ω⁻¹X)⁻¹X′Ω⁻¹Y.
Facts to remember
- Normal equations X′Xβ̂ = X′Y; β̂ = (X′X)⁻¹X′Y.
- E(β̂) = β; D(β̂) = σ²(X′X)⁻¹.
- Gauss-Markov theorem: OLS is BLUE when E(u) = 0, E(uu′) = σ²I and X is fixed with full column rank.
- s² = e′e/(n − k) is unbiased for σ².