Discuss orthogonal polynomial regression on a single variable along with its principal advantages.
15 marks · directive: Discuss
The core of a good answer
In orthogonal polynomial regression y is fitted as α₀ + α₁P₁(x) + ⋯ + α_kP_k(x), where P_j is a polynomial of degree j and ∑ₓP_i(x)P_j(x) = 0 for i ≠ j; the normal equations become diagonal, so α̂_j = ∑yP_j/∑P_j², each degree's sum of squares (∑yP_j)²/∑P_j² is separate, and adding a higher-degree term leaves earlier coefficients unchanged.
An outline written by upsc.help, not an official UPSC answer; every fact checked by a separate review. How we work
What the question asks
A discussion question: explain the problem with ordinary polynomials (ill-conditioned, correlated x, x², x³ columns), define orthogonal polynomials over the observed x values, show how estimation and testing simplify, give the construction (Gram-Schmidt or tabulated coefficients for equally spaced x), and list the advantages clearly.
Answer outline
Introduction
Polynomial model y = β₀ + β₁x + ⋯ + β_kx^k + e. Powers of x are highly correlated, so X′X is nearly singular, and every coefficient changes when the degree is raised.
Definition
- Choose P₀(x) = 1 and polynomials P_j(x) of exact degree j with ∑ᵢ P_r(xᵢ)P_s(xᵢ) = 0 for r ≠ s over the n observed x values.
- Model: y = α₀ + α₁P₁(x) + ⋯ + α_kP_k(x) + e, which is the same fitted curve, reparametrised.
Estimation
- X′X is diagonal with entries ∑P_j², so α̂_j = ∑yᵢP_j(xᵢ)/∑P_j(xᵢ)², V(α̂_j) = σ²/∑P_j².
- The α̂_j are uncorrelated; under normality, independent.
Construction
- Gram-Schmidt on 1, x, x², … with respect to the sum over the x values.
- Equally spaced x, u = (x − x̄)/h: P₁ = u, P₂ = u² − (n² − 1)/12, with tables of integer coefficients (e.g. n = 5: linear −2, −1, 0, 1, 2; quadratic 2, −1, −2, −1, 2; cubic −1, 2, 0, −2, 1).
Analysis of variance
- Regression SS splits into one-d.f. parts SS_j = (∑yP_j)²/∑P_j².
- Test each degree by F = SS_j/MSE; stop when higher terms are not significant.
Principal advantages
- No matrix inversion; computations are simple and numerically stable.
- Adding or dropping a degree does not change the other coefficients or their sums of squares.
- Clear test for the degree of the polynomial needed.
- Removes multicollinearity among x, x², x³.
- Linear, quadratic, cubic trends in designed experiments (equally spaced levels of a factor) are tested by the same contrasts.
Limitations
- Polynomials depend on the particular x values; with unequal spacing they must be constructed afresh.
- Coefficients in terms of powers of x must be recovered by expanding the P_j.
Conclusion
Orthogonal polynomials give the same fit as ordinary polynomial regression but with a diagonal design, independent coefficient estimates and an additive breakdown of the regression sum of squares by degree. This makes choosing the degree simple and stable.
Examples and data to cite
- For x uniform on (0, 1), x and x² have correlation about 0.97, which is why raw powers make a poorly conditioned design.
- For five equally spaced points the linear, quadratic and cubic coefficients are (−2, −1, 0, 1, 2), (2, −1, −2, −1, 2) and (−1, 2, 0, −2, 1) (checked by Gram-Schmidt in Python).
Facts to remember
- ∑P_rP_s = 0 for r ≠ s over the design points.
- α̂_j = ∑yP_j/∑P_j²; SS_j = (∑yP_j)²/∑P_j², 1 d.f. each.
- For n equally spaced points: ∑P₁² = n(n² − 1)/12 when P₁ = u.
- Five-point coefficients: linear (−2, −1, 0, 1, 2), quadratic (2, −1, −2, −1, 2).