Explain how you will combine the ranks of a number of subjects given by several judges.
10 marks · directive: Explain
The core of a good answer
With m judges each ranking the same n subjects, add the ranks of each subject to get Rⱼ and order the subjects by Rⱼ (the smallest total gets the best combined rank), and measure agreement by Kendall's coefficient of concordance W = 12S/[m²(n³ − n)] with S = Σ(Rⱼ − m(n + 1)/2)², 0 ≤ W ≤ 1, tested by χ² = m(n − 1)W on n − 1 d.f.; the combined ranking is meaningful only if W is significant.
An outline written by upsc.help, not an official UPSC answer; every fact checked by a separate review. How we work
What the question asks
Describe the procedure: arrange the m × n rank table, sum ranks, derive the combined ranking, measure and test agreement with Kendall's W (with tie correction), and relate W to the average Spearman correlation. A short numerical sketch helps.
Answer outline
Introduction
Data: an m × n table; row i gives judge i's ranks 1 to n of the n subjects (or items, candidates). Ranks are ordinal, so means of raw scores are not used.
Step 1: Rank sums
- Rⱼ = Σᵢ rᵢⱼ for each subject j; ΣRⱼ = mn(n + 1)/2, mean R̄ = m(n + 1)/2.
- Combined ranking: order subjects by Rⱼ (or by mean rank Rⱼ/m).
Step 2: Coefficient of concordance
- S = Σⱼ (Rⱼ − R̄)².
- Maximum S (perfect agreement) = m²(n³ − n)/12, so W = 12S/[m²(n³ − n)].
- W = 1: all judges agree; W = 0: no agreement (rank sums equal).
- With ties: W = 12S/[m²(n³ − n) − mΣT], T = Σ(t³ − t) over tied groups.
Step 3: Testing W
- H₀: judges' rankings are independent (no agreement).
- For moderately large n: χ² = m(n − 1)W with n − 1 d.f. (equal to Friedman's statistic); for small n use exact tables of S or a permutation test.
Step 4: Relation to Spearman's ρ
- Average of the m(m − 1)/2 pairwise Spearman correlations: ρ̄ = (mW − 1)/(m − 1).
- For m = 2, W and ρ are linearly related: ρ = 2W − 1.
Step 5: Other ways
- Average rank correlation of each judge with the combined ranking identifies outlying judges.
- Friedman's test (χ²_r = 12S/[mn(n + 1)]) is the same statistic used to test differences among subjects.
Conclusion
Sum the ranks across judges to form the combined order, and report Kendall's W with its χ² test to show whether the judges agree enough for the combined ranking to be trusted.
Examples and data to cite
- Kendall and Babington Smith (1939, The Annals of Mathematical Statistics) introduced the coefficient of concordance; W is a normalisation of Friedman's test statistic.
- Typical uses: panels of judges in competitions, ranking of job applicants, wine tasting, and expert ranking of policy options.
Facts to remember
- W = 12S/[m²(n³ − n)], 0 ≤ W ≤ 1.
- χ² = m(n − 1)W with n − 1 d.f. for large n.
- ρ̄ = (mW − 1)/(m − 1).
- Combined ranking is by total (or mean) rank.