See Inside

Do the Math: Why No Ranking System Is No. 1

Rankings are flawed in part because rarely are all votes equal and assigning weights to voters is often not feasible

Image: TONY CORDOZA Getty Images

Decisions concerning the products we buy, the Web sites we click on, the movies we watch and even where we send our children to college are all affected by rankings. But did you ever wonder who or what is making all these rating decisions? Are they subjective opinions, or is something else going on under the covers?

Put yourself in Mark Zuckerberg's place when he rated and ranked the women of Harvard University for his Facemash site that evolved into Facebook. The most straightforward method would be to ask people to vote for their favorite; a coed's rating would simply be the number of votes received. But this doesn't work well, because rarely are all votes equal. For example, an uninformed vote is usually not as valuable as one from a knowledgeable person, or, in the case of Facemash, a voter's gender might matter.

But assigning weights to voters is often not feasible, especially when voters' identities are unknown, so you might try the Bowl Championship Series approach used to rate college football teams. Applied to a top-10 list of coeds, it would work in this way: Ask voters to assign a score of 10 to their favorite, 9 to their second favorite, and so on. Add up the scores for each woman to arrive at a rating.

Most football fans, however, prefer that their sports teams be ranked according to how well they fare in head-to-head competition. In fact, the pressure from fans has become so great that college football commissioners announced in April that they are contemplating play-off games for the 2014 season. Zuckerberg instinctively knew that implementing head-to-head matchups is a better way to establish ratings. He implemented pair-wise comparisons by displaying a pair of photographs and asking, “Which is hotter?” Scoring is easy. Each matchup is scored by allotting one point to the winner and zero to the loser (each gets half of a point in case of a tie).

But how does this get turned into ratings? Arpad Elo, a Hungarian-born physicist and avid chess player, theorized that a reasonable approach would be to establish a mean performance level for each player as competition ensues. Once ratings are assigned, they should be changed only in proportion to the degree to which a player performs above or below his or her mean. Elo's idea was later refined by replacing each player's overall mean performance by a relative measure that reflects the expected performance when one player is matched specifically against another player. The logic is that the difference between two players' ratings before they meet should suggest what to expect when they are matched against each other.

Elo's elegant idea has become ubiquitous throughout the gaming world, as well as in soccer and American football. But in each case, the scheme is tweaked to suit the specifics of the competition. We still cannot say that it is the best way to rate and rank things—because there is no best way. In 1951 Kenneth Arrow, a mathematical economist, proved that there can be no optimal ranking system that also satisfies certain fairness criteria. Thus, the controversy lives on, keeping raters and rankers in business by continually tweaking and tailoring their systems to fit specialized needs.

Already a Digital subscriber? Sign-in Now

Langville, a professor of mathematics at the College of Charleston, and Meyer, a professor of mathematics at North Carolina State University, wrote Who's 1?: The Science of Rating and Ranking (Princeton University Press, 2012).
Langville, a professor of mathematics at the College of Charleston, and Meyer, a professor of mathematics at North Carolina State University, wrote Who's 1?: The Science of Rating and Ranking (Princeton University Press, 2012).

Rights & Permissions

View
1. 1. krohleder 02:05 PM 7/11/12

Interesting! I wonder how this could be applied to political choices?

2. 2. LarryW 05:00 PM 7/11/12

Rankings don't work not because of ignorance alone, but because of the various independent criteria that can be and should be used for evaluation.

Win/Lose comparisons are the easiest in sports but even then, player health, weather conditions, good days/bad days, adaptions over time, location, match ups will likely result in different rankings.

Then there is stability of rankings. They are not. Rankings are ordinal measures, but estimates of stability of the rankings require interval or ratio scales.

3. 3. Cramer 06:10 PM 7/11/12

Why was there no mention of new research by the authors? Elo and Arrow research took place in the 1950s -- nothing new since then? From this article that is what I would guess, especially since Kenneth Arrow "proved that there can be no optimal ranking system that also satisfies certain fairness criteria."

You must sign in or register as a ScientificAmerican.com member to submit a comment.
Click one of the buttons below to register using an existing Social Account.

More from Scientific American

• Scientific American Magazine | 32 minutes ago

Orphaned Chimpanzees Play Rougher Than Their Mother-Reared Counterparts

• Scientific American Magazine | 23 hours ago

Teenage Flu Scientist Shares His Recipe for Prizewinning Research

• Scientific American Magazine | 23 hours ago

• Cocktail Party Physics | Dec 7, 2013

Physics Week in Review: December 7, 2013

• @ScientificAmerican | Dec 6, 2013

More »

Latest from SA Blog Network

• Stream of Thought Description of Teaching James's "Stream of Thought": A Work of Faction

Cross-Check | 20 hours ago
• Physics Week in Review: December 7, 2013

Cocktail Party Physics | Dec 7, 2013
• Wonderful Things: The Pugnacious, Alien-esque Skeleton Shrimp

The Artful Amoeba | Dec 6, 2013
• Can We Harness Disruption to Improve Our World's Future?

STAFF
@ScientificAmerican | Dec 6, 2013
• British Storm Brings Up History's First Work of Social Media

Plugged In | Dec 6, 2013

Science Jobs of the Week

Do the Math: Why No Ranking System Is No. 1: Scientific American Magazine

X

Give a 1 year subscription as low as \$14.99

X

X

Welcome, . Do you have an existing ScientificAmerican.com account?

No, I would like to create a new account with my profile information.

X

Are you sure?

X