See Inside

# Do the Math: Why No Ranking System Is No. 1

Rankings are flawed in part because rarely are all votes equal and assigning weights to voters is often not feasible

Image: TONY CORDOZA Getty Images

Decisions concerning the products we buy, the Web sites we click on, the movies we watch and even where we send our children to college are all affected by rankings. But did you ever wonder who or what is making all these rating decisions? Are they subjective opinions, or is something else going on under the covers?

Put yourself in Mark Zuckerberg's place when he rated and ranked the women of Harvard University for his Facemash site that evolved into Facebook. The most straightforward method would be to ask people to vote for their favorite; a coed's rating would simply be the number of votes received. But this doesn't work well, because rarely are all votes equal. For example, an uninformed vote is usually not as valuable as one from a knowledgeable person, or, in the case of Facemash, a voter's gender might matter.

But assigning weights to voters is often not feasible, especially when voters' identities are unknown, so you might try the Bowl Championship Series approach used to rate college football teams. Applied to a top-10 list of coeds, it would work in this way: Ask voters to assign a score of 10 to their favorite, 9 to their second favorite, and so on. Add up the scores for each woman to arrive at a rating.

Most football fans, however, prefer that their sports teams be ranked according to how well they fare in head-to-head competition. In fact, the pressure from fans has become so great that college football commissioners announced in April that they are contemplating play-off games for the 2014 season. Zuckerberg instinctively knew that implementing head-to-head matchups is a better way to establish ratings. He implemented pair-wise comparisons by displaying a pair of photographs and asking, “Which is hotter?” Scoring is easy. Each matchup is scored by allotting one point to the winner and zero to the loser (each gets half of a point in case of a tie).

But how does this get turned into ratings? Arpad Elo, a Hungarian-born physicist and avid chess player, theorized that a reasonable approach would be to establish a mean performance level for each player as competition ensues. Once ratings are assigned, they should be changed only in proportion to the degree to which a player performs above or below his or her mean. Elo's idea was later refined by replacing each player's overall mean performance by a relative measure that reflects the expected performance when one player is matched specifically against another player. The logic is that the difference between two players' ratings before they meet should suggest what to expect when they are matched against each other.

Elo's elegant idea has become ubiquitous throughout the gaming world, as well as in soccer and American football. But in each case, the scheme is tweaked to suit the specifics of the competition. We still cannot say that it is the best way to rate and rank things—because there is no best way. In 1951 Kenneth Arrow, a mathematical economist, proved that there can be no optimal ranking system that also satisfies certain fairness criteria. Thus, the controversy lives on, keeping raters and rankers in business by continually tweaking and tailoring their systems to fit specialized needs.

Already a Digital subscriber? Sign-in Now

Langville, a professor of mathematics at the College of Charleston, and Meyer, a professor of mathematics at North Carolina State University, wrote Who's 1?: The Science of Rating and Ranking (Princeton University Press, 2012).
Langville, a professor of mathematics at the College of Charleston, and Meyer, a professor of mathematics at North Carolina State University, wrote Who's 1?: The Science of Rating and Ranking (Princeton University Press, 2012).

View
1. 1. RDH 10:44 AM 7/11/12

Last season LSU played Alabama and won. Last season LSU played Alabama and lost. The result? Alabama was given a higher ranking. Calculate that.

2. 2. krohleder 02:05 PM 7/11/12

Interesting! I wonder how this could be applied to political choices?

3. 3. LarryW 05:00 PM 7/11/12

Rankings don't work not because of ignorance alone, but because of the various independent criteria that can be and should be used for evaluation.

Win/Lose comparisons are the easiest in sports but even then, player health, weather conditions, good days/bad days, adaptions over time, location, match ups will likely result in different rankings.

Then there is stability of rankings. They are not. Rankings are ordinal measures, but estimates of stability of the rankings require interval or ratio scales.

4. 4. Cramer 06:10 PM 7/11/12

Why was there no mention of new research by the authors? Elo and Arrow research took place in the 1950s -- nothing new since then? From this article that is what I would guess, especially since Kenneth Arrow "proved that there can be no optimal ranking system that also satisfies certain fairness criteria."

You must sign in or register as a ScientificAmerican.com member to submit a comment.
Click one of the buttons below to register using an existing Social Account.

## More from Scientific American

• Quick and Dirty Tips | 25 minutes ago

### What is Pascal's Triangle?, Part 2

• Plugged In | 11 hours ago

### What unconventional fuels tell us about the global energy system

• News | 13 hours ago

### Shooting the Wheeze: Whooping Cough Vaccine Falls Short of Previous Shot’s Protection

• Guest Blog | 14 hours ago

### Dissecting the controversy about early psychological response to disasters and trauma

• Ask the Experts | 18 hours ago | 13

More »

## Latest from SA Blog Network

• ### Introducing: Kyle Hill

STAFF
The SA Incubator | 1 hour ago
• ### #SciAmBlogs Tuesday - Oklahoma, cervical cancer, Weil's postulate, immunity and reproduction, global energy, UK wallabies, and more.

STAFF
The Network Central | 8 hours ago
• ### Fat Tuesday: Hungry for love

MIND
Illusion Chasers | 10 hours ago
• ### What unconventional fuels tell us about the global energy system

Plugged In | 11 hours ago
• ### Dissecting the controversy about early psychological response to disasters and trauma

Guest Blog | 14 hours ago

Do the Math: Why No Ranking System Is No. 1: Scientific American Magazine

X

### Subscribe Today

Save 66% off the cover price and get a free gift!

X

X

###### Welcome, . Do you have an existing ScientificAmerican.com account?

No, I would like to create a new account with my profile information.

X

Are you sure?

X