Decisions concerning the products we buy, the Web sites we click on, the movies we watch and even where we send our children to college are all affected by rankings. But did you ever wonder who or what is making all these rating decisions? Are they subjective opinions, or is something else going on under the covers?

Put yourself in Mark Zuckerberg's place when he rated and ranked the women of Harvard University for his Facemash site that evolved into Facebook. The most straightforward method would be to ask people to vote for their favorite; a coed's rating would simply be the number of votes received. But this doesn't work well, because rarely are all votes equal. For example, an uninformed vote is usually not as valuable as one from a knowledgeable person, or, in the case of Facemash, a voter's gender might matter.

But assigning weights to voters is often not feasible, especially when voters' identities are unknown, so you might try the Bowl Championship Series approach used to rate college football teams. Applied to a top-10 list of coeds, it would work in this way: Ask voters to assign a score of 10 to their favorite, 9 to their second favorite, and so on. Add up the scores for each woman to arrive at a rating.

Most football fans, however, prefer that their sports teams be ranked according to how well they fare in head-to-head competition. In fact, the pressure from fans has become so great that college football commissioners announced in April that they are contemplating play-off games for the 2014 season. Zuckerberg instinctively knew that implementing head-to-head matchups is a better way to establish ratings. He implemented pair-wise comparisons by displaying a pair of photographs and asking, “Which is hotter?” Scoring is easy. Each matchup is scored by allotting one point to the winner and zero to the loser (each gets half of a point in case of a tie).

But how does this get turned into ratings? Arpad Elo, a Hungarian-born physicist and avid chess player, theorized that a reasonable approach would be to establish a mean performance level for each player as competition ensues. Once ratings are assigned, they should be changed only in proportion to the degree to which a player performs above or below his or her mean. Elo's idea was later refined by replacing each player's overall mean performance by a relative measure that reflects the expected performance when one player is matched specifically against another player. The logic is that the difference between two players' ratings before they meet should suggest what to expect when they are matched against each other.

Elo's elegant idea has become ubiquitous throughout the gaming world, as well as in soccer and American football. But in each case, the scheme is tweaked to suit the specifics of the competition. We still cannot say that it is the best way to rate and rank things—because there is no best way. In 1951 Kenneth Arrow, a mathematical economist, proved that there can be no optimal ranking system that also satisfies certain fairness criteria. Thus, the controversy lives on, keeping raters and rankers in business by continually tweaking and tailoring their systems to fit specialized needs.

4 Comments

1. 1. RDH 10:44 AM 7/11/12

   Let's see how head-to-head match ups determine the outcome.
   
   Last season LSU played Alabama and won. Last season LSU played Alabama and lost. The result? Alabama was given a higher ranking. Calculate that.

   Reply | Report Abuse | Link to this

2. 2. krohleder 02:05 PM 7/11/12

   Interesting! I wonder how this could be applied to political choices?

   Reply | Report Abuse | Link to this

3. 3. LarryW 05:00 PM 7/11/12

   Rankings don't work not because of ignorance alone, but because of the various independent criteria that can be and should be used for evaluation.
   
   Win/Lose comparisons are the easiest in sports but even then, player health, weather conditions, good days/bad days, adaptions over time, location, match ups will likely result in different rankings.
   
   Then there is stability of rankings. They are not. Rankings are ordinal measures, but estimates of stability of the rankings require interval or ratio scales.

   Reply | Report Abuse | Link to this

4. 4. Cramer 06:10 PM 7/11/12

   Why was there no mention of new research by the authors? Elo and Arrow research took place in the 1950s -- nothing new since then? From this article that is what I would guess, especially since Kenneth Arrow "proved that there can be no optimal ranking system that also satisfies certain fairness criteria."

   Reply | Report Abuse | Link to this