Decisions concerning the products we buy, the Web sites we click on, the movies we watch and even where we send our children to college are all affected by rankings. But did you ever wonder who or what is making all these rating decisions? Are they subjective opinions, or is something else going on under the covers?
Put yourself in Mark Zuckerberg's place when he rated and ranked the women of Harvard University for his Facemash site that evolved into Facebook. The most straightforward method would be to ask people to vote for their favorite; a coed's rating would simply be the number of votes received. But this doesn't work well, because rarely are all votes equal. For example, an uninformed vote is usually not as valuable as one from a knowledgeable person, or, in the case of Facemash, a voter's gender might matter.
But assigning weights to voters is often not feasible, especially when voters' identities are unknown, so you might try the Bowl Championship Series approach used to rate college football teams. Applied to a top-10 list of coeds, it would work in this way: Ask voters to assign a score of 10 to their favorite, 9 to their second favorite, and so on. Add up the scores for each woman to arrive at a rating.
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
Most football fans, however, prefer that their sports teams be ranked according to how well they fare in head-to-head competition. In fact, the pressure from fans has become so great that college football commissioners announced in April that they are contemplating play-off games for the 2014 season. Zuckerberg instinctively knew that implementing head-to-head matchups is a better way to establish ratings. He implemented pair-wise comparisons by displaying a pair of photographs and asking, “Which is hotter?” Scoring is easy. Each matchup is scored by allotting one point to the winner and zero to the loser (each gets half of a point in case of a tie).
But how does this get turned into ratings? Arpad Elo, a Hungarian-born physicist and avid chess player, theorized that a reasonable approach would be to establish a mean performance level for each player as competition ensues. Once ratings are assigned, they should be changed only in proportion to the degree to which a player performs above or below his or her mean. Elo's idea was later refined by replacing each player's overall mean performance by a relative measure that reflects the expected performance when one player is matched specifically against another player. The logic is that the difference between two players' ratings before they meet should suggest what to expect when they are matched against each other.
Elo's elegant idea has become ubiquitous throughout the gaming world, as well as in soccer and American football. But in each case, the scheme is tweaked to suit the specifics of the competition. We still cannot say that it is the best way to rate and rank things—because there is no best way. In 1951 Kenneth Arrow, a mathematical economist, proved that there can be no optimal ranking system that also satisfies certain fairness criteria. Thus, the controversy lives on, keeping raters and rankers in business by continually tweaking and tailoring their systems to fit specialized needs.