Image: iStockphoto
-
The Best Science Writing Online 2012
Showcasing more than fifty of the most provocative, original, and significant online essays from 2011, The Best Science Writing Online 2012 will change the way...
Read More »
Web sites such as Amazon, TripAdvisor and Yelp have long depended on customers to rate books, hotels and restaurants. The philosophy behind this so-called crowdsourcing strategy holds that the truest and most accurate evaluations will come from aggregating the opinions of a large and diverse group of people. Yet a closer look reveals that the wisdom of crowds may neither be wise nor necessarily made by a crowd. Its judgments are inaccurate at best, fraudulent at worst.
According to Eric K. Clemons, a professor of operations and systems management at the Wharton School of the University of Pennsylvania, online ranking systems suffer from a number of inherent biases. The first is deceptively obvious: people who rate purchases have already made the purchase. Therefore, they are disposed to like the product. “I happen to love Larry Niven novels,” Clemons says. “So whenever Larry Niven has a novel out, I buy it. Other fans do, too, and so the initial reviews are very high—five stars.” The high ratings draw people who would never have considered a science-fiction novel. And if they hate it, their spite could lead to an overcorrection, with a spate of one-star ratings.
Such negativity exposes another, more pernicious bias: people tend not to review things they find merely satisfactory. They evangelize what they love and trash things they hate. These feelings lead to a lot of one- and five-star reviews of the same product.
A controlled offline survey of some of these supposedly polarizing products revealed that individuals’ true opinions fit a bell-shaped curve—ratings cluster around three or four, with fewer scores of two and almost no ones and fives. Self-selected online voting creates an artificial judgment gap; as in modern politics, only the loudest voices at the furthest ends of the spectrum seem to get heard.
This self-selection process manifests itself in other ways. In a 2009 study of more than 20,000 items on Amazon, Vassilis Kostakos, a computer scientist at the University of Madeira in Portugal, found that a small percentage of users accounted for a huge majority of the reviews. These super-reviewers—often celebrated with “Top Reviewer” badges and ranked against one another to encourage their participation—each contribute thousands of reviews, ultimately drowning out the voices of more typical users (95 percent of Amazon reviewers have rated fewer than eight products). “There is nothing to say that these people are good at what they do,” Kostakos says. “They just do a lot of it.” What appears to be a wise crowd is just an oligarchy of the enthusiastic.
The existence of super-reviewers has one unassailable advantage, though: they are rarely shills. The deliberate manipulation of review sites by people directly involved with a product—the author of the book, say—is one of the oldest and most difficult problems for online-rating communities to solve.
Some sites attempt to remove suspect posts using automated filters that search for extremely positive or negative language, especially when the review comes from someone with a short résumé. But this lack of transparency can breed mistrust—or worse.
Consider the case of the local-business review site Yelp, which filters out suspect reviews. Its CEO and co-founder Jeremy Stoppelman defends the practice by pointing to classified advertisements placed by business owners offering payment for positive reviews. Yet some businesses suspect more sinister forces at work. Earlier this year a coalition of local business owners sued Yelp, accusing the company of running what amounted to a digital extortion racket. The lawsuit claims that sales representatives from Yelp would call businesses and make a simple offer: advertise with us, and we’ll make negative reviews disappear.
The company vigorously denies the allegations and claims that any cuts are automated and coincidental. Still, Yelp has refused to divulge how its filters operate, lest unscrupulous users employ that information to game the system. This lack of transparency has led to the perception that the company itself might be manipulating the playing field.
See what we're tweeting about
22 Comments
Add CommentIMO, most purchasers claim to have never made a bad purchase decision. This is especially true for corporate purchasing agents, who are evaluated by their purchase decisions, as we all are to some extent by our peers.
Reply | Report Abuse | Link to thisThis article was all right.
Reply | Report Abuse | Link to thisIt is time to revise a democratic system which allows 1,000 million people to suffer malnutrition and a similar number to live below the poverty line. And the starting-point should be to fight against corruption and other organised political crimes.
Reply | Report Abuse | Link to thisI think this article is a little negative. I am one of the "Top Reviewers" at Amazon and I do not drown out other voices. I make many 3 star reviews.
Reply | Report Abuse | Link to thisMy negative reviews of bad movies and faulty products are often deleted by the fan boys and corporate sell-outs that usually dominate sites. First read the most negative reviews of any product, because despite exaggeration, they usually have way more honesty about problems: Nobody is paying these shills.
Reply | Report Abuse | Link to thisP.S. The phoniness of early online ratings can be seen in the numerous reviews of products, books and movies that are pre-release! The vast majority of these people didn't have advance copies.
Reply | Report Abuse | Link to thisBetter rating systems must be created, and we can vet them more effectively. Before the advent of the technologies/tools referred to above, there were general principles of supply and demand - coupled with word of mouth. Experts emerged, as did opinion/thought leaders.
Reply | Report Abuse | Link to thisWe must work to improve on the current rating systems and develop new ones that make them more useful. Current one to five star scales are insufficient, even with their other variables and an open-ended field for extensive detail. The more we quantify consumer feedback (e.g. Zagat ratings), the easier it will be to sort and filter.
We also must consider the many reasons consumers are willing to spend a few minutes of their time - and a few degrees of their privacy - to rave or rant, and more importantly, when their feelings are less strong. With respect to the former, the incentive is innate. But perhaps new incentives can be created so that post-purchase review becomes a standard requirement for transactions - so that we not only get more helpful information, but that we get it from a majority, rather than a small minority. I.e. what if you earned $ for your review - perhaps an instant rebate on the product itself? Some review to help others, some to promote their personal brands, some because they are addicted to engaging online, some to give back to the companies/brands they admire, some to feel accepted. The list goes on. But money is the tie that binds.
And although improved systems will directly benefit the same consumers helping to generate this new content - especially as mobile technologies allow quick and easy (and in-store) access at the exact time when real purchasing decisions are made - no one will benefit more than the retailers and marketers themselves, looking to know more about who their customer is, how he/she behaves/thinks, and what they feel about your new products/services as they launch.
Marketers and retailers
Are there any websites that do have unbiased legitimate reviews? Any recommendations?
Reply | Report Abuse | Link to thisyou can't fairly compare restaraunts on a rating scale or movies like Titanic to Toy Story.
Reply | Report Abuse | Link to thisi don't understand our society's inability to process the value of rankings > ratings for relatable items.
@lfshadow - that's precisely what we're doing at http://gorankem.com for the music industry. you can't compare bob dylan to beyonce...that's why we allow the fans of each artist to rank their favorite songs & then aggregate the collective opinion :)
-adam
This article does not mention paid shills who do the reviews. As a freelance writer, this is one of the more common things that I am paid to do. Yes, you heard that right. I get paid to write articles about "products". You should try going to freelancer.com and see for yourselves the prices of reviews, articles, tweets, blog posts, facebook friends, etc.
Reply | Report Abuse | Link to thisYour world is a highly manipulated one.
How gloriously ironic that this article was published with Digg and Stumble buttons, not to mention the comments section, for rating the article.
Reply | Report Abuse | Link to thisAlas, I can only give this article a 2 out of 5. Moyer just rehashes old, misleading statistical theory. "The vast majority," the theory goes, "of honest pushed reviews (where reviewers decide for themselves to post) come from the enthusiastic and the severely dissatisfied." If this were true, though, reviews would tend to fall in an inverse bell curve--highest at the top and the bottom--but this doesn't happen all that often. For the majority of review cases, once you have a dozen or more entries, the bell curve rules.
So, why does this happen? Why would anyone post a mediocre review? Why would anyone waste their time posting reviews of something about which they have only lukewarm feelings? Are the so-called Top Reviewers, what Moyer calls our "oligarcy of the enthusiastic," responsible for all of the middle reviews? Partly. First, I should note, like commenter jackdavis37, my wife is an Amazon Top Reviewer. In fact, she's in a program called Vine, where she gets free products in return for reviewing them. Moyer's word choice is insulting and inappropriate. I would call that class an aristocracy of the articulate. They don't drown anyone out, but they do fill in a lot of gaps. They review because they like writing product reviews. Their reviews tend to be the most comprehensive and the least emotional. They don't fill quite all the gaps, however. The remaining fill-ins come from folks who have a sense of community, a concept apparently alien to Mr. Moyer.
As for dannybuntu and his ilk, when are you guys going to get a clue? We know you're out there, and we know how to spot you. Good review systems allow us to peruse your other reviews, contact you for further information, and otherwise interact with the system. Beside which, paid reviews always have a set of points the marketing guys want you to hit, making the reviews look hokey.
So, overall, yes: caveat emptor. Of course, that's what the reviews are there for: to help the buyer be wary. As with any opinion from an unknown source, of course you have to be careful. That's why the review sites have so many mechanisms for ensuring the validity of the reviews. The one gem you might take from this article is don't rely on a single good or bad reviews. If you have only a small number of reviews and none of those answer the questions you really want answered, look for another source of information.
I agree with Danny -- I'm also an occasional writer, and have been asked on several occasions to write glowing blurbs for products on various sites. This practice is apparently widespread, and isn't dissimilar from the fake "testimonials" given by paid actors on TV commercials.
Reply | Report Abuse | Link to thisI give this article 1 star; here in the comment section. Comments are in fact the redeeming value of rating systems online. First you can tell when it is the fan club, or employees, that post by what they say. Also we all know people do not rate accurately (ourselves excluded - lol) but usually some truth comes out in the comments. For example: "I love my iPhone, there is nothing better, sometimes I have to hold it a certain way to get reception, but over all it is amazing: 4 out of 4 stars!". So we can see issues past the actual rating in the comments.
Reply | Report Abuse | Link to thisNewegg has one of the better ratings systems I've seen, mostly since many items (generally computer hardware) tend to get hundreds of reviews, most of which would be from people who have bought said item. On the whole, there tends to be mostly 5 and 4 ratings, not many 3s, and depending on the product, 1-10% 1s and a few 2s usually. Anything with a lot of 1s is something to be leery of, and you generally end up comparing the ratio of 5s and 4s to 1s.
Reply | Report Abuse | Link to thisMost 3s are from people who noticed a flaw in the item or had a problem with it and thus couldn't give it a 5 (the highest rating), but didn't think it was bad enough to warrant a 1 or 2. Some don't take off much for problems they knew about before buying it (Which would be like buying an iPhone4 with a case and rating it a 4/5).
These don't follow a bell curve, of course, but if the number of 1s were equal to the number of 5s in computer hardware then you'd have a problem on your hands. That's the kind of thing where you want the hardware to be as reliable as possible. (Most 1s are for when the person is shipped something that is DOA or defective, especially if they RMA it for a replacement and the replacement is also DOA (it happens, though I'd think it should be rare), or there's a major problem with it.)
I take in the reviews (when applicable) from a handful of friends and coworkers. Otherwise, if I'm curious, I'll check out the product myself. It's a pretty reliable system.
Reply | Report Abuse | Link to thisThis article totally ignored what happen to the normal consumers who will read these reviews. I think it is silly to assume everyone would buy anything based on how many stars it has. My philosophy to the reviews is that I would not buy things that is lower than 2 stars unless I really need it, and I will read some arbitrary comments. I will decide not to buy something if it has low ratings, but I will not decide to buy something if the average rating is high.
Reply | Report Abuse | Link to thisRational people would know that ratings are biased. The stories coming out of the comments remind the buyers what they will be expecting,
A better system should be able to poll on the comments people wrote, instead of a simple star rating systems.
@Medical Bill Dog.
Reply | Report Abuse | Link to thisThank you for the flattery, however, you are wrong in one part.
"We know you're out there, and we know how to spot you."
Er... No you can't. ;)
Bad news for you is there are actually people who write better reviews than me and get paid to do it.
@Enthilzha Yep. :)
A definite 5 star article! The best ever. If you don't read this article, your life will not be as worthwhile. You should read this article! I'll tell you, I read articles all the time, but it wasn't until I read this article that I finally figured it all out. This article has all the answers! It's better than anyone could ever dream of! It's the best! Buy the issue!!!!
Reply | Report Abuse | Link to thiswe at ratebeer have an incredibly deep and well-maintained database of ratings now stretching back a full decade (we pre-date facebook by 5 years). few of the other sites we're usually mentioned with can say such a thing, and few come as close to 'globally comprehensive' as ratebeer. we also tend to go nuts with finding new ways to measure, sort and display the pulse of beer, another thing very few others do.
Reply | Report Abuse | Link to thisPut your money where your mouth is, Danny? Send me a list of reviews. I'll not only spot the shills, I'll tell you which points the marketing department told them to pitch.
Reply | Report Abuse | Link to thisI am suspicious of the so called filter at Yelp.com. I've posted negative but perfectly honest and factually correct reviews that were rated positively by others, and poff they are gone. The filter removes them without explanation. I stopped reading Yelp.com
Reply | Report Abuse | Link to thisThomas Wikman
I'd never thought about how "people tend not to review things they find merely satisfactory. They evangelize what they love and trash things they hate." This phenomenon would make it difficult to trust online reviews. One thing, though, that I didn't agree with is his negative characterization of those who write with a personal investment (like authors on Amazon, or endorsements by people who <a href="https://www.writersdomain.net/">get paid to write articles</a>). I think it is important, not sneaky, for people to be actively promoting their work. It shows that they really believe in it.
Reply | Report Abuse | Link to this