At the end of September the Pew Forum on Religion and Public Life released a survey that seemed to show that nonbelievers knew more about religion than the faithful. Some media outlets crowed about the results (“Atheists Know More about Religion Than Believers,” Time magazine declared), whereas others turned to comforting the faithful (“We Didn’t Flunk the Religion Test,” FoxNews.com insisted). Few seemed to realize that the polls were far from immaculate. In fact, the episode was a good example of what I call disestimation: the act of taking fuzzy numbers way too seriously.
At first, it might seem like a cut-and-dried story: out of 32 quiz questions, atheists and agnostics, on average, got 20.9 correct, higher than any other group and higher than the overall average of 16.0 questions right. But because Pew managed to reach very few atheists and agnostics—only 212 people out of the 3,412 included in the survey—the 20.9 number masks a tremendous amount of imprecision. Small samples don’t give reliable numbers, and if you present the poll results using a standard graphical technique to represent uncertainty (below), you can see that the distinction between atheists/agnostics and Jews and Mormons evaporates.
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
The story gets even fuzzier because Pew left out one category altogether: those who believe “nothing in particular,” many of whom had specifically said they didn’t believe in God. Interestingly, this group scored worse than the typical American on the religion quiz. Had they been lumped together with atheists and agnostics, the group would have fared a little worse, on average, than evangelical Protestants.
When Pew did a more stringent analysis, correcting for respondents’ education and income (which, sadly, was buried deep in the report), there was no significant difference between believers and nonbelievers. Those who said they did not believe in God scored a mere 0.3 point higher than the national average, a meaningless number, given how big the error bars are.
The press leaped on the atheists versus believers headlines without critically examining the numbers. The Pew study revealed less about our faith in God than it did about our faith in polls—which, far too often, is blind.