At the end of September the Pew Forum on Religion and Public Life released a survey that seemed to show that nonbelievers knew more about religion than the faithful. Some media outlets crowed about the results (“Atheists Know More about Religion Than Believers,” Time magazine declared), whereas others turned to comforting the faithful (“We Didn’t Flunk the Religion Test,” FoxNews.com insisted). Few seemed to realize that the polls were far from immaculate. In fact, the episode was a good example of what I call disestimation: the act of taking fuzzy numbers way too seriously.
At first, it might seem like a cut-and-dried story: out of 32 quiz questions, atheists and agnostics, on average, got 20.9 correct, higher than any other group and higher than the overall average of 16.0 questions right. But because Pew managed to reach very few atheists and agnostics—only 212 people out of the 3,412 included in the survey—the 20.9 number masks a tremendous amount of imprecision. Small samples don’t give reliable numbers, and if you present the poll results using a standard graphical technique to represent uncertainty (below), you can see that the distinction between atheists/agnostics and Jews and Mormons evaporates.
The story gets even fuzzier because Pew left out one category altogether: those who believe “nothing in particular,” many of whom had specifically said they didn’t believe in God. Interestingly, this group scored worse than the typical American on the religion quiz. Had they been lumped together with atheists and agnostics, the group would have fared a little worse, on average, than evangelical Protestants.
When Pew did a more stringent analysis, correcting for respondents’ education and income (which, sadly, was buried deep in the report), there was no significant difference between believers and nonbelievers. Those who said they did not believe in God scored a mere 0.3 point higher than the national average, a meaningless number, given how big the error bars are.
The press leaped on the atheists versus believers headlines without critically examining the numbers. The Pew study revealed less about our faith in God than it did about our faith in polls—which, far too often, is blind.
Already a Digital subscriber? Sign-in Now
If your institution has site license access, enter here.
Rights & Permissions
See what we're tweeting about
19 Comments
Add CommentGood point. The public media should know that every poll has a margin of error, and their reporting should reflect that.
Reply | Report Abuse | Link to thisThe Pew Forum itself has a lot of blame for this, as they should know better, and yet their report places very little emphasis on the margins of error.
However, I take issue with one aspect of the SA article: The author, Charles Seife, combines the results of "Atheist/Agnostic" with "Nothing in Particular" and concludes that atheists and agnostics don't know as much about religion as the poll would indicate. He is reading too much into the results. The poll's results were based on the self-reported religious affiliation of the respondents. Seife doesn't know the true beliefs of those who responded "nothing in particular", or, for that matter, any other affiliation in the survey.
Seife states that those who identified themselves as "nothing in particular" also said that they don't believe in God. Even if true, this doesn't diminish my point that it isn't meaningful to combine the two groups. But on top of that, I couldn't find that statistic anywhere in the full report. I admit that I didn't read all 78 pages, but a quick skim failed to show that statistic.
I don't understand the authors point that "
Reply | Report Abuse | Link to thiscorrecting for respondents’ education and income (which, sadly, was buried deep in the report), there was no significant difference between believers and nonbelievers." It is still accurate to state that nonbelievers did better than believers. All controlling for eduction and income does is provide a possible explanation (i.e., mediating variable) for why this link exists. In other words it is likely that BECAUSE nonbelivers and better educated (this assumption is pretty safe because it would explain the decrease in the initial correlation) they tend to score better on this test. I'm not sure of the authors background -- but I'm guessing it isn't statistics. I suggest reading about mediating variables -- a good place to start is the classic Baron and Kenny (1986) article -- to better grasp these findings.
I've gotta disagree with this one. Claiming that people who don't care about religion and believe "nothing in particular" should be lumped together with atheists is highly disingenuous. There is a big difference between people who self-identify as atheist/agnostic and people who simply say "I know don't care/know". If you carry your logic to its full conclusion you may as well eliminate every subcategory in the survey and just leave it at "religious vs non-religious" and blend the Jews, Mormons, Catholics, Protestants, etc. all together.
Reply | Report Abuse | Link to thisSure you can do that, but again, it would fail to tell you anything meaningful.
You single out the atheist sample as small, but it's no smaller than the Jewish, Mormon, Hispanic, or Black Protestant samples.
It's certainly arguable that the differences between the Jews, Mormons, and Atheists isn't significant overall, though generally the Jews and Atheists did better on general religion questions and the Mormons on Biblical ones indicating different areas of competency, but there is a clear distinction between those groups and the rest.
You then go on to make a VERY false claim in your own analysis. You state:
"When Pew did a more stringent analysis, correcting for respondents’ education and income (which, sadly, was buried deep in the report), there was no significant difference between believers and nonbelievers."
You've setup a completely misleading claim here. First you are mixing all believers and all non-believers together, which you've already said eliminates the distinctions between the two. You aren't comparing atheists, Jews, Mormon, Protestants, etc., now, you're mixing them together. Then you state that education level accounts for all the differences, but this isn't true, the report clearly states that the atheists, Jews, Mormons, outperform the rest *even when* education is taken into consideration. Education only accounts for the difference after you've mixed the atheists with the "nothing in particular" group.
So, as far as I can tell, what this amounts to is that you simply don't like the results of the survey and are attempting to discredit it for no good reason.
Are you seriously going to argue that if the sample sizes were simply larger that all of these distinctions would go away, and that if they had surveyed 5,000 randomly selected people from each group we'd see completely different results, and that there would be no distinctions between Jews, Protestants, Catholics, atheists etc.?
Do you really think that people of all faiths generally have the exact same level of aggregate knowledge of religion? Sorry, but its your position that I'm, highly skeptical of....
When I looked at the numbers in this poll the most obvious thing was that the more-educated sectors of society did better. Jews and atheists/agnostics (usually well-educated) scored high, Hispanic Catholics and black Protestants (usually poorly educated) scored low.
Reply | Report Abuse | Link to thisThus, the poll only tells us that more educated people know more facts. But we already knew that.
Lumping atheists & agnostics also makes the data suspect. Agnostics, as questioners, might be thought to have the highest level of knowledge; while atheists, like the religious, have a firm belief & thus perhaps less interest in learning more.
Reply | Report Abuse | Link to thisThis is patently false, and I addressed the issue in my prior comment. Here is the actual study:
Reply | Report Abuse | Link to thishttp://pewforum.org/other-beliefs-and-practices/u-s-religious-knowledge-survey.aspx
Note: "Atheists and agnostics, Jews and Mormons perform better than other groups on the survey *even after controlling for differing levels of education.*"
The results don't just tell us who is better educated and who isn't. They controlled for education by including non-religious questions, like questions about literature and history, etc. Even if you compare people who got the same number of those questions right to each other, the results still hold, so its not just an issue of "better education".
It would seem to be obvious that people that have for whatever reason been exposed to multiple religions would be both more likely to know about multiple religions and less likely to be as religious. If you have been trained to believe in a particular religion as absolute truth you are likely to follow that religion. If you are aware that there are many religions that all train their followers that they are the only absolute truth, you are less likely to be religious and might rebel against all of them and become religiously atheistic!
Reply | Report Abuse | Link to thisTo add a little to this, from the report:
Reply | Report Abuse | Link to this"This survey and previous Pew Forum studies have shown that Jews and atheists/agnostics have high levels of educational attainment on average, which partially explains their performance on the religious knowledge survey. However, even after controlling for levels of education and other key demographic traits (race, age, gender and region), significant differences in religious knowledge persist among adherents of various faith traditions. Atheists/agnostics, Jews and Mormons still have the highest levels of religious knowledge, followed by evangelical Protestants, then those whose religion is nothing in particular, mainline Protestants and Catholics."
If you think about all the people you know and how educated and how religious they are vs how smart you think they are. What do you come up with? The biggest church in our area, most of the people I know from there, need the church to help them with bigger issues that being educated. I can only think of two that have a Church type degrees, (it's not Harvard), most of the people have High School or less. The people who are in business and have degrees more less support the church. People who are healthy are generally more educated overall. We work for a large apartment complex and we know all most everybody. These people go to church to get help, if it doesn't work...they go to another church. We see this everyday, for over twenty years now, It's very sad.
Reply | Report Abuse | Link to thisThat's not a proper subject for any poll!
Reply | Report Abuse | Link to thisIt's like counting apples, oranges, pears, lemons, and not any of the other fruits. You have to count ALL the fruits. Fairly!
Thanks for looking at the actual report. My earlier post was regarding what CF stated about controlling for eduction (I was lazy and didn't look at the actual report). Given what you found in the report it makes me wonder what CF was thinking with his comments. It is a little sad that a blog post about how the media is misusing polling data is itself misusing that same data to make a point.
Reply | Report Abuse | Link to thisThanks for looking at the actual report. My earlier post was regarding what CF stated about controlling for eduction (I was lazy and didn't look at the actual report). Given what you found in the report it makes me wonder what CF was thinking with his comments. It is a little sad that a blog post about how the media is misusing polling data is itself misusing that same data to make a point.
Reply | Report Abuse | Link to thisWhen I read about the survey, I didn't believe it, mainly because it so obviously reflected the situation in a country, where religion is an important subject. To be an atheist, in such a country, must imply that you aften have to discuss/defend your views against the vast majority of believers.
Reply | Report Abuse | Link to thisBeing a Dane, one of the most secular countries in the world, where the vast majority would be in the "nothing in particular" category, religion is hardly ever discussed. But the few believers will often have to defend their beliefs, so they would need to know something about religion, at least their brand of it. As a matter of fact, apx. 85% are member of the state church (protestant) but are known as four-wheel christians, they only go to church for their baptism, communion, wedding and burial.
And why are there no figures for muslims, which, like jews and christians, are monotheists and share central parts of their religion with the other two middle-eastern religions?
@moderatelymoderate:
Reply | Report Abuse | Link to thisAtheism doesn't imply a 'firm belief' in anything. It's lack of a belief (in theism). It's not unreasonable to lump agnostics and atheists together. The two positions are not mutually exclusive, unlike the rest of the categories. Agnosticism is usually a stance on what we can/could possibly know about the universe and god. Atheism is just the lack of thinking there is a god.
To summarize: They aren't mutually exclusive and I, and a large part of both groups, probably over-lap depending on the nature of the discussion.
This poll result dovetails nicely with observations made over a number of years in a Fidonet newsgroup dedicated to religious discussion. It was called "Holysmoke".
Reply | Report Abuse | Link to thisIn this newsgroup a skeptical view of many religions as being largely superstitious was held and the "faithful" were invited to explain and/or defend their views.
Discussion participants were not just "the usual sort of rant merchants" but often included clergy and academics.
It was appalling and surprising to know how many of the "faithful" were wholly unaware of the contents of even their own religion...and this on the part of commentators who had taken it upon themselves to participate in a discussion on these issues.
There was a major display ignorance of even irrefutable and straighforward historical facts... such as the fact the current version(s) of the Bible are the direct descendents of the compilation created by the Council of Nicea.
Even the idea that Jesus was a Jew was a fact that many of the "faithful" had wholly forgotten or ignored.
This same phenomena could also be observed on the then active discussion boards at PBS's Religion and Ethics Weekly.
It seems that it is too often the case that people-of-faith really don't hold any kind of profound religious views and are spiritually lazy... in that they seem to have operated their entire life using as a guide the idea that "somebody else already figured this out so I don't need to bother".
As a minor addendum.... those who were most ignorant of the even the incontrovertible aspects of their own religious belief often were also the ones who were the most rabid (and rabid is a perfect adjective)in proclaiming their views.
Reply | Report Abuse | Link to thisIt seems to me.. that most people are "a-religious" in that they are utterly bereft of any kind of meaningful spiritual or philosophical views. They sit back figuring "somebody else already took care of this".
They seem to clump up in groups like riot-prone soccer hooligans who irrationally and insanely "take sides" in a contest in which their fistfight results are wholly irrelevant.
Neither team in a World Cup match really cares about or needs the support of the rioters punching each other out in the parking lot.
So it is with most mass religions, it seems.
Consider a soccer match going on with the few dozen players and officials on the field... while several hundred or thousand hooligans trash each other simultaneously.
Taken as a whole.... an alien spacecraft descending to observe might think the riot was the main event and that the soccer match was a pastime indulged by some bystanders.
So it is in the world of mass religions.... pretty much everyone in mass religions are out of the loop... and I would suggest are about as serious about "religion" as the rioting hooligans are serious about soccer... which is to say.. not at all...they are serious about street fights.
The hooligans are there for a riot... and they let the players sort out the soccer score.
So too is it in mass-religions... it seems pretty clear that members of popular mass religions are generally about as engaged in religion as soccer hooligans are engaged in soccer ... which is to say not at all.
The point has already been made, but I'll add my weight to it - it is a complete nonsense to lump together Atheists and NIPs. The NIP could be a Jewish NIP, a Catholic NIP, or a NIP of any other background.
Reply | Report Abuse | Link to thisTo be a NIP means "I don't know and I don't care", whereas an atheist has thought about the issue, reviewed the evidence, and decided that the idea of a god is nonsensical, and religions positively dangerous. (And I speak from personal experience, after 42 years as a practising Anglican).
An example, my Congregationalist wife taught in a Catholic school. None of the other teachers, all practising Catholics, had ever heard of transubstantiation, and would have been amazed had they known they were required to believe in it.
So it is no surprise to me at all that atheists are better informed than religious folk - what does surprise me is that a professor of journalism should practise the dark arts of mathematical deception!
Submitted as letter to the editor on Dec. 16:
Reply | Report Abuse | Link to thisIn his Dec. 14th column (The Science of “Disestimation”: The Shortcomings of Opinion Polls), Charles Seife expresses concerns about the religious knowledge survey recently conducted by the Pew Forum on Religion & Public Life (http://www.pewforum.org/Other-Beliefs-and-Practices/U-S-Religious-Knowledge-Survey.aspx). We welcome a robust discussion of the findings of our polls, and we try to make such discussion possible by publishing our full question wording, sample sizes, methodology and margins of sampling error. However, Seife’s column contains factual errors and misstatements requiring correction.
In particular, Seife sets up a straw man by saying that the survey “seemed to show” that “nonbelievers” know “more about religion than the faithful.” He then knocks down the straw man by saying that once sampling error is taken into account, “the distinction between atheists/agnostics and Jews and Mormons evaporates.”
But our report does not claim that atheists and agnostics outperformed Jews and Mormons. Rather, we say in the very first sentence that “Atheists and agnostics, Jews and Mormons are among the highest-scoring groups on a new survey of religious knowledge,” and go on to point out that “Jews and Mormons do about as well” as atheists and agnostics. The report also shows that all three of these groups “perform better than other (religious) groups on the survey even after controlling for differing levels of education.” We are not sure how the “margin of error” bars in the chart accompanying Seife’s piece were calculated. But Scientific American readers should know that the differences between groups that we describe in the report passed conventional tests of statistical significance. At Pew, we take great care not to highlight differences between groups if those differences are not statistically significant once sampling error is taken into account.
(Cont'd)
(Cont'd)
Reply | Report Abuse | Link to thisSeife erroneously claims that “Pew left out one category altogether: those who believe ‘nothing in particular,’” and he implies that people in this category should have been analyzed together with atheists and agnostics as part of one large group. In fact, those who describe their religion as “nothing in particular” were not left out. Results for this group are presented repeatedly throughout the report, as are results for the total “unaffiliated” population, which is the term we use to describe the umbrella group Seife describes (i.e., atheists and agnostics combined with those describing their religion as nothing in particular).
We also know, however, that the religious, political and social profile of self-described atheists and agnostics is quite distinct from the profile of those who describe their religion as “nothing in particular.” For example, our 2007 Religious Landscape Survey found that fewer than one-in-five atheists and agnostics (18%) say religion is somewhat or very important in their lives. Among those whose religion is “nothing in particular,” nearly half (48%) say religion is at least somewhat important to them. Therefore our report looks at atheists/agnostics and those describing their religion as “nothing in particular” both separately and in combination. This is the same approach we follow for Protestants and Catholics (whom we analyze both as a whole and broken into subgroups).
Seife is correct that smaller samples yield less precise estimates than larger samples. And that is why our report includes information about the margin of error for the full sample and for the religious groups analyzed in the report. But Seife’s assertion that “Pew managed to reach very few atheists and agnostics” is misleading. In fact, the survey reached many more atheists and agnostics than a typical survey would, because we oversampled the group to have enough cases (more than 200) to generalize from.
More details on the survey’s findings and methodology are available on the Pew Forum website (www.pewforum.org), along with a variety of other resources on religion and public life; we hope Scientific American readers will visit, explore and judge our work for themselves.
Gregory A. Smith
Senior Researcher
Pew Research Center's Forum on Religion & Public Life