Posts
That was my initial reaction when my friend Tom Gilson told me that the latest issue of Skeptic magazine features an article extolling Gregory S. Paul’s paper on social problems and religious belief. Mr Paul’s study, published last year in the Journal of Religion and Society, is entitled "Cross-National Correlations of Quantifiable Societal Health with Popular Religiosity and Secularism in the Prosperous Democracies”; the html version is posted here and the pdf version here.
Mr Paul’s paper presents itself as a statistical investigation of social problems—including homicide, prevalence of sexually transmitted diseases, abortion, pregnancy among young teenagers—and religious belief. He collected data from eighteen democracies and claimed to have found positive correlations between social dysfunction and religious belief, thus purportedly showing that religiosity is not necessarily beneficial and secularism is not inherently harmful to social mores.
One year ago next week, I wrote one of my most popular blog posts critically assessing Mr Paul’s paper and showing it to be statistically invalid. I concluded that, in my professional judgment as a statistician, the study should not be taken seriously.
Now comes the Skeptic article, by one Matthew Provonsha, who merely reiterates—in the most uncritical fashion imaginable—the errors and shortcomings in Mr Paul’s original work. For Provonsha, evidently, Paul can say no wrong. (Why was this published in a magazine called Skeptic?)
Thus, Mr Provonsha does not wonder why Mr Paul fails precisely to document his data sources or provide sound reasons for his selection of countries to be included.
The study focuses on the prosperous democracies, because “levels of religious and nonreligious belief and practice, and indicators of societal health and dysfunction, have been most extensively and reliably surveyed” in them. Also, “The cultural and economic similarity of the developed democracies minimizes the variability of factors outside those being examined.” With a database of 800 million people, this study is far more reliable than results based on smaller sample sizes used in other such studies. The data are also current and extensive, collected in the middle and latter half of the 1990s and early 2000s from the International Social Survey Programme, the UN Development Programme, the World Health Organization, Gallup, and other well-documented sources.
From a statistical perspective, the criteria Paul used to include his eighteen chosen countries and omit all others are arbitrary and statistically unsupported, but Provonsha falls for it. Also, Paul does not document his data sources, in that we don’t know from which particular source he obtained each of the various measures of social health.
The claim that Paul utilised “a database of 800 million people” is laughable. The sample size of Paul’s study was precisely eighteen: one data point from each country for each data series. To claim this represents the combined population of all the countries is like a market research firm conducting a survey of 1200 Canadians and then claiming they accessed a database of over 32.5 million people.
I do want to thank Mr Provonsha for re-drawing some of Mr Paul’s charts. They are now larger and much easier to read. Their statistical value, however, is still negligible. Paul seemed to hold the belief that plotting points on charts constitutes statistical analysis. Not in my world. Paul and Provonsha should both read Darrell Huff’s classic, How To Lie With Statistics, which devotes an entire chapter to the perils of graphs.
Visual representations of data, e.g., plotting data points on a chart, can be valuable as aids to understanding, but they are no substitute for calculating correlation coefficients and other analytical statistics. Paul did not calculate any analytical statistics—not one—and Provonsha doesn’t call him on it. Paul’s paper has many descriptive statistics, but no actual statistical analysis, and Provonsha apparently didn’t even notice. That says it all.
h/t: Tom Gilson, Thinking Christian
Other recommended commentary on Gregory Paul’s study:
- “Dogma Bites Man”, by George H. Gallup, Jr. at Touchstone Magazine
- “Debunking Gregory S. Paul”, by my friends John and Scott at Verum Serum
Previous related posts:
Scott, I missed that reference to 800 million people. It’s absolutely preposterous.
This Skeptic magazine, and the like-named Society, are headed by Michael Shermer, who spends a lot of time going around saying Intelligent Design is not science. He could stay closer to home and discover a great example of non-science in his own magazine. But that wouldn’t fit his pre-determined conclusions.
[...] Do you remember the Gregory S. Paul study? Sure you do. It was that study that purported to show that there is a correlation between the religious belief and the dysfunction of a society, but it was shown to be riddled with errors by statistican Scott Gilbreath as well as by my fellow Telician Mike Gene. Now, none of this has prevented the “study” from being featured in Skeptic Magazine, where Matthew Provonshan uncritically repeats Gregory Paul’s flawed claims. This has caused Scott Gilbreath to write another post about the article, finding even more howlers than he did at first. The claim that Paul utilised “a database of 800 million people” is laughable. The sample size of Paul’s study was precisely eighteen: one data point from each country for each data series. To claim this represents the combined population of all the countries is like the market research firm Ipsos Canada conducting a survey of 1200 Canadians and then claiming they accessed a database of over 32.5 million people. [...]
[...] Krauze quotes Scott Gilbreath and MikeGene as pointing out serious, not to say fatal, flaws in the study. And to be honest, they may have a point. I would like to see a more rigorous study of this data carried out. [...]
I am beginning to suspect that this article will be around for years, unfortunately.
Part of the reason, I think, is a confusion on the part of many of its supporters. Most of those who like Paul’s conclusions like it because they want to say that there is no clear link between secularism and social dysfunction. If someone thinks this is probably true, they can do two different things: try to show that there are philosophical reasons for doubting such a link (e.g., reasons that have to do with ethics itself); or try to show that there are empirical reasons for doubting such a link. The Paul study purports to be a first attempt at the latter; but often when people are faced with the problems of the study, they retreat to the more purely philosophical reasons, and treat those not merely as reasons to think the desired conclusion right, but also as reasons to think Paul’s particular argument right. Paul’s reasoning gets conflated with a more reasonable argument (in the sense that people could reasonably be convinced by it, whatever its ultimate merit), and we get an oscillation: Paul’s argument is supposed to provide empirical reasons for the conclusion, and they must be good reasons because they lead to the right conclusion (as shown by some other, more philosophical argument). This is all sophistry, of course, but it’s hard to eradicate.
Of course it’s often slightly more subtly expressed than that. We find a related version of the oscillation in Paul’s own article: he starts out claiming that he’s looking at whether secularization is inconsistent with society’s possession of the moral foundations necessary for a healthy society. This puts it entirely in philosophical territory (i.e., whether secularization is a means rationally consistent with certain social goals). He then shifts over to muddling about with the statistics in order to argue that belief in the creator does not benefit “national cultures” — which is a purely empirical issue. Then he concludes that “The widely held fear that a Godless citizenry must experience societal disaster is therefore refuted.” (”A Godless citizenry must experience societal disaster,” however, is a stronger claim, and much easier to criticize, than the claim that godlessness is not consistent with the goal of establishing moral foundations for a healthy society.) Depending on what societal disaster he means this could either be an ethical claim about consistency of means and ends again (which it would have to be to answer the question he began with), in which case it’s not supported by the argument; or an empirical conclusion about the apparent consequences as they currently stand. But if it’s the latter, it doesn’t require any of the statistics he actually looks at, since all you’d have to do is look to see whether there are any societies that are (1) godless; and (2) not experiencing societal disaster. (This is, in fact, virtually admitted by Paul himself, since he supports the conclusion by pointing out that there are secular societies that “are clearly able to govern themselves and maintain social cohesion”. Now, you don’t have to look at homicide rates to know that the Swedes are able to govern themselves and exist as a coherent society, since the existence of Sweden is proof of both. But this doesn’t tell you much at all about how to answer the original question.) And anyone taken in by that is likely to make the same mistake themselves.
[...] RECYCLED MANURE, still smelly: “I can‚Äôt believe that atrocious article is still being touted” …. (magicstatistics) [...]
[...] I can’t believe that atrocious article is still being touted [...]
[...] This new research is yet another nail in the coffin of the claim, made by Gregory S Paul and Skeptic magazine among others, that secularism is good (or, at least, not bad) for society. [...]
[...] I can’t believe that atrocious article is still being touted [...]
[...] To add a note of surrealism, a commenter at Christian Post absurdly recommends Gregory S Paul’s thoroughly debunked study purporting to show that religious belief is detrimental to societal health. The same commenter even more absurdly recommends the credulous article in Skeptic touting Paul’s misguided study. [...]