Ruth Gledhill’s Times of London report on a newly released public opinion poll is headlined, “Religious people back gay rights”. I would argue, however, that the poll does not necessarily reflect the views of all British religious adherents, but only of the 2009 respondents who participated in the survey.
Most religious people want equal rights for lesbians and homosexuals, according to a survey to be published today.
. . .
The research into public attitudes to homosexuality was commissioned by Stonewall, the gay rights group, and conducted by YouGov. More than 2,000 people were questioned for the report.
The British pro-homosexual news service Pink News trumpet the survey findings here and here. Near the bottom of the latter post, one finds a bare-bones outline of the survey’s methodology, stating that YouGov selected respondents from its database of individuals who had previously registered to serve as survey participants. Thus, the sample was self-selected, which statistics experts view as a fatal flaw for the reliability of survey results.
In order for a survey to provide a sound foundation for making valid inferential statements about a larger population, the survey respondents must constitute a sample that is representative of that larger population (known as the “universe”). The statistically accepted method for assembling such a representative sample is random selection.
For the survey at hand, the universe is all Britons. In order to make valid inferences about Britons in general, the sample must be drawn randomly from the universe of Britons. To be drawn randomly means that every individual in the universe has an equal chance of being included in the sample. That is the accepted methodological understanding of survey procedure, which YouGov forthrightly rejects. See the essay by Chairman Peter Kellner entitled “Down with Random Samples”, posted at YouGov’s website. (The essay is confused and fundamentally misguided, in my view, but a full discussion here would take us too far afield.)
YouGov admits that its survey method does not entail a random selection from the universe of Britons, but maintains that a random selection from its database of 110,000 volunteers, combined with appropriate weighting of respondents, is tantamount to a random selection from the entire population of Great Britain.
The company also claims a track record of successfully estimating the overall popular vote in recent elections, even though it called John Kerry to win the 2004 American presidential election and prompted Tony Blair to muse about government curbs on pollsters after it seriously overestimated support for the United Kingdom Independence Party.
YouGov’s sample frame excludes the 40-50% of Britons who do not have access to the internet, imparting an obvious socio-economic bias. Moreover, its database almost certainly includes an over-representation of individuals who are much more concerned than the average Briton with political issues and public affairs, including homosexual rights.
I also notice in the various news reports a peculiar reticence to define “religious”. (YouGov’s own website contains no information on the survey.) The Times headline and first sentence refer to “religious people”, as do the reports at Pink News. The Guardian story includes this cryptic sentence:
The YouGov survey of 2,000 Britons shows that "people of faith" were as likely to support gay equality as members of the wider population.
Yet none of the available sources includes YouGov’s questionnaire, nor do any specify how YouGov identified “religious people”/”people of faith”. Since the news spin focuses on attitudes of religious people, this is a major reporting gap that prevents intelligent assessment of the alleged findings.
Compounding the difficulties of interpreting the survey results, none of the news sources includes estimates of confidence intervals or margins of error. A technical methodological report would also include a full discussion of the weighting scheme used, but none is available.
h/t for Pink News links: Anglican Mainstream, which see for a critical perspective of the assumptions behind YouGov’s survey.
Previous related posts:
Posts
IMAO has been publishing little-known facts about Fred Thompson every day since mid-March. Here is a small selection: