Magic Statistics

A few days ago, the Washington Post headlined a report on a survey of bloggers: "Cyber-Catharsis: Bloggers Use Web Sites as Therapy". Here’s the crucial quote: "Nearly half of bloggers consider it a form of therapy, according to a recent survey sponsored by America Online Inc." Then the article seems to go off on a tangent, discussing the theory and practice of blogging-as-therapy programs. But it never returns to the AOL-sponsored survey that provided the basis for the whole article.

The press release summarizing the results of the survey is here. Another summary is posted at an AOL website.

Some questions have been raised about the soundness of the survey. How valid is the survey, and how accurate are the results? A webmeister at blogs4god is wondering about this because, in his experience, very few bloggers blog for self-therapy of some kind. Most bloggers that he follows seem to be driven by other, shall we say, more altruistic, motives. By that, I mean a desire to share one’s knowledge, expertise, or opinions with ‘net surfers. For myself, this is my experience of blogging as well. My blog, and the blogs listed in my sidebar, focus on information, expert or other special knowledge, and/or commentary on current events of various kinds. Therapy doesn’t enter into it at all, as far as I can tell.

However, that's just the impression of two bloggers. A well-known saying among statisticians is: "The plural of anecdote is not data." A story or two regarding what one has personally observed does not provide a valid basis for statistical analysis. So, the fact that some or even many bloggers do not agree with the results of the AOL survey does not prove the survey wrong.

Among the necessary elements of a statistically valid survey is a sample drawn randomly from a sample frame that encompasses all of the observations one wishes to makes inferences about. The technical term for the latter—all the observations one wishes to make inferential statements about—is "universe". Ideally, the universe and sample frame—the collection of individual observations available to draw from–should coincide. (In the real world, that’s not always possible; but in any case the sample frame, in every relevant characteristic, must be representative of the universe. More about the real-world sample frame later.) If one wishes to make a statement about, say, political opinions of Canadians, the sample frame cannot be limited to residents of Ontario; it would have to encompass the universe in question: residents of all the provinces and territories of Canada.

I hope that wasn’t too tedious. Now, how do those foundational considerations impinge on the AOL survey of bloggers?

To take a random sample means that all observations in the universe under investigation have an equal chance of being selected for inclusion in the survey. That is the way to ensure that the sample truly represents the entire population one wishes to study. With respect to the AOL survey of bloggers, the universe under study was comprised of all bloggers. Did the sample frame assembled for the survey encompass all bloggers? No, it didn’t; I don’t think AOL anywhere claimed that it did. They used a sample frame containing only some bloggers. So, the question becomes: Did the sample frame represent, in every relevant characteristic, the universe of all bloggers? The relevant characteristics are those that the survey sought to examine and reported on: reasons for blogging, reasons for not blogging, intended audience of bloggers, reasons for reading others’ blogs, and similar questions about attitudes toward blogging.

So, we need to look more closely at the AOL survey’s sample frame. Where and how was a list of bloggers, with contact information, assembled and used in the survey? This is the sum total of information on the survey’s methodology:

The survey was conducted from July 17-22, 2005 through Opinion Place, an online research site operated by DMS. Opinion Place utilizes real-time, randomized assignment of respondents to surveys based on a scientifically process [sic]. A total of 600 Web users completed the survey. The screen qualifications for this study were males and females aged 18 and above who write one or more blogs on the Web.

It says the survey was "randomized", so that sounds good, but important details are not provided: They didn't say exactly how they found the bloggers they surveyed.

No further information on either DMS or Opinion Place was provided, but a minute or two with Google and their websites are located: Digital Marketing Services and Opinion Place. DMS runs online surveys through Opinion Place. At this page, we finally find some information relevant to an assessment of survey validity.

How are respondents drawn into Opinion Place? Respondents become aware of Opinion Places through an extensive promotion campaign that reaches 79% of household internet traffic.

So, internet users have to navigate to and register at Opinion Place to take part in DMS's surveys. This is very bad news for the validity of AOL’s survey. The survey sample frame is composed of what is known as a self-selected sample. Only those who actively and deliberately choose to be included in the sample frame can be selected to take the survey. Internet users who decline to visit and register at Opinion Place have zero chance of being surveyed. It is true that Opinion Place did not administer the survey to all who registered at their site. Only those aged 18 and over who write on blogs were included in the pool of those to be given the survey. But the crucial fact is that you had to register even to be considered for the actual survey.

DMS attempts to circumvent this criticism by saying:

How are respondents selected for Opinion Place surveys?

Selection for a survey is a complex, random process. Multiple points of survey selection ensure that respondents cannot control or manipulate the process of survey assignment.

But, of course, these respondents do control and manipulate the survey in the most fundamental way possible: They themselves choose to be considered for inclusion.

Still, DMS may argue that the sample frame, despite being self-selected, does, in every relevant characteristic, indeed represent the universe under study. But that doesn’t fly, either. Look at it this way: What kind of blogger is more likely to register to take surveys at a site like Opinion Place—one who is motivated to blog as a form of therapy or one who is motivated to blog in order to share information or knowledge? Unless someone can offer a compelling counter-argument, it seems obvious to me that the former kind of blogger is far more likely to want to take part in surveys. If this is the case, then the sample frame is strongly biased toward those who blog for self-therapy, and therefore the results cannot be generalized to the entire universe of bloggers. The survey results would speak only for the 600 bloggers who actually responded.

Given the method of selecting survey respondents, one could argue that the blogger survey is about as valid as those online polls you see at news sites. Except that those polls carry a disclaimer something like this one from CNN's QuickVote:

This QuickVote is not scientific and reflects the opinions of only those Internet users who have chosen to participate. The results cannot be assumed to represent the opinions of Internet users in general, nor the public as a whole.

UPDATE: dean at blogs4god asked me for a summary of this item to post at his site. So here 'tis.

The film version of The Lion, The Witch & The Wardrobe won’t be out until 9 December, but already liberals are becoming unhinged because of it. Last week it was Liam Lacey warning us about "hidden" Christian messages in the film. Now it's Frank Cerabino's turn to take leave of his senses. In a column appearing in the Tallahassee Democrat, he accuses the state of Florida of "blatantly" promoting Christianity because it’s encouraging school children to read the book. Not only that, state governor Jeb Bush is personally implicated:

When you can combine the forces of Disney, the McDonald's Happy Meal and Gov. Jeb Bush in one tidy package - all working together to cram thinly veiled Christian theology down the gullets of Florida's schoolchildren - you've got yourself a hell of a plan.

"Cram" Christian theology down their little "gullets", you say? How charming.

But then, a few paragraphs later, Cerabino writes: "I have read The Lion, the Witch and the Wardrobe as a bedtime story to my son". So, Mr Cerabino, you admit cramming thinly veiled Christian theology down the throat of your own child. You should be ashamed. Someone call Florida’s child protection agency—right now—before he does worse damage to his own son, say, telling him that Narnia author C.S. Lewis regularly went to church to worship Jesus. The horror!

Mr Cerabino thinks Narnia’s theology is "thinly veiled"—or maybe not. He also says the film is "a way to subtly introduce the Christ story to young people". So, which is it: thinly veiled or subtle? Or maybe it’s thinly veiled subtlety?

As Blogger Jinx McHue at Shock & Blog says, "You can’t make this stuff up".

I see a trend here. Two liberals in two weeks have lost all perspective—not to mention common sense—over The Lion, The Witch & The Wardrobe. It will probably only get worse as the film’s release date approaches. Get ready for an avalanche of liberal foolishness.

via Wizbangblog.

The morning dawned a bright, clear, crisp, late fall day here in Whitehorse. Late fall often has a lot of mornings like this, so it can be a beautiful time of year, just before the snow flies. The temperature this morning around 7:00 am was -6 Celsius. Vehicles parked outside overnight had a heavy frost, as did the lawns. Since I’m an early riser, I get to enjoy these fine-looking mornings along with some peace and quiet.

I drove downtown around 8:30 am and stopped by the Yukon River. On mornings like this, the mist rises from the river. Here's what it looks like:

This scene is right across the street from our local Wal-Mart parking lot.

Of all the Wal-Marts in the world, I’ll bet the Whitehorse store has one of the best views.