The heat is turned up one more time on the 2006 study published in The Lancet that claimed that the US-led invasion of Iraq had led to the deaths of over 650,000 Iraqis who would not have died otherwise. The study by Gilbert Burnham, Riyadh Lafta, Shannon Doocey, and Les Roberts, received massive publicity from mainstream media but soon became mired in controversy as scientists and other experts raised pointed questions concerning the study’s survey methodology, field operations, and analytical techniques.
The current issue of National Journal carries an in-depth investigative report by Neil Munro that details the controversy and explores crucial problems with the study.
Over the past several months, National Journal has examined the 2006 Lancet article, and another [PDF] that some of the same authors published in 2004; probed the problems of estimating wartime mortality rates; and interviewed the authors and their critics. NJ has identified potential problems with the research that fall under three broad headings: 1) possible flaws in the design and execution of the study; 2) a lack of transparency in the data, which has raised suspicions of fraud; and 3) political preferences held by the authors and the funders, which include George Soros's Open Society Institute.
Several of my previous blog posts criticising the 2006 study have focused on points 1 and 2. For example, this:
The study’s description of field operations indicates that the survey was administered more quickly and more smoothly than any comparable survey I’ve ever heard of. A team of two surveyors appeared unannounced at the front doors of 1849 Iraqi households asking highly sensitive and intrusive questions about everyone who had lived there since January 2002. Interviews were successfully completed in an average of only fifteen minutes each. We are told that interviewers found family heads at home in all but 16 households. What’s most mind-boggling (to me, anyway) is that only 15 households refused to participate—a refusal rate of only 0.8%.
To answer these and related questions, the Lancet authors need to release the original survey data and field reports to other social scientists and statisticians for independent assessment. This they refuse to do.
Still, the authors have declined to provide the surveyors' reports and forms that might bolster confidence in their findings. Customary scientific practice holds that an experiment must be transparent — and repeatable — to win credence. Submitting to that scientific method, the authors would make the unvarnished data available for inspection by other researchers. Because they did not do this, citing concerns about the security of the questioners and respondents, critics have raised the most basic question about this research: Was it verifiably undertaken as described in the two Lancet articles?
"The authors refuse to provide anyone with the underlying data," said David Kane, a statistician and a fellow at the Institute for Quantitative Social Statistics at Harvard University. Some critics have wondered whether the Iraqi researchers engaged in a practice known as "curb-stoning," sitting on a curb and filling out the forms to reach a desired result. Another possibility is that the teams went primarily into neighborhoods controlled by anti-American militias and were steered to homes that would provide information about the "crimes" committed by the Americans.
In view of the alacrity with which survey operations were completed and data collected, curb-stoning is a real possibility. Deliberately targeting anti-American respondents is, in my view. less likely but, as long as the survey forms are withheld, such scenarios cannot be ruled out.
Fritz Scheuren, vice president for statistics at the National Opinion Research Center and a past president of the American Statistical Association, said, "They failed to do any of the [routine] things to prevent fabrication." The weakest part of the Lancet surveys is their reliance on an unsupervised Iraqi survey team, contended Scheuren, who has recently trained survey workers in Iraq.
That assessment is spot on. It seems to me that manager of survey operations Riyadh Lafta is the key to this whole puzzle. He has consistently refused to respond publicly to criticisms; indeed, he has stonewalled efforts to substantiate or corroborate the work of his surveyors. When questioned about Lafta’s contribution to the study, his co-authors simply proclaim their total and unqualified faith in his work.
Asked if he remains certain that Lafta's Iraqi teams truly collected the data they turned in, Roberts answered, "I'm just absolutely confident this data is not fabricated."
It is increasingly clear that Roberts has no basis for his confidence.
There is much more in the National Journal, so interested readers are encouraged to read the whole thing. Check out the articles in the sidebar, too.
Previous related posts:
Posts
The problem of marriages of underage girls was highlighted last month by UNICEF's photo of the year (at right) showing a 40-year-old Afghan man sitting beside his 11-year-old bride-to-be.