Posts
An exhaustive investigation of the methods used in the analysis of Iraqi deaths conducted in 2006 and published by The Lancet has uncovered many serious violations of accepted ethical practices and possible data fabrication.
“Mortality after the 2003 invasion of Iraq: a cross-sectional cluster sample survey”, by Gilbert Burnham, Riyadh Lafta, Shannon Doocy, and Les Roberts, estimated that, due to the US-led invasion of Iraq, over 650,000 people died between March 2003, when the invasion began, and July 2006. The study has been subjected to wide-ranging and prolonged criticisms, some of which have been discussed at this blog.
Michael Spagat of Royal Holloway College, University of London, has posted a critical review at his website. Here is the abstract:
I consider the second Lancet survey of mortality in Iraq published in 2006. I give evidence of ethical violations against the survey’s respondents including endangerment, privacy breaches and shortcomings in obtaining informed consent. Violations to minimal disclosure standards include non-disclosure of the survey’s questionnaire, data-entry form, data matching anonymized interviewer IDs with households and sample design. I present evidence suggesting data fabrication and falsification that falls into nine broad categories: 1) non-disclosure of key information; 2) implausible data on non-response rates and security-related failures to visit selected clusters; 3) evidence suggesting that the survey’s figure for violent deaths was extrapolated from two earlier surveys; 4) presence of a number of known risk factors for interviewer fabrication listed in a joint document of American Association for Public Opinion Research and the American Statistical Association; 5) a claimed field-work regime that seems impossible without field workers crossing ethical boundaries; 6) large discrepancies with other data sources on the total number of violent deaths and their distribution in time and space; 7) two particular clusters that appears to contain fabricated data; 8 ) irregular patterns suggestive of fabrication in claimed confirmations of violent deaths through death certificates and 9) persistent mishandling of other evidence on mortality in Iraq presented so as to suggest greater support for the survey’s findings from other evidence than is actually the case.
Ethical violations are identified based on a comparison of the Lancet study’s survey methodology with the Code of Professional Ethics & Practices recommended by the American Association for Public Opinion Research (AAPOR). The Code reads, in part,
II. Principles of Professional Responsibility in Our Dealings With People
. . .
D. The Respondent:1. We shall avoid practices or methods that may harm, humiliate, or seriously mislead survey respondents.
2. We shall respect respondents' concerns about their privacy.
3. Aside from the decennial census and a few other surveys, participation in surveys is voluntary. We shall provide all persons selected for inclusion with a description of the survey sufficient to permit them to make an informed and free decision about their participation.
4. We shall not misrepresent our research or conduct other activities (such as sales, fund raising, or political campaigning) under the guise of conducting research.
Prof Spagat argues that the Lancet study authors violated all of those principles, potentially endangering the physical safety of survey respondents.
III. Standards for Minimal DisclosureGood professional practice imposes the obligation upon all public opinion researchers to include, in any report of research results, or to make available when that report is released, certain essential information about how the research was conducted. At a minimum, the following items should be disclosed.
1. Who sponsored the survey, and who conducted it.
2. The exact wording of questions asked, including the text of any preceding instruction or explanation to the interviewer or respondents that might reasonably be expected to affect the response.
3. A definition of the population under study, and a description of the sampling frame used to identify this population.
4. A description of the sample design, giving a clear indication of the method by which the respondents were selected by the researcher, or whether the respondents were entirely self-selected.
5. Sample sizes and, where appropriate, eligibility criteria, screening procedures, and response rates computed according to AAPOR Standard Definitions. At a minimum, a summary of disposition of sample cases should be provided so that response rates could be computed.
Prof Spagat argues that every single one of those professional standards was violated as well.
Prima facie evidence of data fabrication is also presented, including a showing that the study incorporated almost all of the risk factors listed in “Interviewer Falsification in Survey Research: Current Best Methods for Prevention, Detection and Repair of Its Effects”, a document of the AAPOR and the American Statistical Association. Analysis of study findings supports the contention that some data appear to have been falsified.
The Spagat paper’s documentation of shoddy and questionable practices is simply shocking. In my view, it constitutes a damning indictment of the Lancet study. Containing little technical statistical analysis, it should be accessible to interested non-specialists. Although almost 50 pages long, it is well worth a read.
Prof Spagat concludes with a call for a formal investigation of the study. One certainly hopes that would be possible but, in view of the stonewalling by study authors—in particular, manager of survey operations Riyadh Lafta—that would seem a faint hope.
Previous related posts:
Spagat’s speculations have all been successfully refuted, many of them quite a few months ago. He tips his hand from the outset, where he charges ethical violations on the basis on the funding source—it was MIT, not Open Society Institute (Soros), and this fact is widely known. I commissioned the study, and this was told to Neil Munro, who ignored such facts, among many others, disproving his claims of bias.
See: http://www.johntirman.com/Bombs%20Away%20-%20a%20dull%20hatchet%20job.pdf
Spagat’s “main street bias” is easily refuted by the demographic data, the supposed fabications are all gross speculations. The study has perhaps been the most closely vetted of all time. Only the right wing ideologues, embarrassed by the carnage of the war, are taking issue. And, oddly, the alternative they often cite—the “new” Ministry of Health survey published last month in NEJM, also demonstrates 400,000 excess deaths as of 20 months ago—now that would be around 600,000.
Remarkably, there is considerable congruence between the five surveys to date, and differences are not surprising given how difficult gathering such data is. See http://mit.edu/humancostiraq/ There is also remarkable congruence with other incovenient facts—4.5 million displaced, >500,000 new widows, etc.
The Lancet articles were courageous, professional, and highly regarded attempts to estimate the true costs of this war. None of the criticisms on method or political bias have stuck, despite obsessive attempts to do so. The authors, and particularly the field researchers, should be applauded for bringing this important information to the attention of the world.
Lancet authors Burnham and Roberts have said that some of the funding for the project came from George Soros’s Open Society Institute. Are you now claiming that OSI provided no funding? You need to talk with Burnham and Roberts about that. Furthermore, Spagat is not alleging bias because Soros funded the project; he is pointing out that failure to disclose that fact violates an accepted ethical standard.
If you really think this “study has perhaps been the most closely vetted of all time”, well, you need to get out more. How can it be properly vetted when the authors refuse to provide the full dataset and a complete methodological description to other social scientists?
Demographic data alone are insufficient to prove or disprove main street bias. Until a full and unambiguous survey methodology is forthcoming, that and other questions will remain unanswered.
Many of your other points are covered in depth in the Spagat paper I quoted and linked to. That, and your failure to mention the many ethical violations he adduced (except for the Soros connection the import of which, as I pointed out, you appear to misunderstand) causes me to wonder whether you even read it. In particular, Spagat spends several pages showing that your claim of “considerable congruence between the five surveys to date” is incorrect.
Your reference to “right-wing ideologues” reveals your own political bias. As Spagat and other critics (myself included) have said repeatedly, the issue is the Lancet paper’s scientific validity. Is every one of the critics motivated by political support for the current American administration? In my case, you would have a very hard time showing pro-Bush bias from what I’ve written at this blog. Furthermore, that is an ad hominem argument: If you can’t disprove the arguments, attack the individuals making them.
“Spagat’s “main street bias” is easily refuted by the demographic data,”
“The demographic data”? Is Tirman serious (btw, I believe Tirman was himself one of the MIT funders of the Lancet study)? The Lancet study doesn’t have demographic data, which is one of its many problems. It only collected gender info, and even that was not consistently collected, as with the Al-Tameem cluster discussed by Spagat where, supposedly two separate field teams both simultaneously forgot to collect even this limited demographic data for all 40 houses.
Such data could not refute the Main Street Bias issue. Disclosing a sample design might help, but the Lancet authors haven’t done that. What they have done is apparently issue contradictory statements muddying the issue of their sample design, which doesn’t help.
“the supposed fabications are all gross speculations.”
It looks to me like Spagat has identified quite a lot of evidence for fabrication based on a rather rigorous analysis of a wide range of highly dubious patterns in the data which suggest fabrication. The Al-Tameem issue above is one, and is a pretty good indicator that somebody probably just shoddily curb-stoned that cluster. A “closely vetted” study would have noticed right away that separate teams supposedly all simultaneously forgot to collect the gender info in this place, seen a red flag and gone back to re-survey. (Of course, since the field teams were allowed to destroy the details of where they surveyed, the authors would be unable to do any such checks on veracity, as the field teams would have been well aware throughout.)
Putting all of this alongside the fact that the survey had no defenses against fabrication at all, and alongside all the spin-doctoring of facts in the Lancet paper, as listed toward the end of Spagat’s piece, it looks like this study was cooked and should be investigated thoroughly.
I also found the graphic in 3.1 of Spagat’s piece rather interesting. What do you make of this one StatGuy? Just another big coincidence?
The allegation of political bias began with Munro’s falsification of information given to him and Cannon on two separate occasions. The OSI did not fund the survey; this was funded out of internal MIT funds. Period. This has been widely discussed since the NJ hatchet job. OSI contributed, more than six months after the study was commissioned, to public education aspects, and this also came to MIT, not the authors of the Lancet.
Spagat’s allegations have either been refuted publicly, including main st bias (demographics show enormous prevalence of young men as victims, showing how irrelevant the msb argument is), ethics violations (vetted by JHSPH), etc. Use of IBC “data” is ludicrous, akin to collecting crime statistics in the US using only Spanish language newspapers. Also completely misrepresents to MoH data.
If it walks like a duck…..
Tirman seems to be babbling incoherently now.
How exactly can data showing more young male victims “refute” MSB? It seems to be the case that young males are killed disproportionately in Iraq, but MSB is about the study favoring more violent areas because of its non-random street selection procedures. If that bias is there it would tend to show up in inflated numbers, not in victim genders. So much for Tirman’s fanciful “refutation”.
The analogy for IBC seems almost painfully stupid. Then there’s this unexplained accusation of misusing MoH data. Maybe it’s not worth asking. In Tirman-land there’s probably a “remarkable congruence” between MoH and Lancet data.