The Lancet article published online 11 October replicated, with a somewhat larger sample size, a 2004 study, also published in The Lancet and also done by researchers from Johns Hopkins Bloomberg Medical School.  Yet the researchers ignored a United Nations survey of Iraqis, conducted about the same time as the first Lancet survey, that found very different results.

The 2004 Lancet article estimated that, between the US-led invasion of March 2003 and September 2004, 98,000 Iraqis died who would not have died had the invasion not occurred.  The estimate had a 95% confidence interval of 8,000 - 194,000 deaths and was based on a cluster sample made up of 33 clusters of 30 households each for a total sample size of 988 households. (In one or two clusters, the full complement of households was not surveyed.)

The United Nations conducted its Iraq Living Conditions Survey (ILCS) between April and August 2004.  As its name implies, the survey was aimed at gathering data on a broad spectrum of indicators of living conditions about one year after Saddam Hussein was deposed.  Measures surveyed related to housing, infrastructure, demographics, child health, nutrition, education, condition of women, labour market activity, income and wealth, etc. Because the ILCS was a much more comprehensive survey, its findings were not published until March 2005, several months after the 2004 Lancet article.

The ILCS also produced an estimate of deaths since the invasion, but it was much lower than that published in the 2004 Lancet article.  The estimate is discussed on p. 55 of the Analytical Report of the ILCS (pdf).

The number of deaths of civilians and military personnel in Iraq in the aftermath of the 2003 invasion is another set of figures that have raised controversy. The ILCS data indicates 24,000 deaths, with a 95 percent confidence interval from 18,000 to 29,000 deaths.

Although the ILCS estimate of 24,000 is far below the 2004 Lancet estimate of 98,000, there is a statistical sense in which they are not inconsistent, for the huge confidence interval of the Lancet estimate (8,000 - 194,000) easily encompasses the ILCS estimate's confidence interval (18,000 - 29,000).  Statistically, however, one would conclude that the ILCS estimate is to be preferred because of its much smaller confidence interval, other things being equal.

The reason the confidence intervals differ so much in width is because the two estimates are based on different sample sizes.  Both surveys used a cluster methodology of sample selection, but with different numbers of clusters and different number of households surveyed within each cluster.  The Lancet survey selected 33 clusters from all of Iraq and then surveyed 30 households within each cluster.  In the event, a total of 988 households were sampled.  The ILCS selected 110 clusters from 17 of Iraq's 18 Governorates, with an additional 330 clusters from the remaining Governorate of Baghdad.  Within each of the 2200 clusters, 10 households were surveyed.  After removing six clusters due to operational considerations, the total sample size was 21,940 households.

The ILCS used over 66 times as many clusters and surveyed over 22 times as many households as did the Lancet survey.  No wonder the ILCS's confidence interval was much more precise.

Based on the accompanying documentation, the ILCS was far superior to the Lancet survey across the whole gamut of survey operations.  I won't go into details here, but those interested are referred to "Appendix 2: Technical Characteristics of the Living Conditions Survey Sample", found on pages 169-170 of the ILCS Analytical Report.

Despite the obvious superiority of the ILCS to the 2004 Lancet survey, the 2006 Lancet article contains no discussion of the ILCS or its estimate of Iraqi deaths between March 2003 and August 2004.

This I found odd.  Articles in academic and professional journals that address topics of controversy generally include references to previously published studies and discuss the perspective the current article takes vis-à-vis the views and findings of those earlier studies.  That is how scientific knowledge advances—by critically engaging published findings of other scholars and specialists.

The authors of the 2006 Lancet article, however, appear uninterested in critical engagement with the ILCS estimate of Iraqi deaths.  Yet we know that the Lancet researchers are aware of the ILCS, for they refer to it twice in their footnotes.  The first page mentions "surveys that assessed the burden of conflict on the population" and the fact that "insufficient water supplies, non-functional sewerage, and restricted electricity supply . . . create health hazards", and for these the ILCS is footnoted.

But as for critical discussion of the enormous difference between the ILCS estimate of deaths and the estimates generated from both Lancet surveys, the authors don't want to touch that.  They don't even acknowledge its existence.

As I said, in my experience scientific knowledge is not built up by ignoring previous relevant studies, especially ones that differ so radically from one’s own study.  That the researchers behind the 2006 Lancet article did so reinforces the belief that their real agenda is not scientific knowledge but advocacy.

Previous related post: Lancet study of Iraqi deaths is statistically unsound and unreliable