The scientists who provided the information behind my post “’Main street bias’ in Lancet study” have now come up with an equation providing a framework for estimation of the extent of bias in the Lancet study on Iraqi deaths. As discussed in that post, the Lancet study published in October included only urban residents of Iraq and, more specifically, only people living on a residential street crossing a “main street”. From page 2 of the pdf version of the October Lancet article:
The third stage consisted of random selection of a main street within the administrative unit from a list of all main streets. A residential street was then randomly selected from a list of residential streets crossing the main street. On the residential street, houses were numbered and a start household was randomly selected. From this start household, the team proceeded to the adjacent residence until 40 households were surveyed.
People living on residential streets not crossing a “main street” therefore could not have been surveyed. (In statistical survey jargon, such streets, and their residents, were “out of scope”.) It seems likely that residential streets crossing main streets would be scenes of more deadly attacks than away-off-the-main-drag side streets. Residents of streets crossing main streets would face substantially greater probability than residents of other streets of being killed by car bombs or other violent events. These observations point to the conclusion that the Lancet study utilised a sample that was biased toward an overestimate of deaths in Iraq.
(The study estimated that, since the US-led invasion in March 2003, almost 655,000 Iraqis have died who would not have died had the invasion not occurred.)
Neil F. Johnson, Michael Spagat, Sean Gourley, Jukka-Pekka Onnela, and Gesine Reinert have written a paper entitled “Bias in epidemiological studies of conflict mortality”, containing a mathematical specification of that source of bias. An abstract is posted here and the full document here (pdf). Mr Spagat is on faculty at Royal Holloway, University of London, and the other four are associated with the University of Oxford. They plan to submit the paper to an academic journal for publication.
For the full monty with mathematical equations and charts, read the pdf paper. I’ll try to explain it briefly here.
The authors single out four parameters as of crucial significance for precise estimation of bias in the Lancet study’s estimate of excess deaths. First is the ratio of the probability of death faced by those living in residential streets crossing main streets compared to that faced by those living in other residential streets. They designate this ratio by the letter “q”. They argue that this parameter is certainly greater than 1, and probably closer to, say, 5, because targets of terrorist and other attacks would be found on main streets to a much greater degree than on more remote streets.
Secondly, the ratio of out-of-scope residential streets to in-scope residential streets. After looking at maps of Baghdad generated via Google Earth, examples of which are reproduced in the paper, they argue that a reasonable value for this ratio is 10. That is, there appear to be ten times as many residential streets that do not cross main streets as those that intersect main streets. This parameter is designated by the letter “n”.
Thirdly, the probability that a resident of an out-of-scope residential street is present in an out-of-scope residential street; and fourthly, the probability that a resident of an in-scope residential street is found in an in-scope residential street. These parameters are designated “fo” and “fi” respectively. The authors argue that, because of the dangers of traveling far from home in war-torn Iraq, both parameters are close to one (13/14, to be precise).
It is very interesting to note that substantiation for the latter parameter value was provided in the Lancet article itself, where it was reported that the surveyors were able to contact a head of household in a selected house 91.9% of the time. In only 0.9% of houses was no resident found at home when the surveyors came by to ask their questions. Not only were residents of in-scope streets somewhere within the in-scope area, they were in their houses. That constitutes very striking support for that part of the Oxford-Royal Holloway experts’ work.
Given the suggested values for the four parameters—q=5, n=10; fi=13/14, and fo=13/14—the equation specified by the Oxford-Royal Holloway experts generates a value of 2.95 (or 3 with rounding). The survey that formed the basis for the Lancet estimate of increased deaths would thus be expected to yield a very large overestimate.
The effect on bias of alternative values for the crucial parameters can be tested by plugging them into equation (1), found on page 2 of the pdf paper.
A website has been set up at Royal Holloway, University of London, containing a wealth of material related to the Lancet study. In addition to the new bias specification paper, there are links to the controversial Lancet study on Iraqi deaths, maps of Iraq, links to earlier epidemiological studies using cluster sample methodology and discussion of same, information on urban homicide rates around the world, and other related materials. Check it out!
My thanks to Michael Spagat for sending me the links.
Previous related posts:
Posts
