Posts
One of the joys of blogging for me is interacting with people I'd never have met otherwise. My posts on the Lancet study of Iraqi deaths (background here) have afforded many opportunities for that. One in particular prompts this post.
On Friday I received an e-mail from Sean Gourley, a physicist at the University of Oxford and Royal Holloway, University of London, who has just co-authored a critical review of the Lancet study. He has graciously allowed me to report on his findings. His fellow researchers on this project are Neil Johnson, also in the Oxford Dept of Physics, and Michael Spagat of the Dept. of Economics, Royal Holloway, University of London.
As I pointed out in this post, the Lancet survey included only residents of urban areas, thus introducing significant bias into the results. Mr Gourley and his co-researchers argue that the survey methodology also excludes many urban residents, making bias problems even worse. The problem is what they call “main street bias”.
The Lancet surveyors selected clusters by randomly choosing administrative units within Iraq’s Governorates in proportion to population. Then:
The third stage consisted of random selection of a main street within the administrative unit from a list of all main streets. A residential street was then randomly selected from a list of residential streets crossing the main street. On the residential street, houses were numbered and a start household was randomly selected. From this start household, the team proceeded to the adjacent residence until 40 households were surveyed.
Only residential streets crossing a “main street” were eligible for selection. Urban areas typically contain residential streets that do not cross a main street; but the methodology ruled them out. Such streets could never be selected for surveying.
The map below, sent by Mr Gourley, shows a section of Oxford, UK. (The traffic circle near the top left corner is just across a short bridge from Magdalen College at the end of High Street, so it is only a few minutes’ walk from the centre of Oxford.) Three main streets are marked by the three black arrows; each street that does not cross one of them is marked by a red arrow. So, if the Lancet methodology were to be implemented in this section of Oxford, there would appear to be hundreds of households who could never be selected for surveying.
Generally speaking, armed conflict is more common in or near main streets than in side streets. Certainly, given typical traffic patterns, conflict on main streets is likely to endanger more people. So, excluding streets that do not cross main streets would tend to result in overestimation of casualties. Thus, "main street bias".
The crucial question becomes: How exactly did the Lancet surveyors define main streets? That question was put to lead author Gilbert Burnham, epidemiologist at Johns Hopkins, by the Oxford-Royal Holloway researchers. According to an article in Science (behind a subscriber wall, but Sean Gourley sent me a copy), Prof Burnham had two different, and apparently contradictory, answers.
Burnham counters that such streets were included and that the methods section of the published paper is oversimplified. He also told Science that he does not know exactly how the Iraqi team conducted its survey; the details about neighborhoods surveyed were destroyed “in case they fell into the wrong hands and could increase the risks to residents.”
Every time I read a Lancet co-author defend that article, it just gets worse. If Prof Burnham doesn’t “know exactly how the Iraqi team conducted its survey”, how can he know whether the methodological description is oversimplified or not?
Not only that, he admits that data have already been destroyed. To call this bad statistical practice is putting it mildly. Statisticians I know could be reprimanded or even lose their jobs if they destroyed data only a few months after a survey, especially one they knew ahead of time would generate public controversy. In my experience, it is standard procedure to store all survey materials in a secure location for an absolute minimum of three years—and, in practice, usually longer.
Speaking of secure locations, it sounds like the Iraqi surveyors didn’t have one. If they really had no safe place to store completed surveys, they should not have gone out and gathered the data—and not just because of the potential consequences for interviewees if confidential information is leaked. No: the real issue here is the professionalism of the surveyors. Professional surveyors and statisticians take whatever steps are necessary ahead of time to ensure that confidentiality will be protected. If they couldn’t do that, they had no business going into the field in the first place.
Now that essential information has been destroyed, there is no way of verifying Burnham’s claim that all streets, not only those crossing main streets, were included in the sample frame. Failing to ensure that data, analysis, and results can be independently verified is another indication of unprofessional statistical practice.
I’m not the only one who’s irritated that the controversy over the Lancet article’s methodology has turned into a circus.
Michael Spagat, an economist at Royal Holloway, University of London, who specializes in civil conflicts, says the scientific community should call for an in-depth investigation into the researchers’ procedures. “It is almost a crime to let it go unchallenged,” adds [Neil] Johnson.
Fred Kaplan at Slate has also had difficulty getting a straight answer from Gilbert Burnham about his study. Mr Kaplan concludes:
It sounds as if he's saying he didn't destroy the data because they never existed in the first place. If that's the case, how does Burnham know whether his instructions on methodology were followed at all? How can anyone verify the findings? And this is a peer-reviewed article. Who were these peers? And what did they review?
I, too, would be very happy to see a thorough evaluation by independent experts—including statistical methodologists, not just the epidemiologists who seem to be running this little show. The only problem is that essential background information has been destroyed—or was never collected in the first place—so it may already be too late for that.
Previous related posts:
- Lancet study of Iraqi deaths is statistically unsound and unreliable
- Lancet researchers ignored superior study on Iraqi deaths
UPDATE (5 Dec.): Sean Gourley and his colleagues have come up with a specification of the extent of main street bias.
