Deltoid

Sometimes I think that there must be a qualifying exam in order to write for Tech Central Station. Fail the exam and you’re in. They seem to have exams in at least physics, economics, statistics, and epidemiology. Tim Worstall, the author of today’s article seems to have failed both the statistics and epidemiology exams.

Worstall is criticizing a recent study published in the Lancet that found very roughly 100,000 excess deaths in Iraq after the invasion, almost all of which were violent. He writes:

At the very least one would have to add The Lancet to that list of mainstream media which are worth 15% (or is it 5% now, the left have never really been any good at numbers) to John Kerry in the polls. What makes it a great deal worse is this, from the findings to the report. In fact, these are the findings in their totality:

“The risk of death was estimated to be 2.5-fold (95% CI 1.6-4.2) higher after the invasion when compared with the preinvasion period. Two-thirds of all violent deaths were reported in one cluster in the city of Falluja. If we exclude the Falluja data, the risk of death is 1.5-fold (1.1-2.3) higher after the invasion. We estimate that 98 000 more deaths than expected (8000-194 000) happened after the invasion outside of Falluja and far more if the outlier Falluja cluster is included. The major causes of death before the invasion were myocardial infarction, cerebrovascular accidents, and other chronic disorders whereas after the invasion violence was the primary cause of death. Violent deaths were widespread, reported in 15 of 33 clusters, and were mainly attributed to coalition forces. Most individuals reportedly killed by coalition forces were women and children. The risk of death from violence in the period after the invasion was 58 times higher (95% CI 8.1-419) than in the period before the war.”

Have a look at those confidence levels. Yup, 95%. That is, a one in twenty chance that the effect simply does not exist.

No, a 95% confidence interval does not mean that there is a 95% chance that the true value is in the interval. And even if it did, Worstall would still be wrong, since if there was a 95% chance that the true value was in 1.1–2.3, there would be a 2.5% chance that it was more than 2.3 and only a 2.5% chance that it was less than 1.1.

It gets worse:

Look at the relative risk ratios (leave out Falluja; I don’t think anyone is really very surprised to see a higher mortality rate there): 1.1-2.3. It isn’t just that it is an absurdly wide one (note, a relative risk ratio of 1 would mean no effect whatsoever) it is that if this paper was written to generally accepted statistical standards it would never have been published. With a 95% confidence level a relative risk ratio of anything less than three is regarded as statistically insignificant.

Actually, by “generally accepted statistical standards” the result is statistically significant at the 95% level since the 95% confidence interval does not include 1. The risk ratio does not have to be three or more to be statistically significant. Worstall is trying to tell his readers that if the death rate increased by a factor of 2.9 (which would be about 300,000 dead bodies), statistics could not detect this increase. Really. (Worstall’s factor of three rule is probably a confused version of the GEP scam run by Philip Morris to try to show that cigarette smoke was harmless.)

Worstall goes on:

Just to clarify that, by “insignificant” no one is stating that it is not important to those people who undoubtedly have been killed during the War. What is being said is that we don’t have enough information to be able to say anything meaningful about it. “Statistically insignificant” means “we don’t know”.

The result was, in fact, statistically significant, so we can be reasonably confident that the death rate in Iraq went up. The best estimate (not including Falluja) is 100,000 extra deaths, but the confidence interval is wide, so the estimate should be treated with caution.

Of course, it is obvious why Worstall’s article was published despite his ingorance of basic statistics—Tech Central Station is campaigning for Bush and wants to deny that the invasion of Iraq has had bad consequences.

Update: Daniel Davies has a comprehensive takedown of more defective critiques of the Lancet study, including Worstall’s.

Tech Central Station has published Tim Worstall’s admission that his critique of the Lancet Iraq study was completely wrong:

Further to my article of Friday on this subject. I’m afraid I mangled the statistical argument. My inadequate knowledge of the subject led me to make an argument that is incorrect. I stand by my contention that there is something fishy about this study (leaving aside the politically motivated timing of its publication, something the author has been clear about himself) yet have to admit that I have not found it, leaving me with nothing but personal prejudice upon which to stand my argument. I would also like to make clear that this subject was not “assigned” to me, the idea, research, argument and errors were all my own, as was my request for this clarification. Just in case you are wondering, being fact checked by the Pajamahaddin and being found in error does hurt and I hope that future writings will be, where necessary, so corrected.”

On his blog Worstall thanks Daniel Davies and me for the correction. Such decent behaviour is unfortunately not common at Tech Central Station. Authors like John Lott and Iain Murray just repeat their false claims, while Glenn Reynolds posts a correction but does not acknowledge the source.

Tech Central Station didn’t just post Worstall’s correction by itself. They have had a second attempt at debunking the Lancet study, posting an article by Michael Fumento. Fumento argues:

the researchers didn’t feel themselves bound by anything official, like death certificates. Interviews were just fine. “In the Iraqi culture it was unlikely for respondents to fabricate deaths,” they wrote.

Unfortunately, Fumento seems to have missed the immediately preceding sentences in the Lancet paper, where they noted that, when asked, 81% confirmed with death certificates:

In 63 of 78 (81%) households where confirmations were attempted, respondents were able to produce the death certificate for the decedent. When households could not produce the death certificate, interviewers felt in all cases that the explanation offered was reasonable eg, the death had been very recent, the certificate was locked away and only the husband who was not home had the key. We think it is unlikely that deaths were falsely recorded.

Fumento’s “killer” argument is:

Cluster sampling can be valid if it uses reliable data, rather than on inherently unreliable self-reporting. But it can also be easily skewed by picking out hotspots — like determining how much of a nation’s population wears dentures by surveying only nursing homes.

In fact, intentionally or otherwise, that’s pretty much what The Lancet did. Most of the clusters had no deaths whatsoever. But here’s the real bombshell: “Two-thirds of all violent deaths were reported in one cluster in the city of Falluja,” the journal reported. That’s it; game over; report worthless.

Trouble is, Fumento has once more been extraordinarily careless in his reading of the study. Here are the two sentences in the report that follow the one he quoted:

If we exclude the Falluja data, the risk of death is 1.5-fold (1.1-2.3) higher after the invasion. We estimate that 98 000 more deaths than expected (8000-194 000) happened after the invasion outside of Falluja and far more if the outlier Falluja cluster is included.

That’s right, they properly excluded the outlier Falluja in their estimate of 98,000 and Fumento didn’t notice this fact. That’s it; game over; Fumento article worthless.

I’m starting to feel embarrassed for Tech Central Station. Do you think they’ll have a third go at the Lancet?

Also of interest is Chris Lightfoot’s demolition of more lame critiques of the study.

I wrote earlier how it seems that you must fail a qualifying exam before you can write on a topic at Tech Central Station. Now the errors in Fumento’s critique of the Lancet study.aren’t errors in epidemiology—they seem to result from not having read the study. Indeed, in comments at TCS, Fumento seems to be asking for help to find out what it said:

You imply rather strongly that you’ve read the report. If so, please inform us of what the extrapolation was that DID NOT rely on the Falluja cluster. I’m waiting.

and then:

I asked the wrong question. I meant to say how many were in the extrapolation that DID include the Falluja cluster?

and also:

We also know there is absolutely no way to randomly select 33 clusters.

Again, if he had read the study, he would have known that they did randomly select 33 clusters, and that they explained the randomization procedure in great detail.

But now, in a comment on my blog, Fumento has proved that he meets the TCS requirement to write about epidemiology—he demonstrates ignorance of a basic epidemiological principle:

The alleged 100,000 deaths were those above the pre-war baseline. That baseline was predicated on a figure of 5.0 deaths per 1,000. BUT the figure for the US at the time was 8.3 deaths per 1,000. Obviously Iraq was one of the safest countries on the face of earth prior to the Yankee imperialist invasion. In fairness, the CIA Worldbook uses a 5.6 per 1,000 figure for Iraq but what was it’s source? Saddam’s ultra-trustworthy government, of course. Thus The Lancet is using figures even lower than the Land of Baghdad Bob was putting out.

OK, some basic epidemiology:

Comparing Mortality in Different Populations: an important use of the rates—recall that the populations may differ in regard to many factors which affect mortality, of which age distribution is the most important—so to compare, we hold the factor (age) constant.

Consider:

Death Rates by Age and Race, Baltimore City, 1965
	All	<1yr	1-4yr	5-17yr	18-44yr	45-64yr	>65yr
White	14.3	23.9	0.7	0.4	2.5	15.2	69.3
Black	10.2	31.3	1.6	0.6	4.8	22.6	75.9

Overall [this is called CRUDE or UNADJUSTED mortality], the death rate for blacks is LOWER than it is for whites. This is unexpected given the conditions at the time for blacks with respect to health care access and living situations. Why would this happen? Well, if we look at the stratified rates by age, we see that the rate for blacks is HIGHER at every age level. This seeming conundrum is explained by the fact that, while the rates increase markedly for both races in the over 65 bracket, there are more whites left in that age group. So many, in fact, that the overall mortality for whites overwhelmingly occurs in the over 65 age group, while the same is NOT the case for blacks.

Hmmm, so do you think Iraq and the US have the same age distribution? From the CIA Factbook—the very source that Fumento cited—we find that the US has a very different age structure with a much larger proportion of old people. And notice that neighbouring Syria has a similar age structure to Iraq and a death rate of 4.96, consistent with the Lancet study.

Country	US	Iraq	Syria
Death Rate	8.34	5.66	4.96
Median Age	36 years	19.2 years	20 years
Age Structure	0–14 years: 20.8% 15–64 years: 66.9% 65 years and over: 12.4%	0-14 years: 40.3% 15-64 years: 56.7% 65 years and over: 3%	0-14 years: 38% 15-64 years: 58.7% 65 years and over: 3.3%

Fumento’s argument that the 5.66 number for Iraq was falsified by Saddam’s regime makes no sense. If Saddam was cooking the numbers he would have made them higher and claimed that the sanctions were killing vast numbers of people.

And his claim that the Lancet number of 5.0 is inconsistent with the CIA Factbook number of 5.66 is also wrong, since he ignores the fact that the 95% confidence interval for the Lancet study was 3.7–6.3, and 5.66 lies in that interval.

Also of interest might be this Chris Mooney post on Fumento’s book on biotechnology.

Update: Fumento replies:

Much of what you write is simply idiotic, as in saying the clusters were “randomly sampled.” How is that even possible, aside from throwing darts at a dartboard? It also implies you can read the researchers’ minds. Ah, The Amazing Lambert. But at least I’ve found one person who really does believe we’ve been violently killing civilians at a rate of over 180 per day. I was afraid the Lancet propagandists weren’t going to get away with that. You can blog all you want, but my next column is also on this. It goes out to over 350 newspapers and lots of websites that each have more traffic than yours. You occupy the pitiful place of the harmless blogger who blogs because nobody in his right mind would punish him. Rant, rant, rant. The world careth not.

Looks like Fumento flunks statistics as well as epidemiology. Random sampling is a fundamental concept in statistics and despite Fumento’s incredulity it is actually possible to carry it out. I didn’t have to read the researchers’ minds to find out how they randomized the clusters; I just had to read their paper, something that Fumento still has not done. And no, I don’t believe that we are killing civilians at over 180 a day—that isn’t what the paper found. I do believe that Fumento’s critiques of it are worthless, if that is any help.

Update 2: Fumento replies again:

You cannot randomly choose physical households in a place like Iraq. Those data are not in a computer. Do you think it’s like campaign polling in the States?

Here is the methodology that is beyond Fumento’s comprehension:

We assigned clusters to individual communities within the Governorates by creating cumulative population lists for the Governorate and picking a random number between one and the Governorate population. Once a town, village, or urban neighbourhood was selected, the team drove to the edges of the area and stored the site coordinates in a global positioning system (GPS) unit. We assumed the population was living within a rectangle, with the dimensions corresponding to the distances spanned between the site coordinates stored in the GPS unit. The area was drawn as a map subdivided by increments of 100 m. A pair of random numbers was selected between zero and the number of 100 m increments on each axis, corresponding to some point in the village. The GPS unit was used to guide interviewers to the selected point. Once at that point, the nearest 30 households were visited.

And if you are interested in more of the Fumento follies, look here and here.

The Anchorage Daily News has published a new version of Michael Fumento’s attempt to debunk the Lancet study on deaths in Iraq. How does it differ from his previous attempt? Well his key argument was that their estimate was skewed by the inclusion of the Falluja cluster. But it is perfectly clear from the report that Falluja was excluded from their estimate. Fumento knows this because he responded to my post with a comment, and he specifically asked questions about the inclusion of Falluja in the comments to his TCS article. In his new version he tacitly admits his error by quietly dropping his false claim. Of course, he hasn’t corrected his TCS article. He also aware that they did look at death certificates, so it is dishonest for him to repeat his claim that “the researchers didn’t feel bound by anything official like death certificates”.

Unfortunately he has replaced his bogus claim about Falluja with something worse—his ridiculous comparison of crude death rates between the US and Iraq. (I have explained what is wrong with that here.)

It is disgraceful that someone so wilfully ignorant of basic science has published a column on a scientific question in a newspaper. You can send a letter to the editor at the Anchorage Daily News here. (Leave a comment if you do write a letter.) Contact information for Scripps Howard News Service is here. I think it would be particularly helpful if any epidemiologists in my readership contacted SHNS.

Update: Fumento replies:

Actually, the major changes were additions — including quite legitimately pointing out that The Lancet insisted on using as its baseline pre-war mortality a number far lower than Saddam had used. That gave a range in the paper is inconsequential; the figure they used for their all-important 100,000 figure was five per 1,000.

5.0 is not “far lower” than 5.66. The post-war mortality rate was 7.9 (that’s excluding Falluja—if you include Falluja it was 12.3), so whether the pre-war rate was 5.0 or 5.66, it is still a substantial increase. I notice you offered no defence of your ridiculous comparison of crude death rates between the US and Iraq.

Fumento continues:

I pulled the section about Falluja being included because it confused people — like you. Find in the paper where they provide an equivalent to the 100,000 figure but exclude Falluja deaths. You can’t, because it’s not there. The Lancet has lied and you support it because you happen to like its conclusions, not because those conclusions where arrived at scientifically.

I already gave the exact quote from the paper—it was from the same paragraph that Fumento quoted in his TCS piece. Here it is again, with extra emphasis:

We estimate that 98 000 more deaths than expected (8000 194 000) happened after the invasion outside of Falluja and far more if the outlier Falluja cluster is included.

And no, I do not like the conclusions. I find them most unpalatable. But rather than inventing specious grounds for dismissing the study, I think it is better to face up to reality.

Yet another person has tried to refute the Lancet article. John Brignell dismisses the study just because:

A relative risk of 1.5 is not acceptable as significant.

Actually the increased risk was statistically significant. You won’t find support for Brignell’s claim in any conventional statistical text or paper. To support his claim he cites a book called Sorry, wrong number!. Trouble is, that book was written by…. John Brignell. Not only that, it was published by… John Brignell. Brignell is a crank who dismisses the entire field of modern epidemiology as some sort of plot by scientists to scare people. We encountered him before in this post where, armed with no evidence whatsoever, he insisted that the ozone hole had always been present.

To see how silly Brignell’s “relative risk of 1.5 is not acceptable as significant” claim is, consider this: Suppose we had perfect records of every death in Iraq and there were 200,000 in the year before the invasion, and 300,000 in the year after. Then the relative risk would be 1.5 and Brignell would dismiss the increase as not significant even though in this case we have absolutely certainty that there were 100,000 extra deaths.

One interesting feature of blogspace discussion of the Lancet study has been the comments from warbloggers, who, despite not even knowing what cluster sampling is, have been absolutely certain that the methodology of the study has been discredited. For instance, Arthur Chrenkoff admits:

I’m not a statistician

but none the less concludes that Shannon Love had demolished the study. (Daniel Davies deals with that “demolition”).

Or Michael Totten at Instapundit, who is certain that the study uses very bad methodology. Bill Trippe sent him a correction:

Did you even read the paper before you decided Shannon Love’s argument was so brilliant? Probably not, because if you had, you would see that she clearly did not understand what she was reading. Case in point: she makes a grand pronouncement (even putting it in bold), about the inclusion of Fallujah in the conclusions. Guess what? Fallujah was excluded from the results as an outlier.

Totten, of course, did not correct the erroneous post.

Or Cori Dauber. I corresponded with her last year because she bought Lott’s ridiculous claim that the murder rate in Baghdad was very low. and she admitted that she wasn’t good with numbers. But she is sure that the methodology is “garbage” and calls the study a “lie”.

Our last example, by Anthony Rickey, is like a Bizarro world version of this post. He takes issues with the folks who accepted the Lancet’s study, reckoning that their prejudices have blinded them to the obvious (to him) flaws. He even believes that the Lancet study will be another scandal like the affair of Dan Rather and the forged memos. Unfortunately it is Rickey’s prejudices that have blinded him to the flaws in the criticism of the study. (See Daniel Davies again.)

But what about all the people who accepted the study’s result who also didn’t know much about statistics? Well if you don’t know enough to evaluate the study yourself, you’d have to trust the experts on statistics, and I don’t think that it would be unreasonable for you to suppose that a journal with the prestigious reputation of the Lancet would have checked those statistics thoroughly. And you would have been right.

John Fleck commented on my exchange with Fumento here and here. He responded to Fumento’s silly charge that I “occupy the pitiful place of the harmless blogger who blogs because nobody in his right mind would punish (sic) him” with:

That’s of course ad hominem, something of a poor refuge in any argument. But it’s worse than that. It’s plain dumb in this age of Dan Rather and Little Green Footballs for a writer of Fumento’s stature to expect us to think he wins the argument because his work is published in mainstream media.

Sure enough, Fleck got an email from Fumento:

Subject: Ah, another worthless observation from somebody that can’t get published so he blogs.

Earth to Inkstain and Lambert: Other than Inkstain caring what Lambert says and Lambert caring about what Inkstain says (perhaps), nobody cares what either of you says. Not only are you fully contained in the blogosphere, you’re actually in a much tinier realm than that. Meanwhile of the many places my piece on the Lancet trash appeared is today’s Daily News, weekend population above 500,000. You attack not out of a sense that injustice has been done regarding the Lancet report, but out of jealousy. But if you cleaned up your act, you might just find that somebody somewhere, even with a circulation of ten, would occassionally print you. Alas, you will not. You are a lost cause.

Which is pretty funny, since if Fumento had bothered to click on Fleck’s About me link he would have discovered that Fleck is the science writer for the Albuquerque Journal.

Fumento then sent Fleck another abusive email:

First, it seems to me that any nationally syndicated columnist, including those I can’t stand, is a journalist — whether John Fleck acknowledges it or not. Second, I dropped one of my arguments from the TCS piece only because it confused people with simple minds. Like you. As it happens, I also had to cut 200 words even as I added in new information. So we are faced with two possibilities here, neither pleasant. A) You’re not particularly bright; B) You’re not particularly bright.

I wonder if Fumento has managed to read the Lancet paper yet?

On a slightly more serious note: Fumento has managed to get his attack on the Lancet paper published in the Sacramento Bee, the Arizona Daily Star and the Minneapolis-St Paul Star Tribune. Just think how much it would bug him if you wrote a Letter to the Editor about his column.

Oh, and my humble blog is now the second site returned by a Google search for “Fumento”.

The defective refutations of the Lancet study just keep on coming. First, we have Gerard Alexander writing in the Weekly standard:

But the study’s researchers were sure to survey in Falluja, far and away the most violent city in post-invasion Iraq. Falluja turned out to be such a wild statistical outlier that they offer two estimates, one with Falluja included and one with it kept out. But questions about just how representative the sample sites were go deeper than this. The researchers selected their survey sites households for such unclear reasons that we simply can’t extrapolate to the whole country with any confidence. What are the chances that they have over-sampled the most violent parts of the Sunni Triangle and under-sampled the calmest Kurdish and Shia areas? Without better statistics about population and violence, we can’t know, and neither can they. The fact that they don’t explain their strategy doesn’t build confidence in their research design or their conclusions.

Alexander comes from Fumento’s don’t-bother-to-read-the-study school. If Alexander had bothered to read the Lancet study he could not possibly have missed their lengthy and detailed explanation of how they randomly selected the clusters. After this gross deviation from the standards of scholarship, Alexander then has the hide to accuse the Lancet of jettisoning “their standards of fairness, restraint, objectivity, and integrity.”

Next, we have Robert Lichter of STATS who writes:

The crucial assumption is that any increase in deaths after the invasion began on March 19, 2003 is associated with the conflict and subsequent occupation, to the exclusion of any other factor. Specifically, their sample included 46 reported pre-invasion deaths, only one of which was violent, and 89 post-invasion deaths outside Falluja, 21 of which were violent. According to a table that breaks down the causes of death, fewer than half of the “excess deaths” (45 percent) resulted from violence. One in five was accidental, one in six was due to heart attack or stroke, just under one in 10 was caused by infectious disease, and the same proportion consisted of neonatal or infant deaths. Yet all these deaths without exception were attributed to the war and occupation.

It seems pretty unlikely that a doubling in the number of deaths was just a coincidence. If Lichter wants to suggest that something else caused it, he needs to explain what. And I hope it is clear how the disruption to medical services and electricity and water could cause increases in deaths from diseases.

The Economist and Stephen Soldz have discussions of the study that are more balanced than the hatchet jobs by Alexander and Lichter.

Update: I sent my comments to Alexander and he replied:

I appreciate your certainty that I haven’t read the study, but am afraid its misplaced. I’m even sorrier that you missed my point: in the absence of better stats, especially on violence, no-one—no matter how lengthily they think they’re explaining their methods—can survey a sample in whose representativeness we can have confidence. It also means we can’t exclude the study’s findings as accurate, which is why I don’t dismiss them, though I question them. What I got most specific about was the Lancet’s apparent disinterest in their own previous published estimates of child/infant mortality before March 2003. I’m not dismissing those previous findings without explanation; they are.

So Alexander read the extensive explanation of their strategy for selection and then accused the authors of not explaining their strategy. Readers will have to think of the best word to describe Alexander’s conduct. Now he claims that it is not possible at all to get a representative sample. He doesn’t explain why it is impossible, probably because he cannot. Nor do they dismiss previous findings on infant mortality without explanation. They clearly state:

The Coalition Provisional Authority and the Iraqi Ministry of Health have identified the halving of infant mortality as a major objective. In the absence of any surveys, however, they have relied on Ministry of Health records. These data have indicated a decline in young child mortality since February, 2001, but because only a third of all deaths happen in hospitals, these data might not accurately represent trends. No surveys or census-based estimates of crude mortality have been undertaken in Iraq in more than a decade, and the last estimate of under-five mortality was from a UNICEF sponsored demographic survey from 1999.

In other words, infant mortality has decreased since it was last measured.

Alexander claimed that the Lancet chose to throw away its reputation with the article, but it is Alexander who has thrown his own reputation away with his disgraceful article in the Weekly Standard.

Fumento left a comment on my earlier post. Instead of discussing the Lancet article, he boasted how his column had been published in the on the web site of the Lake Wylie Pilot, which is a free weekly newspaper serving a town of 3,000 people. Hey, my little blog has a greater circulation than that.

Eye Doc linked to Fumento’s attack on the Lancet, so I left a comment explaining what was wrong. Fumento replied:

Tim Lambert is on a personal Jihad to debunk my debunking. I did not say death certificates were not used, they were. But so was alleged personal recall. That means that if a family recalled ten deaths of people who were alive and well, they went straight into the pot.

Nowhere in his piece did Fumento say that they used death certificates, instead he implies they did not with this: “the researchers didn’t feel themselves bound by anything official, like death certificates.”

Fumento continues:

The authors claimed to have come up with one set of numbers including Falluja, another without. But strangely, they never present the “without numbers.” Lambert knows this because I told him directly. Anyway, it’s in the study — or rather, it’s NOT in the study.

Who are you going to believe, Fumento, or your lying eyes?

And David Mason piles on here and here, opining: “Michael Fumento is a bitter, bitter man.”.

Correction: In comments, Fumento complains:

you are now lying about what I wrote to you. I didn’t say I appeared in the Lake Wylie Pilot, I said my column is picked up by the McLatchey News Service that posts it automatically to the sites of about a dozen papers.

My apologies. When he wrote that “it goes to” the Lake Wylie Pilot he meant that it appeared on their web site, which is apparently different from appearing in the Lake Wylie Pilot. I apologize unreservedly to Mr Fumento for stating that his column had appeared in the Lake Wylie Pilot when it had merely appeared on the Lake Wylie Pilot web site. I hope that Mr Fumento’s reputation has not been harmed by my erroneous statement.

Daniel Davies has an excellent roundup of the Lancet discussion.

I’ve added an update to my post about Gerard Alexander’s attack on the Lancet.

Chris at Mixing Memory takes down another Lancet critique, this one by John Ray.

The fun continues in this comment thread. Highlights:

Michael Fumento:

The authors claimed to have come up with one set of numbers including Falluja, another without. But strangely, they never present the “without numbers.” Lambert knows this because I told him directly. Anyway, it’s in the study—or rather, it’s NOT in the study.

John Fleck:

A quick refresher on where the Lancet study’s authors included the “without Falluja” numbers. It’s in the paper’s abstract. That’s the thing that comes right at the beginning: “We estimate that 98,000 more deaths than expected (8,000–194,000)happened after the invasion outside of Falluja and far more if the outlier Falluja cluster is included.”

It would be one thing if Fumento simply misspoke in his original TCS piece when he said the inclusion of the Falluja cluster biased the study and made it worthless. But in the face of Lambert and others repeatedly quoting the precise, clear and unambiguous wording with which the study did exactly the opposite, excluding the Falluja data as an outlier, Fumento continues to misrepresent it

Jonathan Dursi:

There is nothing in Fumento’s analysis which even comes close to disputing the Lancet article—indeed, it’s not even clear he read the abstract.

Michael Fumento:

Lambert, Fleck, and Dursi (an epidemiologica expert because his field is astronomy) just won’t let go. The study did not present numbers that included Falluja, either in the abstract or text. Yet they accuse ME of not reading it.

Mark Tyrrell Frank:

Michael Fumento is quite extraordinary. How can he continue to say this? The text in the abstract and in the body quite clearly gives figures both with and without Falluja. E.g. maximum likelihood estimate of death risk is increased 2.5 times including Falluja and only 1.5 times without. This is not so much deception as sheer madness. Or maybe he has a different copy of the paper?

Sheer madness or deception? I report, you decide.

For someone who holds blogs in contempt, Michael Fumento sure spends a lot of time posting comments to blogs. Here he is again: (Hat tip: John Fleck, now the third site on a Google search for “Michael Fumento”)

My writing on the Lancet article has been Fleck’s obsession for over a week, and everything he says is wrong including this latest posting. First, simple subtraction tells you in 19 percent of the households death certificates were NOT used. But that’s not the equivalent of 19 percent of the deaths. If a household said a bomb killed five family members, that’s worth five death certificates. For all we know, half the deaths reported came from recall. Second, apparently Lambert has never seen a death certificate and thinks coroners or medical examiners are psychics. A death certificate will indicate death by violence, but it won’t say something like “500 lb. JDAM dropped by U.S. F-18.” Every day we see pictures of Iraqis blown up by Iraqis and other Arabs. Yes, the people were blown up but no we didn’t do it and the death certificate isn’t going to say one way or another. Finally, he ignores evidence outside of the paper that I presented, including two anti-war groups that said the Lancet figures were far too high and a certain individual who pegged them too high by 85,000. His name? Osama bin Laden.

Note that after his apparent flip-flop on the question of whether the study reported an estimate excluding Falluja, Fumento is silent on that question. Anyway, let me address the points that Fumento does raise. First, hardly any households had multiple deaths, so his notion that half the deaths were not confirmed is wrong. Second, yes, the death certificate will just say that the cause of the death was an explosion without saying who was responsible. For that we must rely on the word of the family. But the person is still dead as a result of the invasion, no matter who did the killing. Third, the numbers from groups like Iraq Body Count are measuring something different from the Lancet survey–the number of confirmed civilian deaths directly caused by the war. The IBC number is certainly an underestimate since not all deaths will be reported. Not does it count indirect effects like increase in disease because of breakdowns in medical infrastructure. The IBC does not say that the Lancet estimate is too high:

Others have asked us to comment on whether the Lancet report’s headline figure of 100,000 is a credible estimate. At present our resources are focused on our own ongoing work, not assessing the work of others.

Fumento, of course, claims otherwise.

Sadly, it looks as if Michael Fumento has retired from the field. All I can offer any folks suffering withdrawal symptoms is this thread. James M describes it like this:

I noticed a truly spectacular example of what I suppose is the unarmed kamikaze approach to debate carrying on in the comments boxes. Not so much being savaged by a dead sheep, as seeing someone punch themselves repeatedly in the face. It is painful to watch.

But yet, like a car wreck, you must look.

The latest pundit to have a go at the Lancet study is Andrew Bolt. Like most of the critics, Bolt just does not have the statistical background to produce a competent critique. In Bolt’s case this is even less excusable, since he had the benefit of the Economist’s excellent article, but unfortunately Bolt does not seem to have understood it.

Bolt writes:

Just ask yourself: Have more than 180 Iraqis, mainly women and children, really died every day, on average, for the past 18 months, usually at the hands of the Americans?

If so, where are all the funerals? Where are the pictures? Where are the news reports

The CIA World Factbook reports that the death rate in Australia is 7.38 per 1,000. That’s more than 400 deaths per day. Just ask yourself: Have more than 400 Australians, really died every day, on average, for the past 18 months? If so, where are all the funerals? Where are the pictures? Where are the news reports?

Bolt continues

But few of the commentators who seized on the survey bothered to ask such basic questions, or even to heed Human Rights Watch, which warned: “The numbers seem to be inflated.”

Perhaps they should read what Human Rights Watch said later:

“I hate the interview I did for The Washington Post,” he says. “I was on the train, I hadn’t read the report yet [when the Post’s reporter called for comment]. In general, I’m not as negative as that [Post] report made me seem. This is raising issues that are not heard of much in the U.S.”

Bolt continues:

Its researchers interviewed 7868 Iraqis in 988 households in 33 neighbourhoods around Iraq, allegedly chosen randomly, and asked who in the house had died in the 14 months before the invasion and who in the 18 months after.

“allegedly chosen randomly”? The researchers carefully explain, in detail, how the randomization worked. If Bolt wants to accuse them of lying about this, why does he bother with the rest of his attack? Why not just accuse them of making up all the results?

Bolt argues that the pre-war infant death rate that the survey found (29 per 1,000) is wrong because:

2002 figures from UNICEF, which in a much bigger survey of 24,000 households found the infant mortality rate in Iraq before the war was actually a tragic 108 deaths per 1000 infants.

But the 2002 UNICEF figures are based on a survey conducted in 1999 (which meant that it was counting deaths that happened in 1998). And the 1999 survey used similar methodology to the new survey, so the difference can’t be because there was something wrong with the methodology of the new survey—the most likely explanation is that the oil-for-food program lowered the infant mortality rate to between 1998 and 2002.

In March 2002 Matt Welch wrote an article debunking extreme claims of deaths caused by sanctions:

Sanctions critics almost always leave out one other salient fact: The vast majority of the horror stats they quote apply to the period before March 1997, when the oil-for-food program delivered its first boatload of supplies (nearly six years after the U.N. first proposed the idea to a reluctant Iraqi government). …

As the U.N. Office for the Iraqi Program stated in a September 28, 2001 report, “With the improved funding level for the program, the Government of Iraq is indeed in a position to address the nutritional and health concerns of the Iraqi people, particularly the nutritional status of the children.” Even two years earlier, Richard Garfield noted in his survey that “the most severe embargo-related damages [have] already ended.” …

Those who get past the initial frustrations of researching the topic usually end up on Richard Garfield’s doorstep. His 1999 report—which included a logistic regression analysis that re-examined four previously published child mortality surveys and added bits from 75 or so other relevant studies—picked apart the faulty methodologies of his predecessors, criticized the bogus claims of the anti-sanctions left, admitted when the data were shaky, and generally used conservative numbers.

So Richard Garfield is the go-to guy on Iraqi infant mortality. Does he think that there is something wrong with the numbers in the Lancet study? Probably not, since he is one of the authors.

Andrew Bolt then demonstrates a lack of understanding of basic statistics. After noting the high death rate in the Falluja cluster he writes:

Truly, these statistics are unbelievable. I suspect the study’s authors thought so, too, which may be why they left the Fallujah figures out—calling them unrepresentative—when they calculated Iraq’s death toll since the invasion.

But the survey techniques they used to give clearly wrong figures in Fallujah are the same ones they used in the other 32 clusters of households that they interviewed elsewhere in Iraq.

An estimate based on Falluja is unreliable because it is based on one cluster, and there is a good chance that that cluster might not be representative. But it does not follow that because an estimate based on one cluster is unreliable that an estimate based on 32 clusters is unreliable. In fact, the size of the sample is the crucial difference—you need a reasonable sample size to get a reliable estimate. If you believe Bolt’s argument, then all surveys can be similarly dismissed. For example, a poll on voting intentions that just surveyed one random person is clearly going to give an unreliable result. Does it follow that a poll of a 1000 people must also give an unreliable result?

Bolt finishes with:

Too many commentators seem too desperate to believe the worst of the Americans and to belittle the liberation of Iraqis from a tyrant.

That desperation means even junk surveys such as this will find many eager believers, ready to hear the very worst. And to recklessly repeat it.

Too many war supporters have a desperate need to deny that the Iraq war has had some bad consequences. This desperation means that even clueless critiques, such as Bolt’s will find credulous believers. This is unfortunate. I believe we need to be clear-eyed about the consequences of the war and face up to reality.

Update: Bolt’s response:

I’m sure you don’t need me to point out the obvious omissions, red herrings, false comparisons of additional deaths to total deaths, and the telling failure even to cite the shocking figure of Iraqi child deaths given even by the “expert” cited as the last word on the issue.

Of course I did cite the infant mortality figures. I don’t know why Bolt would write something so plainly false. And did you notice the clever clever way he used square quotes to disrespect Richard Garfield?

Update 2: Tim Blair thinks that Bolt’s hopelessly innumerate criticism proves that Peter FitzSimons is stupid for citing the 100,000 figure. Unfortunately, all Blair has proved is that he himself has no understanding of basic statistics.

Chris Bertram points out that a new study suggests that the Lancet’s finding of an increase in infant mortality following the invasion of Iraq is correct. The Washington Post reports:

After the rate of acute malnutrition among children younger than 5 steadily declined to 4 percent two years ago, it shot up to 7.7 percent this year, according to a study conducted by Iraq’s Health Ministry in cooperation with Norway’s Institute for Applied International Studies and the U.N. Development Program….

International aid efforts and the U.N. oil-for-food program helped reduce the ruinous impact of sanctions, and the rate of acute malnutrition among the youngest Iraqis gradually dropped from a peak of 11 percent in 1996 to 4 percent in 2002. But the invasion in March 2003 and the widespread looting in its aftermath severely damaged the basic structures of governance in Iraq, and persistent violence across the country slowed the pace of reconstruction almost to a halt.

Via Juan Cole I find that Rod Nordland and Babak Dehghanpisheh have written:

The leading British medical journal, The Lancet, recently published a study that used interviews and extrapolations to estimate the total figure at 100,000 or more, mostly from aerial bombardment. Other statisticians have since dismissed the study’s conclusions as unreliable and speculative.

No no no! The people who dismissed the study’s conclusions were not statisticians and displayed profound ignorance about statistical methodology.

Update: David Adesnik attacks the malnutrition report, calling the 11% and 4% acute malnutrition figures “pseudo-statistics”. Adesnik offers no criticism of the methodology—the only reason he offers for rejecting the figures is that they were compiled under Saddam. But his reason makes no sense at all. Even if, for the sake of argument, we believe that Saddam could force UNICEF into cooking the statistics, why would Saddam have been artificially lowering the figures? Surely he would have been raisning them so that he could point to the harm that the sanctions were inflicting on Iraqi children.

In an update, Adesnik puts his money on the scenario that the Washington Post is belatedly reporting the results of this study conducted 29 April–3 May 2003 which found that 7.7% of Baghdad children were acutely malnourished. Well, he’s lost his money, because that was a different survey. The Post is reporting the results of a national survey conducted in April/May 2004.

Adesnik also argues that even if there is a new survey the 2003 survey, “conducted less than three weeks after the invasion of Iraq”, proves that “the increase from 4 to 7.7 percent was Saddam’s doing.” Adesnik does not seem to have bothered to read the conclusion of the 2003 study (which was conducted six weeks after the start of the war, not “less than three” as Adesnik claimed):

Seven out of 10 children reported had suffered from diarrhoea at some time during the previous 5 weeks. Diarrhoea is likely a major factor in the rise in malnutrition since the war, linked mainly to the poor quality and low quantity of water, poor sanitation, large amounts of uncollected garbage and frequent electricity cuts.

Adesnik seems to be unaware that a sick child can lose a lot of weight in a few weeks.

Now the 2003 study just covered Baghdad, while the 2004 one was a national one, so the numbers are that comparable, but it does seem that malnutrition got worse after the invasion and things still have not improved.

Update 24/11: David Adesnik has replied. First, he is unconvinced that the Post is reporting the results of the IAIS survey:

Hoping to track down the data, I sent an e-mail to IAIS on Sunday asking for further information about their work. In addition, I spent a considerable amount of term searching for related information on Google and Lexis-Nexis, yet found absolutely nothing.

Of course, it may turn out that IAIS really has done a new survey. But for the moment, there is hardly enough evidence to substantiate Tim’s allegation.

Well, I didn’t spend a considerable amount of time searching, I just looked at the IAIS home page, which says:

Malnutrition in Iraq
In recent days data on malnutrition from the IMIRA survey have been published in Iraq and elsewhere (see Washington Post ). According to our findings 7.5 percent, or 216,000, children between 6 months and 59 months of age at the time of interview are acutely malnourished. Acute malnutrition is more widespread in the south than the north of Iraq. Acute malnutrition is measured by comparing a child’s height and weight to a standard reference population. See the Fafo IMIRA web pages.

I trust that clears things up.

Originally, Adesnik wrote that the 2003 study was conducted “less than three weeks after the invasion of Iraq”. He now says that this was an “ambiguity” and that he was counting from the fall of Baghdad. Forgive me, but “invasion of Iraq” is not an ambiguous way to refer to “fall of Baghdad”. And it makes no sense to count from the fall of Baghdad in any case, since the disruptions to water supplies and so on would have started with the bombing campaign at the start of the war.

Next Adesnik writes:

what I doubt is that 200,000 thousand children can get that sick in the space of a few weeks. Major combat operations were fairly localized and coalition bombing raids did not target civilian infrastructure.

While most Iraqis probably were dependent on official food rationing programs that may have been disrupted during the war, I tend to doubt that such a disruption would translate so immediately into a national epidemic of malnutrition. Of course, that is just speculation—but Tim is only offering more of the same.

I am perplexed by this response. I didn’t offer “just speculation”, but quoted the conclusion of the study, which found that “Seven out of 10 children reported had suffered from diarrhoea at some time during the previous 5 weeks.” Adesnik can doubt that the children got sick, but the doctors who examined them seem to think otherwise.

Adesnik next argues that Saddam was cooking the malnutrition figures and that we don’t know whether he was artificially increasing them or decreasing them because “speculation about Saddam’s motives is futile.” Well, it doesn’t seem likely that Saddam could get Unicef to cook their numbers, but we have independent evidence that malnutrition was lower before the war—the recent Lancet study found that infant mortality was quite a bit lower before the war (and was conducted when Saddam was no longer in a position to influence the results).

Moreover, the infant mortality and malnutrition figures seem to show a consistent pattern and support each other. If you study the table below, you will see that things were very bad in the late 90s, then the oil-for-food program made things a lot better and that things have gotten worse since the war.

Acute Malnutrition Rate	Infant Mortality Rate
1996 11% (1996 MICS)	1999 10.8% (Unicef 1999)
2002 4% (Nutrition survey 2002)	2002 2.9% (Lancet 2004)
2004 7.5% (IMIRA 2004)	2003-2004 5.8% (Lancet 2004)

Update 26/11 Further discussion with Adesnik is here.

One of the arguments made against the Lancet study was that the study had greatly underestimated the pre-war mortality rate, because the study found that it was about 29 per 1000 live births, while UNICEF estimated that it was 108. Now the 108 dates from 1999, but sceptics doubted that it could have declined dramatically by 2002. However, other studies (see table below) show that the incidence of acute malnutrition declined dramatically between the late 90s and 2002, so it seems likely that infant mortality would have done so as well.

Acute Malnutrition Rate	Infant Mortality Rate
1991 3% (GOI-UNICEF 2000c)	1990 4.7% (CMMS)
1996 11% (1996 MICS)	1999 10.8% (Unicef 1999)
2002 4% (Nutrition survey 2002)	2002 2.9% (Lancet 2004)
2004 7.5% (IMIRA 2004)	2003-2004 5.8% (Lancet 2004)

Update: Added figures for situation pre-sanctions to the table.

David Adesnik has replied to my post on malnutrition in Iraq. He has conceded that the Washington Post was reporting the results of a new survey rather than the results of one from 2003. But he is still arguing that the war did not cause the increase in malnutrition seen in the 2003 study:

The question isn’t whether a certain child had some diarrhoea during the invasion, but whether that child started to have diarrhoea (or whether the condition intensified) during that five week period.

If we look at this UNICEF press release (which Adesnik already linked to) we find (my emphasis):

UNICEF says that unsafe water from disrupted water services may be playing a significant role in the findings. Poor water quality is largely to blame for a rapid increase in cases of diarrhoea among children in recent weeks.

Speaking from Baghdad, UNICEF Health and Nutrition Officer Dr. Wisam Al-Timini said that the survey found that more than 1 in 10 children were in need of treatment for dehydration.

“This suggests exactly what we know: Poor water and sanitation leads to diarrhoea, and then to dehydration and malnutrition. These children need treatment to stop their bodies from wasting because of an inability to retain vitamins and nutrients from ordinary foods. Those severely malnourished who do not get treatment are at very high risk of dying.”

Adesnik continues:

In order to show that the invasion was the primary cause of rising malnutrition, one has to show that the preponderance of the children’s severe weight loss took place during the six weeks of major combat operations, rather than the preceding year or so.

But the UNICEF report says:

“If we compare these results with earlier findings, we note that children who have generally grown over the past few years because of improved nutrition have suddenly and dramatically wasted. This coincides with war and the breakdown of social services.

Adesnik then looks for trouble by writing:

On a brighter note for OxBlog, Tim doesn’t seem to challenge my assertion that the similar results of the UNICEF and IAIS studies demonstrate that the malnutrition rate has been essentially stable since the beginning of the occupation. Thus, the WaPo was still very wrong to report that malnutrition “shot up…this year”.

Well, they’re not strictly comparable since one is for Baghdad and one is national, but more importantly, let’s look at what the Post said:

Acute malnutrition among young children in Iraq has nearly doubled since the United States led an invasion of the country 20 months ago, according to surveys by the United Nations, aid agencies and the interim Iraqi government.

After the rate of acute malnutrition among children younger than 5 steadily declined to 4 percent two years ago, it shot up to 7.7 percent this year,

The Post did not say that the increase happened this year as Adesnik seems to believe, but that it happened since the invasion. Clearly, “this year” refers to when the new survey was conducted, not to when all of the increase happened. Point to the Washington Post.

I haven’t commented on Kaplan’s shoddy critique of the Lancet because Daniel Davies already demolished it here. Kaplan did have one argument that Davies did not address, so I will deal with that in this post. Kaplan wrote:

The survey team simply could not visit some of the randomly chosen clusters; the roads were blocked off, in some cases by coalition checkpoints. So the team picked other, more accessible areas that had received similar amounts of damage. But it’s unclear how they made this calculation. In any case, the detour destroyed the survey’s randomness; the results are inherently tainted. In other cases, the team didn’t find enough people in a cluster to interview, so they expanded the survey to an adjoining cluster. Again, at that point, the survey was no longer random, and so the results are suspect.

It’s quite remarkable how many things that Kaplan got wrong in just one paragraph:

1. It is not true that they could not visit some clusters because the roads were blocked off.
2. They did not pick other more accessible regions. To reduce travel, they paired governorates with similar violence levels and move all the clusters in the pair into one of the governorates.
3. It was not unclear how they did this calculation since the report explains it in excruciating detail.
4. The moving was clusters was randomized so that it did not destroy the survey’s randomness.
5. It is not true that they expanded the survey to an adjoining clusters if there weren’t enough people in a cluster. The sampling unit was households, not people. And a cluster was the thirty households that were nearest a randomly chosen location.

The last mistake is particularly egregious. The first four seem to have resulted from a bizarre misreading of the study’s Governorate pairing procedure, but the last one seems to have been made up out of whole cloth.

The latest pundit to attack the Lancet study is somebody called John Lott. He writes:

I haven’t spent a lot of time going through the methodology used in this survey by Lancet, but I don’t know how one could assume that those surveyed couldn’t have lied to create a false impression. After all, some do have a strong political motive.

Well, unlike surveys of defensive guns use, where the people questioned can make anything up that they liked, the researchers tried to verify the deaths with death certificates and were succesful in 81% of the times that they asked.

There is also the question of the comparability of the before and after war fatality rates. Andrew Bolt has a very extensive and interesting critique of the Lancet paper:

As I explained earlier, Bolt’s article contains some basic statistical errors. But Lott seems to be endorsing it. What does that say about Lott’s knowledge of statistics?

Lott also links to this New York Times article, claiming

If the New York Times critiques you (even with caveats) from the right, you know that you are in trouble

Which is pretty weird, since the article defends the Lancet study:

Other critics referred to the findings of the Iraq Body Count project, which has constructed a database of war-related civilian deaths from verified news media reports or official sources like hospitals and morgues.

That database recently placed civilian deaths somewhere between 14,429 and 16,579, the range arising largely from uncertainty about whether some victims were civilians or insurgents. But because of its stringent conditions for including deaths in the database, the project has quite explicitly said, “Our own total is certain to be an underestimate.”

Yes, he’s back! Over at his website Fumento has posted Hate Mail, Volume 32, which contains his creatively edited version of our exchange. According to Fumento, it went like this:

Fumento:

And no, the Lancet column I wrote didn’t just appear in the four papers you mentioned. It appears in places you don’t even know about because, unlike your blog, it isn’t confined to the web but also appears in print. Yesterday it was in the Washington Times print edition. But if only the web interests you, you should know it was picked up by the entire McClatchy News Service. That means that in addition to automatically going onto the website of the Sacramento Bee (not the Sacremento Bee) it goes to about a dozen other newspaper websites as well.

Lambert

Instead of discussing the Lancet article, he boasted how his column had been published in the Lake Wylie Pilot, which is a free weekly newspaper serving a town of 3,000 people.

Fumento:

You are so obsessed you are now lying about what I wrote to you. I didn’t say I appeared in the Lake Wylie Pilot, I said my column is picked up by the McClatchey News Service which posts it automatically to the sites of over a dozen papers. You chose the smallest, ignoring such as the Sacramento Bee and the Minneapolis-St. Paul Star Tribune.

Dear me, how naughty of me to say that he was boasting about the Lake Wylie Pilot when he hadn’t mentioned it. Trouble is, if you check what he actually said you find that it was this (my emphasis):

That means that in addition to automatically going onto the website of the Sacramento Bee (not the Sacremento Bee) it goes to: Carolina Newspapers Clover Herald (SC) Fort Mill Times (SC) Lake Wylie Pilot The Bakersfield Californian The Modesto Bee The News & Observer The News Tribune (WA)

So he had mentioned the Lake Wylie Pilot. He just edited it out of the version he posted in an attempt to make me look bad. And of course Fumento does not link, making sure that any readers cannot check his version against the original sources.

Just when you thought you had seen all the different possible attacks on the Lancet study, Helle Dale, writing in the Washington Times, comes up with a new one: the study’s authors are having second thoughts. Dale writes

As the Financial Times reported on Nov. 19, even the Lancet study’s authors are now having second thoughts. Iraq’s Health Ministry estimates by comparison that all told, 3,853 Iraqis have been killed and 155,167 wounded.

Gee, did the Financial Times really report that the authors were having second thoughts? Let’s check. The report (subscription required) says:

Les Roberts, one of the paper’s authors, said that including more clusters would have improved precision, but would have increased to unacceptable levels the risks faced by interviewers. With hindsight, he would have liked interviewers to visit two or three points in Falluja to get an idea of variations between neighbourhoods there.

One other important weakness of such surveys is associated with retrospective recall, which the authors play down. Respondents may have lied or been confused about when deaths took place.

One question that could have thrown light on the precision of the mortality figure was to seek information on the wounded. Dr Roberts said no data were gathered on this: “Injuries are more subjective.”

Roberts did not say he had second thoughts. Dale is just making stuff up.

Now compare Dale’s second sentence with what the Iraqi Health Minister actually reported:

“Every hospital reports daily the number of civilians (which may include insurgents) who have been killed or injured in terrorist incidents or as a result of military action. All casualties are likely to be taken to hospital in these circumstances except for some insurgents (who may fear arrest) and those with minor injuries. The figures show that between 5 April 2004 and 5 October 2004, 3,853 civilians were killed and 15,517 were injured. I am satisfied that this information is the most reliable available.”

Those 3,853 deaths are not “all told” as Dale claimed, but occured in just six months. And the Lancet study’s 100,000 excess deaths includes those that resulted from the increase in infant mortality and homicide that followed the invasion. If we triple the Health Minister’s number to cover the whole period we get 12,000. The Lancet number for military and terrorist deaths is very roughly 40,000. Since the Health Minister’s number only covers what was reported to hospitals, the real number could easily be twice as much. A number like 25,000 military/terrorist deaths seems to be compatible with both the Lancet study and the hospital reports.

Medact, a UK health charity has a new study on the effects of the war on health and the health system in Iraq. Some extracts:

A recent scientific study has suggested that upwards of 100,000 Iraqis may have died since the 2003 coalition invasion, mostly from violence, mainly air strikes by coalition forces. Most of those reportedly killed by coalition forces were women and children. Many thousands of conflict-related injuries were also sustained. Infant mortality has risen because of lack of access to skilled help in childbirth, as well as because of violence….

The health system—all activities whose primary purpose is to promote, restore or maintain health—is in disrepair. The quality of state services is poor owing to chronic underfunding, poor physical infrastructure, shortage and mismanagement of supplies, staff shortages and lack of modern skills and knowledge. The 2004 budget allocation to the Iraq Ministry of Health is only US$38 per citizen. People increasingly rely on self-diagnosis and traditional healing, and buy prescription medicines in the marketplace. Under-the-table payments are required to secure many services, and there is widespread suspicion of criminal involvement in the distribution of pharmaceutical supplies. Health workers are trying to provide services in extremely difficult circumstances.

The poor state of the hospital system may explain part of the rise in infant mortality seen in the Lancet study. It also suggests that the figures for deaths compiled by hospitals may well be undercounts.

Richard Garfield (one of the authors of the Lancet study) comments:

“We need population-based monitoring to know how the Iraqis are doing. The UK and Iraqi ministry rebuttals of our conclusions do nothing to change that. The Lancet study and Medact report paint a realistic picture. What we now need is to understand exactly why people are dying, how to prevent those deaths and how to improve the quality of life. The Medact report helps us to do just that.”

Daniel Davies takes apart another bogus critique of the Lancet study, this one from the British Foreign Office that relies on comparing apples to oranges. Michael Lewis at Iraq Analysis has a more detailed rebuttal.

Remarkably, Tech Central Station has published an article by Iain Murray, who acknowledges that

The study itself is actually much more statistically sound than many commentators (including some in these pages) have suggested, and it certainly suggests that the mortality rate is worse in the unstable insurgency-ridden Iraq after the ouster of Saddam’s regime than during the last days of his tyranny.

and

perfectly good science like the Iraq study has been the subject of unfounded criticism and dismissal

Murray does take issue with the way the Lancet presented the results, objecting to the fact that they described the study as finding “that around 100 000 Iraqi civilians died as a result of the invasion” when the death toll included combatants as well as civilians. This is a legitimate objection, but Murray makes too much of it, since the great majority of the excess deaths were not combatants. Murray also objects to the timing of the publication complaining that it was an attempt by the Lancet to influence another country’s election. This objection strikes me as being without merit. I think it is better that voters have more information about the state of affairs in Iraq rather than less. I suspect that Murray is complaining because this information was damaging to his and Tech Central Station’s preferred candidate. That does not strike me as a reason to delay publication till after the election.

Mike Harwood asked Les Roberts about the breakdown of the violent deaths. Roberts’ reply:

Yes, all 12 non-coalition violent deaths happened outside of Falluja. (1 Kut, 1 Thiqar, 1 Karbala, 7 Baghdad, 1 Diala, 1 Missan, Note Baghdad is about 3-7 times greater in population than these other Governorates so the rates are not so different)

Bombing deaths:

Thiqar

M5, M2, F22 (one family)

Thiqar (different village)

M27

Missan

1mo. & 6mo. in same households (often there are multiple sons with wives under the same roof — interviewer did not record the gender of the infant)

Falluja

10 girls<12 years, 13 boys<12, M14, 25 adult males, 3 adult women (adult defined as 15–59).

The study itself describes the deaths from small arms fire:

only three of 61 incidents (5%) involved coalition soldiers (all reported to be American by the respondents) killing Iraqis with small arms fire. In one of the three cases, the 56-year-old man killed might have been a combatant. In a second case, a 72-year-old man was shot at a checkpoint. In the third, an armed guard was mistaken for a combatant and shot during a skirmish. In the latter two cases, American soldiers apologised to the families of the decedents for the killings, indicating a clear understanding of the adverse consequences of their use of force.

At most two (one from small arms, one adult male from bombing) of the deaths outside Falluja were combatants. In other words, 95% or more of the 100,000 excess deaths were civilians, so it is not wrong to describe the findings as “about 100,000 civilian deaths”.

The biggest limitation of the Lancet study is the small sample size. We can be reasonably confident that deaths have increased in Iraq since the invasion, but the 100,000 estimate is a very rough one. The sample from Falluja found an alarming number of deaths from air strikes, but since it was only one sample it is hard to guess how many others have died in similar ways. Fortunately, it is easy to address both these limitations. For the cost of running the Iraq war for about two minutes it would be possible to do a survey with four times the sample size and which oversampled in violent areas. Unfortunately, Tony Blair doesn’t want to know the answer:

Yesterday Tony Blair rejected a call from more than 40 diplomats, peers, scientists and religious leaders who pressed for an independent inquiry for a civilian death toll.

“Figures from the Iraqi ministry of health, which are a survey from the hospitals there, are in our view the most accurate survey there is,” he told parliament.

The health ministry has produced a figure of 3,853 civilians killed between April and October this year. But it is not clear whether those figures cover the entire country, how they were confirmed or what causes of death. No figures have been produced for the first year since the invasion.

Now, it is a well-established principle of statistics that random sampling will give more accurate estimates than trying to count the entire population. If Blair doesn’t know this, his scientific advisors surely do. I think it is obvious that if he felt that the Iraqi death rate had gone down, he would have been in favour of a survey to establish this.

So here’s a question that should be put to all those who object to the Lancet study because of the wide confidence interval: Do you agree that a larger survey should be carried out?

Also of interest: An interview with Richard Garfield about the study, and a letter to the Independent by the study’s authors (posted by Tribbs in this discussion):

Fallujah is the only insight into those cities experiencing extreme violence (ie Ramadi, Tallafar, Fallujah, Najaf); all the others were passed over in our sample by random chance. If the Fallujah duster is representative, there were about 200,000 excess deaths above the 98,000.

Perhaps Fallujah is so unique that it represents only Fallujah, implying that it represents only 50-70,000 additional deaths. There is a tiny chance that the neighborhood we visited in Fallujah was worse than the average experience, and only corresponds with a couple of tens of thousands of deaths. We also explain why, given study limitations, our estimate is likely to be low. Therefore, when taken in total, we concluded that the civilian death toll was at least around 100,000 and probably higher, not between 8,000 and l94,000 as Mr. Straw states. While far higher than the Iraq Ministry of health surveillance estimates, on 17 August the minister himself described surveillance in Iraq as geographically incomplete, insensitive and missing most health events.

Welcome to the 2004 Deltoid awards. Today we are giving out the Golden Rake Award, named in honour of Sideshow Bob and the rakes in the Simpsons Cape Feare episode:

How many other series would waste valuable prime-time real estate by showing a man whacking himself in the face with a garden rake not once, not twice, but NINE TIMES?!? If ever there was a gag genius in its repetitive stupidity (progressing from funny to not so funny to the funniest thing ever), this is it—merely the sharpest cut in an entire episode that just plain kills.

The award goes to Michael Fumento for claiming over and over again that the Lancet study on excess deaths in Iraq included deaths in Falluja in their estimate, despite being repeatedly shown the clear language of the report, which states:

“We estimate that 98 000 more deaths than expected (8000-194 000) happened after the invasion outside of Falluja and far more if the outlier Falluja cluster is included.”

Notable was the way he would respond to the above quote from the report with a demand that he be shown where the paper gives an estimate that excludes Falluja.

While Sideshow Mike makes his way to the stage to accept his award, read some extracts from his Hate Mail, Volume 33. Sterling Vinson wrote this letter to the Arizona Daily Star

Michael Fumento criticizes a study carried out by the Johns Hopkins Bloomberg School of Public Health and published in the British medical journal The Lancet (”Numbers on Iraqi deaths questionable,” Nov. 6).

Fumento incorrectly states, “The Lancet (claims) the United States has killed 100,000 Iraqi civilians since the invasion.”

The article makes no such claim. The summary says, “We think that about 100,000 excess deaths or more have happened since the 2003 invasion of Iraq.

“Violence accounted for most of the excess deaths, and airstrikes from coalition forces accounted for most violent deaths.”

Note the use of the word “most.” That leaves plenty of room for other causes of death and other killers.

The methodology of the Johns Hopkins study was used in Kosovo and is generally regarded as sound. Moreover, the authors of the Lancet article are careful to qualify their results and to recalculate the number of deaths after excluding those in Fallujah, where the worst fighting is taking place.

Please correct Fumento’s misstatements.

Fumento’s reply:

I reply to a letter sent to The Star concerning an opinion by Michael Fumento published in the paper on 6 November 2004. Dr. Vinson criticizes a study carried out by the Johns Hopkins Bloomberg School of Public Health, and published in the British medical journal, The Lancet.

Actually, Vinson is criticizing Fumento, not the study.

I have read the study, which he just as easily could have since it appeared free and full-text online, but he did not. The term “most” applied only to causes of death other than airstrikes, not causes of death other than coalition actions.

Vinson’s quote comes directly from the study and is completely accurate. Fumento is claiming that the study attributed all the excess deaths to coalition actions, which is not what the study claimed at all. It would seem that Vinson read the study while Fumento did not.

Notice from Dr. Vinson’s own words compared to mine that I actually understated what the Lancet propagandists claimed in that I left out there (sic) modifier “or more.” If anything, perhaps I am guilty of understating just what shoddy work they did.

One of the features of Fumento’s replies is the way he inserts a “(sic)” after any spelling mistakes he finds in the text he quotes. I think that is a kind of petty thing to do, but I thought I’d make an exception for Fumento.

Regarding Kosovo, Dr. Vinson has no idea what methodology was used there. He’s simply repeating what he heard, like an old parrot.

Fumento can’t offer any reason why Vinson was wrong about Kosovo, so he insults him instead.

In any case, I explicitly said that the methodology could be correct but only if it were used in a completely unbiased manner, then went on to show how truly biased the researchers and the journal itself are. I also showed in other ways how the 100,000 figure couldn’t possibly be correct, but Mr. Vinson seems to have neglected that.

Uh oh, is that a rake in front of Sideshow Mike?

Moreover, the authors of the Lancet article CLAIMED to have re-calculated the number of deaths after excluding those in Fallujah, except that strangely enough they never bothered to say what those numbers were.

Thwack! Fumento steps on the rake yet again.

Again, had Dr. Vinson looked at the report itself instead of a 300-word summary published by the AP or some other intermediate media he would know that. Instead he simply establishes what I already written (sic), that people are simply accepting this study because they find it convenient not because they find it convincing.

Actually, Vinson, like everybody else who read the study, including all the other critics, managed to find where the study stated what the number of deaths excluding Falluja were.

Please correct Dr. Vinson’s misstatements and forgive him for his support of Hussein’s henchmen.

“support of Hussein’s henchmen.” I wonder if the Arizona Daily Star approves of one of its columnists abusing its readers?

It’s too long to reproduce here but Fumento also has exchange about the Lancet with A. Michlmayr, where Fumento responds to Michlmayr’s attempt to educate Fumento about confidence intervals and age-adjusted death rates with:

I suggest that if you’re as serious about all this as you seem to be, get on a plane to Syria or Iran and volunteer your services to the terrorists. They might saw your head off on camera, or they might accept you into their ranks.

This exchange is short enough to reproduce in full. No comments from me are necessary. Jonathan Dursi wrote:

I’ve read with some interest your analysis of the recent Roberts et al. in Lancet. As a scientist, I believe science journalism to be a very important connection between the research community and society, and I think science journalists have an important role in informing public debate. Speaking as someone who has actually published scientific papers, your written articles and comments on blogs have been really, really weak and intellectually lazy. If you are as concerned about reporting truth as your protestations about this Lancet article claim, I see no evidence of it in your own writing.

[150 words omitted.]

I’m glad you don’t seem to write about astrophysics. If I were to someday write a paper on something you disagreed with, I imagine you would write just as spiteful articles based on just as weak (or absent) arguments about my own papers. It’s a shame; the world needs good science journalism.

Fumento’s reply:

Dear Mr. Dursi:

Fear not, I don’t write about astrophysics because I don’t know anything about the field. Would that you had such humility. But I do know this definition: “There is a vague notion that astrophysics is more rigorous or quantitativ