Deltoid

Harvard’s David Hemenway has a devastating review of Lott’s new book, The Bias against Guns. Apparently Lott claims that the “impact from closing the gun show ‘loophole’” was a reduction of 102% in Indiana’s auto theft rate, which would have meant that thieves were returning previously stolen cars.

Pro-gun activist Neal Knox has leaped to Lott’s defence. He claims that Lott is in trouble for getting the same result in his survey as Kleck when in fact concerns were raised because Lott’s brandishing number was so very different from Kleck’s. I was going to do a detailed dissection of Knox’s claims, but gzuckier beat me to it.

Mac Diva does not believe the story about the Independence Institute editing Lott’s article. I will post more on this tomorrow.

Julian Sanchez has another thoughtful post on the question of whether it was Lott or Kopel who attributed the 98% to Kleck. I’m still trying to collect my thoughts on this one, but I should correct one statement he makes. Even if the attribution is established to be Kopel’s it does not follow that Lott did not get the 98% by misreading Kleck. Lott got the defensive gun use numbers 2.5 million, 760,000, and 3.6 million,from Kleck but never attributed them to him in his public statements.

Julian Sanchez suggests that if Lott really got the 98% from his survey, then by marrying the 2.5 million Kleck DGU estimate with the 98% brandishing number, Lott is indulging in cherry-picking the numbers most favourable to his position from different surveys. Well, in this case I don’t think that Lott is cherry picking. In statements before May 1999, Lott would say that there were 2.5 million DGUs (Kleck’s DGU number) with 98% of them involving brandishing. After May 1999 (which was when he first claimed that the 98% came from his own survey) Lott switched to saying that there were 2 million DGUs (Lott’s DGU number) with 98% of them involving brandishing. The logical explanation of this is not that he quit cherry-picking in May 1999, but that before May 1999 he believed that the 98% came from Kleck’s survey. Now maybe he did a survey in 1997 that coincidently came up with 98% brandishing, matching the number he thought came from Kleck, but this seems like to much of a coincidence to be plausible.

Noam Alaska doesn’t think much of Lott’s new book.

Mac Diva explains what she thinks is wrong with Lott’s research.

[Note: This is a copy of a document found at this link on John Lott’s website on April 6, 2003. I have added critical commentry, written in italics like this.
Tim Lambert ]

Statement on John Lott’s Survey Work on Self-Defensive Uses of Guns

by David B. Mustard
Monday 10 February 2003

Background

John and I started working on our concealed carry paper in the fall of 1995. I was finishing my Ph.D. in Economics at the University of Chicago, and John was a faculty member. We worked on our paper (more…)

[Note: This is a copy of a document found on John Lott’s website on April 6, 2003. I have added critical commentry, written in italics like this.
Tim Lambert ]

 ------ Forwarded Message From: "Dave Kopel" Date: Sun, 9 Mar 2003 13:07:49 -0700
 To: <cut> Subject: Re: FW: A quick question. John Lott I’ve got no specific recollection of editing the piece, but the evidence seems to indicate that attributing the 98% figure to Kleck was an error by the Independence Institute, rather than an error on the author’s part. Dave Kopel —— End of Forwarded Message 

Apparently, some credence is being given to the claim that I have attributed the 98 percent brandishing estimate to others instead of myself. Some are taking this as evidence that I never conducted the survey. Yet, the fact is I never attributed this number to anyone else other than myself. (more…)

So, was the attribution of the 98% to Kleck’s study in the Lott quote below made by Lott, or did Dave Kopel add it?

“Guns clearly deter criminals, with Americans using guns defensively over 2 million times each year—five times more frequently than the 430,000 times guns were used to commit crimes in 1997, according to research by Florida State University criminologist Gary Kleck. Kleck’s study of defensive gun uses found that ninety-eight percent of the time simply brandishing the weapon is sufficient to stop an attack.”

Our first piece of evidence is Kopel’s recollections:

“I’ve got no specific recollection of editing the piece, but the evidence seems to indicate that attributing the 98% figure to Kleck was an error by the Independence Institute, rather than an error on the author’s part.”

In a later message he says he does “dimly recall adding the attribution”, but this memory may have been been coloured by the fact that other evidence convinced him that he must have done it. I think it best not to rely on on this piece of evidence.

Second, Kopel observes:

“The description of Kleck as a “Florida State University criminologist” is a common phrasing of mine.”

This is a good point. Lott hardly ever mentions Kleck by name, even when repeating Kleck’s results. Consequently it is not hard to find people who think that Kleck’s 2.5 million DGU estimate was the result of Lott’s research (examples are here and here). This piece of evidence suggests that Kopel added the attribution.

Third, Kopel states:

“A version of this same Lott article (without Independence Institute editing) was published in the Rocky Mountain News, and that version does not contain any reference to Kleck.”

While it is possible that Lott produced two versions of his article or that the Rocky Mountain News removed the reference, this also suggests that Kopel added the reference.

Fourth, consider the date the op-ed was published—9th Feb 2000. I made a table (see below) giving the defensive gun use numbers that Lott associated with his 98% claim. The 2.5 million estimate is from Kleck’s survey, and the 760,000 to 3.6 million estimates were produced by Kleck, so associating the 98% with these estimates implies that they also come from Kleck. Lott claims his survey produced an estimate of 2.1 million, so associating the 98% with “about 2 million” implies that it came from Lott’s survey. Five times 430,000 is 2.15 million, so this also implies Lott as a source. Finally, “more than 2 million” could refer to either survey.

Number of defensive uses	Implicit source	Number of citations
		Before May 1999	After May 1999
760,000–3.6 million	Kleck	5	0
2.5 million		5	0
more than 2 million	ambiguous	5	3
about 2 million	Lott	0	13
five times 430,000		0	8

The pattern is clear—before May 1999 Lott consistently implied that the 98% came from Kleck, after May 1999 he consistently implied that it came from his own survey. Since the op-ed was written after May 1999, it is unlikely that he would have attributed the 98% to Kleck.

Fifth, the first sentence of the quote above attributes the “five times” claim to Kleck. In a June 1999 talk.politics.guns posting I suggested that Lott got the “five times more frequently” thing from Kleck and was somewhat surprised to get an email from Lott claiming the five times as his own work. (Lott was not active on Usenet at the time.) Again, it seems unlikely that he would turn around and attribute it to Kleck a few months later.

That is the evidence that points to Kopel doing it. However, there are also two things that point to Lott as the one.

First, why would Kopel make such a major editing change? Kopel’s explanation:

“While editing Lott’s article, I must have seen the figure of two million DGUs in a sentence, and I would have known that the figure came from Kleck, but draft article did not specifically cite Kleck. I then would have seen a 98% figure in the subsequent sentence, and I must have mistakenly thought that both figures came from Kleck. Accordingly, I revised the sentences to credit both “two million” and “98%” to Kleck, in order to help (oops) the readers by giving them the original source of the data.

“In fact, Kleck’s 98% referred to non-injury (no kill, no wound), whereas Lott’s 98% referred to non-firing. Kleck’s 98% figure includes warning shots or missed shots, and Lott’s does not. Kleck’s 98% figure was in very wide use in gun debates from 1988 onward. With the meme of “98% of DGUs are harmless” stuck in my head, I mistakenly failed to notice that Lott’s draft sentence said something different from what Kleck had reported, and accordingly I failed to realize that in the sentence about 98%, Lott was relying on his own research, not Kleck’s. I had read MGLC, but of course the 98% figure only appears in a single sentence; that sentence (2d edition, p. 3), clearly presents the 98% as Lott’s own. But like the vast majority of op-ed editors, I don’t have a staff to cite-check every sentence of everything we publish, and so I make mistakes sometimes. The error was entirely mine, not Lott’s.”

Actually, in Jan 2000 the 2nd edition had not been published. The first edition clearly attributes the 98% to “national surveys”, and the only such “national surveys” that Lott mentions in the book are the ones analyzed by Kleck. Kopel’s belief that the 98% came from Kleck was the only reasonable explanation given the evidence Lott had presented at that time.

I think Kopel presents a plausible case for his having made the changes.

The second reason for thinking that Lott did it, is the fact that Lott is only now denying doing it, despite the attribution being discussed in Lindgren’s report in December and Slate specifically asking him about it in January. Kopel’s suggestion:

“I suspect the answer is that when you show an author something he wrote three years ago, he will not always precisely recall the differences between his original draft and the final published version of an article. Lott may have thought that the phrasing about Kleck was Lott’s own, rather than mine.”

I think that’s correct. But here’s the point: Why did he think it was something he wrote? If the 98% had come from his own survey he would never have attributed it to Kleck, so he would have denied making the attribution. Why did he think he might have made the attribution? The reason is that before May 1999 he thought that the 98% came from Kleck, so he could well have attributed it to Kleck in one of those articles. Probably he didn’t notice that the Independence Institute piece was written after May 1999.

To summarize: I’m not certain, but the attribution in the Independence Institute piece was probably made by Kopel, not Lott. Unfortunately, this does not help Lott very much. His tardy denial indicates that he thought he had made the attribution, and in any case he did falsely attribute the 98% to “national surveys”.

Nor has the Independence Institute or the Heartland Institute acted properly in this matter. Good practice is to indicate that a correction has been made, rather than quietly making a change. For an example, see this Ari Armstrong posting. Speaking of which, I have made a small correction to this post on Julian Sanchez’s comments on cherry picking. He wasn’t saying that Lott was cherry picking, but rather that if Lott really did conduct a survey, then Lott was cherry picking.

After I concluded yesterday that Kopel had probably added the attribution to Kleck in one Lott op-ed, Lott has weighed in, contradicting Kopel’s story. In this posting Lott writes:

“My vague recollection of what happened is that David Kopel (Research Director at the Independence Institute) called me up asking for more information on who had done self-defense surveys and I mentioned that among them was Gary Kleck.”

This is contradicted by Kopel’s account of what he thinks happened (see yesterday for a summary). It is ridiculous to suppose that Kopel would have needed to ask Lott who had done self-defence surveys. Kopel stated he was well aware of Kleck’s work. You can see an earlier reference Kopel made to Kleck’s survey here.

It is very odd that Lott can recollect, even vaguely, something that is so obviously false.

Lott also argues that he “never attributed this number to anyone else other than [him]self”, writing:

“The relevant passage from the op-ed reads:

"Other research shows that guns clearly deter criminals. Polls by the Los Angeles Times, Gallup and Peter Hart Research Associates show that there are at least 760,000, and possibly as many as 3.6 million, defensive uses of guns per year. In 98 percent of the cases, such polls show, people simply brandish the weapon to stop an attack."

If the reference in the second sentence had been to "these" polls and not "such" polls, I would think that the critics would have a much better argument. Instead, I view "such polls" as merely referring back to this type of polls and not those specific polls. Still there is admittedly an error in using the plural. The most plausible explanation is that I was describing what findings had been generated by the polls, in other words I was viewing them in general as a body of research.”

Even if Lott was referring to the polls as a general body of research, this contradicts his claim that he “never attributed this number to anyone else other than [him]self”, since at best, only one of those polls was his.

Lastly, Lott writes:

The bottom line is that there is not a single place where I have directly attributed the 98 percent figure to Kleck or anybody else’s study. The only thing that can be charge is that I likely on a couple of cases must have made some trivial plural/singular mistake.

This is nonsense. Try changing the plural to singular in one of his statements and you get this:

Polls by the Los Angeles Times, Gallup and Peter Hart Research Associates show that there are at least 760,000, and possibly as many as 3.6 million, defensive uses of guns per year. In 98 percent of the cases, such poll show, people simply brandish the weapon to stop an attack.

Changing it to “a poll shows” doesn’t work either. If he really had been trying to say that the number came from his own poll, he would have written “my own poll shows”. This isn’t just some trivial difference.

John Quiggin gives some more examples of folks behaving like the Heartland Institute and the Independence Institute and covering up their mistakes rather than acknowledging them.

Lott has a long message at his website where he discusses Mary Rosh and argues that when he claimed that he had “not participated in the firearms discussion group nor in the apparent online newsgroup discussions”, he was not lying:

Another misunderstanding in the media is that I was lying as to whether I had ever participated in internet chat rooms. I have never made any general statement that I do not participate in such groups. And, obviously, I did participate under my own name for a substantial period of time. There are however two separate statements, one in an email to Glenn Reynolds on 1/13 and one in an email the following day (1/14) to Eugene Volokh that ­ taken out of context - can be made to look like I am making such a nonsensical statement. At issue here was a posting that Glenn Reynolds had made on his web site (1/12), expressing concern that I was not responding to certain questions regarding my surveys. I only learned indirectly that my surveys had been discussed extensively at the Discussion List for Firearms Regulation Scholars. Reynolds’ posting was the first time that I had ever heard of the other blog sites. So my denial of participation was only with reference to why I was not aware of the current debate of my survey work, in particular as to how it had been conducted at the main forum, Discussion List for Firearms Regulation Scholars FIREARMSREGPROF, a list serve site that I was not subscribing to. The statement to Volokh that I had not participated in "the firearms discussion group nor in the apparent online newsgroup discussions" was again specifically for the Discussion List for Firearms Regulation Scholars to let those participating in the discussions there know that I hadn’t been following their debate. (For anybody doubting this, Eugene Volokh can verify that neither Mary Rosh nor I participated in any discussions on this topic in FIREARMSREGPROF prior to 1/14. The same holds true for any other forum debating the merits of my surveys.)

And yet in his confession Lott stated

However, I never subscribed to the firearmsregprof posting hosted by Volokh.

thereby conceding that the part of his statement about participating in the “apparent online discussion groups” was untrue.

At the time that he wrote that message he was an active participant on Usenet, with a posting as recent as Jan 4. Now he is trying to pretend that all he was saying was that he did not participate in online discussion of his survey. This doesn’t make sense. The whole point of the passage was an attempt to explain why he had not responded by claiming that he was not aware of the discussion. In fact, Lott closely followed discussions of Lott on Usenet. It is true that he did not post to online discussion groups about the survey before Jan 14, but that was because he persistently ducked discussion of the survey. For an example, see this posting where I raised the matter of Lott’s survey with him. He responded by asking me what I thought of Bellesiles.

If all Lott was saying was that he hadn’t responded in online discussions to questions about the survey, well that is hardly an explanation for why he hadn’t responded to questions about the survey.

Furthermore, in his Jan 14 email Lott wrote “I am not going be involved in these online groups”. On Jan 15 Mary Rosh posted to a Usenet discussion about Lott’s survey, and joined into discussions in other online groups over the next few days.

John Quiggin has written an article for the Australian Financial Review examining the role weblogs played in the John and Trent Lott affairs. Jim Henley has some interesting comments on gun issues including his assessment of Lott.

I’ve been reading Lott’s new book, The Bias against Guns. Chapter 3 is entitled “How the Government Works against Gun Ownership”. The heart of the chapter is on pages 53–55, where he argues that the National Academy of Sciences stacks its panels against guns.

His first example is their panel on firearms research. He argues that the panel was set up “to examine only the negative side of guns”. Lott writes:

Rather than comparing how firearms facilitate both harm and self-defense, the panel was only asked to examine “firearm violence” or how “firearms may become embedded in community.”

Now look at the first paragraph of the project outline. (I have emphasised the parts that Lott did not tell his readers about.)

In 1997, some 24,000 victims of non-fatal gun crimes suffered gunshot wounds, and 12,382 persons were murdered with a gun. Homicide is outstripped by suicide, which annually accounts for more than half of all people killed with guns. While it is clear that firearms are heavily involved in criminal violence, homicides, and suicides, causal pathways remain uncertain. For example, there is the unresolved question of the potential role of temperament, motivation, and circumstances that is, whether in most cases, persons who commit a suicide or homicide with a gun would, because of these other influences, find another means of doing so if no gun were available. Even if firearms contribute to lethal violence, the development of successful prevention, intervention, and control policies is a complex undertaking. Significant numbers of Americans own guns for sporting, recreational, or defensive purposes. Public officials must sort through important but competing policy objectives involving the protection of Second Amendment rights and legitimate recreational and defensive uses of guns on the one hand, and the lowering the rates of gun related mortality and injury on the other.

Nor can Lott claim to be unaware that the panel considered self-defense, because Lott was a speaker (as well as Gary Kleck) at the panel’s Workshop on Self-Defense, Deterrence and Firearm Markets on Jan 16, 2002.

Lott also claims that as well as rigging the scope of discussion, the government stacked the panel with anti-gun people. He writes that panelist Steve Levitt has been described as “rabidly anti-gun”, citing this article by Dave Kopel and Glenn Reynolds. Lott does not mention that Levitt denies the charge and does not mention that the accuser was anonymous. (For all we know, the accuser could have been Mary Rosh.) Lott goes on to imply that an op-ed that Levitt wrote arguing that swimming pools were more dangerous to children than guns was written for the express purpose of hiding Levitt’s true feelings about guns. (There was some blogspace discussion last September of the Kopel-Reynolds smear of Levitt, with postings from Brad DeLong, Mark Kleiman and Glenn Reynolds.)

Lott’s second example is another NAS panel:

Unfortunately, this is not the only stacked National Academy of Sciences panel. During August 2002, I was asked to participate in a National Academy of Sciences day long workshop on “Children, Youth, and Gun Violence.” I was one of the last people invited for the September 18 meeting. Despite my concerns that I was being included simply so that they could claim that had a “balanced panel,” I was assured by the staff person who invited me, Mary Ellen O’Connell, that the workshop would be balanced. I only attended my session, and at the beginning of my talk I asked the audience of over a hundred people: “How many people here are presenters?” About twenty-five people raised their hands. I then asked of those who were presenters: “How many of you think that it might be possible, not for sure, but just possible, that existing gun control produces more problems than benefits?” All the hands went down. Not one of the presenters was even willing to acknowledge the possibility. Even worse than the bias, the problem was that the academy was unwilling to even acknowledge their biases and were unwilling to engage in a balanced debate.

Unfortunately for Lott, a recording of his talk is available online. At 18:35 in the audio we can find out what he really said:

“How many people are speakers here? How many of those people who are speakers think that on net gun regulations probably cause more problems than good? <pause> Oh, that’s a pretty balanced group.”

And no, the tone of his last sentence was not sarcastic. I also note that one of the people who presumably put his hand down was Dave Kopel, who was a speaker in the previous session. [Update: I am informed by someone who was there that virtually all the hands did go down, so it looks like he was being sarcastic. This doesn’t help Lott very much since the question he actually asked was very different from the question he claimed he asked.]

Just so people don’t have to take my word for the nature of Lott’s attack on Levitt, here are Lott’s exact words. On page 54:

- Another panel member, Steve Levitt, an economist, has been described in media reports as being “rabidly anti-gun.”[10]

On page 289:

[10] Dave Kopel and Glenn Reynolds, ibid., 347. Levitt apparently tried to overcome this image by writing his first op-ed about a week before his name was publicly nominated for the panel. Given that panel members are supposed to not have strong views on the topic that they are studying, it was strange that Levitt would write his first op-ed piece at this time. The op-ed argued that swimming pools posed a greater risk to children than guns, but it is hard to understand why he would choose this very time to write his very first op-ed on this particular topic when this would normally be considered the least appropriate time to do so. When I raised concerns about Levitt’s strong opposition to guns to John Pepper, who was serving as the staff director for the panel, Pepper pointed to the op-ed piece that Levitt had written as evidence that Levitt believed the same things that I believed on guns. Personally knowing Levitt, I know that was not true and one could point to several of Levitt’s academic papers. But the op-ed served its purpose.

Notice how Lott makes it look like Kopel and Reynolds are accusing Levitt of being “rabidly anti-gun”, even though the accuser was anonymous.

Otis Dudley Duncan

This discussion is concerned with four topics: (1) Lott’s references to, remarks about, and discussions of DGU statistics originating in sample surveys or polls carried out by other investigators; (2) Lott’s claims about a survey he says he conducted in 1997; (3) Lott’s reports on a survey he conducted in 2002; (4) several matters that have proved to be distractions from the careful consideration of the foregoing. Section (5) presents my conclusions. They may be read at once by anyone who has followed closely the Internet exchanges about the Lott case.

(more…)

I asked Steve Levitt about Lott’s attack. He comments:

I wrote that op-ed piece on swimming pools and guns almost a full year before it was published. Members of the U of C publicity department can attest to that. I wrote it at the tail end of the summer, so they suggested waiting until the beginning of the next summer to try to publish it, which I did. I had certainly never heard of any NRC panel at the time I wrote it. I wrote it because it is the truth and it is an important point

So Lott was wrong about the timing of when Levitt’s op-ed was written. Lott either knew this or was grossly irresponsible in not finding it out.

I also got some comments from John Pepper, director of the NAS study on firearms:

Lott’s critique, as described on your web page, seems rather premature. We have not finished our work and, as you correctly note, we have heard formal presentations from Lott, Kleck (2x), Bronars, Wright, Blackman, and many others typically associated with the “pro-gun” side of the debate. I can assure you, as I have assured Lott and others (e.g. the NRA and Paul Blackman), that we are considering research and data relevant to both the positive and negative aspects of firearms.

John and I did speak about his concerns regarding Steve Levitt and several other committee members. I did not try to persuade John that Steve Levitt shared his views. Rather, I recall trying to get specific documentation from John and others making similar claims on Steve’s purported biases. At that time (July/Aug 2001), I knew about Levitt’s short theoretical paper in the American Economic Review, and the editorial, neither of which would indicate strong biases for or against the Lott research or firearms research more generally. Is there anything else? I do not recall whether we specifically discussed the Levitt editorial, but it is possible within this context. As an aside, I still have never seen documentation that would indicate that Levitt is biased in favor of any particular side on this debate, and certainly nothing supporting the claims that he is rabidly anti-gun.

Finally, in regards you April 9, 2003 commentary on the August 2002 meeting you should know that this was not an NAS panel but rather was a workshop held an NAS facility at the request of the Packard Foundation to showcase their work on kids and guns. It did not result in an NRC report and the comments of particular speakers were not endorsed by the NRC. Rather, it was a discussion centered around the Packard report, the record of which is up on our web-site (as you know). The Packard report can be found at: http://www.futureofchildren.org/pubs-info2825/pubs-info.htm?doc_id=154414.

It seems that Lott has misrepresented his discussion with Pepper. Lott has not the slightest scrap of evidence for his smear of Levitt. His unprincipled attack on Levitt tells us more about Lott than it does about Levitt.

You can read Steve Levitt’s op-ed on pools and guns here. It is quite clear from the op-ed why he wrote it: he lost his son and he didn’t want another parent to lose a child to a preventable accident. I am totally disgusted with Lott for accusing Levitt of exploiting the death of his child to cover his “rabidly anti-gun” views. I’m too angry to write any more.

First, a recap and a time line on the Kopel/Lott/Reynolds attacks on Steve Levitt:

16 Aug 2001

Glenn Reynolds claims that the NAS panel is “stacked” with “ardent supporters of gun control”, especially Levitt.

29 Aug 2001

Dave Kopel and Glenn Reynolds write an article in National Review Online where they claim that most of the people on the panel are anti-gun and that Levitt has been described as “rabidly antigun”. They offer no evidence to support their attack on Levitt.

29 Aug 2001

Levitt emails Reynolds, denying the charge, pointing to this op-ed as evidence that he is not rabidly anti-gun. Reynolds conceals the identity of the “scholar” who accused Levitt of being “rabidly antigun”.

31 Aug 2001

Fox News has a story about the attacks on the panel, quoting Lott as saying “It’s not a balanced panel” and Kopel charging that the real intent of the committee was to debunk Lott’s work.

18 Sep 2002

Reynolds repeats his attack on the NAS panel. Brad DeLong vigorously refutes Reynold’s attack on Levitt.

19 Sep 2002

Mark Kleiman comments on Reynolds’ unethical behaviour:

I still think the NRO story was shabby journalism: it made a wild, personally damaging charge based on a single anonymous source, it failed to check that charge with its subject to allow a comment, and it misrepresented the document it quoted from by selective quotation and actual misquotation. Nor was it ever updated with links to the denial and the evidence supporting it.

Reynolds has a bizarre response that completely ignores all of Delong’s and Kleiman’s arguments:

As a former official in the Clinton Administration, surely DeLong isn’t arguing that only people’s buddies are entitled to discuss questions of whether they might be biased or not. He should know better than that.

(Err, discussing it is fine, Glenn, but you should either offer actual evidence in support of your claim or retract it.)

10 Mar 2003

Lott’s book The Bias Against Guns is published. On page 54 he writes (referring to the Kopel-Reynolds article):

Another panel member, Steve Levitt, an economist, has been described in media reports as being “rabidly anti-gun.”

He goes on to falsely accuse Levitt of writing his op-ed to cover up his “strong opposition to guns” and misrepresent his discussion with Pepper about Lott. (For details see here.)

Now, who was the anonymous accuser whose identity Reynolds concealed? Well, who do we know who makes anonymous personal attacks on academics like this or this? And who would have a vested interested in keeping someone highly skilled in econometrics off the panel, lest their own firearms research relying on econometrics be debunked? And whose research on the abortion-crime link did Levitt describe (on June 19, 2001, shortly before the anonymous attack on him) as “just garbage”? And since the “rabidly antigun” charge has proven to be false, who do we know who has a habit of making things up?

Levitt informs me that he is almost certain that the anonymous character assassin is Lott. I asked both Reynolds and Kopel to confirm or deny this and neither one denied it. I think we are entitled to conclude that the character assassin is Lott.

It was clearly wrong for Kopel and Reynolds to print Lott’s smear without telling their readers of their source’s interest in the matter. And it was dishonest of Lott to refer to the Kopel-Reynolds article to make it appear that there were other people who were saying that Levitt was “rabidly antigun” when, in fact, Lott was the only source of the claim.

Kevin Drum has a nice summary on Lott’s anonymous attack on Levitt. Kieran Healy tells what Lott’s next step will be. Brian Linse thinks Reynolds and Kopel should offer some answers. Atrios links here. And Tom Spencer has two posts. First, he observes that Reynolds’ cover up for Lott raises questions about Reynolds. Second, he is impressed by Lott:

However, can you imagine the chutzpah on the part of Lott to quote an article in a book that is quoting himself as an unnamed source to bolster an argument he’s advancing in the book? You’ve got to give it to Lott, he certainly has, uh, like I said, chutzpah.

Glenn Reynolds has posted a reply to the firearmsregprof mailing list. Perhaps the most interesting thing about his reply is what he doesn’t say: he doesn’t deny that Lott was his anonymous source. Anyway, he claims that I omitted that he

“published Levitt’s response that he wasn’t rabidly anti-gun, and took him at his word.”

I did not mention that Reynolds “took him at his word” because he didn’t. Reynolds published Levitt’s denial, commented on the op-ed, and wrote that his source was sticking to the charge. Reynolds was quite clearly agnostic on the question of whether Levitt was “rabidly antigun”, writing:

“I suppose the real test of his fair-mindedness will be how he conducts himself on the study”

Reynolds also claims:

“Kopel sent an update to the NRO piece some time ago stressing Levitt’s denial of the charge. Although Lambert doesn’t mention this, I imagine that he’s aware of it.”

Neither Reynolds nor Kopel mentioned this in their emails on this subject. I also have corresponded with Levitt, Kleiman, DeLong and Pepper on this matter, and none of them seemed to be aware of this update.

Notice also that Levitt denied the charge on the same day that the article was posted. It has now been over 18 months and the article still has not been corrected.

Mac Diva is trying to figure out why Lott does the things he does. Atrios explains why he cares about Lott. Brad Delong says that I have “a very strong case”. Matt Yglesias has some thoughtful comments on appropriate behaviour in this case. ArchPundit has one two posts.

On Monday Glenn Reynolds wrote:

Kopel sent an update to the NRO piece some time ago stressing Levitt’s denial of the charge. Although Lambert doesn’t mention this, I imagine that he’s aware of it. I don’t know if it has appeared on their site yet or not.

It turns out that “some time ago” was Reynolds’ special way of saying “yesterday”. Nor, of course, would he have any reason to believe that I would be aware of this update. Kopel says he thinks Reynolds may have “misremembered”.

Anyway, here’s Kopel’s update:

Shortly after this article was published, Steve Levitt wrote to Glenn:

“I don’t understand your National Review article in which I am described as ‘rabidly anti-gun.’

“No one who knows me would describe me that way. I love to shoot guns and would own them if my wife would let me. I recently published an op-ed piece in Chicago Sun-Times entitled ‘Pools more dangerous than guns’ (July 28, 2001) that could only be construed as pro-guns. I have never written anything even remotely anti-gun. I think your sources must have me confused with someone else.”

Levitt’s Sun-Times article argues that the risks of gun accidents are grossly exaggerated by the media compared to other accident risks. I wrote back to Levitt something which I should have also asked to be posted on this article, so I’m belatedly posting it now:

“As Glenn’s instapundit site details, we have checked with our original source. Nevertheless, since I try (not always successfully) to shed light rather than heat on the gun issue, I think that in retrospect the adverb ‘rabidly’ shouldn’t have been used. So I promise to avoid it in the future. I’m glad to know about your swimming pools piece, and I enjoyed reading it. I did check your publications page on the web before I submitted the article, but the pool piece wasn’t there—understandably, since your page just cites journal articles.

“And, as the article said, whatever your views on guns, there’s no dispute about your scholarly abilities. My forthcoming article ‘Lawyers, Guns, and Burglars: Why Mass Tort Litigation Fails to Account for Positive Externalities and the Network Effects of Controversial Products’” 43 Arizona Law Review (no 2, 2001) cites and discusses your excellent LoJack article.

Kopel’s correction is inadequate. He doesn’t withdraw the charge at all, he just allows that it was too inflammatory. Here’s what he should have said:

In my article I quoted an anonymous claim that Steve Levitt was “rabidly antigun”. I have investigated the matter and I find that the charge is false. I withdraw it and apologize to Levitt for the unwarranted attack on his character and to my readers for inadvertently misleading them. I also apologize for not not making this correction in a timely fashion.

Max Sawicky links here, as does Brad Delong and Hesiod.

Meanwhile, in a post that seems to have drifted in from some alternate reality, the William Sjostrom take on the Kopel/Reynolds/Lott attacks on Levitt is that Brad Delong is a sleaze.

In a previous message Glenn Reynolds claimed to have taken Levitt’s word that he wasn’t rabidly anti-gun. In his blog it did not seem that Reynolds had taken Levitt’s word, so I asked him to clarify his position. He wrote:

I was quite clear in my InstaPundit post on this: I don’t know Levitt. Someone who I found credible told me he was rabidly antigun. He says he isn’t. (And, the way things work, both may be honestly giving their opinions). The proof will be in how the Committee works.

So when Reynolds wrote that he

“published Levitt’s response that he wasn’t rabidly anti-gun, and took him at his word.”

he actually meant that he

“published Levitt’s response that he wasn’t rabidly anti-gun, and did not take him at his word.”

I also asked Dave Kopel to clarify his position on the “rabidly antigun” question. He believes that Lott his source is correct, but regrets using an inflammatory word like “rabidly”. I asked for the evidence that his source produced that convinced him and was informed that it was secret. (Apparently presenting any evidence at all in support of the claim would reveal the identity of his source.)

Since neither Reynolds nor Kopel have retracted their attack on Levitt, I believe it is germane to examine the accuracy of the other statements they make in their article.

1. They claim that the panel ignores “research into the benefits of firearms.”. As Mark Kleiman pointed out last year, the project description quite clearly includes research into the benefits of firearms. Reynolds claims that the language about benefits wasn’t in the document that he looked at. However, the Wayback Machine shows that the language was in the page that they linked to. (The date on the Wayback Machine’s copy is a little later than their article, but the internal evidence on that page shows that it hadn’t been changed since before their article was written.)
2. They claim that:

   The committee members were not given even one of the many social-science articles detailing the failures of various gun-control laws.

   This is contradicted by Paul Blackman, who on the firearmsreg list stated that 75-80% of the articles were “anti-gun”.
3. They assert that a 1993 study by Kellermann et al that found that gun ownership was associated with a three times greater risk of becoming a victim of a homicide in the home was “junk science”. They argue that

   hardly any of Kellermann’s murder victims were killed with a gun from their own home, and a significant number of the murder “victims” were lawfully killed by police, and the whole factoid disappears once you account for the true rates of gun ownership among the “control group” of people who weren’t murdered

   All of these claims are false. Enough of the victims were murdered by the gun in the home so that all of the extra risk from guns was associated just with gun homicide and homicide by people with access to the home gun. Only four (out of 400) were killed by police. You can speculate that the correlation may have be caused by gun ownership being more underreported in the control group, but that does not make the speculation true.

With so many inaccuracies in their article, it is probably best to entirely discount their claims about Levitt.

Ted Barlow thinks that Mac Diva overstates her case against Lott’s views on non-gun issues. I agree with him. While it is relevant to note that Lott’s research always seems to produce results supporting a right-wing agenda, in most of those issues he does not indulge in advocacy. You can’t say that he thinks that woman’s suffrage was detrimental because he produces a study purporting to show that it made the government bigger (something that Lott would consider bad). Lott might believe that giving women the vote had some benefit that outweighed any costs. In any case, he is not out there arguing against votes for women like he is arguing against gun laws.

Otis Dudley Duncan has written an excellent article on Lott and defensive gun use surveys. I’ll quote from the conclusion, but you really should read the whole thing:

Investigators are obliged to tell the truth about what they take from the work of other investigators and to provide verifiable evidence and complete documentation for statements made in reports on their own research. They are responsible for telling the “whole truth” about it, to use the legal phraseology, and for enabling others to confirm or falsify their results. As far as his claim about the evidence on gun brandishing obtained in 1997 is concerned, John Lott has failed on these counts. Keep in mind that the burden of proof is his, as in all inquiry. In the vernacular, Put up or shut up.

Despite his admission that he cannot document his 1997 survey, Lott continues to discuss the alleged findings from that project as if they are acceptable as statistical evidence. (See “What Surveys Can Help Us Understand About Guns?” cited earlier.) It appears that he has learned nothing from his critics about the ethical requirements of the scientific enterprise. So it is time for Lott’s supporters to advise him that his best course of action now is to retract his claims concerning the 1997 survey. Say-so and “recall” that gets more elaborate as time goes by are simply not acceptable. And it is long past time for him to retract his manifestly false allegations about what other investigators have found. His failure to do so is much more reprehensible than the Mary Rosh foolishness.

People who violate traffic laws can be sent back to school to relearn the etiquette of the road. Would that there were a comparable resource for those who violate professional ethics.

Roger Ailes comments on Reynolds and Kopel’s failure to show any evidence in support of the “rabidly antigun” claim. Greg Beato also has some extensive comments. Tom Spencer has two comments, as does Atrios here and here.

Last September Lott told Lindgren that he “weighted his respondents by demographic information taken from his main national study in More Guns, Less Crime”

On January 14 he provided more details:

I did not weight the sample by household size but used the state level age, race, and sex data that I had used in the rest of my book. There where 36 categories by state. Lindgren hypotheses why you can get such small weights for some people and I think that this fine of a breakdown easily explains it. I don’t remember who answered what after all these years, but suppose someone who fired a gun was a elderly black in Utah or Vermont.

I tried to weight the data in Lott’s 2002 survey using this procedure. The first difficulty I encountered was that the age categories he said he used were 10-19, 20-29, 30-39, 40-49, 50-64, and 65+, but his survey collected age by decade. There is no way of knowing which category to assign someone in their 60s to. Also, the people in the survey were 18 or older, so using a category of 10-19 makes no sense.

There is, however, a far worse problem. In states with a small population there were only two or three people in the sample. With 36 categories for each state, that meant that most categories in those states had no data. It should be obvious that no amount of weighting can correct for the problem of having no data.

All that can be done for the categories with no data is give them weight zero. But this means that the weighting procedure is systematically biased against minorities and people from small states. An example might make this clearer: Suppose that we just weight by race and sex and 90% of the population is white and 10% is black. We take a sample of ten people, getting nine whites (four men and five women) and one black (a man). The four white men constitute 40% of the sample and 45% of the population so get weight 45/40=1.13. The five white women are 50% of the sample and 45% of the population so get weight 45/50=0.9. The black man is 10% of the sample and 5% of the population so gets weight 5/10=0.5. Notice that the total weight of the whites in the sample is 4×1.13+5×0.9=9, 18 times the total weight of the blacks, even though they are only 9 times as frequent in the general population. In this example whites were weighted twice as heavily, but with 1836 (=36×51 states+DC) categories Lott’s procedure could cause even worse discrepancies.

I didn’t notice this problem until I took his data set from the 2002 survey and tried to follow his procedure. With 6 out 7 people brandishing (86%), after weighting using his procedure, the weighted brandishing percentage is 99.8%. It turns out that the person who fired was a minority woman from a small state, and ends up with a weight about 100 times as small as that for the other defensive gun users. Obviously this is incorrect.

Now in Lott’s new book, The Bias Against Guns (published in March) he claims to have weighted the data by race and sex (only six categories). Unlike the weighting procedure he claimed to have followed in September (and he repeated the claim in January), this is a reasonable procedure. (Though, of course, the sample size is too small to give even a rough estimate of the brandishing percentage).

One explanation for the change in Lott’s story is that in January he had not done any weighting and it was only when he tried to do the weighting that he discovered that the procedure he described in January was not valid.

Incoming links from Tapped and American Politics Journal.

Glenn Reynolds finally gets around to blogging on the accusation in his article that Levitt is “rabidly antigun”. Remarkably, Reynolds does not mention or link to any of the discussion about this that has occurred on many blogs.

Kevin Drum comments on Lott’s weighting scheme. He also links to a January posting which has this explanation from Lott explaining how he might have got a weight of 1/8 from his weighting procedure (my emphasis).

Whether it is possible depends upon how finely you do the weighting. If you do something as simple as national weighting, you are right, it would not be likely. But if you are willing to put in the effort to break things down into enough categories it becomes quite likely. I just looked up some different numbers from 2000 to give you a rough idea. In Montana, black males make up .14 percent of the population. In Mississippi, they make up 18.8 percent. That is a difference of 134 fold, quite a bit bigger than your 8/1 ratio. Obviously, this is an extreme difference and the difference that Lott must have come across is only about 1/17th as large. If he broke things down by age in addition to race and gender, I am sure that you could easily get difference much bigger than 134 fold. My impression is that at least on this point Lindgren is “making a mountain out of a mole hill.”

Lott is now claiming that he did do national weighting, so by his own admission, it is not likely that he could have got a weighting of 1/8.

James Lindgren writes:

The weighting problem you note for the claimed 1997 survey bears repeating. Things are more serious than I think some readers might suspect from your example. I’m not sure that I can convey the problem adequately in words.

The purpose of the sort of weighting Lott claims to have done is to adjust things so that if there are a few more of any one group in your sample in a state and a few less of any other group, you weight their answers accordingly, so the total is still the same. For example, assume you start out with 200 Californians in your sample, with a few more whites in your sample than the population and few less blacks. You reweight the sample so it still comes out at 200 Californians, but some are weighted more than 1 and others are weighted less than 1. The average of all respondents’ weights is exactly 1, and the sum of the weights of all 200 respondents is still 200. (I leave out the complication of more complex sampling designs and design effects that Lott said he did not use and that he did not have the expertise to use.)

So there are 36 categories for each state and 51 jurisdictions, which means a staggering 1836 categories with 1836 different individual weights. But Lott says he had only 2424 respondents in his 1997 study. A small state with about half a million people in the 1990 census (Wyoming, Vermont, DC, North Dakota, etc.) would have only about 4-6 respondents, but 36 categories for weighting those handful of respondents.

This doesn’t work in practice. Let’s say that all 4 of the respondents in a given small state were white, one white male aged 35, one white male aged 51, one white female aged 22, and one white female aged 45. Let’s say that for your sample of 4 people, males and females each make up 7% of the state’s population in each decade. Each would have their answer weighted by .25/.07=.28. These four categories with data together make up 28% of the population and thus would contribute 28% of the state’s total in Lott’s study. But where does the other 72% of the state’s total come from? There are no respondents in those categories making up 72% of the state to weight. In all, 32 of the 36 categories would be empty.

So adding all 4 respondents together, the total weighted count for the small state is only (4*.28)=1.12 of a respondent. Yet the total weighted number of respondents for that state should equal 4 respondents, not 1.12 respondents. The small state would contribute only 1.12 respondents to Lott’s totals, not the 4 people it is supposed to.

Note that in my example above for weighting California, you had 200 respondents both before and after weighting. But in Lott’s methods, in a small state you would get 4 respondents before weighting, and 1.12 respondents after weighting—if you are lucky; if you are unlucky, you might get a respondent from a small group with a low weight or 2 from the same group and get even less of the total. When added to the very large states, which would have enough respondents to cover most of the 36 categories for each state, the resulting data would be garbage, because large states would overcontribute to the total.

In short, weighting by 36 categories in each state sounds like something someone might come up with off the top of his head, but it is hard to believe that someone would really go through using it, once they realized that their weighted totals didn’t come close to adding up to 2424. I think it highly unlikely that a social scientist would actually try to weight by 1836 weighting categories when he realized that the great majority of the weights couldn’t be used. Indeed, Lott didn’t use the unworkable 1836 category weighting in his latest 2002 survey.

If there were some way to estimate the counts for the missing cells (which there isn’t in this case) and most of the cells were not empty, one might consider using such a method.

It doesn’t help Lott’s current account of the sampling and the weighting that he first told me that he drew a random sample from the CD but didn’t remember how, but then wrote: “Not true. I told Jim that one of the students had a program to randomly sample the telephone numbers by state. My guess is that it was part of the CD, but on that point I can’t be sure.”

When Lott called me on this matter, I was listening closely to Lott’s answers to questions about methods to see what details he could remember on the spot. I asked him how he drew the sample from the CD, which he said that he thought he got from a student. Lott said that he didn’t remember how he sampled, but assured me it was drawn randomly. I remember being disappointed in his answer because I thought that a social scientist would probably remember how he solved this problem of getting a random sample (there are several solutions). Also, I am absolutely positive that he did not mention pre-stratifying the sample by state, which is a form of stratified proportional sampling, not random sampling. He seems to be claiming now that he drew the sample proportionally by state then randomly within states. This discrepancy is not large by itself, but it is troubling nonetheless.

This comment about weighting problems is confusing. But I find it hard to believe that Lott (or anyone) would use the weighting method he claims to have used in 1997, not just because it is substandard, but because it doesn’t work at all when most of the cells are empty. Almost anyone trying to use it would immediately realize this and reduce the number of categories, such as removing the age breakdowns, which I gather from comments on the internet is what Lott did in his 2002 survey.

Links from Guy Cabot, Alan Schussman (twice!).

Some responses to Glenn Reynolds’ post yesterday: Tbogg considers Reynolds to be washing his hands and changing the subject. Tom Spencer observes that it is dishonest of Reynolds to respond to criticism without providing a link to that criticism. Roger Ailes reckons that Reynolds is being unfair to Lott by calling him “disingenuous” for not mentioning Levitt’s denial of the “rabidly antigun” charge. After all, Reynolds did not bother correcting the article and it unreasonable to expect Lott to have read the correction on Reynolds’ weblog. However, it is reasonable to expect Lott to be aware of Levitt’s denial if Lott is Reynolds’ anonymous source. Ailes also has some pointed comments about Reynolds’ misrepresentation of what the controversy has been about. Note further that Reynolds still has not explained why he called Levitt an especially ardent supporter of gun control.

Dave Kopel has also blogged on the affair. Like Reynolds, he avoids mentioning or linking to any of the recent discussion about this matter. Anyway, he starts with a false claim:

Perhaps our warnings had some effect; the panel’s “charge,” which we linked to from our article, focused only on examining the negative effects of firearms in society. That link is no longer operative, and a more detailed charge has replaced it; the new charge requires the panel to also consider beneficial aspects to firearms ownership.

You can see the original charge that they linked to here, courtesy of the Wayback Machine. You can compare it with the current version to see that there has been no significant change. Mark Kleiman also checked with someone on the panel who stated that the charge had not been changed. Numerous other errors of fact in the Kopel/Reynolds article are detailed here.

Kopel then points to two articles by Levitt that Kopel claims are “anti-gun”. The most important point is that not even Kopel is willing to claim that that the articles are “rabid”. Having failed yet again to present any evidence that Levitt is “rabidly antigun”, the decent thing to do would be to withdraw the charge and apologize for the unfounded slur, but Kopel does not do this.

Anyway, Kopel somehow construes the two Levitt articles as being anti-gun. He does not dispute the accuracy of Levitt’s statements, so apparently making any correct statement about harm done with guns means that you are “anti-gun”. By this measure, even Lott is “anti-gun”. Nor can Kopel claim that Levitt just concentrates on the harm and avoids the potential benefits, since the first paper models the effects of concealed carry laws on crime.

Kopel finishes with:

There is nothing logically inconsistent with a scholar favoring gun control to address the very large problem of criminal homicide with guns, while also recognizing that the magnitude of the problem of fatal gun accidents involving children is not nearly as large as the media imply.

Nice attempt to shift the burden of proof. If Kopel makes an unsubstantiated claim about Levitt being anti-gun, then it is up to him to prove it. It is not the responsibility of others to disprove it.

It would seem that some wag has had some fun at poor Professor Reynolds’ expense. Reynolds has an update with an email supposedly from one Brendan Dooher that reads:

I worked with the study director at the National Academy of Sciences (he is actually in the National Academy of Engineering) when I was a Fellow there last year. I can tell you in no uncertain terms that the study is heavily biased. I made myself persona non grata there over my year because of my conservative (but always scientifically based) views. The committee’s first meeting had multiple speakers from Hand Gun [Control] Inc and other anti-gun types giving testimony - but no one to speak of the positives. I asked him if he would have Professor John Lott speak and the reply was a sneer. (I should state that the study director was a typical liberal type - goatee, whiny voice, upset at the stolen election - much like most of the people I encountered there (except the goatee…)

However, in the same posting Reynolds writes:

I noted a report by Sam MacDonald of Reason who attended the first meeting of the panel and who thought it seemed reasonably fair.

And if you look at MacDonald’s report you find:

the committee did hear from an NRA spokesman, and there was some talk of trying to calculate the benefits of gun ownership along with the costs.

Reynolds also mentions the “considerable discussion on an email list”. Prominent in that discussion was the fact that Lott had spoken to the panel as you can see here. I don’t think it was right for “Dooher” to play such a mean trick on Reynolds, but Reynolds really should have noticed that the stuff “Dooher” was telling him was plainly false.

Expect to see something like the following at InstaPundit.com soon:

FURTHER UPDATE: Archbishop Heywood Jablome, reader and astronaut, emails:

I can confirm that the NAS Panel is like, totally biased. I talked to the panel commissar and while twirling his moustache he told me: “Our task of creating a socialist America can only succeed when those who would resist us are totally disarmed.”

Interesting. I checked Google and got 529 hits for “Heywood Jablome”, so that establishes his bona fides. Heh.

Tom Spencer comments on Kopel’s “update” to his attack on the NAS panel.

ArchPundit has a short post about Lott’s weighting and a long, interesting post on the Kopel/Reynolds attack on the NAS Panel. I think he nails the problem with the Kopel/Reynolds approach when he writes:

Perhaps it is their background as law professors that is the problem. While law journals serve their purpose, I’m a bit mystified by this almost post modern view of social science Kopel and Reynolds seem to be promoting. A fair panel is one that examines the issue from a social scientific view–not just a balance of pro and con.

David Kaun, who is Professor of Economics at University of California Santa Cruz has an article over at BuzzFlash discussing Lott’s More Guns, Less Crime.

I was mistaken when I suggested that the email Dooher sent Reynolds was a hoax. I emailed Dooher asking him if he had written the letter and when I didn’t get a reply, because of the weird stuff about goatees and because he got every single fact wrong (including the claim about the study director having a goatee) I decided that it was a hoax. Dooher eventually replied, apologizing for his mistakes and asking Reynolds to remove his email. Reynold’s spin? OK, so they had Lott speak, but maybe they were planning a biased study and Dooher talked them out of it. Tom Spencer and Atrios linked to my comments about Dooher’s email. Atrios also has some good comments on ArchPundit’s thoughts on Reynolds’ post-modern approach to science. ArchPundit has an update to his thoughts about Reynolds and the NAS panel. And the Iraqi Information Minister apparently supports Lott and Reynolds.

Donald Kennedy, Editor-in-Chief of Science has an editorial (subscription required) in the April 18 edition entitled “Research Fraud and Public Policy”. Here is some of it:

Michael Bellesiles, of Emory University, supported the gun control case with a book called Arming America. Part of his argument was that guns were rare at much earlier times in U.S. history. Challenged on that claim, he failed to produce the data, claiming that an office flood had destroyed his records. Emory empaneled a committee of scholars to investigate, and its report questioned Bellesiles “scholarly integrity.” He resigned from the Emory faculty, and the Bancroft Prize his book had won was revoked. The pro-gun faction began to chortle with glee; end of story, right?

Not so fast. Here is John Lott: ex-University of Chicago Law School, now at the American Enterprise Institute. His book More Guns, Less Crime claims that on 98% of the occasions in which citizens use guns defensively, the mere production of a weapon causes the criminal to desist. These data were allegedly based on some 2000 interviews conducted by Lott himself. But when pushed for the survey data, Lott gave a hauntingly familiar explanation: His hard drive had been destroyed in a computer crash. Apparently the dogs in this controversy eat everyone’s homework.

Wait. It gets even funnier. As the debate over gun laws spilled over from the scholarly journals to the Internet, Lott was defended passionately by a persistent ally named Mary Rosh. She attacked Lott’s academic critics, including John Donohue of Stanford Law School, claiming in one posting that Lott had been the “best professor I ever had.” Alas for Lott and his case, Mary Rosh now turns out to be—John Lott! The American Enterprise Institute has not yet followed the example Emory set with Bellesiles, though it might think about it.

Meanwhile, though, legislators in a number of states are still considering liberalizing concealed-weapon laws, and Lott’s book plays a continuing role in the debate. That moves this story from high comedy to a troubling challenge in social policy that isn’t funny at all. Death by shooting is a national public health problem. Sound social science, not cooked data, is what we need to work out the tough problems like the relationship between gun ownership and violent crime.

Oh, and if you think that last bit is unfair because no-one has shown that Lott’s data on concealed-weapon laws was cooked, well, stay tuned.

A few days ago I observed that Lott had changed his story from his original, unworkable, claim that he had used 1836 categories (sex, race, age and state) to weight his data to the claim that he had used just six (sex and race). If this is indeed the scheme he used then two things follow:

1. He has incorrectly calculated the brandishing number for his 2002 survey.
2. It is impossible for him to have obtained a 98% brandishing number for his 1997 survey.

Here are the details:

In his new book, Lott tells us the brandishing number he gets after weighting:

“the survey I conducted during the fall of 2002 indicates that simply brandishing a gun stops crimes 95 percent of the time”

After Lott spent years claiming that the brandishing number was 98%, I can now claim victory. He has stopped making the 98% claim. As of this month he now claims:

Something like 95% of the time or so simply brandishing a gun is sufficient to go and cause a criminal to break off an attack.

I guess I should be pleased that he has stopped using a number based on nonexistent data and started using a number based on a real survey. I only have a few remaining quibbles:

1. The sample size of just 7 defensive gun users is far too small for a point estimate like 95% to be at all meaningful.
2. It is dishonest not to mention the markedly different estimates from professionally conducted surveys with samples containing hundreds of defensive gun users.
3. Lott hasn’t even calculated the point estimate correctly. It should be 91%, not 95%.

Weighting Lott’s 2002 survey isn’t particularly complicated. I’ll go through the whole procedure here, so you can check my working. To work out he weights you need to count the number of adults in the US in each category, and the number of people in Lott’s survey in each category:

Category	Population	# in sample	weight
Black Male	10,821,013	32	1.52
Black Female	12,790,596	55	1.04
White Male	72,496,265	333	0.98
White Female	77,992,817	424	0.83
Other Male	17,591,994	41	1.93
Other Female	17,586,464	54	1.46
Total	209,279,149	939

Looking at this we see that other race females make up 17,586,464/209,279,149=8.4% of the population, but only 54/939=5.8% of Lott’s survey. That means responses in that category must be given weight 8.4/5.8=1.46 to compensate for their underrepresentation in the survey. The other weights in the last column are calculated in a similar fashion.

Second, we look at all the defensive gun users in Lott’s survey. Here is the complete data set:

Category	Weight	DGU count	Weighted DGUs	# times gun fired	Weighted times fired
Other Female	1.46	1	1.46	1	1.46
White Male	0.98	2	1.95	0	0
Other Female	1.46	2	2.92	0	0
Black Female	1.04	1	1.04	0	0
White Female	0.83	2	1.65	0	0
Other Male	1.93	2	3.85	0	0
White Male	0.98	3	2.93	0	0
Totals	8.67	13	15.81	1	1.46

All we do is look at which category each case falls into to determine its weight and multiply that weight by the number of defensive uses and firings. The result is that there were 15.81 defensive uses after weighting and 1.46 of these involved firing, so the weighted firing percentage is 1.46/15.81=9%, or, in other words 91% brandishing, contradicting Lott’s 95% figure that he claims came from this survey. My calculations are also available in a spreadsheet.

Back in January, Lott responded to my question

Even Lott cannot possibly be sure that the correct result of his survey was 98% since there is no way to check his calculations. Why did he repeat the figure over and over again?

with

Is there any evidence to suggest that I can’t figure out a weighted average?

Well, there certainly is.

Even more troubling is the fact that Lott now recalls that in his 1997 survey there were two cases of firing. For these two cases to constitute 2% of the (weighted) 25 uses Lott must have had, their total weight must have been 2% of 25 or 0.5. That means that at least one of the weights must have been 0.25 or smaller. And to get a weight of 0.25, Lott’s 1997 survey would have had to have four times as many people in the appropriate category as occur in the general population. Clearly that is not even possible with a category like “White Male” which makes up 34% of the population. The most likely way this could have happened is with the smallest category, “Black Male”, which makes up 5% of the population. Five percent of 2424 people (the size Lott claimed for his 1997 survey), is 125 black males. If that is the expected number, what is the chance of getting 500, four times as many? I worked out the probability and it is less than one in a googol (that’s a one followed by one hundred zeroes). That counts as impossible in my book. In other words, if Lott used the weighting scheme he now claims to have used there is no way he possibly could have obtained the 98% brandishing figure from his 1997 survey.

This is a long post, so I’ll start with two summaries. One sentence summary: It looks as if Lott might have been caught cooking his “more guns, less crime” data.

One paragraph summary: Ian Ayres and John Donohue wrote a paper that found that, if anything, concealed carry laws lead to more crime. Lott, (along with Florenz Plassmann and John Whitley) wrote a reply where they argued that using data up to 2000 confirmed the “more guns, less crime” hypothesis. In Ayres and Donohue’s response to that paper, they found that Lott’s data contained numerous coding errors that, when corrected, reversed the results. Furthermore, this was the second time these sorts of errors had been found in Lott’s data. Lott had presented to the NAS panel figures showing sharp declines in crime following carry laws. Declines which disappeared when the coding errors were corrected. Finally, when Lott saw Ayres and Donohue’s response he had his name removed from the final paper.

The gory details: In a critique of econometric studies such as Lott’s Ted Goertzel makes an important point—for any study that involves a multiple regression that finds a significant association, it seems that there is another study that applies a different model to the same data and gets a different answer. Goertzel argues that

When presented with an econometric model, consumers should insist on evidence that it can predict trends in data other than the data used to create it. Models that fail this test are junk science, no matter how complex the analysis.

In the case of Lott’s model we are in the fortunate position of being able to test its predictive power. Lott’s original data set ended in 1992. Between 1992 and 1996, 14 more jurisdictions (13 states and Philadelphia) adopted carry laws. We can test the predictive power of Lott’s model by seeing if it finds less crime in those jurisdictions. In their just-published paper Ayres and Donahue have done this test. They found that, using Lott’s model, in those later-adopting jurisdictions, carry laws were associated with more crime in all crime categories . Lott’s model fails the predictive test.

Ayres and Donahue go on to examine all the states adopting carry laws using data up to 1999 and find that:

The best, albeit admittedly imperfect, evidence suggests that, for the majority of states, shall-issue laws are associated with higher levels of crime.

Lott’s reply offered two main arguments. First:

Ayres and Donohue have simply misread their own results. Their own most generalized specification that breaks down the impact of the law on a year-by-year basis shows large crime reducing benefits. Virtually none of their claims that their county level hybrid model implies initial significant increases in crime are correct. Overall, the vast majority of their estimates based on data up to 1997 actually demonstrate that right-to-carry laws produce substantial crime reducing benefits.

And second:

Analyzing county level data for the entire United States from 1977 to 2000, we find annual reductions in murder rates between 1.5 and 2.3 percent for each additional year that a right-to-carry law is in effect.

Ayres and Donohue’s response to this is devastating. This is from their introduction:

In our initial article Shooting Down the More Guns, Less Crime Hypothesis—we reached two main conclusions: First, that there was no credible statistical evidence that the adoption of concealed-carry (or “shall-issue”) laws reduced crime; and second, that the best, although admittedly quite imperfect, data suggested that the laws increased the costs of crime to the tune of $1 billion per year (which is a relatively small number given the total cost of FBI index crimes of roughly $114 billion per year). In their response to our article, Florenz Plassmann and John Whitley (PW) offer two sets of evidence in support of their view that that concealed-carry laws are beneficial: First, they argue that some of our regression specifications really buttress their position; and second, they analyze some new county data for the period 1977-2000.

Their first method of proof fails because it simply overlooks—without even a single word of commentary!—the entire thrust of our paper: that aggregated specifications of the effects of these laws are badly marred by “jurisdiction selection” effects. We did not misread these aggregated estimates, as PW suggest; we simply showed that the PW claims based on these aggregated estimates are inaccurate and misleading. The data at every turn reject the idea that concealed-carry laws passed in different jurisdictions have a uniform impact on crime. Therefore, the results of disaggregated regressions must, counter to PW s claim, be taken as a more authoritative assessment of the overall impact of concealed-carry laws.

Their second method of proof fails because PW seriously miscoded their new county dataset in ways that irretrievably undermine every original regression result that they present in their response. As a result, the new PW regressions must simply be disregarded. Correcting PW s empirical mistakes once again shows that the “more guns, less crime” hypothesis is without credible statistical support.

And this:

It is important to note that what we now refer to as the PW response has already been widely circulated as a draft, whose first author is John Lott. Moreover, Lott has repeatedly told the press and/or published to the Internet that Ayres and Donohue have simply misread their own results. But after seeing this Reply to the original Lott, Plassmann, and Whitley paper, Lott asked the Stanford Law Review to take his name off the work. We hope that this indicates that the arguments in our Reply have caused the primary proponent of the more guns, less crime hypothesis to at least partially amend his views. We note that to this day, legislators are still voting for the adoption of concealed-carry laws while citing Lott’s work.

And this is from their conclusion:

While we emphasized the severe selection-effect problem of estimating the effects of concealed handguns by aggregating across all the adopting jurisdictions, PW simply ignore this concern. When they contend that we have misread our own results, it is because they cite the jurisdictionally aggregated regression estimates that we showed were flawed and continue to pin their more guns, less crime hypothesis on this flawed estimation approach. Indeed, if one accepts our view on this point, one has to jettison virtually their entire paper, which probably explains why they did not respond to the issue. PW present twelve figures and thirteen tables in their paper that offer estimates of the effect of concealed-carry laws on crime, and of these, every one but PW table 6 and appendix table 4 is unreliable because they rely on the discredited jurisdictional-aggregation assumption. Moreover, every new regression on PW s extended county dataset is fatally fla