Deltoid

[Originally posted to firearmreg on Aug 15 1996]

Daniel Polsby writes:

Lott’s results are highly plausible and internally consistent.

Highly plausible? Lets look at Dade county:

Lott reckons that the carry law caused a reduction of 8% in murders, 5% in rapes, 7% in aggravated assaults and 2% in robberies. For Dade county that translates to 1,500 fewer aggravated assaults, 450 fewer robberies, 65 fewer rapes and 30 fewer murders each year. From Cramer and Kopel’s paper on CCW (TN Law Review v 62p733) one learns that “the police kept track of every known incident involving [Dade] county’s more than 21,000 handgun permitees over a six-year period.” and there were 12 defensive gun uses by CCW holders against persons known to the police over the six year period. That’s two per year.

It does not seem highly plausible that those two uses prevented 2,000 crimes.

[Originally posted to firearmsreg Aug 16 1996]

Daniel Polsby writes:

Mr. Lambert, and for that matter most others on this list, assume that firearms are used defensively when they are brandished. All of the endless back and forth about survey research techniques of establishing how often this sort of thing happens has embedded this assumption.

No, my comments were directed specifically at the deterrence theory. IF concealed weapon permit holders used their weapons frequently (say two or three times a week) and IF these incidents were given wide publicity (so that criminals were made aware of the risk they faced) THEN you would have a plausible mechanism by which a concealed wepons law could cause a large decrease in crime.

Economists’ model of deterrence treats the prospect of meeting armed resistence as affecting, ex ante any misbehavior, the apparent present value of a given crime. The prospect of meeting an armed defender or an armed good Samaritan should be expected to do two things: (1) raise the absolute expected cost of confrontational crimes and thus (2) increase the relative value of other potential uses of ones time, including non-confrontational crimes.

Compare: 2 encounters with armed CCW holders per year in Dade county with 10,000 arrests by armed police for violent crime per year in Dade county. The chance of meeting an armed CCW holder is negligble. In an economic model of deterrence it will be swamped by the 5,000 times greater chance of being arrested.

Incidentally, in case the point has been lost, I have never questioned the integrity or seriousness of McDowall and his collaborators.

Pardon? You accused McDowall et al “of taking the extra steps necessary to locate a subset of the data that would yield a particular result.” It certainly sounds to me that you are accusing them of dishonest data selection.

Suffice to say that no conversation about causation and correlation can be had without a reasonable grasp of David Hume’s argument on this subject, which was, in brief, that “causation” as such can never be demonstrated. How do you know, if you drop a marble, that your dropping of it has “caused” it to go onto the floor? How do you know that this result is not, simply, highly correlated with dropping?

Absolute certainty is of course impossible. Scientists generally accept a replicable controlled experiment as good enough for showing cause and effect for practical purposes. A non-experimental study like Lott’s is merely suggestive of causation. When you have a pile of them and well-understood and verified mechanisms (like for smoking-cancer) then you can maybe make some causal claims. Lott says: “If those states which did not have right-to-carry concealed gun provisions had adopted them in 1992, approximately 1,570 murders; 4,177 rapes; and over 60,000 aggravate assaults would have been avoided yearly.” This statement needs at least two caveats that Lott does not give us.

jmaraldo writes:

Why assume that only the defensive use deters, that the knowledge the intended victim and civilians in the vicinity may be armed does not also deter? It seems to me that a shall-issue CCW system is like a sign which says “Premises Protected By ** Alarm” in that the many more burglars are deterred than actually trigger the alarm and the sign often works even though there is in fact no alarm installed.

But CCW holders do not carry signs saying “Person Protected By Smith and Wesson”. What is it that is telling criminals that they are facing a risk from CCW holders? It can’t be from personal experience in encountering one, or media reports of CCW holders thwarting crime, since these incidents are almost nonexistent.

Tom Tancredo (really Dave Kopel?) writes:

Tim Lambert writes:

It does not seem highly plausible that those two uses prevented 2,000 crimes. “

Sure it’s plausible. First of all, there is some unknown number of DGUs by permitees that were never reported to authorities. It’s possible that 90% may be unreported, since there’s no real need to tell the police that you drew a gun, and the criminal ran away.

If Kleck’s DGU survey is to be believed 64% of DGUs are known to the police. In any case, even if there are another 20 unreported ones, these aren’t going to have much of a deterrent effect since only one or two criminals will be aware of them.

More important, as Kleck has shown in various cases, wide publicity about the prevalence of armed citizens can sometimes have a huge deterrent effect, all by itself.

Firstly, I do not believe that Kleck has shown this at all. Publicity in Orlando and Kansas City was associated with a temporary but not statistically significant reduction in crime. His arguments about Kennesaw are not even internally consistent.

Secondly, if the mechanism to be invoked is that publicity about the introduction of CCW laws reminded criminals of the risks that they faced from armed victims, then Lott has used the wrong model. Such an effect would be as temporary as the publicity, so his model should be for a temporary effect, not a permanent one.

[Originally posted to firearmsreg Aug 19 1996]

Daniel Polsby writes:

McDowall very freely interprets his five-county study as suggestive of causation. I tend to share the view that one should be slow to change public policy on the basis of a single study, though one might say of Lott-Mustard that it amounts to at least 610 McDowall-Loftin-Wiersema studies, as it covers that many times more counties, to say nothing of controlling for a lot more variables. I should say, however, that ordinarily the heavy lifting of causation involves the existence of a theory which models how the world works. The Lott-Mustard results are generally consistent with such a theory (the price theory model).

Qualitatively consistent in that an increase in the price of crime was associated with a decrease in the volume of crime, yes. Quantitatively consistent, no. Arrest is 5,000 times more likely than having a CCW holder pull a gun on you. If the criminal considers these to be equally bad outcomes, then the CCW law increases the cost by no more than 0.02%. It is absurd to expect a 7% decrease in crime from such an insignificant change in the cost.

Even if (contrary to what criminals said in the Wright-Rossi study) criminals are not afraid of police guns but are afraid of victims guns. The change in the cost is insignificant. If you believe Kleck there are about 20,000 DGUs in Dade county each year. If you believe the NCVS it’s more like 500. Either way, the two by CCW holders is insignificant.

A general point about deterrence theory. The theory doesn’t tell us when a person who is supposed to be deterred comes to believe that a threat is credible. It doesn’t assume, in other words, that there will be a certain number of dead criminals before people start believing that if they commit a crime they may be shot. Nuclear deterrence works in exactly the same way; nobody wants to find out how credible that sort of threat actually is.

Ahem. Since no nation has actually attacked NATO it is reasonable to suppose that if one did attack NATO there is a good chance of starting a nuclear war. On the hand with 50,000 or so violent crimes in Dade county each year and only 2 DGUs by CCW holders it is not reasonable for a criminal pulling a violent crime to expect to be thwarted by a CCW holder.

This post contains extracts from Chapter 23 Private Defence of Textbook of Criminal Law by Glanville L Williams (2nd Ed 1983). (more…)

Otis Dudley Duncan, University of California, Santa Barbara

(from The Criminologist Vol 25, No 1 Jan/Feb 2000 pp 1-7)

We who work hard to produce statistics for public consumption would do well to acquire a little historical perspective. Theodore Porter’s wide-ranging Trust in Numbers: The Pursuit of Objectivity in Science and Public Life (1995) takes note of 19th-century developments illustrating the “creative power of statistics…. Every category has the potential to become a new thing.” Crime did not originate in that century, but “it may be doubted whether there were crime rates” before then (p. 37). To the examples given by Porter we might add the relatively new “thing” in the domain of what used to be called “moral statistics”—the production of counts and rates of DGU (defensive gun uses, or users, according to context). Porter concludes that democratic politics gives rise to distrust of personal judgments and statistical methods are brought into play to nurture confidence where personal knowledge is an insufficient basis for political decisions. A somewhat paradoxical outcome is that “There is a strong incentive to prefer precise and standardizable measures to highly accurate ones” (p. 29). But it is arguable that the study of DGU is not yet at a stage where such a choice is necessary, or even possible. I shall point out some problems of communication that stand in the way of achieving either desidratum soon.

(more…)

(from The Criminologist Vol 25, No 5 Sep/Oct 2000 pp 1,6)

In a recent issue of the Criminologist, Otis Duncan raises concerns about my writings. He discusses a wide range of issues from the estimated number of defensive gun uses and the rate at which defensive uses result in the gun being fired to even the significance of why the NRA doesn’t cite certain aspects of my research. Let me go through the different points raised by Duncan: (more…)

by Steven D. Levitt, Professor of Economics, University of Chicago

[Editor’s note: A version of this piece was published in the Chicago Sun-Times on July 28, 2001 under the title “Pools more dangerous than guns.” ]

What’s more dangerous: a swimming pool or a gun? When it comes to children, there is no comparison: a swimming pool is 100 times more deadly. (more…)

[On Sep 14 2002 I posted this to firearmsregprof. I also emailed it to John Lott. ]

Way back in 1993 in talk.politics.guns, C. D. Tavares wrote:

The answer is that the gun never needs to be fired in 98% of the instances of a successful self-defense with a gun. The criminals just leave abruptly, instead.”

When I queried him about this, he quickly corrected his error:

Kleck says in the magazine “Social Problems” (2/88):

“there were about 8,700-16,600 non-fatal, legally permissible woundings of criminals by gun armed civilians” annually, and “the rest of the one million estimated defensive gun uses, over 98% involved neither killings nor woundings but rather warning shots fired or guns pointed or referred to.”

So I certainly screwed up by not including the warning shots fired, plus the times the victim simply missed. From Kleck’s latest survey as reported here, the warning shots and misses are very significant (~14%).

Now, Lott has repeatedly made the same claim: “98% of the time when people use a gun defensively, merely brandishing the weapon is sufficient to stop an attack.”

A Google search for “98% lott brandishing gun” finds about 120 different sites where his claim is repeated (and only one of those is critical). Lott himself has made the claim on at least 25 different occasions. Otis Dudley Duncan has compiled a list of these that I extracted the examples below from.

Lott’s claims about the source of this statistic have changed over time:

“Polls of American citizens undertaken by organizations like the Los Angeles Times and Gallup showing that Americans defend themselves with guns between 764,000 and 3.6 million times each year, with the vast majority of cases simply involving people brandishing a gun to prevent attack.[1]”
John Lott “Does Allowing Law-Abiding Citizens to Carry Concealed Handguns Save Lives?” Valparaiso University Law Review, 31(2): 355-63, Spring, 1997.
[1]. Gary Kleck & Marc Gertz, Armed Resistance to Crime: The Prevalence and Nature of Self-Defense with a Gun, 86 J. CRIM. L. & CRIMINOLOGY 150, 153, 180, 180-82 (1995).

“Polls by the Los Angeles Times, Gallup and Peter Hart Associates show that there are at least 760,000, and possibly as many as 3.6 million, defensive uses of guns per year. In 98 percent of the cases, such polls show, people simply brandish the weapon to stop an attack.”
John Lott “Gun-Lock Proposal Bound to Misfire” Chicago Tribune, August 6, 1998.

“If national surveys are correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack.”
John Lott “More Guns, Less Crime” (University of Chicago Press, 1998), p. 3. [Date of publication, May, 1998.]

“People used guns defensively to stop violent crimes over 2 million times in 1997. 98 percent of the time, when people use guns defensively, simply brandishing a gun is sufficient to cause a criminal to break off an attack. In less than 2 percent of the time is the gun fired. About three-quarters of those are warning shots.”
Oral statement, paraphrased, TV show, Hardball, CNBC, August 18, 1999

“Guns clearly deter criminals, with Americans using guns defensively over 2 million times each year — five times more frequently than the 430,000 times guns were used to commit crimes in 1997, according to research by Florida State University criminologist Gary Kleck. Kleck’s study of defensive gun uses found that ninety-eight percent of the time simply brandishing the weapon is sufficient to stop an attack.”
John Lott “Gun Locks: Bound to Misfire” online publication of the Independence Institute, Feb. 9, 2000.

“If a national survey that I conducted is correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack.”
More Guns, Less Crime, second edition (University of Chicago Press, 2000), p. 3. [Publication date, May, 2000]

“As I told Duncan last year in a telephone conversation, I had no idea why the estimated 2.5 million defensive gun uses was attributed to me. The 2.5 million estimate obviously comes from Kleck. The about 2 million reference is the average of the 15 national surveys and is very similar to my own estimate of a little over 2 million defensive uses. The survey that I oversaw interviewed 2,424 people from across the United States. It was done in large part to see for myself whether the estimates put together by other researchers (such as Gary Kleck) were accurate. The estimates that I obtained implied about 2.1 million defensive gun uses, a number somewhat lower than Kleck’s. However I also found a significantly higher percentage of them (98 percent) involved simply brandishing a gun. My survey was conducted over 3 months during 1997. I had planned on including a discussion of it in my book, but did not do so because an unfortunate computer crash lost my hard disk right before the final draft of the book had to be turned in.”
John R. Lott, Jr.’s Reply to Otis Duncan’s recent article in The Criminologist, The Criminologist, vol. 25, no. 5, September/October 2000, page 6. http://www.cse.unsw.edu.au/~lambert/guns/lottduncan.html

“In fact, in 98% of the cases, simply brandishing a gun is sufficient to stop a crime. Research at Florida State University and at the University of Chicago indicates that only one out of 1,000 defensive gun uses results in the attacker’s death.”
LA Times Friday, March 30, 2001 Others Fear Being Placed at the Mercy of Criminals by John Lott Jr.

The reason why Lott’s story has changed is that neither the Kleck survey nor any of the polls that he cited found that only 2% of the time the weapon was fired. Kleck says, quite clearly, “24% of the incidents involved the defender firing their gun”. The other polls Lott mentioned give even higher numbers.

Lott is well aware of this, since he changed the wording in the second edition of “More Guns, Less Crime”. However, instead of changing it so that his readers were informed of what national surveys actually found, he claimed that it came from his own survey.

Further, a 2.1 million estimate implies that about 1% of his respondents reported a personal DGU in the past year. With a sample size of 2,424, that’s about 25 DGUs. 2% of 25 is 0.5. Clearly it is not possible for half a person to report firing their weapon. OK, so maybe Lott used a design similar to Kleck’s where the question asked about household use over a five year span. If he got a similar number of positive responses to Kleck, then he would have got 100 DGU reports and is is conceivable that only two reported shooting. However, he also claims that 3/4 were warning shots—3/4 of 2 is 1.5. Again, it is not possible for half a person to report a non-warning shot.

Otis Dudley Duncan has tried to find out more details about this survey. I quote from an email from him:

I queried him [Lott] by mail and e mail (I sent the mail certified but he claims not to have received the e mail, which was identical) and got nothing more from him besides what is in the passage in his “Reply” except for the detail that the interviewing was done by students using their own computers. They fed their results into his computer, and it was this one that experienced the crash. He assures me that others lost data in that crash too. Strangely enough, in the phone conversation of May 1999, presumably long after the crash, he said nothing about it. And he has left the timing rather ambiguous. If he did a survey in early 1997 but too late to mention in his Valparaiso Law Review article, he would still have got the 98% figure by the time his book went to press. Indeed, he says that he had planned to include a discussion of the survey in the book, but the crash occurred just before his deadline for turning in the book. (This could refer to the 2000 edition, but that would raise other problems.) So, it is not at all clear why he couldn’t have said the same thing on p. 3 in 1998 as he did in 2000. The timing strongly suggests that the figure of 98% came first, the story about the survey and its loss in the computer crash came afterward. A second letter to Lott this summer went unanswered. In it I had asked for specific evidence such as a copy of his questionnaire, a copy of the computer printout from which he derived his 98% etc., names of some of the students, etc. He has not answered that letter. I too would like to have an explanation from Lott. I considered the possibility that he indeed thought of doing some kind of survey, and maybe went so far as to have some students do trial interviews, but gave it up for whatever reason. And possibly something of that material went into the computer that subsequently crashed.

No one knows but Lott.

A plausible explanation of what has happened is that Lott made a similar error to CDT, but rather than correct it like CDT did, Lott chose to make up a story about a survey.

I appeal to the readers of this list—if you can think of a better explanation or if you have some knowledge of this mysterious Lott survey, please let me know.

“The reluctance of gun-control advocates to release their data is quite widespread. In May 1997 I tried to obtain data from the Police Foundation about a study that they had recently released by Philip Cook and Jens Ludwig, but after many telephone calls I was told by Earl Hamilton on May 27. “Well, lots of other researchers like Arthur Kellermann do not release their data.” I responded by saying that was true, but it was not something that other researchers approved of, nor did it give people much confidence in his results” John Lott, page 291 of “More Guns, Less Crime”

[On Sep 15 2002 I posted this to firearmsregprof.]

In response to my appeal for any information about this survey Lott claims to have carried out in 1997 I received this email from Geoffrey Huck. Unfortunately, as you can see, he was not able to provide any support for the existence of the survey, so I repeat my appeal: it seems increasingly likely that the survey is fictional—I seek evidence to the contrary.

From: Geoffrey Huck

I was John Lott’s editor at the University of Chicago Press for his book, More Guns, Less Crime, published originally in 1998 and then in a second edition in 2000. John has asked me to confirm for the record an incident that occurred in the summer of 1997, just as he was preparing the final version of the manuscript for submission to the Press. At that time, John reported to me that a bookshelf had fallen on his computer, seriously damaging his hard disk containing not only all his files and data for More Guns, but also work on some other projects as well. I recall that much of what was on the disk was lost and could not be recovered. We did have hard copy of most (but I think not all) of the book manuscript, however, and were able to proceed with that.

[On Sep 17 2002 I posted this to firearmsregprof.]

I’ve had replies to my queries about Lott’s survey from David Mustard and Gary Kleck.

Mustard said that he was not involved with the survey. All he was able to say was what he had been told by Lott: that Lott had conducted the survey in 1997 and lost the results in a computer crash.

Kleck felt that Lott’s survey should be ignored unless and until Lott publishes the results and methodology. This would be reasonable except for the fact of the 25+ occasions that Lott has made the “98% merely brandishing” claim.

Nothing from Lott.

[On Sep 18 2002 I posted this to firearmsregprof and emailed it to Lott.]

Some more information about DGU surveying from a Kleck email:

“We got 4,977 completions over 3 months, using an average of about 10 callers per day (and I believe all of them were FSU students)”

Also, look at the acknowledgments of Kleck and Gertz’s paper:

“The authors wish to thank David Bordua, Gary Mauser, Seymour Sudman, and James Wright for their help in designing the survey instrument. The authors also wish to thank the highly skilled staff responsible for the interviewing: Michael Trapp (Supervisor), David Antonacci, James Belcher, Robert Bunting, Melissa Cross, Sandy Hawker, Dana R. Jones, Harvey Langford, Jr., Susannah R. Maher, Nia Mastin-Walker, Brian Murray, Miranda Ross, Dale Sellers, Esty Zervigon, and for sampling work, Sandy Grguric.”

If there had been a huge fire or something and all of the computer and paper records of Kleck’s survey had been destroyed, we would still have twenty witnesses who could confirm that the survey had been conducted.

Lott says his survey interviewed 2,424 people, over three months, so he would have needed 5 callers operating a day. That would have required a pool of at least eight interviewers, plus a supervisor. I looked through Lott’s vita and I couldn’t see any paper where it looked like he had conducted a phone survey, so he would have had to consult with an expert on the design of phone surveys. And….

Put yourself in Lott’s shoes. It’s the beginning of 1997. You’re busy writing your book and talking to reporters and whatnot about your 1996 paper. You decide to do a survey in order “to see for myself whether the estimates put together by other researchers (such as Gary Kleck) were accurate”. Who are you going to call to discuss the survey design?

Yeah, that’s right. Gary Kleck.

But Lott didn’t. We know this because both Duncan and I have asked Kleck about it.

[I have sent a copy of this message to Lott, just as I have the others on this topic.]

Still nothing from Lott.

[On Sep 20 2002 I posted this to firearmsregprof and emailed it to Lott.]

James Lindgren writes:

After my post to this list saying that "a big national study doesn’t just disappear without a trace" because a computer crashes, John Lott called me and told me a long story about how the study was done (which I don’t choose to share just yet, if ever; Lott can speak for himself on his methods, if he wishes).  He didn’t ask me to do anything about it, and I wasn’t planning on posting anytime soon, but given the recent posts I thought I would.

Whether the study should be given credence is a different question from whether it was ever done.    The latter question is the important one for me. 

While it will probably be possible to get a lot of circumstantial evidence about Lott’s data losses on various projects in 1997 (I got an email from David Mustard to this effect and will check with Bill Landes myself), the crucial question on which direct evidence is needed is whether study was done.  Lott reports that he used UC undergrad volunteers to do the calling, using a sample drawn from a CD ROM with phone numbers.

Volunteers? Based on Kleck’s surveys you would need 300 person days of calling. I can imagine a student doing a day or three as a volunteer, but more than that? How could you get enough volunteers?

When his computer crashed, he decided not to publish the study and says that he made only passing reference to it in his book.

In the 1st edition of MGLC Lott writes: “If national surveys are correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack.” He doesn’t mention his survey anywhere in the 1st edition. In the 2nd edition of MGLC (published in 2000) Lott changed it to: “If a national survey that I conducted is correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack.” As far as I know, this is his first published mention of the survey.

This would not be a big deal, except that Lott has made his 98% claim on literally dozens of occasions. His claim promotes the “magic talisman” theory of gun self-defence — suggesting that all you need to do is wave a gun around a little. This theory is dangerous — it could potentionally lead to people being injured or killed.

Lott does not remember the names of the undergrads.  I offered to contact the UC alumni office to see if I could email the undergrads who were juniors and seniors in the 1996-97 academic year to locate the callers.  I intend to do just that in the next 3-4 weeks, if UC will go along.

Thank you so much for your assistance.

However, I find it disturbing that after discussing with you the means by which he could demonstrate that this survey was actually conducted, the best Lott could come up with was to attempt to contact students whose names he cannot remember. As has been discussed here before, there should be plenty more evidence — phone bills, survey questions, people he discussed the survey with whose names he recalls and so on.

[On Sep 25 2002 I posted this to firearmsregprof. I also emailed it to John Lott. ]

I have replies to queries about Lott’s survey from Tom Smith and Al Alschuler. Neither had heard anything about it. Other than Kleck, Tom Smith would be the logical person to discuss the survey design with—Lott acknowledges his assistance on other survey-related questions in MGLC. He’d also be a good source to obtain a CD of phone numbers from, but apparently Lott obtained this elsewhere.

Alschuler is in the U of Chicago Law School and wrote a reply to the paper where Lott first made the brandishing claim. He told me that the paper was due shortly after the conference it was presented to. That conference was held in mid November 1996. Consequently, this:

“Polls of American citizens undertaken by organizations like the Los Angeles Times and Gallup showing that Americans defend themselves with guns between 764,000 and 3.6 million times each year, with the vast majority of cases simply involving people brandishing a gun to prevent attack.[1]”
John Lott “Does Allowing Law-Abiding Citizens to Carry Concealed Handguns Save Lives?” Valparaiso University Law Review, 31(2): 355-63, Spring, 1997.

was written before Lott even started doing his survey. So how did he know that the “vast majority of cases” involved brandishing before he did this survey of his?

Now, in the quote above, he doesn’t give the exact number, whereas in later version like this one:

“98% of the time when people use guns defensively simply brandishing a gun [is] sufficient to cause a criminal break off an attack. In less than 2% of the time is the gun fired. Most of those are warning shots. Less then one half of one percent of the time is the gun fired at the attacker. And only one out of every thousand times that people use guns defensively does it result in the death of the attacker. In the vast majority of these times simply brandishing a gun is sufficient to make a criminal break off an attack and run away.”
John Lott on NPR’s “Justice Talking” June 28, 1999 Audio available from http://www.justicetalking.org/getshow.asp?showid=97 Quote is at time 55:25 in the audio

he does, so it is possible that he only knew the number approximately in 1996 and his survey gave him a more precise estimate, but the question remains: “Where did the 1996 “vast majority” estimate come from?”

I would be most interested in any suggestions that readers have.

[On Sep 27 2002 I posted this to firearmsregprof and emailed it to Lott.]

Peter Boucher, replying to this post, writes:

I don’t have a copy of Point Blank handy, but I seem to recall the 98% figure either explicitly in the text of that book, or directly derivable from the figures in the book.

Yes, as I noted at the beginning of the discussion, the 98% figure could have come from a misreading of “Point Blank”. However, the problem with this theory is that Lott has specifically denied it. He says that the figure comes from his survey. It also seems far too much of a coincidence for him to believe (mistakenly) in 1996 that the figure was 98% and then conduct a survey in 1997 that comes up with exactly the same number.

I found a few example of others who had reached the same conclusion by doing a google:

http://www.gunowners.org/fs9504.htm “Of the 2.4 million times citizens use their guns to defend themselves every year, 92% merely brandish their gun or fire a warning shot to scare off their attackers. Less than 8% of the time, a citizen will kill or wound his/her attacker. (5) … (5) Kleck, interview with Schulman, op. cit.”

Arrgh! another misreading of Kleck. The 92% includes shots fired at the attacker. Also this assumes that defensive gun use is 100% effective — just because the defender brandishes a gun, it doesn’t mean that the attacker automatically flees.

Anyway, the “vast majority” claim seems to be supported by Kleck-Gertz from 1995.

OK, maybe Lott felt that the 76% of DG users in Kleck-Gertz that said they didn’t fire their weapon was big enough that he considered it a “vast majority”. Now here’s the problem:

In 1996, based on Kleck’s survey, Lott believes the figure is 76%. In 1997 he conducts his own, smaller, survey which comes up with 98%. Quite a remarkable difference. Later in 1997 he summarizes the situation in MGLC. I can think of several ways you might describe this: “Surveys give widely varying estimates for the percentage of defenders that fire their weapons, ranging from 2% to 24%.” Or, maybe if there is some reason, so far not revealed by Lott, that he thinks his survey supercedes Kleck’s he could could write this: “A survey conducted by Gary Kleck found that 24% of defenders shoot, but I survey I conducted shows that the figure is 2% and I believe that figure is more accurate because <insert super-secret reason>”.

What Lott in fact wrote was “If national surveys are correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack.” (MGLC p3) and “Kleck’s study of defensive gun uses found that ninety-eight percent of the time simply brandishing the weapon is sufficient to stop an attack.” (Gun Locks: Bound to Misfire). And on at least three dozen other occasions he presented the 98% figure without ever mentioning that there might be any doubt over its accuracy.

Nowhere did he even mention the existence of this survey of his until 2000 in the 2nd edition of MGLC. And even then he wrote “If a national survey that I conducted is correct, 98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack.” Still no mention of the wildly divergent Kleck estimate.

[On Sep 27 2002 I posted this to firearmsregprof.]

Norman Heath writes:

Just a suggestion, but perhaps Lott simply made a rough comparison between the number of claimed DGUs and the total number of shootings. I.e. if total shootings is (making this up) 120,000 and we subtract 35,000 suicides, 5000 police shootings, 200 hunting accidents, 15,000 gun murders, then even without accounting for non-fatal criminal shootings the highest possible number of live-fire DGUs would be about 64,800 (again, using made-up numbers). If at least 764,00 people claimed DGUs, then the number of those who actually shot somebody in self defense would necessarily be far under 10% of total claimed DGUs.

This is basically how Kleck came by his original estimate that 98% of DGU did not involve anyone getting shot. To get the numbers he would need for this calculation he would have to go to Kleck’s work and would have come up with the same 98% figure for DGUs where no-one was wounded. However, it just seems too much of a coincidence that his own survey would just happen to produce exactly the same percentage for a different number (percentage who don’t fire).

It is not my mission in life to stick up for John Lott, about whom I know next to nothing. But this hopeful little fishing expedition trying to catch J.L. in a lie has a couple of problems, not the least of which is that the inquisitors are trying to prove a negative, the non-existence of a survey. This is very unlike checking an author’s claim against a cited document which does exist and can be examined. Even if no phone bills from the survey can be found, no undergrad students recall participating, etc., the case against the existence of a ‘98 survey can probably never be considered proven. The inquiry has little prospect of ever being conclusive, thus it appears to be more an exercise in insinuation than anything else.

On the other hand, given the amount of evidence that a national survey of the scale that Lott claims he conducted would leave, it should be trivially easy for him to prove that the survey was conducted. But he hasn’t done so.

Of course, anybody is free to try to dig up some dirt against an academic who represents the “other side.” But a little discretion is appropriate, in order that the inquiry itself does not become campaign of slander. It is one thing to use a listserve to ask casually where one might obtain Prof. X’s dissertation. It is another matter entirely to suggest on a listserve that Prof. X committed plagiarism in his dissertation and then ask where one might obtain a copy of the dissertation in order to find the plagiarism.

Unfortunately that approach does not work for the questions that we would like answered. For example: “What, specifically, were the questions in Lott’s survey?” If I asked the list that question, I think the reasonable answer would be that I should ask Lott. Duncan has asked Lott that question and Lott has not answered it. I could not see any way of making the enquiries that I felt were needed without there being some sort of implication that something untoward was going on, so I felt that I should explain my suspicions to the list.

And, raising the matter in a somewhat public forum has prompted Lott to respond to some extent, though I don’t understand why he would not tell the list the story of his survey.

[On Oct 03 2002 I posted this to firearmsregprof.]

Norman Heath writes:

Tim Lambert asked for suggestions as to how John Lott might have formed a belief about the proportion of DGUs that involve gunfire, prior to having conducted a survery. I took this to mean that Lambert was open to those suggestions which involve innocent explanations. But when I offer an innocent explanation, Tim responds by pointing out that a sinister one also exists. I think Lott is owed the courtesy of assuming his innocence, and it still seems to me that there are justifications that Lott might have had, or thought he had, for saying that a “vast majority” of DGUs do not result in gun discharges, prior to his having conducted a survey on the subject.

Well, there certainly are, and I listed them. The most straightforward one is that by “vast majority” he was referring to the 76% that did not fire in Kleck’s survey. Duncan has suggested another possibility — that Lott misread Kleck’s table 1. This is the table that contains the numbers that Kleck derives from several surveys that Lott cites. In that table, directly above the DGU numbers that Lott cites, is a row entitled “% who fired gun”. The numbers in that row range from 2.6% to 16%. It is possible that Lott mistakenly took these as percentages of gun users rather than the percentage of respondents which they are.

One of these is the fact that the number of claimed DGUs looks to me to be much, much higher then the number of shootings that could conceivably be attributed to DGUs. The lowest figure for DGUs in the given poll was 746,000. What is the highest possible number of shootings that could conceivably be attributable to those DGUs, after we subtract known suicides, accidents, etc, from the total of reported shootings? Say the highest conceivable number of shootings attributable to DGUs appears to be 5000 (a wild guess). That would be less than 0.7% of the lowest poll figure for DGUs, and something over 0.1% of the highest. Even if ten or twenty warning shots or misses occurred for every DGU that resulted in a shooting, it would still indicate that a “vast majority” of claimed DGUs do not result in gunfire.

But your calculation is, by your own admission, based on a wild guess. The normal standards of scholarship would require you to tell your readers that and not present it to them as a fact.

An author’s having confidently expressed a belief on a matter prior to confirmation is NOT evidence that he has fabricated anything.

I have explained several times now why I think the matter is significant. You keep ignoring my explanation.

I look out my window and tell everybody in earshot that the vast majority of trees on my property are oaks. Now I will count them … ninety-eight percent of my trees are oaks.

Damn, I lost the paper I wrote the tree-count on. The fact that I expressed a belief while looking out the window at my trees will now be cited as proof that I did not later count my trees.

Moreover, my political enemies will also suggest that looking out my window and claiming that the vast majority of my trees are oaks was a “fabrication” on my part, since I had not yet counted them. That is the reasoning Tim Lambert uses in his last post.

That is not even remotely close to my reasoning. I see no point in repeating my reasoning. I refer you to my previous messages.

I knew that somebody would bring up Bellesiles sooner or later. Imagine the political utility of tainting John Lott just as the Bellesiles scandal comes to a head! Reason enough to pursue the matter on pure speculation!

You object to legitimate questions about Lott’s survey because they carry implications that might reflect on his character. And then you turn around and cast slurs on my character.

For the record: Your allegations as to my motivations are false.

I point out to Mr. Lambert that there is a mountain of evidence against Michael Bellesiles that Bellesiles himself produced. Bellesiles misrepresented the contents of innumerable documents which he then cited, and many of which were available for examination in libraries.

So is this like claiming that Kleck found that 98% of DGUs involved brandishing when, in fact, he made no such finding? Or claiming that “national surveys” gave the 98% figure when, in fact, they didn’t?

The complaint against John Lott is that he has no evidence. Can Tim Lambert see the difference between this and the case above? No evidence means NO EVIDENCE. We cannot contrast Lott’s claim with his evidence, as was done with Bellesiles, because the evidence is not available.

I did not say the cases were the same. The specific parallel I drew was with the probate data:

Bellesiles claims to have collected probate records that gave far lower gun ownership rates than those obtained by anyone else. Some of his results appear to be mathematically impossible. He claims his records were destroyed in a flood.

Lott claims to have conducted survey that gave far lower DGU shooting rates than those obtained by anyone other survey. Some of his results appear to be mathematically impossible. He claims his records were destroyed in a computer crash.

Presumably it was lost. Perhaps Lott diabolically destroyed the data, or perhaps he never had it. I do not know. But I do know that John Lott is entitled to the presumption that the data were lost in a computer crash as he claims.

That’s fine, as long as you give Bellesiles the presumption that his probate data was destroyed in a flood.

Lott’s opponents may dismiss the 98% figure as unverifiable, but they cannot properly accuse him of fabricating it until they have some real evidence. The fact that Duncan does not know what Lott said in legislative testimony several years ago, and does not know how to find out, is not “evidence.” Nor is the fact that Lott made a vague claim prior to nailing down a specific percentage for that claim.

The evidence, yet again, is that all seven other surveys gave a radically different figure. That it is mathematically impossible for a survey of the size of Lott’s to give a number that Lott says that he obtained from his survey. That Lott first claimed (falsely) that the 98% figure came from Kleck, and the first printed mention of this 1997 survey was in 2000 despite Lott citing the 98% figure dozens of times. That no-one, not colleagues, not collaborators, not experts on surveys that he consulted at the time on other matters, that not a single person that Lott can name was aware of a survey that would have required at least 300 person-days of work. That Lott has not provided any of evidence that the survey was carried out, despite having had ample time to do so.

When Lambert/Duncan can examine Lott’s data and demonstrate that he cooked the numbers then they can compare him to that person from Emory.

Why? Has someone examined Bellesiles’ probate data and demonstrated that it was cooked? Oh, wait. They can’t do that since Bellesiles say the data was destroyed.

In the meanwhile, it is perfectly plain from Prof. Duncan’s letter which Lambert forwarded to this list that Duncan is on a vendetta; that he is not merely trying, in scientific fashion, to replicate Lott’s results in the unstated hope that they will prove irreproducible.

For someone who demands that Lott be treated with kid gloves, you are mighty free with the personal attacks. Prof Duncan suspects, as do I, that Lott did not carry out the survey he claims that he did. I have presented several pieces of evidence for this. It is entirely legitimate to look for evidence that will resolve the question one way or the other.

Rather, he is looking for old legislative testimony in what he admits is a “longshot” hope of incriminating Lott personally. He should be embarrassed that Lambert distributed his letter in public. It testifies to Duncan’s lack of objectivity vis-a-vis Lott, and undermines his credibility on related academic matters. The fact that Duncan first made an effort to obtain the data from Lott is irrelevant.

I find it bizarre that you assert that Duncan’s search for evidence on both sides of question proves that he is biased. A biased researcher would only look for evidence that supports one side of the question—apparently you think that Duncan should only look for evidence that exonerates Lott—that is, you want him to be biased, but when he is isn’t you accuse him of bias!

If you have have access to the legislative testimony that Duncan asked about, then I suggest that you look at the testimony, and tell us what you find.

[On Oct 03 2002 I posted this to firearmsregprof.]

Glenn Reynolds writes:

I agree that Mr. Lambert’s “payback for Bellesiles” angle is pretty obvious.

Your allegation is false.

I also note that Lott isn’t accused of publishing fraudulent scholarship, but rather of making public statements that appear to be obviously true, but for which he has not published research as a backup.

Are you actually asserting that it is “obviously true” that “98 percent of the time that people use guns defensively, they merely have to brandish a weapon to break off an attack”? That’s the claim that Lott has made on at least forty (40) occasions.

If it turned out that Lott had pretended to conduct a survey that in fact he never did, that would be serious—not of Bellesiles’, er, caliber, but serious. Of course, if it turned out that Lott were the second gunman at the grassy knoll, that would be serious as well.

At the moment, Lambert appears to have as much evidence for the one proposition as for the other.

The evidence, yet again, is that all seven other surveys gave a radically different figure. That it is mathematically impossible for a survey of the size of Lott’s to give a number that Lott says that he obtained from his survey. That Lott first claimed (falsely) that the 98% figure came from Kleck, and the first printed mention of this 1997 survey was in 2000 despite Lott citing the 98% figure dozens of times. That no-one, not colleagues, not collaborators, not experts on surveys that he consulted at the time on other matters, that not a single person that Lott can name was aware of a survey that would have required at least 300 person-days of work. That Lott has not provided any evidence that the survey was carried out, despite having had ample time to do so.

It would be more constructive if you commented on the evidence instead of pretending that it does not exist.

[On Oct 7 2002 I posted this to firearmsregprof and emailed it to Lott.]

However, that isn’t what I was referring to when I wrote “mathematically impossible”. Lott often goes on to claim that 3/4 of the times the DG User fires the shots are warning shots, that only 0.5% of DG uses involve a shot fired at the offender. Kleck’s survey turned up about 200 incidents. Since Lott’s survey had half the sample size of Kleck’s, the most he could expect to find would be 100 incidents (slightly less if you allow for the slightly lower DGU incidence he found). 0.5% of 100 is 0.5. It is mathematically impossible for half a person to report firing at an attacker.

James Lindgren replies:

Tim, your comments seem to be based on an assumption that figures in national surveys are unweighted.

Yes, I assumed that the percentages Lott gave were based on unweighted numbers.

Most national studies use a multistage stratified sampling design, which typically requires that answers be weighted. Further, a well done sample design by telephone would weight by adult family size (otherwise small households are overcounted relative to large households). A respondent in a household in the US with 2 adults would usually be weighted about .98 and a person in a household with 1 adult would be weighted about .49 (which is awfully close to .5) Lott tells me that he did not do this sort of weighting for adult household size, but he claims to have used what is usually called post-stratification, though Lott didn’t use the term. Lott said he weighted his results to match demographic information from his larger study in the book. That could easily lead to weights of half a person

I agree that it is possible that you could get a weight of one-half that way, so I should not say that the result is mathematically impossible if that is what he did.

(but, of course, one wouldn’t normally report a rate of incidence for behavior for which only one person in a sample reported the behavior unless one had pretty close to 100% of the population in one’s sample, which he doesn’t)

However, that may well be what Lott has done. Another favourite Lott statistic, presented in dozens of opeds, speeches and in his book is that “the probability of serious injury from an attack is 2.5 times greater for women offering no resistance than for women resisting with a gun.” If you examine the NCVS data this is based on, you find that 1 out of 80 women who resisted with a gun was injured, whereas if the injury rate had been the same as for women offering no resistance it would have been about 2 out of 80.

Further, Tim’s computation of significance (not shown above) assumes that the same questions were asked and that Lott’s sample was as efficient as a Simple Random Sample. With demographic weighting, no weighting for household size, and a sample taken from a CD-ROM of phone numbers, Lott’s effective sample size would be considerably smaller than the stated sample size.

I’m not sure I follow you here. Surely the only one of those that effectively reduces the sample size would be the demographic weighting? And if he took a random sample of households, the weights would be mostly close to one, so that the effective sample size would not be much different.

Further, his results would be biased in favor of small households.

I looked at Kleck’s data and the fraction who fired seems to be about the same if I calculate it without weights. (Actually there don’t even seem to be weights in the data set.) So this cannot explain, even in part, Lott’s results.

Lott’s results would be almost impossible if he used the same questions to probe the behavior as other studies, had a sampling design as efficient as a Simple Random Sample, and was not biased against large families, but Lott’s study does not have any of these characteristics, so Tim’s computations of significance would not be accurate.

However, I think I’m within a factor of two, which is still enough to say that his results are almost impossible.

One must be extremely careful in making accusations to avoid error or overstatement (to the extent humanly possible). I’m sure that Tim (who has examined this far more than I have) will correct me if I’m in error–and I urge him to do so. I’d rather that we end up with the right answers and the issues clarified than defend my analysis–which might itself be based on misunderstandings.

I most grateful for your comments. As far as I know, Lott has told you more about the design of his survey than he has told anyone else.

Suspicions have arisen about Lott’s survey because he has failed to answer reasonable questions about it.

compiled by Otis Dudley Duncan and Tim Lambert
revised 23 Oct 2005 by Tim Lambert

Note: With the exception of academic publications, some tapes and some found by LexisNexis search, these were found on the Internet. The web is, of course, not perfectly reliable, and items appearing there can later disappear. This approximately chronological listing is probably incomplete, and we welcome additions and corrections.

This page documents how often Lott has made the false claim that 98% of with-gun defences involve merely brandishing the gun. A summary and the latest update on the controversy is here.

“polls of American citizens undertaken by organizations like the Los Angeles Times and Gallup showing that Americans defend themselves with guns between 764,000 and 3.6 million times each year, with the vast majority of cases simply involving people brandishing a gun to prevent attack.”
Does Allowing Law-Abiding Citizens to Carry Concealed Handguns Save Lives? Valparaiso University Law Review, 31(2): 355-63, Spring, 1997. [The same passage also appears in: John R. Lott Jr “Concealed Handgun Laws Can Save Lives” Agenda 3(4): 499-503, 1996]

“There are surveys that have been done by the Los Angeles Times, Gallup, Roper, Peter Hart, about 15 national survey organizations in total that range from anything from 760,000 times a year to 3.6 million times a year people use guns defensively. About 98 percent of those simply involve people brandishing a gun and not using them.”
Page 41, State of Nebraska, Committee on Judiciary LB465, February 6, 1997, statement of John Lott, Transcript prepared by the Clerk of the Legislature, Transcriber’s Office.

(more…)

Lott’s reply to Duncan’s article raises some disturbing questions about Lott’s honesty. See also James Lindgren’s report on his attempt to find some evidence that Lott actually conducted a DGU survey. (more…)

(more…)

After discussion has simmered along via email, Usenet and mailing list, Marie Gryphon has posted a nice summary on her blog.

Several blogs have picked up on this: Julian Sanchez, Jim Henley, Jane Galt, Kevin Drum and Thomas M. Spencer.

Clayton Cramer has posted a letter from John Lott on his blog. Some highlights:

“The overwhelming majority of the survey work was done at the beginning of the period over which the survey was done. It has obviously been a while, but my recollection is that the small number of people surveyed after the first four or five weeks (mainly January 1997) did not include any more defensive gun uses.”

In an email to Lindgren Lott wrote “I am willing to bet that I don’t start mentioning this [98%] figure until the spring of 1997. If I use it before I said that I did the survey, I will say that they nailed me.” Lott used the figure on Feb 6, 1997 and got caught. Does he say “they nailed me”, like he promised? No, he changes his story yet again. Instead of three months, the survey is now supposed to have taken one month. This is the fourth version (1. “national surveys”, 2. “Kleck”, 3. “three month survey”, 4. “one month survey”) of the source of the 98% figure Lott has now given.

“I did another survey over 10 days this past fall and it will be discussed in a book coming out in a couple of months. The results of the survey are very similar to those previously reported.”

It is surprising that with nine published surveys giving numbers ranging from 21%-67% for the percentage firing, Lott could conduct two surveys that give far lower figures. The questions and methodology of this second survey need to be examined carefully to seek an explanation. [There used to be some comments here about the results of Lott’s new survey. Clayton had posted that Lott’s new survey found a “bit more than 4%” firing. He informs me that he removed this because it was a mistake and could not post the correct figure because Lott does not want the results made public. Consequently I have deleted my comments on this figure.]

“I am not sure that I understand why things should be weighted by household size since I was asking questions about individual experiences.”

Because the sampling unit was the household. Let me demonstrate with a small example. Suppose that half the household have two adults and half have one adult. Suppose further that adults in small and large households are equally likely to experience a defensive gun use (DGU), but adults in large households are more likely to shoot, 30% (large) vs 12% (small), say (because they are defending a spouse and children as well as themselves).

OK, now, since 2/3 of people and hence 2/3 of DGUs are in large households, it follows that 2/330+1/312 = 24% of DGUs involve shooting. However, since Lott is sampling by household, he will find half of his DGUs in small households and half in large households and hence 1/230+1/212 = 21% of his DGUs involve shooting.

Notice, however, that even in this contrived example the difference is small, so this can’t possibly explain Lott’s anomalous 2% shooting figure. I also did an analysis on the 1992-2001 NCVS data set, and it gives 21% shooting whether you weight by household or don’t weight by household.

Next, Clayton gives me too much credit. Otis Dudley Duncan deserves the credit for uncovering this mess. As part of my critique of Lott I listed his 98% claim in a long list of errors in More Guns, Less Crime, but without expressing any doubts that he actually conducted a survey. Duncan contacted me, suggesting that the survey might never have been carried out. While I was initially sceptical, Duncan was able to persuade me that there was a real issue here. (The key was Lott’s failure to respond to a reasonable request from Duncan for evidence that the survey had actually been carried out.) In September 2002, I raised the issue in Eugene Volokh’s firearmsregprof mailing list. The ensuing discussion involving Clayton, myself, Glenn Reynolds and several others prompted James Lindgren’s investigation that he reports on here.

Finally, I should comment on the overall significance of this question. Lott’s 98% claim takes up just one sentence of his book. Whether or not it’s true, it doesn’t affect his main argument, which is about alleged benefits of concealed carry laws. I don’t think any fuss would have been made if Lott hadn’t repeated the claim numerous times on TV shows, on radio shows, and in opinion pieces. I have a list containing 52 examples. By suggesting to millions of people that all you have to do defend yourself is wave a gun around when all the evidence suggests that more serious action is often required, Lott could cause people to act in ways that could get them injured or killed.

Lindgren has updated his report. Main changes are the inclusion of a reply from John Lott and a dissection of Lott’s new “Did I say three months? I meant one month. Yeah, that’s the ticket!” claim.

Lots more people have blogged on this: Glenn Reynolds, Pejman Yousefzadeh, skippy, Ken Parish, Roger Ailes (twice), Atrios and Guy Cabot. And Marie Gryphon, Julian Sanchez, Jane Galt, Kevin Drum and Thomas M. Spencer have updates or new comments.

Glenn Reynolds and Thomas Spencer mention Bellesiles, but from opposite sides. Glenn states that “Lott’s critics want, rather too obviously, for this to be another Bellesiles affair” while Thomas asks “Where are Bellesiles’ critics?”. Uh, guys, the set of Lott critics and the set of Bellesiles critics has an intersection called James Lindgren. So the answer to Thomas’ question is “front and centre!”. And, uh, Glenn, Lindgren really didn’t want this to be another Bellesiles affair. Lindgren deserves our thanks, again, for doing an unpleasant job. As for myself, I’m certainly critical of Lott, but this is not something I wanted—it’s like winning a race because your opponent is disqualified for cheating at the starting line.

Glenn also argues that even if Lott is found to have fabricated data, it would only be something akin to the Jon Ellis affair. This is patently false. Ellis lied about his past, but his dishonesty did not extend to his published work. Lott published his claim about conducting a survey on page 3 of More Guns, Less Crime and repeated it in the Criminologist. Now, the claim is only a very small part of More Guns, Less Crime, but the probate data that got Bellesiles into so much trouble was only a small part of Arming America. Lott went on to make the issue important by publishing his 98% figure in: the Los Angeles Times (three times), Investor’s Business Daily, Wall Street Journal (four times), The American Enterprise, Chicago Tribune (twice), American Bar Association Journal, Consumers’ Research Magazine, National Review, Washington Times, Insight on the News, American Experiment Quarterly, Intellectual Capital, Rocky Mountain News, Detroit News, Christian Science Monitor, National Forum, San Diego Union-Tribune (twice), Boston Herald, Dallas Morning News, Philadelphia Inquirer, New York Post (twice), Chicago Sun-Times, and the Detroit News. He also made the claim in testimony for the Nebraska Committee on Judiciary LB465 and the US House Judiciary Committee. He’s done it on TV—Uncommon Knowledge, a TV pilot hosted by John Stossel and Hardball. He’s done it on radio—South Dakota Public Radio, NPR’s ’s “Justice Talking”, the Zoh show and Radio Liberty. He’s done it in public talks—to the Independent Women’s Forum, at the 2000 Gun Rights Policy Conference and at the Eagle Council Forum XXVIII. And in all these cases he used the figure to argue against gun control. It’s hard to imagine a case more unlike that of Jon Ellis.

Finally, I should comment on the suggestions that Lott can somehow salvage things with his new survey. While I have it on good authority that he really did conduct a survey this time, it is really very easy to alter the interview records to give any result you want. Unless some trusted third party has verified all the records and results of the survey as being unaltered, there will be a big question mark over this survey too.

John Lott has emailed Jim Henley with another response to Lindgren’s report. I’ll comment on this below. Julian Sanchez also got the Lott email. I didn’t.

Comments from Jesse Walker at reason.com, skippy, Guy Cabot, Kevin Drum and Steve Verdon. Brief mentions by Iain Murray, Charles Murtaugh, Tapped, John Quiggin, Natalie Solent, Gene Hoffman, Jr., jeff and Bruce R.

In Lott’s latest response, he changes his story again. He originally told Lindgren that hadn’t discussed the survey with anyone at the time. Now he has recalled the name of an economist he discussed it with at the time. Unfortunately, when Julian Sanchez contacted this economist, he was unable to recall those discussions. He also originally told Lindgren that the survey was conducted by “several University of Chicago undergraduate volunteers” (a survey of the size he claims to have conducted would have required at least ten students, and much more than that if almost everything was done in one month as he is now claiming). Now he seems to be saying that only two students from U of Chicago were involved, and the rest were from other places. This latest story change from Lott is truly disturbing because it appears to be designed to stymie Lindgren’s plan to settle the issue. This was Lindgren’s plan:

“If someone were to email the 1997-98 Chicago college alumni twice, I remain virtually certain that at least one of those who did the study for Lott would come forward, if the study were actually done. If no one came forward after two attempts emailing the entire class, the evidence would tend to point strongly to the probability that the 1997 study was never done.”

If there only two U of Chicago students involved, then, given the frequency at which people change email addresses, it would not be at all surprising to get no response from the mass email. Lott has changed his story and changed his story until he has ended up with one that cannot be proven or disproven.

Julian also has some comments where Lott claims that the difference between the 2% shooting figure he got and the 21% to 67% shooting numbers from all nine published surveys is not statistically significant. He mentions that in his current survey he found 13 defensive gun uses, and in only one of them (8%) was the gun fired. We can calculate a 95% confidence interval for the percentage (via Wilson’s method for those interested in the technical details). It’s 1.4%-33%. (If you’re not sure what a 95% confidence interval means, it’s the same thing people are talking about when they say that a survey is plus or minus 3%.) Since the interval includes 2%, this new survey is consistent with what he alleges he found in his first survey. Trouble is, since it also includes 21%, 24%, 27% and 28% it is also consistent with most of the big published surveys. (These had hundreds of defensive gun uses in the sample.) The trouble is that the sample size of 13 defensive gun uses is much too small to say anything meaningful about the percentage who fire their weapons. Most statisticians would not even print a percentage based on a sample size of 13, instead they would print an asterisk, with a footnote stating that the sample size was too small for the number to be meaningful. Lott’s claim that his new survey confirms his previous one is nonsense.

As for Lott’s original survey, he hasn’t told us how many defensive gun uses were found in that survey (maybe he has forgotten?) so we can’t determine if his 2% shooting number was significantly different from the numbers in the other surveys, but there are only two possibilities:

1. it wasn’t, in which case he should not have repeated the statistic 50 times when there were numbers from other, vastly more statistically reliable surveys available
2. it was, in which case Lott’s latest explanation is wrong, and he still has to explain why his survey gave such a radically different number.

Jim Henley has an excellent blog roundup with thoughtful comments and even ratings.

Julian Sanchez has an update where he includes a comment from an economic researcher that repeats what I wrote below: if his new survey only has 13 defensive gun uses, then this sample is far too small to say anything useful about the frequency with which defenders shoot, and certainly can’t be regarded as “confirming” the alleged 1997 survey.

Thomas Spencer has a post that is critical of some of the folks who were all over Bellesiles for downplaying Lott’s sins.

Marie Gryphon (who broke the story into the blog world) also got the Lott email. I feel so left out.
Brief comments from Jane Galt, skippy, Alan Schussman and jeff. Ted Barlow even has a Lott light bulb joke.

Mark Kleiman has an insightful post that is definitely worth reading. Mark observes that even though Lott’s work on concealed carry laws does not rely on his alleged survey, destroying his credibility means that people cannot trust the results of his concealed carry laws. Actually, there is actually no reason to trust the results of his research on concealed carry laws in any case, since it has been superseded by the work of Ayres and Donahue who repeated it using more data and better models:

“Those who were swayed by the statistical evidence previously offered by Lott and Mustard to believe the more guns, less crime hypothesis should now be more strongly inclined to accept the even stronger statistical evidence suggesting the crime-inducing effect of shall issue laws.”
Ayres, I. & Donohue, J. 2002. “Shooting Down the More Guns, Less Crime Hypothesis” Stanford Law Review

Mark also suggests observes that permit holders very rarely commit crimes, so carry laws could not cause increases in firearms crime. However, carry laws could cause criminals to carry more frequently and increase crimes that way.

I want to comment further on the email Lott sent to some bloggers. Lott states that he has been responsive and implies that he was somehow ambushed on this issue.

Duncan first asked Lott for evidence for his survey last June. Lott did not respond. I sent him the first version of my report last September, and regularly sent him updates. He has not responded to any of them. I’ve tried my best to be open, while Lott has acted like someone with something to hide. For example, this new survey was done months ago, but Lott kept it secret until Dec 26. The results, which Lott claims are “very similar” are still secret.

Here’s a simple question that Lott could respond to if he is going to be responsive: “Why, if it came from your survey, did you attribute the 98% figure to “national surveys” and to Kleck?”

I’ll let Lott have the final word in today’s post:

“The reluctance of gun-control advocates to release their data is quite widespread. In May 1997 I tried to obtain data from the Police Foundation about a study that they had recently released by Philip Cook and Jens Ludwig, but after many telephone calls I was told by Earl Hamilton on May 27. “Well, lots of other researchers like Arthur Kellermann do not release their data.” I responded by saying that was true, but it was not something that other researchers approved of, nor did it give people much confidence in his results”
John Lott, page 291 of “More Guns, Less Crime”

Steve Verdon comments on the Ayres and Donahue study I cited yesterday. Unfortunately he doesn’t seem to have understood what their conclusions were. Try reading the abstract, Steve:

“Estimating more statistically preferred disaggregated models on more complete county data, we show that in most states shall issue laws have been associated with more crime and that the apparent stimulus to crime tends to be especially strong for those states that adopted in the last decade.”

Now this study is correlational, so it certainly doesn’t prove that carry laws cause more crime, but if you believed that Lott’s correlational study proved that more guns caused less crime you should also believe that Ayres and Donahue have now proved that more guns caused more crime.

Otis Dudley Duncan has sent me these comments, which draw attention to a key point that almost all bloggers have missed:

“There are two distinct issues in this case.

1. Lott repeatedly made erroneous statements about the findings of other researchers. None of the national surveys that he cited by name actually had any figure at all for merely brandishing or firing. One of them, the Roper survey (which was mentioned in the Feb. 6, 1997, Nebraska testimony) never even did any survey on defensive gun use. Of the polls that did collect data on firing, none of them obtained a figure anywhere near Lott’s 2 percent. So everything he has said about surveys on this topic done by others is utterly, totally false. There is no room even for reasonable doubt.
2. Did Lott himself carry out the survey that he described in his “Reply” of Sept./Oct. 2000? It appears that he will not be able to produce compelling evidence that such a survey was done. But the internal evidence indicates that no survey with the results he described can have produced responsible statistical estimates of population frequencies: sample size = 2,424; population estimate of 2.1 million defensive uses in a population of about 200 million American adults as of 1997 (rate = about .01); 98% of the uses involving merely brandishing, 1.5% firing warning shots, 0.5% firing at offender. An event (firing in self defense) with a probability of about 1 in 5,000 (.01 x .02) cannot be honestly estimated with a sample size of 2,424, let alone the 1 in 20,000 probability of firing at an offender.

“Thus, while it is important to continue the investigation Lindgren has begun into whether the survey itself was actually faked, there can be no serious question as to whether Lott is guilty of fabricating evidence in his reports on what others found. And at best he has convicted himself of gross incompetence/culpable negligence in his report on his own purported survey. As to the first of these charges, Lott has never seen fit to comment in any way on why, knowing as early as Spring 1997 that the brandishing figure of 98% was obtained in his own survey, he chose to attribute it to surveys done by others. Nor has he explained why the 2000 edition in which the claim about his own survey first appeared there was no explanation of the revision of the statement on page 3.”