The Stop and Frisk sleight of hand
I’m finishing up an essay called “On Being a Data Skeptic” in which I catalog different standard mistakes people make with data – sometimes unintentionally, sometimes intentionally.
It occurred to me, as I wrote it, and as I read the various press conferences with departing mayor Bloomberg and Police Commissioner Raymond Kelly when they addressed the Stop and Frisk policy, that they are guilty of making one of these standard mistakes. Namely, they use a sleight of hand with respect to the evaluation metric of the policy.
Recall that an evaluation metric for a model is the way you decide whether the model works. So if you’re predicting whether someone would like a movie, you should go back and check whether your recommendations were good, and revise your model if not. It’s a crucial part of the model, and a poor choice for it can have dire consequences – you could end up optimizing to the wrong thing.
[Aside: as I’ve complained about before, the Value Added Model for teachers doesn’t have an evaluation method of record, which is a very bad sign indeed about the model. And that’s a Bloomberg brainchild as well.]
So what am I talking about?
Here’s the model: stopping and frisking suspicious-looking people in high-crime areas will improve the safety and well-being of the city as a whole.
Here’s Bloomberg/Kelly’s evaluation method: the death rate by murder has gone down in New York during the policy. However, that rate is highly variable and depends just as much on whether there’s a crack epidemic going on as anything else. Or maybe it’s improved medical care. Truth is people don’t really know. In any case ascribing credit for the plunging death rate to Stop and Frisk is a tenuous causal argument. Plus since Stop and Frisk events have decreased drastically recently, we haven’t seen the murder rate shoot up.
Here’s another possible evaluation method: trust in the police. And considering that 400,000 innocent black and Latino New Yorkers were stopped last year under this policy (here are more stats), versus less than 50,000 whites, and most of them were young men, it stands to reason that the average young minority male feels less trust towards police than the average young white male. In fact, this is an amazing statistic put together by the NYCLU from 2011:
The number of stops of young black men exceeded the entire city population of young black men (168,126 as compared to 158,406).
If I’m a black guy I have an expectation of getting stopped and frisked at least once per year. How does that make me trust cops?
Let’s choose an evaluation method closer to what we can actually control, and let’s optimize to it.
Update: a guest columnist fills in for David Brooks, hopefully not for the last time, and gives us his take on Kelly, Obama, and racial profiling.
mathbabe 
So what evaluation method do you propose?
LikeLike
A WSJ article linked to by the David Brooks guest columnist is instructive:
“The Political War on the NYPD”
http://online.wsj.com/article/SB10001424127887323501004578388311774675612.html?mod=rss_mobile_uber_feed
“96% of shooting victims and 90% of murder victims in New York City are black or Hispanic.”
“The commissioner [Kelly] says the data suggest that blacks are “under-stopped significantly,” while other ethnic groups are “over-stopped.” He reports that 53% of stops involve African-Americans, though blacks commit more than 70% of crimes. Hispanics on the other hand make up 32% of stops but commit 26% of crimes.”
As a rule in population/public health it is best to prevent (eg, stop teens from smoking by raising the cost of tobacco through higher taxes) as opposed to treat (eg, lung cancer or emphysema).
The seizing of illegal guns from those most likely to have and use them would seem to reduce the number of shootings that not only kill some people but injure and possibly maim many more. The seizing of guns would seem to serve as a form of “vaccination” against shootings and cessation of reduction of the vaccination effect may not be immediately felt.
Bloomberg has donated $12 million to air commercials through an organization he co-heads, “Mayors Against Illegal Guns.” In June, Bloomberg asked donors to withhold gifts to four Democratic Senators who voted against a bill that strengthened background checks for gun purchases.
It would seem that skeptical data scientists at least owe Bloomberg and NYPD an examination their claims before the media and competing politicians indite their efforts, especially since lives may be at risk if the program is discontinued.
LikeLike
I’m perplexed as to David18s point. Mathbabe made no claim about the representativeness of the “stop and frisk” stops, only that a large number of Blacks and Latinos have been involved. Plainly, the main point was that the outcome of interest for the Mayor (death rate by murder) is one that is impacted by many different factors (health care, lead paint levels, etc). This makes it very difficult to determine if the policy is working. She proposes an alternative outcome to evaluate the program on – trust in police. Trust in police has the advantage of having (quite possibly) fewer factors impacting it, and thus we might want to center our efforts on that goal (where we may more easily “move the needle”, or rather, if the needle is moved have greater confidence that its something we’ve done that caused it).
Given that homicide rates have been in substantial decline even in places where there is no stop and frisk makes me highly dubious of the effectiveness of the policy. That trend has continued across business cycles, changes to the unemployment rate, and in places where firearm availability has changed. It will be difficult, if not impossible, to ever determine with any real confidence the effectiveness of the policy (IMHO).
If there is any real critique to be made, it’s wondering if trust in police is an outcome we care about. For quality of life of Black and Brown people of NYC, the answer is yes. For people who don’t care about that issue, and they are legion, the real issue is whether trust in police would bring down the death by murder rate or other violent/property crime rates. I suspect it would, but I’m no criminologist. Also, looked at from this angle, trust in police is already so low among minority communities I wonder if its even possible for stop and frisk to have reduced it (of course, that’s an empirical question). So the costs of the policy might actually be low, though it may extend distrust far into the future.
LikeLike
How do you measure “trust in police?” How do you then correlate it with crime statistics? Frankly, I wonder whether any criminal, White, Black, Latino, etc. is deterred by “trust in police.” Fear of police may be a better deterrent.
LikeLike
That part is all fairly straightforward. A survey with a question about trust in police (which if i’m not mistaken is a question asked in the GSS, so you have a nice national comparison, but that’s not really necessary, just a bonus), and a behavioral measure (have you ever cooperated in a police investigation, or some such). You connect the survey to Census tracts within the city (getting addresses or nearest intersections, maybe even just zipcodes), and then you can correlate the two (crime statistics are often recorded by location). There is nothing innovative there, it just takes money.
From a game theoretic point of view, I’ve always thought that it was not the severity of the punishment, but the certainty of capture that produced the greatest effects. Think about it in terms of an expected value calculation – you’d be much better off in deterring crime if you increased the probability of capture (increasing certainty of costs) than if you simply increase the costs (increasing fear of cops, and by extension, penalties). Obviously that’s a really simple model. Essentially, I’d argue that if trust in the police increases cooperation in investigations, capture increases (e.g. more arrests). That seems like a plausible deterrent… but what do I know?
LikeLike
In the WSJ, Ray Kelly deliberately twists the NYC murder rate history. http://goo.gl/CGGr9g
LikeLike