mathbabe

A few weeks ago I was a “data wrangler” at the first Data Without Borders datadive weekend. My group of volunteer data scientists was exploring the NYPD “Stop, Question and Frisk” data from the previous few years. I blogged about it here and here.

One thing we were interested in exploring was the extent to which this policy, whereby people can be stopped, questioned, and frisked for merely looking suspicious (to the cops) is racist. This is what I said in my second post:

We read Gelman, Fagan and Kiss’s article about using the Stop and Frisk data to understand racial profiling, with the idea that we could test it out on more data or modify their methodology to slightly change the goal. However, they used crime statistics data that we don’t have and can’t find and which are essential to a good study.

As an example of how crucial crime data like this is, if you hear the statement, “10% of the people living in this community are black but 50% of the people stopped and frisked are black,” it sounds pretty damning, but if you add “50% of crimes are committed by blacks” then it sound less so. We need that data for the purpose of analysis.

Why is crime statistics data so hard to find? If you go to NYPD’s site and search for crime statistics, you get really very little information, which is not broken down by area (never mind x and y coordinates) or ethnicity. That stuff should be publicly available. In any case it’s interesting that the Stop and Frisk data is but the crime stats data isn’t.

I still think it is outrageous that we don’t have open source crime statistics in New York, where Bloomberg claims to be such a friend to data and to openness.

And I also still think that, in order to prove racism in the strict sense of the above discussion, we need that data.

However, my overall opinion has changed about whether we have enough data already to say if this policy is broadly racist. It is. My mind changed reading this article from the New York Times a couple of weeks ago. It was written by a young black man from New York, describing his experiences first-hand being stopped, questioned, and frisked. The entire article is excellently written and you should take a look; here’s an excerpt:

For young people in my neighborhood, getting stopped and frisked is a rite of passage. We expect the police to jump us at any moment. We know the rules: don’t run and don’t try to explain, because speaking up for yourself might get you arrested or worse. And we all feel the same way — degraded, harassed, violated and criminalized because we’re black or Latino. Have I been stopped more than the average young black person? I don’t know, but I look like a zillion other people on the street. And we’re all just trying to live our lives.

The argument for this policy is that it improves crime statistics. For some people, especially if they aren’t young and aren’t constant targets of the policy, it’s probably a price worth paying to live in a less crime-ridden area.

And we all want there to be less crime, of course, but what we really want is something even more fundamental, which is a high quality of life. Part of that is not being victimized by crooks, but another part of that is not being (singled out and) victimized by authority either.

I think a good thought experiment is to consider how they could make the policy colorblind. One obvious way is to have cops in every neighborhood performing stop, question and frisk to random people. The argument against this is, of course, that we don’t have enough cops or enough money to do something like that.

Instead, to be more realistic about resources, we could have groups of cops randomly be assigned to neighborhoods on a given day for such stops. If you think the policy is such a good crime deterrent, than you can even weight the probability of a given neighborhood by the crime rate in that neighborhood. (As an aside, I would love to see whether there’s statistically significant reason to believe that this policy does, in fact, deter crime. So often mayors and policies take credit for lowered crime rates in a given city when in fact crime rates are going down all over the country in a kind of seasonality way.) So in this model the cops are more likely to land in a high-crime area, but eventually by the laws of statistics they will visit every neighborhood.

My guess is that, the very first time the Upper East Side is chosen randomly, and a white hedge fund manager is stopped, questioned, and frisked by a cop, who takes away his key and enters his apartment, terrorizing his family while he’s handcuffed in the back of a cop car, is the very last day this policy is in place.