Big data, disparate impact, and the neoliberal mindset
When you’re writing a book for the general public’s consumption, you have to keep things pretty simple. You can’t spend a lot of time theorizing about why some stuff is going on, you have to focus on what’s happening, and how bad it is, and who’s getting screwed. Anything beyond that and you’ll be called a conspiracy theorist by some level of your editing team.
But the good thing about writing a blog is that you can actually say anything you like. That’s one reason I cling so strongly to mathbabe; I need to be able to write stuff that’s mildly conspiracy-theoretical. After all, just because you’re paranoid doesn’t mean nobody’s out to get you, right?
Anyhoo, I’m going to throw out a theory about big data, disparate impact, and the neoliberal mindset. First I need to set it up a bit.
Did you hear about this recent story whereby Facebook just got a patent to measure someone’s creditworthiness by looking at who their friends are and what their credit scores are? They idea is, you are more likely to be able to pay back your loans if the people you’re friends with pay back their loans.
On the one hand, it sounds possibly true: richer people tend to have richer friends, and so if there’s not very much information about someone, but that person is nevertheless inferred to be “friends with rich people,” then they might be a better bet for paying back loans.
On the other hand, it also sounds like an unfair way to distribute loans: most of us are friends with a bunch of people from high school, and if I happened to go to a high school filled with poor kids, then loans for me would be ruled out by this method.
This leads to the concept of disparate impact, which was beautifully explained in this recent article called When Big Data Becomes Bad Data (hat tip Marc Sobel). The idea is, when your process (or algorithm) favors one group of people over another, intentionally or not, it might be considered unfair and thus illegal. There’s lots of precedent for this in the courts, and recently the Supreme Court upheld it as a legitimate argument in Fair Housing Act cases.
It’s still not clear whether a “disparate impact” argument can be used in the case of algorithms, though. And there are plenty of people who work in the field of big data who dismiss this possibility altogether, and who even claim that things like the Facebook idea above are entirely legitimate. I had an argument on my Slate Money podcast last Friday about this very question.
Here’s my theory as to why it’s so hard for people to understand. They have been taken over in these matters by a neoliberal thought process, whereby every person is told to behave rationally, as an individual, and to seek maximum profit. It’s like an invisible hand on a miniature scale, acting everywhere and at all times.
Since this ideology has us acting as individuals, and ignoring group dynamics, the disparate impact argument is difficult if not impossible to understand. Why would anyone want to loan money to a poor person? That wouldn’t make economic sense. Or, more relevantly, why would anyone not distinguish between a poor person and a rich person before making a loan? That’s the absolute heart of how the big data movement operates. Changing that would be like throwing away money.
Since every interaction boils down to game theory and strategies for winning, “fairness” doesn’t come into the equation (note, the more equations the better!) of an individual’s striving for more opportunity and more money. Fairness isn’t even definable unless you give context, and context is exactly what this mindset ignores.
Here’s how I talk to someone when this subject comes up. I right away distinguish between the goal of the loaner – namely, accuracy and profit – and the goal of the public at large, namely that we have a reasonable financial system that doesn’t exacerbate the current inequalities or send people into debt spirals. This second goal has a lot to do with fairness and definitely pertains broadly to groups of people. Then, after setting that up, we can go ahead and discuss the newest big data idea, as long as we remember to look at it through both lenses.