This article about recidivism risk algorithms in use in Philadelphia really bothers me (hat tip Meredith Broussard). Here’s the excerpt that gets my goat:
“As a Black male,” Cobb asked Penn statistician and resident expert Richard Berk, “should I be afraid of risk assessment tools?”
“No,” Berk said, without skipping a beat. “You gotta tell me a lot more about yourself. … At what age were you first arrested? What is the date of your most recent crime? What are you charged with?”
Let me translate that for you. Cobb is speaking as a black man, then Berk, who is a criminologist and statistician, responds to Cobb as an individual.
In other words, Cobb is asking whether black men are systematically discriminated against by this recidivism risk model. Berk answers that he, individually, might not be.
This is not a reasonable answer. It’s obviously true that any process, even discriminatory processes that have disparate impact on people of color, might have exceptions. They might not always discriminate. But when someone who is not a statistician asks whether black men should be worried, then the expert needs to interpret that appropriately – as a statistical question.
And maybe I’m overreacting – maybe that was an incomplete quote, and maybe Berk, who has been charged with building a risk tool for $100,000 for the city of Philadelphia, went on to say that risk tools in general are absolutely capable of systematically discriminating against black men.
Even so, it bothers me that he said “no” so quickly. The concern that Cobb brought up is absolutely warranted, and the correct answer would have been “yes, in general, that’s a valid concern.”
I’m glad that later on he admits that there’s a trade-off between fairness and accuracy, and that he shouldn’t be the one deciding how to make that trade-off. That’s true.
However, I’d hope a legal expert could have piped in at that moment to mention that we are constitutionally guaranteed fairness, so the trade-off between accuracy and fairness should not really up for discussion at all.
I don’t know about you, but every time I go into the bathroom at Penn Station I cry a little bit.
That’s because I remember the 1980’s version of them, and believe you me, they’re so much better now. I grew up in the Boston area but I visited a bunch in high school, which means I spent way too much time in the very few available public toilet facilities. So I can appreciate me some improved amenities.
They are relatively clean! They have toilet paper, consistently! There’s soap available next to working sinks! And, probably most importantly, it’s not a threatening experience with dirty needles all over the floor.
For that matter, while I’m on the theme, have you noticed how much nicer JFK is now compared to 1988? Maybe it’s because I’ve been flying JetBlue a lot, but that terminal is nothing like the broken-down middle school experience I remember not so fondly.
That’s all I have today, just gratitude and anti-nostalgia. And I’m sure there are lots of things we miss as well from those days of New York City, but right now I can’t think of any besides cheaper rent.
It’s been an amazing two weeks – or actually, holy crap, only 10 days – since my book launched.
I found out two days ago that my book made it onto the Longlist for the National Book Award in Nonfiction, along with 9 other books. I haven’t had time to read the other books, but I did want to mention that last year’s NBA Nonfiction winner was Ta-Nehisi Coates’s excellent Between The World And Me, which I highly recommend.
What’s exciting about being on this list is that it means the ideas in the book will get exposure. So many really excellent books never get read by many people, because of bad timing, or small marketing and publicity budgets, or just bad luck. I’m so lucky to have a book that’s been given an extremely generous amount of all of that.
This week I’ve been busy on the West Coast going to book events and giving talks. My last one is today at noon in Berkeley (820 Barrows Hall). I’ve gotten almost no sleep what with jetlag, weird traveling requirements, and pure adrenaline, but it’s been absolutely incredible.
It’s been especially fantastic to meet the people who come to these events, which so far have taken place in Seattle, the San Francisco area, and a couple in New York last week. It seems like almost every person has something to tell me, a story of algorithms they encounter at work, or that their friends do, or questions about how to get a job that they can feel proud of in data science. Some of them are lawyers offering to talk to me about FOIA law or the Privacy Act. Incredible.
Some people who have read the book already will tell me it really changed their perspective, and others will tell me they’ve been waiting years for this book to be written, because it echoes their experience and long-held skepticism.
What?! Do you guys know what that means? It means the book is working!
In any case I’m overwhelmed and grateful to be able to talk to all of them and to start and continue the conversation. It’s never been more timely, and although I had hoped to get the book out sooner, I actually ended up thinking the timing couldn’t be better.
There’s only one thing. I wish I could send a message back to myself four years ago, when I decided to write the book, or even better, to Christmas 2014, when I was convinced it was an unwritable book. I’d just want to send some encouragement, a signal that it would eventually cohere. Those were some dark days, as my family can attest to.
Luckily for me, I had good friends who kept me from losing all hope. Thank you, blog readers, and thank you friends, and thank you Jordan and Laura especially, you guys are the best!
One of the best things about having my book out (finally!) is that once people read it, or hear interviews or read blogs about it, they sometimes sending me more examples of creepy models. I have two examples today.
The first is a company called Joberate which scores current employees (with a “J-score”) on their risk of looking for a job (hat tip Marc Sobel). Kind of like an overbearing, nosy boss, but in automated and scaled digital form. They claim they’re not creepy because they use only publicly available data.
…Litify is led by an experienced executive team, including one of the world’s most influential and successful lawyers, well known VC’s and software visionaries. Litify will transform the way legal services are delivered connecting the firm and the client with new apps and will use artificial intelligence to predict the outcome of legal matters, changing the economy of law. Litify.com will become a household consumer name for getting legal assistance and make legal advice dramatically more accessible to the people…
What could possibly go wrong?
I’ll be giving a version of my book talk at the Alt Banking meeting today. Please come!
Here are the deets:
When: 2-3pm today
Where: Room 409 of the International Affairs Building, 118th and Amsterdam
Have you heard of the most recent outrage committed by a bank? Wells Fargo just got fined a total of $185 million for corrupt practices involving the accounts of depositors.
Specifically, a bunch of depositors were given accounts they didn’t sign up for, and then charged for via fees. Wells Fargo claims they have fired 5,300 low-level workers over the past five years for doing stuff like this.
But as many have pointed out, including Naked Capitalism, this is really not about low-level workers. It’s about ridiculous and unattainable sales quotas imposed on bankers, and then a complete disregard for the knock-on effects of those stupid quotas.
The fact that so much fraud went on so widely means that either the top dogs knew about it and didn’t care (I’m voting for this – after all the high pressure sales tactics were probably profitable overall even with this $185 fine) or that they had entirely insufficient controls and didn’t know about it. Either way they’re idiots, and it’s outrageous that only the underlings were fired, and not the management. For that matter the person who came up with rigid sales quotas without thinking for five minutes about what would happen next needs to get canned.
Oh wait, I just remembered: the lowest paid bank workers, who really work for very little money under tremendous pressure, have very little power. It’s time they form a union. This is not a new idea, but it’s never been more obvious.