I’m writing a book called Weapons of Math Destruction
I’m incredibly excited to announce that I am writing a book called Weapons of Math Destruction for Random House books, with my editor Amanda Cook. There will also be a subtitle which we haven’t decided on yet.
Here’s how this whole thing went down. First I met my amazing book agent Jay Mandel from William Morris though my buddy Jordan Ellenberg. As many of you know, Jordan is also writing a book but it’s much farther along in the process and has already passed the editing phase. Jordan’s book is called How Not To Be Wrong and it’s already available for pre-order on Amazon.
Anyhoo, Jay spent a few months with me telling me how to write a book proposal, and it was a pretty substantial undertaking actually and required more than just an outline. It was like a short treatment of all the chapters but then two chapters pretty filled in, including the first, and as you know the first is kind of like an advertisement for the whole rest of the book.
Then, once that proposal was ready, Jay started what he hoped would be a bidding war for the proposal among publishers. He had a whole list of people he talked to from all over the place in the publishing world.
What actually happened though was Amanda Cook from Crown Publishing, which is part of Random House, was the first person who was interested enough to talk to me about it, and then we hit it off really well, and she made a pre-emptive offer for the book so the full on bidding war didn’t end up needing to happen. And then just last week she announced the deal in what’s called the Publisher’s Marketplace, which is for people inside publishing to keep abreast of the deals and news. The actual link is here, but it’s behind a pay wall, so Amanda got me a screen shot:
If that font is too small, it says something like this:
Harvard math Ph.D., former Wall Street quant, and advisor to the Occupy movement Cathy O’Neil’s WEAPONS OF MATH DESTRUCTION, arguing that mathematical modeling has become a pervasive and destructive force in society—in finance, education, medicine, politics, and the workplace—and showing how current models exacerbate inequality and endanger democracy and how we might rein them in, to Amanda Cook at Crown in a pre-empt by Jay Mandel at William Morris Endeavor (NA).
So as you can tell I’m incredibly excited about the book, and I have tons of ideas about it, but of course I’d love my readers to weigh in on crucial examples of models and industries that you think might get overlooked.
Please, post a comment or send me an email (located on my About page) with your favorite example of a family of models (Value Added Model for teachers is already in!) or a specific model (Value-at-Risk model in finance in already!) that is illustrative of feedback loops, or perverted incentives, or creepy modeling, or some such concept that you imagine I’ll be writing about (or should be!). Thanks so much for your input!
One last thing. I’m aiming to finish the writing part by next Spring, and then the book is actually released about 9 months later. It takes a while. I’m super glad I have had the experience of writing a technical book with O’Reilly as well as the homemade brew Occupy Finance with my Occupy group so I know at least some of the ropes, but even so this is a bit more involved.
mathbabe 
First, congrats, sounds like a great book.
Don’t know how you’ll keep the finance section short – there are so many examples to choose from! My favorite quote on the subject comes from “When Genius Failed”:
“The MIT types always want to short volatility,” noted Andrew Hall, an oil trader who had worked at Salomon in the 1980s. “Academics have embedded in their minds the Black-Scholes models that assume the normal distributions. They think these models are the Holy Grail.”
From the last paragraph of the first section of Chapter 7 – “Bank of Volatility”
For me, the most interesting story that really hasn’t been told well, yet, is how the models failed to take counter party credit risk into account. This was true for both the buyers (who seemed happy to ignore it) and the sellers (who seemed blind to the risk). AIG FP is the one example that is well-known, but the public reports on the matter indicate that AIG’s securities lending business required far more capital in their bailout (though the details of the securities lending business didn’t really capture the pubilc’s attention the way the derivative book did, which is odd). One of the books – “All the Devils are Here” I think – quoted someone ( Larry Fink ??) as saying that the securities lending business at AIG was the worst example of financial management that he’d ever seen.
This story of counterparty credit risk is a little hard to tell, but really gets to the heart of some of the big problems back in 2009.
LikeLike
Cathy, that’s pretty exciting, congratulations. From my perspective I’m hoping it explains, for mathematically-challenged people like myself, how modeling has become a tool for justifying unfairness.
LikeLike
That is part of the plan!!
LikeLike
Very exciting, Cathy! Congratulations. We really need some straight talk about modelling and its affects on society. Along that line, you may want to look at Jacques Ellul’s book The Technological Society, which I’ve found quite prescient in describing how society became “technical”, Ellul’s word for the sort of highly systematized global world we have today, and its effects on humanity.
LikeLike
What great news. Congratulations Cathy! I’m looking forward to reading it.
LikeLike
Thanks Julie!!
LikeLike
All must kneel before the literary hegemony of Jay Mandel – represented Mazur students publishing with Penguin Random House!
LikeLike
Congratulations! moosesnsquirrels’ reference looks deeper and frankly more interesting, but here’s another reference from earlier times: Trapped in the Net: The Unanticipated Consequences of Computerization. The book’s captivating title and unusual cover art caught my attention in a library in 1999. Turned out to be a pretty good read too.
LikeLike
Thanks, mathematrucker! Any chance you’ll be driving through Baltimore at the end of this week to catch my panel on Public Facing Mathematics at JMM like you did last year? It was so awesome to meet you in person!!
Cathy
LikeLike
Cathy it was totally awesome meeting you in person last year too! Truly a Wayne’s World moment for me… “I’m not worthy! I’m not worthy!” 🙂
In general I don’t wish my current driving job included trips to the east coast (especially at this time of the year), but for catching your JMM panel, I do wish it did. I’d be there for sure. As it happens, the job never gets me past Denver.
Good chance I’ll make the Portland MathFest this summer, my attendance at the Seattle JMM two years from now will be automatic, and the San Antonio JMM next year is on my radar. I’m already looking forward to catching whatever talks you might be giving at any of those!
LikeLike
You make it sound so simple. You just write an outline and then write two chapters! I have a feeling its not that simple. 🙂
Congrats. 🙂
LikeLike
Simple but not easy! It took me a year and a half.
LikeLike
Sounds great.
The college admissions process has some good examples of Goodhart’s Law. For instance, how standardized testing has been corrupted. Also, how the whole process has become an effort of the colleges to maximize “yield” because ratings systems use it.
LikeLike
Awesome! You go, Cathy!
LikeLike
I don’t know if you have seen this or not but a Backlight documentary about Haim Bodek, formerly of Goldman Sachs and former HFT trading firm that went broke, but he’s got some very interesting things to say…he’s a coder…and also of interest is some history of how order entry began and an old device created to enter trades with “keyboards” as the rules called for…conversations about HFT and what firms line up to get priority on the exchanges..
http://ducknetweb.blogspot.com/2013/11/the-wall-street-codenew-documentaryi.html
LikeLike
I was thinking through a few other ideas from finance this morning with a view to examples that can be made very concrete. So, not just a bunch of equations on paper, but specific numbers that should be a little easier to drive a point home for a mass audience. The two things I thought of are:
(1) The bailout on the AIG derivative book.
This piece of the AIG bailout was, what, $20 billion ish? The collateral calls from the dealers were based on pretty complicated models and the size of the collateral calls proved to be too much for AIG to pay. Contrast the $20 billion in collateral calls with Joe Cassano’s famous statement (from 2007?) – “it is hard for us, without being flippant, to even see a scenario within any kind of realm of reason that would see us losing $1 in any of those transactions.”
With the passage of time we can now look back and see how the collateral calls related to the size of the actual losses that would have been paid by these securities. Which number was a better estimate of the actual losses paid by these securities – the model’s $20 billion number, or Joe Cassano’s $1?
(2) Pension accounting models
The issue of the solvency of pensions, and particularly public pensions, has been in the news a lot lately, and probably will stay in the news for a long time. Much of the coverage has a political bias one way or the other, and there’s probably no better way to put someone to sleep than by discussing the accounting treatment of pensions. But, since pensions touch so many people’s lives, it is an important subject to talk about and one where models play an important role.
Corporate and municipal accounting for pensions is different. Corporates are required to use market based discount rates, while many public pension plans are still using 8% discount rates today. My guess is that the vast majority of people do not appreciate the economic consequences of these long term discount rates.
For example, if someone is required to pay you $1,000 20 years from now and discounts the value of that payment at 4%, they’ll record a liability of roughly $450. If they use 8% as the discount rate they’ll record a liability of about $215. When you hear about shortfalls in public pension plans, the shortfall is usually with regard to the $215 rather than the $450.
What incentives are created by pension funds needing to earn 8% on their invested capital (something you touched on in your “dumb money” post a few weeks ago)? If someone owed you $1,000 20 years from now, how much would you want them to have set aside today? Can the entire pension industry in aggregate earn rates of return that are well in excess of current risk free rates? If not, who is bearing the risk of shortfalls?
With trillions of dollars on the line and liabilities determined largely by accounting / actuarial assumptions, I think that pension models are one of the most important models for the general public to try to understand.
LikeLike
Important ideas and very well said, Thank you!
LikeLike
Love the title! Should be a great book. I view money (accounting) as merely a tool to dominate others. If we were able to trust one another completely and simply give and receive, with no competitive spirit, oh the beauty we could create. Math is a fun and beautiful thing, but the use of it is perverted. Especially in medicine, where it seems to have corrupted the scientific method. It’s time to stop misusing math.
LikeLike
Scientific research is another fertile area — particularly pharmaceuticals. The misuse of statistical tests are repeated trials until the results come out “right” leaves a lot of room for trouble.
Also, I’ve heard people talk about customizing treatment based on analysis of large data sets (and gene studies in particular). While in theory there is a lot of potential there, there is also great potential for problems resulting from bad analysis or over-fitting.
LikeLike
I agree.
LikeLike
Are you contractually obligated to have a subtitle? One of the things which always amuses me is how Ph.D. theses in math usually have only a title, but in the humanities there is typically a title and then a subtitle, as if they can’t just find a title alone that will work.
Obviously you’re not writing a thesis, of course. But the fact that you write that you already know there will be a subtitle even without knowing what it is makes makes me wonder: why? Can you conceive of finding a title for the book that can stand on its own, or do the gods of marketing say subtitles are necessary?
LikeLike
I’ve gotten the feedback that the title alone is insufficiently explanatory.
LikeLike
just another Congrats!! and another ‘love the title’… ought be LOTS of interest in such a volume! …and sounds like it will appear on shelves at just about the same time that Wall St. will have thoroughly demolished the economy 😦
LikeLike
Seems like you can pick and choose. Fat Tail, black swans, etc. Phenomenological roots in visual rhetoric, etc.
Statistics is a liars game, especially when it comes to student loan debt
http://www.theatlantic.com/business/archive/2013/07/a-mystery-behind-the-rise-of-student-debt/277809/
In fact, “the [ever-increasing?] misappropriation of mathematics to support political and corporate agendas” is the reason that Galtonian statistics has survived this long — handmaiden to mass administrative bureaucracy that tyrannizes over us all, and dictates to the elites what they can and cannot do. The standard history for this is “Statistics in Britain, 1865-1930: The social construction of scientific knowledge,” (1981) Donald MacKenzie.
For those interested in this problem as it played out in finance and banking: Physics and Finance: S-Terms and Modern Finance as a Topic for Science Studies
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.137.7441&rep=rep1&type=pdf
This is now especially true, as you say, in public education. See David Labaree’s
Click to access The_Lure_of_Statistics.pdf
LikeLike
I’m glad this is getting attention in the publishing world. I’ve been studying the use of math and computer models ever since Enron and the entire merchant energy business blew up in 2001 (my career has been in the energy industry). Then I launched and helped direct a small hedge fund in the electricity space and realized that investment firms used their computer “models” as sales tools, not content builders, and of course no one was ever privvy to what was in the black box. I even started to pursue a graduate degree in sociology with financial engineering as a focus. Last year, I managed to get an agent on the strength of a book proposal similar to your description but he couldn’t make anything happen. In the meantime, Nate Silver’s The Signal and the Noise came out and I told myself, “well, it’s been done. Time to move on.” Anyway, I’m a newbie to your blog here, but Donald McKenzie’s An Engine, Not A Camera, which I happened on in the University of Chicago bookstore when my daughter entered there in 2006, really laid an excellent foundation for me on the academic side of what I’d been dealing with essentially for most of my career as the electricity industry proceeded through thirty years of deregulation. I have a shelf full of books you probably already know about, but the shelf begins with How to Lie with Statistics. The error analysis exercises I had to conduct for laboratory experiments in engineering school was probably the single most valuable instruction I ever got in how to truly understand the modern world. One day I hope to develop and teach a course on this for math and non-math inspired people everywhere. In the meantime, this publication of mine might interest you and your posse on this blog: http://visar.csustan.edu/aaba/Makansi2009.pdf
Also, these links are the culmination of the research I began pursuing but had to pause because I could no longer devote the time necessary and thought it demeaning that a person my age would be required to take the GREs to get a doctorate degree.
Click to access Makansi_TheSociologyofSystemicRisk_Research%20Statement_SOCIOL7085.pdf
Click to access Makansi_MakingWhiteCollarCrimeLegal_ResearchStatement_SOCIOL8687.pdf
LikeLike
Congratulations on the book deal! I just found your super-awesome blog a few months ago, and have been a regular reader ever since!
One thing I was wondering is whether you think there’s a risk, maybe in any criticism of the socially unjust and simply wrong-headed use of mathematical models, for mathematics itself to get caught in the crossfire. Especially as mathematics is already maligned so much in other contexts in our society. Do you think you will have a chapter on the distinction between math in and of itself, and the use of math in various contexts, or on some positive uses of mathematical modeling as a counterbalance? I guess the reason why this jumped out at me is because the title, while clearly pretty awesome, does put math in the driver’s seat of evil (rather than a twisted combination of math and the people using it, for which I’ll have to read the rest of the book!). Anyway, I was just wondering what your thoughts were on this angle.
I look forward to reading the book when it comes out, and thanks for such a great blog!
LikeLike
congrats on the book. i don’t really have any substantial suggestions, but here are some off-the-wall ones:
studying only functors modeled by a schemes … would your husband consider that a pervasive and destructive force in daily life? perhaps times have changed and it’s no longer so pervasive.
another topic: use of math in astrology. there are really people that believe astrology is legitimate because it uses numbers. i try to stay away from such people, but unfortunately one of them is my mom.
LikeLike
Ok you can shoot me for this one, but I thought it was worth a mention on the automation that’s going on out there with news and writing books..and Charlie Siefe said he’s going to include a chapter in his next book on the “journobot” too…and I mention this as it deals with quality, which you produce of course, but take a look at the videos here and especially the last one where the “journobot” can mass produce books. I look at it this way as having a benefit as far as research in getting material, but on the other side turn the whole bot loose to write book and the news? I try to see both sides but the concern here is again folks wanting to leave the human value out of all of this and produce bot generated news farms. Forbes is trying to sell their bot that they have been using for news articles for a while now on some of what they produce.
The inventor calls his bot “Long Tail” and the video is a bit shocking to see how far the automation has gone here but it can be both a tool or abused and we want quality information so again shoot me if you like but this stuff is out there and working it’s way around in media of all different types today with journalists “interviewing data bases” and if you leave it up to the bot to create “so called value based news and stories”…well I don’t need to say a lot more there as you get the idea..indeed the potential for abuse scares me a bit too.
http://ducknetweb.blogspot.com/2014/01/cms-modifies-policy-on-disclosure-on.html
LikeLike
I am hoping to use quotes from this book to make my math background look impressive.
LikeLike