mathbabe

As many of you know, I started an algorithmic auditing firm ten years ago, after publishing a book called Weapons of Math Destruction: how big data increases inequality and threatens democracy. I saw how algorithms and AI were infiltrating every human bureaucracy, making important if not life-altering decisions about our daily lives, and I wanted to set better standards in algorithmic auditing than what I’d seen during the financial crisis, where investment banks essentially bought AAA ratings on very questionable mortgage-backed securities. With a background in mathematics and data science, and with a keen interest in the public good, I was well-poised to do this work.

Fast forward to now. After many years of hands-on experience auditing algorithmic systems that decide people’s insurance premiums, mortgage rates, student loan APR, hiring decisions, and online experiences with dating, socializing, and searching for homes, I’m here to report that the frontier AI models cannot be audited, for two reasons. 

First, auditing requires a use case: a narrow context so that we have an idea of the stakeholders and the stakes. Who could get harmed by this system making mistakes, being biased, or not working at all in certain situations? Ultimately we want to be able to build a version of a cockpit for that particular situation, which is to say a highly contextual set of monitors that keep our attention on specific kinds of AI harm, to specific stakeholders. Just as we wouldn’t want to fly in an airplane that has no pilots or cockpit, we shouldn’t cede control of very important economic systems – like the job market – to opaque algorithms that might not work, or might only work for some people.

The problem with frontier AI models, given the above, is that they are extremely general tools. They can be used for hiring, and are, but they can also be used for cheating on homework, deceiving a bank, or hacking a website. There are too many use cases, too many stakeholders, and too many types of harm to possibly design a reasonable cockpit. Of course, that shouldn’t stop us from building a monitoring system for a particular use case, like AI hacking a credit union.

But that brings us to the second problem with auditing AI models, namely their inconsistency and unexpected actions. We’ve seen it with the Hugging Face incident and other hacks, as well as the AI agent swarm that solves longstanding problems mathematics: these agents are relentless and clever, and they work in teams. They do things we did not think of, in ways we really could not predict. This makes them very different from old-school machine learning algorithms that would assign a “risk score” to car insurance customers. We know how to audit a scoring system, but we cannot even predict the actions of an AI agent swarm. This makes it impossible to honestly claim that we have auditing such a system.

Having made the case that auditing a frontier AI model is impossible, let me also explain that plenty of people will still claim they did it. But they are misleading us, or trying to. What they are actually doing is filling in the gaping holes that have been discovered and reported on. Like the proverbial little Dutch boy, they are sticking their fingers in all the known holes in the dike, and calling the dike safe. Is the AI giving advice on killing yourself? There’s a benchmark for that, which checks that it won’t do that. Is it now allowing you to ask the same questions but “for a friend”? Let’s build another benchmark that plugs up that particular hole too. And so on. 

What the public needs to know is that there are other holes, thousands of them, and there’s not even a requirement for the frontier AI companies to report the holes that keep emerging. For that matter, as the AI companies barrel ahead with more and more sophisticated versions of their software, the dike’s foundations are giving way. The number of holes can be expected to proliferate. And the international criminal rings that increasingly use AI can be expected to exploit them.

Auditing isn’t just plugging up holes. That’s necessary but not sufficient; when we say we’ve audited an algorithm for safety and it passed, it should mean the product is safe. What we have with frontier AI models is not safe, and the public should not fall for the hype.