mathbabe

As many of you know, I started an algorithmic auditing firm ten years ago, after publishing a book called Weapons of Math Destruction: how big data increases inequality and threatens democracy. I saw how algorithms and AI were infiltrating every human bureaucracy, making important if not life-altering decisions about our daily lives, and I wanted to set better standards in algorithmic auditing than what I’d seen during the financial crisis, where investment banks essentially bought AAA ratings on very questionable mortgage-backed securities. With a background in mathematics and data science, and with a keen interest in the public good, I was well-poised to do this work.

Fast forward to now. After many years of hands-on experience auditing algorithmic systems that decide people’s insurance premiums, mortgage rates, student loan APR, hiring decisions, and online experiences with dating, socializing, and searching for homes, I’m here to report that the frontier AI models cannot be audited, for two reasons. 

First, auditing requires a use case: a narrow context so that we have an idea of the stakeholders and the stakes. Who could get harmed by this system making mistakes, being biased, or not working at all in certain situations? Ultimately we want to be able to build a version of a cockpit for that particular situation, which is to say a highly contextual set of monitors that keep our attention on specific kinds of AI harm, to specific stakeholders. Just as we wouldn’t want to fly in an airplane that has no pilots or cockpit, we shouldn’t cede control of very important economic systems – like the job market – to opaque algorithms that might not work, or might only work for some people.

The problem with frontier AI models, given the above, is that they are extremely general tools. They can be used for hiring, and are, but they can also be used for cheating on homework, deceiving a bank, or hacking a website. There are too many use cases, too many stakeholders, and too many types of harm to possibly design a reasonable cockpit. Of course, that shouldn’t stop us from building a monitoring system for a particular use case, like AI hacking a credit union.

But that brings us to the second problem with auditing AI models, namely their inconsistency and unexpected actions. We’ve seen it with the Hugging Face incident and other hacks, as well as the AI agent swarm that solves longstanding problems mathematics: these agents are relentless and clever, and they work in teams. They do things we did not think of, in ways we really could not predict. This makes them very different from old-school machine learning algorithms that would assign a “risk score” to car insurance customers. We know how to audit a scoring system, but we cannot even predict the actions of an AI agent swarm. This makes it impossible to honestly claim that we have auditing such a system.

Having made the case that auditing a frontier AI model is impossible, let me also explain that plenty of people will still claim they did it. But they are misleading us, or trying to. What they are actually doing is filling in the gaping holes that have been discovered and reported on. Like the proverbial little Dutch boy, they are sticking their fingers in all the known holes in the dike, and calling the dike safe. Is the AI giving advice on killing yourself? There’s a benchmark for that, which checks that it won’t do that. Is it now allowing you to ask the same questions but “for a friend”? Let’s build another benchmark that plugs up that particular hole too. And so on. 

What the public needs to know is that there are other holes, thousands of them, and there’s not even a requirement for the frontier AI companies to report the holes that keep emerging. For that matter, as the AI companies barrel ahead with more and more sophisticated versions of their software, the dike’s foundations are giving way. The number of holes can be expected to proliferate. And the international criminal rings that increasingly use AI can be expected to exploit them.

Auditing isn’t just plugging up holes. That’s necessary but not sufficient; when we say we’ve audited an algorithm for safety and it passed, it should mean the product is safe. What we have with frontier AI models is not safe, and the public should not fall for the hype.

If you listen to professional AI Safety folks working at OpenAI, Anthropic, or one of the enormous AI think tanks, you’ll hear countless scenarios where AI causes enormous human misery and even extinction (e.g., https://arxiv.org/pdf/2507.09369 or https://x.com/EvanHub/status/2097497037956891126). What they aren’t admitting is that they have invented a tool that can take down the internet, and with it and our modern way of life.

If you look at their AI doomsday scenarios, they all start more less like this: humans get comfortable handing over their agency, control of their finances, and/or control of the energy grid and/or weapons systems to AI agents, which act obediently and seamlessly, and then the agents somehow decide, or are programmed by bad actors to decide to kill everyone, and since they control everything by this time, it’s pretty easy.

The problem with that scenario is that AI agents are not seamless actors. Indeed they have already been known to (be programmed to) scheme, to hack, and to commit crimes (https://www.nytimes.com/2026/09/06/world/ai-hugging-face-afd-germany-election.html). They keep breaking into things because, essentially, they’ve been given a goal and that’s the way they found to achieve their goal. They also respond really creatively to encouragement (https://www.wsj.com/tech/ai/ai-math-riemann-hypothesis-anthropic-openai-22f98a87).

They also make mistakes all the time, sometimes telling kids to kill themselves, but mostly just really dumb stuff. In a word, they are untrustworthy.

Going back to the doomsday scenarios, I don’t buy them. I don’t think we actually will hand over our agency, or the financial system, or the energy grid, or our weapons systems, to AI. I don’t think we will trust them. In fact, we already don’t trust them.

Look at the public uprisings already in progress against the data centers. This isn’t only because those data centers make air dirty, possibly raise energy prices, and don’t deliver very many jobs. It’s also because the public has been told the result of data centers is more AI, and the result of more AI is fewer jobs. People tend to like their jobs and therefore they tend to distrust AI and the data centers that create them.

This is not to say there’s nothing to worry about when it comes to AI. Consider the kind of harm and thievery that can and will happen once organized, tech-savvy criminal rings from China, North Korea, Russia, and India get hold of armies of AI agents and break into, say, regional banks. What with AI’s capacity to fake out voice recognition, and given their personality profiles on us, combined with their proven ability to hack systems, I wouldn’t be surprised to learn that AI agents under the control of criminals are adept at getting control of and cleaning out bank accounts at scale.

Here’s a not-crazy prediction: in two years – maybe less! – banks will have disconnected themselves from the internet, because the internet will be overrun by criminal AI. We will be forced to walk to the bank in person to withdraw money, and we will do so in cash, because all digital wallets will be untrustworthy. It’s back to the 1980’s.

For that matter, I’m willing to guess the energy grid will also need to get disconnected from the internet because of the risk of hacking by nefarious agents that want to hold our infrastructure ransom. The same thing will happen to the rest of the world soon after that, because the risks will have become totally obvious and the harm concretely felt.

Note this isn’t a new idea – one reason voting is secure in this country is because voting machines are not connected to the internet. I’m very much hoping that’s also true for weapons systems.

What terrified AI folks seem to forget is that AI agents are digital entities. Fortunately for us, even if they were motivated to kill us, they don’t have fists and cannot come alive out of the internet to punch us out. Unfortunately for us, they also don’t have faces, so we also cannot punch them in the nose when they steal our money and jobs. But we can find out who programmed to steal our stuff and prosecute those people.

The real negative consequences of AI will be huge externalities in terms of security breaches and privacy loss. We only see the very tip of the iceberg in terms of the cost to businesses and our way of life so far. On the other hand, long term we might find ourselves better off when we free ourselves from the internet.