Interrogating algorithms
What with the recent discovery that VW has been using software to cheat on emissions tests, there has been a sudden and widespread conversation taking place on how we can interrogate algorithms.
In an New York Times op-ed from yesterday, Zeynep Tufekci weighed in on both the VW scandal and another recent software problem of public interest, namely voting machines. She concludes that “…the public can’t always know if the device is working properly — but we can check its operation by creating auditable and hard-to-tamper-with logs of how the software is running that regulators can inspect.” She also notes that slot machines in casinos have regular such inspections, so it’s not impossible.
Another New York Times article profiles Columbia Law professor Eben Moglen, quoted as saying that “proprietary software is an unsafe building material,” because “you can’t inspect it.” That was in 2010. Ironically, the article explained, the reason automobile manufacturers gave for not allowing inspection is that individuals would set up their cars to cheat on admissions tests. Of course, that doesn’t explain why you wouldn’t open up the algorithms at least to regulators.
The inspection of algorithms is a concept that’s probably new to a lot of people, first because algorithms are marketed as “objective” and “fair,” second because they are almost by construction too complicated for an average person to understand.
But, as we’ve seen in this example, those are simply not good enough reasons not to do it anyway. There’s a trade-off when we take advantage of automation and algorithms: we get efficiency and scale, on the one hand, and on the other we lose control. In fact, we don’t really know what’s happening and when.
The very least we could do is ask them.
mathbabe 
VAM
LikeLike
I think one of the practical problems with trying to “interrogate algorithms” will be the corporate push-back, claiming things like “proprietary algorithms” or “trade secrets.”
One agreeable remedy might be to allow them to keep the internals of the algorithm a black box, but allow for regulators to send a wide variety of inputs and then inspect the outputs. This is how programming competitions worked way back when: you’d submit a code sample which supposedly solved some problem, the judges wouldn’t look at the code, instead they’d submit the code to a whole bunch of torturous inputs and iff the code outputted what was desired in every situation, the problem would be considered solved.
Of course even in this situation, there’s very little that can prevent a manufacturer from switching the algorithm at some point. So we can also focus on making things easier to audit.
LikeLike
According to the article, it was the EPA who worried about people cheating, not the automobile group. That might be even more ironic 🙂
“And two months ago, the E.P.A. said it, too, opposed such a move because people might try to reprogram their cars to beat emission rules.”
LikeLike
There’s been a fair amount of capture of regulators in the U.S., especially since the Clinton years when, in the name of spending cuts and simplification, much of the oversight was changed from in-house experts going out to the industries generating reports that are then audited (you see it in the FDA, for example, where the drug tests were farmed out to the drug companies which, surprise, often cheat, and in the finance sector). I don’t know off-hand how much of that has happened in the EPA, but this could just be a reflection of that.
LikeLike
Maybe the problem lies in that whole “corporation as person” thing?
LikeLike
In my experience, limiting performance verification/validation to just the algorithm is usually problematic. Removing an algorithm from its native operating environs and composing a simulator in which it runs does not typically tell the whole story for any nontrivial system. System verification/validation should therefore consider the implementation as a whole and not just focus on its algorithmic processes.
For example, automobile emission control is a complicated system of inter-working computer hardware, software algorithms, hardware/software interfaces, mechanical parts, etc. Neglecting any of these components risks the efficacy of the testing.
LikeLike
I think it is interesting that the Nevada Gaming Control Board insists on getting all the source code for gaming machines. I don’t work in that industry but I have never heard of this being a big controversy. But for cars, voting, planes which have a lot more riding on them (literally for the vehicles) than quarter slots in Vegas.
I doubt they actually analyze all the code, but if a suspected fraud came up there would be no doubt what code was actually being run.
I’m actually surprised VW have not done that. “We did write code like that once during development, but it never shipped in real cars.” Pause for a few days. “Oops, apparently it did, silly us.”
LikeLike