Auditing Algorithms
Big news!
I’ve started a company called ORCAA, which stands for O’Neil Risk Consulting and Algorithmic Auditing and is pronounced “orcaaaaaa”. ORCAA will audit algorithms and conduct risk assessments for algorithms, first as a consulting entity and eventually, if all goes well, as a more formal auditing firm, with open methodologies and toolkits.
So far all I’ve got is a webpage and a legal filing (as an S-Corp), but no clients.
No worries! I’m busy learning everything I can about the field, small though it is. Today, for example, my friend Suresh Naidu suggested I read this fascinating study, referred to by those in the know as “Oaxaca’s decomposition,” which separates differences of health outcomes for two groups – referred to as “the poor” and the “nonpoor” in the paper – into two parts: first, the effect of “worse attributes” for the poor, and second, the effect of “worse coefficients.” There’s also a worked-out example of children’s health in Viet Nam which is interesting.
The specific formulas they use depends crucially on the fact that the underlying model is a linear regression, but the idea doesn’t: in practice, we care about both issues. For example, with credit scores, it’s obvious we’d care about the coefficients – the coefficients are the ingredients in the recipe that takes the input and gives the output, so if they fundamentally discriminate against blacks, for example, that would be bad (but it has to be carefully defined!). At the same time, though, we also care about which inputs we choose in the first place, which is why there are laws about not being able to use race or gender in credit scoring.
And, importantly, this analysis won’t necessarily tell us what to do about the differences we pick up. Indeed many of the tests I’ve been learning about and studying have that same limitation: we can detect problems but we don’t learn how to address them.
If you have any suggestions for me on methods for either auditing algorithms or for how to modify problematic algorithms, I’d be very grateful if you’d share them with me.
Also, if there are any artists out there, I’m on the market for a logo.
mathbabe 
can I suggest remeike forbes of the Jacobin Mag http://remeike.com as a designer. He is good and easy to work with (did an excellent Logo for an organisation I was involved in). Great at translating ideas into visuals.
LikeLike
Congrats! I like the name. Might try Jarrod Barretto (http://www.jarrodbarretto.com) for the logo. He did my mathtransit.com logo (rail tracks on a Möbius strip—originally in a Bloomberg.com news article about mass transit costs).
LikeLike
What exciting news!
For logo design, I highly recommend Carlos Carmona Medina. He’s a friend’s partner, and a pro – he designed the new logo for SocArXiv among many other things.
LikeLike
Congrats! As for a logo, I would only suggest be wary of anything with interlocking letters…
LikeLiked by 1 person
Great point!
LikeLike
This is very exciting! Congratulations!
I think there is a huge untapped market for this kind of service. Independent model validation and risk analysis would be an enormous service for many consumers of models. You don’t have a lot of description (yet) on the webpage.
Let me give you a specific example; I’d like to know if this is within scope of what you plan to do. I work in predictive Analytics for a large global company. In addition to the models we build, our company works with independent consultants (McKinsey, Deloitte, IBM, etc.) who also do predictive modeling. Our company can only see the results of the modeling and some model details, but we cannot see the full models, for proprietary reasons. Obviously, we’d like to know whether the models are really rigorous, or are they blowing smoke. The opportunity here would be for a third party audit of these models, which preserves model IP while giving us assurance that the models are valid (or are garbage). Is that a kind of service you would see yourself offering?
Congrats again!
LikeLiked by 1 person
Yes I would expect the same techniques to work for that kind of audit as well. The point is we can no longer assume that algorithms are inherently objective or, for that matter, meaningful.
LikeLike
Congrats on your new venture — if you have the time and location of the event (as with the birth of a child), then you can set-up a birth chart (i.e., horoscope), and understand its prospects and challenges better.
The decomposition paper refers to poverty as a “lack of purchasing power.” Certainly this is a good definition, but linking purchasing power to “malnutrition inequalities between poor and nonpoor children in Vietnam” is far from obvious. For example, what direct relation do “the correlates of poverty” such as poor water and sanitation, low educational levels, have to do with a child’s personal purchasing power? With height?
Part of the problem lies with the epistemology laid down by the World Bank and used by the government of Vietnam that is used by the paper. How closely are decomposition results tied to underlying distinctions? What happens when those distinctions are varied, do the results vary as well?
LikeLike
> “the poor” and the “nonpoor”
There are two types of people, those that divide people into two types and those that don’t. (Robert Benchley circa 1920?) Be careful how you decide on your partitioning.
LikeLike
I liked this! Thanks!
LikeLike
There are three types of people. Those who know math, and those who don’t.
LikeLike
How wonderful! Let us know if you’ll be hiring in the coming months/years…
LikeLike
Congratulations! Great idea.
Why an S-corporation, rather than an LLC that elects to be treated as a C-corporation? Did someone advise you that an S-corp was a better solution?
Also, when I read oneilrisk.com, my brain parses that as ONE-ILRISK.com. Might I suggest OneilRisk.com, which doesn’t change the URL, but makes it easier to read, while browsers are free to display it all in lower case, or, perhaps adding a ‘-‘ in the name, such as oneil-risk.com?
In any case, congratulations and lots of luck.
LikeLike
S-Corp was cheaper to file in New York state.
I’ll look into changing that.
LikeLike
Two methods that might be worthwhile to look at decompositions:
1. Hierarchical linear models; these are nice because you can obtain variance components for various levels in the model. You can look at, for example, individual, neighborhood, and state level variance. It would likely be the most helpful for descriptive purposes, but I often find it really helpful.
2. For looking at algorithms that track change over time, growth mixture models might be particularly helpful. At their heart, they are latent class models that define classes based on trajectory parameters (e.g., intercept and slope in the simplest models). This gets you out of the assumption that errors follow a single distribution. One can then look to see if particular groups end up in particular classes and how much confidence you have (based on, e.g., a multinomial logistic regression).
I’m happy to dig out the best references that I have for these if it would be helpful.
LikeLike
I’d definitely like references for these because I’ve tried in the past to understand them but could not.
LikeLike