Home > data science, finance, rant > How to evaluate a black box financial system

How to evaluate a black box financial system

November 27, 2012

I’ve been blogging about evaluation methods for modeling, for example here and here, as part of the book I’m writing with Rachel Schutt based on her Columbia Data Science class this semester.

Evaluation methods are important abstractions that allow us to measure models based only on their output.

Using various metrics of success, we can contrast and compare two or more entirely different models. And it means we don’t care about their underlying structure – they could be based on neural nets, logistic regression, or decision trees, but for the sake of measuring the accuracy, or the ranking, or the calibration, the evaluation method just treats them like black boxes.

It recently occurred to me a that we could generalize this a bit, to systems rather than models. So if we wanted to evaluate the school system, or the political system, or the financial system, we could ignore the underlying details of how they are structured and just look at the output. To be reasonable we have to compare two systems that are both viable; it doesn’t make sense to talk about a current, flawed system relative to perfection, since of course every version of reality looks crappy compared to an ideal.

The devil is in the articulated evaluation metric, of course. So for the school system, we can ask various questions: Do our students know how to read? Do they finish high school? Do they know how to formulate an argument? Have they lost interest in learning? Are they civic-minded citizens? Do they compare well to other students on standardized tests? How expensive is the system?

For the financial system, we might ask things like: Does the average person feel like their money is safe? Does the system add to stability in the larger economy? Does the financial system mitigate risk to the larger economy? Does it put capital resources in the right places? Do fraudulent players inside the system get punished? Are the laws transparent and easy to follow?

The answers to those questions aren’t looking good at all: for example, take note of the recent Congressional report that blames Jon Corzine for MF Global’s collapse, pins him down on illegal and fraudulent activity, and then does absolutely nothing about it. To conserve space I will only use this example but there are hundreds more like this from the last few years.

Suffice it to say, what we currently have is a system where the agents committing fraud are actually glad to be caught because the resulting fines are on the one hand smaller than their profits (and paid by shareholders, not individual actors), and on the other hand are cemented as being so, and set as precedent.

But again, we need to compare it to another system, we can’t just say “hey there are flaws in this system,” because every system has flaws.

I’d like to compare it to a system like ours except where the laws are enforced.

That may sounds totally naive, and in a way it is, but then again we once did have laws, that were enforced, and the financial system was relatively tame and stable.

And although we can’t go back in a time machine to before Glass-Steagall was revoked and keep “financial innovation” from happening, we can ask our politicians to give regulators the power to simplify the system enough so that something like Glass-Steagall can once again work.

Categories: data science, finance, rant
  1. November 27, 2012 at 5:28 pm

    In engineering you hit the black bock with an impulse function and look at the response. Then you can characterize what is in the black box. In the case of the financial system it is way too fragile for any impulse function but I would recommend putting the usual bankster suspects into a black room and tossing in a grenade to serve as the impulse. A dud grenade for sure and the response should be enlightening.


  2. November 27, 2012 at 9:53 pm

    This is surprisingly Milton Friedman-esque of you, Cathy! Not sure I see the benefit of looking only at the end results, even for a model (isn’t the whole point of modeling the internal dynamics, rather than an individual outcome?). It is an even bigger stretch for an entire system, because treating it like a black box would allow potential disasters to remain invisible until they explode.

    Or to go back to your wheelhouse – wasn’t the problem with the TBTF banks that they were all looking at one evaluation (the ridiculous VaR) but ignoring everything that the measure didn’t catch? Or am I (likely) missing something?

    Anyway, yet another interesting post. Certainly something to think about.


    • November 28, 2012 at 6:41 am

      I would never suggest looking only at the output as a measurement, or looking at only one measurement of the output. As the system becomes more complicated it clearly requires us to think harder about what we are asking from it. Nor would I want the evaluation period to be short – things can look bad for a couple years and still be long-term viable.

      But what I do like about this thought experiment is that, first of all, it requires us to think through what we actually want from the school system or the financial system irrespective of the internal politics, and second of all, under almost any measurement, it seems pretty clear that our current financial regulation system is a failure.


      • November 29, 2012 at 10:28 am

        I completely agree, and would take that a step further. The nice thing about thinking of finance or education as a system is that it forces you to be clear about what is in the system, and what is in the environment. People who work on the education system implicitly treat students as part of the system, which is why outcomes are measured on the basis of student performance (the validity of the current approaches is another issue entirely).
        That’s not at all true of the financial system. Borrowers, savers, 401k investors and other individuals are generally treated as being in the environment of the financial system but not as part of it. I think that’s why your approach is interesting, because measuring their outcomes would force an acknowledgement that they (we) are actually part of the system. Right now, when economists talk about systemic risk in finance they really mean risk to other market entities, with an implicit motherhood & apple pie nod in the vague direction of “the economy.” What you are talking about would be a good step toward forcing that conversation about systemic risk to acknowledge that we’re all part of the system, not extraneous to it.


  3. November 28, 2012 at 1:59 pm

    You might want to look at the Long Finance initiative (www.longfinance.net) which wrestles with the question ‘when would we know our financial system is working’, and has published a number of interesting papers in search of better questions, and maybe eventually some answers. I’m a member of the ‘kitchen cabinet’ and would welcome your input and thoughts on a set of questions or metrics.


  4. mozibur ullah
    December 4, 2012 at 9:43 am

    Why Jon Corzine got a bit of a pubic pilloring, and nothing else probably has to do with the classical problem of who ‘guards the guardians’, given the tightly woven nexus between big money & big politics. If these financial innovaters were innovating, shouldn’t they also have been innovating regulation to keep it all safe? Glass-Steagall was put there for a reason. It was good solid protection. I think Galbraith in his polemic ‘The Great Crash’ is much more honest than the practitioners of finance say & do, in what actually happens during these business cycles/bubbles: He pins it down much more to human/crowd psychology than would the average spokesperson for the industry. But then his role was to be the eagle-eyed observer, the honest & critical judge of that industry.

    Financial innovation bankrupted Iceland, and democracy worked well enough there to toss a few into jail.

    Although the States likes to call itself a democracy, it operates much more like an oligarchy. Does democracy scale? As an analogy, scaling data (Big Data) creates massive problems in itself, how does one coordinate the political decisions of a sub-continent sized country from the classical size of the Polis/City. Does it take on then, only the appearance of democracy, whereas in itself, it is something else?

    Hmm, this sounds a bit like a rant, sorry.


  1. December 1, 2012 at 6:56 am
Comments are closed.
%d bloggers like this: