Home > finance, modeling, news, rant, statistics > We don’t need more complicated models, we need to stop lying with our models

We don’t need more complicated models, we need to stop lying with our models

April 3, 2013

The financial crisis has given rise to a series of catastrophes related to mathematical modeling.

Time after time you hear people speaking in baffled terms about mathematical models that somehow didn’t warn us in time, that were too complicated to understand, and so on. If you have somehow missed such public displays of throwing the model (and quants) under the bus, stay tuned below for examples.

A common response to these problems is to call for those models to be revamped, to add features that will cover previously unforeseen issues, and generally speaking, to make them more complex.

For a person like myself, who gets paid to “fix the model,” it’s tempting to do just that, to assume the role of the hero who is going to set everything right with a few brilliant ideas and some excellent training data.

Unfortunately, reality is staring me in the face, and it’s telling me that we don’t need more complicated models.

If I go to the trouble of fixing up a model, say by adding counterparty risk considerations, then I’m implicitly assuming the problem with the existing models is that they’re being used honestly but aren’t mathematically up to the task.

But this is far from the case – most of the really enormous failures of models are explained by people lying. Before I give three examples of “big models failing because someone is lying” phenomenon, let me add one more important thing.

Namely, if we replace okay models with more complicated models, as many people are suggesting we do, without first addressing the lying problem, it will only allow people to lie even more. This is because the complexity of a model itself is an obstacle to understanding its results, and more complex models allow more manipulation.

Example 1: Municipal Debt Models

Many municipalities are in shit tons of problems with their muni debt. This is in part because of the big banks taking advantage of them, but it’s also in part because they often lie with models.

Specifically, they know what their obligations for pensions and school systems will be in the next few years, and in order to pay for all that, they use a model which estimates how well their savings will pay off in the market, or however they’ve invested their money. But they use vastly over-exaggerated numbers in these models, because that way they can minimize the amount of money to put into the pool each year. The result is that pension pools are being systematically and vastly under-funded.

Example 2: Wealth Management

I used to work at Riskmetrics, where I saw first-hand how people lie with risk models. But that’s not the only thing I worked on. I also helped out building an analytical wealth management product. This software was sold to banks, and was used by professional “wealth managers” to help people (usually rich people, but not mega-rich people) plan for retirement.

We had a bunch of bells and whistles in the software to impress the clients – Monte Carlo simulations, fancy optimization tools, and more. But in the end, the banks and their wealth managers put in their own market assumptions when they used it. Specifically, they put in the forecast market growth for stocks, bonds, alternative investing, etc., as well as the assumed volatility of those categories and indeed the entire covariance matrix representing how correlated the market constituents are to each other.

The result is this: no matter how honest I would try to be with my modeling, I had no way of preventing the model from being misused and misleading to the clients. And it was indeed misused: wealth managers put in absolutely ridiculous assumptions of fantastic returns with vanishingly small risk.

Example 3: JP Morgan’s Whale Trade

I saved the best for last. JP Morgan’s actions around their $6.2 billion trading loss, the so-called “Whale Loss” was investigated recently by a Senate Subcommittee. This is an excerpt (page 14) from the resulting report, which is well worth reading in full:

While the bank claimed that the whale trade losses were due, in part, to a failure to have the right risk limits in place, the Subcommittee investigation showed that the five risk limits already in effect were all breached for sustained periods of time during the first quarter of 2012. Bank managers knew about the breaches, but allowed them to continue, lifted the limits, or altered the risk measures after being told that the risk results were “too conservative,” not “sensible,” or “garbage.” Previously undisclosed evidence also showed that CIO personnel deliberately tried to lower the CIO’s risk results and, as a result, lower its capital requirements, not by reducing its risky assets, but by manipulating the mathematical models used to calculate its VaR, CRM, and RWA results. Equally disturbing is evidence that the OCC was regularly informed of the risk limit breaches and was notified in advance of the CIO VaR model change projected to drop the CIO’s VaR results by 44%, yet raised no concerns at the time.

I don’t think there could be a better argument explaining why new risk limits and better VaR models won’t help JPM or any other large bank. The manipulation of existing models is what’s really going on.

Just to be clear on the models and modelers as scapegoats, even in the face of the above report, please take a look at minute 1:35:00 of the C-SPAN coverage of  former CIO head Ina Drew’s testimony when she’s being grilled by Senator Carl Levin (hat tip Alan Lawhon, who also wrote about this issue here).

Ina Drew firmly shoves the quants under the bus, pretending to be surprised by the failures of the models even though, considering she’d been at JP Morgan for 30 years, she might know just a thing or two about how VaR can be manipulated. Why hasn’t Sarbanes-Oxley been used to put that woman in jail? She’s not even at JP Morgan anymore.

Stick around for a few minutes in the testimony after Levin’s done with Drew, because he’s on a roll and it’s awesome to watch.

Categories: finance, modeling, news, rant, statistics
  1. April 3, 2013 at 12:03 pm

    This is no rant; people don’t listen to what doesn’t suit them.

  2. Benoit Essiambre
    April 3, 2013 at 12:09 pm

    amen

    • April 7, 2013 at 11:32 am

      Nancy
      Thanks for this post. One reason why we try to keep it as simple as possible…. and open (no secrets)
      Thanks for sharing
      Urs
      @ComMetrics

  3. April 3, 2013 at 12:11 pm

    Cathy, it seems as if the chiefs in charge are trying to work backwards from a particular goal they want played out.

    And data science employees are merely sacrifice-able pawns to get there, truth and integrity be damned!

    In particular, this statement

    > We had a bunch of bells and whistles in the software to impress the clients – Monte Carlo simulations, fancy optimization tools, and more.

    suggests that all the fancy math is just a way to signal authority. Because, unlike real science, there’s actually such a thing as the self-fulfilling prophecy in the marketplace — witness pump-and-dump scams.

    • Nanne
      April 18, 2013 at 12:25 pm

      Well in science it is a well known fact in the meta analysis business that if people believe in a hypothesis it is more likely to occur in their results. This is not because of any evil intent, but simply how we interpret results. During a course on epidemiology they gave some pretty interesting examples on alternative medicine, its effects and how the believe of researchers impacts the results. They argue if you know it you are likely to be less influenced.

    • April 29, 2013 at 4:31 pm

      “> We had a bunch of bells and whistles in the software to impress the clients – Monte Carlo simulations, fancy optimization tools, and more.<
      suggests that all the fancy math is just a way to signal authority. Because, unlike real science, there’s actually such a thing as the self-fulfilling prophecy in the marketplace — witness pump-and-dump scams."

      I would like to propose a challenge to a convocation of Quant Proselytizers of the various Derivational Gospels, such as including an Analytics/PDE Fanatic, a Tree-Model Fanatic, a Monte-Carlo/Martingale Fanatic, and a Finite-Difference Fanatic.

      This proposal is in the spirit of William T. Shaw, author of Modelling Financial Derivatives with MATHEMATICA®, who opined : "In fact, all these methods should give the same answer when applied carefully to a problem to which they can all be applied."

      My challenge is that each highly complex model is to attempt validation by simplifying their assumptions, variables, parameters, boundary conditions, etc. until specialized application to a mutually agreed benchmark problem is feasible. The validity criteria should be that the models agree in their answer to the benchmark problem.

      In the highly unlikely event of agreement of all the specialized models, they would each be candidates for usefulness in their generalized, complex domains. If, on the other hand, all of the models would mutually disagree, then they are all nonsense and have no claim to any scientific utility simply based on using impressively sophisticated mathematical techniques.

      The highly complex "modeling" system known as the General Theory of Relativity had to be capable of simplification and validation to agreement with the Special Theory of Relativity which in turn had to be simplifiable to classical 19th century physics.

      If the Apostles of the various and contradictory heresies of quantitative finance want to claim the relevance of their speaking in the tongues of sophisticated mathematics, let them either meet such a challenge or else wash away as exposed false prophets.

  4. Etienne
    April 3, 2013 at 12:20 pm

    I love JPMorgan. It’s fascinating how low they have gone and what kind of people they have been hiring. As many other firms, they are asking their employees to warm or cool the thermometers, like a child who doesn’t want to go to school. It could be cute in a magic thinking way if it wasn’t wrecking the lives of many.

  5. April 3, 2013 at 12:32 pm

    Thank you, great post and it goes on in other industries too, like financials in healthcare too. Agree the quants are used as vehicles since they have the talent and get pushed to venture in areas to where they shouldn’t have to go to keep their jobs. I think you are right on the money as far as some executives are “not dumb” as they might want us to believe when it comes to making a lot money.

  6. John
    April 3, 2013 at 12:46 pm

    Thanks, Cathy. I, too, was impressed by Sen. Levin’s willingness to ask the hard questions. Did you notice Sen McCain just taking up space? No questions??!! The American people deserve better. Still, I’ll be surprised if any of the banksters get prosecuted for willing neglect of the regulation. The only solution for this mess is to have a few of the “best and brightest” crooks spend some serious time in federal prison. But with “crony capitalism,” I doubt we’ll see it…

    • April 3, 2013 at 12:52 pm

      Eric Holder has already told Congress that the banksters are too big to prosecute.

  7. April 3, 2013 at 12:51 pm

    Out of the park, Cathy! One the frustrations I have with people who talk about “lying with statistics” is how they focus on the “statistics” and not the “lying”. Given the lack of mathematical education in our society, it’s easy to blame the tool and not the (lying) worker.

  8. boris
    April 3, 2013 at 12:53 pm

    Are you sure that a quant has “no way of preventing the model from being misused”? How about insisting on adding a bunch of statements along the lines of “assert(stock_return < 0.5)" to the code that parses input parameters?

  9. April 3, 2013 at 1:01 pm

    Thank you so much for this post. I would add to it a bit. I think that lying (in all its forms) exists with the various models. It also exists in the presentation of the output of those models, usually by the sin of omission. This omission is usually done by a salesperson or sales team that has absolutely no idea about the lying that is internal to the model as well as external to the limitations of the models themselves — a sort of plausible deniability. But I would add based upon my experience that there are also some good and well-meaning people who don’t even know that they are lying. That’s not to excuse them or the industry (because I think the industry encourages such a blind eye) but the problem is even more systemic than simple numerical manipulation.

  10. April 3, 2013 at 1:09 pm

    One last thing. You suggest that wealth management firms perhaps overstate expected returns and understate risk. I know that when I was at US trust we limited the risk measure to two standard deviations, in a 5, 7 or more deviation world. But the bigger even bigger lie may be that anyone could put in ANY numbers that would reflect the future. In the quadratic analysis, assumed future correlations among assets do not have to reflect past correlations. Average returns do not have to revert to a “God-set “mean. And risk may not be fully captured by standard deviation. Even the SEC states “past performance does not necessarily predict future results.” I have always wondered why they felt it necessary to include the word “necessarily.” It’s a fudge word and I think this statement would be more true and stronger without the word “necessarily.”

  11. April 3, 2013 at 1:19 pm

    This is arranging the deck chairs on the Titanic. The real models that need to be built are the ones that demonstrate that the current financial system that creates money only through debt is not sustainable without periodic debt jubilees or debt deflation (major recessions and depressions).

    Banks which own and direct all political will have made a debt jubilee a forgotten relic of our past. And have craftily persuaded the MSM to push the meme that debt deflation is the fault of the 99% or jut the poor or was inevitable.

    But other systems of money creation have been proposed and tested on small scales (Social Credit and demmurage currencies).

    I would love to see less discussion about modeling a broken system with a broken model, the model must be broken to mask the fact that the system is broken, and a new discussion about modeling an alternative to the existing system.

    Models that only work under a limited subset of real world conditions are nothing new, engineers have been killing people with them for decades.

  12. April 3, 2013 at 1:19 pm

    And if you ever get around to it, I would love to read your critique of how Monte Carlo simulations are used by wealth managers across the country.

  13. badmax
    April 3, 2013 at 1:31 pm

    From JPMorgan’s own report, I quote

    “Specifically, after subtracting the old rate from the new rate, the spreadsheet divided by their sum instead of their average, as the modeler had intended. This error likely had the effect of muting volatility by a factor of two and of lowering the VaR…. It also remains unclear when this error was introduced in the calculation.”

    http://files.shareholder.com/downloads/ONE/2272984969x0x628656/4cb574a0-0bf5-4728-9582-625e4519b5ab/Task_Force_Report.pdf

    • April 3, 2013 at 1:54 pm

      Too big to fail. Too incompetent to succeed. Too important to prosecute.

  14. pjm
    April 3, 2013 at 1:53 pm

    Put another way, the issue isn’t fancy models, it’s moral hazard which may be an intractable feature of finance from its historical beginnings. It’s interesting that in economics (especially financial economics) the view of finance in society is Panglossian. In other disciplines, say political sociology, the size of the finance sector is considered to be inversely proportional to a nations general economic and social well-being. I wonder who’s right.

    • April 3, 2013 at 1:56 pm

      Both problems can be fixed with appropriate taxation, regulation, and criminal prosecution.

  15. April 3, 2013 at 4:26 pm

    “[JP Morgan-Whale] Bank managers knew about the breaches, but allowed them to continue, lifted the limits, or altered the risk measures…”

    Jamie wasn’t “lying” to himself or to others when he tweaked the fallacious model and then doubled-down on the bad bet. He is a True Believer in the Nobelist-BadMath and the Ivy League economist “scientists” who have convinced the world of finance that the Normal curve eventually rules. These True-Believer Banksters, however, are criminals for the fraud they carry out to further their false prophecies.

  16. grwww
    April 4, 2013 at 9:56 am

    The problem with risk based investment, is that it artificially inflates the economy without there being real value, that can be “amplified” into real money. Instead, the money is just “there”, and that allows “growth” in an unequal formula (no new employment or new hard investments) and thus more cash exists that is not functioning in the economy, as a stimulus to physical growth.

    Risk based investments have stunted the economic development of the world for more than a decade now, and will continue to be the bane of the economy as long as the volumes continue to be so dominate. Risk is good, but the result has to be “hard”, not “imaginary”.

  17. Hinheckle Jones
    April 4, 2013 at 3:07 pm

    The financial models are bad, but the global warming ones are O.K? In both cases the effect is to change how people invest, and what they invest in.
    It reminds me of something I heard once: “four legs good, two legs bad” or was it: “four legs good, two legs better”?

    • April 4, 2013 at 4:07 pm

      What are you saying? That global warming models can never be “scientific” nor “objective”, just because they can be used to change how we live?

      There’s no self-fulfilling prophecy in global warming. If the glaciers are melting, no amount of fervent anti-belief by myself or by persuasion of the masses will prevent that.

      On the other hand, you can jolt the price of anything if enough people believe you. Case in point: Goldman Sachs and “Peak Oil.”

      Behind your comment is the waft of an assumption that no one can never be objective in nothing, especially not those ivory tower types whose leanings are too close to dem tree huggers (another one of those types!) for comfort.

      And behind this huge can of worms is a plaintive yearning: whither science in such a poisoned atmosphere?

  18. madcap
    April 7, 2013 at 3:13 pm

    this was so amusing. Quantitative models are being used for lying? that’s where a bit of history would help. Do you recall people using entrails (or pick your mechanism such as seeds, astrology, palmistry, numerology) to predict the future? they were also doing risk management. Guess what the palmistry or priests used to do? they used to lie.

    the idea that people need to stop lying is like asking people to be nice. Good, thank you for that tip. heh.

  19. April 8, 2013 at 6:23 pm

    Excellent article – excessive complexity and professional lying is the biggest threat to a sustainable economy today.

  20. Dwight Haskins
    April 9, 2013 at 10:47 am

    Excellent discussion. My concern is that compliance and ethics – as mostly understood and practiced – are void of the bigger social consideration of inclusivity, equality, and fairness by those commercial enterprises who by sheer size and dollars, can get regulators to cave in. This creates an uneven playing field and the regular folks end up being disadvantaged. Certainly, this has been the rule of business through all of time. Yet, one has to believe things have slowly gotten worse.

    The government in a rush to placate the public rushed implementation of the Dodd-Frank Act before they understood fully the “root” causes of the crisis. Now, sadly, efforts are being wasted to back into why Dodd-Frank is the solution for the entire industry. Of course, most wise folks know that the crisis was a large bank issue and reforms need to be made in how large banks are supervised.

    While it is important for regulators and the industry to get a better handle on risk identification in banks, the major problem once agains lies with the blind spots in large bank risk models. Unfortunately, risk identification models is still something the federal government agencies still do not understand for the most part.

    As we have come to learn, the predictive value of our models of “systemic causes of conflict” progressively diminishes as conflicts move from emerging to imminent. At that point, precipitating causes become paramount, and precipitating causes are much more likely to be driven by capricious decisions and unforeseeable, random events that defy prediction. This nearly negates the value of all risk map tools out there, save one or two.

    Moreover, as crises mount, decision makers invariably encounter fiercely competing interpretations of events, both from internal analysts and policy wonks and from external analysts. The considerable energies now being devoted to improving the capacity to predict crises or conflict will no doubt yield some fruit, although not enough.

    It is important not to overstate the ability to predict. As the recent crisis has shown us, herd behavior on the part of regulators and blind spots can make even the most elaborate risk models worthless in predicting a crisis. One management complexity and risk identification tool that stands apart from the crowd is Ontonix. I am of the firm opinion that this tool or something similar needs to be available to all central bankers and financial regulators if we are to have a fair chance of identifying the next crisis before it is too late.

    Just as important as the risk tool, we need a better understanding of the “root” causes of financial crises and what role does poverty, “regulatory and political capture” by outside influences, and uneven distribution of resources play. Broader social concerns are not being dealt with although intertwined with banking policy. By ignoring the underlying factors we are only addressing the symptoms rather than the causes of financial crises.

    We know that measures need to be taken towards reducing economic poverty by lifting up the working poor and by achieving broad-based economic growth. This has to be a step toward future conflict and crisis prevention. Preventive strategies must therefore work to promote human rights, to protect minority rights and to see that all members of society share in the American dream. Yet, the burdens placed on commercial enterprises by government need to be lifted to the extent possible (reduce unnecessary/burdensome regulations where redundant.)

  21. April 11, 2013 at 11:29 am

    Thanks for this post. There’s no end to human greed – which is the real reason that these people manipulate the models.

  22. April 23, 2013 at 5:36 pm

    It’s truly very difficult in this active life to listen news on Television, therefore I only use internet for that purpose, and get the most recent information.

  23. October 14, 2013 at 10:55 am

    I’m always late to these comment threads, but you might enjoy formalizing the complexity argument in terms of the computational complexity of manipulating financial derivatives:

    Arora, S., Barak, B., Brunnermeier, M., & Ge, R. (2011). Computational complexity and information asymmetry in financial products. Communications of the ACM, 54 (5)

    CDOs/CDSs increase (instead of the advertised effect of decreasing) the power created by information asymmetry. By using there derivatives, we are giving more power to people who have better access to information. Thus, we are widening an already existing power gap.

  1. April 4, 2013 at 5:00 am
  2. April 5, 2013 at 5:02 pm
  3. April 11, 2013 at 5:03 am
  4. April 11, 2013 at 7:40 am
  5. April 11, 2013 at 12:00 pm
  6. April 11, 2013 at 1:43 pm
  7. April 11, 2013 at 7:30 pm
  8. April 12, 2013 at 8:28 am
  9. July 21, 2013 at 9:19 am
  10. July 21, 2013 at 8:43 pm
  11. September 22, 2013 at 7:58 am
  12. October 6, 2013 at 11:09 am
  13. October 13, 2013 at 11:45 pm
  14. January 23, 2014 at 5:08 pm
  15. January 28, 2014 at 12:01 pm
Comments are closed.
Follow

Get every new post delivered to your Inbox.

Join 986 other followers

%d bloggers like this: