Crappy modeling
I’m here to vent today about crappy modeling I’m seeing in the world of finance. First up is the 14% corrections of home sales from 2007 to 2010 we’ve been seeing from the National Association of Realtors. From the New York Times article we see the following graph:
Supposedly the reason their models went so wrong was that they assumed a bunch of people were selling their houses without real estate agents. But isn’t this something they can check? I’m afraid it doesn’t pass the smell test, especially because it went on for so long and because it worked in their favor, in that the market didn’t seem as bad as it actually was. In other words, they had a reason not to update their model.
Here’s the next on our list, namely unemployment projections from the Fed versus actual unemployment figures, brought to us by Rortybomb:
Again we see outrageously bad modeling, which is always and consistently biased towards good news. Is this better than having no model at all? What kind of model is this biased? At the very least can you shorten your projection lengths to make up for how inaccurate they are, kind of like how weather forecasters don’t predict out past a week?
Finally, I’d like to throw in one last modeling complaint, namely about weekly unemployment filings. It seems to me that every December for the past few years, the projected unemployment filings have been “surprising” economists with how low they are, after seasonal adjustment.
First, seasonal adjustment is a weird thing, and was the subject of one of my earliest posts. We effectively don’t really know what the numbers mean after they are seasonally adjusted. But even so, we can imagine a bit: they look at past years and see how much the filings typically dip or surge (sadly it looks like they typically surge) at the end of the year, and assume the same kind of dip or surge should happen this year.
Here’s my thing. The fact that the filings surge less than expected the fourth or fifth year of an economic slump shouldn’t surprise anyone. These are real people, losing their real jobs with real consequences, right before Christmas. If you were a boss, wouldn’t you have made sure to have already fired someone in the early Fall or be willing to wait til after the holidays, especially when they know the chances of getting hired again quickly are very slim? Bosses are people too. I do not have statistically significant evidence for this by the way, just a theory.
mathbabe 
I don’t think it is entirely fair to blame the models alone.
First consider the way the statistics are collected and released. The rates at which data is collected from various sources mean that the initial data released is some kind of estimated to be revised once or twice over subsequent months. Thus the models are built on shifting sands.
Secondly the models are weakest exactly at turning points in the market, when higher moments are most significant. Since the higher the moments in the data probably got destroyed in data cleaning, even if the time series is long enough to estimate them. Thus this is expected behaviour for models.
The only odd thing is the systematic bias in the sign of the errors. These may reflect political pressures on the modellers or skewed chain of financial incentive.. Professor Bent Flyvberg investigated this sort of issue in transport planning models. He found that while the level of error was large for all transport models, errors in highway models fell above and below the actual outcome about equally. Errors for rail models however were skewed towards overestimating economic benefit. He concluded by looking at political and economic pressure on modellers (and even questioning the ethics of some).
http://books.google.co.uk/books?id=tenILJ-MowQC&pg=PA121&dq=bent+flyvbjerg+rail+highway+model&hl=en&sa=X&ei=WHD2TtSvJcu2hAf147ifAQ&ved=0CEsQ6AEwAA#v=onepage&q=bent%20flyvbjerg%20rail%20highway%20model&f=false
Finally, one wonders how often they recalibrate their models.
LikeLike
Does this really have to do with economists vs. quants? It seems more like just an issue of bias.
LikeLike