Chameleon models
Here’s an interesting paper I’m reading this morning (hat tip Suresh Naidu) entitled Chameleons: The Misuse of Theoretical Models in Finance and Economics written by Paul Pfleiderer. The paper introduces the useful concept of chameleon models, defined in the following diagram:
Pfleiderer provides some examples of chameleon models, and also takes on the Milton Friedman argument that we shouldn’t judge a model by its assumptions but rather by its predictions (personally I think this is largely dependent on the way a model is used; the larger the stakes, the more the assumptions matter).
I like the term, and I think I might use it. I also like the point he makes that it’s really about usage. Most models are harmless until they are used as political weapons. Even the value-added teacher model could be used to identify school systems that need support, although in the current climate of distorted data due to teaching to the test and cheating, I think the signal is probably very slight.
mathbabe 
I can’t seem to get the links to the paper to work.
But I find your comments that whether assumptions matter depends on the stakes, and models are harmless until used as weapons, very troubling. “No harm, no foul” standards of reliability only encourage bad thinking and gives an undeserved air of reliability. If economics had followed the sort of rigourous review standards used in the real sciences, then we’d likely have better policies today. Friedman’s argument was very much meant to create the sort of double standard needed to insinuate a lot of bad ideas—and policies—into public policy.
As they say, you fight like you train. So, training on bad models that are given passes just because “they’re harmless” (who decides that by the way) means that when you really need a good model you won’t know what one looks like or who has the best chance of providing one.
LikeLike
Sorry! I think I fixed the links.
Yeah well what I mean is, if you have a crappy model to decide on how to bet on the market, and the worst case scenario is you lose your money, then I don’t care. But if it’s going to affect the average citizen, it deserves an enormous amount of public scrutiny.
On Mon, Sep 29, 2014 at 10:03 AM, mathbabe wrote:
>
LikeLike
What allows an innocent mathematical construct to be used as a chameleon?
Part of it is that the aspects of the real world that a model does not describe, or doesn’t describe well, are not what the people using the model care about, or not what they perceive.
For example, we might try to measure the value of academic research articles by the number of citations they receive, and try to model the generation of valuable research by a mathematical description of the accumulation of citations to articles. That’s okay, but it leaves out lots of important stuff. Mathematics articles, especially those on abstract, pure, and non-applied topics, do not get cited as much as biochemistry articles. If mathematics, its proofs, and the different characters of various kinds of mathematics are all real things that we want, and are different from the things we want from biochemistry, then the model is not going to allow us to talk about that stuff.
It would be crazy to infer that mathematics articles are less valuable than biochemistry articles.
But people who are not familiar with the distinctively valuable things in pure mathematics might be disinclined to acknowledge that they are missing something. They may feel that those who insist on the importance of that stuff are just mystifying something, or trying to avoid accountability.
It’s similar, I think, for any attempt to assert that good teaching, or teaching practices, have some value that fails to show up in the economic models used to detect it.
There is a basic failure of perception. It’s like trying to talk to a tone-deaf person about harmony.
LikeLike
Friedman was right that scientific theories are not strictly logical. Anything follows from a falsehood, and it is asking too much to require true assumptions. However, that does not mean that anything goes, that the conclusions justify the premises.
IMO, we should in general apply the same standards to assumptions as we do to scientific predictions. What are the errors? If the errors of the assumptions are too great, we ditch them.
LikeLike
Perception, that hits the nail on the head for sure. With healthcare information that I do, they will not question the numbers even if it’s almost obvious that there’s room to be a skeptic and this is scary indeed. I wrote a post about the new head of the DOJ needing some tech knowledge and fat chance of course of it ever happening but it’s the same old story with one model and mathematical dupe after another out there.
CMS right now too is in a world of panic as their models are failing with the imbalance of models and keeping the human in the picture. It’s getting real ugly and there’s no back up models either, so why you see so many jumping ship. People are getting hurt and if you did not see the ridiculous article in the Atlantic by Zeke Emanuel, read his nonsense as he thinks he should not want to live past 75 and we should all think that way. He’s just a walking, talking commercial for Untied Healthcare/Optum anyway and you can find that all over the web. Zeke also was a big contributor to the writing of the ACA, scarier indeed. He’s done a good job on selling false realities and models to the CMS, very scary indeed. It’s to the point now you just kind of look at what United puts in their reports and you see the similarities of what CMS is pushing too. I would say we easily have some chameleon model types over there today.
LikeLike
Russ Roberts interviewed Pfleiderer recently on EconTalk: http://www.econtalk.org/archives/2014/09/paul_pfleiderer.html
LikeLike
Thanks a lot for this post! One sees “chameleons” constantly in economic models and it’s great to have a term for it. In fact I have a published critique of the ambiguity-aversion literature where this is one major point, but one that was a little hard to make snappy in print (in private I could say “These people are slippery about their motivation” but you can’t really print that.)
The point is that in economics you never have good enough data to sort through all possible models, so combining purely empirical analysis with a judgment of plausibility of the story (model) is unavoidable. But when convenient this is denied, as per Milton Friedman, and you hear that the story is just “as if”, so please don’t trouble us with objections that it’s not plausible.
LikeLike
Hey great can you provide a link?
LikeLike
It hits me in retrospect that though in conversations I often mentioned the “chameleon” phenomenon, it hardly appears in the paper, which is possibly only of interest to those who know the field: http://pages.wustl.edu/files/pages/imce/weinstein/AmbiguityPublished.pdf
LikeLike
The bookshelf model – filter – public policy relationship is not a static condition as witness the commentary in the interview with Clifford Asness and Vanguard group Founder John Bogle (http://www.valuewalk.com/2014/09/asness-bogle-investing-pension-funds-full-video/)
The interview begins talking about underfunded pension plans and a policy that the investment strategy of such plans should obtain a return of 8 percent. Very likely the 8 percent figure was a filter based on some measure of stock and bond price trends in a distant past. It’s an assumption-policy standard that’s inapplicable today and for some dominant sectors of the equities market, Asness and Bogle suggest,
not likely to be attainable.
Most of the remainder of the interview deal with active versus passive (index anchored) investing of which Bogle is well-known. Bogle and Asness note that for the long-term, passive investment looks to be an optimal tactic, however b) there are short-term, zero-sum-like opportunities in which some actively invested activities (e.g., “dumb money” such as mom and pop investors or do-it-yourself investors) net poor performances that can be of benefit to those poised to exploit such situations.
Bogle argues that overall extensive information (data) in among active trading operations isn’t a significant cost factor in those operations. It’s an empirical claim on which I have not independently verified.
How to break a suspect, but operative model-filter imbedded into public policy such as 8% returns on pension investments looks to go beyond the otherwise provocative architecture of Paul Pfleiderer’s paper. And as for retirement, be it public plans or the now championed more uncertain individual-directed investment efforts, this looks, at least to me, to have more gravity than sketching an idealized landscape for models and their explanatory and, public policy value given the challenges to overturn existing policies.
LikeLike
Krugman has blogged a lot about the uses and limitations of models, and the way that Freshwater Econ went of the rails when they stopped caring about the fact that their assumptions were ridiculous and their models didn’t match anything in the real-world data…
LikeLike
I think part of the issue is that different social networks can view the same “thing” quite differently, a property which makes the “thing” a chameleon. This, of course, undermines a key Aristotelian dogma; but it shows how important networks and POV are.
LikeLike
Chameleon models? More like bait-and-switch models.
LikeLike