Home > modeling, rant, statistics > The scienciness of economics

The scienciness of economics

October 23, 2013

A few of you may have read this recent New York TImes op-ed (hat tip Suresh Naidu) by economist Raj Chetty entitled “Yes, Economics is a Science.” In it he defends the scienciness of economics by comparing it to the field of epidemiology. Let’s focus on these three sentences in his essay, which for me are his key points:

I’m troubled by the sense among skeptics that disagreements about the answers to certain questions suggest that economics is a confused discipline, a fake science whose findings cannot be a useful basis for making policy decisions.

That view is unfair and uninformed. It makes demands on economics that are not made of other empirical disciplines, like medicine, and it ignores an emerging body of work, building on the scientific approach of last week’s winners, that is transforming economics into a field firmly grounded in fact.

Chetty is conflating two issues in his first sentence. The first is whether economics can be approached as a science, and the second is whether, if you are an honest scientist, you push as hard as you can to implement your “results” as public policy. Because that second issue is politics, not science, and that’s where people like myself get really pissed at economists, when they treat their estimates as facts with no uncertainty.

In other words, I’d have no problem with economists if they behaved like the people in the following completely made-up story based on the infamous Reinhart-Rogoff paper with the infamous excel mistake.

Two guys tried to figure what public policy causes GDP growth by using historical data. They collected their data and did some analysis, and they later released both the spreadsheet and the data by posting them on their Harvard webpages. They also ran the numbers a few times with slightly different countries and slightly different weighting schemes and explained in their write-up that got different answers depending on the initial conditions, so therefore they couldn’t conclude much at all, because the error bars are just so big. Oh well.

You see how that works? It’s called science, and it’s not what economists are known to do. It’s what we all wish they’d do though. Instead we have economists who basically get paid to write papers pushing for certain policies.

Next, let’s talk about Chetty’s comparison of economics with medicine. It’s kind of amazing that he’d do this considering how discredited epidemiology is at this point, and how truly unscientific it’s been found to be, for essentially exactly the same reasons as above – initial conditions, even just changing which standard database you use for your tests, switch the sign of most of the results in medicine. I wrote this up here based on a lecture by David Madigan, but there’s also a chapter in my new book with Rachel Schutt based on this issue.

To briefly summarize, Madigan and his colleagues reproduce a bunch of epidemiological studies and come out with incredible depressing “sensitivity” results. Namely, that the majority of “statistically significant findings” change sign depending on seemingly trivial initial condition changes that the authors of the original studies often didn’t even explain.

So in other words, Chetty defends economics as “just as much science” as epidemiology, which I would claim is in the category “not at all a science.” In the end I guess I’d have to agree with him, but not in a good way.

Finally, let’s be clear: it’s a good thing that economists are striving to be scientists, when they are. And it’s of course a lot easier to do science in microeconomic settings where the data is plentiful than it is to answer big, macro-economic questions where we only have a few examples.

Even so, it’s still a good thing that economists are asking the hard questions, even when they can’t answer them, like what causes recessions and what determines growth. It’s just crucial to remember that actual scientists are skeptical, even of their own work, and don’t pretend to have error bars small enough to make high-impact policy decisions based on their fragile results.

Categories: modeling, rant, statistics
  1. Kevin H
    October 23, 2013 at 8:06 am

    Cathy

    Well said.

    Kevin

  2. Guest2
    October 23, 2013 at 8:29 am

    Back to that first sentence — there’s still more issues than those you describe.

    “… the sense among skeptics that disagreements about the answers to certain questions suggest that economics is a confused discipline, a fake science whose findings cannot be a useful basis for making policy decisions.”

    Skeptics, such as myself, find that economists aren’t even asking the right questions, and this is why their “answers” give the impression of “a confused discipline.” From the outside, the challenge is coming from other disciplines (such as sociology) that are penetrating into the “mature core” of economics. In the case of sociology, political economics, neo-institutional economics, economic sociology, have all challenged the lack of attention to social, cultural and institutional assumptions, so the problem isn’t the misreading of internal debates, but about the legitimacy of economics and its construction of the atomized-self. Both concepts are passé.

    Thus, the problem is how this delegitimated discipline (“fake science”) can *still* maintain standing as the “useful basis for making policy decisions.” The word “science” is itself facing these kinds of pressures, like the other ubiquitous and rhetorical “innovation.” If we know the history well enough, then we know why these are suspect.

    The fact is that the Op-ed author was not aware of this demonstrates to me that he is captive of the cognitive bubble frequently found in academia — it isn’t called the Ivory Tower for no reason. Self-absorption to the point of denial of the obvious, as seen in this first sentence, is the hallmark of the fanatic.

    • October 23, 2013 at 11:20 am

      Couldn’t agree more with the post and comment. Someone wise has said: ‘What happens in Academia should stay in Academia”.

  3. TRN
    October 23, 2013 at 8:58 am

    This would be interesting if economists had not been trying to convince us they were scientists for the last 80 years. Scienciness is a good description of the methods economists use at the macro level. Explaining results mathematically allows the somewhat dismal results to appear scientific.

    • Guest2
      October 23, 2013 at 9:38 am

      In a 2000 paper on parameterization, the sociologist-physicist Harrison C. White of Columbia goes even further in his characterization of economics and economists:

      “Oddly enough, economic theorists of markets, who talked as if they were doing physics-like analysis, disdained parameters. They turned out to resemble in my eyes a Melanesian cult. Led by Paul Samuelson, this cult had happily accepted mysterious cargo dropped from field theory ‘airplanes’ in physics without ever understanding that parameterization was indispensable for tying to realities. These economic theorists just wrote EQUATIONS, perfectly general and remarkably useless equations. Like Melanesian chiefs they — aside from odd-men like Wassily Leontief — were not worried by the lack of operational results. They provided the religious incantations (equations without parameters) while a lower caste … laboriously, and behind the scenes, did drudge work in description. The latter were unguided by theory and did not seem exhilarated by the screed of Holy Equations spread across their economics firmament.” (Sociological Theory 18/3:507-508)

      But, as White even admits, “some version of statistical fitting does come into its own, as in the seminal work of Leo Goodman, mentioned earlier, and of econometrician Zvi Griliches, who taught us about diffusion of hybrid corn as well as of features of Detroit autos.” However, please notice how limited this endorsement is. “The usual modeling formulae for change, in differential and partial differential equations, are inadequate for social science.”

  4. FogOfWar
    October 23, 2013 at 1:22 pm

    Is “an emerging body of work…” a reference to behavioral economics?

  5. Fats Grobnik
    October 23, 2013 at 2:18 pm

    While your criticisms of epidemiology echo mine of a good portion of medical research, it’s never good to claim that an entire discipline is discredited. This is simply arrogant and probably reflects your ignorance of the broader field. I personally know a few epidemiologists whose primary tools are ethnography. They study how social networks (the not-online kind) affect the spread of disease in impoverished, underdeveloped areas. Your stated criticisms, upon which you condemn the entire field, are largely irrelevant to this.

    Second, you’ve missed a key point of what makes something a science. It’s not about studying historical data and making claims based on a fit or error bars. Science is about making **falsifiable predictions** from specific conditions, and then independently testing those predictions. Here, independent testing includes both subsequently gathered data (not just statistical cross-validation, which is quite weak), and using fundamentally different approaches to test the same hypothesis.

    In fact, scientists only really trust a result when it’s survived several independent experimental tests. As a specific example, protein structures are often determined from high-resolution X-ray diffraction data. This involves purifying the protein outside of the cell, crystalizing it, and then “doping” it with heavy atoms to resolve phase issues in the X-ray diffraction data. That’s three steps removed from a protein in cytosol.

    Why trust these structure? It’s not because the error bars are small. The error bars measure convergence of the diffraction data, not how accurately the structure reflects the in vivo structure. The structures are generally trusted them because enough of them have shown good agreement with other observations: biochemical observations (kinetic, chemical structure modifications, etc.), independent structural measures (NMR structure determinations, FRET distance probes, etc.). Even given that, a careful scientist trusts a particular crystal structure only when its implications agree with other, independent measures.

    Error bars reflect precision. They tell us little about the underlying accuracy of measurements or models. Specifically, they tell us next to nothing about systematic error. That only comes from careful work, falsifiable predictions, and independent means of investigation. Science is not curve fitting and statistical hypothesis testing. They form important tools in quantitative work, but they are simply tools. Falsifiable prediction and independent inquiry are the very core.

    • October 23, 2013 at 2:50 pm

      Of course I did not to say every epidemiologist is not a scientist. Same for economists of course. I was using the terms in the same way that Chetty did.

      Your complaint would be reasonable if Chetty was talking about one kind of epidemiologist and I focused exclusively on another. But that is not the case. His example of epidemiology is still a poor one despite the fact that there exist counter-examples to a general rule.

      And as for whether I know what science is, I think we agree, and there’s no need to be rude.

      The point of having error bars is to point out that, if we redid the experiments, we might not expect the same answer. In other words, falsifiable predictions.

      The point of my Reinhart-Rogoff story, since you didn’t seem to get it, is that all kinds of things happened historically, and we can’t say what happened on average and expect that to be the entire story. Indeed given too few data points, and no chance to set up a natural experiment, there are few other ways to establish whether something is predictable besides empirical bootstrapped errorbars.

  6. Abe Kohen
    October 23, 2013 at 11:20 pm

    I’m with Kuhn on this one. Economics is just one more pseudo-science. If you think financial models are bad, what would you say about economic models? Throwing in a few mathematical formulas doesn’t make it a science.

    • Guest2
      October 24, 2013 at 8:48 am

      In a 2000 paper on parameterization, the physicist-sociologist Harrison C. White of Columbia University says much the same thing about economics and economists. Read on!

      “Oddly enough, economic theorists of markets, who talked as if they were doing physics-like analysis, disdained parameters. They turned out to resemble in my eyes a Melanesian cult. Led by Paul Samuelson, this cult had happily accepted mysterious cargo dropped from field theory ‘airplanes’ in physics without ever understanding that parameterization was indispensable for tying to realities. These economic theorists just wrote EQUATIONS, perfectly general and remarkably useless equations. Like Melanesian chiefs they — aside from odd-men like Wassily Leontief — were not worried by the lack of operational results. They provided the religious incantations (equations without parameters) while a lower caste … laboriously, and behind the scenes, did drudge work in description. The latter were unguided by theory and did not seem exhilarated by the screed of Holy Equations spread across their economics firmament.” (Sociological Theory 18/3:507-508)

      But, as White even admits, “some version of statistical fitting does come into its own, as in the seminal work of Leo Goodman, mentioned earlier, and of econometrician Zvi Griliches, who taught us about diffusion of hybrid corn as well as of features of Detroit autos.” However, please notice how limited this endorsement is. “The usual modeling formulae for change, in differential and partial differential equations, are inadequate for social science.”

    • October 28, 2013 at 1:45 pm

      And Hayek, for that matter, and his criticism of “scientism.” No claim about his theories here, just that he called a spade a spade.

  7. aj
    October 24, 2013 at 7:00 am

    “Even so, it’s still a good things that economists are the hard questions,”
    —–>
    “Even so, it’s still a good thing that economists are asking the hard questions,”

  8. October 25, 2013 at 6:19 pm

    With two self-contradictory definitions of “Inflation”, how the hell do economists ever hope to “solve” the real problem we are suffering: Deflation.

    Definition 1: Inflation = Rising CPI;
    Definition 2: Inflation = Too much money.

    Real Problem: Deflation = Chronic shortage of money.
    Cause: Charging of interest on money at-source (i.e. bank-created loans).

    Solution: Social Credit Policy (ref: C.H. Douglas).

  9. November 8, 2013 at 10:32 am

    “…despite the differences between the Nobel winners Robert J. Shiller and Eugene F. Fama, economics is nonetheless a science just like medicine will be vindicated when the Nobel Prize in Medicine is given to two doctors, one of whom accurately predicts the occurrence of heart attacks and the other who declares that heart attacks don’t exist.”

    PAUL DUGUID
    Berkeley, Calif.,

  1. November 25, 2013 at 2:06 am
Comments are closed.
Follow

Get every new post delivered to your Inbox.

Join 1,484 other followers

%d bloggers like this: