mathbabe

Recently in my friend Peter Woit’s blog, he discussed the idea of establishing a ratings agency for physics. From his blog:

In this week’s Nature, Abraham Loeb, the chair of the Harvard astronomy department, has a column proposing the creation of a web-site that would act as a sort of “ratings agency”, implementing some mathematical model that would measure the health of various subfields of physics. This would provide young scientists with more objective information about what subfields are doing well and worth getting involved with, as opposed to those which are lingering on despite a lack of progress.

Abraham Loeb was proposing to describe the field of String Theory as a perfect example of a bubble. And it’s absolutely true that String Theory has provided finance with tons of brilliant young orphans who either got disillusioned with the field or simply couldn’t get a job after writing a Ph.D. or after a post-doc. It provides an extreme example of a mismatch between supply and demand.

Would a ratings agency for scientific theories help? I don’t think so.

The very basic reason, as Peter points out, is that it’s hard to evaluate scientific theories while they are unfolding. There are two underlying causes: first, people in a field are too invested to admit things aren’t working out, and second, by the nature of scientific research, things could not work out for some time but then eventually still work out. It’s not clear when to give up on a theory!

Ignoring those problems, imagine a “mathematical model” which tries to gauge the success of a field. What would the elemental quantities be that would signify success? Would it count the number of proven theorems? Crappy theorems are easy to prove. Would it count the the number of successful experiments? We could always take a successful experiment and change it ever so slightly to get another success. I can’t think of a quantitative way to measure a field that isn’t open for enormous manipulation (which would only happen if people actually cared about the ratings agency’s rating).

Of course the same might be said about financial ratings. It begs the question, why are ratings agencies useful at all?

In finance we have lots of people buying very similar products with very similar contracts. Sometimes these are even sold on exchanges and carry with them the exact same risk profiles. In such a situation it makes sense to assign someone to look into the underlying risks and report back to the community on how risky a product is.

I would claim that the situation is very different in science or math. People enter a field for all sorts of reasons, with all sorts of goals and situations. String Theory is an extreme case where it could be argued that it got such spin that a whole generation of physics students got sucked into the field by sheer momentum. Perhaps it would have been nice to have a trusted institution whose job it was to calm people down and point out the reality, but I’m not sure it would have helped that much with all the excitement, especially if there had been a model which counted theorems and such. People would just have said the model had never seen something this exciting.

Then there’s the issue of trusting the modeler. Right now ratings agencies have a terrible reputation because they are paid by the people they rate products for, and have been known to sell good ratings. I’m hoping we can do better in the future, but it’s hard (but not impossible!) to imagine gathering enough experts in finance to do it well and to have the product be trusted by the community.

What is the analogy for scientific theories? The problem with rating science is that, because of the depth of most fields, only the experts in the field themselves understand it well enough to even talk about it. So that problem of getting an informed and impartial view on the worthiness of a theory is super super hard, assuming it’s possible.

Finally, I’m not sure what the ratings agency would be in charge of warning people about. Even the financial ratings agencies don’t agree- some of them measure default risk and others measure expected loss through default, which can be two really different things (for example if you think the U.S. will technically default but will end up paying their debts).

In science, I guess you could try to measure the risk that “the theory won’t end up being useful” but it’s not even clear how you’d decide that even after the fact. Maybe you could forecast the number of jobs in the field for graduating Ph.D. students, and that would be helpful to grad students but would also not be the best metric of success for the field.

I’m not saying we shouldn’t have people talk about fields and whether fields are failing, because that’s hugely important. But I don’t think there’s a quantitative model there to be created that would help the conversation. Let’s start an open forum, or a wiki, with the goal of talking about the health of various fields of scientific endeavor and have a bunch of good questions about the field and people can each add their two cents.