## An AMS panel to examine public math models?

On Saturday I gave a talk at the AGNES conference to a room full of algebraic geometers. After introducing myself and putting some context around my talk, I focused on a few models:

- VaR,
- VAM,
- Credit scoring,
- E-scores (online version of credit scores), and
- The h-score model (I threw this in for the math people and because it’s an egregious example of a gameable model).

I wanted to formalize the important and salient properties of a model, and I came up with this list:

- Name – note the name often gives off a whiff of political manipulation by itself
- Underlying model – regression? decision tree?
- Underlying assumptions – normal distribution of market returns?
- Input/output – dirty data?
- Purported/political goal – how is it actually used vs. how its advocates claim they’ll use it?
- Evaluation method – every model should come with one. Not every model does. A red flag.
- Gaming potential – how does being modeled cause people to act differently?
- Reach – how universal and impactful is the model and its gaming?

In the case of VAM, *it doesn’t have an evaluation method*. There’s been no way for teachers to know if the model that they get scored on every year is doing a good job, even as it’s become more and more important in tenure decisions (the Chicago strike was largely related to this issue, as I posted here).

Here was my plea to the mathematical audience: this is being done in the name of mathematics. The authority that math is given by our culture, which is enormous and possibly not deserved, is being manipulated by people with vested interests.

So when the objects of modeling, the people and the teachers who get these scores, ask how those scores were derived, they’re often told “it’s math and you wouldn’t understand it.”

That’s outrageous, and mathematicians shouldn’t stand for it. We have to get more involved, as a community, with how mathematics is wielded on the population.

On the other hand, I wouldn’t want mathematicians as a group to get co-opted by these special interest groups either and become shills for the industry. We don’t want to become economists, paid by this campaign or that to write papers in favor of their political goals.

To this end, someone in the audience suggested the AMS might want to publish a book of ethics for mathematicians akin to the ethical guidelines that are published for the society of pyschologists and lawyers. His idea is that it would be case-study based, which seems pretty standard. I want to give this some more thought.

We want to make ourselves available to understand high impact, public facing models to ensure they are sound mathematically, have reasonable and transparent evaluation methods, and are very high quality in terms of proven accuracy and understandability if they are used on people in high stakes situations like tenure.

One suggestion someone in the audience came up with is to have a mathematician “mechanical turk” service where people could send questions to a group of faceless mathematicians. Although I think it’s an intriguing idea, I’m not sure it would work here. The point is to investigate so-called math models that people would rather no mathematician laid their eyes on, whereas mechanical turks only answer questions someone else comes up with.

In other words, there’s a reason nobody has asked the opinion of the mathematical community on VAM. They are using the authority of mathematics without permission.

Instead, I think the math community should form something like a panel, maybe housed inside the American Mathematical Society (AMS), that trolls for models with the following characteristics:

- high impact – people care about these scores for whatever reason
- large reach – city-wide or national
- claiming to be mathematical – so the opinion of the mathematical community matters, or should,

After finding such a model, the panel should publish a thoughtful, third-party analysis of its underlying mathematical soundness. Even just one per year would have a meaningful effect if the models were chosen well.

As I said to someone in the audience (which was amazingly receptive and open to my message), it really wouldn’t take very long for a mathematician to understand these models well enough to have an opinion on them, especially if you compare it to how long it would take a policy maker to understand the math. Maybe a week, with the guidance of someone who is an expert in modeling.

So in other words, being a member of such a “public math models” panel could be seen as a community service job akin to being an editor for a journal: real work but not something that takes over your life.

Now’s the time to do this, considering the explosion of models on everything in sight, and I believe mathematicians are the right people to take it on, considering they know how to admit they’re wrong.

Tell me what you think.

I think you’re my favorite mathematician ever.

LikeLike

I’m really glad the talk went well… I miss Brown!

I volunteer for the panel, if that’s any use.

LikeLike

Why isn’t this already being done in domain specific academic journals or some of the applied statistics/modeling journals? On the surface, it seems like there are tons of incentives for Finance profs to publish critiques of VaR, credit scoring, e-scoring and the like. A nice critique of VAM would be encouraged and appreciated by many types of academic departments and journals. When new public policy initiatives are proposed, academics typically fall all over themselves to analyze the efforts.

If there is a problem developing and publicizing high-impact critiques of these models, why is that? Lack of modeling skill in the domain specific area? Lack of interest/incentives among modeling generalists? Lack of transparency? Lack of interest from high-impact publishers?

Perhaps these critiques exist but just aren’t having much impact. If the vast majority of people are convinced that there are way too many bad teachers out there, they’ll support a deeply flawed VAM to rout them out until something better comes along that promises to address what they perceive to be a problem. Even the best critique of VAM or other models won’t have as much impact as a marginally better implementable solution that is seen to either address the problem or redefine it.

LikeLike

If there is no falsifiable evaluation (in the Popper sense) of a model, there can be no claim to a scientific method being involved! Where’s the professional outrage – outside of our fave mathbabe!

LikeLike

I think the AMS (unlike individual mathematicians and even organized groups of mathematicians) should be very careful in getting involved in this. The problem is exactly that the controversies over the models are increasingly political in different ways than the ways the AMS would like to be political. Would you want the AMS to have taken sides over the strike in Chicago? The members stand behind the AMS when it lobbies for its core political goals like NSF funding or improving the high-school curriculum, even if we don’t all agree on the particulars. But if the AMS started taking positions on issues that are controversial and technical at the same time like whether certain data models used by public entities are good or bad that would be something completely different. See what happened when the APS tried to claim an official opinion on Global Warming, despite most of its members not having professional competence to judge the science.

I would propose instead forming a separate organization, something like “mathematicians in the public interest” with the stated goal of having official positions on such models. Yes, rather than arrogating the political capital of the AMS you’d have to create your own by having a good track record, but I don’t think that’s bad.

LikeLike

Great idea!

LikeLike

There are an infinite number of possible models.

Do not those models group by type? If so, it is possible to characterize ‘meaningful’ and ‘meaningless’ models.

For instance, there is a categorization of elephant personalities into 4 types in the science news just now. Very simple model, and we can say a lot about what kinds of data and kinds and sequences of tests that are needed to say that categorization is meaningful.

Then every instantiated model can more easily be assessed for meaningfulness.

And this is just the beginning : for anything approximating ‘usefulness’ vs the Absolute Truth we would like, we need Full-dimension consideration, with points for and against from every conceivable POV. That takes a long time and a lot of energy, but there is no substitute, and we hope to be able to generalize, to show that many classes of data and models do not produce the desired knowledge.

This will, I believe, produce results that good Progressives hate, so is never done.

LikeLike