Open Source Ratings Model?
August 11, 2011
A couple of days ago I got this comment from a reader, which got me super excited.
His proposal is that we could start an open source ratings model to compete with S&P and Moody’s and Fitch ratings. I have made a few relevant lists which I want to share with you to address this idea.
Reasons to have an open source ratings model:
- The current rating agencies have a reputation for bad modeling; in particular, their models, upon examination, often have extremely unrealistic underlying assumptions. This could be rooted out and modified if a community of modelers and traders did their honest best to realistically model default.
- The current ratings agencies also have enormous power, as exemplified in the past few days of crazy volatile trading after S&P downgraded the debt of the U.S. (although the European debt problems are just as much to blame for that I believe). An alternative credit model, if it was well-known and trusted, would dilute their power.
- Although the rating agency shared descriptions of their models with their clients, they weren’t in fact open-source, and indeed the level of exchange probably served only to allow the clients to game the models. One of the goals of an open-source ratings model would be to avoid easy gaming.
- Just to show you how not open source S&P is currently, check out this article where they argue that they shouldn’t have to admit their mistakes. When you combine the power they wield, their reputation for sloppy reasoning, and their insistence on being protected from their mistakes, it is a pretty idiotic system.
- The ratings agencies also have a virtual lock on their industry- it is in fact incredibly difficult to open a new ratings agency, as I know from my experience at Riskmetrics, where we looked into doing so. By starting an open source ratings model, we can (hopefully) avoid issues like permits or whatever the problem was by not charging money and just listing free opinions.
Obstructions to starting an open source ratings model:
- It’s a lot of work, and we would need to set it up in some kind of wiki way so people could contribute to it. In fact it would have to me more Linux style, where some person or people maintain the model and the suggestions. Again, lots of work.
- Data! A good model requires lots of good data. Altman’s Z-score default model, which friends of mine worked on with him at Riskmetrics and then MSCI, could be the basis of an open source model, since it is being published. But the data that trains the model isn’t altogether publicly available. I’m working on this, would love to hear readers’ comments.
What is an open source model?
- The model itself is written in an open source language such as python or R and is publicly available for download.
- The data is also publicly available, and together with the above, this means people can download the data and model and change the parameters of the model to test for robustness- they can also change or tweak the model themselves.
- There is good documentation of the model describing how it was created.
- There is an account kept of how often different models are tried on the in-sample data. This prevents a kind of data fitting that people generally don’t think about enough, namely trying so many different models on one data set that eventually some model will look really good.
Would love comments on any of my lists.
Categories: data science, finance, news, open source tools
mathbabe 
Everything sounds great, but in my opinion, you’ve left the biggest, most obvious question unanswered: how do you compensate for vandals and those hoping to temporarily game the system? Who gets to wields administrative power, and how can we be sure they’re not being paid by some group or another?
LikeLike
How about a complete open-source investment house? The above question would just be one of the components it relies upon. It would be interesting to see how well, e.g., an open-source-managed mutual fund would perform. (Also, the prior commenter’s questions are clearly important IRL.)
LikeLike
Getting the data you need could be expensive in the end. Also, is volume + storage going to be an issue? There could be some independent funding to offset some of these costs – after all there must be someone out there with a little bit of money who would like to see these things work. Are you thinking of a self-editing community to make sure that contributions are sound? Validating code or the math involved can be tricky on top of being time consuming. You are going to need to have a submission format that elucidates the entries.
LikeLike
“…an open source language such as python or R…”
Is R less complex than C? Sorry, couldn’t help myself.
LikeLike
Wikipedia succeeded in part because “anybody” could add a little bit to the encyclopedia. Could an open-source ratings model be so accessible? Say my grandmother notices on vacation that there seem to be an awful lot of empty houses in Galicia. Could that information be added to the system?
LikeLike