Interactive scoring models: why hasn’t this happened yet?
My friend Suresh just reminded me about this article written a couple of years ago by Malcolm Gladwell and published in the New Yorker.
It concerns various scoring models that claim to be both comprehensive (which means it covers the whole thing, not just one aspect of the thing) and heterogeneous (which means it is broad enough to cover all things in a category), say for cars or for colleges.
Weird things happen when you try to do this, like not caring much about price or exterior detailing for sports cars.
Two things. First, this stuff is actually really hard to do well. I like how Gladwell addresses this issue:
At no point, however, do the college guides acknowledge the extraordinary difficulty of the task they have set themselves.
Second of all, I think the issue of combining heterogeneity and comprehensiveness is addressable, but it has to be addressed interactively.
Specifically, what if instead of a single fixed score, there was a place where a given car-buyer or college-seeker could go to fill out a form of preferences? For each defined and rated aspect, the user would fill answer a question about how much they cared about that aspect. They’d assign a weight to each aspect. A given question would look something like this:
For colleges, some people care a lot about whether their college has a ton of alumni giving, other people care more about whether the surrounding town is urban or rural. Let’s let people create their own scoring system. It’s technically easy.
I’ve suggested this before when I talked about rating math articles on various dimensions (hard, interesting, technical, well-written) and then letting people come and search based on weighting those dimensions and ranking. But honestly we can start even dumber, with car ratings and college ratings.