College ranking models
Last week Obama began to making threats regarding a new college ranking system and its connection to federal funding. Here’s an excerpt of what he was talking about, from this WSJ article:
The president called for rating colleges before the 2015 school year on measures such as affordability and graduation rates—”metrics like how much debt does the average student leave with, how easy is it to pay off, how many students graduate on time, how well do those graduates do in the workforce,” Mr. Obama told a crowd at the University at Buffalo, the first stop on a two-day bus tour.
Interesting! This means that Obama is wading directly into the field of modeling. He’s probably sick of the standard college ranking system, put out by US News & World Reports. I kind of don’t blame him, since that model is flawed and largely gamed. In fact, I made a case for open sourcing that model recently just so that people would look into it and lose faith in its magical properties.
So I’m with Obama, that model sucks, and it’s high time there are other competing models so that people have more than one thing to think about.
On the other hand, what Obama is focusing on seems narrow. Here’s what he supposedly wants to do with that model (again from the WSJ article):
Once a rating system is in place, Mr. Obama will ask Congress to allocate federal financial aid based on the scores by 2018. Students at top-performing colleges could receive larger federal grants and more affordable student loans. “It is time to stop subsidizing schools that are not producing good results,” he said.
His main goal seems to be “to make college more affordable”.
I’d like to make a few comments on this overall plan. The short version is that he’s suggesting something that will have strong, mostly negative effects, and that won’t solve his problem of college affordability.
Why strong negative effects?
What Obama seems to realize about the existing model is that it’s had side effects because of the way college administrators have gamed the model. Presumably, given that this new proposed model will be directly tied to federal funding, it will be high-impact and will thus be thoroughly gamed by administrators as well.
The first complaint, then, is that Obama didn’t address this inevitably gaming directly – and that doesn’t bode well about his ability to put into place a reasonable model.
But let’s not follow his lead. Let’s think about what kind of gaming will occur once such a model is in place. It’s not pretty.
Here are the attributes he’s planning to use for colleges. I’ve substituted reasonably numerical proxies for his descriptions above:
- Cost (less is better)
- Percentage of people able to pay off their loans within 10 years (more is better)
- Graduation rate (more is better)
- Percentage of people graduating within 4 years (more is better)
- Percentage of people who get high-paying jobs after graduating (more is better)
Cost
Nobody is going to argue against optimizing for lower cost. Unfortunately, what with the cultural assumption of the need for a college education, combined with the ignorance and naive optimism of young people, not to mention start-ups like Upstart that allow young people to enter indentured servitude, the pressure is upwards, not downwards.
The supply of money for college is large and growing, and the answer to rising tuition costs is not to supply more money. Colleges have already responded to the existence of federal loans, for example, by raising tuition in the amount of the loan. Ironically, much of the rise in tuition cost has gone to administrators, whose job it is to game the system for more money.
Which is to say, you can penalize certain colleges for being at the front of the pack in terms of price, but if the overall cost is rising constantly, you’re not doing much.
If you really wanted to make costs low, then fund state universities and make them really good, and make them basically free. That would actually make private colleges try to compete on cost.
Paying off loans quickly
Here’s where we get to the heart of the problem with Obama’s plan.
What are you going to do, as an administrator tasked with making sure you never lose federal funding under the new regime?
Are you going to give all the students fairer terms on their debt? Or are you going to select for students that are more likely to get finance jobs? I’m guessing the latter.
So much for liberal arts educations. So much for learning about art, philosophy, or for that matter anything that isn’t an easy entrance into the tech or finance sector. Only colleges that don’t care a whit about federal money will even have an art history department.
Graduation rate
Gaming the graduation rate is easy. Just lower your standards for degrees, duh.
How quickly people graduate
Again, a general lowering of standards is quick and easy.
How well graduates do in the workforce
Putting this into your model is toxic, and measures a given field directly in terms of market forces. Economics, Computer Science, and Business majors will be the kings of the hill. We might as well never produce writers, thinkers, or anything else creative again.
Note this pressure already exists today: many of our college presidents are becoming more and more corporate minded and less interested in education itself, mostly as a means to feed their endowments. As an example, I don’t need to look further than across my street to Barnard, where president Debora Spar somehow decided to celebrate Ina Drew as an example of success in front of a bunch of young Barnard students. I can’t help but think that was related to a hoped-for gift.
Obama needs to think this one through. Do we really want to build the college system in this country in the image of Wall Street and Silicon Valley? Do we want to intentionally skew the balance towards those industries even further?
Building a better college ranking model
The problem is that it’s actually really hard to model quality of education. The mathematical models that already exist and are being proposed are just pathetically bad at it, partly because college, ultimately, isn’t only about the facts you learn, or the job you get, or how quickly you get it. It’s actually a life experience which, in the best of cases, enlarges your world view, and gets you to strive for something you might not have known existed before going.
I’d suggest that, instead of building a new ranking system, we on the one hand identify truly fraudulent colleges (which really do exist) and on the other, invest heavily in state schools, giving them enough security so they can do without their army of expensive administrators.
mathbabe 
Unis are probably the best ever example of Goodhart’s law. http://en.wikipedia.org/wiki/Goodhart%27s_law
Thinking as a consumer (parent), the numbers I would like to see would be % of total income spent directly on teaching and research, and proportion of teaching academics on permanent contract – though these of course would be gamed just as quickly.
LikeLike
Thank you. I was hoping you would write about this. I read the article and was terrified about his solution. Now I can better articulate why.
LikeLike
“Do we really want to build the college system in this country in the image of Wall Street and Silicon Valley?” We don’t. They do.
LikeLike
Cathy, thanks so much for posting this!! My school posted a link to Obama’s speech and a NY Times article about the issue, and asked us all for our input. I was horrified when I first read the article, and posted a response against it talking precisely about the gaming-the-system idea. I am glad you mentioned the bias this plan also has to tech-heavy fields away from liberal arts, which isn’t something I considered before.
I also want to add the fact that this plan seems very prejudiced against students. Although fundamentally all the aspects they’re using to rank the schools are important for students, they’re not necessarily the most important qualifications for students in their college-decision process. I am very afraid that this will end up inhibiting students from attending schools that are good fits for them, better fits than the higher rated schools, purely because they didn’t receive a high enough rating so couldn’t provide enough financial aid.
This plan also seems to favor big-name schools with huge endowments anyway (like Harvard, MIT, etc.) whose graduates thrive in the work force because of institutional clout, among other things…
LikeLike
Great post!!!
Also consider the disadvantage to public universities that serve non traditional students, working students, single moms, single working moms, single working moms with multiple jobs to pay for child care, who typically take much longer to graduate, falling in and out of full time status, impairing the stats. Private universities sure won’t want them under this cost effectiveness reward proposal.
And it would be a shame since those public universities offer a a highly value-added education even without graduation, and their student pool is both vast and varied from which talent can develop.
LikeLike
Being a huge bubble, college ranking and funding is terribly important – thanks for taking that on.
Yet, in terms of content, I disagree with most of it.
First off, I would argue that computer science, which is actually building stuff from nothing, stuff that often betters people’s lives, is more creative than art history that, to a first approximation, describes what other people have created. I actually think this is a perfect example of the American system currently overrating some fields of knowledge despite relatively poor contribution to society and to the students. While job-market value is indeed somewhat extreme of a measure, it reflects the real writers and thinkers to probably do well anyway, while nudging colleges to produce less of the fake ones, that are poorly employable and only took that major because their college failed to restructure priorities for decades.
As for the goal of college as a life experience – this confusion is also an American culture thing, a byproduct of college years taking place away from home in the formative late-teens/early twenties. I believe 4 years on a fishing boat or in the Salvation Army would have a similar effect. The goal of college in the economic ecosystem is to provide education up to some standards, so that employers will know what to expect of graduates.
Also, if the goal is the experience, not the learning, then the administrators are the ones you need, not the professors.
Finally, you punt on measures, pushing for government higher education over the private system. However, positive experience of this administration in K-12 education suggests that setting standards is the key to pushing state schools to self-improve, which is more sustainable than continued federal investments.
LikeLike
Public higher ed and k-12 are apples and oranges. Public university students often attend college at great sacrifice to themselves quite aside from the debt. The underlying challenge in public k-12 is student motivation. We can lose sight of that amidst the current convenient demonization of teachers.
LikeLike
I think you are wildly undervaluing the kind of opportunities liberal arts educations give to students. The concern is less about specifically what the area is and more how it’s presented to the students — how the students can learn and grow and develop as individual thinkers. And one-way job-focused education doesn’t foster that. It fosters developing a set of skills, not a way of thinking, and sets of skills are much less easily applied in general situations, usually not accompanied with problem solving ideals and worse, very restrictive, so if the tech bubble bursts like the housing bubble or the dot com bubble, all the millions of carefully-pruned-techists will be thrown under a bus, so to speak. However, “job-focused” incentives will reduce training to basic skills instead of ways of thinking. (Think of this as the analogy to teaching-to-the-test in K-12, where students learn how to answer the specific problems they see, but don’t understand the principles behind what they’re doing, and still score well on the test with little applicable or generalizable knowledge.)
Also, I don’t think that college as a life experience is an American-specific thing. What is an American-specific thing is America’s focus on specific careers as being “better” or “worse” than others. We don’t value people who learn the skills to be plumbers, for example, but we’d be very sad if we didn’t have plumbers (or bakers or whatever else) in our society. Learning to be an independent thinker and run an art gallery is just as valuable to our society as programming in a Wall Street finance firm, and is much less likely to destroy other peoples’ lives.
LikeLike
Itsik,
I appreciate your desire to be provocative as usual.
First, of course all fields consider themselves creative. As a mathematician I would defend the creativity of mathematics, bien sur. Putting that aside, it’s creative within a certain framework that other fields take way further. The same for computer science.
Second, I am discussing American colleges, so it’s fair for me to take on the mentality of Americans.
Third, I don’t know what positive experiences you’re referring to in setting standards in K-12 education. I know about a lot of shitty experiences though. Maybe a reference or two?
Thanks,
Cathy
LikeLike
“Do we really want to build the college system in this country in the image of Wall Street and Silicon Valley? Do we want to intentionally skew the balance towards those industries even further?”
Are we talking about the same Obama? Of course he wants to do whatever his Wall Street employers tell him to do. Figure out how to take tax dollars away from the 99% leeches at community colleges and give them to the deserving students at Harvard and Princeton instead? Yes sir, we’ll get right on it!
LikeLike
I like Cathy’s idea of ferreting out the educational frauds. Start with for-profit schools that contain the name of a city, state, county or country. They should be required to add the words “for profit” to their names so there is no mistaking them for a state supported institution. The free MOOC movement may take care of the rest.
LikeLike
This is an excellent start, Cathy. It seems to me that universities today are competing in socially unproductive ways–luxury dorms, higher profile athletics, etc. It’s reminiscent of the competition that used to prevail in the airline industry in the old days of regulation. In that industry, pure price competition has brought its own problems–declining service, tendency toward tighter oligopoly, etc. It seems to me that many universities are financial disasters in waiting. Compare what’s begun to happen to law schools facing declining applicants, declining enrollments and declining cash flows. Layoffs and downsizing have already begun. I am a trustee of a seminary that has ditched the residential model in favor of distance learning. We are attracting highly motivated students who could not afford to leave their jobs/families to enroll in a 3-year degree program. Our student body participates in classes through MegaMeeting, they form closer relationships with fellow students and faculty members through Skype and e-mail, and they can proceed at their own speed. We have sold our campus and are moving into a brave new world. Best of all–our students likely will have much less debt when they graduate than they would have had back in the residential system.
LikeLike
Its not about education. Its about control.
Excerpt;
“A Primary Driver of Economic Inequality —
The 199 Jivey League Clique — Xtrevilism’s Mutation Ground…
Most people think of inbreeding as a low brow phenomenon practiced by hillbillies or backward mountain folks that occasionally results in easily recognized, deformed, and severely retarded individuals. That is the more popularized version of the word. But inbreeding can also be a far more insidious and less recognizable spectacle practiced by whole cultures and subsets of cultures.
Webster defines inbreeding as: the interbreeding of closely related individuals especially to preserve and fix desirable characters of and to eliminate unfavorable characters from a stock and: confinement to a narrow range or a local or limited field of choice.
The transnational 199 Jivey League Clique (the terms genesis is rooted in the now well deserved ‘pointy nose and pretentious’ stigma attached to the earlier national term ‘the big eight Ivy League’, and yes, it is meant to be very disparaging of those within it), is comprised of the individuals who matriculate in, and manage, the top 199 elite universities throughout the world. This very small subset group of self anointed elite human beings confines itself to a very narrow field of selection for students. That exclusionary selection process, and the criteria used to arrive at it, amounts to a virtual inbreeding; an incestuous relationship that forms the gene pools of the aberrant sociopathic mental disorders of Evilism and the more pernicious Xtrevilism that it is presently mutating into (see side bar). Most all of these self acclaimed prestigious schools that present themselves as institutions dedicated to liberty, opportunity, meritocracy, equality, and a just rule of law, are nothing of the sort. They are instead breeding grounds for pointy nose, my crap does not stink elitism, and exclusionary, secretive thinking. In the present aggregate they stand as one big monolithic global petri dish for the mutation from Evilism into the growth of the sociopathic disease of Xtrevilism. When one looks at the aggregate yearly income these inbred ‘birds of a feather flock together’ cretins extract in total yearly world income, and their total asset wealth, and worse, their grossly disproportionate influence over the misutilization of total world resources, one can easily see the dis-ingenuousness of their espoused mission statements of liberty, opportunity, meritocracy, equality and a just rule of law. For them yes! For the rest of us no!”
More here…
http://www.boxthefox.com/
Deception is the strongest political force on the planet.
LikeLike
“If you really wanted to make costs low, then fund state universities and make them really good, and make them basically free. That would actually make private colleges try to compete on cost.”
This claim is so brief, it can be either true or false (or both!) depending on what exactly do you mean by “fund”, “make” and “good”. Let me elaborate on this idea a bit.
1) If by “fund” you mean “increase state and federal subsidies” and by “good” you mean some kind of USNews-style ranking, this is most certainly false. The debate over K12 education long ago concluded that more money may lead to marginal improvements, but does not make schools good. And private universities like Stanford and USC with over 40K annual tuition are really not “competing on cost” with their similarly ranked counterparts Berkeley and UCLA whose tuition is around 15K.
2) What we have is a two tier system where state universities monopolize state funds, waste a lot of it on bureaucracy often to satisfy ever increasing state demands, and give students occasionally good but often subpar education (think class sizes). The state schools almost never directly compete for the same students from other states, as non-resident tuition is often outrageously high. If by “make” you mean adding one more layer of bureaucracy, this is a road to more waste. OTOH, top private universities have no incentives to cut tuition as they monopolize on access and “aspiration”. To put it plainly (as someone did I forgot where), if you want to end up in the Supreme Court, you don’t go to a state college – you go to either Harvard or Yale (see bios of current justices). In other words, the choice in both the cost and offerings is so stark, the two types of universities don’t really compete.
3) Rather than give money directly to colleges, states should make private all large state universities (but not community colleges), and issue grants in the number and amount they currently pay in subsidies, Pell grant style. Good poor and middle class state students would then be able to study in the former state universities for the same amount as before, but would have an option to take their money to rich private schools. This would force these state universities compete on class sizes and make private universities think of cutting the cost to attract some of these students. If this sounds like a “school choice” debate, it’s because it is, in a good way, since students don’t have to go to college in their neighborhood (and usually don’t anyway).
LikeLike