Home > data science, math education, open source tools, statistics > The Value Added Teacher Model Sucks

The Value Added Teacher Model Sucks

March 6, 2012

Today I want you to read this post (hat tip Jordan Ellenberg) written by Gary Rubinstein, which is the post I would have written if I’d had time and had known that they released the actual Value-added Model scores to the public in machine readable format here.

If you’re a total lazy-ass and can’t get yourself to click on that link, here’s a sound bite takeaway: a scatter plot of scores for the same teacher, in the same year, teaching the same subject to kids in different grades. So, for example, a teacher might teach math to 6th graders and to 7th graders and get two different scores; how different are those scores? Here’s how different:

Yeah, so basically random. In fact a correlation of 24%. This is an embarrassment, people, and we cannot let this be how we decide whether a teacher gets tenure or how shamed a person gets in a newspaper article.

Just imagine if you got publicly humiliated by a model with that kind of noise which was purportedly evaluating your work, which you had no view into and thus you couldn’t argue against.

I’d love to get a meeting with Bloomberg and show him this scatter plot. I might also ask him why, if his administration is indeed so excited about “transparency,” do they release the scores but not the model itself, and why they refuse to release police reports at all.

  1. March 6, 2012 at 1:28 pm

    What on earth is going on? These plots don’t just look bad, but crazy, which is not what I expected.

    Whoever approved the choice of this model owes the public either an explanation of why it is not as bad as this makes it seem, or an explanation of how such an apparently random system got chosen and why we can trust that future decisions will be made more responsibly.

  2. March 6, 2012 at 2:00 pm

    My reactions, in order: (1) This is insane. (2) My undergraduate students should know better than to trust a model which produces results like this. (3) Now I know what the kids will be analyzing for their next exam. (4) This is insane.

  3. K.J.
    March 6, 2012 at 3:29 pm

    I’ve seen the decisions that get made in education. They are made by politicians, not statisticians or economists or mathematicians. The decisions are made by people who are popular, not people who are smart.

    • March 6, 2012 at 3:30 pm

      Maybe releasing this data to the public was the stupidest move the politicians have ever made. Let’s hope we can make it so.

      • JSE
        March 6, 2012 at 5:21 pm

        Politicians didn’t release this on purpose. They KNOW you’re not supposed to pay any attention to these numbers until you’ve, at the very least, averaged over several years of consistent measurement (and even that may be enough.) But the data are public records, subject to Freedom of Information Act requests, and they had no choice.

        • Darren
          March 6, 2012 at 7:55 pm

          If they KNOW they’re not supposed to pay any attention to these numbers etc.–and they not only pay attention to them but use them to guide decisions, both about educational practices and about hiring/firing–why the hell are they?

          And when did teachers become everyone’s favorite scapegoat?

        • Cynicism
          March 6, 2012 at 10:19 pm

          My guesses:

          1) Because in politics it’s better to have bad ideas than no ideas (Gingrich has been around for how long now?)

          2) Tough to say, but it probably started because the unions of most other professions have been weakened to near-irrelevancy. Now it’s just teachers unions and police unions.

        • dilbert dogbert
          March 7, 2012 at 6:59 pm

          Simple when you look at the cash flow in public education and what Bain Capital could do with this flow. CEO bonuses out the Wahzoooooooo!!!

  4. rtg
  5. unreasonableresponse
    March 7, 2012 at 7:46 am

    Err, I thought that just about everyone knew that there’s no way to tell luck (or call it “external factors not under direct control”) from skill (call it “what you can control”).

    From that perspective, 24% correlation is not that hugely bad (i.e. it’s about to be expected) – althouh basing any perfomance review on this is clearly stupid (basing ANY performance review on two datapoints – or even three is stupid).

    IIRC, CEO and their company performance has about 30% correlation (which translates to about 20% value add compared to a randomly deciding monkey. Clearly overpaid, but still worth some money). So in theory you could use this data and benchmark against CEO performance using the 30% correlation ;)

    I’d be interested whether longer data series are still low-correlation or not.

  6. March 7, 2012 at 8:55 am

    Here’s another article for your readers to look at: http://www.nytimes.com/2012/03/05/nyregion/in-brooklyn-hard-working-teachers-sabotaged-when-student-test-scores-slip.html
    This article describes one of the major mathematical flaws with value added test calculations: in high performing schools small declines in student performance result in large declines in “value added”… Perhaps your readers with a background in statistics could try to explain that to Bill Gates, one of the business leaders who is underwriting this move to value added assessments….

    • unreasonableresponse
      March 7, 2012 at 11:46 am

      Bill should spend a bit of his money to hire a statistican, who then could explain to him that relatively large proportion of small schools at the top does not automaticaly mean they are better than large schools. Or, if he wants to do it on the cheap, someone should point him to http://en.wikipedia.org/wiki/Clustering_illusion

  7. March 7, 2012 at 9:37 am

    Since the purpose of the model is humiliation, for the policy goal of destroying the public school system, it seems to be achieving its goal. Pavlov taught us about the random infliction of pain, right?

  8. nonclassical
    March 7, 2012 at 1:00 pm

    Lambert is accurate…but this “infliction of pain” is not “random”-read Naomi Klein’s, “The Shock Doctrine-rise of disaster capitalism”…or ask Poly-Sci people who know 70’s-80’s
    South-Central American history of U.S. imperialism. Or, read William Blum’s, “Killing Hope”,
    or Perkins’, “Confessions Of An Economic Hit Man”, for “privatization” schemes…

  9. charles 2
    March 7, 2012 at 10:20 pm

    As far as “datasource used in a statistically inept way to push forward preposterous models and conclusions”, the Bloomberg workstation ranks probably #1 . So what you are telling us is not astonishing me the least !

  10. joemac53
    March 9, 2012 at 6:51 pm

    I retired two years ago after 35 years as a public high school math/science teacher. I came from a relatively weak union. If it was eliminated there would be a mass exodus. Just before I left I was given a taste of the value-added model concerning the “trajectory” of students. I think I got out just in time.
    Just as NCLB guaranteed that 90% of schools would be “failing” in less than 10 years, the “trajectory” model would label 90% of teachers “ineffective” within a very short time. Yuck! Of course, we math snobs can’t stand it when some administrator who couldn’t add without a calculator is trying to explain how this all works.

  11. myname
    September 13, 2012 at 8:40 am

    the correlation you present suffers from a problem: you dont compare the same group of students nor the same content. so why expect a high correlation? teacher impact should count and it does but it´s not everything. this is saying it accounts for 25% of your variation, the other 2 sources i mention account for 75%.

    • September 13, 2012 at 9:30 am

      Sorry, no. The point of these scores is that they are supposed to be measuring the value of the teacher. The point of the model is to take into account differences in students. And moreover, the teachers are evaluated on the basis of these numbers, not on the basis of the students.

  1. March 7, 2012 at 6:57 am
  2. March 7, 2012 at 8:52 am
  3. March 9, 2012 at 4:45 pm
  4. August 2, 2012 at 7:00 am
  5. September 14, 2012 at 8:54 am
  6. September 16, 2012 at 11:29 pm
  7. September 25, 2012 at 10:01 am
  8. January 29, 2013 at 9:58 am
  9. February 6, 2013 at 4:55 pm
  10. March 31, 2013 at 8:12 am
  11. July 19, 2013 at 10:13 am
  12. January 29, 2014 at 7:12 am
Comments are closed.

Get every new post delivered to your Inbox.

Join 986 other followers

%d bloggers like this: