Home > data science, rant > Versus what?

Versus what?

March 7, 2012

I’m going to specialize in short, curmudgeony blog posts this week.

Today’s topic: you always need something to compare a new thing with. It’s this versus what?

If it’s a model, compare it to noise. That is, go ahead and test a model by scrambling the “y”s and see how well your model predicts randomness. It’s a really good and inexpensive way of seeing whether your model is better than noise, so go ahead and do it. There’s even a name for this but I forget what it is (update from reader: permutation testing).

If it’s a plan for a system or the world, compare it to the status quo. I’m so sick of people discarding good plans because they’re not perfect. If they’re better than what’s currently going on, then let’s go with that. Which brings me to my last example.

If it’s someone’s proposal (person A), compare it to other proposals (person B). I don’t think it’s fair for people (person C) to nix an idea unless they come up with a better one. If person C is consistently doing that, it’s a good bet that they have something to protect in the status quo situation, which brings us to the previous example.

Categories: data science, rant
  1. JSE
    March 7, 2012 at 10:28 am

    “I don’t think it’s fair for people (person C) to nix an idea unless they come up with a better one.” Sure, as long as “don’t do anything” counts as a “better idea.”


    • March 7, 2012 at 11:42 am

      In which case make the argument that “don’t do anything” counts as a better idea.


  2. March 7, 2012 at 11:34 am

    I think the word you are looking for is “permutation test” but maybe not.


    • March 7, 2012 at 11:41 am

      That’s the one! And I should have said, do it a bunch of times and see the range of possibilities. Thanks!


  3. March 7, 2012 at 11:54 am

    The best test I’ve seen for derivative models is the monkey throwing darts, which easily shows the foolish Nobelist flaw of using bell-curve standard deviations for market volatility.


  4. jj
    March 7, 2012 at 10:28 pm

    I hate to say it, but doesn’t your VAM Sucks post count as nixing an idea without proposing something better?

    Don’t get me wrong; VAM does suck. Enormously. But so does the *current* system, which has been a steady drive downhill for teacher evaluation. A regression to pre-NCLB would be an instant improvement of gigantic proportions. If that’s the proposal for the “something better”, I’m all over it.

    But how do you make that argument without appearing to be feverishly clinging to a cushy, permanent job, in the eyes of the unwashed masses?


    • March 8, 2012 at 5:53 am

      Whenever I talk to someone about this I ask them if they think the testing has done its job of keeping good teachers and getting rid of bad teachers. Specifically, if you were a good teacher in the system, would you be more likely to stay in the system given the current testing environment or given no testing at all?

      It’s tough because some of the best teachers are really truly devoted to teaching and to their kids. But I still think most people want to feel like they are being respected and appreciated when they work hard, and people who have talents and skills also have options. So I think most good teachers would get the hell out of this current system and I don’t blame them one bit.

      We have to stop doing this right now, give more power to the principals to evaluate their teachers, and keep an eye on the principals too. We need to establish trust. It’s hard to even imagine right now.


    • CitizensArrest
      January 31, 2013 at 2:31 pm

      Here’s whats better, it’s called PAR or Peer Assistance and Review. Problem for the politicians is that it’s not math based (no mumbo jumbo to hide behind) and doesn’t produce profits for their campaign contributors. PAR makes use of the most powerful data processor on the planet, the human brain. It succeeds by accessing the huge data base of institutional knowledge, contained in other brains and literature, that is growing, evolving and improving.


  1. No trackbacks yet.
Comments are closed.
%d bloggers like this: