Home > modeling, statistics > An attempt to FOIL request the source code of the Value-added model

An attempt to FOIL request the source code of the Value-added model

March 7, 2014

Last November I wrote to the Department of Education to make a FOIL request for the source code for the teacher value-added model (VAM).

Motivation

To explain why I’d want something like this, I think the VAM model sucks and I’d like to explore the actual source code directly. The white paper I got my hands on is cryptically written (take a look!) and doesn’t explain what the actual sensitivity to inputs are, for example. The best way to get at that is the source code.

Plus, since the New York Times and other news outlets published teacher’s VAM scores after a long battle and a FOIA request (see details about this here), I figured it’s only fair to also publicly release the actual black box which determines those scores.

Indeed without knowledge of what the model consists of, the VAM scoring regime is little more than a secret set of rules, with tremendous power over teachers and the teacher union, and also incorporates outrageous public shaming as described above.

I think teachers deserve better, and I want to illustrate the weaknesses of the model directly on an open models platform.

The FOIL request

Here’s the email I sent to foil@schools.nyc.gov on 11/22/13:

Dear Records Access Officer for the NYC DOE,

I’m looking to get a copy of the source code for the most recent value-added teacher model through a FOIA request. There are various publicly available descriptions of such models, for example here, but I’d like the actual underlying code.

Please tell me if I’ve written to the correct person for this FOIA request, thank you very much.

Best,
Cathy O’Neil

Since my FOIL request

In response to my request, on 12/3/13, 1/6/14, and 2/4/14 I got letters saying stuff was taking a long time since my request was so complicated. Then yesterday I got the following response:
Screen Shot 2014-03-07 at 8.49.57 AM

If you follow the link you’ll get another white paper, this time from 2012-2013, which is exactly what I said I didn’t want in my original request.

I wrote back, not that it’s likely to work, and after reminding them of the text of my original request I added the following:


What you sent me is the newer version of the publicly available description of the model, very much like my link above. I specifically asked for the underlying code. That would be in a programming language like python or C++ or java.

Can you to come back to me with the actual code? Or who should I ask?

Thanks very much,
Cathy

It strikes me as strange that it took them more than 3 months to send me a link to a white paper instead of the source code as I requested. Plus I’m not sure what they mean by “SED” but I’m guessing it means these guys, but I’m not sure of exactly who to send a new FOIL request.

Am I getting the runaround? Any suggestions?

Categories: modeling, statistics
  1. March 7, 2014 at 9:24 am

    Start by blasting out your request to all the email addresses on the following URL:

    http://www.highered.nysed.gov/contact.html

  2. Bill
    March 7, 2014 at 9:55 am

    This is K-12 not college/university, so you probably want:
    http://www.nysed.gov/

    whose contact page is here:
    http://usny.nysed.gov/contact.html

    That page has a link about FOIA requests:
    http://www.oms.nysed.gov/foil/

  3. March 7, 2014 at 10:07 am

    For the current model, State Ed (SED) doesn’t have the code; AIR has it. I wouldn’t be surprised if SED doesn’t even have access to the code. The contract is probably just for the results. Similarly for NYC’s old model – I don’t think NYC ever had the code; VARC (http://varc.wceruw.org/) had it. I’m not sure a FOIL request will have the power to get the code from these non-government sources; will it?

  4. Bill
    March 7, 2014 at 10:17 am

    You could try going via the “Value Added Research Center”

    http://varc.wceruw.org/

    which was apparently involved in writting the reports. As a part of the UWisc system
    they are probably be covered by Wisconsin FOIA rules. Unfortunately, a cursory review of Wisconsin FOIA rules seems to indicate that computer programs are NOT covered.

    You might want to talk to the people at PublicResource.Org who have experience
    trying to get information out of government agencies and then publishing it:
    https://public.resource.org/

  5. Min
    March 7, 2014 at 10:34 am

    It sounds like you sent your first request to someone who had little idea of what source code is. Also, as aaronshumacher points out, the state might not even have the source code. (Although it should belong to the state, either as a work for hire or by contract.)

    Years ago Byte Magazine published a letter to the editor from a bank which had a problem. They had hired someone to write software for them, as had other banks in their town. But the software did not work, for any of the banks. At this point the programmer told the banks that he would fix the software for an additional fee. The other banks took him up on the offer, but this bank refused, chafing at the blackmail and not trusting the guy, anyway. They tried to hire another programmer, but he informed them that he could not fix the software without the source code. The letter ended with this plaintive plea: “What is the source code and how do we break it?”

  6. March 7, 2014 at 10:38 am

    You are most certainly getting the “runaround.” I live and teach in Louisiana and hold a PhD in stats/research methods. One of our state ed board members requested that I have access to VAM. Former TFAer-gone-state-super John White (who cut his teeth under Joel Klein’s direction) will not allow it, of course, because he and his DOE minions play with those formulas to yield whatever results they wish. Access to their code is “inner sanctum” and certainly not allowable by someone who could possibly see right through their game.

    They will give you the runaround as infinitum. The algorithms are likely multiple sets of formulas adjusted as The Powers That Be see fit– and likely hundreds of printed pages given all of the teachers and subjects and “special adjustments” these folks have made in their playing god with the numbers.

    I think that the best course is to try to find someone connected on the inside of DOE and who might be willing to pass to you some VAM-abuse info.

  7. Ted Coltman
    March 7, 2014 at 10:44 am

    You probably are getting the run-around, but I don’t think it’s highered.nysed.gov that you want to focus on in the state Department of Education. Teacher and principal evaluation seems to be under the rubric of the Race to the Top program, for which you can find some leads at http://usny.nysed.gov/rttt/teachers-leaders/home.html . I’d guess that you want to pursue queries at one or more of the contact points listed on http://usny.nysed.gov/rttt/contact.html — probably starting with people working onTeacher and Principal Annual Professional Performance Review (APPR) at educatoreval@mail.nysed.gov .

  8. March 7, 2014 at 11:17 am

    Cathy, Copy your request to every media company and every public official you can think of
    and put the cc’s on your original request. Generally they all contact each other to cover their asses and someone is forced to address the issue. publicity is refreshing.

  9. dan
    March 7, 2014 at 5:54 pm

    this is fascinating. but rewrite the headline of your blogpost to appeal to public school advocates, nyc transparency folks, etc. if i share this on facebook as is, nobody except people who are aware of the VAM would click through.

  10. That Guy
    March 8, 2014 at 8:35 am

    Joanna Cannon is the Executive Director of Teacher and Principal Performance New York City Department of Education. In this role, she is one-down from the Chancellor and directly responsible for VAM as well as managing the relationship with the Wisconsin guys. Reach out to her directly as she will definitely have the answers, even if she has to kick your query out to the media relations group. At least she will know what “source code” is.

  11. March 8, 2014 at 8:46 am

    Well, the NYT did publish the formula:

    http://www.nytimes.com/2011/03/07/education/07winerip.html

    Filed under the tag Mathematical Intimidation.

  12. Thomas
    March 8, 2014 at 8:48 am

    One of the biggest problems with the design of the NYC VAM is that it’s comparing results from the student’s test at the end of the prior school year with this year’s test results. This is patently unfair to teachers. A better design would compare beginning of the school year test results with end of the same school year results. This would measure score changes as a function of the teacher. Of course, they haven’t figured out how to do VAM for middle school and up yet since there are multiple teachers in those grades.

    But admittedly this amounts to an apology for an inherently unfair process…

  13. Stuart Buck
    March 8, 2014 at 11:40 am

    The problem may be asking for “source code” in C++ or Java or whatever. In all likelihood, the value-added model was run in Stata or a similar program, and whoever did it wouldn’t have any idea what the “source code” actually is.

  14. March 9, 2014 at 11:36 pm

    to #13, it might also be Excel macros, the horror, the horror..

    Here in CO the Race to the Top leaves the construction of the VAM model as an exercise for the reader. That is, administration in each school district, already overwhelmed by large and continuing budget cuts, now has to draw up a set of VAM metrics and prepare to defend it. The genius of this is that the failure of VAM to provide meaningful results can now be blamed on local school districts. Brilliant consultants from Pearson will doubtless be hired with public education funds to fix the VAM. In ten years the obvious failure of any of this to produce 100% ‘college-ready’ kids (like the 100% effective schools required by NCLB) will bring forth another monster. Or perhaps we will regain our sanity ? may I live so long.

    http://www.cde.state.co.us/educatoreffectiveness/studentgrowthguide

    The ‘student growth guide’ is worth 50% of a teacher’s evaluation.
    They present an outline of an Excel spreadsheet, see the link for Measures of Student Learning Tool – download and marvel at its substance-free blithering.

    So, it’s even worse here than in NY, hard as it may be to image that.

  15. Guest2
    March 28, 2014 at 9:16 am

    Here is a VAM lawsuit in Florida, just to get access to the data used. The newspaper won.

    MORRIS PUB. GROUP, LLC v. State, 13 So. 3d 120 – Fla: Dist. Court of Appeals, 1st Dist. 2009

  1. March 8, 2014 at 6:29 am
  2. March 15, 2014 at 3:10 pm
Comments are closed.
Follow

Get every new post delivered to your Inbox.

Join 986 other followers

%d bloggers like this: