An attempt to FOIL request the source code of the Value-added model
Last November I wrote to the Department of Education to make a FOIL request for the source code for the teacher value-added model (VAM).
Motivation
To explain why I’d want something like this, I think the VAM model sucks and I’d like to explore the actual source code directly. The white paper I got my hands on is cryptically written (take a look!) and doesn’t explain what the actual sensitivity to inputs are, for example. The best way to get at that is the source code.
Plus, since the New York Times and other news outlets published teacher’s VAM scores after a long battle and a FOIA request (see details about this here), I figured it’s only fair to also publicly release the actual black box which determines those scores.
Indeed without knowledge of what the model consists of, the VAM scoring regime is little more than a secret set of rules, with tremendous power over teachers and the teacher union, and also incorporates outrageous public shaming as described above.
I think teachers deserve better, and I want to illustrate the weaknesses of the model directly on an open models platform.
The FOIL request
Here’s the email I sent to foil@schools.nyc.gov on 11/22/13:
Dear Records Access Officer for the NYC DOE,
I’m looking to get a copy of the source code for the most recent value-added teacher model through a FOIA request. There are various publicly available descriptions of such models, for example here, but I’d like the actual underlying code.
Please tell me if I’ve written to the correct person for this FOIA request, thank you very much.
Best,
Cathy O’Neil
Since my FOIL request
In response to my request, on 12/3/13, 1/6/14, and 2/4/14 I got letters saying stuff was taking a long time since my request was so complicated. Then yesterday I got the following response:
If you follow the link you’ll get another white paper, this time from 2012-2013, which is exactly what I said I didn’t want in my original request.
I wrote back, not that it’s likely to work, and after reminding them of the text of my original request I added the following:
…
What you sent me is the newer version of the publicly available description of the model, very much like my link above. I specifically asked for the underlying code. That would be in a programming language like python or C++ or java.Can you to come back to me with the actual code? Or who should I ask?
Thanks very much,
Cathy
It strikes me as strange that it took them more than 3 months to send me a link to a white paper instead of the source code as I requested. Plus I’m not sure what they mean by “SED” but I’m guessing it means these guys, but I’m not sure of exactly who to send a new FOIL request.
Am I getting the runaround? Any suggestions?
mathbabe 
Start by blasting out your request to all the email addresses on the following URL:
http://www.highered.nysed.gov/contact.html
LikeLike
This is K-12 not college/university, so you probably want:
http://www.nysed.gov/
whose contact page is here:
http://usny.nysed.gov/contact.html
That page has a link about FOIA requests:
http://www.oms.nysed.gov/foil/
LikeLike
For the current model, State Ed (SED) doesn’t have the code; AIR has it. I wouldn’t be surprised if SED doesn’t even have access to the code. The contract is probably just for the results. Similarly for NYC’s old model – I don’t think NYC ever had the code; VARC (http://varc.wceruw.org/) had it. I’m not sure a FOIL request will have the power to get the code from these non-government sources; will it?
LikeLike
You could try going via the “Value Added Research Center”
http://varc.wceruw.org/
which was apparently involved in writting the reports. As a part of the UWisc system
they are probably be covered by Wisconsin FOIA rules. Unfortunately, a cursory review of Wisconsin FOIA rules seems to indicate that computer programs are NOT covered.
You might want to talk to the people at PublicResource.Org who have experience
trying to get information out of government agencies and then publishing it:
https://public.resource.org/
LikeLike
It sounds like you sent your first request to someone who had little idea of what source code is. Also, as aaronshumacher points out, the state might not even have the source code. (Although it should belong to the state, either as a work for hire or by contract.)
Years ago Byte Magazine published a letter to the editor from a bank which had a problem. They had hired someone to write software for them, as had other banks in their town. But the software did not work, for any of the banks. At this point the programmer told the banks that he would fix the software for an additional fee. The other banks took him up on the offer, but this bank refused, chafing at the blackmail and not trusting the guy, anyway. They tried to hire another programmer, but he informed them that he could not fix the software without the source code. The letter ended with this plaintive plea: “What is the source code and how do we break it?”
LikeLike
You are most certainly getting the “runaround.” I live and teach in Louisiana and hold a PhD in stats/research methods. One of our state ed board members requested that I have access to VAM. Former TFAer-gone-state-super John White (who cut his teeth under Joel Klein’s direction) will not allow it, of course, because he and his DOE minions play with those formulas to yield whatever results they wish. Access to their code is “inner sanctum” and certainly not allowable by someone who could possibly see right through their game.
They will give you the runaround as infinitum. The algorithms are likely multiple sets of formulas adjusted as The Powers That Be see fit– and likely hundreds of printed pages given all of the teachers and subjects and “special adjustments” these folks have made in their playing god with the numbers.
I think that the best course is to try to find someone connected on the inside of DOE and who might be willing to pass to you some VAM-abuse info.
LikeLike
You probably are getting the run-around, but I don’t think it’s highered.nysed.gov that you want to focus on in the state Department of Education. Teacher and principal evaluation seems to be under the rubric of the Race to the Top program, for which you can find some leads at http://usny.nysed.gov/rttt/teachers-leaders/home.html . I’d guess that you want to pursue queries at one or more of the contact points listed on http://usny.nysed.gov/rttt/contact.html — probably starting with people working onTeacher and Principal Annual Professional Performance Review (APPR) at educatoreval@mail.nysed.gov .
LikeLike
Cathy, Copy your request to every media company and every public official you can think of
and put the cc’s on your original request. Generally they all contact each other to cover their asses and someone is forced to address the issue. publicity is refreshing.
LikeLike
this is fascinating. but rewrite the headline of your blogpost to appeal to public school advocates, nyc transparency folks, etc. if i share this on facebook as is, nobody except people who are aware of the VAM would click through.
LikeLike
Joanna Cannon is the Executive Director of Teacher and Principal Performance New York City Department of Education. In this role, she is one-down from the Chancellor and directly responsible for VAM as well as managing the relationship with the Wisconsin guys. Reach out to her directly as she will definitely have the answers, even if she has to kick your query out to the media relations group. At least she will know what “source code” is.
LikeLike
Well, the NYT did publish the formula:
Filed under the tag Mathematical Intimidation.
LikeLike
One of the biggest problems with the design of the NYC VAM is that it’s comparing results from the student’s test at the end of the prior school year with this year’s test results. This is patently unfair to teachers. A better design would compare beginning of the school year test results with end of the same school year results. This would measure score changes as a function of the teacher. Of course, they haven’t figured out how to do VAM for middle school and up yet since there are multiple teachers in those grades.
But admittedly this amounts to an apology for an inherently unfair process…
LikeLike
The problem may be asking for “source code” in C++ or Java or whatever. In all likelihood, the value-added model was run in Stata or a similar program, and whoever did it wouldn’t have any idea what the “source code” actually is.
LikeLike
to #13, it might also be Excel macros, the horror, the horror..
Here in CO the Race to the Top leaves the construction of the VAM model as an exercise for the reader. That is, administration in each school district, already overwhelmed by large and continuing budget cuts, now has to draw up a set of VAM metrics and prepare to defend it. The genius of this is that the failure of VAM to provide meaningful results can now be blamed on local school districts. Brilliant consultants from Pearson will doubtless be hired with public education funds to fix the VAM. In ten years the obvious failure of any of this to produce 100% ‘college-ready’ kids (like the 100% effective schools required by NCLB) will bring forth another monster. Or perhaps we will regain our sanity ? may I live so long.
http://www.cde.state.co.us/educatoreffectiveness/studentgrowthguide
The ‘student growth guide’ is worth 50% of a teacher’s evaluation.
They present an outline of an Excel spreadsheet, see the link for Measures of Student Learning Tool – download and marvel at its substance-free blithering.
So, it’s even worse here than in NY, hard as it may be to image that.
LikeLike
Here is a VAM lawsuit in Florida, just to get access to the data used. The newspaper won.
MORRIS PUB. GROUP, LLC v. State, 13 So. 3d 120 – Fla: Dist. Court of Appeals, 1st Dist. 2009
LikeLike