Two great articles about standardized tests
In the past 12 hours I’ve read two fascinating articles about the crazy world of standardized testing. They’re both illuminating and well-written and you should take a look.
First, my data journalist friend Meredith Broussard has an Atlantic piece called Why Poor Schools Can’t Win At Standardized Testing wherein she tracks down the money and the books in the Philadelphia public school system (spoiler: there’s not enough of either), and she makes the connection between expensive books and high test scores.
Here’s a key phrase from her article:
Pearson came under fire last year for using a passage on a standardized test that was taken verbatim from a Pearson textbook.
The second article, in the New Yorker, is written by Rachel Aviv and is entitled Wrong Answer. It’s a close look, with interviews, of the cheating scandal from Atlanta, which I have been studying recently. The article makes the point that cheating is a predictable consequence of the high-stakes “data-driven” approach.
Here’s a key phrase from the Aviv article:
After more than two thousand interviews, the investigators concluded that forty-four schools had cheated and that a “culture of fear, intimidation and retaliation has infested the district, allowing cheating—at all levels—to go unchecked for years.” They wrote that data had been “used as an abusive and cruel weapon to embarrass and punish.”
Putting the two together, it’s pretty clear that there’s an acceptable way to cheat, which is by stocking up on expensive test prep materials in the form of testing company-sponsored textbooks, and then there’s the unacceptable way to cheat, which is where teachers change the answers. Either way the standardized test scoring regime comes out looking like a penal system rather than a helpful teaching aid.
Before I leave, some recent goodish news on the standardized testing front (hat tip Eugene Stern): Chris Christie just reduced the importance of value-added modeling for teacher evaluation down to 10% in New Jersey.
mathbabe 
Politicians, the press, spread-sheet magicians, and the public love standardized tests because they provide seemingly precise data inexpensively. Because the data is “precise” it can be used to rank things like schools, students, and teacher and administrator performance. As your readers know well, precision and accuracy are different from each other… and standardized test results are NOT accurate indicators of the kind of education we supposedly aspire to in our country. Please keep beating the drum against the misuse of testing!
LikeLike
Odd, No one is paying attention to the underlying flaws in the meaurement theory that it is based on: Item Response theory.
http://en.wikipedia.org/wiki/Item_response_theory
LikeLike
The idea that you can always tease out the latent variable in “big data” if you model it well enough has to die.
Sometimes the other latent variables in the data you are trying to measure are just to strong and your attempts to better the fit of your model turns it into an RNG.
LikeLike
“she tracks down the money and the books in the Philadelphia public school system (spoiler: there’s not enough of either)”
One more thing that was really interesting about Meredith’s article. It’s easy (and pretty popular among people with direct instructional roles) to say “the problem is we’re wasting all this money on administration!” And in this article you see what happens when you get rid of all the administrators and there’s nobody left to deal with boring stuff like inventory. Nobody even KNOWS where the books are, and there are books still in boxes that aren’t being used. From the article:
“If the principal doesn’t meet with every parent, deal with every crisis, they get criticized. If they don’t do the invisible stuff, like the paperwork, they’re not going to read about it in the newspaper. So they triage.”
We all get annoyed about bureaucracy, but bureaucracy has a purpose, and you miss it when it’s gone!
LikeLike
Great point. She makes that point in as well when she talks about the attempt to create a district-wide database with book inventories for every school. The problem then is, nobody in the administration is available to update the system.
On Wed, Jul 16, 2014 at 8:02 AM, mathbabe wrote:
>
LikeLike
Regarding Professor Broussard’s article, if her thesis were correct then in NYC public schools – which have access to books and in which children (and their parents) have access to a great public library system – we should see much higher scores than in Philadelphia.
LikeLike
Also I wondered why Professor Broussard is doing her child’s homework? Does her child not have access to books?
LikeLike
I’ve worked in public education for fifteen years and one thing had not changed – we ignore the biggest factor in how well children do in school. Home life. Constant stress from poverty, threat of violence and instability impact development in ways we are finally beginning to understand. Short term stressors also impact test scores (divorce, major illness, death in the family) but we never take these into account. We can improve education and accountability to a point, but we will never “fix” education without also addressing poverty, violence and instability in our communities.
LikeLike
“divorce, major illness, death in the family” occur across all socioeconomic strata. No one is exempt.
LikeLike
They do, but we label those children (no matter economic strata) as failing because scores go down one year without looking at what else is going on in their lives. Teachers are labeled ineffective and students forced into remedial classes they don’t need based on one test alone.
LikeLike
True, but not with the consistency, pervasiveness or intensity that are experienced in disadvantaged communities.
LikeLike
““divorce, major illness, death in the family” occur across all socioeconomic strata. No one is exempt.”
This may get me in moderation, but this is a vile answer.
This is equivalent to saying everyone dies so there is no causation between a groups mortality rates and that groups wealth.
As to your other point, the article very clearly lays out why its not access to books that is the issue, but access to the specific books that teach the test the big 3 give and grade.
Standardized tests whose answers are only available if you purchase them is a very funny definition of “standardized”. It would put a lie to this being about education and not Money if the elites of our society weren’t so corrupt.
LikeLike
Individuals, not groups, deal with individual problems such as divorce, major illness and death in the family. I have no idea why that upsets you. I have dealt with all three, even though that is none of anybody’s business. As for books, when it was time to prepare for the SATs (many years ago) I went to the public library and used a public library book to prepare. And to increase my vocabulary beyond my mother’s poor English I read lots of books from the public library, since I could not afford to buy them on my sweatshop working mother’s income.
Rational people can disagree, yet still respect each other’s perspective.
LikeLike
We are talking about 8 and 12 year children deciding the fate of their schools, teachers, and districts funding based on a measurement on how they perform on a test.
We are not talking about if 8 or 12 year old children should/could deal with stressful life situations in the long term.
We all have hard luck stories that we over came because we are the ones who survived. The ones who didn’t aren’t around to tell their stories.
LikeLike
Ever hear of the “culture of poverty”? It’s also cognitive. Ruby Payne does.
http://en.wikipedia.org/wiki/Culture_of_poverty
LikeLike
For me the key quote from Rachel Aviv’s gripping article is this:
David Berliner, the former dean of the school of education at Arizona State University, told me that, with the passage of the law [No Child Left Behind], teachers were asked to compensate for factors outside their control. He said, “The people who say poverty is no excuse for low performance are now using teacher accountability as an excuse for doing nothing about poverty.”
As one who has taught in inner-city schools (as they were once called), I could readily identify with the teachers that found themselves enmeshed in cheating in order to protect their students, whose vulnerabilities and needs they clearly understood and felt keenly. I know of many, many teachers in these schools who, out of their own pockets like Damany Lewis, regularly provided needy students with basic necessities or who purchased materials for their classrooms that school budgets didn’t afford. The emotional and other kinds of intangible support they provide are further evidence that most of these teachers take the phrase “in loco parentis” seriously.
The persecution and firing of staff who care about them, the culture of fear and intimidation in a system, the wrecking of communities by the closing of local schools, which have been a place of at least some minimal safety and support, are devastating for children whose lives are already beset with problems the rest of us can barely imagine.
Education needs to be emancipated from the whip and shackles of standardized testing and policies like NCLB that lead to such mindless distortion of the process of education.
LikeLike
While acknowledging all of the problems that happen in our schools including those that are a result of standardized testing, I still see no alternative to testing for what I consider a real problem. Please feel free to explain why this isn’t a real problem or suggest alternative ways to solve it that don’t involve standardized tests.
Fundamental Problem: How do we determine as a society if students have learned what we want them to learn?
If students at two different school gets an A in English in the fourth grade, does that tell me anything about what they have learned? I would say no it doesn’t. It might tell me that
the students learned more then a student in their class who got a B, but there is no way
for me to compare between classes let alone schools. Even if both schools are nominally using the same instructional materials, I can’t know because there I don’t have any idea how individual schools/teachers do their grading. What is to stop any school from ending up like the example from Atlanta? From the outside, it was doing all of these wonderful things to to improve things. But without cheating, standardized tests would have shown that they weren’t actually meeting improvement goals. So they cheated. Without standardized tests, I’m not sure that they (let alone outsiders) would have even known that they weren’t measuring well against other schools.
Now it may be that the goals were too high and we should have just accepted the level of improvement that they were achieving, but without testing I don’t see how we would even know.
I’m worried that without testing we would become complacent. If it looks like things are going well we will just assume that they are.
As for the usefullness of testing, I’m old enough that my first real exposure to standardized tests was during the college admission process and I am pretty sure that they were created because colleges had this exact problem. What does an “A” in a particular class actually mean? They couldn’t know so standardized tests were created as a shortcut to extensive interviewing of each applicant (or simply only admitting from a select few schools). While the importance of standardized tests for college admissions has changed over time, I’m pretty sure that the vast majority of colleges still give them significant weight in the admissions process as they have found no other feasible method to solve this problem.
Finally, although Common Core is NOT inherently standardized testing; I feel it necessary to point out that without Common Core standardized testing is problematic. Colleges don’t care because they only need to figure out if the students that they admit will be successful. Students that don’t make the cut aren’t their problem. Nor do colleges care how a student did in 5th grade. All students are our problem and we should care how students are doing at every grade level. It wasn’t until Common Core that we even had national standards for what students at particular grade levels should be learning and so we couldn’t even begin to attack the problem of what students should vs. actually did learn in any particular grade. We don’t have (and probably shouldn’t have) a national standard on actual instructional material. As a result, I see no other choice besides standardized testing for doing between school (or even between classroom) comparisons and I see Common Core as a necessary prerequisite as well.
LikeLike
I find it extremely interesting that my comment above elicited not a single response. I expected most people who follow this blog to be against standardized testing. However, I didn’t expect a complete failure to engage with an honest attempt to point out the positive good that can come from it. I have a hard time seeing a difference between this and the refusal of pro-testing advocates to even acknowledge there might be negative aspects to testing. In both cases, it seems that ideology triumps over reason.
LikeLike
[Fools rush in. Guilty as charged.]
There are so many avenues to approach your seemingly simple question. I’ll only deal with two.
First, the nature of intelligence and learning is manifold. We’ve all heard stories of the kid at the bottom of the class who ends up creating a multimillion-dollar startup. While anecdotal, it does serve to highlight the wide range of knowledge and achievement that each of us is capable of.
Multiple choice and other “pencil and paper” tests challenge only a relatively small range of any student’s skill set. In this respect, children from disadvantaged circumstances (no books, limited opportunities for stimulating verbal interchange, etc.) have been shown generally to do less well on standardized tests. (This is another reason I understand why Lewis and the other teachers went “over to the dark side” and edited their students’ responses.)
The universities cannot stand as the only arbiter of what people “should” learn, let alone what they actually do learn and use for varying degrees of success in life.
Second, most teachers innately understand and are trained in other kinds of assessment, many of these happening on the fly as they interact with their students or watch them interact with others. These perceptions and evaluations have yet to be codified in a way that would satisfy society’s need for a number or letter grade, which is why there is still room for teacher comments on most report cards.
As a teacher myself I was always wary of assessing a student’s work too negatively until I’d had the opportunity to actually discuss it with them to see whether their ability to put words to paper matched the depth of their understanding and insight. This is particularly true of younger students for whom much knowledge is still inchoate, but who will eventually arrive at some clearer understanding with age and further experience. (Piaget’s work seems to have been forgotten by the testers and the education establishment at large. It’s one reason why you can’t teach, say, per cent in a single two-week timeframe and assume that every kid gets it.)
Our society is obsessed with winners and losers, praising, nay, glorifying the one without acknowledging the effort and achievements – the value – of all the others (Did you follow the World Cup?)
Looking at the big picture, humanity seems to have scored high on the biggest test of all, evolution. And yet, we now live at what appears to be the brink of a major fail or series of failures, victim of our own success as measured by numbers: seven billion and growing. Not the kind of achievement we can crow about. It should make us rethink our attitudes to success in general and, more specifically, to how we educate our children for survival on Spaceship Earth.
LikeLike
Somehow I didn’t get see your reply earlier and I thank you for doing so.
While I agree with what you are saying (universities shouldn’t be the arbiter of success), most of our society is running around saying that without a college degree you have no chance of having even “decent” (whatever that means) standard of living. Whether there is any real reason for this state of affairs is in some sense irrelevant. As long as employers use having a college degree as an arbitrary gateway to hiring someone it will be a self fulfilling prophecy. Not acknowledging this when
we educate our children is doing a real disservice to them.
Yes, we should try to change things, but I don’t think we should ignore how things are now and are likely to stay for
some time.
LikeLike
What’s the point of doing comparisons if the instrument you are using to do the measurement is biased and inaccurate? That it doesn’t measure what you think it measures.
I’ll leave it to the American Stats Association –
Research on VAMs has been fairly consistent that aspects of educational effectiveness that are measurable and within teacher control represent a small part of the total variation in student test scores or growth; most estimates in the literature attribute between 1% and 14% of the total variability to teachers. This is not saying that teachers have little effect on students, but that variation among teachers accounts for a small part of the variation in scores. The majority of the variation in test scores is attributable to factors outside of the teacher’s control such as student and family background, poverty, curriculum, and unmeasured influences.
Click to access ASA_VAM_Statement.pdf
LikeLike
You seem to be talking about the issues involved in using standardized tests of students to evaluate teacher performance. I’m quite happy to acknowledge that for that purpose standardized tests of students are very problematic.
That doesn’t mean that they aren’t useful for other purposes. I suggested some other purposes. Do you disagree with the usefulness/importance of the other purposes?
LikeLike
Reblogged this on The Art of Teaching Science and commented:
This is an important article by Dr. O’Neil with references that are applicable to those of us in Georgia, especially. Unfortunately, standardized testing is accelerating, not slowing down, and in Georgia, a new series of standardized tests will be rolled out next year. Targeting higher test scores, and higher graduation rates diverts the real purpose of schooling which should be in interests of students and their families.
LikeLike