The achievement gap: whose problem is it?
On Monday night I went to see Boston College professor Henry Braun speak about the Value-Added Model for teachers (VAM) at Teachers College, right here in my hood (hat tip Sendhil Revuluri).
I wrote about VAM recently, and I’m not a fan, so I was excited for the event. Here’s the poster from Monday:
The room was not entirely filled with anti-VAM activists such as myself, even though it was an informed audience. In fact one of the people I found myself talking to before the talk started mentioned that he’d worked on Wall Street, where they “culled” 10% of the workforce regularly – during downsizing phases – and how fantastic it was, how it kept standards high.
I mentioned that the question is, who gets decide which 10% and why, and he responded that it was all about profit, naturally. Being an easily provoked person, I found myself saying, well right, that’s the definition of success for Wall Street, and we can see how that’s turned out for everyone. He stared blankly at me.
I told that story because it irks me, still, how utterly unscathed individuals feel, who were or are part of the Wall Street culture. They don’t see any lesson to learn from that whole mess.
But even more than that, the same mindset which served the country so poorly is now somehow being held up as a success story, and applied to other fields like public education.
That brings me to the talk itself. Professor Braun did a very good job of explaining the VAM, and the inconsistencies, and the smallish correlations and unaccountable black box nature of the test.
But he then did more: he drew up a (necessarily vague) picture of the entire process by which a teacher is “assessed,” of which VAM plays a varying role, and he asked some important questions: how does this process affect the teaching profession? Does the scrutiny of each teacher in this way make students learn more? Does it make bad teachers get better? Does it make good teachers stay in the profession?
Great questions, but he didn’t even stop there. He went on to point something out that I’d never directly considered. Namely, why do we think individual responsibility – i.e. finger pointing at individual teachers – is going to improve the overall system? Here he suggested that there’s been a huge split in the profession between those who want to improve educational systems and those who want to assess teachers (and think that will “close the achievement gap”). The people who want to improve education talk about increasing communication between teachers in a school or between schools in a district, and they talk about improving and strengthening communities and cultures of learning.
By contrast the “assess the teachers” crowd is convinced that holding teachers individually accountable for the achievement of their students is the only possible approach. Fuck the school culture, fuck communicating with other teachers in the school. Fuck differences in curriculum or having old books or not having enough books due to unequal funding.
It got me thinking, especially since I read that book last week, The New Prophets of Capitalism (review here). That book explained how hollow Oprah’s urging to live a perfect life is to people whose situations are beyond their control. The problem with Oprah’s reasoning is that it ignores real systemic problems and issues that radically affect certain parts of the population and make it much harder to take her advice. It’s context free in a world where context is more and more meaningful.
So, whose problem is the achievement gap? Is it owned in tiny pieces by every teacher who dares to enter the profession? Is it owned by schools or school systems? Or is it owned by all of us, by the country as a whole? And if it is, how are we going to start working together to solve it?
mathbabe 
just to clarify things: if a teacher doesn’t get tenure in New York, this teacher can usually get an extension and go for tenure again, repeatedly, at the same school. So not giving ten percent of teachers tenure is not the same as firing ten percent of people who try to become teachers. It is basically just saying ten percent aren’t ready for tenure.
LikeLike
It is however clearly a punishing message and meant to be so.
LikeLike
I’d come up with a management maxim, following Tony Hoare’s “You can’t control what you can’t measure.” Mine: “You can’t manage what you can’t recognize.” Or a little Tao Te Ching imitation: “The best managers recognize performance; the mediocre managers measure performance; the worst managers just do whatever, for no reason at all.”
Over time, it seems the whole culture has been bent on running everything by impersonal pre-arranged rules. Whether because it reduces responsibility and blame, or because discretion introduces the risk of bias, or something else. A lot of numeric targets (4% inflation?, killing the bottom 10%?) with no visible connection to outcomes.
LikeLike
Thanks so much for sharing, Cathy! (Do you know if his talk was recorded?) His critique is powerful, and connects with an implicit (and in my opinion, severely problematic) assumption of the “education reform” movement: that in so far as current outcomes fall short of our expectations, the gap is one of *motivation* (and thus addressable by incentives, whether positive or negative) rather than one of *capacity*.
It’s odd to me that we seem to think that the fact that some effective practices is known (somewhere, by someone) is sufficient, when it would seem that the really important thing is that it is widely-known, and even more, widely-used. (Especially so in a context where the whole job is to educate students who don’t yet know all of the things that are known.) To me, this is connected with another problematic assumption: that teaching isn’t that hard, when in fact, it’s quite complex. (Elizabeth Green’s book, I think, is a good step forward there.)
You ask, “So, whose problem is the achievement gap?” I can take this in two ways. The first is, “If someone is to fix it, who could that be?” You have addressed that, and I definitely agree that collaboration and joint problem-solving are key. (They require some beliefs: that teaching is complex, that through some work, we can get more effective at it, and that we will benefit by working together.) And it is not a problem for schools alone.
The other way to take “So, whose problem is the achievement gap?” is “If it doesn’t get fixed, who does it affect?” I think the answer to this question is “Everyone.” However, I don’t think this is a widely-shared view. (In his recent book “Our Kids”, Robert Putnam lays out both some reasons for our “shriveled” sense of collective responsibility, and the practical and moral arguments for why we should care.)
(If you are feeling apocalyptic, you might be receptive to the findings of the “cliodynamicists”, especially the “Secular Cycle” of social instability, and additional cycles of “widespread political violence” — see http://www.wired.com/2013/04/cliodynamics-peter-turchin/.)
LikeLike
I keep thinking this can’t possibly be the position of reformers, but I am afraid that it is: we don’t have to change any of the conditions poor students face, not the lack of housing that puts them in shelters, not the lack of decent jobs that keeps their parents overworked and underpaid, not the dangerousness of certain neighborhoods where kids are afraid of walking to and from school. If only those kids had great teachers, they’d be able to overcome every obstacle.
Of course there are inadequate and downright terrible teachers who should leave the profession. In every school I worked at, we all knew who they were. However, teaching is one of the largest professions, by numbers, in the country–it will never be the case that the majority of teachers will be superior educators. In addition to finding legitimate ways to remove bad teachers from schools–I favor getting rid of tenure and having something like five-year contracts–we have to have the common sense to recognize that teachers are only one influence in a child’s life.
Isn’t this another instance of our American individualism run amok? No reference to systems or context or economic tendencies, just bad individuals who should pay for their inadequacy.
LikeLike
John Dewey said that schools as institutions are miniature versions of their surrounding society. At best, educators can make them reflect the better aspects of the society. At worst, schools reflect bad things about our society, such as ruthless program cuts and layoffs in order to meet arbitrary numbers that powerful people have demanded.
So-called reformers like Cuomo are fond of saying, “Money won’t solve the problem.” The denial seems to be a Freudian admission that money has something to do with improving education. In New York, the Campaign for Fiscal Equity lawsuit, and subsequent failure for the state to provide the money that the decision on the case supposedly required, is a piece of this context. At least in New York, this somewhat undercuts another frequent refrain among reformers that equity is their motivation. This is just part of the context.
So-called reformers are also fond of saying, “We know what works.” Maybe if “we” knew exactly what the aims of education are, and if there were an adequate model for how education functions with respect to these aims, then all that’s left is to fix or eliminate whatever parts of the machine that don’t function properly. VAM appears to be the proposed model for the functioning of education. Cathy, you are as well able to make the needed take-down as anyone, and are probably the best.
But that still leaves the question of what the aims of education are. That’s a hard question. That’s philosophy, I’m afraid, and I’m afraid that once anyone starts talking about philosophy, eyes glaze over, or else people become irritated. (I’m trained as a philosopher, now a librarian, so this the blank stares and irritation are familiar.)
In teaching math, some teachers can aim at rigor, with the idea that this focuses on what math really is about, with all the good things that come from that. Or some teachers can aim to get students able to solve the kinds of problems they are likely to encounter at work or in life. Or some can aim to get students to appreciate something beautiful or wonderful, with an aesthetically motivated approach. Or some can aim to prepare students for a standardized test.
In a school with many teachers, it is possible that different teachers are capable of teaching well in different ways. If we don’t hold each individual teacher responsible for failing to meet an arbitrary standard, this allows for the possibility that there are different ways to succeed. This allows for a school that is democratic and tolerant.
LikeLike
Less long-winded:
(1) The question, what are appropriate aims of education?, is unavoidable and mixed up with everything in education policy. But it’s hard to talk about so everyone avoids it. Avoidance (in favor of simplistic ideas that conveniently excuse the powerful) explains the “blank stare.”
(2) One approach to this problem is tolerance of disagreement over aims, and acceptance that good teaching have a variety of aims.
LikeLike
“can have”
LikeLike
Hi Cathy-I just want to thank you for all your excellent coverage/discussion re: VAM and teacher evaluations. I’ve been sharing your posts with other public school parents who have been organizing around the issue. I hope we can all be in touch as we continue campaigning against this craziness in NY.
LikeLike
Absolutely!
LikeLike
Systemic thinking is unfortunately rare. It is easier to cast blame on individuals.
LikeLike
I have no brief for VAM, but so far all I’m seeing is the NULL hypothesis. This implies that we hire teachers at a basic salary, give them raises for a couple of years as they learn to apply theory and then COL raises until they retire. What else is implied by all this data?
LikeLike
In 3 districts in 3 states, I have yet to see a district or state actually manage COL raises year in and year out. Most don’t even pretend to do so. In my current state of NM, the state refuses to commit itself to COL raises because ‘they have to balance a budget every year’. No one seems to care much that the workers have to balance their budget as well. Consistent COL increases until retirement would be an improvement in many places.
LikeLike
you said he asked questions like “Does the scrutiny of each teacher in this way make students learn more? Does it make bad teachers get better? Does it make good teachers stay in the profession?”
Duh, these are questions everyone and their mom and has been asking for years. did he give any answers is what i wanna know. since this fellow comes from ets i assumed he was a vam supporter and i was looking forward to get to the part where you put him in his place, like a boss.
does he support vam or does he not?
LikeLike
Great post.
1) The 10% cull makes me think of the “re-engineering” efforts of the 90’s where invited to “re-engineer” organizations and, frequently, re-engineer themselves out of their jobs.
2) Item 1 makes me think of a a quote (roughly) “A dictator cares not whether his people love him so long as they hate each other.” The 10% cull stimulates the plebs to screw each other rather than collaborate to take down the monarch.
3) “By contrast the “assess the teachers” crowd is convinced that holding teachers individually accountable for the achievement of their students is the only possible approach….” How about fuck the parents who don’t particularly care how their kids do in school? A friend of mine taught in an ‘underperforming’ school a number of years ago. His take was that parental indifference was the underlying cause of many of the issues he had to deal with. How does VAM control for that?
LikeLike
Your friend identified the underlying problem of “failing schools”… which is parent disengagement. This problem, unfortunately, poses a set of thorny issues which ultimately boil down to this question: what is the role of “the government” in child rearing?
LikeLike
Take away their salary. Make them have uneven working hours. Make them take on two or the jobs. Give them predatory loans so they are caught in debt. Then blame it on child rearing.
LikeLike
To clarify my Item 3 above, my friend would comment that he spent 90% of his class time dealing with kids being rowdy. Most of the kids were there to learn but a half a dozen disruptive ones in a class of twenty-five made teaching very difficult. (He felt his successes were mostly in helping kids one-on-one study sessions/study halls.)
To what extent is it reasonable to hold teachers accountable for kids behavior in class and, more the point, to hold them accountable for lack of progress due to kids being disruptive? There are elements of the educational experience which are within the teacher’s control and there are elements which are beyond it. (Providing adequate resources and supportive administrators is important but it’s not everything.) How do you decide when it is appropriate to declare, “That teacher can’t control their classroom?” and when is it appropriate to put the onus on the parents for not making it clear to their kids that it is not acceptable for them to be disruptive in class? How the hell would you distinguish between the two situations by looking at trends in test scores or any other kind of metric? Not to be coy, I think it’s ridiculous to even try.
LikeLike
I suggest that teachers harmed by a system based on test scores sue the test maker and school system in Federal court and demand that professional standards of test validity be proven. The tests do not work; they are all IQ tests; we cannot make a test that is not. Force the people who want to use tests to evaluate schools or teachers to prove there is any significant test variance attributable to those factors.
LikeLike
I’m not an expert on VAM, educational testing, or anything like that. But I do know a bit about applied statistics/econometrics. The challenge to “prove there is any significant test variance attributable to those factors” seems too high a standard. As far as I can tell, VAM is a statistical model and variation in test performance is assessed statistically. Using such methods we don’t prove statistical significance because of the key role probability plays in such assessments. What we typically do is determine if there’s enough evidence to conclude that certain findings are unlikely, assuming the “null hypothesis” is true (at least this is how things are done in the “frequentist” statistical tradition). I’m not saying any of this to argue that VAM is a “good” model to use, teachers should be evaluated by it, or anything like that. I’m just saying that we ought to use a fair standard to judge VAM as we should to judge any other statistical model.
LikeLike
So with regard to wall street, its not about profit. Its about the compensation of senior decision makers. That is the incentive structure and that is the competitive pressure. It will be until the shareholders have more power.
LikeLike
Sigh. I’m all for statistical measurement in school ratings. I’ve never seen a scheme that included the largest factors in a student’s success: parents’ education and income & similar values for their peers & the community. I suspect if you add that in, and account for the students’ school history, most teachers’ contributions (barring really good or really bad ones) aren’t significant.
In most fields (mine included), no one wants to gather the difficult but vital data. “Dear Parent: Please forward us a certified copy of your annual tax return for our statistical analysis of your child’s performance. Include a transcript of your highest educational levels. Signed, Little Ellie’s School.”
LikeLike