Do Charter Schools Cherrypick Students?
Yesterday I looked into quantitatively measuring the rumor I’ve been hearing for years, namely that charter schools cherrypick students – get rid of troublesome ones, keep well-behaved ones, and so on.
Here are two pieces of anecdotal evidence. There was a “Got To Go” list of students at one charter school in the Success Academy network. These were troublesome kids that the school was pushing out.
Also, I recently learned that Success Academy doesn’t accept new kids after the fourth grade. Their reasoning is that older kids wouldn’t be able to catch up with the rest of the kids, but on the other hand it also means that kids kicked out of one school will never land there. This is another form of selection.
Now that I’ve said my two examples I realize they both come from Success Academy. There really aren’t that many of them, as you can see on this map, but they are a politically potent force in the charter school movement.
Also, to be clear, I am not against charter schools as a concept. I love the idea of experimentation, and to the extent that charter schools perform experiments that can inform how public schools run, that’s interesting and worthwhile.
Anyhoo, let’s get to the analysis. I got my data from this DOE website, down at the bottom where I clicked “citywide results” and grabbed the following excel file:
With that data, I built an iPython Notebook which is on github here so you can take a look, reproduce my results with the above data (I removed the first line after turning it in to a csv file), or do more.
From talking to friends of mine who run NYC schools, I learned of two proxies for difficult students. One is ‘Percent Students with Disabilities’ and the other is ‘Percent English Language Learners’ (I also learned that charter schools’ DBN code starts with 84). Equipped with that information, I was able to build the following histograms:
Percent Students with Disabilities, non-Charter
Percent Students with Disabilities, Charter
Percent English Language Learners, non-Charter
Percent English Language Learners, Charter. Please note that the x-axis differs from above.
I also computed statistics which you can look at on the iPython notebook. Finally, I put it all together with a single scatterplot:
The blue dots to the left and all the way down on the x-axis are mostly test schools and “screened” schools, which are actually constructed to cherrypick their students.
The main conclusion of this analysis is to say that, generally speaking, charter schools don’t have as many kids with disabilities or poor language skills, and so when we compare their performance to non-charter schools, we need to somehow take this into account.
A final caveat: we can see just by looking at the above scatter plot that there are plenty of charter schools that are well inside the middle of the blue cloud. So this is not a indictment on any specific charter school, but rather a statistical statement about wanting to compare apples to apples.
Update: I’ve now added t-tests to test the hypothesis that this data comes from the same distribution. The answer is no.
Those very small numbers are the p-values which are much smaller than 0.05. Other t-tests give similar results (but go ahead and try them yourself)
mathbabe 
Some people just love experimenting on Other People’s Kids, but few of those experiments involve informed consent, much less passing the Human Subjects Committee.
LikeLike
Thank you for doing this… There is one other proxy that you might check into: suspension rates. There is a school of thought that schools like the Success Academy have higher rates of student suspension. Can you tackle that for those of us who are not as facile with statistical tools?
LikeLike
Reblogged this on Network Schools – Wayne Gersen and commented:
I trust the Mathbabe’s analytics… and the short answer to her question is: “mostly yes”!
LikeLike
I am a bit concerned with your summation of this analysis. I am not speaking to the correctness of Mathbabe’s process (which is excellently documented), but to the fact that a visual comparison of distributions does not come close to proving the causality that your comment implies. School choice is a two way street and without a complete understanding of bias inherent in these systems, I would need a great deal more than group comparisons to justify the claim that most charter schools cherry pick their students.
LikeLike
Fair enough – I will avoid imputing intention. Let me say it like this: given that the populations are statistically different, it’s difficult to compare results from charter schools and non-charter public schools. In particular, it would be unsafe to assume that what works for charters would work for the general public school population.
LikeLiked by 2 people
Here is one other group of schools that cherrypick their students: elite private schools. Everyone knows the easiest way to have high performing kids coming out of a school is to start with high performing kids going into the school. As one school head explained to me: “it is hard to determine the value we add; we know the kids are great when they leave us, but they were already great before.” I guess it is similar to the old adage about how to make a small fortune: start with a big fortune.
Many academy schools in England (similar to US charter schools) often have another interesting bias: the schools are perceived as being so effective that eager parents have to compete to get places for their children. That creates obviously biased populations, without the usual fraction of kids from “i couldn’t give a f***” families.
Also, the point about not accepting entering students above grade 4 is a tricky one. As a teacher, it is really difficult if you can’t build off certain prior knowledge or experiences or, conversely, much easier if you can. This is a popular complaint among college calculus instructors who find themselves teaching a lot of algebra (or trig!)
By chance, I just today heard the head of Teach for Thailand (sister of TeachFirst) express that this is a common frustration of middle school teachers in his program. They think they’re going to teach about poetry and then find they have to spend all their time on phonics basics.
Does anyone know of a good diagnostic that can be used for an elementary school teacher (and another for high school) to figure out what’s missing in a kid’s understanding so that they can plan the right remediation? I mostly care about math, but other subjects would be of interest, too. Given the scope of material one would like to cover, maybe a computer adaptive test structure would be best?
LikeLike
The existence of such a diagnostic would depend on whether the skill in question can be broken down into steps, so that the missing ones can be located.
With chess, at least a computer can be told how winning is defined. For human language use, that’s a lot more complicated.
Even for the simplest arithmetic or geometry, there are many ways to get the right answer, and many ways of thinking about it correctly.
So how you as a teacher understand your students and how you respond to them both demand all your creativity as a teacher, assuming you are allowed to be creative.
LikeLike
I like this post – thanks. I struggle a lot with the structure of education. I think the key question this analysis leads to, as you say, is whether performance of the child who moves to a charter improves or not. But we should also study whether the performance of the child who does not move (in a system where some do) is impacted which is something that seems often asserted.
I struggle a lot with private education (which is related – but charters are of course public schools). Where I live the public schools aren’t awesome and my daughter attends a modest independent school instead that we love. She definitely gets an advantage by doing so, and the school district basically still gets a per student tax credit for her even though she is 0 cost to them – so that helps assuage my guilt. But I cannot help but think the arrangement isn’t the best way to be doing education in my community. But my community doesn’t support higher tax expenditures (both as a practical and political matter), so I’m left with a very practical decision to make for my kid.
LikeLike
This is a great look at this data! Thank you for documenting your process too. I am also interested in this type of analysis and reproducibility is close to the heart of data science. Would you be able to share the results of the statistical tests you ran when you compared these distributions and/or their summary statistics? (For those of us who don’t use python) Clearly the visualizations paint a picture, but because of the differences in the sizes (and makeups) of these two groups, I would hesitate to make a judgement about whether or not the populations that comprise these two groups are truly different. They may very well be, but these two metrics are not necessarily sufficient to prove that point (which I don’t think you’re claiming, but readers seem to be inferring).
There are some major assumptions being made here about, two at the forefront of my mind being; 1. Students enter charter schools and public schools by the same process and 2. districting is not an influential factor. To these points, readers should keep in mind that an exploratory comparison analysis (are these groups different) does not in any way prove that “Yes charter schools cherry pick students”. I’m not speaking about the accuracy of the analysis that you did, I mean simply that the implied overall assumption that ALL charter schools have autonomy over their student body and only select the students they want, is not proven or disproven by comparison of group makeups. It’s far too easy for implication to imply causation when view through this lens.
LikeLike
I ran t-tests and updated the post.
LikeLike
Its an interesting analysis. There is a bit selection bias with charter schools and English that we have to keep in mind. Charter schools are a “Push” enrollment so parents have to push their kids into one, where as public schools are the default “Pull” enrollment meaning everyone that doesn’t make a choice ends up there. That could easily explain why there are no charter schools in the over 30% ESL range as it is highly unlikely that non English speaking parents know to enroll their kids into a charter school.
LikeLiked by 1 person
Yes, that makes sense. But what about disabilities?
LikeLike
Disabilities are an interesting metric. There seems to be an odd distribution to the high end that I don’t have a really good explanation for. How do some schools have 50%+ disabled kids when about 12.1% of the national student body is and 25% of NY student body is disabled? This should point to some sort of parental selection or strong preference of certain schools for parents with disabled students.
I don’t have the full numbers but just based on the graph it seems that the median % of kids with disabilities in NY is around 25% so that’s 2x the national average so whats driving this tend? Are schools intentionally pushing kinds into the disabled bucket to get more funding? Is there more incentive for public schools to do that than charter schools? It brings up some interesting questions
REF: http://www.data-first.org/data/how-many-students-with-disabilities-are-in-our-schools/
LikeLiked by 1 person
I think we need a definition of “disabilities.” Are we talking about physical disabilities? Or are we talking about “learning disabilities,” which is often gamed by parents (gives a kid extra time on tests, including the SATs) and by schools, for funding, as Vladdy asks.
LikeLike
Regarding disabilities, I recall reading in the past that some elite private schools purposely choose not to provide the necessary staff and/or resources required to enroll such students. The claim was that an independent school doesn’t have the financial resources to provide the array of services for all kinds of different disabilities (even though many of them likely *do* have the financial resources). This way, the private schools turn this from a moral issue (we don’t want disabled students) into a resource issue (we don’t have the staff to serve disabled students).
So it falls on the public schools to accept the majority of kids with disabilities.
I’d be interested to know if charter schools can deny students with disabilities on the same grounds, or at least encourage their parents to look elsewhere.
LikeLike
thank you Cathy, it is good to have actual data and analysis on this.
My son is in a STEM charter school, run by a for-profit company. I had deep misgivings about sending him there, but all his friends were already at the school. In the event it’s a school filled with very smart nerds, geeks, misfits and others, all perfectly happy to subvert the dominant paradigm. At their homecoming two teams of girls from the school played football, with male cheer sections.. the student’s idea, not anything admin or the teachers did.
My boy is, let us say, a challenging student, so we will see how that plays out..
LikeLiked by 1 person
Thank you Cathy for the python notebook. It’s an excellent teaching tool, and I hope to find time to run it myself.
LikeLike
Experimentation is, or can be, fine, but not when it’s largely fictitious and used as a rhetorical mask for privatizing public resources, busting teachers unions, and operating hyper-segregated Skinner Box sweatshops for the Worthy Poor (who are often later deemed Unworthy, and pushed out).
Oh, and then there’s the looting and real estate plays that hide behind charter school proliferation.
Cathy, the canard that charter schools are about “innovation” and “experimentation” has long been debunked by their practices, which, especially among the chains that are intended to be the beneficiaries of the hostile takeover of the public schools, are almost always pedagogically revanchist.
Please catch up with what is really happening to the public schools, and use your wit and quantitative talents to expose it.
LikeLiked by 3 people
What he said …
LikeLike
Thanks for doing this. Anecdotal information is only as fair as the person who shares it (and that can be 2nd or 3rd-hand). The use of data like this is to ensure (or promote) an sensible comparison — if comparison is necessary — of outcomes.
LikeLike
Charter schools have no monopoly on experimentation. Their main innovation is making public education a for-profit enterprise and attacking unions. The knowledge of what makes schools and students successful is known and no secret. More poverty equals less successful schools and lower student performance.
LikeLike
I totally agree with Mr. Fiorillo, (although I had to look up “revanchist.”) Charters, and the appealing rhetoric that accompany them, are a tool of the interests who want to end public education in our country. I took the time to figure out how to log in and comment on this blog because it is so critical that people understand this. They are succeeding. They are beating a drumbeat that our public schools are failing, setting up impossible tests like parcc to prove it, and convincing the public that public schools need to be disbanded at an alarming rate. They are bullying public school teachers until no one wants to do it anymore and creating a self-fulfilling prophecy. The best evidence of the lie is the push to increase the requirements for teachers (only the best for our kids!!) and then, when they can’t fill the positions, they leave classrooms without teachers or waiver the requirements. This is a crisis, and we need to be alert to it.
LikeLiked by 1 person
Certainly in the worst case scenario you’re exactly right. Not all charter schools are like that though.
LikeLiked by 1 person
What charter schools do is providing an environment in which students who want to learn, encouraged by parents who want their children to learn, can succeed. The unruly child who can prevent a whole class from learning, which can and does exist in inner city public schools, does not exist in such an environment. I understand that some feel that this is unfair to other children.
LikeLike
So what you are saying, (if I’m hearing you correctly,) is that we plan to publicly fund two sets of institutions: One set has public oversight and rules, one of which is that they cannot turn any child (no matter how disruptive) away, and another set which has no oversight and can limit themselves to children who fit a particular profile of behavior.
That makes sense. In neither case do we address the unruly child who prevents the class from learning; we just make sure it’s not OUR kid in their class.
Incidentally, that child exists in suburban schools, also. I was teaching in a public middle school in a relatively rich district where a child diagnosed odd (oppositional defian disorder) was allowed to curse and scream at the teacher throughout the class and no one was allowed to address it. The parents in suburban schools have the wherewithal (money) to enforce a ruling like that. So according to you, we need to fund a two tiered system everywhere.
It’s not that I don’t agree with you that there is a problem. I’m going to go in a face it tomorrow. I just disagree that charters are a solution. With over half of the children in the country in poverty, these problems are widespread. The disruptive kid may have watched someone beat the devil out of his mom the night before and feel helpless, or be living in a shelter where they are afraid to go to the bathroom all night….or a million other things.
LikeLiked by 1 person
I hear you, but we have 150-200 year of “free” public schools, and yet the problem has not been resolved. Someone else claimed we know how. Well, I wish they would implement that magic. Charters are public schools and they do have oversight. The ones in Harlem seem to be successful at graduating students with high achievement who otherwise would not have reached that level. I’ve also taught, albeit shortly, in inner city schools, and have had the child who insisted on preventing every one else from learning, and what that child was absent or was in the dean’s office, another child would step up to the plate and provide that function. But I also had parents who really cared and their children did do better than the others. Poverty alone (and I grew up dirt poor but I have a mother who cared) does not mean parents don’t care or children not wanting to learn, but the environment that is created in school, and even in a poverty stricken home, has a major effect, Charter schools are able to create that environment because the students WANT to learn and the parent or guardian CARES. It’s not about the quality of the teacher, as public schools have some great teachers, and it’s not about union busting. It’s about NOT keeping everyone to the same LOW standard. I would rather have some solution than no solution. And somewhat off topic, Common Core is not the solution. Good luck tomorrow and every day.
LikeLike