Guest post: Quatama Elementary
This is a guest post, converted from a letter to me, by Derek Osborne, a father of four and active participant in his community with a strong belief that real change happens at the local level. Derek is a data scientist at Intel where he works on a team that utilizes machine learning techniques to optimize the workforce at Intel. Prior to working at Intel, he earned his Ph.D. from the University of Michigan in Biophysics.
I moved to Hillsboro, Oregon four years ago with my wife and three kids after finishing my Ph.D. at the University of Michigan. Like many parents when choosing a home, I checked on the school scores of the nearby elementary schools and there was a large variance in the Zillow school scores that are taken from greatschools.org. After house hunting for a long time, we finally found a home that was perfect for our family but it was in the school boundaries of Quatama Elementary that was ranked a 5 out of 10 and red. Asking around, other parents told us the reason was because there was low income housing in the area which was driving down the score. We felt that if the only issue with the school was that the school boundaries included low income housing, it shouldn’t stop us from buying the home. We could always transfer to a better school if we didn’t like the experience.
Over the following years we have loved all of our teachers, the principal, and our kid’s classmates and were baffled that it was scoring so low. During this time, we’ve met people that avoided the school when they moved in because of the score they saw on Zillow when they moved to the area. We also have had multiple friends move away because of the school’s ranking. When they would move, we’d ask what in the school do you dislike and they would acknowledge their personal experience was positive but they wanted to move to a “better” school. It was sad to see people trust a single digit score more than a personal experience.
Over this time, I’d check the same single digit ranking every year or so to see if it has gone up but it would remain the same. I felt that our school was a quality school and I was confused why the score never changed. What was even more baffling is that I started to dig into the scores published by the state that go into more detail and Quatama scored nearly the same or higher as its nearby high performing schools. After hearing some other parents say they wouldn’t let their kids go to Quatama, I felt that I needed to figure out why it was “rated low”.
I emailed greatschools.org and explained the situation and I got back a standard cut and paste answer but after a few emails insisting something was wrong they realized there was an error in their publishing system for Quatama. They have now updated the rankings and Quatama is now an 8 out of 10 and “green” which is comparable to its high performing peers. The perception that Quatama is a low performing school was completely erroneous and based off a math system gone wrong.
I’m now working with the principal to see if there is a way for us to measure how this rating has impacted the school. My thought that the same way there are bandwagon fans, there are bandwagon parents. Now that the school is rated higher, will the parents view of the school change? Will the parental support change over the next few years? If it does change, this will open up a large question about the morality of publishing overly simplified data.
mathbabe 
And, perhaps, the morality (or at the very least wisdom) of relying upon published over-simplified data. This is another example of the end user problem. It’s also (along with such things as “eggs are bad for you, no wait eggs are good for you”) a contributor to the alternative facts, post truth, don’t trust the experts ethos. As Derek himself found out, it’s a lot of work to get at the truth sometimes. Most of us either just accept things at face value or give up at some point.
LikeLike
It does raise the meta question, as well. How does one know that the published data is “overly simplified.”
LikeLike
Those test scores used to rank schools do not take into account the impact of poverty on a child’s motivation to learn or how damaging a dysfunctional family can be (we can’t blame poverty for everything). I taught for thirty years in public schools in the same district that had high rates of child poverty, but in every class there were great students that worked hard and many of those motivated children ended up going to college (a few to universities like MIT and Stanford on full scholarships) but in the same classes were children who did not cooperate, do classwork, read books, or do the homework and these children made up most of the students that scored low on those blind, mute, and deaf (often flawed and always secretive) high-stakes tests (that make profits for a huge corporation) that are used to punish teachers and close public schools.
Changing schools does not deal with poverty or dysfunctional families. When our daughter was in 3rd grade she complained about one of her teachers. We had a long talk and I said it is the teacher’s job to teach and your job to learn. You can’t blame the teacher when you don’t learn. When you don’t learn, it is your fault. Even if the teacher is incompetent and boring, that is not an excuse not to learn.
From 3rd grade to high school graduation, she earned straight A’s. In her second year at Stanford (where she graduated with a BA in 2014), I asked her how many incompetent K-12 teachers she had out of the 40 – 50 in the five different school districts she attended. She thought about it and eventually said TWO, and she earned A’s in their classes too and never came home complaining about them or blaming them for anything.
LikeLike
“they realized there was an error in their publishing system for Quatama” and “based off a math system gone wrong”. So which was it, a publishing error or a “math” error?
LikeLike
Great question David. Below is what greatschools.org sent me:
“The school actually appeared on the site twice, and one of the pages was deactivated. That deactivation contained the test scores.”
When I wrote “math system” I did not intend this to be read as just “math” but a full product that ranged from acquiring data, modeling the data, and then publishing data. I can see where the confusion comes from.
LikeLike
I see. So once they had the intended data correctly associated with Quatama, and did whatever “math” is in their model, the number they computed and published was 8/10, “which is comparable to its high performing peers”. To the extent one is going to rate schools on a single dimension, does this mean you feel that this number is reasonable?
LikeLike
Yes, given the inputs of the model and that it is rating a school on a single dimension, I think the score is reasonable. I think that everyone would agree that rating a school on a single dimension is an oversimplification. When looking at greatschools.org, they have multiple disclosures attempting to teach the readers that it is a simple model. After this experience and seeing how many in my community valued this single dimension score more than their own personal experience, I think that we need acknowledge that overly simplistic models can have pernicious impacts on the readers.
LikeLike
Super interesting! Those scores are so powerful, so they should have more controls in place to minimize those kinds of errors. Scores can be so misleading. In our neighborhood they have scores that rank really really high and the public schools get all these awards. But our impression of the schools were that they weren’t right for our kids: they are overly focused on repetitive drills and standardized tests, they don’t care about nurturing a kids inherent curiosity or drive, and spend zero time on social and emotional development. So to the shock and awe of our neighbors we drive them to another town where they go to a private progressive elementary school.
LikeLike
This leads to an interesting legal question as well. Given that many parents who are prospective home buyers rely on school rating systems and the impact of such ratings on homeowner values: could a group of recent home sellers in Quatama sue greatschools.org for deflating the sales price of their homes and/or increasing the number of days their homes remained on the market? If Standard and Poors can be legally liable for their role in the housing bubble why couldn’t greatschools.org be vulnerable to a lawsuit?
LikeLike
It’s really a shame that no one — NO ONE — caught the error, not the school district, not the parents, no one until now.
Yes, a lawsuit — how else can we make websites accountable?
What a horrible story! Just think how many OTHER errors are involved! Ugh!
LikeLike
It would appear that not “everyone would agree that rating a school on a single dimension is an oversimplification.” Those who moved because of the rank and perhaps despite their personal experiences made a pretty significant decision. There must have been other motivating factors around the importance of such a ranking? Any ideas? Compliments to you for pressing this issue.
LikeLike
Jeff, I’m not sure there are other motivating factors around the importance of the ranking but do agree that it is a significant decision to move. One thought I’ve had is that a decision to move (like a school ranking) really can’t be broken down to just a single dimension. Likely, the school ranking was one of multiple reasons for a move. Because the school ranking is public knowledge, it acts as an easy and fast method to justify a move. For example, maybe my neighbors think my family is too noisy and would rather blame the school than talk about how loud my family is. We really don’t know though and this is why I’m excited to see if in a year from now there is an increase in parental support and fundraising money because this would be a better indication that the school ranking was changing behavior. I’d love to get your thoughts.
LikeLike
Derek, You haven’t provided any information on how word of the corrected ranking has been publicized. A “bandwagon” effect can occur only if people know about the correction. You seem to have paid attention to the ranking because you thought there was a discrepency between a rank you noticed when investigating to move into the area and the educational environment your children experienced. It seems an assumption is this ranking is similarly followed by others over time. I don’t know anything about greatschools.com. Are these rankings publicized by schooldistricts? state dept of education? If Quatama had been originally rated “green” would you have continued to follow the ranking?
LikeLike
Derek mentions that his Quatama neighbors said “low income housing” was “driving down the score.” Since we now know the score was erroneous, what becomes of that conclusion and its underlying assumptions?
In Portland, where I live and work for the public school district, “low income” is too often used as a proxy for “minority.” I’d be curious to know if that was a factor in the perceptions of people who avoided or moved away from Quatama.
LikeLiked by 1 person
There are many unanswered questions in this narrative and it is an awful lot to ask Derek to put the time and effort into answering them all. He did good work pursuing the issue and is trying to assess the effects of the change. That’s enough for one person. What can we do? One thing would be to contact greatschools.org referring to this situation and asking for substantiation that their programming has been fixed in general, not just for Quatama, so any other school without test scores would not be rated. Might also inquire how a school without any test scores could still be rated 5 out of 10 and not 0 out of 10? Contact Zillow and apprise them of this problem. Now to be picky! It seems to me greatschools.com is not rating schools on a single dimension.They are using many dimensions to come up with a simple ranking. What information can they possibly have besides test scores?
LikeLike
I didn’t mean to imply that I’m expecting an answer from Derek. He’s done good service bringing this forward. But I did want to point out that the greatschools.org error opened the door for people’s (conscious or unconscious) biases to operate against Quatama ES: it wasn’t the error alone that caused people to avoid the school.
LikeLike
Tiel, I don’t know if people were using “low income” as a way to communicate “minority”. I don’t have the answer though but agree that some use these terms interchangeably.
LikeLike
Take a look at this, Derek— and Cathy!
LikeLike
The rating methodology is actually discussed (in part) here:
Click to access GreatSchools_Ratings_Methodology_Report.pdf
Almost all of the really important information about the whole website is buried in the fine print.
Look at the bottom panel, navy blue, and click on the entries. The privacy policy is a fiction, especially if you are user of Facebook. You can find the “methodology” for rating schools there, with weightings assigned to test scores, which tests are used (including ACT, SAT), how the scores in various subjects get combined and other great feats of statistical wizardry. The pretense of objectivity is fully developed.
In whisper type at the very bottom of the navy blue panel you can find all of the information about leasing data. In addition you can find two bizarre “research reports” that manipulate NAEP scores, and state test scores in math and reading in order to rate schools. One of the reports in based on an invented Education Equity Index for schools based on the percentage of students who qualify for free or reduced price lunches (a proxy for property). That report complains about a change in federal policy for data gathering that shifts attention from poverty rates in individual schools to poverty rates in an entire district.
LikeLike