Bill Gates is naive, data is not objective
In his recent essay in the Wall Street Journal, Bill Gates proposed to “fix the world’s biggest problems” through “good measurement and a commitment to follow the data.” Sounds great!
Unfortunately it’s not so simple.
Gates describes a positive feedback loop when good data is collected and acted on. It’s hard to argue against this: given perfect data-collection procedures with relevant data, specific models do tend to improve, according to their chosen metrics of success. In fact this is almost tautological.
As I’ll explain, however, rather than focusing on how individual models improve with more data, we need to worry more about which models and which data have been chosen in the first place, why that process is successful when it is, and – most importantly – who gets to decide what data is collected and what models are trained.
Take Gates’s example of Ethiopia’s commitment to health care for its people. Let’s face it, it’s not new information that we should ensure “each home has access to a bed net to protect the family from malaria, a pit toilet, first-aid training and other basic health and safety practices.” What’s new is the political decision to do something about it. In other words, where Gates credits the measurement and data-collection for this, I’d suggest we give credit to the political system that allowed both the data collection and the actual resources to make it happen.
Gates also brings up the campaign to eradicate polio and how measurement has helped so much there as well. Here he sidesteps an enormous amount of politics and debate about how that campaign has been fought and, more importantly, how many scarce resources have been put towards it. But he has framed this fight himself, and has collected the data and defined the success metric, so that’s what he’s focused on.
Then he talks about teacher scoring and how great it would be to do that well. Teachers might not agree, and I’d argue they are correct to be wary about scoring systems, especially if they’ve experienced the random number generator called the Value Added Model. Many of the teacher strikes and failed negotiations are being caused by this system where, again, the people who own the model have the power.
Then he talks about college rankings and suggests we replace the flawed US News & World Reports system with his own idea, namely “measures of which colleges were best preparing their graduates for the job market”. Note I’m not arguing for keeping that US News & World Reports model, which is embarrassingly flawed and is consistently gamed. But the question is, who gets to choose the replacement?
This is where we get the closest to seeing him admit what’s really going on: that the person who defines the model defines success, and by obscuring this power behind a data collection process and incrementally improved model results, it seems somehow sanitized and objective when it’s not.
Let’s see some more example of data collection and model design not being objective:
- We see that cars are safer for men than women because the crash-test dummies are men.
- We see that cars are safer for thin people because the crash-test dummies are thin.
- We see drugs are safer and more effective for white people because blacks are underrepresented in clinical trials (which is a whole other story about power and data collection in itself).
- We see that Polaroid film used to only pick up white skin because it was optimized for white people.
- We see that poor people are uninformed by definition of how we take opinion polls (read the fine print).
Bill Gates seems genuinely interested in tackling some big problems in the world, and I wish more people thought long and hard about how they could contribute like that. But the process he describes so lovingly is in fact highly fraught and dangerous.
Don’t be fooled by the mathematical imprimatur: behind every model and every data set is a political process that chose that data and built that model and defined success for that model.
mathbabe 
Более, чем точно. Принятие решений на основе неполных данных – это еще возможно, но на практике, как правило, принимаются на основе недостоверных данных. И это не самое страшное, хуже когда сомнительные ценности лиц, принимающих решение, и исполнителей, как вполне возможно в нашем случае. А Б.Гейтсу желаю удачи.
LikeLike
Drug doses are apparently also safer and/or more effective for men (for the same reason that women, particularly of child-bearing age, are underrepresented at clinical trials). http://well.blogs.nytimes.com/2013/01/28/the-drug-dose-gender-gap/?src=rechp
LikeLike
Richard Feynman told a story about how after WWII, when bureaucrats were newly impressed by physicists because of the success of the Manhattan Project, he was asked by the State Department to go to Brazil as a science advisor. The task he was given to address was how to get clean water to the favelas. He thought this was ridiculous–clearly Brazilian engineers had the intelligence and capability to solve this problem–it was an issue of political will and allocation of resources, not one of science and technology.
On the other hand–perhaps one reason for Gates’s success in various endeavors (aside from the billions, of course) is that he defines problems in such a way that they seem solvable. That is no small thing in a time when we are constantly told that each crisis is so complex and so interconnected with other crises that it is practically insoluble.
LikeLike
I’m gonna go with the money. That, and the potential for more money.
LikeLike
Bitter post
I kind of like the way Feynman gives them what they want by not giving them what they want. It remids me to Xzibit. Smart people are always doing that. I hate it. Or like it. I’m not sure.
LikeLike
Gates is spending his own money so he gets to make the rules. Period. Unlike governments, he’s actually trying to make a difference so why would he choose to waste his money collecting flawed metrics? He might start out collecting flawed data but without seeing a real improvement I’m guessing there would be a re-evaluation. Human error is acceptable and understandable but you seem to be implying less than honorable intent or outright corruption. When you (personally) start paying the bills for improvement projects on the same scale then you can start moaning about the methodologies employed.
LikeLike
Pardon me for questioning a rich person! Anyway I’m not actually questioning his intentions, I’m only questioning whether it’s as simple as he implies.
LikeLike
That is the context I took away from this Cathy. I am sure the Bill and Melinda Gates foundation have good intentions and having met Bill once at Durham Public Schools (I worked there and he happened to sponsoring a project through his foundation) he seems like a genuinely warm person. That his thoughts on solving problems with data might not be spot on is no reflection on the man himself. You are absolutely right.
LikeLike
Unlike Cathy, I will question his intentions. And as for you: No, he doesn’t. His money, true, but where did it come from? Heaven? No. Outer Space? No. From other people, many who gave it unwillingly and gave too much of it for what he gave in exchange. So, no, having the money does not mean you get to make the rules (actually, my money = the rules is the rule of iron, the rule of silver is if my money = your money, then money = the rules, the golden rule is consideration of other things beside money = the rules). You seem to subscribe to the rule of iron, I subscribe to the golden rule.
LikeLike
I completely agree with H. Alexander. Bill Gates wants to control things. His approach to philanthropy is the same as his approach to business and that is to monopolize. For example, his foundation spends hundreds of millions of dollars to influence education policy, much of which seeks to privatize public education. He has also been instrumental in having his own people appointed to key positions in the US Department of Education. Finally, by playing the education-industrial complex game from the safety of his foundation, he has absolutely no responsibility to taxpayers and those who fund public education.
LikeLike
His tax avoidance scheme for which he spends money to make money and gathers “big man” credit for his largesse. Meanwhile in America our infrastructure is collapsing because it might cause a moment of discomfort to tax people in order to pay for public services.
LikeLike
“He’s using his own money, so he makes the rules….He’s trying to make a difference.”
Oliver North! Where have you been all this time?
Gates is an effective shill for Monsanto and Pharma. (Come to think of it, they use their own money, too. Guess they make their own rules as well.)
But all this is quite beside the point.
You make a category error. Gate’s Machiavellian mix of noblesse oblige and the Divine Right of Kings…er…Billionaires is not what Cathy’s post is about.
Cathy is exploring the validity of his methods, not the merit of his goals.
LikeLike
Our most primitive form of measurement is voting. The candidate who talks about what’s most important to voters ultimately wins the election. However, as politics have evolved, the science of managing the debate has developed. As the technology to measure things evolves, the science of managing the debate will find its way into the process as well.
One important thing to remember about educational reform is that the most successful reforms of today are tomorrow’s average and below average methods. And even the Gates Foundation, with its $38 billion, is a small player compared to all public expenditures on local education through time.
A strategy for managing the problem of what to measure is to save everything you can measure. Although it’s costly, the historical data in its most raw form is really the only way to preserve measurements that cannot be conceptualized now.
LikeLike
I think Gates sees “school reform” as just another way for him to get a lot more money. One point none of the charlatans want to address is that we don’t know how to collect the data that shows what makes effective teaching. My opinion is that the push for standardized testing is pretty obviously a scam, designed to take public money from where it might do some good and put it in the pockets of people who don’t really care what the outcome is. I think it’s actually harmful. We have seen one model that works quite well — Finland. It requires paying teachers enough to attract capable people. The current reliance on idealism worked before student loans became debt peonage (thanks, “Bankruptcy Reform Act of 2005”). Half the country actually don’t want better schools; they want schools that will teach Creationism and that this is (and always has been) a Christian country.
LikeLike
Hi Wathbabe,
Today’s post was particularly well done. Good work!
I wonder if you know which Aunt Pythia submissions are from me…? 🙂
Love you, Becky
LikeLike
Hey! Thanks!
I had no idea you ever submit questions! I need more!! 🙂
Love you too,
Cathy
LikeLike
Great piece. I’ll share with my political peeps.
LikeLike
I’m confused. Would you prefer that Gates *not* think this way?
Keep in mind that most charitable organizations don’t think this way at all. As examples, “fair trade” systems usually don’t benefit the farmers they are nominally intended to support, and early education programs generally have their benefits erode once the student has been in a public school for 1-2 years.
So, yes, Gates gets tremendous power by setting the model. For a guy trying to be an effective source of charity, don’t we *want* him to have tremendous power?
LikeLike
Actually, “fair trade” programs usually do benefit the farmers they are intended to support, though less than straight-up price supports (I’ve read through the studies purporting to claim that fair trade programs are counterproductive, and their data doesn’t support their conclusions), and early education programs have benefits which are still measurable by high school.
So the lesson of your comment is: there are people with agendas who spread dishonest propaganda. Or, “figures don’t lie, but liars can figure”.
LikeLike
No, we absolutely DO NOT want Gates or anyone like him to have tremendous power. It should not be a question of power in the first place. Think of it this way. Mathbabe is making the argument that right makes might, and you (and Gates et al) are making the opposite argument, that might makes right. The implicit idea that wealth is a measure of wisdom, ethical behavior and morality in all things is absurd on it’s face. One need look no further than Gates to see the failures of the application of opinion buttressed by wealth alone as a substitute for well constructed policy.
LikeLike
> Our most primitive form of measurement is voting. The candidate who talks about what’s most important to voters ultimately wins the election.
NotRelevant, if only that were true. The political process in the US is broken on many counts, that goes way beyond “managing the debate”. Media access is a huge concern, which mostly (but not entirely) comes down to money. Also, the two parties have the collective, de-facto, extra-constitutional power to shut legitimate, popular viewpoints out of politics that they don’t like. And this leaves the rich and the powerful the ability to simply buy off both sides of the isle.
A lot of what you hear out of other people is no accident, but rather a result of deliberate misinformation. I remember hearing a friend who had just got a job selling houses during the bubble gushing about the financial benefits of owning a house and how houses were an investment that always go up. Or, recently I made a late-night trip to an otherwise deserted sandwich shop, and accidentally got involved in a short, uncomfortable political chat. The lone worker was angry about the economy, angry about the federal deficit, angry at high taxes, angry about cuts to Social Security, angry about cuts to the military, and angry at Obama. Now, there are plenty of things to be angry about, take your pick, but that’s a completely nonsensical set of issues to be upset about.
Now, I’ve been involved in some of the nitty-gritty details of politics enough to know full well that neither of these chance occurrences were truly random. They are deliberate pieces of misinformation placed into our culture, designed to sell houses that people couldn’t afford, and to keep people angry and distrustful of government and to keep attention off of those who really deserve it.
LikeLike
IHME is the institute funded by Gates family that focuses on problems related to global health. Although Bill Gates’s speech may sound naive I beleive the institute itself is very well aware of the bias Cathy is pointing to.
LikeLike
Fan of your blog since day 1 here. Apologies for the wall of text. I used to be a teacher in states with lots of testing (but pre-VAM), and now I am a modeler who works for the polio eradication campaign.
tl;dr: The polio folks have the benefit of something resembling a ground truth to test against: people either have polio or they don’t. This makes modeling polio fundamentally less crappy than teacher assessment.
The editorial paints too pretty a picture, agreed. But for context, the current situation in public health is often about going from doing no data analysis beyond intuition and staring at a spreadsheet to doing some actual math, or from patting folks on the back based on a crappy number to acknowledging that the number is crappy, and maybe, maybe trying to make it better.
The people Gates (and importantly and increasingly oftenly, the CDC, WHO, and country partners) turns to for information in the polio community are pretty open about the issues you bring up around the modeling death spiral. We put a lot of effort into breaking our own and each others’ models, understanding incentives that bias the data, doing fundamental science both in math-land and at various stages of the biology, and seeing if we can actually predict the whens and wheres of polio.
A lot of the work is done collaboratively with in-country partners who do the real work but can’t crunch the numbers they need for support. That is not yet the norm for all the international organizations involved, but openness is becoming increasingly common. The smallest number I’ve seen quoted about the number of people involved in the eradication campaign is 50,000, which is really cool!
You also allude to the controversy about focusing on polio to the possible detriment of more pressing diseases. It’s not often said for money reasons, but Gates is bringing a singular focus to the polio eradication party because it will either succeed in a game-changing way or it will fail. Much of the work in global health is focused on incremental change over decades. The push for worldwide polio eradication, formalized in 1988, had stalled by the early 2000’s. Under the incremental improvements model, stalling and then failing has been the norm (malaria, HIV, TB). Beyond eradicating polio, the goal is to demonstrate that we didn’t just get lucky with smallpox and that fundamental, qualitative change on finite timescales is possible without having to incrementally address all public health failures. That might not be true, but it hasn’t really been tried for 40 years.
All that having been said, I can’t speak for why Gates is so committed to teacher assessment with value-added models when the whole thing stinks and is maybe worse than doing nothing. Speaking as a scientist, the nicest thing about polio is you actually get to observe (imperfectly) the quantity you care about: when and where people are infected. Everything we do has to be testable and verifiably predictive. This is as different as can be compared to teacher evaluation, where the thing best measured by a standardized test is socioeconomic status.
Having spent some time on the professional side of education, I assume he couldn’t find any non-pseudoscientists to work with. In contrast, I like to think it matters that the public health modeling community is made of up of disease experts and physicists, mathematicians, and statisticians who have actively chosen not to work in finance. It might tell you something that I’m betting this stage of my career on it being easier to eradicate polio in Africa and Asia than to participate in sensible education reform in the US.
My last comment is that I agree it’s troubling that nation-state resources are in the hands of one person.
LikeLike
Thanks for clarifying something that has always troubled me.
LikeLike
5. Poor people are uninformed, or unrepresented ?
LikeLike
Wow, Tim, and others, way to miss the point.
First of all, I’m sure that someone as smart as Gates understands that he needs to get input from others in order to solve problems so this article is yet another piece of data that could help him in his endeavors.
Secondly, the point was that you can have all the money in the world, but you will NOT get data unless you have the political power to effect change.
That is, most problems in the world are not caused because people are too stupid or lazy to solve them, but rather because there are interests greater than themselves that stop them. This could be due to government intervention such as those who support monopolies, the creation of a caste system like the one we are building in the US, and the use of military power to destroy minority communities.
Thus, you can collect all the data you want, but unless you change political systems your data will not solve any problems the powers that be do not want solved.
I look forward to continued confusion and distortion of this simple idea.
LikeLike
Positive feedback creates an unstable system. You really want negative feedback for a positive result. 🙂
One person commented that it was Bill Gates’ money, he could do what he wanted to. Actually, it isn’t his money. He got it from a company that is a convicted monopolist (many times in many countries). It is really our money that we had to give to him (try buying a PC without Windows someday) for a product that was free before he came along. Nobody should have $50 billion to spend on their pet, erroneous projects. It skews results terribly. Better to let more people make a better living so that they can contribute to charities, too. Or, in the worst case, give it to the government and have potluck.
LikeLike
Perhaps we should shout a bit more from rooftops about how actively counterproductive the Gates Foundation is where education policy is concerned. Their main thesis seems to be that public schools need to be as similar to humongous corporations as possible. In particular, their goals include subjecting teachers to magic evaluations that determine raises and firing, and increasing “throughput and efficiency” by lowering spending per student and increasing class sizes. These ideas have been tried many times by many organizations, and any perceived positive effects were indistinguishable from noise.
It’s not likely to stop anytime soon. Ultrarich education reformers rarely question their prejudices, even after so many of their expensive projects end up as miserable failures.
LikeLike
This is my rooftop! And I shout as much as I can. But please join me.
LikeLike
A quote from the article from bgates’ mouth himself:
“In the past year, I have been struck by how important measurement is to improving the human condition. You can achieve incredible progress if you set a clear goal and find a measure that will drive progress toward that goal—in a feedback loop similar to the one Mr. Rosen describes.”
Please note the use of the words “In the past year”. To me, that strongly suggests that he didn’t use measurement before, or at the very least, discounted its importance to process improvement.
I find it hard to believe that anyone could not see the importance of using measurement (I _am_ an engineer), but find it ironic in the case of bgates, the richest man in the world. Apparently, there are other methods that can be used to amass unheard-of wealth.
In the early 90s, Microsoft hired away the VP of Management Systems Division at Procter & Gamble. He would be called the CIO at any other company. Measuring operating performance was a mania back then at P&G. Everybody was rated on metrics. One of the shop floor sayings back them was “Any stupid decision can be traced back to a VP’s bonus.” So bgates should have been exposed to those concepts 20 years ago.
One of the sayings back then was that every system does exactly what it is designed to do. Setting metrics will get you better metrics, but usually at the expense of something that is not being measured. Be careful what you ask for!
My prejudice tells me that bgates does what he wants, and everybody else scrambles to make reality adjust to him. I’ve worked for CEOs like that before, just not on that scale.
What Gates misses is that some things are science and some are art. Even highly technical jobs can hold a lot of art. (No pun intended.) I have worked as a troubleshooter many times, and while I operated on a base of science, the actual troubleshooting involved a lot of art. Perhaps even math professors recognize that math is a science, but extending that science drifts into the realm of art frequently. How do I steer my investigations? What path do I take now? How do these unrelated items relate to each other?
Managing people, in particular, I have found to be mostly art. I have also been an industrial instructor, and have done training at many jobs, and that requires a fine eye to people’s subjective reactions to the subject matter. Most people won’t admit, or sometimes just don’t realize that they don’t understand the material.
Enough rambling. Hope I was clear enough.
LikeLike
Good points. College educated, middle class people are easily manipulated on the issue of education (has something to do with status insecurity, interesting literature on this). So “education reform” is largely misguided liberals who have been sold on an “education crisis” and right-wing foundation money aimed to erode public schools and teacher’s unions, largely for ideological reasons. There is no education crisis (when filter out poor schools, U.S. schools do adequately in transnational comparisons) but rather there are poverty and inequality crises. Even if there were such a crisis many of the proposed solutions (NCLB, VAM, charter schools) are highly questionable fixes (which only confirms the idea that “reform” agenda is driven by ideology).
LikeLike
Whoops, was responding to S. Carnahan.
LikeLike
Bill Gates and the “education reformers” all favor and have unquestioning faith in unregulated capitalism and they want public education to play by those rules. Instead of measuring profits, though, they want to use things like VAM to “measure” school and teacher performance… and they have convinced politicians that simplistic metrics like VAM are superior to painstaking observations and benchmarking in multiple areas because those comprehensive approaches would require more staff and, therefore, more costs. Public education is a public good, it cannot operate under the system of unregulated capitalism if we hope to achieve an equal opportunity for all students.
LikeLike
@wgersen: in practice, public schools are not making “painstaking observations and benchmarking in multiple areas”. I’ve been moderately involved in American public schools, and I found teachers to have negligible oversight. Unless you commit a felony, you can do most anything you want in your own classroom.
I am sure those of us who went through the public schools were aware of any number of teachers that just shouldn’t be there. Did you ever stop and wonder how they keep their jobs year after year?
LikeLike
I was a public school superintendent for 29 years. I see two major factors that mitigate against painstaking observations and benchmarking in multiple areas and the dismissal of teachers: money and local politics. Having staff to do more observations requires more administrators and/or more master teachers… which means more money. Benchmarking in multiple areas requires someone to collect the data and present it to administrators in a usable and timely fashion— which requires either staff or contracted assistance… which means more money. The “reform” crowd doesn’t want to face the music on the need for more money to do the job right so they favor reductionist pencil and paper exams and arcane mathematical manipulations that the Mathbabe accurately characterizes as a random number generator. And the local politics isn’t just push back from unions… Having finessed the resignation of an experienced elementary teacher “everyone knew was bad” I was astonished to find a school board meeting packed with parents, young adults, and college students who told of glowing experiences they had with this teacher… your favorite teacher was despised by some… and the teacher you think “just shouldn’t be there” has a following of beloved students… Long story short: oversight and benchmarking require money and teacher dismissals require local political will on the part of the Principal, Superintendent and board… Not easy!
LikeLike
Indeed. :sigh: Don’t get me started on the way school boards generally reward administrators for NOT having any oversight of teachers or junior administrators, while punishing them for doing their job.
LikeLike
Hey mathbabe, here’s a link I’m betting a six pack of my favorite ale that you will really like. Rhetorical bet, FYI, & Thanks! BTW, everyone else should read and save it for future use, it explains a lot about why management has it’s head where the sun never shines. http://www.auroraadvisors.com/articles/Webber-Metrics.pdf
LikeLike
Love it!
Wait, does that mean I owe you beer?
Cathy
LikeLike
That was agreat writeup. If you liked it you might like this book or at the very least you’ll find it intriguing http://www.amazon.com/Obliquity-Goals-Best-Achieved-Indirectly/dp/0143120557
LikeLike
Thank you, a nice set of counterexamples.
“We know only too well that facts don’t speak for themselves: They have to be constructed through legal processes and kept transparent. They have to be defended, too.”
Hernando de Soto
I can’t tell if Gates is naive or pursuing some other agenda.. it seems implausible that he’s not aware of the problems with the education ‘reform’, but there doesn’t seem to be a clear ulterior motive. It’s true his skills are business rather than technical/mathematical, but even so.
In my experience of being subjected to them, metrics used to manage always produce a reality-distortion field: the efforts of those managed are squandered on getting good numbers on the metrics, rather than good performance as such. aka Goodhart’s law, ” Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”
LikeLike
I don’t agree. Bill Gates tries to convey very much with very little communication. The point made in this article is valid, but Gates doesn’t argue that data is infallible.
For me it looks as if it is much easier to improve bad data collection than it is to revise wrong or harmful attitudes without any data.
Another good example: According to pretty much all the data you can find, the world is getting ever more peaceful and wealthy. Yet, noone seems to notice. There is an interesting TED talk on this by Steven Pinker. Not to say we shouldn’t do more against poverty and violence.
LikeLike
Nassim Taleb has roundly refuted Pinker’s premise as pseudo science…
LikeLike
You are apparently not aware that crash-test dummies simulate men, women, three ages of children, and pregnant women. The days of male only crash test dummies are long gone. Even a cursory search of Wikipedia would tell you this and a further 5 min would have given you more information. This is an easily checkable fact, the fact that you did not take the time to look it up sort of undercuts your credibility as it shows you took a “well known fact” that was wrong and put it out as truth.
LikeLike
Don’t be ridiculous, just because some people have pregnant women crash test dummies doesn’t mean that the industry represents them fairly.
LikeLike
If you’re going to ignore the information Dennis posted regarding crash test dummies, don’t accuse HIM of being the ridiculous one.
LikeLike
She didn’t accuse him of being “the ridiculous one”, she was brushing off his thrust of quibbling about an example so he can discredit her argument. Her “Don’t be ridiculous” is an counter thrust, not an ad hominem attack, which your comment likes suspiciously like.
LikeLike
I am deeply impressed how Bill Gates is able to frame the privatization of the US education system and the opening of new markets in Africa for Monsanto as charitable acts. He is a genius! George Soros is fiddling with trivialities compared to that.
LikeLike
“Gates describes a positive feedback loop when good data is collected and acted on”
While it is important to be very careful about how data collection is done, which is the focus of your article, I think the real driver behind this quote is the data must be “acted on”.
While the Gates Foundation may not be using perfect data, B.G. and others are probably very intelligent people who are using at worst ‘okay’ data. What separates the Gates Foundation in my mind is their willingness to get out there and get stuff done. Was the polio campaign the best use of resources? That’s a matter of opinion. Nearly eradicating the disease, however, is pretty awesome, regardless of your opinion. Maybe I’m naive in my own way, that’s okay too.
LikeLike
Thank you for this little gem of wisdom. You could extend your arguments to many other domains, for instance aptitude tests. Our society has a ridiculous tendency to measure everything without pausing to ask what exactly is being measured. Too many PhDs with student loans to repay? And what does not help is the unfortunate tendency to consider anyone with a bit of weath as a fount of wisdom.
LikeLike
I think you missed the entire point of his article. His point wasn’t that “all data is objective/equal/whatever”, but rather, that if you don’t set a goal and define some way of measuring whether you’ve reached that goal or not you’re just flopping around in the dark with no idea how you are actually doing. Furthermore, that choosing an important goal, measuring progress regularly, and improving data collection is good way to achieve it. Having said that, he still recognizes the importance of collecting good data:
“Yet while measurement is critical to making progress in global health, it’s very hard to do well. You have to measure accurately, as well as create an environment where problems can be discussed openly so you can effectively evaluate what’s working and what’s not. ”
He continues: “Ethiopia’s recent effort to monitor the progress of its immunization program is a good example of learning from data and-the hardest part-using data to improve delivery of the right solutions. A recent national survey of Ethiopia’s vaccination coverage reported vastly different results from the government’s own estimates. Ethiopia could have ignored this conflict and reported the most favorable data. Instead, it brought in independent experts to understand why the measurements were so different. They commissioned a detailed independent survey that pinpointed geographic pockets of very high coverage-and very low coverage. The government is now working to develop better plans for the poorer performing regions.”
He clearly recognizes that you have to define your goals properly and that the collection of accurate and meaningful data, and then using that data, is the hardest part.
LikeLike
Mathbabe…. I have loved your take on things for a while but saying Bill Gates is naive is quite over the top… Nassim Taleb coined words for that kind of statement… “Epistemic Arrogance”… I know of no one who is completely immune to that state of mind…
LikeLike
Just a side note, as a sometime engineer and sometime guitar player, a “positive feedback loop” is one where the differences at each cycle of the loop are increased. This does not lead to stability – it leads to screaming guitars though! 😉
LikeLike
Just to further quibble, as an engineer trained and now English teacher; perhaps Cathy was stringing together “positive” words leading to a “positive” goal. Her sentence, “…describes a [1] positive feedback loop when [2] good data is collected and [3] acted on…” could be interpreted as words 1 and 2 give the reader an impression of s/th good being done so the word 3 would, logically in the way feelings are communicated in written English, be a good thing as well. (Yes, you are correct in the engineering sense, but one could say that her writing is for a non-engineering audience, but hey, I’m just jealous that you saw the feedback loop thing before me…)
Ah, the joys of combining the how-to-ness of engineering with the prickliness of a grammarian!
LikeLike
look you people,
if Bill Gates was so keen on acting on solid data, then he would have stopped supporting GM food and Monsanto a long time ago. There is no evidence that GM crops provide benefits and a massive amount of data showing that GM crops and technology are really terrible for bio diversity and farmers and nature in general (monucultures etc).Bill Gates has all of this “measured data” but choses to ignore it.
Bill is a lying Bill-ionaire.
LikeLike
I do think you make good criticisms. Teacher rating systems are so weak that I have really low confidence that this is a good avenue to improving school outcomes. To make education better, we’d have to develop something closer to a science of how people learn –something less problematic than the way social science is commonly done, something that involves experimenting with really varied educational techniques. And we’d have to take homeschooling seriously, and ask why public school kids can’t outperform homeschoolers on tests. It’s nuts to take an arbitrary (and tautological) metric and optimize for it.
I think, however, that folks like Gates *have* to make a simplistic case for data-driven decisionmaking, because there’s resistance to the basic notion that you should count things. Credentialed experts (doctors, teachers, pundits, everyone) tend to be resistant to the possibility that they could be partially replaced by an algorithm. Established interests don’t want to let go of their fiefdoms.
Being a data scientist, you’re probably surrounded with people who love data and, if anything, you have to fight against data triumphalism. But in most of the world, you get a lot of pushback. Quantitative folks who work with doctors have to fight to get them to accept that “no, conventional practice isn’t supported by the evidence here.” Folks who work in efficient charity, like GiveWell, encounter a lot of resistance for the simple idea that people should give to the charities that have been shown to work best.
Gates is trying to do charitable work with a little autonomy. If you want the freedom to innovate, you have to defend the notion that a smart interloper can look at the facts, reach his own conclusions, and successfully compete with established institutions. The data triumphalism is a way of saying, more or less, “If you can count, you can improve the performance of anything.” It’s a strategic and political move, it’s a mandate for technologists/quantitative folks to take over any field they want to. And it’s more or less accurate; statistics give you tremendous optimization power. (Whether it optimizes for what you want is another question.) I’m not sure whether, behind the scenes, I’d be impressed or disappointed by Gates’ level of rigor. But I do think what he’s doing makes sense. If I wanted to have an impact on the world, and I were in his position, I’d absolutely be making the case for “data-driven” decision-making, because that’s the way you get legitimacy these days to go into a field like global health or education and shake up the status quo. Sometimes the status quo is dismal enough that this makes sense; whatever fixes American education, it certainly won’t be more of the same.
LikeLike
I am normally a critic when commenting here, so my great praise of this essay should have more weight 8)
Another POV, set of insights, that enhance our ability to understand the very complex systems that we live within.
You should also have labeled Gates ‘just another oligarch’, and analyzed his recommendations from the POV of how they affect his fortunes, but otherwise, perfect.
LikeLike
We also see a situation in China where a simple amine test for protein causes melamine to be added to milk and pet food… Causing children to die in China, and animals to die in the US.
When you can assume that companies will game a test for profit, you have to blame those wo set up the test/model and collect the data for theose deaths.
LikeLike
Cathy, I think you are wrong. I am no Gates fanboy, but collecting data is fairly important because without data problems either don’t have visibility or attempts to solve it will likely target the wrong aspect of the problem. Worse yet, without some kind of data the program won’t have a way to evaluate whether it should change to improve over time. And as we all know, initial solutions are always wrong, one just can’t design an adequate solution without iterating over and over again.
Take for example the problem of trauma in Africa — anybody ever visiting Africa and seeing the reality there knows that this is a huge issue. And yet, you talk about trauma prevention or provider training and people will immediately say no, HIV is a problem not trauma. Sure you need to politicians to act upon data, but that is not a reason not to collect data. Actually, no matter how screwed up a give political system might be, in the absence of data and the corresponding pressure to fix the problem politicians are even less likely to act upon a problem.
LikeLike
I absolutely agree that it’s important to collect relevant data if you want to solve a problem. But when you say “in the absence of data and the corresponding pressure to fix the problem politicians are even less likely to act upon a problem,” I’d counter that “when politicians decide not to fix a problem, they often do not allow good measurements of that problem to occur.”
My uncle was the ambassador to a small African nation. At some point an enormous amount of drugs, I believe for cholera, were given to that nation for humanitarian purposes in anticipation of an outbreak. When that outbreak occurred, which it did, the medicine was gone. There were no records of where it had gone. The data was inexplicably missing. That’s the kind of data that the government didn’t decide to keep when it sold the medicine.
LikeLike
It’s a feature, not a bug.
LikeLike
Thanks Cathy. One of your best so far. I hope you write some more about the Gates foundation’s metrics specifically as well as the general caveats & warnings.
LikeLike
The main point of the education establishment is that they don’t want to be measured at all. They maintain there is no ‘better teaching’ or ‘worse teaching’ – it’s just important that teachers care and their intentions are good. To the rest of us they say: “just shut up and give us more money”.
LikeLike
I found it interesting that Gates and his wife could easily observe, not determine by looking at data, that the teacher he was observing was a master teacher: “She engaged her students, walking among them and eliciting great participation. We could see why Mary Ann is a master teacher…”
LikeLike