Archive
Climate Convergence march on Sunday
This Sunday there’ll be a huge march to raise awareness about climate change. It’s called the Climate Convergence, and the Alternative Banking group is going to be there.
If you want to join us, come to 79th and Central Park West at 11am, in front of the Natural History Museum. We will have signs and a banner. See you there, I hope!
Why the NFL conversation about Ray Rice is so important to me
My first memory is of my father throwing a plate of eggs at my mother’s head, like a frisbee. My mother had to duck to get out of the way, and the plate exploded on the wall behind her. His eggs hadn’t been cooked well enough, and this was his way of expressing that to my mother, who had cooked them. Then he punched his hand through a glass window. Blood and glass fragments were everywhere. I was 4 years old. I remember running to my bed and crying, and the already familiar feeling of hiding in fear.
My mother was a battered woman who didn’t leave her abuser. And that meant a bunch of things for her and for me and my brother. I cannot explain her reasoning, because I was a small child when most of the abuse occurred. But I can tell you it’s common enough, and it’s not even that hard to understand.
One of the aspects of this decision – to stay with your abuser or not – that I haven’t been hearing a lot of recently, in this whole Ray Rice-inspired nationwide conversation about violence against women, is the economics of it. The worst of my father’s behavior happened when he was unemployed and desperately unhappy with how his life was turning out. Once he got on his feet again he didn’t take stuff out on his wife as much or as often. I imagine that is typical, but what it means is that it’s extra hard to imagine managing a second household, with small children, on one salary, when it’s already a huge struggle to manage one. The economic reality of leaving your husband has to be understood.
Even so, the abuse didn’t completely stop, and it’s not like my mother never considered leaving my father. I remember I went away for a month, to communist Budapest, when I was turning 13, the summer of 1985. When I came back my mother told me that my father had pushed her down the stairs. Then she asked me if she should leave him. I said yes, but then she didn’t do it.
I will probably never really forgive her for asking me that, for putting that kind of responsibility on a child like that, and then not following through. Especially now that I have kids of my own that age, it seems outrageous to put that kind of decision on their plate, or even seem to. It was my last day of childhood, the day I realized there were no responsible people in my family, and that I would have to step up and be the person who negotiated reasonable boundaries or, failing that, call the cops. From then on I was my mother and my brother’s protector.
If anyone ever asks me why I am not intimidated by anyone, I think of that moment. When you are a 13-year-old girl who has decided to stand up for your mother and brother against a large and very strong man, who often becomes an enraged and unreasonable bully, you forget about fear and intimidation, because it’s just something you cannot think about.
—
Many years later, after I left college, my father engaged me in a series of ritualized revisionist history lessons. Every Christmas, every Thanksgiving, maybe even on July 4th, he would bring up the bad old days and he’d mention how much I’d hated him when I was a teenager, and how he hadn’t deserved it, and how even when he’d been abusive to my mother, she had hit him first, and he hadn’t really wanted to do it but there it is. He often distorted facts, and he never explained why he was doing this.
It always sounded so bizarre to me – how could it matter that my mother had hit him first, not to mention that it was unbelievably hard to imagine? How could that be an excuse for what kind of fear and rage he had manifested on her body and on our family for so long? Answer: it isn’t an excuse.
It was very confusing, these inaccurate family history lessons in sermon form. It made me so angry I never could do anything except stay silent. I didn’t even correct him when he lied about the details, because he was evidently saying all of this more for him than for me.
It took me years to figure out why this conversation kept happening, but I think I finally know now. He was working through his guilt with me as his chosen audience. He was, in a sense, asking for my forgiveness. I never gave it, but what those conversations did accomplish for him was almost the same: he made it my problem for being so unkind as to not forgive him. After all, my mother had forgiven him, why couldn’t I? Looking back, I felt increasing pressure to forgive, but I never gave in. I didn’t even really know how.
Here’s why I’m thinking about this now. This Ray Rice and Adrian Peterson conversation, which I’ve been listening to on sports radio, has gotten me to thinking about this stuff. I am listening to these football guys, these pinnacles of macho masculinity, talking about men who abuse women and children, and describing it as unforgivable. Thank god for those men.
Because here’s the thing. It is unforgivable, but until now I hadn’t realized that I was allowed to think so. I’ve been feeling so guilty for so long at not being able to forgive my father, I never realized that I could just be okay with it. But now I do, and I don’t forgive him, and I never will.
—
After much deliberation, I’ve finally decided to publish this. To be clear, I’m not doing so to hurt my father or my mother. I’m writing it in hopes that by reading this, people will realize that this kind of thing happens everywhere, to all kinds of people, and that it’s always fucked up and wrong. We need to know that, the NFL needs to know that, and policy makers need to know that. We need to create stronger laws around this, that don’t buckle when the women refuse to press charges.
If this happened to you as a kid, it wasn’t your fault, and you don’t have to forgive if can’t or you don’t want to, and even if you don’t forgive them, you will probably still love them. Human beings are really good at conflicting emotions. Focus on not being like that yourself. My proudest accomplishment is that I have not perpetuated the cycle of violence on my own family. And good luck.
The green-eyed/ blue-eyed puzzle/ conundrum
Today I want to share a puzzle that my friend Aaron Abrams told me a few days ago. I’m sure some of you have heard it before, but it’s confusing me, so I’m asking for your help.
Set-up
Here’s the setup. There’s an island of people, all of whom have either blue eyes or green eyes. By social convention they never discuss eye color, because there’s a tragic rule that states that, if you ever figure out your eye color, you have to leave the island within 24 hours. Oh, and there are no mirrors.
OK, get it? So think of the island as pretty small, maybe 100 people, so you know everyone else’s eye color but not your own.
Here’s what happens next. Some castaway arrives by swimming onto the island, stays for a few days and hangs out with the folks there eating island food and having island parties, and then after building himself a boat he prepares to leave. Not being trained in the social customs of the island, on the day he leaves he says, “hey, it’s good to see some people with green eyes here!”.
Puzzle
So the puzzle is, what happens next?
Here’s what’s obvious. If you are a person who only sees blue eyes, you know by his statement that you must have green eyes. So you have to leave the next day.
But actually he said “some people.” So even if you only see one other person with green eyes, then you have to leave, with that other green-eyed person, after one day.
With me so far?
But hey, what if you see two other people with green eyes? Well, you might think you’re safe, and you’d wait to see them leave together the next day. But what if they don’t leave after one day? That must mean that you also have green eyes. Then all three of you have to leave, after two days. Get it?
Then you work by induction. If you see N other people with green eyes, they should all leave after N-1 days, or else you have green eyes too and all (N+1) of you leave after N days.
Conundrum
OK, so here’s the conundrum. The guy who started this whole mess really didn’t do much. He just stated what was obvious to everyone already on the island, namely that some people had green eyes. I mean, yes, if there were really only 2 people with green eyes, then he clearly added real information, because both those people had thought only 1 person had green eyes.
But just for the fun of it, let’s assume there were 17 people with green eyes. Then they guy really didn’t add information. And yet, 16 days after the guy left, so do all the green-eyed islanders. So really the guy just started a count-down more than anything.
So, is that it? Is that what happened? Or was the original set-up inconsistent? Is it not an equilibrium at all? Or is it an unstable equilibrium?
Saying
In any case, Aaron and his friend Jamie have developed a saying, it’s a green-eyed/ blue-eyed thing, which means it’s an apparently information-free fact which changes everything. I think I’ll use that.
Christian Rudder’s Dataclysm
Here’s what I’ve spent the last couple of days doing: alternatively reading Christian Rudder’s new book Dataclysm and proofreading a report by AAPOR which discusses the benefits, dangers, and ethics of using big data, which is mostly “found” data originally meant for some other purpose, as a replacement for public surveys, with their carefully constructed data collection processes and informed consent. The AAPOR folk have asked me to provide tangible examples of the dangers of using big data to infer things about public opinion, and I am tempted to simply ask them all to read Dataclysm as exhibit A.
Rudder is a co-founder of OKCupid, an online dating site. His book mainly pertains to how people search for love and sex online, and how they represent themselves in their profiles.
Here’s something that I will mention for context into his data explorations: Rudder likes to crudely provoke, as he displayed when he wrote this recent post explaining how OKCupid experiments on users. He enjoys playing the part of the somewhat creepy detective, peering into what OKCupid users thought was a somewhat private place to prepare themselves for the dating world. It’s the online equivalent of a video camera in a changing booth at a department store, which he defended not-so-subtly on a recent NPR show called On The Media, and which was written up here.
I won’t dwell on that aspect of the story because I think it’s a good and timely conversation, and I’m glad the public is finally waking up to what I’ve known for years is going on. I’m actually happy Rudder is so nonchalant about it because there’s no pretense.
Even so, I’m less happy with his actual data work. Let me tell you why I say that with a few examples.
Who are OKCupid users?
I spent a lot of time with my students this summer saying that a standalone number wouldn’t be interesting, that you have to compare that number to some baseline that people can understand. So if I told you how many black kids have been stopped and frisked this year in NYC, I’d also need to tell you how many black kids live in NYC for you to get an idea of the scope of the issue. It’s a basic fact about data analysis and reporting.
When you’re dealing with populations on dating sites and you want to conclude things about the larger culture, the relevant “baseline comparison” is how well the members of the dating site represent the population as a whole. Rudder doesn’t do this. Instead he just says there are lots of OKCupid users for the first few chapters, and then later on after he’s made a few spectacularly broad statements, on page 104 he compares the users of OKCupid to the wider internet users, but not to the general population.
It’s an inappropriate baseline, made too late. Because I’m not sure about you but I don’t have a keen sense of the population of internet users. I’m pretty sure very young kids and old people are not well represented, but that’s about it. My students would have known to compare a population to the census. It needs to happen.
How do you collect your data?
Let me back up to the very beginning of the book, where Rudder startles us by showing us that the men that women rate “most attractive” are about their age whereas the women that men rate “most attractive” are consistently 20 years old, no matter how old the men are.
Actually, I am projecting. Rudder never actually specifically tells us what the rating is, how it’s exactly worded, and how the profiles are presented to the different groups. And that’s a problem, which he ignores completely until much later in the book when he mentions that how survey questions are worded can have a profound effect on how people respond, but his target is someone else’s survey, not his OKCupid environment.
Words matter, and they matter differently for men and women. So for example, if there were a button for “eye candy,” we might expect women to choose more young men. If my guess is correct, and the term in use is “most attractive”, then for men it might well trigger a sexual concept whereas for women it might trigger a different social construct; indeed I would assume it does.
Since this isn’t a porn site, it’s a dating site, we are not filtering for purely visual appeal; we are looking for relationships. We are thinking beyond what turns us on physically and asking ourselves, who would we want to spend time with? Who would our family like us to be with? Who would make us be attractive to ourselves? Those are different questions and provoke different answers. And they are culturally interesting questions, which Rudder never explores. A lost opportunity.
Next, how does the recommendation engine work? I can well imagine that, once you’ve rated Profile A high, there is an algorithm that finds Profile B such that “people who liked Profile A also liked Profile B”. If so, then there’s yet another reason to worry that such results as Rudder described are produced in part as a result of the feedback loop engendered by the recommendation engine. But he doesn’t explain how his data is collected, how it is prompted, or the exact words that are used.
Here’s a clue that Rudder is confused by his own facile interpretations: men and women both state that they are looking for relationships with people around their own age or slightly younger, and that they end up messaging people slightly younger than they are but not many many years younger. So forty year old men do not message twenty year old women.
Is this sad sexual frustration? Is this, in Rudder’s words, the difference between what they claim they want and what they really want behind closed doors? Not at all. This is more likely the difference between how we live our fantasies and how we actually realistically see our future.
Need to control for population
Here’s another frustrating bit from the book: Rudder talks about how hard it is for older people to get a date but he doesn’t correct for population. And since he never tells us how many OKCupid users are older, nor does he compare his users to the census, I cannot infer this.
Here’s a graph from Rudder’s book showing the age of men who respond to women’s profiles of various ages:
We’re meant to be impressed with Rudder’s line, “for every 100 men interested in that twenty year old, there are only 9 looking for someone thirty years older.” But here’s the thing, maybe there are 20 times as many 20-year-olds as there are 50-year-olds on the site? In which case, yay for the 50-year-old chicks? After all, those histograms look pretty healthy in shape, and they might be differently sized because the population size itself is drastically different for different ages.
Confounding
One of the worst examples of statistical mistakes is his experiment in turning off pictures. Rudder ignores the concept of confounders altogether, which he again miraculously is aware of in the next chapter on race.
To be more precise, Rudder talks about the experiment when OKCupid turned off pictures. Most people went away when this happened but certain people did not:
Some of the people who stayed on went on a “blind date.” Those people, which Rudder called the “intrepid few,” had a good time with people no matter how unattractive they were deemed to be based on OKCupid’s system of attractiveness. His conclusion: people are preselecting for attractiveness, which is actually unimportant to them.
But here’s the thing, that’s only true for people who were willing to go on blind dates. What he’s done is select for people who are not superficial about looks, and then collect data that suggests they are not superficial about looks. That doesn’t mean that OKCupid users as a whole are not superficial about looks. The ones that are just got the hell out when the pictures went dark.
Race
This brings me to the most interesting part of the book, where Rudder explores race. Again, it ends up being too blunt by far.
Here’s the thing. Race is a big deal in this country, and racism is a heavy criticism to be firing at people, so you need to be careful, and that’s a good thing, because it’s important. The way Rudder throws it around is careless, and he risks rendering the term meaningless by not having a careful discussion. The frustrating part is that I think he actually has the data to have a very good discussion, but he just doesn’t make the case the way it’s written.
Rudder pulls together stats on how men of all races rate women of all races on an attractiveness scale of 1-5. It shows that non-black men find their own race attractive and non-black men find black women, in general, less attractive. Interesting, especially when you immediately follow that up with similar stats from other U.S. dating sites and – most importantly – with the fact that outside the U.S., we do not see this pattern. Unfortunately that crucial fact is buried at the end of the chapter, and instead we get this embarrassing quote right after the opening stats:
And an unintentionally hilarious 84 percent of users answered this match question:
Would you consider dating someone who has vocalized a strong negative bias toward a certain race of people?
in the absolute negative (choosing “No” over “Yes” and “It depends”). In light of the previous data, that means 84 percent of people on OKCupid would not consider dating someone on OKCupid.
Here Rudder just completely loses me. Am I “vocalizing” a strong negative bias towards black women if I am a white man who finds white women and asian women hot?
Especially if you consider that, as consumers of social platforms and sites like OKCupid, we are trained to rank all the products we come across to ultimately get better offerings, it is a step too far for the detective on the other side of the camera to turn around and point fingers at us for doing what we’re told. Indeed, this sentence plunges Rudder’s narrative deeply into the creepy and provocative territory, and he never fully returns, nor does he seem to want to. Rudder seems to confuse provocation for thoughtfulness.
This is, again, a shame. A careful conversation about the issues of what we are attracted to, what we can imagine doing, and how we might imagine that will look to our wider audience, and how our culture informs those imaginings, are all in play here, and could have been drawn out in a non-accusatory and much more useful way.
Aunt Pythia’s advice
Do you know what Aunt Pythia has been occupied with recently? Yes, you guessed it, she has a fantabulous new knitting pattern and she just can’t get enough of it. Here’s a recent work-in-progress pic:
IS THAT NOT GORGEOUS?!
I hope you know how much Aunt Pythia must love you considering how hard it was to tear herself away from such a beautiful project. So please, love her back, and after loving her madly, don’t forget to:
please think of something to ask Aunt Pythia at the bottom of the page!
By the way, if you don’t know what the hell Aunt Pythia is talking about, go here for past advice columns and here for an explanation of the name Pythia.
——
My dear Aunt Pythia propagating loving introspective nerdy girls,
This morning, I am going to imagine sitting in between your lovely kids to enjoy crepes and vent with you. Today’s vent is that I have been highly disturbed by this week’s coverage of the Fields medal (putting aside for the moment the question of whether the Fields medal should exist in the first place). One article I read compared being female in math with being a handicapped competitive athlete.
WTF? This is the news that is being reported and the way people are reacting? What is the most healthy way I can respond to this and still enjoy my Saturday morning crepes?
Love,
SINGing Introspective Nerdy Girl
P.S. I also read the following social media post of a male scientist: “I know I’ll get shit for this, but doesn’t it seem a bit weird that the first woman to win this is butch and wears men’s clothing? Is this because she has a man’s brain, or because she got chosen because she’s man-like?”
I’m not sure it would be a good idea to publicize this, but I would like to ask how I should respond in this situation (feel free to paraphrase the quote if you see fit). I would personally love to publicly shame the male scientist, but I also wanted to make sure I am responding in a way that is helpful and positive to anybody who is reading my message.
In case you are able to see his Facebook posts, the male scientist is “Brian Raney” at USC.
Dear SINGING,
Hmmm… not sure what I can add to this post about the topic, but here goes.
I guess the best way to think about this is as a totally non-mathematical PR thing, which is heavily steeped in weird and fucked up expectations due to historical sexism. As for the USC guy, it would obviously have been infinitely better for him to say something like, “Maryam was awarded the Fields Medal because she did some incredible stellar mathematics.” But there you go, some people miss opportunities to say the right thing. Or maybe he first said the right thing and then he added a bunch of other things after that, who knows. I don’t even care enough to check on his Facebook page. Who cares about what one random guys says?
As for overall butchiness and wearing men’s clothes, lots of female mathematicians do that (including myself many days!), and it’s actually not an uninteresting observation about women in math and other STEM fields, but the phenomenon is certainly not limited to Fields Medal winners.
If you don’t mind me going off into a slight tangent (thanks!), let me also mention that men’s clothes are, generally speaking, great for looking totally unobjectionable, not getting harassed or hit on, and not evoking catcalls (a big deal here in NYC!) compared to short skirts and high heels, and if men could wear them they totally would. Oh wait, they already do.
My point being, there are lots of reasonable reasons to wear men’s clothes besides being a lesbian (although being a lesbian is of course a great reason! And please include suspenders when possible! Fetching!). Being taken seriously as a scholar comes to mind. I defend everyone’s rights to trousers and a boring button-down shirt.
Or, you know, a short skirt and heels if you wanna sex it up and get some attention. Or for the more full-figured gal, a bodycon dress:
The key is to get what you want, when you want it.
Keep singing!
Aunt Pythia
——
Dear Aunt Pythia,
I’m a 24-year-old young woman in New York and I consider myself pretty lucky to absolutely LOVE my job as a “data analyst”. I make great money, my boss trusts me in a sort of crazy way, I can work remotely whenever I want to, and after 6 months, I’ve come to truly believe that my company is an awesome place to work and a pretty great group of people (I guess you could say I’ve been drinking the free chai tea + almond milk). Though I did balk for a second and wonder if I’m just a SQL database monkey, I’m proud to say that if I have to spend 1/7 of my day in SQL but get to spend the rest of it messing around with Python pandas and learning to be a command line ninja, give me a banana and call me Koko.
Now, I won’t have this autonomy forever. This is only my first job, and we’re rapidly expanding, which includes building out an ACTUAL data science department. Without going into too much detail, our platform currently delivers some basic analytics to our customers, and we want to beef up these metrics into something they value us for and, ideally, become dependent on.
We are hiring a director (read: a new boss for me) and we’ve interviewed a ton of people. As you’ve mentioned, a good data scientist is hard to find! I’m pretty outspoken and have spoken up about presenting our clients our with not-quite-as-accurate-as-I-myself-would-like metrics (and I drink chai tea here, not the kool aid). I think I could be a GOOD data scientist someday, but I need the right person to guide me. Most of these candidates are Google Analytics or Tableau jockeys who don’t have any interest in my sweet matplotlib graphs with opacity depending on client billing amount! circumference depending on length of time with us! and so forth.
Last week, I met a candidate that I KNOW will never be topped. She (SHE!!) is also outspoken, knows her shit, cares about data AND ALSO cares about stuff besides data (!) and just is certainly my perfect Yoda. Unfortunately, because the job market is a real thing and a good data scientist is hard to find, I fear that she will not take this job in favor of a better offer elsewhere, financially or otherwise (probably just a bigger company with more data than mine).
Aunt Pythia, HOW do I get her to choose my small company?? This feels to me like the kind of career-changing, perhaps even LIFE changing moment that you have to do EVERYTHING you can to make happen. What would you advise a young woman to do? I have scruples in life, but am not above planting bed bugs at the offices of her competing offer.
Most Enthusiastic Neophyte To Ever Enquire
Dear MENTEE,
You are seriously awesome and you don’t need a Yoda to tell you that, although we’d all love a Yoda.
“PATIENCE YOU MUST HAVE my young padawan”
Here’s the thing. I sense in you the power to be a great data scientist someday, not because your fave boss will or will not take that job, but because you have the obvious urge to do something cool and fun with your life, and because you have integrity, and because you are too smart to trick yourself into thinking what you’re doing is great when it isn’t. Trust in yourself. And if your company doesn’t hire someone awesome, go find yourself another job. Keep learning, keep striving.
Love always,
Aunt Pythia
——
Dear Aunt Pythia,
I am a junior mathematician just starting to navigate the depth of academia. I am so disillusioned by what I see. I thought being a mathematician was supposed to be this wonderful thing, wherein I exchange ideas with people of similar interests, make friends, and not working but playing.
Instead, I have met so many mean people, who hide what they are doing from me, some who ignore me because they don’t think that I’m good enough, and some who try to intimidate me. When I was a grad student, I even had a student by the same advisor, who never spoke to me once while we were students together, except to try to embarrass me during my talks.
While there are nice people in academia, and I still love being a mathematician, I sometimes become really sad about the mean people in academia. Sometimes, I feel so disillusioned and burned out, then I am too upset to be thinking about math. I feel that I would be so much more productive if only I could deal with these feelings, and I am often frustrated by the fact. Is leaving academia my only solution?
Disillusioned
Dear Disillusioned,
You are right on all accounts! You would be more productive if you could deal with these feelings, and people are mean, and leaving academia would help, although not in the way you think.
Here’s the thing. I left academic math in part because people were so mean. They were really mean to me, and especially because I was a woman, and especially because I was married to a man who was highly respected. It was a situation.
But after leaving academics, mostly what I’ve realized is how most places contain mean people, and academics are really not all that good at being mean. No offense to mean mathematicians! But really they are like, small-fry mean. If you want to see hugely assholic behavior, work in finance for a few years.
So I’m wondering if this might help – and it might not, of course – but if you can, engage in the following thought experiment: you have left academics, and you go into some other field, and people are mean there too, except for a few nice people with whom you can bitch about the meanies. Then you leave that job and go in search of another job, where maybe there are fewer assholes but also you don’t get paid as well and there are other problems that come up because of that, or because the job stability is rough, or etc. etc.
Then after that long thought experiment, you might realize that as long as there are resources to be fought over, there will be fights, and the question is how to ignore all the stupid bickering and get some math done, because after all math is beautiful and awesome and it’s not math’s fault that all these people are mean.
Good luck!
Auntie P
——
Dear Aunt Pythia,
My colleagues and I at the Militant Grammarians of Massachusetts would like to know why the word “data” is plural while the phrase “big data” is singular.
Your singular,
Big Datum
Dear Big Datum,
OK here’s where I am on this issue. It’s always singular. Always. Look at the data! All the data points to the same conclusion! There might be several data points that offer alternative preferences, but those are outliers. Every time I hear someone say something incredibly awkward like, “Are your organization’s data as clear as they can be?” I just wanna retch. Don’t do it. You just sound like a grammar nazi, and nobody likes those people.
You asked!
Aunt Pythia
——
Please submit your well-specified, fun-loving, cleverly-abbreviated question to Aunt Pythia!
What’s next for mathbabe?
The Columbia J-School program that I have been directing, The Lede Program in Data Journalism, has wound down this past week and in four days my 6-month contract with Columbia will end. I’ve had a fantastic time and I am super proud of what we accomplished this past summer. The students from the program are awesome and many of them are now my friends. About half of them are still engaged in classes and will continue to work this semester with Jonathan Soma, who absolutely rocks, and of course my fabulous colleague Theresa Bradley, who will step in as Director now that I’m leaving.
So, what’s next? I am happy to say that as of today (or at least as of next Monday when my kids are really in school full-time) I’m writing my book Weapons of Math Destruction on a full-time basis. This comes as a huge relief, since the internal pressure I have to finish this book is reminiscent of how I felt when I needed to write my thesis: enormous, but maybe even worse than then since the timeliness of the book could not be overstated, and I want to get this book out before the moment passes.
In the meantime I have some cool talks I’m planning to go to (like this one I went to already!) and some I’m planning to give. So for example, I’m giving a keynote at The Yale Day of Data later this month, which is going to be fun and interesting.
My Yale talk is basically a meditation on what can be achieved by academic data science institutions, what presents cultural and technical obstacles to collaboration, and why we need to do it anyway. It’s no less than a plea for Yale to create a data science institute with a broad definition of data science – so including scholars from law and from journalism as well as the fields you think of already when you think of data science – and a broad mandate to have urgent conversations across disciplines about the “big data revolution.” That conversation has already begun at the Information Society Project at Yale Law School, which makes me optimistic.
I also plan to continue my weekly Slate Money podcasts with Felix Salmon and Jordan Weissmann. Today we’re discussing the economic implications of Scottish independence, Felix’s lifetime earnings calculator, and the Fed’s new liquidity rules and how they affect municipalities, which my friend Marc Joffe guest blogged about yesterday.
Guest post: New Federal Banking Regulations Undermine Obama Infrastructure Stance
This is a guest post by Marc Joffe, a former Senior Director at Moody’s Analytics, who founded Public Sector Credit Solutions in 2011 to educate the public about the risk – or lack of risk – in government securities. Marc published an open source government bond rating tool in 2012 and launched a transparent credit scoring platform for California cities in 2013. Currently, Marc blogs for Bitvore, a company which sifts the internet to provide market intelligence to municipal bond investors.
Obama administration officials frequently talk about the need to improve the nation’s infrastructure. Yet new regulations published by the Federal Reserve, FDIC and OCC run counter to this policy by limiting the market for municipal bonds.
On Wednesday, bank regulators published a new rule requiring large banks to hold a minimum level of high quality liquid assets (HQLAs). This requirement is intended to protect banks during a financial crisis, and thus reduce the risk of a bank failure or government bailout. Just about everyone would agree that that’s a good thing.
The problem is that regulators allow banks to use foreign government securities, corporate bonds and even stocks as HQLAs, but not US municipal bonds. Unless this changes, banks will have to unload their municipal holdings and won’t be able to purchase new state and local government bonds when they’re issued. The new regulation will thereby reduce the demand for bonds needed to finance roads, bridges, airports, schools and other infrastructure projects. Less demand for these bonds will mean higher interest rates.
Municipal bond issuance is already depressed. According to data from SIFMA, total municipal bonds outstanding are lower now than in 2009 – and this is in nominal dollar terms. Scary headlines about Detroit and Puerto Rico, rating agency downgrades and negative pronouncements from market analysts have scared off many investors. Now with banks exiting the market, the premium that local governments have to pay relative to Treasury bonds will likely increase.
If the new rule had limited HQLA’s to just Treasuries, I could have understood it. But since the regulators are letting banks hold assets that are as risky as or even riskier than municipal bonds, I am missing the logic. Consider the following:
- No state has defaulted on a general obligation bond since 1933. Defaults on bonds issued by cities are also extremely rare – affecting about one in one thousand bonds per year. Other classes of municipal bonds have higher default rates, but not radically different from those of corporate bonds.
- Bonds issued by foreign governments can and do default. For example, private investors took a 70% haircut when Greek debt was restructured in 2012.
- Regulators explained their decision to exclude municipal bonds because of thin trading volumes, but this is also the case with corporate bonds. On Tuesday, FINRA reported a total of only 6446 daily corporate bond trades across a universe of perhaps 300,000 issues. So, in other words, the average corporate bond trades less than once per day. Not very liquid.
- Stocks are more liquid, but can lose value very rapidly during a crisis as we saw in 1929, 1987 and again in 2008-2009. Trading in individual stocks can also be halted.
Perhaps the most ironic result of the regulation involves municipal bond insurance. Under the new rules, a bank can purchase bonds or stock issued by Assured Guaranty or MBIA – two major municipal bond insurers – but they can’t buy state and local government bonds insured by those companies. Since these insurance companies would have to pay interest and principal on defaulted municipal securities before they pay interest and dividends to their own investors, their securities are clearly more risky than the insured municipal bonds.
Regulators have expressed a willingness to tweak the new HQLA regulations now that they are in place. I hope this is one area they will reconsider. Mandating that banks hold safe securities is a good thing; now we need a more data-driven definition of just what safe means. By including municipal securities in HQLA, bank regulators can also get on the same page as the rest of the Obama administration.
Wife beating education for sports fan and everyone else
Do you know what I am doing this morning? I’m glued to ESPN talk radio, which is 98.7FM in the NYC area, although it is a national station and can be streamed online as well.
Here’s a statement you might be surprised to hear from me. In the past decade, sports talk radio has become the best, rawest, and most honest source of information about how our culture condones and ignores violence against women, not to mention issues of race and homophobia. True fact. You are not going to hear this stuff from politicians or academics.
Right now I’m listening to the Mike & Mike program, which has guest Jemele Hill, who is killing it. I’m a huge fan of hers.
The specific trigger for the conversation today is the fact that NFL football player Ray Rice has been indefinitely suspended from playing now that a video has emerged of him beating his wife in the elevator. Previously we had only gotten to seen the video of her slumped body after he came out of the elevator with her. The police didn’t do much about it, and then the NFL responded with a paltry 2-game suspension, after which there was such a backlash (partly through sports radio!) that the commissioner promised to enact a stronger policy.
Questions being addressed right now as I type:
- Why didn’t the police give Rice a bigger penalty for beating his wife unconscious?
- Why didn’t the NFL ask for that video before now? Or did they, and now they’re lying?
- What does it say about the NFL that they had the wife, Janay Rice, apologize for her role in the incident?
- What did people think it would look like when a professional football player knocks out a woman?
- Did people really think she did something to deserve it, and now they are shocked to see that she didn’t?
Reverse-engineering the college admissions process
I just finished reading a fascinating article from Bloomberg BusinessWeek about a man who claims to have reverse-engineered the admission processes at Ivy League colleges (hat tip Jan Zilinsky).
His name is Steven Ma, and as befits an ex-hedge funder, he has built an algorithm of sorts to work well with both the admission algorithms at the “top 50 colleges,” and the US News & World Report model which defines which colleges are in the “to 50.” It’s a huge modeling war that you can pay to engage in.
Ma is a salesman too: he guarantees that a given high-school kid will get into a top school, your money back. In other words he has no problem working with probabilities and taking risks that he think are likely to pay off and that make the parents willing to put down huge sums. Here’s an example of a complicated contract he developed with one family:
After signing an agreement in May 2012, the family wired Ma $700,000 over the next five months—before the boy had even applied to college. The contract set out incentives that would pay Ma as much as $1.1 million if the son got into the No. 1 school in U.S. News’ 2012 rankings. (Harvard and Princeton were tied at the time.) Ma would get nothing, however, if the boy achieved a 3.0 GPA and a 1600 SAT score and still wasn’t accepted at a top-100 college. For admission to a school ranked 81 to 100, Ma would get to keep $300,000; schools ranked 51 to 80 would let Ma hang on to $400,000; and for a top-50 admission, Ma’s payoff started at $600,000, climbing $10,000 for every rung up the ladder to No. 1.
He’s also interested in reverse-engineering the “winning essay” in conjunction with after-school activities:
With more capital—ThinkTank’s current valuation to potential investors is $60 million—Ma hopes to buy hundreds of completed college applications from the students who submitted them, along with the schools’ responses, and beef up his algorithm for the top 50 U.S. colleges. With enough data, Ma plans to build an “optimizer” that will help students, perhaps via an online subscription, choose which classes and activities they should take. It might tell an aspiring Stanford applicant with several AP classes in his junior year that it’s time to focus on becoming president of the chess or technology club, for example.
This whole college coaching industry reminds me a lot of financial regulation. We complicate the rules to the point where only very well-off insiders know exactly how to bypass the rules. To the extent that getting into one of these “top schools” actually does give young people access to power, influence, and success, it’s alarming how predictable the whole process has become.
Here’s a thought: maybe we should have disclosure laws about college coaching and prep? Or would those laws be gamed too?
Aunt Pythia gives it up for Polly
Dearest readers. Dearest, dearest readers. Aunt Pythia was just about to crack open her dog-eared google doc of questions when she happened across this Ask Polly column which blew her away (hat tip Julie Steele).
It’s entitled Ask Polly: Why Don’t the Men I Date Ever Truly Love Me? and it’s just about the best advice Aunt Pythia has ever seen for a whole lot of people, men and women. In fact she’s seriously considering stealing certain phrases out of this one column for future use, including the following:
- Is it time to stop being so good and start discovering what’s going to transform your life into something big and vibrant and shocking?
- Block the “other” from this picture. No more audience. You are the cherished and the cherisher.
- Fuck wondering if you’re lovable. Fuck asking someone else, “Am I there yet?” Fuck listening for the answer.
Bravo, Polly! And readers, please go read it.
Friday morning reading
I’m very gratified to say that my Lede Program for data journalism at Columbia is over, or at least the summer program is (some students go on to take Computer Science classes in the Fall).
My adorable and brilliant students gave final presentations on Tuesday and then we had a celebration Tuesday night at my house, and my bluegrass band played (didn’t know I have a bluegrass band? I play the fiddle! You can follow us on twitter!). It was awesome! I’m hoping to get some of their projects online soon, and I’ll definitely link to it when that happens.
It’s been an exciting week, and needless to say I’m exhausted. So instead of a frothy rant I’ll just share some reading with y’all:
- Andrew Gelman has a guest post by Phil Price on the worst infographic ever, which sadly comes from Vox. My students all know better than this. Hat tip Lambert Strether.
- Private equity firms are buying stuff all over the country, including Ferguson. I’m actually not sure this is a bad thing, though, if nobody else is willing to do it. Please discuss.
- Bloomberg has an interesting story about online PayDay loans and the world of investing. I am still on the search for someone who knows exactly how those guys target their ads online. Hat tip Aryt Alasti.
- Felix Salmon, now at Fusion, has set up a nifty interactive to help you figure out your lifetime earnings.
- Felix also set up this cool online game where you can play as a debt collector or a debtor.
- Is it time to end letter grades? Hat tip Rebecca Murphy.
- There’s a reason fast food workers are striking nationwide. The ratio of average CEO pay to average full-time worker pay is around 1252.
- People lie to women in negotiations. I need to remember this.
Have a great weekend!
Student evaluations: very noisy data
I’ve been sent this recent New York Times article by a few people (thanks!). It’s called Grading Teachers, With Data From Class, and it’s about how standardized tests are showing themselves to be inadequate to evaluate teachers, so a Silicon Valley-backed education startup called Panorama is stepping into the mix with a data collection process focused on student evaluations.
Putting aside for now how much this is a play for collecting information about the students themselves, I have a few words to say about the signal which one gets from student evaluations. It’s noisy.
So, for example, I was a calculus teacher at Barnard, teaching students from all over the Columbia University community (so, not just women). I taught the same class two semesters in a row: first in Fall, then in Spring.
Here’s something I noticed. The students in the Fall were young (mostly first semester frosh), eager, smart, and hard-working. They loved me and gave me high marks on all categories, except of course for the few students who just hated math, who would typically give themselves away by saying “I hate math and this class is no different.”
The students in the Spring were older, less eager, probably just as smart, but less hard-working. They didn’t like me or the class. In particular, they didn’t like how I expected them to work hard and challenge themselves. The evaluations came back consistently less excited, with many more people who hated math.
I figured out that many of the students had avoided this class and were taking it for a requirement, didn’t want to be there, and it showed. And the result was that, although my teaching didn’t change remarkably between the two semesters, my evaluations changed considerably.
Was there some way I could have gotten better evaluations from that second group? Absolutely. I could have made the class easier. That class wanted calculus to be cookie-cutter, and didn’t particularly care about the underlying concepts and didn’t want to challenge themselves. The first class, by contrast, had loved those things.
My conclusion is that, once we add “get good student evaluations” to the mix of requirements for our country’s teachers, we are asking for them to conform to their students’ wishes, which aren’t always good. Many of the students in this country don’t like doing homework (in fact most!). Only some of them like to be challenged to think outside their comfort zone. We think teachers should do those things, but by asking them to get good student evaluations we might be preventing them from doing those things. A bad feedback loop would result.
I’m not saying teachers shouldn’t look at student evaluations; far from it, I always did and I found them useful and illuminating, but the data was very noisy. I’d love to see teachers be allowed to see these evaluations without there being punitive consequences.
Guest Post: Bring Back The Slide Rule!
This is a guest post by Gary Cornell, a mathematician, writer, publisher, and recent founder of StemForums.
I was was having a wonderful ramen lunch with the mathbabe and, as is all too common when two broad minded Ph.D.’s in math get together, we started talking about the horrible state math education is in for both advanced high school students and undergraduates.
One amusing thing we discovered pretty quickly is that we had independently come up with the same (radical) solution to at least part of the problem: throw out the traditional sequence which goes through first and second year calculus and replace it with a unified probability, statistics, calculus course where the calculus component was only for the smoothest of functions and moreover the applications of calculus are only to statistics and probability. Not only is everything much more practical and easier to motivate in such a course, students would hopefully learn a skill that is essential nowadays: how to separate out statistically good information from the large amount of statistical crap that is out there.
Of course, the downside is that the (interesting) subtleties that come from the proofs, the study of non-smooth functions and for that matter all the other stuff interesting to prospective physicists like DiffEQ’s would have to be reserved for different courses. (We also were in agreement that Gonick’s beyond wonderful“Cartoon Guide To Statistics” should be required reading for all the students in these courses, but I digress…)
The real point of this blog post is based on what happened next: but first you have to know I’m more or less one generation older than the mathbabe. This meant I was both able and willing to preface my next point with the words: “You know when I was young, in one way students were much better off because…” Now it is well known that using this phrase to preface a discussion often poisons the discussion but occasionally, as I hope in this case, some practices from days gone by ago can if brought back, help solve some of today’s educational problems.
By the way, and apropos of nothing, there is a cure for people prone to too frequent use of this phrase: go quickly to YouTube and repeatedly make them watch Monty Python’s Four Yorkshireman until cured:
Anyway, the point I made was that I am a member of the last generation of students who had to use slide rules. Another good reference is: here. Both these references are great and I recommend them. (The latter being more technical.) For those who have never heard of them, in a nutshell, a slide rule is an analog device that uses logarithms under the hood to do (sufficiently accurate in most cases) approximate multiplication, division, roots etc.
The key point is that using a slide rule requires the user to keep track of the “order of magnitude” of the answers— because slide rules only give you four or so significant digits. This meant students of my generation when taking science and math courses were continuously exposed to order of magnitude calculations and you just couldn’t escape from having to make order of magnitude calculations all the time—students nowadays, not so much. Calculators have made skill at doing order of magnitude calculations (or Fermi calculations as they are often lovingly called) an add-on rather than a base line skill and that is a really bad thing. (Actually my belief that bringing back slide rules would be a good thing goes back a ways: when that when I was a Program Director at the NSF in the 90’s, I actually tried to get someone to submit a proposal which would have been called “On the use of a hand held analog device to improve science and math education!” Didn’t have much luck.)
Anyway, if you want to try a slide rule out, alas, good vintage slide rules have become collectible and so expensive— because baby boomers like me are buying the ones we couldn’t afford when we were in high school – but the nice thing is there are lots of sites like this one which show you how to make your own.
Finally, while I don’t think they will ever be as much fun as using a slide rule, you could still allow calculators in classrooms.
Why? Because it would be trivial to have a mode in the TI calculator or the Casio calculator that all high school students seem to use, called “significant digits only.” With the right kind of problems this mode would require students to do order of magnitude calculations because they would never be able to enter trailing or leading zeroes and we could easily stick them with problems having a lot of them!
But calculators really bug me in classrooms and, so I can’t resist pointing out one last flaw in their omnipresence: it makes students believe in the possibility of ridiculously high precision results in the real world. After all, nothing they are likely to encounter in their work (and certainly not in their lives) will ever need (or even have) 14 digits of accuracy and, more to the point, when you see a high precision result in the real world, it is likely to be totally bogus when examined under the hood.
Distributional Economic Health
I am pushing an unusual way of considering economic health. I call it “distributional thinking.” It requires that you not aggregate everything into one statistic, but rather take a few samples from different parts of the distribution and consider things from those different perspectives.
So instead of saying “things are great because the economy has expanded at a rate of 4%” I’d like us to think about more individual definitions of “great.”
For example, it’s a good time to be rich right now. Really good. The stock market keeps hitting all-time highs, the jobs market is great in tech, and it’s still absolutely possible to hide wealth in off-shore tax havens.
It’s not so good to be middle class. Wages are stagnant and have been forever, and jobs are drying up due to automation and a lack of even maintenance-level infrastructure work. Colleges are super expensive, and the best the government can do is fiddle around the edges with interest rates.
It’s a really bad time to be poor in this country. Jobs are hard to find and conditions are horrible. There are more and more arrests for petty crimes as the violent crime rate goes down. Those petty crime arrests lead to big fees and sometimes jail time if you can’t pay the fee. Look at Ferguson as an example of what this kind of frustration this can lead to.
Once you are caught in the court system, private probation companies act as abusive debt collectors, and nobody controls their fees, which can be outrageous. To be clear, we let this happen in the name of saving money: private for-profit companies like this guarantee that they won’t cost anything to the local government because they make the people on probation pay for services.
And even though that’s an outrageous and predatory system, it’s not likely to go away. Once they are officially branded as criminals, the poor often lose their voting rights, which means they have little political recourse to protect themselves. On the flip side, they are largely silent about their struggles for the same reason.
Once you think about our economic health this way, you realize how comparatively meaningless the GDP is. It is no longer a good proxy to true economic health, where all classes would be more or less better off as it went up.
And until we get on the same page, where we all go up and down together, it is a mathematical fact that no one statistic could possibly capture the progress we are or are not making. Instead, we need to think distributionally.
Aunt Pythia’s advice: the nerdy edition
Aunt Pythia is ginormously and ridonkulously excited to be here. She just got back from a nifty bike ride to the other side of the Hudson and took this picture of this amazing city on this amazing day:
The bike traffic on the GWB is not too bad at 7:10am.
OK, so full disclosure. Aunt Pythia kind of blew her load, so to speak, on the sex questions last week, so she’s making do with coyly answering nerdy questions. Because that’s what we got.
I hope you enjoy her efforts, and even if you despise them – especially if you despise them – don’t forget to:
please think of something to ask Aunt Pythia at the bottom of the page!
By the way, if you don’t know what the hell Aunt Pythia is talking about, go here for past advice columns and here for an explanation of the name Pythia.
——
Hi Aunt Pythia,
I’m a math student at MIT, where you did a postdoc. I’m also into computers, and am considering working in some finance classes. I could see myself being happy working for some big financial company that I don’t really care about, as long as I have interesting problems to work on, make a ton of money, and have bright people I get to work with.
My interests right now are in very pure math, I get chills just thinking about categorical-theoretic concepts. I’m planning to learn commutative algebra and algebraic geometry soon. I’m also likely to take stochastic calculus.
What kind of math did you do? Any tips on if taking the pure math I love will be of use, or at least get me “cred” with financial companies?
I do love math, and seeing that you did math at MIT and have seen this world of things, maybe you have some advice to offer me.
Thank you dearly.
Math Cult
Dear MC,
Don’t do it!
Don’t take the math to get “cred” with financial companies. Do what is sexy and beautiful to you. If you love category theory, do that, then do algebra and algebraic geometry. I did number theory in the form of arithmetic algebraic geometry myself. It’s awesomely beautiful and I don’t regret one moment of it.
Let’s say you do decide to go into the “real world.” At the end of the day, if you can do that math stuff we’ve been talking about, you can learn other stuff too. So I’m not going to worry about you on the technical side of things.
On the other side of things, I’d like you to rethink the idea that you “don’t mind who you work for as long as you have interesting problems.” Is that really true? Once you leave pure math there are real applications of your work, and they affect real people. Shit gets real real quick and stuff matters, and I urge you to think it through some more.
Good luck!
Aunt Pythia
——
Dear Aunt Pythia,
Do all mathematicians visualize their problems? From a logical viewpoint there are a lot of mathematical spaces that don’t map onto an imagined 3d workspace but on limited conversations with working mathematicians they seem to me to do it at least at some stage of problem solving.
(I’m more of a physicist who visualizes nearly everything so maybe I’m misreading them.)
Inner glimmer
Dear Inner,
Most, but not all. I once had a conversation with someone who couldn’t understand my drawing of a geometric map between spaces. I was explaining the concept visually (or at least I thought I was!) but he forced me to write it down with double sums and formulas, and I thought that was the weirdest thing ever, but that’s how it became understandable to him.
In general we do think visually, although we really can’t think beyond three dimensions (even though we pretend we can). I guess time makes it 4. Most geometers I know, ironically, don’t have a very good working sense of 3 dimensions, and definitely don’t have a good sense of direction!
Come to think of it my sample is too small, so I’m mostly just saying that for fun. It would be neat to get actual statistics on that. Maybe if I’m ever pulled into going to JMM again I’ll make people fill out forms. Oh wait, I’m going to JMM this January.
I can ask about this, it’s a nice question! Readers, what else should I poll math nerds on?
Aunt Pythia
——
Dear Aunt Pythia,
I’m an American mutt and for awhile I was annoyed when people asked “Where are you from” or “What’s your nationality”. I think I was sensitive to it because kids wanted to narrow down exactly which ethnic slurs to use. But as an adult, mostly people are just curious, and I’m happy to share since I’m curious about them too.
When I meet someone with an accent, I’m curious about them and their background, what it’s like in their home country, how they came to the US, etc.
What is an appropriate way to ask about someone’s ethnic background or country of origin? It seems like you should be able to ask anyone this question; it just seems rude when that person is different from you. Do you know what I mean?
WHy Ask That Rude qUestion
Dear WHATRU,
I like the subtle sign-off!
Here’s the thing, I think you nailed it. If your intention is to be mean, then don’t ask it. If your intention is to be friendly and to make a connection, then go ahead and ask it! I always ask cabbies where they come from, and then I get to learn about their countries. I have never experienced someone who doesn’t want to talk to me about their home country, and I’ve made quite a few friends. I’ve been invited to so many countries for visits, and that is always so incredibly generous and sweet! People are amazing.
Of course, some people just don’t do this kind of small-talk, and I get that too. It’s not for everyone. But it’s super fun for us extroverts.
Aunt Pythia
——
Dear Aunt Pythia,
First off, you’re blog is both entertaining and informative, and you’ve found the sweet spot combination of the two that makes it addictive.
I find your work with the Lede program at Columbia fascinating and relevant to the growing, amorphous “big data” movement. I am a frequent visitor of websites such as Fivethirtyeight, which Nate Silver has rebranded as a news source that derives its stories from statistics and big data analytics. Even other sources, such as The Atlantic, have begun to follow suit and incorporate large statistical analyses into some of their stories. This experiment of basing our news stories on statistics brings hope that we can move closer to the ideal of an unbiased account.
In light of this new format (and your school), what sources do you consider the best? Are there any that you visit to get an insightful statistical perspective on the news. Or do you side with the criticism that many of these sites fuel a sensationalist, biased view of the world intended to spawn viral stories?
Will we ever find the right place for statistics in the news?
Considering unbiased reality in our ubiquitous (news)stories
Dear Curious,
Holy crap, nice sign-off. And thanks for being addicted to mathbabe! All my evil plans are working. Time to start on the next phase… moo-hooo-hahahahahaha.
OK, so here’s the thing. We will never have unbiased accounts. Never. At the very least we will have bias in the way that data is collected.
What I’ve spent the summer talking to my students about is getting used to the fact that there will always be bias, and how we therefore do our best to be at least somewhat aware of them, and try very hard not to obscure them. Transparency is the new objectivity!
This is of course disappointing to people who want there to be “one truth,” but that’s how science is. After a while we get used to the disappointment and we can all appreciate some really good signal/noise ratios.
As for the right place for statistics in the news, I think we’re figuring that out right now, and I’m excited to be part of it. And holy shit, have you seen the new ProPublica work on the Louisiana coast? Those guys are killing it.
Love,
Aunt Pythia
——
Please submit your well-specified, fun-loving, cleverly-abbreviated question to Aunt Pythia!
The bad teacher conspiracy
Any time I see an article about the evaluation system for teachers in New York State, I wince. People get it wrong so very often. Yesterday’s New York Times article written by Elizabeth Harris was even worse than usual.
First, her wording. She mentioned a severe drop in student reading and math proficiency rates statewide and attributed it to a change in the test to the Common Core, which she described as “more rigorous.”
The truth is closer to “students were tested on stuff that wasn’t in their curriculum.” And as you can imagine, if you are tested on stuff you didn’t learn, your score will go down (the Common Core has been plagued by a terrible roll-out, and the timing of this test is Exhibit A). Wording like this matters, because Harris is setting up her reader to attribute the falling scores to bad teachers.
Harris ends her piece with a reference to a teacher-tenure lawsuit: ‘In one of those cases, filed in Albany in July, court documents contrasted the high positive teacher ratings with poor student performance, and called the new evaluation system “deficient and superficial.” The suit said those evaluations were the “most highly predictive measure of whether a teacher will be awarded tenure.”’
In other words, Harris is painting a picture of undeserving teachers sneaking into tenure in spite of not doing their job. It’s ironic, because I actually agree with the statement that the new evaluation system is “deficient and superficial,” but in my case I think it is overly punitive to teachers – overly random, really, since it incorporates the toxic VAM model – but in her framing she is implying it is insufficiently punitive.
Let me dumb Harris’s argument down even further: How can we have 26% English proficiency among students and 94% effectiveness among teachers?! Let’s blame the teachers and question the legitimacy of tenure.
Indeed, after reading the article I felt like looking into whether Harris is being paid by David Welch, the Silicon Valley dude who has vowed to fight teacher tenure nationwide. More likely she just doesn’t understand education and is convinced by simplistic reasoning.
In either case, she clearly needs to learn something about statistics. For that matter, so do other people who drag out this “blame the teacher” line whenever they see poor performance by students.
Because here’s the thing. Beyond obvious issues like switching the content of the tests away from the curriculum, standardized test scores everywhere are hugely dependent on the poverty levels of students. Some data:
It’s not just in this country, either:
Considering how many poor kids we have in the U.S., we are actually doing pretty well.
The conclusion is that, unless you think bad teachers have somehow taken over poor schools everywhere and booted out the good teachers, and good teachers have taken over rich schools everywhere and booted out the bad teachers (which is supposed to be impossible, right?), poverty has much more of an effect than teachers.
Just to clarify this reasoning, let me give you another example: we could blame bad journalists for lower rates of newspaper readership at a given paper, but since newspaper readership is going down everywhere we’d be blaming journalists for what is a cultural issue.
Or, we could develop a process by which we congratulate specific policemen for a reduced crime rate, but then we’d have to admit that crime is down all over the country.
I’m not saying there aren’t bad teachers, because I’m sure there are. But by only focusing on rooting out bad teachers, we are ignoring an even bigger and harder problem. And no, it won’t be solved by privatizing and corporatizing public schools. We need to address childhood poverty. Here’s one more visual for the road:
A decision tree for decision trees
For a while now I’ve been thinking I should build a decision tree for deciding which algorithm to use on a given data project. And yes, I think it’s kind of cool that “decision tree” would be an outcome on my decision tree. Kind of like a nerd pun.
I’m happy to say that I finally started work on my algorithm decision tree, thanks to this website called gliffy.com which allows me to build flowcharts with an easy online tool. It was one of those moments when I said to myself, this morning at 6am, “there should be a start-up that allows me to build a flowchart online! Let me google for that” and it totally worked. I almost feel like I willed gliffy.com into existence.
So here’s how far I’ve gotten this morning:
Not far! But I also learned how to use the tool.
I looked around the web to see if I’m doing something that’s already been done and I came up with this:
I appreciate the effort but this is way more focused on the size of the data than I intend to be, at least for now. And here’s another one that’s even less like the one I want to build but is still impressive.
Because here’s what I want to focus on: what kind of question are you answering with which algorithm? For example, with clustering algorithms you are, you know, grouping similar things together. That one’s easy, kind of, although plenty of projects have ended up being clustering or classifying algorithms whose motivating questions did not originally take on the form “how would we group these things together?”.
In other words, the process of getting at algorithms from questions is somewhat orthogonal to the normal way algorithms are introduced, and for that reason taking me some time to decide what the questions are that I need to ask in my decision tree. Right about now I’m wishing I had taken notes when my Lede Program students asked me to help them with their projects, because embedded in those questions were some great examples of data questions in search of an algorithm.
Please give me advice!
Aise O’Neil at Gotham Comedy Club
I don’t usually blog about my kids, but my 14-year-old son has explicitly given me his blessing to post his recent stand-up performance at the Gotham Comedy Club:
The look he gives the audience at the end is my favorite part.
The Stubborn Hope of an Urban Teacher
Yesterday I read a book written by Carole Marshall which she called Stubborn Hope: Memoir of an Urban Teacher (thanks to Ernest Davis for sending it to me). Just to give you an idea of how quick this read is, I read it before class. I think it took about 1 hour and 10 minutes in all.
In a nutshell, it was the story of a really hard-working and dedicated urban school teacher who learned how to teach reading skills, and prose and poetry writing skills to her poverty-stricken students in the urban Providence, RI area. She develops curriculum, making it relevant to the kids, and gets them to read every night and to aspire to college. The school that she mostly taught at is profiled in this article from the Brown Daily Herald.
She’s a really good writer herself, and she profiles a bunch of her students with enough details to make you feel enormous empathy for their struggles. In other words, she makes this shit very very real. After reading this you stop wondering why we see a strong negative correlation between standardized tests scores and poverty levels, because it is so obvious.
You might want to check out this video to get a satirical idea of what this woman was like and what she was dealing with (hat tip Jenn Rubinovitz):
Here’s the thing. We need nice white ladies in our schools! And of course nice other people too.
But we are presently losing such dedicated people. Carole Marshall, the author of these memoirs, quit teaching after the school system she worked in was taken over by the mindless testing zombies. She describes her experience like this:
After spending years refining strategies for getting my students to become enthusiastic readers and writers on thoughtful, relevant curriculum, I was being forced to teach canned curriculum purchased for millions of dollars from textbook publishers who knew nothing about urban teaching.
School and district administrators roamed the halls and classrooms, taking notes on shiny new iPads, to make sure teachers were on the same page every day as every other teacher in our grade and subject in the district. All the activities we had used in the past to open our students to a world beyond the narrow constraints of their neighborhoods were no longer permitted; they were seen as time wasted. Every path to good teaching was effectively blocked off.
It had become impossible to do the things with students that I believe teachers need to be able to do. What was going on in the classrooms could no longer be called teaching. When I realized that, it was a sad day. At the end of that year, I left teaching.
That was in 2012, I believe. Since then she’s become more aware of the national disaster that is defined by the testing insanity. She even worked for a time with a test prep company based in Florida that was clearly scamming for the $5 million consultant fee and removing cherry-picked students from important classes so the school would look like it had improved based on the arbitrary measure of the month.
We are so used to pointing at examples of bad and defeated teachers and saying that they are the problem, and that a strict and regimented system of curriculum will improve the classrooms for the students of such teachers. And maybe in some cases that is true.
But when we do that we also push out really talented and inspirational teachers like Carole Marshall. It is painful to imagine how many great teachers have left the educational system because of No Child Left Behind and Race To The Top. Come to think of it, that would be a great data journalism project.
Gillian Tett gets it very wrong on racial profiling
Last Friday Gillian Tett ran a profoundly disturbing article in the Financial Times entitled Mapping Crime – Or Stirring Hate? (hat tip Marcos Carreira), which makes me sad to say this given how much respect I normally have for her regarding her coverage of the financial crisis.
In the article, Tett describes the predictive policing model used by the Chicago police force, which told the police where to go to find criminals based on where people had been arrested in the past.
Her article reads like an advertisement for racist profiling. First she deftly and indirectly claims the model is super successful at lowering the murder rate without actually coming out and saying so (since she actually has only correlative evidence):
And when Weis launched the programme in early 2010, together with a clever policeman-cum-computer expert called Brett Goldstein, it delivered impressive results. In the first year the murder rate fell 5 per cent and then continued to tumble. Indeed by the summer of 2011 it looked as if Chicago’s annual death toll would soon drop below 400, the lowest since 1965. “The homicide rates for that summer were just crazy low compared to what we had been,” Weis observes.
But then, following his departure from the force, the programme was wound down in late 2011. And, tragically, the murder rate immediately rose again.
Here’s the thing, it’s really hard to actually know why murder rates go up and down. In New York City we’ve been using Stop & Frisk as the violent crime rates have been steadily lowering in this city (and many others), and for a long time Bloomberg took credit for that through the Stop & Frisk practice. But when Stop & Frisk rates went down, murder rates didn’t shoot up. Just saying. And that’s ignoring how reliable the police data is, which is another issue. Let’s take a look at her evidence for a longer time frame:
She’s talking about that small uptick at the end, which to the naked eye could well be statistical noise.
The reason I’m pointing out her bad statistics is that she needs them to set up the following, truly disturbing paragraphs (emphasis mine):
But while racism is rightly deemed unacceptable, computer programs pose more subtle questions. If a spreadsheet forecast has a racial imbalance, is this likely to reinforce existing human biases, or racial profiling? Or is a weather map of crime simply a neutral tool? To put it another way, does the benefit of using predictive policing outweigh any worries about political risk?
Personally, I think it does. After all, as the former CPD computer experts point out, the algorithms in themselves are neutral. “This program had absolutely nothing to do with race… but multi-variable equations,” argues Goldstein. Meanwhile, the potential benefits of predictive policing are profound.
No, Gillian Tett, there is no such thing as a neutral tool. No algorithm focused on human behavior is neutral. Anything which is trained on historical human behavior embeds and codifies historical and cultural practices. Specifically, this means that the fact that black Americans are nearly four times as likely as whites to be arrested on charges of marijuana possession even though the two groups use the drug at similar rates would be seen by such a model (or rather, by the people who deploy the model) as a fact of nature that is neutral and true. But it is in fact a direct consequence of systemic racism.
Put it another way: if we allowed a model to be used for college admissions in 1870, we’d still have 0.7% of women going to college. Thank goodness we didn’t have big data back then!
This is very scary to me, when even Gillian Tett, who famously predicted the financial crisis in 2006, can be fooled. We clearly have a lot of work to do.
mathbabe 
