Big Data community, please don’t leave underrepresented students behind

This is a guest post by Nii Attoh-Okine, Professor of Civil and Environmental Engineering and Director of Big Data Center at the University of Delaware. Nii, originally from Ghana, does research in Resilience Engineering and Data Science. His new book, Resilience Engineering: Models and Analysis will be out in December 2016 with Cambridge Press. Nii is also working on a second book, Big Data and Differential Privacy: Analysis Strategies for Railway Track Engineering, which will be out Fall 2016 with John Wiley & Sons. 

Big data has been a major revolutionary area of research in the last few years—although one may argue that the name change has created at least part of the hype. Only time will tell how much. In any case, with all the opportunities, hype, and advancement, it is very clear that underrepresented minority students are virtually missing in the big data revolution.

What do I mean? The big data revolution is addressing and tackling issues within the banking, engineering and technology, health and medical sciences, social sciences, humanities, music, and fashion industries, among others. But visit conferences, seminars, and other activities related to big data: underrepresented minority students are missing.

At a recent Strata and Hadoop conference in New York, one of the premier big data events, it was very disappointing and even alarming that underrepresented minority students (participants and presenters) were virtually nonexistent. The logical question that comes to mind is whether the big data community is not reaching out to underrepresented minority students or if underrepresented minority students are not reaching out to the big data community.

To address the importance of addressing and tackling the issues, there are a two critical facts to know, the first on the supply side, the other on the demand side:

  1. The demographics of the US population are undergoing a dramatic shift. Minority groups underrepresented in STEM fields will soon make up the majority of school-age children in the states (Frey, 2012). This means that currently underrepresented minorities are a rich pool of STEM talent, if we figure out how to tap into it.
  2. “‘Human resource inputs are a critical component to our scientific enterprise. We look to scientists for creative sparks to expand our knowledge base and deepen our understanding of natural and social phenomena. Their contributions provide the basis for technological advances that improve our productivity and the quality of lives. It is not surprising, therefore, that concern about the adequacy of the talent pool, both in number and quality, is a hardy perennial that appears regularly as an important policy issue.’ This statement, borrowed from Pearson and Fechter’s book, Who will Do Science?: Educating the Next Generation, remains a topic of serious debate” (A. James Hicks, Ph.D., NSF/LSAMP Program Director).

The issue at large is how the big data community can involve the underrepresented minority students. On that front I have some suggestions. The big data community can:

  • Develop ‘invested mentors’ from the big data community who show a genuine interest in advising underrepresented minority students about big data.
  • Forge partnerships with colleges and universities, especially minority-serving institutions.
  • Identify professors who have genuine interest in working with underrepresented students in big data related research.
  • Invite some students and researchers from underrepresented minorities to big data conferences and workshops.
  • Attend and organize information sessions during conferences oriented toward underrepresented minority students.

The major advice to the big data community is this: please do make the effort to engage and include underrepresented minority students because there is so much talent within this group.

Categories: Uncategorized

Math and the caveman imagination

This is a guest post by Ernie Davis Professor of Computer Science at NYU. Ernie has a BS in Math from MIT (1977) and a PhD in Computer Science from Yale (1984). He does research in artificial intelligence, knowledge representation, and automated commonsense reasoning. He and his father, Philip Davis, are editors of Mathematics, Substance and Surmise: Views on the Ontology and Meaning of Mathematics, published just last week by Springer.

We hear often that our cognitive limitations and our social and psychological flaws are due to our evolutionary heritage. Supposedly, the characteristics of minds and our psyches reflect the conditions in the primordial savannah or caves and therefore are not a good fit to the very different conditions of the 21st century.

The conditions of our primordial ancestors have been blamed for political conservativism, for religious belief , for vengefulness, and especially – since the subject is so fraught and so enjoyable – for gender differences, particularly in sexual fidelity. These kinds of theories have been extensively criticized, most notably by Steven Jay Gould, as being often “just-so” stories. You find a feature of the human mind that you dislike, or one that you think is an ineradicable part of human nature, and you make up a story about why it was good for the cavemen. You find a feature that some people have and others don’t, like political conservatism, and you explain that the stupid bad guys have inherited it from the cavemen, but that the smart good guys have overcome it. I gave my own opinions of the theories about conservatism and religion here.

This week, our ancestors are the fall guys for the fact that we find math difficult. In this week’s New Yorker, Brian Greene is quoted as saying, “[Math] is not what our brains evolved to do. Our brains evolved to survive in the wilderness. You didn’t have to take exponentials or use imaginary numbers in order to avoid that lion or that tiger or to catch that bison for dinner. So the brain is not wired, literally, to do the kinds of things that we now want it to do.”

The problem with this explanation is that it doesn’t explain. The question is not “Why is math hard in an absolute sense?” That’s hardly even a meaningful question. The question is “Why is math (for many people)particularly hard and unpleasant?”; that is to say, harder than a lot of other cognitive tasks. Saying that math is hard because it was useless for avoiding lions and catching bison doesn’t answer the question, because there are many other tasks that were equally useless but are easy and pleasant for people: reading novels, singing songs, looking at pictures, pretending, telling jokes, talking nonsense, dreaming. Nor can the comparative hardness of math be explained in terms of inherent computational complexity; if our experience with artificial intelligence is any indication, doing basic mathematics is much easier computationally than understanding stories. Until we have a much better understanding of how the mind carries out these various cognitive tasks, no explanation of why one task is harder than another can possibly hold much water.*

Conversely, our cognitive apparatus has all kinds of characteristics that, one has to suppose, were unhelpful for primitive people: our working memory is limited in size, our long-term memory is error-prone, we are susceptible to all manner of cognitive illusions and psychological illnesses, we are easily distracted and misled, we are lousy at three-dimensional mental rotation, our languages have any number of bizarre features. We find it harder to communicate distance and direction than bees; we find it harder to navigate long distances than migratory birds. Granted, imaginary numbers would have been useless in primitive life, but other forms of math which would probably have been useful, such as three-dimensional geometry, are also difficult.

Also, our distant ancestors should not be underestimated. The quotation from Greene seem to reflect Hobbes’ view that primitive life was “poor, nasty, brutish, and short”. These are, after all, the people from whom we inherit number systems, art, and language. They did not spend all their time escaping from lions and hunting bison.

Our ancestors on the savannah saw parabolic motion whenever they threw a stone; they experienced spherical geometry whenever they looked up at the starry sky. They never encountered a magic wand or a magic ring. Nonetheless, most people find it easier and much more enjoyable to read and remember and discuss four volumes of intricate tales about magic rings or seven about magic wands than to read a few dozen pages with basic information about parabolas; and even most mathematicians find spherical geometry unappealing and difficult. Why? We have absolutely no idea.

* “I well remember something that Francis Crick said to me many years ago, … ‘Why do you evolutionists always try to identify the value of something before you know how it’s made?’ At the time I dismissed this comment … Now, having wrestled with the question of adaptation for many years, I understand the wisdom of Crick’s remark. If all structures had a `why’ framed in terms of adaptation, then my original dismissal would be justified for we would know that “whys” exist whether or not we had elucidated the “how”. But I am now convinced that many structures … have no direct adaptational ‘why’. And we discover this by studying pathways of genetics and development — or, as Crick so rightly said to me, by first understanding how a structure is built. In other words, we must first establish ‘how’ in order to know whether or now we should be asking ‘why’ at all.” — Steven Jay Gould, “Male Nipples and Clitoral Ripples”, in Bully for Brontosaurus 1991.

Categories: Uncategorized

Aunt Pythia’s advice

Readers, Aunt Pythia must apologize. She was taken over last weekend with the intense urge to craft. This is a coping mechanism of hers which takes over in times of stress, and the Paris attacks combined with anything Trump says, ever, overwhelmed her for about a week and it started last weekend. The good news is she got her project done:


It’s a baby blanket for my friend Adrian’s new baby. The design is here.



It’s reversible. Except for the name.

As for yesterday, Aunt Pythia had family over and was whipping up 5 dozen pancakes. Forgot to take pictures of them, but there were a lot of bananas and chocolate chips involved.

Anyhoo, that’s the explanation, but rest assured she has recovered and has emerged from her craft cave that exists inside the head. She is here for you and wants nothing more than to listen to your questions and give her half-reasoned and pseudo-sound advice. Before that, though, a small interruption.

Public Service Announcement: Reading Aunt Pythia has been known to improve mental and physical health, if only because it keeps you away from Thanksgiving leftovers for a few extra minutes. This effect is not statistically significant. I repeat, not statistically significant. This has been a Public Service Announcement brought to you by state and local authorities. In the event of an actual emergency, this announcement would not be helpful. I repeat, unhelpful.

If, after reading the below, you want to waste even more time, please don’t hesitate to:

ask Aunt Pythia any question at all at the bottom of the page!

By the way, if you don’t know what the hell Aunt Pythia is talking about, go here for past advice columns and here for an explanation of the name Pythia.


Dear Aunt Pythia,

I’d like to tell you an anecdote, one that is both mortifying and instructive. I offer it in the hope that it will elicit some thoughtful discussion from you, and from your other readers.

I am a male mathematician of a certain age. About twenty years ago, in my capacity as a member of a journal editorial board, I received a paper to handle. The author, a European, was nobody I had heard of. I took a look at the paper and decided that it was not up to the journal’s standards. The theorem was correct, but I could explain it to myself quickly using standard ideas in a routine way, by arguments simpler than those in the paper.

So without sending it to a referee I wrote a polite rejection letter explaining my reasons. From the author’s name, I assumed she was a woman. In fact he was a man.

I learned this about ten years later, when I happened to meet him at a small international conference. I don’t believe that either of us acknowledged the fact that our paths had crossed long ago. Several embarrassing thoughts flew into my head. Had he received a rejection letter starting “Dear Ms ____”? I would like to think that I knew enough back then to write “Dear Professor ____” or “Dear Dr ____”. Maybe I did.

But the thing that really made me squirm was that, as soon as I learned that she was a he, it came to me in a flash that I had written the letter in a sexist frame of mind. Imagining that I was writing to a woman, probably a struggling new-fledged female PhD, I had adopted a condescending tone that I don’t think I would have taken with a struggling new-fledged male PhD.

I can’t say for sure whether the air of condescension would have been obvious to the reader, but it was certainly there in my head. If I couldn’t see my own sexism in that case until this chance discovery made it apparent, how can I guard against similar attitudes when I’m teaching, or writing a reference letter, or reading a job application or a grad school application?

Baffling, Our Own Biases, Yes?

Dear BOOBY?,

I get it. I’m like that too. In fact women are just as sexist as men. I often find myself wondering if I spoke too condescendingly to women. I sometimes wish I could “play back the tape,” as it were, from old conversations so that I could apologize when appropriate.

But of course, you can’t do that in a conversation, because we don’t record conversations, or for that matter take any other anti-bias steps for real-world interactions. On the other hand, we can and should for formal situations like submitting papers.

Here’s a very concrete suggestion: Make it a rule that you obscure the name of every article that comes in to every journal, and that at least one referee sees each one. This way you, who saw the name before it was obscured, will not be tempted to immediately dismiss articles written by women. And as a happy by-product you won’t have to fret later on about your own sexist impulses, which again we all have.

Maybe you can arrange for someone else to do the obscuring-of-the-names process so you can look at the papers yourself. Maybe you could arrange with another editor to obscure their names in return for them obscuring yours. Whatever. Do what it takes to make it a blind audition.

I’d like to add that, whenever possible, do this for grad school applications and job applications as well. I know it’s hard for job applicants, because they are known personally at that level, but try to put processes into place that at least mitigate this kind of thing.

Good luck, and please know that once this system is in place you will have accomplished way more than you are now by kicking yourself needlessly and fruitlessly.

Aunt Pythia


Aunt Pythia,

My wife, her sister and I are all in our late 40’s and empty nesters. My sister-in-law, Naomi, is divorced and currently between boyfriends. As a result, she usually shows up at our place Saturday mornings. She and my wife, Allison, go shopping or to an art exhibit or other places, returning to our house in the late afternoon. They both shower and get dressed and then the three of us go out for dinner and other activities. I’ve edited some of the conversation below to make it more coherent.

Allison showers in the master bath and Naomi in the guest bath. They both come to the double sink in the master bath to fix their makeup, do their hair, and talk. I’ve taken to joining them because they both primp while wrapped in bath towels and I’m hoping for a flash. Allison will usually oblige pretending to need to get something from the under sink. She bends over keeping her legs straight. Then she’ll straighten up and smirk at me.

Several weeks ago Allison was alone in front of her sink when I came in. She smelled so fresh that I couldn’t resist. I knelt in front of her, put a hand behind her thigh for balance, pulled down her towel, and licked her nipples. All of a sudden Naomi showed up. Allison was trapped.

“Ronnie, don’t embarrass Naomi!”

I took one more lick of each nipple making sure Naomi could see my tongue flicking each bud. I stood up.

“Sorry, sometimes I get carried away,” I said to Naomi.

“Looked like fun,” Naomi replied. “I wish I had someone to do that to me.”

The minor exhibitionism revved me up. That night I was harder than I had been in a long time. Allison was hot also. She had a couple of orgasms.

Since that Saturday night had gone so well, I decided to try the same thing the next week. When I walked into our bedroom Allison was naked and still drying off. Unfortunately Naomi was already there. My disappointment showed.

“Aww, no sugar tits this week, Ron,” Naomi said and laughed.

The following week I pulled down Allison’s towel before Naomi showed up. Allison was moaning when Naomi walked in on us.

“Ronnie, stop Naomi’s here.” But she didn’t push me away.

“Yeah, Ron. You’re making me jealous.”

I took one last long lick to make sure Naomi saw.

“Doesn’t your sister have nice breasts?” I cupped one of Allison’s breasts and turned to Naomi.

“Mine are bigger.” Naomi pulled her towel down for about three seconds before covering up again.
That night Allison was all over me as soon as Naomi left.

“Did you like seeing my slut sister’s big boobs?” I was sucking Allison’s breasts so it took me a few seconds to reply.

“They’re okay.”

“Don’t bullshit a bullshitter. I saw the way your eyes bugged out when she flashed you.”

“Honey, I’ve told you before, anything over a mouthful is wasted.”

“So you would waste a lot if you sucked her boobs?”

“I guess so.” After that the conversation waned as I moved lower on her body.

We all felt a certain excitement the next Saturday when the girls got ready for their showers.

Allison was still drying off when I caught her out of the shower. This time as I lowered my lips to her nipples I slipped the hand that I used to balance on the back of her thigh up higher until my fingers were against her sex.

“You be good if Naomi comes in. Don’t embarrass me.” Naomi showed up a minute later.

“Don’t you two ever get enough?” I licked both nipples before raising my head and looking at Allison.

“If Naomi is going to watch us each week, don’t you think she ought to show us something?”

“Yeah, Sis. Show us your boobs again.” I dropped my head back down and watched out of the corner of my eye as Naomi lowered her towel to her waist. After about fifteen seconds I raised my head again.

“Do you think she wants her nipples licked?”

“Sis, do you want Ronnie to lick your nipples?” Allison moaned. Then she spread her legs farther apart. My index finger slipped between her moistening lips.

Naomi didn’t reply, but she drifted over beside me. I turned my head and put my other hand on the back of Naomi ‘s thigh to balance. Of course that hand slipped higher to touch between her legs. I moved my mouth to her nearest nipple. As I sucked, Naomi spread her legs a little. I extended my finger upward. After about five minutes of switching back and forth and tonguing four nipples, I was about to explode.

“Ronnie, we need to finish getting ready for dinner.”

“Yeah, Ron, I bet you’re hungry,” Naomi added hoarsely.

I got up reluctantly.

That night I was licking between Allison’s legs.

“Do you think Naomi would like it if I licked her clit?” I asked as I stopped briefly.

“My sister’s such a slut. I bet she would give you a blowjob and swallow if you licked her clit. But maybe you shouldn’t get confused about which sister you’re married to.” With that she dug her fingernails into my shoulder hard enough to hurt. “Get back to my clit.

So, here’s my question. Is my wife giving me permission to steal third base or is she calling me out and sending me to the dugout?

Wants to Smell the Roses but Afraid of the Thorns

Dear WrStRbAotT,

First of all, I want to thank you for your letter. I appreciate the work you put into it, I really do. It’s obvious what you put yourself through on Aunt Pythia’s behalf, and she appreciates it.

Second of all, I don’t have a sister, but if I did, I am pretty sure I’d never want to be sexual with her or in her presence. I mean, I have a brother, so I know how I feel about that kind of thing, and I’m pretty sure sisters are similar. Even so, it seems like – and I’m generalizing from Happy Days and Fonzie, but who doesn’t – it seems like this is a common enough male fantasy.


I guess to probe just a bit on this topic, how would you react if I talked about having sex with you and your brother at the same time? Would that turn you on? I doubt it. Just saying.

In answer to your question, I think your wife has been giving you mixed signals, and maybe you should take that as a sign of ambivalence, or maybe she’s just saving the best for the next chapter, if you will.

To sum up: I’d definitely let her take the lead if I were you. If she wants you to do the nasty with her sister, believe me, she’ll tell you to. Or maybe show you how (cough).

Good luck!

Aunt Pythia


Dear Aunt Pythia,

About seventeen years ago when I was a grad student I was in the computer room of a national science facility analyzing my data. The only other person there was a fellow grad student analyzing her data when her thesis supervisor came in the room and started to sexually assault her.

Without thinking, I pulled him away from her and dragged him to my desk where I said “here are those plots which I mentioned before” which is true because a few hours earlier I was talking to him about that data. He looked quite livid, but said nothing and left the room.

The grad student was stunned and left the room. Neither of us said anything either then or later. I have not seen XXXXXX (AP: name redacted) for at least ten years. The recent news about the sexual harassment case at Berkeley has made me think of these memories after so many years of not wanting to think of them. I have also started wondering whether what I did really made a difference in her life and so I have thought of contacting her directly and asking that question.

There is no question in my mind that what I did was the right thing, yet somehow it would be nice to get some sort of acknowledgment. Do you think contacting her and asking her whether what I did made any kind of difference in her life be a good idea and if so, how would you word it ?

Old Memories Arise Again

Dear OMAA,

Hmm. I’m thinking, you maybe didn’t do the wrong thing, but I don’t think I’d characterize it as “the perfect thing”. That’s not to blame you at all, because I think your intuition was good, and you definitely defused the situation. But I’m wondering if the internal conflict you’re feeling might arise from the fact that you could have done more. In short, you diverted him but you didn’t keep him from trying it again.

The problem with diversion, as a technique, is that it doesn’t address the underlying issue, and it doesn’t call it out as fucked up. It simply avoids it in a short-term way. So for example, there’s no reason to think that your colleague ever felt safe going back to that computer lab to do work again, even if she got a new advisor.

So, if you’re wondering what you’d do if that situation came up again, I’d suggest 1) telling him he’s doing something illegal while he’s doing it, 2) telling your colleague she has every right to call the police, and 3) calling the police yourself in front of both of them. That way he gets the message, and even if he ends up thinking he did nothing wrong, which is typical in this kind of situation, at least he’s been through enough that he doesn’t think doing it again is worth it. Introduce serious friction into his predatory ways, in other words, and it will at least slow him down, and at best get him fired.

As far as contacting her, I don’t think you should, especially with your current expectation of “acknowledgement” for “doing the right thing.” If I were that woman, I’d kind of want to say, “why the fuck didn’t you speak up?” and I would definitely not appreciate it. If, on the other hand, you wanted to reach out and ask her what you should have done, and what would have helped her the most, then yeah, maybe that could fly. But it would have to be done carefully.

Finally, I think it’s strange that you’d say her name. Is that some kind of signal to me that this is a fake question? In that case, please don’t send me fake questions; better to say what you’re sending me is a hypothetical. Otherwise, I’m not clear on why you’d name the victim at all. Is it an unnecessary pseudonym? I don’t get it.

Aunt Pythia


Dear Aunt Pythia,

What do you think of the heartbreaking story of Anna Stubblefield – to me it reads like a Shakespearean tragedy: brave woman leaves her white cis-male husband for a differently-abled man of color, but instead of being praised she is arrested and send to prison on ridiculous charges without any evidence.


Dear LIM,

Wow. Seriously?

Anna Stubblefield is super nuts. Yes, in a tragic way, but still. The best thing I can say about her is that she’s nuts and I don’t think she had evil intentions. But she’s still nuts.

Otherwise said: people have an amazing ability to believe what they want to believe. I know this because I worked in finance and I get that Lloyd Blankfein was not joking when he said that Goldman Sachs was “doing God’s work.” For reals some people are true believers, and they are the scariest people around.

Aunt Pythia


Readers? Aunt Pythia loves you so much. She wants to hear from you – she needs to hear from you – and then tell you what for in a most indulgent way. Will you help her do that?

Please, pleeeeease ask her a question. She will take it seriously and answer it if she can.

Click here for a form for later or just do it now:


Categories: Uncategorized

What are you thankful for in finance or economics?

It’s been a few days, I’ve been listening to Adele’s new album pretty much on loop while knitting and sewing curtains. So yes, it’s that nesting time of year, where we hunker down and seriously consume creamy spiked drinks.

Screen Shot 2015-11-24 at 7.12.52 AM.png

And by “we” I mean Americans, Canadians, Australians, and New Zealanders. Obviously we blame the hobbits on that last one.

Screen Shot 2015-11-24 at 7.13.05 AM.png

Well, here’s a question for you nog-quaffers: what are you thankful for from finance? I’ll extend it to the economy as well if you’d like.

The reason I’m asking is that this week, the Slate Money podcast I’m on is doing a special “thanksgiving” episode where we all talk about something we’re grateful for, and I’m having trouble coming up with something. Here’s what I’ve got so far:

  1. I’m grateful for consumer loans. After all, they help us out in rough times and allow us to invest in ourselves and our futures through mortgages and student loans. On the other hand, they also raise the price of everything through their availability. In fact I spent a couple of weeks ago on the show arguing that all college debt should be forgiven and that state colleges should be free. So I don’t think this works.
  2. I guess I’m thankful for inflation, in a sense. I mean, inflation makes it easier on debtors, since their debt is constantly dwindling in value, and it’s certainly better for an economy than deflation. But on the other hand, it can get out of hand and that’s bad, and it’s hard to control. So in the end I’m not actually all that excited by inflation.
  3. I could just be grateful for the entire financial system working at all. If you think about how much we depend on its functioning, to take out loans, to use our credit and debit cards, and to get paid monthly, it’s kind of amazing. On the other hand, if you think about the way finance deals with poor people, squeezing them for nickels and dimes, then you kind of lose respect. In fact it makes you want to be grateful for the CFPB instead, but that’s not financial enough.
  4. Finally, I’m thinking about how much I appreciate insurance. Yeah, I know there are plenty of problems with insurance (for example how cray-cray medical prices are for those without insurance, but I tend to blame a lack of reasonable transparency regulation on pricing in medicine on that, not insurance per se). But if you just think about how much insurance actually does for us, whether it’s medical or fire or car or life insurance, then you appreciate that it more or less functions as intended: to even out the bumpy risks of everyday life.

I’m still thinking about this question, and I’d love to hear your ideas!

Categories: Uncategorized

Do Charter Schools Cherrypick Students?

Yesterday I looked into quantitatively measuring the rumor I’ve been hearing for years, namely that charter schools cherrypick students – get rid of troublesome ones, keep well-behaved ones, and so on.

Here are two pieces of anecdotal evidence. There was a “Got To Go” list of students at one charter school in the Success Academy network. These were troublesome kids that the school was pushing out.

Also, I recently learned that Success Academy doesn’t accept new kids after the fourth grade. Their reasoning is that older kids wouldn’t be able to catch up with the rest of the kids, but on the other hand it also means that kids kicked out of one school will never land there. This is another form of selection.

Now that I’ve said my two examples I realize they both come from Success Academy. There really aren’t that many of them, as you can see on this map, but they are a politically potent force in the charter school movement.

Also, to be clear, I am not against charter schools as a concept. I love the idea of experimentation, and to the extent that charter schools perform experiments that can inform how public schools run, that’s interesting and worthwhile.

Anyhoo, let’s get to the analysis. I got my data from this DOE website, down at the bottom where I clicked “citywide results” and grabbed the following excel file:

With that data, I built an iPython Notebook which is on github here so you can take a look, reproduce my results with the above data (I removed the first line after turning it in to a csv file), or do more.

From talking to friends of mine who run NYC schools, I learned of two proxies for difficult students. One is ‘Percent Students with Disabilities’ and the other is ‘Percent English Language Learners’ (I also learned that charter schools’ DBN code starts with 84). Equipped with that information, I was able to build the following histograms:

Percent Students with Disabilities non-charter

Percent Students with Disabilities, non-Charter

Percent Students with Disabilities charter

Percent Students with Disabilities, Charter

Percent English Language Learners non-charter

Percent English Language Learners, non-Charter

Percent English Language Learners, Charter.png

Percent English Language Learners, Charter. Please note that the x-axis differs from above.

I also computed statistics which you can look at on the iPython notebook. Finally, I put it all together with a single scatterplot:

scatter disabilities, english learners

The blue dots to the left and all the way down on the x-axis are mostly test schools and “screened” schools, which are actually constructed to cherrypick their students.

The main conclusion of this analysis is to say that, generally speaking, charter schools don’t have as many kids with disabilities or poor language skills, and so when we compare their performance to non-charter schools, we need to somehow take this into account.

A final caveat: we can see just by looking at the above scatter plot that there are plenty of charter schools that are well inside the middle of the blue cloud. So this is not a indictment on any specific charter school, but rather a statistical statement about wanting to compare apples to apples.

Update: I’ve now added t-tests to test the hypothesis that this data comes from the same distribution. The answer is no.

Screen Shot 2015-11-19 at 11.06.13 AM

Those very small numbers are the p-values which are much smaller than 0.05. Other t-tests give similar results (but go ahead and try them yourself)

Categories: Uncategorized

Line Edits

Right now I’m eyeball deep in line edits for my book, Weapons of Math Destruction: how big data increases inequality and threatens democracy.

Or rather, I’m in the phase of minor(ish) edits from my editor (post-existential threats, anyway, and that’s a big deal!) which is before the next phase, where I’ll be dealing with issues from the actual line editor, the person who knows all about commas and what gets italicized versus quoted and so forth.

Then come the galleys, and along the way of course I help choose a cover design. After that the blurbs start (who should I ask?) and they print a bunch of copies in China and have it all shipped back here by boat. The process takes months and it’s all new to me.

Point is, my brain is completely occupied with this stuff, which is the opposite of sexy but on the other hand is exactly why the book will be better (hopefully!) than a blog – it will be actually carefully edited.

Everyone, fingers crossed, but the tentative launch date is September 5th of 2016. I know, it’s forever from now, but at least it’s an actual date. I’m already planning the party.

Categories: Uncategorized

Open Data Conference this Friday

I’m going to be on a panel Friday at a conference called Responsible Use of Open Data in Government and the Private Sector, which is being co-organized by Berkeley and NYU and is being held at NYU this Thursday evening and Friday all day.

The agenda is here. The first keynote, on Thursday night, is to be given by the Chief Analytics Officer of New York City, who I’m interested to hear from. I’m wondering what de Blasio’s administration is up to with respect to data and predictive modeling.

I’m going to miss Panel 2 because of my podcast taping, which is a shame, but I’m looking forward to Panel 1 which will discuss consequences of data sharing, unintended as well as intended: privacy concerns, discrimination concerns, and so on.

I’m on Panel 3, which with Panel 4 is devoted to the topic of private data use and collection versus the public good. The focus is on health care data and smart cities, but I will probably veer off to all kinds of ways that private companies use data to the detriment of the public, and how that should change.

Panel 5 discusses platforms for sharing data as well as the proposed governance of shared data. To be honest I’m a bit skeptical of the concept I’ve heard floated about recently that private companies will “donate” their data for the public good, but I’d love to be wrong.

Registration is free and open and available here.

Categories: Uncategorized

Get every new post delivered to your Inbox.

Join 3,682 other followers