The SHSAT matching algorithm isn’t that hard

My 13-year-old took the SHSAT in November, but we haven’t heard the results yet. In fact we’re expecting to wait two more months before we do.

What gives? Is it really that complicated to match kids to test schools?

A bit of background. In New York City, kids write down a list of their preferred public high schools that are not “SHSAT” schools. Separately, if they decide to take the SHSAT, they rank their preferences for those, which fall into a separate category and which include Stuyvesant and Bronx Science. They are promised that they will get into the first school on the list that their SHSAT score allows them to.

I often hear people say that the algorithm to figure out what SHSAT school a given kid gets into is super complicated and that’s why it takes 4 months to find out the results. But yesterday at lunch, my husband and I proved that theory incorrect by coming up with a really dumb way of doing it.

  1. First, score all the tests. This is the time-consuming part of the process, but I assume it’s automatically done by a machine somewhere in a huge DOE building in Brooklyn that I’ve heard about.
  2. Next, rank the kids according to score, highest first. Think of it as kids waiting in line at a supermarket check-out line, but in this scenario they just get their school assignment.
  3. Next, repeat the following step until all the schools are filled: take the first kid in line and give them their highest pick. Before moving on to the next kid, check to see if you just gave away the last possible slot to that particular school. If so, label that school with the score of that kid (it will be the cutoff score) and make everyone still in line erase that school from their list because it’s full and no longer available.
  4. By construction, every kid gets the top school that their score warranted, so you’re done.

A few notes and one caveat to this:

  1. Any kid with no schools in their list, either because they didn’t score high enough for the cutoffs or because the schools all filled up before they got to the head of the line, won’t get into an SHSAT school.
  2. The above algorithm would take very little time to actually run. As in, 5 minutes of computer time once the tests are scored.
  3. One caveat: I’m pretty sure they need to make sure that two kids with the same exact score and the same preference would both either get in or get out (because think of the lawsuit if not). So the actual way you’d implement the algorithm is when you ask for the next kid in line, you’d also ask for any other kid with the same score and the same top choice to step forward. Then you’d decide whether there’s room for the whole group or not.

So, why the long wait? I’m pretty sure it’s because the other public schools, the ones where there’s no SHSAT exam to get in (but there are myriad other requirements and processes involved, see e.g. page 4 of this document) don’t want people to be notified of their SHSAT placement 4 months before they get their say. It would foster too much unfair competition between the systems.

Finally, I’m guessing the algorithm for matching non-SHSAT schools is actually pretty complicated, which is I think why people keep talking about a “super complex algorithm.” It’s just not associated to the SHSAT.

Categories: Uncategorized

O’Neil family anthem

I’m working through final edits today, and it’s terribly stressful, so I’m glad I spent last night with my three sons listening to their favorite music.

The most important songs to share with you come from Rob Cantor, who just happens to be incredibly talented. I want to see him live with my kids but so far I haven’t found out about any concerts he’s planning. Here’s my fave Cantor tune (obviously, because I’m an emo):

Next, my 7-year-old’s favorite Cantor tune, Shia LaBeouf:

And my 13-year-old’s favorite, Old Bike:

Just in case you think we only listen to this guy, I wanted to share with you the song that all of us sing regularly, for whatever reason. We make up reasons to sing this song, and it can fairly be called the O’Neil/de Jong family anthem. It’s called First Kiss Today, and made – or constructed anyway – by Songify This. Bonus footage from Biden:

Categories: Uncategorized

Surveillance and wifi in NYC subways

This morning I heard some news from the Cuomo administration (hat tip Maxine Rockoff).

Namely, we’re set to get mobile tickets in the NYC subways:


In addition, they’re saying we will have wi-fi in the stations, as well as surveillance cameras on all the subways and buses. Oh, and charging stations for USB chargers.

My guess: the surveillance cameras will continue to function long after the USB chargers get filled with gum.

Categories: Uncategorized

Which Michigan cities are in receivership?

Yesterday at my Occupy meeting we watched a recent Rachel Maddow piece on the suspension of democracy in Michigan:

If it’s too long, the short version is that instead of having elected officials, some specially chosen towns have instead ‘Emergency Managers,’ who do things like save money by pumping in poisonous water.

So, as usual, my group had a bunch of questions, among them: what is the racial make-up of the towns who are in receivership?

Well first, here’s a list of towns currently under receivership, which I mapped on Google Maps:

Screen Shot 2016-01-11 at 7.47.53 AM

You can interact with my map here.

And next I looked at a census map of where black people live in Michigan:

Screen Shot 2016-01-11 at 7.47.12 AM

Taken from this website which displays 2010 census data

I also wanted to zoom into the Detroit area:

and compared that to the municipalities under receivership in the area:

Screen Shot 2016-01-11 at 7.45.46 AM.png

Take a closer look here.

Just in case you’re wondering, that teal spot on the left is exactly where the Inkster is. And Wayne County’s government is also in receivership, but it’s a county, not a town.

Categories: Uncategorized

The economics of weight loss

Tomorrow’s recording of Slate Money will concern New Year’s resolutions. We’re talking about gym memberships and health classes, Fitbits and other “quantified self” devices, and the economics of Weight Watchers and other weight loss industry companies.

I’m in charge of researching the weight loss industry, which was estimated at $64 billion in 2014. That’s huge, but actually it’s dwindling, as people formally diet less often and instead try to informally “eat healthy.”

In fact, Weight Watchers is an old person’s company; the average age is 48, and Oprah’s recent help notwithstanding, younger people are more likely to be interested in quantified self devices which can track calories burned and so on than they are in getting together in person and talking with people about the struggle.

Also, Obamacare doesn’t cover weight programs outside of a doctor’s office, so that has dried up funds as well.

This is good news, because there’s really no evidence that weight loss programs work long-term, but they are expensive. They keep doing studies but they never come out with any positive results beyond 12 months. That’s because they don’t have any evidence.

For example, if I joined Weight Watchers, I’d pay $44.95 per month, although I get refunded if I lost 10 pounds quickly enough. I’d be able to go to meetings two blocks away from my house every Wednesday. The plan will auto-renew and charge my credit card unless I cancel it, which is tantamount to admitting defeat. I’m wondering what the statistics are on people who are paying monthly but no longer attending meetings.

If you want an extreme example of the current dysfunction around dieting, look no further than the show The Biggest Loser, which the Guardian featured recently with the tag line, “It’s a miracle no one has died yet.”

So, given how much money people put into this stuff even now, why are they doing it? After all, if we were expected to pay a doctor to set the bones of our broken leg, but it only worked for a few months before our leg started breaking again, we’d call the doctor a quack and demand our money back. But somehow with diets it’s different. Why?

I have a complicated theory.

The first level is the “I’m an exception” law of human nature, whereby everyone thinks they somehow will prove to be an exception to statistical rules. It’s the same magical reasoning that makes people buy lottery tickets when they know their chances of winning are slim, and they even know their expected value is negative.

The second level is entertainment. This is also taken directly from the lottery mindset; even if you know you’re not winning the lottery, the momentary fantasy of possibly winning is delicious, and you relish it. The cost of that fantasy is a small price to pay for the freedom to believe in this future for one day.

I think the same kind of thing happens when people join diets. They get to fantasize about how great their lives will be once they’re finally thin. And of course the prevalent fat shaming helps this myth, as does the advertising from the diet industry. It’s all about imagining a “new you,” as if you also get a personality transplant along with losing weight.

But there’s something even more seductive about weight loss regimens that lotteries don’t have, namely public support. When someone announces that they’re on a diet, which happens pretty often, everyone around them has been trained to “be supportive” in their endeavor. At the same time, people rarely announce they’ve gone off their diet. So you’ve got asymmetrical dieting attention.

That attention also has a moral flavor to it. Since people are expected to have control over their weight, they are given moral standing when they announce their diet; it is a sign they are finally “taking control.” Never mind that their chances of long-term success are minimal.

The third and final phase, which is the saddest, is guilt. Because we’ve bought in to the idea that people have direct control over their weight, when people end up giving up, they feel personally guilty and end up paying extra money for basically nothing in return.

Of course, no part of this story is all that different from the story of gym memberships or even Fitbit-like device acquisition. Seen together, it’s just a question of what quasi-moralistic self-help fad happens to be popular at any given moment. And there’s tons of money in all of it.

Categories: Uncategorized

Finishing up Weapons of Math Destruction

Great news, you can now pre-order my book on Amazon. That doesn’t mean it’s completely and utterly finished, but nowadays I’m working on endnote formatting rather than having existential crises about the content. I’m also waiting to see the proposed design of the book’s cover, for which I sent in a couple of screen shots of my python code. And pretty soon I get to talk about stuff like font, which I don’t care about at all.

But here’s the weird part. This means it’s beginning.

You see, when you’ve lived your life as a mathematician and quant, projects are usually wrapped up right around now. You do your research, give talks, and finally write your paper, and then it’s over. I mean, not entirely, because sometimes people read your paper, but actually that mostly doesn’t happen for the published version but instead with the preprint archive. By the time you’ve finished submitting your paper, you’re kind of over your result and you want to move on.

When you do a data science project, a similar thing happens. The bulk of the work happens pre-publishing. Once the model is published, it’s pretty much over for you, and you go on to the next project.

Not so with book publishing. This whole process, as long and as arduous and soul-sucking as it’s been, is just a pre-cursor to the actual event, which is the publication of the book (September 6th of next this year). Then I get to go around talking about this stuff with people for weeks if not months. And although I’m very familiar with the content, the point of writing the book wasn’t simply for me to know the stuff and then move on, it’s for me to spread that message. So it’s also exciting to talk to other people about it.

I also recently got a booking agent, which you can tell if you’ve noticed my new Contact Page. That means that when people invite me to give a talk they’re going to deal with her first, and she’s going to ask for real money (or some other good reason I might want to do it). This might offend some people, especially academics who are used to having people available to donate their time for free, but I’m really glad to have her, given how many talk requests I get on a weekly basis.

Categories: talks

Racial identity and video games

Yesterday I stumbled upon an article entitled The Web is not a post-racial utopia, which concerns a videogame called Rust. It explains that when player enters the world of the game, they are “born” naked and alone. The game consists of surviving the wilderness. I’m guessing it’s like a grown-up version of Minecraft in some sense.

In the initial version of the game, all the players were born bald and white. In a later version, race was handed out randomly. And as you can guess, the complaints came pouring in after the change, as well as a marked increase in racially hostile language.

This is all while blacks and Hispanics play more videogames than whites. They were not complaining about being cast as a white man in the initial version, because it’s so common. Videogame designers are almost all white guys.

I’ll paraphrase from a great interview with one of the newest Star Wars heros John Boyega when I say, I’m pretty sure there wouldn’t have been any complaints if everybody were born a randomly colored alien. White people are okay with being cast as a green alien avatar, but no way they’re going to be cast as a black man. WTF, white people?

Of course, not everyone’s complaining. In fact the reactions are interesting although extreme. They’re thinking of setting up analytics to track the reactions. They’re also thinking of assigning gender and other differences randomly to avatars. And by the way, it looks like they’ve recently been attacked by hackers.

For what it’s worth, I’d love to see men in video games dealing with getting their period. Actually, that’s a great idea. Why not have that as part of the 7th grade ‘Health and Sexuality’ curriculum for both boys and girls? Those who advance to the next level can experience being pregnant and suffering sciatica. Or maybe even hot flashes and menopause, why not?

Categories: Uncategorized

Get every new post delivered to your Inbox.

Join 3,746 other followers