Archive for March, 2012

Hip Hop’s Cambrian Explosion: Part 1

March 12, 2012 15 comments

On her January 28th post entitled Does Hip Hop Still Exist? Mathbabe wrote:

“My oldest friend sent me some mixed CDs for Christmas. I listened to them at work one recent morning, and although I like a few songs, many of them were downright jarring. I mean, so syncopated! So raw and violent! What the hell is this?! It was hip-hop, I think, although that was a word from some far-away time and place. Does hip-hop still exist?”

Fortunately for me, I am that oldest friend, mixer of said CD, and guest blogger this week, here to answer Mathbabe’s question with the first of a three-part post entitled Hip Hop’s Cambrian Explosion.


I discovered Hip Hop around the same time I discovered Mathbabe. In 1987, Hip Hop was a toddler living in Brooklyn while Cathy and I were teenagers living in suburban Massachusetts.  As I walked home from school one afternoon, I popped Boogie Down Production’s debut cassette into my walkman and snapped to attention as KRS One delivered a high-energy critique of public schooling’s systematic omission of Black history from the curriculum. As I listened I found myself considering for the first time the ways in which I had been raised on a steady academic diet of European and American histories and literatures, with no mention of those of Africa, Latin America, or Asia. These were entire continents and peoples whose histories were tacitly deemed peripheral to the central drama of whiteness. I listened closely as KRS One, aptly known as “The Teacher,” educated me about the people studiously ignored in my history textbooks. Here is a delightfully dated video of that first song, You Must Learn:

I am now nearly 40 and recently had the opportunity to meet KRS One at a concert in Berkeley, where I was able to thank him in person for supplementing my education. He is as dynamic as I remember, still using the mic as a vehicle to teach critical thinking, still building community by inviting up-and-coming rappers onto the stage to improvise with him, still innovating by rapping over electric violins spilling amplified Mozart over the surging audience.  In this photo I took from stageside he reaches out to connect with the crowd:


And here I am, looking up at him.


Photo by Hugo Garcia, aka Steelo

As you can see in the photo, I plainly admire him, as I do any iconoclast who has the audacity and clarity to say so when the Emperor has no clothes. So as an avid fan of Hip Hop, I’d like to appeal its case for those of you who are new to the genre or are considering giving it a second listen. Why should you bother listening to Hip Hop? And what exactly is Hip Hop anyway? I offer this primer as a paean.

1. Hip Hop is political. Hip Hop gained national attention in 1989 when Public Enemy’s Fight the Power piqued the paranoia of white America. The now-classic ghetto anthem opens with Martin Luther King’s lilting oratory, not the more tepid, politically-milktoast MLK Jr. of the official public holiday, but the radical MLK Jr, who exhorts Americans not only to refuse to serve in the U.S. Army, but to switch allegiance to fight alongside the Viet Cong.

ImageChuck D and Flavor Flav at Yoshi’s in San Francisco. “Most of my heroes don’t appear on no stamps.” – Chuck D. Photo by Cherie Chavez

True to its origins, Hip Hop remains today the artistic genre of choice and the voicebox for people pushed to the margins of power by historical and social forces. And it’s not afraid to name those forces. Paramount among the themes tackled in Hip Hop is that of white supremacy, a topic — a phrase even — that tends to make white people uncomfortable.  When rapper Brother Ali released Uncle Sam Goddamn, an overview of American racism — past and present — cell phone company Verizon responded by revoking its sponsorship of his tour. Corporations typically don’t profit by talking about racism, unless it’s in that “Rainbow Nation” manner of Benetton, which carefully eschews analysis of power relations. The video for Uncle Sam Goddamn includes some powerful historical footage.

Another of Hip Hop’s recurring themes is poverty. As Somalian-born rapper (and personal favorite) K’Naan explains:

…I remember when I was 7
When rap came mysteriously and made me feel 11
It understood me, and made my ghetto heaven
I understood it as the new poor people’s weapon.

Smart 7-year-old. The excerpt is from The African Way, a funky fusion of American-style rap vocals and East African drum rhythms. As K’Naan recounts in several of his autobiographical songs, he learned English by listening to rap music (it’s no coincidence he sounds so much like Eminem) in order to have a forum for speaking about the violence he experienced as a child growing up in his native Mogadishu. His beautiful Blues for the Horn is both lament and homage to the Horn of Africa. He narrates the story of Somalia himself so that no one can “make a mockery of our struggle like Hollywood plans to.”  And despite the seriousness of his purpose, he carries on another of Hip Hop’s traditions: its sense of humor. Describing Mogadishu, he quips:

If you bring the world hoods to a seminar
We’re from the only place worse than Kandahar —

And that’s kinda hard!

In the song Somalia, he reminisces about his childhood:

We used to take barbed wire
Mold it around discarded bike tires,
Roll em down the hill in foot blazin’ –
Now that was our version of mountain bike racing!
Do you see why it’s amazing
When someone comes out of such a dire situation
And learns the English language,
Just to share his observations?
Probably get a Grammy without a grammar education.

Hip Hop has an unapologetic working class hero sensibility, like John Lennon’s Working Class Hero, but edgier.  How’s this for edgier? 5 Million Ways to Kill a C.E.O.

The racism and classism that inhere in our justice system are the targets of Lauryn Hill’s epic rant in The Mystery of Iniquity. She seems to enter a poetic trance as she excoriates the American judicial system in a style that calls to mind the dogged dirge of Allen Ginsberg’s Howl. This is but a tiny excerpt of Hill’s stream-of-consciousness dressing down:

Do we expect the system made for the elect
To possibly judge correct?
Properly serve and protect?
Materially corrupt
Spiritually amuck…

Mafia with diplomas keeping us in a coma trying to own a piece
of the American Corona.
The revolving door:
Insanity every floor
Skyscraping, paper chasing,
What are we working for?
Empty traditions
Reaching social positions
Teaching ambition to support the family superstition?

With a bass voice like Barry White, rapper Lyrics Born questions our funding priorities in Stop Complaining:

I pay my taxes when I’m asked to.
I’m not enthusiastic about it, but shit, I make it happen.
Yeah, it’s last minute, but goddammit they cash it.
(“This is fiscal harassment, they keep touchin my assets!”)
Now I imagine I might be feeling different about it
If it was given outright, witness it helping somebody
But it just so happens in life, the school district’s too crowded
It ain’t no teachers in sight, that’s why the kids are so rowdy.
I just imagine some asshole with glasses on up at the Capitol
One of a thousand pawns packed in an office cramped up like animals,
Pictures of his sister, his mixture Lapso Apso-poodle
His 2.6 kids, and the missus thumbtacked to his cubicle
So damn detached from the average man’s planet, he cain’t fathom
That we could ever be anything other than stats, fat and taxable
He’s gettin his usual ritual 2 o’clock Cup of Noodles on
While he’s fuckin you on your W2, his John Denver music on.

The ongoing disparities in K-12 schooling and access to higher education are a common theme in Hip Hop.  Shad K, who pursued his career in Hip Hop while simultaneously earning a Masters degree in Business, writes in Exile:

We’re taught not to question the status quo cuz the masses never get heard
unless you’re established
with expert professors in dress-shirts
and glasses that lecture to classes
from lecterns
where next term the best third will pass and
earn cash working as
desk clerks for the best firms in Manhattan.

Shad was born in Kenya, the son of Rwandan refugee Bernadette Kabango, whose autobiographical poetry he incorporates in the chilling song I’ll Never Understand. Shad’s mother reads in her own voice, telling the story of her family’s murder in the 1994 genocide, addressing those who committed violence against her, and raising questions about the possibility of forgiveness. Shad’s rap vocals interlace with his mother’s voice, interjecting questions about whose genocides matter and whose don’t. The video of I’ll Never Understand includes footage that defies commentary from the Rwandan massacres. How could one begin to talk about such atrocities but through art?  As writer Victor Hugo observed, “Music expresses that which cannot be said and on which it is impossible to be silent.”

The musical conversation between mother and son brings me to Hip Hop’s next defining characteristic.

2. Hip Hop is intergenerational. One of the stylistic and structural conventions of the genre is sampling.  The contemporary artist layers his/her vocal track over a repeated excerpt of a melodic track — the sample — by an older artist, alive or dead. The tradition of sampling older artists from a generation prior (e.g., Nina Simone, Ray Charles, etc.) began perhaps for practical reasons, as access to older songs was not limited by royalties and copyright. Regardless of the motivation, sampling has the effect of creating intergenerational dialogue, a musical conversation across time.

Hip Hop has its roots in the oral traditions of West Africa, where people still live in active relationship with their ancestors and respect for elders is a core cultural value. Hip Hop carries on this tradition of talking with the dead and honoring those who have paved the way. Erick Sermon’s Just Like Music is an ode to music’s healing power (“I wish music could adopt me!”), sampling musical legend Marvin Gaye. Sermon cleverly interweaves his contemporary vocals over Marvin Gaye’s melodies so that at one point they appear to be in direct conversation:

Sermon:  Is that true, Marvin?

Gaye: Yeeeeeeaaah!

It’s no surprise that songs from the Civil Rights movement provide a rich pool of sampling material. Movin’ Forward by Collective Efforts samples Civil Rights song Ain’t Gonna Let Nobody Turn Me Around, and references Eyes on the Prize, the comprehensive and inspiring documentary on the history of the Civil Rights movement.

Fort Minor’s Kenji tells the story of the United States internment of Japanese Americans during World War II and honors elder survivors by incorporating original audio interviews of former internees.

I’ve talked a lot about politics here, but if Hip Hop were all politics, it would be two-dimensional, flat like a Soviet-era agitprop poster (you know the ones of the workers with the disproportionately huge fists). It is the next characteristic which gives Hip Hop its complexity, dynamism, and multi-dimensionality.

3. Hip Hop is poetry. On steroids. A bit like aural caffeine. While I first got hooked on rap for its incisive outsider critiques, I equally enjoy the verbal acrobatics and linguistic playfulness of the form. I’m a word nerd, a sesquipedalian, easily wooed by an orator who can wield an adjective, so the highly verbal genre holds a natural appeal for me.

Others have told me that they find the language of Hip Hop itself a barrier to listening, the lyrics so rapid-fire and abstruse as to be unintelligible to the uninitiated.  Perhaps so, but many of us found Shakespeare difficult to parse at the beginning but ultimately worth the effort. As in any specialized field, rap has created a unique language, its own grid of intelligibility, with webs of cross-references and insider lingo that can be opaque to newcomers to the genre. Just as you would read Shakespeare with a dictionary at your side as a reference to make meaning of the text, rap music lyrics must be studied with the right reference materials at hand.  The rate of word evolution in rap music is rapid, which can make it difficult to keep up with the neologisms. Fortunately, there’s Urban Dictionary, with user-entered definitions being added continuously.

Approaching Hip Hop with the same spirit of literary criticism used to analyze Shakespeare or T.S. Eliot reveals that its poets employ all of the literary devices standard in the craft: alliteration, assonance, consonance, onomatopoeia, irony, and variation. They amplify the metrical effects of diction and syntax with the percussion and syncopation provided by the music itself. It is the interplay of the two (or more often, multiple) meters – the rhythm inherent in language, and the nonvocal rhythms of layered drums, piano, trumpet, or kora – that so stimulates the linguistically-inclined mind.

What rap adds to the traditional toolbox available to written poetry is a tool only available to the spoken word artist: something called flow. Flow is a bit difficult to describe, but you know it when you hear it. Flow is that state a rapper gets into when the syllables are tumbling off the tongue in a waterfall of words, the cadences rising and falling, surprising and mesmerizing. Flow is that trance artists crave, that moment when the rational mind steps aside, time telescopes, and the artist becomes hypnotized by the presence of the muse. Flow is when the music comes through the musician rather than from the musician.  Cee-Lo has it.  I couldn’t agree more with his self-review in One for the Road: “Oh, his way with words! I want seconds and thirds!”

I’ll let Cee Lo have the last word for now, and will continue tomorrow with Part 2 of Hip Hop’s Cambrian Explosion.

Categories: guest post, Uncategorized

Why Larry Summers lost the presidency of Harvard

Some people still think Larry Summers got fired from being the president of Harvard because of the ridiculous comments he made about women in math (see my post about this here) or because of the comments he made about Cornel West. Actually, the truth is something worse, and for which he should actually be in jail. It’s also something that makes Harvard look bad, so maybe that’s why it’s less known.

The subtitle of this post is: Why Larry Summers shouldn’t be made head of the World Bank.

I was inspired to write this by being disgusted at continued rumors that he could get yet another prestigious job. It’s like this guy can’t fail spectacularly enough! Let’s give him another chance!

Let’s set the record straight: Summers was directly involved with defrauding the U.S. Government (see below) and Russia. He admitted to not understand conflict of interest issues (see below). It is particularly appalling, knowing these things, that he would be considered for the World Bank head, which presumably requires nuanced understanding of such issues.

I’m using this article, entitled “How Harvard Lost Russia,” and written in 2006 in Institutional Insider (II), as a reference. More on that article and how it led to getting Summers fired below. And by the way, I’m not claiming this story is completely unkown: see this wikipedia article for a quick overview, for example, in addition to the II article. I just think it needs reviving at this crucial moment, before Summers gets more toys to play with.


So why did Summers lose his job at Harvard? It was because of his protecting a buddy, a fellow economist at Harvard named Andrei Shleifer.

Andrei Shleifer managed to get put in charge of helping Russia privatize stuff in the mid 1990’s. His mission was to make things more useful and transparent to the infant capitalist system. Through his wife and friends, Shleifer instead orchestrated a boondoggle on Russia. He invested money through his wife and helped his friend Jonathan Hay and his lover and friends invest theirs, and set up the very first mutual fund as well as thwarting the efforts of other people to set up their own funds. All of these things were strictly against the conflict of interest policy they were working under.

Shleifer got in trouble, and the U.S Government sued and won against Harvard and Shleifer. From the article:

The judge determined that Shleifer and Hay were subject to the conflict-of-interest rules and had tried to circumvent them; that Shleifer engaged in apparent self-dealing; that Hay attempted to “launder” $400,000 through his father and girlfriend; that Hay knew the claims he caused to be submitted to AID were false; and that Shleifer and Hay conspired to defraud the U.S. government by submitting false claims.

On August 3, 2005, the parties announced a settlement under which Harvard was required to pay $26.5 million to the U.S. government, Shleifer $2 million and Hay between $1 million and $2 million, depending on his earnings over the next decade. Shleifer was barred from participating in any AID project for two years and Hay for five years. Shleifer and Zimmerman were required by terms of the settlement to take out a $2 million mortgage on their Newton house. None of the defendants acknowledged any liability under the settlement. (Forum Financial also settled its lawsuit against Harvard, Shleifer and Hay under undisclosed terms.

Summers and Shleifer

Summers was good friends with this criminal, and used his position to protect him. From the article:

Shleifer remained close to his friend and mentor Summers; they talked to and saw each other frequently and continued vacationing together in the summer on the Cape. Then it became known in early 2001 that Summers was on the short list of candidates to succeed Neil Rudenstine as the president of Harvard University. Shleifer and Zimmerman began campaigning for Summers to get the Harvard post, giving meet-and-greet parties for him at their home. Summers stayed with them when he visited Harvard.

In March 2001, Summers was named president of Harvard. Shleifer, who had been courted by New York University’s Stern School of Business, decided to stay put.

Having his close friend as his boss would turn out to be quite helpful to Shleifer. Summers asserted in his deposition that he recused himself from any involvement in the university’s handling of the Shleifer matter, but the new president stayed involved anyway. Early in his presidency he told the dean of the faculty of arts and sciences, Jeremy Knowles, to keep Shleifer at Harvard.

“I expressed to Dean Knowles,” Summers testified in a deposition in 2002, “. . . that I was concerned to make sure that Professor Shleifer remained at Harvard because I felt that he made a great contribution to the economics department . . . and expressed the hope that Dean Knowles would be attentive to that. . . . I think he recognized and shared the concern.”

“Conflict of interest issues should be left to the lawyers” says Summers

This is the testimony that says to me, in no uncertain terms, that Summers cannot be put in charge of something politically sensitive:

Summers said conflict-of-interest “issues,” in his Washington experience, were “left to the lawyers.” He said he was sensitive to “ethics rules,” but testified that “in Washington I wasn’t ever smart enough to predict them . . . things that seemed very ethical to me were thought of as problematic and things that seemed quite problematic to me were thought of as perfectly fine. . . .”

More intervention on behalf of Shleifer

Maybe you’d think that getting sued by the US Government and losing $40 million might lose your job as a Harvard Professor. But you’d be wrong:

Knowles tells Institutional Investor that he does not remember Summers’ approaching him about Shleifer. “I don’t recall this particular conversation, but the president and I shared the goal of recruiting and retaining the best faculty, so it would have been perfectly natural for us to mention to each other the names of people that we certainly wouldn’t want to lose.” However, not long after Summers says he intervened on the professor’s behalf, Knowles promoted Shleifer from professor of economics to a named chair, the Whipple V.N. Jones professorship.

Shleifer’s legal position changed on June 28, 2004, when Judge Woodlock ruled that he and Hay had conspired to defraud the U.S. government and had violated conflict-of-interest regulations. Still, there was no indication that the Summers administration had initiated disciplinary proceedings. To the contrary, efforts were seemingly made to divert attention from the growing scandal. The message from the top at Harvard was, “No problem — Andrei Shleifer is a star,” says one senior Harvard figure.

The Summers-Shleifer friendship flourished. They spoke on the phone more than once a day, on average. Two months after the court ruling against Shleifer, he hosted Summers at a break-the-fast dinner on Yom Kippur.

One instance was a meeting early in the academic year that began in September 2004, less than two months after the federal court formally adjudicated Shleifer’s liability for conspiring to defraud the U.S. government. A faculty member asked Kirby why Harvard should defend a professor who had been found liable for conspiring to commit fraud. The second confrontation came early in the current academic year when another professor asked Kirby why Harvard should pay a settlement of $26.5 million and legal fees estimated at between $10 million and $15 million for legal violations by a single professor and his employee, about which it was unaware. On both occasions Kirby is said to have turned red in the face and angrily cut off discussion.

On at least one other occasion, Summers himself told members of the faculty of arts and sciences that the millions of dollars that Harvard paid in damages did not come from the budget of the faculty of arts and sciences, but didn’t say where the money came from. Those listening inferred he meant that the matter shouldn’t be of concern to the faculty and that they shouldn’t raise it, a curious notion, given that Shleifer was one of their own.

A spokesman for Summers said he was “unable to schedule” an interview with Summers for II in December, when this article was being prepared. As the lawsuit was against the university, not just the faculty of arts and sciences, the settlement came from “university funds available for these purposes,” the spokesman added.

Shleifer has never acknowledged doing anything wrong. Summers has said nothing. And so far as is known, there has been no internal investigation or sanction. “An observer trying to make sense of the University’s position on Shleifer, Ogletree and Tribe is driven to an unhappy conclusion. Defiance seems to be a better way to escape institutional opprobrium than confession and apology. . . . And most of all being a close personal friend of the president probably does one no harm.”

The article gets Summers fired

An anonymous person got a bunch of copies of the II article and stuck one in every Harvard faculty’s mailbox the morning of the no-confidence vote that got Summers ousted.

And just in case you’re wondering, here’s the website of Sheifer, still on faculty of Harvard.

Categories: finance, news

VAM versus what?

A few astute readers pointed out to me that in the past few days I both slammed the Value-added teacher’s model (VAM) and complained about people who reject something without providing an alternative. Good point, and today I’d like to start that discussion.

What should we be doing instead of VAM?

First of all, I do think that not rating teachers at all is better than the current system. So my “compare the the status quo” argument goes through in this instance. Namely, VAM is actively discouraging teachers whereas leaving them alone entirely would neither discourage or encourage anyone. So better than this.

At the same time, I am a realist, and I think there should be, ultimately, a system of evaluating teachers, just as there is a system for evaluating me at work. The difference between my workplace, of 45 people, and the NYC public schools is scale. It makes sense to have a very large and consistent evaluation system in the NYC public schools, whereas my job can have an ad hoc inconsistent system without it being a problem.

There’s another problem which is nearly impossible to tease from this discussion. Namely, the fact that what’s going on in NYC is a disingenuous political game between Bloomberg and the teacher’s union. Just to emphasize how important that fight is, let’s keep in mind that as of now, although the union is much weaker than it historically has been, it still has the tenure system. So any model, VAM or not, of evaluation is somewhat irrelevant for “removing bad teachers” given that they have tenure and tenure still means something.

Probably the best way to decouple the “Bloomberg vs. union/tenure” issue (a massive one here in NYC) from the “VAM versus other” question is to think nationally rather than citywide.

The truth is, the VAM is being tried out all over the country (although I don’t have hard numbers on this) and the momentum is for it to be used more and more. I predict within 10 years it will be done systematically everywhere in the country.

And, sadly, that’s kind of my prediction whether or not the underlying model is any good or not! The truth is, there is a large contingent of technocrats who want control over the evaluation system and believe in the models, whether or not they are producing pure noise or not. In other words, they believe in “data driven decisioning” as a holy grail even though there’s scant evidence that this will work in schools. And they also don’t want to back down now, even though the model sucks, because they feel like they’ll be losing momentum on the overall data-driven approach.

One thing I know for sure is that we should continue to be aware of how badly the current models are, and I want to set up an open source version of the models (see this post to get an idea how it could work) to exhibit that. In other words, even if we don’t turn off the models altogether, can’t we at least minimize their importance while their quality is bad? The first step is to plainly exhibit how bad they are.

It’s hard for me to decide what to do next, though. I’m essentially a modeler who is hugely skeptical of models. In fact, I don’t think using purely quantitative models to evaluate teachers is the right thing to do, period. Yet I feel like if it’s definitely going to happen, better for people like me to be in the middle of it, pointing out how bad the proposed (or in use) models are actually performing, and improving them.

One thing I know I’d do if I were to be put in charge of creating a better model: I’d train on data where the teacher is actually rated as a good teacher or not. In other words, I wouldn’t proxy “good teacher” by “if your students scored better than expected on tests”. A good model would be trained on data where there would be an expert teacher scorer, who would go into 500 classrooms and carefully evaluate the actual teachers, based on things like whether the teacher asked questions, or got the kids engaged, or talked too much or too little, or imposed too much busy work, etc. Then the model would be trying to mimic this expert.

Of course there are lots of really complicated issues to sort out- and they are *totally unavoidable*. This is why I’m so skeptical of models, by the way: people think you can simplify stuff when you actually can’t. There’s nothing simple about teaching and whether someone’s a good teacher. It’s just plain complex. A simple model will be losing too much information.

Here’s one. Different people think good teaching is different. A possible solution: maybe we could have 5 different “expert models” based on different people’s definitions of good teaching, and every teacher could be evaluated based on every model. Still need to find those 5 experts that teachers trust.

Here’s another. The kind of teacher-specific attributes collected for this test would be different from the VAM- things that happen inside a classroom (like percentage of time teacher talks vs. student, the tone of the discussion, the number and percentage of kids involved in the discussion, etc,) and are harder to capture accurately. These are technological hurdles that are hard.

I think one of the most important questions is whether we can come up with an evaluation system that would be sufficiently reasonable and transparent that the teachers themselves would get on board.

I’d to hear more ideas.

Sausage Wall

I’m getting ready to go to Amsterdam this Sunday (get ready for an exciting guest blogger while I’m gone!).

I’ve been to Amsterdam a bunch of times since hooking up with my big-nosed Dutch husband, and I enjoy our visits to his family very much.

But it’s not what you’re thinking. I’m allergic to that stuff (I have very funny, inappropriate stories about that which you’ll have to ask me in person), plus I’m traveling with our 3 sons, so it’s all about bikes, canals, and food. I’m also allergic to art, so the museum scene is kind of irrelevant too. I know that’s blasphemous, but there you go, I just don’t get paintings.

So, about the food. I’ve got street food tastes, and much to the chagrin of my in-law family, I consider true Dutch delicacies to be the stuff you find in carts along the side of the main road between the train station and the place with all the pigeons. Mostly loompjes loompjas (skinny little spring rolls), ollieballen (donuts without holes, literally translated as “balls of oil”), and poffertjes (tiny pancakes). Mmmm… poffertjes.

Anyhoo, what I really wanted to discuss today is the sausage wall, which I dearly dearly love. It’s near the Central Train Station, and I never know exactly where it is but I always find it like a fucking homing device. I’m the pigeon, the sausage wall is my coop.

What is a sausage wall, you ask? It’s a tiny little hole in the wall fast-food restaurant where you put coins into slots, like a vending machine, and you get to open these tiny little doors, inside of which are these delicious sausage sandwiches and other strange things. So, weird little fried things, mostly in buns but sometimes not, of all descriptions, except you never know exactly what anything is made of.

Is it delicious, you may ask? Oh yes, it is. It is, for reals, but my guess is that the crucial ingredient that makes everything so good is that you have about 40,000 high people very nearby getting the munchies, and the result of this is unbelievable turnover.

I have never been to the sausage wall when there are fewer than 15 other people vying for the best sausage windows. On the supply side, there’s an army of Dutch people on the other side of the wall feverishly preparing fresh fried sausages (if that even makes sense). Thank God for those people, and who is the genius I can thank for coming up with this brilliant idea in the first place?

I can’t wait to get to Amsterdam, folks, the sausage wall is calling me and I can hear its cry.

Categories: Uncategorized

Quants for the rest of us

A few days ago I wrote about the idea of having a quant group working for consumers rather than against them. Today I wanted to spell out a few things I think that group could do.

The way I see it, there’s this whole revolution of data and technology and modeling going on right now, but only people with enough dough to pay for the quants are actually actively benefiting from the revolution. So people in finance, obviously, but also internet advertising companies.

The problem with this, besides the lopsidedness of it all, is that the actual models being used are for the most part predatory rather than helpful to the average person.

In other words, most models are answering question of the form:

  1. how can we get you to spend money you don’t necessarily want to spend? or
  2. how good a credit risk are you? or
  3. how likely are you to come back and spend more money? or
  4. how can we anticipate the market responding to end-of-month accounting shenanigans? I just threw this one in to give people a sense of how finance models work.

It all makes sense: these are businesses that are essentially bloodthirsty, making their money off your clicks and purchases. They are not going away.

Then there are some models that are already out there trying to make the user experience more enjoyable. They answer questions of the form:

  1. If you like that, what else may you like? (Pandora, Netflix, Google)
  2. If you bought that last week, maybe you’d like to buy it again this week?
  3. If you bought that, maybe you’d like to buy this now?

The problem, as you can see, is that these second, helpfulish kinds of things quickly devolve into the first, predatory kinds of things. In other words, you’re being bombarded by ads and suggestions for spending more money than you actually want to spend.

What about stuff that you actually don’t want to do but that is probably not directly profitable to anyone? I’d love to see technology used to tackle some of these chores:

  1. Figuring out what the best deal is on loans (credit card, student loans or mortgages), without becoming a lawyer. Here I’m not just saying they should all be clear about their terms- that’s a job for the CFPB. I mean there should be a website that asks me a few questions about what I need a loan for and points me to the best deal available.
  2. Finding the best price for something. I discussed this briefly here.
  3. Warnings about weird conditions and agreements, like mandatory arbitration.
  4. Help finding a good doctor or a good plumber etc.
  5. Help knowing when to go to the DMV or other public services building so the lines are bearable, or even better a way to do stuff on your computer or phone and avoid the trip altogether.
  6. Getting your kids’ medical records to their schools and camps safely and efficiently.
I’m sure I’ve missed a bunch of things (please tell me what I’ve missed!). In general, the consumer is constantly being asked to make decisions in situations where there’s unnecessary information asymmetry. A quant model for consumers should try to rebalance that asymmetry.

Some of these things already exist (like the doctor thing) but aren’t well-known or well-publicized, so wouldn’t it be cool if this quant group also provided an app that would aggregate all these services into one place?

And for the things like #4, where you want to be warned before you buy, it’s too much work to go back to a webpage to check everything before buying. Instead, the app could follow you around the web when you’re shopping and overlay a “bullshit warning” icon on the potential purchase in situations where people have complained about unreasonable terms and conditions.

Let’s use technology in our favor. Instead of the companies collecting information about us, let’s collect information about them. Come to think of it, the CFPB should start this quant group today.

Categories: data science

Versus what?

I’m going to specialize in short, curmudgeony blog posts this week.

Today’s topic: you always need something to compare a new thing with. It’s this versus what?

If it’s a model, compare it to noise. That is, go ahead and test a model by scrambling the “y”s and see how well your model predicts randomness. It’s a really good and inexpensive way of seeing whether your model is better than noise, so go ahead and do it. There’s even a name for this but I forget what it is (update from reader: permutation testing).

If it’s a plan for a system or the world, compare it to the status quo. I’m so sick of people discarding good plans because they’re not perfect. If they’re better than what’s currently going on, then let’s go with that. Which brings me to my last example.

If it’s someone’s proposal (person A), compare it to other proposals (person B). I don’t think it’s fair for people (person C) to nix an idea unless they come up with a better one. If person C is consistently doing that, it’s a good bet that they have something to protect in the status quo situation, which brings us to the previous example.

Categories: data science, rant

The Value Added Teacher Model Sucks

Today I want you to read this post (hat tip Jordan Ellenberg) written by Gary Rubinstein, which is the post I would have written if I’d had time and had known that they released the actual Value-added Model scores to the public in machine readable format here.

If you’re a total lazy-ass and can’t get yourself to click on that link, here’s a sound bite takeaway: a scatter plot of scores for the same teacher, in the same year, teaching the same subject to kids in different grades. So, for example, a teacher might teach math to 6th graders and to 7th graders and get two different scores; how different are those scores? Here’s how different:

Yeah, so basically random. In fact a correlation of 24%. This is an embarrassment, people, and we cannot let this be how we decide whether a teacher gets tenure or how shamed a person gets in a newspaper article.

Just imagine if you got publicly humiliated by a model with that kind of noise which was purportedly evaluating your work, which you had no view into and thus you couldn’t argue against.

I’d love to get a meeting with Bloomberg and show him this scatter plot. I might also ask him why, if his administration is indeed so excited about “transparency,” do they release the scores but not the model itself, and why they refuse to release police reports at all.

Why experts?

I recently read a Bloomberg article (via Naked Capitalism) about wine critics and how they have better powers of taste than we normal people do.

One of the things I love about this article is how it’s both completely dead obvious and at the same time totally outrageous when you think about it.

Obvious because when we see wine experts going on and on about what they can discern in their tiny sip, we know they either have magical powers or they’re lying, and since some of them can do this shit blindfolded we will assume they aren’t lying.

Outrageous because if you think about it, that means we follow the advice of people whose taste is provably different from ours. In other words, the word of experts is fundamentally irrelevant to us, and yet we care about it anyway.

My question is, why do we care?

Just to go over a couple of ground rules. First, yes, let’s assume that the wine experts really do have powers of discernment that are incredible and unusual, even though we have to trust an expert on that, which may seem contradictory. The truth is, this isn’t the first study that’s shown that, and I for one have hung out with these guys and they really do taste that minerally soil in the wine. I’m not even jealous.

Second, I’m not saying you care about wine experts’ opinions, but lets face it, lots of people do. And you could say it’s because of the performance that the experts give when they smell the wine and describe it, and I’ll agree that some of them can be poets, and that’s nice to see. But the truth is they also rate the wines with a number, and these numbers are printed in books, and lots of people carry these books around to wine shops and devote themselves to only buying wines with sufficiently high numbers, even though those people probably don’t themselves have the mouth smarts to tell the difference.

Now that we’ve framed the question, we can go ahead and make guesses as to why. Here are mine:

  1. People are hoping that they themselves are also supertasters. This kind of seems like the most obvious one, but I can’t help think that true supertasters would not need other people’s opinions at all.
  2. People think experts’ opinion of “good”, even if not completely the same as theirs, will be highly correlated, and so is better than nothing.
  3. People want to be seen drinking wine that supertasters would drink, as a sort of cachet thing.

I think #3 above is pretty much the definition of snob, and I think it exists but is not the major reason people do things (but I could be wrong). I’m guessing it’s more typically #2, but even so it doesn’t explain the really expensive high-end wine market’s huge appeal, unless there are way more supertasters than I thought out there.

I think this question, of why people listen to expert advice even when it’s mostly irrelevant, is an important one, because it happens so much in our culture, and clearly not just about wine.

I for one am attracted to the idea of going one step further and ignoring expert advice. I see a natural progression: first, people are ignorant, second, they learn what experts think, and third, they ignore experts and go with their gut.

But even sexier is the idea of never listening to experts at all: skip step two. Am I the only person who thinks that’s sexy? I mean, I guess it mostly means you’re wasting your time, but it also comes out in the end with less herd mentality.

I think this desire I have of skipping the expert advice is very tied into why I despise the echo chamber of the web and how we are profiled online and how our environment is constantly updated and tailored to our profile. It’s in some sense an expert opinion on what we’d like, given our behavior, and I hate the finiteness of that concept, possibly in part because I’ve designed models like that and I know how dumb they are.

Anyway, I’d love to hear your thoughts, and if I’ve missed any reasons why people like to hear partially relevant expert opinion.

Categories: data science

Do not track vs. don’t track

There’s been some buzz about a new “do not track” button that will be installed in coming versions of browsers like google chrome. The idea is to allow people their privacy online, if they want it.

The only problem is, it doesn’t give people privacy. It only blocks some cookies (called third-party cookies) but allows others to stick.

Don’t get me wrong- without third-party cookies, the job I do and every other data scientist working in the internet space will get harder. But please don’t think you’re not being tracked simply by clicking on that.

And as I understand it, it isn’t even clear that third-party cookies won’t be added: I think it’s just an honor system thing, so third-party cookie pasters will be politely asked not to add their cookies.

But don’t believe me, visualize your own cookies as you travel the web. The guy (Atul Varma) who wrote this also open-sourced the code, which is cool. See also the interesting conversation in comments on his blog Toolness.

Let me suggest another option, which we can call “don’t track”. It’s when nothing about what you do is saved. There’s a good explanation of it here, and I suggest you take a look if you aren’t an expert on tracking. They make a great argument for this: if you’re googling “Hepatitis C treatments” you probably don’t want that information saved, packaged, and sold to all of your future employers.

They also have a search engine called “DuckDuckGo” which seems to work well and doesn’t track at all, doesn’t send info to other people, and doesn’t save searches.

I’m glad to see pushback on these privacy issues. As of now we have countless data science teams working feverishly in small companies to act as predators against consumers, profiling them, forecasting them, and manipulating their behavior. I’m composing a post about what a data science team working for consumers would have on their priority list. Suggestions welcome.

Charity auctions and hate crimes

I read an absolutely incredible story last night on Bloomberg.

This Morgan Stanley executive William Jennings (co-head of North American fixed-income capital markets) is being charged with a hate crime. Let me piece it together a bit.

On December 22nd Jennings hosted a charity auction at Morgan Stanley until 6pm, then went to Ink48, a hotel in midtown on the west side. After partying on the rooftops for some time, and drinking, his car service didn’t show up fast enough for him so he hailed a cab to take him to Connecticut, where he lives with his wife and three kids in a $3.4 million house.

When he got to Connecticut, he got into a fight with the cab driver and ended up refusing to pay, stabbing the guy in his hand with a knife (which required 60 stitches) while using ethnic slurs. Then he went away to Florida for two weeks on the DL. My favorite line from the article:

Jennings fell asleep during the trip, the driver said. Once at the destination, though, Jennings said “he did not feel like paying” because he was already home.

Up for debate and the trial: did he really refuse to pay or was he just arguing his fare? Was it really 60 stitches or is that an exaggeration? Did he really use ethnic slurs? I’m throwing in these questions because I want to be correct and because the overall point of my post won’t depend on these details anyway.

Not up for debate: he stabbed the cabbie, it was definitely an argument over money, and he was worried enough to go to Florida for two weeks.

Okay, now that I’ve summed this up I’m gonna connect it to charity auctions. Yes I am.

I’ve been to charity auctions myself. I want to devote an entire post to describing what such an event consists of; for now take it from me that they are orgies of self-congratulatory arrogance. And ironically, they are not at all charitable in the sense of being generous and tolerant.

They are in fact celebrations of self-centeredness, displays of careless overabundance. Yes, I’ll pay $120k to go to Australia for a week to golf, and I’ll do it for the poor children, and by the way also because I can afford to throw away such money and especially by the way because everyone in this room now knows that.

So I think it’s extra deliciously ironic that this guy went from that atmosphere to arguing with an Egyptian cabbie over a $200 fare (or maybe $300, if we want to be generous to Jennings and believe his “extortionist cabbie” sob story).

But my point is that, although the cab ride was a different atmosphere from the charity auction, his was not a different attitude at all: both parts of his evening centered on assumptions of entitlement and selfishness and the idea that he is somehow outside the regular rules and cannot be held accountable like normal people. From the article:

He then went on vacation to Florida, police said. Jennings told officers he subsequently called his lawyer after a friend told him police were looking for a suspect in the stabbing incident, according to the report.

“Jennings said he didn’t know what to do — he just wanted the whole thing to go away,” Darien Police Detective Chester Perkowski said in a court document filed with the report.

The part about the car service not showing up is absolutely key: these guys use car services a lot, and when you do that, you get used to not paying for such trivial little things as rides, or for that matter food or drinks. All such things are handed to you for free when you are this important (read: rich). Paying, writing a check or what have you, is reserved for ostentatious displays of wealth. I know hedge fund guys that don’t even carry money in their wallet because they never use cash. Actually I don’t know them personally but I know that this is true because they brag about it in the elevators.

I’m not trying to generalize this story – most Morgan Stanley execs haven’t been charged with knifing down working class cabbies. But it’s impossible for me not to see the consistency in the two events.

Categories: finance, news, rant

This is water

I just started reading Infinite Jest and it’s blowing my mind.

I’m a nerd so I had never heard of David Foster Wallace before reading his book, but now I’ve officially joined his cult. If I’m too late to this party I will start my own, one-woman cult.

As far as I’m concerned he’s the Elliott Smith of literature.

If you haven’t already, please read this, Wallace’s 2005 commencement speech to Kenyon College. I read it at work yesterday and bawled into my keyboard for about 20 minutes.

Categories: rant

Open Models (part 2)

In my first post about open models, I argued that something needs to be done but I didn’t really say what.

This morning I want to outline how I see an open model platform working, although I won’t be able to resist  mentioning a few more reasons we urgently need this kind of thing to happen.

The idea is for the platform to have easy interfaces both for modelers and for users. I’ll tackle these one at a time.


Say I’m a modeler. I just wrote a paper on something that used a model, and I want to open source my model so that people can see how it works. I go to this open source platform and I click on “new model”. It asks for source code, as well as which version of which open source language (and exactly which packages) it’s written in. I feed it the code.

It then asks for the data and I either upload the data or I give it a url which tells the platform the location of the data. I also need to explain to the platform exactly how to transform the data, if at all, to prepare it for feeding to the model. This may require code as well.

Next, I specify the extent to which the data needs to stay anonymous (hopefully not at all, but sometimes in the case of medical data or something, I need to place security around the data). These anonymity limits will translate into the kinds of visualizations and results that can be requested by users but not the overall model’s aggregated results.

Finally, I specify which parameters in my model were obvious “choices” (like tuning parameters, or prior strengths, or thresholds I chose for cleaning data). This is helpful but not necessary, since other people will be able to come along later and add things. Specifically, they might try out new things like how many signals to use, which ones to use, and how to normalize various signals.

That’s it, I’m done, and just to be sure I “play” the model and make sure that the results jive with my published paper. There’s a suite of visualization tools and metrics of success built into the model platform for me to choose from which emphasize the good news for my model. I’ve created an instance of my model which is available for anyone to take a look at. This alone would be major progress, and the technology already exists for some languages.


Now say I’m a user. First of all, I want to be able to retrain the model and confirm the results, or see a record that this has already been done.

Next, I want to be able to see how the model predicts a given set of input data (that I supply). Specifically, if I’m a teacher and this is the open-sourced value added teacher model, I’d like to see how my score would have varied if I’d had 3 fewer students or they had had free school lunches or if I’d been teaching in a different district. If there were a bunch of different models, I could see what scores my data would have produced in different cities or different years in my city. This is a good start for a robustness test for such models.

If I’m also a modeler, I’d like to be able to play with the model itself. For example, I’d like to tweak the choices that have been made by the original modeler and retrain the model, seeing how different the results are. I’d like to be able to provide new data, or a new url for data, along with instructions for using the data, to see how this model would fare on new training data. Or I’d like to think of this new data as updating the model.

This way I get to confirm the results of the model, but also see how robust the model is under various conditions. If the overall result holds only when you exclude certain outliers and have a specific prior strength, that’s not good news.

I can also change the model more fundamentally. I can make a copy of the model, and add another predictor from the data or from new data, and retrain the model and see how this new model performs. I can change the way the data is normalized. I can visualize the results in an entirely different way. Or whatever.

Depending on the anonymity constraints of the original data, there are things I may not be able to ask as a user. However, most aggregated results should be allowed. Specifically, the final model with its coefficients.


As a user, when I play with a model, there is an anonymous record kept of what I’ve done, which I can choose to put my name on. On the one hand this is useful for users because if I’m a teacher, I can fiddle with my data and see how my score changes under various conditions, and if it changes radically, I have a way of referencing this when I write my op-ed in the New York Times. If I’m a scientist trying to make a specific point about some published result, there’s a way for me to reference my work.

On the other hand this is useful for the original modelers, because if someone comes along and improves my model, then I have a way of seeing how they did it. This is a way to crowdsource modeling.

Note that this is possible even if the data itself is anonymous, because everyone in sight could just be playing with the model itself and only have metadata information.

More on why we need this

First, I really think we need a better credit rating system, and so do some guys in Europe. From the New York Times article (emphasis mine):

Last November, the European Commission proposed laws to regulate the ratings agencies, outlining measures to increase transparency, to reduce the bloc’s dependence on ratings and to tackle conflicts of interest in the sector.

But it’s not just finance that needs this. The entirety of science publishing is in need of more transparent models. From the nature article’s abstract:

Scientific communication relies on evidence that cannot be entirely included in publications, but the rise of computational science has added a new layer of inaccessibility. Although it is now accepted that data should be made available on request, the current regulations regarding the availability of software are inconsistent. We argue that, with some exceptions, anything less than the release of source programs is intolerable for results that depend on computation. The vagaries of hardware, software and natural language will always ensure that exact reproducibility remains uncertain, but withholding code increases the chances that efforts to reproduce results will fail.

Finally, the field of education is going through a revolution, and it’s not all good. Teachers are being humiliated and shamed by weak models, which very few people actually understand. Here’s what the teacher’s union has just put out to prove this point:


Get every new post delivered to your Inbox.

Join 1,147 other followers