Archive
Can clouds think?
Sometimes I have trouble falling asleep. Especially if I’m riled up thinking about the newest stealth bank bailout, or wondering how to model rare, fat-tailed events, I’ll toss and turn, unable to get these problems out of my head.
Luckily I have a husband who is kind enough to tell me his stories at moments like these. I really appreciate his ability to draw out a story. He starts out slowly, and gets slower. He ends up at such a leisurely pace that I get completely distracted from my work-a-day concerns simply wondering what he’ll say next, when he’ll say it, or if he’s just fallen asleep.
It’s not just the slowness of the stories that do the trick, either, it’s also the content. He’s the master of the boring relaxing, abstract, science-fictiony story with exactly one idea. He’s seriously considering starting a blog for his stories which he’d call `Stories that put my wife to sleep’. I honestly think it would work great for lots of people- a public service, really, especially if he made very very very boring podcasts.
It’s efficient too, he’s mentioned to me that he’s told me the same story sometimes 5 or more times but I can never last through to the point of understanding the plot, and it always seems new. I never know what’s going to happen next, if anything.
My favorite story, which I have probably heard 17 times, is the story of whether clouds can think. It’s unresolved, the answer, but it’s wonderful to imagine, very slowly, the decisions a cloud could make, things like very slight changes in its luminosity or which winds to take rides on or how high to fly.
The Great Wealth Transfer, late 1900’s to early 2000’s (part 1)
When historians write about this era of U.S. history, how will it be described? I have a guess: “the Great Wealth Transfer” from the middle class to the wealthy. Let me explain why I say this.
There are lots of different parts to this story, but today I’ll concentrate on the housing wealth transfer.
We all know there was a housing bubble, that millions of people took out mortgages on dubious terms for houses that were already overpriced but that they were each counting on to go even higher. The way this was sold at the time, and even today is described, was as “home ownership for an expanded middle class.” But as Sue Waters from the Alt Banking group pointed out to me, these people didn’t get home ownership, they got debt ownership.
I know, it sounds a bit strange, but that’s just it, the language is important.
When people say they own their home, do they mean they don’t have a mortgage? Probably not. They probably mean they’re in the process of paying a mortgage, but they conflate the two concepts because they assume they will pay off the mortgage eventually. But in the meantime they don’t actually own their home, the bank does. The extent to which this is an important distinction is the extent to which it is likely that they will be able to pay off that mortgage some day in the future.
When you’ve stopped conflating home ownership with debt ownership, and you look back at the housing bubble, it’s a different picture. How many new home owners were there really? It’s not an easy question to answer, but it’s clear that there were way fewer than we thought- many of the mortgages had terms that were clearly very optimistically written. Nobody really thought it would work out well, but the system just kept growing and the optimism kept getting less reasonable. Meanwhile, bankers got extremely rich.
How did this all happen?
This is answered by asking an even larger question: how does the financial system make money? I claim a large part of it is by finding a group of people that are relatively naive and pushing risk to them. For example, the dot com bubble was created by getting normal people to invest in dumb new-fangled things – they were the pawns.
For the housing bubble, it was a bit more devious. One one end, systemic risk was pushed (into the future) to the taxpayers themselves through bailouts of the banks, AIG, Fannie, and Freddie. In other words, taxpayers didn’t know it at the time but they were getting more and more on the hook for losses as the banks and financial system took larger and larger bets on the direction of the housing market.
At the same time and at the other end of the mortgage contracts, the so-called “homeowners” who took on mortgages were the fall guys. As a whole, they signed up for debt (and the right to claim themselves as homeowners) and in return are now hopelessly underwater. The Obama administration, just like the Bush administration before it, is urging these people to do what they are morally compelled to do, namely pay off their unmanageable debts, while changing the laws for the big banks so they can get away with whatever they need to in order to ignore their outrageous undercapitalization.
To sum it up: we found a very large pool of people too naive to understand the risk they were taking on, we signed them up for that risk while painting them a beautiful picture of the American dream, and now we get to accuse them of being immoral for not being able to hold up their end of the contract. It was an amazing swindle.
To be continued in part 2, in which many of the same players who brokered the mortgages to the “new homeowners” are now buying up their foreclosed homes and renting them back.
Get overpaid so people will listen to you
Have you read the recent article in the New York Times about how lower-status monkeys are less healthy and more stressed out than higher-status monkeys? Their gene expression actually responds to changes in social status. Does this resonate with your experiences with humans?
It does with me, and for us people I’d rephrase it this way: your concerns and ideas are given attention in direct relation to your status. Your stress levels rise as you realize your status is lower and your risks have grown.
Here are some examples from work. I’ve been disappointed to notice, time after time, that my ideas are considered important and innovative in direct proportion to how much they are paying me to have them. If I’m underpaid then nobody thinks I am all that smart; nevermind being a friggin’ volunteer (with some exceptions, but don’t stop me, I’m on a roll). This perversely makes me want to get overpaid just so I’ll get listened to.
Cuz why? Didn’t you ever notice that overpaid people’s ideas are about as good as anyone else’s but they are framed as pure brilliance? I have. It even works head-to-head: two people of different status come up with the same exact idea but the one who is more important was listened to and their idea championed. Oh yeah, I’ve seen it, and so have you (example: when I was at D.E. Shaw, we rated other people’s ideas with a “probability of success” in an effort to estimate their expected payouts; someone once showed me their idea, which was identical to one of Larry Summer’s ideas, but had come 2 years before and had scored about half as well. But my fried wasn’t an MD making $5 million per year so clearly his ideas weren’t as good!).
A similar thing happens with problems rather than ideas in a workplace. The worst examples of over-worked and under-appreciated situations clearly don’t happen at the top. For example, when I worked at MSCI, it seemed like the sales guys, who defined the top there, spent more time strutting around making sure each and every one of their efforts went appreciated than doing the actual efforts, whereas the lowly dev-ops guys, and the guys setting up the initial portfolios for the new clients, were treated as an afterthought, only noticed if something went wrong. They’d stay up all night fixing something, probably someone else’s mistake, and nobody would even thank them.
[It still seems so ironic that the most technical people there are also the least appreciated, since the product is essentially technical expertise. Or is it? Maybe I’ve got it wrong, and it’s really about selling technical expertise in a package that makes people feel safe and pious. Maybe the black box we’re selling doesn’t even have to work.]
If you are thinking that everyone at MSCI is in finance and is thus overpaid and pampered, then you’ve got it wrong, it’s a brutal atmosphere, like much of finance. If you don’t believe me, read my friend Katya’s blog, Left with Balls, where she talks about the spell of Wall Street.
Taking one step back, this kind of thing strikes me as unfair and frustrating. The idea that the lowest-ranked also has to deal with ridiculous stress and chronic health problems does not jive with my inherent concept of justice. Although it does seem like a natural response to a system that’s already been created (as in, as a consequence of being frustrated because my ideas are ignored, I want to get overpaid to get listened to, so I’m joining in on the perverted game and furthering the system), it doesn’t seem like we’ve done a particularly good job setting up these systems.
For a country that putatively considers itself a democracy, we seem to have a tremendous amount of respect for a rat-race corporate hierarchy. Is that a contradiction? Or is the American dream actually to start a hierarchy and to sit at the top? Do other people identify with the guys on the bottom or the guy at the top? Or the guys in the middle clawing upwards?
Question: is it really impossible to listen to and evaluate ideas based on their merit? How about anonymous polling of problems? It’s certainly technologically feasible, but we don’t do it.
Question: Is it really impossible to appreciate people who make things work behind the scenes? How about we ask people to sit with other people in entirely different departments in a rotation to witness what other people actually do? I really think that would help with the appreciation problem (but not if the technical people in your company are in India and the salesguys are in New York).
For the record, when I start a company I’ll do these things. Of course, I’ll be sitting up there at the top thinking what a great idea I had to do them.
#OWS update: looking for UX help
I’ve got three updates on Occupy, besides reminding people that the Alternative Banking Working Group meets every Sunday from 3-5pm at Columbia (room 1401 in the International Affairs Building at 118th and Amsterdam).
- Occupy.com has launched! This is a website set up by my friend David Sauvage, and it’s looking awesome and informative.
- The “find a Credit Union webapp” is looking for UX Designer help. I’ve written about this project before, but in a word we’re helping people figure out which credit unions they are eligible for, if any; the rules can get kind of tricky. We’ve got the basic ideas down but we’d love a thoughtful designer to come in, improve the user experience, and help create a appropriate Occupy look which also doesn’t scare away non-Occupy people. We also have a development team from ThoughtWorks helping us out, but it would be very helpful to have a New York- based developer to maintain the knowledge. The eligibility rules based on address (but not necessarily on zipcode or borough) are particularly hard (or interesting, depending on how nerdy you are).
- Not a strictly Occupy issue but did you hear the Vermont Senate has voted to end Corporate Personhood (hat tip reader G. Jones)? Move to Amend has spearheaded this effort. I love their motto: End Corporate Rule, Legalize Democracy. Read about the Vermont vote here.
Should we have a ratings agency for scientific theories?
Recently in my friend Peter Woit’s blog, he discussed the idea of establishing a ratings agency for physics. From his blog:
In this week’s Nature, Abraham Loeb, the chair of the Harvard astronomy department, has a column proposing the creation of a web-site that would act as a sort of “ratings agency”, implementing some mathematical model that would measure the health of various subfields of physics. This would provide young scientists with more objective information about what subfields are doing well and worth getting involved with, as opposed to those which are lingering on despite a lack of progress.
Abraham Loeb was proposing to describe the field of String Theory as a perfect example of a bubble. And it’s absolutely true that String Theory has provided finance with tons of brilliant young orphans who either got disillusioned with the field or simply couldn’t get a job after writing a Ph.D. or after a post-doc. It provides an extreme example of a mismatch between supply and demand.
Would a ratings agency for scientific theories help? I don’t think so.
The very basic reason, as Peter points out, is that it’s hard to evaluate scientific theories while they are unfolding. There are two underlying causes: first, people in a field are too invested to admit things aren’t working out, and second, by the nature of scientific research, things could not work out for some time but then eventually still work out. It’s not clear when to give up on a theory!
Ignoring those problems, imagine a “mathematical model” which tries to gauge the success of a field. What would the elemental quantities be that would signify success? Would it count the number of proven theorems? Crappy theorems are easy to prove. Would it count the the number of successful experiments? We could always take a successful experiment and change it ever so slightly to get another success. I can’t think of a quantitative way to measure a field that isn’t open for enormous manipulation (which would only happen if people actually cared about the ratings agency’s rating).
Of course the same might be said about financial ratings. It begs the question, why are ratings agencies useful at all?
In finance we have lots of people buying very similar products with very similar contracts. Sometimes these are even sold on exchanges and carry with them the exact same risk profiles. In such a situation it makes sense to assign someone to look into the underlying risks and report back to the community on how risky a product is.
I would claim that the situation is very different in science or math. People enter a field for all sorts of reasons, with all sorts of goals and situations. String Theory is an extreme case where it could be argued that it got such spin that a whole generation of physics students got sucked into the field by sheer momentum. Perhaps it would have been nice to have a trusted institution whose job it was to calm people down and point out the reality, but I’m not sure it would have helped that much with all the excitement, especially if there had been a model which counted theorems and such. People would just have said the model had never seen something this exciting.
Then there’s the issue of trusting the modeler. Right now ratings agencies have a terrible reputation because they are paid by the people they rate products for, and have been known to sell good ratings. I’m hoping we can do better in the future, but it’s hard (but not impossible!) to imagine gathering enough experts in finance to do it well and to have the product be trusted by the community.
What is the analogy for scientific theories? The problem with rating science is that, because of the depth of most fields, only the experts in the field themselves understand it well enough to even talk about it. So that problem of getting an informed and impartial view on the worthiness of a theory is super super hard, assuming it’s possible.
Finally, I’m not sure what the ratings agency would be in charge of warning people about. Even the financial ratings agencies don’t agree- some of them measure default risk and others measure expected loss through default, which can be two really different things (for example if you think the U.S. will technically default but will end up paying their debts).
In science, I guess you could try to measure the risk that “the theory won’t end up being useful” but it’s not even clear how you’d decide that even after the fact. Maybe you could forecast the number of jobs in the field for graduating Ph.D. students, and that would be helpful to grad students but would also not be the best metric of success for the field.
I’m not saying we shouldn’t have people talk about fields and whether fields are failing, because that’s hugely important. But I don’t think there’s a quantitative model there to be created that would help the conversation. Let’s start an open forum, or a wiki, with the goal of talking about the health of various fields of scientific endeavor and have a bunch of good questions about the field and people can each add their two cents.
How to teach someone how to prove something
In a couple of my posts (most recently here), I’ve talked about the need for a course early on in undergraduate math classes on proof techniques.
The goals of the class are two-fold: first, teach the students basic skills, and second demystify the concept of proof. The students should come away from the class thinking, no it’s not magic, and I’ve learned how to do this stuff, and there are a few basic techniques which seem to come in handy.
Today I want to go further into what a curriculum for such a course might look like.
And I will, in a moment, but first I want to explain something. It’s actually a really important and dangerous question, how to teach such a course, because it could go wildly wrong, and sometimes does. From my commenter Jordan:
… “Numbers, Equations, and Proofs,” which I started at Princeton in 2002 and which is still going as well. Though here’s an interview with a dude who was an ace math competition dude and found the course so hard as to drive him out of the math major! So maybe it’s no longer as “for everyone” as I designed it to be….
This struck me, how perverted Jordan’s class became. For that matter, Math 55 at Harvard could have started out as a good idea as well, but by the time I got to Harvard as a grad student it was the reason so few math majors ever stuck at Harvard and why there were especially few women.
I remember Noam Elkies taught it while I was there and was famous for asking questions in class and getting students to compete to answer them quickly. It makes sense that he’d run a class like this, because he’s so fast and clever, and he’s naturally wondering, am I the fastest and clevererest of them all? But rather than a place where proof is demystified and people feel safe asking dumb questions, he’d created the polar opposite, a live quiz show of clever competition. Ew!
In order to combat this downfall and decay, I think the class needs to have a clearly stated mission as well as built-in curriculum requirements that works against ostentatious displays of cleverness, which indeed only serve to further the “I got it but you don’t” stereotype of math skills (but which mathematicians themselves are incentivized to further since that magical aura comes in handy).
For example, when I taught it, I let the students hand in homework again and again until they got a score they liked. Of course, this depending on me having an awesome grader (and a relatively small class), which luckily I had.
Also, I asked each student to give a presentation to the class on some proof they particularly enjoyed, and I sat through a preview of their presentation and gave them extensive advice on board work and eye contact, which took a lot of work but really helped them prepare and also boosted their egos while at the same time increased their sympathy with each other and with me.
But of course the most important thing was that I clearly stated at the beginning of each class in the first two weeks that proving things in math was a skill like any other that you get good at through practice. And when I left Barnard Dusa McDuff took over the class and still teaches it, so I know it’s in good hands.
If I hadn’t had Dusa, I’d probably have written a manifesto to be given to each person who would teach the class after me. Of course anyone could have just thrown that away but it’s an idea.
As for content, I taught them really basic proof techniques, so induction, proof by contradiction, the pigeon-hole principle, and some epsilon-delta practice. We covered some basic logic, graph theory, group theory, ordinals, and basic analysis. We constructed the reals two ways and the complex numbers once and talked for a long time about whether “i” is real and what that even means. We used A Transition to Higher Mathematics, which I recommend with a few reservations (please tell me if you’ve found a better text for something like this!).
Everything was done super explicitly and carefully, no rushing. I said things three times in three different ways. I wasn’t expecting people to be fast or clever, because I know intelligence works in different ways and that this stuff was completely new to most of the students. And at least one student in the class, who had been an artist, is now a grad student in math at Berkeley.
Looking over my post I realize I spent way more time talking about the tone of the class than the content, but that’s totally appropriate, since I think of this class as an introduction to the culture of mathematics (or rather the culture I wish we had) just as much as mathematics itself.
After all, there really is no time limit on good ideas, and you do get to do it over if you make a mistake, and going over things slowly gives you more time to ask good questions and find mistakes.
Calling all data scientists! The first ever global data science hackathon
Hey I’m helping organize a NYC data hackathon at Bloomberg Ventures to take place April 28th – 29th, from 8am Saturday to 8am Sunday. I’m looking for outrageously nerdy people to come help. There will be some prizes.
Read the official blurb below carefully and if you’re in, sign up for the event by registering here.
Update: they’ve decided on prizes.
See you there!
——————————————————————————————————————————————–
Are you a smart data scientist? Participate in this hackful event. 24 hours of non-stop, fun data science competition. The first ever global, simultaneous data science hackathon!
In connection with Big Data Week, we’re helping organize a global data science hackathon that will simultaneously take place in various locations around the world (including London, Sydney, and San Francisco). We will host the NYC event at the Bloomberg Ventures office in the West Village.
The aim of the hackathon is to promote data science and show the world what is possible today combining data science with open source, Hadoop, machine learning, and data mining tools.
Data scientists, data geeks, and hackers will self organize around teams of 3-5 members. Contestants will be presented with a ‘big data’ set (hosted on the Kaggle platform). In order to win prizes, the teams will have to use data science tools and develop an analytical model that will solve a specific data science problem specified by the judging tech panel. The contestants will have to report their achievements at specific milestones, and a leader board will be published online at each milestone.
The contestants will spend 24 hours in Bloomberg Ventures’ office space where food, drinks, workspaces, and resting areas will be provided. Teams will compete for both local and global titles and prizes.
The Hackathon runs for 24 hours starting on April 28th at 8am (early start to allow for the event to happen simultaneously in multiple time zones around the world).
If you have questions, please email shivon.zilis@bloombergventures.com
PLEASE READ CAREFULLY:
1. This is a technical competition, not a networking event or an opportunity to learn more about big data techniques and technologies. We have limited space, so we unfortunately need to be strict about who gets to compete. If you’re an entrepreneur looking to recruit, we’re excited to have you as a member of this community, but this specific event is not the right venue, please come our regular Data Business meetups instead! 🙂
2. You should have Mad Skillz at at least one of the following:
- Data grappling and/or cleaning,
- Data modeling and forecasting,
- Data visualization,
- Spontaneous micro- and macro-economic theory creation
3. You should know one or more of the following languages:
- R
- python
- Matlab
- Some statistical package like SPSS or SAS
4. You should bring your hardcore laptop to the event, since we will have on the order of 10 gigs of data to play with.
In which mathbabe becomes insurance claims adjuster
Who knows what I’m talking about with this story.
My husband dislocated his finger sledding with my son last January, so more than a year ago, and the hospital kept sending us bills for the event.
But here’s the thing, we were covered under my medical insurance, which had perhaps recently changed policy numbers when MSCI took over Riskmetrics. So probably what had happened was my husband had given them the old insurance card, but in any case, in the the end I knew I wouldn’t have to pay since we’d definitely been covered.
The hospital called once a month or so, and every time they got hold of me I argued with them and told them to check their records. They kept telling me that the insurance company was refusing payment under any of the policy numbers I gave them.
In the end, last month, I called up the insurance company myself and got them to admit payment, which wasn’t hard since they said they’d already paid for the X-ray from the dislocation on that date. I called up the hospital and straightened it out.
So yeah, I ended up doing their job for them, and that’s both annoying and exciting because now nobody thinks I owe them $2400. In fact I did a victory dance (at work, because you always have to do this during work hours for people to answer the phone).
But why I’m writing about it today is that it’s actually really infuriating how often something like this happens, and I can’t help noticing that I always get out of it but many people wouldn’t. I’m at a huge advantage in this common situation because:
- I worked as a customer service person so I know how to talk to customer service people. Turns out you should always be polite, but never hang up the phone until your problem is solved. Just keep asking, extremely nicely, things like, “Hmmm… that’s confusing, what do you think could have gone wrong?” or “What would you do if you were me?” or if those don’t work, “Do you think you could tell me who to talk to sort this out? I’d really appreciate it.”
- I am always covered by insurance, so I never worry that they are right. This is an enormous advantage over people who sometimes lose coverage between jobs or something.
- I keep all my old paperwork. Impossible for people who don’t have an incredibly
boringstable lifestyle like mine. - I have a job that allows me to make calls like this during work hours. Obviously huge.
- I am completely unafraid of forms and red tape. This comes from experience, but I know most people are afraid of such stuff, and that alone would probably keep most people from arguing.
I really do feel like I am relying on my professional skills in order to get my insurance to pay for setting my husband’s dislocated finger, when that should be a no-brainer. If you are inexperienced and poor, you’d probably be completely at a loss for how to deal with this situation.
I wonder how many people have their credit scores lowered by medical claims which should have been paid but weren’t due to crap like this. It’s a broken system, but it only leaks on the most vulnerable people, and I hate that.
Continuously forecasting
So I’ve been thinking about how to forecast a continuous event. In other words, I don’t want to forecast a “yes” or a “no”, which is something you might do in face recognition using logistic regression, and I don’t want to split the samples into multiple but finite bins, which is something you may want to do in handwriting recognition using neural networks or decision trees or recommender systems.
I want essentially a score, in fact an expected value of something, where the answers could range over the real numbers (but will probably just range over a pretty small subset but I don’t know exactly what smallish subset).
What happens when you look around is that you realize machine learning algorithms pretty much all do the former, except for various types of regression (adding weights, adding prior, nonlinear terms), which I already know about from my finance quant days. So I’m using various types of regression, but it would be fun to also use a new kind of machine learning algorithm to compare the two. But it looks like there’s nothing out there to compare with.
It’s something I hadn’t noticed til now, and I’d love to be wrong about it, so tell me if I’m wrong.
On the making of a girl nerd
Today I want to discuss the process by which girls become math and cs nerds.
I could be tempted to talk primarily about my own story, since I’m a huge nerd. And I will talk about my story, but my focus is going to be on the girls of my generation who could have become nerds but didn’t. I’m hoping we can learn some lessons so that future generations will have more nerd girls.
Both my parents are nerds. My mother has a Ph.D. in applied math and my father has a Ph.D. in pure math. Moreover, I was on the math team in high school, found out about a math camp, and went to it for two summers, with the full support of my family.
I want to go over these details again, because I want to point out that they gave me an enormous advantage to becoming a successful nerd.
First, my parents being nerds: I have found an amazing correlation between women with math Ph.D.’s and women whose fathers are mathematicians. I don’t think this is random- indeed I think it means two things. First, that girls with mathematician dads have an easy time imagining themselves as mathematicians (and an even easier time if their mom is too). Second, that girls without mathematician dads don’t. Otherwise you wouldn’t be able to explain the statistics I have.
Second, the math camp experience. I went to math camp in spite of it being an extremely uncool summer endeavor, according to my classmates at school. Yet I didn’t care, and went anyway, mostly because I was already a complete outsider, a fat girl on the math team (but a mathbabe when I got there!).
Two things about this. First, most smart girls around me in Lexington High School, and there were a lot of them, would not have been willing to go to math camp and ruin their reputations. Most of them were relatively popular, and wanted to keep it that way. I had nothing to lose in that aspect and knew it. This kind of thinking may seem silly to us as grownups but seemed like life or death choices then.
Second, the advantage having been to math camp gave me when I got to college was phenomenal. I knew how to prove things by induction, by contradiction, and using the pigeon-hole principle. I knew basic group theory, graph theory, and real analysis. This gave me a jump-start in all of my undergrad math major classes. I was an elite, and what I could do seemed like magic to the kids who were math majors who didn’t know that stuff.
The thing about math is that people get into this mindset about being good at it: they think that you either have it or you don’t (see this post for more on the mindset). So the experience for the other kids, boys and girls, going to an algebra class and sitting next to me and a few other kids from math camp backgrounds was understandably intimidating and made them think they couldn’t compete. But I believe that, considering the social constructs and the kind of confidence girls and boys are trained to have (or not have), it was particularly daunting for other girls to see their competition in a small group of elite nerds who already knew all the answers.
I’m not advocating closing math camps. In fact, I am going back to teach at my high school math camp in July for three weeks (woohoo!). What I am advocating is thinking seriously about the selection process for young nerds and how much it weeds out girls. We can do better.
For example, Harvey Mudd is doing better by careful thought and attention to the issue. Namely, they are changing the introduction to programming class to be more appealing for non-math-or-cs-camp nerds. From the New York Times article:
Known as CS 5, the course focused on hard-core programming, appealing to a particular kind of student — young men, already seasoned programmers, who dominated the class. This only reinforced the women’s sense that computer science was for geeky know-it-alls.
“Most of the female students were unwilling to go on in computer science because of the stereotypes they had grown up with,” said Zachary Dodds, a computer scientist at Mudd. “We realized we were helping perpetuate that by teaching such a standard course.”
To reduce the intimidation factor, the course was divided into two sections — “gold,” for those with no prior experience, and “black” for everyone else. Java, a notoriously opaque programming language, was replaced by a more accessible language called Python. And the focus of the course changed to computational approaches to solving problems across science.
This sounds like a brilliant idea, and one that we should all consider (and python rocks!). It is reminiscent of the “Introduction to Proofs” class which I started with Karen Edwards and Sara Robinson in 1993 at UC Berkeley as an undergrad and which is still going, as well as the class I started at in 2006 at Barnard College, which is also still going. The dual goals of such a class are to teach basic proof techniques to people interested in the major (who probably didn’t go to math camp) and to show people that being able to prove things isn’t magic, it just takes practice and knowing techniques.
Let’s get more campuses across the country to think about all the math and cs nerds they are missing out on by teaching the same old math (or cs) major classes every year. This is a curriculum change that is easy, fun to teach, and completely worthwhile.
Who is the market?
Oftentimes you’ll read an article in the middle of a market day about how “the market is responding” to the jobs report, or the manufacturing index, or sentiment reports. That kind of makes sense – it is shorthand for the fact that the people betting on the market are, as a group, reacting and changing their bets based on new news. If the expectation was for 200,000 jobs to be added but only 120,000 jobs were added, you’d expect disappointment and a drop in the S&P index.
Even so, this language is pretty confusing, since it’s certainly not true that everyone who invests in the market is doing this – most people with money in the market don’t do anything at all on a given day. Okay then, let’s interpret it as meaning something kind of reasonable like, “of those people who respond to this news by changing their bets, a majority of them are betting in one way which is moving the market.”
It still may not be true, since people who are seriously involved with the market typically don’t have the same expectations as what the official expectation report says – that report may have contained no surprising news at all, but one hedge fund liquidating their portfolio may be dominating the market. So even if there is a reasonable interpretation, the chances are it’s vapid.
Other times you’ll read an article, probably put out by Bloomberg, about how the market is “recalibrating,” or “taking stock” after a rise. This is where I get confused. It’s like I’m expected to imagine a huge man, hunched over thinking about what to do next.
But what does that really mean? As far as I can tell, nothing at all. There’s no man, there are no little men behind the wall representing this man, and everyone betting on the market is just doing their thing. It’s maybe just a way of writing a story because the journalist was told to write a story and the market wasn’t doing anything.
But lately I’m wondering if there’s something more to it. Why are journalists covering the market allowed, day after day, to write vapid articles about the market? What is it about using language like this that makes us comforted?
My guess is that people want there to be such a man, and moreover want him to be understandable and reasonable.
It’s primarily a question of control – control over our lives, as if we can say, as long as we kind of get his (the market’s) sentiments, we can avoid catastrophic risks. Like in those human nature tests where 85% of people consider themselves better than average drivers, we feel that we understand the market and so we’re covered and safe. Even when there’s plenty of evidence that we don’t actually understand the risks, we continue the market myth out of this need to feel in control.
I also think there’s another, secondary effect of this personification. Namely, we feel like the system is massive and powerful and there’s nothing we can do to affect it. It makes us passive.
My friend Hannah, who’s an anthropologist and whom I met through Occupy, likes to say to people, “that good idea you’ve had that someone should do? It’s your idea, and you should do it! There’s no Occupy elf that will go do it for you just because it’s a good idea.” I love that sentiment, and the idea of Occupy elves (why aren’t there Occupy elves?).
It makes me realize how much we expect other people to do stuff just because it’s a good idea, when in fact from experience we should have learned by now that the stuff that gets done by other people is usually because it’s a good idea for them. Stuff that’s a good idea for us, or for everyone, we should consider our personal responsibility. The market is certainly not looking out for us.
More creepy models
I’ve been having fun collecting creepy data-driven models for your enjoyment. My first installment was here, with additional models added by my dear commenters. I’ve got three doozies to add today.
- Girls Around Me. This is a way to find out if you know any girls in your immediate vicinity, which is perfect for my stalker friends, using Foursquare data. My favorite part is that the title of this article about it actually uses the word “creepy”.
- Zestcash is a cash lending, payday-like service that data mines their customers, with a stated APR of up to 350%. On of my favorite misleading quotes in this article about the model: “Better accuracy should translate into lower interest rates for consumers.” Ummmm… yeah for some of them. And I guess the idea goes, those other losers deserve what they get because they’re already poor?
- The creepiest of all by far (because it is so painfully scalable and I could imagine it being everywhere in 2 years) is this one which proposes to embody the “best practices” of medicine into a data science model. Look, we desperately need a good model in health and medicine to do some of the leg-work for us, namely come up with a list of reasonable treatments that your doctor can take a look at and discuss with you. But we definitely don’t need a model which comes up with that list and then decides for you and your doctor which one you should undergo. Decisions like that, which often come down to things like how we care about quality of life, cannot and should not be turned into an algorithm.
By the way, just to balance the creepy models a bit, I also wanted to mention a few cool ideas:
- What about having a Reckoner General, like a surgeon general? That person could answer basic questions to explain how models are being used and to head off creepy models. Proposed by my pal Jordan Ellenberg.
- What about having an F.D.A.-like regulator for financial products? They would be in charge of testing the social utility of a proposed instrument class before it went to market. Can we do the same for data-driven models? Can the regulator be kick-ass and reasonably funded?
- What about having a creep model auditing board that brings together a bunch of nerds from technology and ethics and looks through the new models and formally reprimands creepiness, using the power of social pressure to fend it off? They could publicize a list of creeps who made these models to call people out and shame them. That really doesn’t happen enough, it’s like the modelers are invisible.
- How about a law that, if you add a cookie to your digital signature that says, “don’t track me for reals”, then if you find someone tracking you, as in saving and selling your information, you can sue for $100K in damages and win?
It sucks to be rich
I often find myself uttering the phrase, “you don’t want to be really rich, because it sucks to be rich.” For whatever reason I’m always asked to explain that opinion. I’ll do so here so I can just reference this blog post from now on instead of having to repeat myself.
Just to be clear, it also sucks to be poor. I’m not saying it doesn’t because it really, obviously does. My experience going to Ghana and making friends with dancers who later injured their backs has shown me that, especially when there are unmet medical needs, being poor absolutely bites.
But I would (and will) argue it also sucks to be rich, in a more psychological, and less sympathetic (as in, people don’t have sympathy for you) kind of way.
This recent article from the New York Times, about a reported who lived like a billionaire for a day, is worth a read and is what spurred me to write this post. My favorite line:
“Somebody’s got to live this life,” he says, gesturing to the pristine view from his penthouse villa. “God decided it should be me.”
Not that this line supports my arguments, but it’s just awesomely grandiose and despicable.
Anyhoo, back to why it actually sucks. I am using evidence I gleaned from working at D.E. Shaw with quite a few rich people (as in never have to work in their lives and can take yearly ski vacations in the Alps or wherever) and a few insanely rich people (way more). So it’s a relatively small sample size, but even so it’s not empty.
The main reason I think it sucks, is that human nature has us worrying about stuff no matter what. And rich people don’t have normal things to worry about, so they make up really weird shit to worry about. That’s kind of the whole argument but I’ll give a bunch of examples.
The primary reason it sucks to be rich is that, counter-intuitively, rich people constantly worry about money. If you drew a graph of “have money” versus “worry about money” it would be a “U” shaped graph. I feel very lucky to be in the sweet spot where I make enough money not to worry about paying my bills or being on medical insurance but I don’t make so much money that I have to start worrying about it.
What do I mean? I mean:
- Rich people worry about whether they’ve invested their money correctly (not a concern for me). This sounds like a joke but believe me, they talk about it for many many hours, probably more time than they spend with their kids.
- They worry about whether the charities they give money to are really producing stuff, because the scale of their donations is so large (again, not a concern for me, if I give money it’s to Fair Foods and I know exactly where it goes, usually to paying for insurance for the trucks).
- They scheme and plan how to affect politics and politicians with their money. Maybe not so much sympathy for this.
- They worry about whether their kids will turn into good-for-nothing leeches and so come up with weird estate planning contracts with lawyers to keep money away from their kids, which in turn screws up their kids and their relationship with their kids. This stuff is for real and can get insanely nasty, see this article if you don’t believe me.
Who needs all that? I’m much happier having kids where I’ll say, when they are ready to go to college, hey here’s how much we’ve been able to save, here are your college choices, the rest you’ll have to pay for yourself so choose wisely.
In other words, it’s good to have nice and reasonable worries.
Besides money, what do rich people worry about? The answer is: absolutely everything, and nothing, at the same time.
My favorite two examples come from stories about David Shaw himself, who is massively rich. I didn’t actually meet the people involved, so these are myths I heard working there, but they are really good myths and have the ring of so-absurd-nobody-could-make-this-up.
First example: David hires a Ph.D. in English literature (he has a thing for “geniuses”, even in the mail room) to test mattresses for him. So that person’s job is to sleep on 15 different mattresses, for 8 nights each, and draw up a report to tell him the pros and cons of each mattress. This is to avoid him having an uncomfy night’s sleep. That’s what the risk was that we were avoiding with that.
Second example: David wants to be sure his trip to California goes smoothly, so he hires a Ph.D. in Something to take the exact same trip – same car service to the NY airport, same flight (same seat on plane!), same car service upon arrival, same hotel, exactly a week before his trip (due to understood seasonality issues of air travel) – to make sure there are no snags, and to draw up the report that presumable explains how much leg room there was in his plane.
You could say that he’s just a weirdo, but here’s where I’d disagree. Before making $2.5 billion, he was just a computer science nerd at Columbia. Sure, he was intense and probably competitive, but he had normal worries and isn’t famous for being a total jerk. For that matter he’s still not famous for being a total jerk, but he’s clearly got not enough to worry about.
In other words, I’m convinced that if I had that much money, I’d be doing stuff like that too, and so would you. The existence of asstons of money around you makes you weird and entitled. Add to that that everyone around you is either your servant or someone who assumes you are living a perfect happy life, and you become increasingly isolated and misunderstood on top of it, which leads to more weirdness.
Yuck! I’d rather be saving up for a family trip to somewhere nice, and in the meantime having stay-cations where the biggest expense is a Brazilian barbeque restaurant in Queens.
Thought experiment: witness protection program
I’m often accused by my family members of having no imagination.
I really don’t think that’s fair, I think it’s more that they have outrageous amounts of imagination, and I’m normal. My husband will say something at the dinner table along the lines of, “hey I was thinking about what it would be like to live on a planet that’s attached to another planet by a weird system of ropes,” and without skipping a beat one of my sons will start asking questions: “Are they going straight up in the air or slanted? Can you climb up to the other planet? How far away is it?”. Pretty soon they are, and I’m not kidding, arguing about how thick the ropes are and the question of traveling between the planets and the different civilizations that would evolve on these two conjoined spheres. I’m an observer.
When one of them says something directly to me along the lines of, “mom, what if we lived on a spaceship going to Mars and we could only eat a liquid diet and there were no books, only videogames?”, I am usually pretty stumped (fake example, I just made it up, but you get what I mean). It just doesn’t make sense to me, and I’m constantly going back to one of the assumptions and asking why – why no books? Can’t we get books if we have so much technology? This is when my kids roll their eyes and walk back to their room.
I can’t help it, I’m just a practical-minded person. I want to solve problems but I want those problems to make sense.
In pure defense of my own ability to imagine, I came up with the following thought experiment. For some reason, which doesn’t matter (although it can be fun to come up with ridiculous reasons), we are all put into the witness protection program, and have to move to a tiny little town in the south or the midwest and we have to blend in with the townspeople, and figure out how to make a living and how to make a home. Or at the very least, if we don’t blend in exactly, we have to come up with a good story to explain our eccentricities, and it can’t be, “we’re in the witness protection program!”.
The first question is how we can make a living. I usually imagine waitressing at first, then learning how to be a car mechanic. I think of being a car mechanic as the coolest, nerdiest thing you can count on being able to do in a small town. Plus I love those jumpsuits with the grease stains, I would totally rock my jumpsuit, kinda like these guys:
My husband is a bit harder to place. He’s kind of a huge math nerd, with no actual practical skills, so the best we’ve come up with is that he can be the guy who goes around to people’s houses and helps them with their computer set-ups. But as people are getting better at computers, and as wireless systems are getting easier to set up, this plan is becoming increasingly weak.
Then there’s the issue of his Dutch accent. The idea that our story is that we’ve just moved like 60 miles, so we’re supposed to be heartland Americans, and the accent totally messes up that story. Sometimes (in my mind of course) I make him mute, other times we explain it with some weird speech impediment or maybe a stroke. I know it’s ridiculous, but it’s a toughie!
Finally, there are my kids. They are going to have to play along with the story too, but the problem there is that they’ve been raised (intentionally) to be pretty smart-assed. I’m trying to imagine them going to some random school and not giving away that they’re from Manhattan, watch the Colbert Report every night, and have strong opinions about the GOP race (and those opinions are not positive). Put on top of that the atheism thing, and I’m getting worried. I haven’t spent enough time in little towns (say, in Iowa) to really know how weird that would be, but I’m guessing pretty obviously not-from-around-here weird.
This is one of my favorite dinner topics, I suggest you try it.
Who here reads Dutch? (#OWS)
Hey I was interviewed last week by the Belgian newspaper De Tijd about Occupy Wall Street and the Alternative Banking group. Here are pdf versions of the first page and the second page, but I wanted to show the picture too, event though this jpeg version isn’t good enough to read:
The muppets strike back
No time this morning but you gotta see this video (hat tip Nathan T. and Michael C.).
What is innovation?
I’ve come to pretty much despise the word “innovation.”
First of all, it’s painfully overused, whether you work in finance or a start-up.
In finance, when people complain that banks and hedge funds should be regulated because they take dangerous risks that they don’t understand and that taxpayers have to backstop, the response, typically from a chorus of business professors and economists, is “don’t over-regulate, you might stifle innovation!”
Never mind that if you dig down to what is meant by financial innovation, it usually consists of creating weird mathematical instruments or contracts that require a complicated computer algorithm to price. So, pretty much the stuff that gets us into weird messes in the first place.
If I needed to write a sign to sum this up, it would read something like “Please stifle financial innovation!”. Actually, Volcker said it best: “the only useful banking innovation was the invention of the ATM.”
As it’s used in start-ups, the word “innovation” is also mostly painful to experience. For the most part it’s baldly used as a buzzword, by someone with a spiffy presentation who is clearly not himself (or herself, don’t want to be sexist) planning to be innovative.
In such a buzzword context, innovation becomes meaningless. At best, it’s their attempt to encourage and cajole the people around them to be innovative, and then perhaps take credit for such innovation. At worst, they fetishize Steve Jobs, which usually means channeling his perfectionist asshole side, thinking that may spur extra innovation.
What’s particularly sad about the abuse of this word is that it is inherently meaningful, and that I see and read about true innovation every day, mostly gone unnoticed by the spin doctors. Maybe that’s because most of it is too technical for business guys to get their heads around.
Another thing I’ve noticed, is that usually the most innovative people are also the most high maintenance pain-in-the-ass people to work with. Sometimes (often) downright hostile in fact.
I’ve come to enjoy this phase of “creative hostility” as a way of getting through skeptical or openly suspicious questioning incredibly efficiently. If I propose something and my most innovative, hostile critics immediately jump down my throat, that’s a sign my idea was a good one. I know that sounds weird but it’s true.
Actually maybe it’s not so weird. I watched this TED talk by Brene Brown where she talks about shame, vulnerability, and innovation. You should watch it, it’s only 20 minutes and it’s good.
Specifically, she talks about how, in order to be innovative, one must make oneself vulnerable. Even though she didn’t have time to really argue this, it resonates with me. The most innovative moments I’ve experienced are when everyone involved is willing to be wrong (vulnerable) and to smell each other’s bullshit (skeptical). Opening yourself up to other people’s skepticism takes courage.
She also describes how cultural norms can come into play at moments of shame or vulnerability or courage. In particular, this thing where men cannot be seen as weak. I think it explains why, when I see innovation, I also tend to see overt displays of macho behavior.
I wouldn’t have it any other way. I say that because I’m pretty sure the alternative is passivity and indifference, which is totally unappealing.
Here’s what I think. Real innovation is a mess and brings up all sorts of things that people don’t actually want to talk about. That’s why we only hear about some watered-down a posteriori description of it.
Parents: don’t put your kid on a diet
Yesterday I read this article about a mother, Dara-Lynn, who put her daughter Bea on a diet, tiger-mom style, and then triumphantly wrote about it in Vogue, and more recently got a book deal. This brings up lots of stuff for me personally, and I think it’s time for me to write about it.
First of all, I would like to address the issue of why people care so much about parenting issues in the first place. I mean, I guess if you aren’t a parent and don’t plan to be one, this doesn’t matter quite as much (even though you of course were yourself parented, so it should still be somewhat relevant).
My message is basically, don’t dismiss parenting discussions- they expose who we are and what we aspire to be as human beings. Questions of how we parent and what values we choose to impose on our children, who are of course vulnerable to such things, are question of how we individually form and inform culture, creating a tiny piece of the larger culture inside our homes. As parents we want to both affect that culture and prepare our children for the larger world, and it’s a tricky balance.
So given that, what values are we imposing on our kids when we put them on a diet? I can infer that somewhat from the article but I can also speak from personal experience, because my parents put me on a diet when I was 10. My parents set up system whereby I’d be punished (by losing my allowance) if I didn’t lose at least a certain amount of weight each week. They also explained to me how calories worked. That’s it. They let me loose with that information and ultimatum. (ooh, just remembered: the reward for losing 5 pounds was, ironically, a candy bar.)
It was a little strange, in retrospect, for a few reasons. First, I had been chubby since soon after birth. My parents are both and were both chubby. My grandmother lived with us off and on and constantly hoarded bags of candy and fed them to us constantly while watching soap operas. My older brother was also somewhat chubby. In spite of this, I was the only person on this regimen, and nothing else about our eating habits changed besides that I was expected to keep track of the calories I ate- the food itself was still hamburgers, boiled vegatables, and spaghetti with butter.
I guess this may have been my first experience dealing first-hand with a misleading, pseudo-quantitative model. I was told by my parents that losing weight was a simple concept of calories in and calories out, and thus must be simple to deal with. I was honestly too young to question this, and to also question why they hadn’t achieved their perfect weights if it was so simple.
Now, to the question of values. From my perspective as a kid, the values I learned were the following:
- I am terrible at following simple directions, because I can’t seem to control my eating,
- I don’t look good to other people, and
- it’s really important to look good to other people.
All in all, a pretty nasty set of values that I carried around with me like a sin for years, until something else happened, which I will get back to soon.
You might say that the article with Dara-Lynn and Bea is completely different, because first of all the mother wasn’t offhand in planning her daughter’s diet: she lived and breathed the control that she thought was required to get her daughter to lose weight. Also, it was a “success,” in that Bea lost the weight her mother set out for her. Even so, I see parallels for Bea’s received wisdom from her mother:
- You have to submit entirely to someone else (me, your mother) because you can’t be trusted to follow your instincts,
- You didn’t look good to other people, and
- it’s really important to look good to other people.
But I do feel sorry for her on top of what I went through, because now her mom is not only in a national magazine bragging about her control over her kids, she’s also gotten a book deal to go into the details of this control freakiness. Because it’s all about how a mother can foster good eating habits in her kid. I guess.
If there was any justice in the world, it would be Bea getting the book deal, in advance, to describe what it’s like to live with such a control freak mother. I honestly wish her luck.
A few concluding remarks. First of all, if you were wondering when the nerdy stuff was coming, here it is: there’s enormous selection bias going on. For every mother you hear about who drags their kids kicking and screaming through a diet, there are hundreds of poor kids who ended up like me, with failed enforced diets and incredibly guilty consciences (but at least no pictures in Vogue of their shame). We don’t hear about them, of course, because nobody wants to.
Next, although Bea is a “success story” now, I’m pretty well versed in statistics on teenage dieting and I’m anticipating lots of terrible experiences for the daughters of the women who will buy this book. A generation of girls who are ashamed and self-conscious.
Finally, how I got over my shame. It was mostly a coping mechanism. I went into a hospital when I was in 10th grade, with deep feelings of depression, and missed a few weeks of school. It was a critical moment for me, and I knew it. I had to decide whether to be depressed and passive for the rest of my life or whether to try to live life on my own terms. I basically decided to take on the following “anti-values” in order to obviate the terrible self-image I harbored at that time. I came up with these three anti-values, which I still live by:
- I forgive myself for not being good at controlling myself, because I love my body, even parts of it that confound me,
- I look good to myself, and
- it’s not that important to look good to other people.
Probably not stuff for a book deal, but at least it’s kept me from giving my kids eating disorders.
Vote with your wallet
Today we have a guest post from Elizabeth Friedrich, with whom I work on the Credit Union Findr webapp I blogged about here. Cross posted from Beyond the Bailout.
In 2008, America was in shock seeing the stock market crash, the housing market collapse, and a $12.8 trillion-dollar bailout of financial institutions many felt were responsible for the economic crash. We were paralyzed, unable to see past the madness and despair. At first, our national response was minimal. Americans had lost their homes, jobs, everything, and the anger was evident in the national mood. However, from that desperation and pain-came action and movement! People began to organize in order to decide their own fate, not leave it up to the 1% and/or a complacent government. Action came in many forms, marches in streets, letter-writing and media campaigns, peaceful occupation of public spaces, and of course “Move Your Money.”
The Move Your Money Project actually started several years ago, but had not gained significant momentum until last year, when consumers began to voice their anger and outrage at the very largest for-profit financial institutions, who had been bailed-out with billions in tax-payer dollars, and rather than using those funds to expand credit to communities in need, instead sat on this cheap money and tightened their lending standards. With historic low interest rates set and held by the Federal Reserve system, profit margins became slimmer and many banks responded by increasing their fees across the board, much to the ire of many fed-up consumers. This action was a catalyst to finally moved people to question the role of their so-called trusted financial institutions and on November 5 2011, over 600,000 people moved their money totaling $80 million dollars out of traditional banking institutions into credit unions and community banks across the country. In addition to that single day of action, over the last few years, over 4 million accounts have moved from the nation’s largest Wall Street banks according to Moebs Services, an economic research firm in Lake Bluff, IL. They also project an additional 12 million people will do the same in the next two years.
This mass-exodus from the big banks is by no means accidental and shows the overwhelming, yet untapped energy of the American people who have grown discouraged with a government that was unwilling or unable to enact true, meaningful financial reform. Many of their reasons for this are clear: consumers are looking for ethical practices, re-investment in local communities, fewer fees and more service, and the end of “Too Big to Fail” financial oligopolies. Naturally, people began focusing on credit unions and community development banks, institutions that have the public interest in mind and seek to strengthen local communities. At these community-focused institutions you actually know where you money goes and what is used for.
Convenience over accountability…
Our culture has taught us that convenience is primary tool when making decisions as opposed to accountability and fairness. Just as we make other choices; purchasing food, clothing, and transportation. Convenience is often the factor that carries the most weight in our decisions rather than ethics. This comes with many consequences – often at the expense of the environment and disadvantaged communities. Hopefully, in the future accountability and transparency will be a primary motivation for consumers when making financial decisions.
What to do?
Today we have a choice whether we know it or not. There is a parallel financial industry functioning on the fringe: Community Development Financial Institutions (CDFIs), a national network community development banks, loan funds, and community development credit unions (CDCUs). These are institutions with a primary mission to strengthen vulnerable communities and invest locally. Banks and credit unions are regulated depository institutions; banks by the Federal Deposit Insurance Corporation (FDIC) and credit unions by the National Credit Union Administration (NCUA). Credit unions offer many of the same services as banks: mortgages, car loans, personal loans, small dollar loans, credit cards, savings/checking accounts, international money transfers, Individual Development Accounts (matched-savings accounts), retirement planning, financial literacy education and budgeting, affordable savings and checking accounts, and credit and debit cards with low minimum balances and flexible terms. They are not-for-profit financial institutions created to serve their local communities and members first. Unlike banks, which can serve any customer that walks in the door, credit unions are restricted to specific fields of membership.
This means that consumers have more options than ever with respect to their primary financial institutions, and a major selling-point for many is that the money they deposit in their credit union stays local within the specific field of membership. Rather than profiting shareholders, income earned at a credit union, dividends are returned in different forms from free services to better interest rates or to expand services in the community.
Making the choice to bank at a credit union or a community development bank creates a multiplier effect for the local communities being served, and ultimately in the entire the financial system. When you invest in a community development financial institution you are investing in job creation, building schools, developing housing and financing small businesses.
Some banks may be “too big to fail,” but consumers are waking up and realizing they have a choice where they put their money, and the impact that choice can have in their own communities. Rather than letting too big to fail institutions gamble away their hard-earned cash, people are choosing to exercise their power as consumers and speak with their wallet. In banking this means find the smart, responsible alternative for you, your family and your community, and community development banks and community development credit unions are a logical choice.
What’s Mahout?
Mahout is an Apache project, which means it’s open source software.
Specifically, Mahout consists of machine learning algorithms that are (typically) map-reducable, and implemented in a map-reduce framework (Hadoop), which means you can either use them on the cloud or on your own personal distributed cluster of machines.
So in other words, if you have a massive amount of data up in the cloud, and you want to apply some machine-learning algorithm to your data, then you may want to consider using Mahout.
Yesterday I learned about a recommendation algorithm and how to map-reduce it (i.e. make it as fast as I want by distributing the work on many machines) by reading Mahout in Action. And the cool thing is that it’s already implemented and optimized, which is good because there’s a big difference between thinking I know how to map-reduce something and making it fast.
So if Netflix ever fails, but their data is miraculously left intact, they can send me and a few other nerds in as a kind of data scientist rescue squad and we can help figure out how to reassemble the recommendations of new movies based on what people have already watched and rated.
If that ever really happens, I hope we’d get t-shirts that say “data scientist rescue squad” on the back.
Update: a mahout is also someone who drives an elephant. And Mahout drives Hadoop, which is the name of Doug Cutting‘s son’s toy elephant. Doug is the guy who started Hadoop at Yahoo! but now he’s at Cloudera.





