Yesterday a couple of guys from Mortar came to explain their hadoop platform. You can see a short demo here. I wanted to explain it at a really high level because it’s cool and a big deal for someone like me. I’m not a computer scientist by training, and Mortar allows me to work with huge amounts of data relatively easily. In other words, I’m not sure what ultimately will be the interface for analytics people like me to get access to massive data, but it will be something like this, if not this.
To back up one second, for people who are nodding off, here’s the thing. If you have terabytes of data to crunch, you can’t put it on your computer to take a look at it, and then crunch, because your computer is too small. So you need to pre-crunch. That’s pretty much the problem we need to solve, and people have solved it either one of two ways.
The first is to put your data onto a big relational database, on the cloud or something, and use SQL or some such language to do the crunching (and aggregating and what have you) until it’s small enough to deal with, and then download it and finish it off on your computer. The second solution, called MapReduce (the idea started at Google), or hadoop (the open-source implementation started at Yahoo) allows you to work on the raw data directly where it lies (e.g. on the Amazon cloud (where it’s actually Elastic MapReduce, which I believe is a fork of hadoop)), in iterative steps called mappings and reduction steps.
Actually there’s an argument to be made, apparently, because I heard it at the Strata conference, that data scientists should never use hadoop at all, that we should always just use relational databases. However, that doesn’t seem economical, the way it’s set up at my work anyway. Please comment if you have an opinion about this because it’s interesting to me how split the data science community seems to be about this issue.
On the other hand, if you can make using hadoop as easy as using SQL, then who cares? That’s kind of what’s happened with Mortar. Let me explain.
Mortar has a web-based interface with two windows. On top we have the pig window and on the bottom a python editor. The pig window is in charge and you can call python functions in the pig script if you have defined them below. Pig is something like SQL but is procedural, so you tell it when to join and when to aggregate and what functions to use in what order. Then pig figures out how to turn your code into map-reduce steps, including how many iterations. They say pig is good at this but my guess is that if you really don’t know anything about how map-reduce works then it’s possible to write pig code that’s super inefficient.
One cool feature, which I think comes from pig itself but in any case is nicely viewable through the Mortar interface, is that you can ask it to “illustrate” the resulting map-reduce code and it takes a small sample of your data and shows example data (of “every type” in a certain sense) at every step of the process. This is super useful as a bug-watching feature to see that it’s looking good with small data sets.
The interface is well designed and easy to use. Overall it reduces a pretty scary and giant data job to something that would probably take me about a week to feel comfortable. And new hires who know python can get up to speed really quickly.
There are some issues right now, but the Mortar guys seem eager to improve the product quickly. To name a few:
- it’s not yet connected to git (although you can save pig and python code you’ve already run),
- you can’t import most python modules except super basic ones like math (including ones you’ve written; right now you have to copy and paste into their editor),
- they won’t be able to ever let you import numpy because they are actually using jython and numpy is c-based,
- it doesn’t automatically shut down the cluster after your job is finished, and
- it doesn’t yet allow people to share a cluster
These last two mean that you have to be pretty on top of your stuff, which is too bad if you want to leave for the night and start a job and then bike home and feed your kids and put them to bed. Which is kind of my style.
Please tell me if any of you know other approaches that allow python-savvy (but not java savvy) analytics nerds access to hadoop in an easy way!
So I went to see the Occupy Wall Street protests this morning before work and this evening after work again. Here are some of my comments and observations.
First, if you are interested in checking it out, know that there are small marches at opening and closing bell for the market.
However, the police have made it basically impossible to walk on Wall Street, due to some incredibly annoying barricades.
So for our march this morning we seemed to just circle the city block where the protest is based, although I didn’t stay til the end so it’s possible they decided to very very slowly march on Wall Street proper.
Second, they have “assemblies” twice a day, with guest speakers sometimes (Michael Moore, Susan Sarandon and Cornel West have visited), and this is where general announcements are made. The crowd was quite large tonight and it was difficult to hear what the speaker and the repeaters were saying, which is frustrating. But maybe it’s easier at the 1pm assembly. Also, it seems to be easier to actually discuss issues in the morning- at night it gets loud and kind of crazy and hard to focus in my opinion.
Next, I’d like to address the issue of the message of the protesters being dismissed as incoherent. For the record, I went to a conference at the end of 2009 at Columbia Business School on the financial crisis and what we should do about it, where the speakers were fancy economists from central banks and CEOs of international banks, and they were about as incoherent as these protesters. There was absolutely no getting them to say anything that was an actual plan or even an attempt at a plan for changing the system so this mess wouldn’t happen again. I should know, because there was a question and answer period and I asked.
Having said that, there have been some pretty unconvincing statements reported from some of the protesters in terms of what they would like to see. For example, some of them seem to think that short selling should be banned. As some of you know, I disagree. In fact there are lots of seriously corrupt and ridiculous things going on in the financial system which they should know about and they should protest, and I’d like to invite them to educate themselves.
In particular, if you are someone interested in knowing stuff about how the financial system works, then please ask! A major part of why I blog is to try to inform people about these things who are interested. Please comment below and ask whatever you want, and if I don’t know the answer I will find someone who does, or I will blog about the question.
Having said that, I’d like to add that it’s on the one hand perfectly reasonable that people don’t understand the financial system, because it has essentially been set up to be too complicated to understand, and on the other hand it’s also reasonable to think of the entire financial system as a black box which can be judged by its outputs.
Finally, if we are going to judge the system by looking at its outputs, then these protesters, who are in general young, with educations, huge students debts, and hopeless outlooks, have a pretty dismal view. In other words they have every right to complain that the system is fucking them, even though they don’t know how the system works. I for one am super proud that they’re out there doing something, even if it’s not obviously organized and polished, rather than passively sitting by.
As a long-time (yes since they sucked) Red Sox fan, let me just say, the Tampa Bay Rays totally deserve to be in the play-offs. They made me a fan last night with an absolutely amazing game.
Last night I was talking to a friend of mine about my teaching experiences, and what’s it’s like to be a woman in math and to be taken seriously. We were going over the standard stuff, that women are too self-effacing compared to men and tend not to strut their stuff enough. But then I remembered this story from my early teaching experiences that kind of put a different spin on that.
I was in grad school, and over the summer I went to Berkeley to teach at a women in math program, which was still called the “Mill’s program” even though it was being held at Berkeley. It was a really fun experience, something like 30 days of lecture and problem session, and I led the problem sessions.
It was some time in the second week when, one day because of something or other, I hadn’t prepared completely and I apologized to the class for being slightly unprepared. I said something like, “sorry I’m not completely prepared today”. I remember thinking that, in spite of that, the class went very well and there was no “damage” from my being unprepared. Every other day I was completely, perhaps overly prepared, and that was the only day I ever mentioned something about my preparedness.
At the end of the summer we got back teaching evaluations, and I remember that a full half of the evaluations described me as unprepared.
I made a promise to myself never ever to apologize for anything again. And I never have, and I’ve never been accused like that since. Which isn’t to say I pretend to be a perfect teacher, but there are subtle ways of dealing with imperfections (my favorite: turn a self-criticism into a flattery. Instead of saying, oh how stupid I am for not thinking of that, say oh how smart you are for thinking of that. Generosity is not a negative in my experience!).
Going back to last night, though, it’ a two-way street. Women may be too self-effacing, but other people (including women!) are absolutely too dismissive. It’s a very important thing to keep in mind when you are teaching or presenting.
One other thing, in a one-on-one, professional setting, I believe you can apologize and not be executed for it (sometimes and depending on the person), but in a teacher-students setting, or when you’re presenting to clients in business, or even when you’re presenting to colleagues, you’re giving a performance and need to be flawlessly confident.
In an ideal world, we would use this information to learn to become better audiences, to not be dismissive and overly harsh of self-effacing people, and I do try to keep this in mind when I’m in the audience. But it’s going to take lots of effort for this to happen on a large scale, especially among strangers. It’s a cultural axiom in a certain sense.
My advice to young people, especially women: never apologize.
This is a guest post by FogOfWar.
I was originally going to lead with a tongue-in-cheek comment (later in the post now), but then the NYPD did something colossally stupid. If you haven’t seen it, here’s the video from this last weekend. It pretty much speaks for itself.
There’s a lot to be said about freedom of expression and police overreaction. I’ve been to see the protests a number of times, and they’ve never been violent and in fact seem pretty well trained in the confines of freedom of assembly in the US legal system. Using mace against an imminent threat of violence is OK for the police, but the video seems to show no threatening moves made at all (and it runs for a good period before the police attack so it wasn’t edited out).
I’d suggest the NYPD be shown the following video (taken from the protests in Greece) to demonstrate when things reach a level where force might be an appropriate response. Note that the crowd is attacking with sticks, Molotov cocktails and a fucking bowling ball. In contrast, the NYPD appears to be pepper spraying people for just holding signs and walking down the street. What the fuck?
There are maybe a few hundred people consistently protesting at “Occupy Wall Street” for about 10 days now. It’s got a definite crunchy vibe to the center. Drumming and Mohawks are mandatory:
But also a (growing?) contingent of more mainstream participants like this one:
Here’s a crowd shot for scale:
And some people painting signs:
And then of course, there’s the dreaded “consensus circle”:
It’s hard to tell what they really want to happen—this was up at one of the information booths (but then down the next time I went):
Misspelled “derivatives”, and there are some things on that list that are spot on and then others that are just weird and irrelevant (DTC? Really?). I don’t think you can hold that against them though. I work in the industry, and I’ve been spending the last three years thinking about this stuff and I still find it confusing and hard to come up with a cohesive plan of what I think should be done. At least these people are doing something, even if it’s a bit incoherent at times.
I have to end with my all time favorite sign from the protest. Someone was looking for good cardboard and inadvertently came up with the following:
“Delicious pizza to pay off the taxpayers”. Now that’s a slogan I think we can all rally behind!
Do you remember, back in 2005 or 2006 or even up to early 2008, how absolutely everyone seemed to be buying flat screen TVs? And not only one, they’d actually buy new ones when new models came out, or ones with different high definition properties. And not just people who could afford it, either. The marketers did an excellent job in somehow convincing people that they needed these flat screen TVs so bad that they should just put it on their credit cards, all 3 thousand dollars of it, or whatever those things cost.
I don’t know exactly how much they cost because I never bought one. The last TV we bought was in 1997 and it still works, for the most part, although it’s really hard to turn it on and off. When it finally kicks the bucket I’m thinking we go without a TV, since TV pretty much sucks anyway. When we do watch it, it’s for live sports (local, or nationally televised, since we don’t pay for cable). Baseball we watch or listen to on the computer.
I was reminded of the the “flat screen TV era” by my friend Ian Langmore the other day when we were discussing household debt amnesty. His argument against debt amnesty for consumers was that they might spend it on crappy things. His example was luxury dog poo, but I’ve been obsessed with the flat screen TV phenomenon ever since a friend of mine, who was $120,000 in debt and didn’t have a salary, somehow managed to buy a flat screen TV in 2007. It blew me away in terms of wasteful consumerism. Ian found this unbelievable blog which kind of sums up my concerns.
In Ian’s opinion, the danger of amnesty, or any system where money is put willy-nilly into the hands of consumers, is twofold:
1) We waste time on unproductive activities. E.g. people spent time buying/building cars that are unneeded.
2) If a miscalculation is made, then the over-leveraged money-go-round stops with a huge mis-balance. E.g. home mortgage crisis.
These are very good points, and put together form a lesson we somehow can’t learn, although perhaps that can be partially explained by this article.
I have two thoughts. First, I’m also uncomfortable putting money in the hands of irresponsible consumers. But the truth is, the way I see it is currently working, we are already putting money in the hands of irresponsible bankers (that’s what the term “injection of liquidity” really means), and they are not doing anything with it, so let’s try something else. In other words, an alternative unpleasant idea.
Second, I don’t think we are going to see a new wave of flat screen TV buying any time soon. If we put money into the hands of consumers right now, I think we’d see them pay down their debts, go to the doctor, and buy jeans for their kids. Of course, there is always someone whose pockets burn with cash, and they would waste money in any situation. Let’s face it, though, credit is tight right now compared to the mid-2000’s. In fact, since economists seem to have a tough time spotting bubbles until afterwards, maybe we can take “a huge part of the population starts buying useless gadgets on credit” as almost a definition, or at least a leading indicator. Then at least there would be some point to all of that wasteful spending.
Here are the annotated slides from my Strata talk. The audience consisted of business people interested in big data. Many of them were coming from startups that are newly formed or are currently being formed, and are wondering who to hire.
When do you need a data scientist?
When you have too much data for Excel to handle: data scientists know how to deal with large data sets.
When your data visualization skills are being stretched: as we will see, data scientists are skilled (or should be) at data visualization and should be able to figure out a way to visualize most quantitative things that you can describe with words.
When you aren’t sure if something is noise or information: this is a big one, and we will come back to it.
When you don’t know what a confidence interval is: this is related to the above; it refers to the fact that almost every number you see coming out of your business is actually an estimate of something, and the question you constantly face is, how trustworthy is that estimate?
Let’s take a step back: Should you need a data scientist?
Are you asking the right questions? Is there a business that you’re not in that you could be in if you were thinking more quantitatively? Big data is making things possible that weren’t just a few years ago.
Are you getting the most out of your data? In other words, are you sitting on a bunch of delicious data and not even trying to mine it for your business?
Are you anticipating shocks to your business? As we will see, data scientists can help you do this in ways you may be surprised at.
Are you running your business sufficiently quantitatively? Are you not collecting the data (or not collecting it in a centralized way) that would lead to opportunities for data mining?
So, you’ve decided to hire a Data Scientist (nice move!)
What do you need to get started?
Data storage. You gotta keep all your data in one place and in some unified format.
Data access — usually through a database (payoffs for different types). Specifically, you can pay for someone else to run a convenient SQL database that people know how to use walking in the door without much training, or you could set something up that’s open source and “free” but then it will probably take more time to set up and make take the data scientists longer to figure out how to use. The investment here is to create tools to make it convenient to use.
Larger-scale or less uniform data may require Hadoop access (and someone with real tech expertise to set it up). The larger your data is the more complicated and developed your skills need to be to access it. But it’s getting easier (and other people here at the conference can tell you all you need to know about services like this).
Who and how should you hire? It’s not obvious how to hire a data scientist, especially if your business so far consists of less mathematical people.
A math major? Perhaps a Masters in statistics? Or a Ph.D. in machine learning? If you’re looking for someone to implement a specific thing, then you just need proof that they’re smart and know some relevant stuff. But typically you’re asking more than that: you’re asking for them to design models to answer hard questions and even to figure out what the right questions are. For that reason you need to see that the candidate has the ability to think independently and creatively. A Ph.D. is evidence of this but not the only evidence- some people could get into grad school or even go for a while but decide they are not academically-minded, and that’s okay (but you should be looking for someone who could have gotten a Ph.D. if they’d wanted to). As long as they went somewhere and challenged themselves and did new stuff and created something, that’s what you want to see. I’ll talk about specific skills you’d like in a later section, but keep in mind that these are people who are freaking smart and can learn new skills, so you shouldn’t obsess over something small like whether they already know SQL.
What should the job description include? Things like, super quantitative, can work independently, know machine learning or time series analysis, data visualization, statistics, knows how to program, loves data.
Who even interviews someone like this? Consider getting a data scientist as a consultant just to interview a candidate to see if they are as smart as they claim to be. But at the same time you want to make sure they are good communicators, so ask them to explain their stuff to you (and ask them to explain stuff that has been on your mind lately too) and make sure they can.
Also: don’t confuse a data scientist with a software engineer! Just as software engineers focus on their craft and aren’t expected to be experts at the craft of modeling, data scientists know how to program in the sense that they typically know how to use a scripting language like python to manipulate the data into a form where they can do analytics on it. They sometimes even know a bit of java or C, but they aren’t software engineers, and asking them to be is missing the point of their value to your business.
What do you want from them?
Here are some basic skills you should be looking for when you’re hiring a data scientist. They are general enough that they should have some form of all of them (but again don’t be too choosy about exactly how they can address the below needs, because if they’re super smart they can learn more):
- Data grappling skills: they should know how to move data around and manipulate data with some programming language or languages.
- Data viz experience: they should know how to draw informative pictures of data. That should in fact be the very first thing they do when they encounter new data
- Knowledge of stats, errorbars, confidence intervals: ask them to explain this stuff to you. They should be able to.
- Experience with forecasting and prediction, both general and specific (ex): lots of variety here, and if you have more than one data scientist position open, I’d try to get people from different backgrounds (finance and machine learning for example) because you’ll get great cross-pollination that way
- Great communication skills: data scientists will be a big part of your business and will contribute to communications with big clients.
What does a Data Scientist want from you? This is an important question because data scientists are in high demand and are highly educated and can get poached easily.
Interesting, challenging work. We’re talking about nerds here, and they love puzzles, and they get bored easily. Make sure they have opportunities to work on good stuff or they’ll get other jobs. Make sure they are encouraged to think of their own projects when it’s possible.
Lots of great data (data is sexy!): data scientists love data, they play with it and become intimate with it. Make sure you have lots of data, or at least really high-quality data, or soon will, before asking a data scientist to work for you. Data science is an experimental science and cannot be done without data!
To be needed, and to have central importance to the business. Hopefully it’s obvious that you will want your data scientists to play a central role in your business.
To be part of a team that is building something: this should be true of anyone working in business, especially startups. If your candidate wants to write academic papers and sit around while they get published, then hire someone else.
A good and ethically sound work atmosphere.
Cash money. Most data scientists aren’t totally focused on money though or they would go into finance.
Further business reasons for hiring a Data Scientist
Reporting help: automatically generated daily reports can be a pain to set up and can require lots of tech work and may even require a dedicated person to generate charts. Data scientists can pull together certain kinds of reports in a matter of days or weeks and generate them every day with cronjobs. Here’s a sample picture of something I did at my job:
A/B testing: data scientists help you set up A/B testing rigorously.
Beyond A/B testing: adaptability and customization. What you really want to do is get beyond A/B testing. Instead of having the paradigm where customers come to the ad and respond in a certain way, we want to have the (right) ad come to the customer.
Knowing whether numbers are random (seasonality) or require action. If revenue goes down in a certain week, is that because of noise? Or is it because it always goes down the week after Labor Day? Data scientists can answer questions like this.
What-if analysis: you can ask data scientists to estimate what would happen to revenue (or some other stat) if a client drops you, or if you gain a new client, or if someone doubles their bid at an auction (more on this later).
Help with business planning: Will there be enough data to answer a given question? Will there be enough data to optimize on the answer? These are some of the most difficult and most important questions, and the fact that a data scientist can help you answer them means they will be central to the business.
Education for senior management: senior people who talk to and recruit new clients will need to be able to explain how to think about the data, the signals, the stats, and the errorbars in a rigorous and credible way. Data scientists can and should take on the role of an educator for situations like this.
Mathematically sound communication to clients: you may have situations where you need the data scientists to talk directly to clients or to their data scientists. This is yet another reason to make sure you hire someone with excellent communication skills, because they will be representing your business to really smart people who can see through bullshit.
Case Study: Stress Tests
We can learn from finance: the idea of a stress test is stolen directly from finance, where we look at how replays of things like the credit crisis would affect portfolios. I wanted to do something like that but for general environmental effects that a business like mine, which hosts an advertising platform, encounters.
You know how big changes will affect your business directionally and specifically. But do you know how combinations will play out? Stress tests allow you to combine changes and estimate their overall effect quantitatively. For example, say we want to know how lowering or raising their bids (by some scalar amount) will effect advertisers impression share (the number of times their ads get displayed to users). Then we can run that as a scenario (for each advertiser separately) using the last two weeks (say) of auction data with everything else kept the same, and compare it to what actually happened in the last two weeks. This gives an estimate of how such a change would affect impression change in the future. Here’s a heat map of possible results of such a “stress test”:
We could also:
- run scenarios which combine things like the above
- run scenarios which ask different questions: how would advertisers be affected if a new advertiser entered the auction? If we change the minimum bid? If one of the servers fails? If we grow into new markets?
- run scenarios from the perspective of the business: how would revenue change if the bids change?
In the end stress tests can benefit any client-facing person or anyone who wants to anticipate revenue, so across many of the verticals of the business.
I remember when I moved to New York in 2005. I found it intimidating and shocking how aggressively people vied for seats on the subway. I live near Columbia so the 1 train is my line, and of course everyone thinks their subway line is the most overused and crazy line, but in this case I’m right. I came from Boston, where we have subways too, four little itty bitty ones, and we are extremely polite to each other and, in particular, we never touch. By contrast here were these New Yorkers not only touching but literally squeezing into these tiny seats and sweating all over each other in the summer.
After about 3 months of living here I got really into it. I was in love with this city, and every gritty thing about it, and I considered the shared experience of the subway a sign of a larger public communistic love. Here they were, people from all walks of life, sharing their sweat! Isn’t it beautiful?
That kind of admiration only grew in the two years I stayed a professor at Barnard, which meant I almost never left the cozy neighborhood of Morningside Heights, so subway rides were rather rare, amusing events. I loved the subway and I developed theories about when people start talking on the subway (in three situations: 1) someone who is incredibly smelly gets off the train and everyone needs to talk about how smelly they were, 2) someone who is incredibly sick and coughing up a lung gets off the train and everyone has to talk about how sick and nasty they were, and 3) the train stops in the tunnel and the announcer tells us we have no idea when we will be able to move, and everyone has to talk about their stuck-in-a-tunnel-during-9/11 experiences.)
As soon as I started working at D.E. Shaw in midtown, and commuted during rush hour, I got real. I figured out exactly where to stand, and I mean exactly where on each platform, to maximize my chances of getting a seat once the train came. I figured out, depending on how many people were on which platform in Times Square, and the subsequent stations as we passed them, what the recent train traffic pattern had been in terms of the express 2/3 train and my local 1 train, and sometimes I’d do crazy things like get off the express train early to get on the 1 train because I’d anticipate that if I waited til 96th street like everyone else, there would be no chance I could get on the 1 train. Actually looking back, I almost never sat down at all during these commutes, even when I was pregnant.
Which comes to the turn in my story. When I was heavily pregnant, commuting on the subway was actually hellish. I had no balance, and felt vulnerable, and being squished up against people with no place to hold on was really scary. For the most part commuters are a selfish bunch, and people sitting would pretend not to notice me, so they wouldn’t have to give up their seat. I promised myself I’d never be that jerk.
For the last two weeks of my pregnancy I took a cab to work every day, but even so coming home was another story, since it’s hard to get a cab in Times Square at 5pm. I remember one time some asshole in a suit actually ran to grab a cab that had stopped for me, and he beat me because… I was 9 months pregnant and couldn’t keep up with him. I started crying, on the street, until this nice pedicab guy pulled over and asked me if he could help. I told him I lived all the way uptown and he biked me around until he found me a cab; he refused to let me pay. I still love that guy.
Once I started down the road of getting up for pregnant people, though, it was a short logical step to never sitting down again. After all, there are all kinds of hidden reasons people may need to sit down more than I do. What if their feet are killing them after standing all day at work? What if they have balance problems?
For a while I decided it’s okay to sit if everyone else had an available seat. That seemed safe. But then I’d be sitting there, spaced out or reading, with a sea of empty seats around me, and all of a sudden a huge group of people would converge and somehow I’d be face to face with someone with a murderous look which said, you motherfucker you’re sitting in my seat. In the end, it’s become my policy to just never sit down.
I do of course still think about the question of where’s the best place to stand in the subway. This is a whole different optimization play, which for intellectual property reasons I won’t share with you all, since I don’t want more competition than I already have. Just one hint: don’t get on in the middle of the car. Always get on at one of the ends.
Best article ever about beards.
First, it needs to be said that, as I have learned in this book I’m reading, it’s probably a bad idea to make statements about learning when you make “cohort-to-cohort comparisons” instead of following actual students along in time. In other words, if you compare how well the 3rd grade did in a test one year to the next, then for the most part the difference could be explained by the fact that they are different populations or demographics. Indeed the College Board, which administers the SAT, explains that the scores went down this year because more and more diverse kids are taking the test. So that’s encouraging, and it makes you think that the statement “SAT scores went down” is in this case pretty meaningless.
But is it meaningless for that reason?
Keep in mind that these are small differences we’re talking about, but with a pretty huge sample size overall. Even so, it would be nice to see some errorbars and see the methodology for computing errorbars.
What I’m really worried about though is the “equating” part of the process. That’s the process by which they decide how to compare tests from year to year, mostly by having questions in common that are ungraded. At least that’s what I’m guessing, it’s actually not clear from their website.
My first question is, are they keeping in mind the errors for the equating process? (I find it annoying how often people, when they calculate errors, only calculate based on the very last step they take in a very sketchy overall process with many steps.) For example, is their equating process so good that they can really tell us with statistical significance that American Indians as a group did 2 points worse on the writing test (see this article for numbers like this)? I am pretty sure that’s a best guess with significant error bars.
Additional note: found this quote in a survey paper on equating methodologies (top of page 519):
Almost all test-equating studies ignore the issue of the standard error of the equating
Second, I’m really worried about the equating process and its errorbars for the following reason: the number of repeat testers varies widely depending on the demographic, and also from year to year. How then can we assess performance on the “linking questions” (the questions that are repeated on different tests) if some kids (in fact the kids more likely to be practicing for the test) are seeing them repeatedly? Is that controlled for, and how? Are they removing repeat testers?
This brings me to my main complaint about all of this. Why is the SAT equating methodology not open source? Isn’t the proprietary “intellectual property” in the test itself? Am I missing a link? I’d really like to take a look. Even better of course if the methodology is open source (as in there’s an available script which actually computes the scores starting with raw data) and the data is also available with anonymization of course.
Being a mathematician, I find myself forced to consider statements like “higher taxes kill jobs” as statements of theorems with missing stated assumptions. How could you fill in the assumptions and prove this theorem?
First I think about extreme cases- sometimes extreme situations need fewer assumptions, they kind of spill out as obvious. So here’s one, the tax rate is at 80%, what would happen if we raised taxes? My first reaction is, 80%!? That must mean you have way too much government and regulation and for those reasons businesses are probably already quite pinned down and don’t have lots of freedom- don’t tax them more, that will make their good ideas (if they have them) all the more suffocated. Just think of the paperwork you’d need to go through in a society that government-heavy, to hire someone.
What’s another extreme case? How about taxes are super low, more like fees for doing business. Then no, I don’t think raising them a moderate amount would kill jobs at all, in fact it may introduce enough government to make things less wild west and safer for businesses to operate.
So in other words at some level I buy the anti-regulation anti-government angle. I don’t want super duper high taxes because I think it encourages too much bureaucracy and that stuff is boring (but some amount of it is necessary to make things safe).
Moreover I’m assuming that governments generally use taxes to protect people from food poisoning and the like, regulate to force companies to play fair, and as social safety nets when things go bad, and that they’re not particularly efficient. Those of course are my assumptions, which anyone can disagree with.
But in terms of proving my theorem, I’m stuck thinking it’s more like, there’s some point in between very low and very high taxes where it gradually becomes true that raising taxes more will indeed start to kill jobs.
How about our situation now? Right now we have pretty low taxes by historical measures, and moreover the known loopholes mean that businesses (especially big ones with fancy lawyers) pay much less than their stated tax rate.
Why, in this case, would a moderate bump in their tax rates kill jobs?
Here’s a possible argument: if higher taxes actually encourage more regulation, then that could be a major problem for smaller businesses, who don’t have the margin for dealing with hiring that many lawyers for compliance issues. Although this article argues that “regulation kills jobs” is an invalid statement in general.
Pet peeve of mine: when you hear conservatives talk about killing jobs, they often frame it in terms of struggling small businesses, often run by a woman. But it’s easy enough to imagine that we introduce taxes and regulation that are easier for small businesses to avoid smothering them. It’s really the huge businesses that we want to see start hiring, and it’s the huge businesses that pay so little taxes.
Here’s another one: if you raise taxes people will spend their cash on taxes instead of hiring people. But wait, that doesn’t apply right now when we have so much frigging cash on hand (and hidden in other countries). In other words, companies are not not hiring people for cash flow reasons, it’s because they don’t see the demand.
In the end I can’t see how to prove or even argue that theorem, assuming today’s conditions. Would love to hear the argument I’m missing.
I was delighted to meet a huge number of fun, hopeful, and excited nerds throughout the day. Since my talk was pretty early in the morning, I was able to relax afterwards and just enjoy all the questions and remarks that people wanted to discuss with me.
Some were people with lots of data, looking for data scientists who could analyze it for them, others were working with packs of data scientists (herds? covens?) and were in search of data. It was fun to try to help them find each other, as well as to hear about all the super nerdy and data-driven businesses that are getting off the ground right now. It certainly was an optimistic tone, I didn’t feel like we were in the middle of a double-dip recession for the entire day (well, at least til I got home and looked at the Greek default news).
Conferences like these are excellent; they allow people to get together and learn each others’ languages and the existence of the new tools and techniques in use or in development. They also save people lots of time, make fast connection that would otherwise difficult or impossible, and of course sometimes inspire great new ideas. Too bad they are so expensive!
I also learned that there’s such thing as a “data scientist in residence,” held of course by very few people, which is the equivalent in academic math to having a gig at the Institute for Advanced Study in Princeton. Wow. I still haven’t decided whether I’d want such a cushy job. After all, I think I learn the most when I have reasonable pressure to get stuff done with actual data. On the other hand maybe that much freedom would allow one to do really cool stuff. Dunno.
I’m reading an interesting book by Douglas Harris about the value-added model movement, called Value-added Measures in Education, available here from Harvard Education Press. Harris goes into a very reasonable critique of how “snapshot” views of students, teachers, and school are a very poor assessment of teacher ability, since they are absolute measurements rather than changes in knowledge. Kind of like comparing the Dow to the S&P and concluding that you should definitely invest in Dow stocks since they are ten times better, it’s all about the return on a test score or an index, not the absolute number, when you are trying to gauge learning or profit.
His goal of the book is to explain how value-added models work, how they measure learning, how the take into account things like poverty level and other circumstances beyond the control of the school or the teachers, and other such factors. In his introduction he also promises not to be unreasonable about applying the results of these tests beyond where it makes sense. He certainly seems to be a smart guy; smart enough to know about errors and the problems with badly set up incentives – he uses the financial crisis as a model of how not to do it. I’m hopeful!
Here’s what I am interested in talking about today, which is how the “standardized” gets into standardized testing, because already at this point the mathematical modeling is pretty tricky (and involves lots of choices). There are many ways a test is ultimately standardized, assuming for simplicity that it’s a national test given at many grade levels yearly (pretend it’s an SAT that every grade takes):
- the test is normalized for being harder or easier than it was last year, for each grade’s test separately, and sometimes per question as well,
- the grading is normalized so that a student who learns exactly as much “as is expected” gets the same grade from year to year, and
- the grading is further normalized so that a student who gets 10 more points than expected in 3rd grade is doing as well as if she got 10 extra points in 4th grade.
One way of accomplishing all of the above would be to draw a histogram of raw results per year and per grade and normalize that distribution of raw scores by some standard mean and standard deviation, just as you would make a normal distribution standard, i.e. mean 0 and standard deviation 1. In fact, go ahead and demean it and divide by the standard deviation. That’s the first thing I’d do.
But if you actually do that, then you lose lots of the information you are actually trying to glean. Namely, how could you then conclude if students are doing better or worse than last year? I’m sure you’ve seen the recent news that SAT scores have fallen this year from last. I guess my question is, how can they tell? If we do something as simple as what I suggested, then the definition of doing as well “as is expected” is that you did “as well as the average person did”. But clearly this is not what the SAT people do, since they claim people aren’t doing as well as they used to. So how are they standardizing their test?
It isn’t really explained here or here, but there are clues. Namely, if you give 3rd and 4th graders some of the same questions on a given year, then you can infer how much better 4th graders do on those questions than 3rd graders do, and you can use that as a proxy for how to scale between grades (assuming that those questions represent the general questions well). Next, since you can’t repeat questions (at least questions that count towards the score) between years, because the stakes are too high and people would cheat, you can instead have ungraded sections that have repeated questions which give you a standard against which to compare between years. In fact the SAT does have ungraded sections, and so did the GREs as I recall, and my guess is this is why.
That brings up the question, do all standardized tests have ungraded sections? Is there some other clever way to get around this problem? Also in my mind, how well does standardization work, and what is a way to test it?
This week has been particularly confusing when it comes to the European debt crisis. It’s complicated enough to think about the various countries, with their various current debt problems, future debt problems, and austerity plans, not to mention how they typically interact at the political level versus how the average citizen is affected by it all. But this week we’ve seen weird and coordinated intervention by a bunch of central banks to address a so-called “liquidity crisis”.
What is this all about? Is it actually a credit crisis disguised as a liquidity crisis? Is it just another stealth way to bail out huge banks?
I’m going to take a stab at answering these questions, at the risk of talking out of my ass (and when has that ever stopped me?).
Finance is a big messy system, and it’s hard to know where to begin on the merry-go-round of confusion, but let’s start with European banks since they are the ones in need of funding.
European banks have lots of euros on hand, just as American banks have lots of dollars, because of the actual deposits they hold. However, European banks invest in American things (like businesses) that need them to come up with short term funding denominated in dollars. Similarly American banks invest in Europe, but that’s not really relevant to the discussion yet.
How do European banks get these short term (3 month) loans? Historically they do a large majority of it through money-markets: much of the money people have in banks is funneled to huge vats called money markets, and the fund managers of those vats are very very conservatively trying to make a bit of interest on them. In fact they were burned in the credit crisis, when they famously “broke the buck” on Lehman short-term loans.
Well, guess what, those same American money managers are avoiding European short-term loans right now, because they are super afraid of losing money on them. So that source of funding has dried up. Note that this is a credit problem: the money market managers do not trust the banks to be around in 3 months.
Another source of funding for the European banks’ American investments has been just to use their euros, exchange them to dollars (the currency market is very very large and liquid, especially on this particular exchange), then wait until the term of the short-term financing is over, and then convert the dollars back to euros. What actually happens, in fact, is that they borrow euros (at the going rate of 1%), do the exchange, then financing, and then get their money back in the future.
The guys who work at the European banks and who do this short-term financing aren’t allowed to take on the risk that the exchange rate is going to violently change between now and when the short-term term is over. Therefore they need to hedge the risk, which means they have to have a guarantee that the dollars they get out at the end of the term will be turned into a reasonable number of euros.
This kind of guarantee is called a currency swap, and the market for those is also very large and liquid, but has been less liquid recently because of the one-sidedness of this problem: European banks need short-term dollars but American banks don’t need euros at the same rate at the same maturity. So the end result is that the swaps are very very expensive for European banks.
Let’s put this another way, the way that seems strangest and most confusing: right now the European banks can borrow at 1% in euros but at 4% in dollars (for three month maturity), and more generally the demand for USD seems to be skyrocketing recently from all over the place. Does this mean there’s an arbitrage opportunity somewhere? The swaps market is at 3% so no obvious arbitrage. More likely it means that the markets are expecting the exchange rate to drastically change, or at least they are pricing in the risk of it changing violently in the very near future. (The strangest thing to me is why it hasn’t just changed the spot exchange rate as well.)
By the way, a pet peeve or two I have with people talking about arbitrage: firstly, many people use the term so loosely it means nothing at all, as when they take risk over time (exposing themselves to the possibility of an exchange rate change for example). But even here, I’m misusing the term, since in an arbitrage it’s literally supposed to be a way to make money risk-free, but the whole point of my post is that this is really all about counter-party risk! In other words, there’s no arbitrage opportunity to get into contracts with people where you’d make money except if they go bankrupt tomorrow, when there’s a good chance that will happen.
The bottomline is that although the ECB and the Fed and the other central banks have spun this as a coordinated effort to help out a liquidity squeezed but functional market, it doesn’t pass the smell test. What’s actually happening is that the shoddy accounting and investments of French banks and others is not being trusted by American money market managers who are wise to them.
One more thing: the collateral being asked of the European banks is purportedly of low standard, which is to say the ECB is allowing thing like Greek debt as collateral, which wouldn’t past muster with other institutions (or with U.S. money markets!). In that sense this can be seen as a stealth bailout, although I think not the first one in Europe under that definition. This isn’t going away until they figure out how to deal with the Greek debt problem.
It’s Saturday morning, which means it’s time to conduct a thoroughly absurd thought experiment just for the sake of argument. Today I want to consider the idea of a widespread household debt amnesty: everyone who owes money on their credit cards and payday loans and also perhaps mortgage will be forgiven their debt (although mortgages would have to be rewritten rather than forgiven). What would happen next?
I was discussing this very question, and David Graeber’s book (still not finished- it’s long!) with a friend of mine, specifically how Graeber cites ancient Sumerian civilization as having periodically enacted household debt amnesties to avoid the collapse of their cities (specifically to avoid the debtors from fleeing the cities to avoid their debt problems).
One thing that I realized in that conversation is that, whereas Graeber mentions that it was historically an amnesty for household debt only, so didn’t involve commercial debt between companies and merchants, what we’ve seen in this country in the past 3 years is something like the opposite of that concept. Our (large financial) companies have been granted special considerations while the people who had made the mistake of entering contracts with them have not. And the so-called mortgage modification process has not been sufficiently widespread yet to really consider it an example of this. There is an excellent article here which makes this point, although not in this context.
The objection my friend had to the idea of enacting such an amnesty was that it would be pouring good money after bad; he’s European so he cited the example of Greece, and how the more money Greece gets the more money they spend, so it’s an impossible situation.
Actually I think this is an appealing analogy to make, but it’s a false one. Greece has a large-scale system in place, and giving them money without changing that system clearly isn’t going to solve any long term problems- it just kicks the can down the road.
However, it’s really different with consumer debt (credit cards etc.). Namely, the “system” that a given consumer enters into is a simple relationship (contract) with the credit card company in question. If the debt is forgiven, then the credit card company doesn’t have any obligation to extend more credit to that person. And in many cases, it wouldn’t.
I think the consequences of a household debt amnesty would be something along these lines:
- People who were previously in debt would have some cash on hand and would be able to spend it on consumer stuff (that they can actually afford with no credit) instead of spending it all on minimum payments to old credit card debt
- Credit card companies, burned from their losses, wouldn’t give them new credit cards, or would change the payment arrangements to make sure they got their money back faster
- Since they have no credit, those people who essentially be living in a cash-dominated society. This may actually be a good thing, because it would force people to budget in real time.
- Eventually people could rebuild a credit score over time if they decided to try credit again
In other words, that’s really not so bad and would get money flowing through the system again, which might help with our current recession.
Of course not everyone would be happy about a household debt amnesty. In particular the people who aren’t debtors would feel pretty burned that they’ve been careful (or lucky) with their money and aren’t getting a free ride. And the credit card companies would have to eat a lot of loss. On the other hand they’re going to eat (and have eaten) a lot of loss already, and the slowness of this process is killing the economy.
Is there a way we could set it up to make this work? Even if we ignore the political obstacles?
By guest blogger rwitte
The social structures commonly talked about when discussing finance and economics include individuals, governments and corporations. However the most social structure is family, since it is the structure that results in the perpetuation of society. I was reminded of this by a recent link that mathbabe posted here. And since she invited me to write a guest post, it inspired me to mouth off about family.
What do you think of when someone uses the word family? I am guessing that you think of the so-called nuclear family consisting of a husband a wife and a variable number of children. For sure their are many variants including one-parent families and gay families, but the ideal is thus. It wasn’t always so. I am a fifty year old male of Ashkenazi Jewish descent. In that community my generation is the first to have prioritise the needs of small nuclear families. My grandmother’s idea of family was a much larger group, extending over several generations, and with a healthy side-order of cousins, aunts etc. My wife is Jaimaican, and her mother’s conception of family is similar to my grandmother’s (except the Jamaican version is even more matriarchal because so many of the fathers are absent).
I don’t know if you have ever seen a family home from a hundred and fifty or more years ago. You will be surprised at the size; there are more rooms for a family, at any level in the social scale. This is because the experiences of my wife and I are not unique, they are just a few generations later than those of the majority. Until relatively recently, as in most ‘primitive’ societies, the family is the extended family, and you will commonly find three or four generations living together.
I think the organisation of society into extended families was a great idea, and that the fragmentation into nuclear families sucks. But before I explain what’s so terrible about them I want to take a paragraph to explain why I think the change occured in the first place. Nuclear families are small and relatively mobile. As industrialization progressed and first transcontinental and then transglobal corporations formed, it suited their purpose to be able to move and resettle employees between different sites in their empires. At the other end of society the pull of the factories was encouraging many to move from rural areas to the big city. This also fragmented extended families; the units that moved were nuclear. As a process in the developed countries, it probably peake in the 1950s or 1960s, but it is still going on now in developing countries such as China.
The original myth of the nuclear family was one in which the male was the breadwinner and married women stayed home to provide full time childcare. This idea, obviously sexually discriminatory, is certainly a myth. It has never been the case that poor couples could support themselves on one person’s wages. Manual labour has never been that well payed. And since the rich typically had access to nannies, only a thin stratum of society has ever organized childcare this way.
Today, in an attempt to paper over the cracks in this story, a new myth has arisen. Namely, the ‘superwoman’ who has it all: career, children, and social life (it only take 28 hours a day eight days a week!). In actual fact this myth is probably even more dangerous than the previous one, because millions of women are now trying to live up to this impossible ideal. When they don’t overachieve these impossibly demanding targets, they feel guilty and inadequate. For some, serious mental health issues can ensue as the buckle under the pressure.
We often complain about the poor quality of life our society offers to our elders. They get stuck in some retirement home quitely out of sight where they can slowly die of boredom. The middle classes must pay for child-care and the exorbitant prices push them towards poverty. Meanwhile poor parents simply cannot afford adequate child care and poor children roam the streets; ‘latch-key kids’ with no adequate supervision between the end of school and the parents’ return from the work. Some of these children get really out-of-control; getting involved with drugs, crime, and street-gangs in some combination. I don’t want to get too carried away with arguments about ‘the youth of today’ because the rose-tinted vision painted nostagia never was, but still I hear the cries, something must be done.
I believe that we can overcome all these social problems by returning to a social organisation based on extended families. The grandparents can look after the children while the able-bodied parents go out to work. This leads to an interesting and fulfilling retirement, in which they can pass on the wisdom that they have accumulated over the years to a willing audience. The children get an educational, loving and supportive home-life which will help them to grow up honest and secure. And the fittest adults can commit 100% to the workforce increasing social productivity.
Of course it may be that the grandparents are still too young to retire and the great-grandparents take on the responsibility. Or maybe a cousin or Aunt who particularly enjoys childcare can set up a sort of family creche. Perhaps all the adults work and coordinate their days off in a rota so that who looks after the children depends on the day of the week. Nobody would be surprised to discover that while all families have much in common, every family is different. The role of the greater society is as to encourage and enable relatives to remain geographically close to each other. It would be up to each particular family to work out how to organise themselves for their own convenience (although there is some evidence to suggest that children brought up by maternal grandparents are more psychologically secure than those brought up by paternal grandparents).
Modern advances in information and communication technology may render a society with less geographical relocation of people possible. You don’t have to be in the same office to work with somebody (hell, you don’t even have to be in the same continent). Instead of workers having to move around the globe, they can stay with their parents and children while their information flows around the internet and other computing networks, more quickly, conveniently and cheaply.
I love Elizabeth Warren. I think she’s real, she cares about people, she’s smart and tough, and she’s incorruptible. The bizarre news, under these circumstances, is that she’s running for Congress. Is she too good to win in politics? Don’t you have to be a slime ball? Don’t you need to attract and sell out to megacorporations to succeed? We shall see. It could be really great.
And just in case you think I’m being too cynical, please read this. It’s an absolutely stunning, depressing insider’s view of the Republic Party from Mike Lofgren, who retired in June after 28 years as a Congressional staffer, having served 16 years as a professional staff member on the Republican side of both the House and Senate Budget Committees.
Last night I attended this Meetup on a cool package that Wes McKinney has been writing (in python and in cython, which I guess is like python but is as fast as c). That guys has been ridiculously prolific in his code, and we can all thank him for it, because pandas looks really useful.
To sum up what he’s done: he’s imported the concept of the R dataframe into python, with SQL query-like capabilities as well, and potentially with some map-reduce functionality, although he hasn’t tested it on huge data. He’s also in the process of adding “statsmodel” functionality to the dataframe context (he calls a dataframe a Series), with more to come soon he’s assured us.
So for example he demonstrated how quickly one could regress various stocks against each other, and if we had a column of dates and months (so actually hierarchical labels of the data), then you could use a “groupby” statement to regress within each month and year. Very cool!
He demonstrated all of this within his IPython Notebook, which seems to demonstrate lots of what I liked when I learned about Elastic-R (though not all, like the cloud computing part of Elastic-R is just awesome), namely the ability to basically send your python session to someone like a website url and to collaborate. Note, I just saw the demo I can’t speak from personal experience, but hopefully I will be able to soon! It’s a cool way to remotely use a powerful machine and not need to worry about your local setup.
One of the positive things about working at D.E. Shaw was the discipline shown in determining whether a model had a good chance of working before spending a bunch of time on it. I’ve noticed people could sometimes really use this kind of discipline, both in their data mining projects and in their normal lives (either personal lives or with their jobs).
Some of the relevant modeling questions were asked and quantified:
- How much data do you expect to be able to collect? Can you pool across countries? Is there proxy historical data?
- How much signal do you estimate could be in that data? (Do you even know what the signal is you’re looking for?)
- What is the probability that this will fail? (not good) That it will fail quickly? (good)
- How much time will it take to do the initial phase of the modeling? Subsequent phases?
- What is the scope of the model if it works? International? Daily? Monthly?
- How much money can you expect from a model like this if it works? (takes knowing how other models work)
- How much risk would a model like this impose?
- How similar is this model to other models we already have?
- What are the other models that you’re not doing if you do this one, and how do they compare in overall value?
Even if you can’t answer all of these questions, they’re certainly good to ask. Really we should be asking questions like these about lots of projects we take on in our lives, with smallish tweaks:
- What are the resources I need to do this? Am I really collecting all the resources I need? What are the resources that I can substitute for them?
- How good are my resources? Would better quality resources help this work? Do I even have a well-defined goal?
- What is the probability this will fail? That it will fail quickly?
- How long will I need to work on this before deciding whether it is working? (Here I’d say write down a date and stick to it. People tend to give themselves too much extra time doing stuff that doesn’t seem to work)
- What’s the best case scenario?
- How much am I going to learn from this?
- How much am I going to grow from doing this?
- What are the risks of doing this?
- Have I already done this?
- What am I not doing if I do this?
Next, if you want evidence that it sucks to be rich, or at least it sucks to be a child of rich people, then read this absolutely miserable article about how rich people control and manipulate their children. Note the entire discussion about “problem children” never discusses the possibility that you are actually a lousy, money-obsessed and withholding parent.
Next, how friggin’ cool is this? Makes me want to visit Mars personally.
And also, how cool is it that the World Bank is opening up its data?
Finally, good article here about Bernanke’s lack of understanding of reality.