Archive
Do we really want elite youth to get more elite?
I don’t know if you guys read this recent New York Times editorial entitled Even Gifted Students Can’t Keep Up: In Math and Science, the Best Fend for Themselves.
In it, they claim there’s some kind of crisis going on in this country for smart kids (defined as good test-takers). Mostly their evidence for this is that, among other countries, our super good test takers aren’t as prevalent as in other countries. Turns out we’re in the middle of the pack in terms of super scorers. From the article:
On the 2012 Program for International Student Assessment test, the most recent, 34 of 65 countries and school systems had a higher percentage of 15-year-olds scoring at the advanced levels in mathematics than the United States did.
Why is this a problem? As far as I can see they’ve come up with two reasons.
First, it’s “bad for American competitiveness,” whatever that means. Last time I checked we were still pretty dominant in various ways in terms of technology and science, and there are still plenty of very well-educated young people trying desperately to get visas to enter or stay in this country.
As an aside: it’s a super interesting question to think about how we, as a country, are increasingly ignorant about how our technology works, because so much technical knowledge has been off-shored. But that’s not the crisis these guys are addressing.
Second, it’s bad for the smart kids in this country, because “when the brightest students are not challenged academically, they lose steam and check out.”
I’ll pause in my summary of their article to make the following point. If that’s true, if bright kids who aren’t academically challenged at school start checking out more and more, then it makes just as much sense to me to see if there’s something they can check out towards.
In other words, what else is there for bright teenagers to do besides school? I’ll speak as a former bright teenager. When I lost interest in school, I got a lot of odd jobs in town cleaning houses and raking lawns, and then I used the money to buy lots of softcover books from a local bookstore. I don’t know why I didn’t just take them out of the library, where I also worked. It just didn’t seem as cool as owning my very own Brother Karamazov.
I learned a lot with my odd jobs in high school, which included being a part-time secretary, a barista, a math tutor, and working on a truck at the New England Produce Center. In fact I learned way more about how the world worked than I would have if I’d followed the advice of this editorial, which was to take lots more AP classes and then enter college when I was 14.
My theory is that, instead of obsessing over math scores in standardized tests, we concentrate on allowing our children to enrich their lives with adventures and experiences that they come up with and that are reasonably safe. So let’s start by encouraging widespread internships for younger kids, and not just minimum wage jobs at fast food joints. And not just for super test-takers either. Enrichment happens when kids learn about stuff that’s outside their usual rhythm and when there are no adults scripting their activities and telling them what to do or how many laps to swim.
Notice my emphasis on letting kids choose stuff. What drives me nuts just as much as the idea of further separating and isolating and venerating great test-takers, which as far as I’m concerned is the opposite way you should treat future successful people, is the idea that there should be such a well-defined funnel for children at all.
Yes, kids should all go to school and learn basic things. But the idea that, just because someone’s good at tests they should be treated as if they’re already running the Fed only increases the weird worshippy aspect of how our culture treats math nerds.
Plus, it’s a bizarre time to come up with this idea, considering how many online and live resources there are for nerd kids now compared to when I was a kid. If I’m a nerd teenager now, I can find plenty of ways to share nerdy questions and learn nerdy things online if I decide not to work in a coffee shop.
Finally, let me just take one last swipe at this idea from the perspective of “it’s meritocratic therefore it’s ok”. It’s just plain untrue that test-taking actually exposes talent. It’s well established that you can get better at these tests through practice, and that richer kids practice more. So the idea that we’re going to establish a level playing field and find minority kids to elevate this way is rubbish. If we do end up focusing more on the high end of test-takers, it will be completely dominated by the usual suspects.
In other words, this is a plan to make elite youth even more elite. And I don’t know about you, but my feeling is that’s not going to help our country overall.
Aunt Pythia’s advice, flannel version
Aunt Pythia aint gonna lie, she’s all about flannel this week. Right now she’s wearing flannel pajamas and snuggled underneath a flannel sheet. She’s looking forward to taking a warm bath and then getting into flannel lined pants and a flannel shirt, also lined.
[Aside: it’s all about the lining.]
It’s been a particularly good year in flannel fashion, just in case you’re wondering. And, the clothes horse that Aunt Pythia is, she’s been sampling the various wares.
Conclusion: LLBean knows from comfy warmth.
Now that we’ve addressed the theme of the day, it’s time for some serious advice distribution. And remember, as you enjoy today’s column:
please, think of something flannelly to ask Aunt Pythia at the bottom of the page!
By the way, if you don’t know what the hell Aunt Pythia is talking about, go here for past advice columns and here for an explanation of the name Pythia.
——
Dear Aunt Pythia,
I have two (much) younger brothers, aged 4 and 5. Can you recommend any books/youtube videos/games I can read/play/do with them to help them learn about and enjoy math?
Rapt in Flatland
Dear RiF,
I really think that age range is too young to explicitly learn math from a book. But it’s not too young to have discussions (it’s incredibly important to make this an interactive Socratic dialog, not a monologue from an adult!) about, say,
- the number line (where’s 0? What’s on the other side of 0?)
- why the earth spins
- what the solar system looks like (talking first without pictures or models)
- why the moon looks different on different days
If that all seems cool, you can move on to
- why there are seasons
- why daylight is shorter in the winter
- what really defines a “pattern”
In other words, super concrete things that require mind-expanding 3-dimensional visualizations and/or questioning of basic assumptions and definitions. You’d be surprised how many weeks these conversations can last with a 4- or 5-year old.
Also, please keep in mind that jigsaw puzzles, while not strictly logical exercises, are super awesome. And the Tower of Hanoi puzzle is super mathematical as well, the solution being a coded way to count in binary.
Good luck!
Aunt Pythia
p.s. Just gave my yearly contribution to wikipedia. Love that site.
——
Dear Aunt Pythia,
My girlfriend has a genuinely very complicated family life, and, though she definitely loves me, has much much less time to be with me than I would like. I do love her, but now I’m increasingly irritated at being alone so much even while I nominally have a girlfriend, and it’s affecting my feelings and even my desire for her. But this is absolutely not something we can discuss. She is doing her best and that’s it, and gets upset at any discussion, sometimes even hostile. It may end badly.
Suggestions?
Increasingly Irritated
Dear II,
Yes, I do have a suggestions. Discuss it with her. It’s the only way you’re going to get relief and get to a relationship that works for both of you.
I think it might help to ask her to make an appointment with you to discuss the arrangement between you two, and it might also help to imply that it’s very important to your future as a couple. Give her some time to prepare for this discussion but not an arbitrary amount of time.
To be super explicit, here are some phrases you might want to use. “I’d really like to sit down with you and discuss our relationship and its future. It would be great to have a discussion in the next two weeks. Here are some times that work for me, and where we have as much time as we need to finish the first part of the conversation, which I know will be difficult for both of us.”
That way you’re both respectfully demanding a discussion and giving her the emotional credit that it’s going to be tough on her.
A bit more advice: don’t conflate why it’s difficult with your request for a discussion. Then she’ll ignore the request and focus on how difficult it is for her.
Now, you didn’t ask this, and I don’t know much about your situation, but can we spend just a moment wondering what’s really going on here? I don’t want to sound negative, but whatever it is that’s going on, she’s not spending much time with you and is not even really willing to explain why. Those two things in combination make me want to advise you to just break up with her now, and forget the discussion.
In fact, in breaking up with her, you might get a discussion for free, if she really wants to keep you. And if she doesn’t, you should be the hell out of there anyway and with someone who’s making time to be with you.
Good luck!
Aunt Pythia
——
Dear Aunt Pythia,
As 50 approaches and work opportunities seem to be disappearing, do you really think it is possible to reinvent yourself and move into a different field?
I have applied my expertise in various quantitative and analytical fields to move from number cruncher (and forecasting) to more policy-based analysis roles. Is management the only answer? Every other role seems to be eaten by the progression of technology into various fields.
Grumpy Old Dude
Dear GOD,
My guess is that your salary and experience has made the lower-level analyst jobs both unappealing and out of reach for you. If you’re going to cost that much, the reasoning goes, you need to be doing more. Thus management.
Can I just make one argument in favor of management? If someone with your experience is in charge of telling people with very little or no experience how to do number crunching and analysis well, then things would be much more efficient. In an ideal world, that would be your role as manager – doling out sage advice to inexperienced analysts. So see if that makes sense and excites you.
But, to answer your question, it’s never too late to reinvent yourself. Some of my favorite people did so when they were 75.
Good luck!
Auntie P
——
Dear Aunt Pythia,
I’m a Libertarian. I find it pretty much impossible to have an intelligent discussion with liberals about anything even remotely related to politics. The same is not true with conservatives, even though I disagree with them just as much. Consequently I just don’t participate in such conversations. Unfortunately I both love a good debate and live in a very liberal state, so this has lead me to feel somewhat isolated and starved for good conversation.
I’ve talked to other libertarian minded people and they’ve had similar experiences. Conservatives can be infuriating in some of their beliefs, but at least you can reason with them on most issues. With liberals it’s impossible, unless you already pretty much agree and are just quibbling over details.
You’re about as liberal as people come. Do you have any advice on how one could have a civil discussion with a liberal on issues where you have relatively fundamental disagreements?
Lonely Libertarian
Dear LL,
Yes. Discuss what your commonalities are first, and move cautiously and carefully outwards from there. Suggestions here include: Too Big to Fail banks are not OK, Google and the NSA stealing our private data is not OK, and other currently shitty things about the world.
Next, before going to the inevitable argument about how to fix these problems, talk about exactly why these things are not OK. Then start talking about the concept of fixing them – what characteristics would a solution enjoy? What is not acceptable, and why?
That should take enough time to get through lunch.
Good luck! And don’t forget, liberals make great lovers! They’re really into massages.
Auntie P
——
Dear Aunt Pythia,
So, after all the turkey and pie and so forth, was the ratio of time spent agonising in the kitchen to time spent going ‘Yay, Thanksgiving’ a good one? And how would you determine a metric for Thanksgiving enjoyment?
(Asking in contemplation of what’s likely to happen at Christmas…)
Strange Oblivious Briton
Dear SOB,
I could totally tell you were British from your weird spelling of “agonizing!”.
For me, Thanksgiving was a success if 1) the turkey didn’t make anyone sick and 2) nobody threw plates frisbee-style at each others’ heads in the heat of an argument.
A particularly good sign is when people are still willing to hug each other when they finally leave the building, assuming anyone’s actually still there at all and hasn’t stormed off in a huff.
Haha just kidding, kind of. The truth is these big meals are just so so difficult, I honestly think you should keep standards as low as possible and be pleasantly surprised at every moment of happiness.
And whatever you do, stop agonizing in the kitchen, nobody will appreciate your effort anyway!! Just throw a bird into the oven and come back 6 hours later, seriously.
Merry Christmas,
Aunt Pythia
——
Please submit your well-specified, fun-loving, cleverly-abbreviated question to Aunt Pythia!
International trade agreements and big money bullying #OWS
The New York Times just put out an amazing and outrageous story, entitled Tobacco Industry Tactics Limit Poorer Nations’ Smoking Laws and written by Sabrina Tavernise.
In it she describes the bullying tactics of tobacco companies to small countries over their internal health regulations aiming to protect their citizenry from cancer. From the article:
In Africa, at least four countries — Namibia, Gabon, Togo and Uganda — have received warnings from the tobacco industry that their laws run afoul of international treaties, said Patricia Lambert, director of the international legal consortium at the Campaign for Tobacco Free Kids.
“They’re trying to intimidate everybody,” said Jonathan Liberman, director of the McCabe Center for Law and Cancer in Australia, which gives legal support to countries that have been challenged by tobacco companies. In Namibia, the tobacco industry has said that requiring large warning labels on cigarette packages violates its intellectual property rights and could fuel counterfeiting.
A few comments about this outrage, only tangentially mathematical:
- This happens because, in order to protect companies from being taken over by foreign nations, and in the name of free trade, it’s now possible for companies to sue countries directly. That’s what the tobacco industry is doing to small countries without the means to fight back.
- This is exactly the kind of thing that some people like Yves Smith have warned the TPP is going to do with financial regulation and other stuff. So imagine large companies or industries suing the United States or other nations for regulation that would “harm trade” or “violate their intellectual property rights.”
- In fact, it’s not going too far to say that the proliferation of these kinds of treaties are a serious threat to national sovereignty. Yves makes this case here.
- Think about it this way. It’s kind of a supernational Citizen’s United, in that only companies and industries that have enough lawyer power can get their way. And many companies easily have more resources than many countries. Think about Facebook and Google and their lobbying efforts on behalf of data privacy laws in Europe already. Now think how that battle might look when it’s against Namibia.
- Already, from the article, we saw that Uruguay was only able to fight back against Philip Morris because Bloomberg’s foundation helped them with the legal battle. And I’m glad Bloomberg helped, but if you count up all the money that will go to bullying and compare that to all the money available to help out the bullied, you quickly come to a sad conclusion.
- Once again, we have to remember that the TPP is being negotiated in secret, among a bunch of nations many of whom claim to be democratic.
Occupy the SEC shows us all how to occupy
Today is a remarkably cold day in New York but that’s done nothing to dampen the spirits of the members and supporters of Occupy the SEC, which had a real influence on the final Volcker Rule, as reported by the Wall Street Journal and The Washington Post, among others.
I’m not saying the Volcker Rule is perfect – in fact Occupy the SEC gave it a C- grade overall. And it was maddeningly delayed. As I’ve mentioned before, Occupy the SEC sued the regulators for the delay, or tried to, but the judge found that they had no standing.
Even so, there’s a clear victory here for Occupy the SEC. The final rule, or at least its preamble, references Occupy the SEC’s public commenting letter 284 times. The rule could have been way worse, and probably would have been without OSEC’s contribution.
Just to get some idea of the other voices in this debate – i.e. the lobbyists – take a look at this graphic on access to the rule-writers by parties interested in influencing the final rule:
I remember those guys in the early days of Occupy, meeting in the cold, drafty, and noisy chambers of 60 Wall Street after work and sitting in a circle going through the proposed Volcker Rule line by line. Nothing could have been more workmanline or hopeful. It was a great example of an Occupy group that was all work and no talk. People who wanted to talk just left them alone. I talked to them briefly about some risk model details for the rules because of my background in portfolio risk.
I even remember talking to them about the chances anybody would even read their letter, never mind take it seriously. They knew the rules, though, and the rules were that the regulators would have to read all the letters, although how seriously they took them was a different matter. They were doing it because they knew it was one of the best shots at having their voices heard. It was an incredible commitment to that ideal.
My conclusions:
- Those guys are awesome.
- Public commenting is a critical tool.
- This wouldn’t have worked without a receptive set of regulators.
- This is inspiring and will hopefully help my Occupy group, Alt Banking, plan our next project.
- I particularly like the idea of report cards, which Occupy has created for regulation.
Computer, do I really want to get married?
There’s a new breed of models out there nowadays that reads your face for subtle expressions of emotions, possibly stuff that normal humans cannot pick up on. You can read more about it here, but suffice it to say it’s a perfect target for computers – something that is free information, that can be trained over many many examples, and then deployed everywhere and anywhere, even without our knowledge since surveillance cameras are so ubiquitous.
Plus, there are new studies that show that, whether you’re aware of it or not, a certain “gut feeling”, which researchers can get at by asking a few questions, will expose whether your marriage is likely to work out.
Let’s put these two together. I don’t think it’s too much of a stretch to imagine that surveillance cameras strategically placed at an altar can now make predictions on the length and strength of a marriage.
Oh goodie!
I guess it brings up the following question: is there some information we are better off not knowing? I don’t think knowing my marriage is likely to be in trouble would help me keep the faith. And every marriage needs a good dose of faith.
I heard a radio show about Huntington’s disease. There’s no cure for it, but there is a simple genetic test to see if you’ve got it, and it usually starts in adulthood so there’s plenty of time for adults to see their parents degenerate and start to worry about themselves.
But here’s the thing, only 5% of people who have a 50% chance of having Huntington’s actually take that test. For them the value of not knowing that information is larger than knowing. Of course knowing you don’t have it is better still, but until that happens the ambiguity is preferable.
Maybe what’s critical is that there’s no cure. I mean, if there was therapy that would help Huntington’s disease sufferers delay it or ameliorate it, I think we’d see far more people taking that genetic marker test.
And similarly, if there were ways to save a marriage that is at risk, we might want to know on the altar what the prognosis is. Right?
I still don’t know. Somehow, when things get that personal and intimate, I’d rather be left alone, even if an algorithm could help me “optimize my love life”. But maybe that’s just me being old-fashioned, and maybe in 100 years people will treat their computers like love oracles.
Predictive risk models for prisoners with mental disorders
My friend Jordan Ellenberg sent me an article yesterday entitled Coin-flip judgement of psychopathic prisoners’ risk.
It was written by Seena Fazel, a researcher at the department of psychiatry at Oxford, and it concerns his research into the currently used predictive risk models for violence, repeat offense, and the like, which are supposedly tailored to people who have mental disorders like psychopathy.
Turns out there are a lot of these models, and they’re in use today in a bunch of countries. I did not know that. And they’re not just being used as extra, “good to know” information, but rather as a tool to assess important decisions for the prisoner. From the article:
Many US states use such tools to assess sexual offending risk and to help decide whether to exercise their powers to detain sexual offenders indefinitely after a prison term ends.
In England and Wales, these tools are part of the admission criteria for centres that treat people with dangerous and severe personality disorders. Outside North America, Europe and Australasia, similar approaches are increasingly popular, particularly in clinical settings, and there has been a steady growth of research from middle-income countries, such as China, documenting their use.
Also turns out, according to a meta-analysis done by Fazel, that these models don’t work very well, especially for the highest risk most violent population. And what’s super troubling is, as Fazel says, “In practice, the high false-positive rate probably means that some offenders spend longer in prison and secure hospital than their true risk would suggest.”
Talk about creepy.
This seems to be yet another example of a mathematical obfuscation and intimidation that gives people a false sense of having a good tool at hand. From the article:
Of course, sensible clinicians and judges take into account factors other than the findings of these instruments, but their misuse does complicate the picture. Some have argued that the veneer of scientific respectability surrounding such methods may lead to over-reliance on their findings, and that their complexity is difficult for the courts. Beyond concerns about public protection, liberty and costs of extended detention, there are worries that associated training and administration may divert resources from treatment.
The solution? Get people to acknowledge that the tools suck, and have a more transparent method of evaluating them. In this case, according to Fazel, it’s the researchers who are over-estimating the power of their models. But especially where it involves incarceration and the law, we have to maintain an adherence to a behavior-based methodology. It doesn’t make sense to put people in jail an extra 10 years because a crappy model said so.
This is a case, in my opinion, for an open model with a closed black box data set. The data itself is extremely sensitive and protected, but the model itself should be scrutinized.
What’s the best narrative for revolution?
Yesterday at our weekly Alt Banking meeting we had an extraordinary speaker, Merlyna Lim, come talk to us about social media and grass roots organizing.
Her story was interesting and nuanced; I won’t get everything down here. But there were quite a few sound byte takeaways I can express.
- The organizing which culminated in the Arab Spring started way before Facebook or Twitter came to the region.
- To a large extent social media has replaced chatting in the cafe, which we don’t do anymore.
- But that’s actually a good thing, since many regimes are so oppressive they won’t let large groups of people hold regular meetings (and large can mean 5 or more).
- Whereas social media is pretty good at energizing people to “get rid of their enemy” at a given critical moment, and mobilize on the street, it’s not that great at nuanced discussions for how to build something permanent and lasting afterwards.
One other thing I wanted to mention was Merlyna’s work on Mohamed Bouazizi, the Tunisian street vendor who self-immolated after getting into a dispute with a police officer.
The original story that got people mobilized in Tunis and out on the street, was that the police officer was a woman, that she slapped him, and that he was a college educated street vendor. It turns out these were white lies – he never finished high school, the police officer may have been a man, and there was probably no slap – but they built a narrative that people really loved. Merlyna wrote a paper about this available here if you want to know more.
That brings us to the question of why this particular framing was so appealing. Merlyna put it this way: plenty of other people had self-immolated under similar circumstances in Tunisia in the past 6 months alone. But they didn’t start a revolution because they were just very poor and didn’t have this story with extra (made-up) humiliating details. Killing yourself because you are frustrated at not having enough to eat just isn’t as compelling.
It reminds me of this Bloomberg View piece I’ve been chewing on for a couple of week, written by Peter Turchin and entitled Blame Rich, Overeducated Elites as Our Society Frays. He studies conditions for revolution as well, and claims that having a large unemployed but highly educated population – the “lawyer glut” we’re seeing today – is asking for trouble. From his article:
Elite overproduction generally leads to more intra-elite competition that gradually undermines the spirit of cooperation, which is followed by ideological polarization and fragmentation of the political class. This happens because the more contenders there are, the more of them end up on the losing side. A large class of disgruntled elite-wannabes, often well-educated and highly capable, has been denied access to elite positions.
Food for thought. Does one have more sympathy for people whose foodstamps have been recently cut or for someone who got a law degree and can’t find a job? Or is the real outrage when both happen (or at least are said to happen)? Personally, and this is maybe because I’ve been reading Jonathan Kozol’s Savage Inequalities, I’m not as worried about the lawyer.
Aunt Pythia’s advice: sex at the end
You guys know Aunt Pythia loves you. And Aunt Pythia feels the love from you readers as well, especially in person (some of you are reticent to add comments online, for whatever reason).
So don’t take it the wrong way when I say this: you guys are nerds. I have like a 5-to-1 ratio of math-related versus sex-related questions, and today I’m effectively withholding the sex until the end as a hook to keep you guys.
Don’t get me wrong, I love nerd questions. Happy to answer them. But people! Let’s spice this up! And if you can’t go all the way to sex at least come up with something about breastfeeding in public or thereabouts. As you know, Aunt Pythia doesn’t make up questions – that would be beneath her – but she has no problem with prompts.
In other words, as you enjoy today’s column:
please, think of something sexy to ask Aunt Pythia at the bottom of the page!
By the way, if you don’t know what the hell Aunt Pythia is talking about, go here for past advice columns and here for an explanation of the name Pythia.
——
Aunt Pythia,
What’s the deal with employers being dishonest in their job descriptions, and the general acceptance of this sort of unethical behavior? I work in a somewhat prestigious buy-side shop where I was told I’d be in a front-office quant research position. After I arrive, I find out that my responsibilities are really more like that of a middle-office tech position. Instead of doing research on market inefficiencies, I’m relegated to automating an endless number of reports. My employer knew what the job would entail before I joined and yet portrayed it to be something it’s not. Worst of all, it seems like 80% of the people I consult with say (expressly or implicitly) that I should be glad I got my foot in the door and that this stuff is very common, so it’s nothing to fret about. WTF’s wrong with people?
Perennial Employee
Dear PE,
For whatever reason, which I certainly don’t relate to, there are some people that still desperately want to work in finance as front-office quants. They want it so badly, in fact, that they’re willing to pretend to be doing that while they actually do other stuff. You seem to not be one of those people. Awesome.
My suggestion to you is to get another job, simple as that. You’re not going to change their mind about what your job should be, since they’re clearly perfectly comfortable with lying to people. I mean, once you’ve got another job lined up, there’s no harm in telling them you’re leaving unless you get moved to the position you were promised, but please don’t hold your breath for that to actually happen.
One last thing: look outside finance! There are plenty of other ways to be a nerd.
Aunt Pythia
——
Dear Aunt Pythia,
How does Mathbabe break down a data problem into manageable steps? I’m a mathematician who has tried a few data mining problems on the side for fun, and I get totally overwhelmed whenever I try to start. If I were solving a math problem, I’d read relevant papers to see what is known and get ideas for techniques, I’d break down my desired result into lemmas and work on them one by one, and I’d have a plan in mind throughout (it might change, of course, but I’d always know why I was doing what I was doing).
But if I’m trying to, say, classify a bunch of labeled feature vectors, I’m at a loss. I experiment and play around with the data, but I feel so random about everything. How do I choose how many hidden units to have in a neural net? How do I choose K in K-nearest neighbor classification? And so on. Some stuff works better than other stuff, but I don’t know how to be systematic. I end up getting discouraged, which is too bad because data problems are awesome and I want to master them.
Any tips for this mathematician on how to solve problems whose solutions aren’t proof-based?
Proof Machine
Dear PM,
Great question! And I’m glad you’re asking that. It’s a sign that you want to do things right, and know why you’ve made decisions. I want you to cultivate that desire.
First, (after separating my out-of-sample data from my in-sample data) I spend a lot of time with smallish samples getting the feel of things through “exploratory data analysis.” This helps make sure the data is clean, gives me the overall distribution and feel for the various data sources, and gives me some idea of the kind of relationships I might expect between the inputs and possibly the target, if there’s a well-defined target.
You’d be surprised how much you learn by doing that.
Next, how do you even choose which algorithm to use, never mind how exactly to tune the hyperparameters of a given algorithm? The answer is that it’s a craft, and over time you gain intuition, but at first you just don’t know and you experiment. Put the science in data science. Try a bunch of different ones and see which works better, and hypothesize on why, and try to test that hypothesis.
Here’s another possibility. Start with synthetic data that is “perfectly set up” for a given algorithm – figure out what that means – and then pretend you don’t know that, and see whether the above testing procedure would give you the correct result. Now add noise to that perfect data set, and see how quickly (i.e. with how much noise) your perfect solution doesn’t seem optimal anymore. That gives you an overall way of thinking about optimizing algorithms and hyperparameters. It’s hard, even with linear regression.
Oh, and buy my book. It should hopefully help.
Good luck!
Aunt Pythia
p.s. when I worked in math, I didn’t break things down into lemmas first. I first tried to answer the question, why is this true? (maybe by starting with small examples) and then only later, in order to explain it on paper, would I break things down into lemmas.
——
Dear Aunt Pythia,
I have a tenure-track job in a “hard-core STEM field”; I’m also a very young looking woman. I have a serious and rewarding research program, I really enjoy teaching at the board, and I hear that I give great seminars.
Yet recently, for the first time, I have been overcome with extreme, physiological, panic when I stand at the front of a room to give a seminar. This is not because I’m worried about the material; I’m not. This is also not stage fright; I have iron nerves about performing.
It is a feeling of panic brought on by watching the room fill up with men, with maybe only 1 or 2 very junior women. I start thinking “what happened to all the other young women who, like me, loved mathematics? At what point were they all removed from the community? When will too much get to be too much for me too?”
This started happening about a year ago and it’s only getting worse. I’m not expecting to change all the weird experiences of being a young woman in my field; I just want to figure out how to deal with my own thoughts as I stand in front of my audience.
Feeling like a fox in a room full of hunting dogs
Dear fox,
This is going to sound trite, but here goes: you are not a statistic, you are an individual person. And although you are a woman person, that doesn’t mean you have to do stuff that other women have done. If things are working for you on a minute-to-minute basis, then that means you can be happy and proud of having set up your life to be fulfilled.
Nobody is asking you to explain why other people do the things they do. We can barely explain why we do the things we do – and then half the time the understanding only comes years later. Just focus on who you are, who you want to be and how you want to spend your time.
I’d also like to mention that, as a woman who left math, I also loved teaching and I loved giving seminars – that was the good stuff! For that matter there were lots of great things about being a professor. And I didn’t leave because I was a woman and felt like it was time to leave – nor did I not leave because I wanted to prove a point about women not leaving. I left because, in my individual life and with my individual goals, it was what I wanted.
So I guess I’m suggesting that you be a bit more self-centered and somewhat less identified with women, at least at those moments, if that is possible and if that helps. If that doesn’t help, consider going to a cognitive therapist who specializes in dealing with panic attacks. Good luck!
Aunt Pythia
——
Dear Aunt Pythia,
I feel an eerie compulsion to answer this email. I love the broken grammar and all. What should I do?
Hello,
How are you doing today? My name is Colvin Hostetter. I came across your e-mail under the Graduate Students portal while surfing online for tutorial for my daughter, Debra is a 18 years old girl. She is ready to learn. I would like the lessons to be at your location. Kindly let me know your policy with regard to the fees, cancellations, location and make-up lessons.
Also, get back to me with your area of SPECIALIZATION and any necessary information you think that might help.
The lessons can start by last week of November. Mind you, any break during Thanksgiving and Christmas would be observed respectively.
Looking forward reading from you.
My best regard,
Mr Colvin.
Professor Has Ignored Silly Ignoble New Game
Dear PHISING,
Really? PHISING? I think you really are a bit kinky in the grammar and spelling rules department.
So this must be a spam email, since it’s talking about an 18-year-old girl who is “ready to learn.” It sounds like soft porn. And it doesn’t describe what she needs to learn – math? physics? German? I’d be not at all surprised to hear someone describe the actual financial bamboozling mechanism that would transpire if you did answer this, although a quick Google search doesn’t uncover it.
My suggestion is to mark this, and any other similar emails, as “spam” so that Google will do the work for us in the future and delete this bullshit.
AP
——
Dear Auntie P,
My wife and I have not used any birth control other than rhythm and/or withdrawal for more than 16 years now (~mid late twenties to early mid forties.) We have not had any unwanted pregnancies through this. We did have one successfully planned pregnancy that corresponded exactly to the month she charted to pinpoint ovulation.
So, are we lucky outliers or is this a much more successful strategy than we were both led to believe in high school sex ed?
Any suggestions for here on out?
Thanks,
Lucky in love
Dear Lucky,
The reason that rhythm might not work is if women have irregular ovulations. If your wife doesn’t, though, then cool (although she may experience that as she approaches menopause).
The reason withdrawal doesn’t work is because men often forget the “withdrawal” part of the plan. I mean, it’s certainly possible to get pregnant with the pre-cum (just ask Alice) but super unlikely.
In other words, you are a special, special man with a very excellent memory.
Here on out: don’t forget to remember the plan! And be aware of irregular cycles!
Aunt Pythia
——
Please submit your well-specified, fun-loving, cleverly-abbreviated question to Aunt Pythia!
PDF Liberation Hackathon: January 17-19
This is a guest post by Marc Joffe, the principal consultant at Public Sector Credit Solutions, an organization that provides data and analysis related to sovereign and municipal securities. Previously, Joffe was a Senior Director at Moody’s Analytics.
As Cathy has argued, open source models can bring much needed transparency to scientific research, finance, education and other fields plagued by biased, self-serving analytics. Models often need large volumes of data, and if the model is to be run on an ongoing basis, regular data updates are required.
Unfortunately, many data sets are not ready to be loaded into your analytical tool of choice; they arrive in an unstructured form and must be organized into a consistent set of rows and columns. This cleaning process can be quite costly. Since open source modeling efforts are usually low dollar operations, the costs of data cleaning may prove to be prohibitive. Hence no open model – distortion and bias continue their reign.
Much data comes to us in the form of PDFs. Say, for example, you want to model student loan securitizations. You will be confronted with a large number of PDF servicing reports that look like this. A corporation or well funded research institution can purchase an expensive, enterprise-level ETL (Extract-Transform-Load) tool to migrate data from the PDFs into a database. But this is not much help to insurgent modelers who want to produce open source work.
Data journalists face a similar challenge. They often need to extract bulk data from PDFs to support their reporting. Examples include IRS Form 990s filed by non-profits and budgets issued by governments at all levels.
The data journalism community has responded to this challenge by developing software to harvest usable information from PDFs. Examples include Tabula, a tool written by Knight-Mozilla OpenNews Fellow Manuel Aristarán, extracts data from PDF tables in a form that can be readily imported to a spreadsheet – if the PDF was “printed” from a computer application. Introduced earlier this year, Tabula continues to evolve thanks to the volunteer efforts of Manuel, with help from OpenNews Fellow Mike Tigas and New York Times interactive developer Jeremy Merrill. Meanwhile, DocHive, a tool whose continuing development is being funded by a Knight Foundation grant, addresses PDFs that were created by scanning paper documents. DocHive is a project of Raleigh Public Record and is led by Charles and Edward Duncan.
These open source tools join a number of commercial offerings such as Able2Extract and ABBYY Fine Reader that extract data from PDFs. A more comprehensive list of open source and commercial resources is available here.
Unfortunately, the free and low cost tools available to modelers, data journalists and transparency advocates have limitations that hinder their ability to handle large scale tasks. If, like me, you want to submit hundreds of PDFs to a software tool, press “Go” and see large volumes of cleanly formatted data, you are out of luck.
It is for this reason that I am working with The Sunlight Foundation and other sponsors to stage the PDF Liberation Hackathon from January 17-19, 2014. We’ll have hack sites at Sunlight’s Washington DC office and at RallyPad in San Francisco. Developers can also join remotely because we will publish a number of clearly specified PDF extraction challenges before the hackathon.
Participants can work on one of the pre-specified challenges or choose their own PDF extraction projects. Ideally, hackathon teams will use (and hopefully improve upon) open source tools to meet the hacking challenges, but they will also be allowed to embed commercial tools into their projects as long as their licensing cost is less than $1000 and an unlimited trial is available.
Prizes of up to $500 will be awarded to winning entries. To receive a prize, a team must publish their source code on a GitHub public repository. To join the hackathon in DC or remotely, please sign up at Eventbrite; to hack with us in SF, please sign up via this Meetup. Please also complete our Google Form survey. Also, if anyone reading this is associated with an organization in New York or Chicago that would like to organize an additional hack space, please contact me.
The PDF Liberation Hackathon is going to be a great opportunity to advance the state of the art when it comes to harvesting data from public documents. I hope you can join us.
Algorithmic Accountability Reporting: On the Investigation of Black Boxes
Tonight I’m going to be on a panel over at Columbia’s Journalism School called Algorithmic Accountability Reporting: On the Investigation of Black Boxes. It’s being organized by Nick Diakopoulos, Tow Fellow and previous guest blogger on mathbabe. You can sign up to come here and it will also be livestreamed.
The other panelists are Scott Klein from ProPublica and Clifford Stein from Columbia. I’m super excited to meet them.
Unlike some panel discussions I’ve been on, where the panelists talk about some topic they choose for a few minutes each and then there are questions, this panel will be centered around a draft of a paper coming from the Tow Center at Columbia. First Nick will present the paper and then the panelists will respond to it. Then there will be Q&A.
I wish I could share it with you but it doesn’t seem publicly available yet. Suffice it to say it has many elements in common with Nick’s guest post on raging against the algorithms, and its overall goal is to understand how investigative journalism should handle a world filled with black box algorithms.
Super interesting stuff, and I’m looking forward to tonight, even if it means I’ll miss the New Day New York rally in Foley Square tonight.
The cost savings of food stamps cuts versus the cost increases of diabetes care
As many of you are aware, food stamps were recently cut in this country. This has had a brutal effect on people and families and on neighborhood food pantries, which are being swamped with new customers and increased need among their existing customers.
One thing that I come away with when I read articles describing this problem is how often they detail individuals who have been diagnosed with diabetes but can no longer afford to pay for appropriate food for their condition.
As a person with a family history of diabetes, and someone who has been actively avoiding sugars and carbs to control my blood sugar for the past couple of years, I have a tremendous amount of sympathy for these struggling people.
Let me put it another way. Eating well in this country is expensive, and I’ve had to spend real money on food here in New York City to avoid sugary and fast carb-laden food. I don’t think I could have done that on a skimpy food budget. It’s especially hard to imagine budgeting healthy food on a withering food stamp budget.
Because here’s the thing, and it’s not a secret: shitty food is cheap. If I need to buy lots of food (read: calories) for a small amount of money, I can do it easily, but it will be hell for my blood sugar control. I’m guessing I’d be a full-blown diabetic by now if I were poor and on food stamps.
And that brings me to my nerd question of the morning. How much money are we really saving by decreasing the food stamp allowance in this country, if we consider how many more people will be diagnosed diabetic as a result of the decreased quality of their diet? And how many people’s diabetes will get worse, and how much will that cost?
It’s not over, either: apparently more cuts are coming over the next 10 years (maybe by $4 billion, maybe by $40 billion). And although diabetes care costs have gone up 40% in the last 5 years ($245 billion in 2012 from $174 billion in 2007), that doesn’t mean they won’t go up way more in the next 10.
I’m not an expert on how this all works, but the scale is right – we’re talking billions of dollars nationally, so not small potatoes, and of course we’re also talking about people’s quality of life. Never mind in a moral context – I’m definitely of the mind that people should be able to eat – I’m wondering if the food stamp cuts make sense in a dollars and cents context.
Please tell me if you know of an analysis in this direction.
What are Walmart’s wage incentives?
Suppose you have a employer – think Walmart – that has a ubiquitous presence nationally – think the United States. Is there ever a point, as that employer grows in size and employs more and more of the nation, that its agenda becomes aligned with the national agenda of prosperity and well-being?
That’s the interesting assumption behind a recent Guardian article called Walmart and Downton Abbey: rampant inequality and detachment from reality written by Sadhbh Walshe. From the article:
The best thing the top brass at Walmart could do to preserve their own privileged status would be to raise wages for their workers. A recent study by the progressive thinktank Demos illustrated that the company could afford to pay its workers an additional $5.83 an hour (pdf), enough to bring their wages just above the poverty level, simply by ending the company’s share-buyback program. This way prices could stay as they are but sales would increase as more workers would have more money to spend.
This comes up a lot in my Occupy group as well – the idea that raising wages would be good for low-wage companies like McDonalds and Walmart. The question I have is, is it true? (Aside to readers: if you’re aware of a paper that does this analysis, please tell me!) My guess is no.
Let’s put it this way. If I’m a small company then it’s pretty clear I don’t want to raise wages if I don’t have to. The lower the wages for my workers are, the more I get to keep or spend on other things. But as I grow in size, it might actually make sense, depending on context. If I employ 50% of the population, which is indeed an enormous number of people, then how much I pay them goes straight to the bottomline of how much they spend at my store. But again, it all depends on context.
And there’s and important bit of context going on here, which was beautifully explained recently by Bloomberg columnist Barry Ritholtz in his column entitled How McDonald’s and Wal-Mart Became Welfare Queens. From that article:
Wal-Mart, the nation’s largest private sector employer, is also the biggest consumer of taxpayer supported aid. According to Florida Congressman Alan Grayson, in many states, Wal-Mart employees are the largest group of Medicaid recipients. They are also the single biggest group of food stamp recipients. Wal-mart’s “associates” are paid so little, according to Grayson, that they receive $1,000 on average in public assistance. These amount to massive taxpayer subsidies for private companies.
My point is this. Given their welfare queen status, the agenda of Walmart and other huge low-wage employers is not actually aligned with the nation’s. As long as it can lower wages below poverty level, leaving Uncle Sam to make up some of the difference, and either pocket that difference or distribute them to their shareholders, they’ll continue to do so. The incentives are wrong.
And instead of appealing to their greed and telling them it’s in their best interest to raise wages, which I don’t believe is true (but I’d love to be wrong!), we need to raise political awareness about how the system actually works. And my guess is we should either raise the minimum wage – and then tie it to inflation – or establish a basic income guarantee.
Protest JP Morgan at noon on Wednesday #OccupyJPM #OWS
I’m looking forward to protesting in front of JP Morgan with my #OWS Alt Banking group this Wednesday at noon. The exact location is 270 Park Avenue, near 48th Street.
It’s part of a “Week of Action” being put together by a broad coalition of activist and labor groups here in New York. The overall theme of the week is to try to communicate to New Yorkers, in this time of transition from Bloomberg to de Blasio, that we can effect positive change in our city. The theme of the day on Wednesday, at least for us, is to “be in the know,” which makes it a bit more positive than other protests we’ve been part of.
I think this makes sense. There’s so much widespread distrust and hatred of the big banks at this point that I feel like Occupy’s role has gone from provoking people to be outraged to provoking people to be hopeful. Hopeful about the fact that things could be a whole lot better than this, if we work together.
Anyhoo, we spent yesterday planning the action, and made some signs. Here’s one based on an idea we borrowed from Alexis Goldstein from her recent twitter war with JPMorgan:
and here’s a sign we’ll hold up while playing a “rigged game” with props:
I also made a sign that referenced the London Whale and the risk model, but someone said we might need to give people a copy of our recent book, Occupy Finance, just to understand that sign. Sigh.
The facebook page is here, please share it with people who may be able to join us Wednesday!
Aunt Pythia’s advice: the long and boring edition
If you’re feeling anything like Aunt Pythia is feeling, you don’t want to even look at any food that has been peeled, baked, poured into a pie crust, mashed with butter, or stuffed into a turkey. It’s chopped cucumbers and raw apples from here on out, with plentiful brisk walks in the sunshine. Yes or no?
And also, is it just me, or has it been approximately 40 years since Aunt Pythia’s last column? Or is that just measured in “dishes done” years?
Before Aunt Pythia gets down to the advice part of the column, which is particularly long and boring and for which she apologizes, she wants to draw attention to the Black Friday protests that many of her Occupy friends took part in yesterday in Secaucus, New Jersey at Walmart.
It was a national day of Walmart protests, but here in Secaucus we had a large Occupy presence – note my friend Marni, who is holding up a deceased Walmart employee. More pictures are available here.
Now down to business. As you enjoy today’s column (or as you nod your way through it, as the case may be),
please, think of something to ask Aunt Pythia at the bottom of the page!
By the way, if you don’t know what the hell Aunt Pythia is talking about, go here for past advice columns and here for an explanation of the name Pythia.
——
Dear Aunt Pythia,
Some cosmological theories talk about the universe as being a mathematical space, rather than a pile of floating rocks and other real stuff. I always thought that mathematics consisted of rules for writing squiggles on bits of paper that sometimes produced sets of squiggles that corresponded with the real out-there stuff.
Am I a tiny part of the solution to a humungous equation? I’m happy with being made of fundamental particles but this is something else. What’s your take? Are there any practical consequences?
Perplexed
Dear Perplexed,
I’m no philosopher, but as a mathematician I’m here to tell you that mathematics doesn’t describe the universe. It’s at most used as a tool to understand certain parts of the universe, but only at the level of an approximation.
So for example, there’s no such thing as a circle in reality. It’s an idealized shape we use in mathematics that comes in super handy for various reasons, but because actual matter is made up of stuff, there’s never going to be a true circle, except in our brains. You can extend that concept of approximation to other mathematical models of the universe as well, at least as far as I understand it (Peter, please correct me if I’m wrong here!).
As far as each of us being a tiny part of a solution to a humungous equation, it all depends on how you look at it. I’m sure I can set up an equation that would dictate how many children my parents had, and then by construction I’d be in some sense a part of the solution to that. If you’re thinking more metaphysical than that, I can’t help you, and I doubt it would be more meaningful than that, although it might be wrapped up in fancier wrappers.
Aunt Pythia
——
Dear Aunt Pythia,
I am mulling reinventing myself (again). I used to be a lawyer, but it was entirely too demanding and inhumane to manage that and a family life. So I downshifted. Now I use the law degree writing for a legal publication. It’s a 40 hour work week, I have awesome benefits, I am able to take my kids (including the special needs one) to all their crap.
There are only two problems with this nirvana: I don’t make a lot and I’m bored off my butt. My spouse is, unfortunately, utterly useless at home and has proven that he (1) will always put work first, period – because he thinks he’s making the world a better place, and (2) won’t sell out and make some money, because that would be evil. Look, it’s a package deal and that’s who he is, apparently.
This leaves me with a conundrum, and I’m getting quite tired of being poor. One needs a tutor and one needs more behavioral therapy, and I’m not sure where that money is going to come from. I can go and take a government counsel job, I believe that I can get one, and make substantially more than I make now. Like, twice as much roughly. The hours will be a little worse, the commute will be a lot worse. All told, I figure I’d lose 2 extra hours a day, at least.
There’s no guarantee I’ll like it, of course, but I know I’m bored with the current job. And it has no room for growth. I wonder if I’m better off trying to take a second job or make money contracting rather than going whole hog and jumping careers again? I’ve been where I am about 3 years.
Considering Aunt Pythia has jumped ship quite a few times with her skill set, I’m curious if she has insight for me.
Proudly OK on Rent, But Otherwise Rarely Excited Daily
Dear POoR, BORED,
First, I appreciate your sign-off, and second, you were seriously bumping up against the length limit but the sign-off got you through.
And I get needing to prioritize your kids, but I’m going to take issue with two things: your hubby and your boredom.
First, your boredom. Not cool. You need to be interested in your own life, and being bored off your butt is seriously not cutting it. Be more selfish than that, and do it for your kids. They need a mom who’s also a role model. Go find a better job, that pays enough for your needs and that interests you.
Second, your husband. Also not cool that he’s “utterly useless at home,” both because you are wasting time resenting him and because you genuinely need his help. And don’t give me that “because he’s saving the world” crap. He’s not helpful because he’s gotten away with not being helpful. It’s a deal you made with him, possibly (probably) without thinking enough about it. Time to renegotiate. Oh, and renegotiating shitty deals that don’t work for you is also a good role modeling opportunity for your kids.
Here’s how I’d work this. Sit your husband down when the two of you have time, on a weekend evening after the kids are asleep, and tell him you’re bored, need a more challenging job, and that will mean he needs to help out with the house and the kids, because chances are your new job will have more commuting time or whatever.
Next, explain how you’ve worked out the schedule for both of you (if you need to), or ask him to help work it out with you so that it all works. Don’t ask him for help like he’s got an option, because you need this, and that means the family needs this. You guys are a team, and teams work together to make things work.
Good luck!
Aunt Pythia
——
Dear Aunt Pythia,
Two economists have recently posted a paper arguing that Fields Medalists’ productivity decreases after they get their Fields Medal. While this certainly seems plausible psychologically (after all, proving minor theorems might seem anti-climactic after you’ve solved the major open problem in your field), when I looked at this paper, it seemed that the statistics in it were very naive, in that the authors completely ignored any possible post-conditioning; this leads me to believe that the conclusion is quite likely to be wrong. I have two questions:
Are the statistics used in most economics papers this poor, and if so, how can we trust economists to run our economy?
Would it be worth redoing the statistics in this paper to show up these economists, and maybe to defend Fields Medalists against their charges of being lazy?
Burnt-Out Prizewinner
Dear BOP,
First of all, without even reading the paper I’d say we shouldn’t trust economists to run our economy. They have already proved their vested interests are too distracting for such a responsibility.
Second, I scanned the paper, and I’m not very interested in their model but I am kind of interested in their appendix, where they have the following graph:
After all, I want to see the data, and here it is. Look carefully at the comparison group for the Fields Medalists: people who won another big award besides the Fields Medal and have “above-media per-year citations” during the eligibility period.
My question is, why did they include that second part about citations? It’s muddying the waters, for me at least. Did the actual winners also have above-median per-year citations? Are we assuming that the Fields Medal committee uses that as a criterion for eligibility? It’s weird, and I think the data would be cleaner without that stipulation: we’d just be comparing Fields Medalists versus “other” medalists. Now I’m thinking we’re cherry picking. After all, I can imagine that people who get lots of citations are also like to write more papers.
Next, I’d like to see the data on the individual basis or in some way see what kind of error bars we’re talking about here. The fact that there’s a three-year average in the above graph tells me this data is somewhat noisy. Plus the fact that the three-year average is centered on the middle year is weird. All graphs should reflect data known by a certain date.
But finally, I’m willing to ask, who cares? I guess I don’t care about awards in math much, but even if I did, I’m not willing to agree that the whole point of giving out Fields Medals is to “encourage further achievement” on the part of the recipient, even if Fields himself said that. I’ma go with the other reason, which is to get people to compete against each other (yuck).
Whatever, it’s not like you’re going to get a second Fields Medal or something. If you were doing stuff in order to win a Fields Medal, then after getting it, you’d stop, right, and do something else that’s interesting? Makes sense to me.
Aunt Pythia
——
Dear Aunt Pythia,
I agree with your answers to recent question that often enough folks’ criticism of others stems from their own insecurities. Meditating on the fact was making me depressed, until I found an uneaten cupcake in the cupboard, and then thought: do you think that it also works the other way around?
Even if it doesn’t, your site is da bomb! <fingers crossed>.
Mmmm…a cupcake
Dear Mmmm,
First of all, I don’t trust cupcakes in cupboards. At least at my house that signals something very very wrong, that a cupcake made it into a cupboard. Shoulda at most made it onto the counter. Most cupcakes don’t make it out of the shopping bag around here.
Second of all, are you crossing your fingers because you’re hoping to get your question into the column? Or is it because you’re hoping mathbabe.org really is da bomb? Cuz it is, so your hopes have been fully realized.
Next, to your question. By “the other way around,” I’m going to guess you mean the following: when people are insecure about something, they accuse others of having that flaw. The answer is yes, absolutely.
In fact it’s generally true that when someone is sensitized to an issue, even if it’s not one of insecurity, then they see it everywhere, all the time, as if for the first time. There’s a running joke in my family that whenever someone starts a sentence with “Have you noticed lately that…” then what follows that will be a selection bias in exactly that way. So, have you noticed lately that everyone is wearing incredibly awesome flannel shirts? That’s because I got comfort on da mind over here.
Anyhoo, same thing for insecurities. If one is feeling like one’s acne is out of control, one sees other people’s pimples a mile away. If I am ashamed of myself for being overly bossy, then I see overbearing behavior everywhere and I can’t understand how people can stand it. And although we do our best to not accuse people of stuff we’re aware of being sensitized to, it doesn’t always work out that way.
Best,
Aunt Pythia
——
Please submit your well-specified, fun-loving, cleverly-abbreviated question to Aunt Pythia!
“People analytics” embeds old cultural problems in new mathematical models
Today I’d like to discuss recent article from the Atlantic entitled “They’re watching you at work” (hat tip Deb Gieringer).
In the article they describe what they call “people analytics,” which refers to the new suite of managerial tools meant to help find and evaluate employees of firms. The first generation of this stuff happened in the 1950’s, and relied on stuff like personality tests. It didn’t seem to work very well and people stopped using it.
But maybe this new generation of big data models can be super useful? Maybe they will give us an awesome way of throwing away people who won’t work out more efficiently and keeping those who will?
Here’s an example from the article. Royal Dutch Shell sources ideas for “business disruption” and wants to know which ideas to look into. There’s an app for that, apparently, written by a Silicon Valley start-up called Knack.
Specifically, Knack had a bunch of the ideamakers play a video game, and they presumably also were given training data on which ideas historically worked out. Knack developed a model and was able to give Royal Dutch Shell a template for which ideas to pursue in the future based on the personality of the ideamakers.
From the perspective of Royal Dutch Shell, this represents huge timesaving. But from my perspective it means that whatever process the dudes at Royal Dutch Shell developed for vetting their ideas has now been effectively set in stone, at least for as long as the algorithm is being used.
I’m not saying they won’t save time, they very well might. I’m saying that, whatever their process used to be, it’s now embedded in an algorithm. So if they gave preference to a certain kind of arrogance, maybe because the people in charge of vetting identified with that, then the algorithm has encoded it.
One consequence is that they might very well pass on really excellent ideas that happened to have come from a modest person – no discussion necessary on what kind of people are being invisible ignored in such a set-up. Another consequence is that they will believe their process is now objective because it’s living inside a mathematical model.
The article compares this to the “blind auditions” for orchestras example, where people are kept behind a curtain so that the listeners don’t give extra consideration to their friends. Famously, the consequence of blind auditions has been way more women in orchestras. But that’s an extremely misleading comparison to the above algorithmic hiring software, and here’s why.
In the blind auditions case, the people measuring the musician’s ability have committed themselves to exactly one clean definition of readiness for being a member of the orchestra, namely the sound of the person playing the instrument. And they accept or deny someone, sight unseen, based solely on that evaluation metric.
Whereas with the idea-vetting process above, the training data consisted of “previous winners” which presumable had to go through a series of meetings and convince everyone in the meeting that their idea had merit, and that they could manage the team to try it out, and all sorts of other things. Their success relied, in other words, on a community’s support of their idea and their ability to command that support.
In other words, imagine that, instead of listening to someone playing trombone behind a curtain, their evaluation metric was to compare a given musician to other musicians that had already played in a similar orchestra and, just to make it super success-based, had made first seat.
That you’d have a very different selection criterion, and a very different algorithm. It would be based on all sorts of personality issues, and community bias and buy-in issues. In particular you’d still have way more men.
The fundamental difference here is one of transparency. In the blind auditions case, everyone agrees beforehand to judge on a single transparent and appealing dimension. In the black box algorithms case, you’re not sure what you’re judging things on, but you can see when a candidate comes along that is somehow “like previous winners.”
One of the most frustrating things about this industry of hiring algorithms is how unlikely it is to actively fail. It will save time for its users, since after all computers can efficiently throw away “people who aren’t like people who have succeeded in your culture or process” once they’ve been told what that means.
The most obvious consequence of using this model, for the companies that use it, is that they’ll get more and more people just like the people they already have. And that’s surprisingly unnoticeable for people in such companies.
My conclusion is that these algorithms don’t make things objective, they makes things opaque. And they embeds our old cultural problems in new mathematical models, giving us a false badge of objectivity.
Today I’m thankful for the pope
I’m not religious but I think Pope Francis is an awesome and inspiring thinker and leader. And yes, I’ve invited him to join Occupy. Here’s an excerpt from his recent Apostolic Exhortation in case you haven’t seen it yet:
No to an economy of exclusion
53. Just as the commandment “Thou shalt not kill” sets a clear limit in order to safeguard the value of human life, today we also have to say “thou shalt not” to an economy of exclusion and inequality. Such an economy kills. How can it be that it is not a news item when an elderly homeless person dies of exposure, but it is news when the stock market loses two points? This is a case of exclusion. Can we continue to stand by when food is thrown away while people are starving? This is a case of inequality. Today everything comes under the laws of competition and the survival of the fittest, where the powerful feed upon the powerless. As a consequence, masses of people find themselves excluded and marginalized: without work, without possibilities, without any means of escape.
Human beings are themselves considered consumer goods to be used and then discarded. We have created a “disposable” culture which is now spreading. It is no longer simply about exploitation and oppression, but something new. Exclusion ultimately has to do with what it means to be a part of the society in which we live; those excluded are no longer society’s underside or its fringes or its disenfranchised – they are no longer even a part of it. The excluded are not the “exploited” but the outcast, the “leftovers”.
54. In this context, some people continue to defend trickle-down theories which assume that economic growth, encouraged by a free market, will inevitably succeed in bringing about greater justice and inclusiveness in the world. This opinion, which has never been confirmed by the facts, expresses a crude and naïve trust in the goodness of those wielding economic power and in the sacralized workings of the prevailing economic system. Meanwhile, the excluded are still waiting. To sustain a lifestyle which excludes others, or to sustain enthusiasm for that selfish ideal, a globalization of indifference has developed. Almost without being aware of it, we end up being incapable of feeling compassion at the outcry of the poor, weeping for other people’s pain, and feeling a need to help them, as though all this were someone else’s responsibility and not our own. The culture of prosperity deadens us; we are thrilled if the market offers us something new to purchase; and in the meantime all those lives stunted for lack of opportunity seem a mere spectacle; they fail to move us.
No to the new idolatry of money
55. One cause of this situation is found in our relationship with money, since we calmly accept its dominion over ourselves and our societies. The current financial crisis can make us overlook the fact that it originated in a profound human crisis: the denial of the primacy of the human person! We have created new idols. The worship of the ancient golden calf (cf. Ex 32:1-35) has returned in a new and ruthless guise in the idolatry of money and the dictatorship of an impersonal economy lacking a truly human purpose. The worldwide crisis affecting finance and the economy lays bare their imbalances and, above all, their lack of real concern for human beings; man is reduced to one of his needs alone: consumption.
56. While the earnings of a minority are growing exponentially, so too is the gap separating the majority from the prosperity enjoyed by those happy few. This imbalance is the result of ideologies which defend the absolute autonomy of the marketplace and financial speculation. Consequently, they reject the right of states, charged with vigilance for the common good, to exercise any form of control. A new tyranny is thus born, invisible and often virtual, which unilaterally and relentlessly imposes its own laws and rules. Debt and the accumulation of interest also make it difficult for countries to realize the potential of their own economies and keep citizens from enjoying their real purchasing power. To all this we can add widespread corruption and self-serving tax evasion, which have taken on worldwide dimensions. The thirst for power and possessions knows no limits. In this system, which tends to devour everything which stands in the way of increased profits, whatever is fragile, like the environment, is defenseless before the interests of a deified market, which become the only rule.
No to a financial system which rules rather than serves
57. Behind this attitude lurks a rejection of ethics and a rejection of God. Ethics has come to be viewed with a certain scornful derision. It is seen as counterproductive, too human, because it makes money and power relative. It is felt to be a threat, since it condemns the manipulation and debasement of the person. In effect, ethics leads to a God who calls for a committed response which is outside of the categories of the marketplace. When these latter are absolutized, God can only be seen as uncontrollable, unmanageable, even dangerous, since he calls human beings to their full realization and to freedom from all forms of enslavement. Ethics – a non-ideological ethics – would make it possible to bring about balance and a more humane social order. With this in mind, I encourage financial experts and political leaders to ponder the words of one of the sages of antiquity: “Not to share one’s wealth with the poor is to steal from them and to take away their livelihood. It is not our own goods which we hold, but theirs”.[55]
58. A financial reform open to such ethical considerations would require a vigorous change of approach on the part of political leaders. I urge them to face this challenge with determination and an eye to the future, while not ignoring, of course, the specifics of each case. Money must serve, not rule! The Pope loves everyone, rich and poor alike, but he is obliged in the name of Christ to remind all that the rich must help, respect and promote the poor. I exhort you to generous solidarity and a return of economics and finance to an ethical approach which favours human beings.
No to the inequality which spawns violence
59. Today in many places we hear a call for greater security. But until exclusion and inequality in society and between peoples is reversed, it will be impossible to eliminate violence. The poor and the poorer peoples are accused of violence, yet without equal opportunities the different forms of aggression and conflict will find a fertile terrain for growth and eventually explode. When a society – whether local, national or global – is willing to leave a part of itself on the fringes, no political programmes or resources spent on law enforcement or surveillance systems can indefinitely guarantee tranquility. This is not the case simply because inequality provokes a violent reaction from those excluded from the system, but because the socioeconomic system is unjust at its root. Just as goodness tends to spread, the toleration of evil, which is injustice, tends to expand its baneful influence and quietly to undermine any political and social system, no matter how solid it may appear. If every action has its consequences, an evil embedded in the structures of a society has a constant potential for disintegration and death. It is evil crystallized in unjust social structures, which cannot be the basis of hope for a better future. We are far from the so-called “end of history”, since the conditions for a sustainable and peaceful development have not yet been adequately articulated and realized.
60. Today’s economic mechanisms promote inordinate consumption, yet it is evident that unbridled consumerism combined with inequality proves doubly damaging to the social fabric. Inequality eventually engenders a violence which recourse to arms cannot and never will be able to resolve. This serves only to offer false hopes to those clamouring for heightened security, even though nowadays we know that weapons and violence, rather than providing solutions, create new and more serious conflicts. Some simply content themselves with blaming the poor and the poorer countries themselves for their troubles; indulging in unwarranted generalizations, they claim that the solution is an “education” that would tranquilize them, making them tame and harmless. All this becomes even more exasperating for the marginalized in the light of the widespread and deeply rooted corruption found in many countries – in their governments, businesses and institutions – whatever the political ideology of their leaders.
Cool open-source models?
I’m looking to develop my idea of open models, which I motivated here and started to describe here. I wrote the post in March 2012, but the need for such a platform has only become more obvious.
I’m lucky to be working with a super fantastic python guy on this, and the details are under wraps, but let’s just say it’s exciting.
So I’m looking to showcase a few good models to start with, preferably in python, but the critical ingredient is that they’re open source. They don’t have to be great, because the point is to see their flaws and possible to improve them.
- For example, I put in a FOIA request a couple of days ago to get the current teacher value-added model from New York City.
- A friends of mine, Marc Joffe, has an open source municipal credit rating model. It’s not in python but I’m hopeful we can work with it anyway.
- I’m in search of an open source credit scoring model for individuals. Does anyone know of something like that?
- They don’t have to be creepy! How about a Nate Silver – style weather model?
- Or something that relies on open government data?
- Can we get the Reinhart-Rogoff model?
The idea here is to get the model, not necessarily the data (although even better if it can be attached to data and updated regularly). And once we get a model, we’d build interactives with the model (like this one), or at least the tools to do so, so other people could build them.
At its core, the point of open models is this: you don’t really know what a model does until you can interact with it. You don’t know if a model is robust unless you can fiddle with its parameters and check. And finally, you don’t know if a model is best possible unless you’ve let people try to improve it.
Twitter and its modeling war
I often talk about the modeling war, and I usually mean the one where the modelers are on one side and the public is on the other. The modelers are working hard trying to convince or trick the public into clicking or buying or consuming or taking out loans or buying insurance, and the public is on the other, barely aware that they’re engaging in anything at all resembling a war.
But there are plenty of other modeling wars that are being fought by two sides which are both sophisticated. To name a couple, Anonymous versus the NSA and Anonymous versus itself.
Here’s another, and it’s kind of bland but pretty simple: Twitter bots versus Twitter.
This war arose from the fact that people care about how many followers someone on Twitter has. It’s a measure of a person’s influence, albeit a crappy one for various reasons (and not just because it’s being gamed).
The high impact of the follower count means it’s in a wannabe celebrity’s best interest to juice their follower numbers, which introduces the idea of fake twitter accounts to game the model. This is an industry in itself, and an associated arms race of spam filters to get rid of them. The question is, who’s winning this arms race and why?
Twitter has historically made some strides in finding and removing such fake accounts with the help of some modelers who actually bought the services of a spammer and looked carefully at what their money bought them. Recently though, at least according to this WSJ article, it looks like Twitter has spent less energy pursuing the spammers.
It begs the question, why? After all, Twitter has a lot theoretically at stake. Namely, its reputation, because if everyone knows how gamed the system is, they’ll stop trusting it. On the other hand, that argument only really holds if people have something else to use instead as a better proxy of influence.
Even so, considering that Twitter has a bazillion dollars in the bank right now, you’d think they’d spend a few hundred thousand a year to prevent their reputation from being too tarnished. And maybe they’re doing that, but the spammers seem to be happily working away in spite of that.
And judging from my experience on Twitter recently, there are plenty of active spammers which actively degrade the user experience. That brings up my final point, which is that the lack of competition argument at some point gives way to the “I don’t want to be spammed” user experience argument. At some point, if Twitter doesn’t maintain standards, people will just not spend time on Twitter, and its proxy of influence will fall out of favor for that more fundamental reason.
Aunt Pythia’s advice
Aunt Pythia is hung over from excess rabble rousing and karaoke, but she’s determined not to miss another week of her beloved advice column. Aunt Pythia has missed you! As I’m sure you’ve missed her! Please enjoy today’s column, and
please, don’t forget to ask Aunt Pythia a question at the bottom of the page!
By the way, if you don’t know what the hell Aunt Pythia is talking about, go here for past advice columns and here for an explanation of the name Pythia.
——
Dear Aunt Pythia,
I have an older sister. She’s a lovely and good person. Very generous, very friendly. And very assertive, in an oldest-child-in-the-family way.
I love my family, but I often feel depressed and suffocated when I’m around my sister. Is it because I feel she’s constantly giving input on how I could do things differently, and why she’s chosen to do things the way she does as if she has a PhD on the subject, and I often am left doubting my own abilities to make decisions even though I know that in reality I have a pretty good head on my shoulders? Maybe. Is it because of the authority she speaks on any topic, even ones she knows very little about? Is it because she doesn’t seem to entertain the possibility that anyone else could have anything to add in terms of input? Is it because she rarely shows any kind of vulnerability? Is it because she’s so assertive that it often feels like she’s taking up all of the oxygen in the room? Is it because she does all of these things even while, at the same, she is being utterly helpful, generous, and selfless in most other ways? Yeah, maybe that too.
Whatever it is, it hardly seems like a good reason to get depressed or to distance myself from someone who genuinely loves me and whom I love. I get that this is my issue, and the problem is how I feel about myself when I’m around her. I want to get over this. I just don’t know where to start.
Family Stuff
Dear Family Stuff,
To be honest I double- and triple-checked that I don’t have any younger siblings when I read this, because it could be about me. I could totally be that older sister, and I imagine that many people feel this way about me.
But if I’m right, and if your sister is a lot like me, then I don’t think it’s “your issue” to get over. I’m guessing it’s more like a series of signals that she’s giving out that are not hitting the intended targets. And if I’m right, she actually does want you to add stuff, but she expects you to jump right into the ring and not need an invitation.
So, when she gives advice, think of her words as her unedited thoughts, and do with those thoughts what you may. You can test this theory by every now and then pointing out, “I tried that already, it didn’t work” and see what she says. If she’s like, “Oh cool, how about trying this?” then you know she’s just taking stabs.
And, when she has an opinion on everything, maybe she’s just trying to engage in a provocative conversation and wants to be challenged. I do that all the time (duh). So next time she says something that sounds uninformed, say something like, “Hey that sounds wrong to me – should we check the facts?” and see how she reacts. She might be psyched for the challenge and for the chance to learn something interesting.
As for your sister showing no vulnerability. The funny thing about family is, we are our most vulnerable with our family, and yet we are also very comfortable with them, because we know them so well. You might be surprised by how vulnerable she really is. At the same time, you might not want to test this one, because it’s usually a negative experience to expose vulnerability in someone else. In any case my advice here is to not assume an entire lack of vulnerability around family, even if it looks like that.
Last piece of advice: go read my recent post called “Cathy’s Wager.” It’s about how to react when people are treating you not-so-nicely. I think it’s relevant here, because the overall point is that it’s not about you. Your sister is who she is and she’s very likely not doing all this stuff in order to make you feel stifled and depressed. She’s a know-it-all loudmouth, true, but the sooner you can either get on her wavelength (see above tips) or roll your eyes and love her in spite of her pushy know-it-all ways the better for you and for her. Don’t take it personally.
Either that or just never see her again. That’s totally fine too, honestly. I don’t agree that you have to hang out with family, unless possibly if they’re dying or in need.
I hope that helps!
Aunt Pythia
——
Aunt Pythia,
What ever happened to the proof of the ABC conjecture by Mochizuki that you talked about a year ago?
Thanks,
Curious
Curious,
I unfortunately missed him when he came to Columbia, but Brian Conrad recently came and updated the math community on the status of the alleged proof. I believe the bottomline was that it has not been confirmed by anyone. So, I’d say this means it’s not a proof.
AP
——
Dear Aunt Pythia,
This is a longtime worry of mine. Since you are a master of both abstract as well as the quantitative, let me query you regarding the deep connection that seems to exist between the two. To put it simply, the question is, “Does Size Matter?” More precisely, does Size influence tender feelings of the heart?
Sizeable Confusion
Dear SC,
I’d guess about as much as anything else physical, like boob size or leg length. In other words it might be a pretty big deal initially, as in during the first few minutes, but then when real love sets in it’s a total non-issue.
AP
——
Dear Aunt Pythia,
Many government officials testified that there is no way for them to tell how many people signed up for Obamacare. Can extracting the data from the website be that complicated? I am worried and lost.
Worried about Obamacare
Dear Worried,
Well many people have been busy counting this stuff since you submitted that question, and the final number for the first day of Obamacare seems to be 6. Given how small that number is, I’m going to assume it wasn’t that hard to count, or at least approximate at “0”. In other words, it might have been a political decision to repress the actual number.
On the other hand, engineering large-scale systems is actually pretty complicated, and it might not make sense to have a single repository to put all the enrollment figures – who knows, and I didn’t design this system, so I don’t – so I can imagine that it was actually non-trivial to figure out the answer to this question.
By the way, I’m planning to write some posts on how we are increasingly seeing pure engineering issues become political issues. There’s Knight Capital’s trading mistakes, then there’s Obamacare. Those are just two, but my theory is that they are just the beginning of a very long list. The nerds are taking over, in other words, or at least their mistakes are.
Aunt Pythia
——
Dear Aunt Pythia,
I sent you a question a few weeks back and you didn’t answer it (which is completely fine). What is your criteria for answering a question or not? Maybe your answer might help me rewrite my question in a way that suits you better.
Socially Awkward Dude
Dear SAD,
Here’s the thing. I’m pretty desperate over here, what with a pretty short list of questions, and a stubborn refusal on my part to make up fake questions (although I do accept other people’s fake questions!). So there had to be something about your question that didn’t sit well with me. Here are some possibilities as to why:
- The question was something I couldn’t answer, because it required expertise I don’t have.
- The question was really long and not easily edited down to something shortish.
- The question wasn’t really a question, just a rambling speech.
- The question was spam.
- The question was verbally abusive towards me.
- The question struck me as disingenuous in some way.
- The question is a lot like other questions I’ve already answered (note to the 40 people asking me how to become a data scientist: read my book called Doing Data Science!)
I have no idea which question was yours, but if you’d care to resubmit, making sure it’s to the point, has a specific and earnest question, and is about something I have knowledge about, then I’m guessing it will get through.
I hope that helps!
Auntie P
——
Dear Aunt Pythia,
I am a late-20’s data scientist (working at a large non-tech company) about to apply for Ph.D. programs in machine learning. My reason for doing this is two-fold. One, I enjoy research and feel that I can contribute to humanity through scholarship, even if the contribution may be small. Two, I’ve grown disillusioned with working in a corporate environment – it seems like one needs to be more of a politician than a genuinely nice and high-performing individual to be recognized. But I realize this is partly due to the size of my organization (are start-ups any different?).
However, I’ve heard people tell me that academia is no different. Given the publish-or-perish paradigm, people are more interested in how many citations they have than they are about truly advancing human knowledge (for example, this was a depressing read).
You transitioned from academia into industry. Do you have any advice for someone who’s trying to make the opposite transition?
Naively Bayesian
Dear NB,
First of all, start-ups are sometimes different, although they work you really hard and often expect you to sleep under your desk. This might not work for you, but it might be worth it if you get to have influence. Also, I’d suggest going with a very small start-up: as soon as there are like 60 people, your potential influence typically gets pretty miniscule.
Second, my motto is “You never get rid of your problems, you just get a new set of problems.” So it’s more a question of which kinds of problems suit your personality than anything else.
But there’s one thing I can assure you: there’s politics everywhere. You’re not getting away from that, so if you’re really allergic to politics, I suggest you find a place where you can safely ignore that stuff, like maybe in a cave in the woods.
But seriously, I’d suggest you talk to a lot of people and see what kind of problems are there, without exaggerating them too much (I feel like that link is too aggressive for example, although there are grains of truth in it). And most importantly, try to find something to do that actually interests you in an intellectual way so you can become absorbed in your own sense of curiosity and shut out the real world at least once a day. Good luck!
Aunt Pythia
——
Please submit your well-specified, fun-loving, cleverly-abbreviated question to Aunt Pythia!
People don’t get fired enough
This might surprise some of you – or not, I’m not sure. But one of the most satisfying things about leaving academia and the tenure system and going into industry is how, at least in the ideal situation, you can get fired for not doing your job.
In fact, one of the reasons I decided to leave academia is that I really thought some of my colleagues weren’t doing right by the undergraduates, and the frustrating thing was that there was essentially no way to force them to start. Tenure has great aspects and not-so-great aspects, and a total lack of leverage is not a great one. I feel for deans sometimes.
Here’s the dirty little secret of lots of industry jobs, though: lots of time people also don’t get fired when they should. And sometimes it’s super awful bullies who yell and scream and act inappropriately but also pull in amazing sales numbers. There are things like that, of course. That’s the example of how they don’t abide by the alleged social contract but they perform on the bottomline. Social contracts are hard to quantify and somewhat squishy. You see people getting away with stuff because they’re rainmakers or higher ups.
But there are also plenty of examples of people just not doing their job, and having super awful attitudes, or even just completely apathetic attitudes, and for whatever reason they don’t get fired. This demoralizes and irritates and distracts everyone around them, because they all resent the free-rider.
Plus, retaining people who should by all accounts get fired makes the veneer of the kool-aid drinking camaraderie even more flimsy and scrutinizable – what’s so great about working here if people can just slack off and not care? Why do I give two shits about this project anyway? How does this project in the larger scheme of things? Maybe that scrutiny is a good thing – I engage in it myself – but you don’t want everyone thinking that all the time.
Here’s the thing, before you think I’m super vicious and mean to want people to get fired. These people I’m talking about are generally high skilled and temporarily depressed. They’re in the wrong job. And once fired, they will find another job, which will hopefully be a better one for them. I’m not saying that nobody will ever end up jobless and homeless, but very few, and moreover there are plenty of jobless and homeless people who would be psyched to do that job really well (putting aside how difficult it is for homeless people to get seriously considered for a job).
And I’m not saying you fire people out of the blue. You definitely need to tell people they’re not performing well (or that they are) and keep them in the feedback loop on whether things are working out. But in my experience people who deserve to get fired totally know it and can’t believe their luck that they’ve not been fired yet.
To conclude, I’m going on record saying I kind of agree with Jack Welch on this issue in a way I never thought I would.
mathbabe 