Predictive risk models for prisoners with mental disorders
My friend Jordan Ellenberg sent me an article yesterday entitled Coin-flip judgement of psychopathic prisoners’ risk.
It was written by Seena Fazel, a researcher at the department of psychiatry at Oxford, and it concerns his research into the currently used predictive risk models for violence, repeat offense, and the like, which are supposedly tailored to people who have mental disorders like psychopathy.
Turns out there are a lot of these models, and they’re in use today in a bunch of countries. I did not know that. And they’re not just being used as extra, “good to know” information, but rather as a tool to assess important decisions for the prisoner. From the article:
Many US states use such tools to assess sexual offending risk and to help decide whether to exercise their powers to detain sexual offenders indefinitely after a prison term ends.
In England and Wales, these tools are part of the admission criteria for centres that treat people with dangerous and severe personality disorders. Outside North America, Europe and Australasia, similar approaches are increasingly popular, particularly in clinical settings, and there has been a steady growth of research from middle-income countries, such as China, documenting their use.
Also turns out, according to a meta-analysis done by Fazel, that these models don’t work very well, especially for the highest risk most violent population. And what’s super troubling is, as Fazel says, “In practice, the high false-positive rate probably means that some offenders spend longer in prison and secure hospital than their true risk would suggest.”
Talk about creepy.
This seems to be yet another example of a mathematical obfuscation and intimidation that gives people a false sense of having a good tool at hand. From the article:
Of course, sensible clinicians and judges take into account factors other than the findings of these instruments, but their misuse does complicate the picture. Some have argued that the veneer of scientific respectability surrounding such methods may lead to over-reliance on their findings, and that their complexity is difficult for the courts. Beyond concerns about public protection, liberty and costs of extended detention, there are worries that associated training and administration may divert resources from treatment.
The solution? Get people to acknowledge that the tools suck, and have a more transparent method of evaluating them. In this case, according to Fazel, it’s the researchers who are over-estimating the power of their models. But especially where it involves incarceration and the law, we have to maintain an adherence to a behavior-based methodology. It doesn’t make sense to put people in jail an extra 10 years because a crappy model said so.
This is a case, in my opinion, for an open model with a closed black box data set. The data itself is extremely sensitive and protected, but the model itself should be scrutinized.
What’s the best narrative for revolution?
Yesterday at our weekly Alt Banking meeting we had an extraordinary speaker, Merlyna Lim, come talk to us about social media and grass roots organizing.
Her story was interesting and nuanced; I won’t get everything down here. But there were quite a few sound byte takeaways I can express.
- The organizing which culminated in the Arab Spring started way before Facebook or Twitter came to the region.
- To a large extent social media has replaced chatting in the cafe, which we don’t do anymore.
- But that’s actually a good thing, since many regimes are so oppressive they won’t let large groups of people hold regular meetings (and large can mean 5 or more).
- Whereas social media is pretty good at energizing people to “get rid of their enemy” at a given critical moment, and mobilize on the street, it’s not that great at nuanced discussions for how to build something permanent and lasting afterwards.
One other thing I wanted to mention was Merlyna’s work on Mohamed Bouazizi, the Tunisian street vendor who self-immolated after getting into a dispute with a police officer.
The original story that got people mobilized in Tunis and out on the street, was that the police officer was a woman, that she slapped him, and that he was a college educated street vendor. It turns out these were white lies – he never finished high school, the police officer may have been a man, and there was probably no slap – but they built a narrative that people really loved. Merlyna wrote a paper about this available here if you want to know more.
That brings us to the question of why this particular framing was so appealing. Merlyna put it this way: plenty of other people had self-immolated under similar circumstances in Tunisia in the past 6 months alone. But they didn’t start a revolution because they were just very poor and didn’t have this story with extra (made-up) humiliating details. Killing yourself because you are frustrated at not having enough to eat just isn’t as compelling.
It reminds me of this Bloomberg View piece I’ve been chewing on for a couple of week, written by Peter Turchin and entitled Blame Rich, Overeducated Elites as Our Society Frays. He studies conditions for revolution as well, and claims that having a large unemployed but highly educated population – the “lawyer glut” we’re seeing today – is asking for trouble. From his article:
Elite overproduction generally leads to more intra-elite competition that gradually undermines the spirit of cooperation, which is followed by ideological polarization and fragmentation of the political class. This happens because the more contenders there are, the more of them end up on the losing side. A large class of disgruntled elite-wannabes, often well-educated and highly capable, has been denied access to elite positions.
Food for thought. Does one have more sympathy for people whose foodstamps have been recently cut or for someone who got a law degree and can’t find a job? Or is the real outrage when both happen (or at least are said to happen)? Personally, and this is maybe because I’ve been reading Jonathan Kozol’s Savage Inequalities, I’m not as worried about the lawyer.
Aunt Pythia’s advice: sex at the end
You guys know Aunt Pythia loves you. And Aunt Pythia feels the love from you readers as well, especially in person (some of you are reticent to add comments online, for whatever reason).
So don’t take it the wrong way when I say this: you guys are nerds. I have like a 5-to-1 ratio of math-related versus sex-related questions, and today I’m effectively withholding the sex until the end as a hook to keep you guys.
Don’t get me wrong, I love nerd questions. Happy to answer them. But people! Let’s spice this up! And if you can’t go all the way to sex at least come up with something about breastfeeding in public or thereabouts. As you know, Aunt Pythia doesn’t make up questions – that would be beneath her – but she has no problem with prompts.
In other words, as you enjoy today’s column:
please, think of something sexy to ask Aunt Pythia at the bottom of the page!
By the way, if you don’t know what the hell Aunt Pythia is talking about, go here for past advice columns and here for an explanation of the name Pythia.
——
Aunt Pythia,
What’s the deal with employers being dishonest in their job descriptions, and the general acceptance of this sort of unethical behavior? I work in a somewhat prestigious buy-side shop where I was told I’d be in a front-office quant research position. After I arrive, I find out that my responsibilities are really more like that of a middle-office tech position. Instead of doing research on market inefficiencies, I’m relegated to automating an endless number of reports. My employer knew what the job would entail before I joined and yet portrayed it to be something it’s not. Worst of all, it seems like 80% of the people I consult with say (expressly or implicitly) that I should be glad I got my foot in the door and that this stuff is very common, so it’s nothing to fret about. WTF’s wrong with people?
Perennial Employee
Dear PE,
For whatever reason, which I certainly don’t relate to, there are some people that still desperately want to work in finance as front-office quants. They want it so badly, in fact, that they’re willing to pretend to be doing that while they actually do other stuff. You seem to not be one of those people. Awesome.
My suggestion to you is to get another job, simple as that. You’re not going to change their mind about what your job should be, since they’re clearly perfectly comfortable with lying to people. I mean, once you’ve got another job lined up, there’s no harm in telling them you’re leaving unless you get moved to the position you were promised, but please don’t hold your breath for that to actually happen.
One last thing: look outside finance! There are plenty of other ways to be a nerd.
Aunt Pythia
——
Dear Aunt Pythia,
How does Mathbabe break down a data problem into manageable steps? I’m a mathematician who has tried a few data mining problems on the side for fun, and I get totally overwhelmed whenever I try to start. If I were solving a math problem, I’d read relevant papers to see what is known and get ideas for techniques, I’d break down my desired result into lemmas and work on them one by one, and I’d have a plan in mind throughout (it might change, of course, but I’d always know why I was doing what I was doing).
But if I’m trying to, say, classify a bunch of labeled feature vectors, I’m at a loss. I experiment and play around with the data, but I feel so random about everything. How do I choose how many hidden units to have in a neural net? How do I choose K in K-nearest neighbor classification? And so on. Some stuff works better than other stuff, but I don’t know how to be systematic. I end up getting discouraged, which is too bad because data problems are awesome and I want to master them.
Any tips for this mathematician on how to solve problems whose solutions aren’t proof-based?
Proof Machine
Dear PM,
Great question! And I’m glad you’re asking that. It’s a sign that you want to do things right, and know why you’ve made decisions. I want you to cultivate that desire.
First, (after separating my out-of-sample data from my in-sample data) I spend a lot of time with smallish samples getting the feel of things through “exploratory data analysis.” This helps make sure the data is clean, gives me the overall distribution and feel for the various data sources, and gives me some idea of the kind of relationships I might expect between the inputs and possibly the target, if there’s a well-defined target.
You’d be surprised how much you learn by doing that.
Next, how do you even choose which algorithm to use, never mind how exactly to tune the hyperparameters of a given algorithm? The answer is that it’s a craft, and over time you gain intuition, but at first you just don’t know and you experiment. Put the science in data science. Try a bunch of different ones and see which works better, and hypothesize on why, and try to test that hypothesis.
Here’s another possibility. Start with synthetic data that is “perfectly set up” for a given algorithm – figure out what that means – and then pretend you don’t know that, and see whether the above testing procedure would give you the correct result. Now add noise to that perfect data set, and see how quickly (i.e. with how much noise) your perfect solution doesn’t seem optimal anymore. That gives you an overall way of thinking about optimizing algorithms and hyperparameters. It’s hard, even with linear regression.
Oh, and buy my book. It should hopefully help.
Good luck!
Aunt Pythia
p.s. when I worked in math, I didn’t break things down into lemmas first. I first tried to answer the question, why is this true? (maybe by starting with small examples) and then only later, in order to explain it on paper, would I break things down into lemmas.
——
Dear Aunt Pythia,
I have a tenure-track job in a “hard-core STEM field”; I’m also a very young looking woman. I have a serious and rewarding research program, I really enjoy teaching at the board, and I hear that I give great seminars.
Yet recently, for the first time, I have been overcome with extreme, physiological, panic when I stand at the front of a room to give a seminar. This is not because I’m worried about the material; I’m not. This is also not stage fright; I have iron nerves about performing.
It is a feeling of panic brought on by watching the room fill up with men, with maybe only 1 or 2 very junior women. I start thinking “what happened to all the other young women who, like me, loved mathematics? At what point were they all removed from the community? When will too much get to be too much for me too?”
This started happening about a year ago and it’s only getting worse. I’m not expecting to change all the weird experiences of being a young woman in my field; I just want to figure out how to deal with my own thoughts as I stand in front of my audience.
Feeling like a fox in a room full of hunting dogs
Dear fox,
This is going to sound trite, but here goes: you are not a statistic, you are an individual person. And although you are a woman person, that doesn’t mean you have to do stuff that other women have done. If things are working for you on a minute-to-minute basis, then that means you can be happy and proud of having set up your life to be fulfilled.
Nobody is asking you to explain why other people do the things they do. We can barely explain why we do the things we do – and then half the time the understanding only comes years later. Just focus on who you are, who you want to be and how you want to spend your time.
I’d also like to mention that, as a woman who left math, I also loved teaching and I loved giving seminars – that was the good stuff! For that matter there were lots of great things about being a professor. And I didn’t leave because I was a woman and felt like it was time to leave – nor did I not leave because I wanted to prove a point about women not leaving. I left because, in my individual life and with my individual goals, it was what I wanted.
So I guess I’m suggesting that you be a bit more self-centered and somewhat less identified with women, at least at those moments, if that is possible and if that helps. If that doesn’t help, consider going to a cognitive therapist who specializes in dealing with panic attacks. Good luck!
Aunt Pythia
——
Dear Aunt Pythia,
I feel an eerie compulsion to answer this email. I love the broken grammar and all. What should I do?
Hello,
How are you doing today? My name is Colvin Hostetter. I came across your e-mail under the Graduate Students portal while surfing online for tutorial for my daughter, Debra is a 18 years old girl. She is ready to learn. I would like the lessons to be at your location. Kindly let me know your policy with regard to the fees, cancellations, location and make-up lessons.
Also, get back to me with your area of SPECIALIZATION and any necessary information you think that might help.
The lessons can start by last week of November. Mind you, any break during Thanksgiving and Christmas would be observed respectively.
Looking forward reading from you.
My best regard,
Mr Colvin.
Professor Has Ignored Silly Ignoble New Game
Dear PHISING,
Really? PHISING? I think you really are a bit kinky in the grammar and spelling rules department.
So this must be a spam email, since it’s talking about an 18-year-old girl who is “ready to learn.” It sounds like soft porn. And it doesn’t describe what she needs to learn – math? physics? German? I’d be not at all surprised to hear someone describe the actual financial bamboozling mechanism that would transpire if you did answer this, although a quick Google search doesn’t uncover it.
My suggestion is to mark this, and any other similar emails, as “spam” so that Google will do the work for us in the future and delete this bullshit.
AP
——
Dear Auntie P,
My wife and I have not used any birth control other than rhythm and/or withdrawal for more than 16 years now (~mid late twenties to early mid forties.) We have not had any unwanted pregnancies through this. We did have one successfully planned pregnancy that corresponded exactly to the month she charted to pinpoint ovulation.
So, are we lucky outliers or is this a much more successful strategy than we were both led to believe in high school sex ed?
Any suggestions for here on out?
Thanks,
Lucky in love
Dear Lucky,
The reason that rhythm might not work is if women have irregular ovulations. If your wife doesn’t, though, then cool (although she may experience that as she approaches menopause).
The reason withdrawal doesn’t work is because men often forget the “withdrawal” part of the plan. I mean, it’s certainly possible to get pregnant with the pre-cum (just ask Alice) but super unlikely.
In other words, you are a special, special man with a very excellent memory.
Here on out: don’t forget to remember the plan! And be aware of irregular cycles!
Aunt Pythia
——
Please submit your well-specified, fun-loving, cleverly-abbreviated question to Aunt Pythia!
PDF Liberation Hackathon: January 17-19
This is a guest post by Marc Joffe, the principal consultant at Public Sector Credit Solutions, an organization that provides data and analysis related to sovereign and municipal securities. Previously, Joffe was a Senior Director at Moody’s Analytics.
As Cathy has argued, open source models can bring much needed transparency to scientific research, finance, education and other fields plagued by biased, self-serving analytics. Models often need large volumes of data, and if the model is to be run on an ongoing basis, regular data updates are required.
Unfortunately, many data sets are not ready to be loaded into your analytical tool of choice; they arrive in an unstructured form and must be organized into a consistent set of rows and columns. This cleaning process can be quite costly. Since open source modeling efforts are usually low dollar operations, the costs of data cleaning may prove to be prohibitive. Hence no open model – distortion and bias continue their reign.
Much data comes to us in the form of PDFs. Say, for example, you want to model student loan securitizations. You will be confronted with a large number of PDF servicing reports that look like this. A corporation or well funded research institution can purchase an expensive, enterprise-level ETL (Extract-Transform-Load) tool to migrate data from the PDFs into a database. But this is not much help to insurgent modelers who want to produce open source work.
Data journalists face a similar challenge. They often need to extract bulk data from PDFs to support their reporting. Examples include IRS Form 990s filed by non-profits and budgets issued by governments at all levels.
The data journalism community has responded to this challenge by developing software to harvest usable information from PDFs. Examples include Tabula, a tool written by Knight-Mozilla OpenNews Fellow Manuel Aristarán, extracts data from PDF tables in a form that can be readily imported to a spreadsheet – if the PDF was “printed” from a computer application. Introduced earlier this year, Tabula continues to evolve thanks to the volunteer efforts of Manuel, with help from OpenNews Fellow Mike Tigas and New York Times interactive developer Jeremy Merrill. Meanwhile, DocHive, a tool whose continuing development is being funded by a Knight Foundation grant, addresses PDFs that were created by scanning paper documents. DocHive is a project of Raleigh Public Record and is led by Charles and Edward Duncan.
These open source tools join a number of commercial offerings such as Able2Extract and ABBYY Fine Reader that extract data from PDFs. A more comprehensive list of open source and commercial resources is available here.
Unfortunately, the free and low cost tools available to modelers, data journalists and transparency advocates have limitations that hinder their ability to handle large scale tasks. If, like me, you want to submit hundreds of PDFs to a software tool, press “Go” and see large volumes of cleanly formatted data, you are out of luck.
It is for this reason that I am working with The Sunlight Foundation and other sponsors to stage the PDF Liberation Hackathon from January 17-19, 2014. We’ll have hack sites at Sunlight’s Washington DC office and at RallyPad in San Francisco. Developers can also join remotely because we will publish a number of clearly specified PDF extraction challenges before the hackathon.
Participants can work on one of the pre-specified challenges or choose their own PDF extraction projects. Ideally, hackathon teams will use (and hopefully improve upon) open source tools to meet the hacking challenges, but they will also be allowed to embed commercial tools into their projects as long as their licensing cost is less than $1000 and an unlimited trial is available.
Prizes of up to $500 will be awarded to winning entries. To receive a prize, a team must publish their source code on a GitHub public repository. To join the hackathon in DC or remotely, please sign up at Eventbrite; to hack with us in SF, please sign up via this Meetup. Please also complete our Google Form survey. Also, if anyone reading this is associated with an organization in New York or Chicago that would like to organize an additional hack space, please contact me.
The PDF Liberation Hackathon is going to be a great opportunity to advance the state of the art when it comes to harvesting data from public documents. I hope you can join us.
Algorithmic Accountability Reporting: On the Investigation of Black Boxes
Tonight I’m going to be on a panel over at Columbia’s Journalism School called Algorithmic Accountability Reporting: On the Investigation of Black Boxes. It’s being organized by Nick Diakopoulos, Tow Fellow and previous guest blogger on mathbabe. You can sign up to come here and it will also be livestreamed.
The other panelists are Scott Klein from ProPublica and Clifford Stein from Columbia. I’m super excited to meet them.
Unlike some panel discussions I’ve been on, where the panelists talk about some topic they choose for a few minutes each and then there are questions, this panel will be centered around a draft of a paper coming from the Tow Center at Columbia. First Nick will present the paper and then the panelists will respond to it. Then there will be Q&A.
I wish I could share it with you but it doesn’t seem publicly available yet. Suffice it to say it has many elements in common with Nick’s guest post on raging against the algorithms, and its overall goal is to understand how investigative journalism should handle a world filled with black box algorithms.
Super interesting stuff, and I’m looking forward to tonight, even if it means I’ll miss the New Day New York rally in Foley Square tonight.
The cost savings of food stamps cuts versus the cost increases of diabetes care
As many of you are aware, food stamps were recently cut in this country. This has had a brutal effect on people and families and on neighborhood food pantries, which are being swamped with new customers and increased need among their existing customers.
One thing that I come away with when I read articles describing this problem is how often they detail individuals who have been diagnosed with diabetes but can no longer afford to pay for appropriate food for their condition.
As a person with a family history of diabetes, and someone who has been actively avoiding sugars and carbs to control my blood sugar for the past couple of years, I have a tremendous amount of sympathy for these struggling people.
Let me put it another way. Eating well in this country is expensive, and I’ve had to spend real money on food here in New York City to avoid sugary and fast carb-laden food. I don’t think I could have done that on a skimpy food budget. It’s especially hard to imagine budgeting healthy food on a withering food stamp budget.
Because here’s the thing, and it’s not a secret: shitty food is cheap. If I need to buy lots of food (read: calories) for a small amount of money, I can do it easily, but it will be hell for my blood sugar control. I’m guessing I’d be a full-blown diabetic by now if I were poor and on food stamps.
And that brings me to my nerd question of the morning. How much money are we really saving by decreasing the food stamp allowance in this country, if we consider how many more people will be diagnosed diabetic as a result of the decreased quality of their diet? And how many people’s diabetes will get worse, and how much will that cost?
It’s not over, either: apparently more cuts are coming over the next 10 years (maybe by $4 billion, maybe by $40 billion). And although diabetes care costs have gone up 40% in the last 5 years ($245 billion in 2012 from $174 billion in 2007), that doesn’t mean they won’t go up way more in the next 10.
I’m not an expert on how this all works, but the scale is right – we’re talking billions of dollars nationally, so not small potatoes, and of course we’re also talking about people’s quality of life. Never mind in a moral context – I’m definitely of the mind that people should be able to eat – I’m wondering if the food stamp cuts make sense in a dollars and cents context.
Please tell me if you know of an analysis in this direction.
What are Walmart’s wage incentives?
Suppose you have a employer – think Walmart – that has a ubiquitous presence nationally – think the United States. Is there ever a point, as that employer grows in size and employs more and more of the nation, that its agenda becomes aligned with the national agenda of prosperity and well-being?
That’s the interesting assumption behind a recent Guardian article called Walmart and Downton Abbey: rampant inequality and detachment from reality written by Sadhbh Walshe. From the article:
The best thing the top brass at Walmart could do to preserve their own privileged status would be to raise wages for their workers. A recent study by the progressive thinktank Demos illustrated that the company could afford to pay its workers an additional $5.83 an hour (pdf), enough to bring their wages just above the poverty level, simply by ending the company’s share-buyback program. This way prices could stay as they are but sales would increase as more workers would have more money to spend.
This comes up a lot in my Occupy group as well – the idea that raising wages would be good for low-wage companies like McDonalds and Walmart. The question I have is, is it true? (Aside to readers: if you’re aware of a paper that does this analysis, please tell me!) My guess is no.
Let’s put it this way. If I’m a small company then it’s pretty clear I don’t want to raise wages if I don’t have to. The lower the wages for my workers are, the more I get to keep or spend on other things. But as I grow in size, it might actually make sense, depending on context. If I employ 50% of the population, which is indeed an enormous number of people, then how much I pay them goes straight to the bottomline of how much they spend at my store. But again, it all depends on context.
And there’s and important bit of context going on here, which was beautifully explained recently by Bloomberg columnist Barry Ritholtz in his column entitled How McDonald’s and Wal-Mart Became Welfare Queens. From that article:
Wal-Mart, the nation’s largest private sector employer, is also the biggest consumer of taxpayer supported aid. According to Florida Congressman Alan Grayson, in many states, Wal-Mart employees are the largest group of Medicaid recipients. They are also the single biggest group of food stamp recipients. Wal-mart’s “associates” are paid so little, according to Grayson, that they receive $1,000 on average in public assistance. These amount to massive taxpayer subsidies for private companies.
My point is this. Given their welfare queen status, the agenda of Walmart and other huge low-wage employers is not actually aligned with the nation’s. As long as it can lower wages below poverty level, leaving Uncle Sam to make up some of the difference, and either pocket that difference or distribute them to their shareholders, they’ll continue to do so. The incentives are wrong.
And instead of appealing to their greed and telling them it’s in their best interest to raise wages, which I don’t believe is true (but I’d love to be wrong!), we need to raise political awareness about how the system actually works. And my guess is we should either raise the minimum wage – and then tie it to inflation – or establish a basic income guarantee.
Protest JP Morgan at noon on Wednesday #OccupyJPM #OWS
I’m looking forward to protesting in front of JP Morgan with my #OWS Alt Banking group this Wednesday at noon. The exact location is 270 Park Avenue, near 48th Street.
It’s part of a “Week of Action” being put together by a broad coalition of activist and labor groups here in New York. The overall theme of the week is to try to communicate to New Yorkers, in this time of transition from Bloomberg to de Blasio, that we can effect positive change in our city. The theme of the day on Wednesday, at least for us, is to “be in the know,” which makes it a bit more positive than other protests we’ve been part of.
I think this makes sense. There’s so much widespread distrust and hatred of the big banks at this point that I feel like Occupy’s role has gone from provoking people to be outraged to provoking people to be hopeful. Hopeful about the fact that things could be a whole lot better than this, if we work together.
Anyhoo, we spent yesterday planning the action, and made some signs. Here’s one based on an idea we borrowed from Alexis Goldstein from her recent twitter war with JPMorgan:
and here’s a sign we’ll hold up while playing a “rigged game” with props:
I also made a sign that referenced the London Whale and the risk model, but someone said we might need to give people a copy of our recent book, Occupy Finance, just to understand that sign. Sigh.
The facebook page is here, please share it with people who may be able to join us Wednesday!
Aunt Pythia’s advice: the long and boring edition
If you’re feeling anything like Aunt Pythia is feeling, you don’t want to even look at any food that has been peeled, baked, poured into a pie crust, mashed with butter, or stuffed into a turkey. It’s chopped cucumbers and raw apples from here on out, with plentiful brisk walks in the sunshine. Yes or no?
And also, is it just me, or has it been approximately 40 years since Aunt Pythia’s last column? Or is that just measured in “dishes done” years?
Before Aunt Pythia gets down to the advice part of the column, which is particularly long and boring and for which she apologizes, she wants to draw attention to the Black Friday protests that many of her Occupy friends took part in yesterday in Secaucus, New Jersey at Walmart.
It was a national day of Walmart protests, but here in Secaucus we had a large Occupy presence – note my friend Marni, who is holding up a deceased Walmart employee. More pictures are available here.
Now down to business. As you enjoy today’s column (or as you nod your way through it, as the case may be),
please, think of something to ask Aunt Pythia at the bottom of the page!
By the way, if you don’t know what the hell Aunt Pythia is talking about, go here for past advice columns and here for an explanation of the name Pythia.
——
Dear Aunt Pythia,
Some cosmological theories talk about the universe as being a mathematical space, rather than a pile of floating rocks and other real stuff. I always thought that mathematics consisted of rules for writing squiggles on bits of paper that sometimes produced sets of squiggles that corresponded with the real out-there stuff.
Am I a tiny part of the solution to a humungous equation? I’m happy with being made of fundamental particles but this is something else. What’s your take? Are there any practical consequences?
Perplexed
Dear Perplexed,
I’m no philosopher, but as a mathematician I’m here to tell you that mathematics doesn’t describe the universe. It’s at most used as a tool to understand certain parts of the universe, but only at the level of an approximation.
So for example, there’s no such thing as a circle in reality. It’s an idealized shape we use in mathematics that comes in super handy for various reasons, but because actual matter is made up of stuff, there’s never going to be a true circle, except in our brains. You can extend that concept of approximation to other mathematical models of the universe as well, at least as far as I understand it (Peter, please correct me if I’m wrong here!).
As far as each of us being a tiny part of a solution to a humungous equation, it all depends on how you look at it. I’m sure I can set up an equation that would dictate how many children my parents had, and then by construction I’d be in some sense a part of the solution to that. If you’re thinking more metaphysical than that, I can’t help you, and I doubt it would be more meaningful than that, although it might be wrapped up in fancier wrappers.
Aunt Pythia
——
Dear Aunt Pythia,
I am mulling reinventing myself (again). I used to be a lawyer, but it was entirely too demanding and inhumane to manage that and a family life. So I downshifted. Now I use the law degree writing for a legal publication. It’s a 40 hour work week, I have awesome benefits, I am able to take my kids (including the special needs one) to all their crap.
There are only two problems with this nirvana: I don’t make a lot and I’m bored off my butt. My spouse is, unfortunately, utterly useless at home and has proven that he (1) will always put work first, period – because he thinks he’s making the world a better place, and (2) won’t sell out and make some money, because that would be evil. Look, it’s a package deal and that’s who he is, apparently.
This leaves me with a conundrum, and I’m getting quite tired of being poor. One needs a tutor and one needs more behavioral therapy, and I’m not sure where that money is going to come from. I can go and take a government counsel job, I believe that I can get one, and make substantially more than I make now. Like, twice as much roughly. The hours will be a little worse, the commute will be a lot worse. All told, I figure I’d lose 2 extra hours a day, at least.
There’s no guarantee I’ll like it, of course, but I know I’m bored with the current job. And it has no room for growth. I wonder if I’m better off trying to take a second job or make money contracting rather than going whole hog and jumping careers again? I’ve been where I am about 3 years.
Considering Aunt Pythia has jumped ship quite a few times with her skill set, I’m curious if she has insight for me.
Proudly OK on Rent, But Otherwise Rarely Excited Daily
Dear POoR, BORED,
First, I appreciate your sign-off, and second, you were seriously bumping up against the length limit but the sign-off got you through.
And I get needing to prioritize your kids, but I’m going to take issue with two things: your hubby and your boredom.
First, your boredom. Not cool. You need to be interested in your own life, and being bored off your butt is seriously not cutting it. Be more selfish than that, and do it for your kids. They need a mom who’s also a role model. Go find a better job, that pays enough for your needs and that interests you.
Second, your husband. Also not cool that he’s “utterly useless at home,” both because you are wasting time resenting him and because you genuinely need his help. And don’t give me that “because he’s saving the world” crap. He’s not helpful because he’s gotten away with not being helpful. It’s a deal you made with him, possibly (probably) without thinking enough about it. Time to renegotiate. Oh, and renegotiating shitty deals that don’t work for you is also a good role modeling opportunity for your kids.
Here’s how I’d work this. Sit your husband down when the two of you have time, on a weekend evening after the kids are asleep, and tell him you’re bored, need a more challenging job, and that will mean he needs to help out with the house and the kids, because chances are your new job will have more commuting time or whatever.
Next, explain how you’ve worked out the schedule for both of you (if you need to), or ask him to help work it out with you so that it all works. Don’t ask him for help like he’s got an option, because you need this, and that means the family needs this. You guys are a team, and teams work together to make things work.
Good luck!
Aunt Pythia
——
Dear Aunt Pythia,
Two economists have recently posted a paper arguing that Fields Medalists’ productivity decreases after they get their Fields Medal. While this certainly seems plausible psychologically (after all, proving minor theorems might seem anti-climactic after you’ve solved the major open problem in your field), when I looked at this paper, it seemed that the statistics in it were very naive, in that the authors completely ignored any possible post-conditioning; this leads me to believe that the conclusion is quite likely to be wrong. I have two questions:
Are the statistics used in most economics papers this poor, and if so, how can we trust economists to run our economy?
Would it be worth redoing the statistics in this paper to show up these economists, and maybe to defend Fields Medalists against their charges of being lazy?
Burnt-Out Prizewinner
Dear BOP,
First of all, without even reading the paper I’d say we shouldn’t trust economists to run our economy. They have already proved their vested interests are too distracting for such a responsibility.
Second, I scanned the paper, and I’m not very interested in their model but I am kind of interested in their appendix, where they have the following graph:
After all, I want to see the data, and here it is. Look carefully at the comparison group for the Fields Medalists: people who won another big award besides the Fields Medal and have “above-media per-year citations” during the eligibility period.
My question is, why did they include that second part about citations? It’s muddying the waters, for me at least. Did the actual winners also have above-median per-year citations? Are we assuming that the Fields Medal committee uses that as a criterion for eligibility? It’s weird, and I think the data would be cleaner without that stipulation: we’d just be comparing Fields Medalists versus “other” medalists. Now I’m thinking we’re cherry picking. After all, I can imagine that people who get lots of citations are also like to write more papers.
Next, I’d like to see the data on the individual basis or in some way see what kind of error bars we’re talking about here. The fact that there’s a three-year average in the above graph tells me this data is somewhat noisy. Plus the fact that the three-year average is centered on the middle year is weird. All graphs should reflect data known by a certain date.
But finally, I’m willing to ask, who cares? I guess I don’t care about awards in math much, but even if I did, I’m not willing to agree that the whole point of giving out Fields Medals is to “encourage further achievement” on the part of the recipient, even if Fields himself said that. I’ma go with the other reason, which is to get people to compete against each other (yuck).
Whatever, it’s not like you’re going to get a second Fields Medal or something. If you were doing stuff in order to win a Fields Medal, then after getting it, you’d stop, right, and do something else that’s interesting? Makes sense to me.
Aunt Pythia
——
Dear Aunt Pythia,
I agree with your answers to recent question that often enough folks’ criticism of others stems from their own insecurities. Meditating on the fact was making me depressed, until I found an uneaten cupcake in the cupboard, and then thought: do you think that it also works the other way around?
Even if it doesn’t, your site is da bomb! <fingers crossed>.
Mmmm…a cupcake
Dear Mmmm,
First of all, I don’t trust cupcakes in cupboards. At least at my house that signals something very very wrong, that a cupcake made it into a cupboard. Shoulda at most made it onto the counter. Most cupcakes don’t make it out of the shopping bag around here.
Second of all, are you crossing your fingers because you’re hoping to get your question into the column? Or is it because you’re hoping mathbabe.org really is da bomb? Cuz it is, so your hopes have been fully realized.
Next, to your question. By “the other way around,” I’m going to guess you mean the following: when people are insecure about something, they accuse others of having that flaw. The answer is yes, absolutely.
In fact it’s generally true that when someone is sensitized to an issue, even if it’s not one of insecurity, then they see it everywhere, all the time, as if for the first time. There’s a running joke in my family that whenever someone starts a sentence with “Have you noticed lately that…” then what follows that will be a selection bias in exactly that way. So, have you noticed lately that everyone is wearing incredibly awesome flannel shirts? That’s because I got comfort on da mind over here.
Anyhoo, same thing for insecurities. If one is feeling like one’s acne is out of control, one sees other people’s pimples a mile away. If I am ashamed of myself for being overly bossy, then I see overbearing behavior everywhere and I can’t understand how people can stand it. And although we do our best to not accuse people of stuff we’re aware of being sensitized to, it doesn’t always work out that way.
Best,
Aunt Pythia
——
Please submit your well-specified, fun-loving, cleverly-abbreviated question to Aunt Pythia!
“People analytics” embeds old cultural problems in new mathematical models
Today I’d like to discuss recent article from the Atlantic entitled “They’re watching you at work” (hat tip Deb Gieringer).
In the article they describe what they call “people analytics,” which refers to the new suite of managerial tools meant to help find and evaluate employees of firms. The first generation of this stuff happened in the 1950’s, and relied on stuff like personality tests. It didn’t seem to work very well and people stopped using it.
But maybe this new generation of big data models can be super useful? Maybe they will give us an awesome way of throwing away people who won’t work out more efficiently and keeping those who will?
Here’s an example from the article. Royal Dutch Shell sources ideas for “business disruption” and wants to know which ideas to look into. There’s an app for that, apparently, written by a Silicon Valley start-up called Knack.
Specifically, Knack had a bunch of the ideamakers play a video game, and they presumably also were given training data on which ideas historically worked out. Knack developed a model and was able to give Royal Dutch Shell a template for which ideas to pursue in the future based on the personality of the ideamakers.
From the perspective of Royal Dutch Shell, this represents huge timesaving. But from my perspective it means that whatever process the dudes at Royal Dutch Shell developed for vetting their ideas has now been effectively set in stone, at least for as long as the algorithm is being used.
I’m not saying they won’t save time, they very well might. I’m saying that, whatever their process used to be, it’s now embedded in an algorithm. So if they gave preference to a certain kind of arrogance, maybe because the people in charge of vetting identified with that, then the algorithm has encoded it.
One consequence is that they might very well pass on really excellent ideas that happened to have come from a modest person – no discussion necessary on what kind of people are being invisible ignored in such a set-up. Another consequence is that they will believe their process is now objective because it’s living inside a mathematical model.
The article compares this to the “blind auditions” for orchestras example, where people are kept behind a curtain so that the listeners don’t give extra consideration to their friends. Famously, the consequence of blind auditions has been way more women in orchestras. But that’s an extremely misleading comparison to the above algorithmic hiring software, and here’s why.
In the blind auditions case, the people measuring the musician’s ability have committed themselves to exactly one clean definition of readiness for being a member of the orchestra, namely the sound of the person playing the instrument. And they accept or deny someone, sight unseen, based solely on that evaluation metric.
Whereas with the idea-vetting process above, the training data consisted of “previous winners” which presumable had to go through a series of meetings and convince everyone in the meeting that their idea had merit, and that they could manage the team to try it out, and all sorts of other things. Their success relied, in other words, on a community’s support of their idea and their ability to command that support.
In other words, imagine that, instead of listening to someone playing trombone behind a curtain, their evaluation metric was to compare a given musician to other musicians that had already played in a similar orchestra and, just to make it super success-based, had made first seat.
That you’d have a very different selection criterion, and a very different algorithm. It would be based on all sorts of personality issues, and community bias and buy-in issues. In particular you’d still have way more men.
The fundamental difference here is one of transparency. In the blind auditions case, everyone agrees beforehand to judge on a single transparent and appealing dimension. In the black box algorithms case, you’re not sure what you’re judging things on, but you can see when a candidate comes along that is somehow “like previous winners.”
One of the most frustrating things about this industry of hiring algorithms is how unlikely it is to actively fail. It will save time for its users, since after all computers can efficiently throw away “people who aren’t like people who have succeeded in your culture or process” once they’ve been told what that means.
The most obvious consequence of using this model, for the companies that use it, is that they’ll get more and more people just like the people they already have. And that’s surprisingly unnoticeable for people in such companies.
My conclusion is that these algorithms don’t make things objective, they makes things opaque. And they embeds our old cultural problems in new mathematical models, giving us a false badge of objectivity.
Today I’m thankful for the pope
I’m not religious but I think Pope Francis is an awesome and inspiring thinker and leader. And yes, I’ve invited him to join Occupy. Here’s an excerpt from his recent Apostolic Exhortation in case you haven’t seen it yet:
No to an economy of exclusion
53. Just as the commandment “Thou shalt not kill” sets a clear limit in order to safeguard the value of human life, today we also have to say “thou shalt not” to an economy of exclusion and inequality. Such an economy kills. How can it be that it is not a news item when an elderly homeless person dies of exposure, but it is news when the stock market loses two points? This is a case of exclusion. Can we continue to stand by when food is thrown away while people are starving? This is a case of inequality. Today everything comes under the laws of competition and the survival of the fittest, where the powerful feed upon the powerless. As a consequence, masses of people find themselves excluded and marginalized: without work, without possibilities, without any means of escape.
Human beings are themselves considered consumer goods to be used and then discarded. We have created a “disposable” culture which is now spreading. It is no longer simply about exploitation and oppression, but something new. Exclusion ultimately has to do with what it means to be a part of the society in which we live; those excluded are no longer society’s underside or its fringes or its disenfranchised – they are no longer even a part of it. The excluded are not the “exploited” but the outcast, the “leftovers”.
54. In this context, some people continue to defend trickle-down theories which assume that economic growth, encouraged by a free market, will inevitably succeed in bringing about greater justice and inclusiveness in the world. This opinion, which has never been confirmed by the facts, expresses a crude and naïve trust in the goodness of those wielding economic power and in the sacralized workings of the prevailing economic system. Meanwhile, the excluded are still waiting. To sustain a lifestyle which excludes others, or to sustain enthusiasm for that selfish ideal, a globalization of indifference has developed. Almost without being aware of it, we end up being incapable of feeling compassion at the outcry of the poor, weeping for other people’s pain, and feeling a need to help them, as though all this were someone else’s responsibility and not our own. The culture of prosperity deadens us; we are thrilled if the market offers us something new to purchase; and in the meantime all those lives stunted for lack of opportunity seem a mere spectacle; they fail to move us.
No to the new idolatry of money
55. One cause of this situation is found in our relationship with money, since we calmly accept its dominion over ourselves and our societies. The current financial crisis can make us overlook the fact that it originated in a profound human crisis: the denial of the primacy of the human person! We have created new idols. The worship of the ancient golden calf (cf. Ex 32:1-35) has returned in a new and ruthless guise in the idolatry of money and the dictatorship of an impersonal economy lacking a truly human purpose. The worldwide crisis affecting finance and the economy lays bare their imbalances and, above all, their lack of real concern for human beings; man is reduced to one of his needs alone: consumption.
56. While the earnings of a minority are growing exponentially, so too is the gap separating the majority from the prosperity enjoyed by those happy few. This imbalance is the result of ideologies which defend the absolute autonomy of the marketplace and financial speculation. Consequently, they reject the right of states, charged with vigilance for the common good, to exercise any form of control. A new tyranny is thus born, invisible and often virtual, which unilaterally and relentlessly imposes its own laws and rules. Debt and the accumulation of interest also make it difficult for countries to realize the potential of their own economies and keep citizens from enjoying their real purchasing power. To all this we can add widespread corruption and self-serving tax evasion, which have taken on worldwide dimensions. The thirst for power and possessions knows no limits. In this system, which tends to devour everything which stands in the way of increased profits, whatever is fragile, like the environment, is defenseless before the interests of a deified market, which become the only rule.
No to a financial system which rules rather than serves
57. Behind this attitude lurks a rejection of ethics and a rejection of God. Ethics has come to be viewed with a certain scornful derision. It is seen as counterproductive, too human, because it makes money and power relative. It is felt to be a threat, since it condemns the manipulation and debasement of the person. In effect, ethics leads to a God who calls for a committed response which is outside of the categories of the marketplace. When these latter are absolutized, God can only be seen as uncontrollable, unmanageable, even dangerous, since he calls human beings to their full realization and to freedom from all forms of enslavement. Ethics – a non-ideological ethics – would make it possible to bring about balance and a more humane social order. With this in mind, I encourage financial experts and political leaders to ponder the words of one of the sages of antiquity: “Not to share one’s wealth with the poor is to steal from them and to take away their livelihood. It is not our own goods which we hold, but theirs”.[55]
58. A financial reform open to such ethical considerations would require a vigorous change of approach on the part of political leaders. I urge them to face this challenge with determination and an eye to the future, while not ignoring, of course, the specifics of each case. Money must serve, not rule! The Pope loves everyone, rich and poor alike, but he is obliged in the name of Christ to remind all that the rich must help, respect and promote the poor. I exhort you to generous solidarity and a return of economics and finance to an ethical approach which favours human beings.
No to the inequality which spawns violence
59. Today in many places we hear a call for greater security. But until exclusion and inequality in society and between peoples is reversed, it will be impossible to eliminate violence. The poor and the poorer peoples are accused of violence, yet without equal opportunities the different forms of aggression and conflict will find a fertile terrain for growth and eventually explode. When a society – whether local, national or global – is willing to leave a part of itself on the fringes, no political programmes or resources spent on law enforcement or surveillance systems can indefinitely guarantee tranquility. This is not the case simply because inequality provokes a violent reaction from those excluded from the system, but because the socioeconomic system is unjust at its root. Just as goodness tends to spread, the toleration of evil, which is injustice, tends to expand its baneful influence and quietly to undermine any political and social system, no matter how solid it may appear. If every action has its consequences, an evil embedded in the structures of a society has a constant potential for disintegration and death. It is evil crystallized in unjust social structures, which cannot be the basis of hope for a better future. We are far from the so-called “end of history”, since the conditions for a sustainable and peaceful development have not yet been adequately articulated and realized.
60. Today’s economic mechanisms promote inordinate consumption, yet it is evident that unbridled consumerism combined with inequality proves doubly damaging to the social fabric. Inequality eventually engenders a violence which recourse to arms cannot and never will be able to resolve. This serves only to offer false hopes to those clamouring for heightened security, even though nowadays we know that weapons and violence, rather than providing solutions, create new and more serious conflicts. Some simply content themselves with blaming the poor and the poorer countries themselves for their troubles; indulging in unwarranted generalizations, they claim that the solution is an “education” that would tranquilize them, making them tame and harmless. All this becomes even more exasperating for the marginalized in the light of the widespread and deeply rooted corruption found in many countries – in their governments, businesses and institutions – whatever the political ideology of their leaders.
Cool open-source models?
I’m looking to develop my idea of open models, which I motivated here and started to describe here. I wrote the post in March 2012, but the need for such a platform has only become more obvious.
I’m lucky to be working with a super fantastic python guy on this, and the details are under wraps, but let’s just say it’s exciting.
So I’m looking to showcase a few good models to start with, preferably in python, but the critical ingredient is that they’re open source. They don’t have to be great, because the point is to see their flaws and possible to improve them.
- For example, I put in a FOIA request a couple of days ago to get the current teacher value-added model from New York City.
- A friends of mine, Marc Joffe, has an open source municipal credit rating model. It’s not in python but I’m hopeful we can work with it anyway.
- I’m in search of an open source credit scoring model for individuals. Does anyone know of something like that?
- They don’t have to be creepy! How about a Nate Silver – style weather model?
- Or something that relies on open government data?
- Can we get the Reinhart-Rogoff model?
The idea here is to get the model, not necessarily the data (although even better if it can be attached to data and updated regularly). And once we get a model, we’d build interactives with the model (like this one), or at least the tools to do so, so other people could build them.
At its core, the point of open models is this: you don’t really know what a model does until you can interact with it. You don’t know if a model is robust unless you can fiddle with its parameters and check. And finally, you don’t know if a model is best possible unless you’ve let people try to improve it.
Twitter and its modeling war
I often talk about the modeling war, and I usually mean the one where the modelers are on one side and the public is on the other. The modelers are working hard trying to convince or trick the public into clicking or buying or consuming or taking out loans or buying insurance, and the public is on the other, barely aware that they’re engaging in anything at all resembling a war.
But there are plenty of other modeling wars that are being fought by two sides which are both sophisticated. To name a couple, Anonymous versus the NSA and Anonymous versus itself.
Here’s another, and it’s kind of bland but pretty simple: Twitter bots versus Twitter.
This war arose from the fact that people care about how many followers someone on Twitter has. It’s a measure of a person’s influence, albeit a crappy one for various reasons (and not just because it’s being gamed).
The high impact of the follower count means it’s in a wannabe celebrity’s best interest to juice their follower numbers, which introduces the idea of fake twitter accounts to game the model. This is an industry in itself, and an associated arms race of spam filters to get rid of them. The question is, who’s winning this arms race and why?
Twitter has historically made some strides in finding and removing such fake accounts with the help of some modelers who actually bought the services of a spammer and looked carefully at what their money bought them. Recently though, at least according to this WSJ article, it looks like Twitter has spent less energy pursuing the spammers.
It begs the question, why? After all, Twitter has a lot theoretically at stake. Namely, its reputation, because if everyone knows how gamed the system is, they’ll stop trusting it. On the other hand, that argument only really holds if people have something else to use instead as a better proxy of influence.
Even so, considering that Twitter has a bazillion dollars in the bank right now, you’d think they’d spend a few hundred thousand a year to prevent their reputation from being too tarnished. And maybe they’re doing that, but the spammers seem to be happily working away in spite of that.
And judging from my experience on Twitter recently, there are plenty of active spammers which actively degrade the user experience. That brings up my final point, which is that the lack of competition argument at some point gives way to the “I don’t want to be spammed” user experience argument. At some point, if Twitter doesn’t maintain standards, people will just not spend time on Twitter, and its proxy of influence will fall out of favor for that more fundamental reason.
Aunt Pythia’s advice
Aunt Pythia is hung over from excess rabble rousing and karaoke, but she’s determined not to miss another week of her beloved advice column. Aunt Pythia has missed you! As I’m sure you’ve missed her! Please enjoy today’s column, and
please, don’t forget to ask Aunt Pythia a question at the bottom of the page!
By the way, if you don’t know what the hell Aunt Pythia is talking about, go here for past advice columns and here for an explanation of the name Pythia.
——
Dear Aunt Pythia,
I have an older sister. She’s a lovely and good person. Very generous, very friendly. And very assertive, in an oldest-child-in-the-family way.
I love my family, but I often feel depressed and suffocated when I’m around my sister. Is it because I feel she’s constantly giving input on how I could do things differently, and why she’s chosen to do things the way she does as if she has a PhD on the subject, and I often am left doubting my own abilities to make decisions even though I know that in reality I have a pretty good head on my shoulders? Maybe. Is it because of the authority she speaks on any topic, even ones she knows very little about? Is it because she doesn’t seem to entertain the possibility that anyone else could have anything to add in terms of input? Is it because she rarely shows any kind of vulnerability? Is it because she’s so assertive that it often feels like she’s taking up all of the oxygen in the room? Is it because she does all of these things even while, at the same, she is being utterly helpful, generous, and selfless in most other ways? Yeah, maybe that too.
Whatever it is, it hardly seems like a good reason to get depressed or to distance myself from someone who genuinely loves me and whom I love. I get that this is my issue, and the problem is how I feel about myself when I’m around her. I want to get over this. I just don’t know where to start.
Family Stuff
Dear Family Stuff,
To be honest I double- and triple-checked that I don’t have any younger siblings when I read this, because it could be about me. I could totally be that older sister, and I imagine that many people feel this way about me.
But if I’m right, and if your sister is a lot like me, then I don’t think it’s “your issue” to get over. I’m guessing it’s more like a series of signals that she’s giving out that are not hitting the intended targets. And if I’m right, she actually does want you to add stuff, but she expects you to jump right into the ring and not need an invitation.
So, when she gives advice, think of her words as her unedited thoughts, and do with those thoughts what you may. You can test this theory by every now and then pointing out, “I tried that already, it didn’t work” and see what she says. If she’s like, “Oh cool, how about trying this?” then you know she’s just taking stabs.
And, when she has an opinion on everything, maybe she’s just trying to engage in a provocative conversation and wants to be challenged. I do that all the time (duh). So next time she says something that sounds uninformed, say something like, “Hey that sounds wrong to me – should we check the facts?” and see how she reacts. She might be psyched for the challenge and for the chance to learn something interesting.
As for your sister showing no vulnerability. The funny thing about family is, we are our most vulnerable with our family, and yet we are also very comfortable with them, because we know them so well. You might be surprised by how vulnerable she really is. At the same time, you might not want to test this one, because it’s usually a negative experience to expose vulnerability in someone else. In any case my advice here is to not assume an entire lack of vulnerability around family, even if it looks like that.
Last piece of advice: go read my recent post called “Cathy’s Wager.” It’s about how to react when people are treating you not-so-nicely. I think it’s relevant here, because the overall point is that it’s not about you. Your sister is who she is and she’s very likely not doing all this stuff in order to make you feel stifled and depressed. She’s a know-it-all loudmouth, true, but the sooner you can either get on her wavelength (see above tips) or roll your eyes and love her in spite of her pushy know-it-all ways the better for you and for her. Don’t take it personally.
Either that or just never see her again. That’s totally fine too, honestly. I don’t agree that you have to hang out with family, unless possibly if they’re dying or in need.
I hope that helps!
Aunt Pythia
——
Aunt Pythia,
What ever happened to the proof of the ABC conjecture by Mochizuki that you talked about a year ago?
Thanks,
Curious
Curious,
I unfortunately missed him when he came to Columbia, but Brian Conrad recently came and updated the math community on the status of the alleged proof. I believe the bottomline was that it has not been confirmed by anyone. So, I’d say this means it’s not a proof.
AP
——
Dear Aunt Pythia,
This is a longtime worry of mine. Since you are a master of both abstract as well as the quantitative, let me query you regarding the deep connection that seems to exist between the two. To put it simply, the question is, “Does Size Matter?” More precisely, does Size influence tender feelings of the heart?
Sizeable Confusion
Dear SC,
I’d guess about as much as anything else physical, like boob size or leg length. In other words it might be a pretty big deal initially, as in during the first few minutes, but then when real love sets in it’s a total non-issue.
AP
——
Dear Aunt Pythia,
Many government officials testified that there is no way for them to tell how many people signed up for Obamacare. Can extracting the data from the website be that complicated? I am worried and lost.
Worried about Obamacare
Dear Worried,
Well many people have been busy counting this stuff since you submitted that question, and the final number for the first day of Obamacare seems to be 6. Given how small that number is, I’m going to assume it wasn’t that hard to count, or at least approximate at “0”. In other words, it might have been a political decision to repress the actual number.
On the other hand, engineering large-scale systems is actually pretty complicated, and it might not make sense to have a single repository to put all the enrollment figures – who knows, and I didn’t design this system, so I don’t – so I can imagine that it was actually non-trivial to figure out the answer to this question.
By the way, I’m planning to write some posts on how we are increasingly seeing pure engineering issues become political issues. There’s Knight Capital’s trading mistakes, then there’s Obamacare. Those are just two, but my theory is that they are just the beginning of a very long list. The nerds are taking over, in other words, or at least their mistakes are.
Aunt Pythia
——
Dear Aunt Pythia,
I sent you a question a few weeks back and you didn’t answer it (which is completely fine). What is your criteria for answering a question or not? Maybe your answer might help me rewrite my question in a way that suits you better.
Socially Awkward Dude
Dear SAD,
Here’s the thing. I’m pretty desperate over here, what with a pretty short list of questions, and a stubborn refusal on my part to make up fake questions (although I do accept other people’s fake questions!). So there had to be something about your question that didn’t sit well with me. Here are some possibilities as to why:
- The question was something I couldn’t answer, because it required expertise I don’t have.
- The question was really long and not easily edited down to something shortish.
- The question wasn’t really a question, just a rambling speech.
- The question was spam.
- The question was verbally abusive towards me.
- The question struck me as disingenuous in some way.
- The question is a lot like other questions I’ve already answered (note to the 40 people asking me how to become a data scientist: read my book called Doing Data Science!)
I have no idea which question was yours, but if you’d care to resubmit, making sure it’s to the point, has a specific and earnest question, and is about something I have knowledge about, then I’m guessing it will get through.
I hope that helps!
Auntie P
——
Dear Aunt Pythia,
I am a late-20’s data scientist (working at a large non-tech company) about to apply for Ph.D. programs in machine learning. My reason for doing this is two-fold. One, I enjoy research and feel that I can contribute to humanity through scholarship, even if the contribution may be small. Two, I’ve grown disillusioned with working in a corporate environment – it seems like one needs to be more of a politician than a genuinely nice and high-performing individual to be recognized. But I realize this is partly due to the size of my organization (are start-ups any different?).
However, I’ve heard people tell me that academia is no different. Given the publish-or-perish paradigm, people are more interested in how many citations they have than they are about truly advancing human knowledge (for example, this was a depressing read).
You transitioned from academia into industry. Do you have any advice for someone who’s trying to make the opposite transition?
Naively Bayesian
Dear NB,
First of all, start-ups are sometimes different, although they work you really hard and often expect you to sleep under your desk. This might not work for you, but it might be worth it if you get to have influence. Also, I’d suggest going with a very small start-up: as soon as there are like 60 people, your potential influence typically gets pretty miniscule.
Second, my motto is “You never get rid of your problems, you just get a new set of problems.” So it’s more a question of which kinds of problems suit your personality than anything else.
But there’s one thing I can assure you: there’s politics everywhere. You’re not getting away from that, so if you’re really allergic to politics, I suggest you find a place where you can safely ignore that stuff, like maybe in a cave in the woods.
But seriously, I’d suggest you talk to a lot of people and see what kind of problems are there, without exaggerating them too much (I feel like that link is too aggressive for example, although there are grains of truth in it). And most importantly, try to find something to do that actually interests you in an intellectual way so you can become absorbed in your own sense of curiosity and shut out the real world at least once a day. Good luck!
Aunt Pythia
——
Please submit your well-specified, fun-loving, cleverly-abbreviated question to Aunt Pythia!
People don’t get fired enough
This might surprise some of you – or not, I’m not sure. But one of the most satisfying things about leaving academia and the tenure system and going into industry is how, at least in the ideal situation, you can get fired for not doing your job.
In fact, one of the reasons I decided to leave academia is that I really thought some of my colleagues weren’t doing right by the undergraduates, and the frustrating thing was that there was essentially no way to force them to start. Tenure has great aspects and not-so-great aspects, and a total lack of leverage is not a great one. I feel for deans sometimes.
Here’s the dirty little secret of lots of industry jobs, though: lots of time people also don’t get fired when they should. And sometimes it’s super awful bullies who yell and scream and act inappropriately but also pull in amazing sales numbers. There are things like that, of course. That’s the example of how they don’t abide by the alleged social contract but they perform on the bottomline. Social contracts are hard to quantify and somewhat squishy. You see people getting away with stuff because they’re rainmakers or higher ups.
But there are also plenty of examples of people just not doing their job, and having super awful attitudes, or even just completely apathetic attitudes, and for whatever reason they don’t get fired. This demoralizes and irritates and distracts everyone around them, because they all resent the free-rider.
Plus, retaining people who should by all accounts get fired makes the veneer of the kool-aid drinking camaraderie even more flimsy and scrutinizable – what’s so great about working here if people can just slack off and not care? Why do I give two shits about this project anyway? How does this project in the larger scheme of things? Maybe that scrutiny is a good thing – I engage in it myself – but you don’t want everyone thinking that all the time.
Here’s the thing, before you think I’m super vicious and mean to want people to get fired. These people I’m talking about are generally high skilled and temporarily depressed. They’re in the wrong job. And once fired, they will find another job, which will hopefully be a better one for them. I’m not saying that nobody will ever end up jobless and homeless, but very few, and moreover there are plenty of jobless and homeless people who would be psyched to do that job really well (putting aside how difficult it is for homeless people to get seriously considered for a job).
And I’m not saying you fire people out of the blue. You definitely need to tell people they’re not performing well (or that they are) and keep them in the feedback loop on whether things are working out. But in my experience people who deserve to get fired totally know it and can’t believe their luck that they’ve not been fired yet.
To conclude, I’m going on record saying I kind of agree with Jack Welch on this issue in a way I never thought I would.
Cathy’s Wager
I know you guys might be getting kind of exhausted from all the oversharing that’s been happening on mathbabe this week. I am too. But let me finish the phase with one piece of advice which I hope you find helpful.
I call it “Cathy’s Wager” (h/t Chris Wiggins) in reference to a much more famous and better idea called Pascal’s Wager. That, you may remember, is the argument that you might as well be a good person because there’s either a god, who cares, or there isn’t, in which case you haven’t lost all that much.
So here’s my version, and it refers to how other people treat you and how you react. I’ll assume most people treat you nicely most of the time, and then sometimes someone doesn’t treat you nicely. How do you react?
My theory is that you always assume it’s something they’re going through, and you try to never take it personally. Here are some examples.
- You’re friends with someone and all of a sudden they stop writing back to your emails. Assume they’re going through something, maybe a depression, maybe a break-up, maybe they just fell in love or moved jobs. It’s not about you and you shouldn’t take it personally. Consider writing to them and saying you’re there for them if they need a friend, or just do nothing and let them take their time, depending on how close a friend they are and how likely each of those scenarios is. Err on the side of compassion, not blame.
- You’re trying to set up a meeting with someone professionally and they never get back to you, or even worse, they don’t show up for the planned meeting and never explain why. First, always assume this has nothing to do with you. Maybe the got into a fight with their significant other, maybe they just got fired. You have no idea. But in order to avoid this from happening, do remember to confirm business meetings the day of, if it’s in the afternoon, or the afternoon before, if it’s in the morning, especially if the meeting was made more than a week in advance.
- You have what you think is an interesting if provocative conversation with someone and they never talk to you again, and you hear 2nd or 3rd hand (or both) that they hate your guts. Again, it’s not about you. There was some trigger in that conversation, and yes if you want to be sensitive you could try to go back over the conversation in your head and figure out what the heck happened. Do it once, but if you are convinced you meant no offense, then assume that person is going through something. They might even get over it and want to make up someday. Who knows, maybe part of what they like in life is getting offended and complaining. For example, maybe they take this article to heart entitled “The 14 Habits of Highly Miserable People.”
- Someone gets into your face and tells you you’re an awful person and are being mean to them for whatever reason. Not about you.
As I’m sure you can see, the assumption that “it’s not about me” is super useful and time-saving. I use it a lot, which means I don’t spend a lot of time second guessing myself or trying to change things that I can’t change about other people’s feelings. It’s kind of a selfish version of the Serenity Prayer, if you will, without all the religious stuff.
And this is not to say I don’t spend time trying to mend differences and reach out to my friends! I totally do! I just don’t feel personally affected if it doesn’t work. And I think that actually helps me do it more often in the end.
One caveat: the above examples work pretty well unless you are actually an awful person. I’m assuming you’re not. If you are actually a bad person, please don’t rely on Cathy’s Wager, thanks. Of course that begs the question of whether anyone actually thinks they’re awful, and if you go there, consider the idea that awful people are already using Cathy’s Wager, so you may as well too.
I’m already fat so I may as well be smart
I seem to be in a mood this week for provocative posts about body image and appearance (maybe this is what happens when I skip an Aunt Pythia column). Apologies to people who came for math talk.
I just wanted to mention something positive about the experience of being fat all my life, but especially as a school kid. Because just to be clear, this isn’t a phase. I’ve been pudgy since I was 2 weeks old. And overall it kind of works for me, and I’ll say why.
Namely, being a fat school kid meant that I was so uncool, so outside of normal social activity with boys and the like, that I was freed up to be as smart and as nerdy as I wanted, with very little stress about how that would “look”. You’re already fat, so why not be smart too? You’re not doing anything else, nobody’s paying attention to you, and there’s nothing to gossip about, so might as well join the math team.
It’s really a testament to both the pressure to be thin and the pressure to conform intellectually, i.e. not be a nerd, when you’re a young girl: they are both intense and super unpleasant. The happy truth is, one can be cover for the other. More than that, really: being fat (or “overweight” for people who are squeamish about the word “fat”) has opened up many doors that I honestly think would have, or at least could have, remained shut had I been more socially acceptable.
Going back to dress code at work for a moment: while people claim that corporate dress codes are meant to keep our minds off of sex, that is clearly a huge lie when it comes to many categories of women’s work clothes. Who are we kidding? The mere fact that many women wear high heels to work kind of says it all. And that’s fine, but let’s freaking acknowledge it.
On the other hand, it’s pretty hard to look sexy in a plus-sized suit (although not impossible), and the idea of high heels at work is just nuts. This ends up being a weirdly good thing for me, though: people take me more seriously because I have taken myself out of the sex game altogether – or at least the traditional sex game.
By the way, I’m not saying all fat women have the same perspective on it. I’m lucky enough to have figured out pretty early on how to separate other people’s projected feelings about my body from my own feelings. I am an observer of fat hatred, in other words. That doesn’t make me entirely insulated but it does give me one critical advantage: I have a lot of time on my hands to do stuff that I might otherwise spend fretting about my body.
It also might help partly explain why some girls get on the math team and others don’t. Being fat is something you don’t have control over (the continuing and damaging myth that each person does have control over it notwithstanding) but joining the math team is something you do have control over. And if you aren’t already excluded for some other reason (being fat is one but by no means the only way this could happen of course), you might not want to start that whole thing intentionally. Just a theory.
One reason corporate culture sucks for women
Am I the only person offended by the recent wave of articles wherein “senior women” at corporate offices are going around telling “younger women” about the appropriate dress code?
For example, here’s the beginning of a WSJ piece on just that subject:
Clothes may make the man. Can they undo the woman?
When female employees at Frontier Communications Corp. show up at its headquarters in very short skirts, sweatpants or sneakers, Chief Executive Maggie Wilderotter sometimes pulls them aside for a quick, private chat on dressing for success.
“I want women to be paid attention to for what they say–and not how they look,” explains Ms. Wilderotter.
Later in the article the explain why this is ok:
Women face more pitfalls because they have more clothing choices than men. And because male bosses fear being accused of sexual harassment, it usually falls to female supervisors to confront an associate about her attire.
This is one reason I hate corporate jobs. And yes, it’s because I come from academia and because I’m essentially a hippie, but seriously, why do we need so much policing? Why can’t people just leave each other alone to express themselves? It’s also a double standard:
Rosalind Hudnell, human resources vice president of Intel Corp., occasionally intervenes when she sees young female staffers clad unprofessionally, even though Intel staffers often wear shorts and jeans.
It’s just another in a long list of things you are scrutinized on if you’re a woman. In addition to whether you are a good mother, a feminine-enough-without-being-too-feminine employee, and, as a tertiary issue, if that, whether you actually do your job well. Fuck this.
Question for you readers: what does it really mean that these “senior women” are taking it upon themselves to scrutinize and criticize young women? Am I wrong, is it actually generous? Or is it some kind of hazing thing? Or is it a media invention that doesn’t actually happen?
Debunking Economic Myths #OWS
Yesterday in Alt Banking we were honored to have Suresh Naidu visit us to talk about and debunk economic myths.
The first myth, and the one we spent the most time on, is the idea that people “deserve” the money they earn because it is an accurate measure of their “added value” to society.
There are two parts of this, or actually at least two parts.
First, there’s the idea that you can even dissect the meaning of one person’s value. And if you can, it’s likely a question of a marginal value: what does our society look like without Steve Jobs, and then with him, and what’s the difference between the two worlds? As soon as you say it, you realize that such a thought experiment is complicated, considering the extent to which Steve Jobs’ journey intersected with other people’s like Steve Wozniak and a huge crowd of Chinese workers.
If you think about it some more, you might conclude that the marginal value of a single person is impossible to actually measure, at least with any precision, and not just because of the counterfactual problem, i.e. the problem that we only have one universe and can’t run two parallel universes at the same time. It’s really because any one person succeeds or fails, or more generally contributes, within a context of an entire culture. Even Mozart wrote his symphonies within a cultural context. In another context he would have been a kid who hums to himself a lot.
Second, there’s the assumption that people who earn a lot of money are actually adding value at all. This isn’t clear, and you don’t need to refer to formally criminal acts to make that case (although of course there are plenty of rich people who have committed criminal acts).
In many examples of super rich people, they got that way through not paying for negative externalities like polluting the environment, or because they had control of the legal mechanisms to reap profits off of other peoples’ work. Not technically illegal, then, but also not exactly a fair measure of their added value.
Or, of course, if they worked in finance, they might have made money by keeping stuff incredibly complicated and opaque while providing liquidity to the credit markets. It’s not clear that such work has added any value to society, or if it has, whether it’s balanced the good with the bad.
Some observations about this myth that were brought up include:
- There’s a deep belief in “the markets” at work here which is rather cyclical. The market values you more than other people which is why you’re paid so well for whatever it is you do. Other people who have less to offer the market are get paid less. Anyone who doesn’t have a job doesn’t deserve a job since the market isn’t offering them a job, which must mean they are adding no value.
- There are exceptions where people add obvious value – caretakers of our children for example – but aren’t paid well. This is because of a different mechanism called supply and demand. For whatever reason supply and demand isn’t at work at high ends of the market.
- Or maybe it is and there’s really only one possible person who could do what Steve Jobs did. Personally I don’t buy it. And I chose Steve Jobs because so many people love that guy, but really he’s one of the best examples of someone who might have had a unique talent. Most rich people are generically good at their job and not all that unique.
- It’s mostly the people that benefit from the market system that believe in it. That kind of reminds me of the marshmallow study, or rather one of the many re-interpretations of the marshmallow study. See the latest one here.
- It’s patently difficult to believe in the market system if you consider a lack of equality of opportunity in this country due to extreme differences in school systems and the like. I’m about to start reading this book which explains this issue in depth.
- For other evidence, look at Pimco’s Bill Gross’s recent confessions about being born at the right time with easy access to credit.
- The unequal access of opportunities in this country is becoming increasingly entrenched, and as it does so the myth of the market giving us what we deserve is becoming increasingly difficult to swallow.
Crisis Text Line: Using Data to Help Teens in Crisis
This morning I’m helping out at a datadive event set up by DataKind (apologies to Aunt Pythia lovers).
The idea is that we’re analyzing metadata around a texting hotline for teens in crisis. We’re trying to see if we can use the information we have on these texts (timestamps, character length, topic – which is most often suicide – and outcome reported by both the texter and the counselor) to help the counselors improve their responses.
For example, right now counselors can be in up to 5 conversations at a time – is that too many? Can we figure that out from the data? Is there too much waiting between texts? Other questions are listed here.
Our “hackpad” is located here, and will hopefully be updated like a wiki with results and visuals from the exploration of our group. It looks like we have a pretty amazing group of nerds over here looking into this (mostly python users!), and I’m hopeful that we will be helping the good people at Crisis Text Line.
mathbabe 