Plumping up darts

October 13, 2013 25 comments

Someone asked me a math question the other day and I had fun figuring it out. I thought it would be nice to write it down.

So here’s the problem. You are getting to see sample data and you have to infer the underlying distribution. In fact you happen to know you’re getting draws – which, because I’m a basically violent person, I like to think of as throws of a dart – from a uniform distribution from 0 to some unknown d, and you need to figure out what d is. All you know is your data, so in particular you know how many dart throws you’ve gotten to see so far. Let’s say you’ve seen n draws.

In other words, given x_1, x_2, x_3, \dots, x_n, what’s your best guess for d?

First, in order to simplify, note that all that really matters in terms of the estimate of d is what is max_{i \in \{1, \dots, n\}} (x_i) and how big n is.

Next, note you might as well assume that d=1 and you just don’t know it yet.

With this set-up, you’ve rephrased the question like this: if you throw n darts at the interval [0,1], then where do you expect the right-most dart – the maximum – to land?

It’s obvious from this phrasing that, as n goes to infinity, you can expect a dart to get closer and closer to 1. Moreover, you can look at the simplest case, where n=1, and since the uniform distribution is symmetric, you can see the answer is 1/2. Then you might guess the overall answer, which depends on n and goes to 1 as n goes to infinity, might be n/(n+1). It makes intuitive sense, but how do you prove that?

Start with a small case where you know the answer. For n=1 we just need to know what the expected value of max(x_1) is, and since there’s one dart, the max is just x_1 itself, which is to say we need to compute a simple integral to find the expected value (note it’s coming in handy here that I’ve normalized the interval from 0 to 1 so I don’t have to divide by the width of the interval):

\int_0^1 x \, dx = (x^2/2) |_0^1 = 1/2,

and we recover what we already know. In the next case, we need to integrate over two variables (same comment here, don’t have to divide by area of the 1×1 square base):

\int_0^1 \int_0^1 max(x_1, x_2) \, dx_1 dx_2.

If you think about it, though, x_1 and x_2 play symmetric parts in this matter, so you can assume without loss of generality that x_1 is bigger, as long as we only let x_2 range between 0 and x_1, and then multiply the end result by 2:

 = 2 \int_0^1 \int_0^{x_1} x_1 \, dx_2 dx_1.

But that simplifies to:

= 2 \int_0^1 x_1^2 \, dx = 2 (x_1^3/3) |_0^1 = 2/3.

Let’s do the general case. It’s an n-fold integral over the maximum of all n darts, and again without loss of generality x_1 is the maximum as long as we remember to multiply the whole thing by n. We end up computing:

= n \int_0^1 \int_0^{x_1} \int_0^{x_1} \cdots \int_0^{x_1} x_1 \, dx_n \cdots dx_3 dx_2 dx_1.

But this collapses to:

n \int_0^1 x_1^n \, dx_1 = n (x_1^{n+1}/(n+1)) |_0^1 = n/(n+1).

To finish the original question, take the maximum value in your collection of draws and multiply it by the plumping factor (n+1)/n to get a best estimate of the parameter d.

Categories: math, statistics

Aunt Pythia’s advice

October 12, 2013 10 comments

Hello and good morning, dear Aunt Pythia readers. Aunt Pythia is feeling bright-eyed and bushy tailed this morning and can’t wait to dig into the juicy questions and ethical dilemmas she is sure are awaiting her in her beloved and glamorous google spreadsheet.

Aunt Pythia has taken a few minutes today already to count her blessings, and high among them are the chance to interact with you kind people through this blog and particularly this Saturday morning column. Thank you all! Please feel generous for being here, you are appreciated!

And as always please:

ask a question at the bottom of the page!!

By the way, if you want more, go here for past advice columns and here for an explanation of the name Pythia.

——

Dear Auntie,

1. Do body parts that are not for public purview (read “genitals”) show greater physical diversity because they have not been acted upon by marketing and evolution?

2. Does the use of wigs by Orthodox Jewish women lead to baldness, as they don’t have to demonstrate good hair and so theirs is kind of …meh? I have two data points; albeit from the same family.

No disrespect to genitals or Orthodox Jews intended.

Sexual Evolution Xpounded

Dear SEX,

First of all, I’m in a new phase where I am really into using the phrase “particulars”. So I’m really glad you asked this question, since it gives me tremendous opportunity in that regard. I’m no expert in particulars, of course, but I’ll talk about particulars anyway, since you asked.

First, let’s think about whether particulars have escaped evolution untouched: for sure not, but it has presumably been more about procreation probabilities and not dying in childbirth than about beauty per se.

Here’s my argument along those lines, specifically when it comes to women’s particulars and the issue of marketing standardization: my impression is that no man has ever gotten that close to sex and then said, “whoa, your vagina has a slightly peculiar shape and/or positioning relative to your clitoris. Maybe we should not procreate after all!!”

I mean, it may have happened but I haven’t heard about it. Tell me if you have evidence to the contrary.

That’s not to say there’s no beauty there in something that is varied and idiosyncratic, to be sure. And things might be slightly different for men in this regard, since let’s face it, men’s particular particulars are more obvious pieces of apparatus and therefore more easily scrutinized.

As for baldness and wigs: no freaking clue, but I do have something to say about wigs in general, which is that there are a TON of wigs out there if you know how to spot them. In fact if you go onto the NY subway and take a look around, you’ll see that a good portion of rush hour commuting women are wearing wigs, and I don’t think it’s because New Yorkers are more likely to be bald. It’s just a big thing, particularly for Jewish and for African-American women. Bigger than you might think, and essentially never discussed, which always piques my interest.

Hope that helps,

Aunt Pythia

——

Dear Aunt Pythia,

Here is my Career dilemma. I am what you would consider an “Engineer” in the Analytics industry. I have had a good career in building Analytics Products aimed at analyzing data and finally implementing some ‘algorithms’ after enough study to take the human out of the process (one example is a routing algorithm that considers 10-15 price, quality and other factors).

Lately, I feel less excited about ‘normal’ analytics projects (because initial study is smaller and rest is all about creating pipelines to setup algorithms to work autonomously). Instead the new ‘Data Science’ field seems more interesting, fun and challenging. I had a good math background, but that was a decade ago…ideally, I would be part of a Data Science team and learn in the process, but as soon as I say I am not a math major, nobody takes me seriously.

I am relearning some of my math skills but I can hardly refresh years of algebra, calculus and operations research skills that easily.

I am NOT dreaming of being the math nerd in a Data Science team but I cannot figure out if Data Science teams need people like me, who have years of Decision Science + Data Processing background. Yes building 1 model does not make someone a Data Scientist, on the other hand writing a couple of python mapreduce jobs or a few SQL queries does not make someone a Data Architect either.

I apply for jobs, get no response and get frustrated and stop looking…and then repeat that after few weeks. I am almost at the point of giving up and going back to Analytics + Data Architecture field. Do you think Data Science teams would welcome people who have more traditional Data background?

Confused about Career Options

Dear Confused,

A couple of things. First, my new book with Rachel Schutt is coming out in a week and a half and is ideal for someone like you. Get it, read it, and build a few of the things discussed in it with publicly available data so you have a portfolio of projects.

Next, it’s hard to get hired as a data science person with your background, even with projects under your belt. So try to get a job as an engineer in a data-driven business, and worm your way into the data group. Tell them that is your intention, and that you are willing to prove your mad data skillz. I’d be surprised if someone didn’t pick you up under such conditions.

Good luck!

Aunt Pythia

——

Aunt Pythia,

I have, belatedly, come in contact with the “Youth Sports Industrial Complex” and the insane, existential battle parents wage for their children’s future through travel soccer and the like.

Literally, people seem to think that their kid will get into Harvard on the strength of their parents’ SERIOUS COMMITMENT to youth sports. Winning at all costs seems to be the one and only goal.

The thing is, my kid could be very competitive at this particular sport – if we were to join one of the competitive clubs and hand our souls over to the dark side. I don’t expect to get a scholarship or something, frankly that’s nuts.

Am I a looney for suggesting to my kids that playing well and having fun – and exhibiting excellent sportsmanship – are the goal if they never seem to beat the hyper-aggressive kids? Am I setting them up for a life as outcasts if we reject this ethos? As a mom, what do you think?

Maximize, Or Maintain?

Dear MOM,

What a fucking great question, thank you for asking it.

As a mom, I am definitely on the radical fringe when it comes to this. Specifically, I have taken my kids out of all grown-up organized activities, mostly at their request (but secretly because I think that shit is nuts). That means no sports, no nothing (they do student-organized stuff sometimes). They are expected to exercise but they get to choose how, and they are expected to do interesting stuff – so not play video games after school – but it’s up to them what to do.

Because for my family, it’s not just offensive to think that “winning is the goal” at all times. It’s even offensive to think that adults should define the goal for growing children in their free time.

[Rant to those people: What’s wrong with you people, isn’t it enough that these kids will probably have to live by other people’s rules when they’re working in jobs later? Why do we have to start that crap so soon?]

This stance makes it easy for me to never have to deal with the question you’re currently dealing with, namely having a kid who likes a team sport and is good at it, and how to think about the rest of the lunatics. My kids, to be clear, hate team sports and suck at them, like good nerds.

My advice is to be consistently sane and give them absolute agency on these decisions. Be utterly honest about what you think of the attitude displayed by the other kids, and ask your kid what they want considering the dire conditions. They might want to do it anyway, and they will definitely benefit from having a sane person to look to when emotions and goals get distorted and out of hand. Most importantly, if they decide to quit the team, let them.

Good luck!

Aunt Pythia

——

Dear Aunt Pythia,

With an electrical engineering background but no research experience, I want to study mathematics. I am quite certain that I want to be in research. Without an undergraduate background in mathematics (though I’ve take few applied mathematics courses), what’s the best way to move forward? I don’t know what exactly would end up being the outcome – I would like it to be either in cognitive sciences or mathematical physics/geology. It’s rather broad, because I can’t tell unless I know more. Should I take a year out and preparing for something, get another bachelors (which I dread, I don’t want to do the 4 year university) or …?

Slowkill

Dear Slowkill,

Pardon me for saying it, but WTF?? How would you know you want to do math research if you don’t have experience in math? That makes no sense, because it means you want to devote yourself to something you don’t understand at all and have no experience in. It really has nothing to do with math at all, unless you are assuming that stories you heard about living the math life are true. An I’m here to tell you, they’re not. If Good Will Hunting were to be believed, all math professors have personal secretaries scurrying around getting them coffee – NOT!!

My advice is to think about what it is you really want to do – or to escape. I’m sensing more escapism than desire in your words. Go see Gravity, it’s supposed to be awesome and totally escapist.

Good luck,

Auntie P

——

Please submit your well-specified, fun-loving, cleverly-abbreviated question to Aunt Pythia!

Categories: Aunt Pythia

Cumulative covariance plots

October 11, 2013 4 comments

One thing I do a lot when I work with data is figure out how to visualize my signals, especially with respect to time.

Lots of things change over time – relationships between variables, for example – and it’s often crucial to get deeply acquainted with how exactly that works with your in-sample data.

Say I am trying to predict “y”: so for a data point at time t, we’ll say we try to predict y(t). I’ll take an “x”, a variable that is expected to predict “y”, and I’ll demean both series x and y, hopefully in a causal way, and I will rename them x’ and y’, and then, making sure I’ve ordered everything with respect to time, I’ll plot the cumulative sum of the product x'(t) * y'(t).

In the case that both x'(t) and y'(t) have the both sign – so they’re both bigger than average or they’re both smaller than average, this product is positive, and otherwise it’s negative. So if you plot the cumulative sum, you get an upwards trend if things are positively correlated and downwards trend if things are negatively correlated. If you think about it, you are computing the numerator of the correlation function, so it is indeed just an unscaled version of total correlation.

Plus, since you ordered everything by time first, you can see how the relationship between these variables evolved over time.

Also, in the case that you are working with financial models, you can make a simplifying assumption that both x and y are pretty well demeaned already (especially at short time scales) and this gives you the cumulative PnL plot of your model. In other words, it tells you how much money your model is making.

So I was doing this exercise of plotting the cumulative covariance with some data the other day, and I got a weird picture. It kind of looked like a “U” plot: it went down dramatically at the beginning, then was pretty flat but trending up, then it went straight up at the end. It ended up not quite as high as it started, which is to say that in terms of straight-up overall correlation, I was calculating something negative but not very large.

But what could account for that U-shape? After some time I realized that the data had been extracted from the database in such a way that, after ordering my data by date, it was hugely biased in the beginning and at the end, in different directions, and that this was unavoidable, and the picture helped me determine exactly which data to exclude from my set.

After getting rid of the biased data at the beginning and the end, I concluded that I had a positive correlation here, even though if I’d trusted the overall “dirty” correlation I would have thought it was negative.

This is good information, and confirmed my belief that it’s always better to visualize data over time than it is to believe one summary statistic like correlation.

Categories: data science, modeling

Data Skeptic post

October 10, 2013 3 comments

I wrote a blog post for O’Reilly’s website to accompany my essay, On Being a Data Skeptic. Here’s an excerpt:

I left finance pretty disgusted with the whole thing, and because I needed to make money and because I’m a nerd, I pretty quickly realized I could rebrand myself a “data scientist” and get a pretty cool job, and that’s what I did. Once I started working in the field, though, I was kind of shocked by how positive everyone was about the “big data revolution” and the “power of data science.”

Not to underestimate the power of data––it’s clearly powerful! And big data has the potential to really revolutionize the way we live our lives for the better––or sometimes not. It really depends.

From my perspective, this was, in tenor if not in the details, the same stuff we’d been doing in finance for a couple of decades and that fields like advertising were slow to pick up on. And, also from my perspective, people needed to be way more careful and skeptical of their powers than they currently seem to be. Because whereas in finance we need to worry about models manipulating the market, in data science we need to worry about models manipulating people, which is in fact scarier. Modelers, if anything, have a bigger responsibility now than ever before.

Categories: data science, finance, modeling

Make Rich People Read Chekhov

October 9, 2013 24 comments

There have been two articles in the New York Times very recently concerning empathy.

First, there was this Opinionator piece about how rich people have less empathy. Second, there was this Well blogpost which reports on a study that implies you can improve your empathy skills, at least in the short term, by reading literary fiction like Chekhov.

Empathy means understanding and sharing the feelings of other people. So what do these two columns actually refer to?

For rich people, it’s mostly about attention rather than empathy. The idea is that researchers study how people pay attention to people (answer: they pay attention to high status people more), and found that rich people don’t do it much at all. They claim attention is a prerequisite for empathy, and that there’s a negative feedback loop going on with the rich, a lack of empathy, and increasing inequality.

As for the literary fiction column, it cites a study in which what they measure is something a little bit different, namely the “theory of mind” of a person after reading Checkhov versus something else. The concept of the theory of mind is that we have internal models of other people’s mindset, and actually they claim to be able to separate this into two parts, cognitive and affective. So if I have a realistic impression of what you’re feeling, we say that my affective theory of mind is good, whereas if I have a realistic impression of how you’re planning to act, that’s called nailing a cognitive theory of mind.

A few comments:

  1. I’m not so sure about the attention-leads-to-empathy assumption. Sometimes I am on a subway and I start sensing people’s emotions around me whether I like it or not, even when I’m trying not to pay attention to them. For me empathy is like smell, and some people are incredibly smelly, especially on the subway.
  2. On the other hand it resonates with me that rich people have less empathy. Certainly this seemed to be the case when I worked at D.E. Shaw, although it might have been a self-selection thing: maybe people who are not empathetic are attracted to working at a hedge fund.
  3. In any case, there’s a tremendous disconnect between regular people and the attitude of finance people, along the lines of “I’m smarter than those people so I deserve to be rich”, and I ascribe much of this disconnect to a lack of empathy.
  4. In both of these columns, though, the question was how well do you pay attention to, and read, people in the same room with you. Unfortunately that’s not a good enough question, at least if you’re worried about that negative feedback loop, if you think about the real world. In the real world, even in New York, rich people don’t spend lots of time in the same room with anyone except other rich people. So it’s a bigger problem to address than what you might at first think.
  5. Having said that, I don’t claim that if everyone just had more empathy all our problems would be solved. Even so I do think it might help. Certainly my sensitivity to other people’s emotions deeply affects me and my actions and goals, but of course that’s too little evidence to go by.
  6. In any case it’s an interesting thought experiment to imagine a world of increased empathy. I like that it’s being considered as a basic attribute of interest, and that it seems tweakable.
  7. Conclusion: before talking to someone I perceive as unempathetic, I will bust out a Checkov short story (this one) and demand they read it on the spot. That should really help.
Categories: finance, musing, news

Guest post: Rage against the algorithms

October 8, 2013 8 comments

This is a guest post by , a Tow Fellow at the Columbia University Graduate School of Journalism where he is researching the use of data and algorithms in the news. You can find out more about his research and other projects on his website or by following him on Twitter. Crossposted from engenhonetwork with permission from the author.

shutterstock_125392883_650

How can we know the biases of a piece of software? By reverse engineering it, of course.

When was the last time you read an online review about a local business or service on a platform like Yelp? Of course you want to make sure the local plumber you hire is honest, or that even if the date is dud, at least the restaurant isn’t lousy. A recent survey found that 76 percent of consumers check online reviews before buying, so a lot can hinge on a good or bad review. Such sites have become so important to local businesses that it’s not uncommon for scheming owners to hire shills to boost themselves or put down their rivals.

To protect users from getting duped by fake reviews Yelp employs an algorithmic review reviewer which constantly scans reviews and relegates suspicious ones to a “filtered reviews” page, effectively de-emphasizing them without deleting them entirely. But of course that algorithm is not perfect, and it sometimes de-emphasizes legitimate reviews and leaves actual fakes intact—oops. Some businesses have complained, alleging that the filter can incorrectly remove all of their most positive reviews, leaving them with a lowly one- or two-stars average.

This is just one example of how algorithms are becoming ever more important in society, for everything from search engine personalization, discrimination, defamation, and censorship online, to how teachers are evaluated, how markets work, how political campaigns are run, and even how something like immigration is policed. Algorithms, driven by vast troves of data, are the new power brokers in society, both in the corporate world as well as in government.

They have biases like the rest of us. And they make mistakes. But they’re opaque, hiding their secrets behind layers of complexity. How can we deal with the power that algorithms may exert on us? How can we better understand where they might be wronging us?

Transparency is the vogue response to this problem right now. The big “open data” transparency-in-government push that started in 2009 was largely the result of an executive memo from President Obama. And of course corporations are on board too; Google publishes a biannual transparency report showing how often they remove or disclose information to governments. Transparency is an effective tool for inculcating public trust and is even the way journalists are now trained to deal with the hole where mighty Objectivity once stood.

But transparency knows some bounds. For example, though the Freedom of Information Act facilitates the public’s right to relevant government data, it has no legal teeth for compelling the government to disclose how that data was algorithmically generated or used in publicly relevant decisions (extensions worth considering).

Moreover, corporations have self-imposed limits on how transparent they want to be, since exposing too many details of their proprietary systems may undermine a competitive advantage (trade secrets), or leave the system open to gaming and manipulation. Furthermore, whereas transparency of data can be achieved simply by publishing a spreadsheet or database, transparency of an algorithm can be much more complex, resulting in additional labor costs both in creation as well as consumption of that information—a cognitive overload that keeps all but the most determined at bay. Methods for usable transparency need to be developed so that the relevant aspects of an algorithm can be presented in an understandable way.

Given the challenges to employing transparency as a check on algorithmic power, a new and complementary alternative is emerging. I call it algorithmic accountability reporting. At its core it’s really about reverse engineering—articulating the specifications of a system through a rigorous examination drawing on domain knowledge, observation, and deduction to unearth a model of how that system works.

As interest grows in understanding the broader impacts of algorithms, this kind of accountability reporting is already happening in some newsrooms, as well as in academic circles. At the Wall Street Journal a team of reporters probed e-commerce platforms to identify instances of potential price discrimination in dynamic and personalized online pricing. By polling different websites they were able to spot several, such as Staples.com, that were adjusting prices dynamically based on the location of the person visiting the site. At the Daily Beast, reporter Michael Keller dove into the iPhone spelling correction feature to help surface patterns of censorship and see which words, like “abortion,” the phone wouldn’t correct if they were misspelled. In my own investigation for Slate, I traced the contours of the editorial criteria embedded in search engine autocomplete algorithms. By collecting hundreds of autocompletions for queries relating to sex and violence I was able to ascertain which terms Google and Bing were blocking or censoring, uncovering mistakes in how these algorithms apply their editorial criteria.

All of these stories share a more or less common method. Algorithms are essentially black boxes, exposing an input and output without betraying any of their inner organs. You can’t see what’s going on inside directly, but if you vary the inputs in enough different ways and pay close attention to the outputs, you can start piecing together some likeness for how the algorithm transforms each input into an output. The black box starts to divulge some secrets.

Algorithmic accountability is also gaining traction in academia. At Harvard, Latanya Sweeney has looked at how online advertisements can be biased by the racial association of names used as queries. When you search for “black names” as opposed to “white names” ads using the word “arrest” appeared more often for online background check service Instant Checkmate. She thinks the disparity in the use of “arrest” suggests a discriminatory connection between race and crime. Her method, as with all of the other examples above, does point to a weakness though: Is the discrimination caused by Google, by Instant Checkmate, or simply by pre-existing societal biases? We don’t know, and correlation does not equal intention. As much as algorithmic accountability can help us diagnose the existence of a problem, we have to go deeper and do more journalistic-style reporting to understand the motivations or intentions behind an algorithm. We still need to answer the question of why.

And this is why it’s absolutely essential to have computational journalists not just engaging in the reverse engineering of algorithms, but also reporting and digging deeper into the motives and design intentions behind algorithms. Sure, it can be hard to convince companies running such algorithms to open up in detail about how their algorithms work, but interviews can still uncover details about larger goals and objectives built into an algorithm, better contextualizing a reverse-engineering analysis. Transparency is still important here too, as it adds to the information that can be used to characterize the technical system.

Despite the fact that forward thinkers like Larry Lessig have been writing for some time about how code is a lever on behavior, we’re still in the early days of developing methods for holding that code and its influence accountable. “There’s no conventional or obvious approach to it. It’s a lot of testing or trial and error, and it’s hard to teach in any uniform way,” noted Jeremy Singer-Vine, a reporter and programmer who worked on the WSJ price discrimination story. It will always be a messy business with lots of room for creativity, but given the growing power that algorithms wield in society it’s vital to continue to develop, codify, and teach more formalized methods of algorithmic accountability. In the absence of new legal measures, it may just provide a novel way to shed light on such systems, particularly in cases where transparency doesn’t or can’t offer much clarity.

Categories: data science, guest post, modeling

A.F.R. Transparency Panel coming up on Friday in D.C.

October 7, 2013 6 comments

I’m preparing for a short trip to D.C. this week to take part in a day-long event held by Americans for Financial Reform. You can get the announcement here online, but I’m not sure what the finalized schedule of the day is going to be. Also, I believe it will be recorded, but I don’t know the details yet.

In any case, I’m psyched to be joining this, and the AFR are great guys doing important work in the realm of financial reform.

——

 Opening Wall Street’s Black Box: Pathways to Improved Financial Transparency 

Sponsored By Americans for Financial Reform and Georgetown University Law Center

Keynote Speaker: Gary Gensler Chair, Commodity Futures Trading Commission

October 11, 2013 10 AM – 3 PM

Georgetown Law Center, Gewirz Student Center, 12th Floor

120 F Street NW, Washington, DC (Judiciary Square Metro) (Space is limited. Please RSVP to AFRtransparencyrsvp@gmail.com)

The 2008 financial crisis revealed that regulators and many sophisticated market participants were in the dark about major risks and exposures in our financial system. The lack of financial transparency enabled large-scale fraud and deception of investors, weakened the stability of the financial system, and contributed to the market failure after the collapse of Lehman Brothers. Five years later, despite regulatory efforts, it’s not clear how much the situation has improved.

Join regulators, market participants, and academic experts for an exploration of the progress made – and the work that remains to be done – toward meaningful transparency on Wall Street. How can better information and disclosure make the financial system both fairer and safer?

Panelists include:

Jesse Eisinger, Pulitzer Prize-winning reporter for the New York Times and Pro Publica
Zach Gast, Head of financial sector research, Center on Financial Research and Analysis
Amias Gerety, Deputy Assistant Secretary for the FSOC, United States Treasury
Henry Hu, Alan Shivers Chair in the Law of Banking and Finance, University of Texas Law School
Albert “Pete” Kyle, Charles E. Smith Professor of Finance, University of Maryland
Adam Levitan, Professor of Law, Georgetown University Law Center
Antoine Martin, Vice President, New York Federal Reserve Bank
Brad Miller, Former Representative from North Carolina; Of Counsel, Grais & Ellsworth
Cathy O’Neil, Senior Data Scientist, Johnson Research Labs; Occupy Alternative Banking
Gene Phillips, Director, PF2 Securities Evaluation
Greg Smith, Author of “Why I Left Goldman Sachs”; former Goldman Sachs Executive Director
Categories: finance, modeling, open source tools

Sir Andrew Wiles smacks down unethical use of mathematics for profit

October 6, 2013 10 comments

My buddy Jordan Ellenberg sent me this link to an article which covered Sir Andrew Wiles’ comments at a the opening of the Andrew Wiles Building, a housing complex for math nerds in Oxford. From the article:

Wiles claimed that the abuse of mathematics during the global financial meltdown in 2009, particularly by banks’ manipulation of complex derivatives, had tarnished his chosen subject’s reputation.

He explained that scientists used to worry about the ethical repercussions of their work and that mathematics research, which used to be removed from day-to-day life, has diverged “towards goals that you might not believe in”.

At one point Wiles said the following, which is music to my ears coming from a powerful mathematician:

One has to be aware now that mathematics can be misused and that we have to protect its good name.

Two things.

First, maybe I should invite Wiles to be on my panel of mathematicians for investigating public math models. I originally thought this should be run under the auspices of a society such as the AMS but after talking to some people I’ve given up on that and just want it to be independent.

Second, the Andrew Wiles building was evidently paid for primarily by Landon Clay, who also founded the Clay Institute and was the CEO of Eaton Vance, which an investment management firm which provides its clients with wealth management tools and advice. I’m wondering if that kind of mathematical tool was in Wiles’ mind when he made his speech, and if so, how it went over. Certainly in my experience, wealth management tools are definitely in the “weapons of math destruction” toolbox.

Categories: finance, math

Aunt Pythia’s advice

October 5, 2013 6 comments

Sorry for the lateness of this column. Aunt Pythia slept in this morning and then went for a beautiful bike ride in Central Park. It’s perfect biking weather: somewhat chilly and cloudy, so the sun doesn’t get in your eyes and you don’t get overheated. You guys gotta try it!

However, Aunt Pythia didn’t forget you guys and she wants you to enjoy today’s column and of course she urges you as always to:

ask a question at the bottom of the page!!

By the way, if you want more, go here for past advice columns and here for an explanation of the name Pythia.

——

Dear Aunt Pythia,

Why is it that in this country we can accept that a lobbyist is a valid job description and a valid job but we can’t accept a sex worker?

The profession is legal in Europe, why not the US?

Short and Sweet

Deat S&S,

I wasn’t sure whether, after that first sentence, you wanted lobbyists banned or sex workers made legal. To tell you the truth I coulda gone either way.

So yes, I agree, it’s interesting to think about A) what the hold-up is on legalizing sex work and B) what the pros and cons are of sex-work being legal.

As for the politics, after writing this post about the GOP mindset I’m really not surprised that we haven’t gotten consensus.

As for the pros and cons, I’ve thought about this before, and since I don’t have the actual data I am going on these assumptions I’ve gathered from various reading on the topic, which would all have to be verified:

  • Protecting sex workers makes the profession safer for the workers. It means, for example, that they can call the cops if the clients misbehave, not to mention demand things like health insurance and regular HIV tests like porn industry actors.
  • It also has economic effects. For example, legalized sex workers probably makes buying sex cheaper (and safer), as well as not-quite-sex stuff like topless bars and lap dances.
  • So, in particular, there are plenty of current U.S. establishments that would lose money if sex-working became legalized, specifically places that have super expensive legal almost-sex things and possibly even more expensive illegal sex things for sale. Of course if they moved quickly they could capture the new market.
  • Also, keep in mind that, although safer when legal, sex-work is still dangerous. And if it were more widespread it would affect more people, meaning it might be a net negative thing to do. Kind of like how alcohol is more harmful than heroin because it’s so widely used.

Going back to the original question, how about we just outlaw lobbyists?

Aunt Pythia

——

Dear Aunt Pythia,

We know you live in New York. But what are your favorite cities or places that you’ve visited?

Curious

Dear Curious,

There are two kinds of traveling for me: with my kids and alone or with other grownups. When I travel with my kids, I basically just spend time with them. But when I travel alone or with other grown-ups, I do it to meet the people living in that place. I am not visiting to see historical things or to view what that culture’s elite considers its finest works.

I don’t like museums or monuments or historical sites, I never have. I like talking to the people currently living somewhere, and I like exploring how they actually live day to day. I’d rather see their markets than their art. Partly this is because I don’t get art but mostly it’s because I think it’s fascinating to see the differences in average people’s lives and how that informs their mindset. I walk around for hours in their cities and intentionally fall into random conversations at the shop or at lunch or at coffee or at a live music performance. That’s a perfect day for me.

Since everyone shops, and everyone eats, and most everyone talks to people when they do this, I’m pretty neutral to exactly where I go. I always find something fascinating about any place I visit, be it Vermont or Prague.

The only place I’ve ever gone where I found the surroundings more interesting than the people is on my honeymoon, when we went to Alaska, and I got really into geology. And the most fascinating and engaging people I ever met were in Accra, Ghana.

One last thing: I love traveling and I would do way more of it if I didn’t have 3 kids. In fact that’s one thing I am truly jealous of for people with no kids, that they get to travel so much. Enjoy that!

Aunt Pythia

——

Dear Aunt Pythia,

I’m from Europe. There are some fairly strong cultural differences between countries, but also some common trends. One is anti-US bashing (e.g. NSA stuff, Iraq fiasco, guns and abortion laws…). To the point that I know some academics who actually refuse to travel to the US.

And yet, I’ve been there a couple times and am well aware that there’s a sizable liberal community, especially on the coasts, and some places like the Boston area or the Silicon Valley seem quite attractive to me. 

So to my question: what advice would you give to a hesitant European (who has no family issues yet, but not a large wallet either)? Land an IT job, and then fly there and just give it a try? Or maybe you have observed many Europeans going back after a couple years?

Patriotism Is Dangerous

Dear PID,

Important question: are people objecting to living in a country with those kinds of policies? Or are they objecting to living in a country where everyone wants to personally own a loaded pistol so they can kill anyone trying to have an abortion?

Here’s the thing. There’s the policies, and then there are the people. While it’s reasonable to avoid living in the U.S. because of it’s insane policies, especially as a non-citizen, it’s of course not reasonable to assume that every city is filled with people who are insane.

For that matter a friend of mine, who is not a descendant of Europeans, tells me that Europeans are hugely racist – not everyone, and not everywhere, but it makes him not want to travel to Europe. So we see the flip side of the coin, namely you can also have reason avoid a country that has reasonable policies but unappealing people.

I’m not really answering your question, but I do want to challenge you (and your European academic brethren) to think about it more carefully. In New York or San Francisco you won’t find a lot of people supporting the policies you despise, but then again you will be in some sense a part of that system even so.

As for what you should do: I know LOTS of Europeans who come here and love it, and still hate lots of the policy. My husband, for example. Of course I am less likely to meet people who leave. So do with that what you will.

Aunt Pythia

——

Dear Aunt Pythia,

I have a problem with this guy I was platonically interested in, because he seemed interesting conversation. Unfortunately, he turned out to be quite a self-centered person, so while having intelligent thoughts, his overflowing self confidence makes it less fun to be around.

Worse, he is sexually interested in me, despite my very clear messages that this is never going to happen. He claims freedom of speech of some sort and openness between friends. For a while he used opportunities when we meet at various social circles to kiss me and try to touch me, and once even made some loud embarrassing comments in the presence of a crowd.

I thought I had this under control, as we are both in stable, long term relationships, and I could handle this shit. Indeed he stopped for a while, but recently started texting me again. I don’t want to make a big scene, because innocent people may get hurt, so I try to be civil when we chance to meet, but I do wonder whether there is a particular angle at which I can kick him in the balls to get the message across.

Half Of The Time Intolerably Embarrassed

Dear HOTTIE,

Ooooh I like your sign-off.

OK so just to be SUPER CLEAR about this: have you told him in no uncertain terms to stop? Have you said “I want you to stop trying to be sexual with me, right now”? Have you texted him back with the words “please do not text me”? I will assume you have since it is CLEARLY not enough to think he will get the hint just because you guys are both in stable long-term relationships.

In other words, when you say you’ve given him “very clear messages” I need to believe that you mean “I said no”. Many many men do not hear “no” until you actually say that word, so please promise me you haven’t expected him to pick it up through certain looks or the way you don’t respond to a stolen kiss.

Okay, now that that is out of the way, I am surprised you are willing to talk to him at all. Are you still friends with him? Do you want to be? It sounds like you are somewhat ambivalent, which I think may be the problem here. He might be reading your continued interest in being his friend as sexual interest, or at least as a lack of sexual rejection.

My advice: next time he texts, ignore him altogether, and go ahead and block him now if that is hard to do. Next time you see him in person, if he tries something, take away his hand and say you’re not interested, and that you’re planning to talk to his long-term partner about how he can’t seem to stop trying something even though you’ve rejected him multiple times, and you’re going to ask his partner for advice on how to get him to stop. If he laughs or otherwise ignores you tell him you’re serious and might follow it up with a restraining order.

In other words, make it clear to him that it’s really not OK, the way he’s been acting, and that you are willing to risk real discomfort in relationships (especially his) to get this resolved. It has nothing to do with your relationship, so don’t feel threatened if he says he’ll talk to your long-term partner.

Good luck!

Aunt Pythia

——

Please submit your well-specified, fun-loving, cleverly-abbreviated question to Aunt Pythia!

Categories: Aunt Pythia

Inside the GOP, a report from Democracy Corps

October 4, 2013 14 comments

I was wondering what a lot of other people were wondering yesterday. Namely why, if Republicans were claiming their party was being hijacked by a small minority of Tea Party lunatics, did they actually have a majority vote for closing down the government? It’s a statistical conundrum.

A fascinating report entitled Inside the GOP: Report on focus groups with Evangelical, Tea Party, and moderate Republicans explains that to me. It was put out by Democracy Corps, a non-profit Democratic strategy think tank co-founded by Stan Greenberg, and it explains the current mindset of different factions of the Republican Party, inferred from focus groups made up of three types of Republicans: Evangelicals (55%), moderates (25%), and Tea Partiers (20%).

You should read the whole thing, because it’s absolutely fascinating, but I think the explanation of the above-mentioned statistical conundrum is as follows: Tea Partiers are minority, but the largest group, namely the Evangelicals (think Fox News, anti-gay marriage) are behind the Tea Party’s agenda on the Affordable Care Act as well as Obamacare, and together they represent the majority of the Republican Party. Maybe you knew this but I didn’t.

Other things this report brings up:

  • how deeply race matters to Evangelicals, especially when it comes to Obamacare,
  • how tenuous the alliance is between Tea Party Republicans and Evangelicals, considering Tea Partiers don’t care about social issues like gay marriage and Evangelicals deeply care, and
  • just how much Fox News matters in this world.
Categories: news

I think I understand the revolving door problem

October 3, 2013 18 comments

I was reading this Bloomberg article about the internal risk models at JP Morgan versus Goldman Sachs, and it hit me: I too had an urge for the SEC to hire the insiders at Goldman Sachs to help them “understand risk” at every level. Why not hire a small team of Goldman Sachs experts to help the SEC combat bullshit like what happened with the London Whale?

After all, Goldman people know risk. They probably knew risk even better before 1999, when they went IPO and the partners stopped being personally liable for losses. But even now, of all the big players on the street, Goldman is known for being a few steps ahead of everyone else when it comes to a losing trade.

So it’s natural to want someone from deeply within that culture to come spread their technical risk wisdom to the other side, the regulators.

Unfortunately that’s never what actually happens. Instead of getting the technical knowledge of how to think about risk, how to model a portfolio to squirrel out black holes of mystery, the revolving door instead keeps outputting crazy freaks like Jon Corzine, who blow up firms through, ironically, taking ridiculous risks at the first opportunity.

So, why does this happen? Some possibilities:

  1. Goldman Sachs promotes crazy freaks because they make great leaders while constrained inside a disciplined culture of calculated risks, but when they get outside they go nuts. This is kind of the model of Mormon children who are finally allowed out into the world and engage in tons of sex and drugs.
  2. On the flip side, perhaps Goldman Sachs keeps the people who actually understand the technical part of risk very deep in the machine and these guys never get leave the building at all.
  3. Or maybe, people who understand risk sometimes do go through the revolving door, but they don’t share their knowledge with the other side, because their incentives have changed once they’re outside.
  4. In other words, they don’t help the regulators understand how banks lie and cheat to regulators, because they’re too busy watering down regulation so their buddies can continuously lie and cheat to regulators.

Whatever the case, for whatever reason we keep using the revolving door in hopes that someone will eventually tell us the magic that Goldman Sachs knows, but we never quite get anyone like that, and that means the the SEC and other regulators are woefully unprepared for the kind of tricks that banks have up their sleeves.

Categories: finance, modeling, musing

“The only thing we really learned from the S&L scandal was how CEO’s should lawyer up”

October 2, 2013 4 comments

The title of today’s piece is a quote from economist Chris Thornberg in a Reuters interview I found a couple of days ago.

It’s almost 15 minutes long, and the first couple of minutes are concerned with the housing market, which he has strong opinions on and I don’t, but then he moves on to financial regulation, the question of accountability, municipal bankruptcies, and whether economists suck at their jobs, subjects I care about a lot.

It’s a super interesting interview, and I’m planning to mail Thornberg a copy of Occupy Finance soon. Even though he serves on the advisory board of hedge fund Paulson & Co, he agrees with lots of people who come to Alternative Banking on many issues.

A few examples of what he’s talking about for those of you without 15 minutes to spend on him.

Regulation

He talks about how Dodd-Frank is collapsing in on itself through extensive watering-down and lobbying, and how negotiations over how much “skin in the game” is required around securitization is the death knell of that process.

He talks about how the overall system is under-regulated and not being made accountable. He claims that, instead of trying to draw lines in the sand, we should try to attack the skewed incentives. A lot of people got very rich doing very bad things, he says, and the continued existence of those messed-up incentives will encourage a whole new generation to do the same.

For example, bonuses shouldn’t be given on how many loans you push out into the market but how they perform. We should be able to clawback money, which is to say pull back money from someone who’s done something bad. After all, he points out, Dick Fuld and Angelo Mozilo as men who became filthy rich from doing bad things and are now “untouchable”.

Washington

He also points out that D.C. never went through a recession because of money coming in through lobbying. His top two priorities to improve our system would be:

  1. To simplify the tax code, both corporate and personal. There too many special interests. Get rid of complexity and make it more progressive.
  2. Political reform. We like to say the Chinese are “one-party state”. But are we really two-party? We have very little “choice” especially if you take into account the gerrymandering. California is doing a good job reforming here.

Municipal Bankruptcies

Next, he moves onto municipal emergencies in Detroit, Stockton, and San Bernadino. All three city bankruptcies pose the fundamental decisions: what is the sanctity of public pensions? State laws generally insist that pensions be paid.

The issue here is, says Thornberg, that federal bankruptcy laws trump, not state law. So the relevant federal judge can say “too bad” to the contract, and the pensioners will be forced to take a cut like all the other debtholders.

Lawyers in the know think pensions are gonna go, and that precedent for this kind of thing is being established in these three cities in the very near future.

Economics

The Reuters interviewer addressed the issue that, generally speaking, economists missed the oncoming crisis. Does that mean there’s something wrong with economics?

Thornberg: “Professors don’t get published writing papers about the current economy – nobody cares about that.”

He claims that, yes, economists should be more on top of day-to-day stuff, but he claims that the critique misses the broader point, which is that everyone needs to be a better economist. He claims it is a social science which tries to address why people do what they do. And although economists would like to think of it as a mathematical science, they’re wrong to do so.

His advice: don’t listen to economists on TV. Educate yourself and know your source. It’s amazing how much stock we put into economists working for, say, the National Association of Realtors who get paid to say good things like, “it’s a good time to buy and sell a house!”

Categories: #OWS, finance

New Essay, On Being a Data Skeptic, now out

October 1, 2013 5 comments

It is available here and is based on a related essay written by Susan Webber entitled “Management’s Great Addiction: It’s time we recognized that we just can’t measure everything.” It is being published by O’Reilly as an e-book.

No, I don’t know who that woman is looking skeptical on the cover. I wish they’d asked me for a picture of a skeptical person, I think my 11-year-old son would’ve done a better job.

Categories: data science, modeling, musing

“Here and Now” is shilling for the College Board

September 30, 2013 13 comments

Did you think public radio doesn’t have advertising? Think again.

Last week Here and Now’s host Jeremy Hobson set up College Board’s James Montoya for a perfect advertisement regarding a story on SAT scores going down. The transcript and recording are here (hat tip Becky Jaffe).

To set it up, they talk about how GPA’s are going up on average over the country but how, at the same time, the average SAT score went down last year.

Somehow the interpretation of this is that there’s grade inflation and that kids must be in need of more test prep because they’re dumber.

What is the College Board?

You might think, especially if you listen to this interview, that the college board is a thoughtful non-profit dedicated to getting kids prepared for college.

Make no mistake about it: the College Board is a big business, and much of their money comes from selling test prep stuff on top of administering tests. Here are a couple of things you might want to know about College Board through its wikipedia page:

Consumer rights organization Americans for Educational Testing Reform (AETR) has criticized College Board for violating its non-profit status through excessive profits and exorbitant executive compensation; nineteen of its executives make more than $300,000 per year, with CEO Gaston Caperton earning $1.3 million in 2009 (including deferred compensation).[10][11] AETR also claims that College Board is acting unethically by selling test preparation materials, directly lobbying legislators and government officials, and refusing to acknowledge test-taker rights.[12]

Anyhoo, let’s just say it this way: College Board has the ability to create an “emergency” about SAT scores, by say changing the test or making it harder, and then the only “reasonable response” is to pay for yet more test prep. And somehow Here and Now’s host Jeremy Hobson didn’t see this coming at all.

The interview

Here’s an excerpt:

HOBSON: It also suggests, when you look at the year-over-year scores, the averages, that things are getting worse, not better, because if I look at, for example, in critical reading in 2006, the average being 503, and now it’s 496. Same deal in math and writing. They’ve gone down.

MONTOYA: Well, at the same time that we have seen the scores go down, what’s very interesting is that we have seen the average GPAs reported going up. So, for example, when we look at SAT test takers this year, 48 percent reported having a GPA in the A range compared to 45 percent last year, compared to 44 percent in 2011, I think, suggesting that there simply have to be more rigor in core courses.

HOBSON: Well, and maybe that there’s grade inflation going on.

MONTOYA: Well, clearly, that there is grade inflation. There is no question about that. And it’s one of the reasons why standardized test scores are so important in the admission office. I know that, as a former dean of admission, test scores help gauge the meaning of a GPA, particularly given the fact that nearly half of all SAT takers are reporting a GPA in the A range.

Just to be super clear about the shilling, here’s Hobson a bit later in the interview:

HOBSON: Well – and we should say that your report noted – since you mentioned practice – that as is the case with the ACT, the students who take the rigorous prep courses do better on the SAT.

What does it really mean when SAT scores go down?

Here’s the thing. SAT scores are fucked with ALL THE TIME. Traditionally, they had to make SAT’s harder since people were getting better at them. As test-makers, they want a good bell curve, so they need to adjust the test as the population changes and as their habits of test prep change.

The result is that SAT tests are different every year, so just saying that the scores went down from year to year is meaningless. Even if the same group of kids took those two different tests in the same year, they’d have different scores.

Also, according to my friend Becky who works with kids preparing for the SAT, they really did make substantial changes recently in the math section, changing the function notation, which makes it much harder for kids to parse the questions. In other words, they switched something around to give kids reason to pay for more test prep.

Important: this has nothing to do with their knowledge, it has to do with their training for this specific test.

If you want to understand the issues outside of math, take for example the essay. According to this critique, the number one criterion for essay grade is length. Length trumps clarity of expression, relevance of the supporting arguments to the thesis, mechanics, and all other elements of quality writing. As my friend Becky says:

I have coached high school students on the SAT for years and have found time and again, much to my chagrin, that students receive top scores for long essays even if they are desultory, tangent-filled and riddled with sentence fragments, run-ons, and spelling errors.

Similarly, I have consistently seen students receive low scores for shorter essays that are thoughtful and sophisticated, logical and coherent, stylish and articulate.

As long as the number one criterion for receiving a high score on the SAT essay is length, students will be confused as to what constitutes successful college writing and scoring well on the written portion of the exam will remain essentially meaningless. High-scoring students will have to unlearn the strategies that led to success on the SAT essay and relearn the fundamentals of written expression in a college writing class.

If the College Board (the makers of the SAT) is so concerned about the dumbing down of American children, they should examine their own role in lowering and distorting the standards for written expression.

Conclusion

Two things. First, shame on College Board and James Montoya for acting like SAT scores are somehow beacons of truth without acknowledging the fiddling that goes on time and time again by his company. And second, shame on Here and Now and Jemery Hobson for being utterly naive and buying in entirely to this scare tactic.

Categories: Becky Jaffe, math education, modeling, rant

Sunday morning reading

September 29, 2013 3 comments

I wanted to share with you guys a few things I’ve been interested in this weekend.

First, this TED talk which for whatever reason never made it onto the TED main website. It’s by Nick Hanauer, and in it he dispels some common economic myths, much like Chapter 7 of Occupy Finance. A juicy quote: “So when businesspeople take credit for creating jobs, it’s a little like squirrels taking credit for creating evolution. In fact, it’s the other way around.”

Next, it turns out women are way better than men at orgasms, at least those where you do it all in your head – a “think off”. A full 2% of women can fantasize their way to climax (compared to, I guess, way fewer men) and there’s even training for this skill. Two questions from the mathbabe. First, do wet dreams count? Because if they do I think we’ll have to recount. Second, say I invest my time in this, and get really good at it, since it’s all hands-free and such and will make my life that much more efficient. Is this something that takes a lot of time? Can I do it whilst carrying groceries home, or whilst cooking dinner? On the subway? Between stops? Details please.onetrain

Next, here’s an important discussion of why junior people get criminally prosecuted – in this case, for the debacle that was the JP Morgan London Whale case – while the big bosses just get vaguely complained about in a civil case, and the shareholders end up paying huge fines for their misbehavior. Last two lines: “Yet it remains disquieting when the same actions result in criminal charges for some but only a civil case for others, and no individuals are held responsible for misconduct at a company. In the end, we are left to trust that prosecutors have made good decisions.” There’s no recourse for bad prosecuting either. How do we even protest bad prosecuting?

Next, I’ve been listening to some seriously catchy and funny tunes my boys turned me on to. What makes me old is how long it takes me to catch on to stuff, since I heard this one years ago, but it seems that everything that these guys touch is hilarious. Especially this one (also see “I Just Had Sex” and “Jack Sparrow“):

Finally, I’m really into knitting recently, and I recently figured out how to knit this pattern even though you’d have to pay big bucks to get the official pattern. Email me if you’re interested in the bootleg version.

Not mine - I stole this picture. I am still working on mine.

Not mine – I stole this picture. I am still working on mine.

Categories: musing, news

Aunt Pythia’s advice

September 28, 2013 9 comments

Thanks guys, Aunt Pythia has been feeling some love this week, ever since I threatened to murder her. Nothing like a damsel in distress to get the ethical-dilemma juices flowing. Please keep the questions coming though, we don’t want her continually scared and exhausted, that’s no way to live.

In other words, enjoy today’s advice, but please: 

Don’t forget to ask a question at the bottom!!

By the way, if you don’t know what the hell I’m talking about, go here for past advice columns and here for an explanation of the name Pythia.

——

Dear Aunt Pythia,

My partner needs to find a new job. I believe she needs to (at least partially) reinvent herself, although she’s not very adventurous.

You’ve reinvented yourself a few times, you probably know a great deal about what works in this process. I remember you once posted about creating a spreadsheet and recording what you like, what you don’t etc until you found your dream job.

I’m looking for this type of exercises that would challenge her to find a job she loves as opposed to the job she can easily land. Any other insights from your remodeling thought process? Any other resources/reference you would recommend?

Abelian Grape

Dear Abelian,

I don’t believe much in astrology, but I can dig the next closest thing, which is personality tests. I recently looked in the Myers-Briggs Type Indicator and discovered that I’m a so-called “ENTP”, which is to say extroverted (duh), intuitive, thinking, and perceptive. Who knows why, a test told me[1]. That means I’m:

Quick, ingenious, stimulating, alert, and outspoken. Resourceful in solving new and challenging problems. Adept at generating conceptual possibilities and then analyzing them strategically. Good at reading other people. Bored by routine, will seldom do the same thing the same way, apt to turn to one new interest after another.

Why do I mention this? First of all because everyone loves talking about personality tests – trust me – and second of all because it’s in my nature to reinvent myself. I don’t do it because I’m theoretically excited by reinvention, but because I’m bored and compelled to start something new.

So, two conclusions. First, your partner might just not be like that. Second, she might be like that in special circumstances, but in that case she’d need to get to the point of frustration and boredom that she’s the one writing to Aunt Pythia for advice on self-reinvention rather than you. Once that happens I will indeed point her to my tools of reinvention.

My advice is to be supportive of her but not to push her into “reinvention” if that’s not how she rolls. It just won’t work and it will feel to her like another thing she’s failing at. Wait for her, and if you’re not the kind of person that is patient, then that’s a problem in itself and I’ll expect to hear back from you, although given how impatient I am, the advice won’t be hopeful.

Good luck!

Aunt Pythia

1. Actually, one test told me that, then another one said “ENFJ”, but “ENTP” helps me make my point better.

——

Dear Aunt Pythia,

I hope you’re feeling better. 

This is, I admit, a rather lame question, as I am sure I could’ve answered it myself when I was a student. But now I’m old and hence stupid.

I’ll phrase it as a “sock drawer” question. Suppose my drawer contained 44 black socks and 116 white ones, and I draw them out blindly in pairs. What are the chances of getting exactly 10 black pairs?

More generally, if I have b black and w white socks, what is the probability of getting exactly p pairs of black ones?

Thanks Pythia-babe!

Socks Maniac

Dear Socks,

Are the socks already rolled into pairs? Not clear from your question, but I’ll assume so. Otherwise the question is harder, so please do re-submit if I got it wrong. Also, are you blindly taking out exactly 10 pairs and looking to see if they’re all black? I’ll assume that too since you didn’t specify.

Assuming the above, we’re starting with 22 black pairs and 58 white pairs in a drawer, and we take out 10 pairs, and we’re wondering what the chances are that they’re all black. We just need to count the total ways they could be all black and then divide by the total ways we could have done the extraction.

Start with the “all black” count: there are 22 ways we could choose the first black pair, then 21 ways to choose the second black pair, etc., so we get 22*21* … *13 ways altogether to get 10 black pairs.

Next, count the “anything goes” possibilities: we have 22+58=80 pairs of socks altogether, which means we have 80 ways to choose the first pair, then 79 ways to choose the second pair, etc., giving us 80*79*78*…*71 ways to get all ten pairs. Some of them will be all black, but not many.

In fact if you take that ratio – google “22*21*20*19*18*17*16*15*14*13/(80*79*78*77*76*75*74*73*72*71)” – you will see that the answer is very small indeed: 4e-7. You know it’s small if you need scientific notation.

Auntie P

——

Dear Aunt Pythia,

I’ve spent the last year and a half working as, effectively, project manager to get a fairly cool academic mobile app out the door. We’ve applied for a grant to renew the project, but if the grant fails, I’ll be asked to leave $reallyNiceCountry again. 

How do I manage the sense of powerlessness that stems from being a 30-year-old freshly minted Ph.D. either about to be deported again or offered a job that allows me a sufficient contract window to become a permanent resident?

A sense of loyalty (and major deadlines) mean that I don’t feel right trying to apply for other jobs in case the grant is passed.

Exhausted Academic

Dear Exhausted,

I’m glad you wrote. I really object to your sense of loyalty, and I see this all too often among freshly minted foreign-born Ph.D.’s.

Face it, you are a specialist in a bizarre system (the intersection of the academic system and the U.S. visa system) with ridiculously arbitrary and last-minute changes of plan. There is absolutely no reason for you not to develop other plans while you are waiting around for the grant to come through or not. In fact you’re a fool for not applying for other jobs, straight up. Deadlines are a short-term distraction from making your life in a country where you want to live. Your life plan is your priority, not someone else’s app deadline.

Here’s my advice, to you and to anyone else in a related situation. No wonder you’re feeling helpless, it’s because you’re acting passively and helplessly. Nobody is going to think strategically about your future except you. Never let this happen again, and get thee on the job market immediately. People with your education level and mad skillz will get great jobs if they go and look. But you gotta go and look. And if you need to learn other stuff to get a good job, then go learn that stuff. But don’t act like the stupid NSF is the voice of God.

Good luck!

Aunt Pythia

——

Dear Aunt Pythia,

Why is it that the students for whom you’ve made the most opportunity, and invested the most in, are the ones that ultimately screw you?

Pissed Off Professor

Dear POP,

Without more details, I’m going to have to use my imagination here.

I can understand what you might mean by making opportunities for your students – you help them with their work, you write them letters, you make calls and introductions on their behalf to help them land jobs. Granted, it can be a lot of work and you are staking your reputation on their work ethic and smarts. On the other hand, it is your job, and you get paid for it, and your reputation also grows with theirs.

But I’m getting a bit lost with the them-screwing-you part. If they simply aren’t very good at the jobs you help them get, then I don’t think that can be considered screwing you. It’s hard for me to imagine exactly what that could mean beyond that. Is the student spreading nasty rumors about your work? Are there internal politics in your field and your student isn’t in your camp? Has the student stolen your ideas?

Or is it something totally normal, like the student doesn’t express sufficient gratitude for your help? In this case I’d say, welcome to young people. Being an advisor is a lot like being a parent, and in this society we don’t get lots of gratitude as parents. Move to China if you want that stuff.

Or maybe I missed it altogether, which is why you’d need to say more when you write back.

Aunt Pytha

——

Please submit your well-specified, fun-loving, cleverly-abbreviated question to Aunt Pythia!

Categories: Aunt Pythia

Sometimes, The World Is Telling You To Polish Up Your LinkedIn Profile

September 27, 2013 9 comments

The above title was stolen verbatim from an excellent essay by Dan Milstein on the Hut 8 Labs blog (hat tip Deane Yang). The actual title of the essay is “No Deadlines For You! Software Dev Without Estimates, Specs or Other Lies”

He wrote the essay about how, as an engineer, you can both make yourself invaluable to your company and avoid meaningless and arbitrary deadlines on your projects. So, he’s an engineer, but the advice he gives is surprisingly close to the advice I was trying to give on Monday night when I spoke at the Columbia Data Science Society (my slides are here, by the way). More on that below.

Milstein is an engaging writer. He wrote a book called Coding, Fast and Slow, which I now feel like reading just because I enjoy his insights and style. Here’s a small excerpt:

Let’s say you’ve started at a new job, leading a small team of engineers. On your first day, an Important Person comes by your desk. After some welcome-to-the-business chit chat, he/she hands you a spec. You look it over—it describes a new report to add to the company’s product. Of course, like all specs, it’s pretty vague, and, worse, it uses some jargon you’ve heard around the office, but haven’t quite figured out yet.

You look up from the spec to discover that the Important Person is staring at you expectantly: “So, <Your Name>, do you think you and your team can get that done in 3 months?”

What do you do?

Here are some possible approaches (all of which I’ve tried… and none of which has ever worked out well):

  • Immediately try to flesh out the spec in more detail

“How are we summing up this number? Is this piece of data required? What does <jargon word> mean, here, exactly?”

  • Stall, and take the spec to your new team

“Hmm. Hmm. Hmmmmmmmm. Do you think, um, Bob (that’s his name, right?) has the best handle on these kinds of things?”

  • Give the spec a quick skim, and then listen to the seductive voice of System I

“Sure, yeah, 3 months sounds reasonable” (OMG, I wish this wasn’t something I’ve doneSO MANY TIMES).

  • Push back aggressively

“I read this incredibly convincing blog post 1 about how it’s impossible to commit to deadlines for software projects, sorry, I just can’t do that.”

He then goes on to write that very blog post. In it, he explains what you should do, which is to learn why the project has been planned in the first place, and what the actual business question is, so you have full context for your job and you know what it means to the company for this to succeed or fail.

The way I say this, regularly, to aspiring data scientists I run into, is that you are often given a data science question that’s been filtered from a business question, through a secondary person who has some idea that they’ve molded that business question into a “mathematical question,” and they want you to do the work of answering that question, under some time constraint and resource constraints that they’ve also picked out of the air.

But often that process has perverted the original aims – often because proxies have magically appeared in the place of the original objects of interest – and it behooves a data scientist who doesn’t want to be working on the wrong problem to go to the original source and verify that their work is answering a vital business question, that they’re optimizing for the right thing, and that they understand the actual constraints (like deadlines but also resources) rather than the artificial constraints made up by whoever is in charge of telling the nerds what to do.

In other words, I suggest that each data scientist “becomes part business person,” and talks to the business owner of the given problem directly until they’re sure they know what needs to get done with data.

Milstein has a bunch of great tips on how to go through with this process, including:

  1. Counting on people’s enjoyment of hearing their own ideas repeated and fears understood,
  2. Using a specific template when talking to Important People, namely a) “I’m going to echo that back, make sure I understand”, b) echo it back, c) “Do I have that right?”.
  3. To always think and discuss your work in terms of risks and information for the business. Things like, you need this information to answer this risk. The point here is it always stays relevant to the business people while you do your technical thing. This means always keeping a finger on the pulse of the business problem.
  4. Framing choices for the Important Person in terms of clear trade-offs of risk, investments, and completion. This engages the business in what your process is in a completely understandable way.
  5. Finally, if your manager doesn’t let you talk directly to the Important People in the business, and you can’t convince your manager to change his or her mind, then you might wanna polish up your LinkedIn profile, because otherwise you are fated to work on failed projects. Great advice.
Categories: data science

A Code of Conduct for data scientists from the Bellagio Fellows

September 25, 2013 3 comments

The 2013 PopTech & Rockefeller Foundation Bellagio Fellows – Kate Crawford, Patrick Meier, Claudia Perlich, Amy Luers, Gustavo Faleiros and Jer Thorp – yesterday published “Seven Principles for Big Data and Resilience Projects” on Patrick Meier’s blog iRevolution.

Although they claim that these principles are meant for “best practices for resilience building projects that leverage Big Data and Advanced Computing,” I think they’re more general than that (although I’m not sure exactly what a resilience building project is) I and I really like them. They are looking for public comments too. Go to the post for the full description of each, but here is a summary:

1. Open Source Data Tools

Wherever possible, data analytics and manipulation tools should be open source, architecture independent and broadly prevalent (R, python, etc.).

2. Transparent Data Infrastructure

Infrastructure for data collection and storage should operate based on transparent standards to maximize the number of users that can interact with the infrastructure.

3. Develop and Maintain Local Skills

Make “Data Literacy” more widespread. Leverage local data labor and build on existing skills.

4. Local Data Ownership

Use Creative Commons and licenses that state that data is not to be used for commercial purposes.

5. Ethical Data Sharing

Adopt existing data sharing protocols like the ICRC’s (2013). Permission for sharing is essential. How the data will be used should be clearly articulated. An opt in approach should be the preference wherever possible, and the ability for individuals to remove themselves from a data set after it has been collected must always be an option.

6. Right Not To Be Sensed

Local communities have a right not to be sensed. Large scale city sensing projects must have a clear framework for how people are able to be involved or choose not to participate.

7. Learning from Mistakes

Big Data and Resilience projects need to be open to face, report, and discuss failures.

Categories: data science, modeling, news, open source tools

Are you cliterate?

September 24, 2013 1 comment

Not much time this morning for blogging, but I wanted everyone to get a chance to read this amazing Huffington Post article about learning more than you ever thought possible about the female sexual organ, and then celebrating that knowledge in style.

The article is actually more inspiring than you’d think, and I found myself weeping with joy at times. I’m an easy cry, but still.

Plus, any article that has this picture is worth reading:

SOLID-GOLD-CLIT

Categories: musing, news

Upcoming talks

September 23, 2013 4 comments

Tomorrow evening I’m meeting with the Columbia Data Science Society and talking to them – who as I understand it are mostly engineers – about “how to think like a data scientist”.

On October 11th I’ll be in D.C. sitting on a panel discussion organized by the Americans for Financial Reform. It’s part of a day-long event on the topic of transparency in financial regulation. The official announcement isn’t out yet but I’ll post it here as soon as I can. I’ll be giving my two cents on what mathematical tools can do and cannot do with respect to this stuff.

On October 16th I’ll again be in D.C. giving a talk in the MAA Distinguished Lecture Series to a mostly high school math teacher audience. My talk is entitled, “Start Your Own Netflix”.

Finally, I’m going to Harvard on October 30th to give a talk in their Applied Statistics Workshop series. I haven’t figured out exactly what I’m talking about but it will be something nerdy and skeptical.

Categories: data science, finance