Archive for July, 2011

What kind of math nerd job should you have?

Say you’re a math nerd, finishing your Ph.D. or a post-doc, and you’re wondering whether academics is really the place for you. Well I’ve got some advice for you! Actually I will have some advice for you, after you’ve answered a few questions. It’s all about fit. Since I know them best, I will center my questions and my advice around academic math vs. hedge fund quant vs. data scientist at a startup.

By the way, this is the advice I find myself telling people when they ask. It’s supposed to be taken over a beer and with lots of tongue in cheek.

1) What are your vices?

It turns out that the vices of the three jobs we are considering are practically disjoint! If you care about a good fit for your vices, then please pay attention.

NOTE: I am not saying that everyone in these fields has all of these vices! Far from it! It’s more like, if one or more of these vices drives you nuts, then you may get frustrated when you encounter them in these fields.

In academics, the major vices are laziness, envy, and arrogance. It’s perhaps true that laziness (at least outside of research) is typically not rewarded until after tenure, but at that point it’s pretty much expected, unless you want to be the fool who spends all of his(her) time writing recommendation letters and actually advising undergraduates. Envy is, of course, a huge deal in academics, because the only actual feedback is in the form of adulating rumor. Finally, arrogance in academics is kind of too obvious to explain.

At a hedge fund, the major vices are greed, covetousness, and arrogance. The number one source of feedback is pay, after all, so it’s all about how much you got (and how much your officemate got). Plus the isolation even inside your own office can lead to the feeling that you know more and more interesting, valuable, things than anyone else, thus the arrogance.

Finally, at a startup, the major vices are vanity, impatience, and arrogance. People really care about their image- maybe because they are ready to jump ship and land a better job as soon as they start to smell something bad. Plus it’s pretty easy in startups as well to live inside a bubble of self-importance and coolness and buzz. Thus the arrogance. On the flip side of vanity, startups are definitely the sexiest of the three, and the best source by far for good karaoke singers.

Okay it turns out they all have arrogance. Maybe that’s just a property of any job category.

2) What do you care about?

Do you care about titles? Don’t work at a startup.

Do you care about stability? Don’t work at a startup. Actually you might think I’d say don’t work at a hedge fund either, but I’ve found that hedge funds are surprisingly stable, and are full of people who are surprisingly risk averse. Maybe small hedge funds are unstable.

Do you care about feedback? Don’t work in academics.

Do you care about publishing? Don’t work outside academics (it’s sometimes possible to publish outside of academics but it’s not always possible and it’s not always easy).

Do you care about making lots of money? Don’t work in academics. In a startup you make a medium amount of money but there are stock options which may pan out someday, so it’s kind of in between academics and Wall St.

Do you care about being able to decide what you’re working on? Definitely stay in academics.

Do you care about making the world a better place? I’m still working on that one. There really should be a way of doing that if you’re a math nerd. It’s probably not Wall Street.

3) What do you not care about?

If you just like math, and don’t care exactly what kind of math you’re doing, then any of these choices can be really interesting and challenging.

If you don’t mind super competitive and quasi-ethical atmospheres, then you may really enjoy hedge fund quant work- the modeling is really interesting, the pay is good, and you are part of the world of finance and economics, which leaks into politics as well and is absolutely fascinating.

If you don’t mind getting nearly no vacation days and yet feeling like your job may blow up any minute, you may like working at a startup. The people there are real risk lovers, care about their quality of life (at least at the office!), and know how to throw a great party.

If you don’t mind being relatively isolated mathematically, and have enormous internal motivation and drive, then academics is a pretty awesome job, and teaching is really fun and rewarding. Also academic jobs have lots of flexibility as well as cool things like sabbaticals.

4) What about for women who want kids?

Let’s face it, the tenure clock couldn’t have been set up worse for women who want children. And startups have terrible vacation policies and child-care policies as well; it’s just the nature of living on a Venture Capitalist’s shoestring. So actually I’d say the best place to balance work and life issues is at an established hedge fund or bank, where the maternity policies are good; this is assuming though that your personality otherwise fits well with a Wall St. job. Actually many of the women I’ve met who have left academics for government research jobs (like at NASA or the NSA) are very happy as well.

Three strikes against the mortgage industry

There’s a great example here of mortgage lenders lying through their teeth with statistics. Felix Salmon uncovers a ridiculous attempt to make loans look safe by cutting up the pile of mortgages in a tricky way- sound familiar at all?

And there’s a great article here about why they are lying. Namely, there is proposed legislation that would require the banks to keep 5% of the packaged mortgages on their books.

And finally here’s a great description of why they should know better. A breakdown of what banks are currently doing to avoid marking down their mortgage book.

Categories: finance, news, rant

Historical volatility on the S&P index

In a previous post I described the way people in finance often compute historical volatility, in order to try to anticipate future moves in a single stock. I’d like to give a couple of big caveats to this method as well as a worked example, namely on daily returns of the S&P index, with the accompanying python code. I will use these results in a future post I’m planning about errorbars and how people abuse and misuse them.

Two important characteristics of returns

First, market returns in general have fat-tailed distributions; things can seem “quiet” for long stretches of time (longer than any lookback window), during which the sample volatility is a possibly severe underestimate of the “true” standard of deviation of the underlying distribution (if that even makes sense – for the sake of this discussion let’s assume it does). Then when a fat-tailed event occurs, the sample volatility typically spikes to being an overestimate of the standard of deviation for that distribution.

Second, in the markets, there is clustering of volatility- another way of saying this is that volatility itself is rather auto-correlated, so even if we can’t predict the direction of the return, we can still estimate the size of the return. This is particularly true right after a shock, and there are time series models like ARCH and its cousins that model this phenomenon; they in fact allow you to model an overall auto-correlated volatility, which can be thought of as scaling for returns, and allows you to then approximate the normalized returns (returns divided by current volatility) as independent, although still not normal (because they are still fat-tailed even after removing the clustered volatility effect). See below for examples of normalized daily S&P returns with various decays.

Example: S&P daily returns

I got this data from Yahoo Finance, where they let you download daily S&P closes since 1950 to an excel spreadsheet. I could have used some other instrument class, but the below results would be stronger (especially for things like credit default swamps), not weaker- the S&P, being an index, is already the sum of a bunch of things and tends to be more normal as a result; in other words, the Central Limit Theorem is already taking effect on an intraday basis.

First let’s take a look at the last 3 years of closes, so starting in the summer of 2008:

Next we can look at the log returns for the past 3 years:

Now let’s look at how the historical volatility works out with different decays (decays are numbers less than 1 which you use to downweight old data: see this post for an explanation):

For each choice of the above decays, we can normalize the log returns. to try to remove the “volatility clustering”:

As we see, the long decay doesn’t do a very good job. In fact, here are the histograms, which are far from normal:

Here’s the python code I used to generate these plots from the data (see also R code below):

#!/usr/bin/env python

import csv
from matplotlib.pylab import *
from numpy import *
from math import *
import os

dataReader = csv.DictReader(open(‘SandP_data.txt’, ‘rU’), delimiter=’,’, quotechar=’|’)

close_list = []
for row in dataReader:
#print row[“Date”], row[“Close”]
close_array = array(close_list)
close_log_array = array([log(x) for x in close_list])
log_rets = array(diff(close_log_array))
perc_rets = array([exp(x)-1 for x in log_rets])

plot(close_array[-780:-1], label = “raw closes”)
title(“S&P closes for the last 3 years”)
#plot(log_rets, label = “log returns”)
#hist(log_rets, 100, label = “log returns”)
#hist(perc_rets, 100, label = “percentage returns”)

def get_vol(d):
var = 0.0
lam = 0.0
var_list = []
for r in log_rets:
lam = lam*(1.0-1.0/d) + 1
var = (1-1.0/lam)*var + (1.0/lam)*r**2
return [sqrt(x) for x in var_list]

for d in [10, 30, 100]:
plot(get_vol(d)[-780:-1], label = “decay factor %.2f” %(1-1.0/d))
title(“Volatility in the S&P in the past 3 years with different decay factors”)
for d in [10, 30, 100]:
these_vols = get_vol(d)
plot([log_rets[i]/these_vols[i-1] for i in range(len(log_rets) – 780, len(log_rets)-1)], label = “decay %.2f” %(1-1.0/d))
title(“Volatility normalized log returns (last three years)”)
plot([log_rets[i] for i in range(len(log_rets) – 780, len(log_rets)-1)], label = “raw log returns”)
title(“Raw log returns (last three years)”)
for d in [10, 30, 100]:
these_vols = get_vol(d)
normed_rets = [log_rets[i]/these_vols[i-1] for i in range(len(log_rets) – 780, len(log_rets)-1)]
hist(normed_rets, 100,label = “decay %.2f” %(1-1.0/d))
title(“Histogram of volatility normalized log returns (last three years)”)

Here’s the R code Daniel Krasner kindly wrote for the same plots:


dataReader <- read.csv(“SandP_data.txt”, header=T)

close_list <- as.numeric(dataReader$Close)

close_list <- rev(close_list)

close_log_list <- log(close_list)

log_rets <- diff(close_log_list)

perc_rets = exp(log_rets)-1


plot(close_list[(length(close_list)-779):(length(close_list))], type=’l’, main=”S&P closes for the last 3 years”, col=’blue’)

legend(125, 1300, “raw closes”, cex=0.8, col=”blue”, lty=1)

get_vol <- function(d){

var = 0


var_list <- c()

for (r in log_rets){

lam <- lam*(1 – 1/d) + 1

var = (1 – 1/lam)*var + (1/lam)*r^2

var_list <- c(var_list, var)


return (sqrt(var_list))


L <- (length(close_list))


plot(get_vol(10)[(L-779):L], type=’l’, main=”Volatility in the S&P in the past 3 years with different decay factors”, col=1)

lines(get_vol(30)[(L-779):L],  col=2)

lines(get_vol(100)[(L-779):L],  col=3)

legend(550, 0.05, c(“decay factor .90”, “decay factor .97″,”decay factor .99”) , cex=0.8, col=c(1,2,3), lty = 1:3)



plot((log_rets[2:L]/get_vol(10))[(L-779):L], type=’l’,  col=1, lty=1, ylab=”)

legend(620, 3, “decay factor .90”, cex=0.6, col=1, lty = 1)

plot((log_rets[2:L]/get_vol(30))[(L-779):L], type=’l’, col=2, lty =2, ylab=”)

legend(620, 3, “decay factor .97”, cex=0.6, col=2, lty = 2)

plot((log_rets[2:L]/get_vol(100))[(L-779):L], type=’l’, col=3, lty =3, ylab=”)

legend(620, 3, “decay factor .99”, cex=0.6, col=3, lty = 3)


plot(log_rets[(L-779):L], type=’l’, main = “raw log returns”, col=”blue”, ylab=”)


hist((log_rets[2:L]/get_vol(10))[(L-779):L],  breaks=200, col=1, lty=1, ylab=”, xlab=”, main=”)

legend(2, 15, “decay factor .90”, cex=.8, col=1, lty = 1)

hist((log_rets[2:L]/get_vol(30))[(L-779):L],  breaks=200, col=2, lty =2, ylab=”,  xlab=”, main=”)

legend(2, 40, “decay factor .97”, cex=0.8, col=2, lty = 2)

hist((log_rets[2:L]/get_vol(100))[(L-779):L],  breaks=200,  col=3, lty =3, ylab=”,  xlab=”, main=”)

legend(3, 50, “decay factor .99”, cex=0.8, col=3, lty = 3)

Is too big to fail a good thing?

I read this blog post a couple of days and it really got me thinking. This guy John Hempton from Australia is advocating the too big to fail model- in fact he things we should merge more big banks together (Citigroup and Wells Fargo) because we haven’t gone far enough!

His overall thesis is that competition in finance increases as a function of how many banks there are out there and is a bad thing for stockholders and for society, because it makes people desperate for profit, and in particular people increase their risk profiles in pursuit of profit and they blow up:

What I am advocating is – that as a matter of policy – you should deliberately give up competition in financial services – and that you should do this by hide-bound regulation and by deliberately inducing financial service firms to merge to create stronger, larger and (most importantly) more anti-competitive entities.

He acknowledges that the remaining banks will be hugely profitable, and perhaps also extremely lazy, but claims this is a good thing: we would, as a culture, essentially be paying a fee for stability. It’s something we do all the time in some sense, when we buy insurance. Insurance is a fee we pay so that disruptions and small disasters in our lives don’t completely wipe us out. So perhaps, as a culture, this would be a price worth paying?

The biggest evidence he has that this setup works well is that it works in Australia- they have four huge incompetent yet profitable banks there, and they don’t blow up. People who work there are sitting pretty, I guess, because they really are just living in a money press. There is no financial innovation because there’s no competition.

I guess I have a few different reactions to this scenario. First, it’s kind of an interesting twist on the too-big-to-fail debate, in that it’s combined with the idea I already talked about here of having a system of banks that are utilities. John is saying that, really, we don’t need to make that official, that as soon as banks are this huge, we are already done, they are essentially going to act like utilities. This is super interesting to me, but I’m not convinced it’s a necessary or even natural result of huge banks.

Second, I don’t buy that what happened in Australia will happen here- perhaps Australia squelched financial innovation through regulations and the existing boring system, but maybe the people who would have been financial innovators all just moved to the U.S. and became innovators here (there are plenty of examples of that!). In other words Australia may have made it just a bit too difficult to be competitive relative to what else is out there- if everyone tried to be that repressive to financial innovation, we may see people moving back into Australia’s financial waters (like sharks).

Third, I think what John is talking about is an example of a general phenomenon, namely that, in the limit as regulations go to infinity, there is only one bank left standing. This is because every additional regulation requires a lawyer to go over the requirements and a compliance person to make sure the rules are being followed continuously. So the more regulation, the more it behooves banks to merge so that they can share those lawyers and compliance officers to save costs. In the end the regulations have defined the environment to such an extent that there’s only one bank that can possibly follow all the rules, and knows how to because of historical reasons. And that one, last bank may as well be a government institution, albeit with better pay, especially for its managers.

But we don’t have that kind of regulatory environment, and hedge funds are alive and well. They have to follow some rules, it’s absolutely true, but it’s still possible to start a smallish hedge fund without a million lawyers.

I guess what I’m concluding is that if we had formed our very few, very huge banks because of a stifling regulatory environment, then maybe we would have an environment that is sufficiently anti-competitive to think that our banks would serve us as slightly overpaid utilities. However, that’s not why we have them – it was because of the credit crisis, and the rules and regulations haven’t changed that much since then.

At the same time, I don’t totally disagree that huge banks do become anti-competitive, just by dint of how long it takes them to make decisions and do things. But I’m not sure anti-competitive is the same thing as low-risk.

Categories: finance, hedge funds, rant

Elizabeth Warren: Moses and the Promised Land

July 28, 2011 Comments off

This is a guest post by FogOfWar

In Biblical style, Elizabeth Warren (EW) was not nominated to head the CFPB (Consumer Financial Protection Bureau).  Having spearheaded the movement to create the institution, pushed to make it part of the otherwise-generally-useless* Dodd Frank “Financial Reform” Bill, and spent the better part of the last two years staffing the actual CFPB and moving it into gear, she has now been deemed too controversial by what passes for a President these days.

One of my favorite EW quotes: “My first choice is a strong consumer agency.  My second choice is no agency at all and plenty of blood and teeth left on the floor.”  This still remains to be seen, as opposition to the CPFB (and filibuster threats to any appointment to head the Bureau) remains in the face of nominee Richard Cordray.  In fact, if one were inclined to be an Obama apologist (I gave up apologizing for Obama right about here), one might view the Warren-Cordray switch as a potentially brilliant tactical maneuver, with the emphasis on “potentially”.  If the opposition to the CPFB took its persona in EW, then sidestepping her personally to get the agency up and running would be worthwhile, particularly as Cordray seems at least as assertively pro-consumer as EW (a bank lobbyist described him as “Elizabeth Warren without the charm”).

Barney Frank believes gender bias played a role.  Maybe yes, maybe no and the Cordray confirmation will give some evidence to that question.  I suspect the Republican opposition isn’t stupid and knows that Cordray will run a good agency.  If that’s right then passing over EW doesn’t really serve any purpose.

Hard to tell what a public figure is really like, but my sense is EW doesn’t have any ego attached to running the agency personally.  And what she does next is really up to her, I mean who really cares what we think she should do?

Wait—this is a blog!  Our Raison d’être is practically making suggestions that no one will listen to, so let’s go…

1.     Run for Congress

The biggest idea floated around.  Yves Smith thinks it’s a terrible idea. I’m not entirely convinced—there are many ways to make a difference in this world, and being one minority member of a large and powerful body, and thus moving the body incrementally in the right direction can be a very good thing.

Two questions though: can she win (a few early stage polls seemed to indicate no, but do early stage polls really have much predictive value on final election results?  Cathy?  Fivethirtyeight?), and on which party platform would she run (I vote for running as an Independent)?  Any thoughts from the ground from our MA-registered voters?

2.     The “Al Gore” option

EW could continue to advocate, lecture and write outside of political office.  She’s good television and would be free to speak without the gag order of elected office.  Definitely something to be said for this option.  Just realized pulling links for this post that EW was the person from the movie “Maxed Out”.  Part of me thinks “damn that was effective and she should do more of that because it was so effective” and part of me thinks “wait, that movie came out in 2006 and no one listened and no one will listen”, and then the other part goes “but it can happen—you’ve actually seen social perceptions change in the wake of Al Gore (and yes, lots and lots of other people, but sparks do matter) with real and deep impacts.”

3.     The “Colin Powell” option

Y’now, being in the public light kinda sucks ass.  Colin Powell passed up a run for President, and largely retired to private life, and doesn’t seem to have any complaints about it.  One legitimate option is to say “I did my part, you guys fight the good fight & I’m going to hang out with my grandkids on the beach.”

Any other suggestions?

*-Paul Volker deserves a parallel post of equal length for pushing the Volker Rule through this legislation and similarly receiving the thanks of being sidelined by the TBTF bank-capital-must-increase-even-if-the-peasants-have-to-eat-cake crowd.

Categories: finance, FogOfWar, news, rant

Quit your job and become a data miner!?

Today my friend sent me this link, which is a pretty interesting and inspiring video of a talk from a guy from Google named Steve Yegge talking at an O’Reilly conference about how he’s sick of working on uninspiring projects involving social media and cat pictures, and wants to devote himself (and wants you to devote yourself) to more important questions about the nature of human existence. And he things the way to go about this is to become a data miner. I dig it! Of course he’s preaching to the choir at that conference. I wonder what other people will make of his appeal. Can one nerd change an entire culture of endless cat pic collections?

And lest you think that data mining is the answer to everything, here’s an article about how much data mining (in the form of “Value-added modeling”) can screw up other peoples’ lives when it’s misdirected. It’s written by John Ewing, who is the fabulous president of MfA, an organization that trains and mentors excellent college math majors to become effective math teachers in the New York Public School system and beyond- the “beyond” part is partly due to the crazy state of the budgets for new teachers here in NYC- we now have access to these wonderful MfA graduates but have hiring freezes so we can’t hire them. Also, my good friend Japheth Wood, a.k.a. the Math Wizard, is one of the MfA mentors.

I’m planning to post more soon on how crappy the value-added modeling (VAM) system is and how’s it’s a perfect example of mathematics being used to make things seem magical and therefore inaccessible, the exact opposite of what should be going on.

Categories: math education, news

The Bad Food Tax

There’s an interesting op-ed article in today’s New York Times. The author, Mark Bittman, is proposing that we tax bad foods to the point where people will naturally select healthy food because they will be subsidized and cheap.

He has lots of statistics to back him up, and if you’re someone like me who reads this kind of thing widely, nothing surprised me. Of course Americans eat crappy food and it’s terrible for our bodies. We know that, it’s old news.

And we all want to know how to fix this- clearly education about nutrition isn’t doing the trick by itself. And I’m the first person who would love to use quantitative methods to solve a really important, big problem. Moreover, if we start to get rid of the evil farm subsidies that are currently creating a ridiculous market for corn sugar (a major reason we have some much soda on the shelves at such low prices to begin with) as well as screwing up the farmers in Africa and other places, that will be a good thing.

Unfortunately, I really think his tax plan stinks. The main problem is something he actually brings up and dismisses- namely:

Some advocates for the poor say taxes like these are unfair because low-income people pay a higher percentage of their income for food and would find it more difficult to buy soda or junk. But since poor people suffer disproportionately from the cost of high-quality, fresh foods, subsidizing those foods would be particularly beneficial to them.

Yes they would, if they could actually buy them in their neighborhood! If he has the idea that the reason poor people buy crappy food is because they go into their neighborhood grocery store with a museum-like display of fresh fruits and vegetables, bypass those foods (because they are too expensive) to go straight to the back and find junk, then I guess his plan would make sense. Unfortunately the truth is, there is no fresh fruit at most of the food stores in poor urban areas – they are typically small and carry long-lasting packaged goods and groceries, from canned evaporated milk to diapers, and don’t have extra space. Moreover, I don’t think a pure price comparison is going to convince them to carry fruit, because it’s not just the higher prices that makes bodegas carry no fruit- it’s also the convenience of packages that don’t go bad. In fact it’s an entirely different business model, which is unfortunately a pretty tough nut to crack, but is essential in this discussion.

In other words, the result of this tax plan would be, for poor people, even higher prices for crappy food, not access to fresh cheap food. Unless the plan has worked out a system for how to get fresh fruit into poor areas, it really is missing the very audience it wishes to target.

Categories: news, rant